Restart Sentry and upgrade ExceptionLess version, Docker troubleshooting notes

Preface

Ever since Sentry went down again last year, and given that most of our projects are in C# and ExceptionLess has been working pretty well for us, we haven't used Sentry since 🦎

Just the day before yesterday, I finished configuring Gitlab. Now, there are some projects in other languages ☕️ that need monitoring, so I thought I’d bring Sentry back online as well. Meanwhile, ExceptionLess hasn’t been updated for a long time either—the official version is already at 8.x, while we’re still on 7.x. So, I decided to upgrade it this time as well ⛰️

Who would’ve thought this whole process would take an entire day 🥱, leaving me totally drained 🐎

sentry

About Sentry 📄

In the world of software development, error tracking and performance monitoring are key to ensuring application health and user satisfaction. This is where Sentry comes into play. Sentry is an open-source, real-time error tracking tool that effectively helps developers monitor and fix errors across various programming languages. Whether it's a frontend JavaScript error or a backend Python exception, Sentry can capture it and provide detailed error reports and analysis.

Core Features

• Real-time Monitoring: Sentry provides real-time error monitoring. As soon as an error occurs in the application, it is immediately captured and the developer is notified.
• Cross-Platform Support: Supports a wide range of programming languages and frameworks, including JavaScript, Python, Java, Ruby, PHP, and more.
• Detailed Error Reports: Each error report contains sufficient information to help developers quickly pinpoint the source of the problem. This information includes the stack trace where the error occurred, the affected users, the time of the error, and the application state before and after the error.
• Performance Monitoring: It can not only track errors but also monitor the performance of the application, such as page load times, API response times, etc., helping developers optimize the user experience.
• Issue Analysis and Resolution: Provides trend analysis of issues, assessment of their impact, and solution suggestions, making issue resolution more efficient.

Use Cases

Sentry has a wide range of use cases. It is not only suitable for complex applications in large enterprises but also for small development teams and individual projects. For example, integrating Sentry during development allows for continuous monitoring of the application's state. If an issue arises after deploying a new version, Sentry can help quickly roll back and fix it. Additionally, its user-friendly interface and powerful configuration capabilities make it a valuable tool for handling errors and improving application quality.

self hosted

First, deploying Sentry isn't actually that difficult.

The reason it took so long to get things working was mainly due to network issues, followed by version problems. The official documentation uses 24.1.0, but I initially tried using the latest 24.7.0 from GitHub, which didn't work out at all 🙅‍♀️.

In the end, I solved it through Cloudflare Worker. You can refer to my previous article: Using Cloudflare Worker to Accelerate Docker Image.

---

The official documentation is very detailed: https://develop.sentry.dev/self-hosted/

Just follow the official steps and you're good to go.

After solving the network issues, there were basically no more obstacles 😴.

Installing Sentry⏬

VERSION="24.1.0"
git clone https://github.com/getsentry/self-hosted.git
cd self-hosted
git checkout ${VERSION}
sudo ./install.sh

If you want a more verbose installation process, you can add the DEBUG=1 environment variable and also disable telemetry📴

sudo DEBUG=1 ./install.sh --no-report-self-hosted-issues

Problem Log📝

The installation was truly a series of twists and turns 😭. Here I'm recording some of the issues encountered.

• Configured the .env.custom file, which caused the upgrade-clickhouse.sh script to fail. Initially, I hadn't used Cloudflare Worker to accelerate Docker, so I planned to use a proxy to pull images. The official documentation says to write the proxy settings in /etc/environment, but I didn't want to do that. So I put the proxy configuration in .env.custom, which resulted in continuous errors.
• Then there was the Docker Compose version issue. The local Compose version was lower than the required one. Upgrading Compose also presented some small problems. My Compose executable was located at ~/.docker/cli-plugins/docker-compose, appearing as a Docker plugin, but it wasn't recognized, so I couldn't use the docker plugins command to update it. In the end, I directly downloaded the executable from GitHub to replace it. The real solution is often this down-to-earth.
• The remaining issues were network 🌍 problems; several images couldn't be pulled.

Main Components ⚙️

Sentry's backend is built on Django, leveraging Python's flexibility and rich ecosystem of libraries, making it an ideal choice for developing complex applications like Sentry. The frontend pages are developed with React.

The entire Sentry system is quite complex, and the current version runs 28 containers, requiring a certain level of server performance.

The main components include:

• ClickHouse: A high-performance columnar database (列式数据库) designed for large-scale data analysis. Sentry uses it to store and query event data.
• Redis: An in-memory data structure store used as a database, cache, and message broker. Sentry uses it to handle queues and cache tasks.
• PostgreSQL: A powerful open-source relational database (关系数据库). Sentry uses it to store core data, including project configurations and user data.
• Kafka: A distributed stream processing platform (流处理平台). Sentry uses Kafka to process event streams and ensure resilient data handling.
• Zookeeper: Primarily used for service coordination and configuration management, working with Kafka in Sentry to maintain Kafka's service state.
• Snuba: One of Sentry's service components, written in Python, mainly responsible for querying data from ClickHouse.
• Symbolicator: A service for handling native stack traces, helping to parse and symbolize unprocessed stack traces.
• Relay: A standalone service used to handle events sent from clients to Sentry, providing data sanitization and security controls.

Introduction to Each Container 📂

These containers together form Sentry's complex architecture, ensuring that every stage from data collection to processing to storage operates efficiently and stably.

It can be seen that Sentry is a very resource-intensive monitoring service.

1. attachments-consumer: Processes and stores attachment data in error reports.
2. billing-metrics-consumer: Manages and processes billing-related metric data.
3. cron: Handles scheduled tasks, such as cleanup jobs and triggering periodic tasks.
4. events-consumer: Receives and processes event data, ensuring it is properly stored and indexed.
5. generic-metrics-consumer: Processes generic metric data for monitoring and analysis.
6. ingest-monitors: Processes and monitors events and statuses during the data ingestion process.
7. ingest-occurrences: Handles the data ingestion of event occurrences.
8. ingest-profiles: Manages the ingestion of user performance profiling data.
9. ingest-replay-recordings: Processes the data ingestion of replay recordings for error reproduction.
10. memcached: Provides in-memory caching services, enhancing data read/write speed and efficiency.
11. metrics-consumer: Processes and aggregates Application Performance Monitoring (APM) metric data.
12. nginx: Acts as a reverse proxy server, handling HTTP requests and load balancing.
13. post-process-forwarder-errors: Handles post-processing and forwarding of error events.
14. post-process-forwarder-issue-platform: Handles post-processing and forwarding of issue platform events.
15. post-process-forwarder-transactions: Handles post-processing and forwarding of transaction events.
16. postgres: Provides relational database services for Sentry, storing configuration and user data.
17. redis: Provides queue management and caching services, supporting event processing and data caching.
18. relay: Receives events from clients, performs cleansing and security checks, then forwards them to Sentry.
19. smtp: Handles sending emails for notifications and reports.
20. snuba-api: Provides API interfaces to access data stored in ClickHouse.
21. snuba-replacer: Processes and replaces data in Snuba for data correction and deduplication.
22. subscription-consumer-events: Manages event data subscriptions and processes related notifications.
23. subscription-consumer-generic-metrics: Manages subscriptions and notifications for generic metric data.
24. subscription-consumer-metrics: Manages subscriptions and processing of metric data.
25. subscription-consumer-transactions: Manages subscriptions and processing of transaction data.
26. transactions-consumer: Processes and stores transaction data, supporting application performance monitoring.
27. vroom: Handles Sentry-specific internal requests and jobs.
28. zookeeper: Acts as a coordinator for distributed services, managing state information for Kafka and other services.

Does Sentry have alternatives?

🈶️

Throughout my tinkering, I once thought about giving up, but then I found two open-source alternatives that seem pretty promising:

• https://glitchtip.com/
• https://www.highlight.io/

I haven't tried them yet, but I'll get around to deploying and testing them when I have time. Although Sentry works now, it is still too resource-heavy.

ExceptionLess

🌟 Meet Exceptionless: Your Code's Little Guardian Fairy! 🌟

Hey, dear developer friends! Have you ever thought that if there were a little guardian fairy in your code, it could help you catch all those mischievous errors and exceptions, then secretly tell you where things went wrong, making your debugging work easy and pleasant? Great news, this little fairy already exists, and its name is Exceptionless! 🎉

🚀 What is Exceptionless?

Exceptionless is more than just a tool; it's a super cute, powerful open-source error and event reporting service. Think of it as a faithful companion on your programming journey. Once integrated into your application, it automatically captures unhandled exceptions and events. Imagine whenever your code hits a minor hiccup, Exceptionless catches them instantly and quietly places the information on your dashboard for you to review. 🐞

🌈 Why is it so special?

• Automated Monitoring: Just with a simple integration, Exceptionless will automatically start monitoring. You don't even need to tell it where to begin!
• Real-Time Feedback: Receive real-time error reports, allowing you to know instantly what's happening, like having a little fairy whispering in your ear.
• Deep Insights: It doesn't just tell you "something went wrong." Exceptionless also explains why it went wrong, where it happened, and even who encountered the error. 🕵️‍♂️

💖 The Developer's Best Friend

Is your log file driving you crazy? Let Exceptionless join your development life! It will bring you a whole new experience of error tracking with its adorable style. It not only helps you fix bugs but also optimizes your app to take the user experience to the next level!

So, let's welcome Exceptionless together and make it our little hero in the code world! With Exceptionless, let's turn those "Oops" moments into "Fixed it!" moments! 🌟

Upgrading Version

OK, as mentioned before, I'm upgrading from 7.x to 8.x.

The documentation for ExceptionLess is way worse than Sentry's—poorly written and unclear. Fortunately, I'm already proficient with this tool.

This is also simpler than Sentry, with just a few components.

• The ExceptionLess official documentation is fake documentation ✖
• The Github repository is the real documentation ✅

Just use this configuration: https://github.com/exceptionless/Exceptionless/blob/main/samples/docker-compose.yml

However, for better deployment, a few minor modifications are needed:

• Added container_name configuration to each container
• Added restart to ensure containers start automatically
• Disabled HTTPS for the app container—just let swag handle it
• Turned off all port mappings
• Changed all volume mappings to the current directory—remember to create these directories and assign permissions, e.g., chmod -R 1000:1000 esdata/
• Bridged with the swag network so the containers are not externally accessible, only reachable via swag's reverse proxy
• Configured email settings—see the previous ExceptionLess installation and configuration article for details

services:
  app:
    restart: always
    depends_on:
      - elasticsearch
      - redis
    image: exceptionless/app:latest
    container_name: exless_app
    environment:
      EX_AppMode: Production
      EX_ConnectionStrings__Cache: provider=redis
      EX_ConnectionStrings__Elasticsearch: server=http://elasticsearch:9200
      EX_ConnectionStrings__MessageBus: provider=redis
      #EX_ConnectionStrings__Metrics: provider=statsd;server=statsd;
      EX_ConnectionStrings__Queue: provider=redis
      EX_ConnectionStrings__Redis: server=redis,abortConnect=false
      EX_ConnectionStrings__Storage: provider=folder;path=/app/storage
      #ASPNETCORE_URLS: http://+;https://+
      ASPNETCORE_URLS: http://+:80
      #ASPNETCORE_HTTPS_PORT: 5001
      #ASPNETCORE_Kestrel__Certificates__Default__Password: password
      #ASPNETCORE_Kestrel__Certificates__Default__Path: /https/aspnetapp.pfx
      EX_RunJobsInProcess: 'false'
    volumes:
      - ./appdata:/app/storage
      - ./ssldata:/https
    networks:
      - default
      - swag

  jobs:
    restart: always
    depends_on:
      - app
    image: exceptionless/job:latest
    container_name: exless_jobs
    environment:
      EX_AppMode: Production
      EX_BaseURL: http://app:80
      EX_ConnectionStrings__Cache: provider=redis
      EX_ConnectionStrings__Elasticsearch: server=http://elasticsearch:9200
      EX_ConnectionStrings__Email: smtps://邮箱(需转义):密码@smtp.exmail.qq.com:465
      EX_SmtpFrom: 邮箱地址
      EX_ConnectionStrings__MessageBus: provider=redis
      #EX_ConnectionStrings__Metrics: provider=statsd;server=statsd;
      EX_ConnectionStrings__Queue: provider=redis
      EX_ConnectionStrings__Redis: server=redis,abortConnect=false
      EX_ConnectionStrings__Storage: provider=folder;path=/app/storage
    volumes:
      - ./appdata:/app/storage
    networks:
      - default

  elasticsearch:
    image: exceptionless/elasticsearch:8.14.1
    restart: always
    container_name: exless_es
    environment:
      discovery.type: single-node
      xpack.security.enabled: "false"
      ES_JAVA_OPTS: -Xms1g -Xmx2g
    expose:
      - 9200
      - 9300
    volumes:
      - ./esdata:/usr/share/elasticsearch/data
    networks:
      - default

  kibana:
    restart: always
    depends_on:
      - elasticsearch
    image: docker.elastic.co/kibana/kibana:8.14.1
    container_name: exless_kibana
    expose:
      - 5601
    networks:
      - default

  redis:
    restart: always
    image: redis:7.2-alpine
    container_name: exless_redis
    expose:
      - 6379
    networks:
      - default

networks:
  default:
    name: exless
  swag:
    name: swag
    external: true

Troubleshooting Issues

• Elasticsearch Version Upgrade Issue: Previously running version 7.x, after the upgrade to 8.x, the data files between these two versions are incompatible and require migration. However, because the deployment of Sentry the day before had messed up the environment, this time ExceptionLess also couldn't run, and in the end, the entire Elasticsearch was simply deleted.
• Elasticsearch Health Check Failure Issue: Persistent error messages like [02:22:16 ERR] Health check Elasticsearch with status Unhealthy completed after 242.2644ms with message 'null' <Microsoft.Extensions.Diagnostics.HealthChecks.DefaultHealthCheckService>. This is a multifaceted issue, which will be discussed later.
• DNS Resolution Problem Inside Docker: This is strange—other projects don’t behave this way. The issue was later resolved inexplicably. It’s really puzzling; I’d guess it might be somewhat related to the proxy? 🥱

Elasticsearch Unhealthy Issue

This is the most maddening part.

It started as an issue of upgrading from version 7 to 8.

Then, I cleaned up the data and started from scratch. I was already able to start ES8 normally.

But the app and jobs containers still couldn't connect.

Then began the long troubleshooting phase, including using busybox and alpine containers to test the network in various ways.

The mystical issues are self-evident.

One moment, Docker DNS fails.

Next, all sorts of connection failures. Finally, I discovered a bunch of red-flagged elasticsearch:9200 access entries in the proxy server logs, only to find the problem was with the proxy.

But even after removing the proxy settings from the environment variables and ~/.docker/config.json, it still went through the proxy. I was really at a loss—this had become a mystical issue too.

In the end, I had to add a rule on the proxy server...

Docker Troubleshooting

It took a lot of time and effort to tinker.

Didn't learn much technical knowledge,

but my Docker troubleshooting experience did increase a little bit.

Check Container IP Address

Actually, using tools like portainer makes it easy to view, but using commands isn't difficult either.

exless-es is the container name, and this will be used as an example below.

docker inspect -f '{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}' exless-es

Using Temporary Containers to Test Network Connectivity

Nowadays, containers are generally built based on lightweight distributions like Alpine and may not have tools like curl to test the network.

No worries! We can create a temporary container and add it to this compose network to test. In this article, the network name is exless.

curl

The following command starts a curl container to test if the http://elasticsearch:9200 address is accessible.

docker run --rm --network=exless appropriate/curl http://elasticsearch:9200

busybox/alpine

docker run --rm -it --network=exless busybox

or 🔥

docker run --rm -it --network=exless alpine

This way, you can use tools like ping, nslookup, dig, etc., to troubleshoot network issues.

Add a Temporary Container in the Compose Configuration

This method also works

services:
  busybox:
    image: busybox
    container_name: exless-busybox
    command: sleep 3600  # 让容器运行一个小时，足够进行测试
    networks:
      - default  # 确保这是你其他服务使用的网络

Summary

That's it.

The entire process was extremely tortuous.

Writing this article felt like reliving the torture...

References

• https://develop.sentry.dev/self-hosted/
• https://github.com/exceptionless/Exceptionless