Observability & OpenTelemetry

mzchat supports OpenTelemetry (OTel) for tracing and monitoring the backend service. This allows you to track the request lifecycle, analyze performance bottlenecks, and aggregate errors across the system.

Key Features

Tracing: All RPC endpoints are automatically traced with OpenTelemetry.
Request IDs: Every request is assigned a unique x-request-id, which is propagated through headers and spans.
Error Aggregation: Uncaught exceptions and unhandled rejections are recorded as error spans and logged.
Source Map Support: Stack traces in production (or when running compiled code) are automatically mapped back to the original TypeScript source code for better visibility.

Configuration

To enable OpenTelemetry in the backend, set the following environment variables:

Variable	Description	Default
`ENABLE_OTEL`	Set to `true` to enable OTel SDK.	`false`
`OTEL_EXPORTER_OTLP_ENDPOINT`	The endpoint of your OTel Collector or backend.	`http://localhost:4318`

Local Development Setup

The easiest way to view traces during development is to run Jaeger v2, which provides a native OpenTelemetry Collector and OTLP endpoints.

Docker Compose Example (Jaeger v2)

Create an observability.yml file (or add to your existing docker-compose.yml):

services:
  jaeger:
    image: jaegertracing/jaeger:2.0
    ports:
      - "16686:16686" # UI
      - "4317:4317"   # OTLP gRPC
      - "4318:4318"   # OTLP HTTP

Running with OTel

Start the Jaeger v2 stack:

docker-compose -f observability.yml up -d

Run the backend with OTel enabled. The OTEL_EXPORTER_OTLP_ENDPOINT should point to the OTLP HTTP port (4318).
Terminal window
```
ENABLE_OTEL=true OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4318 bun run dev
```
Access the Jaeger UI at http://localhost:16686 to view your traces.

Request Tracking

Every RPC call includes an x-request-id header in both the request and response. This ID is also attached to the OTel span as the request_id attribute. You can use this ID to correlate logs, traces, and specific user reports.

Runtime Error Visibility

The backend uses source-map-support to ensure that even when running bundled or transpiled code, stack traces point to the original TypeScript files. This is essential for debugging runtime errors effectively in production environments.