OpenTelemetry trace export arrives in GoodMem 1.0.324

Announcements
1

GoodMem 1.0.324 is out, and the server now exports OpenTelemetry traces. Each API request becomes a trace, with spans underneath it for the retrieval stages, the provider calls, and any background job attempts the request triggers. Export goes over OTLP, to a collector of your own or directly to Jaeger, Tempo, Langfuse, or LangSmith.

A quick tour:

  • Export is off by default and turns on with one variable, OTEL_TRACES_EXPORTER=otlp. OTLP goes over gRPC or HTTP/protobuf, so a standard collector, Jaeger, or Tempo works with no translation.
  • Two vendor profiles, langfuse and langsmith, add the metadata those backends expect, so GoodMem spans show up there as retriever and embedding runs rather than generic spans. We verified against Jaeger 2.21, Tempo 3.0.3, Langfuse 4.46, and LangSmith as of this month.
  • If your application already propagates W3C trace context, GoodMem continues the caller’s trace by default, so one trace runs from your agent through retrieval and back. A link mode starts a separate trace with a link to the caller instead, which LangSmith needs.
  • Spans carry operation names, durations, outcomes, counts, and the principal, API key, space, and memory identifiers. They never carry query text, memory content, prompts, generated text, credentials, or SQL values, and exception messages and stack traces are stripped before export. There is no content-capture switch to turn on by accident.
  • Telemetry starts fail-open. If the exporter cannot initialize, the server logs a sanitized diagnostic, exposes a gauge on /metrics, and keeps serving requests.

In our own runs we could not measure a cost from leaving export on. Over 200 retrievals with the exporter on and then off, the median latencies were within a millisecond of each other. We have not measured it under production load, so treat that as a first data point rather than a guarantee.

The minimum configuration for a collector on the same network is:

OTEL_TRACES_EXPORTER=otlp
OTEL_EXPORTER_OTLP_TRACES_PROTOCOL=http/protobuf
OTEL_EXPORTER_OTLP_TRACES_ENDPOINT=http://otel-collector:4318/v1/traces

The docs cover it in reading order: OpenTelemetry support, Export traces from GoodMem, Configuration, Spans and attributes, and Troubleshoot tracing. The Langfuse and LangSmith guides have the per-backend header and profile settings.

Metrics and logs are unchanged. Metrics stay on the Prometheus /metrics endpoint, and log lines now include the trace and span ID when a span is current.

To move to 1.0.324, run goodmem upgrade for the CLI, then goodmem system upgrade for the server.

If traces do not show up, or show up somewhere strange, post here with the backend you are using and we’ll sort it out.

1 Like