Evaluate OpenTelemetry as a vendor-neutral layer for request-path instrumentation

[mdorf]

Context

Raised by @alexskr after the /search instrumentation work (#243, #244, #245): the method tracers we added are New Relic-specific. OpenTelemetry is more versatile, New Relic supports the standard, migration to other APM solutions would be easier, and OTel-based instrumentation could get wider adoption in the OntoPortal community.

This issue documents the current state and a recommended path so the work can be picked up when prioritized.

What is actually New Relic-specific today

The custom surface added by #243/#245 is deliberately small and centralized:

• config/newrelic_method_tracers.rb: a declarative registry of 14 traced methods; only the registration mechanism (add_method_tracer) is vendor-specific. The registry itself (which methods matter) is the transferable knowledge.
• One NewRelic::Agent.add_custom_attributes call (search_scoped, helpers/search_helper.rb).
• A registry-driven regression test.

Porting this layer to OTel (a module that wraps the same registry methods in tracer.in_span) is roughly a day of work. The registry is the abstraction seam; new instrumentation should keep going through it.

The larger dependency is the newrelic_rpm agent itself, which predates this work: Rack/Sinatra transaction naming, Redis and Net::HTTP segments, transaction traces, breakdown tables, error tracking, and the production dashboards/alerting built on them. That is the real switching cost, not the tracer file.

OpenTelemetry assessment (as of mid-2026)

• New Relic ingests OTLP natively, so an OTel-instrumented app can keep reporting to the existing NR account. Data arrives as generic spans and dimensional metrics: NRQL span queries (the basis of most of the GET /search: reduce response time (~830 ms of ~890 ms average is in application code) #244 analysis) work the same, but curated APM UI features (transaction breakdown tables, Apdex, thread profiler) are native-agent only.
• opentelemetry-ruby is mature for traces, with auto-instrumentation gems for Sinatra, Rack, Redis, and Net::HTTP; peripheral features are weaker than the native agent. Traces are what the /search and /tree work actually needed.
• Strongest argument: community portability. The current tracer file only benefits deployments with New Relic; every other OntoPortal installation gets no-ops. OTel instrumentation in the shared codebase would work with any OTLP backend (Jaeger, Grafana Tempo, SigNoz, New Relic, ...), and instrumentation contributions would compound across adopters.

Recommended path (incremental, not a same-day swap)

1. Keep the native NR agent as-is for now; production monitoring and the in-flight GET /search: reduce response time (~830 ms of ~890 ms average is in application code) #244 verification depend on it.
2. Keep routing new instrumentation through the registry so the vendor-specific surface stays one file.
3. As a scoped project: add the OTel SDK + auto-instrumentation, port the registry to emit OTel spans, export OTLP (to New Relic for NCBO; backend of choice for other deployments).
4. Run both pipelines briefly, compare overhead and data quality, then decide whether to retire newrelic_rpm.
5. Coordinate with the OntoPortal community on whether this becomes a shared feature (config knobs for OTLP endpoint, docs).

Open questions

• Overhead comparison: native agent vs OTel SDK under our Unicorn setup.
• Metrics/logs: New Relic application logging is in use for staging; OTel Ruby's logs story should be re-validated at implementation time.
• Whether dashboards/alerts currently defined on native-agent data need rebuilding on span/metric queries before the agent can be retired.

Related

• Perf: instrument /search request path with New Relic method tracers #243 (method tracer registry), GET /search: reduce response time (~830 ms of ~890 ms average is in application code) #244 (search performance investigation), Perf: cache the ontology list, removing ~80% of /search response time #245 (ontology list cache)