Distributor: add experimental OTLP metrics ingestion over gRPC - #7873
Open
friedrichg wants to merge 1 commit into
Open
friedrichg wants to merge 1 commit into
friedrichg wants to merge 1 commit into
Conversation
friedrichg
force-pushed
the
otlp-grpc-ingestion
branch
from
October 1, 2026 18:08
550f92f to
049a978
Compare
Add the OTLP metrics gRPC service (opentelemetry.proto.collector.metrics.v1.MetricsService/Export) to the distributor gRPC server port, so the OpenTelemetry Collector otlp exporter can push to Cortex. Enable it with -distributor.otlp.grpc-enabled. The HTTP and gRPC receivers now share convertOTLPToWriteRequest. The gRPC receiver maps distributor errors to the gRPC codes that OTLP clients use to decide if they retry, and it returns a partial success when some metrics cannot be converted. The Cortex "proto" gRPC codec replaced the pdata codec, because it is registered after it. The Cortex codec now encodes and decodes pdata messages, so OTLP gRPC requests work with either init order. Signed-off-by: Friedrich Gonzalez <1517449+friedrichg@users.noreply.github.com>
friedrichg
force-pushed
the
otlp-grpc-ingestion
branch
from
October 1, 2026 18:38
049a978 to
fd39350
Compare
SungJin1212
reviewed
Oct 2, 2026
| |---|---|---| | ||
| | Request deduplicated by the HA tracker | `OK` | No | | ||
| | Invalid request (HTTP 4xx) | `INVALID_ARGUMENT` | No | | ||
| | Missing or wrong tenant | `UNAUTHENTICATED` / `PERMISSION_DENIED` | No | |
Member
There was a problem hiding this comment.
Current ServerUserHeaderInterceptor behavior emits UNKNOWN as ServerUserHeaderInterceptor before Export runs in the missing tenant case, so the client gets UNKNOWN, the same as for every other gRPC method.
I think we should change the interceptor in fakeauth.SetupAuthMiddleware to return codes.Unauthenticated for user.ErrNoOrgID, since this is the first gRPC API we expose to external clients.
SungJin1212
reviewed
Oct 2, 2026
| | Client canceled the request | `CANCELED` | No | | ||
| | Deadline exceeded | `DEADLINE_EXCEEDED` | Yes | | ||
|
|
||
| _Requires [authentication](#authentication)._ |
Member
There was a problem hiding this comment.
The authentication link points to a section that describes the X-Scope-OrgID HTTP header, not gRPC metadata.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this PR does:
Adds experimental OTLP metrics ingestion over gRPC to the distributor. The distributor serves the standard
opentelemetry.proto.collector.metrics.v1.MetricsService/Exporton the existing gRPC server port, so the OpenTelemetry Collectorotlpexporter can push to Cortex. Enable it with-distributor.otlp.grpc-enabled(defaultfalse).X-Scope-OrgIDgRPC metadata. The request size is limited by-server.grpc-max-recv-msg-size-bytes.UNAVAILABLE, 4xx becomesINVALID_ARGUMENT). When some metrics cannot be converted, the response is a partial success with an error message.protogRPC codec is registered after the pdata codec and replaced it, so no OTLP gRPC request could be decoded. The Cortex codec now handles pdata messages.Tested with unit tests (bufconn and the real
pmetricotlpclient) and the integration testTestOTLPGRPC.AI disclosure: I wrote this feature with Claude Code. I reviewed every line and I can explain every change.
Which issue(s) this PR fixes:
Fixes #7872
Checklist
CHANGELOG.mdupdated - the order of entries should be[CHANGE],[FEATURE],[ENHANCEMENT],[BUGFIX]docs/configuration/v1-guarantees.mdupdated if this PR introduces experimental flags