Repository navigation
trunk-merge/pr-105591/6af9b020-599b-4266-9e6e-276d05a18a11 - #114826
Closed
trunk-io[bot] wants to merge 36 commits into
Closed
trunk-io[bot] wants to merge 36 commits into
trunk-io[bot] wants to merge 36 commits into
Conversation
capture-logs now adds a min_timestamp header to every logs message: the earliest row timestamp, after the clamp, in epoch microseconds. The Node consumer will use it to tell, without decoding a message, whether any row in it can already be past the team's retention. Traces and metrics messages do not carry it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Rows from a backfill request can be recent, so the consumer can't tell them apart by age. capture-logs now forwards the granted window as a header on every message from such a request, so the consumer can apply the per-project backfill flag. Firehose and Datadog never send it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A backdated log row whose timestamp plus its retention is already in the past is now dropped in the logs consumer. Written, it would stay visible until a TTL merge removed its whole part, and every materialized view on logs34 would count it. The check needs a decode, so it only runs on a message whose min_timestamp header says its oldest row is past the shortest retention any row can get. Messages without the header come from a capture-logs that clamps every timestamp to within 24 hours, so they skip the check. Drops go through the existing drop accounting: they are credited in billing under LOGS_BILLING_PRORATE_ENABLED, an emptied message is attributed to retention_expired_all_dropped, and each team gets a records_dropped_retention_expired usage metric. Two accounting fixes were needed for a second measuring filter stage: the sampling drop counter now counts only sampling drops, and the billing pro-rate keeps the first stage's whole-batch content total instead of the last stage's. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
When no sampling stage runs, a transformation can drop rows before the expiry filter sees the batch. The filter measured the content total over the survivors only, while the consumer applies the dropped fraction to the header bytes of the whole batch, so the credit came out too large. The pipeline now measures the total over every decoded row before any stage runs, when a stage asks for it. The capture-logs README now states in one place which backfilled records are kept and when the kept ones expire. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The pipeline now measures the batch content total, counts drops per stage and names the stage that emptied a message, from the records it passes in and gets back. Stages report only what they dropped and why. This replaces the opt-in flag a filter set to have the total measured up front, and the rule that the first measuring stage won. The sampling stage reads the total from the pipeline instead of scanning the batch again. The all-dropped reason is the stage name plus "_all_dropped", so a new filter no longer needs its own branch in the consumer. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The expiry check now falls back to the team default when a row's retention_days is 0 or less, the same as kafka_logs34_avro_mv, instead of treating it as zero days and dropping a row ClickHouse would keep. An empty min_timestamp header now reads as absent. It used to parse as 1970-01-01, which forced a decode of the whole message. The sampling and expiry stage names are exported constants, so the consumer's per-stage counters cannot silently read zero after a rename. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A capture-logs build older than the min_timestamp header still accepts backfill_days, so a headerless message can hold old rows. The gate skips it anyway, which is safe only while ClickHouse counts expiry from the ingest time. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The expiry stage credited a dropped row by its size at the time of the drop, while the pro-rate denominator was measured before any stage ran. A hog transformation runs between the two, so a team that can edit transformations could grow an expired row and claim a credit up to the whole batch while its fresh rows were still stored. This mattered only with LOGS_BILLING_PRORATE_ENABLED on. The pipeline now fixes each row's content weight before any stage runs and passes it to the filter stages. Sampling and expiry both credit by that weight, and a row a stage created weighs nothing. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
sampleRecords now takes the batch context that runPipelineStages builds, so the integration test calls the sampling stage through the pipeline. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… flag capture-logs marks a backfill request with a backfill_days header. The consumer drops those messages whole unless the team has the logs-backfill-enabled flag, before quota and rate limiting. Row age can't stand in for the marker, because a backfill request can carry recent rows. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This predates the expiry stage. On master, sampling drop rules already credited a batch by body, attributes and event name only. A sender could pair a dropped one-byte row with a surviving row whose content sat in resource attributes, and get the whole batch refunded under LOGS_BILLING_PRORATE_ENABLED. Expiry was a second way in. The weight now counts every sender-controlled field. Resource and scope bytes are split across the rows that share them, because the request carries them once. Counting them per row would let a sender inflate the weight of rows it expects to be dropped. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
LOGS_BACKFILL_ENABLED_TEAMS replaces the logs-backfill-enabled flag, matching how the logs consumer gates every other per-team feature. It needs no PostHog client, no flag cache and no new charts secrets. The list is empty in production by default, so no team can backfill until one is added. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Loki guarantees only three things per entry: a nanosecond timestamp, a line, and stream labels whose names are arbitrary. It has no schema and no reserved label names, so service.name and severity have to be told rather than inferred. This layer is the file a customer writes to tell us, and the command that reads it. Validation rejects what would otherwise surface hours into a multi-terabyte run: a range outside what Loki nanosecond epochs express, a shard that overflows or exceeds max_query_length, a body size the intake answers with 413, a regex whose capture group does not exist, and an empty resource_labels list, which means the opposite of omitting the key. Extractors dispatch on the YAML node kind rather than deriving untagged, so a misspelled key inside an extractor names the field instead of reporting that no variant matched. Running an import errors: this layer validates the config and stops there. The Loki client, mapping, batching, checkpointing and send loop follow. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Every extractor field takes one extractor or a list, not just severity. A reader following the old text would never use a fallback chain on trace_id, which is the case that needs one when newer streams carry structured metadata and older ones do not. The --config help pointed at the help output it was printed in. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Reads each selector's range in half-open shards, pages query_range forward, maps entries onto OTLP records, batches them under the intake's body limit, and records a shard only once every batch in it is acknowledged. Pages over-fetch by one to tell a full page from an exhausted shard, and resume at the boundary timestamp rather than past it: Loki truncates by entry count, not by timestamp, so entries sharing that instant would otherwise be skipped and never re-queried. A hash set of lines already sent at the boundary suppresses the re-read. Batching sizes each record once through the same serializer the encoder uses, so escaping cannot push a body past the ceiling, and encodes once per flush rather than re-encoding the batch per record. The checkpoint remembers the range it was built under. Widening range.from discards it and says so, rather than producing no windows and reporting success without importing the newly requested history. --dry-run sizes the job and reports how many sampled records each extraction rule matched, which is the only warning before a long run that a rule will produce nothing. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The boundary set was replaced on each page rather than extended, so a timestamp holding more entries than one page lost its earlier pages from the set and sent them again two pages later. The key hashed the line alone, so two pods emitting an identical line in the same nanosecond collided and one was suppressed as a duplicate. That is data loss rather than duplication, and the key now covers the stream labels. Both lived inside the paging loop with no test seam, which is why two review passes and the whole suite missed them. The logic is now a BoundaryFilter with three tests: an instant spanning several pages keeps every page suppressed, two streams sharing a line stay distinct, and moving off an instant forgets it so the set cannot grow for a whole shard. Transport failures now retry on the same schedule as HTTP ones instead of ending a multi-hour run on one connection reset, and the last attempt no longer sleeps before giving up. The dry run reads each sample value from the first record that carried the field, rather than reporting NOT FOUND for a field the first record happened to lack. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The checkpoint tests built a predictable path under the shared temp directory, which rust.lang.security.temp-dir blocks. tempfile is already a dependency and gives a unique directory that cleans itself up. project_key_from_env needed an unsafe env mutation to test its message, which rust.lang.security.unsafe-usage blocks and which is unsound in a multithreaded test binary anyway. The validation is now split from the environment read, the shape LokiAuth::resolve already uses in this module. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The CLI compiled in the public CA list and read nothing else, so a host signed by an internal authority could not be verified at all. Installing the CA on the machine had no effect, because a compiled-in list cannot be extended from outside. Enable `rustls-tls-native-roots` next to `rustls-tls`. reqwest extends one RootCertStore from each enabled source, so the two combine rather than replace. The public list still verifies PostHog Cloud on a machine with no trust store, and the native store now covers internally signed hosts. This is what the Loki import needs to reach a self-hosted cluster, and it also removes the reason `sourcemap upload` shipped `--skip-ssl-verification` in #35872, which stops verifying certificates rather than trusting one authority. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Status failures from Loki already carry a hint naming the setting to check: 401 points at the auth variables, 400 at the selector, 429 at rate limiting. Transport failures carried none, so the hardest failure to diagnose was the only one with no guidance. An internally signed Loki is the normal self-hosted setup, and it surfaces as `invalid peer certificate: UnknownIssuer` several levels down a reqwest error chain. Both `send` calls now route through a helper that walks that chain and, on a trust failure, names the root certificate and SSL_CERT_FILE. The guidance goes in the error rather than a separate log line, so it appears once, where the run stopped, in the same shape `decode` already uses. reqwest exposes no typed variant for a trust failure, so the match is on the message. A miss costs only the hint, never the error, so a rustls rewording degrades to the previous behavior. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The dry run took its whole mapping sample from the first selector that returned enough records, so a field only later selectors carried was reported as NOT FOUND. Each selector now gets an equal share of the sample. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Loki answers every query_range that starts at an instant holding more entries than a page with the same first page, so the import stopped with "Loki made no progress" and a rerun stopped at the same shard. When a page brings nothing new at the cursor's instant, the importer now reads that one instant in a single request ([T, T+1ns), limit 5000, the default max_entries_limit_per_query), sends the entries it has not sent, and moves the cursor past it. An instant with 5000 or more entries still stops the run, with a message to narrow the selector. Checked against a local Loki 3.4.2: start is inclusive and end exclusive, the same first page comes back each time, and a limit above 5000 is rejected with a 400. An import over 1300 entries at one instant failed before this change and now sends all 1312 entries in range exactly once. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A full Loki page can cut its last instant short. The importer used to send that partial instant, resume at it, and suppress the re-read with a hash set of entries already sent. That set needed a key, and the key left out structured metadata, so two entries differing only in metadata could lose one of them. A page now keeps only the instants before the first entry past the limit, which Loki returns in full, and the next page starts at that instant. A full page holding a single instant is read whole straight away, without first fetching the same page again. Nothing is sent twice, so the dedup set and its key are gone. The dry run asks Loki for each selector's share of the sample instead of fetching a full page and discarding most of it, and the shares now sum to at most 100 records. Checked against a local Loki 3.4.2: an import over 1300 entries at one instant, with entries on either side, sends all 1312 exactly once, and an instant of 5100 entries stops with the narrow-the-selector message. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Intake drops a record older than the project's logs retention without telling the sender, so an import whose range reaches past retention used to look successful while most of it was thrown away. The import now reads the project's default logs retention and states which part of the range intake would drop. The dry run prints it. A real run asks before it goes ahead, then skips the shards wholly past the cutoff instead of sending them. Without a terminal it stops before sending anything unless --skip-expired is passed. When the retention cannot be read, it says so and asks the same question. The check reads the default retention only; the README says that retention rules can keep some logs longer or drop them sooner. Checked against a local Loki with a fake PostHog: the run stops without a terminal, skips the expired shards with --skip-expired (Loki saw no query before the cutoff), and treats a cancelled prompt as a no. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…fill Adds GET /api/projects/:id/logs/backfill_status, which reports the logs-backfill-enabled flag for the project. The Loki import reads it before sending anything. A refusal stops the run, because intake would drop every record after answering 200. An unreadable answer only warns, because the consumer enforces the flag either way. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The retention and backfill checks read the project from posthog-cli login, but the import writes to the project that owns POSTHOG_PROJECT_API_KEY. When they differ, both checks answered for the wrong project, so a run could skip records the target keeps or send records the target drops. The import now compares the two api tokens and stops on a mismatch. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Backfill is now gated by LOGS_BACKFILL_ENABLED_TEAMS on the logs consumer instead of a feature flag. Django cannot see that env var, so the backfill_status endpoint and the CLI pre-check that read it go. The README tells operators to contact support before the first import. The project-match check stays, because the retention check still reads the logged-in project. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The dashboard query snapshots were regenerated by CI against an older base and now conflict with master. The stack does not change dashboard queries, so the file goes back to the base version and master's copy wins on merge. The stray blank line in the logs API goes too, so this PR no longer touches Django. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
trunk-io
Bot
deleted the
trunk-merge/pr-105591/6af9b020-599b-4266-9e6e-276d05a18a11
branch
October 10, 2026 00:16
Author
|
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This pull request was created and is being managed by Trunk Merge.
This pull request is based on the master branch at SHA 4c1b5996efacb03ee7961b53efe6779f46246b00.
See more details here.
When CI completes, this pull request will be closed automatically.
Pull Requests Being Tested
This pull request is testing the changes from pull request 105591, stacked on pull requests 112110, 112111, and 105590.