fix: stringify JSON search sidebar filters

[mfroembgen]

Summary

• Detect JSON-backed source columns and render sidebar filter keys as ClickHouse string expressions, for example toString(ResourceAttributes.\k8s`.`namespace`.`name`), instead of JSON map access or a fixed .:String` suffix.
• Canonicalize stale URL/saved sidebar filters that still use JSON map access once source columns load.
• Reuse the same JSON string rendering for search URL filters, sidebar value queries, load-more queries, and distribution queries.
• Prioritize active and pinned fields when fetching initial facet values so selected combinations show their matching options before Load more.
• Add unit coverage for JSON filter serialization, hydration, stale persisted filter canonicalization, and metadata value/distribution SQL rendering.

Why toString(...)

Testing against ClickHouse 26.5.1.882 with JSON(max_dynamic_types=8, max_dynamic_paths=64) columns showed that .:String is only safe for JSON paths whose active dynamic type is actually String. Integer, boolean, and mixed-type paths can silently return no values with .:String, while toString(<JSON path>) returned usable values for all of these real path shapes:

• ResourceAttributes.k8s.namespace.name (string)
• ResourceAttributes.cloud.account.id (integer)
• LogAttributes.endOfBatch (boolean)
• LogAttributes.level (mixed integer/string)

The same toString(...) expression also worked for distribution/grouping queries.

Manual verification

• Verified directly against ClickHouse 26.5.1.882, where ResourceAttributes and LogAttributes are JSON(max_dynamic_types=8, max_dynamic_paths=64) columns.
• Metadata-style query over a live data window returned non-empty values for string, int, bool, and mixed JSON paths with toString(...).
• Distribution-style grouping returned values for toString(ResourceAttributes.\cloud`.`account`.`id`)andtoString(LogAttributes.`endOfBatch`)`.
• Verified a local dashboard flow against a live API: stale ResourceAttributes['k8s.namespace.name'] filters were canonicalized, results loaded without the ClickHouse arrayElement JSON error, Show Distribution loaded, and selected/pinned facets refreshed with matching values plus Load more.

Automated verification

• mise exec node@22.16.0 -- node .yarn/releases/yarn-4.13.0.cjs lint:fix
• mise exec node@22.16.0 -- node ../../.yarn/releases/yarn-4.13.0.cjs ci:lint in packages/common-utils
• mise exec node@22.16.0 -- node ../../.yarn/releases/yarn-4.13.0.cjs ci:unit in packages/common-utils
• mise exec node@22.16.0 -- node .yarn/releases/yarn-4.13.0.cjs workspace @hyperdx/app jest packages/app/src/components/__tests__/DBSearchPageFilters.test.tsx packages/app/src/components/DBSearchPageFilters/utils.test.ts packages/app/src/__tests__/searchFilters.test.ts --runInBand --coverage=false
• mise exec node@22.16.0 -- bash -lc 'corepack enable >/dev/null 2>&1 || true; make dev-e2e FILE=filter-key-edge-cases' (18 passed)

References

Fixes #2549.
Related to #2482 and #2537.

[changeset-bot]

🦋 Changeset detected

Latest commit: 7f46a87

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 4 packages

Name	Type
@hyperdx/app	Patch
@hyperdx/common-utils	Patch
@hyperdx/api	Patch
@hyperdx/otel-collector	Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

[vercel]

@mfroembgen is attempting to deploy a commit to the HyperDX Team on Vercel.

A member of the Team first needs to authorize it.

[greptile-apps]

Greptile Summary

This PR replaces ClickHouse JSON typed-subcolumn access (.:String) with toString(...) wrappers for sidebar filters, facet queries, and distribution queries, fixing silent empty-result bugs on integer/boolean JSON paths. It also canonicalizes stale URL/saved bracket-form JSON filters and adds a generation-counter guard to discard stale load-more results after the query scope changes.

• toString(...) serialization: renderJsonStringSubcolumn is replaced by renderJsonStringExpression, a now-exported function that emits toString(col.\seg1`.`seg2`)instead ofcol.`seg1`.`seg2`.:String`; all downstream callers are updated consistently.
• Stale filter canonicalization: canonicalizeFilterQuery rewrites persisted bracket-form or typed-subcolumn JSON filters to the new canonical form, with a round-trip guard that prevents rewriting compound SQL predicates.
• Load-more race safety: A loadMoreGenerationRef counter is incremented whenever the query scope resets; in-flight callbacks that arrive after a scope change are silently discarded.

Confidence Score: 5/5

Safe to merge — all changed paths are well-covered by new and updated unit tests, and the generation-counter and round-trip guards prevent the two most dangerous edge cases.

The toString(...) replacement is consistently applied across all four query surfaces and the backslash-escaping round-trip is explicitly tested. The canonicalizeFilterQuery round-trip guard means stale filters are rewritten only when they fully parse and reproduce. The load-more generation counter correctly discards stale callbacks, confirmed by two dedicated race-condition tests.

No files require special attention. The parsing helpers in metadata.ts are the densest new code but each has targeted unit coverage including backslash and complex type suffix edge cases.

Important Files Changed

Filename	Overview
packages/common-utils/src/core/metadata.ts	Core change: replaces renderJsonStringSubcolumn with renderJsonStringExpression (toString wrapper); adds stripClickHouseJsonTypeSuffix, parseRenderedJsonStringExpression, and related parsing helpers; exports quoteJsonPathSegment and quoteIdentifierIfNeeded. Wires parseRenderedJsonStringExpression into getAllKeyValues so toString(...) expressions resolve to the correct rollup (column, key) pair.
packages/app/src/searchFilters.tsx	Adds canonicalizeFilterQuery with a round-trip guard and threads jsonColumns through escapeFilterStateKeys and useSearchPageFilterState. A jsonColumnsRef mirrors the stable knownColumnsRef pattern.
packages/app/src/components/DBSearchPageFilters/utils.ts	Imports renderJsonStringExpression and stripClickHouseJsonTypeSuffix from common-utils; removes local quoteIdentifierIfNeeded duplicate; extends cleanClickHouseExpression to strip generic ClickHouse JSON type suffixes; adds jsonColumns option to toQuotedClickHouseKeyExpression.
packages/app/src/components/DBSearchPageFilters/hooks.ts	Passes jsonColumns to toQuotedClickHouseKeyExpression and escapeFilterStateKeys in both pipelines; adds loadMoreGenerationRef counter incremented on scope resets to discard stale load-more callbacks and eagerly clear loading-keys state.
packages/app/src/DBSearchPage.tsx	Moves inputSourceTableConnection memo earlier; adds canonicalizedFilters memo and useEffect that rewrites stale persisted JSON filters on source load; passes jsonColumns to useSearchPageFilterState.
packages/app/src/components/DBSearchPageFilters.tsx	Passes jsonColumns to toQuotedClickHouseKeyExpression call sites; replaces isDefaultExpanded-only auto-expand with shouldExpandForLoadMore condition that also expands empty facet groups not yet loaded.
packages/app/src/tests/searchFilters.test.ts	Adds unit tests for JSON filter serialization/hydration, idempotent canonicalization from all three legacy forms, and guards against rewriting compound SQL predicates.
packages/common-utils/src/tests/metadata.test.ts	Updates assertions from .:String to toString(...) form; adds backslash-escaping coverage and getAllKeyValues tests verifying toString(...) expressions decompose to correct rollup (column, key) pairs.

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart TD
    A[Filter key in sidebar] --> B{Is key in jsonColumns?}
    B -- Yes --> C[renderJsonStringExpression toString wrapper]
    B -- No --> D{Bracket-form or backtick-quoted?}
    D -- Yes --> E[Return unchanged map access]
    D -- No --> F[toClickHouseKeyExpression bracket form]
    C --> G[SQL filter condition]
    E --> G
    F --> G
    H[Persisted filter loaded] --> I{canonicalizeFilterQuery}
    I -- hasJsonFilter AND round-trip match --> J[Re-serialize with escapeFilterStateKeys]
    I -- no JSON filter OR compound predicate --> K[Return original filters unchanged]
    J --> L[setValue plus setSearchedConfig]
    L --> M[canonicalizedFilters memo returns null - cycle stops]

Loading

%%{init: {'theme': 'base', 'themeVariables': {"darkMode": true, "background": "#0d1117", "primaryColor": "#21262d", "primaryTextColor": "#e6edf3", "primaryBorderColor": "#8b949e", "lineColor": "#8b949e", "textColor": "#e6edf3", "edgeLabelBackground": "#161b22", "actorBkg": "#21262d", "actorBorder": "#8b949e", "actorTextColor": "#e6edf3", "actorLineColor": "#8b949e", "signalColor": "#8b949e", "signalTextColor": "#e6edf3", "noteBkgColor": "#373320", "noteBorderColor": "#d4a72c", "noteTextColor": "#f0e6c0", "labelBoxBkgColor": "#21262d", "labelBoxBorderColor": "#8b949e", "labelTextColor": "#e6edf3", "loopTextColor": "#e6edf3", "activationBkgColor": "#30363d", "activationBorderColor": "#8b949e"}}}%%
flowchart TD
    A[Filter key in sidebar] --> B{Is key in jsonColumns?}
    B -- Yes --> C[renderJsonStringExpression toString wrapper]
    B -- No --> D{Bracket-form or backtick-quoted?}
    D -- Yes --> E[Return unchanged map access]
    D -- No --> F[toClickHouseKeyExpression bracket form]
    C --> G[SQL filter condition]
    E --> G
    F --> G
    H[Persisted filter loaded] --> I{canonicalizeFilterQuery}
    I -- hasJsonFilter AND round-trip match --> J[Re-serialize with escapeFilterStateKeys]
    I -- no JSON filter OR compound predicate --> K[Return original filters unchanged]
    J --> L[setValue plus setSearchedConfig]
    L --> M[canonicalizedFilters memo returns null - cycle stops]

Loading

_{Reviews (18): Last reviewed commit: "fix: stringify JSON search sidebar filte..." | Re-trigger Greptile}

[github-actions]

🔴 P0/P1 -- must fix

• packages/app/src/components/DBSearchPageFilters/utils.ts:25 -- cleanClickHouseExpression strips backticks but never reverses the backslash-doubling quoteJsonPathSegment applies, so canonicalizeFilterQuery is non-idempotent for a JSON path segment containing a backslash or backtick; the DBSearchPage memo/effect then re-setValues an ever-growing filter each render and crashes with "Maximum update depth exceeded".
  • Fix: Make cleanClickHouseExpression the exact inverse of quoteJsonPathSegment (halve doubled backslashes and undouble backticks), or reuse the metadata-side parseRenderedJsonStringExpression, so escape∘unescape reaches a fixed point.
  • _{adversarial, correctness, maintainability, kieran-typescript}

🟡 P2 -- recommended

• packages/app/src/DBSearchPage.tsx:1315 -- The canonicalization effect snapshots live form state via getValues() into setSearchedConfig, so when useJsonColumns resolves asynchronously after the user has typed an unsubmitted where/select, it commits and executes that draft and rewrites the URL.
  • Fix: Update only the filters field from the already-searched searchedConfig, not the live form, so canonicalization cannot pick up unsubmitted edits.
  • _{adversarial, julik-frontend-races}
• packages/app/src/components/DBSearchPageFilters.tsx:945 -- The shouldExpandForLoadMore effect only ever calls setExpanded(true) and re-fires on every false→true transition (e.g. live-tail refetch toggling optionsLoading while extraFacetKeys resets), re-expanding a facet group the user just collapsed.
  • Fix: Make auto-expand one-shot per key (track already-auto-expanded keys in a ref) so it never re-asserts after a manual collapse.
  • _{adversarial, julik-frontend-races}
• packages/common-utils/src/core/metadata.ts:185 -- The new security-sensitive JSON-path parsers (parseRenderedJsonStringExpression, parseClickHouseIdentifier, parseRenderedJsonPath, stripClickHouseJsonTypeSuffix) are only exercised through happy-path callers, with no direct negative or adversarial-escaping coverage.
  • Fix: Add direct tests for malformed inputs (unterminated toString(/backtick, unquoted segments, empty path) and round-trip cases with doubled backticks and backslashes.
  • _{kieran-typescript, testing}

🔵 P3 nitpicks (6)

• packages/common-utils/src/core/metadata.ts:2217 -- parseRenderedJsonStringExpression uses bracket indexing innerExpression[parsedColumn.endIdx] while the rest of the module uses charAt(), relying on noUncheckedIndexedAccess being off.
  • Fix: Use charAt() for consistency and undefined-safety.
• packages/common-utils/src/core/metadata.ts:60 -- quoteJsonPathSegment was newly exported but has no consumer outside metadata.ts, and four low-level quoting helpers now widen the published @hyperdx/common-utils API surface.
  • Fix: Keep quoteJsonPathSegment module-private and confirm the other helper exports are intended public contract.
• packages/app/src/components/DBSearchPageFilters/utils.ts:5 -- The imported quoteIdentifierIfNeeded unquotes and escapes backslashes before quoting, diverging from the removed SqlString.escapeId version for already-quoted or backslash-containing flat column names.
  • Fix: Add a regression assertion that flat-column quoting still emits identical SQL for special-character column names.
• packages/app/src/components/DBSearchPageFilters/hooks.ts:140 -- Call sites pass { jsonColumns: jsonColumns ?? [] } even though toQuotedClickHouseKeyExpression already defaults options.jsonColumns ?? [] internally, duplicating the empty-array default in ~6 places.
  • Fix: Pass jsonColumns directly and let one layer own the default.
• packages/app/src/components/DBSearchPageFilters/hooks.ts:130 -- The escapedKeysToFetch memo gates only on isColumnsLoading, so if useColumns resolves before useJsonColumns (still []), JSON sub-keys are briefly escaped as invalid map-access SQL until the memo re-runs.
  • Fix: Also gate the memo on jsonColumns having loaded before fetching JSON-backed keys.
• packages/common-utils/src/core/metadata.ts:1695 -- getKeyValues rollup parsing only recognizes the new toString(...) form, so legacy .:String keys that were never canonicalized fall through parseKeyPath and mis-split into rollupColumn/rollupKey, degrading facet value suggestions.
  • Fix: Recognize and strip the legacy .:String typed-subcolumn form in the keyExpressions parse step.

---

Reviewers (8): correctness, security, adversarial, julik-frontend-races, kieran-typescript, testing, maintainability, api-contract.

Testing gaps:

• DBSearchPage canonicalizedFilters memo/effect has no coverage of its loop-avoidance guard or its non-overwrite behavior.
• canonicalizeFilterQuery non-sql early bail, length-mismatch bail, dateTimeColumns argument, and ordering-difference skip paths are untested.
• No adversarial round-trip tests for JSON path segments containing backslashes, backticks, closing parens, or newlines.