Consolidate bundle schema annotations into one tree-structured file#5574
Draft
pietern wants to merge 7 commits into
Draft
Consolidate bundle schema annotations into one tree-structured file#5574pietern wants to merge 7 commits into
pietern wants to merge 7 commits into
Conversation
Drop the checked-in annotations_openapi.yml: the schema generator now derives those annotations from .codegen/cli.json in memory on every run. Merge annotations.yml and annotations_openapi_overrides.yml into a single annotations.yml whose structure mirrors the bundle configuration tree. Docs absent from the file inherit from cli.json; entries override them. The generated jsonschema.json and jsonschema_for_docs.json are unchanged. Co-authored-by: Isaac
Contributor
Approval status: pending
|
Collaborator
Integration test reportCommit: 4f4fe2c
52 interesting tests: 28 FAIL, 15 SKIP, 7 KNOWN, 2 flaky
Top 24 slowest tests (at least 2 minutes):
|
Replace the "_" entry inside "fields" with an explicit "type" key on the node, sibling to "fields". For map and sequence fields it documents each entry; for scalar fields it documents the value's type (enum values live here). Purely a file-format spelling change; the in-memory representation and the generated schema are unchanged. Co-authored-by: Isaac
Fold the near-identical inline-descriptor and type-docs serialization paths into one descriptorToMap helper and hoist the shared key order to a package var. Document RootTypeKey as the in-memory sentinel. No change to the generated annotations file or schema. Co-authored-by: Isaac
The in-memory annotations keyed each type's docs by field name plus a
magic "_" entry for the type itself, forcing "_" to dodge real field
names (config has fields named type, fields, description). Model it
instead as TypeAnnotation{Self, Fields}: Self is the type's own docs
(applied to its $defs entry; also where enum values live), Fields is the
per-field docs. The "_" sentinel and RootTypeKey are gone.
No change to the generated annotations file or schema.
Co-authored-by: Isaac
structFieldNames re-implemented the generator's readonly/internal/ exported/json-tag skip rules and then cross-checked the result against the emitted schema. The schema's property set is already authoritative, so order the names by struct declaration and keep only those the schema emitted; reflection now supplies order alone. A count check still catches a struct walk that fails to reach a property. Co-authored-by: Isaac
A config field named "type" (artifacts.*.type, variables.*.type) was told apart from the structural type-docs key only by nesting level. Prefix the two structural keys with "$" — matching the JSON-Schema convention this file documents — so they can never be confused with or collide with a field of the same name, which always appears as a bare key inside "$fields". Also tidy the saver: a takeSelf accessor mirrors takeField, and the ordering line numbers are named constants. No change to the generated schema. Co-authored-by: Isaac
It wrapped clijson.Parse with an empty-schemas check and was called once. Fold it into generateSchema and drop cli_json.go. Co-authored-by: Isaac
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Changes
The bundle JSON schema annotations move from three type-name-keyed YAML files to a single
bundle/internal/schema/annotations.ymlstructured like the bundle configuration tree:annotations_openapi.yml(~7k lines of derived data) is deleted. The generator now computes those annotations in memory from.codegen/cli.jsonon every run, so upstream docs are never materialized in the repo.annotations_openapi_overrides.ymland the oldannotations.ymlare consolidated into the newannotations.yml. A node documents one field inline;type:documents the type the field's value resolves to (for map and sequence fields: each entry) and carries enum values — these docs land on the type's shared$defsentry;fields:holds the block of that type's fields (maps/sequences unwrapped). Each type is expanded exactly once, at its first occurrence in struct declaration order, so shared types (e.g. everything undertargets) are documented in one place.description: PLACEHOLDERmarks fields with no docs anywhere.task generate-schemakeeps the file in sync via reflection over the config types: placeholders are added for new undocumented fields, and entries whose field or type no longer exists are dropped with a warning. The file's structure is validated against the samejsonschema.FromTypewalk that generates the schema, so it cannot drift.generate-annotationstask is gone; the generator takes the cli.json path as a required argument (theDATABRICKS_CLI_JSONenv var is removed).Why
The type-name-keyed files didn't show where a field lives in the configuration, and most of their content was a copy of cli.json data that went stale between regenerations. With a single structured file there is exactly one place to write CLI-owned docs and overrides, it reads like the config it documents, and everything inherited from cli.json stays out of the repo.
Tests
bundle/schema/jsonschema.jsonandjsonschema_for_docs.jsonare byte-identical tomainafter regeneration; the annotations file rewrite is byte-idempotent across runs (this is also asserted in CI by the existing drift gate and the docs workflow).TestRequiredAnnotationsForNewFieldsandTestNoDetachedAnnotationsare rewritten for the new format. Migration dropped 23 entries that referenced fields no longer present in the config (verified against the structs)../task generate-check, fulllint,fmt,checks, andtestpass locally.This pull request and its description were written by Isaac.