Generated schemas go stale because each source writes into the shared tree on its own #428

What problem are you facing?

Generated schemas go stale because each source writes into the shared tree on its own. This ends up resulting in a case where users are running rm -rf schemas before builds.

Summary

crossplane project build writes the language schemas under schemas/ (Go,
Python, KCL, JSON) one source at a time. Each source's output is copied on top
of the existing tree, and nothing is ever removed. The lock file records only
each source's version, not the tree as a whole. As a result, the tree on disk
can keep models that no source produces anymore, or miss models the project
should have, and the build never notices.

I found this while adding TypeScript support (#170), which can't work at all
with the current design. I'd like to fix it in its own PR first so the
behavior change can be reviewed separately from TypeScript.

Current impact (on main today)

None of these cause an error. Users see wrong or leftover models in their
editor and in function builds, and the fix ("clean the cache") isn't obvious.

This happens in real projects. Modelplane now deletes schemas/ before
every build to work around it (modelplaneai/modelplane#170, commit
a3e9ba9,
June 2026). The commit message says:

Both generators are also additive: they overwrite the files they emit but
never remove models or lock entries for XRDs or dependencies that have
been dropped or renamed. Stale schemas and lock entries accumulated across
regenerations. This is how schemas/.lock.json came to carry entries for
provider versions the project no longer referenced.

The workaround costs a full regeneration on every build: about 100s for
modelplane's 13 providers, even when only an XRD changed.

TypeScript (#170) is fully blocked. Its generator writes one npm package per
run. The root index.js, package.json and the shared _schemas/ folder
describe everything in that run. When each source runs on its own, each one
rewrites those root files to describe only itself, and the last one wins. Only
the last source's models can be imported.

Where the problem is in the code

Line numbers are from main @ 833d6e0.

1. Two separate write passes in the build.
   internal/project/build.go:269 calls dependencyManager.AddAll (one write
   per dependency). Then build.go:277 calls schemaManager.Generate for
   apis/ (one more write). Nothing ever sees all sources at once.

2. Writes only add files, never remove them.
   internal/schemas/manager/manager.go:72 (Generate) copies the generator
   output into schemas/<lang>/ with CopyFilesBetweenFs (lines 131 and 135).
   Nothing clears the old output first. This causes the leftover kind and
   leftover dependency rows in the table.

3. Freshness is checked one source at a time.
   manager.go:51 (Add) skips a source if its recorded version matches. It
   never checks the language set or whether the directories still exist. The
   dependency manager calls Add at internal/dependency/manager.go:226 and
   :379-383. This causes the language rows in the table.

4. The lock only stores versions.
   internal/schemas/manager/lock.go:22 has just
   Packages map[string]string. It can't record which languages were
   generated, or whether the tree on disk was produced from exactly these
   sources.

5. apis/ is never cached.
   build.go:277 calls Generate (always regenerates) instead of Add. The
   version hash for apis/ is computed and written to the lock, but nothing
   ever reads it back.

Proposed fix

Collect every source first, then generate in two cached groups and swap the
finished tree into place.

Two source groups

• Dependencies: every source from crossplane-project.yaml, including
  transitive xpkg dependencies.
• APIs: the project's own apis/ directory.

Each group is generated, cached and checked for freshness on its own. An XRD
edit regenerates only the APIs group; the dependency output is reused from the
cache. A dependency change regenerates only the dependencies group, which is
rare.

Within a group, sources are generated once per source type (CRD and OpenAPI).
A generator whose output describes the whole run (TypeScript) can implement a
small schemaMerger interface to combine parts into one tree. Other
generators copy parts one after another, and the copy fails on a conflict (see
below).

Lock file (internal/schemas/manager/lock.go)

{
  "version": 2,
  "generator": "<fingerprint>",
  "languages": ["go", "python"],
  "groups": {
    "dependencies": { "xpkg://...": "sha256:..." },
    "apis": { "fs://apis": "<hash>" }
  }
}

A group is fresh only when all of these hold:

• its source map exactly matches the current sources,
• the language set matches,
• the generator fingerprint matches,
• its cached output exists.

This one rule covers every row in the impact table:

• A removed dependency changes the source map.
• A changed language list changes languages.
• A deleted directory fails the "cached output exists" check.

The generator fingerprint is built from the CLI version, the generator
image digests and the generator options, such as the Go runtime-object and
accessor settings. A CLI upgrade that changes generated code then regenerates.
Today it doesn't.

An old lock has no version field, so it reads as stale and the project
regenerates once. An unreadable lock is treated as stale rather than failing
the build. Today getLock returns the decode error, so a lock damaged by an
interrupted write breaks every later build.