feat: versioned artifact DAG reconciliation for #1679, with tombstone fix and artifact retention

[lyingbug]

Closes Tencent#1679.

This takes PR #2360 by @Even1115 as the base — of all the submissions for this issue it is the only one whose migrations load cleanly both on its own head and after merging main, so the service actually starts — and adds the two things it was missing.

Base: what Tencent#2360 provides

Reparse stops being a delete-and-rebuild and becomes a reconciliation against a desired state.

• internal/artifact/key.go defines a versioned canonical key envelope: KeyVersion, Stage, ordered DirectInputs (upstream artifact digests, not upstream configuration), Processor identity, the rendered request, and the output schema version. Changing a VLM prompt therefore invalidates the VLM layer without cascading into layers whose inputs did not move. RejectSecretFields fails closed rather than silently dropping credential fields.
• internal/artifact/identity.go anchors chunk identity on the content digest with a DuplicateOrdinal for repeated text, so inserting a paragraph does not shift the IDs of everything after it.
• knowledge_reconcile.go computes Added / Updated / Stale, applies additions and updates before cleanup, and preserveLiveChunkFields keeps user-owned state (enabled flag, tags, metadata, relations) across a rebuild.
• Migrations are complete: PostgreSQL 000079, SQLite 000002, plus the MySQL and ParadeDB bootstrap schemas.

Added here

1. Merge main and resolve the conflict correctly. main landed markKnowledgeProcessing, which clears a stale error_message when a run starts; Tencent#2360 replaced the same write with an attempt-conditional publish so a delayed worker cannot overwrite a newer attempt. Both are kept: the status write clears error_message and goes through UpdateKnowledgeIfAttemptCurrent.

2. Reclaim tombstoned chunk IDs. This was the one functional defect in the base. Because chunk IDs are content-addressed, a paragraph removed in one reparse and restored in a later one resolves to the same primary key. Stale cleanup soft-deletes, so that key is still held by a row the reconciliation read path cannot see, and the restore fails:

reconciliation sees 0 chunks (tombstone invisible)
UNIQUE constraint failed: chunks.id

PurgeSoftDeletedChunks hard-deletes only tombstoned rows among an explicit ID list, and processChunks calls it before inserting desired.Added. The predicate is tenant + explicit IDs + deleted_at IS NOT NULL, so a live row owned by a concurrent attempt can never be dropped.

3. Prune cold processing artifacts. processing_artifacts carries last_hit_at and a (tenant_id, created_at) index, but nothing ever reclaimed rows — a cache holding parse output and embedding payloads grew without bound. A new housekeeping sweep drops entries untouched for the retention window (30 days by default, retunable with WEKNORA_ARTIFACT_RETENTION_DAYS, 0 disables). Work per tick is bounded so a first run against a neglected table cannot hold locks for minutes, and offloaded payloads are skipped so their objects cannot be orphaned.

Verification

go build ./..., go vet ./internal/... and go test ./internal/... all pass — 65 packages, no failures.

New regression coverage:

• TestPurgeSoftDeletedChunksAllowsStableIDReinsert — the restore path that previously failed.
• TestPurgeSoftDeletedChunksLeavesLiveRowsIntact / ...IsTenantScoped — the purge cannot reach live rows or another tenant.
• TestBuildDesiredDocumentChunksRestoresRemovedContentUnderSameID — proves at the reconciliation layer that a restored paragraph really does come back under the tombstoned key.
• Five tests for the artifact sweep, covering the retention window, renewal on read, offloaded rows, the disable switch, and a missing table.

Follow-ups worth separate PRs

• planChunkReuse from PR #2035 skips the vector write entirely for chunks that are already indexed with an unchanged embedding fingerprint. This branch still rewrites vectors; that change is a few hundred lines and is easier to review on its own.
• The StableIdentity column from PR #2467 decouples identity from the row primary key, which makes the tombstone class of problem structurally impossible rather than handled.
• Image sub-chunks (image_ocr, image_caption) still use random UUIDs, so their vectors churn on every rebuild even though the VLM text itself is cached.