Skip to content

Host the compressed dictionary editor on view's binary engine (6b V2) - #169

Merged
ww-mw merged 4 commits into
mainfrom
adopt/binary-engine
Oct 30, 2026
Merged

ww-mw merged 4 commits into
mainfrom
adopt/binary-engine

Conversation

@ww-mw

@ww-mw ww-mw commented Oct 30, 2026 •

Copy link
Copy Markdown
Member

Step 6b, V2: the compressed dictionary (binary .sldd) editor becomes a host over view's binary engine (BinaryEngine, data-explorer-view/host, view v1.2.3). The parsed package, the tree, its edits, the Status column's baseline, the save gate, the paints and every gesture are the engine's, lifted in 6a from this file at c73da3c (unchanged since, up to main). This extension keeps VS Code's end of each hook and what is its own: the disk (stamp, save conflict, watcher), the save's sequence (saveSettled), the hot-exit backup, the webview, and the messages every table answers alike.

One behaviour change, named (D44); nothing else is intended to change. The P4 runners give the same saves as main's V0c run, file by file, and the same answers. The screenshot and posted-message A/B is identical. The integration suites are unchanged and pass, apart from the D44 test this PR adds and the two re-pointed assertions below.

Commits:

  1. Pin what a compressed dictionary's save does after an older save lands last. On main's code: a new integration test, binarySlddSaveOrder.test.ts, of today's behaviour, and the seam it needs (the provider's writeUserFile as a property, as askSaveConflict is one; it defaults to diskFollow's). It passes on main's provider in VS Code 1.144.0.
  2. Pin the data-explorer-view v1.2.3 release. Same shape as Pin the data-explorer-view v1.2.0 release #165 and V1's pin. Since v1.2.1, view gained for this PR (Pin core v1.36.5: a class or field named toString no longer stops a file opening #54, Bump to 1.36.7: a name like toString no longer breaks the file that holds it #55, Pin core v1.36.6: a damaged .mat no longer runs the host out of memory, and saves keep what the file had #56): the binary engine's trailerAfterDelivery; opened(view) matched to its paint by identity in both engines (review168's race); BinaryEngine.forTests, a test seam for this extension's suites only; and a dispose that skips a suite's stand-in views.
  3. Host the compressed dictionary editor on view's binary engine. The switch, and the D44 test flipped to the new behaviour.
  4. Spell both editors' setRows in engineTable.ts, where the relay's tests read them (review T1). Both setRows builders, matRows and binaryRows, are exported by engineTable.ts (vscode-free). Each provider hands its own builder over as setRows, which is no longer optional, and engineTable.test.ts posts through the real ones instead of a copy. The literals are unchanged, key for key.

D44: the one behaviour change

  • Before: when a save's write lands after a newer save's (VS Code cancels the first save's token and ignores its result, but its write goes on: a slow or network disk, a large dictionary, a double save or autosave beside Ctrl+S), the file holds the older bytes, and the next save is refused as a fault in the editor: "Refusing to save: entry "gravity" would be written with bytes other than the file's, though no edit changed it. This is a fault in the editor…".
  • After: the engine gives the newer saves' edits back when the older save lands (view e759101, in 1.2.0), so the next save writes the newest edits.
  • The proof: commit 1's test passes on main expecting the refusal; commit 3 flips its last step to "save 3 writes, and the file holds gravity 9.6 and scalarD 42.0", and it passes. The flow: edit gravity to 9.7 and save with the write held; cancel; edit gravity to 9.6 and save (that write lands); release the first write (it lands last); edit scalarD to 42 and save.
  • Also changes (D44b), no test: a Save As of a JSON dictionary over a compressed .sldd that this editor has open no longer resets that tab's Modified marks. The baseline is now the engine's, not the uri-keyed map the JSON editor shares (open-questions 16). VS Code's Save As dialog is its only way in.
  • What the test drives: the provider, as VS Code calls it: saveCustomDocument with a token the test cancels, as VS Code does when it starts a newer save (review T2). Driving VS Code's own save pipeline instead (workspace.save(uri) twice with the first write held, then the tab's isDirty) is not possible without a production hook. activate exposes nothing. The suites' providers are their own instances in the test bundle, not the registered one. A real tab's edits arrive only from its webview, which the extension host cannot script. So the suite pins the provider's behaviour, and VS Code's overlap (it cancels the first save's token and ignores its result) rests on settledSave.ts's note, measured against VS Code 1.143's workbench.
  • Also changes (U1), favourable, no test: the uri-keyed baseline map that the JSON editor uses no longer reaches this editor. Closing a JSON-table tab on the same file therefore no longer wipes this tab's Modified marks (Bump to 1.36.4: binary-dictionary hex values open as what they hold, and sparse arrays show as MATLAB does #47 L1). That route is unreachable in VS Code anyway: the Bump to 1.36.4: binary-dictionary hex values open as what they hold, and sparse arrays show as MATLAB does #47 probe in real VS Code 1.144 opened the JSON table editor (dataExplorer.tableView) on a compressed .sldd, both in a second editor group and in a split. Both times VS Code refused it ("File seems to be binary and cannot be opened as text"), and only the binary tab stayed open. The JSON tab's unregisterSourceDeleter(uri) on close behaves as before.
  • Not fixed, pre-existing (review P1, medium): after such a late landing VS Code shows the document clean over the older file, until the next save, so closing the tab with no prompt loses the newer edit. The MAT-file editor has the same window. D44 makes the next save succeed, but only if the user makes one. This PR stays a no-change switch apart from D44. The fix is for both editors after 6b (open-questions 16): for example, the host saves again when a save lands older than one already landed, or the engine re-baselines from what was written.

What moves where

  • BinarySlddEditorProvider.ts goes from 1,803 lines to 321. It keeps openCustomDocument (the read, the misroute guard, the watcher), resolveCustomEditor, the save, Save As, revert and backup. The webview's messages still go through dispatchCommon and the provider's own dispatch chain (no try around it, as before), whose branches now call the engine's table.
  • binaryDocument.ts (new) is the document: the engine plus VS Code's end of its hooks, as matDocument.ts is for the MAT-file editor.
    • EditSink: onDidChangeCustomDocument, with VS Code's undo/redo commands.
    • EditIndex: view's EditStackIndex.
    • notify: V1's mapping (this engine says only a transfer's warning).
    • registers: hubRegisters(), shared with the MAT and JSON editors until V4.
    • No usage: this editor fills no Usage column.
    • It keeps what the host owns: stamp, conflict state, the saving flag. A file whose bytes are no zip never builds an engine (misrouted).
    • The suites' seam: chunkXml, pushEdit, resetParts, views and modified(node) forward to BinaryEngine.forTests; zipMeta reads engine.parts. No bracket access.
  • engineTable.ts gains the binary editor's timing and spelling:
    • awaitRows: false: this editor's post() never awaited its rows, so the rows' post (through postOrReport), the trailer and drainNavigateSelect go out in one run, and ready is answered in the run that asked, by a repaint;
    • setRows: each editor's own literal, exported from engineTable.ts (binaryRows: {type, docUri, rows, columns, columnLabels, columnGroups, editable, warnings}, no notice or add; matRows, V1's), which the provider hands over;
    • opened(view): both relays now pass the delivered view, so the engine pairs a trailer with its own paint.
  • diskConflict.ts: EditStackMirror is now export { EditStackIndex as EditStackMirror } from view/host, the same rule line for line, so diskConflict.test.ts and settledSave.test.ts stay unchanged. The name goes in V6.
  • Kept, shared until V3–V4: xmlStructuralEdit (binary-only in src/, but 10 unit test files import it; it goes with the structuralEdit family in V4), slddBaseline (the JSON editor), transfer, clipboardAction, clipMarks, refusal, editTextRequest, entryOps and structuralEdit.

Test changes, and why

The vscode integration suites are unchanged except for these two files:

  • test-integration/suite/binarySlddSaveOrder.test.ts (new in commit 1, flipped in commit 3) is D44's test, above. Its name says the behaviour, so it changes with it: "… and the next save is refused as a fault in the editor" becomes "… and the next save writes the newest edits".
  • test-integration/suite/binarySlddSaveGate.test.ts, two assertions (lines 178 and 228). Both read the save's baseline with slddBaseline.computeModified(uri, root). The engine keeps its own baseline and no longer feeds that map, so the call would answer [] whatever the save did: a vacuous pass. Both now read the engine's (doc.modified(root), which forwards to BinaryEngine.forTests.modified; it throws if no baseline was ever taken). The test names, the expected [] and the messages are unchanged.
    • Mutation check, run in VS Code on this branch's build: with the capture of the engine's baseline after a save taken out of view's dist (reBaseline's this.baseline.capture(node)), :228 fails with "No baseline has been taken for binary_save_race_copy.sldd." (its test opens no panel, so the save's baseline is the only one), and the suite's other 3 tests pass. Put back, the bundles are byte-identical to before. The old read could not have caught it: nothing feeds slddBaseline's map for this file any more, and computeModified answers [] for a file it has no baseline for.
    • :178's test saves without an edit, so it cannot tell a missing re-baseline from a kept one; it checks that the save registered the tree it read and that this tree reads unmodified. :228 is the one that would catch a save that forgot to re-baseline.
    • No other suite reads a baseline the engine took over: every other private read of a binary document (chunkXml, zipMeta, srcId, views, repaintAll, pushEdit, resetParts) reaches the engine through the shell, and computeModified is left only in a comment of twoTabBaseline (the JSON editor's case).

Unit tests (vitest) that read this extension's source follow the code there, with their names kept unless said:

  • test/binarySaveGate.test.ts, the describe "writeTo — the provider's own half of the save gate" becomes "the save's own half of the gate: the engine's beginSave, before the save's first await". Its two tests keep their names. "refuses a chunk that does not read back, in the words the user is shown" runs the engine (forTests.setChunk of an unreadable chunk, then beginSave throws "Refusing to save: the document did not re-parse (…)."), where it read the literal from the provider's source. "reads the chunk, gates it and zips it with no await between" checks that beginSave is one synchronous call and that writeDocument calls it before its first await.
  • test/messageDispatch.test.ts, "every setRows says which document it is describing": a provider that calls engineTable is read with the one builder it names (setRows: binaryRows) from engineTable.ts, where both literals are now spelled. Still 4 posts, each with docUri.
  • test/multiSelectInvariants.test.ts, " pastes and drops through runTransfer": a provider on an engine is checked to hand paste and drop to the engine's table (which runs view's runTransfer, pinned in view); the JSON editor's check is unchanged.
  • test/slddParts.test.ts: "and the two files that navigate the part take the names from core" becomes "and the file that navigates the part takes the names from core", for jsonEntryScan.ts only. The binary editor no longer navigates the package; the engine reads and writes it through core's readBinarySlddPackage/writeBinarySlddPackage.
  • vitest.config.ts adds src/host/binaryDocument.ts to the coverage exclude list (it imports vscode), as V1 did for matDocument.ts.

Inventory

  • 2,223 tests (main, 419b43f) become 2,227, by name: 5 renamed, 4 new, none removed or moved. The inventory compare (main against this branch) reports 0 missing apart from the 5 renames, and 0 no longer passing.
  • Renamed: binarySaveGate's two cases (their describe, above); slddParts' "and the file that navigates the part takes the names from core"; viewPin's two cases now name v1.2.3.
  • New:
    • engineTable.test.ts (2), the binary editor's mode: a paint posts the rows, then in the same run what follows them (sectionRules, clipboardState), then the drain, with the keys of the real binaryRows, in order; rows it cannot deliver say "Failed to show" after what follows them, as the unawaited post did;
    • editStackMirror.test.ts (1): EditStackMirror is view's EditStackIndex;
    • integrationSleeps.test.ts (1, generated per suite file): binarySlddSaveOrder.test.ts sleeps for no fixed time of its own.

Gate

  • npm run verify: typecheck (bench included), both builds, 138 files and 2,227 tests, and check:leak.
  • npm run test:integration (with the runners' environment set): 0 failing in either run. With the MAT and gesture runners set: 461 passing, 3 pending. With editorSaveSweep set: 460 passing, 4 pending. Each run's pending tests are the runners not set in it, plus the DEX_GATE_FIXTURES live gate (pending on main too). Both add up to 464 tests: the 463 of main's V0c run plus the D44 test.
  • Goldens 159 of 159; the full export (469 files) is identical to main's.
  • Screenshot + posted-message A/B, main 419b43f against this branch, same machine and inputs: 5,712 of 5,712 screenshots identical (12 skipped, as on main); all 5,724 scenarios' posted messages identical; shells identical, and the webview bundles byte-identical. (Run before the SaveGate re-point, on view Pin core v1.36.6: a damaged .mat no longer runs the host out of memory, and saves keep what the file had #56's tree, which v1.2.3 tags unchanged. The A/B renders the webview bundles over the goldens' payloads; on the final build, 6ee2919, the webview bundles are again byte-identical to main's, and the goldens and the full export were re-run equal.)
  • vsce ls: the same 136 files as main.
  • Bundles: dist/webview is byte-identical. The host bundles grow by 14,222 B each (desktop and web) at commit 3, and by 15 B more at commit 4 (the exports):
    • view's binaryView.js +11,698 B and binaryEngine.js +6,221 B, against the provider's −11,372 B;
    • core's edit modules the engine uses (entryOps, structuralEdit, xmlStructuralEdit, dictionaryPaste, BinarySlddPackage), about +6.4 KB, bundled beside this extension's copies (still used by the JSON editor) until V4; this extension's xmlStructuralEdit leaves the bundle (−960 B; its tests stay);
    • view's slddBaseline and dictionaryTarget, +759 B;
    • binaryDocument.ts +1,908 B, engineTable.ts +112 B, diskConflict.ts −495 B;
    • the rest is minifier renames.

P4, through the real editors (#166's runners), on this branch

On commit 3's build (6ee2919). Commit 4 moves the two setRows builders and changes no literal: in the built bundle both still read the same keys in the same order. So P4 was not rerun on it; npm run verify and CI's integration suite were. The output is in /tmp/eb-run-v2 (MAT and gestures) and /tmp/eb-run-v2b (plain edits).

set vs main's V0c run (/tmp/eb-run-v0c-main) vs the 6a references
MAT edits 2,455 of 2,455 equal: answers, and saves byte for byte 2,455 byte-equal to /tmp/eb-run-p4mat
MAT gestures 201 of 201 equal: 172 written (bytes and added names), 29 refused in the same words gesture-core: 172 byte-equal with the same added names; the same 29 refused in the same words
sldd gestures (365 JSON + 358 binary) 723 of 723 EQUAL (compare-gesture) 660 of 660 vs the replay (p4sldd); 723 of 723 vs real VS Code (p4real, with the 62 cross-document moves)
JSON undo / redo 350 undo saves equal the original, 350 redo saves equal the gesture's save as main
binary undo / redo 345 + 345 (zip members, LastMod masked) as main
plain JSON edits 2,434 of 2,434 EQUAL (compare-p4), 63 whole files byte-equal 2,434 of 2,434 EQUAL vs /tmp/eb-run-p4c1521, 63 whole files byte-equal
plain binary edits 2,469 of 2,469 EQUAL (zip members in order, LastMod masked) 2,469 of 2,469 EQUAL vs /tmp/eb-run-p4c1521

Run 2's first attempt was cut short when every VS Code window on the machine was closed by hand mid-sweep; it was rerun whole on the same build, and these are the rerun's numbers.

Risks, data loss first, and what holds each

  • R1, the save gate. engine.beginSave() runs synchronously before the save's first await: it reads, gates and zips (level 6) and takes the edited and unchecked marks. On any throw the host calls notSaved, which gives them back. On success, saved holds the next save to the text this one wrote, re-baselines, then repaints from the registered tree. saveSettled reads the same EditStackIndex. Held by binarySlddSaveGate, SaveSafety, SaveLanding, KeepsUntouched, SaveOrder (D44) and P4.
  • R2, revert and reload. A file that is no dictionary throws in engine.replace before anything changes; the host sets the stamp only after it. A failed revert read falls back to revertToSaved(). Held by binarySlddOutsideChange and the revert cases.
  • R3, hot exit. engine.backup() is ungated, level 1; the restore opens dirty with no stamp. Held by the backup and restore cases.
  • R4, the catalog part. A rename carried into the System Composer part travels with the chunk through undo, redo and the save. Held by archRenameCarry.
  • R5, the misroute. A JSON .sldd handed to this editor never builds an engine. Held by the misroute case.
  • R6, the registers. Cross-document moves binary↔JSON↔MAT; the deleter on the last view. Held by crossDocumentMove, matClipboard and the gesture sweep.
  • R7, message shape and timing. The binary setRows keys, with no notice or add; the trailer in the same run as the rows; ready answered at once. Held by the A/B messages and engineTable.test.ts.

For later

  • V3 (JSON) moves the JSON editor onto JsonEngine and takes slddBaseline with it.
  • V4 replaces hubRegisters with view's createRegisters() and deletes the shared copies, xmlStructuralEdit included.
  • V6 maps the engine-logic integration tests to view's unit tests (D48), and retires the EditStackMirror name and the forTests seam's uses with them.
  • After 6b: the clean-over-older-file residual of D44 (bug list).

ww-mw added 4 commits October 30, 2026 14:34
…s last

A test of today's behaviour, ahead of the binary editor's move onto view's
engine: two saves whose writes land out of order. VS Code cancels a save's
token when it starts another save of the same document, and ignores the
first one's result, but the first save's write goes on and can land after
the newer save's. The file then holds the older bytes, and the next save
is refused as a fault in the editor ("Refusing to save: entry "gravity"
would be written with bytes other than the file's"), though the user
only saved twice.

The provider gains writeUserFile as a property, as askSaveConflict is
one, so that the suite can hold the first save's write as a slow disk
would. It defaults to diskFollow's.
v1.2.2 and v1.2.3 add what the compressed dictionary editor needs to move
onto view's binary engine, on ./host only: the engine's
trailerAfterDelivery, opened(view) matched by identity in both engines,
a seam for this extension's integration suites (BinaryEngine.forTests:
setChunk, pushEdit, resetParts, views and modified), and closing a
document closes only its real tables. Nothing changes for apps.

The lockfile resolves the tag to its commit, 618edfc (the ^{} line of
ls-remote), at version 1.2.3, with no integrity field; view's core peer
is still ^1.54.0. viewPin.test.ts moves with it.
BinarySlddEditorProvider is now a host over data-explorer-view's
BinaryEngine: the parsed package, the tree, its edits, the Status
column's baseline, the save gate, the paints and every gesture are the
engine's, lifted from this file. What stays is VS Code's end of each
hook and what is this extension's own: the disk (its stamp, a save
conflict, the watcher), the save's sequence, the hot-exit backup, the
webview and the messages every table answers alike.

- binaryDocument.ts: the document, the engine with VS Code's end of its
  hooks (the edit stack and VS Code's Undo and Redo, view's
  EditStackIndex for its position and save point, notices, the
  registers), and the engine's forTests seam that the integration
  suites read and write through (chunkXml, pushEdit, the views, the
  Status column's set).
- engineTable.ts: an editor whose post() did not await its rows
  (awaitRows: false) posts them and what follows them in one run, and
  answers ready at once, as this editor's post() did. A setRows of the
  editor's own spelling (binaryRows) replaces the MAT-file editor's.
- diskConflict.ts: EditStackMirror is view's EditStackIndex, the same
  rule, under its old name until the suites are rebalanced.

One behaviour changes, on purpose: a save whose write lands after a
newer save's leaves the file holding the older bytes, and the next save
was refused as a fault in the editor ("Refusing to save: entry ... would
be written with bytes other than the file's"). The engine gives the
newer saves' edits back when the older save lands, so the next save
writes the newest edits. binarySlddSaveOrder.test.ts, which pinned the
refusal, now pins that.

Tests that follow the code:
- binarySlddSaveGate.test.ts reads a save's baseline from the engine
  (modified(root), which throws if no baseline was taken) where it read
  slddBaseline's computeModified, which the engine no longer feeds and
  so would answer [] whatever the save did. Names and expected values
  are unchanged.
- binarySaveGate.test.ts checks the save's own half of the gate in the
  engine's beginSave, and that writeDocument calls it before its first
  await.
- messageDispatch, multiSelectInvariants and slddParts read the
  provider's source; they follow it to engineTable's own setRows, the
  engine table's paste and drop, and jsonEntryScan.ts alone.
- editStackMirror.test.ts: EditStackMirror is EditStackIndex.
…s read them

The compressed dictionary editor's setRows builder lived in its provider,
which imports vscode, so engineTable.test.ts pinned a copy of it: a key
reordered or added in the provider's literal would have passed that test.
Both builders, matRows and binaryRows, are now exported by engineTable.ts
(vscode-free), each provider hands its own over as `setRows` (no longer
optional), and the test posts through the real ones and pins their keys
in order. The literals are unchanged, key for key.

messageDispatch.test.ts's "every setRows says which document it is
describing" reads, with a provider that calls engineTable, the one
builder it names in engineTable.ts. Still four posts, each with docUri.
@ww-mw
ww-mw merged commit cc8cb1b into main Oct 30, 2026
4 checks passed
@ww-mw
ww-mw deleted the adopt/binary-engine branch October 30, 2026 21:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant