UserEdit document can exceed MongoDB's 16 MB limit, permanently blocking goal/step writes

[imnasnainaec]

Describe the bug

Once a user's UserEdit document for a project grows close to MongoDB's hard
16 MB per-document limit, every subsequent goal/step write to it fails with a 500,
and the user is permanently unable to record new work in that project.

Reads still succeed, so the document is not corrupt — the failure is purely on write.

Observed in production (Honeycomb):

• Route: POST v1/projects/{projectId}/useredits/{userEditId}
• http.response.status_code: 500
• error.type: MongoDB.Driver.MongoWriteException
• exception.message: WriteError { Code: 17419, Message: "Resulting document after update is larger than 16777216" }
• Stack: UserEditRepository.Replace (line 94) ← UserEditService.AddGoalToUserEdit (line 55) ← UserEditController.UpdateUserEditGoal (line 136)

The affected UserEdit had 500+ entries in its edits array, sitting just under 16 MB;
the payload to add was only 78 bytes.

Root cause

Two design factors combine:

1. The edits array is unbounded (UserEdit.Edits), and each Edit carries a
   changes JSON blob plus stepData. It grows with every goal a user completes and
   is never trimmed.
2. Writes rewrite the entire document. AddGoalToUserEdit, AddStepToGoal, and
   UpdateStepInGoal all load the full UserEdit, mutate the array in memory, and call
   UserEditRepository.Replace → ReplaceOneAsync with the whole document. When the
   already-large document plus one more edit crosses 16 MB, Mongo rejects the write.

Because it's a full-document replace, there is no way for the user to make further
progress in that project once the threshold is reached — the array can only grow.

To Reproduce

1. As a single user, accumulate a very large number of goal edits in one project
   (or seed a UserEdit whose edits array approaches 16 MB).
2. Attempt to start/advance another goal (POST to .../useredits/{userEditId}) or a
   step (PUT to the same route).
3. Backend returns 500; Honeycomb shows MongoWriteException code 17419.

Expected behavior

Recording goals/steps should not fail as edit history accumulates. The stored history
should be bounded (or stored such that no single document can hit the 16 MB limit), and
writes should not require rewriting the whole document.

Proposed fixes (for discussion)

• Short term: replace the append-then-Replace pattern with an atomic $push +
  $slice that caps the edits array length server-side, bounding growth going forward.
• Consider trimming/limiting what is stored in each Edit.changes.
• Longer term: move edits to their own collection (one document per edit) so no single
  document is bounded by the 16 MB limit. (Schema migration.)

Note on already-affected data

At least one production UserEdit is already at the limit and needs its edits array
trimmed in place to unblock the user. Trim in place — do not delete the document, since
the user's WorkedProjects map references its _id.

Environment

• Component: Backend (UserEditController / UserEditService / UserEditRepository)
• Server: production (thecombine.app)
• Database: MongoDB (16 MB BSON document limit)

[imnasnainaec]

4 user edit documents were over 14MB (with 387-976 array items). Backed all 4 up to external file, then trimmed in-place to the several hundred most recent edits.

There remain 2 user edit documents over 8MB.

[imnasnainaec]

mongosh 1-liner to find any documents over 8 MB:

let targetDb = db.getSiblingDB('CombineDatabase'); targetDb.getCollectionNames().forEach(c => { targetDb[c].find({ "\u0024expr": { "\u0024gt": [{ "\u0024bsonSize": "\u0024\u0024ROOT" }, 8 * 1024 * 1024] } }, { _id: 1 }).forEach(doc => print('Collection: ' + c + ' | ID: ' + doc._id)); });