Skip to content

DOC-6792 add agent experience-notes system: /reflect capture + /finalize distiller#3550

Merged
andy-stark-redis merged 8 commits into
mainfrom
DOC-6792-investigate-ai-experience-notes-in-commit-messages
Jun 26, 2026
Merged

DOC-6792 add agent experience-notes system: /reflect capture + /finalize distiller#3550
andy-stark-redis merged 8 commits into
mainfrom
DOC-6792-investigate-ai-experience-notes-in-commit-messages

Conversation

@andy-stark-redis

@andy-stark-redis andy-stark-redis commented Jun 25, 2026

Copy link
Copy Markdown
Contributor

Note: even if you think this is looking OK, it's probably best not to start using it yet. I'll test run it on a real doc PR before recommending and documenting it :-)

What this is

A skill-based system for recording agent experience notes — the lived reasoning behind a
change (what was learned, what was rejected, what a future editor must not break) — in git
itself
rather than a standalone lessons file. Body prose carries the reflection (for humans
and the semantic history bot); git trailers carry atomic, queryable facts.

It chooses our own thin vocabulary over Lore
(arXiv:2603.15566): Lore's trailer mechanism is sound, but its CLI/vocabulary are young, and
it has no field for process lessons — which is our Learned: addition.

The three-tier pipeline

/reflect          →  WIP commit messages         [episodic, provisional, disposable]
PR review + bots  →  PR comments                 [external critique]
        ↓  /finalize, just before squash
/finalize         →  squash commit message        [durable, location-reachable]
        ↓  fork
promotion         →  learning skills / memory      [cross-cutting, always-loaded]
  • .claude/skills/reflect/ — captures a note into a commit message (or a marked PR comment
    if already pushed). A worth-it gate keeps it silent on mechanical commits.
  • .claude/skills/finalize/ — reconciles the notes across a PR's commits and the review
    feedback into the one commit that survives the squash; proposes promoting cross-cutting
    lessons to learning skills/memory. Prepares the message + gh pr merge command; does not
    merge.
  • .claude/skills/_shared/commit-trailers.md — the single source of truth for the trailer
    vocabulary, subject convention, format rules, and the author-note marker. Both skills
    reference it so capture and distillation can't drift.

Design notes for reviewers

  • Capture lands in commit messages, not PR comments — no PR exists at the first commit, the
    note binds to its diff, and author reflection stays distinct from reviewer critique.
  • Core vocabulary: Constraint / Rejected / Directive / Learned; situational
    Reversibility / Gaps / Ticket / Recheck. A field earns a slot only if reading it
    would change a future agent's decision.
  • /finalize reconciles to the end state, it doesn't concatenate — notes the PR later
    overturned are dropped, not frozen onto main. It reads the arc oldest-first (--reverse).
  • Squash mechanics: the repo squashes via COMMIT_MESSAGES, so /finalize overrides the
    body explicitly with gh pr merge --squash --body-file rather than relying on the default.

Dogfooded on its own commits

Every commit here uses the convention on itself; the worth-it gate fired yes on the
substantive commits and no on a doc tweak. /finalize has been dry-run against this PR's
own arc and correctly dropped discharged directives and refined an overturned Rejected.
Several commits are Bugbot fixes — notably a chain of cross-skill contract bugs that surfaced
only once both skills existed.

Not done / follow-ups

  • Not yet exercised on a real content PR — validated only against this self-referential PR
    about the skills themselves. First non-self test is the next docs change that runs the
    pipeline.
  • No automated merge-time backstop/finalize is human-triggered by design; an
    optional GitHub Action backstop is possible later (it would also need --body-file).
  • Long-term, these may move into the docs: plugin beside assess-comments.

Ticket: DOC-6792


Note

Low Risk
Documentation-only Claude skill definitions; no application, CI, or merge behavior changes until humans invoke the skills.

Overview
Adds a three-tier pipeline for persisting agent reasoning in git: /reflect captures provisional notes in WIP commit bodies and trailers; /finalize reconciles those notes plus PR review into the squash commit and hands off gh pr merge --squash --body-file (it does not merge).

Introduces .claude/skills/_shared/commit-trailers.md as the single source for trailer vocabulary (Constraint, Rejected, Directive, Learned, etc.), DOC-XXXX subject rules, parseable trailer blocks, and the <!-- reflect-note --> marker so author notes in PR comments are not treated as reviewer critique.

/finalize is defined to reconcile to end state (oldest-first git log --reverse, drop overturned WIP notes) rather than concatenate WIP messages—important because the repo’s squash default is COMMIT_MESSAGES. Both skills split location-bound notes (commit trailers) from cross-cutting lessons (proposed promotion to learning skills/memory).

Reviewed by Cursor Bugbot for commit 1eeb2ff. Bugbot is set up for automated code reviews on this repo. Configure here.

…ages

Adds a skill that captures the lived reasoning behind a change into the
commit message — body prose for the reflection, trailers for atomic facts —
so a future agent gets it back via git log/blame at the spot it matters.
Capture lands in commit messages rather than PR comments: there's no PR at
the first commit, the note binds to its diff, and author-side reflection
stays distinct from reviewer critique. WIP notes are deliberately disposable,
so squash isn't a problem to fight — it's the point where a later distiller
compresses them into the one surviving commit. The merge-time bot is a
backstop only: the context-rich durable write wants an agent in the loop.

Learned: WIP-commit immutability is harmless once notes are treated as disposable; that's what unblocked choosing commits over PR comments as the capture medium.
Rejected: PR-comments-as-capture | chicken-and-egg before a PR exists, and it muddles author notes with reviewer critique
Constraint: durable trailers are written once at squash, after review — WIP notes must never be promoted to "final" mid-cycle
Directive: when the distiller is built, factor the trailer vocabulary into one shared reference both skills read, or they will drift
Gaps: not yet run on a real content PR; the trailer vocabulary is unvalidated against notes written in anger
Ticket: DOC-6792
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@github-actions

github-actions Bot commented Jun 25, 2026

Copy link
Copy Markdown
Contributor

DOC-6792

Clarifies the verify section: downstream readers (distiller, dashboards,
git log queries) need %(trailers:...,unfold) or folded multi-line values
come back wrapped. Diff is self-explanatory — no experience note (gate: skip).

Ticket: DOC-6792
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@CLAassistant

Copy link
Copy Markdown

CLA assistant check
Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
You have signed the CLA already but the status is still pending? Let us recheck it.

@jit-ci

jit-ci Bot commented Jun 25, 2026

Copy link
Copy Markdown

🛡️ Jit Security Scan Results

CRITICAL HIGH MEDIUM

✅ No security findings were detected in this PR


Security scan by Jit

@andy-stark-redis andy-stark-redis marked this pull request as ready for review June 25, 2026 14:25
Comment thread .claude/skills/reflect/SKILL.md Outdated
andy-stark-redis and others added 3 commits June 25, 2026 15:44
Bugbot flagged that the location-bound vs cross-cutting sort ran *after* the
note was already committed, so a "this belongs in a learning skill, not the
commit" verdict couldn't be honoured without a rewrite. Swapped the steps: the
sort (now Step 3) decides the destination first, and only the location-bound
part is carried into the landing step (now Step 4). A note can split — part to
the commit, part promoted — which is why the decision has to come first.

Learned: the step that picks a note's destination must run before the step that writes it; deciding after the commit can't be honoured without rewriting history.
Rejected: commit-then-sort | the sort decides where the note goes, so it cannot follow the write
Ticket: DOC-6792
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Acts on the Directive left on ce920c2: before adding /finalize (the second
consumer of the trailer vocabulary), factor the vocabulary, subject convention,
format rules and unfold guidance into one shared reference so capture and
distillation can't drift. /reflect now points at _shared/commit-trailers.md
instead of inlining its own copy. Also fixed a stale "see Step 4" cross-ref
left behind by the earlier step reorder.

Learned: the moment to extract shared config is when the second consumer is created — not after both already hold copies, because that's the point at which drift starts.
Directive: /finalize must reference _shared/commit-trailers.md, never inline its own vocabulary copy
Ticket: DOC-6792
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Adds the squash-time distiller: it reconciles the /reflect notes across a PR's
commits with the review feedback into the one commit message that survives the
squash, and proposes promoting cross-cutting lessons to learning skills/memory.
References the shared vocabulary rather than inlining it, per the directive on
de582b4. Prepares the distilled message and the gh merge command but stops
short of merging.

Learned: distillation is reconciliation, not concatenation — the hard part is dropping WIP notes the PR later overturned, which only reading the commit/comment arc in order reveals.
Rejected: auto-merge inside /finalize | reconciliation is judgment-heavy and a squash-merge is hard to reverse, so the human stays the trigger
Constraint: the distilled body must be passed via `gh pr merge --squash --body-file` — the repo's COMMIT_MESSAGES squash default would otherwise concatenate every raw WIP note onto main
Ticket: DOC-6792
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Comment thread .claude/skills/reflect/SKILL.md Outdated
Comment thread .claude/skills/finalize/SKILL.md Outdated
Bugbot caught two real cross-skill issues. (1) /reflect's "already pushed"
fallback promised the squash-time distiller would pick a note up from
"working notes", but /finalize only reads git log and PR comments — an
uncommitted note is invisible to it. The fallback now routes to a PR comment,
a channel /finalize actually reads. (2) /finalize gathered author notes with
the default newest-first git log while Step 2 reconciles oldest-first; added
--reverse so an overturned note can't out-rank the commit that overturned it.

Learned: a promise one skill makes about another's behaviour is a contract — "the distiller will pick this up" is only true for channels /finalize actually reads (commits, PR comments), never uncommitted working notes.
Constraint: /finalize must gather with `git log --reverse` — reconciliation is order-sensitive, and the default newest-first order lets overturned notes win
Ticket: DOC-6792
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Comment thread .claude/skills/finalize/SKILL.md
Comment thread .claude/skills/_shared/commit-trailers.md Outdated
Two more Bugbot findings, both valid. (1) Routing already-pushed notes to PR
comments (the previous fix) left /finalize unable to tell author reflection
from reviewer critique — it treats all comments as critique, so an author note
would be reconciled as a review verdict or dropped. Added a `<!-- reflect-note -->`
marker defined once in the shared reference: /reflect prefixes the comment with
it, /finalize classifies marked comments as author-side (part of the arc) and
unmarked ones as critique. (2) The shared "reading trailers" branch-wide example
omitted --reverse, contradicting /finalize's oldest-first requirement; fixed so
consumers reading only the shared file get the right order.

Learned: opening a new channel is only half a fix — the consumer also needs to classify what arrives on it. Routing author notes to PR comments wasn't done until /finalize could tell them from critique, and a marker is the cheapest classifier.
Constraint: author-reflection PR comments must carry the `<!-- reflect-note -->` marker; /finalize keys its author-vs-critique split on it
Ticket: DOC-6792
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes using default effort and found 2 potential issues.

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, have a team admin enable autofix in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit 7b23623. Configure here.

Comment thread .claude/skills/reflect/SKILL.md Outdated
Comment thread .claude/skills/finalize/SKILL.md
@andy-stark-redis andy-stark-redis self-assigned this Jun 25, 2026
@andy-stark-redis andy-stark-redis added infrastructure AI-friendliness Features that help AI tools understand and use the doc pages more easily. labels Jun 25, 2026
@andy-stark-redis andy-stark-redis changed the title [DO NOT MERGE — feedback only] DOC-6792 add /reflect skill for agent experience notes DOC-6792 add agent experience-notes system: /reflect capture + /finalize distiller Jun 25, 2026
@andy-stark-redis andy-stark-redis requested a review from a team June 25, 2026 15:50
Two more Bugbot findings, both valid. (1) /reflect verified a not-yet-committed
note with `git log -1`, which reads the previous commit — a false pass while the
pending message could have a broken trailer block. Split the guidance: use
`git interpret-trailers --parse <file>` to check a pending message, `git log -1`
only after the note is committed or amended. (2) /finalize's body-file was
described as subject+body+trailers, but Step 5 also passes --subject, so a file
that repeats the subject duplicates the title into the commit body; the body-file
now holds body + trailers only.

Learned: verify the artifact you actually changed, not a proxy — `git log -1` checks HEAD rather than a pending message, and a --body-file that repeats the --subject duplicates the title. Both bugs were the command's target not matching the intended artifact.
Constraint: the /finalize body-file holds the body and trailers only; the subject is passed separately via `gh pr merge --subject`
Ticket: DOC-6792
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

@dwdougherty dwdougherty left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Interesting. No notes.

@andy-stark-redis

Copy link
Copy Markdown
Contributor Author

Interesting. No notes.

@dwdougherty I'm not sure if you mean this is good or bad? (I'm guessing it's at least tolerable since you approved the PR.) Anyway, thanks for the review :-)

@andy-stark-redis andy-stark-redis merged commit 6628c5d into main Jun 26, 2026
73 of 74 checks passed
@andy-stark-redis andy-stark-redis deleted the DOC-6792-investigate-ai-experience-notes-in-commit-messages branch June 26, 2026 09:48
EliShteinman added a commit to EliShteinman/docs that referenced this pull request Jun 26, 2026
Add agent experience-notes system (/reflect + /finalize skills).
@dwdougherty

Copy link
Copy Markdown
Collaborator

Interesting. No notes.

@dwdougherty I'm not sure if you mean this is good or bad? (I'm guessing it's at least tolerable since you approved the PR.) Anyway, thanks for the review :-)

Certainly not bad. I guess "cool" would have been a better word. I think I wrote that when I was in the middle of fisticuffs with Claude. 🤣

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

AI-friendliness Features that help AI tools understand and use the doc pages more easily. infrastructure

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants