[seayang-nv]

Summary

Adding Part 2 dev note on ABCD (agent conversation) experiments.
This PR includes the second part of a two post series.

• First post describes the ABCD dataset, how to transform the data shape to fit NSS requirements, training/generation, utility and privacy analysis. (PR 774)
• Second post includes prompt only (in-context learning), hyper-parameter sweep and privacy/utility tradeoffs, and data augmentation. (This PR)

Some of the reviews were provided in PR 774

Proposal doc

Human review

Human review

Select exactly one. The default is no human review; uncheck it when choosing another level.

• ⚪ No human review; automated review only
• 🟡 Partially reviewed or spot-checked by a human
• 🔵 Complete diff reviewed by a human
• 🟢 Complete diff reviewed and verified by a human

Verification performed:

Pre-Review Checklist

Ensure that the following pass:

• mise run format leaves only intended formatting changes.
• mise run check ::: test passes locally.
• mise run test:e2e passes locally
• mise run test:ci-container passes locally (recommended)
• GPU CI status check passes -- comment /sync on this PR to trigger a run (auto-triggers on ready-for-review)

Pre-Merge Checklist

• New or updated tests for any fix or new behavior
• Updated documentation for new features and behaviors, including docstrings for API docs.

Other Notes

• Closes #

Summary by CodeRabbit

• Documentation
  • Added a post detailing three experiments with synthetic ABCD conversations, including comparisons of prompt-only generation and fine-tuning, results across 12 configuration settings, and the effect of adding rare-action prefixes.
  • The post reports methods, metrics, trade-offs, and commands for generating and filtering enriched data.
  • Updated Part 2 references in the replay-filtering discussion and “Coming Up in Part 2” section to link to the new post.

[coderabbitai]

Walkthrough

The pull request adds a post about synthetic ABCD conversation experiments, including measured utility, replay, and rare-action enrichment results. It also updates two Part 2 references in the related post to link to the new post.

Changes

ABCD synthetic-conversation experiments

Layer / File(s)	Summary
Experiment report and related-post links docs/dev-notes/posts/tuning-synthetic-conversations-abcd.md, docs/dev-notes/posts/abcd-conversations-to-synthetic-training-data.md	The new post documents baseline metrics, prompting and configuration results, and rare-action enrichment outcomes. Two Part 2 references in the related post now link to it.

Priority: Low

Estimated code review effort: 2 (Simple) | ~10 minutes

Change: Other

Suggested reviewers: binaryaaron

Merge Risk: Low · up to 4e3b4

The documentation is mergeable with a bounded reproducibility gap; record the experiment seeds before relying on the reported measurements.

Pre-merge checks: 5

Check name	Status	Explanation
Description Check		Check skipped - CodeRabbit’s high-level summary is enabled.
Title check		The title clearly and concisely describes the addition of the ABCD synthetic agent conversation Part 2 documentation post.
Docstring Coverage		No eligible functions found in the changed files. Skipping docstring coverage check. Docstring coverage is scoped to functions touched by this diff with cyclomatic complexity of at least 2. Documentat…
Linked Issues check		Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check		Check skipped because no linked issues were found for this pull request.

Full details: Docstring Coverage

Explanation

No eligible functions found in the changed files. Skipping docstring coverage check. Docstring coverage is scoped to functions touched by this diff with cyclomatic complexity of at least 2. Documentation-only edits and deletions within surviving functions are included; unchanged diff context and fully deleted functions are excluded. Analyzed 0 functions across 0 files. (2 skipped: 2 unsupported.)

Finishing Touches

🧪 Generate unit tests (beta)

• Commit to this branch
• Create a new PR

• Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

_{Comment @coderabbitai help to get the list of available commands.}

[greptile-apps]

Confidence Score: 5/5

[Low impact] The PR appears safe to merge, with all three previous findings addressed and no new blocking issue found.

Summary

Adds Part 2 of the ABCD experiment notes and links it from Part 1.

• The prompt experiment shows whether examples can replace fine-tuning.
• The tuning study maps utility against replay across model settings.
• The enrichment study adds rare-action examples with their dialogue context.

The full PR diff was reviewed because the previous and current commits have different merge bases with the target commit. PNG contents could not be checked because image input is unavailable.

_{Reviews (4) · Last reviewed commit: "docs: add ABCD synthetic agent conversat..." · Reviewed by Greptile}

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

[coderabbitai]

Actionable comments posted: 1

---

ℹ️ Review info

⚙️ Run configuration

• Configuration used: Repository: NVIDIA-NeMo/Safe-Synthesizer/.coderabbit.yaml
• Review profile: CHILL
• Plan: Enterprise
• Run ID: cee9df1c-b263-4cc2-8e70-9c4234d44f6f

📥 Commits

Reviewing files that changed from the base of the PR and between c8e38ed and 92ccd0b.

⛔ Files ignored due to path filters: 12

• docs/dev-notes/posts/assets/abcd-conversations-to-synthetic-training-data/action-mix.png is excluded by !**/*.png
• docs/dev-notes/posts/assets/abcd-conversations-to-synthetic-training-data/conversation-length.png is excluded by !**/*.png
• docs/dev-notes/posts/assets/abcd-conversations-to-synthetic-training-data/downstream-utility.png is excluded by !**/*.png
• docs/dev-notes/posts/assets/abcd-conversations-to-synthetic-training-data/membership-inference.png is excluded by !**/*.png
• docs/dev-notes/posts/assets/tuning-synthetic-conversations-abcd/action-long-tail.png is excluded by !**/*.png
• docs/dev-notes/posts/assets/tuning-synthetic-conversations-abcd/enrichment-relative-gains.png is excluded by !**/*.png
• docs/dev-notes/posts/assets/tuning-synthetic-conversations-abcd/knob-effects.png is excluded by !**/*.png
• docs/dev-notes/posts/assets/tuning-synthetic-conversations-abcd/prompt-budget.png is excluded by !**/*.png
• docs/dev-notes/posts/assets/tuning-synthetic-conversations-abcd/prompting-diagnostics.png is excluded by !**/*.png
• docs/dev-notes/posts/assets/tuning-synthetic-conversations-abcd/prompting-results.png is excluded by !**/*.png
• docs/dev-notes/posts/assets/tuning-synthetic-conversations-abcd/targeted-actions-before-after.png is excluded by !**/*.png
• docs/dev-notes/posts/assets/tuning-synthetic-conversations-abcd/utility-replay-operating-points.png is excluded by !**/*.png

📒 Files selected for processing: 4

• docs/css/style.css
• docs/dev-notes/posts/abcd-conversations-to-synthetic-training-data.md
• docs/dev-notes/posts/tuning-synthetic-conversations-abcd.md
• docs/hooks.py

Included review availability: This review used your included allowance. Your plan provides up to 12 included reviews per hour; 10 remain after this review.

📜 Review details

⏰ Context from checks skipped due to timeout: 5

• GitHub Check: Unit Tests (3.12)
• GitHub Check: Unit Tests (3.13)
• GitHub Check: Unit Tests (3.14)
• GitHub Check: Unit Tests (3.11)
• GitHub Check: Greptile Review

🧰 Additional context used

📚 Code guidelines: 2

CONTRIBUTING.md — configured
STYLE_GUIDE.md — configured

📓 Path-based instructions: 7

Review documentation as MkDocs Material content.

⚙️ CodeRabbit configuration file

Files:

• docs/css/style.css
• docs/dev-notes/posts/abcd-conversations-to-synthetic-training-data.md
• docs/hooks.py
• docs/dev-notes/posts/tuning-synthetic-conversations-abcd.md

Review as a senior maintainer for NeMo Safe Synthesizer.

⚙️ CodeRabbit configuration file

Files:

• docs/css/style.css
• docs/dev-notes/posts/abcd-conversations-to-synthetic-training-data.md
• docs/hooks.py
• docs/dev-notes/posts/tuning-synthetic-conversations-abcd.md

Source excerpt: All source files (`.py`, `.sh`, `.yaml`, `.yml`, `.md`) require SPDX copyright headers.

📄 CodeRabbit inference engine (CONTRIBUTING.md)

Files:

• docs/dev-notes/posts/abcd-conversations-to-synthetic-training-data.md
• docs/hooks.py
• docs/dev-notes/posts/tuning-synthetic-conversations-abcd.md

Source excerpt: `__all__` defines the public API surface.

📄 CodeRabbit inference engine (STYLE_GUIDE.md)

Files:

• docs/hooks.py

Source excerpt: Create or edit the `.md` file under the appropriate `docs/` subdirectory.

📄 CodeRabbit inference engine (CONTRIBUTING.md)

Files:

• docs/dev-notes/posts/abcd-conversations-to-synthetic-training-data.md
• docs/dev-notes/posts/tuning-synthetic-conversations-abcd.md

Source excerpt: Documentation pages (in `docs/`): classify as tutorial, how-to, explanation, or reference per the [Diataxis framework](https://diataxis.fr/).

📄 CodeRabbit inference engine (STYLE_GUIDE.md)

Files:

• docs/dev-notes/posts/abcd-conversations-to-synthetic-training-data.md
• docs/dev-notes/posts/tuning-synthetic-conversations-abcd.md

Source excerpt: Use `--` (em-dash) for asides, not `-` (hyphen).

📄 CodeRabbit inference engine (STYLE_GUIDE.md)

Files:

• docs/dev-notes/posts/abcd-conversations-to-synthetic-training-data.md
• docs/dev-notes/posts/tuning-synthetic-conversations-abcd.md

🪛 LanguageTool

docs/dev-notes/posts/abcd-conversations-to-synthetic-training-data.md

[style] ~118-~118: If you’re making a request, consider inserting ‘please’ to sound more courteous and respectful.
Context: ...rance | | hi! i need to return an item, can you help me with that? | return_size | | []...

(CAN_YOU)

docs/dev-notes/posts/tuning-synthetic-conversations-abcd.md

[style] ~84-~84: You have already used ‘right’ in nearby sentences. Consider using an alternative word to let your writing stand out and sound more polished.
Context: ...ing the right action and filling in the right values. One-shot retained 14.99% utilit...

(REP_RIGHT)

---

[style] ~191-~191: Three successive sentences begin with the same word. Consider rewording the sentence or use a thesaurus to find a synonym.
Context: ... With a 2% ceiling, it is S9 at 81.73%. With a 2.5% ceiling, it is Mistral-7B at 89....

(ENGLISH_WORD_REPEAT_BEGINNING_RULE)

---

[style] ~251-~251: Consider a more concise word here.
Context: ...ee the dialogue that leads to an action in order to learn when to take it and which values ...

(IN_ORDER_TO_PREMIUM)

🪛 markdownlint-cli2 (0.23.3)

docs/dev-notes/posts/abcd-conversations-to-synthetic-training-data.md

[warning] 148-148: Code block style
Expected: fenced; Actual: indented

(MD046, code-block-style)

---

[warning] 176-176: Code block style
Expected: fenced; Actual: indented

(MD046, code-block-style)

docs/dev-notes/posts/tuning-synthetic-conversations-abcd.md

[warning] 38-38: Code block style
Expected: indented; Actual: fenced

(MD046, code-block-style)

---

[warning] 211-211: Code block style
Expected: indented; Actual: fenced

(MD046, code-block-style)

---

[warning] 220-220: Code block style
Expected: indented; Actual: fenced

(MD046, code-block-style)

---

[warning] 239-239: Code block style
Expected: indented; Actual: fenced

(MD046, code-block-style)

🔇 Additional comments: 4

docs/hooks.py: 1

29-32: LGTM!

Also applies to: 36-36, 38-40

docs/css/style.css: 1

79-97: LGTM!

docs/dev-notes/posts/abcd-conversations-to-synthetic-training-data.md: 1

1-380: LGTM!

docs/dev-notes/posts/tuning-synthetic-conversations-abcd.md: 1

1-295: LGTM!

[coderabbitai]

Actionable comments posted: 1

---

Review info

Run configuration

• Source mode: Auto
• Repository: NVIDIA-NeMo/Safe-Synthesizer
• Branch: seayang/abcd-dev-notes-part2
• File: .coderabbit.yaml
• Commit: 4e3b443
• Layers: Repository YAML
• Review profile: CHILL
• Plan: Enterprise
• Run ID: 39919a9c-68e1-4068-802d-111bdfe5a228

Commits

Reviewing files that changed from the base of the PR and between 97a155b and 4e3b443.

Files ignored due to path filters: 8

• docs/dev-notes/posts/assets/tuning-synthetic-conversations-abcd/action-long-tail.png is excluded by !**/*.png
• docs/dev-notes/posts/assets/tuning-synthetic-conversations-abcd/enrichment-relative-gains.png is excluded by !**/*.png
• docs/dev-notes/posts/assets/tuning-synthetic-conversations-abcd/knob-effects.png is excluded by !**/*.png
• docs/dev-notes/posts/assets/tuning-synthetic-conversations-abcd/prompt-budget.png is excluded by !**/*.png
• docs/dev-notes/posts/assets/tuning-synthetic-conversations-abcd/prompting-diagnostics.png is excluded by !**/*.png
• docs/dev-notes/posts/assets/tuning-synthetic-conversations-abcd/prompting-results.png is excluded by !**/*.png
• docs/dev-notes/posts/assets/tuning-synthetic-conversations-abcd/targeted-actions-before-after.png is excluded by !**/*.png
• docs/dev-notes/posts/assets/tuning-synthetic-conversations-abcd/utility-replay-operating-points.png is excluded by !**/*.png

Files selected for processing: 2

• docs/dev-notes/posts/abcd-conversations-to-synthetic-training-data.md
• docs/dev-notes/posts/tuning-synthetic-conversations-abcd.md

Files skipped from review as they are similar to previous changes: 1

• docs/dev-notes/posts/abcd-conversations-to-synthetic-training-data.md

Included review availability: This review used your included allowance. Your plan provides up to 12 included reviews per hour; 11 remain after this review.

Review details

Context from checks skipped due to timeout: 2

• GitHub Check: CodeRabbit
• GitHub Check: Greptile Review

Additional context used

Code guidelines: 3

CONTRIBUTING.md — configured
STYLE_GUIDE.md — configured
.cursor/rules/writing-docs.mdc — configured

Path-based instructions: 8

Review documentation as MkDocs Material content.

CodeRabbit configuration file

Files:

• docs/dev-notes/posts/tuning-synthetic-conversations-abcd.md

Review as a senior maintainer for NeMo Safe Synthesizer.

CodeRabbit configuration file

Files:

• docs/dev-notes/posts/tuning-synthetic-conversations-abcd.md

Source excerpt: All source files (`.py`, `.sh`, `.yaml`, `.yml`, `.md`) require SPDX copyright headers.

CodeRabbit inference engine (CONTRIBUTING.md)

Files:

• docs/dev-notes/posts/tuning-synthetic-conversations-abcd.md

Applicability reference: AGENTS.md: # AGENTS.md See [STYLE_GUIDE.md](STYLE_GUIDE.md) for detailed code style conventions (Python, markdown, Dockerfiles, shell scripts, testing, config files, docstrings).

CodeRabbit inference engine (STYLE_GUIDE.md)

Files:

• docs/dev-notes/posts/tuning-synthetic-conversations-abcd.md

Source excerpt: Create or edit the `.md` file under the appropriate `docs/` subdirectory.

CodeRabbit inference engine (CONTRIBUTING.md)

Files:

• docs/dev-notes/posts/tuning-synthetic-conversations-abcd.md

Source excerpt: Each page should fit ONE type.

CodeRabbit inference engine (.cursor/rules/writing-docs.mdc)

Files:

• docs/dev-notes/posts/tuning-synthetic-conversations-abcd.md

Source excerpt: All documentation lives under `docs/`.

CodeRabbit inference engine (CONTRIBUTING.md)

Files:

• docs/dev-notes/posts/tuning-synthetic-conversations-abcd.md

Applicability reference: AGENTS.md: # AGENTS.md See [STYLE_GUIDE.md](STYLE_GUIDE.md) for detailed code style conventions (Python, markdown, Dockerfiles, shell scripts, testing, config files, docstrings).

CodeRabbit inference engine (STYLE_GUIDE.md)

Files:

• docs/dev-notes/posts/tuning-synthetic-conversations-abcd.md

LanguageTool

docs/dev-notes/posts/tuning-synthetic-conversations-abcd.md

[style] ~84-~84: You have already used ‘right’ in nearby sentences. Consider using an alternative word to let your writing stand out and sound more polished.
Context: ...ing the right action and filling in the right values. One-shot retained 14.99% utilit...

(REP_RIGHT)

---

[style] ~191-~191: Three successive sentences begin with the same word. Consider rewording the sentence or use a thesaurus to find a synonym.
Context: ... With a 2% ceiling, it is S9 at 81.73%. With a 2.5% ceiling, it is Mistral-7B at 89....

(ENGLISH_WORD_REPEAT_BEGINNING_RULE)

---

[style] ~255-~255: Consider a more concise word here.
Context: ...ee the dialogue that leads to an action in order to learn when to take it and which values ...

(IN_ORDER_TO_PREMIUM)

markdownlint-cli2 (0.23.3)

docs/dev-notes/posts/tuning-synthetic-conversations-abcd.md

[warning] 38-38: Code block style
Expected: indented; Actual: fenced

(MD046, code-block-style)

---

[warning] 211-211: Code block style
Expected: indented; Actual: fenced

(MD046, code-block-style)

---

[warning] 221-221: Code block style
Expected: indented; Actual: fenced

(MD046, code-block-style)

---

[warning] 240-240: Code block style
Expected: indented; Actual: fenced

(MD046, code-block-style)

Additional comments: 1

docs/dev-notes/posts/tuning-synthetic-conversations-abcd.md: 1

38-38: 📐 Maintainability & Code Quality

Reconcile the repeated MD046 warnings.

markdownlint-cli2 reports MD046 for the fenced YAML, Python, text, and Bash blocks. The documentation instruction requires fenced MkDocs Material code blocks, so do not convert them to indented blocks without checking the applicable docs override. Confirm whether MD046 is blocking in the repository check; if it is, add a docs-specific override or adjust the lint configuration.

Also applies to: 211-211, 221-221, 240-240

Sources: Path instructions, Linters/SAST tools