Skip to content

Add experimental::GetDbIdentityFromManifest() (#15696) - #15696

Open
omkarhgawde wants to merge 1 commit into
meta:mainfrom
omkarhgawde:export-D131983435
Open

omkarhgawde wants to merge 1 commit into
meta:mainfrom
omkarhgawde:export-D131983435

Conversation

@omkarhgawde

@omkarhgawde omkarhgawde commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor

Summary:

The DB ID (DB::GetDbIdentity()) identifies a DB and its physical copies. RocksDB records it in every MANIFEST file that it writes while write_dbid_to_manifest is true, which is the default. A caller that has a MANIFEST file but no DB open on it has no public way to read that DB ID. One such caller keeps a secondary instance on copies of a source DB's files and refreshes it from checkpoints of the source DB. When a new checkpoint has a MANIFEST file with another name, the caller must tell whether it comes from the same DB (the source DB reopened or rolled its MANIFEST) or from another DB, and today it can do that only by parsing the MANIFEST format itself.

This change adds experimental::GetDbIdentityFromManifest(fs, manifest_path, &db_id), next to experimental::GetFileChecksumsFromCurrentManifest(). It takes the path of a MANIFEST file rather than a DB directory, so it also works for a MANIFEST file with no CURRENT file next to it, such as a copy in a checkpoint store. It reads records with log::Reader, verifying record checksums, decodes each record as a VersionEdit, and returns the DB ID of the first record that has one. RocksDB writes that record first in each new MANIFEST file, so a call usually reads one record.

  • A MANIFEST file that records no DB ID (written with write_dbid_to_manifest=false) is read to its end, and the call returns NotFound.
  • A read error, a record that fails its checksum, or a record that cannot be decoded returns that error.
  • The function does not write, so it works through a read-only FileSystem. Like GetFileChecksumsFromCurrentManifest(), it can read the MANIFEST file of an open DB.

Limitations: an empty or truncated MANIFEST file also returns NotFound, because log::Reader in its default mode treats a record truncated at the end of the file as the end of the file. RepairDB() gives the DB a new DB ID, so MANIFEST files from before and after a repair record different DB IDs. Unlike DB recovery, the function does not retry a read with verify_and_reconstruct_read after it finds corruption. There is no C or Java binding.

Bonus: the code that opens a MANIFEST file for a log::Reader moves into a helper that both functions use. Its reporter keeps the first error with Status::UpdateIfOk(), which also checks each later report. Previously, GetFileChecksumsFromCurrentManifest() left a second report in one ReadRecord() call unchecked, which aborts an ASSERT_STATUS_CHECKED build. Other builds see no behavior change.

Reviewed By: xingbowang

Differential Revision: D131983435

@meta-cla meta-cla Bot added the CLA Signed label Nov 16, 2026
@meta-codesync

meta-codesync Bot commented Nov 16, 2026

Copy link
Copy Markdown

@omkarhgawde has exported this pull request. If you are a Meta employee, you can view the originating Diff in D131983435.

@github-actions

github-actions Bot commented Nov 16, 2026 •

Copy link
Copy Markdown

✅ clang-tidy: No findings on changed lines

Completed in 461.7s.

@meta-codesync meta-codesync Bot changed the title Add experimental::GetDbIdentityFromManifest() Add experimental::GetDbIdentityFromManifest() (#15696) Nov 16, 2026
omkarhgawde added a commit to omkarhgawde/rocksdb that referenced this pull request Nov 16, 2026
Summary:

The DB ID (`DB::GetDbIdentity()`) identifies a DB and its physical copies. RocksDB records it in every MANIFEST file that it writes while `write_dbid_to_manifest` is true, which is the default. A caller that has a MANIFEST file but no DB open on it has no public way to read that DB ID. One such caller keeps a secondary instance on copies of a source DB's files and refreshes it from checkpoints of the source DB. When a new checkpoint has a MANIFEST file with another name, the caller must tell whether it comes from the same DB (the source DB reopened or rolled its MANIFEST) or from another DB, and today it can do that only by parsing the MANIFEST format itself.

This change adds `experimental::GetDbIdentityFromManifest(fs, manifest_path, &db_id)`, next to `experimental::GetFileChecksumsFromCurrentManifest()`. It takes the path of a MANIFEST file rather than a DB directory, so it also works for a MANIFEST file with no CURRENT file next to it, such as a copy in a checkpoint store. It reads records with `log::Reader`, verifying record checksums, decodes each record as a `VersionEdit`, and returns the DB ID of the first record that has one. RocksDB writes that record first in each new MANIFEST file, so a call usually reads one record.

- A MANIFEST file that records no DB ID (written with `write_dbid_to_manifest=false`) is read to its end, and the call returns `NotFound`.
- A read error, a record that fails its checksum, or a record that cannot be decoded returns that error.
- The function does not write, so it works through a read-only `FileSystem`. Like `GetFileChecksumsFromCurrentManifest()`, it can read the MANIFEST file of an open DB.

Bonus: the code that opens a MANIFEST file for a `log::Reader` moves into a helper that both functions use. Its reporter keeps the first error with `Status::UpdateIfOk()`, which also checks each later report. Previously, `GetFileChecksumsFromCurrentManifest()` left a second report in one `ReadRecord()` call unchecked, which aborts an `ASSERT_STATUS_CHECKED` build. Other builds see no behavior change.

Differential Revision: D131983435
@github-actions

github-actions Bot commented Nov 16, 2026 •

Copy link
Copy Markdown

Claude Code Review - OBSOLETE

Superseded by a newer AI review. Expand to see the original review.

✅ Claude Code Review

Auto-triggered after CI passed — reviewing commit 0e56835


Summary

Clean, well-scoped PR that adds a useful read-only utility following existing patterns. The refactoring of GetFileChecksumsFromCurrentManifest is a net improvement (fixes an ASSERT_STATUS_CHECKED issue). No high-severity findings.

High-severity findings (0):
No high-severity findings.

Full review (click to expand)

Findings

🟡 MEDIUM

M1. *db_id not cleared on entry — db/experimental.cc
  • Issue: GetDbIdentityFromManifest does not clear *db_id on entry. If a caller passes a non-empty string and the function returns NotFound or an error, the caller is left with stale data in *db_id. The sibling function GetFileChecksumsFromCurrentManifest calls checksum_list->reset() on entry.
  • Root cause: Missing db_id->clear() at the top of the function, after null checks.
  • Suggested fix: Add db_id->clear(); after the null-pointer validation, before creating the reader. This matches the pattern of checksum_list->reset() in the sibling function and ensures the caller never sees stale data on any error path.
M2. Missing null check for manifest_path emptiness — db/experimental.cc
  • Issue: The function validates fs and db_id for nullptr but does not check whether manifest_path is empty. An empty path will produce an unhelpful IOError from NewSequentialFile. The sibling GetFileChecksumsFromCurrentManifest has this same gap, so this is consistent, but it's a small usability issue.
  • Root cause: Following existing pattern, which lacks this check.
  • Suggested fix: Not blocking — consistent with existing code. Could optionally add an InvalidArgument check for empty path.

🟢 LOW / NIT

L1. ManifestReporter::status_ptr defensive assert — db/experimental.cc:50
  • Issue: ManifestReporter has Status* status_ptr = nullptr; default and no null check in Corruption(). While status_ptr is always set immediately after construction in practice, a defensive assert(status_ptr) in Corruption() would prevent future misuse.
  • Suggested fix: Add assert(status_ptr); as the first line of ManifestReporter::Corruption().

Cross-Component Analysis

Context Affected? Analysis
Concurrent MANIFEST writes Partial log::Reader with default kTolerateCorruptedTailRecords returns false on truncated tail — function returns NotFound if no DB ID was found yet. Documented and acceptable.
Unknown VersionEdit tags Safe DecodeFrom handles unknown tags via kTagSafeIgnoreMask. kDbId has this bit set. Forward-compatible.
Non-MANIFEST files Safe Reader will fail on checksum or decode, function propagates the error.

Positive Observations

  • Good refactoring: ManifestReporter with UpdateIfOk fixes ASSERT_STATUS_CHECKED bug in GetFileChecksumsFromCurrentManifest.
  • Thorough test coverage: happy path (open/reopened/closed DB), error paths (no DB ID, corruption, missing file), edge cases (later record, corruption blocking later records).
  • Well-documented API with clear behavior descriptions for all outcomes.
  • Justified API design choice to take manifest_path directly instead of dbname.

ℹ️ About this response

Generated by Claude Code.
Review methodology: claude_md/ci_review_prompt.md

Limitations:

  • Claude may miss context from files not in the diff
  • Large PRs may be truncated
  • Always apply human judgment to AI suggestions

Commands:

  • /claude-review [context] — Request a code review
  • /claude-query <question> — Ask about the PR or codebase

Summary:

The DB ID (`DB::GetDbIdentity()`) identifies a DB and its physical copies. RocksDB records it in every MANIFEST file that it writes while `write_dbid_to_manifest` is true, which is the default. A caller that has a MANIFEST file but no DB open on it has no public way to read that DB ID. One such caller keeps a secondary instance on copies of a source DB's files and refreshes it from checkpoints of the source DB. When a new checkpoint has a MANIFEST file with another name, the caller must tell whether it comes from the same DB (the source DB reopened or rolled its MANIFEST) or from another DB, and today it can do that only by parsing the MANIFEST format itself.

This change adds `experimental::GetDbIdentityFromManifest(fs, manifest_path, &db_id)`, next to `experimental::GetFileChecksumsFromCurrentManifest()`. It takes the path of a MANIFEST file rather than a DB directory, so it also works for a MANIFEST file with no CURRENT file next to it, such as a copy in a checkpoint store. It reads records with `log::Reader`, verifying record checksums, decodes each record as a `VersionEdit`, and returns the DB ID of the first record that has one. RocksDB writes that record first in each new MANIFEST file, so a call usually reads one record.

- A MANIFEST file that records no DB ID (written with `write_dbid_to_manifest=false`) is read to its end, and the call returns `NotFound`.
- A read error, a record that fails its checksum, or a record that cannot be decoded returns that error.
- The function does not write, so it works through a read-only `FileSystem`. Like `GetFileChecksumsFromCurrentManifest()`, it can read the MANIFEST file of an open DB.

Limitations: an empty or truncated MANIFEST file also returns `NotFound`, because `log::Reader` in its default mode treats a record truncated at the end of the file as the end of the file. `RepairDB()` gives the DB a new DB ID, so MANIFEST files from before and after a repair record different DB IDs. Unlike DB recovery, the function does not retry a read with `verify_and_reconstruct_read` after it finds corruption. There is no C or Java binding.

Bonus: the code that opens a MANIFEST file for a `log::Reader` moves into a helper that both functions use. Its reporter keeps the first error with `Status::UpdateIfOk()`, which also checks each later report. Previously, `GetFileChecksumsFromCurrentManifest()` left a second report in one `ReadRecord()` call unchecked, which aborts an `ASSERT_STATUS_CHECKED` build. Other builds see no behavior change.

Reviewed By: xingbowang

Differential Revision: D131983435
@github-actions

Copy link
Copy Markdown

✅ Claude Code Review

Auto-triggered after CI passed — reviewing commit 6036cde


Summary

Clean, well-scoped addition of a read-only utility that follows existing patterns. The refactoring of the shared ManifestReporter/NewManifestReader helper is a nice improvement that also fixes an ASSERT_STATUS_CHECKED issue. Test coverage is thorough with good edge-case handling.

High-severity findings (0):

No high-severity findings.

Full review (click to expand)

Findings

🔴 HIGH

None.

🟡 MEDIUM

M1. *db_id not cleared on entry or on error/NotFound paths — db/experimental.cc
  • Issue: The function does not clear *db_id on entry. On error paths (DecodeFrom failure, reporter corruption, IOError from NewManifestReader) and on the NotFound path, *db_id is returned in whatever state the caller provided it. If the caller passes a non-empty string and the function returns NotFound or an error, the caller may mistakenly believe the string contains a valid DB ID. The existing tests happen to pass because they clear db_id before calling, but this is a contract gap.
  • Root cause: Unlike GetFileChecksumsFromCurrentManifest which calls checksum_list->reset() on entry, GetDbIdentityFromManifest does not clear *db_id.
  • Suggested fix: Add db_id->clear() after the nullptr checks, before any operation that might fail. This ensures *db_id is always empty on non-OK return.

🟢 LOW / NIT

L1. Test GetDbIdentityFromManifestErrors does not clear db_id before the corruption test call — db/db_etc2_test.cc
  • Issue: After the NotFound test (ASSERT_TRUE(db_id.empty())), the corruption test does not assert that db_id remains empty after the Corruption return. This is minor since the function doesn't modify *db_id on error, but adding the assertion would document the expected contract. (The GetDbIdentityFromManifestInLaterRecord test does check ASSERT_TRUE(db_id.empty()) after corruption, but only because it explicitly calls db_id.clear() beforehand.)
  • Suggested fix: No action needed if M1 is addressed (clearing on entry).

Cross-Component Analysis

Context Does code execute? Assumptions hold? Action needed?
Open DB (concurrent MANIFEST writes) Yes Yes — DB ID is first record, sequential reads of completed data are safe None
ReadOnly DB / secondary instance Yes Yes — read-only by design None
WritePreparedTxnDB N/A (standalone utility) N/A None
User-defined timestamps N/A N/A None
MANIFEST with write_dbid_to_manifest=false Yes Yes — returns NotFound None

Assumption stress-test results:

  1. "Damaged records are never skipped" — Verified correct. The while (ReadRecord(...) && s.ok()) pattern ensures the reporter-set error prevents processing later records. Test GetDbIdentityFromManifestInLaterRecord explicitly covers this.

  2. "UpdateIfOk is equivalent to the old pattern" — Verified correct. UpdateIfOk keeps first error and marks discarded statuses as checked, fixing ASSERT_STATUS_CHECKED builds.

  3. "It can read the MANIFEST file of an open DB" — Verified correct. The DB ID is the first record, already written and stable before the DB is operational.

Refactoring correctness: The GetFileChecksumsFromCurrentManifest refactoring is behaviorally equivalent (same file open parameters, reader config, reporter semantics) with the ASSERT_STATUS_CHECKED fix as bonus.

Positive Observations

  • Thorough test coverage with 4 tests covering happy path, errors, corruption-then-valid, and truncation/undecodable cases
  • Good use of SCOPED_TRACE for parameterized truncation sizes
  • Corruption handling correctly prevents returning a DB ID after damaged data
  • Clean helper extraction reduces duplication
  • ReadOnlyFileSystem test verifies the read-only contract
  • Well-written, thorough header documentation
  • Appropriate release note length

ℹ️ About this response

Generated by Claude Code.
Review methodology: claude_md/ci_review_prompt.md

Limitations:

  • Claude may miss context from files not in the diff
  • Large PRs may be truncated
  • Always apply human judgment to AI suggestions

Commands:

  • /claude-review [context] — Request a code review
  • /claude-query <question> — Ask about the PR or codebase

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant