Repository navigation
Add experimental::GetDbIdentityFromManifest() (#15696) - #15696
omkarhgawde wants to merge 1 commit into
Conversation
|
@omkarhgawde has exported this pull request. If you are a Meta employee, you can view the originating Diff in D131983435. |
✅ clang-tidy: No findings on changed linesCompleted in 461.7s. |
9f0570e to
0e56835
Compare
Summary: The DB ID (`DB::GetDbIdentity()`) identifies a DB and its physical copies. RocksDB records it in every MANIFEST file that it writes while `write_dbid_to_manifest` is true, which is the default. A caller that has a MANIFEST file but no DB open on it has no public way to read that DB ID. One such caller keeps a secondary instance on copies of a source DB's files and refreshes it from checkpoints of the source DB. When a new checkpoint has a MANIFEST file with another name, the caller must tell whether it comes from the same DB (the source DB reopened or rolled its MANIFEST) or from another DB, and today it can do that only by parsing the MANIFEST format itself. This change adds `experimental::GetDbIdentityFromManifest(fs, manifest_path, &db_id)`, next to `experimental::GetFileChecksumsFromCurrentManifest()`. It takes the path of a MANIFEST file rather than a DB directory, so it also works for a MANIFEST file with no CURRENT file next to it, such as a copy in a checkpoint store. It reads records with `log::Reader`, verifying record checksums, decodes each record as a `VersionEdit`, and returns the DB ID of the first record that has one. RocksDB writes that record first in each new MANIFEST file, so a call usually reads one record. - A MANIFEST file that records no DB ID (written with `write_dbid_to_manifest=false`) is read to its end, and the call returns `NotFound`. - A read error, a record that fails its checksum, or a record that cannot be decoded returns that error. - The function does not write, so it works through a read-only `FileSystem`. Like `GetFileChecksumsFromCurrentManifest()`, it can read the MANIFEST file of an open DB. Bonus: the code that opens a MANIFEST file for a `log::Reader` moves into a helper that both functions use. Its reporter keeps the first error with `Status::UpdateIfOk()`, which also checks each later report. Previously, `GetFileChecksumsFromCurrentManifest()` left a second report in one `ReadRecord()` call unchecked, which aborts an `ASSERT_STATUS_CHECKED` build. Other builds see no behavior change. Differential Revision: D131983435
Claude Code Review - OBSOLETESuperseded by a newer AI review. Expand to see the original review.✅ Claude Code ReviewAuto-triggered after CI passed — reviewing commit 0e56835 SummaryClean, well-scoped PR that adds a useful read-only utility following existing patterns. The refactoring of High-severity findings (0): Full review (click to expand)Findings🟡 MEDIUMM1.
|
| Context | Affected? | Analysis |
|---|---|---|
| Concurrent MANIFEST writes | Partial | log::Reader with default kTolerateCorruptedTailRecords returns false on truncated tail — function returns NotFound if no DB ID was found yet. Documented and acceptable. |
| Unknown VersionEdit tags | Safe | DecodeFrom handles unknown tags via kTagSafeIgnoreMask. kDbId has this bit set. Forward-compatible. |
| Non-MANIFEST files | Safe | Reader will fail on checksum or decode, function propagates the error. |
Positive Observations
- Good refactoring:
ManifestReporterwithUpdateIfOkfixes ASSERT_STATUS_CHECKED bug inGetFileChecksumsFromCurrentManifest. - Thorough test coverage: happy path (open/reopened/closed DB), error paths (no DB ID, corruption, missing file), edge cases (later record, corruption blocking later records).
- Well-documented API with clear behavior descriptions for all outcomes.
- Justified API design choice to take
manifest_pathdirectly instead ofdbname.
ℹ️ About this response
Generated by Claude Code.
Review methodology: claude_md/ci_review_prompt.md
Limitations:
- Claude may miss context from files not in the diff
- Large PRs may be truncated
- Always apply human judgment to AI suggestions
Commands:
/claude-review [context]— Request a code review/claude-query <question>— Ask about the PR or codebase
Summary: The DB ID (`DB::GetDbIdentity()`) identifies a DB and its physical copies. RocksDB records it in every MANIFEST file that it writes while `write_dbid_to_manifest` is true, which is the default. A caller that has a MANIFEST file but no DB open on it has no public way to read that DB ID. One such caller keeps a secondary instance on copies of a source DB's files and refreshes it from checkpoints of the source DB. When a new checkpoint has a MANIFEST file with another name, the caller must tell whether it comes from the same DB (the source DB reopened or rolled its MANIFEST) or from another DB, and today it can do that only by parsing the MANIFEST format itself. This change adds `experimental::GetDbIdentityFromManifest(fs, manifest_path, &db_id)`, next to `experimental::GetFileChecksumsFromCurrentManifest()`. It takes the path of a MANIFEST file rather than a DB directory, so it also works for a MANIFEST file with no CURRENT file next to it, such as a copy in a checkpoint store. It reads records with `log::Reader`, verifying record checksums, decodes each record as a `VersionEdit`, and returns the DB ID of the first record that has one. RocksDB writes that record first in each new MANIFEST file, so a call usually reads one record. - A MANIFEST file that records no DB ID (written with `write_dbid_to_manifest=false`) is read to its end, and the call returns `NotFound`. - A read error, a record that fails its checksum, or a record that cannot be decoded returns that error. - The function does not write, so it works through a read-only `FileSystem`. Like `GetFileChecksumsFromCurrentManifest()`, it can read the MANIFEST file of an open DB. Limitations: an empty or truncated MANIFEST file also returns `NotFound`, because `log::Reader` in its default mode treats a record truncated at the end of the file as the end of the file. `RepairDB()` gives the DB a new DB ID, so MANIFEST files from before and after a repair record different DB IDs. Unlike DB recovery, the function does not retry a read with `verify_and_reconstruct_read` after it finds corruption. There is no C or Java binding. Bonus: the code that opens a MANIFEST file for a `log::Reader` moves into a helper that both functions use. Its reporter keeps the first error with `Status::UpdateIfOk()`, which also checks each later report. Previously, `GetFileChecksumsFromCurrentManifest()` left a second report in one `ReadRecord()` call unchecked, which aborts an `ASSERT_STATUS_CHECKED` build. Other builds see no behavior change. Reviewed By: xingbowang Differential Revision: D131983435
0e56835 to
6036cde
Compare
✅ Claude Code ReviewAuto-triggered after CI passed — reviewing commit 6036cde SummaryClean, well-scoped addition of a read-only utility that follows existing patterns. The refactoring of the shared ManifestReporter/NewManifestReader helper is a nice improvement that also fixes an ASSERT_STATUS_CHECKED issue. Test coverage is thorough with good edge-case handling. High-severity findings (0): No high-severity findings. Full review (click to expand)Findings🔴 HIGHNone. 🟡 MEDIUMM1.
|
| Context | Does code execute? | Assumptions hold? | Action needed? |
|---|---|---|---|
| Open DB (concurrent MANIFEST writes) | Yes | Yes — DB ID is first record, sequential reads of completed data are safe | None |
| ReadOnly DB / secondary instance | Yes | Yes — read-only by design | None |
| WritePreparedTxnDB | N/A (standalone utility) | N/A | None |
| User-defined timestamps | N/A | N/A | None |
| MANIFEST with write_dbid_to_manifest=false | Yes | Yes — returns NotFound | None |
Assumption stress-test results:
-
"Damaged records are never skipped" — Verified correct. The
while (ReadRecord(...) && s.ok())pattern ensures the reporter-set error prevents processing later records. TestGetDbIdentityFromManifestInLaterRecordexplicitly covers this. -
"UpdateIfOk is equivalent to the old pattern" — Verified correct.
UpdateIfOkkeeps first error and marks discarded statuses as checked, fixing ASSERT_STATUS_CHECKED builds. -
"It can read the MANIFEST file of an open DB" — Verified correct. The DB ID is the first record, already written and stable before the DB is operational.
Refactoring correctness: The GetFileChecksumsFromCurrentManifest refactoring is behaviorally equivalent (same file open parameters, reader config, reporter semantics) with the ASSERT_STATUS_CHECKED fix as bonus.
Positive Observations
- Thorough test coverage with 4 tests covering happy path, errors, corruption-then-valid, and truncation/undecodable cases
- Good use of
SCOPED_TRACEfor parameterized truncation sizes - Corruption handling correctly prevents returning a DB ID after damaged data
- Clean helper extraction reduces duplication
- ReadOnlyFileSystem test verifies the read-only contract
- Well-written, thorough header documentation
- Appropriate release note length
ℹ️ About this response
Generated by Claude Code.
Review methodology: claude_md/ci_review_prompt.md
Limitations:
- Claude may miss context from files not in the diff
- Large PRs may be truncated
- Always apply human judgment to AI suggestions
Commands:
/claude-review [context]— Request a code review/claude-query <question>— Ask about the PR or codebase
Summary:
The DB ID (
DB::GetDbIdentity()) identifies a DB and its physical copies. RocksDB records it in every MANIFEST file that it writes whilewrite_dbid_to_manifestis true, which is the default. A caller that has a MANIFEST file but no DB open on it has no public way to read that DB ID. One such caller keeps a secondary instance on copies of a source DB's files and refreshes it from checkpoints of the source DB. When a new checkpoint has a MANIFEST file with another name, the caller must tell whether it comes from the same DB (the source DB reopened or rolled its MANIFEST) or from another DB, and today it can do that only by parsing the MANIFEST format itself.This change adds
experimental::GetDbIdentityFromManifest(fs, manifest_path, &db_id), next toexperimental::GetFileChecksumsFromCurrentManifest(). It takes the path of a MANIFEST file rather than a DB directory, so it also works for a MANIFEST file with no CURRENT file next to it, such as a copy in a checkpoint store. It reads records withlog::Reader, verifying record checksums, decodes each record as aVersionEdit, and returns the DB ID of the first record that has one. RocksDB writes that record first in each new MANIFEST file, so a call usually reads one record.write_dbid_to_manifest=false) is read to its end, and the call returnsNotFound.FileSystem. LikeGetFileChecksumsFromCurrentManifest(), it can read the MANIFEST file of an open DB.Limitations: an empty or truncated MANIFEST file also returns
NotFound, becauselog::Readerin its default mode treats a record truncated at the end of the file as the end of the file.RepairDB()gives the DB a new DB ID, so MANIFEST files from before and after a repair record different DB IDs. Unlike DB recovery, the function does not retry a read withverify_and_reconstruct_readafter it finds corruption. There is no C or Java binding.Bonus: the code that opens a MANIFEST file for a
log::Readermoves into a helper that both functions use. Its reporter keeps the first error withStatus::UpdateIfOk(), which also checks each later report. Previously,GetFileChecksumsFromCurrentManifest()left a second report in oneReadRecord()call unchecked, which aborts anASSERT_STATUS_CHECKEDbuild. Other builds see no behavior change.Reviewed By: xingbowang
Differential Revision: D131983435