Repository navigation
Description
What happened
An interrupted extraction can cache one successful FileSlice as the complete parent file.
A later semantic cache check then reports no uncached files, although later slices never completed.
This can silently omit content after the interruption.
Expected: a strict cache check must keep the parent file uncached until every required slice completes successfully.
Environment and scope
- Reproduced twice with the installed
graphifyy==0.9.82package, Python 3.13.11, macOS arm64, installed throughuv tool. - The example uses the native
extract_corpus_parallelandcheck_semantic_cachefunctions. - Only
extract_files_directis mocked. No model request, API key, provider response, or private source is required. - The first call returns a complete result for its slice. The second call raises
KeyboardInterrupt. - This is an installed-package repro. I did not run the current upstream checkout or its full test suite.
Steps to reproduce
Run this script with the Python interpreter that contains Graphify.
import tempfile,json
from pathlib import Path
from unittest.mock import patch
from importlib.metadata import version
from graphify import llm
from graphify.cache import check_semantic_cache
with tempfile.TemporaryDirectory() as directory:
root=Path(directory);p=root/'guide.md';p.write_text('# Early\nEarly fact.\n'+'filler '*6000+'\n# Tail\nTAIL_ASSERTION must remain visible.\n');calls=[]
def endpoint(files,**kwargs):
calls.append(len(files))
if len(calls)>1:raise KeyboardInterrupt()
return {'nodes':[{'id':'early','label':'Early fact.','source_file':str(p),'source_location':'Early'}],'edges':[],'hyperedges':[],'input_tokens':1,'output_tokens':1,'finish_reason':'stop','model':'fixture'}
try:
with patch.object(llm,'extract_files_direct',endpoint):llm.extract_corpus_parallel([p],backend='openai',root=root,cache_root=root/'out',max_concurrency=1,token_budget=1,max_retry_depth=0)
except KeyboardInterrupt:pass
nodes,edges,hyp,miss=check_semantic_cache([str(p)],root=root,cache_root=root/'out',prompt=llm._extraction_system())
entries=list((root/'out/graphify-out/cache/semantic').rglob('*.json'));data=json.loads(entries[0].read_text()) if entries else {}
print(json.dumps({'version':version('graphifyy'),'attempted_chunks':len(calls),'successful_chunks':1,'cache_entries':len(entries),'partial_flag':data.get('partial',False),'warm_uncached_files':len(miss),'warm_nodes':len(nodes),'tail_present':any('TAIL_ASSERTION' in str(n) for n in nodes)}))Actual output
{"version": "0.9.82", "attempted_chunks": 2, "successful_chunks": 1, "cache_entries": 1, "partial_flag": false, "warm_uncached_files": 0, "warm_nodes": 1, "tail_present": false}The displayed successful_chunks: 1 value is a fixture constant, not a separate measured counter.
This repro exercises only the serial branch (max_concurrency=1).
The cache contains one entry with partial_flag: false.
warm_uncached_files: 0 means the native checker treats the entire file as cached.
The tail assertion is absent because extraction stopped before its slice completed.
Source investigation
The current v8 source at commit 32e5efe935e6844ef39a5e33409e4efe848f2172 appears to retain the relevant mechanism.
This is source inspection, not a runtime reproduction on that commit.
_checkpoint_chunk maps each slice to its parent path with unit_path.
It calls save_semantic_cache with merge_existing=True.
Its partial set comes from _partial_source_files(result), which does not establish completion of sibling slices.
A successful individual slice can therefore produce an authoritative entry for an incomplete file.
A possible fix is to track completion per parent file and keep interim checkpoints partial.
Any fix also needs native API finalization and final-save protection when one required slice fails.
Existing entries from interrupted runs may need targeted invalidation because their metadata can falsely claim completeness.
These are proposed fix requirements. No patch is included.
Related issues checked
- Incremental semantic cache checkpoint fails on sliced docs: _checkpoint_chunk reads FileSlice.rel, which does not exist #1870 fixed a
FileSlice.relaccess error that prevented checkpoints. Here the checkpoint succeeds but incorrectly represents file completeness. - Semantic extraction cannot tell a file with no entities from an omitted file, so empty files arm the shrink guard #4547 concerns empty-file acknowledgements and omitted sources. This repro returns a nonempty successful slice before interruption.
- [Bug]: a .graphify_partial.txt left by an --update can hold a cached doc back in a later full build #4814 concerns stale partial-state instructions. This repro demonstrates a native semantic cache entry that lacks the partial marker.
I checked the README Troubleshooting section and its linked documentation before reporting this issue.
Activity
github-actions commentedon Nov 2, 2026
Thanks for opening this issue, @CBEPX. A maintainer will take a look soon.
If you would like to discuss it in real time, come say hi on our Discord server. For longer-form questions and ideas there is also GitHub Discussions.
To help us triage, please make sure the report includes what you expected, what actually happened, and the steps (and a small sample) to reproduce it.
CBEPX commentedon Nov 2, 2026
Retested with the released 0.9.100 wheel. The serial interruption case still reproduces.
Artifact:
graphifyy-0.9.100-py3-none-any.whlfrom the GitHub release.SHA-256:
577d917d1282a87cc9e31e237b0073bf7e0f4f258503a698f38d8e0af4eed072.The imported package reported 0.9.100. Its module path pointed to the extracted wheel.
I ran the same synthetic fixture with existing dependencies in an isolated process.
The wrapper blocked
socket.socket.connectand mocked only the extraction endpoint.The fixture used no authentication, model request, or private corpus.
{"version": "0.9.100", "attempted_chunks": 2, "successful_chunks": 1, "cache_entries": 1, "partial_flag": false, "warm_uncached_files": 0, "warm_nodes": 1, "tail_present": false}successful_chunks: 1remains a fixture constant, not an independently measured counter.The scope is the serial case only (
max_concurrency=1).I did not test the parallel branch or a complete fresh installation.
The warm cache check still treats the incomplete parent file as fully cached.
fix(llm): stamp sliced-file checkpoints partial when some slices neve…
test(chunking): regression test for interrupted multi-slice checkpoin…
fix(llm): cache a sliced file only once every slice has succeeded (#4943
safishamsi commentedon Nov 2, 2026
Fixed in 0.9.101 (published to PyPI) via #4947. A sliced file is now cached only once every slice has succeeded, and failed slices are stamped for retry.