Skip to content

[Bug]: interrupted FileSlice extraction marks an incomplete file as fully cached #4943

Closed

Description

@CBEPX

What happened

An interrupted extraction can cache one successful FileSlice as the complete parent file.
A later semantic cache check then reports no uncached files, although later slices never completed.
This can silently omit content after the interruption.

Expected: a strict cache check must keep the parent file uncached until every required slice completes successfully.

Environment and scope

  • Reproduced twice with the installed graphifyy==0.9.82 package, Python 3.13.11, macOS arm64, installed through uv tool.
  • The example uses the native extract_corpus_parallel and check_semantic_cache functions.
  • Only extract_files_direct is mocked. No model request, API key, provider response, or private source is required.
  • The first call returns a complete result for its slice. The second call raises KeyboardInterrupt.
  • This is an installed-package repro. I did not run the current upstream checkout or its full test suite.

Steps to reproduce

Run this script with the Python interpreter that contains Graphify.

import tempfile,json
from pathlib import Path
from unittest.mock import patch
from importlib.metadata import version
from graphify import llm
from graphify.cache import check_semantic_cache
with tempfile.TemporaryDirectory() as directory:
 root=Path(directory);p=root/'guide.md';p.write_text('# Early\nEarly fact.\n'+'filler '*6000+'\n# Tail\nTAIL_ASSERTION must remain visible.\n');calls=[]
 def endpoint(files,**kwargs):
  calls.append(len(files))
  if len(calls)>1:raise KeyboardInterrupt()
  return {'nodes':[{'id':'early','label':'Early fact.','source_file':str(p),'source_location':'Early'}],'edges':[],'hyperedges':[],'input_tokens':1,'output_tokens':1,'finish_reason':'stop','model':'fixture'}
 try:
  with patch.object(llm,'extract_files_direct',endpoint):llm.extract_corpus_parallel([p],backend='openai',root=root,cache_root=root/'out',max_concurrency=1,token_budget=1,max_retry_depth=0)
 except KeyboardInterrupt:pass
 nodes,edges,hyp,miss=check_semantic_cache([str(p)],root=root,cache_root=root/'out',prompt=llm._extraction_system())
 entries=list((root/'out/graphify-out/cache/semantic').rglob('*.json'));data=json.loads(entries[0].read_text()) if entries else {}
 print(json.dumps({'version':version('graphifyy'),'attempted_chunks':len(calls),'successful_chunks':1,'cache_entries':len(entries),'partial_flag':data.get('partial',False),'warm_uncached_files':len(miss),'warm_nodes':len(nodes),'tail_present':any('TAIL_ASSERTION' in str(n) for n in nodes)}))

Actual output

{"version": "0.9.82", "attempted_chunks": 2, "successful_chunks": 1, "cache_entries": 1, "partial_flag": false, "warm_uncached_files": 0, "warm_nodes": 1, "tail_present": false}

The displayed successful_chunks: 1 value is a fixture constant, not a separate measured counter.
This repro exercises only the serial branch (max_concurrency=1).

The cache contains one entry with partial_flag: false.
warm_uncached_files: 0 means the native checker treats the entire file as cached.
The tail assertion is absent because extraction stopped before its slice completed.

Source investigation

The current v8 source at commit 32e5efe935e6844ef39a5e33409e4efe848f2172 appears to retain the relevant mechanism.
This is source inspection, not a runtime reproduction on that commit.

_checkpoint_chunk maps each slice to its parent path with unit_path.
It calls save_semantic_cache with merge_existing=True.
Its partial set comes from _partial_source_files(result), which does not establish completion of sibling slices.
A successful individual slice can therefore produce an authoritative entry for an incomplete file.

A possible fix is to track completion per parent file and keep interim checkpoints partial.
Any fix also needs native API finalization and final-save protection when one required slice fails.
Existing entries from interrupted runs may need targeted invalidation because their metadata can falsely claim completeness.
These are proposed fix requirements. No patch is included.

Related issues checked

I checked the README Troubleshooting section and its linked documentation before reporting this issue.

Activity

  1. github-actions commented on Nov 2, 2026

    @github-actions

    Thanks for opening this issue, @CBEPX. A maintainer will take a look soon.

    If you would like to discuss it in real time, come say hi on our Discord server. For longer-form questions and ideas there is also GitHub Discussions.

    To help us triage, please make sure the report includes what you expected, what actually happened, and the steps (and a small sample) to reproduce it.

  2. CBEPX commented on Nov 2, 2026

    @CBEPX
    Author

    Retested with the released 0.9.100 wheel. The serial interruption case still reproduces.

    Artifact: graphifyy-0.9.100-py3-none-any.whl from the GitHub release.
    SHA-256: 577d917d1282a87cc9e31e237b0073bf7e0f4f258503a698f38d8e0af4eed072.
    The imported package reported 0.9.100. Its module path pointed to the extracted wheel.

    I ran the same synthetic fixture with existing dependencies in an isolated process.
    The wrapper blocked socket.socket.connect and mocked only the extraction endpoint.
    The fixture used no authentication, model request, or private corpus.

    {"version": "0.9.100", "attempted_chunks": 2, "successful_chunks": 1, "cache_entries": 1, "partial_flag": false, "warm_uncached_files": 0, "warm_nodes": 1, "tail_present": false}

    successful_chunks: 1 remains a fixture constant, not an independently measured counter.
    The scope is the serial case only (max_concurrency=1).
    I did not test the parallel branch or a complete fresh installation.
    The warm cache check still treats the incomplete parent file as fully cached.

  3. safishamsi commented on Nov 2, 2026

    @safishamsi
    Member

    Fixed in 0.9.101 (published to PyPI) via #4947. A sliced file is now cached only once every slice has succeeded, and failed slices are stamped for retry.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions