Repository navigation
fix: encode SARIF artifact URIs per RFC 3986 and route batch progress to stderr - #789
dajiaohuang wants to merge 3 commits into
Conversation
… to stderr - Fix NVIDIA#757: Encode file paths in SARIF artifactLocation.uri using urllib.parse.quote to handle special characters (spaces, unicode, etc.) per RFC 3986 - Fix NVIDIA#753: Route all batch scanner progress, header, and status messages to stderr so JSON/markdown output on stdout remains machine-parseable Signed-off-by: dajiaohuang <dajiaohuang@bytedance.com>
yashrajp22
left a comment
There was a problem hiding this comment.
Two paths in these fixes still need attention: batch diagnostics with Rich installed, and SARIF notification locations. Regular/suppressed finding URI encoding and the plain stderr path work.
Verified this commit with fresh BASE/HEAD wheels, source/install identity checks, 48 sample scans, 12 focused SARIF scans, stream/exit probes, and selected existing SARIF/batch tests. Existing tests: BASE 178 passed per mode; HEAD 177 passed and 1 failed per mode. The failure expects the old plain-output error stream and needs its assertion updated.
One BASE/source Markdown batch invocation recorded a worker timeout; I have not attributed it to this PR or claimed complete batch parity. These offline checks do not establish live-provider behavior or global detection accuracy.
| f"{len(skill_dirs)} skill(s) in [dim]{display(root)}[/dim]" | ||
| f" ([cyan]{args.workers} workers[/cyan]{pool_note})\n" | ||
| f" ([cyan]{args.workers} workers[/cyan]{pool_note})\n", | ||
| file=sys.stderr, |
There was a problem hiding this comment.
Could you make _print honor file=sys.stderr when Rich is available? Its Rich branch drops the file argument and uses the stdout-backed Console. With the normal Rich dependency installed, I reproduced --format json emitting the banner/progress before the JSON with empty stderr, so json.loads(stdout) still fails; Markdown has the same prefix. Please route diagnostics through a stderr-backed console and cover both Rich/plain branches. The existing plain-error test also needs to read stderr.
| occurrence = occurrence or {} | ||
| file_path = str(occurrence.get("file", finding.file)).replace("\\", "/").lstrip("/") | ||
| raw_file_path = str(occurrence.get("file", finding.file)).replace("\\", "/").lstrip("/") | ||
| file_path = quote(raw_file_path, safe="/") |
There was a problem hiding this comment.
Could you apply this encoding to invocation notification locations too? _build_sarif.notification_from_exception still constructs SarifArtifactLocation(uri=path) directly. Scanning a malformed file named broken#old.zip produces error/warning notifications whose URI is still broken#old.zip, so SARIF consumers treat old.zip as a fragment instead of part of the filename. I reproduced this from source and the installed wheel; regular finding URIs are correctly encoded. Please use the same path-normalization/encoding policy for notification locations.
Signed-off-by: dajiaohuang <mikewushuwen@outlook.com>
|
Addressed both requested paths. |
Signed-off-by: dajiaohuang <mikewushuwen@outlook.com>
|
Updated the nested archive inspection-limit regression to assert the percent-encoded URI in the SARIF notification location, while retaining the raw display-path assertion for terminal, JSON and Markdown. This addresses the sole unit-test failure in run 37672509169. Validation: 7 relevant report tests passed; Ruff check src/ tests/ passed; formatting check for the changed test passed. The full test suite and live-provider integration tests were not rerun locally. Current-head CI and re-review remain pending. |
Summary
Two minimal fixes:
Fix SARIF artifact URIs do not encode literal filename characters #757: Encode file paths in SARIF
artifactLocation.uriusingurllib.parse.quoteto handle special characters (spaces, unicode, etc.) per RFC 3986. This ensures SARIF consumers correctly parse artifact URIs that contain reserved or non-ASCII characters.Fix Batch scanner mixes progress messages into JSON stdout #753: Route all batch scanner progress, header, and status messages to stderr so JSON/markdown output on stdout remains machine-parseable. Previously, progress lines like
[1/10] skill-name → 80/100 HIGH (2 issues)were printed to stdout, corrupting piped JSON output.Testing
src/skillspector/nodes/report.py: Addedurllib.parse.quoteimport + URI encoding in_sarif_artifact_locationcontrib/batch_scan/batch_scan.py: Addedfile=sys.stderrto all progress/status_printcallsSigned-off-by: dajiaohuang dajiaohuang@bytedance.com