[chmjkb]

Description

Phase 3 of #70: runntime quantize, a small Python CLI that writes int8/int4 checkpoints in format v1, the format the loader from #80–#82 reads.

cd tools/python
uv run runntime quantize model.safetensors model.int8.safetensors --bits 8 --f16

• Project: a uv package in tools/python (Python 3.11+, numpy only). pyright strict on src, ruff with a broad rule set, pytest.
• Input: one .safetensors file or a Hugging Face model folder, sharded or not. Output is one .safetensors file. Untouched tensors are copied as stored (BF16 included) and the header metadata is kept, with runntime.quant added.
• Selection: 2D *.weight tensors; embeddings (*embed*), position embeddings (*pos_emb*) and lm_head stay as stored. --include / --exclude globs replace those defaults. A tensor is skipped when it would not load: in features a multiple of 8 and of the group size, out features a multiple of 4 (int8) or 8 (int4), the same rule as Linear.loadQuantized. Tensors that are not 2D (convs, experts) are skipped.
• Options: --group-size defaults to 32, which divides the 288 and 416 inputs of Moonshine Tiny and Base. --f16 stores the tensors that stay float as F16, and fails on a value beyond the f16 range. --drop leaves matching tensors out of the output, like MiniLM's unused pooler.
• Algorithm: per group, scale = absmax / (2^(bits-1) - 1) rounded to f16, int = round(w / scale) (half to even) clamped to the symmetric range. int4 packs the even index in the low nibble.
• Report: one line per tensor, the max and mean absolute error for quantized ones and the one rule a skipped tensor breaks.
• Golden files: tools/python/tests/golden holds a tiny float checkpoint and its int4 and int8 versions. The Python tests check the CLI reproduces them byte for byte and that the official safetensors library reads them; a new test in safetensors.test.ts loads the same files through fromSafetensors and checks every value is within half a step of the float source.

Introduces a breaking change?

• Yes
• No

Type of change

• Bug fix (change which fixes an issue)
• New feature (change which adds functionality)
• Documentation update (improves or adds clarity to existing documentation)
• Performance (kernel tuning, graph building, anything related to performance)
• Other (chores, tests, code style improvements etc.)

Testing instructions

1. cd tools/python && uv sync && uv run pytest (29 pass)
2. uv run ruff check && uv run ruff format --check && uv run pyright
3. pnpm typecheck && pnpm lint && pnpm --filter runntime exec vitest run --project unit test/core/weights/safetensors.test.ts (103 pass, 2 new)

Related issues

Part of #70. Stacked on #82.

Checklist

• I have performed a self-review of my code
• I have commented my code, particularly in hard-to-understand areas
• I have updated the documentation accordingly
• My changes generate no new warnings

Additional notes

• Checked on the real Moonshine Base f32 checkpoint: --bits 8 --f16 writes 81.5 MB, against 123 MB for the f16 file.
• The tool reads and writes safetensors itself (about 40 lines) instead of depending on the safetensors package, whose numpy API cannot read or write BF16. The package is a dev dependency, used to check the output.
• No CI job runs the Python checks yet. Proposed job, to add in this PR once agreed:

  python-tools:
    if: github.repository == 'software-mansion/runntime'
    runs-on: ubuntu-latest
    permissions:
      contents: read
    defaults:
      run:
        working-directory: tools/python
    steps:
      - uses: actions/checkout@v6
      - uses: astral-sh/setup-uv@v7
      - run: uv sync --locked
      - run: uv run ruff check
      - run: uv run ruff format --check
      - run: uv run pyright
      - run: uv run pytest

[xxpopielxx]

If the output path is the same as the input, the input gets wiped. Runntime quantize m.safetensors m.safetensors --bits 8 crashes and leaves an empty file. Maybe write to a temp file first and rename it at the end?

[xxpopielxx]

Running the tool on an already quantized file gives a file runntime can't load: --bits 4 on an int8 file exits 0, but the old int8 weights stay and the header says int4. I guess refuse a source that's already quantized?

[xxpopielxx]

Don't we want to test this in CI too, on every change to tools/python?