Add frequency_penalty, presence_penalty, and logit_bias sampling options by stikves · Pull Request #294 · apple/coreai-models

stikves · 2026-09-27T01:22:20Z

Summary

Adds three chat-completions sampling controls: frequency_penalty, presence_penalty, and logit_bias to /v1/chat/completions, wired through every sampling path (CPU sequential/static/VLM engines, the pipelined GPU engine, and guided/constrained generation).

Implementation

Request surface (ServerAPITypes): three new optional fields, validated in ChatHandler — penalties in [-2, 2], logit_bias values in [-100, 100], and token-id keys bounds-checked against the vocabulary; invalid input returns 400.
CPU sampling (SamplingConfiguration, new AdditivePenaltyProcessor): additive, count-scaled frequency penalty; once-per-token presence penalty; and per-token logit bias. These are additive in logit space and distinct from the existing multiplicative repetition penalty.
Pipelined GPU engine (MPSGraphSamplers, new AdditivePenaltyGPUState): a fused additive-delta stage in both the composite and argmax samplers, so the options take effect on the default engine. Greedy decoding with penalties/bias is supported; the per-step delta carries the same N-1 pipeline-delay handling as the repetition penalty.
Guided generation: penalties and bias are applied before the grammar mask, so grammar token-bans remain authoritative.
Cross-request correctness: the pipelined engine's cached sampler is invalidated when the penalty/bias configuration changes across requests that share a prefix (PenaltySignature), preventing one request's options from leaking into the next.

Enabling it

HTTP (chat-completions):

curl http://localhost:8080/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{
    "messages": [{"role": "user", "content": "Write a short limerick about the sea."}],
    "temperature": 0.8,
    "frequency_penalty": 1.2,
    "presence_penalty": 0.5,
    "logit_bias": {"1917": -100}
  }'

Swift (SamplingConfiguration):

let config = SamplingConfiguration(
    temperature: 0.8,
    frequencyPenalty: 1.2,   // additive, count-scaled
    presencePenalty: 0.5,    // additive, once per seen token
    logitBias: [1917: -100]  // token id -> bias
)
// hand `config` to the engine's generate(...) call

Testing

Unit tests cover the additive CPU processor, the GPU delta state, and request decoding.
Validated across an 11-model campaign (agentic harness + cross-request stress) on real hardware. The invariants; determinism, logit_bias forcing, and cross-request non-contamination hold on every model, on both the sequential and pipelined engines. Two Phi variants (phi-3-mini, phi-3.5-mini) show a pre-existing base issue unrelated to this change.

Notes

These are sampler-level controls on the CoreAI engine path. The FoundationModels GenerationOptions API does not expose per-token penalties or logit bias (only temperature, sampling mode, and max tokens), so they are offered through the server and the Swift sampler API.
Guided generation and logprobs require the sequential engine; the pipelined engine samples on-GPU and does not return logits.
On the pipelined engine, frequency/presence penalties carry the engine's N-1 pipeline delay (as the repetition penalty does), so their effect is not bit-identical to the sequential path. This is expected behavior.

Adds the three chat-completions sampling controls to /v1/chat/completions and wires them through every sampling path. - ServerAPITypes: parse frequency_penalty, presence_penalty, logit_bias (token-id-string -> bias map); validated in ChatHandler (penalties in [-2, 2], bias in [-100, 100], token ids bounds-checked; invalid input returns 400). - SamplingConfiguration + AdditivePenaltyProcessor: additive, count-scaled frequency penalty, once-per-token presence penalty, and per-token logit bias, applied on the CPU sampler path (sequential/static/VLM engines). Distinct from the existing multiplicative repetition penalty. - Pipelined GPU engine: a fused additive-delta stage in both the composite and argmax MPSGraph samplers (AdditivePenaltyGPUState), so the params take effect on the default engine. Greedy + penalties/bias is supported. The additive delta carries the same N-1 pipeline-delay handling as the repetition penalty. - Guided generation: penalties/bias applied before the grammar mask so token bans stay authoritative. - Fixes a latent cross-request issue: the pipelined engine's cached sampler is now invalidated when the penalty/bias configuration changes across requests that share a prefix (PenaltySignature). Unit tests cover the CPU processor, the GPU delta state, and request decoding.

carinapeng · 2026-10-08T00:09:19Z

+        self.pipelineDepth = pipelineDepth
+        self.frequencyPenalty = Float(frequencyPenalty)
+        self.presencePenalty = Float(presencePenalty)
+        self.windowSize = windowSize ?? 256


With frequency_penalty set and no window, a 600-token generation for instance applies the penalty to a token used at step 10 when sampled on CPU (constrained decoding) but not on the pipelined GPU engine. The same request produces different output depending on the engine

Let's pick one default and apply it to both paths

the actual divergence fix lives at the engine call sites

kevchengcodes · 2026-10-08T18:40:41Z

        self.logits = logitsRef
        self.cachedSampler = nil
        self.cachedSamplerTemperature = nil
+        self.cachedSamplerPenaltySignature = nil


should we initialize self.additivePenaltyState = nil here too?

kevchengcodes · 2026-10-08T23:06:39Z

+        frequencyPenalty: Double,
+        presencePenalty: Double,


why are these Doubles when we immediately cast them to floats anyway?

The external public API is Double

kevchengcodes · 2026-10-08T23:07:03Z

+        self.frequencyPenalty = Float(frequencyPenalty)
+        self.presencePenalty = Float(presencePenalty)


Can make these optional and default to 0.0 to simplify API surface

kevchengcodes · 2026-10-08T23:11:26Z

+                counts[tokenId, default: 0] += 1
+            }
+            for (tokenId, count) in counts {
+                let idx = Int(tokenId)


feels like a lot of casts in this function. please see which ones we can remove

kevchengcodes · 2026-10-09T00:08:15Z

+            for i in 0..<vocabSize {
+                ptr[i] = Float16(0.0)
+            }


double check this is the fastest way to do this, or that the impact is negligible

kevchengcodes · 2026-10-09T00:11:20Z

+    ///
+    /// Valid range is [-2, 2]. Unlike `repetitionPenalty` (multiplicative, count-agnostic),
+    /// this is additive and scales with occurrence count.
+    public let frequencyPenalty: Double?


this gets cast to float In AdditivePenaltyGPUState, can we just make this float?

All other fields are Double, if needed this should be a future refactor.

Co-authored-by: kevchengcodes <59463423+kevchengcodes@users.noreply.github.com>

- Fix cross-engine divergence (carinapeng): a nil repetitionPenaltyWindow means "all history" per the SamplingConfiguration contract. The CPU path honored it, but the GPU state silently capped at 256, so the same request could diverge by engine. Size the GPU ring to maxContextLength when the window is nil, for both the additive (frequency/presence) and repetition penalties. - Reset penaltyState / additivePenaltyState in the pipelined engine init for symmetry with the other cached-sampler fields. - Default AdditivePenaltyGPUState's frequency/presence/logitBias/window params so callers can omit them. - Replace the neutral additive buffer's per-element Float16 fill with a single memset (0.0 is all-zero bits).

stikves force-pushed the sukru/sampler-options branch 2 times, most recently from a032a4d to d80621f Compare September 27, 2026 01:28

stikves force-pushed the sukru/sampler-options branch from d80621f to 9d679af Compare September 27, 2026 01:36

stikves marked this pull request as ready for review September 27, 2026 01:37

stikves self-assigned this Sep 27, 2026

stikves requested review from alejandro-isaza, carinapeng, guru-desh, pkmandke and tjia1818 September 28, 2026 16:40

stikves added 2 commits October 2, 2026 11:37

Merge branch 'main' into sukru/sampler-options

eed363b

Merge branch 'main' into sukru/sampler-options

8bd7489

carinapeng reviewed Oct 8, 2026

View reviewed changes

kevchengcodes reviewed Oct 9, 2026

View reviewed changes

stikves and others added 2 commits October 8, 2026 18:56

Apply suggestion from @kevchengcodes

add27a9

Co-authored-by: kevchengcodes <59463423+kevchengcodes@users.noreply.github.com>

Navigation Menu

Repository navigation

Add frequency_penalty, presence_penalty, and logit_bias sampling options - #294

Add frequency_penalty, presence_penalty, and logit_bias sampling options#294
stikves wants to merge 5 commits into
apple:mainfrom
stikves:sukru/sampler-options

stikves commented Sep 27, 2026 •

edited

Loading

Uh oh!

carinapeng Oct 8, 2026

Uh oh!

stikves Oct 9, 2026

Uh oh!

stikves Oct 9, 2026

Uh oh!

kevchengcodes Oct 8, 2026

Uh oh!

kevchengcodes Oct 8, 2026

Uh oh!

stikves Oct 9, 2026

Uh oh!

kevchengcodes Oct 8, 2026

Uh oh!

stikves Oct 9, 2026

Uh oh!

Uh oh!

kevchengcodes Oct 8, 2026

Uh oh!

kevchengcodes Oct 9, 2026

Uh oh!

kevchengcodes Oct 9, 2026

Uh oh!

stikves Oct 9, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

		self.frequencyPenalty = Float(frequencyPenalty)
		self.presencePenalty = Float(presencePenalty)

Conversation

stikves commented Sep 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Implementation

Enabling it

Testing

Notes

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

stikves commented Sep 27, 2026 •

edited

Loading