08. Sampler Overrides

Overview

Clients can send sampler parameters that users are unaware of, or even not send any parameters at all!

Sampler overrides is tabbyAPI's flexible proxy for adding or forcing sampler values before generation.

Setting up

All supported samplers are located in sampler_overrides/sample_preset.yml.

Note

Sampler overrides can also be switched at runtime via the /v1/sampling/override endpoint set. Please read the API docs for more information.

Steps:

1. Create a new YML in the sampler_overrides folder or duplicate sample_preset.yml
2. Adjust the samplers you want to override using the examples below
3. Save and rename the file (if you haven't already)
4. Open config.yml and inside the sampling block, set the sampler_override key to the preset name from step 3

Examples

Let's say a client doesn't send a value for top_p:

top_p:
	override: 0.7
	force: False

This override will change the default fallback of top_p from a neutral value of 1.0 to 0.7.

Now, let's say a client forces a value that you don't like (ex. top-k: 30):

top_k:
	override: 0
	force: True

This override forces top_k to be 0 no matter what the client sends.

Finally, let's say a client sends a list, but you want to add something else:

stop:
	override: ["a"]
	force: False
	additive: True

Now this override appends the given stop string to whatever the client sends.

Per-model presets

A model can carry its own overrides, applied on top of the global ones while that model is loaded. Add a sampling section under model in the model folder's tabby_config.yml (or in a /v1/model/load request); it has the same syntax as the top-level sampling section:

# models/qwen3.8-27b/tabby_config.yml
model:
  sampling:
    override_preset: qwen3.8-27b

Inline overrides

Both layers can also carry overrides written directly in the config instead of, or on top of, a preset file, using the same per-sampler syntax:

# config.yml
sampling:
  override_preset: safe_defaults
  # Applies on top of the preset above, or instead of it if no preset is given
  temperature:
    override: 0.5
    force: false
  min_p:
    override: 0.1
    force: true

# models/qwen3.8-27b/tabby_config.yml
model:
  sampling:
    override_preset: qwen3.8-27b
    top_k:
      override: 20
      force: false

Within a layer an inline entry wins over the same key from the preset. The full resolution order for each parameter is therefore: the request, then the model's inline overrides, then the model preset, then the global inline overrides, then the global override_preset, then TabbyAPI's built-in default. Forced overrides (force: True) in either preset still win over the request, with the model preset taking precedence over the global one for the same key. The model layer is replaced when another model is loaded (including inline loading) and cleared on unload; the /v1/sampling/override endpoints only change the global layer, and /v1/sampling/override/list reports both.

Separate settings for reasoning

Models that reason before answering can use different sampler settings for the reasoning block than for the answer that follows. A model prone to reasoning in circles can, for example, reason at a higher temperature with a phrase banned, while the answer keeps the regular settings.

In a request to the chat completions endpoint, pass the settings that differ as an object:

{
  "temperature": 0.6,
  "reasoning_override": {
    "temperature": 1.0,
    "banned_strings": ["but wait,"]
  }
}

In config.yml, a model's sampling section or a preset file, nest the same override entries under reasoning_override:

sampling:
  override_preset: safe_defaults
  temperature:
    override: 0.5
    force: false
  reasoning_override:
    temperature:
      override: 1.5
      force: true
    temperature_last:
      override: true
      force: true

Anything not listed is inherited from the regular settings. The settings that can be overridden are the sampler parameters (temperature, top-k/p, min-p, penalties, DRY, XTC, adaptive-P, logit bias, banned tokens) and banned_strings. Limits, stop strings and grammars apply to the whole response and are ignored here with a warning.

For each setting the order is: a forced entry under reasoning_override in the config, then the request's reasoning_override, then a plain config entry under reasoning_override, then the regular value. A config entry under reasoning_override therefore outranks a regular value sent with the request, since it is the more specific of the two. The layers combine as above: inline over preset, model over global.

The switch is made when TabbyAPI sees the reasoning block end, so the first few tokens of the answer may still be sampled with the reasoning settings, more so with speculative decoding. A warning is logged if a stop string or banned string contains the end-of-reasoning tag, as that can delay or hide the tag.

Grammar constraints (response_format, json_schema, regex_pattern, grammar_string) follow the same split: they constrain the answer only, and the reasoning block stays free-form.