Eval bug: ling 3 tiny (BailingMoeV3 architecture) repeating slashes

[rafasmour]

Name and Version

llama-server --version
version: 0.3.0-dev (build 10621, commit c1d0e7a)
built with GNU 13.3.0 for Linux x86_64

Operating systems

Linux

GGML backends

Vulkan

Hardware

Ryzen 7 1800x
16gb ddr4 ram
Rx 570 8gb

Models

Ling 3 Tiny using q4 and q6 quantinization (Ling-3.0-tiny-Q4_K_M.gguf Ling-3.0-tiny-Q6_K.gguf) from the official ggufs (https://huggingface.co/bloomer010/Ling-3.0-tiny-GGUF) released for the llama cpp pull request that added the model support (#26608).

Problem description & steps to reproduce

After ~30k context when the model calls a tool it starts outputting '/////////////////////////' infinitely, I have firstly installed the Q4_M and then I tested the Q6 to see if this is a quantinization issue. the logs without verbose only show the inference metrics, with the --verbose flag

Arguments:

llama-server \
    --model /models/ling/ling-3-tiny/Ling-3.0-tiny-Q6_K.gguf \
    --alias ling-3-tiny \
    --host 0.0.0.0 \
    --port 8080 \
    --metrics \
    --verbose \
    --jinja \
    --flash-attn on \
    --n-gpu-layers auto \
    --threads 8 \
    --threads-batch 16 \
    --parallel 1 \
    --kv-unified \
    --ctx-size 65536 \
    --batch-size 1024 \
    --ubatch-size 512 \
    --cache-type-k q4_0 \
    --cache-type-v q4_0 \
    --temp 1.0 \
    --top-p 0.95 \
    --top-k 20 \
    --repeat-penalty 1.3 \
    --repeat-last-n 2048

First Bad Commit

No response

Relevant log output

Logs

llama-server  | [51439] 15.50.343.156 D que  yield_to_que: done yielding to queue
llama-server  | [51439] 15.50.344.952 D res          send: sending result for task id = 2229
llama-server  | [51439] 15.50.344.954 D res          send: task id = 2229 pushed to result queue
llama-server  | [51439] 15.50.344.958 D slot process_toke: id  0 | task 2229 | n_gen = 907, n_remaining = -1, next token:    14 '/'
llama-server  | [51439] 15.50.344.959 D que    start_loop: waiting for new tasks
llama-server  | [51439] 15.50.344.960 D que    start_loop: processing new tasks
llama-server  | [51439] 15.50.344.962 D que  process_new_: processing task, id = 3145
llama-server  | [51439] 15.50.344.963 D que    start_loop: update slots
llama-server  | [51439] 15.50.344.964 D srv  update_slots: posting NEXT_RESPONSE
llama-server  | [51439] 15.50.344.966 D que          post: new task, id = 3146, front = 0
llama-server  | [51439] 15.50.344.970 D slot handle_last_: id  0 | task 2229 | slot decode token, id=14, n_ctx = 65536, n_tokens = 23599, truncated = 0
llama-server  | [51439] 15.50.344.973 D set_adapters_lora: adapters = (nil)
llama-server  | [51439] 15.50.344.974 D adapters_lora_are_same: adapters = (nil)
llama-server  | [51439] 15.50.344.974 D set_embeddings: value = 0
llama-server  | [51439] 15.50.344.975 D srv        decode: n_batch (effective) = 1024, off = 0
llama-server  | [51439] 15.50.344.975 D que  yield_to_que: yielding to queue
llama-server  | [51439] 15.50.344.994 D que  process_new_: processing task, id = 3146
llama-server  | [51439] 15.50.344.995 D srv  process_sing: decoding, decline task, id_task = 3146
llama-server  | [51439] 15.50.344.995 D que  process_new_: task declined, id = 3146
llama-server  | [51439] 15.50.345.132 D srv    operator(): http: streamed chunk: data: {"choices":[{"finish_reason":null,"index":0,"delta":{"reasoning_content":"/"}}],"created":1787919924,"id":"chatcmpl-dBm2QdLlXGc0CyhxZfCwt3l1KR5K81b3","model":"ling-3-tiny","system_fingerprint":"b10621-c1d0e7a00","object":"chat.completion.chunk","timings":{"cache_n":14622,"prompt_n":8071,"prompt_ms":204038.612,"prompt_per_token_ms":25.28046239623343,"prompt_per_second":39.55623850254382,"predicted_n":907,"predicted_ms":53292.486,"predicted_per_token_ms":58.821728476821185,"predicted_per_second":17.000520486133823}}
llama-server  | [51439] 
llama-server  | [51439] 
llama-server  | 21.25.924.098 D srv    operator(): http: streamed chunk: data: {"choices":[{"finish_reason":null,"index":0,"delta":{"reasoning_content":"/"}}],"created":1787919924,"id":"chatcmpl-dBm2QdLlXGc0CyhxZfCwt3l1KR5K81b3","model":"ling-3-tiny","system_fingerprint":"b10621-c1d0e7a00","object":"chat.completion.chunk","timings":{"cache_n":14622,"prompt_n":8071,"prompt_ms":204038.612,"prompt_per_token_ms":25.28046239623343,"prompt_per_second":39.55623850254382,"predicted_n":907,"predicted_ms":53292.486,"predicted_per_token_ms":58.821728476821185,"predicted_per_second":17.000520486133823}}