ikawrakow asked the seq_rm zeroing be hoisted out of the per-cell loop and
preferably done as part of the compute graph. This rewrites the fix:
- llama_kv_cache gets a pending_recurrent_reset bitmap, sized to
qnext_state_slots and indexed by slot index.
- llama_kv_cache_seq_rm only marks the bitmap; no CPU-side tensor
writes, no work inside the per-cell loop.
- delta_net::build_layer_attn_linear ORs the bitmap into the existing
reset_state flag, so the in-graph ggml_scale(state, 0.0f) path that
already handled batch.pos[0] == 0 now also handles slot reset.
- build_qwen3next and build_qwen35 clear the bitmap after
ggml_build_forward_expand, once per graph build.
- llama_kv_cache_clear also drops the bitmap, since the full-buffer
clear it already does makes any pending in-graph reset redundant.
Previous behavior is preserved (every recurrent layer column for the
released slot is zeroed before the next request reuses it), but the work
moves from the CPU into the compute graph, runs once per graph instead
of once per cell, and reuses the existing reset path referenced in the
review.
When `enable_thinking` is false (e.g. via `chat_template_kwargs: {thinking: false}`),
the Kimi-K2 parser was still extracting reasoning tokens and routing all output to
`reasoning_content` instead of `content`.
The fix aligns with how other thinking models (Qwen3, etc.) handle this: check both
`reasoning_format != NONE` and `enable_thinking` before enabling reasoning extraction.
Fixes#1685
* server: spec checkpoints for recurrent models
* fix: save/restore sampler state during speculative checkpoint
When speculative decoding rejects draft tokens and restores the
recurrent state checkpoint, the sampler (RNG, grammar, prev tokens)
must also be restored to maintain consistency. Without this, the
sampler state reflects the rejected draft tokens, leading to
potential divergence.
Uses common_sampler_clone() to snapshot the sampler before the
speculative batch decode, and restores it on rejection.
* server: snapshot recurrent state in tensor
* reset ngram mod state for rejected tokens
* server: refactor checkpoint state logic
* speculative: fix sampler for checkpoints
* recurrent model: implement recurrent kernel checkpoint
* recurrent model: refactor api
* spec: free rbudget before overwriting
Previously, the end-of-turn token would be added to prompt cache by
`slot.cache_tokens.push_back(slot.sampled)`, without going through
`llama_decode()` first. As such, the token doesn't exist yet in the
actual KV cache. Then, in the next turn, processing of this EOT token
will be skipped since it is already in the prompt cache.
The expected sequence (with Qwen3.5 chat template):
</tool_call><|im_end|>
<|im_start|>user
<tool_response>
The actual sequence that the model sees:
</tool_call>
<|im_start|>user
<tool_response>
As the conversation goes on, the model starts losing the ability to
generate an EOT token after `</tool_call>`, since it hasn't seen this
pattern in the context. With `--parallel-tool-calls`, the grammar
allows `</tool_call>` to be followed by either another tool call or an
EOT token. So, this eventually leads to the model making infinite tool
calls.
We fix this by deferring all the processing of the EOT token to the
next turn.
Fixes#1661
Two related issues that manifest as 'llama_decode ret=-3' on hybrid
architectures (e.g. Qwen3.5/3.6 MoE, Qwen3-Next), matching the symptom
reported in #1576.
1) server_context::apply_checkpoint() was written around transformer KV
semantics (pos_min / pos_max per-token window). For hybrid and pure
recurrent models the per-token pos_min threshold does not apply: the
recurrent state is a single snapshot, and the server-side checkpoint
is a whole-prefix record. The old selector 'cur.pos_min < pos_min_thold'
can succeed on a checkpoint whose pos_max is past the current n_past,
and — more commonly — fall through to do_reset = true, which zeros
slot.n_past / slot.n_past_prompt. Zeroing in-place while the recurrent
state in the context is still populated makes the next decode batch
disagree with the live state, returning ret=-3.
This change gates the checkpoint path on
llama_model_has_recurrent(llama_get_model(slot.ctx)):
- selector uses pos_max <= slot.n_past && pos_max < pos_next
(whole-prefix match, leaves at least one token to decode);
- on miss, slot state is preserved rather than zeroed, letting
update_slots() continue from the already-valid n_past_prompt;
- the erase loop drops any checkpoint whose pos_max > pos_next,
matching the rewind semantics for recurrent state.
Transformer behavior is unchanged.
2) stop_internal_decode is a file-static global in src/llama.cpp, set by
llama_decode_stop() (called on client disconnect) and polled inside
the decode loop to bail out with ret=-3. The flag is only cleared on
one conditional path in server_slot::release(), so a stop signal that
arrives after the interrupted llama_decode() has already returned
bleeds into the NEXT decode call and causes an immediate ret=-3 with
no work performed. Clear it at the top of the public llama_decode()
entry so the signal is scoped to the in-flight decode it was meant
for.
Build-verified: llama-server with GGML_CUDA=ON, -DCMAKE_CUDA_ARCHITECTURES=86
(sm_86), IQK flash-attn + matmul enabled. No new APIs introduced —
llama_model_has_recurrent is already public and already used elsewhere in
server-context.cpp.
Closes#1576
* Autoparser - complete refactoring of parser architecture
Autoparser: add optional argument reshuffle capability
Autoparser: True streaming (#20177)
* Relax atomicity constraint for nicer, more pleasent, True Streaming parsing
* Whitespace
* Remove redundant atomics
Revert to OAI-compatible args (#20213)
* Revert to OAI-compatible args
* Apply workaround::func_args_not_string
Fix structured outputs (#20223)
* Fix structured outputs
* Update common/chat-auto-parser-generator.cpp
Co-authored-by: Aldehir Rojas <hello@alde.dev>
---------
Co-authored-by: Aldehir Rojas <hello@alde.dev>
Fix compile bug (#20203)
* Fix compile bug
* Update common/chat-auto-parser-helpers.cpp
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com>
---------
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com>
# Conflicts:
# common/chat-auto-parser-helpers.cpp
common : gracefully handle incomplete output (#20191)
* common : handle incomplete UTF-8 at end of input in PEG parser
* cont : if reached end prematurely, emit needs_more_input to propagate partial output
* cont: refactor peg parse context to add lenient flag
* cont : remove partial flag, keep lenient flag
PEG parser for LFM2 (#20251)
* PEG parser for LFM2
* Simplify using python_value()
common: map developer role to system (#20215)
* Map developer role to system
* Simplify
common: consolidate PEG string parsers (#20263)
* common : consolidate PEG string parsers
* cont : fix json_string_content()
examples : fix empty items in json_schema_to_grammar.py [no ci] (#19968)
* Fix logic for retrieving schema items in `json_schema_to_grammar.py`
If `schema['items']` is `{}` and `prefixItems not in schema', as `{}` is Falsy, the original code here will raise an error.
I think if `schema['items']` is `{}`, them items should just be `{}`
* Apply suggestion from @CISC
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com>
* Add tests for arrays with empty items
Add two unit tests to `tests/test-json-schema-to-grammar.cpp` that validate handling of arrays when 'items' is an empty schema and when 'prefixItems' is present alongside an empty 'items'. Both tests expect the same generated grammar, ensuring the JSON Schema->grammar conversion treats an empty 'items' schema (and the presence of 'prefixItems') correctly and covering this edge case.
---------
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com>
Reduce level of content parser warning message to avoid log spam on non-debug verbosity (#20347)
do not return if template parse failed
add arg to enable parallel tool call
common : fix incorrect uses of stoul (#20313)
# Conflicts:
# common/arg.cpp
# src/llama-grammar.cpp
examples : fix empty items in json_schema_to_grammar.py [no ci] (#19968)
* Fix logic for retrieving schema items in `json_schema_to_grammar.py`
If `schema['items']` is `{}` and `prefixItems not in schema', as `{}` is Falsy, the original code here will raise an error.
I think if `schema['items']` is `{}`, them items should just be `{}`
* Apply suggestion from @CISC
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com>
* Add tests for arrays with empty items
Add two unit tests to `tests/test-json-schema-to-grammar.cpp` that validate handling of arrays when 'items' is an empty schema and when 'prefixItems' is present alongside an empty 'items'. Both tests expect the same generated grammar, ensuring the JSON Schema->grammar conversion treats an empty 'items' schema (and the presence of 'prefixItems') correctly and covering this edge case.
---------
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com>
Add support for MiroThinker with new jinja template
common/parser: handle reasoning budget (#20297)
* v1
* Finished!
* Handlie cli
* Reasoning sampler
* Apply suggestions from code review
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com>
* Less explosive terminology :)
* Add utf-8 case and tests
* common : migrate reasoning budget sampler to common
* cont : clean up
* cont : expose state and allow passing as initial state
* cont : remove unused imports
* cont : update state machine doc string
---------
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com>
Co-authored-by: Alde Rojas <hello@alde.dev>
common/parser: use nlohmann::ordered_json to preserve parameter order (#20385)
common/parser: add GigaChatV3/3.1 models support (#19931)
Co-authored-by: Mishusha <pmv26021975@gmail.com>
common/parser: gracefully handle undetected tool parser, print error message. (#20286)
fix: prevent nullptr dereference (#20552)
common : fix iterator::end() dereference (#20445)
# Conflicts:
# common/regex-partial.cpp
jinja : add capability check for object args (#20612)
common/parser: add `--skip-chat-parsing` to force a pure content parser. (#20289)
* Add `--force-pure-content` to force a pure content parser.
* Update common/arg.cpp
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com>
common : rework gpt-oss parser (#20393)
* common : rework gpt-oss parser
* cont : fix gpt-oss tests
* cont : add structured output test
* cont : rename final to final_msg
common : fix gpt-oss content removal (#20745)
common/parser: add proper reasoning tag prefill reading (#20424)
* Implement proper prefill extraction
* Refactor cli parameters, update docs, move reasoning budget sampler part to common/reasoning-budget.cpp
* Update tools/server/server-task.cpp
* refactor: move grammars to variant, remove grammar_external, handle exception internally
* Make code less C++y
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
chat : handle tool calls with no required args in TAG_WITH_TAGGED format (#20764)
* chat : handle tool calls with no required args in TAG_WITH_TAGGED format
* Update tests/test-chat.cpp [no ci]
Co-authored-by: Aldehir Rojas <hello@alde.dev>
---------
Co-authored-by: Piotr Wilkin (ilintar) <piotr.wilkin@syndatis.com>
Co-authored-by: Aldehir Rojas <hello@alde.dev>
common/parser : fix out_of_range crash in throw path (#20424 regression) (#20777)
* chat : fix out_of_range crash in throw path (#20424 regression)
#20424 introduced effective_input = generation_prompt + input, but the
throw path uses input.substr(result.end) where result.end is a position
within effective_input. Every thinking model with a non-empty
generation_prompt crashes with std::out_of_range instead of the intended
error message.
Test crashes on unpatched master, passes with fix:
cmake -B build -DLLAMA_BUILD_TESTS=ON -DLLAMA_BUILD_TOOLS=OFF
cmake --build build --target test-chat
./build/bin/test-chat
* Update test-chat.cpp
* Update test-chat.cpp
* Update test-chat.cpp
---------
Co-authored-by: Piotr Wilkin (ilintar) <piotr.wilkin@syndatis.com>
jinja : fix heap OOB read in value equality comparison (#20782)
Address GHSA-q9j6-4hhc-rq9p and GHSA-2q4c-9gq5-5vfp.
The three-iterator overload of std::equal in value_array_t::equivalent()
and value_object_t::equivalent() reads past the end of the shorter
container when comparing arrays or objects of different lengths.
Use the four-iterator overload (C++14) which checks both range lengths.
Found-by: Pwno
common : fix typo in debug log ('extracft' -> 'extract') (#20807)
common/parser: fix nasty bug causing subtle corruption of generation prompt (#20825)
jinja : refactor token advancement (#20864)
* refactor token advancement
* exercise sub-expressions
common/autoparser : detect reasoning markers when enable_thinking changes system prompt (#20859)
common : replace wrap_for_generation with a prefix convenience function and fix gpt-oss (#20912)
jinja: fix macro with kwargs (#20960)
* jinja: fix macro with kwargs
* Apply suggestions from code review
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com>
* fix newline problem
---------
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com>
common : inhibit lazy grammar sampler while reasoning is active (#20970)
* common : inhibit grammar while reasoning budget is active
* cont : update force_pos in accept
* cont : fix tests
* cont : tweak should apply logic
* cont : return early not using grammar sampler
* Add tests
* cont : prevent backend sampling when reasoning budget enabled
* cont : fix typo
---------
Co-authored-by: Piotr Wilkin <piotr.wilkin@syndatis.com>
# Conflicts:
# common/reasoning-budget.h
# common/sampling.cpp
# tools/cli/cli.cpp
# tools/server/server-common.cpp
# tools/server/server-task.cpp
common/parser: fix reasoning whitespace bugs + extra parser tests (#21085)
* fix whitespace reasoning issues + add reconstruction tests
* Proper fix
* fix Nemotron autoparser test expectations to include newline in marker
common : add reasoning_format = none support to gpt-oss (#21094)
common/json-schema: fix: handle non-capturing groups (?:...) in JSON schema pattern converter (#21124)
The regex-to-grammar converter in _visit_pattern() crashes with SIGSEGV
when a JSON schema "pattern" field contains a non-capturing group (?:...).
Root cause: when the parser sees '(' followed by '?', it pushes a warning
but does not advance past '?:'. The recursive transform() call then
interprets '?' as a quantifier and calls seq.back() on an empty vector,
causing undefined behavior.
This commonly occurs when serving OpenAI-compatible tool calls from
clients that include complex regex patterns in their JSON schemas (e.g.,
date validation patterns like ^(?:(?:\d\d[2468][048]|...)-02-29|...)$).
The fix:
- Skip '?:' after '(' to treat non-capturing groups as regular groups
- For unsupported syntax (?=, ?!, etc.), skip to matching ')' safely,
handling escaped characters to avoid miscounting parenthesis depth
- Adjust the ')' unbalanced-parentheses check using direct char
comparisons instead of substr
- Add test cases for non-capturing groups (C++ only, as the JS/Python
implementations do not yet support this syntax)
common/parser: fix handling of tool definition with missing properties key (#21128)
jinja : handle empty expressions correctly (#20913)
* Reject empty computed member expressions before returning slices[0] from parse_member_expression_arguments().
* Treat empty computed member expressions with Jinja2 undefined semantics
Treat empty computed member expressions like `a[]` as undefined instead of
raising a parser error, to match Jinja2 behavior.
- return a noop expression for empty computed member arguments
- return undefined when a computed member key evaluates to undefined
- add Jinja tests covering `a[]|default('fallback')` and `a[] is undefined`
* Handle undefined computed member properties
Move undefined-property handling to the common member access path, and add a test covering `a[undefined] is undefined`.
* Use default undefined value in member access
Initialize val and then return it when property is undefined.
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com>
* empty statement parses to blank_expression instead of noop_statement
---------
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com>
common : gpt-oss handle builtin and unsolicited tool calls (#21213)
fix: tool call parsing for LFM2 and LFM2.5 models (#21242)
* fix: tool call parsing for LFM2 and LFM2.5 models'
* refactor: add test / break out lfm2 and lfm2.5 parsing logic
# Conflicts:
# common/chat.cpp
Relax prefill parser to allow space. (#21240)
* Relax prefill parser to allow space.
* Move changes from prefix() to parser generation
* Only allow spaces if we're not having a pure content parser next
common : add commentary rules for gpt-oss-20b (#21286)
add reasoning budget
model, mtmd: fix gguf conversion for audio/vision mmproj (#21309)
* fix gguf conversion for audio/vision mmproj
* fix test
# Conflicts:
# convert_hf_to_gguf.py
# examples/eval-callback/eval-callback.cpp
# examples/mtmd/CMakeLists.txt
# examples/mtmd/clip-impl.h
# examples/mtmd/mtmd.cpp
# gguf-py/gguf/constants.py
# gguf-py/gguf/gguf_writer.py
# gguf-py/gguf/tensor_mapping.py
# src/CMakeLists.txt
# src/llama-arch.cpp
# src/llama-arch.h
# src/llama-model.cpp
# src/llama-model.h
# src/llama-vocab.cpp
# src/models/models.h
# tests/test-llama-archs.cpp
# tools/mtmd/clip-graph.h
# tools/mtmd/clip-model.h
# tools/mtmd/clip.cpp
# tools/mtmd/models/models.h
fix: gemma 4 template (#21326)
chat : avoid including json in chat.h (#21306)
jinja: coerce input for string-specific filters (#21370)
common : fix tool call type detection for nullable and enum schemas (#21327)
* common : fix tool call type detection for nullable and enum schemas
* common, tests : fix grammar delegation for nullable/enum schemas and add tests
Fix enum type inference to scan all enum values (not just index 0) so
schemas like {"enum": [0, "celsius"]} correctly detect string type.
Fix schema_delegates in peg-parser to handle nullable type arrays
(["string", "null"]) and typeless enum schemas in raw mode, allowing
the tagged parser to use raw text instead of JSON-formatted strings.
Add test cases for Qwen3-Coder (TAG_WITH_TAGGED format):
- nullable string ["string", "null"]
- nullable string with null first ["null", "string"]
- nullable integer ["integer", "null"]
- enum without explicit type key
common/parser: fix call ID detection (Mistral parser mostly) + atomicity for tag-json parsers (#21230)
* Fix call ID detection (Mistral parser mostly) + atomicity for tag-json parsers
* Rename
* Update common/chat-auto-parser-generator.cpp
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com>
---------
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com>
common : add gemma 4 specialized parser (#21418)
* common : add gemma4 dedicated parser
* cont : add '<|tool_response>' as eog
* cont : emit JSON from Gemma4 tool call AST
* cont : more fixes
* cont : refactor convert function
* cont : refine rules and mapping
* cont : add more tests
* cont : clean up
* cont : remove autoparser gemma4 implementation
* cont : more cleanup
* cont : rename gemma4.jinja to match the others
* cont : add custom template to support interleaved thinking
* cont : preserve reasoning in model turns
* cont : fix initializer error
* cont : fix unused vars
* cont : fix accidental static
* cont : fix specialized_template signature
* fix extra semicolon
* remove debug line and extra space [no ci]
fix reasoning budget
parser: fix MiniMax handling (#21573)
jinja : support ensure_ascii=true, string repetition and int/float self-filtering (#21623)
* feat: jinja engine improvements for reka-edge
Port three Jinja engine improvements needed for the reka-edge model:
1. Python-style string repetition ("ab" * 3 → "ababab")
2. ensure_ascii=true support for tojson filter (escapes non-ASCII to \uXXXX)
3. int() builtin on value_int_t (identity, needed for Reka Edge template)
* fix: escape invalid utf8 bytes when ensure_ascii=true
The json_ensure_ascii_preserving_format function does not correctly
handle an edge case where if UTF-8 parsing fails, it adds the non-ascii
character back to the output as a raw byte.
This commit fixes that by adding the unicode standard replacement
character \\ufffd to the output instead. This is the standard behavior
for various programming languages like Python, Rust, Go, etc.
* chore: address PR comments
1. Add todo comment for supporting string repetition for array/tuples
2. Add support for float identity operation
3. Move invalid ascii test case to test_fuzzing
* chore: accept suggestion for common/jinja/value.cpp
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com>
---------
Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com>
common : simplify autoparser tagged parser rules (#21216)
* common : simplify autoparser tagged parser rules
* cont : remove upper limit on optional args
* cont : revert changes to parsing at the end
* cont : undo arbitrary ordering of optional args
* cont : fix uninitialized required parameters
* revert to simplify merge
* re-apply patches
* restore flexible optional arg ordering tests
common : fix ambiguous grammar rule in gemma4 (#21661)
* common : fix ambiguous grammar rule in gemma4
* cont : fix missing comma...
common : enable reasoning budget sampler for gemma4 (#21697)
* fix: enable reasoning budget sampler for gemma4
Add thinking_start_tag and thinking_end_tag to
common_chat_params_init_gemma4(). Without these, the reasoning
budget sampler never activates for gemma4.
Make the newline after "thought" optional in the PEG parser to
handle budget=0 (sampler forces end tag before the newline).
Add test case for empty thinking block.
Fixes#21487
* use p.space() instead of p.optional(p.literal("\n")) in gemma4 thought parser
common : better align to the updated official gemma4 template (#21704)
fix: Fix broken structured output when using $refs in json_schema (#21699)
chat: dedicated DeepSeek v3.2 parser + "official" template (#21785)
Hide render_message_to_json warning
common/gemma4 : handle parsing edge cases (#21760)
common: skip reasoning budget sampler when no budget is requested (#21870)
* common: skip reasoning budget sampler when no budget is requested
After I added thinking_start_tag / thinking_end_tag for gemma4 in #21697, the reasoning budget sampler gets unconditionally created even when no budget is configured (the default -1). The same applies to kimi_k2, lfm2, lfm2_5, and ministral_3 which also set these tags. The budget gets converted to INT_MAX, so the sampler never actually forces any tokens but still runs per-token checks (start tag matching in IDLE state, token-to-piece conversion + UTF-8 checks in COUNTING state).
More importantly, the mere existence of the sampler (non-null rbudget) disables backend sampling. Backend sampling lets the GPU select tokens directly, avoiding a full logits transfer from GPU to CPU every token. This could explain the 30% speed regression reported in #21784 (98 t/s to 70 t/s on Vulkan).
So I added a reasoning_budget_tokens >= 0 check to the sampler creation condition. When the budget is unlimited, the sampler is not created, backend sampling stays enabled, and no per-token overhead is added. When a budget is explicitly set (0, 128, 1024, etc.), the sampler is created and works as before.
* common: preserve rbudget when grammar is lazy
Following up on the review feedback on #21870: keep the reasoning budget sampler when grammar_lazy is true, so the thinking-block grammar suppression from #20970 still works when tools are in use. This way, we only skip the sampler when both no budget is set AND grammar is not lazy.
autoparser: support case of JSON_NATIVE with per-call markers (test case: Reka-Edge) (#21892)
* fix grammar
* fix add sampled token
---------
Co-authored-by: Piotr Wilkin (ilintar) <piotr.wilkin@syndatis.com>
Co-authored-by: firecoperana <firecoperana>
The token loop reads sK[] in the state update (bottom of loop) but has
no barrier before the next iteration overwrites sK[] (top of loop).
Without an explicit memory fence, hardware/compiler reordering can
cause non-deterministic reads from shared memory.
Per review: with this barrier in place, the prior __syncthreads() after
the cross-warp reduction and the one immediately after loop exit are
both redundant. The new barrier is a full block-level fence that also
orders all_sum1/all_sum2 reads vs. the next iteration's writes, and
every thread reaches it before leaving the loop. Both redundant
barriers removed.
No performance impact — GPU utilization is 31-33% during inference,
bottlenecked by CPU MoE expert computation, not the CUDA kernel.
Co-authored-by: Mark Alonzo <mark.alonzo@outlook.com>
Upstream: llama.cpp#21428. GGUF stores bools as 1-byte int8_t but the
loader was casting through (const bool*) which has implementation-defined
sizeof. Use explicit int8_t cast with != 0 normalization.
Co-authored-by: Mark Alonzo <mark.alonzo@outlook.com>
* WIP: Gemma4 vision
Crashes on the GPU because of rms_norm requiring ne0 to be multiple
of warp_size.
Runs on the CPU, but produces garbage.
* Remove unnecessary assert in CUDA rms_norm
* GLU was not advertised as supported on CUDA
* Still not working
* This seems to work
* fix: Updating cleanup step
* fix: Updated trigger for build-container
* fix: Unset token for cleanup step
* fix: Set build-container cleanup step without run-dry
* fix: Removed 100 commits from checkout actions
* fix: Enable the whole history
* test: Suggestion from mcm007 on build-container
* fix: add token to package cleanup step
Explicitly pass GITHUB_TOKEN to the delete-package-versions action to ensure it has sufficient authorization for package deletion.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix: - Error: delete version API failed. Package not found.
* fix: Deleted LLAMA_COMMIT from build
* fix: Removed id-token permission
---------
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
get_mmq_x_max_host() returned 128 for NVIDIA Volta when built with
GGML_CUDA_FORCE_MMQ, but get_mmq_x_max_device() caps at
MMQ_DP4A_MAX_BATCH_SIZE (= 64) for the same configuration because the
wider mmq_x device kernels are not instantiated on sm_70. The host
dispatcher selects an mmq_x in [8, mmq_x_max] and instantiates the
matching template; any pick > 64 hits NO_DEVICE_CODE at runtime and
the kernel returns without writing output, producing zero-valued
activations and incoherent generation.
Repro on Tesla V100 (sm_70) with a Q6_K Qwen3.5-MoE model:
cmake -B build -DGGML_CUDA=ON -DGGML_CUDA_FORCE_MMQ=ON \
-DGGML_CUDA_NO_PEER_COPY=ON -DCMAKE_CUDA_ARCHITECTURES=70
./llama-cli -m model.gguf -ngl 99 -p "<~200-token prompt>"
# stderr: mmq.cuh:3941: ERROR: CUDA kernel mul_mat_q has no
# device code compatible with CUDA arch 700.
Cap the host bound at MMQ_DP4A_MAX_BATCH_SIZE on Volta regardless of
GGML_CUDA_FORCE_MMQ; this matches the device template availability and
yields correct output. Measured on a single V100-SXM2:
before patch: PP unusable (NO_DEVICE_CODE), TG ~100 t/s
after patch: PP ~630 t/s, TG ~102 t/s
The change is a no-op on architectures that have int8 MMA (Turing+),
on AMD/HIP, and on builds without GGML_CUDA_FORCE_MMQ.
In llama_kv_cache_init() call, params.type_v_first was incorrectly passed
twice instead of params.type_v_last.
This caused V cache in the last N layers to use type_v_first instead of type_v_last.
Fix: Replace second params.type_v_first with params.type_v_last.
* ci: implement build matrix for CUDA/CPU containers with dynamic tagging
* fix: Updated Docker images/build-container.yml
* fix: Updated the documentation about Docker
* fix: Set Arch for 3090s
* fix: Updated build step name.
* fix: Set target ARCH as a variable
* feat: Added cleanup step
* feat: Added docker-bake and updated workflow
* fix: Issue with REPO_OWNER variable
* fix: Updated workflow to solve errors
* fix: Updated branch format
* fix: Wrong naming
* Update docker-bake.hcl
* Update build-container.yml
* Update ik_llama-cuda.Containerfile
* Update ik_llama-cpu.Containerfile
* Update docker-bake.hcl
* Update build-container.yml
* Removed action/cache
* added -sSL for reliability and fixed the URL path
* added -sSL for reliability and fixed the URL path CUDA containerfile
* fix: correct Dockerfile RUN command syntax errors
- Combine split apt-get install commands in both Containerfiles
- Fix broken cmake command continuation in ik_llama-cuda.Containerfile
* fix: correct llama-swap download URL in Containerfiles
- Fix broken line continuation in curl download URL for llama-swap
* perf: improve ccache configuration in Containerfiles
- Add CCACHE_UMASK=000 for cache accessibility across stages
- Add CCACHE_MAXSIZE=1G to prevent unbounded growth
- Initialize ccache with ccache -i during build stage
* fix: remove problematic ccache initialization from Containerfiles
- ccache -i fails because CCACHE_DIR mount doesn't exist yet at build time
* fix: add git to CPU Containerfile build dependencies
- Resolves CMake warning about missing Git for build info
* chore: optimize Containerfile with smaller images and better healthchecks
- Add --no-install-recommends to all apt-get commands for smaller image size
- Add ca-certificates to base stage for HTTPS support
- Merge redundant build copy commands from 3 layers to 1
- Fix llama-swap version from 198 to v199 (latest release)
- Add HEALTHCHECK configuration with interval/timeout/retries to server and swap stages
- Copy /app/lib in server stage to fix container startup
* chore: fix CUDA Containerfile healthchecks and swap version
- Add /app/lib copy in server stage to fix container startup
- Fix llama-swap version from 198 to v199 (latest release)
- Add HEALTHCHECK configuration with interval/timeout/retries
* chore: fix indentation in Containerfiles and add LD_LIBRARY_PATH for server target
* fix: add --break-system-packages flag for pip in CPU Containerfile
* feat: add git bind mount for build info and NCCL support for CUDA
* fix: remove libnccl-dev from CUDA build (already included in base image)
* fix: added Markdown files to ignore files
* feat: use BUILD_NUMBER-COMMIT pattern for docker image tags
- Add BUILD_NUMBER and LLAMA_COMMIT to build workflow
- Update docker-bake.hcl to use version tag format matching llama-server --version output
- Format: VARIANT-BUILD_NUMBER-COMMIT (e.g., cu12-full-4406-3bc90dfd)
* fix: fetch full git history for accurate BUILD_NUMBER
- Add fetch-depth: 0 to actions/checkout to get all commits
- This ensures git rev-list --count HEAD returns correct total commit count
* fix: fetch full git history in Dockerfile for accurate BUILD_NUMBER
- Add git fetch --unshallow to get complete commit history during build
- This ensures build-info.cpp is generated with correct LLAMA_BUILD_NUMBER
* chore: update GitHub Actions to latest versions for Node.js 24 compatibility
- docker/setup-buildx-action@v3 -> v4
- docker/login-action@v3 -> v4
* chore: update all GitHub Actions to Node.js 24 compatible versions
- actions/checkout@v4 -> v6
- docker/setup-buildx-action@v3 -> v5
- docker/login-action@v3 -> v6
- docker/bake-action@v5 -> v7
* fix: use CI-passed BUILD_NUMBER and LLAMA_COMMIT in Dockerfile
- Add BUILD_NUMBER and LLAMA_COMMIT as build args
- Fall back to git commands if not provided
- Pass values explicitly to cmake for accurate build info
* fix: pass BUILD_NUMBER and LLAMA_COMMIT as Docker build args
- Add BUILD_NUMBER and LLAMA_COMMIT to docker bake args
- These will be used by the Containerfile for accurate build info
* fix: revert docker actions to v4 (latest available versions)
* fix: calculate BUILD_NUMBER and LLAMA_COMMIT directly in Containerfile
- Removed ARG defaults since we calculate from git during build
- Use git rev-list --count HEAD and git rev-parse for accurate version info
- Falls back to 0/unknown if git commands fail
* feat: calculate BUILD_NUMBER and LLAMA_COMMIT in Containerfiles
- Add git-based version calculation in both CPU and CUDA Containerfiles
- Remove .git bind mount (git is copied with COPY .)
- Pass build info to CMake for accurate llama-server --version output
* feat: calculate BUILD_NUMBER and LLAMA_COMMIT in Containerfiles
- Add git-based version calculation using git rev-list and git rev-parse
- Copy .git directory separately to ensure git commands work during build
- Pass build info to CMake for accurate llama-server --version output
* fix: cache improvements for CUDA and CPU builds
* fix: "/.git": not found
* fix: Unnecessary mv llama-swap
* fix: Remove BUILD_NUMBER and LLAMA_COMMIT from docker file, calculated by cmake proc
* fix: remove .git from dockerignore for local and CI builds
- Enables cmake to access .git directory during Docker build
- Required for version calculation in llama-server binary
- GitHub Actions uses explicit mount via bake action set parameter
* fix: Remove mounts key from Build and Push step in gh workflow
* ci: add .git verification step before build
* refactor: standardize Containerfile structure and remove .git mount dependency
- Remove --mount=type=bind,source=.git,target=.git from both Containerfiles
- Replace COPY . . with git clone for cleaner build context
- Add CUSTOM_COMMIT ARG for optional custom commit switching
- Standardize ARG/ENV ordering and comment formatting across CPU/CUDA variants
- Install ca-certificates before git clone to fix SSL verification issues
- Rename 'Structured artifact collection' to 'Collect build artifacts'
* ci: remove broken cache pruning step
* ci: remove broken prune-cache job
- Remove prune-cache job that was failing due to missing .git directory
- The job required a checkout step and the cache pruning logic was non-critical
* chore: Removed step for Verifying .git existance in GH workflow
* fix: ensure build always proceeds even if git switch fails
- Add '|| true' to git switch command so build continues on failure
- This prevents the entire RUN step from failing when CUSTOM_COMMIT is invalid
* fix: resolve Docker build pipeline issues
- Remove external git clone from Containerfiles, use build context directly
- Add BUILD_NUMBER and BUILD_COMMIT as CMake cache variables in build-info.cmake
- Fix .devops/tools.sh inclusion by using explicit COPY for hidden directories
- Set USE_CCACHE=true for CI builds
- Clean up unused SHA_SHORT variable from docker-bake.hcl
Fixes: Build steps were cached incorrectly due to external git clone ignoring the actual build context source.
* fix: include .git in Docker build context and add verification
* ci: add .git directory verification step after checkout
* build: fix .git mount path for Docker build context compatibility
* build: fix .git mount path for Docker build context compatibility
* docker: include .git in build context for version calculation
* ci: add .git directory verification step after checkout
* chore: Removed unecessary Verify .git step (It was a test)
* docs: update README with docker-bake and build-local.sh instructions
* docs: remove build-local.sh reference (not in repo)
* ci: optimize disk usage by limiting fetch depth and cleaning workspace
* fix: cleanup step in workflow
---------
Co-authored-by: HP Prodesk <sourceupdev@gmail.com>
* ci: implement build matrix for CUDA/CPU containers with dynamic tagging
* fix: Updated Docker images/build-container.yml
* fix: Updated the documentation about Docker
* fix: Set Arch for 3090s
* fix: Updated build step name.
* fix: Set target ARCH as a variable
* feat: Added cleanup step
* feat: Added docker-bake and updated workflow
* fix: Issue with REPO_OWNER variable
* fix: Updated workflow to solve errors
* fix: Updated branch format
* fix: Wrong naming
* Update docker-bake.hcl
* Update build-container.yml
* Update ik_llama-cuda.Containerfile
* Update ik_llama-cpu.Containerfile
* Update docker-bake.hcl
* Update build-container.yml
* Removed action/cache
* added -sSL for reliability and fixed the URL path
* added -sSL for reliability and fixed the URL path CUDA containerfile
* fix: correct Dockerfile RUN command syntax errors
- Combine split apt-get install commands in both Containerfiles
- Fix broken cmake command continuation in ik_llama-cuda.Containerfile
* fix: correct llama-swap download URL in Containerfiles
- Fix broken line continuation in curl download URL for llama-swap
* perf: improve ccache configuration in Containerfiles
- Add CCACHE_UMASK=000 for cache accessibility across stages
- Add CCACHE_MAXSIZE=1G to prevent unbounded growth
- Initialize ccache with ccache -i during build stage
* fix: remove problematic ccache initialization from Containerfiles
- ccache -i fails because CCACHE_DIR mount doesn't exist yet at build time
* fix: add git to CPU Containerfile build dependencies
- Resolves CMake warning about missing Git for build info
* chore: optimize Containerfile with smaller images and better healthchecks
- Add --no-install-recommends to all apt-get commands for smaller image size
- Add ca-certificates to base stage for HTTPS support
- Merge redundant build copy commands from 3 layers to 1
- Fix llama-swap version from 198 to v199 (latest release)
- Add HEALTHCHECK configuration with interval/timeout/retries to server and swap stages
- Copy /app/lib in server stage to fix container startup
* chore: fix CUDA Containerfile healthchecks and swap version
- Add /app/lib copy in server stage to fix container startup
- Fix llama-swap version from 198 to v199 (latest release)
- Add HEALTHCHECK configuration with interval/timeout/retries
* chore: fix indentation in Containerfiles and add LD_LIBRARY_PATH for server target
* fix: add --break-system-packages flag for pip in CPU Containerfile
* feat: add git bind mount for build info and NCCL support for CUDA
* fix: remove libnccl-dev from CUDA build (already included in base image)
* fix: added Markdown files to ignore files
* feat: use BUILD_NUMBER-COMMIT pattern for docker image tags
- Add BUILD_NUMBER and LLAMA_COMMIT to build workflow
- Update docker-bake.hcl to use version tag format matching llama-server --version output
- Format: VARIANT-BUILD_NUMBER-COMMIT (e.g., cu12-full-4406-3bc90dfd)
* fix: fetch full git history for accurate BUILD_NUMBER
- Add fetch-depth: 0 to actions/checkout to get all commits
- This ensures git rev-list --count HEAD returns correct total commit count
* fix: fetch full git history in Dockerfile for accurate BUILD_NUMBER
- Add git fetch --unshallow to get complete commit history during build
- This ensures build-info.cpp is generated with correct LLAMA_BUILD_NUMBER
* chore: update GitHub Actions to latest versions for Node.js 24 compatibility
- docker/setup-buildx-action@v3 -> v4
- docker/login-action@v3 -> v4
* chore: update all GitHub Actions to Node.js 24 compatible versions
- actions/checkout@v4 -> v6
- docker/setup-buildx-action@v3 -> v5
- docker/login-action@v3 -> v6
- docker/bake-action@v5 -> v7
* fix: use CI-passed BUILD_NUMBER and LLAMA_COMMIT in Dockerfile
- Add BUILD_NUMBER and LLAMA_COMMIT as build args
- Fall back to git commands if not provided
- Pass values explicitly to cmake for accurate build info
* fix: pass BUILD_NUMBER and LLAMA_COMMIT as Docker build args
- Add BUILD_NUMBER and LLAMA_COMMIT to docker bake args
- These will be used by the Containerfile for accurate build info
* fix: revert docker actions to v4 (latest available versions)
* fix: calculate BUILD_NUMBER and LLAMA_COMMIT directly in Containerfile
- Removed ARG defaults since we calculate from git during build
- Use git rev-list --count HEAD and git rev-parse for accurate version info
- Falls back to 0/unknown if git commands fail
* feat: calculate BUILD_NUMBER and LLAMA_COMMIT in Containerfiles
- Add git-based version calculation in both CPU and CUDA Containerfiles
- Remove .git bind mount (git is copied with COPY .)
- Pass build info to CMake for accurate llama-server --version output
* feat: calculate BUILD_NUMBER and LLAMA_COMMIT in Containerfiles
- Add git-based version calculation using git rev-list and git rev-parse
- Copy .git directory separately to ensure git commands work during build
- Pass build info to CMake for accurate llama-server --version output
* fix: cache improvements for CUDA and CPU builds
* fix: "/.git": not found
* fix: Unnecessary mv llama-swap
* fix: Remove BUILD_NUMBER and LLAMA_COMMIT from docker file, calculated by cmake proc
* fix: remove .git from dockerignore for local and CI builds
- Enables cmake to access .git directory during Docker build
- Required for version calculation in llama-server binary
- GitHub Actions uses explicit mount via bake action set parameter
* fix: Remove mounts key from Build and Push step in gh workflow
* ci: add .git verification step before build
* refactor: standardize Containerfile structure and remove .git mount dependency
- Remove --mount=type=bind,source=.git,target=.git from both Containerfiles
- Replace COPY . . with git clone for cleaner build context
- Add CUSTOM_COMMIT ARG for optional custom commit switching
- Standardize ARG/ENV ordering and comment formatting across CPU/CUDA variants
- Install ca-certificates before git clone to fix SSL verification issues
- Rename 'Structured artifact collection' to 'Collect build artifacts'
* ci: remove broken cache pruning step
* ci: remove broken prune-cache job
- Remove prune-cache job that was failing due to missing .git directory
- The job required a checkout step and the cache pruning logic was non-critical
* chore: Removed step for Verifying .git existance in GH workflow
* fix: ensure build always proceeds even if git switch fails
- Add '|| true' to git switch command so build continues on failure
- This prevents the entire RUN step from failing when CUSTOM_COMMIT is invalid
* fix: resolve Docker build pipeline issues
- Remove external git clone from Containerfiles, use build context directly
- Add BUILD_NUMBER and BUILD_COMMIT as CMake cache variables in build-info.cmake
- Fix .devops/tools.sh inclusion by using explicit COPY for hidden directories
- Set USE_CCACHE=true for CI builds
- Clean up unused SHA_SHORT variable from docker-bake.hcl
Fixes: Build steps were cached incorrectly due to external git clone ignoring the actual build context source.
* fix: include .git in Docker build context and add verification
* ci: add .git directory verification step after checkout
* build: fix .git mount path for Docker build context compatibility
* build: fix .git mount path for Docker build context compatibility
* docker: include .git in build context for version calculation
* ci: add .git directory verification step after checkout
* chore: Removed unecessary Verify .git step (It was a test)
* docs: update README with docker-bake and build-local.sh instructions
* docs: remove build-local.sh reference (not in repo)
* ci: optimize disk usage by limiting fetch depth and cleaning workspace
---------
Co-authored-by: HP Prodesk <sourceupdev@gmail.com>
* wip: separate llama_context for MTP with graph reuse
* wip: fix KV cache desync with separate MTP context
* refactor: remove dead mtp logic code, encapsulate KV mirroring
* mtp-context: derive args directly from the main model's context
* mtp: fix kv cache positions
* clean small comments
* minor refactor for context shift
* Use build_std_attention for Gemma4 when possible
It is possible for the 26b MoE and 31b dense models.
It is not possible for the E4B/E2B vaiants because they
don't have KV cache in each layer.
* Standardize Gemma4 dense ffn
* WIP: Gemma4 split mode graph
Runs but produces NaNs
* WIP: Gemma4 split mode graph
Runs but very high PPL. At least it is no longer NaN.
* WIP
* This works!
* Put attn_norm, attn_post_norm, ffn_norm, ffn_post_norm on all GPUs
* Fix crash when saving/loading KV cache
* WIP: split mode graph for Gemma4-MoE - crashes
* Split mode graph for Gemma4-MoE - this works
* Disable SWA optimization
Something goes wrong there
* Consolidate MoE and dense graph parallel
