mirror of https://github.com/ikawrakow/ik_llama.cpp.git synced 2026-06-28 04:30:15 -05:00

Go to file

Autoparser - complete refactoring of parser architecture (#1376 )

* Autoparser - complete refactoring of parser architecture

Autoparser: add optional argument reshuffle capability

Autoparser: True streaming (#20177)

* Relax atomicity constraint for nicer, more pleasent, True Streaming parsing

* Whitespace

* Remove redundant atomics

Revert to OAI-compatible args (#20213)

* Revert to OAI-compatible args

* Apply workaround::func_args_not_string

Fix structured outputs (#20223)

* Fix structured outputs

* Update common/chat-auto-parser-generator.cpp

Co-authored-by: Aldehir Rojas <hello@alde.dev>

---------

Co-authored-by: Aldehir Rojas <hello@alde.dev>

Fix compile bug (#20203)

* Fix compile bug

* Update common/chat-auto-parser-helpers.cpp

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com>

---------

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com>
# Conflicts:
#	common/chat-auto-parser-helpers.cpp

common : gracefully handle incomplete output (#20191)

* common : handle incomplete UTF-8 at end of input in PEG parser

* cont : if reached end prematurely, emit needs_more_input to propagate partial output

* cont: refactor peg parse context to add lenient flag

* cont : remove partial flag, keep lenient flag

PEG parser for LFM2 (#20251)

* PEG parser for LFM2

* Simplify using python_value()

common: map developer role to system (#20215)

* Map developer role to system
* Simplify

common: consolidate PEG string parsers (#20263)

* common : consolidate PEG string parsers
* cont : fix json_string_content()

examples : fix empty items in json_schema_to_grammar.py [no ci] (#19968)

* Fix logic for retrieving schema items in `json_schema_to_grammar.py`

If `schema['items']` is `{}` and `prefixItems not in schema', as `{}` is Falsy, the original code here will raise an error.

I think if `schema['items']` is `{}`, them items should just be `{}`

* Apply suggestion from @CISC

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com>

* Add tests for arrays with empty items

Add two unit tests to `tests/test-json-schema-to-grammar.cpp` that validate handling of arrays when 'items' is an empty schema and when 'prefixItems' is present alongside an empty 'items'. Both tests expect the same generated grammar, ensuring the JSON Schema->grammar conversion treats an empty 'items' schema (and the presence of 'prefixItems') correctly and covering this edge case.

---------

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com>

Reduce level of content parser warning message to avoid log spam on non-debug verbosity (#20347)

do not return if template parse failed

add arg to enable parallel tool call

common : fix incorrect uses of stoul (#20313)
# Conflicts:
#	common/arg.cpp
#	src/llama-grammar.cpp

examples : fix empty items in json_schema_to_grammar.py [no ci] (#19968)

* Fix logic for retrieving schema items in `json_schema_to_grammar.py`

If `schema['items']` is `{}` and `prefixItems not in schema', as `{}` is Falsy, the original code here will raise an error.

I think if `schema['items']` is `{}`, them items should just be `{}`

* Apply suggestion from @CISC

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com>

* Add tests for arrays with empty items

Add two unit tests to `tests/test-json-schema-to-grammar.cpp` that validate handling of arrays when 'items' is an empty schema and when 'prefixItems' is present alongside an empty 'items'. Both tests expect the same generated grammar, ensuring the JSON Schema->grammar conversion treats an empty 'items' schema (and the presence of 'prefixItems') correctly and covering this edge case.

---------

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com>

Add support for MiroThinker with new jinja template

common/parser: handle reasoning budget (#20297)

* v1

* Finished!

* Handlie cli

* Reasoning sampler

* Apply suggestions from code review

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com>

* Less explosive terminology :)

* Add utf-8 case and tests

* common : migrate reasoning budget sampler to common

* cont : clean up

* cont : expose state and allow passing as initial state

* cont : remove unused imports

* cont : update state machine doc string

---------

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com>
Co-authored-by: Alde Rojas <hello@alde.dev>

common/parser: use nlohmann::ordered_json to preserve parameter order (#20385)

common/parser: add GigaChatV3/3.1 models support (#19931)

Co-authored-by: Mishusha <pmv26021975@gmail.com>

common/parser: gracefully handle undetected tool parser, print error message. (#20286)

fix: prevent nullptr dereference (#20552)

common : fix iterator::end() dereference (#20445)
# Conflicts:
#	common/regex-partial.cpp

jinja : add capability check for object args (#20612)

common/parser: add `--skip-chat-parsing` to force a pure content parser. (#20289)

* Add `--force-pure-content` to force a pure content parser.

* Update common/arg.cpp

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com>

common : rework gpt-oss parser (#20393)

* common : rework gpt-oss parser

* cont : fix gpt-oss tests

* cont : add structured output test

* cont : rename final to final_msg

common : fix gpt-oss content removal (#20745)

common/parser: add proper reasoning tag prefill reading (#20424)

* Implement proper prefill extraction

* Refactor cli parameters, update docs, move reasoning budget sampler part to common/reasoning-budget.cpp

* Update tools/server/server-task.cpp

* refactor: move grammars to variant, remove grammar_external, handle exception internally

* Make code less C++y

Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>

chat : handle tool calls with no required args in TAG_WITH_TAGGED format (#20764)

* chat : handle tool calls with no required args in TAG_WITH_TAGGED format

* Update tests/test-chat.cpp [no ci]

Co-authored-by: Aldehir Rojas <hello@alde.dev>

---------

Co-authored-by: Piotr Wilkin (ilintar) <piotr.wilkin@syndatis.com>
Co-authored-by: Aldehir Rojas <hello@alde.dev>

common/parser : fix out_of_range crash in throw path (#20424 regression) (#20777)

* chat : fix out_of_range crash in throw path (#20424 regression)

#20424 introduced effective_input = generation_prompt + input, but the
throw path uses input.substr(result.end) where result.end is a position
within effective_input. Every thinking model with a non-empty
generation_prompt crashes with std::out_of_range instead of the intended
error message.

Test crashes on unpatched master, passes with fix:

  cmake -B build -DLLAMA_BUILD_TESTS=ON -DLLAMA_BUILD_TOOLS=OFF
  cmake --build build --target test-chat
  ./build/bin/test-chat

* Update test-chat.cpp

* Update test-chat.cpp

* Update test-chat.cpp

---------

Co-authored-by: Piotr Wilkin (ilintar) <piotr.wilkin@syndatis.com>

jinja : fix heap OOB read in value equality comparison (#20782)

Address GHSA-q9j6-4hhc-rq9p and GHSA-2q4c-9gq5-5vfp.

The three-iterator overload of std::equal in value_array_t::equivalent()
and value_object_t::equivalent() reads past the end of the shorter
container when comparing arrays or objects of different lengths.

Use the four-iterator overload (C++14) which checks both range lengths.

Found-by: Pwno

common : fix typo in debug log ('extracft' -> 'extract') (#20807)

common/parser: fix nasty bug causing subtle corruption of generation prompt (#20825)

jinja : refactor token advancement (#20864)

* refactor token advancement

* exercise sub-expressions

common/autoparser : detect reasoning markers when enable_thinking changes system prompt (#20859)

common : replace wrap_for_generation with a prefix convenience function and fix gpt-oss (#20912)

jinja: fix macro with kwargs (#20960)

* jinja: fix macro with kwargs

* Apply suggestions from code review

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com>

* fix newline problem

---------

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com>

common : inhibit lazy grammar sampler while reasoning is active (#20970)

* common : inhibit grammar while reasoning budget is active

* cont : update force_pos in accept

* cont : fix tests

* cont : tweak should apply logic

* cont : return early not using grammar sampler

* Add tests

* cont : prevent backend sampling when reasoning budget enabled

* cont : fix typo

---------

Co-authored-by: Piotr Wilkin <piotr.wilkin@syndatis.com>
# Conflicts:
#	common/reasoning-budget.h
#	common/sampling.cpp
#	tools/cli/cli.cpp
#	tools/server/server-common.cpp
#	tools/server/server-task.cpp

common/parser: fix reasoning whitespace bugs + extra parser tests (#21085)

* fix whitespace reasoning issues + add reconstruction tests

* Proper fix

* fix Nemotron autoparser test expectations to include newline in marker

common : add reasoning_format = none support to gpt-oss (#21094)

common/json-schema: fix: handle non-capturing groups (?:...) in JSON schema pattern converter (#21124)

The regex-to-grammar converter in _visit_pattern() crashes with SIGSEGV
when a JSON schema "pattern" field contains a non-capturing group (?:...).

Root cause: when the parser sees '(' followed by '?', it pushes a warning
but does not advance past '?:'. The recursive transform() call then
interprets '?' as a quantifier and calls seq.back() on an empty vector,
causing undefined behavior.

This commonly occurs when serving OpenAI-compatible tool calls from
clients that include complex regex patterns in their JSON schemas (e.g.,
date validation patterns like ^(?:(?:\d\d[2468][048]|...)-02-29|...)$).

The fix:
- Skip '?:' after '(' to treat non-capturing groups as regular groups
- For unsupported syntax (?=, ?!, etc.), skip to matching ')' safely,
  handling escaped characters to avoid miscounting parenthesis depth
- Adjust the ')' unbalanced-parentheses check using direct char
  comparisons instead of substr
- Add test cases for non-capturing groups (C++ only, as the JS/Python
  implementations do not yet support this syntax)

common/parser: fix handling of tool definition with missing properties key (#21128)

jinja : handle empty expressions correctly (#20913)

* Reject empty computed member expressions before returning slices[0] from parse_member_expression_arguments().

* Treat empty computed member expressions with Jinja2 undefined semantics

Treat empty computed member expressions like `a[]` as undefined instead of
raising a parser error, to match Jinja2 behavior.

- return a noop expression for empty computed member arguments
- return undefined when a computed member key evaluates to undefined
- add Jinja tests covering `a[]|default('fallback')` and `a[] is undefined`

* Handle undefined computed member properties

Move undefined-property handling to the common member access path, and add a test covering `a[undefined] is undefined`.

* Use default undefined value in member access

Initialize val and then return it when property is undefined.

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com>

* empty statement parses to blank_expression instead of noop_statement

---------

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com>

common : gpt-oss handle builtin and unsolicited tool calls (#21213)

fix: tool call parsing for LFM2 and LFM2.5 models (#21242)

* fix: tool call parsing for LFM2 and LFM2.5 models'

* refactor: add test / break out lfm2 and lfm2.5 parsing logic
# Conflicts:
#	common/chat.cpp

Relax prefill parser to allow space. (#21240)

* Relax prefill parser to allow space.

* Move changes from prefix() to parser generation

* Only allow spaces if we're not having a pure content parser next

common : add commentary rules for gpt-oss-20b (#21286)

add reasoning budget

model, mtmd: fix gguf conversion for audio/vision mmproj (#21309)

* fix gguf conversion for audio/vision mmproj

* fix test
# Conflicts:
#	convert_hf_to_gguf.py
#	examples/eval-callback/eval-callback.cpp
#	examples/mtmd/CMakeLists.txt
#	examples/mtmd/clip-impl.h
#	examples/mtmd/mtmd.cpp
#	gguf-py/gguf/constants.py
#	gguf-py/gguf/gguf_writer.py
#	gguf-py/gguf/tensor_mapping.py
#	src/CMakeLists.txt
#	src/llama-arch.cpp
#	src/llama-arch.h
#	src/llama-model.cpp
#	src/llama-model.h
#	src/llama-vocab.cpp
#	src/models/models.h
#	tests/test-llama-archs.cpp
#	tools/mtmd/clip-graph.h
#	tools/mtmd/clip-model.h
#	tools/mtmd/clip.cpp
#	tools/mtmd/models/models.h

fix: gemma 4 template (#21326)

chat : avoid including json in chat.h (#21306)

jinja: coerce input for string-specific filters (#21370)

common : fix tool call type detection for nullable and enum schemas (#21327)

* common : fix tool call type detection for nullable and enum schemas

* common, tests : fix grammar delegation for nullable/enum schemas and add tests

Fix enum type inference to scan all enum values (not just index 0) so
schemas like {"enum": [0, "celsius"]} correctly detect string type.

Fix schema_delegates in peg-parser to handle nullable type arrays
(["string", "null"]) and typeless enum schemas in raw mode, allowing
the tagged parser to use raw text instead of JSON-formatted strings.

Add test cases for Qwen3-Coder (TAG_WITH_TAGGED format):
- nullable string ["string", "null"]
- nullable string with null first ["null", "string"]
- nullable integer ["integer", "null"]
- enum without explicit type key

common/parser: fix call ID detection (Mistral parser mostly) + atomicity for tag-json parsers (#21230)

* Fix call ID detection (Mistral parser mostly) + atomicity for tag-json parsers

* Rename

* Update common/chat-auto-parser-generator.cpp

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com>

---------

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com>

common : add gemma 4 specialized parser (#21418)

* common : add gemma4 dedicated parser

* cont : add '<|tool_response>' as eog

* cont : emit JSON from Gemma4 tool call AST

* cont : more fixes

* cont : refactor convert function

* cont : refine rules and mapping

* cont : add more tests

* cont : clean up

* cont : remove autoparser gemma4 implementation

* cont : more cleanup

* cont : rename gemma4.jinja to match the others

* cont : add custom template to support interleaved thinking

* cont : preserve reasoning in model turns

* cont : fix initializer error

* cont : fix unused vars

* cont : fix accidental static

* cont : fix specialized_template signature

* fix extra semicolon

* remove debug line and extra space [no ci]

fix reasoning budget

parser: fix MiniMax handling (#21573)

jinja : support ensure_ascii=true, string repetition and int/float self-filtering (#21623)

* feat: jinja engine improvements for reka-edge

Port three Jinja engine improvements needed for the reka-edge model:
1. Python-style string repetition ("ab" * 3 → "ababab")
2. ensure_ascii=true support for tojson filter (escapes non-ASCII to \uXXXX)
3. int() builtin on value_int_t (identity, needed for Reka Edge template)

* fix: escape invalid utf8 bytes when ensure_ascii=true

The json_ensure_ascii_preserving_format function does not correctly
handle an edge case where if UTF-8 parsing fails, it adds the non-ascii
character back to the output as a raw byte.

This commit fixes that by adding the unicode standard replacement
character \\ufffd to the output instead. This is the standard behavior
for various programming languages like Python, Rust, Go, etc.

* chore: address PR comments

1. Add todo comment for supporting string repetition for array/tuples
2. Add support for float identity operation
3. Move invalid ascii test case to test_fuzzing

* chore: accept suggestion for common/jinja/value.cpp

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com>

---------

Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@scala.com>

common : simplify autoparser tagged parser rules (#21216)

* common : simplify autoparser tagged parser rules

* cont : remove upper limit on optional args

* cont : revert changes to parsing at the end

* cont : undo arbitrary ordering of optional args

* cont : fix uninitialized required parameters

* revert to simplify merge

* re-apply patches

* restore flexible optional arg ordering tests

common : fix ambiguous grammar rule in gemma4 (#21661)

* common : fix ambiguous grammar rule in gemma4

* cont : fix missing comma...

common : enable reasoning budget sampler for gemma4 (#21697)

* fix: enable reasoning budget sampler for gemma4

Add thinking_start_tag and thinking_end_tag to
common_chat_params_init_gemma4(). Without these, the reasoning
budget sampler never activates for gemma4.

Make the newline after "thought" optional in the PEG parser to
handle budget=0 (sampler forces end tag before the newline).

Add test case for empty thinking block.

Fixes #21487

* use p.space() instead of p.optional(p.literal("\n")) in gemma4 thought parser

common : better align to the updated official gemma4 template (#21704)

fix: Fix broken structured output when using $refs in json_schema (#21699)

chat: dedicated DeepSeek v3.2 parser + "official" template (#21785)

Hide render_message_to_json warning

common/gemma4 : handle parsing edge cases (#21760)

common: skip reasoning budget sampler when no budget is requested (#21870)

* common: skip reasoning budget sampler when no budget is requested

After I added thinking_start_tag / thinking_end_tag for gemma4 in #21697, the reasoning budget sampler gets unconditionally created even when no budget is configured (the default -1). The same applies to kimi_k2, lfm2, lfm2_5, and ministral_3 which also set these tags. The budget gets converted to INT_MAX, so the sampler never actually forces any tokens but still runs per-token checks (start tag matching in IDLE state, token-to-piece conversion + UTF-8 checks in COUNTING state).

More importantly, the mere existence of the sampler (non-null rbudget) disables backend sampling. Backend sampling lets the GPU select tokens directly, avoiding a full logits transfer from GPU to CPU every token. This could explain the 30% speed regression reported in #21784 (98 t/s to 70 t/s on Vulkan).

So I added a reasoning_budget_tokens >= 0 check to the sampler creation condition. When the budget is unlimited, the sampler is not created, backend sampling stays enabled, and no per-token overhead is added. When a budget is explicitly set (0, 128, 1024, etc.), the sampler is created and works as before.

* common: preserve rbudget when grammar is lazy

Following up on the review feedback on #21870: keep the reasoning budget sampler when grammar_lazy is true, so the thinking-block grammar suppression from #20970 still works when tools are in use. This way, we only skip the sampler when both no budget is set AND grammar is not lazy.

autoparser: support case of JSON_NATIVE with per-call markers (test case: Reka-Edge) (#21892)

* fix grammar

* fix add sampled token

---------

Co-authored-by: Piotr Wilkin (ilintar) <piotr.wilkin@syndatis.com>
Co-authored-by: firecoperana <firecoperana>

2026-04-22 10:04:13 +02:00

.devops

Update nix flake: sync with upstream, fix for newer nixpkgs (#1371 )

2026-03-24 08:17:52 +01:00

.github

Fix Build and Push Container Image (#1633 )

2026-04-16 11:24:19 +02:00

common : introduce composable PEG parser combinators for chat parsing and new jinja template engine (#1369 )

2026-03-09 11:03:33 +01:00

cmake

Graph parallel: the next generation (#1080 )

2025-12-24 08:31:48 +01:00

common

Autoparser - complete refactoring of parser architecture (#1376 )

2026-04-22 10:04:13 +02:00

docker

Added workflow to build container images (#1279 )

2026-04-10 08:06:47 +02:00

docs

Autoparser - complete refactoring of parser architecture (#1376 )

2026-04-22 10:04:13 +02:00

examples

Autoparser - complete refactoring of parser architecture (#1376 )

2026-04-22 10:04:13 +02:00

ggml

Add all quantization types to Mistral4 MLA on the CPU (#1666 )

2026-04-20 16:11:06 +02:00

gguf-py

Vibe coded script + constants from mainline + pip requirements (#1405 )

2026-03-11 15:41:17 +01:00

github-data

Add GitHub data: filename sanitization (#640 )

2025-07-23 13:31:53 +02:00

grammars

llama : add token matching support to llama-grammar (#1220 )

2026-02-03 07:57:17 +02:00

include

Autoparser - complete refactoring of parser architecture (#1376 )

2026-04-22 10:04:13 +02:00

media

README: add graphic for matrix multiplication (#6881 )

2024-04-24 21:29:13 +02:00

models

Autoparser - complete refactoring of parser architecture (#1376 )

2026-04-22 10:04:13 +02:00

pocs

build: rename main → llama-cli, server → llama-server, llava-cli → llama-llava-cli, etc... (#7809 )

2024-06-13 00:41:52 +01:00

prompts

llama : add Qwen support (#4281 )

2023-12-01 20:16:31 +02:00

requirements

Vibe coded script + constants from mainline + pip requirements (#1405 )

2026-03-11 15:41:17 +01:00

scripts

Autoparser - complete refactoring of parser architecture (#1376 )

2026-04-22 10:04:13 +02:00

spm-headers

Merge mainline llama.cpp (#3 )

2024-07-27 07:55:01 +02:00

src

Autoparser - complete refactoring of parser architecture (#1376 )

2026-04-22 10:04:13 +02:00

tests

Autoparser - complete refactoring of parser architecture (#1376 )

2026-04-22 10:04:13 +02:00

vendor

common : introduce composable PEG parser combinators for chat parsing and new jinja template engine (#1369 )

2026-03-09 11:03:33 +01:00

.dockerignore

Added workflow to build container images (#1279 )

2026-04-10 08:06:47 +02:00

.ecrc

Nomic Vulkan backend (#4456 )

2024-01-29 15:50:50 -05:00

.editorconfig

Merge mainline llama.cpp (#3 )

2024-07-27 07:55:01 +02:00

.flake8

py : logging and flake8 suppression refactoring (#7081 )

2024-05-05 08:07:48 +03:00

.gitignore

Add --webui arg to launch llama.cpp new webui (#786 )

2025-10-27 14:22:02 +02:00

.gitmodules

Merge mainline llama.cpp (#3 )

2024-07-27 07:55:01 +02:00

.mailmap

Add .mailmap

2025-07-22 14:53:50 +03:00

.pre-commit-config.yaml

convert.py : add python logging instead of print() (#6511 )

2024-05-03 22:36:41 +03:00

AUTHORS

Update AUTHORS

2026-04-16 08:55:50 +02:00

CMakeLists.txt

server: stop processing the prompt when client disconnects (#1134 )

2026-01-13 07:56:59 +02:00

CMakePresets.json

Merge mainline llama.cpp (#3 )

2024-07-27 07:55:01 +02:00

CONTRIBUTING.md

Merge mainline - Aug 12 2024 (#17 )

2024-08-12 15:14:32 +02:00

convert_hf_to_gguf_update.py

model : Port Minimax M2 from mainline (#907 )

2025-11-06 18:09:24 +02:00

convert_hf_to_gguf.py

Add support for Seed-OSS (#1218 )

2026-02-03 07:39:45 +02:00

convert_imatrix_gguf_to_dat.py

Vibe coded script + constants from mainline + pip requirements (#1405 )

2026-03-11 15:41:17 +01:00

convert_llama_ggml_to_gguf.py

Merge mainline llama.cpp (#3 )

2024-07-27 07:55:01 +02:00

convert_lora_to_gguf.py

Fix missing rope_freqs with convert_hf_to_gguf (#402 )

2025-05-09 09:17:41 -05:00

docker-bake.hcl

Fix Build and Push Container Image (#1633 )

2026-04-16 11:24:19 +02:00

docker-bake.override.hcl

Added workflow to build container images (#1279 )

2026-04-10 08:06:47 +02:00

flake.lock

Update nix flake: sync with upstream, fix for newer nixpkgs (#1371 )

2026-03-24 08:17:52 +01:00

flake.nix

Update nix flake: sync with upstream, fix for newer nixpkgs (#1371 )

2026-03-24 08:17:52 +01:00

LICENSE

Use links for ggml/llama.cpp authors (#318 )

2025-04-07 17:25:06 +02:00

Makefile

Add Unicode allowlist (#1597 )

2026-04-10 18:22:57 +02:00

mypy.ini

convert : partially revert PR #4818 (#5041 )

2024-01-20 18:14:18 -05:00

Package.swift

Merge mainline llama.cpp (#3 )

2024-07-27 07:55:01 +02:00

poetry.lock

Merge mainline llama.cpp (#3 )

2024-07-27 07:55:01 +02:00

pyproject.toml

Merge mainline llama.cpp (#3 )

2024-07-27 07:55:01 +02:00

pyrightconfig.json

Merge mainline llama.cpp (#3 )

2024-07-27 07:55:01 +02:00

README.md

Update docs (#1606 )

2026-04-10 18:20:28 +02:00

requirements.txt

Vibe coded script + constants from mainline + pip requirements (#1405 )

2026-03-11 15:41:17 +01:00

test-function-calls.md

Function calling support for Kimi-K2 (#628 )

2025-07-23 18:11:42 +02:00

README.md

ik_llama.cpp: llama.cpp fork with better CPU performance

TL;DR

This repository is a fork of llama.cpp with better CPU and hybrid GPU/CPU performance, new SOTA quantization types, first-class Bitnet support, better DeepSeek performance via MLA, FlashMLA, fused MoE operations and tensor overrides for hybrid GPU/CPU inference, row-interleaved quant packing, etc.

Note

The only fully functional and performant compute backends are CPU (AVX2 or better, ARM_NEON or better) and CUDA (Turing or newer). Please do not enter issues related to ROCm, Vulkan, Metal, old Nvidia GPUs, AVX CPUs, etc. They will not get resolved unless you roll up your sleeves and help bring your favorite backend up to speed. With the current regular contributors this project simply does not have the bandwidth to work on all backends available in llama.cpp.

Important

Do not use quantized models from Unsloth that have _XL in their name. These are likely to not work with ik_llama.cpp.

The above has caused some stir, so to clarify: the Unsloth _XL models that are likely to not work are those that contain f16 tensors (which is never a good idea in the first place). All others are fine.

Note

Some users have reported issues with graph parallel (a.k.a. split mode graph) and partial GPU offload (using --cpu-moe or --n-cpu-moe or tensor overrides). If you are using/want to use split mode graph and observe gibberish/incoherent responses, try adding -cuda graphs=0 to your command line.

Quickstart

Prerequisites

git clone https://github.com/ikawrakow/ik_llama.cpp

cd ik_llama.cpp

On Debian/Ubuntu Linux, install the required packages (if using another Linux distro, you need to find the corresponding packages and adapt):

apt-get update && apt-get install build-essential git libcurl4-openssl-dev curl libgomp1 cmake

Build for CPU

cmake -B build -DGGML_NATIVE=ON

cmake --build build --config Release -j$(nproc)

Build for GPU

Install Nvidia Drivers and CUDA Toolkit.

cmake -B build -DGGML_NATIVE=ON -DGGML_CUDA=ON

cmake --build build --config Release -j$(nproc)

Step-by-step instructions for a case of a successful Windows build

https://github.com/ikawrakow/ik_llama.cpp/blob/main/docs/build.md

Run

Download .gguf model files (e.g. bartowski/Qwen_Qwen3-0.6B-IQ4_NL.gguf) to your favorite directory (e.g. /my_local_files/gguf).

Start the server with one of the commands (CPU or GPU):

./build/bin/llama-server --model /my_local_files/gguf/Qwen_Qwen3-0.6B-IQ4_NL.gguf --ctx-size 4096

./build/bin/llama-server --model /my_local_files/gguf/Qwen_Qwen3-0.6B-IQ4_NL.gguf --ctx-size 4096 -ngl 999

That's all! Open http://127.0.0.1:8080 in Browser start chatting.

Step by step guide for ik_llama.cpp in podman/docker container including llama-swap

Common parameters and options

Latest News

Model Support

LlaMA-3-Nemotron PR 377, Qwen3 PR 355, GLM-4 PR 344, Command-A PR 341, bitnet-b1.58-2B-4T PR 337, LLaMA-4 PR 321, Gemma3 PR 276, DeepSeek-V3 PR 176, Kimi-2 PR 609, dots.llm1 PR 573, Hunyuan PR 565, GLM-4.5 PR 668 (4.5/4.6/4.7/AIR), Ernie 4.5 MOE and 0.3B PR 759, grok-2 PR 782, Ling/Ring (Bailing-MoE2) PR 833, Qwen3-VL PR 883, SmolLM3 PR 934, GigaChat3 PR 995, ministral3 PR 1030, Mimo-V2-Flash PR 1096, GLM-4.7-Flash PR 1168, Seed-OSS PR 1218, Step-3.5-Flash PR 1231, GLM-5 PR 1268, Qwen3-Next PR 1266, Qwen3.5-MoE PR 1288 and dense Qwen-3.5 1326, Mistral 4 PR 1450, Bonsai 1-bit PR 1570, Gemma4 PR 1581

Quantization

Quantization additions

Trellis quants (`IQ1_KT`, `IQ2_KT`, `IQ3_KT`, `IQ4_KT`)

Information and the original CUDA implementation in PR 113. Additional implementations: Metal PR 475, Neon PR 471, CPU PR 441. IQ1_KT was added more recently in PR 616. Note: these are base on a novel, integer-base trellis, which allows to achieve reasonable CPU performance, see PR 529 and PRs quoted there for details.

IQK quants

Information can be found in Discussion 8.

Initial implementations (Zen4, AVX2, NEON): IQ5_KS_R4 PR 426, IQ5_KS PR 422, IQ4_KS_R4 PR 150, IQ5_K_R4 PR 149, IQ2_K_R4 PR 146, IQ3_K_R4 PR 145, IQ4_K_R4 PR 138, IQ4_KSS PR 89, IQ2_KS PR 85, IQ4_KS PR 83, IQ6_K PR 14, IQ2_K, IQ3_K and IQ5_K PR 7, IQ4_K PR 6

Cuda implementations: IQ4_KS_R4 and IQ5_KS_R4 PR 493, IQ1_S_R4 PR 492, IQ1_M_R4 PR 494. IQ4_KS_R4 and IQ5_KS_R4 PR 462, IQ2_K_R4, IQ3_K_R4, IQ4_K_R4, IQ5_K_R4 PR 461, IQ4_K, IQ5_K, IQ6_K PR 417, IQ2_KS, IQ2_K, IQ3_K PR 418

IQ2_KL is a more recent addition in PR 602

Hadamard transforms for K-cache

CPU PR 1033 and CUDA PR 1034

Hadamard transforms for V-cache

PR 1527

MXFP4 as used in gpt-oss models

Implemented for Zen4, AVX2, ARM_NEON, Metal, CUDA PR 682

Quantization improvements

IQ1_M PR 327, IQ2_XS PR 312, Q2_K, Q4_K, Q5_K, Q4_1, Q5_1 PR 302, Q4_0, Q5_0, Q6_0, Q3_K, Q6_K, IQ4_XS, IQ4_NL PR 295
Low perplexity Q4_0 KV cache PR 1547 PR 1556

Quantization performance improvements

Much faster CPU prompt processing for all non-interleaved quants. Initial idea in PR 515 and PR 531, with many follow up PRs to apply to all quantization types for the 3 supported CPU platforms.
All quantization types now have quantized matrix multiplication CUDA kernels, see PR 557 and several others
Faster CPU prompt processing for Trellis quants and MoE models. PR 488
Trellis quants: faster CPU prompt processing PR 482.
Minor (~2%) iq2_ks TG performance improvement on CUDA PR 468
Faster IQ3_KT and IQ4_KT PR 453
Zen4: Faster PP for IQ2_KS, IQ4_KS, IQ5_KS PR 428
Fast GEMM/GEMV for IQ1_S PR 212
AVX-VNNI optimizations PR 1446 PR 1455 PR 1467 PR 1474 PR 1482

Features

New split mode "graph" for multi GPU setups PR 1022
Fused delta-net for Qwen3-Next and Qwen3.5-MoE PR 1315 PR 1333 PR 1362 PR 1373
Hadamard transforms for K-cache and V-cache PR 1033 PR 1034 PR 1527
Auto-fit offloaded tensors to available VRAM (MoE and dense models) PR 1501 PR 1504
Checkpoints for recurrent models PR 1310 PR 1398
String ban function for all completions PR 1185 PR 1243
OpenAI /v1/responses API endpoint PR 1184
Function call support PR 628
jinja template support PR 677
Webui: New Features for Conversations, Settings, and Chat Messages PR 618
MTP decoding support for GLM-4.x MoE 1270
Self speculative decoding, ngram PR 1261
Dynamic control vector management endpoints PR 1223
Legacy quants conversion schemes in convert_hf_to_gguf.py PR 449, Q6_0 in PR 483
Adaptive-P Sampler PR 1100 implemented as designed by it's author; supported on Webui
Multi-modal Vision support in llama-mtmd-cli PR 798 and in llama-server PR 901
mikupad as an alternative WebUI PR 558
June 8 2025: Webui updated (legacy still available when --path ./examples/server/public_legacy is passed) PR 481
June 8 2025: RPC improvements PR 480
June 7 2025: Add an endpoint that lists all the saved prompt caches to server PR 502
June 6 2025: Make prompt cache saving and restoring MLA aware PR 497
June 3 2025: Added samplers, XTC PR 486, top-n σ PR 489.
May 22 2025: Refactor iqk_mul_mat.cpp which speeds up compilation time significantly. PR 435
May 17 2025: Option to enable or disable the CPU FA kernels PR 429.
May 12 2025: User can now control if/which operations with tensors held in RAM are offloaded to the GPU. See PR 405
May 12 2025: Compatibility issues with mainline llama.cpp GGUFs for DeepSeek models with MLA enabled were resolved in PR 394. The lower prompt processing performance resulting from using llama.cpp-style MLA GGUFs was recovered in PR 409.
April 21 2025: ik_llama.cpp builds and runs successfully on Android (using termux), see PR 336
March 1 2025: Smart Expert Reduction for faster DeepSeek inference PR 239
Feb 25 2025: Tensor overrides for better control where model weights are stored (GPU or CPU) PR 232
Feb 23 2025: sweep-bench - better performance benchmarking PR 225
Feb 19 2025: Q8_KV - new type for 8-bit KV-cache quantization PR 208
March 7 2025: Custom quantization mixes using regular expressions PR 244

Performance improvements

Better GPU offload strategy for MoE models when using hybrid HPU/CPU inference, see PR 520
Much faster rng sampling PR 1187
May 13 2025: Better CPU FA performance for DeepSeek-Lite. PR 410
May 11 2025: Slightly faster flash attention for DeepSeek models on CUDA, along with extending compatibility to Touring or newer GPUs. PR 408
May 4 2025: Significant token generation performance improvement on CUDA with Flash Attention for GQA models. For details and benchmarks. PR 370
April 17 2025: Better CPU Flash Attention token generation performance. PR 332
April 3 2025: Much faster MoE implementation on Metal. PR 307
March 25 2025: Better MoE performance on CUDA PR 283
March 23 2025: Better batched processing speed for DeepSeek models PR 282
March 18 2025: Reduce compute buffer size PR 237
March 10 2025: Better TG performance for MoE models on CUDA PR 248
Feb 23 2025: Fused FFN ops for faster MoE inference PR 229

Flash-MLA

May 7 2025: 🚀 FlashMLA-3 for DeepSeek models on CUDA. PR 386. Caveat: Ampere or newer Nvidia GPU required
March 21 2025: 🚀 FlashMLA-3: fastest CPU-only inference for DeepSeek models PR 273
March 17 2025: 🚀 FlashMLA-2 performance improvements PR 253
March 12 2025: Allow Q8_0 KV cache with FlashMLA-2 on CUDA PR 265
March 9 2025: 🚀 FlashMLA on CUDA PR 247
March 8 2025: 🚀 Faster FlashMLA CPU implementation PR 243
March 3 2025: 🚀 Introducing FlashMLA - MLA with Flash Attention PR 240
Feb 27 2025: MLA without transposed cache PR 235
Feb 13 2025: Allow Q8_0 quantized cache with MLA PR 206
Feb 11 2025: 🚀 Flash Attention support for DeepSeek models PR 200
Feb 9 2025: 🚀 MLA for DeepSeek models PR 188

Fixes

Fix bug in MMVQ kernel PR 446
Fix AVX2 implementation of IQ4_K, IQ4_KS, IQ5_K, IQ6_K PR 427
Fix standard attention on the CPU PR 421
Fix imatrix calculation for MLA models PR 411
Fix new CUDA FA on Touring PR 413
Fix SER. CPU: PR 415 CUDA: PR 416

Resources

There is no single point of reference describing all new ik_llama.cpp features. Pull requests often contain detailed information, so browsing the PRs is often the best way to learn about new features and how to use them. In addition

The Wiki page has performance comparisons to mainline llama.cpp
This guide is a good place to start if you came here because of DeepSeek models
This discussion is about running DeepSeek-V3/R1 on a 16 x 3090 setup
This discussion describes the new quantization types available in ik_llama.cpp

Testing

Function Calls Tests

To run the function calls test suite:

cd build
cmake --build . --target test-function-calls
./bin/test-function-calls

The test suite covers parser functionality, streaming, error handling, content cleaning, and server integration. All tests should pass to ensure production readiness.

Contributing

Contributions in form of pull requests, issue submissions (bug reports, feature requests), or general discussions, are welcome.

License

subprocess.h - Single-header process launching solution for C and C++ - Public domain
server
GBNF grammars

Development documentation

Seminal papers and background on the models

If your issue is with model generation quality, then please at least scan the following links and papers to understand the limitations of LLaMA models. This is especially important when choosing an appropriate model size and appreciating both the significant and subtle differences between LLaMA models and ChatGPT:

LLaMA:
- Introducing LLaMA: A foundational, 65-billion-parameter large language model
- LLaMA: Open and Efficient Foundation Language Models
GPT-3
- Language Models are Few-Shot Learners
GPT-3.5 / InstructGPT / ChatGPT:
- Aligning language models to follow instructions
- Training language models to follow instructions with human feedback

Completions

Command-line completion is available for some environments.

Bash Completion

$ build/bin/llama-cli --completion-bash > ~/.llama-completion.bash
$ source ~/.llama-completion.bash

Optionally this can be added to your .bashrc or .bash_profile to load it automatically. For example:

$ echo "source ~/.llama-completion.bash" >> ~/.bashrc

Dependencies

yhirose/cpp-httplib - Single-header HTTP server, used by llama-server - MIT license
stb-image - Single-header image format decoder, used by multimodal subsystem - Public domain
nlohmann/json - Single-header JSON library, used by various tools/examples - MIT License
miniaudio.h - Single-header audio format decoder, used by multimodal subsystem - Public domain
subprocess.h - Single-header process launching solution for C and C++ - Public domain

Languages

C++ 59.4%

C 14.4%

Cuda 12.9%

Python 5.2%

Metal 2.6%

Other 5.4%

README.md Unescape Escape

ik_llama.cpp: llama.cpp fork with better CPU performance

TL;DR

Quickstart

Prerequisites

Build for CPU

Build for GPU

Step-by-step instructions for a case of a successful Windows build

Run

Step by step guide for ik_llama.cpp in podman/docker container including llama-swap

Common parameters and options

Latest News

Model Support

Quantization

Quantization additions

Trellis quants (IQ1_KT, IQ2_KT, IQ3_KT, IQ4_KT)

IQK quants

Hadamard transforms for K-cache

Hadamard transforms for V-cache

MXFP4 as used in gpt-oss models

Quantization improvements

Quantization performance improvements

Features

Performance improvements

Flash-MLA

Fixes

Resources

Testing

Function Calls Tests

Contributing

License

Development documentation

Seminal papers and background on the models

Completions

Bash Completion

Dependencies

README.md

Trellis quants (`IQ1_KT`, `IQ2_KT`, `IQ3_KT`, `IQ4_KT`)