* Add interface is_model_splitted() to check the c-graph is splited or not
* Infer and propagate dynamic-dimension indices for all tensors in the GGML graph in api compute_model_outputs()
* Only do this for fallback sub graph
* Move dynamic dims compute in graph missmatch
* ggml-openvino: fix tensor data handling for PERMUTE/VIEW ops in split models
* ggml-openvino:add comments
* ggml-openvino: override VIEW op_case to 0 for split model inputs
* openvino backend: Handle unsupported VIEW shape-mismatch in OpenVINO backend
* Enable additional mul_mat tests and add tensor data saving function (#81)
* ggml-openvino: fix CONT/TRANSPOSE mapping and improve dynamic-dimension handling
* OpenVINO: add NORM/TANH support and rework SOFT_MAX translation
* ggml-openvino: extend VIEW handling
* Enable -fa off (#118)
* Enable --context-shift
* Fix llm param compute error for normal softmax not the softmax in attention
* OpenVINO backend: fix error for attention size compute in llm param
* use tensor->extra in infer_request i/o
* OpenVINO backend: refacter the compute_llm_params() func add get_attention_pattern_case to easy extand
* OpenVINO backend: clean unused code
* 1to1 match op update (#146)
* added translate_1to1_match_1_input function and updated gelu and tanh translations
* Remove unused translation function calls
---------
Co-authored-by: Mustafa Cavus <mustafacavus@intel.com>
* initial gemma4 support
* removed hardcoded names for kv cache slicing
* OpenVINO backend: Add new attention pattern for llm parameters compute
* flash attn Q shape static conversion
* Remove slice in permute translation when n_seq is 1
* return optional in extract_layer_from_name
* OpenVINO backend: refactor VIEW related operation (#148)
* OpenVINO backend: refactor VIEW related operation
* Enable VIEW handling in following ops
* OpenVINO backend does not support GGML_OP_NORM & GGML_OP_L2_NORM with VIEW input accuracy issue from OpenVINO
* OpenVINO backend: Add ops l2_norm & pad
* OpenVINO backend does not support CPY with non-contiguous data or mismatched types
* add op SSM_CONV GATED_DELTA_NET
* OpenVINO backend: fix error for bf16 in OV gpu plugin
* reverted static Q input shape for attention layer
* OpenVINO backend: remove hardcode name inp_tokens, which ignore some leaf case
* Disable remote tensor due to bug in ov gpu
* Disable n_token > 1 GATED_DELTA_NET on gpu
* OpenVINO backend: fix the view op dynamic handling issue in gemma4 & enable view + get_row
* OpenVINO backend: clean code
* OpenVINO backend: enable view + norm/rms_norm
* OpenVINO backend: concat op
* OpenVINO backend: argsort op
* OpenVINO backend: enable unary + view & GGML_UNARY_OP_SOFTPLUS
* Fix issue for test-backend-ops in TOPK_MOE, which compare VIEW ops result, VIEW node in OpenVINO no need compare, the whole graph result is correct
* OpenVINO backend: enable sum_rows
* OpenVINO backend: enable clamp
* OpenVINO backend: enable DIV
* OpenVINO backend: enable GGML_OP_MUL_MAT_ID
* OpenVINO backend: disable MUL_MAT_ID_FUSION case with large mem needed
* OpenVINO backend: Disable GGML_OP_ARGSORT, cause test_backend-ops failed
* OpenVINO backend: fix issue in mul_mat_id
* OpenVINO backend: Disable DIV with broadcast on GPU
* OpenVINO backend: update DIV
* use ov internal op GatedDeltaNet
* OpenVINO backend: enable llama erch test qwen3next
* OpenVINO backend: enable RMS_NORM + VIEW & remove op_case 2 for rope
* OpenVINO backend: fix error
* suggested changes, need review
* suggested changes, need review
* OpenVINO backend: clean unused code & fix build warning
* OpenVINO backend: enable minicpm3 for arch test
* Disable GDN op (#177)
* disable gated_delta_net
* update stateful_kv_size correctly in mismatch case
* OpenVINO backend: enable arch test for qwen3vl
* OpenVINO backend: enable cohere2 for arch test
* OpenVINO backend: enable t5 for arch test
* OpenVINO backend: enable jamba for arch test
* OpenVINO backend: remove warning for tmp
* OpenVINO backend: enable kimi-linear for arch test
* Remove unused
* Fix gpt-oss accuracy issue
* OpenVINO backend: enable arctic for arch test
* OpenVINO backend: enable grok for arch test
* Gemma4 initial npu support (#179)
* Initiall gemma4 npu support
* temp. fix for gemma4 accuracy bug on npu
* Remove hardcoded names for npu-fold handling
* revert static n tokens for cont translation as it is not needed
* removed unused variable
* ggml-openvino: add GGML_OPENVINO_ENABLE_CACHE env var to control decoder cache. Add environment variable GGML_OPENVINO_ENABLE_CACHE (default: YES). When set to NO, the decoder_cache is bypassed and models are rebuilt from the cgraph on every inference call in both dynamic and static compute paths. This is useful for debugging and verifying correctness without caching interference.
* Revert "Gemma4 initial npu support (#179)"
This reverts commit 0d29a9c4a52dc2c8aa52990f1a3854cfb01768ad.
* OpenVINO backend: disable debug log print
* Update TBB discovery. Delegated to OpenVINOs own config.
* OpenVINO backend: GGML_OPENVINO_ENABLE_CACHE YES -> 1
* OpenVINO backend: fallback FLASH_ATTN_EXT in gemma3n to CPU backend
* Add raw ov infer profiling metric
* Add OV raw infer time metric to static compute path
Co-authored-by: virajwad <84867530+virajwad@users.noreply.github.com>
* Modify precision of static profiling
* update to OV 2026.2, add OV windows CI
* fix editorconfig-checks
* Initiall gemma4 npu support
* temp. fix for gemma4 accuracy bug on npu
* Remove hardcoded names for npu-fold handling
* revert static n tokens for cont translation as it is not needed
* removed unused variable
* test-llama-archs fix
* Fix gemma4 flash_attn fallback
* support im2col
* fix code style
* disable add_rope_sin_cos optimization
* stateless boradcast and rope optimizations
* Enable manual gqa attn by default for stateless gpu
* manual gqa: fixed static batch
* gemma4 llama-bench ctx update fix
* Update OV win CI
* stateful rope fusion temp. fix
* OpenVINO backend: Conslolidate supported ops
* Exclude unsupported GGML_OP_SUB cases
* Exclude unsupported TOPK_MOE cases
* OpenVINO Backend: MUL_MAT enhancements
* Update OV CI
* support f16 mask input for npu
* Make GGML_OPENVINO_* env vars usage uniform
Standardize all GGML_OPENVINO_* env flags:
positive integers >0 to enable. Unset, empty, =0, or non-numeric values to disable.
This fixes cases where text values or empty strings enabled features.
* OpenVINO backend: Enhance envvar handling
* more cleanup
* move ggml_openvino_env_flag to appropriate place
* OpenVINO backend: add REPEAT translator, Q5_1 weights, and GLU view-input fix
* ggml-openvino: fix -Werror=cast-qual in extract_q5_1_data
* Update openvino.Dockerfile
Use BuildKit cache mounts for faster Docker rebuilds.
Use apt instead of dpkg, remove unused .ddeb downloads, add DLLAMA_BUILD_TESTS=OFF.
* ggml-openvino: centralize env var access via *getenv_str/getenv_int helpers
Replace getenv and legacy flags with _str and _int helpers.Minor cleanup, doc updates.
* OpenVINO backend: Enable GGML_OP_ADD_ID
* Uptade openvino backend clamg-format
* clang-format
* Update OPENVINO.md (#211)
* OpenVINO backend: fix accuracy issue for op CONCAT with i64 precision
* Remove strict concurrency for gpu-openvino-low-perf
* Update openvino CI keynames; add ccache-clear
* Apply suggestions from code review
Co-authored-by: Sigbjørn Skjæret <1629204+CISC@users.noreply.github.com>
* Fix formatting
---------
Co-authored-by: Xuejun Zhai <Xuejun.Zhai@intel.com>
Co-authored-by: Mustafa Cavus <mustafa.cavus@intel.com>
Co-authored-by: Mustafa Cavus <mustafacavus@intel.com>
Co-authored-by: Xuejun <XuejunZhai@intel.com>
Co-authored-by: Wang Yang <yang4.wang@intel.com>
Co-authored-by: Ravi Panchumarthy <ravi.panchumarthy@intel.com>
Co-authored-by: virajwad <84867530+virajwad@users.noreply.github.com>
Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: Mostafa Faheem <mostafaaafaheem@gmail.com>
Co-authored-by: Sigbjørn Skjæret <1629204+CISC@users.noreply.github.com>
* SYCL: fix a bug with async memcpy
* make mmid_row_mapping_host persistent
* comment on stream->wait
* Apply suggestion from @sanmai
* Apply suggestion from @sanmai
* Apply suggestion from @sanmai
This introduces an optional feature to allocate large GPU buffers (≥ 1GB)
using USM system allocations if supported by the device. It allows using
buffers from the system allocator then letting the system manage memory
migrations between host and device as necessary.
This feature is disabled by default and requires the GGML_SYCL_USM_SYSTEM
environment variable to enable. If USM system allocations are not supported
by the device or the system, we fallback to regular allocations.
This feature can allow VRAM overcommit. For example, the test below fails
on B580 due to lack of memory for allocation, but it passes when enabling
USM system allocations:
./examples/sycl/test.sh -m Qwen3.5-27B-Q3_K_M.gguf -lv 4
Signed-off-by: Francois Dugast <francois.dugast@intel.com>
* sycl: support reordered Q4_K and Q5_K MoE MUL_MAT_ID
Extend reordered-weight handling to fused MoE MUL_MAT_ID for Q4_K and Q5_K expert tensors and add Q5_K reordered DMMV coverage. Unsupported 3D reorder cases now fall back instead of aborting.
* sycl: extend MoE reorder to Q6_K mul_mat_id
* vulkan: add GGML_OP_COL2IM_1D, follow-up to the CPU op
* vulkan: col2im_1d bounded gather loop instead of full-K scan with modulo
* vulkan: col2im_1d address review from @jeffbolznv
* vulkan: col2im_1d return nullptr for unsupported types, address review from @0cc4m
* [SYCL] Centralize Level Zero detection in ggml_sycl_init
* use the same wording
* get back the warning
* [SYCL] Remove per-allocation getenv() for GGML_SYCL_ENABLE_LEVEL_ZERO
* bring back the comment
* move it up to make sure devices call the shots
* move the env detection early
* replace g_ggml_sycl_enable_level_zero with a direct call to .ext_oneapi_level_zero
* update the comment
* switch back to g_ggml_sycl_enable_level_zero with a sentinel
* remove the check
* Reduce the diff
* reword, move lower
* move things aroudn
* remove forward declaration if favor of a full replace
* pre-cache results of zeDeviceGetProperties
* put ggml_sycl_get_env back
* replace get_sycl_env with ggml_sycl_get_env
* add whitespace back
* Apply suggestion from @sanmai
* vulkan: support non-contig unary/glu ops
Change unary/glu ops to pass in all strides and use fastdiv for the index
calculation. Put all unary ops in one file, similar to glu, to share the
code. codex went ahead and added expm1 without me asking, but I had to
make it do a real precision analysis rather than just making stuff up.
unary.comp initially couldn't use generic_unary_head because there wasn't
space for xielu's additional constants. Fixing this required packing the
fastdiv 'L' values.
* attempt to workaround compiler bug
* resolve conflict from #23991
* use expm1
* Make ggml_gated_delta_net take only the initial recurrent state (D, 1, n_seqs) and passes the snapshot count K as an op parameter instead of inferring it from state->ne[1].
Remove the padding hack and copy all emitted snapshots into the recurrent cache with a single strided ggml_cpy
* Make GDN changes in all backends. Address review comments.
* Fix CI build errors
* vulkan: add support for valve fp16 dot2 extension
* use macro for dot2 path choice
* properly check for the feature
* add dot_product abstraction to reduce preprocessor branching
* cpu: add GGML_OP_COL2IM_1D
Add the overlap-add (scatter-add) step of a 1D transposed convolution.
A ConvTranspose1d factorizes as a GEMM followed by col2im: a weight
pre-permuted to [IC, K*OC] is contracted against the [IC, T_in] input
with mul_mat to produce a column matrix [K*OC, T_in], and col2im_1d
scatters those columns back into the [T_out, OC] signal, with
T_out = (T_in - 1)*s0 + K - 2*p0.
Keeping the contraction as a plain mul_mat leaves the heavy work on the
optimized (and quantizable) matmul kernels, so col2im_1d only does the
cheap overlap-add.
CPU uses a gather formulation parallelized over output channels,
supporting F32, F16 and BF16 with an F32 accumulator.
* tests: add backend coverage for GGML_OP_COL2IM_1D
Add test_col2im_1d next to the conv_transpose_1d cases, covering F32,
F16 and BF16 across eight geometries: the canonical kernel = 2*stride
DAC upsampling shape, overlap, no overlap, cropping (p0 = 1 and
p0 = stride/2), kernel < stride with zeroed gaps, kernel not a
multiple of stride, and a single column unfold.
Perf mode gets three real vocoder stage shapes reporting memory
bandwidth. max_nmse_err relaxes to 5e-4 for F16 and BF16.
* cpu: harden GGML_OP_COL2IM_1D
ggml_col2im_1d validates s0, oc, p0 and input contiguity at graph
build time, before the oc division, protecting every backend at once.
The kernel asserts the contiguity its flat indexing assumes and its
doc states the full output length including the crop term.
The kernel parallelizes over the time axis: the split stays balanced
down to OC = 1, where the previous channel split was single threaded.
Values are bit identical on the three real vocoder chains, two out of
three improve.
* tests: extend the GGML_OP_COL2IM_1D grid
The eval grid grows to eleven geometries: OC = 1 (mono output stage),
K = 1 with stride > 1 (sparse scatter, every gap position zeroed) and
a crop down to T_out = 2 where all the gather bounds act at once.
* tests: add col2im_1d equivalence test
tests/test-col2im-1d.cpp proves mul_mat + col2im_1d matches the
native ggml_conv_transpose_1d on the CPU backend, F32 bit exact, F16
and BF16 through casts of the column matrix. test-backend-ops cannot
cover this for a CPU only op since the CPU backend is its own
reference there.
* rpc: bump protocol patch version for GGML_OP_COL2IM_1D
GGML_OP_COUNT goes from 96 to 97 with the new op, which trips the
static_assert in ggml-rpc.h. Bump RPC_PROTO_PATCH_VERSION since the
op is appended and no existing op code shifts.
* Only run webgpu CI on my fork
* Add webgpu only workflow
* handle buffer overlap case for concat operator
* restore build-webgpu.yml
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* Run clang-format
* Update ggml/src/ggml-webgpu/wgsl-shaders/concat.wgsl
---------
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-authored-by: Reese Levine <reeselevine1@gmail.com>
* Only run webgpu CI on my fork
* Add webgpu only workflow
* Implement 2d workgroups for more operations
* fix
* Fix type
* Move back to global_invocation_id
This allows vec4 loads of the B elements. Also increase BK to 64 when this is
enabled. Neither of these alone is consistently faster, but together these give
a nice speedup.
In ggml-vulkan.cpp, we need to make sure the B matrix alignment and stride are
multiples of 4.
* cuda: reset device in get_memory function if no backend is active
* also count device and host buffers
* exclude hip and musa from counting and device reset
* use device mutex instead of atomic
* undo backend_free function move
