[GPU] Fix dynamic_quantize kernel sharing between different IFM sizes - #37222
[GPU] Fix dynamic_quantize kernel sharing between different IFM sizes#37222jade-cho wants to merge 1 commit into
Conversation
…openvinotoolkit#37140) ### Description of the issue(symptom, root-cause, how it was resolved) - **Symptom**: int4 VLMs produce garbage output on GPU in per-token dyn-quant mode. Reproduced on `gemma-4-31B-it-int4-ov` and `gemma-4-26b-a4b-it-int4-ov`. - **Root-cause**: Two `dynamic_quantize` nodes with a dynamic innermost dim (`?x?x?`) get identical `kernel_impl_params`, so `add_kernels_source()` drops the second one and both share a single kernel. In per-token mode the group count is baked into the JIT from `fc_ifm_size`, so the shared kernel is correct for only one of them. These models hit it because their sliding/full attention layers feed o_proj FCs with different IFM. - **How it was resolved**: The resolved innermost length is now carried by the primitive (`dynamic_quantize::innermost_size`) and joins `hash()` / `operator==`, so the two nodes no longer collide. `CreateDynamicQuantizeOp` resolves it once from the static dim, or from the weights IFM of the consuming `FullyConnectedCompressed`. #### The code and line that caused this issue (if it is not changed directly) - `intel_gpu/src/graph/impls/ocl/kernels_cache.cpp` - `add_kernels_source()` skips duplicate params - `dynamic_quantize_kernel_opt.cpp` - `GetJitConstants()` bakes `TOTAL_BLOCK_NUM` from `fc_ifm_size` - `intel_gpu/src/graph/impls/ocl/dynamic_quantize.cpp` - the old `get_users().front()` lookup gave a per-node value that never reached the cache key (openvinotoolkit#26850) #### Reproduction step and snapshot (if applicable. Do not attach for customer model) - `$ python vlm_test.py gemma-4-31B-it-int4-ov cat.jpg GPU` - Expected `The picture shows a grey tabby cat ...`, got unrelated tokens. - `gemma-4-26b-a4b-it-int4-ov` reproduces the same with `DYNAMIC_QUANTIZATION_GROUP_SIZE=18446744073709551615`. #### Problematic graph ``` sliding layer: SDPA -> Reshape(?x?x?) -> DynamicQuantize -> FC(W: [N, 8192]) full layer: SDPA -> Reshape(?x?x?) -> DynamicQuantize -> FC(W: [N, 16384]) ``` - Same layouts on both dq nodes; only the FC weights differ, which was not in the cache key. #### Checklist - [x] Is it a proper fix? (not a workaround) K (the innermost dimension) determines the quantization group count baked into the kernel at compile time, so it is now part of the primitive's hash()/operator==. - [x] Did you include test case for this fix, if necessary? `primitive_comparison.dynamic_quantize_innermost_size` — two dynamic_quantize primitives differing only in K must compare unequal and hash differently. - [x] Did you review existing test that can be extended to cover this scenario? Which test did you review? `primitive_comparison_test.cpp` — the fix only changes the primitive's hash/comparison, so this is where it belongs. It had no dynamic_quantize case; extended with one. ### Tickets: - *191435* (cherry picked from commit ede9266)
There was a problem hiding this comment.
Pull request overview
Fixes an Intel GPU kernel-cache collision for dynamic_quantize in per-token dynamic quantization mode by making the resolved innermost (IFM/K) size part of the primitive identity, preventing different IFM sizes from incorrectly sharing a single JIT-compiled kernel.
Changes:
- Adds
dynamic_quantize::innermost_sizeand includes it inhash()/operator==/ serialization to avoid kernel cache key collisions. - Resolves
innermost_sizeduring primitive creation (from static input shape or from consumingFullyConnectedCompressedweights) and propagates it into kernel selector params. - Extends unit coverage to ensure
dynamic_quantizeprimitives with differentinnermost_sizecompare/hash differently.
Reviewed changes
Copilot reviewed 6 out of 6 changed files in this pull request and generated 1 comment.
Show a summary per file
| File | Description |
|---|---|
| src/plugins/intel_gpu/tests/unit/module_tests/primitive_comparison_test.cpp | Adds a regression test asserting dynamic_quantize hash/compare differs when only innermost_size differs. |
| src/plugins/intel_gpu/tests/unit/fusions/fully_connected_fusion_test.cpp | Passes explicit innermost size in a fusion test where the input layout remains fully dynamic. |
| src/plugins/intel_gpu/src/plugin/ops/dynamic_quantize.cpp | Resolves innermost size (including from FullyConnectedCompressed weights) and passes it into the primitive. |
| src/plugins/intel_gpu/src/kernel_selector/kernels/dynamic_quantize/dynamic_quantize_kernel_opt.cpp | Makes opt-kernel applicability depend on having a resolvable innermost size; uses fc_ifm_size only when needed. |
| src/plugins/intel_gpu/src/graph/impls/ocl/dynamic_quantize.cpp | Stops deriving IFM size from graph users at impl time; consumes the primitive’s stored innermost_size. |
| src/plugins/intel_gpu/include/intel_gpu/primitives/dynamic_quantize.hpp | Introduces innermost_size field and integrates it into hashing, equality, and binary save/load. |
Suppressed comments (1)
src/plugins/intel_gpu/src/plugin/ops/dynamic_quantize.cpp:56
- [HIGH] The fix relies on
get_innermost_size(op)resolvinginnermost_sizefrom the consumingFullyConnectedCompressedweights when the DQ input innermost dim is dynamic. Current coverage adds a hash/operator==regression test, but it doesn't exercise this new resolution path, so a regression here would silently reintroduce kernel-cache collisions.
Please consider adding a focused unit test that builds a small model/topology with dynamic DQ input shape feeding two FullyConnectedCompressed ops with different static K, and verifies that the created cldnn::dynamic_quantize primitives end up with different innermost_size (and thus different cache keys).
auto prim = cldnn::dynamic_quantize(primitive_name,
inputs[0],
op->get_attrs(),
op->get_input_partial_shape(0).size(),
get_innermost_size(op));
| static size_t get_innermost_size(const std::shared_ptr<ov::op::internal::DynamicQuantize>& op) { | ||
| const auto& in_shape = op->get_input_partial_shape(0); | ||
| const auto& innermost_dim = in_shape[in_shape.size() - 1]; | ||
| if (innermost_dim.is_static()) | ||
| return innermost_dim.get_length(); | ||
|
|
||
| size_t innermost_size = 0; | ||
| for (const auto& target_input : op->get_output_target_inputs(0)) { | ||
| const auto* fc = ov::as_type<const op::FullyConnectedCompressed>(target_input.get_node()); | ||
| if (fc == nullptr || target_input.get_index() != 0) | ||
| continue; | ||
|
|
||
| // Weights are [N, K] when transposed, [K, N] otherwise | ||
| const auto& weights_shape = fc->get_input_partial_shape(1); | ||
| const auto& ifm = weights_shape[weights_shape.size() - (fc->get_transpose_b() ? 1 : 2)]; | ||
| if (!ifm.is_static()) | ||
| continue; |
Description of the issue(symptom, root-cause, how it was resolved)
gemma-4-31B-it-int4-ovandgemma-4-26b-a4b-it-int4-ov.dynamic_quantizenodes with a dynamic innermost dim (?x?x?) get identicalkernel_impl_params, soadd_kernels_source()drops the second one and both share a single kernel. In per-token mode the group count is baked into the JIT fromfc_ifm_size, so the shared kernel is correct for only one of them. These models hit it because their sliding/full attention layers feed o_proj FCs with different IFM.dynamic_quantize::innermost_size) and joinshash()/operator==, so the two nodes no longer collide.CreateDynamicQuantizeOpresolves it once from the static dim, or from the weights IFM of the consumingFullyConnectedCompressed.The code and line that caused this issue (if it is not changed directly)
intel_gpu/src/graph/impls/ocl/kernels_cache.cpp-add_kernels_source()skips duplicate paramsdynamic_quantize_kernel_opt.cpp-GetJitConstants()bakesTOTAL_BLOCK_NUMfromfc_ifm_sizeintel_gpu/src/graph/impls/ocl/dynamic_quantize.cpp- the oldget_users().front()lookup gave a per-node value that never reached the cache key ([GPU] Use ifm size of fc for dynamic quantize #26850)Reproduction step and snapshot (if applicable. Do not attach for customer model)
$ python vlm_test.py gemma-4-31B-it-int4-ov cat.jpg GPUThe picture shows a grey tabby cat ..., got unrelated tokens.gemma-4-26b-a4b-it-int4-ovreproduces the same withDYNAMIC_QUANTIZATION_GROUP_SIZE=18446744073709551615.Problematic graph
Checklist
primitive_comparison.dynamic_quantize_innermost_size— two dynamic_quantize primitives differing only in K must compare unequal and hash differently.primitive_comparison_test.cpp— the fix only changes the primitive's hash/comparison, so this is where it belongs. It had no dynamic_quantize case; extended with one.Tickets:
(cherry picked from commit ede9266)