Skip to content

[GPU] Fix dynamic_quantize kernel sharing between different IFM sizes - #37222

Open
jade-cho wants to merge 1 commit into
openvinotoolkit:releases/2026/3from
jade-cho:cherry-pick-37140-rel-2026-3
Open

[GPU] Fix dynamic_quantize kernel sharing between different IFM sizes#37222
jade-cho wants to merge 1 commit into
openvinotoolkit:releases/2026/3from
jade-cho:cherry-pick-37140-rel-2026-3

Conversation

@jade-cho

@jade-cho jade-cho commented Aug 4, 2026

Copy link
Copy Markdown
Contributor

Description of the issue(symptom, root-cause, how it was resolved)

  • Symptom: int4 VLMs produce garbage output on GPU in per-token dyn-quant mode. Reproduced on gemma-4-31B-it-int4-ov and gemma-4-26b-a4b-it-int4-ov.
  • Root-cause: Two dynamic_quantize nodes with a dynamic innermost dim (?x?x?) get identical kernel_impl_params, so add_kernels_source() drops the second one and both share a single kernel. In per-token mode the group count is baked into the JIT from fc_ifm_size, so the shared kernel is correct for only one of them. These models hit it because their sliding/full attention layers feed o_proj FCs with different IFM.
  • How it was resolved: The resolved innermost length is now carried by the primitive (dynamic_quantize::innermost_size) and joins hash() / operator==, so the two nodes no longer collide. CreateDynamicQuantizeOp resolves it once from the static dim, or from the weights IFM of the consuming FullyConnectedCompressed.

The code and line that caused this issue (if it is not changed directly)

  • intel_gpu/src/graph/impls/ocl/kernels_cache.cpp - add_kernels_source() skips duplicate params
  • dynamic_quantize_kernel_opt.cpp - GetJitConstants() bakes TOTAL_BLOCK_NUM from fc_ifm_size
  • intel_gpu/src/graph/impls/ocl/dynamic_quantize.cpp - the old get_users().front() lookup gave a per-node value that never reached the cache key ([GPU] Use ifm size of fc for dynamic quantize #26850)

Reproduction step and snapshot (if applicable. Do not attach for customer model)

  • $ python vlm_test.py gemma-4-31B-it-int4-ov cat.jpg GPU
  • Expected The picture shows a grey tabby cat ..., got unrelated tokens.
  • gemma-4-26b-a4b-it-int4-ov reproduces the same with DYNAMIC_QUANTIZATION_GROUP_SIZE=18446744073709551615.

Problematic graph

  sliding layer:  SDPA -> Reshape(?x?x?) -> DynamicQuantize -> FC(W: [N, 8192])
  full    layer:  SDPA -> Reshape(?x?x?) -> DynamicQuantize -> FC(W: [N, 16384])
  • Same layouts on both dq nodes; only the FC weights differ, which was not in the cache key.

Checklist

  • Is it a proper fix? (not a workaround) K (the innermost dimension) determines the quantization group count baked into the kernel at compile time, so it is now part of the primitive's hash()/operator==.
  • Did you include test case for this fix, if necessary? primitive_comparison.dynamic_quantize_innermost_size — two dynamic_quantize primitives differing only in K must compare unequal and hash differently.
  • Did you review existing test that can be extended to cover this scenario? Which test did you review?
    primitive_comparison_test.cpp — the fix only changes the primitive's hash/comparison, so this is where it belongs. It had no dynamic_quantize case; extended with one.

Tickets:

  • 191435

(cherry picked from commit ede9266)

…openvinotoolkit#37140)

### Description of the issue(symptom, root-cause, how it was resolved)

- **Symptom**: int4 VLMs produce garbage output on GPU in per-token
dyn-quant mode. Reproduced on `gemma-4-31B-it-int4-ov` and
`gemma-4-26b-a4b-it-int4-ov`.
- **Root-cause**: Two `dynamic_quantize` nodes with a dynamic innermost
dim (`?x?x?`) get identical `kernel_impl_params`, so
`add_kernels_source()` drops the second one and both share a single
kernel. In per-token mode the group count is baked into the JIT from
`fc_ifm_size`, so the shared kernel is correct for only one of them.
These models hit it because their sliding/full attention layers feed
o_proj FCs with different IFM.
- **How it was resolved**: The resolved innermost length is now carried
by the primitive (`dynamic_quantize::innermost_size`) and joins `hash()`
/ `operator==`, so the two nodes no longer collide.
`CreateDynamicQuantizeOp` resolves it once from the static dim, or from
the weights IFM of the consuming `FullyConnectedCompressed`.

#### The code and line that caused this issue (if it is not changed
directly)

- `intel_gpu/src/graph/impls/ocl/kernels_cache.cpp` -
`add_kernels_source()` skips duplicate params
- `dynamic_quantize_kernel_opt.cpp` - `GetJitConstants()` bakes
`TOTAL_BLOCK_NUM` from `fc_ifm_size`
- `intel_gpu/src/graph/impls/ocl/dynamic_quantize.cpp` - the old
`get_users().front()` lookup gave a per-node value that never reached
the cache key (openvinotoolkit#26850)

#### Reproduction step and snapshot (if applicable. Do not attach for
customer model)

 - `$ python vlm_test.py gemma-4-31B-it-int4-ov cat.jpg GPU`
- Expected `The picture shows a grey tabby cat ...`, got unrelated
tokens.
 - `gemma-4-26b-a4b-it-int4-ov` reproduces the same with
   `DYNAMIC_QUANTIZATION_GROUP_SIZE=18446744073709551615`.

#### Problematic graph

```
  sliding layer:  SDPA -> Reshape(?x?x?) -> DynamicQuantize -> FC(W: [N, 8192])
  full    layer:  SDPA -> Reshape(?x?x?) -> DynamicQuantize -> FC(W: [N, 16384])
```

- Same layouts on both dq nodes; only the FC weights differ, which was
not in the cache key.

#### Checklist

 - [x] Is it a proper fix? (not a workaround)
K (the innermost dimension) determines the quantization group count
baked into the kernel at compile time, so it is now part of the
primitive's hash()/operator==.
 - [x] Did you include test case for this fix, if necessary?
`primitive_comparison.dynamic_quantize_innermost_size` — two
dynamic_quantize primitives differing only in K must compare unequal and
hash differently.
- [x] Did you review existing test that can be extended to cover this
scenario? Which test did you review?
`primitive_comparison_test.cpp` — the fix only changes the primitive's
hash/comparison, so this is where it belongs. It had no dynamic_quantize
case; extended with one.

### Tickets:
 - *191435*

(cherry picked from commit ede9266)
@jade-cho
jade-cho requested review from a team as code owners August 4, 2026 08:42
@github-actions github-actions Bot added the category: GPU OpenVINO GPU plugin label Aug 4, 2026
@maxnick
maxnick requested a lite review from Copilot August 4, 2026 16:06
@maxnick maxnick self-assigned this Aug 4, 2026
@maxnick maxnick added this to the 2026.4 milestone Aug 4, 2026

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Fixes an Intel GPU kernel-cache collision for dynamic_quantize in per-token dynamic quantization mode by making the resolved innermost (IFM/K) size part of the primitive identity, preventing different IFM sizes from incorrectly sharing a single JIT-compiled kernel.

Changes:

  • Adds dynamic_quantize::innermost_size and includes it in hash() / operator== / serialization to avoid kernel cache key collisions.
  • Resolves innermost_size during primitive creation (from static input shape or from consuming FullyConnectedCompressed weights) and propagates it into kernel selector params.
  • Extends unit coverage to ensure dynamic_quantize primitives with different innermost_size compare/hash differently.

Reviewed changes

Copilot reviewed 6 out of 6 changed files in this pull request and generated 1 comment.

Show a summary per file
File Description
src/plugins/intel_gpu/tests/unit/module_tests/primitive_comparison_test.cpp Adds a regression test asserting dynamic_quantize hash/compare differs when only innermost_size differs.
src/plugins/intel_gpu/tests/unit/fusions/fully_connected_fusion_test.cpp Passes explicit innermost size in a fusion test where the input layout remains fully dynamic.
src/plugins/intel_gpu/src/plugin/ops/dynamic_quantize.cpp Resolves innermost size (including from FullyConnectedCompressed weights) and passes it into the primitive.
src/plugins/intel_gpu/src/kernel_selector/kernels/dynamic_quantize/dynamic_quantize_kernel_opt.cpp Makes opt-kernel applicability depend on having a resolvable innermost size; uses fc_ifm_size only when needed.
src/plugins/intel_gpu/src/graph/impls/ocl/dynamic_quantize.cpp Stops deriving IFM size from graph users at impl time; consumes the primitive’s stored innermost_size.
src/plugins/intel_gpu/include/intel_gpu/primitives/dynamic_quantize.hpp Introduces innermost_size field and integrates it into hashing, equality, and binary save/load.
Suppressed comments (1)

src/plugins/intel_gpu/src/plugin/ops/dynamic_quantize.cpp:56

  • [HIGH] The fix relies on get_innermost_size(op) resolving innermost_size from the consuming FullyConnectedCompressed weights when the DQ input innermost dim is dynamic. Current coverage adds a hash/operator== regression test, but it doesn't exercise this new resolution path, so a regression here would silently reintroduce kernel-cache collisions.

Please consider adding a focused unit test that builds a small model/topology with dynamic DQ input shape feeding two FullyConnectedCompressed ops with different static K, and verifies that the created cldnn::dynamic_quantize primitives end up with different innermost_size (and thus different cache keys).

    auto prim = cldnn::dynamic_quantize(primitive_name,
                                        inputs[0],
                                        op->get_attrs(),
                                        op->get_input_partial_shape(0).size(),
                                        get_innermost_size(op));

Comment on lines +18 to +34
static size_t get_innermost_size(const std::shared_ptr<ov::op::internal::DynamicQuantize>& op) {
const auto& in_shape = op->get_input_partial_shape(0);
const auto& innermost_dim = in_shape[in_shape.size() - 1];
if (innermost_dim.is_static())
return innermost_dim.get_length();

size_t innermost_size = 0;
for (const auto& target_input : op->get_output_target_inputs(0)) {
const auto* fc = ov::as_type<const op::FullyConnectedCompressed>(target_input.get_node());
if (fc == nullptr || target_input.get_index() != 0)
continue;

// Weights are [N, K] when transposed, [K, N] otherwise
const auto& weights_shape = fc->get_input_partial_shape(1);
const auto& ifm = weights_shape[weights_shape.size() - (fc->get_transpose_b() ? 1 : 2)];
if (!ifm.is_static())
continue;
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

category: GPU OpenVINO GPU plugin

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants