GPU: Optimize conv/permute/Softplus kernels for f16 + dynamic batch, fixing numerical instability and layout reorders - #37216
Draft
deepaks2 wants to merge 6 commits into
Draft
GPU: Optimize conv/permute/Softplus kernels for f16 + dynamic batch, fixing numerical instability and layout reorders#37216deepaks2 wants to merge 6 commits into
deepaks2 wants to merge 6 commits into
Conversation
Add IsSwappingFX() predicate to Validate() accepting cldnn permutation order [0,2,1,3] (4D/5D/6D), which corresponds to ONNX perm [0,3,2,1] (NCHW→NWHC). The tiled transpose kernel already performs the correct memory access pattern for this order; only the OUTPUT_TILED_ORDER macro arguments need to be exchanged (args 2 and 3 of OUTPUT_GET_INDEX) so that the output feature index maps to the input x-tile and the output x-tile maps to the input feature index. Update GetTiledOutputOrder() and GetFusedOpOrderVector() to emit the swapped argument string when IsSwappingFX() is true. Remove the previous is_rotating_to_end predicate (which targeted [0,2,3,1] cldnn order and was incorrect) and the temporary file-based debug log. This eliminates the permute_ref__f16 fallback kernel for permutations of the form [?,C,H,W]→[?,H,W,C], replacing it with the tiled kernel.
Two changes allow ActivationKernelOpt to handle SOFTPLUS with half-precision inputs on dynamic-batch models: 1. jitter.cpp: replace scalar-only SOFTPLUS F16 formula with type-generic one. Old formula used convert_float(input)/convert_half() which are only valid for scalar half, not half4. The new formula log(exp(input) + 1) uses OpenCL built-ins that are defined for half, half2, half4, float, float4, etc. This makes the macro valid in both the scalar ref kernel and the vectorised opt kernel. Precision behaviour for |x| > 11 (half exp overflow) is identical to the previous float-upcast path. 2. activation_kernel_opt.cpp + activation_opt.cl: add dynamic shape support. Add EnableDynamicShapesSupport() to GetSupportedKey() so the opt kernel is considered for models with dynamic batch dimensions. Add OPTIONAL_SHAPE_INFO_ARG to the CL kernel signature (required when is_shape_agnostic=true so that OV can pass the shape-info buffer without CL_INVALID_ARG_INDEX). Skip the static totalSize % NUM_COLS_WI divisibility check for dynamic inputs (GWS is corrected at runtime by update_dispatch_data_func via SetDefault).
… models
Add dynamic-shape support to ConvolutionKernel_bfyx_to_bfyx_f16, which
directly produces b_fs_yx_fsv16 output from bfyx input with ≤4 input
channels (e.g. RGB first conv). Without this fix the kernel was rejected
for any model with dynamic dimensions, causing OV to fall back to
convolution_gpu_bfyx_os_iyx_osv16 (bfyx output) followed by an expensive
bfyx→b_fs_yx_fsv16 reorder node.
Changes:
- convolution_kernel_bfyx_to_b_fs_yx_fsv16.cpp:
* EnableDynamicShapesSupport() added to GetSupportedKey()
* SetDefault() guards gws[2] against batch=0 (unknown at JIT time)
* GetKernelsPriority() simplified to FORCE_PRIORITY_2 for all batches
(this kernel is the sole candidate for bfyx(≤4ch)→b_fs_yx_fsv16)
- convolution_gpu_bfyx_to_bfyx_f16.cl:
* OPTIONAL_SHAPE_INFO_ARG added as first kernel argument (required
when EnableDynamicShapesSupport is set, otherwise CL_INVALID_ARG_INDEX)
- layout_optimizer.cpp (convolution_b_fs_yx_fsv16_opt):
* Relax correct_batch for small-channel (≤4) bfyx→fsv16 convolutions:
ConvolutionKernel_bfyx_to_bfyx_f16 supports any batch size for this
case, so the f16+batch>1 restriction was overly conservative.
Validated on FlashOCC image_encoder (ResNet-50, batch=6, f16, Xe3):
Kernel selector now correctly considers bfyx_to_bfyx_f16 for
dynamic-batch models where it was previously silently skipped.
…1+exp(-|x|)) The previous implementation log(exp(x)+1) overflows at x≈11.09 in float16 (half max ≈ 65504), producing +INF in exp() and propagating NaN/Inf through subsequent FullyConnected layers. Replace with the branchless numerically stable identity: softplus(x) = max(x,0) + log(1 + exp(-|x|)) This is mathematically equivalent but exp(-|x|) ∈ [0,1] for all x, so it never overflows in float16 or float32. The formula uses type-dispatched max_func and abs_func JIT helpers, compiling correctly for both scalar (half, float) and vector (half4, float4) kernel variants without branching. Validated on FlashOCC bev_trunk with INFERENCE_PRECISION_HINT=f16: before: NaN/Inf in 22M+ output voxels; after: zero NaN/Inf, correct logits.
…t reorder For convolutions with ≤4 input channels (e.g. RGB/RGBD backbone first conv), select_preferred_formats was querying choose_impl with format::any, causing oneDNN to be selected as the implementation (it supports any output format). oneDNN outputs bfyx for these shallow convolutions, which then requires a costly bfyx→b_fs_yx_fsv16 layout reorder before layer1. Fix: when the input has ≤4 features, bfyx input format, and ≥16 output features, first try to find a clDNN (ocl) implementation that supports b_fs_yx_fsv16 output. If one is found (ConvolutionKernel_bfyx_to_bfyx_f16), use it as the preferred factory. Its query_formats then returns b_fs_yx_fsv16 as the preferred output format, which the layout optimizer respects — eliminating the reorder entirely.
resample_opt (FORCE_PRIORITY_3, SIMD16 vectorized) previously only handled BILINEAR_INTERP and NEAREST_NEIGHBOR modes. LINEAR_ONNX mode fell back to resample_onnx (FORCE_PRIORITY_4) because the opt kernel was not declared compatible with that type. Changes: - resample_kernel_opt.cpp: add EnableResampleType(LINEAR_ONNX) so the selector considers resample_opt for LINEAR_ONNX requests. - resample_kernel_opt.cpp: relax INT8/UINT8 dtype guard to also pass for LINEAR_ONNX (matches guard logic for BILINEAR_INTERP). - resample_opt.cl: implement SAMPLE_TYPE_LINEAR_ONNX code path using get_original_coordinate() for correct half_pixel / align_corners / pytorch_half_pixel coordinate transformation, replacing the previous #error stub. The bilinear interpolation follows the same dx1/dx2 weight convention as resample_onnx.cl for numerical equivalence. The 5D guard (NEAREST_NEIGHBOR only for dims==5) is preserved.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Details:
A series of Intel GPU plugin optimizations for half-precision inference, primarily targeting FlashOCC and similar models with shallow convolutions, transpose ops, and activation functions.
Changes:
(overflow at x≈11.09).
OPTIONAL_SHAPE_INFO_ARG , runtime GWS correction). Softplus cost drops from 435→358 Mcs (-18%).
3.1ms of transpose cost; bev_trunk latency drops.
b_fs_yx_fsv16 output without a layout reorder.
supports b_fs_yx_fsv16 output — avoiding a costly bfyx→b_fs_yx_fsv16 reorder before layer1.
Tickets:
AI Assistance: