Batch the non-grouped Conv GEMM with gemmStridedBatched

[harz05]

Implements #29

Changes in ROperator_Conv.hxx (non-grouped path only; grouped unchanged):

• Initialize: _xcol sized to hold all B samples' im2col instead of one slice.
• Generate_GPU_ALPAKA: each sample's im2col writes its own slice; the per-sample matmul loop is replaced by one gemmStridedBatched over the batch; the inter-sample alpaka::wait calls are removed now
• GetBlasConfig: returns empty for the non-grouped path (legacy cuBLAS, no cuBLASLt layout).

Test (output is bit-identical to the loop; existing Conv tests + ConvBatch4 pass):

• ConvBatchModelGenerator.py (model + numpy reference)
• input_models/ConvBatch4.onnx, references/ConvBatch4.ref.hxx, references/ConvBatch4_input.ref.hxx
• ConvBatch4 TEST_F in TestCustomModelsFromONNXForAlpakaCuda.cxx

Benchmarked two ways on T4 Colab (baseline = the per-sample loop):

1. Fixed model, varying batch (8-layer conv stack, C=16, 16x16): for batch 1 the improvement was ~1x (neutral) and for the following batches it was as follows- batch 4 2.6x, batch 8 3.4x, batch 16 4.1x

2. Single conv layer, varying GEMM size at batch 8: C16 8x8 2.6x, C64 32x32 1.9x, C128 28x28 1.17x

Memory tradeoff (from the code, not separately measured on Colab): _xcol grows from one slice to B, so the extra is (B-1) * colElements * 4 bytes per conv layer, where colElements = gemm_k * gemm_m. For the configs here that is ~19 MB total for the 8-layer stack at batch 16, and ~50 MB for the single C64 56x56 layer at batch 8. Modest on a 16 GB T4, but it scales with batch x spatial x channels, so it can grow at large scale batch configs.

Thus to conclude, the benchmarks point to the gain coming from dropping the per-sample syncs and letting cuBLAS batch the small GEMMs: it grows with batch and shrinks as the GEMM gets large enough to saturate the GPU on its own. Gain is neutral at batch=1, and no regression in any case tested so far.

Colab test notebook- Link

EDIT: benchmarking results are in comments

[harz05]

Following are the benchmarking results @sanjibansg

Benchmark: strided-batched GEMM vs per-sample loop (Tesla T4)

Following tables are the results using the SOFIE profiler, 200 iterations; A/B testing by reverting only ROperator_Conv.hxx to the base. Here Speedup is the Overall_Time ratio (batched vs current per-sample loop) and Extra memory = the Intermediate device bufs delta (where the grown _xcol buffer lands).

8-layer conv stack (C=16, 16x16), batch sweep

batch	baseline ms	batched ms	speedup	extra GPU mem
1	0.350	0.332	1.05x (neutral)	+0.0 MB
4	1.026	0.471	2.18x	+3.4 MB
8	1.915	0.689	2.78x	+7.9 MB
16	3.728	1.127	3.31x	+16.9 MB

Single conv layer, size/batch sweep

model	baseline ms	batched ms	speedup	extra GPU mem
C32 S16 B1	0.065	0.060	1.08x	+0.0 MB
C32 S16 B8	0.449	0.186	2.42x	+2.0 MB
C32 S16 B32	1.515	0.467	3.24x	+8.7 MB
C64 S32 B8	0.751	0.590	1.27x	+15.8 MB
C128 S28 B8	0.967	0.832	1.16x	+24.1 MB
C64 S56 B8	1.204	1.054	1.14x	+48.2 MB

Tradeoff

• Extra memory grows linearly as (B-1) * Cin * kH * kW * Hout * Wout * 4B, one im2col buffer per extra sample. The profiler-measured values match this formula (predicted ~17 / ~50 MB for stack_b16 / C64_S56, measured 16.9 / 48.2).
• Best value is the small-GEMM batched regime: 2 to 3.3x for a few MB.
• The largest memory cost lands exactly where the speedup is smallest (big single-layer GEMMs that already saturate the GPU)
• Batch 1 is neutral in both time and memory (the batched path degenerates to the single-sample case).

Note: profiler mode adds an alpaka::wait after every op, so absolute times are inflated, but the A/B ratios hold and match the throughput run.

[sanjibansg]

The speedup could be important, but considering the memory growth, it could also be a disadvantage. Can we make this configurable? Say we have batched as the default, and then the user could turn it off while instantiating the session object?