Coverage is every {@code @AllocFree} kernel except {@code variance(mode)}, which appears here as the flag canary + * instead — see its comment. The two suites cannot be factored into a shared collection: the assertion helpers have to + * stay {@code inline} so each call site gets a monomorphic measurement loop, and driving the kernels from a + * {@code List[() => Unit]} would reintroduce the megamorphic {@code Function0.apply()} dispatch that stops C2 inlining + * through to the Vector API calls. So the two lists are kept in step by hand, and drifting apart is a real hazard: + * this suite sat at fourteen kernels while D1Suite grew to twenty-nine. * * '''Original design intent vs. reality:''' * @@ -31,6 +38,19 @@ import vecxt.all.{*, given} * D1EAOffSuite asserts the same ≤ 8 bytes threshold under EA-off: if a kernel has regressed to the software fallback * path, D1EAOffSuite fails here while D1Suite would continue to pass. * + *
How much that is worth depends on how reliably EA rescues a software-path kernel, and the honest answer is "not
+ * always". D6's canary — species from a method parameter — is a software-path kernel that allocates with EA
+ * enabled, so D1Suite catches it unaided. The gap this suite closes is therefore narrower than "detects lost
+ * intrinsification": it is the fallbacks whose objects happen not to escape, which EA removes and D1Suite then reads
+ * as a pass. Worth having, and cheap, but not a substitute for confirming intrinsification directly.
+ *
+ * '''Why the flag canary is not optional:'''
+ *
+ * Every kernel assertion here reads "still zero with EA off", which is exactly what a run with EA still on
+ * would produce. Without a test that fails when the flag is absent, this entire scope passes whether or not
+ * {@code -XX:-DoEscapeAnalysis} reached the JVM — the same "silently became a no-op" hazard D6 exists to prevent for
+ * D1Suite. The canary is the first test in the file.
+ *
* '''DCE guard:'''
*
* Pure reductions store their results into {@code @volatile} fields, preventing C2 from dead-code-eliminating the
@@ -80,6 +100,48 @@ class D1EAOffSuite extends FunSuite:
)
end assertAllocFreeWithEAOff
+ /** The inverted assertion, used by exactly one test: the flag canary below.
+ *
+ * Every other assertion in this suite reads "still zero with EA off", which is indistinguishable from "the flag was
+ * silently dropped and EA is on". This is the assertion that tells those two apart.
+ */
+ private inline def assertAllocatesWithEAOff(label: String)(inline body: => Unit): Unit =
+ val total = AllocMeter.measureAlloc(Warmup, Reps)(body)
+ if total < 0L && sys.env.contains("CI") then
+ fail(s"[D1-EAOff] $label: CI detected but ThreadMXBean allocation tracking is unavailable.")
+ end if
+ assume(total >= 0L, s"[D1-EAOff] skip $label — ThreadMXBean allocation tracking not available on this JVM")
+ val perOp = total.toDouble / Reps
+ assert(
+ perOp > Eps,
+ s"D1-EAOff github.com/Quafadas/vecxt/issues/105: $label allocated ${perOp.toLong} bytes/op with " +
+ s"-XX:-DoEscapeAnalysis, and was expected to allocate. This is the flag canary, not a kernel check: " +
+ s"the allocation it looks for is one escape analysis would have removed, so measuring zero here means " +
+ s"EA is still on and -XX:-DoEscapeAnalysis did not take effect. Every other assertion in this suite " +
+ s"is then passing for the wrong reason. Check this scope's forkArgs before believing any of them."
+ )
+ end assertAllocatesWithEAOff
+
+ // ── The flag canary ─────────────────────────────────────────────────────────
+
+ /** Proves `-XX:-DoEscapeAnalysis` reached the JVM. Without this the whole scope is unfalsifiable — the absence of
+ * such a check is the most likely reason the CI step for it sat commented out.
+ *
+ * `variance(mode)` is the one kernel in D1Suite whose zero comes from escape analysis rather than from
+ * intrinsification. Its body reads one field out of the [[vecxt.MeanAndVariance]] that `meanAndVarianceTwoPass`
+ * returns and discards the other, so the object is dead and EA removes it — D1Suite measures 0 bytes/op. That object
+ * is an ordinary `final class`, not a `Vector`, so nothing intrinsifies it away. With EA off it must reach the heap.
+ *
+ * Which is also why it is absent from the kernel assertions below rather than merely inverted here: asserting "≤ 8
+ * bytes/op with EA off" for an EA-dependent kernel would be asserting the opposite of what the flag does.
+ */
+ test("D1-EAOff canary: doublearrays.variance(mode) must allocate with EA off") {
+ val arr = Array.tabulate(N)(i => (i % 100).toDouble)
+ assertAllocatesWithEAOff("doublearrays.variance(mode)") {
+ doubleSink = arr.variance(VarianceMode.Population)
+ }
+ }
+
// ── Double ──────────────────────────────────────────────────────────────────
test("D1-EAOff: doublearrays.sumSIMD") {
@@ -123,6 +185,20 @@ class D1EAOffSuite extends FunSuite:
assertAllocFreeWithEAOff("doublearrays.fillLinspace")(fillLinspace(dest, 0.0, 1.0))
}
+ // The two in-place unary kernels. NEG and ABS are intrinsified lanewise operations and the masked
+ // tail's VectorMask is `_VectorFromBitsCoerced`, so nothing here depends on EA — which is exactly
+ // what makes them worth asserting with EA off. The transcendentals (exp!, log!, …) are not
+ // annotated @AllocFree and so are not measured in either suite.
+ test("D1-EAOff: doublearrays.-!") {
+ val arr = Array.fill(N)(1.0)
+ assertAllocFreeWithEAOff("doublearrays.-!")(arr.`-!`)
+ }
+
+ test("D1-EAOff: doublearrays.abs!") {
+ val arr = Array.tabulate(N)(i => if i % 2 == 0 then i.toDouble else -i.toDouble)
+ assertAllocFreeWithEAOff("doublearrays.abs!")(arr.`abs!`)
+ }
+
// ── Float ───────────────────────────────────────────────────────────────────
test("D1-EAOff: floatarrays.sumSIMD") {
@@ -150,6 +226,38 @@ class D1EAOffSuite extends FunSuite:
assertAllocFreeWithEAOff("floatarrays.-=(Float)")(arr -= 0.1f)
}
+ test("D1-EAOff: floatarrays.-!") {
+ val arr = Array.fill(N)(1.0f)
+ assertAllocFreeWithEAOff("floatarrays.-!")(arr.`-!`)
+ }
+
+ test("D1-EAOff: floatarrays.abs!") {
+ val arr = Array.tabulate(N)(i => if i % 2 == 0 then i.toFloat else -i.toFloat)
+ assertAllocFreeWithEAOff("floatarrays.abs!")(arr.`abs!`)
+ }
+
+ test("D1-EAOff: floatarrays.clamp!") {
+ val arr = Array.tabulate(N)(i => (i % 10).toFloat)
+ assertAllocFreeWithEAOff("floatarrays.clamp!")(arr.`clamp!`(2.0f, 7.0f))
+ }
+
+ test("D1-EAOff: floatarrays.+=(Array[Float])") {
+ val arr = Array.fill(N)(1.0f)
+ val arr2 = Array.fill(N)(0.0f)
+ assertAllocFreeWithEAOff("floatarrays.+=(Array[Float])")(arr += arr2)
+ }
+
+ test("D1-EAOff: floatarrays.*=(Array[Float])") {
+ val arr = Array.fill(N)(2.0f)
+ val arr2 = Array.fill(N)(1.0f)
+ assertAllocFreeWithEAOff("floatarrays.*=(Array[Float])")(arr *= arr2)
+ }
+
+ test("D1-EAOff: floatarrays.*=(Float)") {
+ val arr = Array.fill(N)(2.0f)
+ assertAllocFreeWithEAOff("floatarrays.*=(Float)")(arr *= 1.0f)
+ }
+
// ── Int ─────────────────────────────────────────────────────────────────────
test("D1-EAOff: intarrays.sumSIMD") {
@@ -157,4 +265,37 @@ class D1EAOffSuite extends FunSuite:
assertAllocFreeWithEAOff("intarrays.sumSIMD") { intSink = arr.sumSIMD }
}
+ test("D1-EAOff: intarrays.dot") {
+ val arr = Array.tabulate(N)(i => i % 100)
+ val arr2 = Array.tabulate(N)(i => (i + 1) % 100)
+ assertAllocFreeWithEAOff("intarrays.dot") { intSink = arr.dot(arr2) }
+ }
+
+ test("D1-EAOff: intarrays.minSIMD") {
+ val arr = Array.tabulate(N)(i => i % 1000)
+ assertAllocFreeWithEAOff("intarrays.minSIMD") { intSink = arr.minSIMD }
+ }
+
+ test("D1-EAOff: intarrays.maxSIMD") {
+ val arr = Array.tabulate(N)(i => i % 1000)
+ assertAllocFreeWithEAOff("intarrays.maxSIMD") { intSink = arr.maxSIMD }
+ }
+
+ test("D1-EAOff: intarrays.+=(Array[Int])") {
+ val arr = Array.fill(N)(1)
+ val arr2 = Array.fill(N)(0)
+ assertAllocFreeWithEAOff("intarrays.+=(Array[Int])")(arr += arr2)
+ }
+
+ test("D1-EAOff: intarrays.-=(Array[Int])") {
+ val arr = Array.fill(N)(1)
+ val arr2 = Array.fill(N)(0)
+ assertAllocFreeWithEAOff("intarrays.-=(Array[Int])")(arr -= arr2)
+ }
+
+ test("D1-EAOff: intarrays.-=(Int)") {
+ val arr = Array.fill(N)(1)
+ assertAllocFreeWithEAOff("intarrays.-=(Int)")(arr -= 0)
+ }
+
end D1EAOffSuite
diff --git a/jitAudit/package.mill b/jitAudit/package.mill
index 67e18ed5..d881a073 100644
--- a/jitAudit/package.mill
+++ b/jitAudit/package.mill
@@ -26,6 +26,43 @@ import mill.*, scalalib.*
* produce results that look like allocation from interpreted code. {@code --add-modules jdk.incubator.vector} exposes
* the Vector API.
*
+ * ==What was deliberately not built: D2 and D5==
+ *
+ * The plan specified two further checks in this tier, both of which work by running with
+ * {@code -XX:+LogCompilation} and parsing the XML: D2 confirms Vector API intrinsics were applied by looking for
+ * {@code Coverage is deliberately every method carrying {@code @AllocFree}, and the two are kept in step by hand: an
+ * annotation with no test here is an assertion nobody has checked, which is how {@code intarrays.dot} carried
+ * {@code @AllocFree} through two releases while allocating a dead array per call, and how {@code **!} kept it until
+ * #110 measured it. The list is written out rather than driven from a collection on purpose — see the note on
+ * {@code assertAllocFree} below.
*
* {@code assertAllocFree} is declared {@code inline} so that each call site gets its own specialised measurement loop
* inside {@code AllocMeter.measureAlloc}. Without inlining the body would be dispatched through a shared megamorphic
@@ -179,6 +185,30 @@ class D1Suite extends FunSuite:
assertAllocFree("floatarrays.abs!")(arr.`abs!`)
}
+ test("D1: floatarrays.clamp!") {
+ val arr = Array.tabulate(N)(i => (i % 10).toFloat)
+ assertAllocFree("floatarrays.clamp!")(arr.`clamp!`(2.0f, 7.0f))
+ }
+
+ test("D1: floatarrays.+=(Array[Float])") {
+ val arr = Array.fill(N)(1.0f)
+ val arr2 = Array.fill(N)(0.0f) // adding zero keeps the values stable across the measurement windows
+ assertAllocFree("floatarrays.+=(Array[Float])")(arr += arr2)
+ }
+
+ test("D1: floatarrays.*=(Array[Float])") {
+ val arr = Array.fill(N)(2.0f)
+ // See doublearrays.*= — a multiplier of 1.0 avoids driving the array to denormals and then to
+ // zero partway through, which would make most of the measured workload operate on zeros.
+ val arr2 = Array.fill(N)(1.0f)
+ assertAllocFree("floatarrays.*=(Array[Float])")(arr *= arr2)
+ }
+
+ test("D1: floatarrays.*=(Float)") {
+ val arr = Array.fill(N)(2.0f)
+ assertAllocFree("floatarrays.*=(Float)")(arr *= 1.0f)
+ }
+
// ── Int ─────────────────────────────────────────────────────────────────────
test("D1: intarrays.sumSIMD") {
@@ -195,4 +225,31 @@ class D1Suite extends FunSuite:
assertAllocFree("intarrays.dot") { intSink = arr.dot(arr2) }
}
+ test("D1: intarrays.minSIMD") {
+ val arr = Array.tabulate(N)(i => i % 1000)
+ assertAllocFree("intarrays.minSIMD") { intSink = arr.minSIMD }
+ }
+
+ test("D1: intarrays.maxSIMD") {
+ val arr = Array.tabulate(N)(i => i % 1000)
+ assertAllocFree("intarrays.maxSIMD") { intSink = arr.maxSIMD }
+ }
+
+ test("D1: intarrays.+=(Array[Int])") {
+ val arr = Array.fill(N)(1)
+ val arr2 = Array.fill(N)(0) // adding zero keeps the values from overflowing across the windows
+ assertAllocFree("intarrays.+=(Array[Int])")(arr += arr2)
+ }
+
+ test("D1: intarrays.-=(Array[Int])") {
+ val arr = Array.fill(N)(1)
+ val arr2 = Array.fill(N)(0)
+ assertAllocFree("intarrays.-=(Array[Int])")(arr -= arr2)
+ }
+
+ test("D1: intarrays.-=(Int)") {
+ val arr = Array.fill(N)(1)
+ assertAllocFree("intarrays.-=(Int)")(arr -= 0)
+ }
+
end D1Suite
diff --git a/site/docs/blog/2026-07-28-Inlining.md b/site/docs/blog/2026-07-28-Inlining.md
index e8182935..7810fd0e 100644
--- a/site/docs/blog/2026-07-28-Inlining.md
+++ b/site/docs/blog/2026-07-28-Inlining.md
@@ -65,6 +65,33 @@ Two things this also corrects:
`MaxInlineLevel` (15 frames) is the related limit: Panama's Vector API works by inlining a deep chain of `@ForceInline` intrinsics, and when that chain runs out of depth `DoubleVector` stops being register-resident and becomes a heap allocation per operation. The practical effect of bloat is usually inward rather than outward — a method already carrying expanded loop bodies has a large IR graph and no room left to absorb the inlining Panama depends on.
+### The limit that is not measured in bytecodes
+
+Every threshold above is a bytecode count, which is why static analysis can enforce them. There is a fifth one that is not, and for SIMD kernels it is the *tightest* of them.
+
+`InlineSmallCode` (2500) applies to a callee that has **already been compiled**: if its nmethod's machine code exceeds the limit, C2 will not inline it into a new caller, however hot that caller is. Nothing about bytecode size predicts this, because the ratio between the two is not a constant.
+
+Measured on a `LogCompilation` run of the `jitAudit` kernels — Microsoft OpenJDK 25.0.4, x86-64, 8-lane double species — reading `stub_offset - insts_offset` off each `c2` ` What the audit asserts about a {@code @HotPath} method (check C2 of
* #105): its emitted bytecode fits inside
- * HotSpot's {@code FreqInlineSize}, so C2 will inline it into its callers once it is hot. A kernel
- * larger than that budget is not inlined however hot it gets, which costs the surrounding loop the
- * optimisations that only happen across an inlined boundary.
+ * HotSpot's {@code FreqInlineSize}. A kernel larger than that budget is not inlined however hot it
+ * gets, which costs the surrounding loop the optimisations that only happen across an inlined
+ * boundary.
+ *
+ * That is a necessary condition, not a sufficient one. An earlier version of this note
+ * claimed the check meant "C2 will inline it into its callers once it is hot". It does not.
+ * {@code InlineSmallCode} (2500) applies to an already-compiled callee and is measured in
+ * machine code, not bytecode — and for vectorised kernels the observed expansion is 7–10×,
+ * so 2500 bytes of machine code is reached at roughly 260–300 bytecodes, below
+ * {@code FreqInlineSize}'s 325. A kernel can satisfy this annotation and still be one C2 declines
+ * to inline.
+ *
+ * Bytecode analysis cannot see that, so C2 asserts what it can, and nothing measures the
+ * compiled size. Check D2 would have — by reading {@code stub_offset - insts_offset} off
+ * {@code LogCompilation}'s {@code c2} {@code What does cover it, indirectly: a kernel that stops being inlined also stops having its
+ * {@code Vector} temporaries scalarised, which {@code jitAudit}'s D1 and its EA-off cross-check
+ * measure as allocation. That catches the consequence rather than the cause, and only for kernels
+ * carrying {@link AllocFree}. See {@code site/docs/blog/2026-07-28-Inlining.md}, "The limit that is
+ * not measured in bytecodes", for the measurements behind this note.
*
* Only meaningful on a method that is actually emitted. An {@code inline def} body is
* expanded into its callers instead of being compiled on its own, so it has no bytecode to measure
diff --git a/vecxt/src-jvm/java/vecxt/annotations/Thin.java b/vecxt/src-jvm/java/vecxt/annotations/Thin.java
index 74ab2e6f..589ed6fb 100644
--- a/vecxt/src-jvm/java/vecxt/annotations/Thin.java
+++ b/vecxt/src-jvm/java/vecxt/annotations/Thin.java
@@ -15,6 +15,24 @@
* called. That is the property that makes the public API zero-cost at a cold or lukewarm call
* site, where {@code FreqInlineSize} does not apply yet.
*
+ * The budget is in bytecodes and says nothing about the compiled form. Measured on a
+ * {@code LogCompilation} run: {@code vecxt.all.clamp!} is an eleven-bytecode {@code export}
+ * forwarder whose {@code c2} nmethod is 1696 bytes of machine code, because C2 inlined the kernel
+ * into it — 68% of {@code InlineSmallCode}. It would satisfy this annotation's 35-byte budget by a
+ * factor of three while being one of the largest compiled methods in the library.
+ *
+ * Two reasons that matters more here than for {@link HotPath}. Forwarders are where the
+ * bytecode-to-machine-code ratio is most extreme, precisely because the body they forward to gets
+ * pulled in. And the {@code vecxt.all} export forwarders are excluded from the checked-in baseline
+ * by {@code Audit.primaryAnnotated} — deliberately, to keep the baseline readable — so their
+ * compiled size is unmeasured twice over.
+ *
+ * Unverified, and worth confirming in the HotSpot source before relying on either answer: whether
+ * the {@code MaxTrivialSize}/{@code MaxInlineSize} fast paths let a small callee bypass the
+ * {@code InlineSmallCode} veto. If they do, the forwarder above is a curiosity. If they do not, a
+ * small forwarder with a large nmethod stops being inlinable, and this annotation is asserting the
+ * wrong quantity for exactly the methods it was written for.
+ *
* It also asserts the method contains no backward branch. A loop in a forwarder means the method
* does per-element work, so the annotation is simply the wrong one — {@link HotPath} is.
*