My Questions: Better as a discussion?

[Quafadas]

Is there a path, to Cyfra acting on a java.lang.foreign.MemorySegment, i.e. integrate with the JVMs project Panama datastructures?

I'm not entirely sure, if the question is sensible (perhaps this library, is an alternative to that!), so please feel free to say that too.

[szymon-rd]

Most of the memory mapping and operations we do with Vulkan's API native libraries or native bytebuffers. But I have ideas on some prospects of a broader application of cyfra - to support other kinds of accelerators.

Could you tell me in some detail about a specific use case, if you have one in mind, of an integration with Panama data structures?

[Quafadas]

So, yes... my hobby project is this;

https://github.com/Quafadas/vecxt

Which looks at linear algebra. My original goal was consistent cross platform syntax, and I'm happy with where it got to on that front. I successfully used it to train a neural network, (slowly).

When I looked at where all the performance was going, I concluded that it was lost in no small part being forced to make extra memory allocations. I've concluded that JVM data structures e.g. Array[Double] and existing libraries lead inevitably to being forced to make large extra memory allocations (for reasons - there's a blog post - but I guess it's not super interesting for here).

So, because it's a hobby project, I can look left and right and am thinking about going back to the start again, and adding in a new goal, that being pure performance on a (single, JVM) platform, via integration with native libraries. What I have at the moment, is a very simple data structure,

opaque type DoubleVector = MemorySegment

And a very small clutch of extension methods.

But from here, I've already proven that via jextract I can plug into Blis, and I think that gets great CPU performance. For performance performance, I'm also looking left and right to see if there's some sane pathway to shunting this off onto a GPU. Hence, this question.

I would say again, this is 100% a hobby project, not an important one, and it's my first foray into GPU land. I would not want anyone to greatly inconvenience themselves on its behalf, but, it's there's some sane (i.e. not expert level) path to integration, then I'd have a crack at it.

[szymon-rd]

We are now doing a rework on the pipeline and memory management: https://github.com/ComputeNode/cyfra/pull/53/files
It's not ready, we are currently progresing on it.

In our new model, it seems that MemorySession corresponds to our GBufferRegion, and MemorySegment corresponds to our GBuffer. I think there could be a path for integration. However, our library uses a DSL that mirrors entire universe of arithmetic operations, sequences, etc. So it is not possible to run external custom code on it, it needs to be developed with our DSL and API.

[Quafadas]

Thanks for the response, if I may ask some more questions. GBuffer allocates on the GPU?

So perhaps there could be some extension method, that shunts data over from a MemorySegement to a GBuffer? As I understand it, that would have to be done anyway - stuff has to get onto the GPU somehow (!).

From what I saw, it's possible to allocate structs? So something like

  class Matrix (
      val raw: DoubleVector,
      val rows: Int,
      val cols: Int,
      val rowStride: Int,
      val colStride: Int,
      val offset: Int = 0
  )

Would be possible to allocate from (and crucially back into <- this might be the painful part?) a GBuffer? This would need to be done by the third party seeking integration, I believe.

If I've understood, from this point, anything further must be done in Cyfra's DSL - we'd simply have a GBuffer of our data instead of a MemorySegement, and something like Matrix multiplication would be expressed interns of (making this up) cyfra.gemm( ...)?

Presumably, it's possible to compose these operations on the GPU?

[Quafadas]

I've started working through "hello world" type problems in Cyfra and have some (simple, I hope!) questions.

1. Mutating the Input Array

This is a "hello world" written by a human (me) copying out the doc examples to try to internalise the concepts.
https://github.com/Quafadas/cyfra_play/blob/main/project.scala

Would it be considered "pathological" in any sense - it mutates the input array (deliberately) as opposed to allocating new memory for output.

2. Sum Example - Not a race condition?

In this example;
https://cyfra.computenode.io/docs/gpu-pipelines#building-a-simple-pipeline

Cyfra sums an array. If I did this is "parallel CPU land", I would expect the answer to be wrong, because it is writing to the "same variable" in parallel, which I would expect at some point to form a race condition. I assume that my mental model here is wrong - but why? Is there any particular layer of the stack which "protects" against such things? I'm trying to understand where the boundaries of aggregation operations might be.

3. SIMD != SIMT - Example to help arrange mental model

In SIMD land to sum an array, one would sum across the "lanes" - i.e. keep 4 sums running - and then tidy them up in a tail loop. At first glance, it seems this strategy might also work with "threads" instead of "lanes"... if one knew the number of threads and they had an easy identifier.

This is SIMT land (?) - can one implement a similar model here? Could I keep a running sum per thread, then tidy up on the CPU at the end? Would there be a performance benefit to such a strategy? Is there something obviously wrong with my mental model here (I looked at implementing it, but I don't think one can identify "threads" which looks problematic for such an idea)? Fundamental or a design choice?

4 . Matrix Multiplication - Not BLAS like?

In CPU land, the naive implementation of matrix multiplication is easy, but a good implementation of matrix multiplication is terribly difficult, which more or less dictates that one finds some BLAS implementation and shim to it by calling it's dgemm function.

Is there some equivalent strategy on the GPU? I naively assumed, that Vulkan itself would offer some sort of "BLAS-like" API, but my google-fu turned up some quite deep looking presentations. ChatGPT pointed me toward IREE. I have no knowledge of these things. Is this "dgemm" like problem already solved in cyfra?

5. Double

I originally attempted to use Float64 - it appears int he docs here.
https://cyfra.computenode.io/docs/gpu-functions#cyfra-dsl

But I can't find it in the code. I'm not sure if I missed it. When I asked chatGPT about Float32 vs Float64 on GPUs, and Float vs Double in Java, the answer was "not simple", and summarised as "GPU's use Float32".

Is it intentional, that Float64 isn't in the implementation or have I missed it somehow? It appears in OpCodes but not the Cyfra DSL, as far as I can tell.

---

Apologies, on a read back this got long. I hope the questions are sensible though.

[szymon-rd]

Apologies, on a read back this got long

No issue, you are one of our first users and your experience and concerns are quite valuable to us.

Would it be considered "pathological" in any sense - it mutates the input array (deliberately) as opposed to allocating new memory for output.

I don't think that's a problem. Memory transfered to GPU tends to be quite large, sharing the allocation may not be a bad idea. You may want to think about operating on ByteBuffers at some point and using some abstraction. I am looking into providing some soon, with an execution rewrite I am doing.

Cyfra sums an array

Oh no, it just adds a value to each element, doing a "sum". I don't really know why I put the condition on 256 there... I have to fix the docs a bit i think. To really sum, you may want to do a parallel reduction, Spamegg wrote a nice article on doing a filter in parallel-reduction approach: https://spamegg1.github.io/gsoc-2025/#fs2-filtering-through-cyfra

However, we will soon be merging group reduction operators that are the best for things like that. This MR introduces a lot of operators for more complex usages: #82. Not yet merged, unfortunately, and will take couple of weeks before it's merged a we release the Cyfra with those. But you may be interested to take a look, a full llama inference pipeline is implemented there, and it works quite fast (~90% of llama.cpp performance on GPU level, cpu sync still a bit suboptimal).

In SIMD land to sum an array, one would sum across the "lanes" - i.e. keep 4 sums running - and then tidy them up in a tail loop

Yes, to really sum a buffer, you run a number of threads that do a subgroup reduction, then reduce those partial sums into final result. Like in the MR: https://github.com/ComputeNode/cyfra/pull/82/changes#diff-17d8d0438ec1351b101fb686f4aa0a60e92e45d312117cc0934cb399e7e3b5f0R134

This uses a quite heavy for-comprehension, direct-style is in works in another MR.

Is this "dgemm" like problem already solved in cyfra?

Really good question. In my current llama implementation it is not that bad, but the implementation of matmul is naive: https://github.com/ComputeNode/cyfra/blob/3761c7019265a36f7c277103e226acd7aea64cc1/cyfra-llama/src/main/scala/io/computenode/cyfra/llama/programs/f16/F16MatmulVecHybridProgram.scala

It does not do that bad though, we do e.g. subgroup reduction, but there are some optimizations to be added. For example GL_KHR_cooperative_matrix is a low hanging fruit. But a really well optimized BLAS-like library is another question... We may be behind ~10-20% until we figure out how to do something similar with Vulkan.

I originally attempted to use Float64 - it appears int he docs here.

Oh yes, that is a mistake on my part. There should be only Float32, and now we will also be able to add Float16 with that MR.

[Quafadas]

Thanks for the responses. I'm sorry it's taken me a while to write back - I've had to have my headspace elsewhere.

My interest is more or less attempting to integrate some of this into my pet linear algebra library... Do you have any objection if I give proceed with giving that a go? It's open source, with a userbase of 1 (me) . I'm happy to offer as much (or as little :-)!) feedback on the experience / outcome as you wish.

Note: If matmul is only off by 10-20% already, that's an absolute triumph!

[szymon-rd]

Do you have any objection if I give proceed with giving that a go?

You may want to wait 1-2 months before we merge all that was required to build cyfra-llama. Then you will have a lot of linear algebra stuff out of the box. Otherwise, of course we don't have objections, and feel free to ask questions along the way.

[Quafadas]

Hehe cool - I shall wait excitedly and it seems to just about fit my current time available :-)…

[Quafadas]

Just in case it is of interest as I had some time. I sat down with Claude and did a POC on this. Went way better than I expected. Here's the AI assisted writeup, with the summary being that I'd describe early results as very encouraging. It crushes a reasonably well optimised (SIMD) CPU pipeline on some random composition of available operations.

https://github.com/Quafadas/vecxt/blob/gvecxt/site/notes/gpu.md

Heavy pipeline: sin(exp(a+b)) * cos(a*b) + atan(a/b) |> exp |> sqrt |> log — ~15 FLOPs/element

Size	GPU	CPU unfused	CPU fused
1M	13ms	174ms	158ms
10M	96ms	747ms	1596ms
100M	2.1s	8.9s	16.2s

In fairness, Claude did nearly all the heavy lifting here, but the key points are that;

• It absorbed Cyfra's design and seemed to like it
• Burped a pipeline which seems to compose array / matrix type workloads very nicely.

Quodos to your efforts in Cyfra. Wow.

[szymon-rd]

That's great! I've glanced over the report. A lot of issues it highlights are resolved in the branch where I implemented the llm inference - like primitives for matmul, subgroup reductions. Once it's merged, it should unblock it.