I’m really glad to see MVE float support finally being added to kernels like add, mul, and softmax.
I imagine this is also becoming increasingly relevant through the LLM/ExecuTorch work, where models such as SmolLM2 show that some operations still need to remain in floating point even when the linear layers are quantized.
From my experience with larger models, I think the next important addition should be MVE-optimized quantize/dequantize/requantize kernels. For example, I already ported Gemma-3-270m to an edge device (M85+U85) with TFLM, and optimizing these conversion operations can save up to ~10%, depending on the context window. Every small performance improvement is relevant for tokens per second maxxing.
I hope integrating the new MVE float kernels into TFLM through tensorflow/lite/micro/kernels/cmsis_nn is also on the roadmap, so TFLM can make use of them directly.
I’m really glad to see MVE float support finally being added to kernels like
add,mul, andsoftmax.I imagine this is also becoming increasingly relevant through the LLM/ExecuTorch work, where models such as SmolLM2 show that some operations still need to remain in floating point even when the linear layers are quantized.
From my experience with larger models, I think the next important addition should be MVE-optimized quantize/dequantize/requantize kernels. For example, I already ported Gemma-3-270m to an edge device (M85+U85) with TFLM, and optimizing these conversion operations can save up to ~10%, depending on the context window. Every small performance improvement is relevant for tokens per second maxxing.
I hope integrating the new MVE float kernels into TFLM through
tensorflow/lite/micro/kernels/cmsis_nnis also on the roadmap, so TFLM can make use of them directly.