From-scratch Ring-AllReduce on CPU (C++20) and GPU (CUDA, multi-device): real multi-GPU benchmarks, Nsight profiling, and a bandwidth-optimal ring implementation built without NCCL.

Distributed training from scratch: ring all-reduce built from send/recv, data/tensor/pipeline parallelism, and honest scaling efficiency measurements

Ring-allreduce collective built from scratch in C++20 over raw MPI point-to-point primitives, benchmarked against Open MPI 5.0.10's MPI_Allreduce across 8B-128MiB and N=2..16. An alpha-beta cost model shows the ring's small-message loss is the algorithm's 2(N-1) step count, not implementation quality: Open MPI's own forced ring is just as slow.

Distributed training from first principles: ring, tree and halving-doubling all-reduce over raw TCP, with data-parallel, fully-sharded, tensor and pipeline parallelism composed into 2D and 3D meshes