Skip to content

feat: Add cross-platform deterministic math via fdlibm#2602

Open
Okladnoj wants to merge 14 commits intoTheSuperHackers:mainfrom
Okladnoj:okji/feat/deterministic-math
Open

feat: Add cross-platform deterministic math via fdlibm#2602
Okladnoj wants to merge 14 commits intoTheSuperHackers:mainfrom
Okladnoj:okji/feat/deterministic-math

Conversation

@Okladnoj
Copy link
Copy Markdown

@Okladnoj Okladnoj commented Apr 15, 2026

Replace hardware-dependent x87 FPU trig functions (fsin, fcos) in WWMath with fdlibm 5.3 — a portable, bit-exact IEEE 754 C implementation. This ensures lockstep CRC parity between macOS ARM64/x64 and Windows x86 clients, eliminating multiplayer desyncs caused by floating-point precision divergence.

Changes:

  • Integrate fdlibm 5.3 (Sin, Cos, Sqrt, Acos, Asin, Atan, Atan2) into Core/Libraries/Source/WWVegas/WWMath/fdlibm/
  • Replace all x87 asm blocks in wwmath.h with fdlibm wrappers
  • Route ~80+ direct sin/cos/sqrt/atan2 calls in GameLogic through WWMath (Locomotor, Weapon, PhysicsUpdate, PartitionManager, AIStates, etc.)
  • Replace Inv_Sqrt Quake-era hack with 1.0f/WWMath::Sqrt()
  • Gate all changes behind USE_DETERMINISTIC_MATH (RETAIL_COMPATIBLE_CRC)
  • Clean Weapon.cpp diff to contain only functional WWMath replacements
  • Preserve Fast_Sin/Fast_Cos LUT (already deterministic)
  • Leave render/UI layer (GameClient, WW3D2) on system math (no CRC impact)

Tested: 3+ hours of cross-platform multiplayer (macOS Wine × Windows) with zero desyncs. Visual regression (health bars, veterancy icons) verified absent on native macOS Metal port.

Testing results

System Compiler Math Library CRC value Perf (10000 iters)
macOS ARM64 Apple Clang fdlibm (deterministic) 🟩 7BE96687 3.23 ms
macOS ARM64 Apple Clang system math (native) 🟩 7BE96687 1.71 ms
Windows x86 vc6 fdlibm (deterministic) 🟩 TBD (Waiting for test) -
Windows x86 vc6 system math (x87 FPU) 🟥 TBD (Waiting for test) -

Note: On ARM64, the native system math is already strictly IEEE 754 compliant, meaning it natively achieves parity with fdlibm without 80-bit precision drift.

The performance delta (~89% overhead for fdlibm) is measured in a pure trigonometry micro-benchmark. In actual gameplay, math calls represent a negligible fraction of the main game loop, and 3+ hours of testing confirmed no perceptible impact on FPS.

@stephanmeesters
Copy link
Copy Markdown

stephanmeesters commented Apr 15, 2026

I think it would be better to import fdlibm as an external library we can grab with FetchContent_Declare see the examples in cmake/

Although there was an imgui PR that also tried to do it directly like this. Maybe discuss first then.

Could you make a table of CRC values like in #2100? Using vc6 and win32 and Windows/Mac pairs?

@greptile-apps
Copy link
Copy Markdown

greptile-apps Bot commented Apr 15, 2026

Greptile Summary

This PR replaces hardware-dependent x87 FPU trigonometric functions in WWMath with fdlibm-backed wrappers from TheSuperHackers/GameMath, gating all changes behind USE_DETERMINISTIC_MATH / RETAIL_COMPATIBLE_CRC to achieve lockstep CRC parity across macOS ARM64 and Windows x86 clients. Around 80+ direct sin/cos/sqrt/atan2 calls across GameLogic are routed through WWMath, and trig.h/Trig.cpp are removed in favour of the new wrappers.

  • P1 — SimulationMathCrc.cpp still calls system math directly: only WWMath::Sin and WWMath::Cos are deterministic; the remaining ten operations (tanf, sqrtf, atan2f, sinhf, coshf, tanhf, expf, log10f, logf, powf, asinf, acosf, atanf) bypass WWMath, so the CRC this function returns will still diverge between clients.

Confidence Score: 3/5

Not safe to merge: the dedicated CRC validation function still calls non-deterministic system math for 10 of 12 operations, undermining the core goal of the PR.

One confirmed P1 defect — SimulationMathCrc::appendSimulationMathCrc routes only Sin/Cos through WWMath while the remaining ten math calls (including sqrtf, atan2f, tanf, hyperbolic functions, and log/exp/pow) bypass deterministic wrappers entirely, producing a platform-divergent CRC. Several additional P1s were raised in prior review rounds (Float_To_Long rounding, Bresenham loop change, Inv_Sqrt UB) and their resolution status is unclear from the current diff alone.

Core/GameEngine/Source/Common/Diagnostic/SimulationMathCrc.cpp needs all raw math calls replaced with WWMath equivalents; Core/Libraries/Source/WWVegas/WWMath/wwmath.h should be checked for prior-round P1 resolution status.

Important Files Changed

Filename Overview
Core/Libraries/Source/WWVegas/WWMath/wwmath.h Core change: gates Sin/Cos/Sqrt/Acos/Asin/Atan/Atan2/Inv_Sqrt behind USE_DETERMINISTIC_MATH with gmath.h wrappers; removes always.h transitive include; adds redundant SinTrig/CosTrig aliases; several P1 issues flagged in prior review threads remain open
Core/GameEngine/Source/Common/Diagnostic/SimulationMathCrc.cpp Only Sin/Cos are routed through WWMath; 10 other math functions (tanf, sqrtf, atan2f, etc.) still call system library directly, making the CRC non-deterministic and defeating the PR's purpose
cmake/gamemath.cmake Integrates TheSuperHackers/GameMath via FetchContent with a full 40-char SHA pin; disables intrinsics and tests; links as PUBLIC so consumers get include paths automatically
Generals/Code/GameEngine/Source/GameLogic/Object/Behavior/DumbProjectileBehavior.cpp All atan2/sqrt/sin/cos/asin calls correctly migrated to WWMath equivalents; no issues found
GeneralsMD/Code/GameEngine/Source/GameLogic/Object/Behavior/DumbProjectileBehavior.cpp All atan2/sqrt/sin/cos/asin calls correctly migrated to WWMath equivalents; no issues found
Generals/Code/GameEngine/Source/GameLogic/Object/Update/PhysicsUpdate.cpp sqrt/atan2/acos calls correctly replaced with WWMath wrappers; uses WWMath::SinTrig instead of WWMath::Sin (redundant alias)
Core/Libraries/Source/WWVegas/WWMath/CMakeLists.txt Adds PUBLIC gamemath link for non-VS6 builds, correctly propagating include paths to consumers of core_wwmath

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart TD
    A[GameLogic Call Sites\nLocomot / Weapon / Physics / AI] -->|USE_DETERMINISTIC_MATH| B[WWMath wrappers\nSin, Cos, Sqrt, Atan2, ...]
    A -->|non-deterministic build| C[System math\nsinf / cosf / sqrtf]
    B -->|RETAIL_COMPATIBLE_CRC defined| D[GameMath / gmath.h\ngm_sinf, gm_cosf, gm_sqrtf ...]
    D --> E[fdlibm 5.3\nIEEE 754 bit-exact]
    B -->|else| C
    F[SimulationMathCrc.cpp] -->|Sin, Cos| B
    F -->|tanf / sqrtf / atan2f\n+ 7 more| C
    style F fill:#f96,color:#000
    style E fill:#6c6,color:#fff
Loading

Comments Outside Diff (1)

  1. Core/GameEngine/Source/Common/Diagnostic/SimulationMathCrc.cpp, line 40-51 (link)

    P1 Non-deterministic math bypasses WWMath in the CRC function

    Only WWMath::Sin and WWMath::Cos are routed through the deterministic wrappers; the remaining ten calls (tanf, asinf, acosf, atanf, atan2f, sinhf, coshf, tanhf, sqrtf, expf, log10f, logf, powf) invoke the system math library directly. When USE_DETERMINISTIC_MATH is active, these will still produce platform-divergent results on macOS vs. Windows x87 FPU, meaning the CRC returned by appendSimulationMathCrc will differ between clients — the exact failure mode this PR is designed to eliminate.

    Replace each system call with its WWMath counterpart (e.g. WWMath::Sqrt, WWMath::Atan2, WWMath::Tan, WWMath::Log10, WWMath::Pow, etc.).

    Prompt To Fix With AI
    This is a comment left during a code review.
    Path: Core/GameEngine/Source/Common/Diagnostic/SimulationMathCrc.cpp
    Line: 40-51
    
    Comment:
    **Non-deterministic math bypasses `WWMath` in the CRC function**
    
    Only `WWMath::Sin` and `WWMath::Cos` are routed through the deterministic wrappers; the remaining ten calls (`tanf`, `asinf`, `acosf`, `atanf`, `atan2f`, `sinhf`, `coshf`, `tanhf`, `sqrtf`, `expf`, `log10f`, `logf`, `powf`) invoke the system math library directly. When `USE_DETERMINISTIC_MATH` is active, these will still produce platform-divergent results on macOS vs. Windows x87 FPU, meaning the CRC returned by `appendSimulationMathCrc` will differ between clients — the exact failure mode this PR is designed to eliminate.
    
    Replace each system call with its `WWMath` counterpart (e.g. `WWMath::Sqrt`, `WWMath::Atan2`, `WWMath::Tan`, `WWMath::Log10`, `WWMath::Pow`, etc.).
    
    How can I resolve this? If you propose a fix, please make it concise.
Prompt To Fix All With AI
This is a comment left during a code review.
Path: Core/GameEngine/Source/Common/Diagnostic/SimulationMathCrc.cpp
Line: 40-51

Comment:
**Non-deterministic math bypasses `WWMath` in the CRC function**

Only `WWMath::Sin` and `WWMath::Cos` are routed through the deterministic wrappers; the remaining ten calls (`tanf`, `asinf`, `acosf`, `atanf`, `atan2f`, `sinhf`, `coshf`, `tanhf`, `sqrtf`, `expf`, `log10f`, `logf`, `powf`) invoke the system math library directly. When `USE_DETERMINISTIC_MATH` is active, these will still produce platform-divergent results on macOS vs. Windows x87 FPU, meaning the CRC returned by `appendSimulationMathCrc` will differ between clients — the exact failure mode this PR is designed to eliminate.

Replace each system call with its `WWMath` counterpart (e.g. `WWMath::Sqrt`, `WWMath::Atan2`, `WWMath::Tan`, `WWMath::Log10`, `WWMath::Pow`, etc.).

How can I resolve this? If you propose a fix, please make it concise.

---

This is a comment left during a code review.
Path: Core/Libraries/Source/WWVegas/WWMath/wwmath.h
Line: 146-178

Comment:
**`SinTrig`/`CosTrig`/`TanTrig`/`ACosTrig`/`ASinTrig` duplicate existing methods**

`WWMath::Sin`, `WWMath::Cos`, `WWMath::Acos`, and `WWMath::Asin` already exist and are routed through the same `gm_*f` calls under `USE_DETERMINISTIC_MATH`. The new `SinTrig`/`CosTrig` aliases were added to replace deleted `trig.h` free functions, but they create two public API paths for the identical operation. Call sites in `DumbProjectileBehavior.cpp`, `PhysicsUpdate.cpp`, etc. should use `WWMath::Sin`/`WWMath::Cos` directly, and the `*Trig` aliases can be removed to avoid future confusion.

How can I resolve this? If you propose a fix, please make it concise.

Reviews (23): Last reviewed commit: "feat: Replace WWMath::Cos/Sin with CosTr..." | Re-trigger Greptile

Comment thread Core/Libraries/Source/WWVegas/WWMath/wwmath.h
@Okladnoj Okladnoj force-pushed the okji/feat/deterministic-math branch from a3fce09 to 8f7952d Compare April 15, 2026 18:21
Comment thread GeneralsMD/Code/GameEngine/Source/GameLogic/Object/PartitionManager.cpp Outdated
@Okladnoj Okladnoj force-pushed the okji/feat/deterministic-math branch from 8f7952d to 93f6fea Compare April 15, 2026 18:31
Comment thread Core/Libraries/Source/WWVegas/WWMath/wwmath.h
Comment thread Core/Libraries/Source/WWVegas/WWMath/wwmath.h Outdated
Comment thread cmake/fdlibm.cmake Outdated
Comment thread Core/Libraries/Source/WWVegas/WWMath/wwmath.h
Comment thread Core/Libraries/Source/WWVegas/WWMath/wwmath.h Outdated
@Okladnoj Okladnoj force-pushed the okji/feat/deterministic-math branch from 93f6fea to 5dcc907 Compare April 15, 2026 22:40
Comment thread cmake/fdlibm.cmake Outdated
Replace hardware-dependent x87 FPU trig functions (fsin, fcos) in
WWMath with fdlibm 5.3 — a portable, bit-exact IEEE 754 C implementation.
This ensures lockstep CRC parity between macOS ARM64/x64 and Windows x86
clients, eliminating multiplayer desyncs caused by floating-point
precision divergence.

Changes:
- Integrate fdlibm 5.3 via FetchContent from Okladnoj/fdlibm-deterministic
- Replace all x87 asm blocks in wwmath.h with fdlibm wrappers
- Route ~80+ direct sin/cos/sqrt/atan2 calls in GameLogic through WWMath
- Replace Inv_Sqrt Quake-era hack with 1.0f/WWMath::Sqrt()
- Gate all changes behind USE_DETERMINISTIC_MATH (RETAIL_COMPATIBLE_CRC)
- Clean Weapon.cpp diff to contain only functional WWMath replacements
- Preserve Fast_Sin/Fast_Cos LUT (already deterministic)
- Leave render/UI layer (GameClient, WW3D2) on system math (no CRC impact)
- Add SimulationMathCrc dual-path diagnostic (fdlibm vs system math)
@Okladnoj Okladnoj force-pushed the okji/feat/deterministic-math branch from 5dcc907 to 3e28af3 Compare April 15, 2026 22:46
@Mauller
Copy link
Copy Markdown

Mauller commented Apr 16, 2026

There are many other places in the codebase that use Sin and Cos, so the implementation is incomplete.

@OmniBlade
Copy link
Copy Markdown

I evaluated fdlib when I was looking at a replacement math library for Thyme but I decided against it as its very old. I cobbled together a similar library from a newer (though still old at this point) snapshot of the FreeBSD library https://github.com/TheAssemblyArmada/GameMath which also supports float precision (another limitation of the fdmath lib is that it only provides double precision versions of functions).

@stephanmeesters
Copy link
Copy Markdown

There are many other places in the codebase that use Sin and Cos, so the implementation is incomplete.

You mean ones potentially affecting logic CRC?

@Mauller
Copy link
Copy Markdown

Mauller commented Apr 16, 2026

There are many other places in the codebase that use Sin and Cos, so the implementation is incomplete.

You mean ones potentially affecting logic CRC?

Yes, in quite a few places.

@Mauller
Copy link
Copy Markdown

Mauller commented Apr 16, 2026

There should be 3 modes of usage here, Retail compatible, standard library and the new deterministic math.

One of the first things that really needs doing is having all game code call math library functions from a single place where we can substitute in the various optional variants of the functions.

@Okladnoj
Copy link
Copy Markdown
Author

There are many other places in the codebase that use Sin and Cos, so the implementation is incomplete.

You mean ones potentially affecting logic CRC?

Yes, in quite a few places.

@Mauller You were absolutely right. After thoroughly checking the GameLogic/ directory, I found ~11 remaining sin/cos/atan2 calls that could affect CRC.

I just pushed a commit that fixes them. To avoid cluttering the codebase with wwmath.h includes everywhere, I seamlessly routed them through the global SAGE wrappers Atan2/Sin/Cos via Trig.h.

(By the way, the audit confirmed that the standard sqrt is fully deterministic between x86 and ARM64 per the IEEE 754 standard, so those calls were safely left as-is).

The math should now be completely ready for testing.

@xezon
Copy link
Copy Markdown

xezon commented Apr 17, 2026

Please look into GameMath as brought forward by OmniBlade.

@Okladnoj
Copy link
Copy Markdown
Author

Please look into GameMath as brought forward by OmniBlade.

Sure! Since I've already abstracted all direct math calls behind WWMath and Trig.h, swapping the backend from fdlibm to GameMath to get native float-precision will be very straightforward. I'll look into substituting it today/tomorrow.

@xezon
Copy link
Copy Markdown

xezon commented Apr 17, 2026

@Mauller
Copy link
Copy Markdown

Mauller commented Apr 17, 2026

@Mauller You were absolutely right. After thoroughly checking the GameLogic/ directory, I found ~11 remaining sin/cos/atan2 calls that could affect CRC.

I just pushed a commit that fixes them. To avoid cluttering the codebase with wwmath.h includes everywhere, I seamlessly routed them through the global SAGE wrappers Atan2/Sin/Cos via Trig.h.

(By the way, the audit confirmed that the standard sqrt is fully deterministic between x86 and ARM64 per the IEEE 754 standard, so those calls were safely left as-is).

The math should now be completely ready for testing.

There are still missed files that will affect the CRC

Comment thread GeneralsMD/Code/GameEngine/Source/Common/System/Trig.cpp Outdated
Comment thread cmake/gamemath.cmake Outdated
@Okladnoj Okladnoj force-pushed the okji/feat/deterministic-math branch from 186a237 to e49cf3f Compare April 17, 2026 10:15
@greptile-apps
Copy link
Copy Markdown

greptile-apps Bot commented Apr 17, 2026

Want your agent to iterate on Greptile's feedback? Try greploops.

@Okladnoj
Copy link
Copy Markdown
Author

Hello again @xezon @stephanmeesters @Mauller @OmniBlade! Lately I've been working almost exclusively in the GameClient repo. Since we currently only support Zero Hour there, my previous adjustments regarding CRC math didn't touch the vanilla Generals engine. Also, in the process of making these edits, I found and fixed a few remaining non-deterministic math operations in the ZH directory as well.

Important note: since I develop and test everything on macOS (which only runs the Zero Hour client), I was only able to properly test and confirm network sync for the ZH fixes. For the base game, I just did a pure mirror copy of those exact same fixes. Therefore, it would be great if someone could run vanilla on a 32-bit Windows build and just double-check that I didn't break anything there. Thanks!

@Okladnoj
Copy link
Copy Markdown
Author

@xezon
Regarding the suggestion to completely delete the old Trig files:
I don't think that's a good idea. This is original EA legacy code, and it's much safer to leave the file structure intact so we don't accidentally break some implicit or overridden dependencies hidden somewhere deep in the engine. It is far more reliable to just reroute the internal logic inside Trig.cpp to the new WWMath functions.

@xezon
Copy link
Copy Markdown

xezon commented Apr 17, 2026

This is original EA legacy code, and it's much safer to leave the file structure intact so we don't accidentally break some implicit or overridden dependencies hidden somewhere deep in the engine.

Can you give an example for it? Generally it is a bad sign if simplifying code would break something. If so, it needs to be fixed.

The only real concern would be if it broke VC6 retail compat.

…WMath wrappers

Replaces non-deterministic trigonometric and generic floating point math function calls with deterministic counterparts from the WWMath library across the core codebase. This ensures mathematical parity to prevent frame desyncs during multiplayer lockstep execution, and includes build-fixes for missing WWMath namespaces and header removals.
@Okladnoj
Copy link
Copy Markdown
Author

@xezon

Can you give an example for it? Generally it is a bad sign if simplifying code would break something. If so, it needs to be fixed.

The only real concern would be if it broke VC6 retail compat.

The old Trig files have been deleted. BaseType.h was fixed since it depended on Trig, and all math in the codebase has been rewritten via WWMath::.

@stephanmeesters
Copy link
Copy Markdown

Therefore, it would be great if someone could run vanilla on a 32-bit Windows build and just double-check that I didn't break anything there. Thanks!

It's not compiling on win32 or vc6 right now. I think it's important you can compile and test both these builds, so not to shift the burden to the reviewers, perhaps use the Docker build or run Parallels on your Mac?

- Guard gamemath.cmake and link with NOT IS_VS6_BUILD (VC6 lacks long long, stdint.h)
- Guard gmath.h include in wwmath.h with _MSC_VER >= 1300 check
- Fix ACos -> Acos case mismatch in BaseType.h (3 call sites)
Comment thread cmake/gamemath.cmake Outdated
@Caball009
Copy link
Copy Markdown

Please restore the original spacing, particularly for wwmath.h.

@Okladnoj Okladnoj force-pushed the okji/feat/deterministic-math branch 2 times, most recently from a0a6ec1 to ea2411c Compare April 18, 2026 20:57
Comment thread cmake/gamemath.cmake
@Okladnoj Okladnoj force-pushed the okji/feat/deterministic-math branch from 1dfb766 to be7ac16 Compare April 18, 2026 22:26
@Okladnoj
Copy link
Copy Markdown
Author

It's not compiling on win32 or vc6 right now. I think it's important you can compile and test both these builds, so not to shift the burden to the reviewers, perhaps use the Docker build or run Parallels on your Mac?

@stephanmeesters @xezon
The builds have now compiled successfully! The only thing failing is the replay checks, but that's failing on the asset download step which I don't have access to fix.

The root cause was that our isolated console utilities heavily depended on the math from the old removed Trig files.

Regarding your suggestion about testing locally: unfortunately, Docker and Parallels on Apple Silicon Macs don't support win32 environments. However, iterating through GitHub CI turned out to be a great solution for this!


image

@Okladnoj Okladnoj force-pushed the okji/feat/deterministic-math branch 2 times, most recently from e74de2c to 5edf2b9 Compare April 18, 2026 23:04
Extracted wwmath.h from tight coupling with the 3D engine's always.h
macro definitions to establish WWMath as an independent Level 1 base library
interface. Exposes Libraries/Source/WWVegas/WWMath and gamemath
directly to corei_libraries_include targets, ensuring low-level utilities
like Babylon and versionUpdate can compile BaseType.h without transitive
rendering dependencies.
@Okladnoj Okladnoj force-pushed the okji/feat/deterministic-math branch from 5edf2b9 to 5e3d599 Compare April 19, 2026 09:28
@stephanmeesters
Copy link
Copy Markdown

stephanmeesters commented Apr 19, 2026

The builds have now compiled successfully!

Nice one. I have tested the replays but they all fail unfortunately.

Current PR is hard to review for errors now because it tries to do many things, also there are many unnecessary formatting changes, and you made changes to both Generals and Zero Hour already. But it is valuable exploration

So we would need multiple smaller PR's that are checked very carefully. I think @Skyaero42 will be picking this up

…ig.cpp replacement

SinTrig/CosTrig use cosf/sinf (matching original Trig.cpp CRT behavior) without
USE_DETERMINISTIC_MATH, and gm_cosf/gm_sinf with it enabled.

This preserves CRC compatibility with retail replays in non-deterministic builds
while providing cross-platform determinism when USE_DETERMINISTIC_MATH is active.
@Okladnoj Okladnoj force-pushed the okji/feat/deterministic-math branch from 129eae9 to d28bcb2 Compare April 27, 2026 22:50
…erals + GeneralsMD)

All game logic files that originally used global Cos()/Sin() from Trig.cpp
now use WWMath::CosTrig()/SinTrig() which preserves cosf()/sinf() behavior
without USE_DETERMINISTIC_MATH, and routes through gm_cosf()/gm_sinf()
when deterministic math is enabled.
@Okladnoj Okladnoj force-pushed the okji/feat/deterministic-math branch from c7679c8 to 9daa569 Compare April 27, 2026 23:40
@xezon
Copy link
Copy Markdown

xezon commented Apr 30, 2026

The consolidation attempt of trig and wwmath should be a standalone change, to allow for clean review and testing.

@Okladnoj
Copy link
Copy Markdown
Author

Okladnoj commented May 1, 2026

The consolidation attempt of trig and wwmath should be a standalone change, to allow for clean review and testing.

@xezon Agreed. These past 2 weeks, the fact that replays were breaking on Windows with the new math kept bugging me, so I started fresh on a separate branch (okji/feat/deterministic-math-v2) with a cleaner approach — no radical code changes.

In the process I realized that a single math function in this project can have 3 variants: VC6 (x87 inline asm), CRT (standard library), and our new Deterministic (GameMath).

Right now all replays pass perfectly with deterministic math both enabled and disabled — which is strange to me. With USE_DETERMINISTIC_MATH enabled, win32 replays should fail against old golden replays (recorded with x87), while vc6 should succeed (it doesn't use GameMath)... or am I missing something?

I've opened a clean PR (#2670), but I'm still working on it. If anyone has ideas why win32 doesn't fail with USE_DETERMINISTIC_MATH on old replays — please let me know.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

8 participants