Add tile-backed proof generation

[Amaury Chamayou (achamayou)]

Summary

Part 3 of the tiled-storage work. This branch is rebased onto current main (6d70d02) after #52 and #53 merged.

• introduce abstract HashSourceT plus tile-backed, memory-backed, and combined implementations;
• add ProofEngineT roots and inclusion proofs, byte-identical to TreeT::path() / past_path(), and RFC 6962 consistency proofs and verification;
• add the TreeT::subtree_root() accessor for complete resident subtrees with an optional result and unchanged hashing semantics;
• make full-subtree arithmetic defined through the maximum size_t tree height;
• avoid redundant tile-cache copies and consistency-proof copies, and use one allocation-free reducer for perfect subtree roots and tile roll-ups;
• use the current default Tree::hash_function aliases and reject unsafe index conversions and arithmetic overflow;
• cover mixed tile/memory ranges, tile boundaries and tampering, hostile indices and large-height arithmetic, and completed level-2 tile resolution;
• document format compatibility, source composition, proof algorithms, and the remaining lifecycle-wrapper phase.

Compatibility

Tile hashes and proof assembly use the tree's existing HASH_FUNCTION; leaf and node hashing behavior is unchanged. Inclusion proofs remain byte-for-byte compatible with existing merklecpp paths, and the default SHA-256 implementation remains OpenSSL-independent.

CI

CMake detects each runner's logical processor count and exports the standard build and CTest parallelism variables. Tests use CMake's default-on BUILD_TESTING option. tiles_level2 remains behind LONG_TESTS, which is enabled for every Release matrix job. The test retains a 900-second timeout on Linux and a 3600-second timeout on Windows. CodeQL also builds the long-test target.

Rebase status

• base: main at 6d70d02;
• head: achamayou/tiles-proofs at 741fb70;
• divergence: 0 commits behind, 10 commits ahead.

Validation

• focused tiles_proofs and tiles_writer tests;
• Debug Clang 18 ASan/UBSan tiles_proofs test;
• Release build with LONG_TESTS=ON;
• full parallel test suite (17/17 passed on Linux, including tiles_level2);
• SHA-384 and SHA-512 reducer equivalence probes;
• CI-equivalent clang-tidy checks for merklecpp.h and merklecpp_tiles.h.

[Copilot]

🟡 Not ready to approve

There are actionable API/performance issues in the new proof/tile-reading code paths that should be addressed before approval.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

This review doesn't count toward merge requirements. Sign up for the private preview to control whether Copilot approvals count.

Pull request overview

This PR adds tile-backed proof generation infrastructure for merklecpp’s tiled-storage stack, including abstract subtree-root sources (memory, tiles, and combined), a proof engine that produces inclusion/consistency proofs compatible with existing TreeT paths, and test/CI wiring to validate large tile hierarchies.

Changes:

• Add TreeT::subtree_root() to expose resident perfect-subtree roots without changing hashing semantics.
• Introduce HashSourceT implementations and ProofEngineT for roots, inclusion proofs, RFC 6962 consistency proofs, and verification.
• Add new proof and level-2 tile end-to-end tests plus LONG_TESTS gating and CI configuration to run long tests selectively.

File summaries

File	Description
test/tiles_proofs.cpp	Adds tile+memory proof equivalence tests vs existing TreeT::path() / past_path() and consistency verification.
test/tiles_level2.cpp	Adds an end-to-end long test that materializes a full level-2 tile and validates roll-ups and TileHashSource resolution.
test/CMakeLists.txt	Registers new tiles proof tests and gates tiles_level2 behind LONG_TESTS with a timeout.
merklecpp.h	Adds TreeT::subtree_root() accessor for resident perfect subtrees with overflow/flush checks.
merklecpp_tiles.h	Adds HashSourceT abstraction, tile/memory/combined implementations, and ProofEngineT (inclusion + consistency proofs).
doc/design/tlog-tiles.md	Updates design doc to reflect the proof engine/hash-source architecture and lifecycle plan changes.
CMakeLists.txt	Introduces LONG_TESTS CMake option.
.github/workflows/codeql-analysis.yml	Enables LONG_TESTS during CodeQL build configuration.
.github/workflows/ci.yml	Runs LONG_TESTS only on a single selected Linux job to avoid Windows timeouts.

Review details

• Files reviewed: 10/10 changed files
• Comments generated: 3
• Review effort level: Lite

We're testing this review assessment. Please use 👍 or 👎 to tell us if it's correct.

[Copilot]

🟡 Not ready to approve

Proof-generation hot-path code introduces avoidable heap allocations, and the new tests rely on transitive standard header includes that should be made explicit.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

This review doesn't count toward merge requirements. Sign up for the private preview to control whether Copilot approvals count.

Review details

Suppressed comments (3)

merklecpp_tiles.h:1025

• roll_up() allocates a temporary std::vector slice for every subtree roll-up, which is a hot path during proof generation and defeats the goal of avoiding redundant copies. Since span is always a power-of-two <= TILE_WIDTH, this can be rolled up in-place using a fixed buffer and HASH_FUNCTION without heap allocation.

        }
        return perfect_root<HASH_SIZE, HASH_FUNCTION>(std::vector<Hash>(
          tile.begin() + (std::ptrdiff_t)off,
          tile.begin() + (std::ptrdiff_t)(off + span)));

test/tiles_proofs.cpp:15

• This test throws std::runtime_error but does not include directly. Relying on transitive includes from merklecpp headers is not guaranteed and can break with different standard library implementations or include order.

#include <iostream>
#include <limits>
#include <merklecpp.h>
#include <merklecpp_tiles.h>
#include <string>

test/tiles_level2.cpp:18

• This test throws std::runtime_error but does not include directly. Relying on transitive includes from other headers is not guaranteed and can break with different standard library implementations or include order.

#include <ctime>
#include <filesystem>
#include <iostream>
#include <merklecpp.h>
#include <merklecpp_tiles.h>

• Files reviewed: 10/10 changed files
• Comments generated: 0 new
• Review effort level: Lite

We're testing this review assessment. Please use 👍 or 👎 to tell us if it's correct.

[Copilot]

🟡 Not ready to approve

The new subtree-root accessor introduces a confirmed UB path (via is_full() shifting) and the new tests’ temp-dir naming can collide under the newly enabled parallel CI test execution.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

This review doesn't count toward merge requirements. Sign up for the private preview to control whether Copilot approvals count.

Review details

Suppressed comments (3)

merklecpp_tiles.h:1025

• TileHashSourceT::roll_up() allocates/copies a temporary std::vector<Hash> for every span>1 roll-up. This can become a noticeable hotspot during proof generation (many small roll-ups). You can compute the perfect root directly over the tile slice without allocating.

        }
        return perfect_root<HASH_SIZE, HASH_FUNCTION>(std::vector<Hash>(
          tile.begin() + (std::ptrdiff_t)off,
          tile.begin() + (std::ptrdiff_t)(off + span)));
      }

test/tiles_proofs.cpp:281

• The temp directory name is derived from time() + rand() seeded with time(). With ctest --parallel enabled in CI, multiple test binaries can start in the same second and produce the same directory name, causing flaky collisions. Use a nonce that includes the process id and a monotonic/sequence component.

  const fs::path base = fs::temp_directory_path() /
    ("merklecpp_tiles_proofs_" + std::to_string((unsigned long long)seed) +
     "_" + std::to_string(std::rand()));

test/tiles_level2.cpp:50

• The temp directory name is based only on time(nullptr), which can collide when tests run in parallel (CI now uses ctest --parallel). Use a name that incorporates process id and a monotonic/sequence nonce to avoid flaky collisions.

  const fs::path dir = fs::temp_directory_path() /
    ("merklecpp_tiles_level2_" +
     std::to_string((unsigned long long)std::time(nullptr)));

• Files reviewed: 10/10 changed files
• Comments generated: 1
• Review effort level: Lite

We're testing this review assessment. Please use 👍 or 👎 to tell us if it's correct.

[Copilot]

🟡 Not ready to approve

The CMake testing toggle change can silently break existing builds that still pass -DTESTS=ON/OFF, so compatibility mapping (or equivalent) should be added before approval.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

This review doesn't count toward merge requirements. Sign up for the private preview to control whether Copilot approvals count.

Review details

Suppressed comments (1)

CMakeLists.txt:10

• The build option for enabling/disabling tests switched from the project-specific TESTS cache variable to CTest's BUILD_TESTING. As written, -DTESTS=ON/OFF is now silently ignored (and BUILD_TESTING defaults to ON via include(CTest)), which can break existing scripts and make it hard to turn tests off in downstream builds. Consider adding a small compatibility shim that maps TESTS to BUILD_TESTING (with a deprecation warning) before including CTest.

include(GNUInstallDirs)
include(CTest)

• Files reviewed: 10/10 changed files
• Comments generated: 0 new
• Review effort level: Lite

We're testing this review assessment. Please use 👍 or 👎 to tell us if it's correct.

[Copilot]

🟡 Not ready to approve

The switch to BUILD_TESTING without top-level gating can unexpectedly enable tests and FetchContent downloads when merklecpp is consumed as a subproject, which can break downstream builds.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

This review doesn't count toward merge requirements. Sign up for the private preview to control whether Copilot approvals count.

Review details

Suppressed comments (1)

CMakeLists.txt:49

• include(CTest) makes BUILD_TESTING default to ON, so this condition will add the test/ subtree (and FetchContent-download doctest) even when merklecpp is consumed as a dependency via add_subdirectory/FetchContent. That can unexpectedly pull network dependencies and slow or break downstream builds; consider only adding tests when merklecpp is the top-level project.

if(BUILD_TESTING)

• Files reviewed: 10/10 changed files
• Comments generated: 0 new
• Review effort level: Lite

We're testing this review assessment. Please use 👍 or 👎 to tell us if it's correct.

[Amaury Chamayou (achamayou)]

The switch to BUILD_TESTING without top-level gating can unexpectedly enable tests and FetchContent downloads when merklecpp is consumed as a subproject, which can break downstream builds.

There aren't downstream builds, we are doing this cleanup now.

[Copilot]

🟡 Human review recommended

It changes core merkle tree internals and introduces new proof/verification logic plus CI/test wiring, which warrants final human review despite the added test coverage.

Review details

• Files reviewed: 10/10 changed files
• Comments generated: 0 new
• Review effort level: Lite

We're testing this review assessment. Please use 👍 or 👎 to tell us if it's correct.

[Copilot]

🟢 Ready to approve

The changes are cohesive, well-tested via new targeted and long-running tests, and the build/CI updates align with the new testing and proof-generation components without introducing evident correctness issues.

This review doesn't count toward merge requirements. Sign up for the private preview to control whether Copilot approvals count.

Review details

• Files reviewed: 10/10 changed files
• Comments generated: 0 new
• Review effort level: Lite

We're testing this review assessment. Please use 👍 or 👎 to tell us if it's correct.