[ntuple] Support multiple column representations in the merger

tree/ntuple/doc/BinaryFormatSpecification.md

@@ -1,4 +1,4 @@
-# RNTuple Binary Format Specification 1.0.2.0
+# RNTuple Binary Format Specification 1.1.0.0
 
 ## Versioning Notes
 
@@ -167,7 +167,12 @@ That means that readers need to continue reading feature flags as long as their
 
 Readers should gracefully abort reading when they encounter unknown bits set.
 
-At the moment, there are no feature flag bits defined.
+Here is the list of all currently-defined feature flags. Note that the flag name is only for informational purposes
+and is not normative.
+
+| Flag Bit | Introduced in | Name                    | Meaning                                      |
+|----------|---------------|-------------------------|----------------------------------------------|
+| 0        | 1.1.0.0       | Nested Deferred Columns | Signals that the RNTuple contains at least one deferred column that is part of a collection and was extended<br>(i.e. it appears in the footer). This can happen when merging two RNTuples that have the same collection field<br>backed by columns with different encoding, e.g. a `vector<float>` whose elements are represented by SplitReal32<br>in the first ntuple and by Real32 in the second. |
 
 
 ## Frames

tree/ntuple/doc/Merging.md

@@ -15,11 +15,13 @@ Please note that the RNTupleMerger is currently experimental and the content of
 
 Currently there is no guarantee for the user about which mode will be used to generate the merged RNTuple.
 At the moment, this is how it works:
-- if both compression and encoding of the target column match those of the source column, L1 is used;
-- otherwise, if compression matches but encoding doesn't, L2 is used;
-- otherwise L3 is used.
+- if the compression of the target column match that of the source column, L1 is used;
+- otherwise, L2 is used.
 
-Note that L0 and L4 are currently never used.
+L0, L3 and L4 are currently never used.
+
+**NOTE**: prior to ROOT 6.42, if two columns had the same compression but different encoding they would undergo L3 merging (implying a recompression and resealing);
+from 6.42 onwards the RNTupleMerger will instead attach a new column to the parent field as a new representation and L1-merge them.
 
 ## Goal
 The goal of the RNTuple merging process is producing one output RNTuple from *N* input RNTuples that can be used as if it were produced directly in the merged state. This means that:
@@ -44,15 +46,16 @@ Consequences of R3 and R4:
 The following properties are currently true but they are subject to change:
 
 * P1: all output pages have the **same compression** (which may be different from the input pages' compression);
-* P2: all pages in the same output column have the **same encoding** (which may be different from the inputs' encoding);
-* P3: the output clusters are **the same as the input clusters**;
-* P4: the output RNTuple **always has 1 cluster group**
+* P2: the output clusters are **the same as the input clusters**;
+* P3: the output RNTuple **always has 1 cluster group**
+
+Note that these properties influence and are influenced by the level of merging used. 
+E.g. P1 is currently true because we only support L1 merging of pages with identical compressions. This is a limitation that we intend to lift at some point (both for L1 and L0 if we ever support it).
+P2 and P3 would not necessarily be true with L4 support (which might be desirable in some cases, e.g. to group pages into smaller/larger clusters).
 
-Note that these properties influence and are influenced by the level of merging used.
-E.g. P1 and P2 are currently true because we only support L1 merging of pages with identical compressions. This is a limitation that we intend to lift at some point (both for L1 and L0 if we ever support it).
-P3 and P4 would not necessarily be true with L4 support (which might be desirable in some cases, e.g. to group pages into smaller/larger clusters).
+Also note that the output pages coming from matching columns of a field may use mixed encodings.
 
-Therefore we *will* want to drop these properties at some point, in order to improve the capabilities of the Merger.
+Therefore we *will* want to drop at least some of these properties at some point, in order to improve the capabilities of the Merger.
 
 ## High-level description
 The merging process requires at least 1 input, in the form of an `RPageSource`.
@@ -64,14 +67,15 @@ In `Union` mode only, we allow any subsequent input RNTuple to define new fields
 ## Descriptor compatibility and validation
 Whenever a new input is processed, we compare its descriptor with the output descriptor to verify that merging is possible.
 
-The comparison function does 3 main things:
+The comparison function does 4 main things:
 - collect all "extra destination fields" (i.e. fields that exist in the output but not in this input RNTuple)
 - collect all "extra source fields" from the input RNTuple
-- collect and validate all common fields.
+- collect and validate all common fields
+- collect all columns that need to be extended with additional representations.
 
-If the Merging Mode is set to **Filter** we require the "extra destination fields" list to be empty.
-If the Merging Mode is set to **Strict** we require both the "extra destination fields" and "extra source fields" lists to be empty.
-If the Merging Mode is set to **Union**, the "extra source fields" list is used to late model extend the destination model.
+If the merging mode is set to **Filter** we require the "extra destination fields" list to be empty.
+If the merging mode is set to **Strict** we require both the "extra destination fields" and "extra source fields" lists to be empty.
+If the merging mode is set to **Union**, the "extra source fields" list is used to late model extend the destination model.
 
 As for common fields, they are matched by name and validated as follows:
 - any field that is projected in the destination must be also projected in the source and must be projected to the same field;
@@ -90,3 +94,27 @@ As for common fields, they are matched by name and validated as follows:
 
 
 <sup>1</sup>: these restrictions will likely not be required for L4 merging.
+
+## Column representation extension
+In all merging modes, we allow new column representations to be attached to the source fields. This is done to allow for L1 merging of columns with different encodings, which would otherwise require recompressing.
+These new column representations are added to the output RNTuple's footer and become part of its Schema Extension section. Note that in general these columns will be added as deferred *and* suppressed.
+
+**Technical note**: this is *not* done via the regular late model extension API, but uses internal functionality.
+
+We add new (physical) column representations in the following cases:
+
+- when one or more columns of a field has a different type than its matching counterpart in the destination RNTuple;
+- when one or more columns of a field has the same type but different metadata than its matching counterpart in the destination RNTuple (e.g. in case of a Real32Quant column, different bit width or value range).
+
+Whenever we extend a physical column that is referred to by one or more alias columns in some projected fields, we also add a corresponding new alias column in those fields.
+
+#### Example
+Suppose we merge source RNTuples **S1** and **S2**, each with the following fields:
+
+1. `foo` of type `int`
+1. `fooProj` projecting onto field `foo`
+
+Suppose that S1 is compressed and thus its `foo` field is represented by a column of type `kSplitInt32`, whereas S2 is uncompressed and its `foo` field is represented by a column `kInt32`.
+When merging S1 and S2 we collate those two representations under the same field `foo`, so that it will now have representatives: `{kSplitInt32, kInt32}`.
+At the same time, we add a second alias column to the field `fooProj`, which will now have its first column aliasing the `kSplitInt32` column (column 0 of field `foo`) and its second one aliasing the `kInt32` one (column 1 of field `foo`).
+

tree/ntuple/inc/ROOT/RColumnElementBase.hxx

@@ -17,6 +17,7 @@
 #include <ROOT/RError.hxx>
 #include <ROOT/RFloat16.hxx>
 #include <ROOT/RNTupleTypes.hxx>
+#include <ROOT/RNTupleDescriptor.hxx>
 
 #include <Byteswap.h>
 #include <TError.h>
@@ -147,6 +148,13 @@ std::unique_ptr<RColumnElementBase> RColumnElementBase::Generate(ROOT::ENTupleCo
 template <>
 std::unique_ptr<RColumnElementBase> RColumnElementBase::Generate<void>(ROOT::ENTupleColumnType onDiskType);
 
+struct RColumnFormat {
+   ENTupleColumnType fType = ENTupleColumnType::kUnknown;
+   // 0 means "use default". Only valid for fixed-bitwidth column types.
+   std::uint16_t fBitWidth = 0;
+   std::optional<RColumnDescriptor::RValueRange> fValueRange;
+};
+
 } // namespace ROOT::Internal
 
 #endif

tree/ntuple/inc/ROOT/RField/RFieldFundamental.hxx

@@ -399,11 +399,11 @@ protected:
             fAvailableColumns.emplace_back(ROOT::Internal::RColumn::Create<T>(onDiskTypes[0], 0, representationIndex));
          if (onDiskTypes[0] == ROOT::ENTupleColumnType::kReal32Trunc) {
             const auto &fdesc = desc.GetFieldDescriptor(Base::GetOnDiskId());
-            const auto &coldesc = desc.GetColumnDescriptor(fdesc.GetLogicalColumnIds()[0]);
+            const auto &coldesc = desc.GetColumnDescriptor(fdesc.GetLogicalColumnIds()[representationIndex]);
             column->SetBitsOnStorage(coldesc.GetBitsOnStorage());
          } else if (onDiskTypes[0] == ROOT::ENTupleColumnType::kReal32Quant) {
             const auto &fdesc = desc.GetFieldDescriptor(Base::GetOnDiskId());
-            const auto &coldesc = desc.GetColumnDescriptor(fdesc.GetLogicalColumnIds()[0]);
+            const auto &coldesc = desc.GetColumnDescriptor(fdesc.GetLogicalColumnIds()[representationIndex]);
             assert(coldesc.GetValueRange().has_value());
             const auto [valMin, valMax] = *coldesc.GetValueRange();
             column->SetBitsOnStorage(coldesc.GetBitsOnStorage());

tree/ntuple/inc/ROOT/RFieldBase.hxx

@@ -260,14 +260,15 @@ private:
          func(target);
    }
 
-   /// Translate an entry index to a column element index of the principal column and vice versa. These functions
-   /// take into account the role and number of repetitions on each level of the field hierarchy as follows:
+   /// Translate an entry index to a column element index of the principal column. This function
+   /// takes into account the role and number of repetitions on each level of the field hierarchy as follows:
    /// - Top level fields: element index == entry index
    /// - Record fields propagate their principal column index to the principal columns of direct descendant fields
    /// - Collection and variant fields set the principal column index of their children to 0
    ///
    /// The column element index also depends on the number of repetitions of each field in the hierarchy, e.g., given a
-   /// field with type `std::array<std::array<float, 4>, 2>`, this function returns 8 for the innermost field.
+   /// field with type `std::array<std::array<float, 4>, 2>`, this function called with `globalIndex == 1`
+   /// returns 8 for the innermost field.
    ROOT::NTupleSize_t EntryToColumnElementIndex(ROOT::NTupleSize_t globalIndex) const;
 
    /// Flushes data from active columns

tree/ntuple/inc/ROOT/RNTuple.hxx

@@ -76,8 +76,8 @@ class RNTuple final {
 
 public:
    static constexpr std::uint16_t kVersionEpoch = 1;
-   static constexpr std::uint16_t kVersionMajor = 0;
-   static constexpr std::uint16_t kVersionMinor = 2;
+   static constexpr std::uint16_t kVersionMajor = 1;
+   static constexpr std::uint16_t kVersionMinor = 0;
    static constexpr std::uint16_t kVersionPatch = 0;
 
    /// Returns the RNTuple version in the following form:

tree/ntuple/inc/ROOT/RNTupleDescriptor.hxx

@@ -514,6 +514,12 @@ public:
    ROOT::NTupleSize_t GetFirstEntryIndex() const { return fFirstEntryIndex; }
    ROOT::NTupleSize_t GetNEntries() const { return fNEntries; }
    const RColumnRange &GetColumnRange(ROOT::DescriptorId_t physicalId) const { return fColumnRanges.at(physicalId); }
+   const RColumnRange *TryGetColumnRange(ROOT::DescriptorId_t physicalId) const
+   {
+      if (auto it = fColumnRanges.find(physicalId); it != fColumnRanges.end())
+         return &it->second;
+      return nullptr;
+   }
    const RPageRange &GetPageRange(ROOT::DescriptorId_t physicalId) const { return fPageRanges.at(physicalId); }
    /// Returns an iterator over pairs { columnId, columnRange }. The iteration order is unspecified.
    RColumnRangeIterable GetColumnRangeIterable() const;
@@ -768,6 +774,12 @@ public:
    /// All known feature flags.
    /// Note that the flag values represent the bit _index_, not the already-bitshifted integer.
    enum EFeatureFlags {
+      /// Signals that the RNTuple contains at least one deferred column that is part of a collection and was extended
+      /// (i.e. it appears in the footer). This can happen when merging two RNTuples that have the same collection field
+      /// backed by columns with different encoding, e.g. a vector<float> whose elements are represented by SplitReal32
+      /// in the first ntuple and by Real32 in the second.
+      /// Added in version 1.1.0.0 of the binary format.
+      kFeatureFlag_NestedDeferredColumns = 0,
       // Insert new feature flags here, with contiguous values. If at any point a "hole" appears in the valid feature
       // flags values, the check in RNTupleSerialize must be updated.
 

tree/ntuple/inc/ROOT/RNTupleMerger.hxx

@@ -117,16 +117,16 @@ class RNTupleMerger final {
    std::unique_ptr<ROOT::RNTupleModel> fModel;
 
    [[nodiscard]]
-   ROOT::RResult<void> MergeCommonColumns(ROOT::Internal::RClusterPool &clusterPool,
-                                          const ROOT::RClusterDescriptor &clusterDesc,
-                                          std::span<const RColumnMergeInfo> commonColumns,
-                                          const ROOT::Internal::RCluster::ColumnSet_t &commonColumnSet,
-                                          std::size_t nCommonColumnsInCluster, RSealedPageMergeData &sealedPageData,
-                                          const RNTupleMergeData &mergeData, ROOT::Internal::RPageAllocator &pageAlloc);
+   ROOT::RResult<void>
+   MergeCommonColumns(ROOT::Internal::RClusterPool &clusterPool, const ROOT::RClusterDescriptor &clusterDesc,
+                      std::span<RColumnMergeInfo> commonColumns,
+                      const ROOT::Internal::RCluster::ColumnSet_t &commonColumnSet, std::size_t nCommonColumnsInCluster,
+                      RSealedPageMergeData &sealedPageData, const RNTupleMergeData &mergeData,
+                      ROOT::Internal::RPageAllocator &pageAlloc);
 
    [[nodiscard]]
    ROOT::RResult<void>
-   MergeSourceClusters(ROOT::Internal::RPageSource &source, std::span<const RColumnMergeInfo> commonColumns,
+   MergeSourceClusters(ROOT::Internal::RPageSource &source, std::span<RColumnMergeInfo> commonColumns,
                        std::span<const RColumnMergeInfo> extraDstColumns, RNTupleMergeData &mergeData);
 
    /// Creates a RNTupleMerger with the given destination.

tree/ntuple/inc/ROOT/RPageStorage.hxx

@@ -543,6 +543,15 @@ public:
    [[nodiscard]] std::unique_ptr<RNTupleModel>
    InitFromDescriptor(const ROOT::RNTupleDescriptor &descriptor, bool copyClusters);
 
+   /// Adds a new column representation to the given field.
+   /// \return The physical id of the first newly added column.
+   ROOT::DescriptorId_t AddColumnRepresentation(const ROOT::RFieldDescriptor &field,
+                                                std::span<const ROOT::Internal::RColumnFormat> newRepresentation);
+
+   /// Adds a new alias column pointing to an existing column with the given physical id to the given field.
+   void AddAliasColumn(const ROOT::RNTupleDescriptor &desc, const ROOT::RFieldDescriptor &field,
+                       ROOT::DescriptorId_t physicalId);
+
    void CommitSuppressedColumn(ColumnHandle_t columnHandle) final;
    void CommitPage(ColumnHandle_t columnHandle, const ROOT::Internal::RPage &page) final;
    void CommitSealedPage(ROOT::DescriptorId_t physicalColumnId, const RPageStorage::RSealedPage &sealedPage) final;

tree/ntuple/src/RFieldBase.cxx

@@ -683,14 +683,14 @@ void ROOT::RFieldBase::Attach(std::unique_ptr<ROOT::RFieldBase> child, std::stri
 
 ROOT::NTupleSize_t ROOT::RFieldBase::EntryToColumnElementIndex(ROOT::NTupleSize_t globalIndex) const
 {
-   std::size_t result = globalIndex;
+   ROOT::NTupleSize_t result = globalIndex;
    for (auto f = this; f != nullptr; f = f->GetParent()) {
       auto parent = f->GetParent();
       if (parent && (parent->GetStructure() == ROOT::ENTupleStructure::kCollection ||
                      parent->GetStructure() == ROOT::ENTupleStructure::kVariant)) {
          return 0U;
       }
-      result *= std::max(f->GetNRepetitions(), std::size_t{1U});
+      result *= std::max<ROOT::NTupleSize_t>(f->GetNRepetitions(), ROOT::NTupleSize_t{1U});
    }
    return result;
 }
@@ -850,10 +850,7 @@ void ROOT::RFieldBase::SetColumnRepresentatives(const RColumnRepresentations::Se
       if (itRepresentative == std::end(validTypes))
          throw RException(R__FAIL("invalid column representative"));
 
-      // don't add a duplicate representation
-      if (std::find_if(fColumnRepresentatives.begin(), fColumnRepresentatives.end(),
-                       [&r](const auto &rep) { return r == rep.get(); }) == fColumnRepresentatives.end())
-         fColumnRepresentatives.emplace_back(*itRepresentative);
+      fColumnRepresentatives.emplace_back(*itRepresentative);
    }
 }
 

tree/ntuple/src/RNTupleDescriptor.cxx

@@ -931,8 +931,21 @@ ROOT::Internal::RClusterDescriptorBuilder::AddExtendedColumnRanges(const RNTuple
                // `ROOT::RFieldBase::EntryToColumnElementIndex()`, i.e. it is a principal column reachable from the
                // field zero excluding subfields of collection and variant fields.
                if (c.IsDeferredColumn()) {
-                  columnRange.SetFirstElementIndex(fCluster.GetFirstEntryIndex() * nRepetitions);
-                  columnRange.SetNElements(fCluster.GetNEntries() * nRepetitions);
+                  if (c.GetRepresentationIndex() == 0) {
+                     columnRange.SetFirstElementIndex(fCluster.GetFirstEntryIndex() * nRepetitions);
+                     columnRange.SetNElements(fCluster.GetNEntries() * nRepetitions);
+                  } else {
+                     // Deferred representations which are not the first cannot count on the number of elements being
+                     // equal to Entries * nRepetitions because they might have been added in a later cluster. But they
+                     // can rely on the first representation having the correct FirstElement/NElements (by definition
+                     // the first representation cannot be an "extended" one), therefore they can just copy the value
+                     // from it.
+                     const auto &field = desc.GetFieldDescriptor(fieldId);
+                     const auto firstReprColumnId = field.GetLogicalColumnIds()[c.GetIndex()];
+                     const auto &firstReprColumnRange = fCluster.fColumnRanges[firstReprColumnId];
+                     columnRange.SetFirstElementIndex(firstReprColumnRange.GetFirstElementIndex());
+                     columnRange.SetNElements(firstReprColumnRange.GetNElements());
+                  }
                   if (!columnRange.IsSuppressed()) {
                      auto &pageRange = fCluster.fPageRanges[physicalId];
                      pageRange.fPhysicalColumnId = physicalId;
@@ -1350,6 +1363,14 @@ void ROOT::Internal::RNTupleDescriptorBuilder::ShiftAliasColumns(std::uint32_t o
       R__ASSERT(fDescriptor.fColumnDescriptors.count(c.fLogicalColumnId) == 0);
       fDescriptor.fColumnDescriptors.emplace(c.fLogicalColumnId, std::move(c));
    }
+
+   // Patch up column ids in the header extension
+   if (auto &xHeader = fDescriptor.fHeaderExtension) {
+      for (auto &columnId : xHeader->fExtendedColumnRepresentations) {
+         if (columnId >= fDescriptor.GetNPhysicalColumns())
+            columnId += offset;
+      }
+   }
 }
 
 ROOT::RResult<void> ROOT::Internal::RNTupleDescriptorBuilder::AddCluster(RClusterDescriptor &&clusterDesc)