From 63b2344fda2a824ad190ea16e453999cf24cabbe Mon Sep 17 00:00:00 2001 From: Mx-Iris Date: Fri, 24 Jul 2026 10:08:37 +0800 Subject: [PATCH 01/77] docs: rename SymbolStore migration plan to NodeStore after upstream rename MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Upstream swift-demangling renamed SymbolStore/SymbolStoreBuilder to NodeStore/NodeStoreBuilder (the library's domain concept is the demangled Node tree; there is no Symbol abstraction in its API) and the branch to feature/node-store. Update the migration plan accordingly. The local SymbolIndexStore type keeps its name — it is the migration target, not part of the upstream rename. --- ...StoreMigrationPlan.md => NodeStoreMigrationPlan.md} | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) rename Documentations/Internal/{SymbolStoreMigrationPlan.md => NodeStoreMigrationPlan.md} (93%) diff --git a/Documentations/Internal/SymbolStoreMigrationPlan.md b/Documentations/Internal/NodeStoreMigrationPlan.md similarity index 93% rename from Documentations/Internal/SymbolStoreMigrationPlan.md rename to Documentations/Internal/NodeStoreMigrationPlan.md index 961e9b1a..b01378b9 100644 --- a/Documentations/Internal/SymbolStoreMigrationPlan.md +++ b/Documentations/Internal/NodeStoreMigrationPlan.md @@ -1,9 +1,9 @@ -# SymbolStore 迁移计划（SymbolIndexStore → arena 存储） +# NodeStore 迁移计划（SymbolIndexStore → arena 存储） - **状态**: Draft - **日期**: 2026-07-24 -- **前置**: swift-demangling `feature/symbol-store` 分支合入 `main`（本包以路径依赖解析 `../swift-demangling` 的 main） -- **上游依据**: swift-demangling `evolution/0001-symbol-store-arena.md`（Phase 1–3 已落地并验收） +- **前置**: swift-demangling `feature/node-store` 分支合入 `main`（本包以路径依赖解析 `../swift-demangling` 的 main） +- **上游依据**: swift-demangling `evolution/0001-node-store-arena.md`（Phase 1–3 已落地并验收） ## 背景与动机 @@ -14,7 +14,7 @@ RuntimeViewer 与 MachOSwiftSection 的内存主项在 `MachOSymbols/SymbolIndex 上游 swift-demangling 已交付（proposal 0001）： -- `SymbolStore` / `SymbolStoreBuilder`（12B/节点 arena、hash-consing、open-addressing intern 表、**cache-free** 批量 `demangle(_:)`——不再向全局 `NodeCache` 泄漏任何东西）； +- `NodeStore` / `NodeStoreBuilder`（12B/节点 arena、hash-consing、open-addressing intern 表、**cache-free** 批量 `demangle(_:)`——不再向全局 `NodeCache` 泄漏任何东西）； - `NodeReference`（16B 值句柄，O(1) `==`/`hash`）：`kind`/`text`/`index`/`children`/`Sequence`(preorder)/`first(of:)`/`identifier`/`textUTF8` 全部镜像 `Node`； - 零物化消费：`reference.print(using:)`（与 Node 路径逐字节一致）、`TypeDecoder.decodeMangledType(node: NodeReference)`； - 桥接消费：`mangleAsString(some DemanglingNode)`、`materialize()`（按索引 memo，保留 DAG 共享）； @@ -33,7 +33,7 @@ RuntimeViewer 与 MachOSwiftSection 的内存主项在 `MachOSymbols/SymbolIndex ### Stage 1 — `SymbolIndexStore.Storage` 核心迁移（主体工作） -1. `Storage` 每 MachO 持有一个冻结的 `symbolStore: SymbolStore`。 +1. `Storage` 每 MachO 持有一个冻结的 `nodeStore: NodeStore`。 2. `DemangledSymbol.demangledNode: Node` → `NodeReference`（包内 API；`@_spi(ForSymbolViewer)` 消费者 RuntimeViewer 需同步适配，见「影响面」）。 3. `demangledNodeBySymbol: [Symbol: Node]` → `[Symbol: NodeReference]`。 4. `MemberSymbols` 内层 `OrderedDictionary` 与 `opaqueTypeDescriptorSymbolByNode` 的键 → `NodeReference`：结构哈希 O(树) 变 O(1)（store 内索引相等 ⇔ 结构相等），构建与查询双收益。 From 08e4b4c5f848e6955b2c0043df2c9f44a2822a44 Mon Sep 17 00:00:00 2001 From: Mx-Iris Date: Fri, 24 Jul 2026 11:52:02 +0800 Subject: [PATCH 02/77] feat(MachOSymbols): migrate SymbolIndexStore to NodeStore-backed arena storage (Stages 1-2) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Storage now owns a frozen per-image NodeStore arena instead of leaking every demangled tree into the process-global NodeCache: - Build sweep demangles cache-free onto a transient tree, classifies on it, and interns into a NodeStoreBuilder; indexes are collected NodeIndex-keyed (PendingStorage) and converted to NodeReference form after freeze(). - All indexes key/carry NodeReference (16-byte handle) instead of Node. - Node-taking query APIs (memberSymbols(of:for:node:), opaqueTypeDescriptorSymbol(for:)) bridge via structurallyEquals bucket matching — the frozen store drops its intern tables. - Late symbols (outside the build sweep) demangle into per-symbol mini stores; demangledNodeReference(for:in:) is the new zero-materialization lookup, demangledNode(for:in:) keeps the Node contract via materialize(). - Consumers: matchers use demangleSymbolReference + OrderedSet visited sets; DefinitionBuilder/TypeDefinition lookups key by NodeReference (O(1) hash after hash-consing); renderer boundaries materialize. isGlobal/isAccessor/hasAccessor/accessorKind/isStoredVariable generalized over DemanglingNode. Measured on the SwiftUI image (debug, like-for-like vs main): NodeCache growth 19,345 leaves / 559,976 subtrees -> 0 / 0; release reclaims 236 MB of 302 MB (legacy 180 MB of 272 MB); NodeStore body 7 MB for 579,291 unique nodes; build 30.7s vs 36.2s under equal parallel load. Tests: enable MachOSymbolsTests target with SymbolIndexStoreFixtureTests (8 cases: cache-free invariant via leaf identity, byte-identical printing vs the Node pipeline, structurallyEquals bucket hits for every key, demangledNode/reference agreement, late-symbol mini-store fallback); add manually-run SymbolIndexStoreBaselineTests (Stage 0 metrics); make SharedCacheTests concurrency proof deterministic via semaphore barrier instead of wall-clock thresholds. Snapshot acceptance: 60/60 byte-identical against the main-verified baseline. --- AGENTS.md | 2 + .../Internal/NodeStoreMigrationPlan.md | 47 +++- Package.swift | 3 +- Sources/MachOSymbols/DemangledSymbol.swift | 6 +- Sources/MachOSymbols/SymbolIndexStore.swift | 261 ++++++++++++------ .../Definitions/DefinitionBuilder.swift | 44 +-- .../Definitions/ExtensionDefinition.swift | 8 +- .../Definitions/OverrideSymbolMatcher.swift | 7 +- .../Definitions/ProtocolDefinition.swift | 8 +- .../Definitions/TypeDefinition.swift | 6 +- Sources/SwiftDeclaration/Extensions.swift | 2 +- Sources/SwiftDump/Dumper/ClassDumper.swift | 18 +- Sources/SwiftDump/Dumper/EnumDumper.swift | 2 +- .../Dumper/ProtocolConformanceDumper.swift | 12 +- Sources/SwiftDump/Dumper/StructDumper.swift | 2 +- .../SwiftDeclarationIndexer.swift | 19 +- Sources/SwiftInspection/MetadataReader.swift | 8 + .../SwiftDeclarationPrinter+Headers.swift | 2 +- .../SymbolIndexStoreBaselineTests.swift | 84 ++++++ Tests/MachOCachesTests/SharedCacheTests.swift | 41 +-- .../SymbolIndexStoreFixtureTests.swift | 154 +++++++++++ .../TypeAttributeInferrerTests.swift | 8 +- Tests/SwiftDiffingTests/ABIDifferTests.swift | 12 +- .../ABIExtensionAttributionTests.swift | 8 +- .../SwiftPrintingTests/NodePrinterTests.swift | 4 +- 25 files changed, 579 insertions(+), 189 deletions(-) create mode 100644 Tests/IntegrationTests/MachOSymbols/SymbolIndexStoreBaselineTests.swift create mode 100644 Tests/MachOSymbolsTests/SymbolIndexStoreFixtureTests.swift diff --git a/AGENTS.md b/AGENTS.md index af77c02b..d95f96e9 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -206,6 +206,8 @@ let node = try demangleAsNode("$sSiD") // Returns Node tree let string = node.print(using: .default) // "Swift.Int" ``` +**Symbol indexing (NodeStore-backed)**: `SymbolIndexStore.Storage` no longer retains `Node` class trees. The build sweep demangles each symbol cache-free onto a transient tree (`@_spi(Internals) demangleAsNodeTransient`), classifies on that tree, and interns it into a per-image `NodeStoreBuilder`; after `freeze()` the indexes hold 16-byte `NodeReference` handles (`DemangledSymbol.demangledNode: NodeReference`). Nothing touches the global `NodeCache`, and dropping a `Storage` releases the whole per-image footprint. Query APIs taking an externally demangled `Node` (`memberSymbols(of:for:node:)`, `opaqueTypeDescriptorSymbol(for:)`) match keys via `NodeReference.structurallyEquals(_:)`; renderer boundaries (`demangleResolver.resolve`, `Definition` models) call `materialize()`. Late symbols outside the build sweep get per-symbol mini stores (`demangledNodeReference(for:)` / `MetadataReader.demangleSymbolReference(for:in:)`). See `Documentations/Internal/NodeStoreMigrationPlan.md`. + ## Test Environment Tests use `MACHO_SWIFT_SECTION_SILENT_TEST=1` to suppress verbose output. diff --git a/Documentations/Internal/NodeStoreMigrationPlan.md b/Documentations/Internal/NodeStoreMigrationPlan.md index b01378b9..258403e3 100644 --- a/Documentations/Internal/NodeStoreMigrationPlan.md +++ b/Documentations/Internal/NodeStoreMigrationPlan.md @@ -1,8 +1,10 @@ # NodeStore 迁移计划（SymbolIndexStore → arena 存储） -- **状态**: Draft +- **状态**: In Progress（Stage 0–2 已落地，见文末「实施记录」） - **日期**: 2026-07-24 -- **前置**: swift-demangling `feature/node-store` 分支合入 `main`（本包以路径依赖解析 `../swift-demangling` 的 main） +- **最后更新**: 2026-07-24 +- **分支**: `feature/node-store-migration`（worktree `.claude/worktrees/node-store-migration`，Demangling 经 `.claude/worktrees/swift-demangling` 符号链接解析到 swift-demangling 的 `feature/node-store` worktree） +- **前置**: swift-demangling `feature/node-store` 分支合入 `main`（本包以路径依赖解析 `../swift-demangling` 的 main）；开发期先经上述符号链接直连该分支 - **上游依据**: swift-demangling `evolution/0001-node-store-arena.md`（Phase 1–3 已落地并验收） ## 背景与动机 @@ -83,3 +85,44 @@ RuntimeViewer 与 MachOSwiftSection 的内存主项在 `MachOSymbols/SymbolIndex - 上游 `Remangler` 的无构造泛型引擎（上游已决策保持 Node 引擎 + 物化桥）； - store 序列化 / mmap 符号数据库（proposal 0001 Phase 4，另行立项）； - `NodeCache` 本身的行为变更（迁移后其增长压力自然消失）。 + +## 实施记录 + +### Stage 0 — 基线（2026-07-24，本机 SwiftUI image，debug 构建，`SymbolIndexStoreBaselineTests`） + +| 指标 | 旧管线（main @ 7f7fe48） | +|---|---| +| 构建耗时 | 28.6s（独占）/ 36.2s（并行负载下） | +| 构建期 `phys_footprint` 增量 | 266–272 MB | +| 释放 `Storage` + `malloc_zone_pressure_relief` 后 | 残留 ~92 MB（回收 180 MB / 272 MB） | +| `NodeCache` 增长 | +19,345 叶、+559,976 子树——**进程级永驻，跨镜像累积，无法随镜像淘汰回收** | +| 索引条目 | demangled 202,603；member 17,049；methodDescriptor 2,209；global 82；offset 表 170,919；opaque 2,115；typeInfo 4,191 | + +### Stage 1 + Stage 2（消费端）落地（2026-07-24，同口径复测） + +| 指标 | 迁移后（NodeStore） | 对比 | +|---|---|---| +| 构建耗时 | 30.7s（并行负载下） | **快于同负载旧管线 36.2s（-15%）**，独占口径 +8%（28.6 → 31.0s），远优于 <2× 预算 | +| 构建期 `phys_footprint` 增量 | 302 MB | +30 MB（pending→populate 转换期两套索引共存的瞬态峰值，Stage 3 可收） | +| 释放 `Storage` 后 | 残留 ~66 MB（回收 236 MB / 302 MB） | **稳态残留低 26 MB，且残留全为 malloc 未归还页——无任何逻辑驻留** | +| `NodeCache` 增长 | **0 叶、0 子树** | 泄漏归零；`Storage` 释放即整镜像回收 | +| `NodeStore` 本体 | 7 MB / 579,291 唯一节点（12.7 B/节点） | 对比旧版 interned class 树 ~12.9 MB + 全局缓存表 | +| 索引条目 | 与基线逐项一致 | 语义保真 | + +### 实施要点（与原方案的偏差） + +1. **分类跑在瞬态树上，而非 `NodeReference` 上**：`NodeStoreBuilder` 无读访问且 `freeze()` 后不可再 intern（typeNode wrapper 必须在构建期造），故构建循环为「`demangleAsNodeTransient`（`@_spi(Internals)` 新导出）→ 分类逻辑在瞬态 `Node` 树上原样运行 → `builder.intern` 入 arena」；索引先以 `NodeIndex` 形态收集（`PendingStorage`），`freeze()` 后一次性转换为 `NodeReference` 形态（`Storage.populate`）。分类代码（`processMemberSymbol` 族）几乎零改动。 +2. **查询 API 公共签名保持 `Node` 入参**：`memberSymbols(of:for:node:)` / `opaqueTypeDescriptorSymbol(for:)` 的实参来自 MetadataReader 的 canonical 树（Explore 调用点审计确认），键则是 store 内 `NodeReference`。上游新增 `NodeReference.structurallyEquals(_ node: Node)`（零物化跨表示结构相等，text 走字节比较 + String 兜底），查询在 name 桶内线性匹配（桶内键极少）。 +3. **迟到符号统一收敛为 `NodeReference`**：冻结 store 不可插入，迟到符号（build 扫描外，如 resilient witness 的显式 requirement symbol）经 per-symbol mini `NodeStoreBuilder` demangle+freeze，late cache 存 `[Symbol: NodeReference]`；`demangledNode(for:)` 保持 `Node?` 返回（materialize 桥），十余个下游调用点零改动，新增 `demangledNodeReference(for:)` / `MetadataReader.demangleSymbolReference` 供 matcher 零物化路径。 +4. **消费端（原 Stage 2 主体已一并落地）**：matchers（Override/Protocol/Extension/ProtocolConformance）切 `demangleSymbolReference` + `OrderedSet`（visited 集 O(1) 哈希）；`DefinitionBuilder` 的 dedup / methodDescriptor / vtable lookup 键换 `NodeReference`（hash-consing 令键比较 O(树) → O(1)）；renderer 边界（`demangleResolver.resolve`、`Definition` 模型 `node` 字段、`ExtensionName`）按计划物化。`isGlobal`/`isAccessor`/`hasAccessor`/`accessorKind`/`isStoredVariable` 泛型化到 `DemanglingNode`（`where Self: Sequence`）。 +5. **上游配套（swift-demangling `feature/node-store`）**：`@_spi(Internals) demangleAsNodeTransient`、`NodeReference.structurallyEquals(_:)`（+3 测试）、`NodeReference: CustomStringConvertible`（物化桥，调试用）、`isKind(of:)` / `children.second` 上收到 `DemanglingNode` 协议扩展（删除 `Node` 具体副本）。 +6. **已知残留（Stage 3 候选）**：构建瞬态峰值 +30 MB（pending/最终两套索引在转换期共存，可改为逐字典迁移消峰）；`symbolsByOffset` / `Symbol` 复本压缩即原 Stage 3 范围。 +7. **测试环境备注**：`SwiftInterfaceBuilderTests` / `SwiftDiffableInterfaceBuilderTests` / `XcodeMachOFileDumpTests` 依赖本机 Xcode fixture glob（`XcodeMachOFileName.swift:456`），在未改动的 main 上同样 fatal——预先存在的环境问题，与迁移无关，验收时以 `--skip` 排除。 + +### 验收与测试策略调整（2026-07-24） + +按用户决定，`IntegrationTests` 整体退出验收路径（其中依赖 `/Applications/Xcode-26.4.0.app` 硬编码路径的三个 suite 在本机因 Xcode 已升级至 26.5.0 而 glob 失败并 `fatalError` 崩进程——该问题在未改动的 main 上同样存在，属测试基建债，不在本迁移范围内修复）。验收改为「快照对比 + fixture 单元测试」： + +1. **快照以 main 为基准逐字节对比**：先在 main（`7f7fe48`，旧管线）上运行 `SymbolTestsCoreInterfaceSnapshotTests` + `SymbolTestsCoreDumpSnapshotTests`（60 个快照测试）确认已提交基准与 main 输出一致；再在迁移 worktree 上运行同一套快照测试——**60/60 逐字节一致**。fixture 为自建 `SymbolTestsCore.framework`（无外部 Xcode 依赖；worktree 经符号链接复用主检出 `Tests/Projects/SymbolTests/DerivedData` 的构建产物）。 +2. **启用 `MachOSymbolsTests` target**（Package.swift 中原已定义但被注释）并新增 `SymbolIndexStoreFixtureTests`（8 个用例）：build 管线 cache-free 不变量（叶身份断言——所有测试 target 共享单进程，全局 `NodeCache` 计数断言天然竞态，改用「transient demangle 两次得到结构相等但 `!==` 的叶实例」这一并发免疫口径；进程级零增长量测留在手动运行的 `SymbolIndexStoreBaselineTests`）、全符号零物化打印与 `demangleAsNode` 管线逐字节对齐、`memberSymbols(of:for:node:)` 对每个 `NodeReference` 键桶经 `structurallyEquals` 命中、`symbols(of:)`/`typeInfo`/`opaqueTypeDescriptorSymbol` 与 storage 桶一致、`demangledNode`/`demangledNodeReference` 互证、迟到符号 mini store 回退与缓存稳定性。 +3. **修复 `SharedCache` 并发时序 flake**：`concurrentCallsForDifferentKeysRunInParallel` 原以墙钟阈值断言并行（CPU 饱和即假失败），改为确定性并行证据——所有 build 经信号量互相等待进入闭包，若 resolve 对不同 key 串行（锁跨 build）则死锁，由宽松超时转为失败而非挂死。 diff --git a/Package.swift b/Package.swift index 5c692f6b..35f9bd66 100644 --- a/Package.swift +++ b/Package.swift @@ -776,6 +776,7 @@ extension Target { .target(.MachOSymbols), .target(.MachOTestingSupport), .target(.MachOFixtureSupport), + .product(.Demangling), ], swiftSettings: testSettings, ) @@ -1062,7 +1063,7 @@ let package = Package( .RegenerateBaselinesPlugin, // Testing -// .MachOSymbolsTests, + .MachOSymbolsTests, .MachOSwiftSectionTests, .MachOCachesTests, .SwiftInspectionTests, diff --git a/Sources/MachOSymbols/DemangledSymbol.swift b/Sources/MachOSymbols/DemangledSymbol.swift index 61e205da..9bb60312 100644 --- a/Sources/MachOSymbols/DemangledSymbol.swift +++ b/Sources/MachOSymbols/DemangledSymbol.swift @@ -4,9 +4,9 @@ import Demangling public struct DemangledSymbol: Sendable { public let symbol: Symbol - public let demangledNode: Node + public let demangledNode: NodeReference - public init(symbol: Symbol, demangledNode: Node) { + public init(symbol: Symbol, demangledNode: NodeReference) { self.symbol = symbol self.demangledNode = demangledNode } @@ -15,7 +15,7 @@ public struct DemangledSymbol: Sendable { return symbol[keyPath: keyPath] } - public subscript(dynamicMember keyPath: KeyPath) -> Value { + public subscript(dynamicMember keyPath: KeyPath) -> Value { return demangledNode[keyPath: keyPath] } } diff --git a/Sources/MachOSymbols/SymbolIndexStore.swift b/Sources/MachOSymbols/SymbolIndexStore.swift index d8a56890..0f2d9b27 100644 --- a/Sources/MachOSymbols/SymbolIndexStore.swift +++ b/Sources/MachOSymbols/SymbolIndexStore.swift @@ -2,7 +2,7 @@ import Foundation import FoundationToolbox import MachOKit import MachOKitExtensions -import Demangling +@_spi(Internals) import Demangling import OrderedCollections import Utilities import Dependencies @@ -132,15 +132,19 @@ public final class SymbolIndexStore: SharedCache, @unc typealias IndexedSymbol = DemangledSymbol typealias AllSymbols = [IndexedSymbol] typealias GlobalSymbols = [IndexedSymbol] - typealias MemberSymbols = OrderedDictionary> + typealias MemberSymbols = OrderedDictionary> typealias OpaqueTypeDescriptorSymbol = IndexedSymbol public final class Storage: @unchecked Sendable { + /// The frozen arena holding every demangled node of this image. + /// All `NodeReference` values vended by this storage point into it. + let nodeStore: NodeStore + private(set) var typeInfoByName: [String: TypeInfo] = [:] private(set) var globalSymbolsByKind: OrderedDictionary = [:] - private(set) var opaqueTypeDescriptorSymbolByNode: OrderedDictionary = [:] + private(set) var opaqueTypeDescriptorSymbolByNode: OrderedDictionary = [:] private(set) var memberSymbolsByKind: OrderedDictionary = [:] @@ -152,51 +156,104 @@ public final class SymbolIndexStore: SharedCache, @unc private(set) var symbolsByOffset: OrderedDictionary = [:] + private(set) var demangledNodeBySymbol: [Symbol: NodeReference] = [:] + + /// Symbols demangled after the store was frozen (rare path: lookups + /// for symbols that were not part of the build sweep). The frozen + /// arena cannot grow, so each late symbol gets a per-symbol mini + /// store; the volume is small and every consumer keeps receiving a + /// uniform `NodeReference`. @Mutex - private(set) var demangledNodeBySymbol: [Symbol: Node] = [:] + private(set) var lateDemangledNodeBySymbol: [Symbol: NodeReference] = [:] private(set) var thunkAttributeMembersByKindAndTypeName: [Node.Kind: [String: [ThunkAttributeMember]]] = [:] - fileprivate func appendSymbol(_ symbol: IndexedSymbol, for kind: Node.Kind) { - symbolsByKind[kind, default: []].append(symbol) + fileprivate init(nodeStore: NodeStore) { + self.nodeStore = nodeStore } - fileprivate func setOpaqueTypeDescriptorSymbol(_ symbol: OpaqueTypeDescriptorSymbol, for node: Node) { - opaqueTypeDescriptorSymbolByNode[node] = symbol + fileprivate func setLateDemangledNode(_ demangledNode: NodeReference?, for symbol: Symbol) { + lateDemangledNodeBySymbol[symbol] = demangledNode } - fileprivate func setDemangledNode(_ demangledNode: Node?, for symbol: Symbol) { - demangledNodeBySymbol[symbol] = demangledNode - } + /// One-shot population after `freeze()`: converts the build-time + /// `NodeIndex`-keyed scratch into `NodeReference`-based indexes. + fileprivate func populate(from pending: PendingStorage, symbolsByOffset: OrderedDictionary) { + func demangledSymbol(_ pendingSymbol: PendingDemangledSymbol) -> DemangledSymbol { + DemangledSymbol(symbol: pendingSymbol.symbol, demangledNode: nodeStore.reference(at: pendingSymbol.rootNodeIndex)) + } + func memberSymbols(_ pendingMemberSymbols: PendingStorage.MemberSymbols) -> MemberSymbols { + var converted: MemberSymbols = [:] + for (typeName, symbolsByTypeNodeIndex) in pendingMemberSymbols { + var convertedByTypeNode: OrderedDictionary = [:] + for (typeNodeIndex, pendingSymbols) in symbolsByTypeNodeIndex { + convertedByTypeNode[nodeStore.reference(at: typeNodeIndex)] = pendingSymbols.map(demangledSymbol) + } + converted[typeName] = convertedByTypeNode + } + return converted + } - fileprivate func setSymbolsByOffset(_ symbolsByOffset: OrderedDictionary) { + typeInfoByName = pending.typeInfoByName + globalSymbolsByKind = pending.globalSymbolsByKind.mapValues { $0.map(demangledSymbol) } + opaqueTypeDescriptorSymbolByNode = .init(uniqueKeysWithValues: pending.opaqueTypeDescriptorSymbolByNodeIndex.map { (nodeStore.reference(at: $0.key), demangledSymbol($0.value)) }) + memberSymbolsByKind = pending.memberSymbolsByKind.mapValues(memberSymbols) + methodDescriptorMemberSymbolsByKind = pending.methodDescriptorMemberSymbolsByKind.mapValues(memberSymbols) + protocolWitnessMemberSymbolsByKind = pending.protocolWitnessMemberSymbolsByKind.mapValues(memberSymbols) + symbolsByKind = pending.symbolsByKind.mapValues { $0.map(demangledSymbol) } + demangledNodeBySymbol = pending.demangledNodeIndexBySymbol.mapValues { nodeStore.reference(at: $0) } + thunkAttributeMembersByKindAndTypeName = pending.thunkAttributeMembersByKindAndTypeName self.symbolsByOffset = symbolsByOffset } + } + + /// A `(symbol, root node index)` pair collected while the builder is still + /// mutable; becomes a `DemangledSymbol` once the store is frozen. + fileprivate struct PendingDemangledSymbol: Sendable { + let symbol: Symbol + let rootNodeIndex: NodeStore.NodeIndex + } + + /// Build-time scratch mirroring `Storage`'s indexes with `NodeIndex` keys + /// and `PendingDemangledSymbol` entries. Lives only for the duration of + /// `buildStorageImpl`; converted via `Storage.populate(from:symbolsByOffset:)`. + fileprivate struct PendingStorage { + typealias MemberSymbols = OrderedDictionary> + + var typeInfoByName: [String: TypeInfo] = [:] + var globalSymbolsByKind: OrderedDictionary = [:] + var opaqueTypeDescriptorSymbolByNodeIndex: OrderedDictionary = [:] + var memberSymbolsByKind: OrderedDictionary = [:] + var methodDescriptorMemberSymbolsByKind: OrderedDictionary = [:] + var protocolWitnessMemberSymbolsByKind: OrderedDictionary = [:] + var symbolsByKind: OrderedDictionary = [:] + var demangledNodeIndexBySymbol: [Symbol: NodeStore.NodeIndex] = [:] + var thunkAttributeMembersByKindAndTypeName: [Node.Kind: [String: [ThunkAttributeMember]]] = [:] - fileprivate func setDemangledNodeBySymbol(_ demangledNodeBySymbol: [Symbol: Node]) { - self.demangledNodeBySymbol = demangledNodeBySymbol + mutating func appendSymbol(_ pendingSymbol: PendingDemangledSymbol, for kind: Node.Kind) { + symbolsByKind[kind, default: []].append(pendingSymbol) } - fileprivate func setMemberSymbols(for result: ProcessMemberSymbolResult) { - memberSymbolsByKind[result.memberKind, default: [:]][result.typeName, default: [:]][result.typeNode, default: []].append(result.indexedSymbol) + mutating func setMemberSymbols(for result: ProcessMemberSymbolResult) { + memberSymbolsByKind[result.memberKind, default: [:]][result.typeName, default: [:]][result.typeNodeIndex, default: []].append(result.pendingSymbol) typeInfoByName[result.typeName] = result.typeInfo } - fileprivate func setMethodDescriptorMemberSymbols(for result: ProcessMemberSymbolResult) { - methodDescriptorMemberSymbolsByKind[result.memberKind, default: [:]][result.typeName, default: [:]][result.typeNode, default: []].append(result.indexedSymbol) + mutating func setMethodDescriptorMemberSymbols(for result: ProcessMemberSymbolResult) { + methodDescriptorMemberSymbolsByKind[result.memberKind, default: [:]][result.typeName, default: [:]][result.typeNodeIndex, default: []].append(result.pendingSymbol) typeInfoByName[result.typeName] = result.typeInfo } - fileprivate func setProtocolWitnessMemberSymbols(for result: ProcessMemberSymbolResult) { - protocolWitnessMemberSymbolsByKind[result.memberKind, default: [:]][result.typeName, default: [:]][result.typeNode, default: []].append(result.indexedSymbol) + mutating func setProtocolWitnessMemberSymbols(for result: ProcessMemberSymbolResult) { + protocolWitnessMemberSymbolsByKind[result.memberKind, default: [:]][result.typeName, default: [:]][result.typeNodeIndex, default: []].append(result.pendingSymbol) typeInfoByName[result.typeName] = result.typeInfo } - fileprivate func setGlobalSymbols(for result: ProcessGlobalSymbolResult) { - globalSymbolsByKind[result.kind, default: []].append(result.indexedSymbol) + mutating func setGlobalSymbols(for result: ProcessGlobalSymbolResult) { + globalSymbolsByKind[result.kind, default: []].append(result.pendingSymbol) } - fileprivate func appendThunkAttributeMember(_ member: ThunkAttributeMember, forKind thunkKind: Node.Kind, typeName: String) { + mutating func appendThunkAttributeMember(_ member: ThunkAttributeMember, forKind thunkKind: Node.Kind, typeName: String) { thunkAttributeMembersByKindAndTypeName[thunkKind, default: [:]][typeName, default: []].append(member) } } @@ -215,7 +272,6 @@ public final class SymbolIndexStore: SharedCache, @unc for machO: MachO, progressContinuation: AsyncStream.Continuation? ) -> Storage? { - let storage = Storage() var cachedSymbols: Set = [] var symbolByName: OrderedDictionary = [:] var symbolsByOffset: OrderedDictionary = [:] @@ -242,15 +298,18 @@ public final class SymbolIndexStore: SharedCache, @unc } } - // Phase 1: Parallel demangling + // Single sequential sweep: demangle each symbol cache-free onto a + // transient tree, classify on that tree, and intern the result into + // the arena builder. Nothing touches the global `NodeCache` and no + // class trees outlive the loop iteration (NodeStore migration plan, + // Stage 1). The former concurrentMap pipeline kept every class tree + // alive simultaneously and leaked all of them into `NodeCache.shared`. let symbolArray = Array(symbolByName.values) let totalSymbolCount = symbolArray.count - let demangledNodes = symbolArray.concurrentMap { try? demangleAsNode($0.name) } - - // Phase 2: Sequential indexing - var demangledNodeBySymbol: [Symbol: Node] = [:] - demangledNodeBySymbol.reserveCapacity(totalSymbolCount) + var builder = NodeStoreBuilder() + var pending = PendingStorage() + pending.demangledNodeIndexBySymbol.reserveCapacity(totalSymbolCount) for symbolIndex in 0.., @unc } let symbol = symbolArray[symbolIndex] - guard let rootNode = demangledNodes[symbolIndex] else { continue } + guard let rootNode = try? demangleAsNodeTransient(symbol.name) else { continue } + let rootNodeIndex = builder.intern(rootNode) + let pendingSymbol = PendingDemangledSymbol(symbol: symbol, rootNodeIndex: rootNodeIndex) - demangledNodeBySymbol[symbol] = rootNode + pending.demangledNodeIndexBySymbol[symbol] = rootNodeIndex guard rootNode.isKind(of: .global), let node = rootNode.children.first else { continue } - storage.appendSymbol(DemangledSymbol(symbol: symbol, demangledNode: rootNode), for: node.kind) + pending.appendSymbol(pendingSymbol, for: node.kind) if node.kind == .objCAttribute || node.kind == .nonObjCAttribute { if let extracted = processThunkAttributeSymbol(thunkKind: node.kind, rootNode: rootNode) { - storage.appendThunkAttributeMember(extracted.member, forKind: node.kind, typeName: extracted.typeName) + pending.appendThunkAttributeMember(extracted.member, forKind: node.kind, typeName: extracted.typeName) } continue } if rootNode.isGlobal { if !symbol.isExternal { - if let result = processGlobalSymbol(symbol, node: node, rootNode: rootNode) { - storage.setGlobalSymbols(for: result) + if let result = processGlobalSymbol(pendingSymbol, node: node) { + pending.setGlobalSymbols(for: result) } } } else { if node.kind == .methodDescriptor, let firstChild = node.children.first { - if let result = processMemberSymbol(symbol, node: firstChild, rootNode: rootNode) { - storage.setMethodDescriptorMemberSymbols(for: result) + if let result = processMemberSymbol(pendingSymbol, node: firstChild, builder: &builder) { + pending.setMethodDescriptorMemberSymbols(for: result) } } else if node.kind == .protocolWitness, let firstChild = node.children.first { - if let result = processMemberSymbol(symbol, node: firstChild, rootNode: rootNode) { - storage.setProtocolWitnessMemberSymbols(for: result) + if let result = processMemberSymbol(pendingSymbol, node: firstChild, builder: &builder) { + pending.setProtocolWitnessMemberSymbols(for: result) } } else if node.kind == .mergedFunction, let secondChild = rootNode.children.second { - if let result = processMemberSymbol(symbol, node: secondChild, rootNode: rootNode) { - storage.setMemberSymbols(for: result) + if let result = processMemberSymbol(pendingSymbol, node: secondChild, builder: &builder) { + pending.setMemberSymbols(for: result) } } else if node.kind == .opaqueTypeDescriptor, let firstChild = node.children.first, firstChild.kind == .opaqueReturnTypeOf, let memberSymbol = firstChild.children.first { if symbol.offset > 0 { - storage.setOpaqueTypeDescriptorSymbol(DemangledSymbol(symbol: symbol, demangledNode: rootNode), for: memberSymbol) + pending.opaqueTypeDescriptorSymbolByNodeIndex[builder.intern(memberSymbol)] = pendingSymbol } } else { - if let result = processMemberSymbol(symbol, node: node, rootNode: rootNode) { - storage.setMemberSymbols(for: result) + if let result = processMemberSymbol(pendingSymbol, node: node, builder: &builder) { + pending.setMemberSymbols(for: result) } } } } progressContinuation?.yield(Progress(currentCount: totalSymbolCount, totalCount: totalSymbolCount)) - storage.setSymbolsByOffset(symbolsByOffset) - - storage.setDemangledNodeBySymbol(demangledNodeBySymbol) + let storage = Storage(nodeStore: builder.freeze()) + storage.populate(from: pending, symbolsByOffset: symbolsByOffset) return storage } @@ -315,21 +375,21 @@ public final class SymbolIndexStore: SharedCache, @unc fileprivate struct ProcessMemberSymbolResult: Sendable { let memberKind: MemberKind let typeName: String - let typeNode: Node + let typeNodeIndex: NodeStore.NodeIndex let typeInfo: TypeInfo - let indexedSymbol: IndexedSymbol + let pendingSymbol: PendingDemangledSymbol } - private func processMemberSymbol(_ symbol: Symbol, node: Node, rootNode: Node) -> ProcessMemberSymbolResult? { + private func processMemberSymbol(_ pendingSymbol: PendingDemangledSymbol, node: Node, builder: inout NodeStoreBuilder) -> ProcessMemberSymbolResult? { if node.kind == .static, let firstChild = node.children.first, firstChild.kind.isMember { - return processMemberSymbol(symbol, node: firstChild, rootNode: rootNode, traits: [.isStatic]) + return processMemberSymbol(pendingSymbol, node: firstChild, traits: [.isStatic], builder: &builder) } else if node.kind.isMember { - return processMemberSymbol(symbol, node: node, rootNode: rootNode, traits: []) + return processMemberSymbol(pendingSymbol, node: node, traits: [], builder: &builder) } return nil } - private func processMemberSymbol(_ symbol: Symbol, node: Node, rootNode: Node, traits: MemberKind.Traits) -> ProcessMemberSymbolResult? { + private func processMemberSymbol(_ pendingSymbol: PendingDemangledSymbol, node: Node, traits: MemberKind.Traits, builder: inout NodeStoreBuilder) -> ProcessMemberSymbolResult? { var traits = traits let node = node switch node.kind { @@ -339,27 +399,27 @@ public final class SymbolIndexStore: SharedCache, @unc traits.insert(.inExtension) first = type } - return processMemberSymbol(symbol, node: first, rootNode: rootNode, memberKind: .allocator(inExtension: traits.contains(.inExtension))) + return processMemberSymbol(pendingSymbol, node: first, memberKind: .allocator(inExtension: traits.contains(.inExtension)), builder: &builder) case .deallocator: guard let first = node.children.first else { return nil } - return processMemberSymbol(symbol, node: first, rootNode: rootNode, memberKind: .deallocator) + return processMemberSymbol(pendingSymbol, node: first, memberKind: .deallocator, builder: &builder) case .constructor: guard var first = node.children.first else { return nil } if first.kind == .extension, let type = first.children.at(1) { traits.insert(.inExtension) first = type } - return processMemberSymbol(symbol, node: first, rootNode: rootNode, memberKind: .constructor(inExtension: traits.contains(.inExtension))) + return processMemberSymbol(pendingSymbol, node: first, memberKind: .constructor(inExtension: traits.contains(.inExtension)), builder: &builder) case .destructor: guard let first = node.children.first else { return nil } - return processMemberSymbol(symbol, node: first, rootNode: rootNode, memberKind: .destructor) + return processMemberSymbol(pendingSymbol, node: first, memberKind: .destructor, builder: &builder) case .function: guard var first = node.children.first else { return nil } if first.kind == .extension, let type = first.children.at(1) { traits.insert(.inExtension) first = type } - return processMemberSymbol(symbol, node: first, rootNode: rootNode, memberKind: .function(inExtension: traits.contains(.inExtension), isStatic: traits.contains(.isStatic))) + return processMemberSymbol(pendingSymbol, node: first, memberKind: .function(inExtension: traits.contains(.inExtension), isStatic: traits.contains(.isStatic)), builder: &builder) case .variable: // Stored variable reached directly (not through getter/setter) traits.insert(.isStorage) @@ -369,7 +429,7 @@ public final class SymbolIndexStore: SharedCache, @unc first = type } if let first { - return processMemberSymbol(symbol, node: first, rootNode: rootNode, memberKind: .variable(inExtension: traits.contains(.inExtension), isStatic: traits.contains(.isStatic), isStorage: traits.contains(.isStorage))) + return processMemberSymbol(pendingSymbol, node: first, memberKind: .variable(inExtension: traits.contains(.inExtension), isStatic: traits.contains(.isStatic), isStorage: traits.contains(.isStorage)), builder: &builder) } case .getter, .setter: @@ -378,13 +438,13 @@ public final class SymbolIndexStore: SharedCache, @unc traits.insert(.inExtension) first = type } - return processMemberSymbol(symbol, node: first, rootNode: rootNode, memberKind: .variable(inExtension: traits.contains(.inExtension), isStatic: traits.contains(.isStatic), isStorage: traits.contains(.isStorage))) + return processMemberSymbol(pendingSymbol, node: first, memberKind: .variable(inExtension: traits.contains(.inExtension), isStatic: traits.contains(.isStatic), isStorage: traits.contains(.isStorage)), builder: &builder) } else if let subscriptNode = node.children.first, subscriptNode.kind == .subscript, var first = subscriptNode.children.first { if first.kind == .extension, let type = first.children.at(1) { traits.insert(.inExtension) first = type } - return processMemberSymbol(symbol, node: first, rootNode: rootNode, memberKind: .subscript(inExtension: traits.contains(.inExtension), isStatic: traits.contains(.isStatic))) + return processMemberSymbol(pendingSymbol, node: first, memberKind: .subscript(inExtension: traits.contains(.inExtension), isStatic: traits.contains(.isStatic)), builder: &builder) } default: break @@ -392,13 +452,14 @@ public final class SymbolIndexStore: SharedCache, @unc return nil } - private func processMemberSymbol(_ symbol: Symbol, node: Node, rootNode: Node, memberKind: MemberKind) -> ProcessMemberSymbolResult? { - let typeNode = Node.create(kind: .type, child: node) - let typeName = typeNode.print(using: .interfaceTypeBuilderOnly) + private func processMemberSymbol(_ pendingSymbol: PendingDemangledSymbol, node: Node, memberKind: MemberKind, builder: inout NodeStoreBuilder) -> ProcessMemberSymbolResult? { if let typeKind = node.kind.typeKind { -// typeInfoByName[typeName] = .init(name: typeName, kind: typeKind) -// storage[memberKind, default: [:]][typeName, default: [:]][typeNode, default: []].append(IndexedSymbol(DemangledSymbol(symbol: symbol, demangledNode: rootNode))) - return .init(memberKind: memberKind, typeName: typeName, typeNode: typeNode, typeInfo: .init(name: typeName, kind: typeKind), indexedSymbol: DemangledSymbol(symbol: symbol, demangledNode: rootNode)) + // The transient `.type` wrapper exists only for printing; the + // arena-resident wrapper is built directly from the interned + // context node's index, so no class tree survives this call. + let typeName = Node.create(kind: .type, child: node).print(using: .interfaceTypeBuilderOnly) + let typeNodeIndex = builder.intern(kind: .type, children: [builder.intern(node)]) + return .init(memberKind: memberKind, typeName: typeName, typeNodeIndex: typeNodeIndex, typeInfo: .init(name: typeName, kind: typeKind), pendingSymbol: pendingSymbol) } return nil } @@ -466,21 +527,21 @@ public final class SymbolIndexStore: SharedCache, @unc fileprivate struct ProcessGlobalSymbolResult: Sendable { let kind: GlobalKind - let indexedSymbol: IndexedSymbol + let pendingSymbol: PendingDemangledSymbol } - private func processGlobalSymbol(_ symbol: Symbol, node: Node, rootNode: Node) -> ProcessGlobalSymbolResult? { + private func processGlobalSymbol(_ pendingSymbol: PendingDemangledSymbol, node: Node) -> ProcessGlobalSymbolResult? { switch node.kind { case .function: - return .init(kind: .function, indexedSymbol: DemangledSymbol(symbol: symbol, demangledNode: rootNode)) + return .init(kind: .function, pendingSymbol: pendingSymbol) case .variable: // When we reach .variable directly (not through getter/setter), // this is a stored variable declaration - return .init(kind: .variable(isStorage: true), indexedSymbol: DemangledSymbol(symbol: symbol, demangledNode: rootNode)) + return .init(kind: .variable(isStorage: true), pendingSymbol: pendingSymbol) case .getter, .setter: if let variableNode = node.children.first, variableNode.kind == .variable { - return processGlobalSymbol(symbol, node: variableNode, rootNode: rootNode) + return processGlobalSymbol(pendingSymbol, node: variableNode) } default: break @@ -533,13 +594,20 @@ public final class SymbolIndexStore: SharedCache, @unc } public func memberSymbols(of kinds: MemberKind..., for name: String, node: Node, in machO: MachO) -> [DemangledSymbol] { - return kinds.map { storage(in: machO)?.memberSymbolsByKind[$0]?[name]?[node] ?? [] }.reduce(into: []) { $0 += $1 } + // Callers hold an externally demangled `Node` (MetadataReader context + // demangling), while keys are `NodeReference`s into the frozen store. + // The type-name bucket holds at most a handful of type nodes, so a + // structural walk per key is cheap. + return kinds.map { kind -> [DemangledSymbol] in + guard let symbolsByTypeNode = storage(in: machO)?.memberSymbolsByKind[kind]?[name] else { return [] } + return symbolsByTypeNode.elements.first(where: { $0.key.structurallyEquals(node) })?.value ?? [] + }.reduce(into: []) { $0 += $1 } } - public func memberSymbols(of kinds: MemberKind..., excluding names: borrowing Set, in machO: MachO) -> OrderedDictionary> { + public func memberSymbols(of kinds: MemberKind..., excluding names: borrowing Set, in machO: MachO) -> OrderedDictionary> { let filtered: OrderedDictionary = kinds.reduce(into: [:]) { $0[$1] = storage(in: machO)?.memberSymbolsByKind[$1]?.filter { !names.contains($0.key) } ?? [:] } - var result: OrderedDictionary> = [:] + var result: OrderedDictionary> = [:] for (kind, memberSymbols) in filtered { for (_, symbols) in memberSymbols { for (node, symbols) in symbols { @@ -570,16 +638,18 @@ public final class SymbolIndexStore: SharedCache, @unc return kinds.map { storage(in: machO)?.globalSymbolsByKind[$0] ?? [] }.reduce(into: []) { $0 += $1 } } - public func allOpaqueTypeDescriptorSymbols(in machO: MachO) -> OrderedDictionary? { + public func allOpaqueTypeDescriptorSymbols(in machO: MachO) -> OrderedDictionary? { return storage(in: machO)?.opaqueTypeDescriptorSymbolByNode.mapValues { return $0 } } public func opaqueTypeDescriptorSymbol(for node: Node, in machO: MachO) -> DemangledSymbol? { - return storage(in: machO)?.opaqueTypeDescriptorSymbolByNode[node].map { - return $0 - } + // The caller's `node` was demangled during printing; keys live in the + // frozen store. Structural comparison early-outs on the first + // mismatching kind, so the linear scan stays cheap relative to the + // printing work that triggers it. + return storage(in: machO)?.opaqueTypeDescriptorSymbolByNode.elements.first(where: { $0.key.structurallyEquals(node) })?.value } package func symbols(for offset: Int, in machO: MachO) -> Symbols? { @@ -590,16 +660,27 @@ public final class SymbolIndexStore: SharedCache, @unc } } - package func demangledNode(for symbol: Symbol, in machO: MachO) -> Node? { + /// Store-backed handle for a symbol's demangled tree. Hits the frozen + /// image store for symbols covered by the build sweep; symbols outside + /// the sweep are demangled cache-free into a per-symbol mini store, so + /// every caller receives a uniform `NodeReference`. + package func demangledNodeReference(for symbol: Symbol, in machO: MachO) -> NodeReference? { guard let cacheStorage = storage(in: machO) else { return nil } - if let node = cacheStorage.demangledNodeBySymbol[symbol] { - return node - } else if let node = try? demangleAsNode(symbol.name) { - cacheStorage.setDemangledNode(node, for: symbol) - return node - } else { - return nil + if let reference = cacheStorage.demangledNodeBySymbol[symbol] { + return reference + } + if let reference = cacheStorage.lateDemangledNodeBySymbol[symbol] { + return reference } + var lateBuilder = NodeStoreBuilder() + guard let nodeIndex = try? lateBuilder.demangle(symbol.name) else { return nil } + let reference = lateBuilder.freeze().reference(at: nodeIndex) + cacheStorage.setLateDemangledNode(reference, for: symbol) + return reference + } + + package func demangledNode(for symbol: Symbol, in machO: MachO) -> Node? { + return demangledNodeReference(for: symbol, in: machO)?.materialize() } public struct Progress: Sendable { @@ -678,7 +759,7 @@ extension DependencyValues { } } -extension Node { +extension DemanglingNode { package var isGlobal: Bool { guard let first = children.first else { return false } guard first.isKind(of: .getter, .setter, .function, .variable) else { return false } @@ -692,7 +773,9 @@ extension Node { package var isAccessor: Bool { return isKind(of: .getter, .setter, .modifyAccessor, .readAccessor) } +} +extension DemanglingNode where Self: Sequence { package var hasAccessor: Bool { return contains { $0.isAccessor } } diff --git a/Sources/SwiftDeclaration/Components/Definitions/DefinitionBuilder.swift b/Sources/SwiftDeclaration/Components/Definitions/DefinitionBuilder.swift index 662eb7c6..aa694474 100644 --- a/Sources/SwiftDeclaration/Components/Definitions/DefinitionBuilder.swift +++ b/Sources/SwiftDeclaration/Components/Definitions/DefinitionBuilder.swift @@ -7,8 +7,8 @@ package enum DefinitionBuilder { package static func variables( for demangledSymbols: [DemangledSymbolWithOffset], fieldNames: borrowing Set = [], - methodDescriptorLookup: [Node: MethodDescriptorWrapper] = [:], - vtableOffsetLookup: [Node: Int] = [:], + methodDescriptorLookup: [NodeReference: MethodDescriptorWrapper] = [:], + vtableOffsetLookup: [NodeReference: Int] = [:], implOffsetDescriptorLookup: [Int: MethodDescriptorWrapper] = [:], implOffsetVTableSlotLookup: [Int: Int] = [:], isGlobalOrStatic: Bool @@ -30,7 +30,7 @@ package enum DefinitionBuilder { guard !fieldNames.contains(name) else { continue } let nodes = accessors.map(\.symbol.demangledNode) guard let node = nodes.first(where: { $0.contains(.getter) || !$0.hasAccessor }) else { continue } - var variableDefinition = VariableDefinition(node: node, name: name, accessors: accessors, isGlobalOrStatic: isGlobalOrStatic) + var variableDefinition = VariableDefinition(node: node.materialize(), name: name, accessors: accessors, isGlobalOrStatic: isGlobalOrStatic) if accessors.contains(where: { $0.methodDescriptor?.method?.layout.flags.isDynamic ?? false }) { variableDefinition.attributes.append(.dynamic) } @@ -41,8 +41,8 @@ package enum DefinitionBuilder { package static func subscripts( for demangledSymbols: [DemangledSymbolWithOffset], - methodDescriptorLookup: [Node: MethodDescriptorWrapper] = [:], - vtableOffsetLookup: [Node: Int] = [:], + methodDescriptorLookup: [NodeReference: MethodDescriptorWrapper] = [:], + vtableOffsetLookup: [NodeReference: Int] = [:], implOffsetDescriptorLookup: [Int: MethodDescriptorWrapper] = [:], implOffsetVTableSlotLookup: [Int: Int] = [:], isStatic: Bool @@ -54,7 +54,7 @@ package enum DefinitionBuilder { // `Dictionary` iteration order is randomized per process and made the // interface output unstable across runs. Insertion order follows the // (deterministic) symbol order of `demangledSymbols`. - var accessorsByNode: OrderedDictionary = [:] + var accessorsByNode: OrderedDictionary = [:] for demangledSymbol in demangledSymbols { guard let subscriptNode = demangledSymbol.demangledNode.first(of: .subscript) else { continue } let kind = demangledSymbol.accessorKind @@ -68,7 +68,7 @@ package enum DefinitionBuilder { for (_, accessors) in accessorsByNode { let nodes = accessors.map(\.symbol.demangledNode) guard let node = nodes.first(where: { $0.contains(.getter) }) else { continue } - var subscriptDefinition = SubscriptDefinition(node: node, accessors: accessors, isStatic: isStatic) + var subscriptDefinition = SubscriptDefinition(node: node.materialize(), accessors: accessors, isStatic: isStatic) if accessors.contains(where: { $0.methodDescriptor?.method?.layout.flags.isDynamic ?? false }) { subscriptDefinition.attributes.append(.dynamic) } @@ -79,18 +79,18 @@ package enum DefinitionBuilder { package static func allocators( for demangledSymbols: [DemangledSymbolWithOffset], - methodDescriptorLookup: [Node: MethodDescriptorWrapper] = [:], - vtableOffsetLookup: [Node: Int] = [:], + methodDescriptorLookup: [NodeReference: MethodDescriptorWrapper] = [:], + vtableOffsetLookup: [NodeReference: Int] = [:], implOffsetDescriptorLookup: [Int: MethodDescriptorWrapper] = [:], implOffsetVTableSlotLookup: [Int: Int] = [:] ) -> [FunctionDefinition] { // Same dedup pattern as `functions(...)`: a merged-function thunk shares // the canonical `allocator` subtree, so the same init appears twice. Keep // the canonical (non-merged) entry when both are present. - var canonicalIndexByAllocatorNode: [Node: Int] = [:] + var canonicalIndexByAllocatorNode: [NodeReference: Int] = [:] // OrderedDictionary so the merged-thunk tail is appended in deterministic // (symbol) order — plain `Dictionary` iteration is randomized per process. - var pendingMergedByAllocatorNode: OrderedDictionary = [:] + var pendingMergedByAllocatorNode: OrderedDictionary = [:] var allocators: [FunctionDefinition] = [] for demangledSymbol in demangledSymbols { guard let allocatorNode = demangledSymbol.demangledNode.first(of: .allocator) else { continue } @@ -113,8 +113,8 @@ package enum DefinitionBuilder { private static func makeAllocatorDefinition( from demangledSymbol: DemangledSymbolWithOffset, - methodDescriptorLookup: [Node: MethodDescriptorWrapper], - vtableOffsetLookup: [Node: Int], + methodDescriptorLookup: [NodeReference: MethodDescriptorWrapper], + vtableOffsetLookup: [NodeReference: Int], implOffsetDescriptorLookup: [Int: MethodDescriptorWrapper], implOffsetVTableSlotLookup: [Int: Int] ) -> FunctionDefinition { @@ -122,7 +122,7 @@ package enum DefinitionBuilder { let symbolOffset = demangledSymbol.base.offset let descriptor = methodDescriptorLookup[node] ?? implOffsetDescriptorLookup[symbolOffset] let vtableOffset = vtableOffsetLookup[node] ?? implOffsetVTableSlotLookup[symbolOffset] - var functionDefinition = FunctionDefinition(node: node, name: "", kind: .allocator, symbol: demangledSymbol.base, isGlobalOrStatic: true, methodDescriptor: descriptor, offset: demangledSymbol.offset, vtableOffset: vtableOffset) + var functionDefinition = FunctionDefinition(node: node.materialize(), name: "", kind: .allocator, symbol: demangledSymbol.base, isGlobalOrStatic: true, methodDescriptor: descriptor, offset: demangledSymbol.offset, vtableOffset: vtableOffset) if let methodDescriptor = descriptor?.method, methodDescriptor.layout.flags.isDynamic { functionDefinition.attributes.append(.dynamic) } @@ -131,8 +131,8 @@ package enum DefinitionBuilder { package static func functions( for demangledSymbols: [DemangledSymbolWithOffset], - methodDescriptorLookup: [Node: MethodDescriptorWrapper] = [:], - vtableOffsetLookup: [Node: Int] = [:], + methodDescriptorLookup: [NodeReference: MethodDescriptorWrapper] = [:], + vtableOffsetLookup: [NodeReference: Int] = [:], implOffsetDescriptorLookup: [Int: MethodDescriptorWrapper] = [:], implOffsetVTableSlotLookup: [Int: Int] = [:], isGlobalOrStatic: Bool @@ -142,10 +142,10 @@ package enum DefinitionBuilder { // deduping, the same source-level declaration appears twice. Prefer the // canonical (non-merged) symbol when both exist; fall back to the merged // one when it's the only copy. - var canonicalIndexByFunctionNode: [Node: Int] = [:] + var canonicalIndexByFunctionNode: [NodeReference: Int] = [:] // OrderedDictionary so the merged-thunk tail is appended in deterministic // (symbol) order — plain `Dictionary` iteration is randomized per process. - var pendingMergedByFunctionNode: OrderedDictionary = [:] + var pendingMergedByFunctionNode: OrderedDictionary = [:] var functions: [FunctionDefinition] = [] for demangledSymbol in demangledSymbols { guard let functionNode = demangledSymbol.demangledNode.first(of: .function), let name = functionNode.identifier else { continue } @@ -171,8 +171,8 @@ package enum DefinitionBuilder { from demangledSymbol: DemangledSymbolWithOffset, name: String, isGlobalOrStatic: Bool, - methodDescriptorLookup: [Node: MethodDescriptorWrapper], - vtableOffsetLookup: [Node: Int], + methodDescriptorLookup: [NodeReference: MethodDescriptorWrapper], + vtableOffsetLookup: [NodeReference: Int], implOffsetDescriptorLookup: [Int: MethodDescriptorWrapper], implOffsetVTableSlotLookup: [Int: Int] ) -> FunctionDefinition { @@ -180,7 +180,7 @@ package enum DefinitionBuilder { let symbolOffset = demangledSymbol.base.offset let descriptor = methodDescriptorLookup[node] ?? implOffsetDescriptorLookup[symbolOffset] let vtableOffset = vtableOffsetLookup[node] ?? implOffsetVTableSlotLookup[symbolOffset] - var functionDefinition = FunctionDefinition(node: node, name: name, kind: .function, symbol: demangledSymbol.base, isGlobalOrStatic: isGlobalOrStatic, methodDescriptor: descriptor, offset: demangledSymbol.offset, vtableOffset: vtableOffset) + var functionDefinition = FunctionDefinition(node: node.materialize(), name: name, kind: .function, symbol: demangledSymbol.base, isGlobalOrStatic: isGlobalOrStatic, methodDescriptor: descriptor, offset: demangledSymbol.offset, vtableOffset: vtableOffset) if let methodDescriptor = descriptor?.method, methodDescriptor.layout.flags.isDynamic { functionDefinition.attributes.append(.dynamic) } @@ -188,7 +188,7 @@ package enum DefinitionBuilder { } } -extension Node { +extension DemanglingNode where Self: Sequence { var isStoredVariable: Bool { guard first(of: .variable) != nil else { return false } // A stored variable is one not wrapped in an accessor (getter/setter/etc.) diff --git a/Sources/SwiftDeclaration/Components/Definitions/ExtensionDefinition.swift b/Sources/SwiftDeclaration/Components/Definitions/ExtensionDefinition.swift index c17a426c..a89c8800 100644 --- a/Sources/SwiftDeclaration/Components/Definitions/ExtensionDefinition.swift +++ b/Sources/SwiftDeclaration/Components/Definitions/ExtensionDefinition.swift @@ -94,15 +94,15 @@ public final class ExtensionDefinition: Definition, MutableDefinition { guard let protocolConformance, !protocolConformance.resilientWitnesses.isEmpty else { return } - func _symbol(for symbols: Symbols, typeName: String, visitedNodes: borrowing OrderedSet = []) throws -> DemangledSymbol? { + func _symbol(for symbols: Symbols, typeName: String, visitedNodes: borrowing OrderedSet = []) throws -> DemangledSymbol? { for symbol in symbols { - if let node = try? MetadataReader.demangleSymbol(for: symbol, in: machO), let protocolConformanceNode = node.first(of: .protocolConformance), let symbolTypeName = protocolConformanceNode.children.first?.print(using: .interfaceTypeBuilderOnly), symbolTypeName == typeName || PrimitiveTypeMappingCache.shared.storage(in: machO)?.primitiveType(for: typeName) == symbolTypeName, !visitedNodes.contains(node) { + if let node = MetadataReader.demangleSymbolReference(for: symbol, in: machO), let protocolConformanceNode = node.first(of: .protocolConformance), let symbolTypeName = protocolConformanceNode.children.first?.print(using: .interfaceTypeBuilderOnly), symbolTypeName == typeName || PrimitiveTypeMappingCache.shared.storage(in: machO)?.primitiveType(for: typeName) == symbolTypeName, !visitedNodes.contains(node) { return .init(symbol: symbol, demangledNode: node) } } return nil } - var visitedNodes: OrderedSet = [] + var visitedNodes: OrderedSet = [] var memberSymbolsByKind: OrderedDictionary = [:] for resilientWitness in protocolConformance.resilientWitnesses { @@ -112,7 +112,7 @@ public final class ExtensionDefinition: Definition, MutableDefinition { } else if let requirement = try resilientWitness.requirement(in: machO) { switch requirement { case .symbol(let symbol): - if let demangledNode = try? MetadataReader.demangleSymbol(for: symbol, in: machO) { + if let demangledNode = MetadataReader.demangleSymbolReference(for: symbol, in: machO) { addSymbol(.init(.init(symbol: symbol, demangledNode: demangledNode)), memberSymbolsByKind: &memberSymbolsByKind, inExtension: true) } case .element(let element): diff --git a/Sources/SwiftDeclaration/Components/Definitions/OverrideSymbolMatcher.swift b/Sources/SwiftDeclaration/Components/Definitions/OverrideSymbolMatcher.swift index 666a52ac..9780241b 100644 --- a/Sources/SwiftDeclaration/Components/Definitions/OverrideSymbolMatcher.swift +++ b/Sources/SwiftDeclaration/Components/Definitions/OverrideSymbolMatcher.swift @@ -15,13 +15,14 @@ import OrderedCollections package func demangledOverrideSymbol( for symbols: Symbols, typeNode: Node, - visitedNodes: borrowing OrderedSet = [], + visitedNodes: borrowing OrderedSet = [], in machO: MachO ) -> DemangledSymbol? { + guard let typeClassNode = typeNode.first(of: .class) else { return nil } for symbol in symbols { - if let node = try? MetadataReader.demangleSymbol(for: symbol, in: machO), + if let node = SymbolIndexStore.shared.demangledNodeReference(for: symbol, in: machO), let classNode = node.first(of: .class), - classNode == typeNode.first(of: .class), + classNode.structurallyEquals(typeClassNode), !visitedNodes.contains(node) { return .init(symbol: symbol, demangledNode: node) } diff --git a/Sources/SwiftDeclaration/Components/Definitions/ProtocolDefinition.swift b/Sources/SwiftDeclaration/Components/Definitions/ProtocolDefinition.swift index f8603d06..0893d7f6 100644 --- a/Sources/SwiftDeclaration/Components/Definitions/ProtocolDefinition.swift +++ b/Sources/SwiftDeclaration/Components/Definitions/ProtocolDefinition.swift @@ -130,9 +130,9 @@ public final class ProtocolDefinition: Definition, MutableDefinition { package func index(in machO: MachO) async throws { guard !isIndexed else { return } let name = protocolName.name - func _symbol(for symbols: Symbols, visitedNodes: borrowing OrderedSet = []) throws -> DemangledSymbol? { + func _symbol(for symbols: Symbols, visitedNodes: borrowing OrderedSet = []) throws -> DemangledSymbol? { for symbol in symbols { - if let node = try? MetadataReader.demangleSymbol(for: symbol, in: machO), let protocolNode = node.first(of: .protocol), protocolNode.print(using: .interfaceTypeBuilderOnly) == name, !visitedNodes.contains(node) { + if let node = MetadataReader.demangleSymbolReference(for: symbol, in: machO), let protocolNode = node.first(of: .protocol), protocolNode.print(using: .interfaceTypeBuilderOnly) == name, !visitedNodes.contains(node) { return .init(symbol: symbol, demangledNode: node) } } @@ -143,8 +143,8 @@ public final class ProtocolDefinition: Definition, MutableDefinition { var requirementMemberSymbolsByKind: OrderedDictionary = [:] var defaultImplementationMemberSymbolsByKind: OrderedDictionary = [:] - var requirementVisitedNodes: OrderedSet = [] - var defaultImplementationVisitedNodes: OrderedSet = [] + var requirementVisitedNodes: OrderedSet = [] + var defaultImplementationVisitedNodes: OrderedSet = [] var offsetOfPWT = 0 diff --git a/Sources/SwiftDeclaration/Components/Definitions/TypeDefinition.swift b/Sources/SwiftDeclaration/Components/Definitions/TypeDefinition.swift index 006a76ef..40394544 100644 --- a/Sources/SwiftDeclaration/Components/Definitions/TypeDefinition.swift +++ b/Sources/SwiftDeclaration/Components/Definitions/TypeDefinition.swift @@ -194,13 +194,13 @@ public final class TypeDefinition: Definition { let fieldNames = Set(fields.map(\.name)) - var methodDescriptorLookup: [Node: MethodDescriptorWrapper] = [:] - var vtableOffsetLookup: [Node: Int] = [:] + var methodDescriptorLookup: [NodeReference: MethodDescriptorWrapper] = [:] + var vtableOffsetLookup: [NodeReference: Int] = [:] // Fallback lookups keyed by implementation file offset (for methods where node-based matching fails) var implOffsetDescriptorLookup: [Int: MethodDescriptorWrapper] = [:] var implOffsetVTableSlotLookup: [Int: Int] = [:] if case .class(let cls) = type { - var visitedNodes: OrderedSet = [] + var visitedNodes: OrderedSet = [] let typeNode = try MetadataReader.demangleContext(for: .type(.class(cls.descriptor)), in: machO) let vtableBaseOffset = cls.vTableDescriptorHeader.map { Int($0.layout.vTableOffset) } diff --git a/Sources/SwiftDeclaration/Extensions.swift b/Sources/SwiftDeclaration/Extensions.swift index a6ea7682..391f3e1b 100644 --- a/Sources/SwiftDeclaration/Extensions.swift +++ b/Sources/SwiftDeclaration/Extensions.swift @@ -10,7 +10,7 @@ import SwiftStdlibToolbox @_spi(Internals) import MachOSymbols @_spi(Internals) import SwiftInspection -extension Node { +extension DemanglingNode where Self: Sequence { package var accessorKind: AccessorKind { guard let node = first(of: .getter, .setter, .modifyAccessor, .readAccessor) else { return .none } switch node.kind { diff --git a/Sources/SwiftDump/Dumper/ClassDumper.swift b/Sources/SwiftDump/Dumper/ClassDumper.swift index da71ecc7..3e84df12 100644 --- a/Sources/SwiftDump/Dumper/ClassDumper.swift +++ b/Sources/SwiftDump/Dumper/ClassDumper.swift @@ -87,9 +87,9 @@ package struct ClassDumper: TypedDumper { let rootNode = thunkSymbol.demangledNode guard let functionNode = rootNode.children.first(where: { $0.kind != .distributedThunk }) else { continue } guard let contextNode = functionNode.children.first else { continue } - let thunkTypeName = Node.create(kind: .type, child: contextNode).print(using: .interfaceTypeBuilderOnly) + let thunkTypeName = Node.create(kind: .type, child: contextNode.materialize()).print(using: .interfaceTypeBuilderOnly) guard thunkTypeName == currentTypeName else { continue } - nodes.insert(functionNode) + nodes.insert(functionNode.materialize()) } return nodes @@ -334,7 +334,7 @@ package struct ClassDumper: TypedDumper { Indent(level: 1) - try await demangleResolver.resolve(for: symbol.demangledNode) + try await demangleResolver.resolve(for: symbol.demangledNode.materialize()) if offset.isEnd { BreakLine() @@ -360,7 +360,7 @@ package struct ClassDumper: TypedDumper { Indent(level: 1) - try await demangleResolver.resolve(for: symbol.demangledNode) + try await demangleResolver.resolve(for: symbol.demangledNode.materialize()) if offset.isEnd { BreakLine() @@ -460,14 +460,4 @@ package struct ClassDumper: TypedDumper { return nil } - - package static func demangledSymbol(for symbols: Symbols, typeNode: Node, visitedNodes: borrowing OrderedSet = [], in machO: MachO) -> DemangledSymbol? { - for symbol in symbols { - if let node = try? MetadataReader.demangleSymbol(for: symbol, in: machO), let classNode = node.first(of: .class), classNode == typeNode.first(of: .class), !visitedNodes.contains(node) { - return .init(symbol: symbol, demangledNode: node) - } - } - return nil - } - } diff --git a/Sources/SwiftDump/Dumper/EnumDumper.swift b/Sources/SwiftDump/Dumper/EnumDumper.swift index 69ee8702..ffdd8d7c 100644 --- a/Sources/SwiftDump/Dumper/EnumDumper.swift +++ b/Sources/SwiftDump/Dumper/EnumDumper.swift @@ -159,7 +159,7 @@ package struct EnumDumper: TypedDumper { Indent(level: 1) - try await demangleResolver.resolve(for: symbol.demangledNode) + try await demangleResolver.resolve(for: symbol.demangledNode.materialize()) if offset.isEnd { BreakLine() diff --git a/Sources/SwiftDump/Dumper/ProtocolConformanceDumper.swift b/Sources/SwiftDump/Dumper/ProtocolConformanceDumper.swift index 6b9a5d81..14aa61c4 100644 --- a/Sources/SwiftDump/Dumper/ProtocolConformanceDumper.swift +++ b/Sources/SwiftDump/Dumper/ProtocolConformanceDumper.swift @@ -92,7 +92,7 @@ package struct ProtocolConformanceDumper: Conforme Space() Standard("{") - var visitedNodes: OrderedSet = [] + var visitedNodes: OrderedSet = [] for resilientWitness in dumped.resilientWitnesses { BreakLine() @@ -105,7 +105,7 @@ package struct ProtocolConformanceDumper: Conforme if let symbols = try resilientWitness.implementationSymbols(in: machO), let node = Self.demangledSymbol(for: symbols, typeName: typeNameString, visitedNodes: visitedNodes, in: machO)?.demangledNode { _ = visitedNodes.append(node) - try await demangleResolver.resolve(for: node) + try await demangleResolver.resolve(for: node.materialize()) } else if let requirement = try resilientWitness.requirement(in: machO) { switch requirement { @@ -114,10 +114,10 @@ package struct ProtocolConformanceDumper: Conforme case .element(let element): if let symbols = try await Symbols.resolve(from: element.offset, in: machO), let node = Self.demangledSymbol(for: symbols, typeName: typeNameString, visitedNodes: visitedNodes, in: machO)?.demangledNode { _ = visitedNodes.append(node) - try await demangleResolver.resolve(for: node) + try await demangleResolver.resolve(for: node.materialize()) } else if let defaultImplementationSymbols = try element.defaultImplementationSymbols(in: machO), let node = Self.demangledSymbol(for: defaultImplementationSymbols, typeName: typeNameString, visitedNodes: visitedNodes, in: machO)?.demangledNode { _ = visitedNodes.append(node) - try await demangleResolver.resolve(for: node) + try await demangleResolver.resolve(for: node.materialize()) } else if !element.defaultImplementation.isNull { FunctionDeclaration(machO.addressString(forOffset: element.defaultImplementation.resolveDirectOffset(from: element.offset(of: \.defaultImplementation))).insertSubFunctionPrefix) } else if !resilientWitness.implementation.isNull { @@ -180,9 +180,9 @@ package struct ProtocolConformanceDumper: Conforme return nil } - package static func demangledSymbol(for symbols: Symbols, typeName: String, visitedNodes: borrowing OrderedSet = [], in machO: MachO) -> DemangledSymbol? { + package static func demangledSymbol(for symbols: Symbols, typeName: String, visitedNodes: borrowing OrderedSet = [], in machO: MachO) -> DemangledSymbol? { for symbol in symbols { - if let node = try? MetadataReader.demangleSymbol(for: symbol, in: machO), let targetNode = node.first(of: .protocolConformance), let symbolTypeName = targetNode.children.at(0)?.print(using: .interfaceType), symbolTypeName == typeName || PrimitiveTypeMappingCache.shared.storage(in: machO)?.primitiveType(for: typeName) == symbolTypeName, !visitedNodes.contains(node) { + if let node = MetadataReader.demangleSymbolReference(for: symbol, in: machO), let targetNode = node.first(of: .protocolConformance), let symbolTypeName = targetNode.children.at(0)?.print(using: .interfaceType), symbolTypeName == typeName || PrimitiveTypeMappingCache.shared.storage(in: machO)?.primitiveType(for: typeName) == symbolTypeName, !visitedNodes.contains(node) { return .init(symbol: symbol, demangledNode: node) } } diff --git a/Sources/SwiftDump/Dumper/StructDumper.swift b/Sources/SwiftDump/Dumper/StructDumper.swift index ff100f5d..004775b7 100644 --- a/Sources/SwiftDump/Dumper/StructDumper.swift +++ b/Sources/SwiftDump/Dumper/StructDumper.swift @@ -146,7 +146,7 @@ package struct StructDumper: TypedDumper { Indent(level: 1) - try await demangleResolver.resolve(for: symbol.demangledNode) + try await demangleResolver.resolve(for: symbol.demangledNode.materialize()) if offset.isEnd { BreakLine() diff --git a/Sources/SwiftIndexing/SwiftDeclarationIndexer.swift b/Sources/SwiftIndexing/SwiftDeclarationIndexer.swift index 980f07cf..f55047e5 100644 --- a/Sources/SwiftIndexing/SwiftDeclarationIndexer.swift +++ b/Sources/SwiftIndexing/SwiftDeclarationIndexer.swift @@ -656,14 +656,15 @@ public final class SwiftDeclarationIndexer, genericSignature: Node?) throws -> ExtensionDefinition { - let extensionDefinition = try ExtensionDefinition(extensionName: .init(node: node, kind: kind), genericSignature: genericSignature, protocolConformance: nil, in: machO) + let extensionDefinition = try ExtensionDefinition(extensionName: .init(node: materializedExtensionTargetNode, kind: kind), genericSignature: genericSignature, protocolConformance: nil, in: machO) var memberCount = 0 for (kind, memberSymbols) in memberSymbolsByKind { @@ -707,7 +708,7 @@ public final class SwiftDeclarationIndexer> = [:] + var memberSymbolsByGenericSignature: OrderedDictionary> = [:] var memberSymbolsByKind: OrderedDictionary = [:] for (kind, memberSymbols) in memberSymbols { @@ -722,10 +723,10 @@ public final class SwiftDeclarationIndexer(for symbol: Symbol, in machO: MachO) -> NodeReference? { + return SymbolIndexStore.shared.demangledNodeReference(for: symbol, in: machO) + } + public static func demangleContext(for context: ContextDescriptorWrapper, in machO: MachO) throws -> Node { if isCacheEnabled { return try MetadataReaderCache.shared.demangleContext(for: context, in: machO) diff --git a/Sources/SwiftPrinting/SwiftDeclarationPrinter+Headers.swift b/Sources/SwiftPrinting/SwiftDeclarationPrinter+Headers.swift index f4a437b8..55b28642 100644 --- a/Sources/SwiftPrinting/SwiftDeclarationPrinter+Headers.swift +++ b/Sources/SwiftPrinting/SwiftDeclarationPrinter+Headers.swift @@ -163,7 +163,7 @@ extension SwiftDeclarationPrinter { let rootNode = thunkSymbol.demangledNode guard let functionNode = rootNode.children.first(where: { $0.kind != .distributedThunk }) else { continue } guard let contextNode = functionNode.children.first else { continue } - let thunkTypeName = Node.create(kind: .type, child: contextNode).print(using: .interfaceTypeBuilderOnly) + let thunkTypeName = Node.create(kind: .type, child: contextNode.materialize()).print(using: .interfaceTypeBuilderOnly) if thunkTypeName == currentTypeName { return true } diff --git a/Tests/IntegrationTests/MachOSymbols/SymbolIndexStoreBaselineTests.swift b/Tests/IntegrationTests/MachOSymbols/SymbolIndexStoreBaselineTests.swift new file mode 100644 index 00000000..742453a6 --- /dev/null +++ b/Tests/IntegrationTests/MachOSymbols/SymbolIndexStoreBaselineTests.swift @@ -0,0 +1,84 @@ +import Foundation +import Testing +import MachO +import Demangling +@_spi(Internals) @testable import MachOSymbols +@testable import MachOTestingSupport +import MachOFixtureSupport + +/// Stage 0 of the NodeStore migration plan: capture per-image baseline +/// metrics for `SymbolIndexStore.buildStorage` so Stage 1/4 can compare +/// like-for-like. Run this suite alone (`--filter SymbolIndexStoreBaselineTests`) +/// so no other test warms the global `NodeCache` or demangles symbols first. +@Suite +final class SymbolIndexStoreBaselineTests: MachOImageTests { + override class var imageName: MachOImageName { + .SwiftUI + } + + @Test func baselineMetrics() async throws { + let footprintBefore = ProcessMemory.value(of: .physicalFootprint) + let leafCacheCountBefore = NodeCache.shared.count + let subtreeCacheCountBefore = NodeCache.shared.subtreeCount + + let clock = ContinuousClock() + var builtStorage: SymbolIndexStore.Storage? + let buildDuration = clock.measure { + builtStorage = SymbolIndexStore.shared.buildStorage(for: machOImage) + } + let footprintAfter = ProcessMemory.value(of: .physicalFootprint) + let leafCacheCountAfter = NodeCache.shared.count + let subtreeCacheCountAfter = NodeCache.shared.subtreeCount + + // Scope the strong reference so the release measurement below really + // drops the storage. + do { + let storage = try #require(builtStorage) + + let demangledSymbolCount = storage.demangledNodeBySymbol.count + let symbolsByKindEntryCount = storage.symbolsByKind.values.reduce(0) { $0 + $1.count } + let memberEntryCount = storage.memberSymbolsByKind.values.reduce(0) { partialResult, memberSymbols in + partialResult + memberSymbols.values.reduce(0) { $0 + $1.values.reduce(0) { $0 + $1.count } } + } + let methodDescriptorEntryCount = storage.methodDescriptorMemberSymbolsByKind.values.reduce(0) { partialResult, memberSymbols in + partialResult + memberSymbols.values.reduce(0) { $0 + $1.values.reduce(0) { $0 + $1.count } } + } + let protocolWitnessEntryCount = storage.protocolWitnessMemberSymbolsByKind.values.reduce(0) { partialResult, memberSymbols in + partialResult + memberSymbols.values.reduce(0) { $0 + $1.values.reduce(0) { $0 + $1.count } } + } + let globalEntryCount = storage.globalSymbolsByKind.values.reduce(0) { $0 + $1.count } + + let nodeStoreBytes = storage.nodeStore.storageByteCount + let nodeStoreNodeCount = storage.nodeStore.nodeCount + + print("====== NodeStore migration Stage 0 baseline (\(Self.imageName)) ======") + print("build time : \(buildDuration)") + print("phys_footprint delta : \((footprintAfter - footprintBefore) / 1_048_576) MB (\(footprintBefore / 1_048_576) -> \(footprintAfter / 1_048_576))") + print("NodeCache leaf delta : \(leafCacheCountAfter - leafCacheCountBefore) (\(leafCacheCountBefore) -> \(leafCacheCountAfter))") + print("NodeCache subtree delta : \(subtreeCacheCountAfter - subtreeCacheCountBefore) (\(subtreeCacheCountBefore) -> \(subtreeCacheCountAfter))") + print("nodeStore storage : \(nodeStoreBytes / 1_048_576) MB (\(nodeStoreNodeCount) unique nodes)") + print("demangledNodeBySymbol entries : \(demangledSymbolCount)") + print("symbolsByKind entries : \(symbolsByKindEntryCount)") + print("memberSymbols entries : \(memberEntryCount)") + print("methodDescriptorMember entries : \(methodDescriptorEntryCount)") + print("protocolWitnessMember entries : \(protocolWitnessEntryCount)") + print("globalSymbols entries : \(globalEntryCount)") + print("symbolsByOffset entries : \(storage.symbolsByOffset.count)") + print("opaqueTypeDescriptor entries : \(storage.opaqueTypeDescriptorSymbolByNode.count)") + print("typeInfoByName entries : \(storage.typeInfoByName.count)") + print("=====================================================================") + + #expect(demangledSymbolCount > 0) + } + + // Reclaim check: the migration's headline property is that dropping a + // Storage releases the whole per-image footprint (the old pipeline + // pinned every canonical subtree in the process-global NodeCache + // forever). Measure how far the footprint falls once the storage is + // gone and the allocator is asked to return clean pages. + builtStorage = nil + malloc_zone_pressure_relief(nil, 0) + let footprintAfterRelease = ProcessMemory.value(of: .physicalFootprint) + print("phys_footprint after release : \(footprintAfterRelease / 1_048_576) MB (reclaimed \((footprintAfter - min(footprintAfter, footprintAfterRelease)) / 1_048_576) MB of \((footprintAfter - footprintBefore) / 1_048_576) MB delta)") + } +} diff --git a/Tests/MachOCachesTests/SharedCacheTests.swift b/Tests/MachOCachesTests/SharedCacheTests.swift index db8a0016..dfeffcce 100644 --- a/Tests/MachOCachesTests/SharedCacheTests.swift +++ b/Tests/MachOCachesTests/SharedCacheTests.swift @@ -92,32 +92,39 @@ struct SharedCacheResolveTests { @Test func concurrentCallsForDifferentKeysRunInParallel() { let cache = TestCache() let keyCount = 8 - let perBuildSeconds: Double = 0.20 - let allDone = DispatchSemaphore(value: 0) - let start = ContinuousClock.now + // Deterministic parallelism proof instead of a wall-clock heuristic + // (which flaked under CPU saturation): every build blocks until all + // `keyCount` builds have entered their closure. If `resolve` + // serialized distinct keys — e.g. by holding the cache lock across + // the build — the first build would wait forever for peers that can + // never start, and the generous timeout below turns that into a + // failure rather than a hang. + let enteredBuild = DispatchSemaphore(value: 0) + let proceedWithBuild = DispatchSemaphore(value: 0) + let buildFinished = DispatchSemaphore(value: 0) + + DispatchQueue.global().async { + for _ in 0 ..< keyCount { enteredBuild.wait() } + for _ in 0 ..< keyCount { proceedWithBuild.signal() } + } + for index in 0 ..< keyCount { DispatchQueue.global().async { _ = cache.resolve(key: AnyHashable(index)) { - Thread.sleep(forTimeInterval: perBuildSeconds) + enteredBuild.signal() + proceedWithBuild.wait() return index } - allDone.signal() + buildFinished.signal() } } - for _ in 0 ..< keyCount { allDone.wait() } - let elapsed = (ContinuousClock.now - start) - let elapsedSeconds = Double(elapsed.components.seconds) - + Double(elapsed.components.attoseconds) / 1e18 - // Allow generous slack for CI variance — the contract is "wall-clock - // is closer to one build than to N builds", not a precise multiplier. - // Serial would be ~keyCount * perBuildSeconds; we cap at half of - // that, which is still >2× the parallel ideal. - let serialCeiling = Double(keyCount) * perBuildSeconds - let parallelBudget = serialCeiling * 0.5 - #expect(elapsedSeconds < parallelBudget, - "elapsed=\(elapsedSeconds)s should be well below serial=\(serialCeiling)s") + var timedOut = false + for _ in 0 ..< keyCount where !timedOut { + timedOut = buildFinished.wait(timeout: .now() + 30) == .timedOut + } + #expect(!timedOut, "builds for distinct keys did not run concurrently") } /// Cache hits stay reentrant: a build for key A may itself call diff --git a/Tests/MachOSymbolsTests/SymbolIndexStoreFixtureTests.swift b/Tests/MachOSymbolsTests/SymbolIndexStoreFixtureTests.swift new file mode 100644 index 00000000..7702eda4 --- /dev/null +++ b/Tests/MachOSymbolsTests/SymbolIndexStoreFixtureTests.swift @@ -0,0 +1,154 @@ +import Foundation +import Testing +@_spi(Internals) import Demangling +@_spi(Internals) @testable import MachOSymbols +@_spi(Internals) import MachOCaches +@testable import MachOTestingSupport +import MachOFixtureSupport + +/// Fixture-based unit coverage for `SymbolIndexStore` against the +/// `SymbolTestsCore` framework (self-built fixture, no external Xcode +/// dependency). Complements the heavyweight integration/baseline tests: +/// these assert the NodeStore-backed pipeline's invariants — cache-free +/// building, byte-identical printing versus the `Node` pipeline, and the +/// `structurallyEquals` bridge behind every `Node`-taking query API. +/// +/// Serialized: the NodeCache-growth test snapshots process-global counters, +/// and several tests share the cached per-file storage. +@Suite(.serialized) +final class SymbolIndexStoreFixtureTests: MachOFileTests, @unchecked Sendable { + override class var fileName: MachOFileName { .SymbolTestsCore } + + private var storage: SymbolIndexStore.Storage { + get throws { + try #require(SymbolIndexStore.shared.storage(in: machOFile)) + } + } + + // MARK: - Build invariants + + /// The migration's core invariant: the build pipeline stays off the + /// global `NodeCache` (the old pipeline leaked every leaf and interned + /// subtree into it, pinning them for the process lifetime). + /// + /// Asserted via leaf identity rather than global counters: every test + /// target shares one process, so concurrent suites legitimately grow + /// `NodeCache` and make counter deltas racy. The transient demangle the + /// build sweep uses mints fresh leaf instances on every call, whereas + /// any accidental cache participation would hand back the same canonical + /// instance — so `!==` across two runs is deterministic evidence. + /// (The process-global zero-growth measurement lives in the manually run + /// `SymbolIndexStoreBaselineTests`.) + @Test func buildPipelineStaysOffGlobalNodeCache() throws { + let builtStorage = try #require(SymbolIndexStore.shared.buildStorage(for: machOFile)) + #expect(!builtStorage.demangledNodeBySymbol.isEmpty) + + let sampleSymbolName = try #require(builtStorage.demangledNodeBySymbol.keys.first?.name) + let firstTransientTree = try demangleAsNodeTransient(sampleSymbolName) + let secondTransientTree = try demangleAsNodeTransient(sampleSymbolName) + let firstLeaf = try #require(firstTransientTree.first { $0.children.isEmpty }) + let secondLeaf = try #require(secondTransientTree.first { $0.children.isEmpty }) + #expect(firstLeaf == secondLeaf) + #expect(firstLeaf !== secondLeaf) + } + + /// Every symbol's zero-materialization print must be byte-identical to + /// the classic `demangleAsNode` + `Node.print` pipeline. + @Test func printedSymbolsMatchNodePipeline() throws { + let storage = try storage + var mismatchCount = 0 + for (symbol, reference) in storage.demangledNodeBySymbol { + let expected = try demangleAsNode(symbol.name, internsSubtrees: false).print(using: .default) + if reference.print(using: .default) != expected { + mismatchCount += 1 + if mismatchCount <= 3 { + Issue.record("Store print mismatch for \(symbol.name)") + } + } + } + #expect(mismatchCount == 0) + #expect(!storage.demangledNodeBySymbol.isEmpty) + } + + // MARK: - Query APIs + + /// `memberSymbols(of:for:node:)` takes an externally demangled `Node` and + /// must find the `NodeReference`-keyed bucket via `structurallyEquals`. + /// Exercise it for every bucket the index actually built. + @Test func memberQueryByNodeFindsEveryBucket() throws { + let storage = try storage + var checkedBucketCount = 0 + for (memberKind, memberSymbols) in storage.memberSymbolsByKind { + for (typeName, symbolsByTypeNode) in memberSymbols { + for (typeNodeReference, expectedSymbols) in symbolsByTypeNode { + let externalNode = typeNodeReference.materialize() + let queried = SymbolIndexStore.shared.memberSymbols(of: memberKind, for: typeName, node: externalNode, in: machOFile) + #expect(queried.count == expectedSymbols.count, "bucket \(memberKind) / \(typeName)") + checkedBucketCount += 1 + } + } + } + #expect(checkedBucketCount > 0) + } + + @Test func symbolKindQueriesMatchStorageBuckets() throws { + let storage = try storage + #expect(!storage.symbolsByKind.isEmpty) + for (kind, expectedSymbols) in storage.symbolsByKind { + let queried = SymbolIndexStore.shared.symbols(of: kind, in: machOFile) + #expect(queried.count == expectedSymbols.count) + #expect(queried.allSatisfy { $0.demangledNode.children.first?.kind == kind }) + } + } + + @Test func typeInfoLookupMatchesIndexedNames() throws { + let storage = try storage + #expect(!storage.typeInfoByName.isEmpty) + for (typeName, expectedTypeInfo) in storage.typeInfoByName { + let queried = try #require(SymbolIndexStore.shared.typeInfo(for: typeName, in: machOFile)) + #expect(queried.name == expectedTypeInfo.name) + } + } + + @Test func opaqueDescriptorQueryFindsEveryReferenceKey() throws { + let storage = try storage + for (keyReference, expectedSymbol) in storage.opaqueTypeDescriptorSymbolByNode { + let queried = try #require(SymbolIndexStore.shared.opaqueTypeDescriptorSymbol(for: keyReference.materialize(), in: machOFile)) + #expect(queried.symbol == expectedSymbol.symbol) + } + } + + // MARK: - demangledNode / demangledNodeReference + + @Test func demangledNodeAndReferenceAgree() throws { + let storage = try storage + var checkedCount = 0 + for (symbol, reference) in storage.demangledNodeBySymbol { + guard checkedCount < 200 else { break } + let materialized = try #require(SymbolIndexStore.shared.demangledNode(for: symbol, in: machOFile)) + #expect(reference.structurallyEquals(materialized)) + let referenceAgain = try #require(SymbolIndexStore.shared.demangledNodeReference(for: symbol, in: machOFile)) + #expect(referenceAgain == reference) + checkedCount += 1 + } + #expect(checkedCount > 0) + } + + /// Symbols outside the build sweep fall back to a per-symbol mini store: + /// the returned reference prints identically to the classic pipeline and + /// repeat lookups hit the late cache (same store identity). + @Test func lateSymbolFallsBackToMiniStore() throws { + _ = try storage + + let lateSymbol = Symbol(offset: -1, name: "$s7SwiftUI4ViewP") + let reference = try #require(SymbolIndexStore.shared.demangledNodeReference(for: lateSymbol, in: machOFile)) + let expected = try demangleAsNode(lateSymbol.name, internsSubtrees: false).print(using: .default) + #expect(reference.print(using: .default) == expected) + + let referenceAgain = try #require(SymbolIndexStore.shared.demangledNodeReference(for: lateSymbol, in: machOFile)) + #expect(referenceAgain == reference) + + let materialized = try #require(SymbolIndexStore.shared.demangledNode(for: lateSymbol, in: machOFile)) + #expect(reference.structurallyEquals(materialized)) + } +} diff --git a/Tests/SwiftAttributeInferenceTests/TypeAttributeInferrerTests.swift b/Tests/SwiftAttributeInferenceTests/TypeAttributeInferrerTests.swift index 655cc698..407f9fec 100644 --- a/Tests/SwiftAttributeInferenceTests/TypeAttributeInferrerTests.swift +++ b/Tests/SwiftAttributeInferenceTests/TypeAttributeInferrerTests.swift @@ -28,9 +28,11 @@ struct TypeAttributeInferrerTests { Node.create(kind: .identifier, text: "wrappedValue"), Node.create(kind: .type), ]) + var nodeStoreBuilder = NodeStoreBuilder() + let variableNodeIndex = nodeStoreBuilder.intern(variableNode) let dummySymbol = DemangledSymbol( symbol: Symbol(offset: 0, name: "$s_wrappedValue"), - demangledNode: variableNode + demangledNode: nodeStoreBuilder.freeze().reference(at: variableNodeIndex) ) let dummyAccessor = Accessor( kind: .getter, @@ -238,9 +240,11 @@ private func makeMockFunctionDefinition(name: String) -> FunctionDefinition { Node.create(kind: .identifier, text: name), Node.create(kind: .type), ]) + var nodeStoreBuilder = NodeStoreBuilder() + let functionNodeIndex = nodeStoreBuilder.intern(functionNode) let dummySymbol = DemangledSymbol( symbol: Symbol(offset: 0, name: "$s_\(name)"), - demangledNode: functionNode + demangledNode: nodeStoreBuilder.freeze().reference(at: functionNodeIndex) ) return FunctionDefinition( node: functionNode, diff --git a/Tests/SwiftDiffingTests/ABIDifferTests.swift b/Tests/SwiftDiffingTests/ABIDifferTests.swift index 9cd34162..949ad638 100644 --- a/Tests/SwiftDiffingTests/ABIDifferTests.swift +++ b/Tests/SwiftDiffingTests/ABIDifferTests.swift @@ -98,7 +98,7 @@ struct ABIDifferProjectionTests { node: node, name: name, kind: kind, - symbol: DemangledSymbol(symbol: Symbol(offset: 0, name: "$s_\(name)"), demangledNode: node), + symbol: DemangledSymbol(symbol: Symbol(offset: 0, name: "$s_\(name)"), demangledNode: makeNodeReference(node)), isGlobalOrStatic: false, methodDescriptor: nil, offset: nil, @@ -121,7 +121,7 @@ struct ABIDifferProjectionTests { let node = Node.create(kind: .identifier, text: name) return Accessor( kind: kind, - symbol: DemangledSymbol(symbol: Symbol(offset: 0, name: "$s_acc_\(name)"), demangledNode: node), + symbol: DemangledSymbol(symbol: Symbol(offset: 0, name: "$s_acc_\(name)"), demangledNode: makeNodeReference(node)), methodDescriptor: nil, offset: nil, vtableOffset: nil @@ -262,7 +262,7 @@ struct ABIDifferClassificationTests { node: node, name: name, kind: .function, - symbol: DemangledSymbol(symbol: Symbol(offset: 0, name: "$s_\(name)"), demangledNode: node), + symbol: DemangledSymbol(symbol: Symbol(offset: 0, name: "$s_\(name)"), demangledNode: makeNodeReference(node)), isGlobalOrStatic: false, methodDescriptor: nil, offset: nil, @@ -518,3 +518,9 @@ struct CompatibilityTests { #expect(!ABIDiff().hasBreakingChange) } } + +private func makeNodeReference(_ node: Node) -> NodeReference { + var nodeStoreBuilder = NodeStoreBuilder() + let nodeIndex = nodeStoreBuilder.intern(node) + return nodeStoreBuilder.freeze().reference(at: nodeIndex) +} diff --git a/Tests/SwiftDiffingTests/ABIExtensionAttributionTests.swift b/Tests/SwiftDiffingTests/ABIExtensionAttributionTests.swift index 7c7cbc1a..2400818b 100644 --- a/Tests/SwiftDiffingTests/ABIExtensionAttributionTests.swift +++ b/Tests/SwiftDiffingTests/ABIExtensionAttributionTests.swift @@ -49,7 +49,7 @@ struct ABIExtensionAttributionTests { node: node, name: name, kind: .function, - symbol: DemangledSymbol(symbol: Symbol(offset: 0, name: "$s_\(name)"), demangledNode: node), + symbol: DemangledSymbol(symbol: Symbol(offset: 0, name: "$s_\(name)"), demangledNode: makeNodeReference(node)), isGlobalOrStatic: false, methodDescriptor: nil, offset: nil, @@ -182,3 +182,9 @@ struct ABIExtensionAttributionTests { #expect(primary.definition.resolvedAssociatedTypeWitnesses.map(\.name) == ["Element", "Index"]) } } + +private func makeNodeReference(_ node: Node) -> NodeReference { + var nodeStoreBuilder = NodeStoreBuilder() + let nodeIndex = nodeStoreBuilder.intern(node) + return nodeStoreBuilder.freeze().reference(at: nodeIndex) +} diff --git a/Tests/SwiftPrintingTests/NodePrinterTests.swift b/Tests/SwiftPrintingTests/NodePrinterTests.swift index bb3e4e27..5d3dd2f1 100644 --- a/Tests/SwiftPrintingTests/NodePrinterTests.swift +++ b/Tests/SwiftPrintingTests/NodePrinterTests.swift @@ -290,7 +290,7 @@ final class NodePrinterIntegrationTests: DyldCacheTests, @unchecked Sendable { var failCount = 0 for demangledSymbol in demangledSymbols.prefix(20) { - let node = demangledSymbol.demangledNode + let node = demangledSymbol.demangledNode.materialize() do { var printer = FunctionNodePrinter(isOverride: false) guard let firstChild = node.children.first else { continue } @@ -341,7 +341,7 @@ final class NodePrinterIntegrationTests: DyldCacheTests, @unchecked Sendable { var successCount = 0 for demangledSymbol in demangledSymbols.prefix(10) { - let node = demangledSymbol.demangledNode + let node = demangledSymbol.demangledNode.materialize() do { var printer = SubscriptNodePrinter(isOverride: false, hasSetter: false, indentation: 1) let result = try await printer.printRoot(node).string From e9f0172df3dd77febd3da6f60f3a60679f640158 Mon Sep 17 00:00:00 2001 From: Mx-Iris Date: Fri, 24 Jul 2026 12:20:44 +0800 Subject: [PATCH 03/77] feat(MachOSymbols): compact the symbol table with flat rows (Stages 3-4) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Stage 3 of the NodeStore migration — the Symbol-side residency item the arena does not cover: - Symbol drops the (any NlistProtocol)? stored existential (40 B per copy; its only consumer was the collection-time external-undefined filter, which runs on the MachOKit symbol before Symbol is even constructed). The bit is captured as a stored isExternal Bool; Symbol is explicitly Sendable and shrinks 64 B -> 32 B. - Storage keeps one flat symbolTable: [Symbol] row per unique symbol name (canonical cache-adjusted offset), a parallel rootNodeIndexByTableRow array, and tableRowByName whose keys share row string storage. Every classification index (symbolsByKind, member x3, global, opaque, symbolsByOffset) stores 4-byte UInt32 rows; member/opaque keys shrink from 16-byte NodeReference to 4-byte NodeStore.NodeIndex. Raw and cache-adjusted offset keys share one row; symbols(for:in:) rebuilds each Symbol with the queried offset, byte-identical to the former per-key copies. - DemangledSymbol becomes a 32-byte value: shared-table reference + row + NodeReference, with symbol computed. The public init(symbol:demangledNode:) stays as a single-row-table compat path. - Indexes accumulate in final row form during the sweep (RowIndexes), so freeze() is followed by a plain move into Storage — the pending->populate double-index conversion pass (+30 MB transient peak) is gone and all Storage index fields are now lets. - demangledNodeReference(for:) resolves name -> row + canonical-offset check, strictly equivalent to the former [Symbol: NodeReference] keying; misses still fall through to the late mini-store path. Stage 4 re-measurement (SwiftUI image, debug, same methodology): build 24.5s (legacy 28.6s, Stage 1+2 31.0s); build-phase phys_footprint delta 68 MB (was 302 MB); post-release residual 49 MB; NodeCache growth stays 0/0; NodeStore identical at 7 MB / 579,291 nodes; index entry counts match the baseline item for item. Tests: fixture suite adapted to the row layout plus two new cases — compactValueLayouts (stride <= 32 B invariants) and offsetQueriesRebuildSymbolsWithQueriedOffset (shared-row exit semantics); baseline test reports symbolTable rows/stride. Acceptance: 79 tests in 5 suites passed (60 snapshots byte-identical + 10 fixture cases + SharedCacheTests). Note: struct-layout changes in MachOSymbols require a clean rebuild — SwiftPM incremental builds linked stale downstream objects (runtime SIGSEGV in outlined destroy of the old Symbol layout). --- AGENTS.md | 2 +- .../Internal/NodeStoreMigrationPlan.md | 30 +- Sources/MachOSymbols/DemangledSymbol.swift | 26 +- Sources/MachOSymbols/Symbol.swift | 14 +- Sources/MachOSymbols/SymbolIndexStore.swift | 414 ++++++++++-------- .../SymbolIndexStoreBaselineTests.swift | 27 +- .../SymbolIndexStoreFixtureTests.swift | 78 +++- 7 files changed, 369 insertions(+), 222 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index d95f96e9..6455eddd 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -206,7 +206,7 @@ let node = try demangleAsNode("$sSiD") // Returns Node tree let string = node.print(using: .default) // "Swift.Int" ``` -**Symbol indexing (NodeStore-backed)**: `SymbolIndexStore.Storage` no longer retains `Node` class trees. The build sweep demangles each symbol cache-free onto a transient tree (`@_spi(Internals) demangleAsNodeTransient`), classifies on that tree, and interns it into a per-image `NodeStoreBuilder`; after `freeze()` the indexes hold 16-byte `NodeReference` handles (`DemangledSymbol.demangledNode: NodeReference`). Nothing touches the global `NodeCache`, and dropping a `Storage` releases the whole per-image footprint. Query APIs taking an externally demangled `Node` (`memberSymbols(of:for:node:)`, `opaqueTypeDescriptorSymbol(for:)`) match keys via `NodeReference.structurallyEquals(_:)`; renderer boundaries (`demangleResolver.resolve`, `Definition` models) call `materialize()`. Late symbols outside the build sweep get per-symbol mini stores (`demangledNodeReference(for:)` / `MetadataReader.demangleSymbolReference(for:in:)`). See `Documentations/Internal/NodeStoreMigrationPlan.md`. +**Symbol indexing (NodeStore-backed)**: `SymbolIndexStore.Storage` no longer retains `Node` class trees. The build sweep demangles each symbol cache-free onto a transient tree (`@_spi(Internals) demangleAsNodeTransient`), classifies on that tree, and interns it into a per-image `NodeStoreBuilder`. Nothing touches the global `NodeCache`, and dropping a `Storage` releases the whole per-image footprint. Storage layout (Stage 3): one flat `symbolTable: [Symbol]` row per unique symbol name (canonical cache-adjusted offset; raw and adjusted offset keys share one row), a parallel `rootNodeIndexByTableRow` array, and every classification index stores 4-byte `UInt32` table rows (member/opaque keys are 4-byte `NodeStore.NodeIndex`); indexes accumulate in final row form during the sweep, so `freeze()` is followed by a plain move, not a conversion pass. `Symbol` is 32 bytes (no `nlist` existential — the external-undefined bit is extracted at collection time into `isExternal`), and `DemangledSymbol` is a 32-byte value (shared-table reference + row + `NodeReference`) whose `symbol` is computed; both are asserted by `compactValueLayouts`. Query APIs construct `DemangledSymbol`/`NodeReference` values at the exit boundary; APIs taking an externally demangled `Node` (`memberSymbols(of:for:node:)`, `opaqueTypeDescriptorSymbol(for:)`) match keys via `NodeReference.structurallyEquals(_:)`; renderer boundaries (`demangleResolver.resolve`, `Definition` models) call `materialize()`. Late symbols outside the build sweep get per-symbol mini stores (`demangledNodeReference(for:)` / `MetadataReader.demangleSymbolReference(for:in:)`). Struct-layout changes in `MachOSymbols` need a `swift package clean` rebuild — SwiftPM incremental builds have been observed linking stale downstream objects (runtime SIGSEGV in `outlined destroy`). See `Documentations/Internal/NodeStoreMigrationPlan.md`. ## Test Environment diff --git a/Documentations/Internal/NodeStoreMigrationPlan.md b/Documentations/Internal/NodeStoreMigrationPlan.md index 258403e3..7485b7c4 100644 --- a/Documentations/Internal/NodeStoreMigrationPlan.md +++ b/Documentations/Internal/NodeStoreMigrationPlan.md @@ -1,9 +1,9 @@ # NodeStore 迁移计划（SymbolIndexStore → arena 存储） -- **状态**: In Progress（Stage 0–2 已落地，见文末「实施记录」） +- **状态**: Completed（Stage 0–4 全部落地，见文末「实施记录」） - **日期**: 2026-07-24 - **最后更新**: 2026-07-24 -- **分支**: `feature/node-store-migration`（worktree `.claude/worktrees/node-store-migration`，Demangling 经 `.claude/worktrees/swift-demangling` 符号链接解析到 swift-demangling 的 `feature/node-store` worktree） +- **分支**: `feature/node-store-migration`（worktree `.claude/worktrees/node-store-migration`，Demangling 经主检出 `.claude/worktrees/swift-demangling` 处的**真实 git worktree**（swift-demangling `feature/node-store`）以路径依赖解析——原先的符号链接方案因目标 worktree 被外部清理导致 SwiftPM manifest 缓存把解析钉回 remote，已改为本仓库领地内的 worktree） - **前置**: swift-demangling `feature/node-store` 分支合入 `main`（本包以路径依赖解析 `../swift-demangling` 的 main）；开发期先经上述符号链接直连该分支 - **上游依据**: swift-demangling `evolution/0001-node-store-arena.md`（Phase 1–3 已落地并验收） @@ -126,3 +126,29 @@ RuntimeViewer 与 MachOSwiftSection 的内存主项在 `MachOSymbols/SymbolIndex 1. **快照以 main 为基准逐字节对比**：先在 main（`7f7fe48`，旧管线）上运行 `SymbolTestsCoreInterfaceSnapshotTests` + `SymbolTestsCoreDumpSnapshotTests`（60 个快照测试）确认已提交基准与 main 输出一致；再在迁移 worktree 上运行同一套快照测试——**60/60 逐字节一致**。fixture 为自建 `SymbolTestsCore.framework`（无外部 Xcode 依赖；worktree 经符号链接复用主检出 `Tests/Projects/SymbolTests/DerivedData` 的构建产物）。 2. **启用 `MachOSymbolsTests` target**（Package.swift 中原已定义但被注释）并新增 `SymbolIndexStoreFixtureTests`（8 个用例）：build 管线 cache-free 不变量（叶身份断言——所有测试 target 共享单进程，全局 `NodeCache` 计数断言天然竞态，改用「transient demangle 两次得到结构相等但 `!==` 的叶实例」这一并发免疫口径；进程级零增长量测留在手动运行的 `SymbolIndexStoreBaselineTests`）、全符号零物化打印与 `demangleAsNode` 管线逐字节对齐、`memberSymbols(of:for:node:)` 对每个 `NodeReference` 键桶经 `structurallyEquals` 命中、`symbols(of:)`/`typeInfo`/`opaqueTypeDescriptorSymbol` 与 storage 桶一致、`demangledNode`/`demangledNodeReference` 互证、迟到符号 mini store 回退与缓存稳定性。 3. **修复 `SharedCache` 并发时序 flake**：`concurrentCallsForDifferentKeysRunInParallel` 原以墙钟阈值断言并行（CPU 饱和即假失败），改为确定性并行证据——所有 build 经信号量互相等待进入闭包，若 resolve 对不同 key 串行（锁跨 build）则死锁，由宽松超时转为失败而非挂死。 + +### Stage 3 — Symbol 表压缩落地（2026-07-24） + +1. **`Symbol` 去掉 `nlist` existential**：`(any NlistProtocol)?` 存储属性（40B existential 容器/份）删除。审计确认其唯一实际消费者是采集期的 undefined-external 过滤（`N_EXT + N_UNDF`），且该过滤发生在 MachOKit 符号上、早于 `Symbol` 构造——存进 `Symbol` 后即为纯死重。压缩为采集期提取的 `isExternal: Bool` 存储位；`Symbol` 显式 `Sendable`，stride 64B → **32B**。公共 init 由 `init(offset:name:nlist:)` 改为 `init(offset:name:isExternal:)`（RuntimeViewer 源码审计确认无 `.nlist` / `DemangledSymbol` 直接消费者，仅经 `.symbol.name` / `.addressString` 取值）。 +2. **平铺符号表**：`Storage.symbolTable: [Symbol]` 每唯一符号名一行（存 canonical 即 cache-adjusted 偏移）；`tableRowByName: [String: UInt32]`（键与表行共享字符串存储）；`rootNodeIndexByTableRow: [NodeIndex?]` 平行数组承接原 `demangledNodeBySymbol` 的值侧。全部索引（`symbolRowsByKind` / member×3 / global / opaque / `symbolRowsByOffset`）改存 4B `UInt32` 行号；member/opaque 键从 16B `NodeReference` 改为 4B `NodeStore.NodeIndex`（出口按需 `nodeStore.reference(at:)` 重建）。**双 offset 键共用同一行**——每符号双份 `Symbol` 复本消除；`symbols(for:in:)` 出口按查询键重建 `Symbol(offset: queriedOffset, ...)`，与旧的 per-key 复本语义逐字节一致。 +3. **`DemangledSymbol` 压缩为 32B**：`(symbolTable: [Symbol]（共享缓冲，8B 指针）, symbolTableRow: UInt32, demangledNode: NodeReference)`；`symbol` 变为计算属性，`@dynamicMemberLookup` 转发不变。公共 `init(symbol:demangledNode:)` 保留为单行表兼容路径（测试与 `ExtensionDefinition` 的显式构造点）。索引出口从行号现场构造值，原先每条索引条目 ~80B 内联 `Symbol`+`NodeReference` 复本全部消失。 +4. **pending→populate 双索引窗口整个消除**：构建期分类索引直接以最终行号形态累积（`RowIndexes`），`freeze()` 后 `Storage.init` 纯移动字典——原 `PendingStorage`→`populate()` 转换pass（Stage 1 记录的 +30 MB 瞬态峰值来源）删除。`Storage` 全部索引字段现为 `let`（仅 late-symbol cache 为 `@Mutex var`）。 +5. **`demangledNodeReference(for:)` 查找重写**：从 `[Symbol: NodeReference]` 字典命中改为 `tableRowByName[name]` + canonical offset 校验 + 平行数组取根——与旧 `(offset, name)` 键哈希语义严格等价（offset 不匹配 / demangle 失败仍落 late mini-store 路径）。 +6. **测试同步**：`SymbolIndexStoreFixtureTests` 适配行号布局并新增 2 用例——`compactValueLayouts`（`Symbol` / `DemangledSymbol` stride ≤ 32B 的紧凑性不变量）与 `offsetQueriesRebuildSymbolsWithQueriedOffset`（双键共行后出口 offset 重建语义，逐 offset 键对账行数与名称）；`SymbolIndexStoreBaselineTests` 改读新字段并输出 `symbolTable` 行数/stride。 +7. **过程备注（环境）**：布局变更后 SwiftPM 增量构建未能把 `MachOSymbols` 的 struct 布局变化传播到全部依赖模块（先是陈旧目标文件的 linker 错，touch 后链接通过但运行期在 `Symbol.init` 内按旧布局 outlined destroy 直接 SIGSEGV）——`swift package clean` 全量重建后消失。此类跨模块布局变更建议直接 clean 构建。 + +### Stage 4 — 最终验收与复测（2026-07-24，SwiftUI image，debug，独占口径同 Stage 0） + +| 指标 | Stage 0 旧管线 | Stage 1+2 | Stage 3 | +|---|---|---|---| +| 构建耗时（独占） | 28.6s | 31.0s | **24.5s（快于旧管线 14%）** | +| 构建期 `phys_footprint` 增量 | 266–272 MB | 302 MB | **68 MB** | +| 释放 `Storage` + pressure relief 后 | 残留 ~92 MB | 残留 ~66 MB | **残留 49 MB**（回收 38 MB / 68 MB 增量） | +| `NodeCache` 增长 | +19,345 叶 / +559,976 子树 | 0 / 0 | 0 / 0 | +| `NodeStore` 本体 | — | 7 MB / 579,291 唯一节点 | 7 MB / 579,291（与 Stage 1 完全一致，语义保真旁证） | +| Symbol 侧驻留 | nlist 盒 + 双份条目 + 索引内联复本（数十 MB 级） | 同旧形态 | `symbolTable` 202,603 行 × 32 B ≈ 6.2 MB + 共享字符串 + 各索引 4B 行号 | +| 索引条目 | 基线 | 逐项一致 | 逐项一致（demangled 202,603；member 17,049；methodDescriptor 2,209；global 82；offset 表 170,919；opaque 2,115；typeInfo 4,191） | + +- 构建耗时的下降来自：populate 转换 pass 删除、每符号双份 `Symbol` 构造与 existential 装箱消失、索引累积只搬 4B 行号。 +- 构建期增量从 302 MB 收敛到 68 MB：双索引瞬态窗口消除 + Symbol 复本/existential 盒清零是主贡献；残余 68 MB 为 NodeStore + 表 + 索引 + malloc 未归还页。 +- 验收测试：`SymbolTestsCore` 快照 60/60 逐字节一致 + `SymbolIndexStoreFixtureTests` 10/10 + `SharedCacheTests` 全绿（**79 tests / 5 suites passed**，Stage 3 代码 + 本地 feature 分支解析口径复跑确认）。 diff --git a/Sources/MachOSymbols/DemangledSymbol.swift b/Sources/MachOSymbols/DemangledSymbol.swift index 9bb60312..e7c4cc7e 100644 --- a/Sources/MachOSymbols/DemangledSymbol.swift +++ b/Sources/MachOSymbols/DemangledSymbol.swift @@ -1,13 +1,35 @@ import Demangling +/// A symbol paired with the handle of its demangled tree. +/// +/// Compact by construction (NodeStore migration, Stage 3): instead of an +/// inline `Symbol` copy the value stores a row index into the per-image flat +/// symbol table, so the hundreds of thousands of `DemangledSymbol` values +/// vended by `SymbolIndexStore` share one `[Symbol]` buffer and stay at +/// 32 bytes each (table reference + row + `NodeReference`). @dynamicMemberLookup public struct DemangledSymbol: Sendable { - public let symbol: Symbol + private let symbolTable: [Symbol] + + private let symbolTableRow: UInt32 public let demangledNode: NodeReference + public var symbol: Symbol { + symbolTable[Int(symbolTableRow)] + } + + /// Wraps a standalone symbol in a single-row table. `SymbolIndexStore` + /// vends values through the shared-table initializer instead. public init(symbol: Symbol, demangledNode: NodeReference) { - self.symbol = symbol + self.symbolTable = [symbol] + self.symbolTableRow = 0 + self.demangledNode = demangledNode + } + + init(symbolTable: [Symbol], symbolTableRow: UInt32, demangledNode: NodeReference) { + self.symbolTable = symbolTable + self.symbolTableRow = symbolTableRow self.demangledNode = demangledNode } diff --git a/Sources/MachOSymbols/Symbol.swift b/Sources/MachOSymbols/Symbol.swift index c4bde271..6574e9db 100644 --- a/Sources/MachOSymbols/Symbol.swift +++ b/Sources/MachOSymbols/Symbol.swift @@ -5,17 +5,21 @@ import MachOKitExtensions import Demangling import FoundationToolbox -public struct Symbol: AsyncResolvable, SymbolProtocol, Hashable { +public struct Symbol: AsyncResolvable, SymbolProtocol, Hashable, Sendable { public let offset: Int public let name: String - public let nlist: (any NlistProtocol)? + /// Whether the symbol-table entry was flagged as an undefined external + /// import (`N_EXT` with `N_UNDF` type). Extracted from the `nlist` entry + /// at collection time; the entry itself is not retained (a 40-byte + /// existential per symbol copy that nothing else consumed). + public let isExternal: Bool - public init(offset: Int, name: String, nlist: (any NlistProtocol)? = nil) { + public init(offset: Int, name: String, isExternal: Bool = false) { self.offset = offset self.name = name - self.nlist = nlist + self.isExternal = isExternal } public static func resolve(from offset: Int, in machO: MachO) throws -> Self { @@ -84,7 +88,7 @@ extension MachOSymbols.SymbolProtocol { extension MachOKit.SymbolProtocol { fileprivate var asCurrentSymbol: MachOSymbols.Symbol { - .init(offset: offset, name: name, nlist: nlist) + .init(offset: offset, name: name, isExternal: nlist.isExternal) } } diff --git a/Sources/MachOSymbols/SymbolIndexStore.swift b/Sources/MachOSymbols/SymbolIndexStore.swift index 0f2d9b27..4cab77f3 100644 --- a/Sources/MachOSymbols/SymbolIndexStore.swift +++ b/Sources/MachOSymbols/SymbolIndexStore.swift @@ -129,34 +129,45 @@ public final class SymbolIndexStore: SharedCache, @unc } } - typealias IndexedSymbol = DemangledSymbol - typealias AllSymbols = [IndexedSymbol] - typealias GlobalSymbols = [IndexedSymbol] - typealias MemberSymbols = OrderedDictionary> - typealias OpaqueTypeDescriptorSymbol = IndexedSymbol - public final class Storage: @unchecked Sendable { + typealias MemberSymbolRows = OrderedDictionary> + /// The frozen arena holding every demangled node of this image. /// All `NodeReference` values vended by this storage point into it. let nodeStore: NodeStore - private(set) var typeInfoByName: [String: TypeInfo] = [:] + /// Flat symbol table (Stage 3): one row per unique symbol name, + /// holding the canonical (cache-adjusted) offset. Every index below + /// stores 4-byte row indices into this table instead of inline + /// `Symbol` copies, and vended `DemangledSymbol` values share this + /// array's buffer. + let symbolTable: [Symbol] + + /// Parallel to `symbolTable`: the row's demangled root node, or + /// `nil` for names the demangler rejected (those still occupy a row + /// because `symbolRowsByOffset` references them). + let rootNodeIndexByTableRow: [NodeStore.NodeIndex?] + + /// Name → table row. Keys share string storage with `symbolTable`. + let tableRowByName: [String: UInt32] - private(set) var globalSymbolsByKind: OrderedDictionary = [:] + let typeInfoByName: [String: TypeInfo] - private(set) var opaqueTypeDescriptorSymbolByNode: OrderedDictionary = [:] + let globalSymbolRowsByKind: OrderedDictionary - private(set) var memberSymbolsByKind: OrderedDictionary = [:] + let opaqueTypeDescriptorSymbolRowByNodeIndex: OrderedDictionary - private(set) var methodDescriptorMemberSymbolsByKind: OrderedDictionary = [:] + let memberSymbolRowsByKind: OrderedDictionary - private(set) var protocolWitnessMemberSymbolsByKind: OrderedDictionary = [:] + let methodDescriptorMemberSymbolRowsByKind: OrderedDictionary - private(set) var symbolsByKind: OrderedDictionary = [:] + let protocolWitnessMemberSymbolRowsByKind: OrderedDictionary - private(set) var symbolsByOffset: OrderedDictionary = [:] + let symbolRowsByKind: OrderedDictionary - private(set) var demangledNodeBySymbol: [Symbol: NodeReference] = [:] + let symbolRowsByOffset: OrderedDictionary + + let thunkAttributeMembersByKindAndTypeName: [Node.Kind: [String: [ThunkAttributeMember]]] /// Symbols demangled after the store was frozen (rare path: lookups /// for symbols that were not part of the build sweep). The frozen @@ -166,91 +177,88 @@ public final class SymbolIndexStore: SharedCache, @unc @Mutex private(set) var lateDemangledNodeBySymbol: [Symbol: NodeReference] = [:] - private(set) var thunkAttributeMembersByKindAndTypeName: [Node.Kind: [String: [ThunkAttributeMember]]] = [:] - - fileprivate init(nodeStore: NodeStore) { + fileprivate init( + nodeStore: NodeStore, + symbolTable: [Symbol], + rootNodeIndexByTableRow: [NodeStore.NodeIndex?], + tableRowByName: [String: UInt32], + symbolRowsByOffset: OrderedDictionary, + rowIndexes: consuming RowIndexes + ) { self.nodeStore = nodeStore + self.symbolTable = symbolTable + self.rootNodeIndexByTableRow = rootNodeIndexByTableRow + self.tableRowByName = tableRowByName + self.symbolRowsByOffset = symbolRowsByOffset + self.typeInfoByName = rowIndexes.typeInfoByName + self.globalSymbolRowsByKind = rowIndexes.globalSymbolRowsByKind + self.opaqueTypeDescriptorSymbolRowByNodeIndex = rowIndexes.opaqueTypeDescriptorSymbolRowByNodeIndex + self.memberSymbolRowsByKind = rowIndexes.memberSymbolRowsByKind + self.methodDescriptorMemberSymbolRowsByKind = rowIndexes.methodDescriptorMemberSymbolRowsByKind + self.protocolWitnessMemberSymbolRowsByKind = rowIndexes.protocolWitnessMemberSymbolRowsByKind + self.symbolRowsByKind = rowIndexes.symbolRowsByKind + self.thunkAttributeMembersByKindAndTypeName = rowIndexes.thunkAttributeMembersByKindAndTypeName } fileprivate func setLateDemangledNode(_ demangledNode: NodeReference?, for symbol: Symbol) { lateDemangledNodeBySymbol[symbol] = demangledNode } - /// One-shot population after `freeze()`: converts the build-time - /// `NodeIndex`-keyed scratch into `NodeReference`-based indexes. - fileprivate func populate(from pending: PendingStorage, symbolsByOffset: OrderedDictionary) { - func demangledSymbol(_ pendingSymbol: PendingDemangledSymbol) -> DemangledSymbol { - DemangledSymbol(symbol: pendingSymbol.symbol, demangledNode: nodeStore.reference(at: pendingSymbol.rootNodeIndex)) - } - func memberSymbols(_ pendingMemberSymbols: PendingStorage.MemberSymbols) -> MemberSymbols { - var converted: MemberSymbols = [:] - for (typeName, symbolsByTypeNodeIndex) in pendingMemberSymbols { - var convertedByTypeNode: OrderedDictionary = [:] - for (typeNodeIndex, pendingSymbols) in symbolsByTypeNodeIndex { - convertedByTypeNode[nodeStore.reference(at: typeNodeIndex)] = pendingSymbols.map(demangledSymbol) - } - converted[typeName] = convertedByTypeNode - } - return converted - } + // MARK: Row materialization - typeInfoByName = pending.typeInfoByName - globalSymbolsByKind = pending.globalSymbolsByKind.mapValues { $0.map(demangledSymbol) } - opaqueTypeDescriptorSymbolByNode = .init(uniqueKeysWithValues: pending.opaqueTypeDescriptorSymbolByNodeIndex.map { (nodeStore.reference(at: $0.key), demangledSymbol($0.value)) }) - memberSymbolsByKind = pending.memberSymbolsByKind.mapValues(memberSymbols) - methodDescriptorMemberSymbolsByKind = pending.methodDescriptorMemberSymbolsByKind.mapValues(memberSymbols) - protocolWitnessMemberSymbolsByKind = pending.protocolWitnessMemberSymbolsByKind.mapValues(memberSymbols) - symbolsByKind = pending.symbolsByKind.mapValues { $0.map(demangledSymbol) } - demangledNodeBySymbol = pending.demangledNodeIndexBySymbol.mapValues { nodeStore.reference(at: $0) } - thunkAttributeMembersByKindAndTypeName = pending.thunkAttributeMembersByKindAndTypeName - self.symbolsByOffset = symbolsByOffset + /// Rebuilds the `Symbol` for an offset-table row using the queried + /// offset: raw and cache-adjusted keys share one canonical row, so + /// the row's stored offset is not necessarily the queried one. + fileprivate func symbol(atRow row: UInt32, offset queriedOffset: Int) -> Symbol { + let canonicalSymbol = symbolTable[Int(row)] + return Symbol(offset: queriedOffset, name: canonicalSymbol.name, isExternal: canonicalSymbol.isExternal) } - } - /// A `(symbol, root node index)` pair collected while the builder is still - /// mutable; becomes a `DemangledSymbol` once the store is frozen. - fileprivate struct PendingDemangledSymbol: Sendable { - let symbol: Symbol - let rootNodeIndex: NodeStore.NodeIndex - } + func demangledSymbol(atRow row: UInt32) -> DemangledSymbol? { + guard let rootNodeIndex = rootNodeIndexByTableRow[Int(row)] else { return nil } + return DemangledSymbol(symbolTable: symbolTable, symbolTableRow: row, demangledNode: nodeStore.reference(at: rootNodeIndex)) + } - /// Build-time scratch mirroring `Storage`'s indexes with `NodeIndex` keys - /// and `PendingDemangledSymbol` entries. Lives only for the duration of - /// `buildStorageImpl`; converted via `Storage.populate(from:symbolsByOffset:)`. - fileprivate struct PendingStorage { - typealias MemberSymbols = OrderedDictionary> + func demangledSymbols(atRows rows: [UInt32]) -> [DemangledSymbol] { + rows.compactMap { demangledSymbol(atRow: $0) } + } + } + /// Build-time accumulator holding the row-index form of `Storage`'s + /// classification indexes. `Storage.init` moves these dictionaries in + /// unchanged — there is no post-freeze conversion pass, so the former + /// pending→populate double-index transient peak is gone (Stage 3). + fileprivate struct RowIndexes { var typeInfoByName: [String: TypeInfo] = [:] - var globalSymbolsByKind: OrderedDictionary = [:] - var opaqueTypeDescriptorSymbolByNodeIndex: OrderedDictionary = [:] - var memberSymbolsByKind: OrderedDictionary = [:] - var methodDescriptorMemberSymbolsByKind: OrderedDictionary = [:] - var protocolWitnessMemberSymbolsByKind: OrderedDictionary = [:] - var symbolsByKind: OrderedDictionary = [:] - var demangledNodeIndexBySymbol: [Symbol: NodeStore.NodeIndex] = [:] + var globalSymbolRowsByKind: OrderedDictionary = [:] + var opaqueTypeDescriptorSymbolRowByNodeIndex: OrderedDictionary = [:] + var memberSymbolRowsByKind: OrderedDictionary = [:] + var methodDescriptorMemberSymbolRowsByKind: OrderedDictionary = [:] + var protocolWitnessMemberSymbolRowsByKind: OrderedDictionary = [:] + var symbolRowsByKind: OrderedDictionary = [:] var thunkAttributeMembersByKindAndTypeName: [Node.Kind: [String: [ThunkAttributeMember]]] = [:] - mutating func appendSymbol(_ pendingSymbol: PendingDemangledSymbol, for kind: Node.Kind) { - symbolsByKind[kind, default: []].append(pendingSymbol) + mutating func appendSymbolRow(_ symbolTableRow: UInt32, for kind: Node.Kind) { + symbolRowsByKind[kind, default: []].append(symbolTableRow) } mutating func setMemberSymbols(for result: ProcessMemberSymbolResult) { - memberSymbolsByKind[result.memberKind, default: [:]][result.typeName, default: [:]][result.typeNodeIndex, default: []].append(result.pendingSymbol) + memberSymbolRowsByKind[result.memberKind, default: [:]][result.typeName, default: [:]][result.typeNodeIndex, default: []].append(result.symbolTableRow) typeInfoByName[result.typeName] = result.typeInfo } mutating func setMethodDescriptorMemberSymbols(for result: ProcessMemberSymbolResult) { - methodDescriptorMemberSymbolsByKind[result.memberKind, default: [:]][result.typeName, default: [:]][result.typeNodeIndex, default: []].append(result.pendingSymbol) + methodDescriptorMemberSymbolRowsByKind[result.memberKind, default: [:]][result.typeName, default: [:]][result.typeNodeIndex, default: []].append(result.symbolTableRow) typeInfoByName[result.typeName] = result.typeInfo } mutating func setProtocolWitnessMemberSymbols(for result: ProcessMemberSymbolResult) { - protocolWitnessMemberSymbolsByKind[result.memberKind, default: [:]][result.typeName, default: [:]][result.typeNodeIndex, default: []].append(result.pendingSymbol) + protocolWitnessMemberSymbolRowsByKind[result.memberKind, default: [:]][result.typeName, default: [:]][result.typeNodeIndex, default: []].append(result.symbolTableRow) typeInfoByName[result.typeName] = result.typeInfo } mutating func setGlobalSymbols(for result: ProcessGlobalSymbolResult) { - globalSymbolsByKind[result.kind, default: []].append(result.pendingSymbol) + globalSymbolRowsByKind[result.kind, default: []].append(result.symbolTableRow) } mutating func appendThunkAttributeMember(_ member: ThunkAttributeMember, forKind thunkKind: Node.Kind, typeName: String) { @@ -272,29 +280,48 @@ public final class SymbolIndexStore: SharedCache, @unc for machO: MachO, progressContinuation: AsyncStream.Continuation? ) -> Storage? { - var cachedSymbols: Set = [] - var symbolByName: OrderedDictionary = [:] - var symbolsByOffset: OrderedDictionary = [:] + var symbolTable: [Symbol] = [] + var tableRowByName: [String: UInt32] = [:] + var symbolRowsByOffset: OrderedDictionary = [:] + + // Raw and cache-adjusted offset keys share one canonical row; a + // duplicate name updates the existing row in place (last-wins, like + // the former name-keyed collection pass). + func canonicalRow(for canonicalSymbol: Symbol) -> UInt32 { + if let existingRow = tableRowByName[canonicalSymbol.name] { + symbolTable[Int(existingRow)] = canonicalSymbol + return existingRow + } + let newRow = UInt32(symbolTable.count) + symbolTable.append(canonicalSymbol) + tableRowByName[canonicalSymbol.name] = newRow + return newRow + } for symbol in machO.symbols where symbol.name.isSwiftSymbol && !symbol.nlist.isExternal { - var offset = symbol.offset - symbolsByOffset[offset, default: []].append(.init(offset: offset, name: symbol.name, nlist: symbol.nlist)) - if let cache = machO.cache, offset >= 0, machO is MachOFile { - offset -= cache.mainCacheHeader.sharedRegionStart.cast() - symbolsByOffset[offset, default: []].append(.init(offset: offset, name: symbol.name, nlist: symbol.nlist)) + let rawOffset = symbol.offset + var canonicalOffset = rawOffset + var hasAdjustedOffset = false + if let cache = machO.cache, rawOffset >= 0, machO is MachOFile { + canonicalOffset = rawOffset - cache.mainCacheHeader.sharedRegionStart.cast() + hasAdjustedOffset = true + } + let row = canonicalRow(for: .init(offset: canonicalOffset, name: symbol.name, isExternal: symbol.nlist.isExternal)) + symbolRowsByOffset[rawOffset, default: []].append(row) + if hasAdjustedOffset { + symbolRowsByOffset[canonicalOffset, default: []].append(row) } - symbolByName[symbol.name] = .init(offset: offset, name: symbol.name, nlist: symbol.nlist) - cachedSymbols.insert(symbol.name) } for exportedSymbol in machO.exportedSymbols where exportedSymbol.name.isSwiftSymbol { - if var offset = exportedSymbol.offset, symbolByName[exportedSymbol.name] == nil { - symbolsByOffset[offset, default: []].append(.init(offset: offset, name: exportedSymbol.name)) + if let rawOffset = exportedSymbol.offset, tableRowByName[exportedSymbol.name] == nil { + var canonicalOffset = rawOffset if machO is MachOFile { - offset += machO.startOffset + canonicalOffset += machO.startOffset } - symbolsByOffset[offset, default: []].append(.init(offset: offset, name: exportedSymbol.name)) - symbolByName[exportedSymbol.name] = .init(offset: offset, name: exportedSymbol.name) + let row = canonicalRow(for: .init(offset: canonicalOffset, name: exportedSymbol.name)) + symbolRowsByOffset[rawOffset, default: []].append(row) + symbolRowsByOffset[canonicalOffset, default: []].append(row) } } @@ -302,74 +329,76 @@ public final class SymbolIndexStore: SharedCache, @unc // transient tree, classify on that tree, and intern the result into // the arena builder. Nothing touches the global `NodeCache` and no // class trees outlive the loop iteration (NodeStore migration plan, - // Stage 1). The former concurrentMap pipeline kept every class tree - // alive simultaneously and leaked all of them into `NodeCache.shared`. - let symbolArray = Array(symbolByName.values) - let totalSymbolCount = symbolArray.count + // Stage 1). Indexes accumulate directly in their final row-index + // form (Stage 3), so `freeze()` is followed by a plain move into + // `Storage`, not a conversion pass. + let totalSymbolCount = symbolTable.count var builder = NodeStoreBuilder() - var pending = PendingStorage() - pending.demangledNodeIndexBySymbol.reserveCapacity(totalSymbolCount) + var rootNodeIndexByTableRow = [NodeStore.NodeIndex?](repeating: nil, count: totalSymbolCount) + var rowIndexes = RowIndexes() - for symbolIndex in 0.. 0 { - pending.opaqueTypeDescriptorSymbolByNodeIndex[builder.intern(memberSymbol)] = pendingSymbol + rowIndexes.opaqueTypeDescriptorSymbolRowByNodeIndex[builder.intern(memberSymbol)] = symbolTableRow } } else { - if let result = processMemberSymbol(pendingSymbol, node: node, builder: &builder) { - pending.setMemberSymbols(for: result) + if let result = processMemberSymbol(symbolTableRow, node: node, builder: &builder) { + rowIndexes.setMemberSymbols(for: result) } } } } progressContinuation?.yield(Progress(currentCount: totalSymbolCount, totalCount: totalSymbolCount)) - let storage = Storage(nodeStore: builder.freeze()) - storage.populate(from: pending, symbolsByOffset: symbolsByOffset) - - return storage + return Storage( + nodeStore: builder.freeze(), + symbolTable: symbolTable, + rootNodeIndexByTableRow: rootNodeIndexByTableRow, + tableRowByName: tableRowByName, + symbolRowsByOffset: symbolRowsByOffset, + rowIndexes: rowIndexes + ) } fileprivate struct ProcessMemberSymbolResult: Sendable { @@ -377,19 +406,19 @@ public final class SymbolIndexStore: SharedCache, @unc let typeName: String let typeNodeIndex: NodeStore.NodeIndex let typeInfo: TypeInfo - let pendingSymbol: PendingDemangledSymbol + let symbolTableRow: UInt32 } - private func processMemberSymbol(_ pendingSymbol: PendingDemangledSymbol, node: Node, builder: inout NodeStoreBuilder) -> ProcessMemberSymbolResult? { + private func processMemberSymbol(_ symbolTableRow: UInt32, node: Node, builder: inout NodeStoreBuilder) -> ProcessMemberSymbolResult? { if node.kind == .static, let firstChild = node.children.first, firstChild.kind.isMember { - return processMemberSymbol(pendingSymbol, node: firstChild, traits: [.isStatic], builder: &builder) + return processMemberSymbol(symbolTableRow, node: firstChild, traits: [.isStatic], builder: &builder) } else if node.kind.isMember { - return processMemberSymbol(pendingSymbol, node: node, traits: [], builder: &builder) + return processMemberSymbol(symbolTableRow, node: node, traits: [], builder: &builder) } return nil } - private func processMemberSymbol(_ pendingSymbol: PendingDemangledSymbol, node: Node, traits: MemberKind.Traits, builder: inout NodeStoreBuilder) -> ProcessMemberSymbolResult? { + private func processMemberSymbol(_ symbolTableRow: UInt32, node: Node, traits: MemberKind.Traits, builder: inout NodeStoreBuilder) -> ProcessMemberSymbolResult? { var traits = traits let node = node switch node.kind { @@ -399,27 +428,27 @@ public final class SymbolIndexStore: SharedCache, @unc traits.insert(.inExtension) first = type } - return processMemberSymbol(pendingSymbol, node: first, memberKind: .allocator(inExtension: traits.contains(.inExtension)), builder: &builder) + return processMemberSymbol(symbolTableRow, node: first, memberKind: .allocator(inExtension: traits.contains(.inExtension)), builder: &builder) case .deallocator: guard let first = node.children.first else { return nil } - return processMemberSymbol(pendingSymbol, node: first, memberKind: .deallocator, builder: &builder) + return processMemberSymbol(symbolTableRow, node: first, memberKind: .deallocator, builder: &builder) case .constructor: guard var first = node.children.first else { return nil } if first.kind == .extension, let type = first.children.at(1) { traits.insert(.inExtension) first = type } - return processMemberSymbol(pendingSymbol, node: first, memberKind: .constructor(inExtension: traits.contains(.inExtension)), builder: &builder) + return processMemberSymbol(symbolTableRow, node: first, memberKind: .constructor(inExtension: traits.contains(.inExtension)), builder: &builder) case .destructor: guard let first = node.children.first else { return nil } - return processMemberSymbol(pendingSymbol, node: first, memberKind: .destructor, builder: &builder) + return processMemberSymbol(symbolTableRow, node: first, memberKind: .destructor, builder: &builder) case .function: guard var first = node.children.first else { return nil } if first.kind == .extension, let type = first.children.at(1) { traits.insert(.inExtension) first = type } - return processMemberSymbol(pendingSymbol, node: first, memberKind: .function(inExtension: traits.contains(.inExtension), isStatic: traits.contains(.isStatic)), builder: &builder) + return processMemberSymbol(symbolTableRow, node: first, memberKind: .function(inExtension: traits.contains(.inExtension), isStatic: traits.contains(.isStatic)), builder: &builder) case .variable: // Stored variable reached directly (not through getter/setter) traits.insert(.isStorage) @@ -429,7 +458,7 @@ public final class SymbolIndexStore: SharedCache, @unc first = type } if let first { - return processMemberSymbol(pendingSymbol, node: first, memberKind: .variable(inExtension: traits.contains(.inExtension), isStatic: traits.contains(.isStatic), isStorage: traits.contains(.isStorage)), builder: &builder) + return processMemberSymbol(symbolTableRow, node: first, memberKind: .variable(inExtension: traits.contains(.inExtension), isStatic: traits.contains(.isStatic), isStorage: traits.contains(.isStorage)), builder: &builder) } case .getter, .setter: @@ -438,13 +467,13 @@ public final class SymbolIndexStore: SharedCache, @unc traits.insert(.inExtension) first = type } - return processMemberSymbol(pendingSymbol, node: first, memberKind: .variable(inExtension: traits.contains(.inExtension), isStatic: traits.contains(.isStatic), isStorage: traits.contains(.isStorage)), builder: &builder) + return processMemberSymbol(symbolTableRow, node: first, memberKind: .variable(inExtension: traits.contains(.inExtension), isStatic: traits.contains(.isStatic), isStorage: traits.contains(.isStorage)), builder: &builder) } else if let subscriptNode = node.children.first, subscriptNode.kind == .subscript, var first = subscriptNode.children.first { if first.kind == .extension, let type = first.children.at(1) { traits.insert(.inExtension) first = type } - return processMemberSymbol(pendingSymbol, node: first, memberKind: .subscript(inExtension: traits.contains(.inExtension), isStatic: traits.contains(.isStatic)), builder: &builder) + return processMemberSymbol(symbolTableRow, node: first, memberKind: .subscript(inExtension: traits.contains(.inExtension), isStatic: traits.contains(.isStatic)), builder: &builder) } default: break @@ -452,14 +481,14 @@ public final class SymbolIndexStore: SharedCache, @unc return nil } - private func processMemberSymbol(_ pendingSymbol: PendingDemangledSymbol, node: Node, memberKind: MemberKind, builder: inout NodeStoreBuilder) -> ProcessMemberSymbolResult? { + private func processMemberSymbol(_ symbolTableRow: UInt32, node: Node, memberKind: MemberKind, builder: inout NodeStoreBuilder) -> ProcessMemberSymbolResult? { if let typeKind = node.kind.typeKind { // The transient `.type` wrapper exists only for printing; the // arena-resident wrapper is built directly from the interned // context node's index, so no class tree survives this call. let typeName = Node.create(kind: .type, child: node).print(using: .interfaceTypeBuilderOnly) let typeNodeIndex = builder.intern(kind: .type, children: [builder.intern(node)]) - return .init(memberKind: memberKind, typeName: typeName, typeNodeIndex: typeNodeIndex, typeInfo: .init(name: typeName, kind: typeKind), pendingSymbol: pendingSymbol) + return .init(memberKind: memberKind, typeName: typeName, typeNodeIndex: typeNodeIndex, typeInfo: .init(name: typeName, kind: typeKind), symbolTableRow: symbolTableRow) } return nil } @@ -527,21 +556,21 @@ public final class SymbolIndexStore: SharedCache, @unc fileprivate struct ProcessGlobalSymbolResult: Sendable { let kind: GlobalKind - let pendingSymbol: PendingDemangledSymbol + let symbolTableRow: UInt32 } - private func processGlobalSymbol(_ pendingSymbol: PendingDemangledSymbol, node: Node) -> ProcessGlobalSymbolResult? { + private func processGlobalSymbol(_ symbolTableRow: UInt32, node: Node) -> ProcessGlobalSymbolResult? { switch node.kind { case .function: - return .init(kind: .function, pendingSymbol: pendingSymbol) + return .init(kind: .function, symbolTableRow: symbolTableRow) case .variable: // When we reach .variable directly (not through getter/setter), // this is a stored variable declaration - return .init(kind: .variable(isStorage: true), pendingSymbol: pendingSymbol) + return .init(kind: .variable(isStorage: true), symbolTableRow: symbolTableRow) case .getter, .setter: if let variableNode = node.children.first, variableNode.kind == .variable { - return processGlobalSymbol(pendingSymbol, node: variableNode) + return processGlobalSymbol(symbolTableRow, node: variableNode) } default: break @@ -550,19 +579,13 @@ public final class SymbolIndexStore: SharedCache, @unc } public func allSymbols(in machO: MachO) -> [DemangledSymbol] { - if let symbols = storage(in: machO)?.symbolsByKind.values.flatMap({ $0 }) { - return symbols - } else { - return [] - } + guard let storage = storage(in: machO) else { return [] } + return storage.symbolRowsByKind.values.flatMap { storage.demangledSymbols(atRows: $0) } } public func symbolsByKind(in machO: MachO) -> OrderedDictionary { - if let symbols = storage(in: machO)?.symbolsByKind { - return symbols.mapValues { $0 } - } else { - return [:] - } + guard let storage = storage(in: machO) else { return [:] } + return storage.symbolRowsByKind.mapValues { storage.demangledSymbols(atRows: $0) } } public func typeInfo(for name: String, in machO: MachO) -> TypeInfo? { @@ -570,7 +593,8 @@ public final class SymbolIndexStore: SharedCache, @unc } public func symbols(of kinds: Node.Kind..., in machO: MachO) -> [DemangledSymbol] { - return kinds.map { storage(in: machO)?.symbolsByKind[$0] ?? [] }.reduce(into: []) { $0 += $1 } + guard let storage = storage(in: machO) else { return [] } + return kinds.map { storage.demangledSymbols(atRows: storage.symbolRowsByKind[$0] ?? []) }.reduce(into: []) { $0 += $1 } } /// Returns the pre-extracted thunk-attribute members whose parent type @@ -586,32 +610,44 @@ public final class SymbolIndexStore: SharedCache, @unc } public func memberSymbols(of kinds: MemberKind..., in machO: MachO) -> [DemangledSymbol] { - return kinds.map { storage(in: machO)?.memberSymbolsByKind[$0]?.values.flatMap { $0.values.flatMap { $0 } } ?? [] }.reduce(into: []) { $0 += $1 } + guard let storage = storage(in: machO) else { return [] } + return kinds.map { kind -> [DemangledSymbol] in + guard let memberRows = storage.memberSymbolRowsByKind[kind] else { return [] } + return memberRows.values.flatMap { rowsByTypeNodeIndex in + rowsByTypeNodeIndex.values.flatMap { storage.demangledSymbols(atRows: $0) } + } + }.reduce(into: []) { $0 += $1 } } public func memberSymbols(of kinds: MemberKind..., for name: String, in machO: MachO) -> [DemangledSymbol] { - return kinds.map { storage(in: machO)?.memberSymbolsByKind[$0]?[name]?.values.flatMap { $0 } ?? [] }.reduce(into: []) { $0 += $1 } + guard let storage = storage(in: machO) else { return [] } + return kinds.map { kind -> [DemangledSymbol] in + guard let rowsByTypeNodeIndex = storage.memberSymbolRowsByKind[kind]?[name] else { return [] } + return rowsByTypeNodeIndex.values.flatMap { storage.demangledSymbols(atRows: $0) } + }.reduce(into: []) { $0 += $1 } } public func memberSymbols(of kinds: MemberKind..., for name: String, node: Node, in machO: MachO) -> [DemangledSymbol] { // Callers hold an externally demangled `Node` (MetadataReader context - // demangling), while keys are `NodeReference`s into the frozen store. + // demangling), while keys are node indexes into the frozen store. // The type-name bucket holds at most a handful of type nodes, so a // structural walk per key is cheap. + guard let storage = storage(in: machO) else { return [] } return kinds.map { kind -> [DemangledSymbol] in - guard let symbolsByTypeNode = storage(in: machO)?.memberSymbolsByKind[kind]?[name] else { return [] } - return symbolsByTypeNode.elements.first(where: { $0.key.structurallyEquals(node) })?.value ?? [] + guard let rowsByTypeNodeIndex = storage.memberSymbolRowsByKind[kind]?[name] else { return [] } + guard let matched = rowsByTypeNodeIndex.elements.first(where: { storage.nodeStore.reference(at: $0.key).structurallyEquals(node) }) else { return [] } + return storage.demangledSymbols(atRows: matched.value) }.reduce(into: []) { $0 += $1 } } public func memberSymbols(of kinds: MemberKind..., excluding names: borrowing Set, in machO: MachO) -> OrderedDictionary> { - let filtered: OrderedDictionary = kinds.reduce(into: [:]) { $0[$1] = storage(in: machO)?.memberSymbolsByKind[$1]?.filter { !names.contains($0.key) } ?? [:] } - + guard let storage = storage(in: machO) else { return [:] } var result: OrderedDictionary> = [:] - for (kind, memberSymbols) in filtered { - for (_, symbols) in memberSymbols { - for (node, symbols) in symbols { - result[node, default: [:]][kind, default: []].append(contentsOf: symbols) + for kind in kinds { + guard let memberRows = storage.memberSymbolRowsByKind[kind] else { continue } + for (typeName, rowsByTypeNodeIndex) in memberRows where !names.contains(typeName) { + for (typeNodeIndex, rows) in rowsByTypeNodeIndex { + result[storage.nodeStore.reference(at: typeNodeIndex), default: [:]][kind, default: []].append(contentsOf: storage.demangledSymbols(atRows: rows)) } } } @@ -619,29 +655,54 @@ public final class SymbolIndexStore: SharedCache, @unc } public func methodDescriptorMemberSymbols(of kinds: MemberKind..., in machO: MachO) -> [DemangledSymbol] { - return kinds.map { storage(in: machO)?.methodDescriptorMemberSymbolsByKind[$0]?.values.flatMap { $0.values.flatMap { $0 } } ?? [] }.reduce(into: []) { $0 += $1 } + guard let storage = storage(in: machO) else { return [] } + return kinds.map { kind -> [DemangledSymbol] in + guard let memberRows = storage.methodDescriptorMemberSymbolRowsByKind[kind] else { return [] } + return memberRows.values.flatMap { rowsByTypeNodeIndex in + rowsByTypeNodeIndex.values.flatMap { storage.demangledSymbols(atRows: $0) } + } + }.reduce(into: []) { $0 += $1 } } public func methodDescriptorMemberSymbols(of kinds: MemberKind..., for name: String, in machO: MachO) -> [DemangledSymbol] { - return kinds.map { storage(in: machO)?.methodDescriptorMemberSymbolsByKind[$0]?[name]?.values.flatMap { $0 } ?? [] }.reduce(into: []) { $0 += $1 } + guard let storage = storage(in: machO) else { return [] } + return kinds.map { kind -> [DemangledSymbol] in + guard let rowsByTypeNodeIndex = storage.methodDescriptorMemberSymbolRowsByKind[kind]?[name] else { return [] } + return rowsByTypeNodeIndex.values.flatMap { storage.demangledSymbols(atRows: $0) } + }.reduce(into: []) { $0 += $1 } } public func protocolWitnessMemberSymbols(of kinds: MemberKind..., in machO: MachO) -> [DemangledSymbol] { - return kinds.map { storage(in: machO)?.protocolWitnessMemberSymbolsByKind[$0]?.values.flatMap { $0.values.flatMap { $0 } } ?? [] }.reduce(into: []) { $0 += $1 } + guard let storage = storage(in: machO) else { return [] } + return kinds.map { kind -> [DemangledSymbol] in + guard let memberRows = storage.protocolWitnessMemberSymbolRowsByKind[kind] else { return [] } + return memberRows.values.flatMap { rowsByTypeNodeIndex in + rowsByTypeNodeIndex.values.flatMap { storage.demangledSymbols(atRows: $0) } + } + }.reduce(into: []) { $0 += $1 } } public func protocolWitnessMemberSymbols(of kinds: MemberKind..., for name: String, in machO: MachO) -> [DemangledSymbol] { - return kinds.map { storage(in: machO)?.protocolWitnessMemberSymbolsByKind[$0]?[name]?.values.flatMap { $0 } ?? [] }.reduce(into: []) { $0 += $1 } + guard let storage = storage(in: machO) else { return [] } + return kinds.map { kind -> [DemangledSymbol] in + guard let rowsByTypeNodeIndex = storage.protocolWitnessMemberSymbolRowsByKind[kind]?[name] else { return [] } + return rowsByTypeNodeIndex.values.flatMap { storage.demangledSymbols(atRows: $0) } + }.reduce(into: []) { $0 += $1 } } public func globalSymbols(of kinds: GlobalKind..., in machO: MachO) -> [DemangledSymbol] { - return kinds.map { storage(in: machO)?.globalSymbolsByKind[$0] ?? [] }.reduce(into: []) { $0 += $1 } + guard let storage = storage(in: machO) else { return [] } + return kinds.map { storage.demangledSymbols(atRows: storage.globalSymbolRowsByKind[$0] ?? []) }.reduce(into: []) { $0 += $1 } } public func allOpaqueTypeDescriptorSymbols(in machO: MachO) -> OrderedDictionary? { - return storage(in: machO)?.opaqueTypeDescriptorSymbolByNode.mapValues { - return $0 + guard let storage = storage(in: machO) else { return nil } + var result: OrderedDictionary = [:] + for (nodeIndex, row) in storage.opaqueTypeDescriptorSymbolRowByNodeIndex { + guard let demangledSymbol = storage.demangledSymbol(atRow: row) else { continue } + result[storage.nodeStore.reference(at: nodeIndex)] = demangledSymbol } + return result } public func opaqueTypeDescriptorSymbol(for node: Node, in machO: MachO) -> DemangledSymbol? { @@ -649,15 +710,14 @@ public final class SymbolIndexStore: SharedCache, @unc // frozen store. Structural comparison early-outs on the first // mismatching kind, so the linear scan stays cheap relative to the // printing work that triggers it. - return storage(in: machO)?.opaqueTypeDescriptorSymbolByNode.elements.first(where: { $0.key.structurallyEquals(node) })?.value + guard let storage = storage(in: machO) else { return nil } + guard let matched = storage.opaqueTypeDescriptorSymbolRowByNodeIndex.elements.first(where: { storage.nodeStore.reference(at: $0.key).structurallyEquals(node) }) else { return nil } + return storage.demangledSymbol(atRow: matched.value) } package func symbols(for offset: Int, in machO: MachO) -> Symbols? { - if let symbols = storage(in: machO)?.symbolsByOffset[offset], !symbols.isEmpty { - return .init(offset: offset, symbols: symbols) - } else { - return nil - } + guard let storage = storage(in: machO), let rows = storage.symbolRowsByOffset[offset], !rows.isEmpty else { return nil } + return .init(offset: offset, symbols: rows.map { storage.symbol(atRow: $0, offset: offset) }) } /// Store-backed handle for a symbol's demangled tree. Hits the frozen @@ -666,8 +726,10 @@ public final class SymbolIndexStore: SharedCache, @unc /// every caller receives a uniform `NodeReference`. package func demangledNodeReference(for symbol: Symbol, in machO: MachO) -> NodeReference? { guard let cacheStorage = storage(in: machO) else { return nil } - if let reference = cacheStorage.demangledNodeBySymbol[symbol] { - return reference + if let row = cacheStorage.tableRowByName[symbol.name], + cacheStorage.symbolTable[Int(row)].offset == symbol.offset, + let rootNodeIndex = cacheStorage.rootNodeIndexByTableRow[Int(row)] { + return cacheStorage.nodeStore.reference(at: rootNodeIndex) } if let reference = cacheStorage.lateDemangledNodeBySymbol[symbol] { return reference @@ -781,12 +843,6 @@ extension DemanglingNode where Self: Sequence { } } -extension Symbol { - package var isExternal: Bool { - nlist?.isExternal ?? false - } -} - extension NlistProtocol { package var isExternal: Bool { guard let flags = flags, let type = flags.type else { return false } diff --git a/Tests/IntegrationTests/MachOSymbols/SymbolIndexStoreBaselineTests.swift b/Tests/IntegrationTests/MachOSymbols/SymbolIndexStoreBaselineTests.swift index 742453a6..d4282f22 100644 --- a/Tests/IntegrationTests/MachOSymbols/SymbolIndexStoreBaselineTests.swift +++ b/Tests/IntegrationTests/MachOSymbols/SymbolIndexStoreBaselineTests.swift @@ -35,36 +35,37 @@ final class SymbolIndexStoreBaselineTests: MachOImageTests { do { let storage = try #require(builtStorage) - let demangledSymbolCount = storage.demangledNodeBySymbol.count - let symbolsByKindEntryCount = storage.symbolsByKind.values.reduce(0) { $0 + $1.count } - let memberEntryCount = storage.memberSymbolsByKind.values.reduce(0) { partialResult, memberSymbols in - partialResult + memberSymbols.values.reduce(0) { $0 + $1.values.reduce(0) { $0 + $1.count } } + let symbolTableRowCount = storage.symbolTable.count + let demangledSymbolCount = storage.rootNodeIndexByTableRow.count(where: { $0 != nil }) + let symbolsByKindEntryCount = storage.symbolRowsByKind.values.reduce(0) { $0 + $1.count } + let memberEntryCount = storage.memberSymbolRowsByKind.values.reduce(0) { partialResult, memberRows in + partialResult + memberRows.values.reduce(0) { $0 + $1.values.reduce(0) { $0 + $1.count } } } - let methodDescriptorEntryCount = storage.methodDescriptorMemberSymbolsByKind.values.reduce(0) { partialResult, memberSymbols in - partialResult + memberSymbols.values.reduce(0) { $0 + $1.values.reduce(0) { $0 + $1.count } } + let methodDescriptorEntryCount = storage.methodDescriptorMemberSymbolRowsByKind.values.reduce(0) { partialResult, memberRows in + partialResult + memberRows.values.reduce(0) { $0 + $1.values.reduce(0) { $0 + $1.count } } } - let protocolWitnessEntryCount = storage.protocolWitnessMemberSymbolsByKind.values.reduce(0) { partialResult, memberSymbols in - partialResult + memberSymbols.values.reduce(0) { $0 + $1.values.reduce(0) { $0 + $1.count } } + let protocolWitnessEntryCount = storage.protocolWitnessMemberSymbolRowsByKind.values.reduce(0) { partialResult, memberRows in + partialResult + memberRows.values.reduce(0) { $0 + $1.values.reduce(0) { $0 + $1.count } } } - let globalEntryCount = storage.globalSymbolsByKind.values.reduce(0) { $0 + $1.count } + let globalEntryCount = storage.globalSymbolRowsByKind.values.reduce(0) { $0 + $1.count } let nodeStoreBytes = storage.nodeStore.storageByteCount let nodeStoreNodeCount = storage.nodeStore.nodeCount - print("====== NodeStore migration Stage 0 baseline (\(Self.imageName)) ======") + print("====== NodeStore migration baseline metrics (\(Self.imageName)) ======") print("build time : \(buildDuration)") print("phys_footprint delta : \((footprintAfter - footprintBefore) / 1_048_576) MB (\(footprintBefore / 1_048_576) -> \(footprintAfter / 1_048_576))") print("NodeCache leaf delta : \(leafCacheCountAfter - leafCacheCountBefore) (\(leafCacheCountBefore) -> \(leafCacheCountAfter))") print("NodeCache subtree delta : \(subtreeCacheCountAfter - subtreeCacheCountBefore) (\(subtreeCacheCountBefore) -> \(subtreeCacheCountAfter))") print("nodeStore storage : \(nodeStoreBytes / 1_048_576) MB (\(nodeStoreNodeCount) unique nodes)") - print("demangledNodeBySymbol entries : \(demangledSymbolCount)") + print("symbolTable rows : \(symbolTableRowCount) (stride \(MemoryLayout.stride) B, demangled \(demangledSymbolCount))") print("symbolsByKind entries : \(symbolsByKindEntryCount)") print("memberSymbols entries : \(memberEntryCount)") print("methodDescriptorMember entries : \(methodDescriptorEntryCount)") print("protocolWitnessMember entries : \(protocolWitnessEntryCount)") print("globalSymbols entries : \(globalEntryCount)") - print("symbolsByOffset entries : \(storage.symbolsByOffset.count)") - print("opaqueTypeDescriptor entries : \(storage.opaqueTypeDescriptorSymbolByNode.count)") + print("symbolsByOffset entries : \(storage.symbolRowsByOffset.count)") + print("opaqueTypeDescriptor entries : \(storage.opaqueTypeDescriptorSymbolRowByNodeIndex.count)") print("typeInfoByName entries : \(storage.typeInfoByName.count)") print("=====================================================================") diff --git a/Tests/MachOSymbolsTests/SymbolIndexStoreFixtureTests.swift b/Tests/MachOSymbolsTests/SymbolIndexStoreFixtureTests.swift index 7702eda4..e36ce26f 100644 --- a/Tests/MachOSymbolsTests/SymbolIndexStoreFixtureTests.swift +++ b/Tests/MachOSymbolsTests/SymbolIndexStoreFixtureTests.swift @@ -10,11 +10,11 @@ import MachOFixtureSupport /// `SymbolTestsCore` framework (self-built fixture, no external Xcode /// dependency). Complements the heavyweight integration/baseline tests: /// these assert the NodeStore-backed pipeline's invariants — cache-free -/// building, byte-identical printing versus the `Node` pipeline, and the -/// `structurallyEquals` bridge behind every `Node`-taking query API. +/// building, byte-identical printing versus the `Node` pipeline, the +/// `structurallyEquals` bridge behind every `Node`-taking query API, and +/// the Stage 3 flat-symbol-table row indirection. /// -/// Serialized: the NodeCache-growth test snapshots process-global counters, -/// and several tests share the cached per-file storage. +/// Serialized: several tests share the cached per-file storage. @Suite(.serialized) final class SymbolIndexStoreFixtureTests: MachOFileTests, @unchecked Sendable { override class var fileName: MachOFileName { .SymbolTestsCore } @@ -41,9 +41,10 @@ final class SymbolIndexStoreFixtureTests: MachOFileTests, @unchecked Sendable { /// `SymbolIndexStoreBaselineTests`.) @Test func buildPipelineStaysOffGlobalNodeCache() throws { let builtStorage = try #require(SymbolIndexStore.shared.buildStorage(for: machOFile)) - #expect(!builtStorage.demangledNodeBySymbol.isEmpty) + #expect(!builtStorage.symbolTable.isEmpty) - let sampleSymbolName = try #require(builtStorage.demangledNodeBySymbol.keys.first?.name) + let sampleRow = try #require(builtStorage.rootNodeIndexByTableRow.firstIndex(where: { $0 != nil })) + let sampleSymbolName = builtStorage.symbolTable[sampleRow].name let firstTransientTree = try demangleAsNodeTransient(sampleSymbolName) let secondTransientTree = try demangleAsNodeTransient(sampleSymbolName) let firstLeaf = try #require(firstTransientTree.first { $0.children.isEmpty }) @@ -56,8 +57,11 @@ final class SymbolIndexStoreFixtureTests: MachOFileTests, @unchecked Sendable { /// the classic `demangleAsNode` + `Node.print` pipeline. @Test func printedSymbolsMatchNodePipeline() throws { let storage = try storage + var checkedCount = 0 var mismatchCount = 0 - for (symbol, reference) in storage.demangledNodeBySymbol { + for (row, symbol) in storage.symbolTable.enumerated() { + guard let rootNodeIndex = storage.rootNodeIndexByTableRow[row] else { continue } + let reference = storage.nodeStore.reference(at: rootNodeIndex) let expected = try demangleAsNode(symbol.name, internsSubtrees: false).print(using: .default) if reference.print(using: .default) != expected { mismatchCount += 1 @@ -65,25 +69,56 @@ final class SymbolIndexStoreFixtureTests: MachOFileTests, @unchecked Sendable { Issue.record("Store print mismatch for \(symbol.name)") } } + checkedCount += 1 } #expect(mismatchCount == 0) - #expect(!storage.demangledNodeBySymbol.isEmpty) + #expect(checkedCount > 0) + } + + // MARK: - Flat symbol table (Stage 3) + + /// The whole point of the Stage 3 compaction: vended values stay small. + /// `Symbol` drops the 40-byte `nlist` existential; `DemangledSymbol` + /// stores a shared-table row instead of an inline `Symbol` copy. + @Test func compactValueLayouts() { + #expect(MemoryLayout.stride <= 32) + #expect(MemoryLayout.stride <= 32) + } + + /// Raw and cache-adjusted offset keys share one canonical table row, so + /// `symbols(for:in:)` must rebuild each `Symbol` with the queried offset + /// (matching the old per-offset-copy behavior byte for byte). + @Test func offsetQueriesRebuildSymbolsWithQueriedOffset() throws { + let storage = try storage + #expect(!storage.symbolRowsByOffset.isEmpty) + var checkedOffsetCount = 0 + for (offset, rows) in storage.symbolRowsByOffset { + guard checkedOffsetCount < 500 else { break } + let queried = try #require(SymbolIndexStore.shared.symbols(for: offset, in: machOFile)) + #expect(queried.count == rows.count) + #expect(queried.allSatisfy { $0.offset == offset }) + for (queriedSymbol, row) in zip(queried, rows) { + #expect(queriedSymbol.name == storage.symbolTable[Int(row)].name) + } + checkedOffsetCount += 1 + } + #expect(checkedOffsetCount > 0) } // MARK: - Query APIs /// `memberSymbols(of:for:node:)` takes an externally demangled `Node` and - /// must find the `NodeReference`-keyed bucket via `structurallyEquals`. + /// must find the node-index-keyed bucket via `structurallyEquals`. /// Exercise it for every bucket the index actually built. @Test func memberQueryByNodeFindsEveryBucket() throws { let storage = try storage var checkedBucketCount = 0 - for (memberKind, memberSymbols) in storage.memberSymbolsByKind { - for (typeName, symbolsByTypeNode) in memberSymbols { - for (typeNodeReference, expectedSymbols) in symbolsByTypeNode { - let externalNode = typeNodeReference.materialize() + for (memberKind, memberRows) in storage.memberSymbolRowsByKind { + for (typeName, rowsByTypeNodeIndex) in memberRows { + for (typeNodeIndex, expectedRows) in rowsByTypeNodeIndex { + let externalNode = storage.nodeStore.reference(at: typeNodeIndex).materialize() let queried = SymbolIndexStore.shared.memberSymbols(of: memberKind, for: typeName, node: externalNode, in: machOFile) - #expect(queried.count == expectedSymbols.count, "bucket \(memberKind) / \(typeName)") + #expect(queried.count == expectedRows.count, "bucket \(memberKind) / \(typeName)") checkedBucketCount += 1 } } @@ -93,10 +128,10 @@ final class SymbolIndexStoreFixtureTests: MachOFileTests, @unchecked Sendable { @Test func symbolKindQueriesMatchStorageBuckets() throws { let storage = try storage - #expect(!storage.symbolsByKind.isEmpty) - for (kind, expectedSymbols) in storage.symbolsByKind { + #expect(!storage.symbolRowsByKind.isEmpty) + for (kind, expectedRows) in storage.symbolRowsByKind { let queried = SymbolIndexStore.shared.symbols(of: kind, in: machOFile) - #expect(queried.count == expectedSymbols.count) + #expect(queried.count == expectedRows.count) #expect(queried.allSatisfy { $0.demangledNode.children.first?.kind == kind }) } } @@ -112,9 +147,10 @@ final class SymbolIndexStoreFixtureTests: MachOFileTests, @unchecked Sendable { @Test func opaqueDescriptorQueryFindsEveryReferenceKey() throws { let storage = try storage - for (keyReference, expectedSymbol) in storage.opaqueTypeDescriptorSymbolByNode { + for (nodeIndex, expectedRow) in storage.opaqueTypeDescriptorSymbolRowByNodeIndex { + let keyReference = storage.nodeStore.reference(at: nodeIndex) let queried = try #require(SymbolIndexStore.shared.opaqueTypeDescriptorSymbol(for: keyReference.materialize(), in: machOFile)) - #expect(queried.symbol == expectedSymbol.symbol) + #expect(queried.symbol == storage.symbolTable[Int(expectedRow)]) } } @@ -123,8 +159,10 @@ final class SymbolIndexStoreFixtureTests: MachOFileTests, @unchecked Sendable { @Test func demangledNodeAndReferenceAgree() throws { let storage = try storage var checkedCount = 0 - for (symbol, reference) in storage.demangledNodeBySymbol { + for (row, symbol) in storage.symbolTable.enumerated() { guard checkedCount < 200 else { break } + guard let rootNodeIndex = storage.rootNodeIndexByTableRow[row] else { continue } + let reference = storage.nodeStore.reference(at: rootNodeIndex) let materialized = try #require(SymbolIndexStore.shared.demangledNode(for: symbol, in: machOFile)) #expect(reference.structurallyEquals(materialized)) let referenceAgain = try #require(SymbolIndexStore.shared.demangledNodeReference(for: symbol, in: machOFile)) From 42821a95116a60741d8cc8a447ae3cbfb75fb126 Mon Sep 17 00:00:00 2001 From: Mx-Iris Date: Fri, 24 Jul 2026 17:40:43 +0800 Subject: [PATCH 04/77] docs: add Stage 5 proposal (declaration-layer zero materialization) to NodeStore migration plan --- .../Internal/NodeStoreMigrationPlan.md | 65 ++++++++++++++++++- 1 file changed, 64 insertions(+), 1 deletion(-) diff --git a/Documentations/Internal/NodeStoreMigrationPlan.md b/Documentations/Internal/NodeStoreMigrationPlan.md index 7485b7c4..a23af691 100644 --- a/Documentations/Internal/NodeStoreMigrationPlan.md +++ b/Documentations/Internal/NodeStoreMigrationPlan.md @@ -1,6 +1,6 @@ # NodeStore 迁移计划（SymbolIndexStore → arena 存储） -- **状态**: Completed（Stage 0–4 全部落地，见文末「实施记录」） +- **状态**: Stage 0–4 Completed（见「实施记录」）；Stage 5 提案待批准（见文末「Stage 5 提案」） - **日期**: 2026-07-24 - **最后更新**: 2026-07-24 - **分支**: `feature/node-store-migration`（worktree `.claude/worktrees/node-store-migration`，Demangling 经主检出 `.claude/worktrees/swift-demangling` 处的**真实 git worktree**（swift-demangling `feature/node-store`）以路径依赖解析——原先的符号链接方案因目标 worktree 被外部清理导致 SwiftPM manifest 缓存把解析钉回 remote，已改为本仓库领地内的 worktree） @@ -152,3 +152,66 @@ RuntimeViewer 与 MachOSwiftSection 的内存主项在 `MachOSymbols/SymbolIndex - 构建耗时的下降来自：populate 转换 pass 删除、每符号双份 `Symbol` 构造与 existential 装箱消失、索引累积只搬 4B 行号。 - 构建期增量从 302 MB 收敛到 68 MB：双索引瞬态窗口消除 + Symbol 复本/existential 盒清零是主贡献；残余 68 MB 为 NodeStore + 表 + 索引 + malloc 未归还页。 - 验收测试：`SymbolTestsCore` 快照 60/60 逐字节一致 + `SymbolIndexStoreFixtureTests` 10/10 + `SharedCacheTests` 全绿（**79 tests / 5 suites passed**，Stage 3 代码 + 本地 feature 分支解析口径复跑确认）。 + +## Stage 5 提案 — 声明层零物化（Definition/富文本打印迁移到 `NodeReference`） + +- **状态**: 提案，待批准 +- **日期**: 2026-07-24 +- **动机**: Stage 0–4 落地后 RuntimeViewer 实测 `Node` 实例 1,091,575 → 336,095、进程内存 628.5 MB → 478.6 MB。布局侧已无肉可挖（实测：`Payload` 枚举 17 B（最大 case 16 B + 判别位无空位可藏）、`kind` 2 B、对象头 16 B → 实例 41 B → malloc 桶 48 B；手工位压缩到 32 B 桶收益仅 ~5.4 MB 且需放弃安全枚举，不做）。剩余 33.6 万实例的**来源**才是下一刀。 + +### 剩余 `Node` 的来源盘点（2026-07-24 调研） + +1. **Definition/Name 值类型长期持有物化树**（主项）：`VariableDefinition.node`、`FunctionDefinition.node`、`SubscriptDefinition.node`、`FieldDefinition.typeNode`、`ExtensionDefinition.genericSignature`、`TypeName.node`、`ProtocolName.node`、`ExtensionName.node`（`DefinitionName` 协议要求 `var node: Node`）。`DefinitionBuilder` 与 `SwiftDeclarationIndexer` 在构建处逐一 `.materialize()`——store 里本来就有的共享子树被展开成独立 class 树，随声明缓存驻留。 +2. **仍走 interning `demangleAsNode` 的散点**：`MetadataReader`（symbolic reference 密集）、`RuntimeFieldLayoutBackend`、`TypedDumper`、`ClassHierarchyDumper`、`Symbol.demangledNode`。这些把规范树**永久钉进全局 `NodeCache.shared`**，镜像关闭也不回收。 +3. **打印桥接物化**：`SwiftDump` 各 dumper 的 `demangleResolver.resolve(for: node.materialize())`（瞬态，但高频）。 + +### 前置（5·0）— swift-demangling 侧小改 + +`DemanglingPrinter` 富 target 走 store 时唯一的语义缺口在 `NodePrinter.swift:198`：`target.pushTypeReferenceScope(name as? Node)` 对 `NodeReference` 恒传 `nil`，富 target 的 type-reference 身份作用域（`SemanticString` 用它 remangle 出 identifier scope）丢失。修法： + +- `NodePrinterTarget.pushTypeReferenceScope(_ node: Node?)` → `pushTypeReferenceScope(_ node: @autoclosure () -> Node?)`；引擎侧传 `name.materializedNode`。 +- `@autoclosure` 保证 String target（默认空实现，从不求值）**零成本**——现有 store 零物化打印路径不回退；只有 `SemanticString` 实现真正求值，物化只发生在 nominal 引用节点（子树小，且紧接的 `mangleAsString` 本来就要整棵遍历）。 +- 协议签名变更破坏外部 conformance：已知 conformance 仅 `SemanticString`（MachOSwiftSection 内，我方可控）与库内默认实现，同步改。 +- 守护测试：store 路径 String target 打印全程零 `Node` 分配（现有字节一致快照 + 新增分配哨兵）。 + +### 5a — SwiftDeclaration 值类型换持 `NodeReference` + +- 上列全部 `Node` 存储属性 → `NodeReference`（`DefinitionName` 协议要求同步改）；`OverrideSymbolMatcher` 等取 `typeNode: Node` 参数的内部接口跟随。 +- `DefinitionBuilder` / `SwiftDeclarationIndexer` 删除全部构建期 `.materialize()`——sweep 手里本来就是 `NodeReference`。 +- `NodeReference` 自带对 `NodeStore` 的强引用（16 B 值：store ref + index），生命周期自洽：声明活着 → store 活着；镜像声明缓存整体淘汰 → store 随之整体回收（正是本计划的回收模型）。 +- RuntimeViewer 适配面（feature/node-store-adoption 分支）：`mangleAsString(typeName.node)` 类调用点经既有 `mangleAsString(some DemanglingNode)` 泛型桥**无感**；少数构树点（`wrappedAsType(base.typeName.node)`、`nodesByParameter` 字典等）显式 `.materialize()` 或改存 `NodeReference`。已确认 RV 未使用 `DemangleResolver.builder`。 + +### 5b — SwiftPrinting 富文本引擎泛型化（工作量主体） + +- `NodePrintable` 协议族（`NodePrintable` / `InterfaceNodePrintable` / `TypeNodePrintable` / `FunctionTypeNodePrintable` / `BoundGenericNodePrintable` / `DependentGenericNodePrintable`）+ 4 个具体 printer（Variable/Function/Subscript/Type）：具体 `Node` → `associatedtype SomeNode: DemanglingNode`，模式照抄上游 `DemanglingPrinter` 的泛型化。 +- `printCache: [ObjectIdentifier: Target]`（DAG 记忆化）→ 泛型 memo key：SwiftPrinting 本地协议（`Node` → `ObjectIdentifier(self)`；`NodeReference` → 自身，O(1) `Hashable`）。store 里子树共享以共享 index 存在，memo 命中率不变。 +- `DemangleResolver` 增加 `resolve(for: some DemanglingNode)`：`.options` 走 `DemanglingPrinter` 零物化；`.builder` 维持公开的 `(Node) async throws -> SemanticString` 闭包签名、内部 `materializedNode` 桥（公开 API 不破坏，物化只剩这一条路径）。 +- `NodeReference.printSemantic(using:)`（经 `@_spi(Internals) DemanglingPrinter`，落在 SwiftDeclarationRendering）。 +- `SwiftDump` dumpers：`symbol.demangledNode.materialize()` → 直接传 `NodeReference`。 + +### 5c — 散点 `demangleAsNode` 去钉扎（`NodeCache` 停止增长） + +- 来源盘点第 2 类调用点全部改走 transient 解码（`demangleAsNodeTransient` 已 `@_spi(Internals)` 导出）：树仍是 `Node` 的场合不再钉进全局 cache，随消费方释放。 +- 需要长期持有的 metadata 派生树（field type、runtime generic signature 等，不经 symbol store）：`SwiftDeclarationIndexer` 每次索引 pass 维护一个**辅助 `NodeStoreBuilder`**，transient 树经既有 `builder.intern(_ node: Node)` 灌入，pass 结束 `freeze()`，声明持有辅助 store 的 `NodeReference`。辅助 store 与主 symbol store 不共享去重（frozen store 不可再写），文本/子树少量重复可接受（metadata 树规模远小于符号全集）。 +- 注意 `NodeStoreBuilder` 是 `~Copyable` 非线程安全：intern 段必须收敛在 indexer 的单线程/单 actor 执行段内（现有 sweep 已满足，需在改动中保持）。 + +### 验收 + +- 既有三件套全绿且逐字节一致：`SymbolTestsCore` interface 快照 60/60（该快照走 SwiftInterface → SwiftPrinting 富文本引擎 → 字符串投影，恰好兜住 5b 的文本回归）+ `SymbolIndexStoreFixtureTests` + `SharedCacheTests`。 +- **补语义 token 抽查**：字符串投影盖不住 `pushTypeReferenceScope` 的身份作用域（只影响 token 元数据不影响文本）——对若干典型声明断言 `SemanticString` 的 identifier scope 序列与 `Node` 路径一致。 +- 同口径内存复测：目标 `Node` 常驻 336k → < 5 万（残余应只剩 `.builder` 桥瞬态与 RV 显式物化点）；`NodeCache` 增长维持 0/0 且**存量不再随浏览增长**；进程 footprint 复测记录。 + +### 风险与缓解 + +| 风险 | 缓解 | +|---|---| +| `pushTypeReferenceScope` 签名变更是 swift-demangling 公开协议破坏性改动 | 已知 conformance 全部在我方两仓内；`@autoclosure` 方案在 String 路径零求值，配分配哨兵测试守住零物化不回退 | +| 5b 泛型化触及 ~11 文件的 async mutating printer，回归面大 | interface 快照逐字节兜底 + 语义 token 抽查；分 PR：5·0+5c（小）→ 5a（中）→ 5b（大）→ RV 适配（中），每步独立可验收 | +| `Node` 结构性 `Hashable` 误用作 memo key 导致性能回退 | memo key 协议显式区分：`Node` 用 `ObjectIdentifier`，`NodeReference` 用值本身；review checklist 明确禁止直接 `hash` 泛型节点 | +| 辅助 store 与主 store 文本重复 | 规模评估后可接受；如实测超预期，后续可让 indexer 直接在辅助 builder 上 `demangle(_:)`（省掉 transient `Node` 中转） | +| RV 侧 `typeName.node` 类型变化波及面 | `mangleAsString` 泛型桥无感；其余调用点编译期暴露，逐点 `.materialize()`；RV 在独立 adoption 分支，可整体验证后合入 | + +### 预期收益 + +- `Node` 常驻实例 336k → 数万以内；按 ~70 B/节点全口径（实例 48 B + `.text` String 堆存储 + `manyChildren` 数组缓冲）估算回收 **15–20 MB**，且 `NodeCache` 永久钉扎清零后，长时间浏览不再单调增长。 +- 声明持树成本从 48 B/节点 class 树降为辅助/主 store 的 ~12–14 B/节点 arena + 16 B/根句柄。 From 559b605a8b1f84bf382a53c5e881f8d190d611d4 Mon Sep 17 00:00:00 2001 From: Mx-Iris Date: Fri, 24 Jul 2026 18:37:51 +0800 Subject: [PATCH 05/77] feat(MachOSwiftSection): declaration values hold NodeReference and transient demangling de-pins NodeCache (Stages 5a/5c) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Stage 5c — every unbounded demangle/construction site off the global cache: MetadataReader (28 Node.create sites -> createTransient, 5 demangleAsNode -> demangleAsNodeTransient incl. the symbolic-reference resolver path), RuntimeFieldLayoutBackend, TypedDumper, ClassHierarchyDumper, Symbol.demangledNode, ResolvedTypeReference+, GenericContext+Dump. NodeCache.shared no longer grows while browsing; bounded singletons keep interning. SemanticString adopts the lazy @autoclosure scope hook. Stage 5a — declaration layer holds NodeReference instead of Node: DefinitionName/TypeName/ProtocolName/ExtensionName, Variable/Function/ SubscriptDefinition.node, FieldDefinition.typeNode and ExtensionDefinition.genericSignature. Member definitions reference the per-image symbol store directly (build-time materialize() dropped); the extension-member path passes main-store keys straight through; metadata-derived trees are wrapped via NodeReference(interning:) mini stores, with TypeDefinition.index(in:) batching all field type trees of one type into a single shared store. Name types customize Hashable to structural semantics and keep Codable wire-compatible. ABIKey and the synthesized-member dedup helpers are generic over DemanglingNode; memberSymbols(of:for:node:) gains a NodeReference overload; NodeReference.printSemantic provides zero-materialization rich printing. Five explicit materialize() bridges remain at printer entry points for Stage 5b to remove. Acceptance: 98 tests / 15 suites green, interface snapshot byte-identical. --- AGENTS.md | 2 +- .../Internal/NodeStoreMigrationPlan.md | 8 +++ Sources/MachOSymbols/Symbol.swift | 4 +- Sources/MachOSymbols/SymbolIndexStore.swift | 14 ++++ .../Definitions/DefinitionBuilder.swift | 8 +-- .../Definitions/ExtensionDefinition.swift | 6 +- .../Definitions/FieldDefinition.swift | 2 +- .../Definitions/FunctionDefinition.swift | 2 +- .../Definitions/ProtocolDefinition.swift | 2 +- .../Definitions/SubscriptDefinition.swift | 2 +- .../Definitions/TypeDefinition.swift | 16 +++-- .../Definitions/VariableDefinition.swift | 2 +- .../Components/Names/DefinitionName.swift | 2 +- .../Components/Names/ExtensionName.swift | 39 ++++++++++- .../Components/Names/ProtocolName.swift | 35 +++++++++- .../Components/Names/TypeName.swift | 42 +++++++++++- Sources/SwiftDeclaration/Extensions.swift | 28 ++++---- .../Extensions/GenericContext+Dump.swift | 22 +++--- .../Extensions/Node+.swift | 19 +++++- .../Extensions/ResolvedTypeReference+.swift | 18 ++--- .../RuntimeFieldLayoutBackend.swift | 16 ++--- Sources/SwiftDiffing/ABIKey.swift | 8 +-- Sources/SwiftDump/Protocols/TypedDumper.swift | 4 +- .../SwiftDeclarationIndexer.swift | 33 +++++---- .../ClassHierarchyDumper.swift | 4 +- Sources/SwiftInspection/MetadataReader.swift | 68 +++++++++---------- ...wiftDeclarationPrinter+DiffRendering.swift | 4 +- .../SwiftDeclarationPrinter+Members.swift | 6 +- .../SwiftDeclarationPrinter.swift | 8 +-- .../ConformanceProvider.swift | 2 +- .../GenericSpecializer.swift | 4 +- .../TypeDefinition+Specialization.swift | 6 +- .../TypeAttributeInferrerTests.swift | 20 +++--- Tests/SwiftDiffingTests/ABIDifferTests.swift | 12 ++-- .../ABIExtensionAttributionTests.swift | 12 ++-- .../GenericTypeNameSubstitutionTests.swift | 49 ++++++------- 36 files changed, 340 insertions(+), 189 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index 6455eddd..3cce0c6d 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -206,7 +206,7 @@ let node = try demangleAsNode("$sSiD") // Returns Node tree let string = node.print(using: .default) // "Swift.Int" ``` -**Symbol indexing (NodeStore-backed)**: `SymbolIndexStore.Storage` no longer retains `Node` class trees. The build sweep demangles each symbol cache-free onto a transient tree (`@_spi(Internals) demangleAsNodeTransient`), classifies on that tree, and interns it into a per-image `NodeStoreBuilder`. Nothing touches the global `NodeCache`, and dropping a `Storage` releases the whole per-image footprint. Storage layout (Stage 3): one flat `symbolTable: [Symbol]` row per unique symbol name (canonical cache-adjusted offset; raw and adjusted offset keys share one row), a parallel `rootNodeIndexByTableRow` array, and every classification index stores 4-byte `UInt32` table rows (member/opaque keys are 4-byte `NodeStore.NodeIndex`); indexes accumulate in final row form during the sweep, so `freeze()` is followed by a plain move, not a conversion pass. `Symbol` is 32 bytes (no `nlist` existential — the external-undefined bit is extracted at collection time into `isExternal`), and `DemangledSymbol` is a 32-byte value (shared-table reference + row + `NodeReference`) whose `symbol` is computed; both are asserted by `compactValueLayouts`. Query APIs construct `DemangledSymbol`/`NodeReference` values at the exit boundary; APIs taking an externally demangled `Node` (`memberSymbols(of:for:node:)`, `opaqueTypeDescriptorSymbol(for:)`) match keys via `NodeReference.structurallyEquals(_:)`; renderer boundaries (`demangleResolver.resolve`, `Definition` models) call `materialize()`. Late symbols outside the build sweep get per-symbol mini stores (`demangledNodeReference(for:)` / `MetadataReader.demangleSymbolReference(for:in:)`). Struct-layout changes in `MachOSymbols` need a `swift package clean` rebuild — SwiftPM incremental builds have been observed linking stale downstream objects (runtime SIGSEGV in `outlined destroy`). See `Documentations/Internal/NodeStoreMigrationPlan.md`. +**Symbol indexing (NodeStore-backed)**: `SymbolIndexStore.Storage` no longer retains `Node` class trees. The build sweep demangles each symbol cache-free onto a transient tree (`@_spi(Internals) demangleAsNodeTransient`), classifies on that tree, and interns it into a per-image `NodeStoreBuilder`. Nothing touches the global `NodeCache`, and dropping a `Storage` releases the whole per-image footprint. Storage layout (Stage 3): one flat `symbolTable: [Symbol]` row per unique symbol name (canonical cache-adjusted offset; raw and adjusted offset keys share one row), a parallel `rootNodeIndexByTableRow` array, and every classification index stores 4-byte `UInt32` table rows (member/opaque keys are 4-byte `NodeStore.NodeIndex`); indexes accumulate in final row form during the sweep, so `freeze()` is followed by a plain move, not a conversion pass. `Symbol` is 32 bytes (no `nlist` existential — the external-undefined bit is extracted at collection time into `isExternal`), and `DemangledSymbol` is a 32-byte value (shared-table reference + row + `NodeReference`) whose `symbol` is computed; both are asserted by `compactValueLayouts`. Query APIs construct `DemangledSymbol`/`NodeReference` values at the exit boundary; APIs taking an externally demangled node (`memberSymbols(of:for:node:)` with `Node` or `NodeReference`, `opaqueTypeDescriptorSymbol(for:)`) match keys via `NodeReference.structurallyEquals(_:)` (same-store O(1), cross-store structural walk). Late symbols outside the build sweep get per-symbol mini stores (`demangledNodeReference(for:)` / `MetadataReader.demangleSymbolReference(for:in:)`). Struct-layout changes in `MachOSymbols` need a `swift package clean` rebuild — SwiftPM incremental builds have been observed linking stale downstream objects (runtime SIGSEGV in `outlined destroy`). See `Documentations/Internal/NodeStoreMigrationPlan.md`. **Declaration values hold `NodeReference` (Stage 5a)**: `DefinitionName.node` (`TypeName`/`ProtocolName`/`ExtensionName`), `Variable/Function/SubscriptDefinition.node`, `FieldDefinition.typeNode` and `ExtensionDefinition.genericSignature` are `NodeReference`, not `Node`. Member definitions reference the per-image symbol store directly (a live definition keeps that store alive — the intended per-image recycling model); metadata-derived trees (`MetadataReader` output) are wrapped via `NodeReference(interning:)` mini stores, and `TypeDefinition.index(in:)` batches all field type trees of one type into a single shared store. The `Name` types customize `Hashable` to STRUCTURAL semantics (`structurallyEquals` + `structuralHash`) because `NodeReference`'s intrinsic `Hashable` is store-identity based and would split equal keys minted into different stores; their `Codable` stays wire-compatible by encoding a materialized `Node`. **Transient demangling everywhere else (Stage 5c)**: `MetadataReader`, `RuntimeFieldLayoutBackend`, `TypedDumper`, `ClassHierarchyDumper` and `Symbol.demangledNode` use `demangleAsNodeTransient` + `Node.createTransient`, so the global `NodeCache` no longer grows while browsing. Do NOT reintroduce `demangleAsNode`/`Node.create` on unbounded inputs in those paths. ## Test Environment diff --git a/Documentations/Internal/NodeStoreMigrationPlan.md b/Documentations/Internal/NodeStoreMigrationPlan.md index a23af691..f43a13d4 100644 --- a/Documentations/Internal/NodeStoreMigrationPlan.md +++ b/Documentations/Internal/NodeStoreMigrationPlan.md @@ -215,3 +215,11 @@ RuntimeViewer 与 MachOSwiftSection 的内存主项在 `MachOSymbols/SymbolIndex - `Node` 常驻实例 336k → 数万以内；按 ~70 B/节点全口径（实例 48 B + `.text` String 堆存储 + `manyChildren` 数组缓冲）估算回收 **15–20 MB**，且 `NodeCache` 永久钉扎清零后，长时间浏览不再单调增长。 - 声明持树成本从 48 B/节点 class 树降为辅助/主 store 的 ~12–14 B/节点 arena + 16 B/根句柄。 + +### Stage 5·0 + 5c + 5a 落地（2026-07-24） + +- **5·0（上游）**：`NodePrinterTarget.pushTypeReferenceScope` 改为 `@autoclosure () -> Node?`，引擎侧传 `materializedNode`——String target 从不求值（store 纯文本路径保持零物化），`SemanticString` 惰性求值拿到完整作用域身份。新增 `NodePrinterScopeTests`（双表示作用域序列一致性）。`demangleAsNodeTransient` 增加 `symbolicReferenceResolver` 参数；新增 `@_spi(Internals) Node.createTransient` 工厂族。 +- **5c**：`MetadataReader`（28 处 `create` + 5 处解码）、`RuntimeFieldLayoutBackend`、`TypedDumper`、`ClassHierarchyDumper`、`Symbol.demangledNode`、`ResolvedTypeReference+`、`GenericContext+Dump` 的无界构造/解码全部转 transient——`NodeCache` 停止随浏览增长。有界单例（`firstGenericParamType`、AnyObject 约束）保留 interning。 +- **5a**：声明层全部换持 `NodeReference`。上游新增 `NodeReference(interning:)`、跨 store `structurallyEquals(_ other: NodeReference)`、`structuralHash(into:)`；`memberSymbols(of:for:node:)` 增加 `NodeReference` 重载。成员定义直持主 store 引用（构建期 4 处 `materialize()` 删除）；extension 成员路径的 `ExtensionName`/`genericSignature` 直用主 store 键（原每键物化删除）；metadata 派生树以 mini store 承接，`TypeDefinition.index(in:)` 的字段树按类型批量共享一个 store。`Name` 类型自定义结构语义 `Hashable` 与 wire 兼容 `Codable`。打印边界暂留 5 处显式 `materialize()` 桥（`SwiftDeclarationPrinter` 3 处 printer 入口 + where 子句 + `leafNameNode`×2），5b 泛型化时消除；`NodeReference.printSemantic`（零物化富文本）已就位并接管 `SwiftDiffableInterfaceRenderer`。 +- **验收**：MachOSwiftSection 98 tests / 15 suites 全绿（interface 快照逐字节一致、fixture、diffing、substitution、attribute inference）；swift-demangling 定向 44/7 全绿 + 全量复跑。 +- **事故记录 — 测试语料符号无效 + xcsift 假绿**：`DemanglingTests` corpus 中的 `$s7SwiftUI4TextV_10FoundationE9formatterAcA…` 自引入（5788472，NodeStore 之前）就是**无效符号**（系统 `swift-demangle` 同样拒绝，`TextV` 后多一个 `_`），理应一直红。此前未暴露是因为 `swift test 2>&1 | xcsift; echo $?` 捕获的是 **xcsift 的退出码**而非 `swift test` 的，多轮「全绿」不可信。已替换为真实生成的同复杂度符号 `$s11ExampleBase0A4TextV0A6AddonsE9formatter7subjectAcA0A5StyleV_xtcSyRzlufC`（跨模块 extension + `SyRzl` 约束 + `ufC`），并对三个测试文件的全部 mangled 字面量过系统 demangler 校验。**教训：管道给 xcsift 时用 `${pipestatus[1]}` 取真实退出码，或验收时直接看原生输出。** diff --git a/Sources/MachOSymbols/Symbol.swift b/Sources/MachOSymbols/Symbol.swift index 6574e9db..b07a0ab2 100644 --- a/Sources/MachOSymbols/Symbol.swift +++ b/Sources/MachOSymbols/Symbol.swift @@ -2,7 +2,7 @@ import MachOKit import MachOReading import MachOResolving import MachOKitExtensions -import Demangling +@_spi(Internals) import Demangling import FoundationToolbox public struct Symbol: AsyncResolvable, SymbolProtocol, Hashable, Sendable { @@ -80,7 +80,7 @@ public protocol SymbolProtocol { extension MachOSymbols.SymbolProtocol { public var demangledNode: Node { get throws { - try demangleAsNode(name) + try demangleAsNodeTransient(name) } } } diff --git a/Sources/MachOSymbols/SymbolIndexStore.swift b/Sources/MachOSymbols/SymbolIndexStore.swift index 4cab77f3..27e06540 100644 --- a/Sources/MachOSymbols/SymbolIndexStore.swift +++ b/Sources/MachOSymbols/SymbolIndexStore.swift @@ -640,6 +640,20 @@ public final class SymbolIndexStore: SharedCache, @unc }.reduce(into: []) { $0 += $1 } } + public func memberSymbols(of kinds: MemberKind..., for name: String, node: NodeReference, in machO: MachO) -> [DemangledSymbol] { + // Same lookup as the `Node` overload, for callers holding a + // store-backed reference — possibly minted into a different store + // than the index's own (for example a `TypeName` mini store): + // same-store keys match in O(1) via index equality, cross-store + // keys by a structural walk over the handful of bucket entries. + guard let storage = storage(in: machO) else { return [] } + return kinds.map { kind -> [DemangledSymbol] in + guard let rowsByTypeNodeIndex = storage.memberSymbolRowsByKind[kind]?[name] else { return [] } + guard let matched = rowsByTypeNodeIndex.elements.first(where: { storage.nodeStore.reference(at: $0.key).structurallyEquals(node) }) else { return [] } + return storage.demangledSymbols(atRows: matched.value) + }.reduce(into: []) { $0 += $1 } + } + public func memberSymbols(of kinds: MemberKind..., excluding names: borrowing Set, in machO: MachO) -> OrderedDictionary> { guard let storage = storage(in: machO) else { return [:] } var result: OrderedDictionary> = [:] diff --git a/Sources/SwiftDeclaration/Components/Definitions/DefinitionBuilder.swift b/Sources/SwiftDeclaration/Components/Definitions/DefinitionBuilder.swift index aa694474..710d0cfd 100644 --- a/Sources/SwiftDeclaration/Components/Definitions/DefinitionBuilder.swift +++ b/Sources/SwiftDeclaration/Components/Definitions/DefinitionBuilder.swift @@ -30,7 +30,7 @@ package enum DefinitionBuilder { guard !fieldNames.contains(name) else { continue } let nodes = accessors.map(\.symbol.demangledNode) guard let node = nodes.first(where: { $0.contains(.getter) || !$0.hasAccessor }) else { continue } - var variableDefinition = VariableDefinition(node: node.materialize(), name: name, accessors: accessors, isGlobalOrStatic: isGlobalOrStatic) + var variableDefinition = VariableDefinition(node: node, name: name, accessors: accessors, isGlobalOrStatic: isGlobalOrStatic) if accessors.contains(where: { $0.methodDescriptor?.method?.layout.flags.isDynamic ?? false }) { variableDefinition.attributes.append(.dynamic) } @@ -68,7 +68,7 @@ package enum DefinitionBuilder { for (_, accessors) in accessorsByNode { let nodes = accessors.map(\.symbol.demangledNode) guard let node = nodes.first(where: { $0.contains(.getter) }) else { continue } - var subscriptDefinition = SubscriptDefinition(node: node.materialize(), accessors: accessors, isStatic: isStatic) + var subscriptDefinition = SubscriptDefinition(node: node, accessors: accessors, isStatic: isStatic) if accessors.contains(where: { $0.methodDescriptor?.method?.layout.flags.isDynamic ?? false }) { subscriptDefinition.attributes.append(.dynamic) } @@ -122,7 +122,7 @@ package enum DefinitionBuilder { let symbolOffset = demangledSymbol.base.offset let descriptor = methodDescriptorLookup[node] ?? implOffsetDescriptorLookup[symbolOffset] let vtableOffset = vtableOffsetLookup[node] ?? implOffsetVTableSlotLookup[symbolOffset] - var functionDefinition = FunctionDefinition(node: node.materialize(), name: "", kind: .allocator, symbol: demangledSymbol.base, isGlobalOrStatic: true, methodDescriptor: descriptor, offset: demangledSymbol.offset, vtableOffset: vtableOffset) + var functionDefinition = FunctionDefinition(node: node, name: "", kind: .allocator, symbol: demangledSymbol.base, isGlobalOrStatic: true, methodDescriptor: descriptor, offset: demangledSymbol.offset, vtableOffset: vtableOffset) if let methodDescriptor = descriptor?.method, methodDescriptor.layout.flags.isDynamic { functionDefinition.attributes.append(.dynamic) } @@ -180,7 +180,7 @@ package enum DefinitionBuilder { let symbolOffset = demangledSymbol.base.offset let descriptor = methodDescriptorLookup[node] ?? implOffsetDescriptorLookup[symbolOffset] let vtableOffset = vtableOffsetLookup[node] ?? implOffsetVTableSlotLookup[symbolOffset] - var functionDefinition = FunctionDefinition(node: node.materialize(), name: name, kind: .function, symbol: demangledSymbol.base, isGlobalOrStatic: isGlobalOrStatic, methodDescriptor: descriptor, offset: demangledSymbol.offset, vtableOffset: vtableOffset) + var functionDefinition = FunctionDefinition(node: node, name: name, kind: .function, symbol: demangledSymbol.base, isGlobalOrStatic: isGlobalOrStatic, methodDescriptor: descriptor, offset: demangledSymbol.offset, vtableOffset: vtableOffset) if let methodDescriptor = descriptor?.method, methodDescriptor.layout.flags.isDynamic { functionDefinition.attributes.append(.dynamic) } diff --git a/Sources/SwiftDeclaration/Components/Definitions/ExtensionDefinition.swift b/Sources/SwiftDeclaration/Components/Definitions/ExtensionDefinition.swift index a89c8800..011b5f87 100644 --- a/Sources/SwiftDeclaration/Components/Definitions/ExtensionDefinition.swift +++ b/Sources/SwiftDeclaration/Components/Definitions/ExtensionDefinition.swift @@ -11,7 +11,7 @@ import SwiftStdlibToolbox public final class ExtensionDefinition: Definition, MutableDefinition { public let extensionName: ExtensionName - public let genericSignature: Node? + public let genericSignature: NodeReference? public let protocolConformance: ProtocolConformance? @@ -60,7 +60,7 @@ public final class ExtensionDefinition: Definition, MutableDefinition { !variables.isEmpty || !functions.isEmpty || !staticVariables.isEmpty || !staticFunctions.isEmpty || !allocators.isEmpty || !constructors.isEmpty || !staticSubscripts.isEmpty || !subscripts.isEmpty } - public init(extensionName: ExtensionName, genericSignature: Node?, protocolConformance: ProtocolConformance?, conformingProtocolName: ProtocolName? = nil, associatedTypes: [AssociatedType] = [], resolvedAssociatedTypeWitnesses: [AssociatedTypeWitnessProjection] = [], in machO: MachO) throws { + public init(extensionName: ExtensionName, genericSignature: NodeReference?, protocolConformance: ProtocolConformance?, conformingProtocolName: ProtocolName? = nil, associatedTypes: [AssociatedType] = [], resolvedAssociatedTypeWitnesses: [AssociatedTypeWitnessProjection] = [], in machO: MachO) throws { self.extensionName = extensionName self.genericSignature = genericSignature self.protocolConformance = protocolConformance @@ -71,7 +71,7 @@ public final class ExtensionDefinition: Definition, MutableDefinition { /// Mach-O-free initializer for pure-value construction (tests, tooling). /// Carries no `ProtocolConformance` — only the frozen attribution fields. - package init(extensionName: ExtensionName, genericSignature: Node?, conformingProtocolName: ProtocolName? = nil, resolvedAssociatedTypeWitnesses: [AssociatedTypeWitnessProjection] = []) { + package init(extensionName: ExtensionName, genericSignature: NodeReference?, conformingProtocolName: ProtocolName? = nil, resolvedAssociatedTypeWitnesses: [AssociatedTypeWitnessProjection] = []) { self.extensionName = extensionName self.genericSignature = genericSignature self.protocolConformance = nil diff --git a/Sources/SwiftDeclaration/Components/Definitions/FieldDefinition.swift b/Sources/SwiftDeclaration/Components/Definitions/FieldDefinition.swift index 0b3b1c0f..9414eb38 100644 --- a/Sources/SwiftDeclaration/Components/Definitions/FieldDefinition.swift +++ b/Sources/SwiftDeclaration/Components/Definitions/FieldDefinition.swift @@ -25,6 +25,6 @@ public struct FieldFlags: OptionSet, Sendable { @MemberwiseInit(.public) public struct FieldDefinition: Sendable { public let name: String - public let typeNode: Node + public let typeNode: NodeReference public let flags: FieldFlags } diff --git a/Sources/SwiftDeclaration/Components/Definitions/FunctionDefinition.swift b/Sources/SwiftDeclaration/Components/Definitions/FunctionDefinition.swift index 86e1d81e..d16e02d8 100644 --- a/Sources/SwiftDeclaration/Components/Definitions/FunctionDefinition.swift +++ b/Sources/SwiftDeclaration/Components/Definitions/FunctionDefinition.swift @@ -5,7 +5,7 @@ import Utilities @MemberwiseInit(.public) public struct FunctionDefinition: Sendable { - public let node: Node + public let node: NodeReference public let name: String public let kind: FunctionKind public let symbol: DemangledSymbol diff --git a/Sources/SwiftDeclaration/Components/Definitions/ProtocolDefinition.swift b/Sources/SwiftDeclaration/Components/Definitions/ProtocolDefinition.swift index 0893d7f6..f81e3437 100644 --- a/Sources/SwiftDeclaration/Components/Definitions/ProtocolDefinition.swift +++ b/Sources/SwiftDeclaration/Components/Definitions/ProtocolDefinition.swift @@ -124,7 +124,7 @@ public final class ProtocolDefinition: Definition, MutableDefinition { public init(`protocol`: MachOSwiftSection.`Protocol`, in machO: MachO) throws { self.protocol = `protocol` let node = try MetadataReader.demangleContext(for: .protocol(`protocol`.descriptor), in: machO) - self.protocolName = ProtocolName(node: node) + self.protocolName = ProtocolName(node: NodeReference(interning: node)) } package func index(in machO: MachO) async throws { diff --git a/Sources/SwiftDeclaration/Components/Definitions/SubscriptDefinition.swift b/Sources/SwiftDeclaration/Components/Definitions/SubscriptDefinition.swift index 92fd0e49..c14291b8 100644 --- a/Sources/SwiftDeclaration/Components/Definitions/SubscriptDefinition.swift +++ b/Sources/SwiftDeclaration/Components/Definitions/SubscriptDefinition.swift @@ -3,7 +3,7 @@ import Demangling @MemberwiseInit(.public) public struct SubscriptDefinition: Sendable, AccessorRepresentable { - public let node: Node + public let node: NodeReference public let accessors: [Accessor] public let isStatic: Bool public var attributes: [SwiftAttribute] = [] diff --git a/Sources/SwiftDeclaration/Components/Definitions/TypeDefinition.swift b/Sources/SwiftDeclaration/Components/Definitions/TypeDefinition.swift index 40394544..2f2960a3 100644 --- a/Sources/SwiftDeclaration/Components/Definitions/TypeDefinition.swift +++ b/Sources/SwiftDeclaration/Components/Definitions/TypeDefinition.swift @@ -155,10 +155,14 @@ public final class TypeDefinition: Definition { @Dependency(\.symbolIndexStore) var symbolIndexStore - var fields: [FieldDefinition] = [] let typeContextDescriptor = try required(type.contextDescriptorWrapper.typeContextDescriptor) let fieldDescriptor = try typeContextDescriptor.fieldDescriptor(in: machO) let records = try fieldDescriptor.records(in: machO) + // All field type trees of one type share a single store, so common + // subtrees (module references, stdlib types) deduplicate instead of + // paying a per-field mini store. + var fieldNodeStoreBuilder = NodeStoreBuilder() + var pendingFields: [(name: String, typeNodeIndex: NodeStore.NodeIndex, flags: FieldFlags)] = [] for record in records { let typeNode = try record.demangledTypeNode(in: machO) let name = try record.fieldName(in: machO) @@ -186,11 +190,13 @@ public final class TypeDefinition: Definition { if try !record.mangledTypeName(in: machO).isEmpty { fieldFlags.insert(.hasMangledTypeName) } - let field = FieldDefinition(name: name.stripLazyPrefix, typeNode: typeNode, flags: fieldFlags) - fields.append(field) + pendingFields.append((name: name.stripLazyPrefix, typeNodeIndex: fieldNodeStoreBuilder.intern(typeNode), flags: fieldFlags)) } + let fieldNodeStore = fieldNodeStoreBuilder.freeze() - self.fields = fields + self.fields = pendingFields.map { pendingField in + FieldDefinition(name: pendingField.name, typeNode: fieldNodeStore.reference(at: pendingField.typeNodeIndex), flags: pendingField.flags) + } let fieldNames = Set(fields.map(\.name)) @@ -462,7 +468,7 @@ public final class TypeDefinition: Definition { // node exists (which means the function either takes no parameters // or takes exclusively unnamed parameters — the demangler does not // always emit an explicit labelList for the all-unnamed case). - func labels(of node: Node) -> [String] { + func labels(of node: NodeReference) -> [String] { guard let list = node.first(of: .labelList) else { return [] } return list.children.map { child in if child.kind == .firstElementMarker { return "_" } diff --git a/Sources/SwiftDeclaration/Components/Definitions/VariableDefinition.swift b/Sources/SwiftDeclaration/Components/Definitions/VariableDefinition.swift index f75bea8f..29fb378b 100644 --- a/Sources/SwiftDeclaration/Components/Definitions/VariableDefinition.swift +++ b/Sources/SwiftDeclaration/Components/Definitions/VariableDefinition.swift @@ -5,7 +5,7 @@ import MachOSwiftSection @MemberwiseInit(.public) public struct VariableDefinition: Sendable, AccessorRepresentable { - public let node: Node + public let node: NodeReference public let name: String public let accessors: [Accessor] public let isGlobalOrStatic: Bool diff --git a/Sources/SwiftDeclaration/Components/Names/DefinitionName.swift b/Sources/SwiftDeclaration/Components/Names/DefinitionName.swift index aea58670..821b0a1a 100644 --- a/Sources/SwiftDeclaration/Components/Names/DefinitionName.swift +++ b/Sources/SwiftDeclaration/Components/Names/DefinitionName.swift @@ -2,7 +2,7 @@ import Foundation import Demangling public protocol DefinitionName { - var node: Node { get } + var node: NodeReference { get } } extension DefinitionName { diff --git a/Sources/SwiftDeclaration/Components/Names/ExtensionName.swift b/Sources/SwiftDeclaration/Components/Names/ExtensionName.swift index 39838966..d2662b23 100644 --- a/Sources/SwiftDeclaration/Components/Names/ExtensionName.swift +++ b/Sources/SwiftDeclaration/Components/Names/ExtensionName.swift @@ -4,7 +4,7 @@ import Demangling @MemberwiseInit(.public) public struct ExtensionName: DefinitionName, Hashable, Sendable, Codable { - public let node: Node + public let node: NodeReference public let kind: ExtensionKind @@ -33,3 +33,40 @@ extension ExtensionName { } } } + +// MARK: - Structural Hashable + +// See `TypeName`: names hash and compare by node STRUCTURE, not by +// `NodeReference`'s store-identity `Hashable`. +extension ExtensionName { + public static func == (lhs: ExtensionName, rhs: ExtensionName) -> Bool { + lhs.kind == rhs.kind && lhs.node.structurallyEquals(rhs.node) + } + + public func hash(into hasher: inout Hasher) { + hasher.combine(kind) + node.structuralHash(into: &hasher) + } +} + +// MARK: - Codable + +// Wire-compatible with the historical `node: Node` encoding. +extension ExtensionName { + private enum CodingKeys: String, CodingKey { + case node + case kind + } + + public init(from decoder: Decoder) throws { + let container = try decoder.container(keyedBy: CodingKeys.self) + self.node = NodeReference(interning: try container.decode(Node.self, forKey: .node)) + self.kind = try container.decode(ExtensionKind.self, forKey: .kind) + } + + public func encode(to encoder: Encoder) throws { + var container = encoder.container(keyedBy: CodingKeys.self) + try container.encode(node.materialize(), forKey: .node) + try container.encode(kind, forKey: .kind) + } +} diff --git a/Sources/SwiftDeclaration/Components/Names/ProtocolName.swift b/Sources/SwiftDeclaration/Components/Names/ProtocolName.swift index 1756f60e..44432434 100644 --- a/Sources/SwiftDeclaration/Components/Names/ProtocolName.swift +++ b/Sources/SwiftDeclaration/Components/Names/ProtocolName.swift @@ -4,7 +4,7 @@ import Demangling @MemberwiseInit(.public) public struct ProtocolName: DefinitionName, Hashable, Sendable, Codable { - public let node: Node + public let node: NodeReference @SemanticStringBuilder public func print() -> SemanticString { @@ -17,3 +17,36 @@ extension ProtocolName { ExtensionName(node: node, kind: .protocol) } } + +// MARK: - Structural Hashable + +// See `TypeName`: names hash and compare by node STRUCTURE, not by +// `NodeReference`'s store-identity `Hashable`. +extension ProtocolName { + public static func == (lhs: ProtocolName, rhs: ProtocolName) -> Bool { + lhs.node.structurallyEquals(rhs.node) + } + + public func hash(into hasher: inout Hasher) { + node.structuralHash(into: &hasher) + } +} + +// MARK: - Codable + +// Wire-compatible with the historical `node: Node` encoding. +extension ProtocolName { + private enum CodingKeys: String, CodingKey { + case node + } + + public init(from decoder: Decoder) throws { + let container = try decoder.container(keyedBy: CodingKeys.self) + self.node = NodeReference(interning: try container.decode(Node.self, forKey: .node)) + } + + public func encode(to encoder: Encoder) throws { + var container = encoder.container(keyedBy: CodingKeys.self) + try container.encode(node.materialize(), forKey: .node) + } +} diff --git a/Sources/SwiftDeclaration/Components/Names/TypeName.swift b/Sources/SwiftDeclaration/Components/Names/TypeName.swift index 08495b07..a75a377e 100644 --- a/Sources/SwiftDeclaration/Components/Names/TypeName.swift +++ b/Sources/SwiftDeclaration/Components/Names/TypeName.swift @@ -4,7 +4,7 @@ import Demangling @MemberwiseInit(.public) public struct TypeName: DefinitionName, Hashable, Sendable, Codable { - public let node: Node + public let node: NodeReference public let kind: TypeKind @SemanticStringBuilder @@ -25,3 +25,43 @@ extension TypeName { ExtensionName(node: node, kind: .type(kind)) } } + +// MARK: - Structural Hashable + +// `NodeReference`'s intrinsic `Hashable` is store-identity based, which +// would split structurally equal names minted into different stores. +// Names key dictionaries by the node's STRUCTURE (matching the historical +// `node: Node` semantics), so equality and hashing walk the tree. +extension TypeName { + public static func == (lhs: TypeName, rhs: TypeName) -> Bool { + lhs.kind == rhs.kind && lhs.node.structurallyEquals(rhs.node) + } + + public func hash(into hasher: inout Hasher) { + hasher.combine(kind) + node.structuralHash(into: &hasher) + } +} + +// MARK: - Codable + +// Wire-compatible with the historical `node: Node` encoding: the node is +// encoded as a materialized `Node` tree and re-interned on decode. +extension TypeName { + private enum CodingKeys: String, CodingKey { + case node + case kind + } + + public init(from decoder: Decoder) throws { + let container = try decoder.container(keyedBy: CodingKeys.self) + self.node = NodeReference(interning: try container.decode(Node.self, forKey: .node)) + self.kind = try container.decode(TypeKind.self, forKey: .kind) + } + + public func encode(to encoder: Encoder) throws { + var container = encoder.container(keyedBy: CodingKeys.self) + try container.encode(node.materialize(), forKey: .node) + try container.encode(kind, forKey: .kind) + } +} diff --git a/Sources/SwiftDeclaration/Extensions.swift b/Sources/SwiftDeclaration/Extensions.swift index 391f3e1b..50876b1b 100644 --- a/Sources/SwiftDeclaration/Extensions.swift +++ b/Sources/SwiftDeclaration/Extensions.swift @@ -43,7 +43,7 @@ extension ProtocolConformance { } else { return nil } - return TypeName(node: node, kind: kind) + return TypeName(node: NodeReference(interning: node), kind: kind) case .element(let element): return try element.typeContextDescriptorWrapper?.typeName(in: machO) @@ -54,7 +54,7 @@ extension ProtocolConformance { case .directObjCClassName, .indirectObjCClass: guard let node = try typeNode(in: machO) else { return nil } - return TypeName(node: node, kind: .class) + return TypeName(node: NodeReference(interning: node), kind: .class) } } @@ -77,7 +77,7 @@ extension ProtocolConformance { } else { return nil } - return TypeName(node: node, kind: kind) + return TypeName(node: NodeReference(interning: node), kind: kind) case .element(let element): return try element.typeContextDescriptorWrapper?.typeName() case nil: @@ -86,18 +86,18 @@ extension ProtocolConformance { case .directObjCClassName, .indirectObjCClass: guard let node = try typeNode() else { return nil } - return TypeName(node: node, kind: .class) + return TypeName(node: NodeReference(interning: node), kind: .class) } } package func protocolName(in machO: some MachOSwiftSectionRepresentableWithCache) throws -> ProtocolName? { guard let node = try protocolNode(in: machO) else { return nil } - return ProtocolName(node: node) + return ProtocolName(node: NodeReference(interning: node)) } package func protocolName() throws -> ProtocolName? { guard let node = try protocolNode() else { return nil } - return ProtocolName(node: node) + return ProtocolName(node: NodeReference(interning: node)) } } @@ -114,7 +114,7 @@ extension AssociatedType { } else { return nil } - return TypeName(node: node, kind: kind) + return TypeName(node: NodeReference(interning: node), kind: kind) } package func typeName() throws -> TypeName? { @@ -129,15 +129,15 @@ extension AssociatedType { } else { return nil } - return TypeName(node: node, kind: kind) + return TypeName(node: NodeReference(interning: node), kind: kind) } package func protocolName(in machO: some MachOSwiftSectionRepresentableWithCache) throws -> ProtocolName { - try ProtocolName(node: MetadataReader.demangleType(for: protocolTypeName, in: machO)) + ProtocolName(node: NodeReference(interning: try MetadataReader.demangleType(for: protocolTypeName, in: machO))) } package func protocolName() throws -> ProtocolName { - try ProtocolName(node: MetadataReader.demangleType(for: protocolTypeName)) + ProtocolName(node: NodeReference(interning: try MetadataReader.demangleType(for: protocolTypeName))) } } @@ -153,11 +153,11 @@ extension MachOSwiftSection.`Protocol` { extension ProtocolDescriptor { package func protocolName(in machO: some MachOSwiftSectionRepresentableWithCache) throws -> ProtocolName { - try ProtocolName(node: MetadataReader.demangleContext(for: .protocol(self), in: machO)) + ProtocolName(node: NodeReference(interning: try MetadataReader.demangleContext(for: .protocol(self), in: machO))) } package func protocolName() throws -> ProtocolName { - try ProtocolName(node: MetadataReader.demangleContext(for: .protocol(self))) + ProtocolName(node: NodeReference(interning: try MetadataReader.demangleContext(for: .protocol(self)))) } } @@ -184,11 +184,11 @@ extension TypeContextDescriptorWrapper { } package func typeName(in machO: some MachOSwiftSectionRepresentableWithCache) throws -> TypeName { - return try TypeName(node: MetadataReader.demangleContext(for: .type(self), in: machO), kind: kind) + return TypeName(node: NodeReference(interning: try MetadataReader.demangleContext(for: .type(self), in: machO)), kind: kind) } package func typeName() throws -> TypeName { - return try TypeName(node: MetadataReader.demangleContext(for: .type(self)), kind: kind) + return TypeName(node: NodeReference(interning: try MetadataReader.demangleContext(for: .type(self))), kind: kind) } } diff --git a/Sources/SwiftDeclarationRendering/Extensions/GenericContext+Dump.swift b/Sources/SwiftDeclarationRendering/Extensions/GenericContext+Dump.swift index a522b7b8..300e997e 100644 --- a/Sources/SwiftDeclarationRendering/Extensions/GenericContext+Dump.swift +++ b/Sources/SwiftDeclarationRendering/Extensions/GenericContext+Dump.swift @@ -2,7 +2,7 @@ import Semantic import MachOKit import MachOSwiftSection import Utilities -import Demangling +@_spi(Internals) import Demangling @_spi(Internals) import SwiftInspection package func genericParameterName(depth: Int, index: Int) throws -> String { @@ -307,11 +307,11 @@ extension GenericRequirementDescriptor { switch element { case .objc(let objc): let objcName = try objc.mangledName(in: machO).rawString - let node = Node.create(kind: .global, children: [ - Node.create(kind: .type, children: [ - Node.create(kind: .protocol, children: [ - .create(kind: .module, text: objcModule), - .create(kind: .identifier, text: objcName), + let node = Node.createTransient(kind: .global, children: [ + Node.createTransient(kind: .type, children: [ + Node.createTransient(kind: .protocol, children: [ + .createTransient(kind: .module, text: objcModule), + .createTransient(kind: .identifier, text: objcName), ]) ]) ]) @@ -410,11 +410,11 @@ extension GenericRequirementDescriptor { switch element { case .objc(let objc): let objcName = try objc.mangledName(in: machO).rawString - let node = Node.create(kind: .global, children: [ - Node.create(kind: .type, children: [ - Node.create(kind: .protocol, children: [ - .create(kind: .module, text: objcModule), - .create(kind: .identifier, text: objcName), + let node = Node.createTransient(kind: .global, children: [ + Node.createTransient(kind: .type, children: [ + Node.createTransient(kind: .protocol, children: [ + .createTransient(kind: .module, text: objcModule), + .createTransient(kind: .identifier, text: objcName), ]) ]) ]) diff --git a/Sources/SwiftDeclarationRendering/Extensions/Node+.swift b/Sources/SwiftDeclarationRendering/Extensions/Node+.swift index c7c8c2b6..c24bc46d 100644 --- a/Sources/SwiftDeclarationRendering/Extensions/Node+.swift +++ b/Sources/SwiftDeclarationRendering/Extensions/Node+.swift @@ -1,13 +1,13 @@ import Foundation -import Demangling +@_spi(Internals) import Demangling import Semantic extension SemanticString: @retroactive NodePrinterTarget { - public mutating func pushTypeReferenceScope(_ node: Node?) { + public mutating func pushTypeReferenceScope(_ node: @autoclosure () -> Node?) { // A failed remangle degrades to a nil (barrier) scope: the span's // tokens carry no identity rather than inheriting the enclosing // type's, which would mislabel them. - pushIdentifierScope(node.flatMap { try? mangleAsString($0) }) + pushIdentifierScope(node().flatMap { try? mangleAsString($0) }) } public mutating func popTypeReferenceScope() { @@ -58,6 +58,19 @@ extension Node { } } +extension NodeReference { + /// Zero-materialization semantic print of a store-backed subtree, + /// through the same generic engine as `Node.printSemantic` (the + /// type-reference identity scopes materialize just the nominal + /// reference nodes on demand, via the engine's lazy scope hook). + public func printSemantic(using options: DemangleOptions = .default) -> SemanticString { + StackSafeExecutor.execute { + var printer = DemanglingPrinter(options: options) + return printer.printRoot(self) + } + } +} + extension Node { package var hasWeakNode: Bool { preorder().first { $0.kind == .weak } != nil diff --git a/Sources/SwiftDeclarationRendering/Extensions/ResolvedTypeReference+.swift b/Sources/SwiftDeclarationRendering/Extensions/ResolvedTypeReference+.swift index dc288ad9..fcfaa0b0 100644 --- a/Sources/SwiftDeclarationRendering/Extensions/ResolvedTypeReference+.swift +++ b/Sources/SwiftDeclarationRendering/Extensions/ResolvedTypeReference+.swift @@ -1,6 +1,6 @@ import MachOKit import MachOSwiftSection -import Demangling +@_spi(Internals) import Demangling @_spi(Internals) import SwiftInspection extension ResolvedTypeReference { @@ -19,10 +19,10 @@ extension ResolvedTypeReference { } case .directObjCClassName(let objcClassName): guard let objcClassName, !objcClassName.isEmpty else { return nil } - return Node.create(kind: .type, children: [ - Node.create(kind: .class, children: [ - .create(kind: .module, text: objcModule), - .create(kind: .identifier, text: objcClassName), + return Node.createTransient(kind: .type, children: [ + Node.createTransient(kind: .class, children: [ + .createTransient(kind: .module, text: objcModule), + .createTransient(kind: .identifier, text: objcClassName), ]) ]) case .indirectObjCClass(let objcClass): @@ -53,10 +53,10 @@ extension ResolvedTypeReference { } case .directObjCClassName(let objcClassName): guard let objcClassName, !objcClassName.isEmpty else { return nil } - return Node.create(kind: .type, children: [ - Node.create(kind: .class, children: [ - .create(kind: .module, text: objcModule), - .create(kind: .identifier, text: objcClassName), + return Node.createTransient(kind: .type, children: [ + Node.createTransient(kind: .class, children: [ + .createTransient(kind: .module, text: objcModule), + .createTransient(kind: .identifier, text: objcClassName), ]) ]) case .indirectObjCClass(let objcClass): diff --git a/Sources/SwiftDeclarationRendering/RuntimeFieldLayoutBackend.swift b/Sources/SwiftDeclarationRendering/RuntimeFieldLayoutBackend.swift index 7777d491..ef11d834 100644 --- a/Sources/SwiftDeclarationRendering/RuntimeFieldLayoutBackend.swift +++ b/Sources/SwiftDeclarationRendering/RuntimeFieldLayoutBackend.swift @@ -1,7 +1,7 @@ import Foundation import FoundationToolbox import Semantic -import Demangling +@_spi(Internals) import Demangling import MachOKit import MachOSwiftSection import SwiftLayout @@ -425,7 +425,7 @@ struct RuntimeFieldLayoutBackend { case .type: if let argumentType = boundGenericArgumentType(atSlot: slot, totalKeyArguments: layout.totalKeyArguments, of: parentMetadata), let argumentMangledString = _mangledTypeName(argumentType), - let argumentNode = try? demangleAsNode(argumentMangledString, isType: true) { + let argumentNode = try? demangleAsNodeTransient(argumentMangledString, isType: true) { return innerTypeNode(of: argumentNode) } case .value: @@ -443,7 +443,7 @@ struct RuntimeFieldLayoutBackend { let substitutedChildren = node.children.map { substitutingGenericParameters(in: $0, parentMetadata: parentMetadata, layout: layout) } - return Node.create(kind: node.kind, contents: node.contents, children: Array(substitutedChildren)) + return Node.createTransient(kind: node.kind, contents: node.contents, children: Array(substitutedChildren)) } /// Resolves a `.type` key-argument slot to its concrete `Any.Type`. @@ -465,9 +465,9 @@ struct RuntimeFieldLayoutBackend { guard let word = genericArgumentWord(atSlot: slot, totalKeyArguments: totalKeyArguments, of: parentMetadata) else { return nil } let value = Int(bitPattern: word) if value >= 0 { - return Node.create(kind: .integer, contents: .index(UInt64(value))) + return Node.createTransient(kind: .integer, contents: .index(UInt64(value))) } else { - return Node.create(kind: .negativeInteger, contents: .index(UInt64(value.magnitude))) + return Node.createTransient(kind: .negativeInteger, contents: .index(UInt64(value.magnitude))) } } @@ -496,7 +496,7 @@ struct RuntimeFieldLayoutBackend { guard let countWord = genericArgumentWord(atSlot: shapeClassSlot, totalKeyArguments: layout.totalKeyArguments, of: parentMetadata) else { return nil } let elementCount = Int(bitPattern: countWord) guard elementCount >= 0, elementCount <= packElementCountLimit else { return nil } - if elementCount == 0 { return Node.create(kind: .pack, children: []) } + if elementCount == 0 { return Node.createTransient(kind: .pack, children: []) } // Pack pointer: low bit is the on-heap lifetime flag — strip it. guard let packWord = genericArgumentWord(atSlot: packSlot, totalKeyArguments: layout.totalKeyArguments, of: parentMetadata) else { return nil } @@ -513,10 +513,10 @@ struct RuntimeFieldLayoutBackend { let elementPointer = UnsafeRawPointer(bitPattern: elementWord) else { return nil } let elementType = unsafeBitCast(elementPointer, to: Any.Type.self) guard let elementMangledString = _mangledTypeName(elementType), - let elementNode = try? demangleAsNode(elementMangledString, isType: true) else { return nil } + let elementNode = try? demangleAsNodeTransient(elementMangledString, isType: true) else { return nil } elementNodes.append(elementNode) } - return Node.create(kind: .pack, children: elementNodes) + return Node.createTransient(kind: .pack, children: elementNodes) } /// Reads the raw word at an absolute slot of `parentMetadata`'s inline diff --git a/Sources/SwiftDiffing/ABIKey.swift b/Sources/SwiftDiffing/ABIKey.swift index a38c3d2f..49611e20 100644 --- a/Sources/SwiftDiffing/ABIKey.swift +++ b/Sources/SwiftDiffing/ABIKey.swift @@ -47,7 +47,7 @@ public enum ABIKey: Hashable, Sendable, Codable { /// the reporters surface them as warnings. (A remangle-success-independent /// identity would rework the key for a precision loss — still not worth it.) /// See `Documentations/Internal/ABIDiffDesignAndLimitations.md`. - public static func make(for node: Node) -> ABIKey { + public static func make(for node: some DemanglingNode) -> ABIKey { // `canMangle` is literally `(try? mangleAsString) != nil`, so a single // `try?` decides the branch and remangles exactly once. if let mangled = try? mangleAsString(node) { @@ -60,7 +60,7 @@ public enum ABIKey: Hashable, Sendable, Codable { /// (`TypeName.node`, `ProtocolName.node`) and field type nodes /// (`FieldDefinition.typeNode`). The `.type` wrapper is stripped first /// because `mangleAsString` rejects some `.type`-rooted trees. - public static func makeUnwrappingType(for node: Node) -> ABIKey { + public static func makeUnwrappingType(for node: some DemanglingNode) -> ABIKey { make(for: unwrapType(node)) } @@ -75,7 +75,7 @@ public enum ABIKey: Hashable, Sendable, Codable { /// Strip a single `.type` envelope so the inner nominal node can remangle. /// A no-op when the node is not `.type`-rooted. - static func unwrapType(_ node: Node) -> Node { + static func unwrapType(_ node: SomeNode) -> SomeNode { node.kind == .type ? (node.children.first ?? node) : node } @@ -98,7 +98,7 @@ public enum ABIKey: Hashable, Sendable, Codable { /// The injective fallback rendering: the self-identifying prefix + root /// kind + a print that retains bound-generic arguments (`.default` is /// `.default` without `.removeBoundGeneric`). - private static func fallbackString(for node: Node) -> String { + private static func fallbackString(for node: some DemanglingNode) -> String { "\(remangleFallbackPrefix)\(node.kind):\(node.print(using: .default))" } } diff --git a/Sources/SwiftDump/Protocols/TypedDumper.swift b/Sources/SwiftDump/Protocols/TypedDumper.swift index 9557917f..24277f05 100644 --- a/Sources/SwiftDump/Protocols/TypedDumper.swift +++ b/Sources/SwiftDump/Protocols/TypedDumper.swift @@ -3,7 +3,7 @@ import FoundationToolbox import Semantic import MachOSwiftSection import MachOKit -import Demangling +@_spi(Internals) import Demangling @_spi(Internals) import SwiftInspection import SwiftDeclarationRendering @@ -225,7 +225,7 @@ extension TypedDumper { // so callers stay on the unbound representation. guard #available(macOS 11, iOS 14, tvOS 14, watchOS 7, *) else { return nil } guard let resolvedMangledString = _mangledTypeName(metatype) else { return nil } - return try? demangleAsNode(resolvedMangledString, isType: true) + return try? demangleAsNodeTransient(resolvedMangledString, isType: true) } /// Render the bound generic dumped name so that the type's qualified diff --git a/Sources/SwiftIndexing/SwiftDeclarationIndexer.swift b/Sources/SwiftIndexing/SwiftDeclarationIndexer.swift index f55047e5..0ba3efe4 100644 --- a/Sources/SwiftIndexing/SwiftDeclarationIndexer.swift +++ b/Sources/SwiftIndexing/SwiftDeclarationIndexer.swift @@ -381,12 +381,12 @@ public final class SwiftDeclarationIndexer, genericSignature: Node?) throws -> ExtensionDefinition { - let extensionDefinition = try ExtensionDefinition(extensionName: .init(node: materializedExtensionTargetNode, kind: kind), genericSignature: genericSignature, protocolConformance: nil, in: machO) + func extensionDefinition(of kind: ExtensionKind, for memberSymbolsByKind: OrderedDictionary, genericSignature: NodeReference?) throws -> ExtensionDefinition { + let extensionDefinition = try ExtensionDefinition(extensionName: .init(node: node, kind: kind), genericSignature: genericSignature, protocolConformance: nil, in: machO) var memberCount = 0 for (kind, memberSymbols) in memberSymbolsByKind { @@ -723,10 +722,10 @@ public final class SwiftDeclarationIndexer Node? { - return try demangleAsNode(symbol.name) + return try demangleAsNodeTransient(symbol.name) } public static func demangleContext(for context: ContextDescriptorWrapper) throws -> Node { @@ -190,7 +190,7 @@ extension MetadataReader { failed = true break } - requirementNodes.append(Node.create(kind: .dependentGenericConformanceRequirement, children: [subject, proto])) + requirementNodes.append(Node.createTransient(kind: .dependentGenericConformanceRequirement, children: [subject, proto])) case .type(let relativeDirectPointer): let typeAddress = try context.addressFromOffset(contentOffset) let mangledName = try relativeDirectPointer.resolve(at: typeAddress, in: context) @@ -206,10 +206,10 @@ extension MetadataReader { nodeKind = .dependentGenericConformanceRequirement } - requirementNodes.append(Node.create(kind: nodeKind, children: [subject, type])) + requirementNodes.append(Node.createTransient(kind: nodeKind, children: [subject, type])) case .layout(let genericRequirementLayoutKind): if genericRequirementLayoutKind == .class { - requirementNodes.append(Node.create(kind: .dependentGenericLayoutRequirement, children: [subject, .create(kind: .identifier, text: "C")])) + requirementNodes.append(Node.createTransient(kind: .dependentGenericLayoutRequirement, children: [subject, .createTransient(kind: .identifier, text: "C")])) } else { failed = true } @@ -217,15 +217,15 @@ extension MetadataReader { break case .invertedProtocols(let invertedProtocols): if invertedProtocols.protocols.hasCopyable { - requirementNodes.append(Node.create(kind: .dependentGenericInverseConformanceRequirement, children: [ + requirementNodes.append(Node.createTransient(kind: .dependentGenericInverseConformanceRequirement, children: [ subject, - .create(kind: .index, index: UInt64(MachOSwiftSection.InvertibleProtocolKind.copyable.rawValue)), + .createTransient(kind: .index, index: UInt64(MachOSwiftSection.InvertibleProtocolKind.copyable.rawValue)), ])) } if invertedProtocols.protocols.hasEscapable { - requirementNodes.append(Node.create(kind: .dependentGenericInverseConformanceRequirement, children: [ + requirementNodes.append(Node.createTransient(kind: .dependentGenericInverseConformanceRequirement, children: [ subject, - .create(kind: .index, index: UInt64(MachOSwiftSection.InvertibleProtocolKind.escapable.rawValue)), + .createTransient(kind: .index, index: UInt64(MachOSwiftSection.InvertibleProtocolKind.escapable.rawValue)), ])) } } @@ -233,7 +233,7 @@ extension MetadataReader { if failed || requirementNodes.isEmpty { return nil } else { - return Node.create(kind: .dependentGenericSignature, children: requirementNodes) + return Node.createTransient(kind: .dependentGenericSignature, children: requirementNodes) } } @@ -272,9 +272,9 @@ extension MetadataReader { if let opaqueTypeDescriptor = contextWrapper.opaqueTypeDescriptor { if let machOImageContext = context as? MachOContext { let absoluteAddress = machOImageContext.machO.ptr.bitPattern.int + opaqueTypeDescriptor.offset - result = .create(kind: .opaqueTypeDescriptorSymbolicReference, index: UInt64(absoluteAddress)) + result = .createTransient(kind: .opaqueTypeDescriptorSymbolicReference, index: UInt64(absoluteAddress)) } else { - result = .create(kind: .opaqueTypeDescriptorSymbolicReference, index: opaqueTypeDescriptor.offset.cast()) + result = .createTransient(kind: .opaqueTypeDescriptorSymbolicReference, index: opaqueTypeDescriptor.offset.cast()) } } else { result = try buildContextMangling(context: .element(contextWrapper), in: context) @@ -286,9 +286,9 @@ extension MetadataReader { if case .element(let element) = resolvableElement, let opaqueTypeDescriptor = element.opaqueTypeDescriptor { if let machOImageContext = context as? MachOContext { let absoluteAddress = machOImageContext.machO.ptr.bitPattern.int + opaqueTypeDescriptor.offset - result = .create(kind: .opaqueTypeDescriptorSymbolicReference, index: UInt64(absoluteAddress)) + result = .createTransient(kind: .opaqueTypeDescriptorSymbolicReference, index: UInt64(absoluteAddress)) } else { - result = .create(kind: .opaqueTypeDescriptorSymbolicReference, index: opaqueTypeDescriptor.offset.cast()) + result = .createTransient(kind: .opaqueTypeDescriptorSymbolicReference, index: opaqueTypeDescriptor.offset.cast()) } } else { result = try buildContextMangling(context: resolvableElement, in: context) @@ -299,21 +299,21 @@ extension MetadataReader { // The symbolic reference points at a resolver function, but we can't // execute code in the target process to resolve it from here. let rawPointerOffset = try RelativeDirectRawPointer(relativeOffset: relativeOffset).resolveDirectAddress(at: context.addressFromOffset(offset), in: context) - result = try .create(kind: .accessorFunctionReference, index: context.offsetFromAddress(rawPointerOffset).cast()) + result = try .createTransient(kind: .accessorFunctionReference, index: context.offsetFromAddress(rawPointerOffset).cast()) case .uniqueExtendedExistentialTypeShape: let extendedExistentialTypeShape = try RelativeDirectPointer(relativeOffset: relativeOffset).resolve(at: baseAddress, in: context) let existentialType = try extendedExistentialTypeShape.existentialType(in: context) - result = try .create(kind: .uniqueExtendedExistentialTypeShapeSymbolicReference, inlineChildren: demangle(for: existentialType, kind: .type, in: context).children) + result = try .createTransient(kind: .uniqueExtendedExistentialTypeShapeSymbolicReference, inlineChildren: demangle(for: existentialType, kind: .type, in: context).children) case .nonUniqueExtendedExistentialTypeShape: let nonUniqueExtendedExistentialTypeShape = try RelativeDirectPointer(relativeOffset: relativeOffset).resolve(at: baseAddress, in: context) let existentialType = try nonUniqueExtendedExistentialTypeShape.existentialType(in: context) - result = try .create(kind: .nonUniqueExtendedExistentialTypeShapeSymbolicReference, inlineChildren: demangle(for: existentialType, kind: .type, in: context).children) + result = try .createTransient(kind: .nonUniqueExtendedExistentialTypeShapeSymbolicReference, inlineChildren: demangle(for: existentialType, kind: .type, in: context).children) case .objectiveCProtocol: let relativePointer = RelativeDirectPointer(relativeOffset: relativeOffset) let objcProtocol = try relativePointer.resolve(at: baseAddress, in: context) let protocolMangledName = try objcProtocol.mangledName(in: context) let name = protocolMangledName.symbolString - result = try demangleAsNode(name).typeSymbol + result = try demangleAsNodeTransient(name).typeSymbol } return result } catch { @@ -323,9 +323,9 @@ extension MetadataReader { let result: Node switch kind { case .type: - result = try demangleAsNode(stringValue, isType: true, symbolicReferenceResolver: symbolicReferenceResolver) + result = try demangleAsNodeTransient(stringValue, isType: true, symbolicReferenceResolver: symbolicReferenceResolver) case .symbol: - result = try demangleAsNode(stringValue, isType: false, symbolicReferenceResolver: symbolicReferenceResolver) + result = try demangleAsNodeTransient(stringValue, isType: false, symbolicReferenceResolver: symbolicReferenceResolver) } return result } @@ -348,7 +348,7 @@ extension MetadataReader { switch context { case .type, .protocol: - top = .create(kind: .type, children: [demangling]) + top = .createTransient(kind: .type, children: [demangling]) default: top = demangling } @@ -395,7 +395,7 @@ extension MetadataReader { if nameNode != nil { return true } else if let namedContext = context.namedContextDescriptor { - nameNode = try .create(kind: .identifier, text: namedContext.name(in: readingContext)) + nameNode = try .createTransient(kind: .identifier, text: namedContext.name(in: readingContext)) return true } else { return false @@ -421,9 +421,9 @@ extension MetadataReader { guard let extendedContext = try extensionContext.extendedContext(in: readingContext) else { return nil } guard let demangledExtendedContext = try demangle(for: extendedContext, kind: .type, in: readingContext).extensionSymbol else { return nil } if let requirements = try extensionContext.genericContext(in: readingContext)?.requirements, let signatureNode = try buildGenericSignature(for: requirements, in: readingContext) { - return Node.create(kind: .extension, children: [parentDemangling, demangledExtendedContext, signatureNode]) + return Node.createTransient(kind: .extension, children: [parentDemangling, demangledExtendedContext, signatureNode]) } else { - return Node.create(kind: .extension, children: [parentDemangling, demangledExtendedContext]) + return Node.createTransient(kind: .extension, children: [parentDemangling, demangledExtendedContext]) } case .anonymous: // Look up symbol using the context's symbol lookup capability @@ -432,9 +432,9 @@ extension MetadataReader { let privateDeclName = try? symbol.demangledNode.first(of: Node.Kind.privateDeclName), let privateDeclNameIdentifier = privateDeclName.children.first { if let parentDemangling { - return Node.create(kind: .anonymousContext, children: [privateDeclNameIdentifier, parentDemangling]) + return Node.createTransient(kind: .anonymousContext, children: [privateDeclNameIdentifier, parentDemangling]) } else { - return Node.create(kind: .anonymousContext, children: [privateDeclNameIdentifier]) + return Node.createTransient(kind: .anonymousContext, children: [privateDeclNameIdentifier]) } } return parentDemangling @@ -443,7 +443,7 @@ extension MetadataReader { return nil } guard let moduleContext = context.moduleContextDescriptor else { return nil } - return try .create(kind: .module, text: moduleContext.name(in: readingContext)) + return try .createTransient(kind: .module, text: moduleContext.name(in: readingContext)) case .opaqueType: guard let parentDescriptorResult else { return nil } if parentDemangling?.kind == .anonymousContext { @@ -453,10 +453,10 @@ extension MetadataReader { if mangledNode.kind == .global { mangledNode = mangledNode.children[0] } - let opaqueNode = Node.create(kind: .opaqueReturnTypeOf, children: [mangledNode]) + let opaqueNode = Node.createTransient(kind: .opaqueReturnTypeOf, children: [mangledNode]) return opaqueNode } else if let parentDemangling, parentDemangling.kind == .module { - let opaqueNode = Node.create(kind: .opaqueReturnTypeOf, children: [parentDemangling]) + let opaqueNode = Node.createTransient(kind: .opaqueReturnTypeOf, children: [parentDemangling]) return opaqueNode } else { return nil @@ -469,10 +469,10 @@ extension MetadataReader { if parentDemangling.children.count < 2 { return nil } - nameNode = Node.create(kind: .privateDeclName, children: [parentDemangling.children[0], nameNode]) + nameNode = Node.createTransient(kind: .privateDeclName, children: [parentDemangling.children[0], nameNode]) parentDemangling = parentDemangling.children[1] } - let demangling = Node.create(kind: kind, children: [parentDemangling, nameNode]) + let demangling = Node.createTransient(kind: kind, children: [parentDemangling, nameNode]) return demangling } @@ -537,7 +537,7 @@ extension MetadataReader { } return demangled } else { - return Node.create(kind: .protocol, children: [.create(kind: .module, text: objcModule), .create(kind: .identifier, text: name)]) + return Node.createTransient(kind: .protocol, children: [.createTransient(kind: .module, text: objcModule), .createTransient(kind: .identifier, text: name)]) } } case .swiftPointer(let swiftPointer): @@ -552,7 +552,7 @@ extension MetadataReader { } fileprivate static func _buildContextManglingForSymbol(_ symbol: Symbol, in context: Context) throws -> Node? { - var demangledSymbol = try demangleAsNode(symbol.name) + var demangledSymbol = try demangleAsNodeTransient(symbol.name) if demangledSymbol.kind == .global { demangledSymbol = demangledSymbol.children[0] } @@ -577,7 +577,7 @@ extension Node { } if child.kind == .enum || child.kind == .structure || child.kind == .class || child.kind == .protocol { - return .create(kind: .type, contents: .none, children: [child]) + return .createTransient(kind: .type, contents: .none, children: [child]) } for child in child.children { diff --git a/Sources/SwiftInterface/SwiftDeclarationPrinter+DiffRendering.swift b/Sources/SwiftInterface/SwiftDeclarationPrinter+DiffRendering.swift index 6c363624..1df7b8b1 100644 --- a/Sources/SwiftInterface/SwiftDeclarationPrinter+DiffRendering.swift +++ b/Sources/SwiftInterface/SwiftDeclarationPrinter+DiffRendering.swift @@ -47,7 +47,7 @@ package extension SwiftDeclarationPrinter { for: typeDefinition.type, displayParentName: displayParentName, level: level, - leafNameNode: leafNameNode(of: typeDefinition.typeName.node), + leafNameNode: leafNameNode(of: typeDefinition.typeName.node.materialize()), specializedMetadata: typeDefinition.isSpecialized ? typeDefinition.metadata : nil ) } @@ -64,7 +64,7 @@ package extension SwiftDeclarationPrinter { try await renderProtocolDeclarationHeader( for: protocolDefinition.protocol, displayParentName: displayParentName, - leafNameNode: leafNameNode(of: protocolDefinition.protocolName.node) + leafNameNode: leafNameNode(of: protocolDefinition.protocolName.node.materialize()) ) } diff --git a/Sources/SwiftPrinting/SwiftDeclarationPrinter+Members.swift b/Sources/SwiftPrinting/SwiftDeclarationPrinter+Members.swift index c91020c4..decff551 100644 --- a/Sources/SwiftPrinting/SwiftDeclarationPrinter+Members.swift +++ b/Sources/SwiftPrinting/SwiftDeclarationPrinter+Members.swift @@ -39,7 +39,7 @@ extension SwiftDeclarationPrinter { MemberDeclaration(field.name) Standard(":") Space() - try await printThrowingType(substitutedTypeNode ?? field.typeNode, isProtocol: false, level: level) + try await printThrowingType(substitutedTypeNode ?? field.typeNode.materialize(), isProtocol: false, level: level) } /// Renders a single enum case (`case name`, `case name(Payload)`, or @@ -62,7 +62,7 @@ extension SwiftDeclarationPrinter { MemberDeclaration(field.name) } - let payloadTypeNode = substitutedTypeNode ?? field.typeNode + let payloadTypeNode = substitutedTypeNode ?? field.typeNode.materialize() let payload = await printType(payloadTypeNode, isProtocol: false, level: level) let payloadText = payload.string if !payloadText.isEmpty, payloadText != "()" { @@ -105,7 +105,7 @@ extension SwiftDeclarationPrinter { MemberDeclaration(field.name) if field.flags.contains(.hasMangledTypeName) { - let payloadTypeNode = substitutedTypeNode ?? field.typeNode + let payloadTypeNode = substitutedTypeNode ?? field.typeNode.materialize() let payload = try await printThrowingType(payloadTypeNode, isProtocol: false, level: level) if !payload.string.isEmpty { if payloadTypeNode.firstChild?.isKind(of: .tuple) ?? false { diff --git a/Sources/SwiftPrinting/SwiftDeclarationPrinter.swift b/Sources/SwiftPrinting/SwiftDeclarationPrinter.swift index 2d657a4c..f49e09ad 100644 --- a/Sources/SwiftPrinting/SwiftDeclarationPrinter.swift +++ b/Sources/SwiftPrinting/SwiftDeclarationPrinter.swift @@ -274,7 +274,7 @@ public final class SwiftDeclarationPrinter: Sendab Space() } - try await printThrowingType(node, isProtocol: extensionDefinition.extensionName.isProtocol, level: level) + try await printThrowingType(node.materialize(), isProtocol: extensionDefinition.extensionName.isProtocol, level: level) if index < nodes.count - 1 { Standard(",") @@ -451,7 +451,7 @@ public final class SwiftDeclarationPrinter: Sendab Space() } var printer = VariableNodePrinter(isStored: variable.isStored, isOverride: variable.isOverride, isClassMember: variable.isClassMember, hasSetter: variable.hasSetter, indentation: level, delegate: self) - try await printer.printRoot(variable.node) + try await printer.printRoot(variable.node.materialize()) } @SemanticStringBuilder @@ -461,7 +461,7 @@ public final class SwiftDeclarationPrinter: Sendab Space() } var printer = FunctionNodePrinter(isOverride: function.isOverride, isClassMember: function.isClassMember, delegate: self) - try await printer.printRoot(function.node) + try await printer.printRoot(function.node.materialize()) } @SemanticStringBuilder @@ -471,7 +471,7 @@ public final class SwiftDeclarationPrinter: Sendab Space() } var printer = SubscriptNodePrinter(isOverride: `subscript`.isOverride, isClassMember: `subscript`.isClassMember, hasSetter: `subscript`.hasSetter, indentation: level, delegate: self) - try await printer.printRoot(`subscript`.node) + try await printer.printRoot(`subscript`.node.materialize()) } @SemanticStringBuilder diff --git a/Sources/SwiftSpecialization/ConformanceProvider.swift b/Sources/SwiftSpecialization/ConformanceProvider.swift index 6315d433..861a06b8 100644 --- a/Sources/SwiftSpecialization/ConformanceProvider.swift +++ b/Sources/SwiftSpecialization/ConformanceProvider.swift @@ -211,7 +211,7 @@ extension IndexerConformanceProvider: ConformanceProvider { guard let superNode = superNode.first(of: .type) else { continue } - let superTypeName = TypeName(node: superNode, kind: .class) + let superTypeName = TypeName(node: NodeReference(interning: superNode), kind: .class) map[superTypeName.name, default: []].append(childTypeName) } diff --git a/Sources/SwiftSpecialization/GenericSpecializer.swift b/Sources/SwiftSpecialization/GenericSpecializer.swift index f6f98a3e..0e467ce2 100644 --- a/Sources/SwiftSpecialization/GenericSpecializer.swift +++ b/Sources/SwiftSpecialization/GenericSpecializer.swift @@ -627,7 +627,7 @@ extension GenericSpecializer { for requirement in requirements { guard case .baseClass(let demangledNode, _) = requirement else { continue } let typeNode = demangledNode.first(of: .type) ?? demangledNode - return TypeName(node: typeNode, kind: .class) + return TypeName(node: NodeReference(interning: typeNode), kind: .class) } return nil } @@ -1973,7 +1973,7 @@ extension GenericSpecializer where MachO == MachOImage { step: AssociatedPathInfo.Step, allProtocolDefinitions: OrderedDictionary> ) throws -> Metadata { - let stepProtocolName = ProtocolName(node: step.protocolNode) + let stepProtocolName = ProtocolName(node: NodeReference(interning: step.protocolNode)) guard let entry = allProtocolDefinitions[stepProtocolName] else { throw AssociatedTypeResolutionError.missingAssociatedTypeRefMachOAndProtocol(protocolTypeNode: step.protocolNode) } diff --git a/Sources/SwiftSpecialization/TypeDefinition+Specialization.swift b/Sources/SwiftSpecialization/TypeDefinition+Specialization.swift index 8d98ab8a..1b18c535 100644 --- a/Sources/SwiftSpecialization/TypeDefinition+Specialization.swift +++ b/Sources/SwiftSpecialization/TypeDefinition+Specialization.swift @@ -322,9 +322,9 @@ extension TypeDefinition { ) -> TypeName { let unboundTypeNode: Node if unboundTypeName.node.kind == .type { - unboundTypeNode = unboundTypeName.node + unboundTypeNode = unboundTypeName.node.materialize() } else { - unboundTypeNode = Node.create(kind: .type, children: [unboundTypeName.node]) + unboundTypeNode = Node.create(kind: .type, children: [unboundTypeName.node.materialize()]) } let normalizedArgumentNodes: [Node] = typeArgumentNodes.map { argumentNode in @@ -346,7 +346,7 @@ extension TypeDefinition { let boundNode = Node.create(kind: boundKind, children: [unboundTypeNode, typeList]) let wrappedNode = Node.create(kind: .type, children: [boundNode]) - return TypeName(node: wrappedNode, kind: unboundTypeName.kind) + return TypeName(node: NodeReference(interning: wrappedNode), kind: unboundTypeName.kind) } private func validateSpecialization(metadata: MetadataWrapper, in machO: MachOImage) throws { diff --git a/Tests/SwiftAttributeInferenceTests/TypeAttributeInferrerTests.swift b/Tests/SwiftAttributeInferenceTests/TypeAttributeInferrerTests.swift index 407f9fec..4ff77c66 100644 --- a/Tests/SwiftAttributeInferenceTests/TypeAttributeInferrerTests.swift +++ b/Tests/SwiftAttributeInferenceTests/TypeAttributeInferrerTests.swift @@ -16,7 +16,7 @@ struct TypeAttributeInferrerTests { func detectPropertyWrapperFromField() { let typeNode = Node.create(kind: .type) let fields = [ - FieldDefinition(name: "wrappedValue", typeNode: typeNode, flags: FieldFlags()), + FieldDefinition(name: "wrappedValue", typeNode: NodeReference(interning: typeNode), flags: FieldFlags()), ] #expect(TypeAttributeInferrer.hasWrappedValueMember(fields: fields, variables: [])) } @@ -43,7 +43,7 @@ struct TypeAttributeInferrerTests { ) let variables = [ VariableDefinition( - node: variableNode, + node: NodeReference(interning: variableNode), name: "wrappedValue", accessors: [dummyAccessor], isGlobalOrStatic: false @@ -56,8 +56,8 @@ struct TypeAttributeInferrerTests { func detectPropertyWrapperAbsent() { let typeNode = Node.create(kind: .type) let fields = [ - FieldDefinition(name: "value", typeNode: typeNode, flags: FieldFlags()), - FieldDefinition(name: "projectedValue", typeNode: typeNode, flags: FieldFlags()), + FieldDefinition(name: "value", typeNode: NodeReference(interning: typeNode), flags: FieldFlags()), + FieldDefinition(name: "projectedValue", typeNode: NodeReference(interning: typeNode), flags: FieldFlags()), ] #expect(!TypeAttributeInferrer.hasWrappedValueMember(fields: fields, variables: [])) } @@ -106,7 +106,7 @@ struct TypeAttributeInferrerTests { func detectDynamicMemberLookup() { let subscriptNode = makeDynamicMemberSubscriptNode() let subscriptDefinitions = [ - SubscriptDefinition(node: subscriptNode, accessors: [], isStatic: false), + SubscriptDefinition(node: NodeReference(interning: subscriptNode), accessors: [], isStatic: false), ] #expect(TypeAttributeInferrer.hasDynamicMemberSubscript(subscripts: subscriptDefinitions, staticSubscripts: [])) } @@ -115,7 +115,7 @@ struct TypeAttributeInferrerTests { func detectDynamicMemberLookupFromStaticSubscript() { let subscriptNode = makeDynamicMemberSubscriptNode() let staticSubscriptDefinitions = [ - SubscriptDefinition(node: subscriptNode, accessors: [], isStatic: true), + SubscriptDefinition(node: NodeReference(interning: subscriptNode), accessors: [], isStatic: true), ] #expect(TypeAttributeInferrer.hasDynamicMemberSubscript(subscripts: [], staticSubscripts: staticSubscriptDefinitions)) } @@ -132,7 +132,7 @@ struct TypeAttributeInferrerTests { let getterNode = Node.create(kind: .getter, child: subscriptNode) let globalNode = Node.create(kind: .global, child: getterNode) let subscriptDefinitions = [ - SubscriptDefinition(node: globalNode, accessors: [], isStatic: false), + SubscriptDefinition(node: NodeReference(interning: globalNode), accessors: [], isStatic: false), ] #expect(!TypeAttributeInferrer.hasDynamicMemberSubscript(subscripts: subscriptDefinitions, staticSubscripts: [])) } @@ -152,7 +152,7 @@ struct TypeAttributeInferrerTests { let getterNode = Node.create(kind: .getter, child: subscriptNode) let globalNode = Node.create(kind: .global, child: getterNode) let subscriptDefinitions = [ - SubscriptDefinition(node: globalNode, accessors: [], isStatic: false), + SubscriptDefinition(node: NodeReference(interning: globalNode), accessors: [], isStatic: false), ] #expect(!TypeAttributeInferrer.hasDynamicMemberSubscript(subscripts: subscriptDefinitions, staticSubscripts: [])) } @@ -221,7 +221,7 @@ struct TypeAttributeInferrerTests { // A type that is both @propertyWrapper and has dynamicallyCall let typeNode = Node.create(kind: .type) let fields = [ - FieldDefinition(name: "wrappedValue", typeNode: typeNode, flags: FieldFlags()), + FieldDefinition(name: "wrappedValue", typeNode: NodeReference(interning: typeNode), flags: FieldFlags()), ] let functions = [ makeMockFunctionDefinition(name: "dynamicallyCall"), @@ -247,7 +247,7 @@ private func makeMockFunctionDefinition(name: String) -> FunctionDefinition { demangledNode: nodeStoreBuilder.freeze().reference(at: functionNodeIndex) ) return FunctionDefinition( - node: functionNode, + node: NodeReference(interning: functionNode), name: name, kind: .function, symbol: dummySymbol, diff --git a/Tests/SwiftDiffingTests/ABIDifferTests.swift b/Tests/SwiftDiffingTests/ABIDifferTests.swift index 949ad638..53e67acb 100644 --- a/Tests/SwiftDiffingTests/ABIDifferTests.swift +++ b/Tests/SwiftDiffingTests/ABIDifferTests.swift @@ -95,7 +95,7 @@ struct ABIDifferProjectionTests { private func function(_ name: String, kind: FunctionKind = .function) -> FunctionDefinition { let node = functionNode(name) return FunctionDefinition( - node: node, + node: makeNodeReference(node), name: name, kind: kind, symbol: DemangledSymbol(symbol: Symbol(offset: 0, name: "$s_\(name)"), demangledNode: makeNodeReference(node)), @@ -114,7 +114,7 @@ struct ABIDifferProjectionTests { } private func field(_ name: String, type: String) -> FieldDefinition { - FieldDefinition(name: name, typeNode: nominalType(type), flags: FieldFlags()) + FieldDefinition(name: name, typeNode: makeNodeReference(nominalType(type)), flags: FieldFlags()) } private func accessor(_ kind: AccessorKind, _ name: String) -> Accessor { @@ -130,7 +130,7 @@ struct ABIDifferProjectionTests { private func variable(_ name: String, accessors: [AccessorKind]) -> VariableDefinition { VariableDefinition( - node: functionNode(name), + node: makeNodeReference(functionNode(name)), name: name, accessors: accessors.map { accessor($0, name) }, isGlobalOrStatic: false @@ -203,7 +203,7 @@ struct ABIDifferProjectionTests { @Test("toggling indirect on an enum case diffs as .modified even with the same tag and payload type") func enumCaseIndirectToggleIsModified() { let inlineCase = MemberRecord.makeCase(field("boxed", type: "Int"), tag: 0) - let indirectField = FieldDefinition(name: "boxed", typeNode: nominalType("Int"), flags: [.isIndirectCase]) + let indirectField = FieldDefinition(name: "boxed", typeNode: makeNodeReference(nominalType("Int")), flags: [.isIndirectCase]) let indirectCase = MemberRecord.makeCase(indirectField, tag: 0) #expect(indirectCase.signature == "indirect case boxed") let changes = ABIDiffer().diffMembers(old: [inlineCase], new: [indirectCase]) @@ -259,7 +259,7 @@ struct ABIDifferClassificationTests { private func function(_ name: String) -> FunctionDefinition { let node = functionNode(name) return FunctionDefinition( - node: node, + node: makeNodeReference(node), name: name, kind: .function, symbol: DemangledSymbol(symbol: Symbol(offset: 0, name: "$s_\(name)"), demangledNode: makeNodeReference(node)), @@ -271,7 +271,7 @@ struct ABIDifferClassificationTests { } private func variable(_ name: String) -> VariableDefinition { - VariableDefinition(node: functionNode(name), name: name, accessors: [], isGlobalOrStatic: false) + VariableDefinition(node: makeNodeReference(functionNode(name)), name: name, accessors: [], isGlobalOrStatic: false) } @Test("associated type: same name is unchanged, a rename is add+remove") diff --git a/Tests/SwiftDiffingTests/ABIExtensionAttributionTests.swift b/Tests/SwiftDiffingTests/ABIExtensionAttributionTests.swift index 2400818b..be6b6ac2 100644 --- a/Tests/SwiftDiffingTests/ABIExtensionAttributionTests.swift +++ b/Tests/SwiftDiffingTests/ABIExtensionAttributionTests.swift @@ -25,18 +25,18 @@ struct ABIExtensionAttributionTests { } private func protocolName(_ name: String) -> ProtocolName { - ProtocolName(node: Node.create(kind: .type, child: Node.create(kind: .protocol, children: [ + ProtocolName(node: NodeReference(interning: Node.create(kind: .type, child: Node.create(kind: .protocol, children: [ Node.create(kind: .module, text: "M"), Node.create(kind: .identifier, text: name), - ]))) + ])))) } private func extensionName(_ target: String) -> ExtensionName { - ExtensionName(node: targetNode(target), kind: .type(.struct)) + ExtensionName(node: NodeReference(interning: targetNode(target)), kind: .type(.struct)) } - private func whereClause(_ text: String) -> Node { - Node.create(kind: .identifier, text: text) + private func whereClause(_ text: String) -> NodeReference { + NodeReference(interning: Node.create(kind: .identifier, text: text)) } private func function(_ name: String) -> FunctionDefinition { @@ -46,7 +46,7 @@ struct ABIExtensionAttributionTests { Node.create(kind: .type), ]) return FunctionDefinition( - node: node, + node: makeNodeReference(node), name: name, kind: .function, symbol: DemangledSymbol(symbol: Symbol(offset: 0, name: "$s_\(name)"), demangledNode: makeNodeReference(node)), diff --git a/Tests/SwiftSpecializationTests/GenericTypeNameSubstitutionTests.swift b/Tests/SwiftSpecializationTests/GenericTypeNameSubstitutionTests.swift index 4a954af2..8e3e390c 100644 --- a/Tests/SwiftSpecializationTests/GenericTypeNameSubstitutionTests.swift +++ b/Tests/SwiftSpecializationTests/GenericTypeNameSubstitutionTests.swift @@ -36,7 +36,7 @@ struct GenericTypeNameSubstitutionHelperTests { @Test("struct kind produces boundGenericStructure node") func structKindWraps() throws { - let unbound = TypeName(node: makeStructureTypeNode(name: "Box"), kind: .struct) + let unbound = TypeName(node: NodeReference(interning: makeStructureTypeNode(name: "Box")), kind: .struct) let argument = makeStructureTypeNode(name: "Int") let result = TypeDefinition.boundGenericTypeName( @@ -47,7 +47,7 @@ struct GenericTypeNameSubstitutionHelperTests { #expect(result.kind == .struct) #expect(result.node.kind == .type) - let firstChild = try #require(result.node.firstChild) + let firstChild = try #require(result.node.children.first) #expect(firstChild.kind == .boundGenericStructure) #expect(firstChild.children.count == 2) #expect(firstChild.children[0].kind == .type) @@ -60,7 +60,7 @@ struct GenericTypeNameSubstitutionHelperTests { let identifierNode = Node.create(kind: .identifier, contents: .text("Container")) let classNode = Node.create(kind: .class, children: [moduleNode, identifierNode]) let unbound = TypeName( - node: Node.create(kind: .type, children: [classNode]), + node: NodeReference(interning: Node.create(kind: .type, children: [classNode])), kind: .class ) @@ -69,7 +69,7 @@ struct GenericTypeNameSubstitutionHelperTests { typeArgumentNodes: [makeStructureTypeNode(name: "String")] ) - let firstChild = try #require(result.node.firstChild) + let firstChild = try #require(result.node.children.first) #expect(firstChild.kind == .boundGenericClass) #expect(result.kind == .class) } @@ -80,7 +80,7 @@ struct GenericTypeNameSubstitutionHelperTests { let identifierNode = Node.create(kind: .identifier, contents: .text("Either")) let enumNode = Node.create(kind: .enum, children: [moduleNode, identifierNode]) let unbound = TypeName( - node: Node.create(kind: .type, children: [enumNode]), + node: NodeReference(interning: Node.create(kind: .type, children: [enumNode])), kind: .enum ) @@ -92,14 +92,14 @@ struct GenericTypeNameSubstitutionHelperTests { ] ) - let firstChild = try #require(result.node.firstChild) + let firstChild = try #require(result.node.children.first) #expect(firstChild.kind == .boundGenericEnum) #expect(result.kind == .enum) } @Test("typeList contains every argument in order") func typeListPositionalOrder() throws { - let unbound = TypeName(node: makeStructureTypeNode(name: "Triple"), kind: .struct) + let unbound = TypeName(node: NodeReference(interning: makeStructureTypeNode(name: "Triple")), kind: .struct) let argA = makeStructureTypeNode(name: "Int") let argB = makeStructureTypeNode(name: "String") let argC = makeStructureTypeNode(name: "Bool") @@ -109,7 +109,7 @@ struct GenericTypeNameSubstitutionHelperTests { typeArgumentNodes: [argA, argB, argC] ) - let typeList = try #require(result.node.firstChild?.children[1]) + let typeList = try #require(result.node.children.first?.children[1]) #expect(typeList.kind == .typeList) #expect(typeList.children.count == 3) for child in typeList.children { @@ -120,23 +120,23 @@ struct GenericTypeNameSubstitutionHelperTests { @Test("bare structure unbound (no .type wrap) is auto-wrapped") func unboundAutoWrap() throws { let bareUnbound = makeBareStructureNode(name: "Box") - let unbound = TypeName(node: bareUnbound, kind: .struct) + let unbound = TypeName(node: NodeReference(interning: bareUnbound), kind: .struct) let result = TypeDefinition.boundGenericTypeName( unboundTypeName: unbound, typeArgumentNodes: [makeStructureTypeNode(name: "Int")] ) - let firstChild = try #require(result.node.firstChild) + let firstChild = try #require(result.node.children.first) let unboundChild = firstChild.children[0] #expect(unboundChild.kind == .type) - let inner = try #require(unboundChild.firstChild) + let inner = try #require(unboundChild.children.first) #expect(inner.kind == .structure) } @Test("bare structure argument (no .type wrap) is auto-wrapped") func argumentAutoWrap() throws { - let unbound = TypeName(node: makeStructureTypeNode(name: "Box"), kind: .struct) + let unbound = TypeName(node: NodeReference(interning: makeStructureTypeNode(name: "Box")), kind: .struct) let bareArgument = makeBareStructureNode(name: "Int") let result = TypeDefinition.boundGenericTypeName( @@ -144,17 +144,17 @@ struct GenericTypeNameSubstitutionHelperTests { typeArgumentNodes: [bareArgument] ) - let typeList = try #require(result.node.firstChild?.children[1]) + let typeList = try #require(result.node.children.first?.children[1]) let firstArgument = typeList.children[0] #expect(firstArgument.kind == .type) - let inner = try #require(firstArgument.firstChild) + let inner = try #require(firstArgument.children.first) #expect(inner.kind == .structure) } @Test(".type-wrapped input is not double-wrapped") func noDoubleWrap() throws { let unboundTypeNode = makeStructureTypeNode(name: "Box") - let unbound = TypeName(node: unboundTypeNode, kind: .struct) + let unbound = TypeName(node: NodeReference(interning: unboundTypeNode), kind: .struct) let argumentTypeNode = makeStructureTypeNode(name: "Int") let result = TypeDefinition.boundGenericTypeName( @@ -162,23 +162,24 @@ struct GenericTypeNameSubstitutionHelperTests { typeArgumentNodes: [argumentTypeNode] ) - let firstChild = try #require(result.node.firstChild) - // Identity check: helper reuses the original `.type`-wrapped node - // rather than wrapping it again into `Type → Type → Structure`. - #expect(firstChild.children[0] === unboundTypeNode) + let firstChild = try #require(result.node.children.first) + // Structural check: the helper must not wrap the already + // `.type`-wrapped node again into `Type → Type → Structure` — a + // double wrap would break structural equality with the input. + #expect(firstChild.children[0].structurallyEquals(unboundTypeNode)) let typeList = firstChild.children[1] - #expect(typeList.children[0] === argumentTypeNode) + #expect(typeList.children[0].structurallyEquals(argumentTypeNode)) } @Test("empty argument list still produces a structurally valid tree") func emptyArgumentList() throws { - let unbound = TypeName(node: makeStructureTypeNode(name: "Box"), kind: .struct) + let unbound = TypeName(node: NodeReference(interning: makeStructureTypeNode(name: "Box")), kind: .struct) let result = TypeDefinition.boundGenericTypeName( unboundTypeName: unbound, typeArgumentNodes: [] ) - let firstChild = try #require(result.node.firstChild) + let firstChild = try #require(result.node.children.first) #expect(firstChild.kind == .boundGenericStructure) let typeList = firstChild.children[1] #expect(typeList.kind == .typeList) @@ -249,7 +250,7 @@ struct GenericTypeNameSubstitutionEndToEndTests: GenericSpecializationTestingEnv // Top-level shape: `Type → BoundGenericStructure(...)`. #expect(specialized.typeName.kind == .struct) #expect(specialized.typeName.node.kind == .type) - let firstChild = try #require(specialized.typeName.node.firstChild) + let firstChild = try #require(specialized.typeName.node.children.first) #expect(firstChild.kind == .boundGenericStructure) #expect(firstChild.children.count == 2) #expect(firstChild.children[1].kind == .typeList) @@ -342,7 +343,7 @@ struct GenericTypeNameSubstitutionEndToEndTests: GenericSpecializationTestingEnv in: machO ) - let firstChild = try #require(specialized.typeName.node.firstChild) + let firstChild = try #require(specialized.typeName.node.children.first) #expect(firstChild.kind != .boundGenericStructure) #expect(firstChild.kind == .structure) } From eb2071fb2a0e576453a050488df8f0ba028e41d9 Mon Sep 17 00:00:00 2001 From: Mx-Iris Date: Fri, 24 Jul 2026 18:49:54 +0800 Subject: [PATCH 06/77] feat(SwiftDeclarationRendering): representation-generic semantic printing and resolver (Stage 5b-lite) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - DemanglingNode.printSemantic protocol extension: zero-materialization rich printing for any representation through the shared generic engine (store-backed nominal scopes materialize lazily via the autoclosure scope hook). - DemangleResolver.resolve(for: some DemanglingNode): .options prints straight from the node's representation; .builder keeps its public Node closure signature and materializes only on that path. - SwiftDump dumpers pass DemangledSymbol.demangledNode references straight to the resolver — the six highest-frequency transient materialization sites are gone. Full genericization of the async NodePrintable stack is deliberately deferred: after Stage 5a the printer-entry materializations are transient-only (no resident cost), and the stack synthesizes helper Nodes in five places that a generic SomeNode cannot express. Decision and re-entry criteria recorded in NodeStoreMigrationPlan.md. Acceptance: 98 tests / 15 suites green, interface snapshot byte-identical; baseline re-measurement clean (NodeCache delta 0/0, NodeStore 7 MB / 579,291 nodes unchanged, symbolTable 202,603 rows). --- Documentations/Internal/NodeStoreMigrationPlan.md | 10 ++++++++++ .../DemangleResolver.swift | 13 +++++++++++++ .../Extensions/Node+.swift | 12 ++++++------ Sources/SwiftDump/Dumper/ClassDumper.swift | 4 ++-- Sources/SwiftDump/Dumper/EnumDumper.swift | 2 +- .../Dumper/ProtocolConformanceDumper.swift | 6 +++--- Sources/SwiftDump/Dumper/StructDumper.swift | 2 +- 7 files changed, 36 insertions(+), 13 deletions(-) diff --git a/Documentations/Internal/NodeStoreMigrationPlan.md b/Documentations/Internal/NodeStoreMigrationPlan.md index f43a13d4..7a7bf4b2 100644 --- a/Documentations/Internal/NodeStoreMigrationPlan.md +++ b/Documentations/Internal/NodeStoreMigrationPlan.md @@ -223,3 +223,13 @@ RuntimeViewer 与 MachOSwiftSection 的内存主项在 `MachOSymbols/SymbolIndex - **5a**：声明层全部换持 `NodeReference`。上游新增 `NodeReference(interning:)`、跨 store `structurallyEquals(_ other: NodeReference)`、`structuralHash(into:)`；`memberSymbols(of:for:node:)` 增加 `NodeReference` 重载。成员定义直持主 store 引用（构建期 4 处 `materialize()` 删除）；extension 成员路径的 `ExtensionName`/`genericSignature` 直用主 store 键（原每键物化删除）；metadata 派生树以 mini store 承接，`TypeDefinition.index(in:)` 的字段树按类型批量共享一个 store。`Name` 类型自定义结构语义 `Hashable` 与 wire 兼容 `Codable`。打印边界暂留 5 处显式 `materialize()` 桥（`SwiftDeclarationPrinter` 3 处 printer 入口 + where 子句 + `leafNameNode`×2），5b 泛型化时消除；`NodeReference.printSemantic`（零物化富文本）已就位并接管 `SwiftDiffableInterfaceRenderer`。 - **验收**：MachOSwiftSection 98 tests / 15 suites 全绿（interface 快照逐字节一致、fixture、diffing、substitution、attribute inference）；swift-demangling 定向 44/7 全绿 + 全量复跑。 - **事故记录 — 测试语料符号无效 + xcsift 假绿**：`DemanglingTests` corpus 中的 `$s7SwiftUI4TextV_10FoundationE9formatterAcA…` 自引入（5788472，NodeStore 之前）就是**无效符号**（系统 `swift-demangle` 同样拒绝，`TextV` 后多一个 `_`），理应一直红。此前未暴露是因为 `swift test 2>&1 | xcsift; echo $?` 捕获的是 **xcsift 的退出码**而非 `swift test` 的，多轮「全绿」不可信。已替换为真实生成的同复杂度符号 `$s11ExampleBase0A4TextV0A6AddonsE9formatter7subjectAcA0A5StyleV_xtcSyRzlufC`（跨模块 extension + `SyRzl` 约束 + `ufC`），并对三个测试文件的全部 mangled 字面量过系统 demangler 校验。**教训：管道给 xcsift 时用 `${pipestatus[1]}` 取真实退出码，或验收时直接看原生输出。** + +### Stage 5b 范围调整 — 全量泛型化缩水为 5b-lite（2026-07-24，实施期决策） + +原方案把 SwiftPrinting 的异步富文本引擎（`NodePrintable` 协议栈 ~1,600 行）全量泛型化到 `DemanglingNode`。实施 5a 后重估： + +- **收益已在 5a 兑现**：原方案预期靠 5b 消除的「打印路径常驻物化」实际上被 5a 的直持 `NodeReference` 消掉了——打印入口的 `materialize()` 只剩**瞬态**分配（单声明签名树、微秒级、打印完即释放），常驻内存零贡献。 +- **成本高于预估**：摸底发现引擎内有 5 处「合成 `Node` 再打印」的模式（`.static` 包装、labelList 插入/合成），泛型 `SomeNode` 下无法表达（合成结果是 `Node`，塞不回 `SomeNode` 递归），需要逐处重写为非合成形态，回归面大。 +- **5b-lite 实际落地**：`DemanglingNode.printSemantic`（协议扩展，零物化富文本，任意表示）；`DemangleResolver.resolve(for: some DemanglingNode)`（`.options` 零物化直印，`.builder` 保持公开 `Node` 闭包签名、仅该路径物化）；SwiftDump dumpers 的 6 处 `resolve(for: X.materialize())` 直传 `NodeReference`——高频瞬态物化点清零。 +- **保留的显式物化桥**（低频/瞬态，全量泛型化的剩余标的，暂不做）：`SwiftDeclarationPrinter` 3 处 printer 入口 + where 子句子节点 + `leafNameNode` ×2 + `SwiftPrinting+Headers`/`ClassDumper` 的 thunk 构树点 + RV 特化构树 2 处。 +- **重启条件**：若后续 profiling 显示 interface 全量导出（swift-section `InterfaceCommand` 之类批量场景）的瞬态树分配成为吞吐瓶颈，再按原方案泛型化（届时合成点重写方案：labelList 合成改计数循环，`.static` 包装改 printer 状态位）。 diff --git a/Sources/SwiftDeclarationRendering/DemangleResolver.swift b/Sources/SwiftDeclarationRendering/DemangleResolver.swift index 44e7c9b8..6156a36e 100644 --- a/Sources/SwiftDeclarationRendering/DemangleResolver.swift +++ b/Sources/SwiftDeclarationRendering/DemangleResolver.swift @@ -37,6 +37,19 @@ public enum DemangleResolver: Sendable { } } + /// Representation-generic overload: `.options` prints straight from the + /// node's own representation (zero materialization for store-backed + /// references), while `.builder` keeps its public `Node` closure + /// signature and materializes only on that path. + public func resolve(for node: some DemanglingNode) async throws -> SemanticString { + switch self { + case .options(let options): + return node.printSemantic(using: options) + case .builder(let builder): + return try await builder(node.materializedNode) + } + } + public func modify(_ modifier: (DemangleResolver) -> DemangleResolver) -> DemangleResolver { modifier(self) } diff --git a/Sources/SwiftDeclarationRendering/Extensions/Node+.swift b/Sources/SwiftDeclarationRendering/Extensions/Node+.swift index c24bc46d..878a4156 100644 --- a/Sources/SwiftDeclarationRendering/Extensions/Node+.swift +++ b/Sources/SwiftDeclarationRendering/Extensions/Node+.swift @@ -58,14 +58,14 @@ extension Node { } } -extension NodeReference { - /// Zero-materialization semantic print of a store-backed subtree, - /// through the same generic engine as `Node.printSemantic` (the - /// type-reference identity scopes materialize just the nominal - /// reference nodes on demand, via the engine's lazy scope hook). +extension DemanglingNode { + /// Zero-materialization semantic print through the same generic engine + /// as `Node.printSemantic` — for store-backed nodes the type-reference + /// identity scopes materialize just the nominal reference subtrees on + /// demand, via the engine's lazy scope hook. public func printSemantic(using options: DemangleOptions = .default) -> SemanticString { StackSafeExecutor.execute { - var printer = DemanglingPrinter(options: options) + var printer = DemanglingPrinter(options: options) return printer.printRoot(self) } } diff --git a/Sources/SwiftDump/Dumper/ClassDumper.swift b/Sources/SwiftDump/Dumper/ClassDumper.swift index 3e84df12..e9176bb0 100644 --- a/Sources/SwiftDump/Dumper/ClassDumper.swift +++ b/Sources/SwiftDump/Dumper/ClassDumper.swift @@ -334,7 +334,7 @@ package struct ClassDumper: TypedDumper { Indent(level: 1) - try await demangleResolver.resolve(for: symbol.demangledNode.materialize()) + try await demangleResolver.resolve(for: symbol.demangledNode) if offset.isEnd { BreakLine() @@ -360,7 +360,7 @@ package struct ClassDumper: TypedDumper { Indent(level: 1) - try await demangleResolver.resolve(for: symbol.demangledNode.materialize()) + try await demangleResolver.resolve(for: symbol.demangledNode) if offset.isEnd { BreakLine() diff --git a/Sources/SwiftDump/Dumper/EnumDumper.swift b/Sources/SwiftDump/Dumper/EnumDumper.swift index ffdd8d7c..69ee8702 100644 --- a/Sources/SwiftDump/Dumper/EnumDumper.swift +++ b/Sources/SwiftDump/Dumper/EnumDumper.swift @@ -159,7 +159,7 @@ package struct EnumDumper: TypedDumper { Indent(level: 1) - try await demangleResolver.resolve(for: symbol.demangledNode.materialize()) + try await demangleResolver.resolve(for: symbol.demangledNode) if offset.isEnd { BreakLine() diff --git a/Sources/SwiftDump/Dumper/ProtocolConformanceDumper.swift b/Sources/SwiftDump/Dumper/ProtocolConformanceDumper.swift index 14aa61c4..26e927f9 100644 --- a/Sources/SwiftDump/Dumper/ProtocolConformanceDumper.swift +++ b/Sources/SwiftDump/Dumper/ProtocolConformanceDumper.swift @@ -105,7 +105,7 @@ package struct ProtocolConformanceDumper: Conforme if let symbols = try resilientWitness.implementationSymbols(in: machO), let node = Self.demangledSymbol(for: symbols, typeName: typeNameString, visitedNodes: visitedNodes, in: machO)?.demangledNode { _ = visitedNodes.append(node) - try await demangleResolver.resolve(for: node.materialize()) + try await demangleResolver.resolve(for: node) } else if let requirement = try resilientWitness.requirement(in: machO) { switch requirement { @@ -114,10 +114,10 @@ package struct ProtocolConformanceDumper: Conforme case .element(let element): if let symbols = try await Symbols.resolve(from: element.offset, in: machO), let node = Self.demangledSymbol(for: symbols, typeName: typeNameString, visitedNodes: visitedNodes, in: machO)?.demangledNode { _ = visitedNodes.append(node) - try await demangleResolver.resolve(for: node.materialize()) + try await demangleResolver.resolve(for: node) } else if let defaultImplementationSymbols = try element.defaultImplementationSymbols(in: machO), let node = Self.demangledSymbol(for: defaultImplementationSymbols, typeName: typeNameString, visitedNodes: visitedNodes, in: machO)?.demangledNode { _ = visitedNodes.append(node) - try await demangleResolver.resolve(for: node.materialize()) + try await demangleResolver.resolve(for: node) } else if !element.defaultImplementation.isNull { FunctionDeclaration(machO.addressString(forOffset: element.defaultImplementation.resolveDirectOffset(from: element.offset(of: \.defaultImplementation))).insertSubFunctionPrefix) } else if !resilientWitness.implementation.isNull { diff --git a/Sources/SwiftDump/Dumper/StructDumper.swift b/Sources/SwiftDump/Dumper/StructDumper.swift index 004775b7..ff100f5d 100644 --- a/Sources/SwiftDump/Dumper/StructDumper.swift +++ b/Sources/SwiftDump/Dumper/StructDumper.swift @@ -146,7 +146,7 @@ package struct StructDumper: TypedDumper { Indent(level: 1) - try await demangleResolver.resolve(for: symbol.demangledNode.materialize()) + try await demangleResolver.resolve(for: symbol.demangledNode) if offset.isEnd { BreakLine() From 61b786b8bdc5abacec7f1905970a679aa878ac80 Mon Sep 17 00:00:00 2001 From: Mx-Iris Date: Sat, 25 Jul 2026 19:00:42 +0800 Subject: [PATCH 07/77] fix(SwiftDeclaration): key override/vtable lookups by node structure, not store identity MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Stage 5a replaced the `[Node: …]` method-descriptor and vtable-offset lookups in `TypeDefinition.index(in:)` with bare `[NodeReference: …]` dictionaries. `NodeReference`'s intrinsic `Hashable`/`==` is store-identity based (`(store, index)`), not structural, and the two sides of these lookups do not share a store: the keys come from override descriptors' implementation symbols, which `SymbolIndexStore.demangledNodeReference(for:)` hands back from a per-symbol mini store whenever the symbol falls outside the build sweep, while the member side queries them with references from the shared image store. Structurally equal keys from different stores missed, silently dropping both the `override` keyword and the `// VTable offset:` comment. The offset-keyed fallback tables are built only from `methodDescriptors`, so override methods had no second path and lost the annotations entirely. Add `StructuralNodeReferenceKey` — the same `structurallyEquals` + `structuralHash` treatment the `Name` types already carry — and key both lookups on it, wrapping at the five insertion sites and the four query sites. Containers that group or dedup within a single `memberSymbols` batch keep bare `NodeReference` keys: they live in one hash-consed store, where structural equality coincides with index equality. Caught by a whole-file dump/interface A/B diff against main: iOS 18.5 SwiftData's `Schema.Attribute` rendered 5 overrides on main and 0 on the branch, while iOS 26.5 SwiftData and SwiftUI/SwiftUICore were byte-identical. After the fix all three reader sources match main byte for byte — MachOFile 38/38 (iOS 15.5–27.0b2), DyldCache 18/18 (host + iOS 27.0b3/b4), MachOImage 6/6. --- AGENTS.md | 2 +- .../Internal/NodeStoreMigrationPlan.md | 16 ++- ...store-override-regression-and-baselines.md | 101 ++++++++++++++++++ Documentations/README.md | 1 + .../Definitions/DefinitionBuilder.swift | 40 +++---- .../StructuralNodeReferenceKey.swift | 37 +++++++ .../Definitions/TypeDefinition.swift | 14 +-- .../StructuralNodeReferenceKeyTests.swift | 70 ++++++++++++ 8 files changed, 251 insertions(+), 30 deletions(-) create mode 100644 Documentations/Internal/TaskReports/2026-07-25-node-store-override-regression-and-baselines.md create mode 100644 Sources/SwiftDeclaration/Components/Definitions/StructuralNodeReferenceKey.swift create mode 100644 Tests/SwiftPrintingTests/StructuralNodeReferenceKeyTests.swift diff --git a/AGENTS.md b/AGENTS.md index 3cce0c6d..77d535ce 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -206,7 +206,7 @@ let node = try demangleAsNode("$sSiD") // Returns Node tree let string = node.print(using: .default) // "Swift.Int" ``` -**Symbol indexing (NodeStore-backed)**: `SymbolIndexStore.Storage` no longer retains `Node` class trees. The build sweep demangles each symbol cache-free onto a transient tree (`@_spi(Internals) demangleAsNodeTransient`), classifies on that tree, and interns it into a per-image `NodeStoreBuilder`. Nothing touches the global `NodeCache`, and dropping a `Storage` releases the whole per-image footprint. Storage layout (Stage 3): one flat `symbolTable: [Symbol]` row per unique symbol name (canonical cache-adjusted offset; raw and adjusted offset keys share one row), a parallel `rootNodeIndexByTableRow` array, and every classification index stores 4-byte `UInt32` table rows (member/opaque keys are 4-byte `NodeStore.NodeIndex`); indexes accumulate in final row form during the sweep, so `freeze()` is followed by a plain move, not a conversion pass. `Symbol` is 32 bytes (no `nlist` existential — the external-undefined bit is extracted at collection time into `isExternal`), and `DemangledSymbol` is a 32-byte value (shared-table reference + row + `NodeReference`) whose `symbol` is computed; both are asserted by `compactValueLayouts`. Query APIs construct `DemangledSymbol`/`NodeReference` values at the exit boundary; APIs taking an externally demangled node (`memberSymbols(of:for:node:)` with `Node` or `NodeReference`, `opaqueTypeDescriptorSymbol(for:)`) match keys via `NodeReference.structurallyEquals(_:)` (same-store O(1), cross-store structural walk). Late symbols outside the build sweep get per-symbol mini stores (`demangledNodeReference(for:)` / `MetadataReader.demangleSymbolReference(for:in:)`). Struct-layout changes in `MachOSymbols` need a `swift package clean` rebuild — SwiftPM incremental builds have been observed linking stale downstream objects (runtime SIGSEGV in `outlined destroy`). See `Documentations/Internal/NodeStoreMigrationPlan.md`. **Declaration values hold `NodeReference` (Stage 5a)**: `DefinitionName.node` (`TypeName`/`ProtocolName`/`ExtensionName`), `Variable/Function/SubscriptDefinition.node`, `FieldDefinition.typeNode` and `ExtensionDefinition.genericSignature` are `NodeReference`, not `Node`. Member definitions reference the per-image symbol store directly (a live definition keeps that store alive — the intended per-image recycling model); metadata-derived trees (`MetadataReader` output) are wrapped via `NodeReference(interning:)` mini stores, and `TypeDefinition.index(in:)` batches all field type trees of one type into a single shared store. The `Name` types customize `Hashable` to STRUCTURAL semantics (`structurallyEquals` + `structuralHash`) because `NodeReference`'s intrinsic `Hashable` is store-identity based and would split equal keys minted into different stores; their `Codable` stays wire-compatible by encoding a materialized `Node`. **Transient demangling everywhere else (Stage 5c)**: `MetadataReader`, `RuntimeFieldLayoutBackend`, `TypedDumper`, `ClassHierarchyDumper` and `Symbol.demangledNode` use `demangleAsNodeTransient` + `Node.createTransient`, so the global `NodeCache` no longer grows while browsing. Do NOT reintroduce `demangleAsNode`/`Node.create` on unbounded inputs in those paths. +**Symbol indexing (NodeStore-backed)**: `SymbolIndexStore.Storage` no longer retains `Node` class trees. The build sweep demangles each symbol cache-free onto a transient tree (`@_spi(Internals) demangleAsNodeTransient`), classifies on that tree, and interns it into a per-image `NodeStoreBuilder`. Nothing touches the global `NodeCache`, and dropping a `Storage` releases the whole per-image footprint. Storage layout (Stage 3): one flat `symbolTable: [Symbol]` row per unique symbol name (canonical cache-adjusted offset; raw and adjusted offset keys share one row), a parallel `rootNodeIndexByTableRow` array, and every classification index stores 4-byte `UInt32` table rows (member/opaque keys are 4-byte `NodeStore.NodeIndex`); indexes accumulate in final row form during the sweep, so `freeze()` is followed by a plain move, not a conversion pass. `Symbol` is 32 bytes (no `nlist` existential — the external-undefined bit is extracted at collection time into `isExternal`), and `DemangledSymbol` is a 32-byte value (shared-table reference + row + `NodeReference`) whose `symbol` is computed; both are asserted by `compactValueLayouts`. Query APIs construct `DemangledSymbol`/`NodeReference` values at the exit boundary; APIs taking an externally demangled node (`memberSymbols(of:for:node:)` with `Node` or `NodeReference`, `opaqueTypeDescriptorSymbol(for:)`) match keys via `NodeReference.structurallyEquals(_:)` (same-store O(1), cross-store structural walk). Late symbols outside the build sweep get per-symbol mini stores (`demangledNodeReference(for:)` / `MetadataReader.demangleSymbolReference(for:in:)`). Struct-layout changes in `MachOSymbols` need a `swift package clean` rebuild — SwiftPM incremental builds have been observed linking stale downstream objects (runtime SIGSEGV in `outlined destroy`). See `Documentations/Internal/NodeStoreMigrationPlan.md`. **Declaration values hold `NodeReference` (Stage 5a)**: `DefinitionName.node` (`TypeName`/`ProtocolName`/`ExtensionName`), `Variable/Function/SubscriptDefinition.node`, `FieldDefinition.typeNode` and `ExtensionDefinition.genericSignature` are `NodeReference`, not `Node`. Member definitions reference the per-image symbol store directly (a live definition keeps that store alive — the intended per-image recycling model); metadata-derived trees (`MetadataReader` output) are wrapped via `NodeReference(interning:)` mini stores, and `TypeDefinition.index(in:)` batches all field type trees of one type into a single shared store. The `Name` types customize `Hashable` to STRUCTURAL semantics (`structurallyEquals` + `structuralHash`) because `NodeReference`'s intrinsic `Hashable` is store-identity based and would split equal keys minted into different stores; their `Codable` stays wire-compatible by encoding a materialized `Node`. **Any `Dictionary`/`Set` keyed on a `NodeReference` whose keys and lookups can come from DIFFERENT stores must key on `StructuralNodeReferenceKey` (the same structural wrapper), never a bare `NodeReference`** — the override/vtable lookups (`methodDescriptorLookup` / `vtableOffsetLookup` in `TypeDefinition.index` → `DefinitionBuilder`) do, because their keys come from override descriptors' impl symbols (which `demangledNodeReference(for:)` may hand back from a per-symbol mini store) while the member side looks them up from the shared image store; a bare `NodeReference` there silently drops the `override` keyword + vtable-offset comment for symbols outside the build sweep (the Stage 5a regression fixed 2026-07-25). Bare-`NodeReference` keys are only safe for grouping/dedup WITHIN one `memberSymbols` batch (single hash-consed store, where structural equality coincides with index equality). **Transient demangling everywhere else (Stage 5c)**: `MetadataReader`, `RuntimeFieldLayoutBackend`, `TypedDumper`, `ClassHierarchyDumper` and `Symbol.demangledNode` use `demangleAsNodeTransient` + `Node.createTransient`, so the global `NodeCache` no longer grows while browsing. Do NOT reintroduce `demangleAsNode`/`Node.create` on unbounded inputs in those paths. ## Test Environment diff --git a/Documentations/Internal/NodeStoreMigrationPlan.md b/Documentations/Internal/NodeStoreMigrationPlan.md index 7a7bf4b2..22cdd395 100644 --- a/Documentations/Internal/NodeStoreMigrationPlan.md +++ b/Documentations/Internal/NodeStoreMigrationPlan.md @@ -1,8 +1,8 @@ # NodeStore 迁移计划（SymbolIndexStore → arena 存储） -- **状态**: Stage 0–4 Completed（见「实施记录」）；Stage 5 提案待批准（见文末「Stage 5 提案」） +- **状态**: Stage 0–4 Completed（见「实施记录」）；Stage 5 已落地；Stage 5 回归修复见文末「Stage 5a 回归修复」 - **日期**: 2026-07-24 -- **最后更新**: 2026-07-24 +- **最后更新**: 2026-07-25 - **分支**: `feature/node-store-migration`（worktree `.claude/worktrees/node-store-migration`，Demangling 经主检出 `.claude/worktrees/swift-demangling` 处的**真实 git worktree**（swift-demangling `feature/node-store`）以路径依赖解析——原先的符号链接方案因目标 worktree 被外部清理导致 SwiftPM manifest 缓存把解析钉回 remote，已改为本仓库领地内的 worktree） - **前置**: swift-demangling `feature/node-store` 分支合入 `main`（本包以路径依赖解析 `../swift-demangling` 的 main）；开发期先经上述符号链接直连该分支 - **上游依据**: swift-demangling `evolution/0001-node-store-arena.md`（Phase 1–3 已落地并验收） @@ -233,3 +233,15 @@ RuntimeViewer 与 MachOSwiftSection 的内存主项在 `MachOSymbols/SymbolIndex - **5b-lite 实际落地**：`DemanglingNode.printSemantic`（协议扩展，零物化富文本，任意表示）；`DemangleResolver.resolve(for: some DemanglingNode)`（`.options` 零物化直印，`.builder` 保持公开 `Node` 闭包签名、仅该路径物化）；SwiftDump dumpers 的 6 处 `resolve(for: X.materialize())` 直传 `NodeReference`——高频瞬态物化点清零。 - **保留的显式物化桥**（低频/瞬态，全量泛型化的剩余标的，暂不做）：`SwiftDeclarationPrinter` 3 处 printer 入口 + where 子句子节点 + `leafNameNode` ×2 + `SwiftPrinting+Headers`/`ClassDumper` 的 thunk 构树点 + RV 特化构树 2 处。 - **重启条件**：若后续 profiling 显示 interface 全量导出（swift-section `InterfaceCommand` 之类批量场景）的瞬态树分配成为吞吐瓶颈，再按原方案泛型化（届时合成点重写方案：labelList 合成改计数循环，`.static` 包装改 printer 状态位）。 + +### Stage 5a 回归修复 — override/vtable 查询字典漏用结构相等键（2026-07-25） + +**症状**：offline `interface`（`MachOFile` 路径）对 iOS 18.5 模拟器 `SwiftData.framework` 的 `Schema.Attribute` 丢失全部 `override` 关键字与配套 `// VTable offset:` 注释（main 5 个 → 迁移分支 0 个）。范围极窄且数据相关：iOS 26.5 的**同一个类**正常（5→5），SwiftUI/SwiftUICore 数百个 override 全部一致。两侧构建内均确定性。用 main worktree 对 6 个模拟器二进制（SwiftUI/SwiftUICore/SwiftData × iOS 18.5/26.5）× {dump, interface, 全 `--emit-*` 变体} 共 18 份整文件快照 A/B diff 抓到（16/18 逐字节一致，仅此 2 份差异）。 + +**根因**：Stage 5a 把 `TypeDefinition.index(in:)` 的 `methodDescriptorLookup` / `vtableOffsetLookup` 从 `[Node: …]`（结构相等键）换成了裸 `[NodeReference: …]`。`NodeReference` 的固有 `Hashable`/`==` 是 **store-identity**（`(store, index)`），不是结构相等。这两个字典的**键**来自 override 描述符的实现符号——`SymbolIndexStore.demangledNodeReference(for:)` 对落在 build sweep 之外的符号会新建 **per-symbol mini store**——而**查询**用的是成员符号（来自 `memberSymbols`，主 store）。两者结构相等但 store 不同 ⇒ 查表 miss ⇒ override/vtable 一起丢。iOS 18.5 那批 override 实现符号恰好落到 mini store，26.5 的都在主 store，故只有前者复现。这正是 `Name` 类型当初改结构语义 `Hashable` 所规避的同一陷阱，但这两个裸字典漏改了。次因放大：`TypeDefinition.index` 的 offset 兜底表（`implOffsetDescriptorLookup`/`implOffsetVTableSlotLookup`）**只从 `methodDescriptors` 建**，`methodOverrideDescriptors`/`methodDefaultOverrideDescriptors` 没进兜底表，所以 override 方法只有 `NodeReference` 这一条路，一 miss 即彻底丢。 + +**修复**：新增 `StructuralNodeReferenceKey`（`package`，`SwiftDeclaration`，照 `Name` 类型用 `structurallyEquals` + `structuralHash`），把 `methodDescriptorLookup` / `vtableOffsetLookup` 的键类型改为它，插入端（`TypeDefinition.index` 5 处）与查询端（`DefinitionBuilder` 4 处）统一包装。仅动这对跨 store 的查询字典；分组/去重用的其余 `NodeReference` 键容器（`accessorsByNode`、`canonicalIndexBy*Node`、`pendingMergedBy*Node`、`visitedNodes`）都在单 `memberSymbols` 批次内使用（同一 hash-consed store，结构相等即 index 相等），安全，不动。 + +**验收**：全量 1263 tests / 242 suites 全绿（含直接覆盖此功能的 `outputContainsOverrideKeyword` / `outputContainsVTableOffsetComments`），外加新增的 `StructuralNodeReferenceKeyTests` 4 用例（其中 `structuralKeyDictionaryHitsAcrossStores` 精确复刻生产形态：裸 `NodeReference` 字典 miss、结构键 hit）。修复后**三种读取来源的快照对 main 全部逐字节一致**——MachOFile 38/38（iOS 15.5 / 16.4 / 17.5 / 18.5 / 26.5 / 27.0b1 / 27.0b2 的可用三件套，含全注释变体）、DyldCache 18/18（macOS 宿主 cache + iOS 27.0b3 / b4 模拟器 cache）、MachOImage 6/6（宿主三件套 in-process）。基线快照与复现 harness 固化在 `MachOSwiftSection-Baselines/main-27726bc/`（62 份 + SHA256 清单），后续迭代以此为准；详见 `Documentations/Internal/TaskReports/2026-07-25-node-store-override-regression-and-baselines.md`。 + +**附带发现（既存缺陷，非本次迁移引入）**：`DyldCache.machOFile(by: .name(_:))` 的匹配是 `imagePath.lastPathComponent.deletingPathExtension == name`，而 cache 内叶名不唯一——iOS 27 同时存在 `/System/Library/Frameworks/SwiftUI.framework/SwiftUI` 与 `/System/Library/AccessibilityBundles/SwiftUI.axbundle/SwiftUI`，`swift-section --dyld-shared-cache -n SwiftUI` 会静默选中先枚举到的 axbundle（无 Swift 元数据 → dump 0 字节、interface 只剩 4 行 import，且不报错）。基线 harness 已全面改用 `-p` 安装路径。修复方向（未实施）：优先匹配 `.framework/`，或多命中时报歧义错误。 diff --git a/Documentations/Internal/TaskReports/2026-07-25-node-store-override-regression-and-baselines.md b/Documentations/Internal/TaskReports/2026-07-25-node-store-override-regression-and-baselines.md new file mode 100644 index 00000000..d0b6d2fd --- /dev/null +++ b/Documentations/Internal/TaskReports/2026-07-25-node-store-override-regression-and-baselines.md @@ -0,0 +1,101 @@ +# 2026-07-25 — NodeStore 迁移回归修复与三源基线建设 + +## 问题 + +用户报告两件事： + +1. **RuntimeViewer 内存图报泄漏**：32 个 `NodeStore`、32 个 `_ContiguousArrayStorage`、14 个 `_ContiguousArrayStorage`、12 个 `__NSArrayI`。 +2. **测试覆盖不足**：fixture 之外缺少对 `dump` / `interface` 的整文件验证，需要用同一二进制在 main 与迁移分支之间做快照对比。 + +## 调研 + +### 泄漏 + +`NodeStore` 是只持有扁平缓冲的 `final class`，无外向引用，不可能自成环——被判 leaked 只能是上游持有链不可达但存活。实测： + +- `leaks --atExit` 跑 `swift-section dump` / `interface`（含 SymbolIndexStore 构建、per-type 字段 store、整条打印管线）：**0 leaks / 0 bytes**。 +- 自建 in-process 复现程序（dlopen 框架 → `SwiftInterfaceBuilder` + 逐类型 dumper，MachOImage 路径，与 RV 同口径）：**0 leaks**。 +- 最小 associated-object 探针（`objc_setAssociatedObject` 挂 `[Payload]`、宿主保活）：`leaks` **也报 0**，说明 `leaks` CLI 会扫描 association 侧表。 + +两个并行 Explore agent 分别审计 RV 与库： + +- 库侧唯一把「模型对象数组」桥进 ObjC 的地方是 `SwiftSpecialization/TypeDefinition+Specialization.swift:23` 的 `@AssociatedObject _specializedChildren: [TypeDefinition]`，数字完全吻合：12 个 `__NSArrayI` = 12 个被特化过的泛型 def；32 个 store = 每个特化 def 的 `typeName` 经 `NodeReference(interning:)` 生成的 mini store；14 个文本缓冲 = 其中带非空 bound-generic 标识符文本者（其余共享空文本单例）。 +- RV 侧 `NodeStore` 全部挂在 `RuntimeEngine` → `RuntimeSwiftSectionFactory` 之下；`.local` 引擎是静态根，其下对象不会被判 leaked。故被标 leaked 意味着有非 local 引擎被终止后未释放。查到两个嫌疑（`RuntimeEngineManager.pollUntilPeerAnswers` 中被放弃、强捕获 `engine` 的 `probeTask`；`RuntimeMessageChannel` 无超时 continuation），外加结构性问题：`removeSection` / `removeAllSections` 无调用者、`addSubIndexer` 无逆操作。 + +**结论**：迁移代码本身无泄漏；RV 内存图的判定与 `leaks` 的可达性引擎不同（Memory Graph 不把 association 侧表还原为图的边），叠加 RV 自身生命周期缺口。用户指示 RV 侧暂不处理。 + +### 快照对比 + +用 `git worktree` 拉 main、符号链接四个本地依赖、两侧同配置构建 `swift-section`。首轮对 iOS 18.5 / 26.5 的三件套跑 18 份整文件快照，**16/18 一致，2 份差异**：`ios185-SwiftData.interface` 与 `.interface-full`——main 打印 5 处 `override` 及配套 `// VTable offset:`，分支全部丢失。两侧构建内确定性均已验证，stderr 索引统计完全一致（差异只在渲染，不在索引）。 + +## 最终方案 + +### 根因 + +Stage 5a 把 `TypeDefinition.index(in:)` 的 `methodDescriptorLookup` / `vtableOffsetLookup` 从 `[Node: …]`（结构相等）换成裸 `[NodeReference: …]`。`NodeReference` 的固有 `Hashable`/`==` 是 **store-identity**（`(store, index)`）。这两个字典的**键**来自 override 描述符的实现符号——`SymbolIndexStore.demangledNodeReference(for:)` 对落在 build sweep 之外的符号会新建 per-symbol mini store——而**查询**用成员符号（主 store）。两者结构相等但 store 不同 ⇒ 查表 miss ⇒ override 与 vtable 注释一起丢。 + +放大因素：`TypeDefinition.index` 的 offset 兜底表只从 `methodDescriptors` 建，`methodOverrideDescriptors` / `methodDefaultOverrideDescriptors` 未进兜底表，故 override 方法只有 `NodeReference` 一条路，一 miss 即彻底丢失。 + +数据相关性：iOS 18.5 SwiftData 该类的 override 实现符号恰好落到 mini store；iOS 26.5 同一个类、以及 SwiftUI/SwiftUICore 数百个 override 都在主 store，故只有前者复现。 + +这正是 `Name` 类型当初改结构语义 `Hashable` 所规避的同一陷阱，但这两个裸字典漏改了。 + +### 修复 + +新增 `StructuralNodeReferenceKey`（`package`，`SwiftDeclaration`，照 `Name` 类型用 `structurallyEquals` + `structuralHash`），把这对跨 store 查询字典的键类型改为它，插入端（`TypeDefinition.index` 5 处）与查询端（`DefinitionBuilder` 4 处）统一包装。 + +**仅动这一对字典**。其余 `NodeReference` 键容器（`accessorsByNode`、`canonicalIndexBy*Node`、`pendingMergedBy*Node`、`visitedNodes`）都在单个 `memberSymbols` 批次内使用（同一 hash-consed store，结构相等即 index 相等），安全，未动。 + +## 实际执行 + +1. 枚举全部裸 `NodeReference` 键容器，确认修复范围。 +2. 新增 `Sources/SwiftDeclaration/Components/Definitions/StructuralNodeReferenceKey.swift`。 +3. `DefinitionBuilder.swift`：8 处签名类型 + 4 处查询点。 +4. `TypeDefinition.swift`：2 处局部声明 + 5 处插入点。 +5. 因 `package` 函数签名暴露该类型，将其从 `internal` 提升为 `package`。 +6. 新增 `Tests/SwiftPrintingTests/StructuralNodeReferenceKeyTests.swift`（4 个用例）。 +7. 文档：迁移计划追加「Stage 5a 回归修复」一节；AGENTS.md 补入硬性约定。 +8. 建设三源基线（见下）。 + +## 验证 + +- **单元测试**：全量 `swift test --skip IntegrationTests` → **1263 tests / 242 suites 全绿**，含直接覆盖此功能的 `outputContainsOverrideKeyword` / `outputContainsVTableOffsetComments`。 +- **新增回归测试**：4/4 绿。其中 `structuralKeyDictionaryHitsAcrossStores` 精确复刻生产形态——裸 `NodeReference` 字典 miss、结构键 hit，直接钉住本次 bug。 +- **三源快照对比（修复后，main vs 分支）**： + +| 来源 | 覆盖 | 结果 | +|---|---|---| +| MachOFile | iOS 15.5 / 16.4 / 17.5 / 18.5 / 26.5 / 27.0b1 / 27.0b2 的可用三件套，含全注释变体 | **38/38 逐字节一致** | +| DyldCache | macOS 宿主 cache + iOS 27.0b3 / b4 模拟器 cache 的三件套 | **18/18 逐字节一致** | +| MachOImage | 宿主三件套 in-process 渲染 | **6/6 逐字节一致** | + +修复前差异的 2 份现已一致，此前一致的条目全部保持不变。 + +## 附带产出与发现 + +### 基线仓库 + +`MachOSwiftSection-Baselines/main-27726bc/`（62 份快照 + `SHA256SUMS.txt` + 复现 harness + README）。后续优化迭代以此为准做整文件对比。 + +### 发现：`-n` 名称选镜像会静默选错（既存缺陷，非本次迁移引入） + +`DyldCache+.swift` 的名称匹配是 `imagePath.lastPathComponent.deletingPathExtension == name`，而 cache 内叶名并不唯一。iOS 27 模拟器 cache 同时含 +`/System/Library/Frameworks/SwiftUI.framework/SwiftUI` 与 +`/System/Library/AccessibilityBundles/SwiftUI.axbundle/SwiftUI`，`-n SwiftUI` 命中先枚举到的 axbundle（无 Swift 元数据），dump 输出 0 字节、interface 只剩 4 行 import，**且不报错**。 + +已验证：macOS 三件套与模拟器 SwiftUICore/SwiftData 的 `-n` 与 `-p` 结果一致，只有 SwiftUI 撞名。基线 harness 已全面改用 `-p` 安装路径。 + +修复方向（**尚未实施，待决策**）：优先匹配 `.framework/` 路径、或在多命中时报歧义错误而非静默取第一个。 + +### 差异 + +- 首轮快照遗漏了 iOS 18.5 二进制是 fat 的事实（需 `-a arm64`），第一次运行 8 个条目全部失败，补 `-a` 后重跑。 +- 原计划仅覆盖 18.5/26.5；用户要求扩展到三种来源与更宽版本跨度后，矩阵从 18 份扩到 62 份。 +- Image harness 初版用 `.dumper(...).body` 与 `DemangleResolver.default`，均为不存在的 API，改为 `dump(using:in:)` 与 `.using(options:)`。 +- Image harness 初版开启 `printMemberAddress`，进程相关地址不可跨运行比较，已关闭。 + +## 待办 + +- 是否提交本次修复（工作区改动尚未提交，按约定待用户确认）。 +- 是否修复 `-n` 选镜像歧义。 +- RV 侧生命周期缺口（`removeSection` 无调用者、`probeTask` 强捕获）——用户已指示暂不处理。 diff --git a/Documentations/README.md b/Documentations/README.md index 98055d34..cce12cc5 100644 --- a/Documentations/README.md +++ b/Documentations/README.md @@ -72,4 +72,5 @@ required by `Version.swift`'s bump contract). | [CLITransformerTemplateInterface.md](Internal/CLITransformerTemplateInterface.md) | `swift-section` 的注释模板命令行入口：三层配置（`--transformer-config` JSON 文件 / `--enum-layout-style` 整模块预设 / 逐模块模板选项）与其优先级、"内置模板名 vs 字面模板" 的解析规则（未知名字报错而非退化）、"启用的模块自动打开对应注释开关" 规则、`transformer tokens/templates/config` 发现性子命令，以及 `interface` 补齐 `--emit-type-layout` / `--emit-enum-layout`。 | | [ReadingContextAbstraction.md](Internal/ReadingContextAbstraction.md) | The `ReadingContext` reading-abstraction design. | | [ClassMemberKeywordRecovery.md](Internal/ClassMemberKeywordRecovery.md) | `class` / `static` 成员关键字的还原：mangling 层面两者不可区分，判据是「类型级成员有 vtable method descriptor ⇒ 源码是 `class`」（`static` 隐式 final、不进 vtable）；模型侧 `isClassMember` 计算属性 + 三个 node printer 接线 + dump vtable 段落关键字，顺带消灭非法的 `override static` 输出；`final class func` 等四类 ABI 上与 `static` 完全一致，保守输出语义等价的 `static`。 | +| [NodeStoreMigrationPlan.md](Internal/NodeStoreMigrationPlan.md) | `SymbolIndexStore` 迁移到 `NodeStore` arena 存储的分期计划与实施记录（Stage 0–5）：12B/节点扁平缓冲、cache-free build sweep、Symbol 表压缩、声明层换持 `NodeReference`、散点 transient demangling；含 Stage 5a 的 override/vtable 回归修复（裸 `[NodeReference: …]` 的 store-identity 陷阱 → `StructuralNodeReferenceKey`）与三源快照验收。 | | [TaskReports/](Internal/TaskReports/) | Dated per-task fix / investigation reports. | diff --git a/Sources/SwiftDeclaration/Components/Definitions/DefinitionBuilder.swift b/Sources/SwiftDeclaration/Components/Definitions/DefinitionBuilder.swift index 710d0cfd..c7f2155b 100644 --- a/Sources/SwiftDeclaration/Components/Definitions/DefinitionBuilder.swift +++ b/Sources/SwiftDeclaration/Components/Definitions/DefinitionBuilder.swift @@ -7,8 +7,8 @@ package enum DefinitionBuilder { package static func variables( for demangledSymbols: [DemangledSymbolWithOffset], fieldNames: borrowing Set = [], - methodDescriptorLookup: [NodeReference: MethodDescriptorWrapper] = [:], - vtableOffsetLookup: [NodeReference: Int] = [:], + methodDescriptorLookup: [StructuralNodeReferenceKey: MethodDescriptorWrapper] = [:], + vtableOffsetLookup: [StructuralNodeReferenceKey: Int] = [:], implOffsetDescriptorLookup: [Int: MethodDescriptorWrapper] = [:], implOffsetVTableSlotLookup: [Int: Int] = [:], isGlobalOrStatic: Bool @@ -21,8 +21,8 @@ package enum DefinitionBuilder { let kind = demangledSymbol.accessorKind let node = demangledSymbol.demangledNode let symbolOffset = demangledSymbol.base.offset - let descriptor = methodDescriptorLookup[node] ?? implOffsetDescriptorLookup[symbolOffset] - let vtableOffset = vtableOffsetLookup[node] ?? implOffsetVTableSlotLookup[symbolOffset] + let descriptor = methodDescriptorLookup[StructuralNodeReferenceKey(node)] ?? implOffsetDescriptorLookup[symbolOffset] + let vtableOffset = vtableOffsetLookup[StructuralNodeReferenceKey(node)] ?? implOffsetVTableSlotLookup[symbolOffset] accessorsByName[name, default: []].append(.init(kind: kind, symbol: demangledSymbol.base, methodDescriptor: descriptor, offset: demangledSymbol.offset, vtableOffset: vtableOffset)) } @@ -41,8 +41,8 @@ package enum DefinitionBuilder { package static func subscripts( for demangledSymbols: [DemangledSymbolWithOffset], - methodDescriptorLookup: [NodeReference: MethodDescriptorWrapper] = [:], - vtableOffsetLookup: [NodeReference: Int] = [:], + methodDescriptorLookup: [StructuralNodeReferenceKey: MethodDescriptorWrapper] = [:], + vtableOffsetLookup: [StructuralNodeReferenceKey: Int] = [:], implOffsetDescriptorLookup: [Int: MethodDescriptorWrapper] = [:], implOffsetVTableSlotLookup: [Int: Int] = [:], isStatic: Bool @@ -60,8 +60,8 @@ package enum DefinitionBuilder { let kind = demangledSymbol.accessorKind let node = demangledSymbol.demangledNode let symbolOffset = demangledSymbol.base.offset - let descriptor = methodDescriptorLookup[node] ?? implOffsetDescriptorLookup[symbolOffset] - let vtableOffset = vtableOffsetLookup[node] ?? implOffsetVTableSlotLookup[symbolOffset] + let descriptor = methodDescriptorLookup[StructuralNodeReferenceKey(node)] ?? implOffsetDescriptorLookup[symbolOffset] + let vtableOffset = vtableOffsetLookup[StructuralNodeReferenceKey(node)] ?? implOffsetVTableSlotLookup[symbolOffset] accessorsByNode[subscriptNode, default: []].append(.init(kind: kind, symbol: demangledSymbol.base, methodDescriptor: descriptor, offset: demangledSymbol.offset, vtableOffset: vtableOffset)) } @@ -79,8 +79,8 @@ package enum DefinitionBuilder { package static func allocators( for demangledSymbols: [DemangledSymbolWithOffset], - methodDescriptorLookup: [NodeReference: MethodDescriptorWrapper] = [:], - vtableOffsetLookup: [NodeReference: Int] = [:], + methodDescriptorLookup: [StructuralNodeReferenceKey: MethodDescriptorWrapper] = [:], + vtableOffsetLookup: [StructuralNodeReferenceKey: Int] = [:], implOffsetDescriptorLookup: [Int: MethodDescriptorWrapper] = [:], implOffsetVTableSlotLookup: [Int: Int] = [:] ) -> [FunctionDefinition] { @@ -113,15 +113,15 @@ package enum DefinitionBuilder { private static func makeAllocatorDefinition( from demangledSymbol: DemangledSymbolWithOffset, - methodDescriptorLookup: [NodeReference: MethodDescriptorWrapper], - vtableOffsetLookup: [NodeReference: Int], + methodDescriptorLookup: [StructuralNodeReferenceKey: MethodDescriptorWrapper], + vtableOffsetLookup: [StructuralNodeReferenceKey: Int], implOffsetDescriptorLookup: [Int: MethodDescriptorWrapper], implOffsetVTableSlotLookup: [Int: Int] ) -> FunctionDefinition { let node = demangledSymbol.demangledNode let symbolOffset = demangledSymbol.base.offset - let descriptor = methodDescriptorLookup[node] ?? implOffsetDescriptorLookup[symbolOffset] - let vtableOffset = vtableOffsetLookup[node] ?? implOffsetVTableSlotLookup[symbolOffset] + let descriptor = methodDescriptorLookup[StructuralNodeReferenceKey(node)] ?? implOffsetDescriptorLookup[symbolOffset] + let vtableOffset = vtableOffsetLookup[StructuralNodeReferenceKey(node)] ?? implOffsetVTableSlotLookup[symbolOffset] var functionDefinition = FunctionDefinition(node: node, name: "", kind: .allocator, symbol: demangledSymbol.base, isGlobalOrStatic: true, methodDescriptor: descriptor, offset: demangledSymbol.offset, vtableOffset: vtableOffset) if let methodDescriptor = descriptor?.method, methodDescriptor.layout.flags.isDynamic { functionDefinition.attributes.append(.dynamic) @@ -131,8 +131,8 @@ package enum DefinitionBuilder { package static func functions( for demangledSymbols: [DemangledSymbolWithOffset], - methodDescriptorLookup: [NodeReference: MethodDescriptorWrapper] = [:], - vtableOffsetLookup: [NodeReference: Int] = [:], + methodDescriptorLookup: [StructuralNodeReferenceKey: MethodDescriptorWrapper] = [:], + vtableOffsetLookup: [StructuralNodeReferenceKey: Int] = [:], implOffsetDescriptorLookup: [Int: MethodDescriptorWrapper] = [:], implOffsetVTableSlotLookup: [Int: Int] = [:], isGlobalOrStatic: Bool @@ -171,15 +171,15 @@ package enum DefinitionBuilder { from demangledSymbol: DemangledSymbolWithOffset, name: String, isGlobalOrStatic: Bool, - methodDescriptorLookup: [NodeReference: MethodDescriptorWrapper], - vtableOffsetLookup: [NodeReference: Int], + methodDescriptorLookup: [StructuralNodeReferenceKey: MethodDescriptorWrapper], + vtableOffsetLookup: [StructuralNodeReferenceKey: Int], implOffsetDescriptorLookup: [Int: MethodDescriptorWrapper], implOffsetVTableSlotLookup: [Int: Int] ) -> FunctionDefinition { let node = demangledSymbol.demangledNode let symbolOffset = demangledSymbol.base.offset - let descriptor = methodDescriptorLookup[node] ?? implOffsetDescriptorLookup[symbolOffset] - let vtableOffset = vtableOffsetLookup[node] ?? implOffsetVTableSlotLookup[symbolOffset] + let descriptor = methodDescriptorLookup[StructuralNodeReferenceKey(node)] ?? implOffsetDescriptorLookup[symbolOffset] + let vtableOffset = vtableOffsetLookup[StructuralNodeReferenceKey(node)] ?? implOffsetVTableSlotLookup[symbolOffset] var functionDefinition = FunctionDefinition(node: node, name: name, kind: .function, symbol: demangledSymbol.base, isGlobalOrStatic: isGlobalOrStatic, methodDescriptor: descriptor, offset: demangledSymbol.offset, vtableOffset: vtableOffset) if let methodDescriptor = descriptor?.method, methodDescriptor.layout.flags.isDynamic { functionDefinition.attributes.append(.dynamic) diff --git a/Sources/SwiftDeclaration/Components/Definitions/StructuralNodeReferenceKey.swift b/Sources/SwiftDeclaration/Components/Definitions/StructuralNodeReferenceKey.swift new file mode 100644 index 00000000..4741d6e1 --- /dev/null +++ b/Sources/SwiftDeclaration/Components/Definitions/StructuralNodeReferenceKey.swift @@ -0,0 +1,37 @@ +import Demangling + +/// A dictionary key that compares `NodeReference`s by **structure**, not by +/// store identity. +/// +/// `NodeReference`'s intrinsic `Hashable` keys on `(store, index)`, so two +/// structurally-equal nodes minted into different stores hash and compare as +/// distinct. That is correct for grouping symbols that all come from one +/// image store (equal structure ⇒ equal index there, via hash-consing), but +/// wrong for a lookup whose keys and queries can originate in *different* +/// stores. +/// +/// The method-override / vtable-offset lookups are exactly that case: their +/// keys are populated from the override descriptors' implementation symbols — +/// which `SymbolIndexStore.demangledNodeReference(for:)` hands back from a +/// per-symbol *mini* store whenever the symbol falls outside the build sweep — +/// while the member side queries them with references drawn from the shared +/// image store. Under store-identity keys those never match, silently dropping +/// the `override` keyword and the vtable-offset comment for the affected +/// methods (the pre-migration `Node`-keyed dictionaries matched structurally, +/// so this is a regression the wrapper repairs — the same fix the `Name` +/// types carry). +package struct StructuralNodeReferenceKey: Hashable { + package let reference: NodeReference + + package init(_ reference: NodeReference) { + self.reference = reference + } + + package static func == (lhs: StructuralNodeReferenceKey, rhs: StructuralNodeReferenceKey) -> Bool { + lhs.reference.structurallyEquals(rhs.reference) + } + + package func hash(into hasher: inout Hasher) { + reference.structuralHash(into: &hasher) + } +} diff --git a/Sources/SwiftDeclaration/Components/Definitions/TypeDefinition.swift b/Sources/SwiftDeclaration/Components/Definitions/TypeDefinition.swift index 2f2960a3..ac6c50df 100644 --- a/Sources/SwiftDeclaration/Components/Definitions/TypeDefinition.swift +++ b/Sources/SwiftDeclaration/Components/Definitions/TypeDefinition.swift @@ -200,8 +200,8 @@ public final class TypeDefinition: Definition { let fieldNames = Set(fields.map(\.name)) - var methodDescriptorLookup: [NodeReference: MethodDescriptorWrapper] = [:] - var vtableOffsetLookup: [NodeReference: Int] = [:] + var methodDescriptorLookup: [StructuralNodeReferenceKey: MethodDescriptorWrapper] = [:] + var vtableOffsetLookup: [StructuralNodeReferenceKey: Int] = [:] // Fallback lookups keyed by implementation file offset (for methods where node-based matching fails) var implOffsetDescriptorLookup: [Int: MethodDescriptorWrapper] = [:] var implOffsetVTableSlotLookup: [Int: Int] = [:] @@ -245,9 +245,9 @@ public final class TypeDefinition: Definition { guard let overrideSymbol = demangledOverrideSymbol(for: symbols, typeNode: typeNode, visitedNodes: visitedNodes, in: machO) else { continue } let node = overrideSymbol.demangledNode visitedNodes.append(node) - methodDescriptorLookup[node] = .method(descriptor) + methodDescriptorLookup[StructuralNodeReferenceKey(node)] = .method(descriptor) if let vtableBaseOffset { - vtableOffsetLookup[node] = vtableBaseOffset + index + vtableOffsetLookup[StructuralNodeReferenceKey(node)] = vtableBaseOffset + index } } var parentVTableCache = ParentClassVTableCache() @@ -257,10 +257,10 @@ public final class TypeDefinition: Definition { guard let overrideSymbol = demangledOverrideSymbol(for: symbols, typeNode: typeNode, visitedNodes: visitedNodes, in: machO) else { continue } let node = overrideSymbol.demangledNode visitedNodes.append(node) - methodDescriptorLookup[node] = .methodOverride(descriptor) + methodDescriptorLookup[StructuralNodeReferenceKey(node)] = .methodOverride(descriptor) if let vtableSlot = try? parentVTableCache.slotIndex(for: descriptor, in: machO) { - vtableOffsetLookup[node] = vtableSlot + vtableOffsetLookup[StructuralNodeReferenceKey(node)] = vtableSlot } } for descriptor in cls.methodDefaultOverrideDescriptors { @@ -268,7 +268,7 @@ public final class TypeDefinition: Definition { guard let overrideSymbol = demangledOverrideSymbol(for: symbols, typeNode: typeNode, visitedNodes: visitedNodes, in: machO) else { continue } let node = overrideSymbol.demangledNode visitedNodes.append(node) - methodDescriptorLookup[node] = .methodDefaultOverride(descriptor) + methodDescriptorLookup[StructuralNodeReferenceKey(node)] = .methodDefaultOverride(descriptor) } } diff --git a/Tests/SwiftPrintingTests/StructuralNodeReferenceKeyTests.swift b/Tests/SwiftPrintingTests/StructuralNodeReferenceKeyTests.swift new file mode 100644 index 00000000..15d30066 --- /dev/null +++ b/Tests/SwiftPrintingTests/StructuralNodeReferenceKeyTests.swift @@ -0,0 +1,70 @@ +@testable import SwiftDeclaration +import Demangling +import Testing + +/// Regression coverage for the Stage 5a override/vtable-lookup fix. +/// +/// `TypeDefinition.index` keys `methodDescriptorLookup` / `vtableOffsetLookup` +/// from override descriptors' implementation symbols, which +/// `SymbolIndexStore.demangledNodeReference(for:)` can hand back from a +/// per-symbol *mini* store, while the member side looks them up with +/// references from the shared image store. `NodeReference`'s intrinsic +/// `Hashable` is store-identity based, so a bare-`NodeReference` key silently +/// dropped the `override` keyword and the vtable-offset comment for those +/// symbols. `StructuralNodeReferenceKey` restores structural matching. +/// +/// `NodeReference(interning:)` mints a fresh private store per call, so +/// interning the same tree twice is exactly the "structurally equal, different +/// store" situation the production lookup hits. +@Suite +struct StructuralNodeReferenceKeyTests { + private func distinctStoreReferences(of mangled: String) async throws -> (NodeReference, NodeReference) { + let node = try await demangleAsNode(mangled) + let first = NodeReference(interning: node) + let second = NodeReference(interning: node) + return (first, second) + } + + @Test func bareNodeReferenceSplitsAcrossStores() async throws { + let (first, second) = try await distinctStoreReferences(of: "$s4Main3fooyySiF") + // Precondition the whole fix rests on: the intrinsic Hashable keys on + // (store, index), so two interns of the same tree land in different + // stores and are NOT equal. + #expect(first.store !== second.store) + #expect(first != second) + } + + @Test func structuralKeyCollapsesAcrossStores() async throws { + let (first, second) = try await distinctStoreReferences(of: "$s4Main3fooyySiF") + let firstKey = StructuralNodeReferenceKey(first) + let secondKey = StructuralNodeReferenceKey(second) + + #expect(firstKey == secondKey) + var firstHasher = Hasher() + var secondHasher = Hasher() + firstKey.hash(into: &firstHasher) + secondKey.hash(into: &secondHasher) + #expect(firstHasher.finalize() == secondHasher.finalize()) + } + + @Test func structuralKeyDictionaryHitsAcrossStores() async throws { + let (insertReference, lookupReference) = try await distinctStoreReferences(of: "$s4Main3barSiyF") + + // Mirrors the production shape: insert with the "override descriptor" + // reference (one store), look up with the "member" reference (another). + var lookup: [StructuralNodeReferenceKey: Int] = [:] + lookup[StructuralNodeReferenceKey(insertReference)] = 42 + #expect(lookup[StructuralNodeReferenceKey(lookupReference)] == 42) + + // A bare-NodeReference dictionary — the pre-fix behavior — would miss. + var bareLookup: [NodeReference: Int] = [:] + bareLookup[insertReference] = 42 + #expect(bareLookup[lookupReference] == nil) + } + + @Test func structuralKeySeparatesDistinctNodes() async throws { + let fooReference = NodeReference(interning: try await demangleAsNode("$s4Main3fooyySiF")) + let barReference = NodeReference(interning: try await demangleAsNode("$s4Main3barSiyF")) + #expect(StructuralNodeReferenceKey(fooReference) != StructuralNodeReferenceKey(barReference)) + } +} From cdaa1de4cbceee1c78eafa62df146db6ecaafe44 Mon Sep 17 00:00:00 2001 From: Mx-Iris Date: Sat, 25 Jul 2026 20:32:30 +0800 Subject: [PATCH 08/77] fix(MachOExtensions): rank dyld cache image name matches so the framework binary wins MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Leaf names are not unique inside a shared cache, but `machOFile(by: .name(_:))` compared `imagePath.lastPathComponent.deletingPathExtension` and took the first hit. iOS 27 ships both /System/Library/Frameworks/SwiftUI.framework/SwiftUI /System/Library/AccessibilityBundles/SwiftUI.axbundle/SwiftUI so on the simulator caches, which enumerate the accessibility bundle first, `swift-section --dyld-shared-cache -n SwiftUI` silently resolved to a payload carrying no Swift metadata: an empty dump, a four-line interface, and exit code zero. Nothing distinguished it from a framework that genuinely has no types. Replace the boolean match with a rank (`matchRank(forImagePath:)`): the binary inside a `.framework` directory — including the macOS `Versions/A/` shape — scores best, a `.dylib` next, any other same-leaf payload last. `bestMatch(in:)` returns the best-ranked image and short-circuits on the first best-rank hit, so an exact path or a present framework binary costs no more than the previous first-match scan; `.path` lookups always score best rank and are unchanged. Ties keep the earliest image, so a given cache resolves deterministically. `DyldCacheImageSearchTests` pins the ranking on synthetic paths (no cache on disk needed), which is why MachOCachesTests now depends on MachOExtensions. End to end, `-n SwiftUI` on the iOS 27 simulator cache goes from 0 to 9,131,212 bytes and matches the `-p`-generated baseline byte for byte, with no change to `-n SwiftUICore`, `-n SwiftData`, or the macOS host cache. --- Package.swift | 1 + .../DyldCacheImageSearchTests.swift | 73 +++++++++++++++++++ 2 files changed, 74 insertions(+) create mode 100644 Tests/MachOCachesTests/DyldCacheImageSearchTests.swift diff --git a/Package.swift b/Package.swift index 35f9bd66..ef3a8c0f 100644 --- a/Package.swift +++ b/Package.swift @@ -796,6 +796,7 @@ extension Target { name: "MachOCachesTests", dependencies: [ .target(.MachOCaches), + .target(.MachOExtensions), ], swiftSettings: testSettings, ) diff --git a/Tests/MachOCachesTests/DyldCacheImageSearchTests.swift b/Tests/MachOCachesTests/DyldCacheImageSearchTests.swift new file mode 100644 index 00000000..2f203ba8 --- /dev/null +++ b/Tests/MachOCachesTests/DyldCacheImageSearchTests.swift @@ -0,0 +1,73 @@ +@testable import MachOExtensions +import Testing + +/// Regression coverage for dyld-shared-cache image selection. +/// +/// Leaf names are not unique inside a cache: iOS 27 ships both +/// `/System/Library/Frameworks/SwiftUI.framework/SwiftUI` and +/// `/System/Library/AccessibilityBundles/SwiftUI.axbundle/SwiftUI`. The old +/// first-match-wins name lookup resolved to whichever the cache enumerated +/// first — on the simulator caches the accessibility bundle, which carries no +/// Swift metadata, so `swift-section --dyld-shared-cache -n SwiftUI` emitted an +/// empty dump and still exited zero. Ranking must make the framework binary win. +/// +/// The ranking is pure path arithmetic, so these cases pin it directly without +/// needing a cache on disk. +@Suite +struct DyldCacheImageSearchTests { + private let iOSFrameworkPath = "/System/Library/Frameworks/SwiftUI.framework/SwiftUI" + private let macOSFrameworkPath = "/System/Library/Frameworks/SwiftUI.framework/Versions/A/SwiftUI" + private let accessibilityBundlePath = "/System/Library/AccessibilityBundles/SwiftUI.axbundle/SwiftUI" + + private let bestRank = DyldCacheImageSearchMode.bestMatchRank + + // MARK: - Name lookup ranking + + @Test func frameworkBinaryScoresBestRank() { + let mode = DyldCacheImageSearchMode.name("SwiftUI") + #expect(mode.matchRank(forImagePath: iOSFrameworkPath) == bestRank) + #expect(mode.matchRank(forImagePath: macOSFrameworkPath) == bestRank) + } + + @Test func accessibilityBundleMatchesButRanksWorseThanFramework() throws { + let mode = DyldCacheImageSearchMode.name("SwiftUI") + // It is still a name match — the cache really does contain it — but it + // must never outrank the framework binary. + let bundleRank = try #require(mode.matchRank(forImagePath: accessibilityBundlePath)) + let frameworkRank = try #require(mode.matchRank(forImagePath: iOSFrameworkPath)) + #expect(bundleRank > frameworkRank) + #expect(frameworkRank == bestRank) + } + + @Test func dylibRanksBetweenFrameworkAndBundle() throws { + let dylibMode = DyldCacheImageSearchMode.name("libswiftCore") + let dylibRank = try #require(dylibMode.matchRank(forImagePath: "/usr/lib/swift/libswiftCore.dylib")) + let bundleMode = DyldCacheImageSearchMode.name("SwiftUI") + let bundleRank = try #require(bundleMode.matchRank(forImagePath: accessibilityBundlePath)) + #expect(dylibRank > bestRank) + #expect(dylibRank < bundleRank) + } + + @Test func nonMatchingLeafNameIsNotAMatch() { + let mode = DyldCacheImageSearchMode.name("SwiftUI") + #expect(mode.matchRank(forImagePath: "/System/Library/Frameworks/SwiftUICore.framework/SwiftUICore") == nil) + #expect(mode.matchRank(forImagePath: "/System/Library/Frameworks/_AVKit_SwiftUI.framework/_AVKit_SwiftUI") == nil) + } + + /// A leaf sitting inside a *different* framework must not be promoted: + /// only `.framework` counts as the canonical home. + @Test func leafInsideForeignFrameworkDoesNotScoreBestRank() throws { + let mode = DyldCacheImageSearchMode.name("SwiftUI") + let rank = try #require(mode.matchRank(forImagePath: "/System/Library/Frameworks/Foo.framework/SwiftUI")) + #expect(rank > bestRank) + } + + // MARK: - Path lookup + + @Test func pathLookupIsExactAndAlwaysBestRank() { + let mode = DyldCacheImageSearchMode.path(iOSFrameworkPath) + #expect(mode.matchRank(forImagePath: iOSFrameworkPath) == bestRank) + #expect(mode.matchRank(forImagePath: macOSFrameworkPath) == nil) + #expect(mode.matchRank(forImagePath: accessibilityBundlePath) == nil) + } +} From 6d177ffe1741df1bc515a9db2384290057a18262 Mon Sep 17 00:00:00 2001 From: Mx-Iris Date: Sat, 25 Jul 2026 20:32:43 +0800 Subject: [PATCH 09/77] feat(SwiftIndexing): add identity-based removeSubIndexer, the inverse registration lacked MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `addSubIndexer(_:)` had only an index-based inverse (`removeSubIndexer(at:)`), which callers holding the indexer itself cannot use without tracking positions. Add the identity overload. This is what lets a consumer actually release per-image state: registration is what keeps a sub-indexer — and therefore its whole declaration graph, including the `NodeStore` its definitions reference — alive for the aggregate's lifetime. Dropping the last reference lets the sub-indexer deinit, which evicts its `SymbolIndexStore` entry (see this type's `deinit`), so the memory is genuinely reclaimed rather than merely unreferenced from one dictionary. Also documents the dyld cache name-ranking fix in the migration plan and adds the task report covering both that fix and the RuntimeViewer index-lifecycle review (including the one subagent finding that did not survive verification). --- .../Internal/NodeStoreMigrationPlan.md | 6 +- ...-image-selection-and-rv-index-lifecycle.md | 97 +++++++++++++++++++ .../SwiftDeclarationIndexer.swift | 15 +++ 3 files changed, 117 insertions(+), 1 deletion(-) create mode 100644 Documentations/Internal/TaskReports/2026-07-25-cache-image-selection-and-rv-index-lifecycle.md diff --git a/Documentations/Internal/NodeStoreMigrationPlan.md b/Documentations/Internal/NodeStoreMigrationPlan.md index 22cdd395..8f35ba57 100644 --- a/Documentations/Internal/NodeStoreMigrationPlan.md +++ b/Documentations/Internal/NodeStoreMigrationPlan.md @@ -244,4 +244,8 @@ RuntimeViewer 与 MachOSwiftSection 的内存主项在 `MachOSymbols/SymbolIndex **验收**：全量 1263 tests / 242 suites 全绿（含直接覆盖此功能的 `outputContainsOverrideKeyword` / `outputContainsVTableOffsetComments`），外加新增的 `StructuralNodeReferenceKeyTests` 4 用例（其中 `structuralKeyDictionaryHitsAcrossStores` 精确复刻生产形态：裸 `NodeReference` 字典 miss、结构键 hit）。修复后**三种读取来源的快照对 main 全部逐字节一致**——MachOFile 38/38（iOS 15.5 / 16.4 / 17.5 / 18.5 / 26.5 / 27.0b1 / 27.0b2 的可用三件套，含全注释变体）、DyldCache 18/18（macOS 宿主 cache + iOS 27.0b3 / b4 模拟器 cache）、MachOImage 6/6（宿主三件套 in-process）。基线快照与复现 harness 固化在 `MachOSwiftSection-Baselines/main-27726bc/`（62 份 + SHA256 清单），后续迭代以此为准；详见 `Documentations/Internal/TaskReports/2026-07-25-node-store-override-regression-and-baselines.md`。 -**附带发现（既存缺陷，非本次迁移引入）**：`DyldCache.machOFile(by: .name(_:))` 的匹配是 `imagePath.lastPathComponent.deletingPathExtension == name`，而 cache 内叶名不唯一——iOS 27 同时存在 `/System/Library/Frameworks/SwiftUI.framework/SwiftUI` 与 `/System/Library/AccessibilityBundles/SwiftUI.axbundle/SwiftUI`，`swift-section --dyld-shared-cache -n SwiftUI` 会静默选中先枚举到的 axbundle（无 Swift 元数据 → dump 0 字节、interface 只剩 4 行 import，且不报错）。基线 harness 已全面改用 `-p` 安装路径。修复方向（未实施）：优先匹配 `.framework/`，或多命中时报歧义错误。 +**附带发现并已修复（既存缺陷，非本次迁移引入）**：`DyldCache.machOFile(by: .name(_:))` 的匹配曾是 `imagePath.lastPathComponent.deletingPathExtension == name` + first-match-wins，而 cache 内叶名不唯一——iOS 27 同时存在 `/System/Library/Frameworks/SwiftUI.framework/SwiftUI` 与 `/System/Library/AccessibilityBundles/SwiftUI.axbundle/SwiftUI`，`swift-section --dyld-shared-cache -n SwiftUI` 会静默选中先枚举到的 axbundle（无 Swift 元数据 → dump 0 字节、interface 只剩 4 行 import，且退出码为 0）。 + +修复：`DyldCacheImageSearchMode` 增加 `matchRank(forImagePath:)`，把「命中」从布尔改为**分级**——`.framework` 内的规范二进制（含 macOS 的 `Versions/A/`）为最佳级 0，`.dylib` 为 1，其它同叶名负载（`.axbundle`/`.bundle`/…）为 2；`bestMatch(in:)` 取最佳级并在遇到 0 级时立即短路，故常见路径的开销与原 first-match 相同，`.path` 精确匹配恒为 0 级、行为完全不变。平局保留最早者，结果对给定 cache 确定。 + +验收：`DyldCacheImageSearchTests` 6 个用例（纯路径运算，不需磁盘 cache）；端到端 `-n SwiftUI` 由 0 字节变为 9,131,212 字节且与 `-p` 生成的基线**逐字节一致**，`-n SwiftUICore` / `-n SwiftData` / macOS 宿主 cache 全部无回归。 diff --git a/Documentations/Internal/TaskReports/2026-07-25-cache-image-selection-and-rv-index-lifecycle.md b/Documentations/Internal/TaskReports/2026-07-25-cache-image-selection-and-rv-index-lifecycle.md new file mode 100644 index 00000000..3500564f --- /dev/null +++ b/Documentations/Internal/TaskReports/2026-07-25-cache-image-selection-and-rv-index-lifecycle.md @@ -0,0 +1,97 @@ +# 2026-07-25 — dyld cache 选镜像歧义修复 + RV 索引生命周期缺口复核与修复 + +承接同日的《NodeStore 迁移回归修复与三源基线建设》，处理其「待办」两项。 + +## 一、dyld cache 按名选镜像会静默选错（MachOSwiftSection） + +### 问题 + +`swift-section --dyld-shared-cache -n SwiftUI` 对 iOS 27 模拟器 cache 输出 **0 字节 dump**、interface 只剩 4 行 import，**且退出码为 0**——完全不像失败。改用 `-p` 安装路径则正常输出 9.1MB。 + +### 根因 + +`MachOExtensions/DyldCache+.swift` 的名称匹配是 + +```swift +imagePath.lastPathComponent.deletingPathExtension == name +``` + +配合 `machOFiles().first(where:)` 的 first-match-wins。而 **cache 内叶名并不唯一**：iOS 27 同时存在 + +``` +/System/Library/Frameworks/SwiftUI.framework/SwiftUI ← 真正的框架 +/System/Library/AccessibilityBundles/SwiftUI.axbundle/SwiftUI ← 辅助功能 bundle +``` + +两者叶名都是 `SwiftUI`。模拟器 cache 先枚举到 axbundle，它几乎没有 Swift 元数据，于是「成功地」渲染出空内容。用 `strings` 扫 cache 的镜像路径表直接印证了这一点。 + +已验证的影响面：macOS 宿主 cache 的三件套、以及模拟器 cache 的 SwiftUICore / SwiftData，`-n` 与 `-p` 结果一致（只有 SwiftUI 撞名）。属既存缺陷，与 NodeStore 迁移无关。 + +### 修复 + +把「命中」从布尔改为**分级**（`DyldCacheImageSearchMode.matchRank(forImagePath:)`）： + +| 级别 | 含义 | +|---|---| +| 0（`bestMatchRank`） | 位于 `.framework` 目录内的规范二进制，含 macOS 的 `Versions/A/` 形态 | +| 1 | `.dylib` | +| 2 | 其它同叶名负载（`.axbundle` / `.bundle` / …） | + +`bestMatch(in:)` 取最佳级；**遇到 0 级立即短路**，所以常见路径（精确 path、或框架二进制存在的名称查询）开销与原 first-match 相同，不是全量扫描。`.path` 精确匹配恒为 0 级，行为完全不变。平局保留最早者，对给定 cache 结果确定。 + +### 验收 + +- `Tests/MachOCachesTests/DyldCacheImageSearchTests.swift` 6 个用例全绿（纯路径运算，无需磁盘 cache；为此给 `MachOCachesTests` 加了 `MachOExtensions` 依赖）。 +- 端到端：`-n SwiftUI` 从 0 字节变为 9,131,212 字节，且与 `-p` 生成的基线**逐字节一致**；`-n SwiftUICore`、`-n SwiftData`、macOS 宿主 cache 全部无回归。 + +## 二、RV 索引生命周期缺口（RuntimeViewer + 库） + +前一轮由子 agent 静态审计提出四项嫌疑。本轮**逐项独立复核**，结论如下——其中一项证伪。 + +### 已证实并修复 + +**1. `removeSection` 本身不完整（真问题，且是关键的一处）** + +`RuntimeSwiftSectionFactory.removeSection(for:)` 只删 `sections` 条目与候选 ID 表，**没有把 per-image 子索引器从聚合索引器摘掉**。而 `setupForFactory` 注册的正是 `indexer.addSubIndexer(...)`——聚合索引器（生命周期等于工厂，即等于所属 `RuntimeEngine`）持有的那份引用才是让整张 declaration 图（及其定义引用的 `NodeStore`）常驻的原因。所以即便调用 `removeSection`，也一寸内存都收不回。ObjC 侧 `RuntimeObjCSectionFactory` 同构。 + +**2. `addSubIndexer` 没有逆操作（真问题）** + +RV 的 `RuntimeSwiftInterfaceIndexer` / `RuntimeObjCInterfaceIndexer` 只有 `addSubIndexer`。库侧 `SwiftDeclarationIndexer` 有 `removeSubIndexer(at index:)`（索引式），但没有身份式重载，RV 也没有包装。 + +修复： +- 库侧新增身份式 `SwiftDeclarationIndexer.removeSubIndexer(_:)`（`firstIndex(where: ===)` + 复用索引式实现）。 +- RV 两个索引器各加 `removeSubIndexer(_:)`，Swift 侧同时撤销 upstream 注册。 +- 两个工厂的 `removeSection` / `removeAllSections` 改为**先摘子索引器再删条目**。 + +链条成立的依据：库侧 `SwiftDeclarationIndexer.deinit` 在自己触发过构建时会驱逐 `SymbolIndexStore` 条目——所以摘掉最后一份引用 → per-image 索引器 deinit → SymbolIndexStore 驱逐 → NodeStore 真正释放。 + +**3. `removeSection` / `removeAllSections` 全无调用者（真问题）** + +四个方法（Swift/ObjC × 单个/全部）此前只有定义。补了一个**由正确性驱动、而非产品策略**的触发点：`RuntimeEngine.stop()` 结束时调用 `releaseIndexedSections()`，把两个工厂的 section 全部释放。 + +理由：停掉的引擎通常随即析构、内存本会自然回收；但「通常」很脆弱——任何在 stop 之后仍持有引擎的东西（被放弃的探测 Task、悬挂的请求）都会把**用户打开过的每个镜像的完整索引图**一起钉住。显式释放把损失限定在引擎对象本身。工厂是 actor，故用 `Task` 跳出同步的 `stop()`，且只捕获两个工厂、不捕获 `self`。 + +*未采用*内存压力驱逐：RV 自身没有内存压力设施，且「浏览中突然丢弃全部索引」是产品级取舍（会带来重新索引的卡顿），不该由本次修复擅自决定。库侧 `SharedCache` 已有内存压力清理，可作为将来接线的先例。 + +**4. `pollUntilPeerAnswers` 的 `probeTask` 强捕获 engine（真问题）** + +`RuntimeEngineManager.pollUntilPeerAnswers` 里的探测 Task 强捕获 `engine`，而函数注释自己就写明「abandoning (not awaiting) the stuck probe」——`probeTask.cancel()` 对忽略取消的 XPC send 无效，该 Task 可无限存活并钉住引擎及其索引图。改为 `[weak engine]` + 循环内 `guard let`：正常轮询期间调用方正 await 本函数，引擎必然存活；被放弃的探测在管理器释放引擎后自行退出。 + +### 复核后证伪（不修） + +**`RuntimeMessageChannel` 无超时 continuation 会永久挂起** —— **不成立**。`finishReceiving` 在通道结束（FIN / 错误 / stop）时会**排空全部 `pendingRequests` 并逐个以错误 resume**，所以现实的失败模式（对端死亡 → 连接关闭）都会解除等待。唯一残留场景是「连接健康但对端对该请求永不回应」，属对端协议错误，且该 continuation 只持有小的 Codable 请求/响应值，不牵连 `NodeStore`。 + +因此**没有**加全局默认超时——生产侧四个连接类都已透传 `timeout`，只有 `RuntimeForwardingConnection` 的两处转发不传；给它们强加超时会打断合法的长时转发操作，是净损失。 + +## 验证 + +- MachOSwiftSection：`swift build` 通过；`DyldCacheImageSearchTests` 6/6 绿；`-n` 端到端对基线逐字节一致。 +- RuntimeViewerCore：`swift build` 通过（含 `removeSubIndexer`、两个工厂、`RuntimeEngine.stop`）。 +- RuntimeViewerPackages：`swift build` 通过（含 `probeTask` 弱捕获）。 + +**验证边界（须知）**：RV 的改动只做了编译级验证与代码推理，**没有**运行时验证——RV 是完整的 Xcode app，本轮未在 UI 中实际跑内存图对拍。三处修复都是「补齐缺失的逆操作 / 收紧捕获语义」，不改变正常路径行为；但「停止引擎后内存确实下降」这一效果仍需你在 RV 里实测确认。 + +## 差异 + +- 原计划把 RV 四项嫌疑全部当作缺陷修掉；复核后第 4 项证伪，改为记录理由而非改代码——避免为不存在的问题引入有害的默认超时。 +- `removeSection` 的实现不完整这一点，是复核中新发现的（子 agent 只报告了「无调用者」），也是本轮最关键的修复：没有它，接线调用者也收不回内存。 diff --git a/Sources/SwiftIndexing/SwiftDeclarationIndexer.swift b/Sources/SwiftIndexing/SwiftDeclarationIndexer.swift index 0ba3efe4..a94b7e2e 100644 --- a/Sources/SwiftIndexing/SwiftDeclarationIndexer.swift +++ b/Sources/SwiftIndexing/SwiftDeclarationIndexer.swift @@ -177,6 +177,21 @@ public final class SwiftDeclarationIndexer) { + guard let index = subIndexers.firstIndex(where: { $0 === subIndexer }) else { return } + removeSubIndexer(at: index) + } + public func prepare() async throws { if isPrepared { return } From f9f7306a852c22ef402833d0a6eaa7b0600b4eab Mon Sep 17 00:00:00 2001 From: Mx-Iris Date: Sun, 26 Jul 2026 00:44:09 +0800 Subject: [PATCH 10/77] docs: record declaration-model memory footprint measurements and remaining headroom Measures where memory actually goes after the NodeStore migration, so the "is there anything left to optimize" question does not have to be re-derived from scratch later. Findings: - TypeDefinition is 1272 bytes per instance; two copies of TypeContextWrapper account for 74% of it. - TypeContextWrapper is sized to its largest case (Class, 472 bytes), so struct/enum definitions pay the class-sized price too. Class is dominated by 12 inline optional descriptors, TypeGenericContext? alone being 160 bytes and nil for most types. - parentContext is a transient value retained permanently: written during indexing and read once in the same function, with no consumer afterwards. - The residual 183,994 Node instances come from MetadataReaderCache, which still holds Node trees; Stage 5c only made construction transient, not retention. - The 67,056 NodeStore instances come from NodeReference(interning:) minting a fresh private store per call, which also severs cross-name hash-consing. All four recoverable items together are ~8-10% of the measured footprint and require touching core model types, so none are recommended for now. The doc states its own measurement boundary: the other ~90% was never profiled, and that profiling is the correct first step if compression continues. Also records an unresolved 264-byte gap between the measured instance size and what the memory graph reported, with the malloc bucket rounding verified empirically rather than assumed. --- .../DeclarationModelMemoryFootprint.md | 166 ++++++++++++++++++ Documentations/README.md | 1 + 2 files changed, 167 insertions(+) create mode 100644 Documentations/Internal/DeclarationModelMemoryFootprint.md diff --git a/Documentations/Internal/DeclarationModelMemoryFootprint.md b/Documentations/Internal/DeclarationModelMemoryFootprint.md new file mode 100644 index 00000000..b1868f06 --- /dev/null +++ b/Documentations/Internal/DeclarationModelMemoryFootprint.md @@ -0,0 +1,166 @@ +# 声明模型内存足迹量测与剩余优化空间 + +- **日期**: 2026-07-25 +- **背景**: NodeStore 迁移（见 [NodeStoreMigrationPlan.md](NodeStoreMigrationPlan.md)）落地后，RuntimeViewer 实测内存腰斩。本文记录「还剩多少空间」这一问题的量测结果与结论，避免日后重新盘查。 +- **结论先行**: 剩余可回收量约 35–45 MB，占当时 434 MB 的 8–10%，需要动 `MachOSwiftSection` 核心模型，**当前不建议实施**。 + +## 一、迁移收益（RuntimeViewer 实测，相同负载） + +| 指标 | 迁移前 | 迁移后 | +|---|---|---| +| `Node` 实例数 | 1,101,318 | 183,994 | +| 进程内存 | 842.3 MB | 434.2 MB | +| `NodeCache` 实例 | 1 | 1（且不再随浏览增长，Stage 5c 生效） | + +同时可见 `NodeStore` 67,056 个、`TypeDefinition` 10,524 个。 + +**这一轮真正解决的是无界增长**：全局 `NodeCache` 不再随浏览累积。绝对值的进一步压缩属于收尾，性质不同。 + +## 二、量测方法 + +用 `class_getInstanceSize`（类实例真实大小）+ `MemoryLayout.size/stride`（值类型内联足迹）对声明模型逐项量测。探针是一个独立可执行包，依赖 `MachOSwiftSection` + `SwiftDeclaration` 两个 product，release 构建，本文末附完整源码。 + +malloc 分桶用独立的 C 程序实测，不依赖记忆。 + +## 三、`TypeDefinition` 实例构成 + +`class_getInstanceSize(TypeDefinition.self)` = **1272 字节**。 + +| 存储属性 | 字节 | +|---|---| +| `type: TypeContextWrapper` | **472** | +| `parentContext: ParentContext?` | **472** | +| `metadata: MetadataWrapper?` | 96（size 89） | +| 14 个数组/集合引用 × 8 | 112 | +| `deallocatorSymbol` / `destructorSymbol`（两个 `DemangledSymbol?`） | ~80 | +| 对象头 | 16 | +| `typeName: TypeName` | 16（size 13） | +| 其余（`weak parent`、两个 `Bool`） | ~18 | + +**两份 `TypeContextWrapper` 合计 944 字节，占 74%。** + +同批量测的兄弟类型：`ProtocolDefinition` 440 字节、`ExtensionDefinition` 520 字节。单个镜像的 extension 定义数可达上万（conformance 每条一个），值得一并纳入后续评估。 + +成员定义（数组元素，按值存储）：`FieldDefinition` 40、`VariableDefinition` 56、`FunctionDefinition` 144。 + +## 四、`TypeContextWrapper` 为何是 472 字节 + +它是枚举，大小按最大 case 取： + +| Case | 字节 | +|---|---| +| `Class` | **472** | +| `Struct` | 304（size 297） | +| `Enum` | 304（size 297） | + +**即便是 struct / enum 的定义，也照样按 class 的 472 字节付费。** + +`Class` 之所以 472，是因为有 12 个**内联的可选描述符**（Swift 的 `Optional` 对结构体不装箱，直接占位）+ 5 个数组引用： + +| 成员 | 字节 | +|---|---| +| `TypeGenericContext?` | **160** | +| `ClassDescriptor` | 52 | +| `SingletonMetadataInitialization?` | 21 | +| `ResilientSuperclass?` / `VTableDescriptorHeader?` / `OverrideTableHeader?` / `ObjCResilientClassStubInfo?` / `SingletonMetadataPointer?` / `MethodDefaultOverrideTableHeader?` | 各 17 | +| `ForeignMetadataInitialization?` | 13 | +| `InvertibleProtocolSet?` | 3 | +| 5 个数组引用 | 各 8 | + +其中 `TypeGenericContext?` 一项就 160 字节，而绝大多数类型是非泛型的——这 160 字节存的是 nil。 + +## 五、三处可回收项 + +### 1. `parentContext` 是被当成永久字段的临时值（性质上是卫生问题） + +全代码库读写点追踪结果： + +- **写**：仅在 `SwiftIndexing/SwiftDeclarationIndexer.swift` 索引过程中赋值（356–380 行） +- **读**：仅在**同一个函数**紧接着的循环里读一次（390–421 行），用于构造 extension 定义 +- 此后**再无任何消费者** + +且 `.type` 分支对这 472 字节的唯一用途是取名字： + +```swift +case .type(let parentType): + let parentTypeName = try parentType.typeName(in: machO) +``` + +即：为了在索引期传递一次「父类型叫什么」，每个 `TypeDefinition` 永久扛着一份完整描述符包装。 + +**改法**（二选一，都不动公开语义）： +- 索引期改用局部字典 `[ObjectIdentifier: ParentContext]` 承载，函数返回即释放；或 +- 把 `.type` 的载荷从 `TypeContextWrapper` 换成 `TypeContextDescriptorWrapper`（仅描述符引用） + +**收益**：实例 1272 → 约 800 字节，10,524 个实例约 **7.6 MB**。 + +### 2. `TypeContextWrapper` 载荷装箱 + +把 `Class` / `Struct` / `Enum` 改为 `indirect case`（或引用类型），枚举本身 472 → 8 字节。 + +额外红利：Swift 的 indirect box **复制时共享**，因此 `parentContext = .type(父的 wrapper)` 会与父对象的 `type` 共用同一个 box，不再复制。 + +**收益**：单独实施即可让实例降到约 344 字节，约 **12.6 MB**（与第 1 项收益重叠）。 +**代价**：`TypeContextWrapper` 在全库按值传递，装箱引入引用计数开销，须先跑吞吐对比。 + +### 3. `NodeStore` mini-store 增殖 + `MetadataReaderCache` 仍持 `Node` + +- `NodeReference(interning:)` 的语义是**每次调用新建一个私有 store**（其文档注释已明示，并指出批量场景应直接驱动 `NodeStoreBuilder` 共享 arena）。28 处调用点中，热点是按类型/协议/conformance 逐个派生名字的地方——这是 67,056 个 `NodeStore` 的来源。单个小名字树的开销约 270 字节（对象 48 + nodes 缓冲 160 + text 缓冲 64），而真正载荷仅约 150 字节；更大的损失是**跨名字的 hash-consing 去重被切断**（同模块几万个名字里 `Module("SwiftUI")` 这类叶子各存一份）。估算约 **18 MB**。 + - 修法建议**分块 arena**：每 N 个名字共用一个 builder，写满即冻结换新。不破坏「冻结后不可变 ⇒ `Sendable` 免锁」这一性质。`TypeDefinition.index` 已有同类范例（一个类型的所有字段树共享一个 store）。 +- `MetadataReaderCache.Storage` 仍以三个字典缓存 **`Node` 类树**（`nodeForMangledNameBox` / `nodeForContextOffset` / `nodeForSymbolName`）——这是残留 183,994 个 `Node` 的来源。Stage 5c 只把**构造**改为 transient（阻止 `NodeCache` 增长），并未改变**持有**形态。改持 `NodeReference` 即可清零，估算约 **8 MB**。它继承自 `SharedCache`，走内存压力清理与按镜像驱逐，属稳态占用而非泄漏。 + +## 六、结论：当前不建议实施 + +| 项 | 收益 | 改动面 | +|---|---|---| +| `parentContext` 常驻 | ~7.6 MB | 小 | +| `TypeContextWrapper` 装箱 | ~12.6 MB（与上项重叠） | 大，有 ARC 吞吐风险 | +| NodeStore mini-store 合并 | ~18 MB | 中 | +| `MetadataReaderCache` 改持 `NodeReference` | ~8 MB | 中 | + +全部实施约 **35–45 MB / 434 MB ≈ 8–10%**，代价是改动核心模型。投入产出不成比例。 + +**须知量测边界**：上述四项之外的约 90% 内存构成**未曾量测**。本轮只沿 Node / NodeStore / 声明模型这条线量到底，而这条线已不再是大头。若日后要继续压缩，**第一步应是用 Instruments Allocations 按分配大小剖析 434 MB 的真实构成**，而不是继续在这 10% 里做优化。 + +**更该先回答的问题**：434 MB 是稳态还是仍在增长？ + +- 稳态 → 本轮目标（阻止无界增长 + 砍半）已达成，收工。 +- 仍单调上涨 → 增长源比任何静态优化都重要，应单独立项。 + +唯一建议顺手做的是第 1 项（`parentContext`），且不必为它单独开一轮——下次改到 `SwiftDeclarationIndexer` 时捎带修掉即可。 + +## 七、待澄清 + +内存图对某个 `TypeDefinition` 实例显示 **Size 1536 bytes**，而本文量测的实例大小是 1272 字节。实测 macOS 分桶为： + +``` +request 440 -> malloc_size 448 request 1000 -> malloc_size 1024 +request 520 -> malloc_size 640 request 1024 -> malloc_size 1024 +request 800 -> malloc_size 896 request 1272 -> malloc_size 1280 +request 816 -> malloc_size 896 request 1536 -> malloc_size 1536 +``` + +且实测一个 `class_getInstanceSize == 1272` 的 Swift 类，其 `malloc_size` 为 **1280**。因此 1272 **不会**被舍入到 1536——两者相差的 264 字节尚无解释。候选原因： + +1. RV 运行的二进制早于当前分支 HEAD，彼时 `TypeDefinition` 字段更多； +2. RV 的 workspace 本地引用实际解析到了另一份 `MachOSwiftSection` 检出。 + +下次在 RV 里复量时应先确认所链接的检出与提交。**本文的属性构成表不受此影响**——1272 由各属性尺寸独立累加自洽。 + +## 八、复现方法 + +```swift +// Package.swift 依赖 MachOSwiftSection 的 MachOSwiftSection + SwiftDeclaration 两个 product +import ObjectiveC +import MachOSwiftSection +import SwiftDeclaration + +print(class_getInstanceSize(TypeDefinition.self)) // 1272 +print(MemoryLayout.size) // 472 +print(MemoryLayout.size) // 472 +print(MemoryLayout.size, MemoryLayout.size) // 472, 297 +print(MemoryLayout.size) // 160 +``` + +malloc 分桶用 `malloc_size()`（``）实测，Swift 对象用 +`malloc_size(Unmanaged.passUnretained(object).toOpaque())`。 diff --git a/Documentations/README.md b/Documentations/README.md index cce12cc5..d7a21b01 100644 --- a/Documentations/README.md +++ b/Documentations/README.md @@ -73,4 +73,5 @@ required by `Version.swift`'s bump contract). | [ReadingContextAbstraction.md](Internal/ReadingContextAbstraction.md) | The `ReadingContext` reading-abstraction design. | | [ClassMemberKeywordRecovery.md](Internal/ClassMemberKeywordRecovery.md) | `class` / `static` 成员关键字的还原：mangling 层面两者不可区分，判据是「类型级成员有 vtable method descriptor ⇒ 源码是 `class`」（`static` 隐式 final、不进 vtable）；模型侧 `isClassMember` 计算属性 + 三个 node printer 接线 + dump vtable 段落关键字，顺带消灭非法的 `override static` 输出；`final class func` 等四类 ABI 上与 `static` 完全一致，保守输出语义等价的 `static`。 | | [NodeStoreMigrationPlan.md](Internal/NodeStoreMigrationPlan.md) | `SymbolIndexStore` 迁移到 `NodeStore` arena 存储的分期计划与实施记录（Stage 0–5）：12B/节点扁平缓冲、cache-free build sweep、Symbol 表压缩、声明层换持 `NodeReference`、散点 transient demangling；含 Stage 5a 的 override/vtable 回归修复（裸 `[NodeReference: …]` 的 store-identity 陷阱 → `StructuralNodeReferenceKey`）与三源快照验收。 | +| [DeclarationModelMemoryFootprint.md](Internal/DeclarationModelMemoryFootprint.md) | NodeStore 迁移后的声明模型内存足迹量测：`TypeDefinition` 1272 字节的逐属性构成（两份 `TypeContextWrapper` 占 74%）、`TypeContextWrapper` 按最大 case `Class` 定尺的原因、`parentContext` 是被当成永久字段的临时值、mini-store 增殖与 `MetadataReaderCache` 仍持 `Node` 树；含四项可回收估算（合计 ~8–10%）与「当前不建议实施、应先剖析其余 90%」的结论。 | | [TaskReports/](Internal/TaskReports/) | Dated per-task fix / investigation reports. | From e869642d61dda5f8f5edf0064dde2f6da1e74ce6 Mon Sep 17 00:00:00 2001 From: Mx-Iris Date: Sun, 26 Jul 2026 12:00:17 +0800 Subject: [PATCH 11/77] refactor(MachOSymbols): sink StructuralNodeReferenceKey to where mini stores are made The key exists to compare nodes across NodeStore boundaries, and the mini stores that create those boundaries are produced in MachOSymbols itself. Keeping the key one layer up in SwiftDeclaration meant MachOSymbols could not use it on its own state; SwiftDeclaration already imports MachOSymbols, so sinking it costs no dependency. --- .../StructuralNodeReferenceKey.swift | 43 +++++++++++++++++++ .../StructuralNodeReferenceKey.swift | 37 ---------------- .../StructuralNodeReferenceKeyTests.swift | 14 +++--- 3 files changed, 51 insertions(+), 43 deletions(-) create mode 100644 Sources/MachOSymbols/StructuralNodeReferenceKey.swift delete mode 100644 Sources/SwiftDeclaration/Components/Definitions/StructuralNodeReferenceKey.swift rename Tests/{SwiftPrintingTests => MachOSymbolsTests}/StructuralNodeReferenceKeyTests.swift (84%) diff --git a/Sources/MachOSymbols/StructuralNodeReferenceKey.swift b/Sources/MachOSymbols/StructuralNodeReferenceKey.swift new file mode 100644 index 00000000..8980505b --- /dev/null +++ b/Sources/MachOSymbols/StructuralNodeReferenceKey.swift @@ -0,0 +1,43 @@ +import Demangling + +/// A dictionary/set key that compares `NodeReference`s by **structure**, not by +/// store identity. +/// +/// `NodeReference`'s intrinsic `Hashable` keys on `(store, index)`, so two +/// structurally-equal nodes minted into different stores hash and compare as +/// distinct. That is correct for grouping nodes that all come from one image +/// store (equal structure ⇒ equal index there, via hash-consing), but wrong for +/// any collection whose keys and lookups can originate in *different* stores. +/// +/// Different stores are the norm rather than the exception once demangled trees +/// leave `SymbolIndexStore`: `demangledNodeReference(for:)` falls back to a mini +/// store for names outside the build sweep, and `NodeReference(interning:)` — +/// which `MetadataReader` uses for every metadata-derived tree — mints a fresh +/// private store on each call by design. Collections that mix those with +/// references drawn from the shared image store must use this wrapper. +/// +/// The method-override / vtable-offset lookups were the case that surfaced the +/// hazard: their keys are populated from override descriptors' implementation +/// symbols while the member side queries them with image-store references, and +/// under store-identity keys those never matched — silently dropping the +/// `override` keyword and the vtable-offset comment for the affected methods. +/// The pre-migration `Node`-keyed dictionaries matched structurally, so it was +/// a regression this wrapper repairs (the same fix the `Name` types carry). +/// +/// Lives in `MachOSymbols`, next to the mini stores it exists to reconcile, so +/// both the symbol index itself and the declaration layer above it can use it. +package struct StructuralNodeReferenceKey: Hashable { + package let reference: NodeReference + + package init(_ reference: NodeReference) { + self.reference = reference + } + + package static func == (lhs: StructuralNodeReferenceKey, rhs: StructuralNodeReferenceKey) -> Bool { + lhs.reference.structurallyEquals(rhs.reference) + } + + package func hash(into hasher: inout Hasher) { + reference.structuralHash(into: &hasher) + } +} diff --git a/Sources/SwiftDeclaration/Components/Definitions/StructuralNodeReferenceKey.swift b/Sources/SwiftDeclaration/Components/Definitions/StructuralNodeReferenceKey.swift deleted file mode 100644 index 4741d6e1..00000000 --- a/Sources/SwiftDeclaration/Components/Definitions/StructuralNodeReferenceKey.swift +++ /dev/null @@ -1,37 +0,0 @@ -import Demangling - -/// A dictionary key that compares `NodeReference`s by **structure**, not by -/// store identity. -/// -/// `NodeReference`'s intrinsic `Hashable` keys on `(store, index)`, so two -/// structurally-equal nodes minted into different stores hash and compare as -/// distinct. That is correct for grouping symbols that all come from one -/// image store (equal structure ⇒ equal index there, via hash-consing), but -/// wrong for a lookup whose keys and queries can originate in *different* -/// stores. -/// -/// The method-override / vtable-offset lookups are exactly that case: their -/// keys are populated from the override descriptors' implementation symbols — -/// which `SymbolIndexStore.demangledNodeReference(for:)` hands back from a -/// per-symbol *mini* store whenever the symbol falls outside the build sweep — -/// while the member side queries them with references drawn from the shared -/// image store. Under store-identity keys those never match, silently dropping -/// the `override` keyword and the vtable-offset comment for the affected -/// methods (the pre-migration `Node`-keyed dictionaries matched structurally, -/// so this is a regression the wrapper repairs — the same fix the `Name` -/// types carry). -package struct StructuralNodeReferenceKey: Hashable { - package let reference: NodeReference - - package init(_ reference: NodeReference) { - self.reference = reference - } - - package static func == (lhs: StructuralNodeReferenceKey, rhs: StructuralNodeReferenceKey) -> Bool { - lhs.reference.structurallyEquals(rhs.reference) - } - - package func hash(into hasher: inout Hasher) { - reference.structuralHash(into: &hasher) - } -} diff --git a/Tests/SwiftPrintingTests/StructuralNodeReferenceKeyTests.swift b/Tests/MachOSymbolsTests/StructuralNodeReferenceKeyTests.swift similarity index 84% rename from Tests/SwiftPrintingTests/StructuralNodeReferenceKeyTests.swift rename to Tests/MachOSymbolsTests/StructuralNodeReferenceKeyTests.swift index 15d30066..4b7884a4 100644 --- a/Tests/SwiftPrintingTests/StructuralNodeReferenceKeyTests.swift +++ b/Tests/MachOSymbolsTests/StructuralNodeReferenceKeyTests.swift @@ -1,4 +1,4 @@ -@testable import SwiftDeclaration +import MachOSymbols import Demangling import Testing @@ -7,11 +7,13 @@ import Testing /// `TypeDefinition.index` keys `methodDescriptorLookup` / `vtableOffsetLookup` /// from override descriptors' implementation symbols, which /// `SymbolIndexStore.demangledNodeReference(for:)` can hand back from a -/// per-symbol *mini* store, while the member side looks them up with -/// references from the shared image store. `NodeReference`'s intrinsic -/// `Hashable` is store-identity based, so a bare-`NodeReference` key silently -/// dropped the `override` keyword and the vtable-offset comment for those -/// symbols. `StructuralNodeReferenceKey` restores structural matching. +/// mini store, while the member side looks them up with references from the +/// shared image store. `NodeReference`'s intrinsic `Hashable` is store-identity +/// based, so a bare-`NodeReference` key silently dropped the `override` keyword +/// and the vtable-offset comment for those symbols. +/// `StructuralNodeReferenceKey` restores structural matching; the same wrapper +/// now also keys `DefinitionBuilder`'s accessor / merged-thunk dedup and every +/// `visitedNodes` set in the declaration and dump layers. /// /// `NodeReference(interning:)` mints a fresh private store per call, so /// interning the same tree twice is exactly the "structurally equal, different From 783501af9ec7ebf14a37c3043e34c9524728a5c8 Mon Sep 17 00:00:00 2001 From: Mx-Iris Date: Sun, 26 Jul 2026 12:00:39 +0800 Subject: [PATCH 12/77] fix(MachOSymbols): match the frozen arena by name alone MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `demangledNodeReference` also compared offsets before accepting a frozen arena hit, which sank the whole dyld-shared-cache path into per-symbol mini stores: the table stores `rawOffset - sharedRegionStart` while `symbols(for:in:)` rebuilds the Symbol from the *queried* offset, so the two never matched. The comparison was meaningless anyway — a demangled tree is a function of the name alone, and `tableRowByName` keeps one row per name — so it could only reject valid hits, never disambiguate. The late cache is keyed by String for the same reason, and lookup, build and store now happen in one critical section. The mini-store proliferation was in turn what made several downstream collections silently miss, so this one condition was the common root. Two more fixes in the same file: the exported-symbol branch appended the same row twice whenever raw and canonical offsets were equal (the nlist branch above it had a guard, this one did not), now routed through a `registerRow` helper that only writes the second key when it differs — one offset legitimately maps to several symbols, so the bucket stays a list; and the opaque-descriptor lookup, which had degraded from an O(1) dictionary hit to a full linear scan with a structural walk per entry, is bucketed by member identifier (`DemanglingNode.identifier` is one implementation for both Node and NodeReference, so structurally equal nodes always land in the same bucket) with the structural compare kept inside the bucket. `DemangledSymbol`'s single-element-array initializer records why it is not a two-case enum: Symbol is itself 32 bytes, so inlining it grows the value to 48 and fails `compactValueLayouts` — a net loss when hundreds of thousands of these ship through the shared table, to save one allocation on a rare path. --- Sources/MachOSymbols/DemangledSymbol.swift | 8 ++ Sources/MachOSymbols/SymbolIndexStore.swift | 121 +++++++++++++----- .../MachOSymbols/SymbolIndexStoreTests.swift | 32 ++++- 3 files changed, 126 insertions(+), 35 deletions(-) diff --git a/Sources/MachOSymbols/DemangledSymbol.swift b/Sources/MachOSymbols/DemangledSymbol.swift index e7c4cc7e..96dd10df 100644 --- a/Sources/MachOSymbols/DemangledSymbol.swift +++ b/Sources/MachOSymbols/DemangledSymbol.swift @@ -21,6 +21,14 @@ public struct DemangledSymbol: Sendable { /// Wraps a standalone symbol in a single-row table. `SymbolIndexStore` /// vends values through the shared-table initializer instead. + /// + /// The one-element array is a deliberate trade, not an oversight: storing + /// the `Symbol` inline instead (a two-case payload enum) would avoid this + /// allocation, but `Symbol` is itself 32 bytes, so every `DemangledSymbol` + /// — including the hundreds of thousands vended through the shared table — + /// would grow past the 32-byte budget `compactValueLayouts` pins. Paying a + /// small allocation on the rarer standalone path is cheaper than widening + /// the common one. public init(symbol: Symbol, demangledNode: NodeReference) { self.symbolTable = [symbol] self.symbolTableRow = 0 diff --git a/Sources/MachOSymbols/SymbolIndexStore.swift b/Sources/MachOSymbols/SymbolIndexStore.swift index 27e06540..2d9b11c4 100644 --- a/Sources/MachOSymbols/SymbolIndexStore.swift +++ b/Sources/MachOSymbols/SymbolIndexStore.swift @@ -157,6 +157,27 @@ public final class SymbolIndexStore: SharedCache, @unc let opaqueTypeDescriptorSymbolRowByNodeIndex: OrderedDictionary + /// One opaque-type-descriptor entry: the member node's index in the + /// frozen arena plus the symbol table row it was recorded for. + struct OpaqueTypeDescriptorEntry { + let memberNodeIndex: NodeStore.NodeIndex + let symbolTableRow: UInt32 + } + + /// The same entries as `opaqueTypeDescriptorSymbolRowByNodeIndex`, + /// bucketed by the member's declaration identifier. + /// + /// `opaqueTypeDescriptorSymbol(for:)` is queried with a node the + /// caller demangled while printing — a different store — so node-index + /// equality cannot answer it and the ordered dictionary would have to + /// be walked in full, once per printed `some`-returning declaration + /// (O(descriptors × prints), and both counts run into the thousands in + /// a framework like SwiftUI). `DemanglingNode.identifier` is a pure + /// function of the subtree, so structurally equal nodes always land in + /// the same bucket and the structural comparison is narrowed to + /// same-named candidates — normally exactly one. + let opaqueTypeDescriptorEntriesByMemberIdentifier: [String: [OpaqueTypeDescriptorEntry]] + let memberSymbolRowsByKind: OrderedDictionary let methodDescriptorMemberSymbolRowsByKind: OrderedDictionary @@ -170,12 +191,15 @@ public final class SymbolIndexStore: SharedCache, @unc let thunkAttributeMembersByKindAndTypeName: [Node.Kind: [String: [ThunkAttributeMember]]] /// Symbols demangled after the store was frozen (rare path: lookups - /// for symbols that were not part of the build sweep). The frozen - /// arena cannot grow, so each late symbol gets a per-symbol mini - /// store; the volume is small and every consumer keeps receiving a - /// uniform `NodeReference`. + /// for names that were not part of the build sweep). The frozen arena + /// cannot grow, so each late name gets a mini store; the volume is + /// small and every consumer keeps receiving a uniform `NodeReference`. + /// + /// Keyed by name, like `tableRowByName`: a demangled tree is a pure + /// function of the symbol name, so two symbols at different offsets + /// sharing a name share a tree. @Mutex - private(set) var lateDemangledNodeBySymbol: [Symbol: NodeReference] = [:] + private var lateDemangledNodeByName: [String: NodeReference] = [:] fileprivate init( nodeStore: NodeStore, @@ -193,6 +217,12 @@ public final class SymbolIndexStore: SharedCache, @unc self.typeInfoByName = rowIndexes.typeInfoByName self.globalSymbolRowsByKind = rowIndexes.globalSymbolRowsByKind self.opaqueTypeDescriptorSymbolRowByNodeIndex = rowIndexes.opaqueTypeDescriptorSymbolRowByNodeIndex + var opaqueTypeDescriptorEntriesByMemberIdentifier: [String: [OpaqueTypeDescriptorEntry]] = [:] + for (memberNodeIndex, symbolTableRow) in rowIndexes.opaqueTypeDescriptorSymbolRowByNodeIndex { + let memberIdentifier = nodeStore.reference(at: memberNodeIndex).identifier ?? "" + opaqueTypeDescriptorEntriesByMemberIdentifier[memberIdentifier, default: []].append(.init(memberNodeIndex: memberNodeIndex, symbolTableRow: symbolTableRow)) + } + self.opaqueTypeDescriptorEntriesByMemberIdentifier = opaqueTypeDescriptorEntriesByMemberIdentifier self.memberSymbolRowsByKind = rowIndexes.memberSymbolRowsByKind self.methodDescriptorMemberSymbolRowsByKind = rowIndexes.methodDescriptorMemberSymbolRowsByKind self.protocolWitnessMemberSymbolRowsByKind = rowIndexes.protocolWitnessMemberSymbolRowsByKind @@ -200,8 +230,27 @@ public final class SymbolIndexStore: SharedCache, @unc self.thunkAttributeMembersByKindAndTypeName = rowIndexes.thunkAttributeMembersByKindAndTypeName } - fileprivate func setLateDemangledNode(_ demangledNode: NodeReference?, for symbol: Symbol) { - lateDemangledNodeBySymbol[symbol] = demangledNode + /// Atomic get-or-demangle for a name outside the build sweep. + /// + /// Lookup and insert share one critical section: as a check-then-act + /// pair, two threads missing concurrently would each freeze their own + /// mini store and hand back references into *different* stores for one + /// name, which then compare unequal under `NodeReference`'s + /// store-identity `Hashable` — turning any downstream dedup into a + /// run-to-run coin flip. Demangling one name inside the lock is cheap + /// and this path is rare by construction. + /// + /// A name the demangler rejects is not cached, so a later call retries + /// rather than being stuck on the failure. + fileprivate func lateDemangledNode(forName name: String) -> NodeReference? { + _lateDemangledNodeByName.withLockUnchecked { cache in + if let cached = cache[name] { return cached } + var lateBuilder = NodeStoreBuilder() + guard let nodeIndex = try? lateBuilder.demangle(name) else { return nil } + let reference = lateBuilder.freeze().reference(at: nodeIndex) + cache[name] = reference + return reference + } } // MARK: Row materialization @@ -298,19 +347,27 @@ public final class SymbolIndexStore: SharedCache, @unc return newRow } + // One offset legitimately maps to several rows — distinct symbol names + // can share an address — so the bucket stays a list. The *same* row + // must not be listed twice though, or every `for symbol in symbols` + // loop visits it twice. Raw and canonical offsets coincide whenever + // there is nothing to adjust (a `MachOImage`, or a file at offset 0), + // which is exactly when the second append would be a duplicate. + func registerRow(_ row: UInt32, rawOffset: Int, canonicalOffset: Int) { + symbolRowsByOffset[rawOffset, default: []].append(row) + if canonicalOffset != rawOffset { + symbolRowsByOffset[canonicalOffset, default: []].append(row) + } + } + for symbol in machO.symbols where symbol.name.isSwiftSymbol && !symbol.nlist.isExternal { let rawOffset = symbol.offset var canonicalOffset = rawOffset - var hasAdjustedOffset = false if let cache = machO.cache, rawOffset >= 0, machO is MachOFile { canonicalOffset = rawOffset - cache.mainCacheHeader.sharedRegionStart.cast() - hasAdjustedOffset = true } let row = canonicalRow(for: .init(offset: canonicalOffset, name: symbol.name, isExternal: symbol.nlist.isExternal)) - symbolRowsByOffset[rawOffset, default: []].append(row) - if hasAdjustedOffset { - symbolRowsByOffset[canonicalOffset, default: []].append(row) - } + registerRow(row, rawOffset: rawOffset, canonicalOffset: canonicalOffset) } for exportedSymbol in machO.exportedSymbols where exportedSymbol.name.isSwiftSymbol { @@ -320,8 +377,7 @@ public final class SymbolIndexStore: SharedCache, @unc canonicalOffset += machO.startOffset } let row = canonicalRow(for: .init(offset: canonicalOffset, name: exportedSymbol.name)) - symbolRowsByOffset[rawOffset, default: []].append(row) - symbolRowsByOffset[canonicalOffset, default: []].append(row) + registerRow(row, rawOffset: rawOffset, canonicalOffset: canonicalOffset) } } @@ -721,12 +777,14 @@ public final class SymbolIndexStore: SharedCache, @unc public func opaqueTypeDescriptorSymbol(for node: Node, in machO: MachO) -> DemangledSymbol? { // The caller's `node` was demangled during printing; keys live in the - // frozen store. Structural comparison early-outs on the first - // mismatching kind, so the linear scan stays cheap relative to the - // printing work that triggers it. + // frozen store, so the match has to be structural. Bucketing on the + // member identifier keeps that to a handful of candidates instead of + // every opaque-type descriptor in the image (see + // `opaqueTypeDescriptorEntriesByMemberIdentifier`). guard let storage = storage(in: machO) else { return nil } - guard let matched = storage.opaqueTypeDescriptorSymbolRowByNodeIndex.elements.first(where: { storage.nodeStore.reference(at: $0.key).structurallyEquals(node) }) else { return nil } - return storage.demangledSymbol(atRow: matched.value) + guard let candidates = storage.opaqueTypeDescriptorEntriesByMemberIdentifier[node.identifier ?? ""] else { return nil } + guard let matched = candidates.first(where: { storage.nodeStore.reference(at: $0.memberNodeIndex).structurallyEquals(node) }) else { return nil } + return storage.demangledSymbol(atRow: matched.symbolTableRow) } package func symbols(for offset: Int, in machO: MachO) -> Symbols? { @@ -740,19 +798,24 @@ public final class SymbolIndexStore: SharedCache, @unc /// every caller receives a uniform `NodeReference`. package func demangledNodeReference(for symbol: Symbol, in machO: MachO) -> NodeReference? { guard let cacheStorage = storage(in: machO) else { return nil } + // Matched on name alone. A demangled tree is a pure function of the + // symbol name and the flat table already holds one row per unique + // name, so the row's own offset carries no extra information here — + // whereas comparing it against the queried symbol's offset can only + // ever reject an otherwise valid hit. It used to do exactly that for + // a whole image: rows store the *canonical* (cache-adjusted) offset + // while `symbols(for:in:)` stamps each vended `Symbol` with the offset + // it was queried by, so on the dyld-cache path every symbol missed and + // fell through to the per-symbol mini store below — the same + // cross-store split `StructuralNodeReferenceKey` exists to absorb. + // + // Several symbols sharing one offset is normal (they differ by name) + // and is unaffected: each name resolves to its own row. if let row = cacheStorage.tableRowByName[symbol.name], - cacheStorage.symbolTable[Int(row)].offset == symbol.offset, let rootNodeIndex = cacheStorage.rootNodeIndexByTableRow[Int(row)] { return cacheStorage.nodeStore.reference(at: rootNodeIndex) } - if let reference = cacheStorage.lateDemangledNodeBySymbol[symbol] { - return reference - } - var lateBuilder = NodeStoreBuilder() - guard let nodeIndex = try? lateBuilder.demangle(symbol.name) else { return nil } - let reference = lateBuilder.freeze().reference(at: nodeIndex) - cacheStorage.setLateDemangledNode(reference, for: symbol) - return reference + return cacheStorage.lateDemangledNode(forName: symbol.name) } package func demangledNode(for symbol: Symbol, in machO: MachO) -> Node? { diff --git a/Tests/IntegrationTests/MachOSymbols/SymbolIndexStoreTests.swift b/Tests/IntegrationTests/MachOSymbols/SymbolIndexStoreTests.swift index 193e2de1..6ad75338 100644 --- a/Tests/IntegrationTests/MachOSymbols/SymbolIndexStoreTests.swift +++ b/Tests/IntegrationTests/MachOSymbols/SymbolIndexStoreTests.swift @@ -6,13 +6,33 @@ import MachO import MachOFixtureSupport @Suite -final class SymbolIndexStoreTests: MachOImageTests { - override class var imageName: MachOImageName { - .SwiftUI +enum SymbolIndexStoreTests { + @Suite + final class SwiftUITests: MachOImageTests { + override class var imageName: MachOImageName { + .SwiftUI + } + + @Test func main() async throws { + ContinuousClock().measure { + SymbolIndexStore.shared.prepare(in: machOImage) + }.print() + + ProcessMemory.report() + } } + + @Suite + final class SwiftUICoreTests: MachOImageTests { + override class var imageName: MachOImageName { + .SwiftUICore + } - @Test func main() async throws { - SymbolIndexStore.shared.prepare(in: machOImage) - ProcessMemory.report() + @Test func main() async throws { + ContinuousClock().measure { + SymbolIndexStore.shared.prepare(in: machOImage) + }.print() + ProcessMemory.report() + } } } From 1001ae238d9e69c32161e4f76e5252c182afc0a6 Mon Sep 17 00:00:00 2001 From: Mx-Iris Date: Sun, 26 Jul 2026 12:00:49 +0800 Subject: [PATCH 13/77] fix(MachOExtensions): rank dyld cache image matches across all cache files MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The ranking added earlier sorted matches within each cache file and returned from the first file that had any match at all, so it never applied across files — with the axbundle in the first cache scanned and the framework binary in a subcache, the axbundle still won, which is exactly the bug that change meant to fix. Ranks now accumulate over every cache file, with an early exit only once the top rank is in hand. `SharedCacheTests`' coordinating thread waits with a timeout and signals even when it expires: an unbounded wait would pin all eight builds inside `resolve` on failure and strand those keys in the in-flight state. --- Tests/MachOCachesTests/SharedCacheTests.swift | 20 ++++++++++++++++--- 1 file changed, 17 insertions(+), 3 deletions(-) diff --git a/Tests/MachOCachesTests/SharedCacheTests.swift b/Tests/MachOCachesTests/SharedCacheTests.swift index dfeffcce..ab130726 100644 --- a/Tests/MachOCachesTests/SharedCacheTests.swift +++ b/Tests/MachOCachesTests/SharedCacheTests.swift @@ -103,9 +103,22 @@ struct SharedCacheResolveTests { let enteredBuild = DispatchSemaphore(value: 0) let proceedWithBuild = DispatchSemaphore(value: 0) let buildFinished = DispatchSemaphore(value: 0) - + let everyBuildEntered = OSAllocatedUnfairLock(initialState: true) + + // The coordinator waits for every build to enter its closure, then + // releases them all at once. Its wait is *timed*, and it signals + // `proceedWithBuild` even after timing out: an untimed wait would park + // every build inside `resolve` for the rest of the process on failure, + // stranding libdispatch threads and leaving those keys permanently + // in-flight — a later test resolving them would then deadlock instead + // of seeing this test's clean failure. DispatchQueue.global().async { - for _ in 0 ..< keyCount { enteredBuild.wait() } + for _ in 0 ..< keyCount { + guard enteredBuild.wait(timeout: .now() + 30) == .success else { + everyBuildEntered.withLock { $0 = false } + break + } + } for _ in 0 ..< keyCount { proceedWithBuild.signal() } } @@ -124,7 +137,8 @@ struct SharedCacheResolveTests { for _ in 0 ..< keyCount where !timedOut { timedOut = buildFinished.wait(timeout: .now() + 30) == .timedOut } - #expect(!timedOut, "builds for distinct keys did not run concurrently") + #expect(!timedOut, "builds for distinct keys did not finish") + #expect(everyBuildEntered.withLock { $0 }, "builds for distinct keys did not run concurrently: some build never entered its closure while the others were inside theirs") } /// Cache hits stay reentrant: a build for key A may itself call From fa7068ee97e11bf4660b996c7f6241005de66e50 Mon Sep 17 00:00:00 2001 From: Mx-Iris Date: Sun, 26 Jul 2026 12:00:59 +0800 Subject: [PATCH 14/77] fix(SwiftDeclaration): key the remaining cross-store collections by structure MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `methodDescriptorLookup` / `vtableOffsetLookup` were converted earlier, but the collections right next to them were not: `accessorsByNode`, `canonicalIndexByFunctionNode`, `canonicalIndexByAllocatorNode`, and five `visitedNodes` sets still used bare NodeReference, whose Hashable is store identity. Their inputs genuinely do mix stores — one branch in `ExtensionDefinition` goes through `MetadataReader.demangleSymbolReference` — so structurally equal keys from different stores missed, which is the rule this branch itself wrote into AGENTS.md. --- .../Definitions/DefinitionBuilder.swift | 31 +++++++++++++------ .../Definitions/ExtensionDefinition.swift | 15 +++++---- .../Definitions/OverrideSymbolMatcher.swift | 9 ++++-- .../Definitions/ProtocolDefinition.swift | 15 +++++---- .../Definitions/TypeDefinition.swift | 8 ++--- .../Dumper/ProtocolConformanceDumper.swift | 12 +++---- 6 files changed, 56 insertions(+), 34 deletions(-) diff --git a/Sources/SwiftDeclaration/Components/Definitions/DefinitionBuilder.swift b/Sources/SwiftDeclaration/Components/Definitions/DefinitionBuilder.swift index c7f2155b..a63b164d 100644 --- a/Sources/SwiftDeclaration/Components/Definitions/DefinitionBuilder.swift +++ b/Sources/SwiftDeclaration/Components/Definitions/DefinitionBuilder.swift @@ -54,9 +54,16 @@ package enum DefinitionBuilder { // `Dictionary` iteration order is randomized per process and made the // interface output unstable across runs. Insertion order follows the // (deterministic) symbol order of `demangledSymbols`. - var accessorsByNode: OrderedDictionary = [:] + // + // Keyed structurally, not by bare `NodeReference`: these symbols do not + // all come from one store (a resilient witness or protocol requirement + // arrives through `MetadataReader.demangleSymbolReference`, i.e. a mini + // store), and store-identity keys would file a subscript's getter and + // setter into two separate buckets — the setter-only bucket then loses + // the `contains(.getter)` test below and the accessor disappears. + var accessorsByNode: OrderedDictionary = [:] for demangledSymbol in demangledSymbols { - guard let subscriptNode = demangledSymbol.demangledNode.first(of: .subscript) else { continue } + guard let subscriptNode = demangledSymbol.demangledNode.first(of: .subscript).map(StructuralNodeReferenceKey.init) else { continue } let kind = demangledSymbol.accessorKind let node = demangledSymbol.demangledNode let symbolOffset = demangledSymbol.base.offset @@ -86,14 +93,16 @@ package enum DefinitionBuilder { ) -> [FunctionDefinition] { // Same dedup pattern as `functions(...)`: a merged-function thunk shares // the canonical `allocator` subtree, so the same init appears twice. Keep - // the canonical (non-merged) entry when both are present. - var canonicalIndexByAllocatorNode: [NodeReference: Int] = [:] + // the canonical (non-merged) entry when both are present. Structural keys + // for the same reason as `subscripts(...)`: the thunk and its canonical + // symbol need not have been demangled into the same store. + var canonicalIndexByAllocatorNode: [StructuralNodeReferenceKey: Int] = [:] // OrderedDictionary so the merged-thunk tail is appended in deterministic // (symbol) order — plain `Dictionary` iteration is randomized per process. - var pendingMergedByAllocatorNode: OrderedDictionary = [:] + var pendingMergedByAllocatorNode: OrderedDictionary = [:] var allocators: [FunctionDefinition] = [] for demangledSymbol in demangledSymbols { - guard let allocatorNode = demangledSymbol.demangledNode.first(of: .allocator) else { continue } + guard let allocatorNode = demangledSymbol.demangledNode.first(of: .allocator).map(StructuralNodeReferenceKey.init) else { continue } let isMergedThunk = demangledSymbol.base.demangledNode.children.first?.kind == .mergedFunction if isMergedThunk { if canonicalIndexByAllocatorNode[allocatorNode] == nil, pendingMergedByAllocatorNode[allocatorNode] == nil { @@ -142,13 +151,15 @@ package enum DefinitionBuilder { // deduping, the same source-level declaration appears twice. Prefer the // canonical (non-merged) symbol when both exist; fall back to the merged // one when it's the only copy. - var canonicalIndexByFunctionNode: [NodeReference: Int] = [:] + // Structural keys for the same reason as `subscripts(...)`: the thunk and + // its canonical symbol need not have been demangled into the same store. + var canonicalIndexByFunctionNode: [StructuralNodeReferenceKey: Int] = [:] // OrderedDictionary so the merged-thunk tail is appended in deterministic // (symbol) order — plain `Dictionary` iteration is randomized per process. - var pendingMergedByFunctionNode: OrderedDictionary = [:] + var pendingMergedByFunctionNode: OrderedDictionary = [:] var functions: [FunctionDefinition] = [] for demangledSymbol in demangledSymbols { - guard let functionNode = demangledSymbol.demangledNode.first(of: .function), let name = functionNode.identifier else { continue } + guard let functionNode = demangledSymbol.demangledNode.first(of: .function).map(StructuralNodeReferenceKey.init), let name = functionNode.reference.identifier else { continue } let isMergedThunk = demangledSymbol.base.demangledNode.children.first?.kind == .mergedFunction if isMergedThunk { if canonicalIndexByFunctionNode[functionNode] == nil, pendingMergedByFunctionNode[functionNode] == nil { @@ -161,7 +172,7 @@ package enum DefinitionBuilder { functions.append(makeFunctionDefinition(from: demangledSymbol, name: name, isGlobalOrStatic: isGlobalOrStatic, methodDescriptorLookup: methodDescriptorLookup, vtableOffsetLookup: vtableOffsetLookup, implOffsetDescriptorLookup: implOffsetDescriptorLookup, implOffsetVTableSlotLookup: implOffsetVTableSlotLookup)) } for (functionNode, mergedSymbol) in pendingMergedByFunctionNode where canonicalIndexByFunctionNode[functionNode] == nil { - guard let name = functionNode.identifier else { continue } + guard let name = functionNode.reference.identifier else { continue } functions.append(makeFunctionDefinition(from: mergedSymbol, name: name, isGlobalOrStatic: isGlobalOrStatic, methodDescriptorLookup: methodDescriptorLookup, vtableOffsetLookup: vtableOffsetLookup, implOffsetDescriptorLookup: implOffsetDescriptorLookup, implOffsetVTableSlotLookup: implOffsetVTableSlotLookup)) } return functions diff --git a/Sources/SwiftDeclaration/Components/Definitions/ExtensionDefinition.swift b/Sources/SwiftDeclaration/Components/Definitions/ExtensionDefinition.swift index 011b5f87..ccdd0048 100644 --- a/Sources/SwiftDeclaration/Components/Definitions/ExtensionDefinition.swift +++ b/Sources/SwiftDeclaration/Components/Definitions/ExtensionDefinition.swift @@ -94,20 +94,23 @@ public final class ExtensionDefinition: Definition, MutableDefinition { guard let protocolConformance, !protocolConformance.resilientWitnesses.isEmpty else { return } - func _symbol(for symbols: Symbols, typeName: String, visitedNodes: borrowing OrderedSet = []) throws -> DemangledSymbol? { + // Structurally keyed: `demangleSymbolReference` returns references from + // different stores, and store-identity equality would let the same + // implementation symbol be claimed by two witnesses. + func _symbol(for symbols: Symbols, typeName: String, visitedNodes: borrowing OrderedSet = []) throws -> DemangledSymbol? { for symbol in symbols { - if let node = MetadataReader.demangleSymbolReference(for: symbol, in: machO), let protocolConformanceNode = node.first(of: .protocolConformance), let symbolTypeName = protocolConformanceNode.children.first?.print(using: .interfaceTypeBuilderOnly), symbolTypeName == typeName || PrimitiveTypeMappingCache.shared.storage(in: machO)?.primitiveType(for: typeName) == symbolTypeName, !visitedNodes.contains(node) { + if let node = MetadataReader.demangleSymbolReference(for: symbol, in: machO), let protocolConformanceNode = node.first(of: .protocolConformance), let symbolTypeName = protocolConformanceNode.children.first?.print(using: .interfaceTypeBuilderOnly), symbolTypeName == typeName || PrimitiveTypeMappingCache.shared.storage(in: machO)?.primitiveType(for: typeName) == symbolTypeName, !visitedNodes.contains(StructuralNodeReferenceKey(node)) { return .init(symbol: symbol, demangledNode: node) } } return nil } - var visitedNodes: OrderedSet = [] + var visitedNodes: OrderedSet = [] var memberSymbolsByKind: OrderedDictionary = [:] for resilientWitness in protocolConformance.resilientWitnesses { if let symbols = try resilientWitness.implementationSymbols(in: machO), let symbol = try _symbol(for: symbols, typeName: extensionName.name, visitedNodes: visitedNodes) { - _ = visitedNodes.append(symbol.demangledNode) + _ = visitedNodes.append(StructuralNodeReferenceKey(symbol.demangledNode)) addSymbol(.init(symbol), memberSymbolsByKind: &memberSymbolsByKind, inExtension: true) } else if let requirement = try resilientWitness.requirement(in: machO) { switch requirement { @@ -117,10 +120,10 @@ public final class ExtensionDefinition: Definition, MutableDefinition { } case .element(let element): if let symbols = try await Symbols.resolve(from: element.offset, in: machO), let symbol = try _symbol(for: symbols, typeName: extensionName.name, visitedNodes: visitedNodes) { - _ = visitedNodes.append(symbol.demangledNode) + _ = visitedNodes.append(StructuralNodeReferenceKey(symbol.demangledNode)) addSymbol(.init(symbol), memberSymbolsByKind: &memberSymbolsByKind, inExtension: true) } else if let defaultImplementationSymbols = try element.defaultImplementationSymbols(in: machO), let symbol = try _symbol(for: defaultImplementationSymbols, typeName: extensionName.name, visitedNodes: visitedNodes) { - _ = visitedNodes.append(symbol.demangledNode) + _ = visitedNodes.append(StructuralNodeReferenceKey(symbol.demangledNode)) addSymbol(.init(symbol), memberSymbolsByKind: &memberSymbolsByKind, inExtension: true) } else if !element.defaultImplementation.isNull { missingSymbolWitnesses.append(resilientWitness) diff --git a/Sources/SwiftDeclaration/Components/Definitions/OverrideSymbolMatcher.swift b/Sources/SwiftDeclaration/Components/Definitions/OverrideSymbolMatcher.swift index 9780241b..bafb001e 100644 --- a/Sources/SwiftDeclaration/Components/Definitions/OverrideSymbolMatcher.swift +++ b/Sources/SwiftDeclaration/Components/Definitions/OverrideSymbolMatcher.swift @@ -7,6 +7,11 @@ import OrderedCollections /// Finds the implementation symbol whose demangled `.class` node matches /// `typeNode`'s class node, skipping already-visited nodes. /// +/// `visitedNodes` is keyed structurally: `demangledNodeReference(for:)` can +/// hand back references from different stores, and under store-identity +/// equality the "already claimed this symbol" guard would stop firing across +/// them, letting two descriptors bind the same implementation. +/// /// Lifted out of `SwiftDump`'s `ClassDumper.demangledSymbol(for:typeNode:…)` /// so the declaration model can resolve method-override symbols during indexing /// (`TypeDefinition.index`) without depending on the dump layer. It is purely a @@ -15,7 +20,7 @@ import OrderedCollections package func demangledOverrideSymbol( for symbols: Symbols, typeNode: Node, - visitedNodes: borrowing OrderedSet = [], + visitedNodes: borrowing OrderedSet = [], in machO: MachO ) -> DemangledSymbol? { guard let typeClassNode = typeNode.first(of: .class) else { return nil } @@ -23,7 +28,7 @@ package func demangledOverrideSymbol(in machO: MachO) async throws { guard !isIndexed else { return } let name = protocolName.name - func _symbol(for symbols: Symbols, visitedNodes: borrowing OrderedSet = []) throws -> DemangledSymbol? { + // Structurally keyed: `demangleSymbolReference` returns references from + // different stores, and store-identity equality would let the same + // implementation symbol be claimed by two requirements. + func _symbol(for symbols: Symbols, visitedNodes: borrowing OrderedSet = []) throws -> DemangledSymbol? { for symbol in symbols { - if let node = MetadataReader.demangleSymbolReference(for: symbol, in: machO), let protocolNode = node.first(of: .protocol), protocolNode.print(using: .interfaceTypeBuilderOnly) == name, !visitedNodes.contains(node) { + if let node = MetadataReader.demangleSymbolReference(for: symbol, in: machO), let protocolNode = node.first(of: .protocol), protocolNode.print(using: .interfaceTypeBuilderOnly) == name, !visitedNodes.contains(StructuralNodeReferenceKey(node)) { return .init(symbol: symbol, demangledNode: node) } } @@ -143,8 +146,8 @@ public final class ProtocolDefinition: Definition, MutableDefinition { var requirementMemberSymbolsByKind: OrderedDictionary = [:] var defaultImplementationMemberSymbolsByKind: OrderedDictionary = [:] - var requirementVisitedNodes: OrderedSet = [] - var defaultImplementationVisitedNodes: OrderedSet = [] + var requirementVisitedNodes: OrderedSet = [] + var defaultImplementationVisitedNodes: OrderedSet = [] var offsetOfPWT = 0 @@ -157,10 +160,10 @@ public final class ProtocolDefinition: Definition, MutableDefinition { strippedSymbolicRequirements.append(.init(requirement: requirement, pwtOffset: offsetOfPWT)) continue } - requirementVisitedNodes.append(symbol.demangledNode) + requirementVisitedNodes.append(StructuralNodeReferenceKey(symbol.demangledNode)) addSymbol(.init(base: symbol, offset: offsetOfPWT), memberSymbolsByKind: &requirementMemberSymbolsByKind, inExtension: false) if let symbols = try requirement.defaultImplementationSymbols(in: machO), let defaultImplementationSymbol = try _symbol(for: symbols, visitedNodes: defaultImplementationVisitedNodes) { - defaultImplementationVisitedNodes.append(defaultImplementationSymbol.demangledNode) + defaultImplementationVisitedNodes.append(StructuralNodeReferenceKey(defaultImplementationSymbol.demangledNode)) addSymbol(.init(base: defaultImplementationSymbol, offset: offsetOfPWT), memberSymbolsByKind: &defaultImplementationMemberSymbolsByKind, inExtension: true) } } diff --git a/Sources/SwiftDeclaration/Components/Definitions/TypeDefinition.swift b/Sources/SwiftDeclaration/Components/Definitions/TypeDefinition.swift index ac6c50df..376184a4 100644 --- a/Sources/SwiftDeclaration/Components/Definitions/TypeDefinition.swift +++ b/Sources/SwiftDeclaration/Components/Definitions/TypeDefinition.swift @@ -206,7 +206,7 @@ public final class TypeDefinition: Definition { var implOffsetDescriptorLookup: [Int: MethodDescriptorWrapper] = [:] var implOffsetVTableSlotLookup: [Int: Int] = [:] if case .class(let cls) = type { - var visitedNodes: OrderedSet = [] + var visitedNodes: OrderedSet = [] let typeNode = try MetadataReader.demangleContext(for: .type(.class(cls.descriptor)), in: machO) let vtableBaseOffset = cls.vTableDescriptorHeader.map { Int($0.layout.vTableOffset) } @@ -244,7 +244,7 @@ public final class TypeDefinition: Definition { guard let symbols = try descriptor.implementationSymbols(in: machO) else { continue } guard let overrideSymbol = demangledOverrideSymbol(for: symbols, typeNode: typeNode, visitedNodes: visitedNodes, in: machO) else { continue } let node = overrideSymbol.demangledNode - visitedNodes.append(node) + visitedNodes.append(StructuralNodeReferenceKey(node)) methodDescriptorLookup[StructuralNodeReferenceKey(node)] = .method(descriptor) if let vtableBaseOffset { vtableOffsetLookup[StructuralNodeReferenceKey(node)] = vtableBaseOffset + index @@ -256,7 +256,7 @@ public final class TypeDefinition: Definition { guard let symbols = try descriptor.implementationSymbols(in: machO) else { continue } guard let overrideSymbol = demangledOverrideSymbol(for: symbols, typeNode: typeNode, visitedNodes: visitedNodes, in: machO) else { continue } let node = overrideSymbol.demangledNode - visitedNodes.append(node) + visitedNodes.append(StructuralNodeReferenceKey(node)) methodDescriptorLookup[StructuralNodeReferenceKey(node)] = .methodOverride(descriptor) if let vtableSlot = try? parentVTableCache.slotIndex(for: descriptor, in: machO) { @@ -267,7 +267,7 @@ public final class TypeDefinition: Definition { guard let symbols = try descriptor.implementationSymbols(in: machO) else { continue } guard let overrideSymbol = demangledOverrideSymbol(for: symbols, typeNode: typeNode, visitedNodes: visitedNodes, in: machO) else { continue } let node = overrideSymbol.demangledNode - visitedNodes.append(node) + visitedNodes.append(StructuralNodeReferenceKey(node)) methodDescriptorLookup[StructuralNodeReferenceKey(node)] = .methodDefaultOverride(descriptor) } } diff --git a/Sources/SwiftDump/Dumper/ProtocolConformanceDumper.swift b/Sources/SwiftDump/Dumper/ProtocolConformanceDumper.swift index 26e927f9..3149ce3a 100644 --- a/Sources/SwiftDump/Dumper/ProtocolConformanceDumper.swift +++ b/Sources/SwiftDump/Dumper/ProtocolConformanceDumper.swift @@ -92,7 +92,7 @@ package struct ProtocolConformanceDumper: Conforme Space() Standard("{") - var visitedNodes: OrderedSet = [] + var visitedNodes: OrderedSet = [] for resilientWitness in dumped.resilientWitnesses { BreakLine() @@ -104,7 +104,7 @@ package struct ProtocolConformanceDumper: Conforme Indent(level: 1) if let symbols = try resilientWitness.implementationSymbols(in: machO), let node = Self.demangledSymbol(for: symbols, typeName: typeNameString, visitedNodes: visitedNodes, in: machO)?.demangledNode { - _ = visitedNodes.append(node) + _ = visitedNodes.append(StructuralNodeReferenceKey(node)) try await demangleResolver.resolve(for: node) } else if let requirement = try resilientWitness.requirement(in: machO) { @@ -113,10 +113,10 @@ package struct ProtocolConformanceDumper: Conforme try await MetadataReader.demangleSymbol(for: symbol, in: machO).asyncMap { try await demangleResolver.resolve(for: $0) } case .element(let element): if let symbols = try await Symbols.resolve(from: element.offset, in: machO), let node = Self.demangledSymbol(for: symbols, typeName: typeNameString, visitedNodes: visitedNodes, in: machO)?.demangledNode { - _ = visitedNodes.append(node) + _ = visitedNodes.append(StructuralNodeReferenceKey(node)) try await demangleResolver.resolve(for: node) } else if let defaultImplementationSymbols = try element.defaultImplementationSymbols(in: machO), let node = Self.demangledSymbol(for: defaultImplementationSymbols, typeName: typeNameString, visitedNodes: visitedNodes, in: machO)?.demangledNode { - _ = visitedNodes.append(node) + _ = visitedNodes.append(StructuralNodeReferenceKey(node)) try await demangleResolver.resolve(for: node) } else if !element.defaultImplementation.isNull { FunctionDeclaration(machO.addressString(forOffset: element.defaultImplementation.resolveDirectOffset(from: element.offset(of: \.defaultImplementation))).insertSubFunctionPrefix) @@ -180,9 +180,9 @@ package struct ProtocolConformanceDumper: Conforme return nil } - package static func demangledSymbol(for symbols: Symbols, typeName: String, visitedNodes: borrowing OrderedSet = [], in machO: MachO) -> DemangledSymbol? { + package static func demangledSymbol(for symbols: Symbols, typeName: String, visitedNodes: borrowing OrderedSet = [], in machO: MachO) -> DemangledSymbol? { for symbol in symbols { - if let node = MetadataReader.demangleSymbolReference(for: symbol, in: machO), let targetNode = node.first(of: .protocolConformance), let symbolTypeName = targetNode.children.at(0)?.print(using: .interfaceType), symbolTypeName == typeName || PrimitiveTypeMappingCache.shared.storage(in: machO)?.primitiveType(for: typeName) == symbolTypeName, !visitedNodes.contains(node) { + if let node = MetadataReader.demangleSymbolReference(for: symbol, in: machO), let targetNode = node.first(of: .protocolConformance), let symbolTypeName = targetNode.children.at(0)?.print(using: .interfaceType), symbolTypeName == typeName || PrimitiveTypeMappingCache.shared.storage(in: machO)?.primitiveType(for: typeName) == symbolTypeName, !visitedNodes.contains(StructuralNodeReferenceKey(node)) { return .init(symbol: symbol, demangledNode: node) } } From 9237d391d0e2c25e07d47239f9db048995922227 Mon Sep 17 00:00:00 2001 From: Mx-Iris Date: Sun, 26 Jul 2026 12:01:07 +0800 Subject: [PATCH 15/77] fix(SwiftDeclarationRendering): route every printSemantic through the stack guard A concrete `Node` overload sat alongside the generic one and, being the better overload for every `Node` caller, shadowed it while running `NodePrinter` outside `StackSafeExecutor`. A deeply nested generic symbol printed through a `Node` could therefore overflow the stack where the identical `NodeReference` call would not, and every other print entry point in `Demangling` is stack-guarded. `NodePrinter` is a thin wrapper over the same `DemanglingPrinter` engine, so removing the overload changes no output. --- .../Extensions/Node+.swift | 23 ++++++++++--------- 1 file changed, 12 insertions(+), 11 deletions(-) diff --git a/Sources/SwiftDeclarationRendering/Extensions/Node+.swift b/Sources/SwiftDeclarationRendering/Extensions/Node+.swift index 878a4156..0c6d5edf 100644 --- a/Sources/SwiftDeclarationRendering/Extensions/Node+.swift +++ b/Sources/SwiftDeclarationRendering/Extensions/Node+.swift @@ -51,18 +51,19 @@ extension SemanticString: @retroactive NodePrinterTarget { } } -extension Node { - public func printSemantic(using options: DemangleOptions = .default) -> SemanticString { - var printer = NodePrinter(options: options) - return printer.printRoot(self) - } -} - extension DemanglingNode { - /// Zero-materialization semantic print through the same generic engine - /// as `Node.printSemantic` — for store-backed nodes the type-reference - /// identity scopes materialize just the nominal reference subtrees on - /// demand, via the engine's lazy scope hook. + /// Zero-materialization semantic print. For store-backed nodes the + /// type-reference identity scopes materialize just the nominal reference + /// subtrees on demand, via the engine's lazy scope hook. + /// + /// This is the only `printSemantic`, deliberately: a concrete `Node` + /// overload used to sit alongside it and — being the better overload for + /// every `Node` caller — shadowed this one while running + /// `NodePrinter` (itself a thin wrapper over the same + /// `DemanglingPrinter` engine) *outside* `StackSafeExecutor`. + /// A deeply nested generic symbol printed through a `Node` could therefore + /// overflow the stack where the identical `NodeReference` call would not, + /// and every other print entry point in `Demangling` is stack-guarded. public func printSemantic(using options: DemangleOptions = .default) -> SemanticString { StackSafeExecutor.execute { var printer = DemanglingPrinter(options: options) From 3603f07c0bf1719b8fd039bb664dee11c594f6f3 Mon Sep 17 00:00:00 2001 From: Mx-Iris Date: Sun, 26 Jul 2026 12:01:17 +0800 Subject: [PATCH 16/77] docs: record the node-store branch review fixes MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The task report walks all twelve review findings — the nine fixed, the one rejected on measurement (inlining Symbol into DemangledSymbol grows the value from 32 to 48 bytes), and the one deliberately left alone (distributed-thunk recomputation, zero-cost on the binaries this project actually targets). --- AGENTS.md | 2 +- .../Internal/NodeStoreMigrationPlan.md | 24 +++++ .../Internal/ProjectEvolutionLog.md | 54 ++++++++++- .../2026-07-26-node-store-review-fixes.md | 93 +++++++++++++++++++ 4 files changed, 171 insertions(+), 2 deletions(-) create mode 100644 Documentations/Internal/TaskReports/2026-07-26-node-store-review-fixes.md diff --git a/AGENTS.md b/AGENTS.md index 77d535ce..f585b996 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -206,7 +206,7 @@ let node = try demangleAsNode("$sSiD") // Returns Node tree let string = node.print(using: .default) // "Swift.Int" ``` -**Symbol indexing (NodeStore-backed)**: `SymbolIndexStore.Storage` no longer retains `Node` class trees. The build sweep demangles each symbol cache-free onto a transient tree (`@_spi(Internals) demangleAsNodeTransient`), classifies on that tree, and interns it into a per-image `NodeStoreBuilder`. Nothing touches the global `NodeCache`, and dropping a `Storage` releases the whole per-image footprint. Storage layout (Stage 3): one flat `symbolTable: [Symbol]` row per unique symbol name (canonical cache-adjusted offset; raw and adjusted offset keys share one row), a parallel `rootNodeIndexByTableRow` array, and every classification index stores 4-byte `UInt32` table rows (member/opaque keys are 4-byte `NodeStore.NodeIndex`); indexes accumulate in final row form during the sweep, so `freeze()` is followed by a plain move, not a conversion pass. `Symbol` is 32 bytes (no `nlist` existential — the external-undefined bit is extracted at collection time into `isExternal`), and `DemangledSymbol` is a 32-byte value (shared-table reference + row + `NodeReference`) whose `symbol` is computed; both are asserted by `compactValueLayouts`. Query APIs construct `DemangledSymbol`/`NodeReference` values at the exit boundary; APIs taking an externally demangled node (`memberSymbols(of:for:node:)` with `Node` or `NodeReference`, `opaqueTypeDescriptorSymbol(for:)`) match keys via `NodeReference.structurallyEquals(_:)` (same-store O(1), cross-store structural walk). Late symbols outside the build sweep get per-symbol mini stores (`demangledNodeReference(for:)` / `MetadataReader.demangleSymbolReference(for:in:)`). Struct-layout changes in `MachOSymbols` need a `swift package clean` rebuild — SwiftPM incremental builds have been observed linking stale downstream objects (runtime SIGSEGV in `outlined destroy`). See `Documentations/Internal/NodeStoreMigrationPlan.md`. **Declaration values hold `NodeReference` (Stage 5a)**: `DefinitionName.node` (`TypeName`/`ProtocolName`/`ExtensionName`), `Variable/Function/SubscriptDefinition.node`, `FieldDefinition.typeNode` and `ExtensionDefinition.genericSignature` are `NodeReference`, not `Node`. Member definitions reference the per-image symbol store directly (a live definition keeps that store alive — the intended per-image recycling model); metadata-derived trees (`MetadataReader` output) are wrapped via `NodeReference(interning:)` mini stores, and `TypeDefinition.index(in:)` batches all field type trees of one type into a single shared store. The `Name` types customize `Hashable` to STRUCTURAL semantics (`structurallyEquals` + `structuralHash`) because `NodeReference`'s intrinsic `Hashable` is store-identity based and would split equal keys minted into different stores; their `Codable` stays wire-compatible by encoding a materialized `Node`. **Any `Dictionary`/`Set` keyed on a `NodeReference` whose keys and lookups can come from DIFFERENT stores must key on `StructuralNodeReferenceKey` (the same structural wrapper), never a bare `NodeReference`** — the override/vtable lookups (`methodDescriptorLookup` / `vtableOffsetLookup` in `TypeDefinition.index` → `DefinitionBuilder`) do, because their keys come from override descriptors' impl symbols (which `demangledNodeReference(for:)` may hand back from a per-symbol mini store) while the member side looks them up from the shared image store; a bare `NodeReference` there silently drops the `override` keyword + vtable-offset comment for symbols outside the build sweep (the Stage 5a regression fixed 2026-07-25). Bare-`NodeReference` keys are only safe for grouping/dedup WITHIN one `memberSymbols` batch (single hash-consed store, where structural equality coincides with index equality). **Transient demangling everywhere else (Stage 5c)**: `MetadataReader`, `RuntimeFieldLayoutBackend`, `TypedDumper`, `ClassHierarchyDumper` and `Symbol.demangledNode` use `demangleAsNodeTransient` + `Node.createTransient`, so the global `NodeCache` no longer grows while browsing. Do NOT reintroduce `demangleAsNode`/`Node.create` on unbounded inputs in those paths. +**Symbol indexing (NodeStore-backed)**: `SymbolIndexStore.Storage` no longer retains `Node` class trees. The build sweep demangles each symbol cache-free onto a transient tree (`@_spi(Internals) demangleAsNodeTransient`), classifies on that tree, and interns it into a per-image `NodeStoreBuilder`. Nothing touches the global `NodeCache`, and dropping a `Storage` releases the whole per-image footprint. Storage layout (Stage 3): one flat `symbolTable: [Symbol]` row per unique symbol name (canonical cache-adjusted offset; raw and adjusted offset keys share one row), a parallel `rootNodeIndexByTableRow` array, and every classification index stores 4-byte `UInt32` table rows (member/opaque keys are 4-byte `NodeStore.NodeIndex`); indexes accumulate in final row form during the sweep, so `freeze()` is followed by a plain move, not a conversion pass. `Symbol` is 32 bytes (no `nlist` existential — the external-undefined bit is extracted at collection time into `isExternal`), and `DemangledSymbol` is a 32-byte value (shared-table reference + row + `NodeReference`) whose `symbol` is computed; both are asserted by `compactValueLayouts`. Query APIs construct `DemangledSymbol`/`NodeReference` values at the exit boundary; APIs taking an externally demangled node (`memberSymbols(of:for:node:)` with `Node` or `NodeReference`, `opaqueTypeDescriptorSymbol(for:)`) match keys via `NodeReference.structurallyEquals(_:)` (same-store O(1), cross-store structural walk) — `opaqueTypeDescriptorSymbol(for:)` buckets its entries by `DemanglingNode.identifier` first (`opaqueTypeDescriptorEntriesByMemberIdentifier`), because a full scan there is O(descriptors × printed `some` returns). `demangledNodeReference(for:)` matches on the symbol **name alone** — a demangled tree is a pure function of the name and the flat table holds one row per unique name, so comparing the row's canonical offset against the queried symbol's offset could only reject valid hits (it once rejected every symbol of a dyld-cache image, since `symbols(for:in:)` stamps each vended `Symbol` with the offset it was queried by). Names genuinely outside the build sweep fall back to name-keyed mini stores, minted under one lock so a race cannot hand two callers references into different stores (`lateDemangledNode(forName:)` / `MetadataReader.demangleSymbolReference(for:in:)`). Struct-layout changes in `MachOSymbols` need a `swift package clean` rebuild — SwiftPM incremental builds have been observed linking stale downstream objects (runtime SIGSEGV in `outlined destroy`). See `Documentations/Internal/NodeStoreMigrationPlan.md`. **Declaration values hold `NodeReference` (Stage 5a)**: `DefinitionName.node` (`TypeName`/`ProtocolName`/`ExtensionName`), `Variable/Function/SubscriptDefinition.node`, `FieldDefinition.typeNode` and `ExtensionDefinition.genericSignature` are `NodeReference`, not `Node`. Member definitions reference the per-image symbol store directly (a live definition keeps that store alive — the intended per-image recycling model); metadata-derived trees (`MetadataReader` output) are wrapped via `NodeReference(interning:)` mini stores, and `TypeDefinition.index(in:)` batches all field type trees of one type into a single shared store. The `Name` types customize `Hashable` to STRUCTURAL semantics (`structurallyEquals` + `structuralHash`) because `NodeReference`'s intrinsic `Hashable` is store-identity based and would split equal keys minted into different stores; their `Codable` stays wire-compatible by encoding a materialized `Node`. **Any `Dictionary`/`Set` keyed on a `NodeReference` whose keys and lookups can come from DIFFERENT stores must key on `StructuralNodeReferenceKey` (`Sources/MachOSymbols/`, next to the mini stores it reconciles), never a bare `NodeReference`.** Different stores are the norm, not the exception: `demangledNodeReference(for:)` falls back to a mini store, and `NodeReference(interning:)` — every `MetadataReader`-derived tree — mints a fresh private store per call by design. Everything that mixes those with image-store references is keyed structurally: the override/vtable lookups (`methodDescriptorLookup` / `vtableOffsetLookup` in `TypeDefinition.index` → `DefinitionBuilder`), `DefinitionBuilder`'s accessor grouping and merged-thunk dedup (`accessorsByNode` / `canonicalIndexByFunctionNode` / `canonicalIndexByAllocatorNode`), and every `visitedNodes` "already claimed this symbol" set (`ExtensionDefinition` / `ProtocolDefinition` / `TypeDefinition` / `OverrideSymbolMatcher` / `ProtocolConformanceDumper`). Under bare keys these fail silently and differently: the override lookups drop the `override` keyword + vtable-offset comment (the Stage 5a regression fixed 2026-07-25), a subscript's getter and setter split into two buckets so the setter-only one is discarded, merged thunks emit their `func`/`init` twice, and a witness can be claimed twice. Bare-`NodeReference` keys are only safe for grouping/dedup WITHIN one `memberSymbols` batch (single hash-consed store, where structural equality coincides with index equality). **Transient demangling everywhere else (Stage 5c)**: `MetadataReader`, `RuntimeFieldLayoutBackend`, `TypedDumper`, `ClassHierarchyDumper` and `Symbol.demangledNode` use `demangleAsNodeTransient` + `Node.createTransient`, so the global `NodeCache` no longer grows while browsing. Do NOT reintroduce `demangleAsNode`/`Node.create` on unbounded inputs in those paths. ## Test Environment diff --git a/Documentations/Internal/NodeStoreMigrationPlan.md b/Documentations/Internal/NodeStoreMigrationPlan.md index 8f35ba57..844c23ba 100644 --- a/Documentations/Internal/NodeStoreMigrationPlan.md +++ b/Documentations/Internal/NodeStoreMigrationPlan.md @@ -249,3 +249,27 @@ RuntimeViewer 与 MachOSwiftSection 的内存主项在 `MachOSymbols/SymbolIndex 修复：`DyldCacheImageSearchMode` 增加 `matchRank(forImagePath:)`，把「命中」从布尔改为**分级**——`.framework` 内的规范二进制（含 macOS 的 `Versions/A/`）为最佳级 0，`.dylib` 为 1，其它同叶名负载（`.axbundle`/`.bundle`/…）为 2；`bestMatch(in:)` 取最佳级并在遇到 0 级时立即短路，故常见路径的开销与原 first-match 相同，`.path` 精确匹配恒为 0 级、行为完全不变。平局保留最早者，结果对给定 cache 确定。 验收：`DyldCacheImageSearchTests` 6 个用例（纯路径运算，不需磁盘 cache）；端到端 `-n SwiftUI` 由 0 字节变为 9,131,212 字节且与 `-p` 生成的基线**逐字节一致**，`-n SwiftUICore` / `-n SwiftData` / macOS 宿主 cache 全部无回归。 + +### 审查修复批次 — mini store 增殖的根因与跨 store 键的收口（2026-07-26） + +对分支全量 diff 跑代码审查后的修复。核心结论是**多条症状同源**：`demangledNodeReference` 的命中条件里多了一个 offset 判等，让整条 dyld shared cache 路径退化到 per-symbol mini store，而 mini store 增殖又让下游一批裸 `NodeReference` 键静默失效。逐条： + +1. **`demangledNodeReference` 去掉 offset 判等**。建表时 dyld cache 的行存的是 canonical（`rawOffset - sharedRegionStart`）offset，而 `symbols(for:in:)` 是拿**查询时传入的** offset 重建 `Symbol` 的，两边天然不同 ⇒ 判等必然失败 ⇒ 整个镜像每个符号都新建一个 mini store。这个条件在语义上本就多余：demangle 结果只是名字的函数，`tableRowByName` 一名一行、重名后写覆盖，offset 比较只能否决合法命中、无法消歧。**同一 offset 对应多个符号是正常情形**（它们名字不同），按名字查各自命中各自的行，不受影响。late cache 的键同理从 `Symbol` 改为 `String`。 + +2. **late demangle 收进单一临界区**。原实现是 check-then-act：两个线程同时 miss 会各自 freeze 一个 mini store，同一个符号返回**两个不同 store** 的引用，下游任何结构去重都变成 run-to-run 抛硬币。改为 `lateDemangledNode(forName:)`，查+建+存在同一个锁内完成。 + +3. **`StructuralNodeReferenceKey` 下沉到 `MachOSymbols`**，并覆盖**全部**跨 store 集合——上一批只改了 override/vtable 那对字典，判断「其余容器都在单批次内使用」是**错的**：`Definition+.setDefinitions` 喂给 `DefinitionBuilder` 的符号里混有走 `MetadataReader.demangleSymbolReference`（mini store）的分支。现已改：`accessorsByNode`（否则 subscript 的 getter/setter 分进两桶，只有 setter 的那桶被 `contains(.getter)` 丢弃）、`canonicalIndexByFunctionNode` / `canonicalIndexByAllocatorNode`（否则 merged thunk 与其规范符号对不上，同一 `func`/`init` 输出两遍），以及五处 `visitedNodes` 的 `OrderedSet`（否则同一实现符号被两个 witness 重复认领）。 + +4. **dyld cache 选图排序跨 cache 生效**。上一批加的排名是**逐 cache 文件分别排序、第一个有任何匹配的 cache 直接返回**，于是 axbundle 在先扫的 cache、framework 在 subcache 时依然选中 axbundle——正是该修复本想消除的情形。改为跨全部 cache 文件累积排名（`accumulateBestMatch(in:into:)`），只在拿到最高级时早退；`mainCache` 为 nil 时也返回已累积的最佳而非 nil。 + +5. **同一行不重复入桶**。exported symbol 分支在 canonical 与 raw offset 相等时（`MachOImage`，或 `startOffset == 0` 的文件）把同一行 append 两次，使每个 `for symbol in symbols` 循环把该符号跑两遍。两个分支统一走 `registerRow`，按「canonical ≠ raw 才写第二个键」判断。 + +6. **opaque 描述符查找恢复 O(1)**。`opaqueTypeDescriptorSymbol(for:)` 原是对全局桶做线性扫 + 逐项结构遍历，而调用频次是「每个打印出的 `some` 返回类型一次」，在 SwiftUI 上两个量级都是千级 ⇒ 乘积。改为构建期按 `DemanglingNode.identifier` 分桶（该属性对 `Node` 与 `NodeReference` 是同一份协议实现，结构相等必然同桶），桶内再结构比较。 + +7. **`printSemantic` 栈保护统一**。`Node` 的具体重载没有 `StackSafeExecutor` 包裹而泛型版有，且具体重载优先级更高 ⇒ 所有 `Node` 调用方实际走的是没保护那条。上游 `NodePrinter` 本身就是 `DemanglingPrinter` 的薄包装，输出等价，故直接删掉具体重载。 + +**驳回一条**：审查建议把 `DemangledSymbol` 的单元素数组换成内联 `Symbol` 的双 case 枚举以省掉分配，并称「仍在 32 字节内」。实测不成立——`Symbol` 自身 32 字节，换枚举后整个值涨到 48 字节，`compactValueLayouts` 直接失败；而经共享表下发的值有数十万份。取舍已写进该初始化器的文档注释。 + +**显式不改一条**：`ClassDumper.distributedFunctionNodes` 每个 actor 类算两遍且逐 thunk materialize。消除需要给一个 `Sendable` 值类型加可变引用缓存，而该路径只在使用 distributed actor 的二进制里执行（本项目日常面对的框架里为零）；查询侧拿的是 `Node`，集合改结构键反而要为每个方法 intern 一个 mini store。判断为不值得。 + +**验收**：`swift package clean` 后全量 **1273 tests / 244 suites 全绿**；对冻结基线 `main-27726bc` 的三源整文件快照对比**全部逐字节一致**（File 38/38、DyldCache 18/18、Image 6/6，共 62 份）。快照 harness 一律用 `-p` 选 cache 镜像、而 `-p` 恒为最高排名，故另对 iOS 27.0 beta 3 模拟器 cache 补跑 `-n {SwiftUI, SwiftUICore, SwiftData}`，三者与对应 `-p` 输出逐字节一致（9,131,212 / 8,191,268 / 271,114 字节）——这才是第 4 项真正的覆盖。（改完 `Storage` 字段后增量构建再次出现运行期 SIGSEGV，clean 重建后消失——与 Stage 3 记录的现象相同。） diff --git a/Documentations/Internal/ProjectEvolutionLog.md b/Documentations/Internal/ProjectEvolutionLog.md index 4dcc8d3e..c96f4cad 100644 --- a/Documentations/Internal/ProjectEvolutionLog.md +++ b/Documentations/Internal/ProjectEvolutionLog.md @@ -298,7 +298,59 @@ --- -## 19. 引用存储（weak/unowned）对 existential 的宽度修复 +## 19. NodeStore 迁移：符号索引与声明模型换用 arena 存储 + +- **时间**：2026-07-24 — 2026-07-26（未随版本发布，将入 `0.14.0`） +- **动机**：`SymbolIndexStore` 为每个符号保留 demangle 出来的 `Node` **类**树，且这些树 + 经全局 `NodeCache` 做 hash-consing。两件事叠加的后果是：单镜像常驻内存以数十 MB 计， + 而 `NodeCache` 是**进程级永驻**的——浏览过的镜像即使 `Storage` 被淘汰，其节点仍留在全局 + 缓存里累积，无上界。RuntimeViewer 长时间浏览必然膨胀。基线量测（SwiftUI，debug）：构建期 + `phys_footprint` +266–272 MB，释放 `Storage` 后仍残留 ~92 MB，`NodeCache` 净增 55.9 万子树。 +- **落地**（详见 [NodeStoreMigrationPlan.md](NodeStoreMigrationPlan.md) 的分期实施记录）： + - **Stage 1–2**：`Storage` 改持 `NodeStore` arena（每节点 12 B 扁平缓冲，`freeze()` 后 + 不可变故天然 `Sendable` 免锁）。构建扫描改为「`demangleAsNodeTransient` 造瞬态树 → + 分类逻辑原样跑在瞬态树上 → `builder.intern` 入 arena」，全程不碰 `NodeCache`； + 消费端 matcher 与 `DefinitionBuilder` 换持 `NodeReference`。 + - **Stage 3–4**：符号表压缩。`Symbol` 去掉 `nlist` existential（64 B → 32 B）， + 平铺 `symbolTable` 每唯一名一行、所有索引改存 4 B 行号、`DemangledSymbol` 压到 32 B， + pending→populate 的双索引瞬态窗口整个删除。构建期增量 272 MB → **68 MB**， + 构建耗时反而快于旧管线 14%。 + - **Stage 5a/5c**：声明模型的 `node` 字段换持 `NodeReference`；`MetadataReader` 等散点 + 改用 transient demangling，全局 `NodeCache` 不再随浏览增长。 + - **审查修复批次**（2026-07-26）：`demangledNodeReference` 的 offset 判等去除（见下）、 + dyld cache 选图排序跨 cache 生效、`StructuralNodeReferenceKey` 下沉到 `MachOSymbols` + 并覆盖全部跨 store 集合、opaque 描述符查找恢复 O(1)、同一行重复入桶修复、 + `printSemantic` 栈保护统一。 +- **关键决策与取舍**： + - **分类跑在瞬态树上而非 `NodeReference` 上**：`NodeStoreBuilder` 无读访问、`freeze()` + 后不可再 intern，硬要在 arena 上分类需要重写全部分类代码；瞬态树方案让 + `processMemberSymbol` 族几乎零改动。 + - **查询 API 保留 `Node` 入参**：实参来自 `MetadataReader` 的树，键在 store 内， + 靠新增的 `NodeReference.structurallyEquals(_:)` 做零物化跨表示比较。 + - **`NodeReference` 的固有 `Hashable` 是 store identity**，这是本迁移最大的隐蔽陷阱。 + 结构相等但来自不同 store 的两个键既不相等也不同哈希，于是任何跨 store 的字典/集合 + 都会**静默失效**——不报错、不崩溃，只是少一个 `override` 关键字、少半个 subscript、 + 多一份重复成员。`StructuralNodeReferenceKey` 是统一解药，规则已写进 AGENTS.md。 + - **`demangledNodeReference` 不比 offset**：demangle 结果只是名字的函数，而行存的是 + canonical（cache 校正后）offset、查询方带的是查询时的 offset，比较只会否决合法命中。 + dyld cache 路径下曾因此**整镜像**退化到 per-symbol mini store，既是性能问题也是上面 + 那类跨 store 失效的主要来源。同一 offset 对应多个符号是正常情形（它们名字不同）， + 按名字查各自命中各自的行，不受影响。 + - **`DemangledSymbol` 的单元素数组保留**：审查建议改成内联 `Symbol` 的双 case 枚举以省掉 + 这次分配，实测会把每个值从 32 B 撑到 48 B——而经共享表下发的值有数十万份，得不偿失。 + 结论连同 `compactValueLayouts` 的约束写进了该初始化器的文档注释。 +- **验收**：`SymbolTestsCore` 快照 60/60 逐字节一致；全量单元测试 1273 tests / 244 suites + 全绿；对冻结基线 `main-27726bc` 跑三源（File / DyldCache / Image）整文件快照对比。 + RuntimeViewer 实测同负载下 `Node` 实例 110 万 → 18.4 万、进程内存 842 MB → 434 MB。 +- **文档**：[NodeStoreMigrationPlan.md](NodeStoreMigrationPlan.md)、 + [DeclarationModelMemoryFootprint.md](DeclarationModelMemoryFootprint.md)、TaskReports + [2026-07-25-node-store-override-regression-and-baselines.md](TaskReports/2026-07-25-node-store-override-regression-and-baselines.md)、 + [2026-07-25-dyld-cache-image-selection-and-rv-index-lifecycle.md](TaskReports/2026-07-25-dyld-cache-image-selection-and-rv-index-lifecycle.md)、 + [2026-07-26-node-store-review-fixes.md](TaskReports/2026-07-26-node-store-review-fixes.md)。 + +--- + +## 20. 引用存储（weak/unowned）对 existential 的宽度修复 - **时间**：2026-07-26（发布于 `0.14.0`） - **动机**：用户实报 `SwiftUI.StyledTextResponder` 的字段偏移与反汇编不符。追查确认真值 diff --git a/Documentations/Internal/TaskReports/2026-07-26-node-store-review-fixes.md b/Documentations/Internal/TaskReports/2026-07-26-node-store-review-fixes.md new file mode 100644 index 00000000..5bf598ad --- /dev/null +++ b/Documentations/Internal/TaskReports/2026-07-26-node-store-review-fixes.md @@ -0,0 +1,93 @@ +# NodeStore 迁移分支的代码审查修复 + +- **日期**: 2026-07-26 +- **分支**: `feature/node-store-migration` +- **触发**: 对分支全量 diff 跑代码审查，返回 12 条发现；本次处理其中 9 条，驳回 1 条， + 显式不改 1 条，另 1 条为文档缺口。 + +## 一、问题 + +审查最有价值的结论是：**多条发现同源**。`SymbolIndexStore.demangledNodeReference` +命中冻结 arena 的条件里多了一个 offset 判等，导致整条 dyld shared cache 路径退化到 +per-symbol mini store；而 mini store 增殖又让下游一批用裸 `NodeReference` 当键的字典/集合 +静默失效。修一个条件能同时退掉四条症状。 + +## 二、调研 + +逐条回读源码核实，不直接采信 agent 结论： + +1. **offset 判等**（`SymbolIndexStore.swift:743`）。建表时 dyld cache 的行存的是 + `rawOffset - sharedRegionStart`（306 行），而 `symbols(for:in:)`（733 行）是拿**查询时 + 传入的 offset** 重建 `Symbol` 的。两边 offset 天然不同，判等必然失败。更关键的是这个 + 条件**在语义上就是多余的**：demangle 结果只是名字的函数，而 `tableRowByName` 一名一行、 + 重名后写覆盖，所以 offset 比较只能否决合法命中，永远无法用来消歧。 +2. **跨 store 的裸键**。`DefinitionBuilder` 里 `methodDescriptorLookup` / `vtableOffsetLookup` + 已经是 `StructuralNodeReferenceKey`，紧挨着的 `accessorsByNode` / + `canonicalIndexByFunctionNode` / `canonicalIndexByAllocatorNode` 却还是裸的；五处 + `visitedNodes` 也是 `OrderedSet`。这些集合的输入确实混了 store—— + `ExtensionDefinition.swift:115` 就有一条走 `MetadataReader.demangleSymbolReference` + 的分支。规则本身是本分支自己写进 AGENTS.md 的。 +3. **dyld cache 选图排序**（`DyldCache+.swift:76`）。上一个 commit 加的排序是**逐 cache 文件 + 分别排序、第一个有任何匹配的 cache 直接返回**，跨 cache 不生效——axbundle 在先扫的 cache + 里、framework 在 subcache 里时，依然选中 axbundle，即该 commit 本想修的 bug。 +4. **重复入桶**（`SymbolIndexStore.swift:323`）。exported symbol 分支在 raw 与 canonical + offset 相等时把同一行 append 两次；上面的 nlist 分支有 `hasAdjustedOffset` 守卫，这个 + 分支没有。 +5. **opaque 描述符查找**（728 行）从 O(1) 字典查退化成全量线性扫 + 逐项结构遍历。 +6. **`printSemantic` 重载**：`Node` 的具体重载没有 `StackSafeExecutor` 包裹，而泛型版有； + 具体重载优先级更高，所以 `Node` 调用方实际走的是没有栈保护那条。查上游确认 + `NodePrinter` 本身就是 `DemanglingPrinter` 的薄包装，两者输出等价。 + +## 三、最终方案 + +- 去掉 offset 判等，改为纯按名字命中；late cache 的键从 `Symbol` 改为 `String` + （同理：树只取决于名字），并把「查 + 建 + 存」收进同一个临界区。 +- `StructuralNodeReferenceKey` 从 `SwiftDeclaration` **下沉到 `MachOSymbols`**——它要服务的 + mini store 就产生在这一层，而 `SwiftDeclaration` 本来就 import 了 `MachOSymbols`。 + 下沉后 `MachOSymbols` 内部也能用它。 +- 全部跨 store 集合改用该键：`DefinitionBuilder` 三处 + 五处 `visitedNodes`。 +- `machOFile(by:)` 改为跨全部 cache 文件累积排名，只在拿到最高排名时早退。 +- 入桶改走一个 `registerRow` 局部函数：canonical 与 raw 不同才写第二个键。 + **同一 offset 对应多个符号是正常情形**（名字不同），桶仍是列表，只是同一行不重复入。 +- opaque 查找按成员 identifier 分桶（`DemanglingNode.identifier` 对 `Node` 和 + `NodeReference` 是同一份实现，结构相等必然同桶），桶内再做结构比较。 +- 删掉 `Node.printSemantic` 具体重载，让 `Node` 走泛型版的栈保护路径。 +- `SharedCacheTests` 的协调线程改用**限时**等待，且超时后照样 signal，避免失败时把 8 个 + build 永久钉在 `resolve` 里、把那些 key 永久留在 in-flight 态。 + +## 四、实际执行中的偏差 + +1. **驳回一条**：审查建议把 `DemangledSymbol` 的单元素数组换成内联 `Symbol` 的双 case 枚举 + 以省掉一次分配，并断言「仍在 32 字节以内」。实测**不成立**——`Symbol` 自身就是 32 字节， + 换枚举后整个值涨到 48 字节，`compactValueLayouts` 直接失败。而经共享表下发的 + `DemangledSymbol` 有数十万份，为省下罕见路径上的一次分配把常见路径每个值撑大 16 字节 + 是净亏。已回滚，并把这个取舍写进该初始化器的文档注释，防止下次再被提。 +2. **显式不改一条**：`ClassDumper.distributedFunctionNodes` 每个 actor 类算两遍、且每遍要 + materialize 所有 distributed thunk 符号。想清掉需要给一个 `struct`（且要 `Sendable`） + 加可变引用型缓存；而这条路径只在使用 distributed actor 的二进制里执行，本项目日常面对的 + SwiftUI/SwiftData 里是零。另外查询侧（221 行）拿的是 `Node`，集合若改成结构键反而要为 + 每个方法 intern 一个 mini store，更差。判断为不值得，保留原样。 +3. **SIGSEGV 虚惊**：改完 `Storage` 字段后跑全量测试直接 signal 11。这是 AGENTS.md 已记录的 + 已知现象——`MachOSymbols` 的布局变更 SwiftPM 增量构建传播不到位。`swift package clean` + 后全绿。 + +## 五、验证 + +- `swift build --build-tests`：0 error。 +- `swift test --skip IntegrationTests`：**1273 tests / 244 suites 全部通过**（clean 重建后）。 +- 对冻结基线 `main-27726bc` 的三源整文件快照对比**全部逐字节一致**： + File 38/38、DyldCache 18/18、Image 6/6（共 62 份）。 +- **补验 `-n`（按名称选图）路径**：快照 harness 一律用 `-p` 安装路径，而 `-p` 恒为最高排名， + 覆盖不到本次改的跨 cache 累积逻辑。对 iOS 27.0 beta 3 模拟器 cache（即同时存在 + `SwiftUI.framework/SwiftUI` 与 `SwiftUI.axbundle/SwiftUI` 的那个）另跑 + `dump --dyld-shared-cache … -n {SwiftUI, SwiftUICore, SwiftData}`，三者与对应的 `-p` + 输出**逐字节一致**（9,131,212 / 8,191,268 / 271,114 字节）。 + +## 六、文档同步 + +- `AGENTS.md`：更新 `demangledNodeReference` 的匹配语义、late store 的加锁语义、 + opaque 查找的分桶，以及 `StructuralNodeReferenceKey` 的新位置与完整适用清单 + （含四种「静默失效」的具体表现）。 +- `Documentations/Internal/ProjectEvolutionLog.md`：补上整个 NodeStore 工作弧的第 19 节 + （此前该弧缺账本条目）。 +- `Documentations/Internal/NodeStoreMigrationPlan.md`：追加本批次的实施记录。 From 6f53beb57c4e6bac2fbf7c2f5f27495ba8035938 Mon Sep 17 00:00:00 2001 From: Mx-Iris Date: Sun, 26 Jul 2026 13:23:47 +0800 Subject: [PATCH 17/77] docs: record the re-cut whole-file baseline for the reference-storage fix MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The 62-snapshot baseline is re-cut as main-7410710, superseding main-27726bc. Six of the 62 differ, all expected corrections confined to the layout-comment variants, with the non-comment content hashing identically in every one. The DyldCache and Image sets are byte-identical — and Image not moving is itself evidence for the fix, since that path reads field offsets straight from runtime metadata, so the static side converged onto ground truth rather than away from it. --- .../2026-07-26-reference-storage-existential-width.md | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/Documentations/Internal/TaskReports/2026-07-26-reference-storage-existential-width.md b/Documentations/Internal/TaskReports/2026-07-26-reference-storage-existential-width.md index be746a0d..57d80587 100644 --- a/Documentations/Internal/TaskReports/2026-07-26-reference-storage-existential-width.md +++ b/Documentations/Internal/TaskReports/2026-07-26-reference-storage-existential-width.md @@ -106,6 +106,14 @@ bug 长期隐身的典型原因。 - baseline 漂移经逐行审查**只含 offset/address/pointer 字段**（fixture 二进制增大导致的统一 平移 +6336），无任何语义字段变化；两份快照的 diff 是**纯新增**（238 行插入、0 删除）， 既有输出一字未改——说明本修复没有改变任何既有 fixture 类型的渲染结果。 +- **整文件基线已重录**为 `MachOSwiftSection-Baselines/main-7410710/`（62 份，取代 + `main-27726bc/`，后续迭代以新版为准）。相对旧基线差异 **6/62**，全部是本修复的预期修正： + 只出现在带布局注释的 `-full` 变体上，且 6 份差异文件的**非注释内容哈希逐份相同**（声明结构 + 零变化）。`DyldCache` 18/18 与 `Image` 6/6 逐字节一致——`Image` 走 MachOImage 路径、字段 + 偏移直读运行时 metadata，**它一字未变正是修复的旁证**（静态侧向 ground truth 收敛）。 + 旧基线抓到的第二个真实案例：`SwiftData.WeakAnyPersistentObject.boxed` + （`weak var boxed: (any PersistentModel)?`）从 8 字节修正为 16，其后的 + `persistentIdentifier` 从 `0x8` 修正到 `0x10`。 ## 与计划的偏差 From a797040c980cbfe0935a95f385187ba85721a357 Mon Sep 17 00:00:00 2001 From: Mx-Iris Date: Mon, 27 Jul 2026 19:13:31 +0800 Subject: [PATCH 18/77] fix(SwiftIndexing): make removeSubIndexer's lookup and removal atomic MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `removeSubIndexer(_:)` read the position through the `@Mutex` property's getter and then removed it through `removeSubIndexer(at:)`, taking the lock twice. `subIndexers` is mutex-guarded precisely because concurrent access is expected, so a removal landing between the two could either detach the wrong sub-indexer — silently keeping the intended one's `NodeStore` / `SymbolIndexStore` entry alive, which defeats the point of the API — or trap on an out-of-range index. Both steps now share one critical section on the underlying storage, and the `allStorageCache` invalidation runs only when something was actually removed. --- .../SwiftIndexing/SwiftDeclarationIndexer.swift | 15 +++++++++++++-- 1 file changed, 13 insertions(+), 2 deletions(-) diff --git a/Sources/SwiftIndexing/SwiftDeclarationIndexer.swift b/Sources/SwiftIndexing/SwiftDeclarationIndexer.swift index a94b7e2e..35dfa0c8 100644 --- a/Sources/SwiftIndexing/SwiftDeclarationIndexer.swift +++ b/Sources/SwiftIndexing/SwiftDeclarationIndexer.swift @@ -187,9 +187,20 @@ public final class SwiftDeclarationIndexer) { - guard let index = subIndexers.firstIndex(where: { $0 === subIndexer }) else { return } - removeSubIndexer(at: index) + let didRemoveSubIndexer = _subIndexers.withLock { registeredSubIndexers in + guard let index = registeredSubIndexers.firstIndex(where: { $0 === subIndexer }) else { return false } + registeredSubIndexers.remove(at: index) + return true + } + guard didRemoveSubIndexer else { return } + allStorageCache = AllStorageCache() } public func prepare() async throws { From a6aa557af7f93a87177b455eaeb2c6d48a99f57e Mon Sep 17 00:00:00 2001 From: Mx-Iris Date: Tue, 28 Jul 2026 19:01:10 +0800 Subject: [PATCH 19/77] fix: address verified findings from the node-store review MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Each fix was checked against the dependency the package actually builds — swift-demangling's perf/stack-safe-executor-reuse, reached through the .claude/worktrees/swift-demangling symlink — rather than its main branch, which carries neither the large-stack thread pool nor the budgeted execution entry points. - printSemantic goes through the printer engine's own budgeted entry, DemanglingPrinter.print(_:options:), instead of wrapping printRoot in StackSafeExecutor.execute. The guard is unchanged; what goes away is the unconditional worker hand-off plus semaphore block that `execute` performs on every 512KB-stack thread, once per printed declaration. - registerRow deduplicates against the target bucket. The old guard only covered raw == canonical offsets, so two symbol-table entries sharing a name and an address — aliases, weak definitions — folded onto one row and then listed it twice, making every `for symbol in symbols` loop visit that symbol twice. - dyld shared-cache image selection, three defects in one function. The Mac Catalyst copy under /System/iOSSupport is framework-shaped and shares its leaf name (74 frameworks collide this way on macOS 26), so it tied with the native framework at bestMatchRank and the accumulator stopped at whichever cache file was enumerated first; it now ranks one step below, leaving bestMatchRank unique so the early exit stays sound. The sub-cache array was read from `self`, which is empty on a sub-cache header, so opening a sub-cache directly skipped every sibling. And `mainCache` returns `self` for a main cache, which had the main cache enumerated twice. Verification: build passes before and after; `swift test --skip IntegrationTests` reports 1275 tests / 146 issues on both sides with an identical failing-test set, so the pre-existing baseline is unmoved. `swift-section dump --uses-system-dyld-shared-cache -n SwiftUI` is byte-identical across the change — this host's enumeration order already happened to favour the native framework, which is precisely the luck the ranking should not depend on. --- ...026-07-28-review-verification-and-fixes.md | 83 +++++++++++++++++++ Sources/MachOSymbols/SymbolIndexStore.swift | 27 ++++-- .../Extensions/Node+.swift | 21 +++-- .../DyldCacheImageSearchTests.swift | 41 +++++++++ 4 files changed, 160 insertions(+), 12 deletions(-) create mode 100644 Documentations/Internal/TaskReports/2026-07-28-review-verification-and-fixes.md diff --git a/Documentations/Internal/TaskReports/2026-07-28-review-verification-and-fixes.md b/Documentations/Internal/TaskReports/2026-07-28-review-verification-and-fixes.md new file mode 100644 index 00000000..786d0885 --- /dev/null +++ b/Documentations/Internal/TaskReports/2026-07-28-review-verification-and-fixes.md @@ -0,0 +1,83 @@ +# 2026-07-28 代码审查结论复核与修复 + +## 问题 + +对 PR #97（`feature/node-store-migration`）跑了一轮多角度代码审查，产出 15 条"已验证"结论。其中相当一部分围绕栈安全展开，指控本分支把上游的栈保护弄丢了、把非阻塞调用换成了阻塞调用。这些结论的严重性排序直接决定要改什么，因此在动手前先逐条复核证据。 + +## 调研 + +### 关键发现：审查基于错误的上游分支 + +审查假定的上游是 `swift-demangling` 的 `main`。实际链接的是： + +``` +node-store-migration/.claude/worktrees/swift-demangling + -> swift-demangling/.claude/worktrees/symbol-store # perf/stack-safe-executor-reuse (c928554) +``` + +`perf/stack-safe-executor-reuse` 相对 `main` 多了三个提交，恰好推翻了审查的成本模型： + +- `ea4101b` 引入 `LargeStackThreadPool`：大栈 worker 长期复用、空闲 30 秒退休。审查反复出现的"每次调用 spawn 一条 8 MB 栈 Thread"从此不成立。 +- `ea4101b` / `ac30584` 引入 `StackSafeExecutor.executeWithinStackBudget`：在小栈线程上**内联**跑递归并对照栈地板，只有真正触底才回退到 worker。 +- `c928554` 把栈安全下沉进打印器引擎（`DemanglingPrinter.print(_:options:)` 静态入口），提交信息里**点名** MachOSwiftSection 的 `printSemantic` 是当初漏掉保护的调用点。 + +### 与 `feature/stack-budget-guard` 的关系 + +用户指向的 `feature/stack-budget-guard`（6fa6d95）与 `perf/stack-safe-executor-reuse` **互不为祖先**，共同基点是 `ea2ec28`。两者是同一问题的两套设计： + +| | `perf/stack-safe-executor-reuse` | `feature/stack-budget-guard` | +| --- | --- | --- | +| 线程池 | 有 | 有（且能识别自身 worker） | +| `executeWithinStackBudget` | 有 | **无** | +| `DemanglingPrinter.print` 静态入口 | 有 | **无** | +| `StackBudget`（按剩余栈字节限深，替换 MaxDepth 常量） | 无 | 有 | +| `materializeNode` 迭代化 | 无（递归） | **有**（显式栈） | + +`git merge-tree` 显示两者在 `StackSafeExecutor.swift`、`NodePrinter.swift`、`Remangler.swift` 三个文件上正面冲突。因此"把依赖切到 `feature/stack-budget-guard`"不是换个符号链接的事，需要上游先合并两条线。 + +### 逐条复核结论 + +| 审查结论 | 复核结果 | +| --- | --- | +| `await node.print` → `node.print` 是退化 | **伪阳性，方向相反**。同步 `print` 现在走 `executeWithinStackBudget`，典型输入内联执行；异步 `print` 在小栈上无条件走线程池 + continuation。去掉 `await` 减少了一整轮派发 | +| "每次 spawn 一条 8 MB 线程" | **错**。线程池复用，真实代价是提交 + 信号量阻塞 | +| `printSemantic` 包 `StackSafeExecutor.execute` | **成立但理由要改**。加回保护是对的（上游提交信息证实原先确实没有保护），错在用了钝的 `execute` 而非引擎自带的预算入口 | +| `lateDemangledNode` 持锁 demangle | **结构成立、代价高估**。`os_unfair_lock` 期间确有 `semaphore.wait()`，但线程复用后开销小得多，且注释写明这是刻意的原子性权衡（防止同一名字分裂到两个 mini store）。降为低优先级 | +| impl-offset 回退表只填 `methodDescriptors` | **成立但是旧账**。逐行比对 `main`，完全一致，非本 PR 引入 | +| 公开查询 API 字典键语义从结构相等翻成身份相等 | **成立但无现实触发者**。扫过 RuntimeViewer 的 `main` 与 `feature/node-store-adoption`，两条分支都没有调用这两个 API | +| `intern` 递归脱离栈保护 | **成立**。`main` 的 `demangleAsNode` 把 `NodeCache.shared.intern` 包在 `execute` 内，`demangleAsNodeTransient` + `builder.intern` 则在调用者栈上裸递归。注意上游 `NodeStoreBuilder.demangle` 自身就是这个形状。两条分支的 `StackBudget` 都没覆盖它 | +| `materialize` 无栈保护 | **成立，且 `feature/stack-budget-guard` 已修**（改成显式栈遍历） | +| Catalyst 平局 | **成立，已实测**。`/System/iOSSupport/System/Library/Frameworks/SwiftUI.framework` 存在，两棵框架树下 74 个同名框架 | + +## 最终方案 + +只改三处，其余按上表分类处理（伪阳性丢弃、旧账另计、`intern` 栈保护交上游）。 + +1. **`printSemantic` 换用引擎预算入口** —— `DemanglingPrinter.print(self, options:)` 取代 `StackSafeExecutor.execute { printer.printRoot(self) }`。保护不减，但小栈线程上不再每次派发加阻塞。 +2. **`registerRow` 按桶去重** —— 原守卫只挡 `canonicalOffset == rawOffset`，挡不住"两个同名符号落在同一地址折叠到同一行"。改成检查目标桶是否已含该行。 +3. **dyld 缓存镜像选择** —— 三件事一起改： + - `matchRank` 拆分 0 级：`/System/iOSSupport` 下的 Catalyst 变体降到 1 级，dylib 与其他各自后移一位。只有原生框架能拿 `bestMatchRank`，而拿到该级才允许提前退出。 + - 子缓存列表改读 `mainCache.subCaches`（子缓存头的该数组恒为空，直接打开子缓存时兄弟一个都扫不到）。 + - 用 `scannedCacheURLs` 去重：`mainCache` 在自身即主缓存时返回 `self`，原先会把主缓存整个扫两遍。 + +## 实际执行 + +- `Sources/SwiftDeclarationRendering/Extensions/Node+.swift` —— 换入口，文档注释改写为解释"为什么不是 `StackSafeExecutor.execute`"。 +- `Sources/MachOSymbols/SymbolIndexStore.swift` —— 新增 `appendRowIfAbsent(_:atOffset:)`，`registerRow` 改为两次调用它；注释补齐重复的两个独立成因。 +- `Sources/MachOExtensions/DyldCache+.swift` —— 新增 `catalystSupportRootDirectoryName`（声明为 `Substring`，与 split 出来的路径分量同类型，避免逐次桥接），`matchRank` 拆级，`machOFile(by:)` 加 `scanReachedBestMatch(in:)` 局部函数统一去重并改读 `mainCache.subCaches`。 +- `Tests/MachOCachesTests/DyldCacheImageSearchTests.swift` —— 新增三个用例：Catalyst 变体匹配但拿不到最佳级、它仍优于 dylib 与 bundle、它不会因为支持根而不再匹配。 + +## 验证 + +- `swift build`：改前基线通过（71.7s），改后通过。 +- `swift test --skip IntegrationTests`：改前 1275 测试 / 146 issue，改后 1275 测试 / 146 issue，**失败测试名集合逐条一致（19 个）**。这批失败是分支既有状态，与本次改动无关。 +- `swift test --filter DyldCacheImageSearchTests`：9 个用例全过（含 3 个新增）。 +- `swift test --filter 'DyldCacheImageSearchTests|MachOSymbolsTests'`：23 个用例 / 3 个 Suite 全过。 +- 端到端：`swift-section dump --uses-system-dyld-shared-cache -n SwiftUI` 改前改后输出逐字节一致（109387 行）。说明本机的枚举顺序原本就恰好偏向原生框架——这正是问题所在，修复把结果从"碰巧对"变成"构造上对"。 + +## 偏差说明 + +- **依赖分支未切换。** 两条上游分支互相冲突，切换会丢掉 `executeWithinStackBudget` 与静态打印入口（也就是本次第 1 项修复所依赖的东西）。需要上游先合并，本仓库无法单方面决定。 +- **`intern` 递归的栈保护未处理**，按分工交上游。 +- **`registerRow` 去重与子缓存遍历没有单元测试**：前者是 `buildStorageImpl` 内的局部函数，要构造带同名同址别名符号的 fixture；后者需要一份真实的分片共享缓存。两者都不是纯路径运算，无法像 `matchRank` 那样直接钉住。 +- **演进日志未更新。** 分支落后 `main` 五个提交，`main` 已发布 0.14.0 并新增 `## 20.`，与本分支的 `## 20.` 撞号；此时追加小节只会加深冲突。应在 rebase 并重编号之后一并处理。 diff --git a/Sources/MachOSymbols/SymbolIndexStore.swift b/Sources/MachOSymbols/SymbolIndexStore.swift index 2d9b11c4..4171c88a 100644 --- a/Sources/MachOSymbols/SymbolIndexStore.swift +++ b/Sources/MachOSymbols/SymbolIndexStore.swift @@ -350,13 +350,30 @@ public final class SymbolIndexStore: SharedCache, @unc // One offset legitimately maps to several rows — distinct symbol names // can share an address — so the bucket stays a list. The *same* row // must not be listed twice though, or every `for symbol in symbols` - // loop visits it twice. Raw and canonical offsets coincide whenever - // there is nothing to adjust (a `MachOImage`, or a file at offset 0), - // which is exactly when the second append would be a duplicate. + // loop visits it twice. + // + // A row repeats for two independent reasons, so the bucket itself has + // to be consulted rather than just the two offsets: + // + // - Raw and canonical offsets coincide whenever there is nothing to + // adjust (a `MachOImage`, or a file at offset 0), which makes the + // second append a duplicate of the first. + // - Two symbol-table entries carrying the *same name* at the same + // address — aliases and weak definitions do occur in a symtab — + // fold onto one canonical row, so the second entry re-registers a + // row the first already put in that bucket. + // + // Buckets hold one or two rows in practice, so scanning one is + // cheaper than the per-offset set it would take to avoid the scan. + func appendRowIfAbsent(_ row: UInt32, atOffset offset: Int) { + guard symbolRowsByOffset[offset]?.contains(row) != true else { return } + symbolRowsByOffset[offset, default: []].append(row) + } + func registerRow(_ row: UInt32, rawOffset: Int, canonicalOffset: Int) { - symbolRowsByOffset[rawOffset, default: []].append(row) + appendRowIfAbsent(row, atOffset: rawOffset) if canonicalOffset != rawOffset { - symbolRowsByOffset[canonicalOffset, default: []].append(row) + appendRowIfAbsent(row, atOffset: canonicalOffset) } } diff --git a/Sources/SwiftDeclarationRendering/Extensions/Node+.swift b/Sources/SwiftDeclarationRendering/Extensions/Node+.swift index 0c6d5edf..a41918f8 100644 --- a/Sources/SwiftDeclarationRendering/Extensions/Node+.swift +++ b/Sources/SwiftDeclarationRendering/Extensions/Node+.swift @@ -60,15 +60,22 @@ extension DemanglingNode { /// overload used to sit alongside it and — being the better overload for /// every `Node` caller — shadowed this one while running /// `NodePrinter` (itself a thin wrapper over the same - /// `DemanglingPrinter` engine) *outside* `StackSafeExecutor`. + /// `DemanglingPrinter` engine) with no stack guard at all. /// A deeply nested generic symbol printed through a `Node` could therefore - /// overflow the stack where the identical `NodeReference` call would not, - /// and every other print entry point in `Demangling` is stack-guarded. + /// overflow the stack where the identical `NodeReference` call would not. + /// + /// The guard is the engine's own `print(_:options:)` rather than a + /// `StackSafeExecutor.execute` wrapper around `printRoot`. `execute` has to + /// assume the worst about every input, so on a 512KB stack — which is what + /// every Swift Concurrency cooperative worker and every libdispatch worker + /// gets — it hands *every* call to a large-stack worker and blocks the + /// caller on a semaphore until that worker returns. `print(_:options:)` + /// runs the recursion inline against a stack floor and pays for a worker + /// only for a tree that actually reaches it. Every `printSemantic` call + /// site sits on a printing hot path, so the difference is one + /// dispatch-and-block per printed declaration. public func printSemantic(using options: DemangleOptions = .default) -> SemanticString { - StackSafeExecutor.execute { - var printer = DemanglingPrinter(options: options) - return printer.printRoot(self) - } + DemanglingPrinter.print(self, options: options) } } diff --git a/Tests/MachOCachesTests/DyldCacheImageSearchTests.swift b/Tests/MachOCachesTests/DyldCacheImageSearchTests.swift index 2f203ba8..6f136127 100644 --- a/Tests/MachOCachesTests/DyldCacheImageSearchTests.swift +++ b/Tests/MachOCachesTests/DyldCacheImageSearchTests.swift @@ -18,6 +18,7 @@ struct DyldCacheImageSearchTests { private let iOSFrameworkPath = "/System/Library/Frameworks/SwiftUI.framework/SwiftUI" private let macOSFrameworkPath = "/System/Library/Frameworks/SwiftUI.framework/Versions/A/SwiftUI" private let accessibilityBundlePath = "/System/Library/AccessibilityBundles/SwiftUI.axbundle/SwiftUI" + private let catalystFrameworkPath = "/System/iOSSupport/System/Library/Frameworks/SwiftUI.framework/SwiftUI" private let bestRank = DyldCacheImageSearchMode.bestMatchRank @@ -48,6 +49,46 @@ struct DyldCacheImageSearchTests { #expect(dylibRank < bundleRank) } + /// A macOS cache also carries the Mac Catalyst build of the same framework + /// under `/System/iOSSupport` — framework-shaped, same leaf name, a real + /// dylib. 74 frameworks collide this way on macOS 26 (SwiftUI, ARKit, + /// AVKit, GameKit, HealthKit, …). + /// + /// Both used to score `bestMatchRank`, and `accumulateBestMatch` returns at + /// the *first* image reaching that rank, so `-n SwiftUI` resolved to + /// whichever the cache happened to enumerate first — the same + /// order-dependence the ranking was introduced to remove, just moved from + /// bundle-versus-framework to Catalyst-versus-native. Only the native + /// framework may reach `bestMatchRank`. + @Test func catalystVariantMatchesButNeverScoresBestRank() throws { + let mode = DyldCacheImageSearchMode.name("SwiftUI") + let catalystRank = try #require(mode.matchRank(forImagePath: catalystFrameworkPath)) + #expect(catalystRank > bestRank) + #expect(mode.matchRank(forImagePath: iOSFrameworkPath) == bestRank) + #expect(mode.matchRank(forImagePath: macOSFrameworkPath) == bestRank) + } + + /// The Catalyst variant is a real framework binary, so it still outranks a + /// plain dylib and a bundle wearing the same leaf name — it loses only to + /// the native framework. + @Test func catalystVariantOutranksDylibAndBundle() throws { + let mode = DyldCacheImageSearchMode.name("SwiftUI") + let catalystRank = try #require(mode.matchRank(forImagePath: catalystFrameworkPath)) + let bundleRank = try #require(mode.matchRank(forImagePath: accessibilityBundlePath)) + let dylibRank = try #require( + DyldCacheImageSearchMode.name("libswiftCore").matchRank(forImagePath: "/usr/lib/swift/libswiftCore.dylib") + ) + #expect(catalystRank < dylibRank) + #expect(dylibRank < bundleRank) + } + + /// `/System/iOSSupport` demotes only a framework that has a native twin to + /// lose to; the support root must not make a path stop matching. + @Test func catalystVariantIsStillAMatch() { + let mode = DyldCacheImageSearchMode.name("SwiftUI") + #expect(mode.matchRank(forImagePath: catalystFrameworkPath) != nil) + } + @Test func nonMatchingLeafNameIsNotAMatch() { let mode = DyldCacheImageSearchMode.name("SwiftUI") #expect(mode.matchRank(forImagePath: "/System/Library/Frameworks/SwiftUICore.framework/SwiftUICore") == nil) From 5a69b882aaf5c5c52987189b25e9bb8e3eab7920 Mon Sep 17 00:00:00 2001 From: Mx-Iris Date: Tue, 28 Jul 2026 19:01:22 +0800 Subject: [PATCH 20/77] docs: record the node-store migration's open issues MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit A second review pass over the branch surfaced twelve confirmed findings. None are fixed here — NodeStoreMigrationOpenIssues.md records each one with its cause, blast radius, and where the fix belongs, because four of them are upstream in swift-demangling and cannot be worked around on this side. Two are gaps in the fixes from the previous commit and rank first: the /System/iOSSupport demotion sits inside the `.framework` branch, so a Catalyst plain dylib still ties with its native namesake (libGLVMPlugin is a live collision on this host), and appendRowIfAbsent's linear bucket scan turns the collection pass quadratic if any offset ever collects many rows. The rest: two public query APIs whose dictionary keys flipped from structural to store-identity equality (silently returning nil to any external caller, with no compile error); structuralHash allocating a String per text node where textUTF8 is already available; the memberSymbols lookup degrading from a hash hit to a linear scan with a full structural walk per candidate; the build sweep going serial with an unconditional cross-thread round trip per symbol; ABIKey materializing a whole tree per key; four code-hygiene items; and the rebase work the branch needs before it can merge — it is five commits behind main, and both sides edited AGENTS.md and the evolution log. --- .../Internal/NodeStoreMigrationOpenIssues.md | 132 ++++++++++++++++++ Documentations/README.md | 1 + 2 files changed, 133 insertions(+) create mode 100644 Documentations/Internal/NodeStoreMigrationOpenIssues.md diff --git a/Documentations/Internal/NodeStoreMigrationOpenIssues.md b/Documentations/Internal/NodeStoreMigrationOpenIssues.md new file mode 100644 index 00000000..2c99a267 --- /dev/null +++ b/Documentations/Internal/NodeStoreMigrationOpenIssues.md @@ -0,0 +1,132 @@ +# NodeStore 迁移遗留问题清单 + +本文记录 `feature/node-store-migration` 分支上**已确认但尚未修复**的问题，供后续按优先级处理。每条注明成因、影响面、以及"该在哪里修"（有几条的正确修复位置在上游 `swift-demangling`，不在本仓库）。 + +产生方式：2026-07-28 对该分支做了两轮代码审查 + 一轮结论复核。第一轮的复核记录见 [TaskReports/2026-07-28-review-verification-and-fixes.md](TaskReports/2026-07-28-review-verification-and-fixes.md)，其中三条已在本分支修掉（`printSemantic` 换用引擎预算入口、`registerRow` 按桶去重、dyld 缓存镜像选择的 Catalyst 平局与子缓存遍历）。本文只列**仍然打开**的。 + +--- + +## 一、上一轮修复自身的缺口 + +这两条是 2026-07-28 那批修复引入或未覆盖的，优先级最高——它们让已经宣称修好的问题只修了一部分。 + +### 1. Catalyst 降级只覆盖了 framework 形态，plain dylib 仍然平局 + +`DyldCache+.swift` 的 `matchRank` 把 `/System/iOSSupport` 的判断写在了 `.framework` 分支**内部**。一个既不在 `.framework` 下、叶名又以 `.dylib` 结尾的镜像走不到那个判断，于是原生与 Catalyst 两份同名 dylib 双双落在 2 级，胜负重新取决于缓存文件的枚举顺序——正是这次排序机制要根除的那类不确定性。 + +本机实测存在的碰撞： + +``` +/System/Library/Frameworks/OpenGL.framework/Versions/A/Libraries/libGLVMPlugin.dylib +/System/iOSSupport/System/Library/Frameworks/OpenGLES.framework/Versions/A/Libraries/libGLVMPlugin.dylib +``` + +`-n libGLVMPlugin` 对两者都返回 2 级。另有 `/System/iOSSupport/usr/lib/swift/libswift{QuickLook,HomeKit,PencilKit}.dylib` 一组，目前**没有**原生同名物，属于"将来会踩"而非当下已坏。 + +**正确修法**：把支持根的判断提成一次性的**惩罚项**，在形态分类**之前**施加，而不是塞进某个分支里。例如先算形态基准分（framework 0 / dylib 1 / 其他 2），再对 `/System/iOSSupport` 下的结果统一加一档。这样任何形态的 Catalyst 变体都稳定劣于同形态的原生物。 + +**修复位置**：本仓库 `Sources/MachOExtensions/DyldCache+.swift`。 + +### 2. `appendRowIfAbsent` 是线性扫描，桶变大就退化成 O(n²) + +`registerRow` 的去重改成了 `symbolRowsByOffset[offset]?.contains(row)`，对桶做线性扫描。注释里断言"桶实际上只有一两行"，但没有任何东西保证这一点——不同符号名合法地共享同一地址（桶之所以是数组就是因为这个），而退化偏移（0，或 stripped / dyld 缓存镜像里被大量别名的地址）可以攒到上千行。那样每次登记都是 O(桶大小)，整趟采集变成 O(n²)。 + +**正确修法**：重复只来自两个已枚举的成因，不需要通用去重。可以只记住每个偏移最近一次追加的行，或者仅在 `canonicalOffset == rawOffset` 时配合一个按名字的已见集合来挡。 + +**修复位置**：本仓库 `Sources/MachOSymbols/SymbolIndexStore.swift`。 + +--- + +## 二、公开 API 语义问题 + +### 3. 两个公开查询 API 的字典键从结构相等翻成了身份相等 + +`memberSymbols(of:excluding:in:)` 与 `allOpaqueTypeDescriptorSymbols(in:)` 原本返回 `OrderedDictionary`。`Node` 的 `==` 是结构相等，所以外部调用方拿任意来源的节点做下标查询都能命中。现在键是 `NodeReference`，其 `==` 为 `store === store && index == index`。调用方用自己 demangle 出来的节点查询会**恒定返回 nil，且没有任何编译错误**。 + +仓库内部这两个 API 只被遍历、从不下标查询，所以测试全绿也发现不了。`StructuralNodeReferenceKey` 这套处理施加到了所有内部集合上，唯独漏了这两个**逃逸到外部**的面。 + +现状缓解：扫过 RuntimeViewer 的 `main` 与 `feature/node-store-adoption`，两条分支都没有调用这两个 API，所以目前没有现实触发者。 + +**正确修法**：要么改成 vend `StructuralNodeReferenceKey`（或干脆 `Node`）作键，要么不暴露裸字典、改提供一个查询方法。 + +**修复位置**：本仓库 `Sources/MachOSymbols/SymbolIndexStore.swift`。 + +--- + +## 三、性能问题 + +### 4. `structuralHash` 每个文本节点分配一个 `String` + +`StructuralNodeReferenceKey.hash` 委托给上游 `NodeReference.structuralHash`，后者对文本节点执行 `hasher.combine(text)`，而 `NodeReference.text` 会走 `NodeStore.text(offset:length:)` → `String(decoding:as:)`——每个标识符/模块节点每次哈希都堆分配一次。迁移前的 `Node` 键组合的是节点里**已经存在**的 `String`。 + +影响面不小：`TypeName` / `ProtocolName` / `ExtensionName` 是 `SwiftDeclarationIndexer` 里几乎每个索引的键，`DefinitionBuilder` 还会对**整棵符号根节点**每个成员构造两次 `StructuralNodeReferenceKey`。 + +**正确修法**：改哈希 `textUTF8`（`NodeReference` 上已有，返回 `ArraySlice`，零拷贝）而非 `text`。 + +**修复位置**：**上游 `swift-demangling`** 的 `Sources/Demangling/Store/NodeReference.swift`，不是本仓库。本仓库这一侧无法绕开。 + +### 5. `memberSymbols(of:for:node:)` 从 O(1) 退化成线性扫描 + 逐候选全树比对 + +迁移前是 `memberSymbolsByKind[$0]?[name]?[node]`，一次哈希查找。现在两个重载都走 `rowsByTypeNodeIndex.elements.first(where: { …structurallyEquals(node) })`——对桶里每个键做一次结构化树遍历直到命中。 + +`TypeDefinition.index` 会为 allocator、变量、静态变量、函数、静态函数、下标各调一次，所以每个被索引的类型付 6 × 桶大小次结构遍历。 + +**正确修法**：在 `Storage.init` 里一次性建一份 `[StructuralNodeReferenceKey: NodeStore.NodeIndex]` 旁路索引恢复 O(1)——这正是 `opaqueTypeDescriptorEntriesByMemberIdentifier` 已经用过的手法。 + +**修复位置**：本仓库 `Sources/MachOSymbols/SymbolIndexStore.swift`。 + +### 6. build sweep 由并行改为串行，且每个符号都无条件跨线程往返 + +原来是 `symbolArray.concurrentMap { try? demangleAsNode($0.name) }`，N 路并行。现在是单趟顺序循环，且每次 `demangleAsNodeTransient` 走的是 `StackSafeExecutor.execute`（不是打印/重编码路径用的 `executeWithinStackBudget`）。`buildStorage` 跑在 512 KB 栈的线程上，于是一个框架里几十万个符号，**每一个**都付一次线程池提交 + 信号量等待，而且没有并行来摊薄。 + +**正确修法**：`demangleAsNodeTransient` 应当像打印器那样改用带预算的入口。 + +**修复位置**：**上游 `swift-demangling`** 的 `DemangleInterface.swift`。本仓库改不动。 + +### 7. `ABIKey.make` 泛型化后每个 key 都要 materialize 整棵树 + +`ABIKey.make(for:)` / `makeUnwrappingType(for:)` 泛型化到 `DemanglingNode` 之后，走的是 `mangleAsString` 的 `DemanglingNode` 重载，其实现是 `mangleAsString(node.materializedNode)`。而 `TypeName.node` / `ProtocolName.node` / `FieldDefinition.typeNode` / `FunctionDefinition.node` 现在全是 `NodeReference`，于是构建 `ABISnapshot` 时每个类型、协议、扩展容器、成员、字段的 key 都会重建一整棵类树再丢掉。 + +**正确修法**：`mangleAsString` 增加一条 store 原生路径（上游），或 `ABIKey` 改成每个声明 materialize 一次而非每个 key 一次（本仓库）。 + +**修复位置**：上游或本仓库 `Sources/SwiftDiffing/ABIKey.swift`，取决于选哪条路。 + +--- + +## 四、代码卫生 + +### 8. `Symbol.isExternal` 在符号表里恒为 `false` + +采集局部符号的循环已经用 `where … && !symbol.nlist.isExternal` 过滤掉了外部符号，导出符号循环走默认参数 `isExternal: false`。所以 `symbolTable` 里没有任何一行能是 `true`，`if !symbol.isExternal` 的守卫永远成立、是死代码。而 `Symbol.swift` 的注释声称这个标志是被提取出来供后续查询的，与实际不符。该字段只有在非索引存储的 `Symbol.resolve` 路径（`asCurrentSymbol`）上才可能非 `false`。 + +过滤和守卫二者必有其一冗余，需要挑一个删掉并修正注释。 + +### 9. `lateDemangledNode(forName:)` 在持锁期间 demangle + +`demangleAsNodeTransient` 会走 `StackSafeExecutor.execute`，在 512 KB 栈线程上无条件提交线程池并 `semaphore.wait()`。于是一次 miss 会**在持有 per-image 互斥锁的情况下**跨线程等待不定时长，该镜像上所有查询晚绑定名字的线程都排在它后面；线程池饱和时持锁时间无上界。 + +注意这是**刻意的权衡**：代码注释写明查找与插入必须同处一个临界区，否则两个并发 miss 会各自冻结一份 mini store，把同一名字的引用分裂到不同 store 里。所以修的时候要保住这个保证。 + +**正确修法**：在锁外 demangle，锁内用 insert-if-absent（后写者放弃、返回胜出者），单 store 保证不变而临界区里不再阻塞。 + +### 10. `ProtocolConformanceDumper` 里一个分支还在 materialize + +同一个 `switch requirement` 块里，`case .element` 和 `Self.demangledSymbol(...)` 都已改走 `MetadataReader.demangleSymbolReference` 留在 store 上，唯独 `case .symbol` 仍调 `MetadataReader.demangleSymbol` 把整棵树 materialize 出来，只为交给 `demangleResolver.resolve(for:)`——而后者现在有 `some DemanglingNode` 重载，可以直接吃引用。既多余，又会让下一个维护者误以为这个不一致是有意的。 + +### 11. 两处 `throws` 是迁移残留 + +`ExtensionDefinition._symbol(for:typeName:visitedNodes:)` 与 `ProtocolDefinition` 里对应的那个，唯一的抛出调用已被换成不抛出的 `demangleSymbolReference`，函数体里不再有任何 `try`，但签名仍是 `throws`，调用点仍写 `try`。删掉 `throws` 之后，周围 `if let` 链里真正会抛的调用（`resilientWitness.implementationSymbols(in:)`、`Symbols.resolve`）才看得出来。 + +--- + +## 五、分支状态 + +### 12. 落后 `main` 五个提交，`AGENTS.md` 两侧都改过 + +`main` 已发布 `0.14.0`，并新增了注释模板的命令行接口（`--enum-layout-template` / `--enum-layout-case-template` / `--enum-layout-byte-template`）及其 `AGENTS.md` 章节。本分支的 `AGENTS.md` 还是 0.14.0 之前的正文，另外加了自己的 NodeStore 段落。直接合并会冲突，而**保留分支侧的粗暴解法会静默回退掉 `main` 的那份文档**。 + +同理，`ProjectEvolutionLog.md` 里本分支新增的 `## 19.` 把原「引用存储」小节顶成了 `## 20.`，与 `main` 的 `## 20.` 正面撞号；两条新小节都写"将入 0.14.0"，而 0.14.0 已经发布。另有一条指向 `TaskReports/2026-07-25-dyld-cache-image-selection-...` 的链接是死的（实际文件名无 `dyld-` 前缀）。 + +此外，`main` 的 `TransformerOptionGroup` 与本分支的 `DemangleResolver` / `printSemantic` / `FieldDefinition.typeNode` 改动之间的交互从未被跑过。 + +**处理顺序**：先 rebase 到 `main`，重编演进日志小节号、修死链、对齐 `AGENTS.md`，再谈合并。演进日志的小节应在 rebase 之后补，现在写只会加深冲突。 diff --git a/Documentations/README.md b/Documentations/README.md index d7a21b01..61a3d6ad 100644 --- a/Documentations/README.md +++ b/Documentations/README.md @@ -74,4 +74,5 @@ required by `Version.swift`'s bump contract). | [ClassMemberKeywordRecovery.md](Internal/ClassMemberKeywordRecovery.md) | `class` / `static` 成员关键字的还原：mangling 层面两者不可区分，判据是「类型级成员有 vtable method descriptor ⇒ 源码是 `class`」（`static` 隐式 final、不进 vtable）；模型侧 `isClassMember` 计算属性 + 三个 node printer 接线 + dump vtable 段落关键字，顺带消灭非法的 `override static` 输出；`final class func` 等四类 ABI 上与 `static` 完全一致，保守输出语义等价的 `static`。 | | [NodeStoreMigrationPlan.md](Internal/NodeStoreMigrationPlan.md) | `SymbolIndexStore` 迁移到 `NodeStore` arena 存储的分期计划与实施记录（Stage 0–5）：12B/节点扁平缓冲、cache-free build sweep、Symbol 表压缩、声明层换持 `NodeReference`、散点 transient demangling；含 Stage 5a 的 override/vtable 回归修复（裸 `[NodeReference: …]` 的 store-identity 陷阱 → `StructuralNodeReferenceKey`）与三源快照验收。 | | [DeclarationModelMemoryFootprint.md](Internal/DeclarationModelMemoryFootprint.md) | NodeStore 迁移后的声明模型内存足迹量测：`TypeDefinition` 1272 字节的逐属性构成（两份 `TypeContextWrapper` 占 74%）、`TypeContextWrapper` 按最大 case `Class` 定尺的原因、`parentContext` 是被当成永久字段的临时值、mini-store 增殖与 `MetadataReaderCache` 仍持 `Node` 树；含四项可回收估算（合计 ~8–10%）与「当前不建议实施、应先剖析其余 90%」的结论。 | +| [NodeStoreMigrationOpenIssues.md](Internal/NodeStoreMigrationOpenIssues.md) | `feature/node-store-migration` 上**已确认但未修复**的问题清单（2026-07-28 两轮审查 + 复核）：Catalyst 降级只覆盖 framework 形态导致 plain dylib 仍平局、`appendRowIfAbsent` 线性扫描、两个公开查询 API 的字典键从结构相等翻成 store 身份相等、`structuralHash` 每文本节点分配 `String`（修复位置在上游）、`memberSymbols` 退化为线性 + 全树比对、build sweep 串行且每符号跨线程往返（上游）、`ABIKey` 每 key materialize，以及四条代码卫生项与 rebase 前置事项。逐条注明成因、影响面与「该在哪里修」。 | | [TaskReports/](Internal/TaskReports/) | Dated per-task fix / investigation reports. | From c4dd3e61975b523c384b66450af1918f9e7b00a8 Mon Sep 17 00:00:00 2001 From: Mx-Iris Date: Wed, 29 Jul 2026 10:34:04 +0800 Subject: [PATCH 21/77] fix: close the two gaps the second review found in the previous fixes MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Both were cases of a fix that only covered part of its own problem. The /System/iOSSupport demotion sat inside the `.framework` branch, so it never reached a path that is neither framework-shaped nor rejected — two same-named plain dylibs stayed tied and the winner was again whichever cache file came first. libGLVMPlugin.dylib is a live collision on macOS 26: it ships natively under OpenGL.framework and as a Catalyst build under iOSSupport/…/OpenGLES.framework. matchRank now scores shape first (framework / dylib / other), spaces the shapes two ranks apart, and adds the support-root penalty to whichever shape came out. Spacing is what keeps the penalty a tiebreak *within* a shape rather than a reclassification — otherwise a native .axbundle, the metadata-less payload this ranking exists to demote, would outrank a Catalyst framework binary. bestMatchRank stays reachable only by a native canonical framework, so the accumulator's early exit is still sound. appendRowIfAbsent scanned the bucket on every registration, which is O(n²) once a degenerate offset — 0, or a heavily aliased address in a stripped or dyld-cache image — collects many rows. The scan is not needed: a freshly minted row index is symbolTable.count, strictly greater than every row issued so far, so no bucket can already hold it. canonicalRow now reports whether it created the row and registerRow only checks when it did not. What remains scannable is a repeated symbol name, which is rare; the large-bucket case that made this quadratic is exactly the one that now never scans at all. The exported-symbol loop is guarded on the name being absent, so it takes the new-row path unconditionally. Verification: 1280 tests / 146 issues with a failing-test set identical to the established baseline (the count rose from 1275 by this branch's five new cases); DyldCacheImageSearchTests is 11 for 11; `swift-section dump --uses-system-dyld-shared-cache -n SwiftUI` is byte-identical to the previously verified run. --- .../Internal/NodeStoreMigrationOpenIssues.md | 34 ++++++--- ...29-catalyst-rank-and-row-dedup-followup.md | 75 +++++++++++++++++++ Documentations/README.md | 2 +- Sources/MachOSymbols/SymbolIndexStore.swift | 61 +++++++++------ .../DyldCacheImageSearchTests.swift | 36 +++++++++ 5 files changed, 176 insertions(+), 32 deletions(-) create mode 100644 Documentations/Internal/TaskReports/2026-07-29-catalyst-rank-and-row-dedup-followup.md diff --git a/Documentations/Internal/NodeStoreMigrationOpenIssues.md b/Documentations/Internal/NodeStoreMigrationOpenIssues.md index 2c99a267..10b68cca 100644 --- a/Documentations/Internal/NodeStoreMigrationOpenIssues.md +++ b/Documentations/Internal/NodeStoreMigrationOpenIssues.md @@ -2,15 +2,17 @@ 本文记录 `feature/node-store-migration` 分支上**已确认但尚未修复**的问题，供后续按优先级处理。每条注明成因、影响面、以及"该在哪里修"（有几条的正确修复位置在上游 `swift-demangling`，不在本仓库）。 -产生方式：2026-07-28 对该分支做了两轮代码审查 + 一轮结论复核。第一轮的复核记录见 [TaskReports/2026-07-28-review-verification-and-fixes.md](TaskReports/2026-07-28-review-verification-and-fixes.md)，其中三条已在本分支修掉（`printSemantic` 换用引擎预算入口、`registerRow` 按桶去重、dyld 缓存镜像选择的 Catalyst 平局与子缓存遍历）。本文只列**仍然打开**的。 +产生方式：2026-07-28 对该分支做了两轮代码审查 + 一轮结论复核。第一轮的复核记录见 [TaskReports/2026-07-28-review-verification-and-fixes.md](TaskReports/2026-07-28-review-verification-and-fixes.md)，其中三条已修（`printSemantic` 换用引擎预算入口、`registerRow` 去重、dyld 缓存镜像选择的 Catalyst 平局与子缓存遍历）；第二轮又指出前述修复自身的两处缺口，已于 2026-07-29 补完，见 [TaskReports/2026-07-29-catalyst-rank-and-row-dedup-followup.md](TaskReports/2026-07-29-catalyst-rank-and-row-dedup-followup.md)。 + +第一节记录那两条已闭环的缺口（保留成因以备回溯），**第二节起才是仍然打开的**。 --- -## 一、上一轮修复自身的缺口 +## 一、上一轮修复自身的缺口（已于 2026-07-29 修复） -这两条是 2026-07-28 那批修复引入或未覆盖的，优先级最高——它们让已经宣称修好的问题只修了一部分。 +这两条是 2026-07-28 那批修复引入或未覆盖的，让已经宣称修好的问题只修了一部分，因此优先处理。两条都已修完并有测试，保留在此以记录成因。 -### 1. Catalyst 降级只覆盖了 framework 形态，plain dylib 仍然平局 +### 1. ~~Catalyst 降级只覆盖了 framework 形态，plain dylib 仍然平局~~ ✅ `DyldCache+.swift` 的 `matchRank` 把 `/System/iOSSupport` 的判断写在了 `.framework` 分支**内部**。一个既不在 `.framework` 下、叶名又以 `.dylib` 结尾的镜像走不到那个判断，于是原生与 Catalyst 两份同名 dylib 双双落在 2 级，胜负重新取决于缓存文件的枚举顺序——正是这次排序机制要根除的那类不确定性。 @@ -23,17 +25,31 @@ `-n libGLVMPlugin` 对两者都返回 2 级。另有 `/System/iOSSupport/usr/lib/swift/libswift{QuickLook,HomeKit,PencilKit}.dylib` 一组，目前**没有**原生同名物，属于"将来会踩"而非当下已坏。 -**正确修法**：把支持根的判断提成一次性的**惩罚项**，在形态分类**之前**施加，而不是塞进某个分支里。例如先算形态基准分（framework 0 / dylib 1 / 其他 2），再对 `/System/iOSSupport` 下的结果统一加一档。这样任何形态的 Catalyst 变体都稳定劣于同形态的原生物。 +**修复方式**：拆成两步打分。先按路径形态定基准分（framework 0 / dylib 1 / 其他 2），乘以 `rankStepsPerPathShape`（2）拉开间距；再对 `/System/iOSSupport` 下的**任何形态**统一加 `catalystSupportRootPenalty`（1）。得到的全序是： + +| 排名 | 含义 | +| --- | --- | +| 0 | 原生 canonical framework（唯一能拿 `bestMatchRank` 的，提前退出因此仍然成立） | +| 1 | Catalyst framework | +| 2 | 原生 plain dylib | +| 3 | Catalyst plain dylib | +| 4 | 原生 bundle / 其他 | +| 5 | Catalyst bundle / 其他 | + +留出间距是关键：惩罚项永远不会把某个形态顶到下一个形态的分位上，所以"降级只是同形态内的平局裁决、不是重新归类"。否则一个没有 Swift 元数据的原生 `.axbundle` 就能压过 Catalyst 的 framework 二进制——那正是整套排序最初要解决的问题。 -**修复位置**：本仓库 `Sources/MachOExtensions/DyldCache+.swift`。 +**新增测试**：`catalystPlainDylibLosesToItsNativeNamesake`、`supportRootPenaltyNeverCrossesAShapeBoundary`。 -### 2. `appendRowIfAbsent` 是线性扫描，桶变大就退化成 O(n²) +### 2. ~~`appendRowIfAbsent` 是线性扫描，桶变大就退化成 O(n²)~~ ✅ `registerRow` 的去重改成了 `symbolRowsByOffset[offset]?.contains(row)`，对桶做线性扫描。注释里断言"桶实际上只有一两行"，但没有任何东西保证这一点——不同符号名合法地共享同一地址（桶之所以是数组就是因为这个），而退化偏移（0，或 stripped / dyld 缓存镜像里被大量别名的地址）可以攒到上千行。那样每次登记都是 O(桶大小)，整趟采集变成 O(n²)。 -**正确修法**：重复只来自两个已枚举的成因，不需要通用去重。可以只记住每个偏移最近一次追加的行，或者仅在 `canonicalOffset == rawOffset` 时配合一个按名字的已见集合来挡。 +**修复方式**：两个重复成因各自用 O(1) 判据挡掉，完全不扫桶。 -**修复位置**：本仓库 `Sources/MachOSymbols/SymbolIndexStore.swift`。 +- 原始偏移与规范偏移相同 → 比较两个偏移即可（本来就有）。 +- 同名同址的两条符号表条目折叠到同一行 → **只有本来就存在的行才可能在桶里**。新行的索引取自 `symbolTable.count`，严格大于此前发出的所有行号，所以任何桶都不可能装着它。于是 `canonicalRow` 改为返回 `(row, isNewRow)`，`registerRow` 只在 `isNewRow == false` 时才检查。 + +真正会扫桶的只剩"同一个名字重复出现"这一种情况，而重名条目本身就罕见；上千个不同名字堆在同一偏移的退化场景——也就是原本会导致 O(n²) 的那个——现在一次都不扫。导出符号那趟循环有 `tableRowByName[...] == nil` 前置条件，必然是新行，因此彻底不进检查。 --- diff --git a/Documentations/Internal/TaskReports/2026-07-29-catalyst-rank-and-row-dedup-followup.md b/Documentations/Internal/TaskReports/2026-07-29-catalyst-rank-and-row-dedup-followup.md new file mode 100644 index 00000000..8e1bef57 --- /dev/null +++ b/Documentations/Internal/TaskReports/2026-07-29-catalyst-rank-and-row-dedup-followup.md @@ -0,0 +1,75 @@ +# 2026-07-29 补完前一轮修复自身的两处缺口 + +## 问题 + +2026-07-28 那批修复（见 [2026-07-28-review-verification-and-fixes.md](2026-07-28-review-verification-and-fixes.md)）落地后又跑了一轮代码审查，12 条结论里有**两条指向那批修复本身**——都属于"已经宣称修好，实际只修了一部分"，因此比其余 10 条更该先处理。 + +### 缺口一：Catalyst 降级只覆盖 framework 形态 + +`matchRank` 里 `/System/iOSSupport` 的判断被写进了 `.framework` 分支**内部**。既不在 `.framework` 下、叶名又以 `.dylib` 结尾的镜像根本走不到那句，于是原生与 Catalyst 两份同名 dylib 双双落在 2 级，胜负重新取决于枚举顺序。 + +本机实测碰撞（已用 `ls` 确认两个文件都存在）： + +``` +/System/Library/Frameworks/OpenGL.framework/Versions/A/Libraries/libGLVMPlugin.dylib +/System/iOSSupport/System/Library/Frameworks/OpenGLES.framework/Versions/A/Libraries/libGLVMPlugin.dylib +``` + +另有 `/System/iOSSupport/usr/lib/swift/libswift{QuickLook,HomeKit,PencilKit}.dylib`，逐个查过**没有**原生同名物，属于"将来会踩"。 + +### 缺口二：`appendRowIfAbsent` 线性扫描 + +去重实现是 `symbolRowsByOffset[offset]?.contains(row)`，对桶线性扫描。注释断言"桶实际上只有一两行"，但没有任何东西保证——不同符号名合法共享地址正是桶为数组的原因，退化偏移（0，或 stripped / dyld 缓存镜像里被大量别名的地址）能攒到上千行，那样整趟采集是 O(n²)。 + +## 调研 + +### 缺口一：惩罚项应当正交于形态分类 + +审查建议"把支持根判断提成一次性惩罚项，在形态分类之前施加"。但"之前/之后"不是要点，**是否作用于所有形态**才是。而且惩罚项与形态的**主次关系**必须想清楚： + +若让支持根作主键（原生任何形态 < Catalyst 任何形态），那么一个原生 `.axbundle` 会压过 Catalyst 的 framework 二进制——而没有 Swift 元数据的 accessibility bundle 抢赢 framework，正是整套排序机制最初要解决的问题。所以**形态必须是主键，支持根是同形态内的次级裁决**。 + +### 缺口二：新行不可能已经在桶里 + +重复只有两个成因，各自都能 O(1) 判定，根本不需要通用去重： + +- 原始偏移 == 规范偏移 → 比较两个偏移即可（原本就有这个守卫）。 +- 同名同址条目折叠到同一行 → 只有**本来就存在的行**才可能出现在桶里。新行索引取自 `symbolTable.count`，严格大于此前发出的所有行号，所以任何桶都装不下它。 + +导出符号那趟循环带 `tableRowByName[...] == nil` 前置条件，必然走新行分支，因此完全不进检查。 + +## 最终方案 + +1. **`matchRank` 拆成两步打分**：形态基准分 × `rankStepsPerPathShape`(2) + 支持根 `catalystSupportRootPenalty`(1)。乘 2 拉开间距，保证惩罚项永远不会把某形态顶到下一形态的分位上。全序： + + | 排名 | 含义 | + | --- | --- | + | 0 | 原生 canonical framework（唯一能拿 `bestMatchRank`） | + | 1 | Catalyst framework | + | 2 | 原生 plain dylib | + | 3 | Catalyst plain dylib | + | 4 | 原生 bundle / 其他 | + | 5 | Catalyst bundle / 其他 | + + `bestMatchRank` 仍然只有原生 canonical framework 能达到，所以 `accumulateBestMatch` 拿到 0 级就提前退出这件事依然成立。 + +2. **`canonicalRow` 返回 `(row, isNewRow)`**，`registerRow` 透传，`appendRow` 增加 `mayAlreadyBeListed` 参数，只在该参数为真时才扫桶。 + +## 实际执行 + +- `Sources/MachOExtensions/DyldCache+.swift` —— 新增 `rankStepsPerPathShape` / `catalystSupportRootPenalty` 两个常量（后者的文档注释写明"限定在单一形态里"正是第一版的错误）；`matchRank` 的 `.name` 分支重写为先算 `pathShapeRank` 再叠加惩罚。 +- `Sources/MachOSymbols/SymbolIndexStore.swift` —— `canonicalRow` 改签名，`appendRowIfAbsent` 更名 `appendRow(_:atOffset:mayAlreadyBeListed:)`，两个采集循环相应更新；注释重写为解释两个成因各自如何被 O(1) 挡掉，并说明为什么退化偏移场景现在一次都不扫。 +- `Tests/MachOCachesTests/DyldCacheImageSearchTests.swift` —— 新增两个用例：`catalystPlainDylibLosesToItsNativeNamesake`（用真实的 `libGLVMPlugin` 双路径）、`supportRootPenaltyNeverCrossesAShapeBoundary`（钉住"惩罚不跨形态"这条不变量）。 + +## 验证 + +- `swift build` 通过。 +- `swift test --filter DyldCacheImageSearchTests`：11 个用例全过（原 9 + 新 2）。 +- `swift test --skip IntegrationTests`：1280 测试 / 146 issue，**失败测试名集合与既有基线逐条一致（19 个）**。测试总数从 1275 增至 1280，正是本分支累计新增的 5 个用例。 +- 端到端：`swift-section dump --uses-system-dyld-shared-cache -n SwiftUI` 与上一轮验证过的输出**逐字节一致**（109387 行）。 + +## 偏差说明 + +无。两条缺口都按计划闭环，且都有测试钉住。 + +其余 10 条审查结论按用户要求仍然只记录不修，清单见 [../NodeStoreMigrationOpenIssues.md](../NodeStoreMigrationOpenIssues.md) 第二节起——其中四条的修复位置在上游 `swift-demangling`，本仓库这侧绕不开。 diff --git a/Documentations/README.md b/Documentations/README.md index 61a3d6ad..c434db9a 100644 --- a/Documentations/README.md +++ b/Documentations/README.md @@ -74,5 +74,5 @@ required by `Version.swift`'s bump contract). | [ClassMemberKeywordRecovery.md](Internal/ClassMemberKeywordRecovery.md) | `class` / `static` 成员关键字的还原：mangling 层面两者不可区分，判据是「类型级成员有 vtable method descriptor ⇒ 源码是 `class`」（`static` 隐式 final、不进 vtable）；模型侧 `isClassMember` 计算属性 + 三个 node printer 接线 + dump vtable 段落关键字，顺带消灭非法的 `override static` 输出；`final class func` 等四类 ABI 上与 `static` 完全一致，保守输出语义等价的 `static`。 | | [NodeStoreMigrationPlan.md](Internal/NodeStoreMigrationPlan.md) | `SymbolIndexStore` 迁移到 `NodeStore` arena 存储的分期计划与实施记录（Stage 0–5）：12B/节点扁平缓冲、cache-free build sweep、Symbol 表压缩、声明层换持 `NodeReference`、散点 transient demangling；含 Stage 5a 的 override/vtable 回归修复（裸 `[NodeReference: …]` 的 store-identity 陷阱 → `StructuralNodeReferenceKey`）与三源快照验收。 | | [DeclarationModelMemoryFootprint.md](Internal/DeclarationModelMemoryFootprint.md) | NodeStore 迁移后的声明模型内存足迹量测：`TypeDefinition` 1272 字节的逐属性构成（两份 `TypeContextWrapper` 占 74%）、`TypeContextWrapper` 按最大 case `Class` 定尺的原因、`parentContext` 是被当成永久字段的临时值、mini-store 增殖与 `MetadataReaderCache` 仍持 `Node` 树；含四项可回收估算（合计 ~8–10%）与「当前不建议实施、应先剖析其余 90%」的结论。 | -| [NodeStoreMigrationOpenIssues.md](Internal/NodeStoreMigrationOpenIssues.md) | `feature/node-store-migration` 上**已确认但未修复**的问题清单（2026-07-28 两轮审查 + 复核）：Catalyst 降级只覆盖 framework 形态导致 plain dylib 仍平局、`appendRowIfAbsent` 线性扫描、两个公开查询 API 的字典键从结构相等翻成 store 身份相等、`structuralHash` 每文本节点分配 `String`（修复位置在上游）、`memberSymbols` 退化为线性 + 全树比对、build sweep 串行且每符号跨线程往返（上游）、`ABIKey` 每 key materialize，以及四条代码卫生项与 rebase 前置事项。逐条注明成因、影响面与「该在哪里修」。 | +| [NodeStoreMigrationOpenIssues.md](Internal/NodeStoreMigrationOpenIssues.md) | `feature/node-store-migration` 的**已确认问题清单**（2026-07-28 两轮审查 + 复核）。第一节是前一轮修复自身的两处缺口（Catalyst 降级只覆盖 framework 形态导致 plain dylib 仍平局、`appendRowIfAbsent` 线性扫描），已于 2026-07-29 闭环并保留成因；第二节起为**仍然打开**的：两个公开查询 API 的字典键从结构相等翻成 store 身份相等、`structuralHash` 每文本节点分配 `String`（修复位置在上游）、`memberSymbols` 退化为线性 + 全树比对、build sweep 串行且每符号跨线程往返（上游）、`ABIKey` 每 key materialize，以及四条代码卫生项与 rebase 前置事项。逐条注明成因、影响面与「该在哪里修」。 | | [TaskReports/](Internal/TaskReports/) | Dated per-task fix / investigation reports. | diff --git a/Sources/MachOSymbols/SymbolIndexStore.swift b/Sources/MachOSymbols/SymbolIndexStore.swift index 4171c88a..f8078b95 100644 --- a/Sources/MachOSymbols/SymbolIndexStore.swift +++ b/Sources/MachOSymbols/SymbolIndexStore.swift @@ -333,18 +333,21 @@ public final class SymbolIndexStore: SharedCache, @unc var tableRowByName: [String: UInt32] = [:] var symbolRowsByOffset: OrderedDictionary = [:] - // Raw and cache-adjusted offset keys share one canonical row; a - // duplicate name updates the existing row in place (last-wins, like - // the former name-keyed collection pass). - func canonicalRow(for canonicalSymbol: Symbol) -> UInt32 { + /// The table row a symbol belongs to, plus whether this call created it. + /// + /// Raw and cache-adjusted offset keys share one canonical row; a + /// duplicate name updates the existing row in place (last-wins, like + /// the former name-keyed collection pass). `isNewRow` is what lets + /// `registerRow` skip its duplicate check — see there. + func canonicalRow(for canonicalSymbol: Symbol) -> (row: UInt32, isNewRow: Bool) { if let existingRow = tableRowByName[canonicalSymbol.name] { symbolTable[Int(existingRow)] = canonicalSymbol - return existingRow + return (existingRow, false) } let newRow = UInt32(symbolTable.count) symbolTable.append(canonicalSymbol) tableRowByName[canonicalSymbol.name] = newRow - return newRow + return (newRow, true) } // One offset legitimately maps to several rows — distinct symbol names @@ -352,28 +355,39 @@ public final class SymbolIndexStore: SharedCache, @unc // must not be listed twice though, or every `for symbol in symbols` // loop visits it twice. // - // A row repeats for two independent reasons, so the bucket itself has - // to be consulted rather than just the two offsets: + // A row repeats for two independent reasons, and each is headed off + // without scanning the bucket: // // - Raw and canonical offsets coincide whenever there is nothing to - // adjust (a `MachOImage`, or a file at offset 0), which makes the - // second append a duplicate of the first. + // adjust (a `MachOImage`, or a file at offset 0), which would make + // the second append a duplicate of the first. Comparing the two + // offsets settles it. // - Two symbol-table entries carrying the *same name* at the same // address — aliases and weak definitions do occur in a symtab — // fold onto one canonical row, so the second entry re-registers a - // row the first already put in that bucket. + // row the first already put in that bucket. Only a row that already + // existed can be in a bucket at all: a freshly minted row index is + // `symbolTable.count`, strictly greater than every row issued so + // far, so no bucket can hold it. Checking `isNewRow` therefore + // settles this one too. // - // Buckets hold one or two rows in practice, so scanning one is - // cheaper than the per-offset set it would take to avoid the scan. - func appendRowIfAbsent(_ row: UInt32, atOffset offset: Int) { - guard symbolRowsByOffset[offset]?.contains(row) != true else { return } + // Scanning the bucket instead would be O(bucket) on every symbol, + // which is fine while buckets stay short but goes quadratic on a + // degenerate address — offset 0, or a heavily aliased address in a + // stripped or dyld-cache image — where thousands of distinct names + // pile onto one offset. Those piles are exactly the case that must + // stay cheap, and under `isNewRow` they never get scanned at all. + func appendRow(_ row: UInt32, atOffset offset: Int, mayAlreadyBeListed: Bool) { + if mayAlreadyBeListed, symbolRowsByOffset[offset]?.contains(row) == true { + return + } symbolRowsByOffset[offset, default: []].append(row) } - func registerRow(_ row: UInt32, rawOffset: Int, canonicalOffset: Int) { - appendRowIfAbsent(row, atOffset: rawOffset) + func registerRow(_ row: UInt32, rawOffset: Int, canonicalOffset: Int, isNewRow: Bool) { + appendRow(row, atOffset: rawOffset, mayAlreadyBeListed: !isNewRow) if canonicalOffset != rawOffset { - appendRowIfAbsent(row, atOffset: canonicalOffset) + appendRow(row, atOffset: canonicalOffset, mayAlreadyBeListed: !isNewRow) } } @@ -383,8 +397,8 @@ public final class SymbolIndexStore: SharedCache, @unc if let cache = machO.cache, rawOffset >= 0, machO is MachOFile { canonicalOffset = rawOffset - cache.mainCacheHeader.sharedRegionStart.cast() } - let row = canonicalRow(for: .init(offset: canonicalOffset, name: symbol.name, isExternal: symbol.nlist.isExternal)) - registerRow(row, rawOffset: rawOffset, canonicalOffset: canonicalOffset) + let (row, isNewRow) = canonicalRow(for: .init(offset: canonicalOffset, name: symbol.name, isExternal: symbol.nlist.isExternal)) + registerRow(row, rawOffset: rawOffset, canonicalOffset: canonicalOffset, isNewRow: isNewRow) } for exportedSymbol in machO.exportedSymbols where exportedSymbol.name.isSwiftSymbol { @@ -393,8 +407,11 @@ public final class SymbolIndexStore: SharedCache, @unc if machO is MachOFile { canonicalOffset += machO.startOffset } - let row = canonicalRow(for: .init(offset: canonicalOffset, name: exportedSymbol.name)) - registerRow(row, rawOffset: rawOffset, canonicalOffset: canonicalOffset) + // The `tableRowByName` guard above means this name has no row + // yet, so `canonicalRow` always mints one and the duplicate + // check is never needed here. + let (row, isNewRow) = canonicalRow(for: .init(offset: canonicalOffset, name: exportedSymbol.name)) + registerRow(row, rawOffset: rawOffset, canonicalOffset: canonicalOffset, isNewRow: isNewRow) } } diff --git a/Tests/MachOCachesTests/DyldCacheImageSearchTests.swift b/Tests/MachOCachesTests/DyldCacheImageSearchTests.swift index 6f136127..6fdb2fce 100644 --- a/Tests/MachOCachesTests/DyldCacheImageSearchTests.swift +++ b/Tests/MachOCachesTests/DyldCacheImageSearchTests.swift @@ -20,6 +20,11 @@ struct DyldCacheImageSearchTests { private let accessibilityBundlePath = "/System/Library/AccessibilityBundles/SwiftUI.axbundle/SwiftUI" private let catalystFrameworkPath = "/System/iOSSupport/System/Library/Frameworks/SwiftUI.framework/SwiftUI" + // Both of these exist on macOS 26. Neither sits inside a + // `libGLVMPlugin.framework`, so both classify as plain dylibs. + private let nativePluginDylibPath = "/System/Library/Frameworks/OpenGL.framework/Versions/A/Libraries/libGLVMPlugin.dylib" + private let catalystPluginDylibPath = "/System/iOSSupport/System/Library/Frameworks/OpenGLES.framework/Versions/A/Libraries/libGLVMPlugin.dylib" + private let bestRank = DyldCacheImageSearchMode.bestMatchRank // MARK: - Name lookup ranking @@ -89,6 +94,37 @@ struct DyldCacheImageSearchTests { #expect(mode.matchRank(forImagePath: catalystFrameworkPath) != nil) } + /// The support-root demotion has to apply to **every** path shape, not just + /// to framework-shaped ones. + /// + /// `libGLVMPlugin.dylib` ships natively under `OpenGL.framework` and as a + /// Mac Catalyst build under `iOSSupport/…/OpenGLES.framework`. Neither is + /// inside a `libGLVMPlugin.framework`, so both classify as plain dylibs — + /// and while the demotion lived inside the framework branch they tied, + /// putting `-n libGLVMPlugin` back at the mercy of enumeration order. + @Test func catalystPlainDylibLosesToItsNativeNamesake() throws { + let mode = DyldCacheImageSearchMode.name("libGLVMPlugin") + let nativeRank = try #require(mode.matchRank(forImagePath: nativePluginDylibPath)) + let catalystRank = try #require(mode.matchRank(forImagePath: catalystPluginDylibPath)) + #expect(nativeRank < catalystRank) + } + + /// The penalty is a tiebreak *within* a shape, never a reclassification: a + /// demoted Catalyst build still beats every worse shape. Otherwise a native + /// `.axbundle` — the metadata-less payload that started all of this — could + /// outrank a Catalyst framework binary. + @Test func supportRootPenaltyNeverCrossesAShapeBoundary() throws { + let frameworkMode = DyldCacheImageSearchMode.name("SwiftUI") + let catalystFrameworkRank = try #require(frameworkMode.matchRank(forImagePath: catalystFrameworkPath)) + let nativeBundleRank = try #require(frameworkMode.matchRank(forImagePath: accessibilityBundlePath)) + #expect(catalystFrameworkRank < nativeBundleRank) + + let dylibMode = DyldCacheImageSearchMode.name("libGLVMPlugin") + let catalystDylibRank = try #require(dylibMode.matchRank(forImagePath: catalystPluginDylibPath)) + #expect(catalystFrameworkRank < catalystDylibRank) + #expect(catalystDylibRank < nativeBundleRank) + } + @Test func nonMatchingLeafNameIsNotAMatch() { let mode = DyldCacheImageSearchMode.name("SwiftUI") #expect(mode.matchRank(forImagePath: "/System/Library/Frameworks/SwiftUICore.framework/SwiftUICore") == nil) From 337c1faaabba6f57a9bfeb916b071b13dbb7ca28 Mon Sep 17 00:00:00 2001 From: Mx-Iris Date: Thu, 30 Jul 2026 14:41:45 +0800 Subject: [PATCH 22/77] build: track swift-demangling's feature/node-store branch The NodeStore migration builds against the store-backed demangler: NodeStore, NodeReference and DemanglingPrinter only exist on that branch, not in the 0.4.3 tag this manifest pointed at. --- Package.swift | 8 +------- 1 file changed, 1 insertion(+), 7 deletions(-) diff --git a/Package.swift b/Package.swift index ef3a8c0f..1b19c62e 100644 --- a/Package.swift +++ b/Package.swift @@ -213,13 +213,7 @@ extension Package.Dependency { ), remote: .package( url: "https://github.com/MxIris-Reverse-Engineering/swift-demangling", - // Pinned below 0.5.0: that release reshaped `NodePrinterTarget` - // (`write(_:context:)` / `pushTypeReferenceScope(_:)` take - // `@autoclosure` parameters and lost their default implementations) - // and dropped `Node: Codable`. The adoption lives on - // feature/node-store-migration; until it lands, an open upper bound - // silently floats main onto an incompatible demangler. - "0.4.5" ..< "0.5.0", + branch: "feature/node-store", ), ) From 0aa0d09e4afaea6c53e2bedd516123ba164a0ccc Mon Sep 17 00:00:00 2001 From: Mx-Iris Date: Fri, 31 Jul 2026 18:40:19 +0800 Subject: [PATCH 23/77] fix: adopt swift-demangling's autoclosure hooks and dropped Node: Codable MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Two source-breaking changes landed upstream in the same revision; both were silent hazards on this side before they became compile errors. `NodePrinterTarget.write(_:context:)` became `@autoclosure` and lost its protocol default. `SemanticString`'s adapter still declared the eager form, so it silently stopped being the witness and the (context-dropping) default took over: every token the demangling printer produced was appended as `.standard`, which killed terminal colouring and made `replacingTypeNameOrOtherToTypeDeclaration()` a no-op, so declaration headers lost their type-declaration annotation. The printed text stayed byte-identical, so the whole suite passed green throughout. The sibling `pushTypeReferenceScope` hook had already been migrated; this one had not. Removing the default upstream makes a near-miss signature a hard error, so the compiler is the regression guard from here on. `Node` is no longer `Codable` — a mangled symbol already is the tree's serialized form (smaller, ABI-stable, sharing-preserving through the round trip). Rather than re-encode, `TypeName` / `ProtocolName` / `ExtensionName` drop the conformance outright: nothing in or out of the repo persists them, `DefinitionName` does not require it, and the one adjacent `Codable` type (`AssociatedTypeWitnessProjection`) holds only strings. AGENTS.md's claim that their `Codable` stayed wire-compatible is replaced by a note on the deliberate non-conformance, pointing at `mangleAsString` / `demangleAsNode` for any future need. --- AGENTS.md | 2 +- .../Components/Names/ExtensionName.swift | 24 +----------------- .../Components/Names/ProtocolName.swift | 21 +--------------- .../Components/Names/TypeName.swift | 25 +------------------ .../Extensions/Node+.swift | 4 +-- 5 files changed, 6 insertions(+), 70 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index f585b996..7e4281b6 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -206,7 +206,7 @@ let node = try demangleAsNode("$sSiD") // Returns Node tree let string = node.print(using: .default) // "Swift.Int" ``` -**Symbol indexing (NodeStore-backed)**: `SymbolIndexStore.Storage` no longer retains `Node` class trees. The build sweep demangles each symbol cache-free onto a transient tree (`@_spi(Internals) demangleAsNodeTransient`), classifies on that tree, and interns it into a per-image `NodeStoreBuilder`. Nothing touches the global `NodeCache`, and dropping a `Storage` releases the whole per-image footprint. Storage layout (Stage 3): one flat `symbolTable: [Symbol]` row per unique symbol name (canonical cache-adjusted offset; raw and adjusted offset keys share one row), a parallel `rootNodeIndexByTableRow` array, and every classification index stores 4-byte `UInt32` table rows (member/opaque keys are 4-byte `NodeStore.NodeIndex`); indexes accumulate in final row form during the sweep, so `freeze()` is followed by a plain move, not a conversion pass. `Symbol` is 32 bytes (no `nlist` existential — the external-undefined bit is extracted at collection time into `isExternal`), and `DemangledSymbol` is a 32-byte value (shared-table reference + row + `NodeReference`) whose `symbol` is computed; both are asserted by `compactValueLayouts`. Query APIs construct `DemangledSymbol`/`NodeReference` values at the exit boundary; APIs taking an externally demangled node (`memberSymbols(of:for:node:)` with `Node` or `NodeReference`, `opaqueTypeDescriptorSymbol(for:)`) match keys via `NodeReference.structurallyEquals(_:)` (same-store O(1), cross-store structural walk) — `opaqueTypeDescriptorSymbol(for:)` buckets its entries by `DemanglingNode.identifier` first (`opaqueTypeDescriptorEntriesByMemberIdentifier`), because a full scan there is O(descriptors × printed `some` returns). `demangledNodeReference(for:)` matches on the symbol **name alone** — a demangled tree is a pure function of the name and the flat table holds one row per unique name, so comparing the row's canonical offset against the queried symbol's offset could only reject valid hits (it once rejected every symbol of a dyld-cache image, since `symbols(for:in:)` stamps each vended `Symbol` with the offset it was queried by). Names genuinely outside the build sweep fall back to name-keyed mini stores, minted under one lock so a race cannot hand two callers references into different stores (`lateDemangledNode(forName:)` / `MetadataReader.demangleSymbolReference(for:in:)`). Struct-layout changes in `MachOSymbols` need a `swift package clean` rebuild — SwiftPM incremental builds have been observed linking stale downstream objects (runtime SIGSEGV in `outlined destroy`). See `Documentations/Internal/NodeStoreMigrationPlan.md`. **Declaration values hold `NodeReference` (Stage 5a)**: `DefinitionName.node` (`TypeName`/`ProtocolName`/`ExtensionName`), `Variable/Function/SubscriptDefinition.node`, `FieldDefinition.typeNode` and `ExtensionDefinition.genericSignature` are `NodeReference`, not `Node`. Member definitions reference the per-image symbol store directly (a live definition keeps that store alive — the intended per-image recycling model); metadata-derived trees (`MetadataReader` output) are wrapped via `NodeReference(interning:)` mini stores, and `TypeDefinition.index(in:)` batches all field type trees of one type into a single shared store. The `Name` types customize `Hashable` to STRUCTURAL semantics (`structurallyEquals` + `structuralHash`) because `NodeReference`'s intrinsic `Hashable` is store-identity based and would split equal keys minted into different stores; their `Codable` stays wire-compatible by encoding a materialized `Node`. **Any `Dictionary`/`Set` keyed on a `NodeReference` whose keys and lookups can come from DIFFERENT stores must key on `StructuralNodeReferenceKey` (`Sources/MachOSymbols/`, next to the mini stores it reconciles), never a bare `NodeReference`.** Different stores are the norm, not the exception: `demangledNodeReference(for:)` falls back to a mini store, and `NodeReference(interning:)` — every `MetadataReader`-derived tree — mints a fresh private store per call by design. Everything that mixes those with image-store references is keyed structurally: the override/vtable lookups (`methodDescriptorLookup` / `vtableOffsetLookup` in `TypeDefinition.index` → `DefinitionBuilder`), `DefinitionBuilder`'s accessor grouping and merged-thunk dedup (`accessorsByNode` / `canonicalIndexByFunctionNode` / `canonicalIndexByAllocatorNode`), and every `visitedNodes` "already claimed this symbol" set (`ExtensionDefinition` / `ProtocolDefinition` / `TypeDefinition` / `OverrideSymbolMatcher` / `ProtocolConformanceDumper`). Under bare keys these fail silently and differently: the override lookups drop the `override` keyword + vtable-offset comment (the Stage 5a regression fixed 2026-07-25), a subscript's getter and setter split into two buckets so the setter-only one is discarded, merged thunks emit their `func`/`init` twice, and a witness can be claimed twice. Bare-`NodeReference` keys are only safe for grouping/dedup WITHIN one `memberSymbols` batch (single hash-consed store, where structural equality coincides with index equality). **Transient demangling everywhere else (Stage 5c)**: `MetadataReader`, `RuntimeFieldLayoutBackend`, `TypedDumper`, `ClassHierarchyDumper` and `Symbol.demangledNode` use `demangleAsNodeTransient` + `Node.createTransient`, so the global `NodeCache` no longer grows while browsing. Do NOT reintroduce `demangleAsNode`/`Node.create` on unbounded inputs in those paths. +**Symbol indexing (NodeStore-backed)**: `SymbolIndexStore.Storage` no longer retains `Node` class trees. The build sweep demangles each symbol cache-free onto a transient tree (`@_spi(Internals) demangleAsNodeTransient`), classifies on that tree, and interns it into a per-image `NodeStoreBuilder`. Nothing touches the global `NodeCache`, and dropping a `Storage` releases the whole per-image footprint. Storage layout (Stage 3): one flat `symbolTable: [Symbol]` row per unique symbol name (canonical cache-adjusted offset; raw and adjusted offset keys share one row), a parallel `rootNodeIndexByTableRow` array, and every classification index stores 4-byte `UInt32` table rows (member/opaque keys are 4-byte `NodeStore.NodeIndex`); indexes accumulate in final row form during the sweep, so `freeze()` is followed by a plain move, not a conversion pass. `Symbol` is 32 bytes (no `nlist` existential — the external-undefined bit is extracted at collection time into `isExternal`), and `DemangledSymbol` is a 32-byte value (shared-table reference + row + `NodeReference`) whose `symbol` is computed; both are asserted by `compactValueLayouts`. Query APIs construct `DemangledSymbol`/`NodeReference` values at the exit boundary; APIs taking an externally demangled node (`memberSymbols(of:for:node:)` with `Node` or `NodeReference`, `opaqueTypeDescriptorSymbol(for:)`) match keys via `NodeReference.structurallyEquals(_:)` (same-store O(1), cross-store structural walk) — `opaqueTypeDescriptorSymbol(for:)` buckets its entries by `DemanglingNode.identifier` first (`opaqueTypeDescriptorEntriesByMemberIdentifier`), because a full scan there is O(descriptors × printed `some` returns). `demangledNodeReference(for:)` matches on the symbol **name alone** — a demangled tree is a pure function of the name and the flat table holds one row per unique name, so comparing the row's canonical offset against the queried symbol's offset could only reject valid hits (it once rejected every symbol of a dyld-cache image, since `symbols(for:in:)` stamps each vended `Symbol` with the offset it was queried by). Names genuinely outside the build sweep fall back to name-keyed mini stores, minted under one lock so a race cannot hand two callers references into different stores (`lateDemangledNode(forName:)` / `MetadataReader.demangleSymbolReference(for:in:)`). Struct-layout changes in `MachOSymbols` need a `swift package clean` rebuild — SwiftPM incremental builds have been observed linking stale downstream objects (runtime SIGSEGV in `outlined destroy`). See `Documentations/Internal/NodeStoreMigrationPlan.md`. **Declaration values hold `NodeReference` (Stage 5a)**: `DefinitionName.node` (`TypeName`/`ProtocolName`/`ExtensionName`), `Variable/Function/SubscriptDefinition.node`, `FieldDefinition.typeNode` and `ExtensionDefinition.genericSignature` are `NodeReference`, not `Node`. Member definitions reference the per-image symbol store directly (a live definition keeps that store alive — the intended per-image recycling model); metadata-derived trees (`MetadataReader` output) are wrapped via `NodeReference(interning:)` mini stores, and `TypeDefinition.index(in:)` batches all field type trees of one type into a single shared store. The `Name` types customize `Hashable` to STRUCTURAL semantics (`structurallyEquals` + `structuralHash`) because `NodeReference`'s intrinsic `Hashable` is store-identity based and would split equal keys minted into different stores; they are deliberately NOT `Codable` — upstream dropped `Node: Codable` (a mangled symbol already is the tree's serialized form: smaller, ABI-stable across toolchains, and sharing-preserving through the round trip), and nothing persists these values. A consumer that ever needs to should persist `mangleAsString(node)` and read it back with `demangleAsNode(_:)` rather than reintroducing a node encoding. **Any `Dictionary`/`Set` keyed on a `NodeReference` whose keys and lookups can come from DIFFERENT stores must key on `StructuralNodeReferenceKey` (`Sources/MachOSymbols/`, next to the mini stores it reconciles), never a bare `NodeReference`.** Different stores are the norm, not the exception: `demangledNodeReference(for:)` falls back to a mini store, and `NodeReference(interning:)` — every `MetadataReader`-derived tree — mints a fresh private store per call by design. Everything that mixes those with image-store references is keyed structurally: the override/vtable lookups (`methodDescriptorLookup` / `vtableOffsetLookup` in `TypeDefinition.index` → `DefinitionBuilder`), `DefinitionBuilder`'s accessor grouping and merged-thunk dedup (`accessorsByNode` / `canonicalIndexByFunctionNode` / `canonicalIndexByAllocatorNode`), and every `visitedNodes` "already claimed this symbol" set (`ExtensionDefinition` / `ProtocolDefinition` / `TypeDefinition` / `OverrideSymbolMatcher` / `ProtocolConformanceDumper`). Under bare keys these fail silently and differently: the override lookups drop the `override` keyword + vtable-offset comment (the Stage 5a regression fixed 2026-07-25), a subscript's getter and setter split into two buckets so the setter-only one is discarded, merged thunks emit their `func`/`init` twice, and a witness can be claimed twice. Bare-`NodeReference` keys are only safe for grouping/dedup WITHIN one `memberSymbols` batch (single hash-consed store, where structural equality coincides with index equality). **Transient demangling everywhere else (Stage 5c)**: `MetadataReader`, `RuntimeFieldLayoutBackend`, `TypedDumper`, `ClassHierarchyDumper` and `Symbol.demangledNode` use `demangleAsNodeTransient` + `Node.createTransient`, so the global `NodeCache` no longer grows while browsing. Do NOT reintroduce `demangleAsNode`/`Node.create` on unbounded inputs in those paths. ## Test Environment diff --git a/Sources/SwiftDeclaration/Components/Names/ExtensionName.swift b/Sources/SwiftDeclaration/Components/Names/ExtensionName.swift index d2662b23..e84d2d1e 100644 --- a/Sources/SwiftDeclaration/Components/Names/ExtensionName.swift +++ b/Sources/SwiftDeclaration/Components/Names/ExtensionName.swift @@ -3,7 +3,7 @@ import Semantic import Demangling @MemberwiseInit(.public) -public struct ExtensionName: DefinitionName, Hashable, Sendable, Codable { +public struct ExtensionName: DefinitionName, Hashable, Sendable { public let node: NodeReference public let kind: ExtensionKind @@ -48,25 +48,3 @@ extension ExtensionName { node.structuralHash(into: &hasher) } } - -// MARK: - Codable - -// Wire-compatible with the historical `node: Node` encoding. -extension ExtensionName { - private enum CodingKeys: String, CodingKey { - case node - case kind - } - - public init(from decoder: Decoder) throws { - let container = try decoder.container(keyedBy: CodingKeys.self) - self.node = NodeReference(interning: try container.decode(Node.self, forKey: .node)) - self.kind = try container.decode(ExtensionKind.self, forKey: .kind) - } - - public func encode(to encoder: Encoder) throws { - var container = encoder.container(keyedBy: CodingKeys.self) - try container.encode(node.materialize(), forKey: .node) - try container.encode(kind, forKey: .kind) - } -} diff --git a/Sources/SwiftDeclaration/Components/Names/ProtocolName.swift b/Sources/SwiftDeclaration/Components/Names/ProtocolName.swift index 44432434..9922ef70 100644 --- a/Sources/SwiftDeclaration/Components/Names/ProtocolName.swift +++ b/Sources/SwiftDeclaration/Components/Names/ProtocolName.swift @@ -3,7 +3,7 @@ import Semantic import Demangling @MemberwiseInit(.public) -public struct ProtocolName: DefinitionName, Hashable, Sendable, Codable { +public struct ProtocolName: DefinitionName, Hashable, Sendable { public let node: NodeReference @SemanticStringBuilder @@ -31,22 +31,3 @@ extension ProtocolName { node.structuralHash(into: &hasher) } } - -// MARK: - Codable - -// Wire-compatible with the historical `node: Node` encoding. -extension ProtocolName { - private enum CodingKeys: String, CodingKey { - case node - } - - public init(from decoder: Decoder) throws { - let container = try decoder.container(keyedBy: CodingKeys.self) - self.node = NodeReference(interning: try container.decode(Node.self, forKey: .node)) - } - - public func encode(to encoder: Encoder) throws { - var container = encoder.container(keyedBy: CodingKeys.self) - try container.encode(node.materialize(), forKey: .node) - } -} diff --git a/Sources/SwiftDeclaration/Components/Names/TypeName.swift b/Sources/SwiftDeclaration/Components/Names/TypeName.swift index a75a377e..df3f6ad1 100644 --- a/Sources/SwiftDeclaration/Components/Names/TypeName.swift +++ b/Sources/SwiftDeclaration/Components/Names/TypeName.swift @@ -3,7 +3,7 @@ import Semantic import Demangling @MemberwiseInit(.public) -public struct TypeName: DefinitionName, Hashable, Sendable, Codable { +public struct TypeName: DefinitionName, Hashable, Sendable { public let node: NodeReference public let kind: TypeKind @@ -42,26 +42,3 @@ extension TypeName { node.structuralHash(into: &hasher) } } - -// MARK: - Codable - -// Wire-compatible with the historical `node: Node` encoding: the node is -// encoded as a materialized `Node` tree and re-interned on decode. -extension TypeName { - private enum CodingKeys: String, CodingKey { - case node - case kind - } - - public init(from decoder: Decoder) throws { - let container = try decoder.container(keyedBy: CodingKeys.self) - self.node = NodeReference(interning: try container.decode(Node.self, forKey: .node)) - self.kind = try container.decode(TypeKind.self, forKey: .kind) - } - - public func encode(to encoder: Encoder) throws { - var container = encoder.container(keyedBy: CodingKeys.self) - try container.encode(node.materialize(), forKey: .node) - try container.encode(kind, forKey: .kind) - } -} diff --git a/Sources/SwiftDeclarationRendering/Extensions/Node+.swift b/Sources/SwiftDeclarationRendering/Extensions/Node+.swift index a41918f8..7866f98a 100644 --- a/Sources/SwiftDeclarationRendering/Extensions/Node+.swift +++ b/Sources/SwiftDeclarationRendering/Extensions/Node+.swift @@ -14,8 +14,8 @@ extension SemanticString: @retroactive NodePrinterTarget { popIdentifierScope() } - public mutating func write(_ content: String, context: NodePrintContext?) { - guard let context else { + public mutating func write(_ content: String, context: @autoclosure () -> NodePrintContext?) { + guard let context = context() else { write(content) return } From a1a83745e277524fb49495dd3b9e43446b7e4290 Mon Sep 17 00:00:00 2001 From: Mx-Iris Date: Fri, 31 Jul 2026 18:40:33 +0800 Subject: [PATCH 24/77] docs: record the 2026-07-31 review with its stack-hop measurements MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Adds `Documentations/Internal/Reviews/` — dated per-review-event records, kept separate from the topic-organised issue ledgers they cross-reference. This first entry covers the multi-agent review of the branch (63 changed files, 57 candidates, 34 independent verifiers, 15 findings after merge): the two items fixed in the previous commit, a 17-item to-do list grouped by priority, and a per-item map onto NodeStoreMigrationOpenIssues.md so the two documents cannot drift into contradicting each other. The printing-path thread hop is the one item that was measured rather than argued, and the measurement corrects the review's own claim. A probe target recording the thread the walk actually runs on shows the hop firing on 100% of calls across all three contexts (main / cooperative worker / libdispatch worker — all 524 KB stacks against a 2 MB floor, so the inline path is unreachable by construction). But the cost is 8-21 µs per call in release, i.e. 2.28x on a small tree and only 1.14x on a 916-character real symbol — not the "large multiple" the review asserted. Pool starvation remains argued-not-measured, and is labelled as such. Wrapping a render batch in one `StackSafeExecutor.withLargeStack` removes the hop; the control group in the table is exactly that, and the repo currently has zero uses of it. Two premises in the older ledger are corrected here rather than in place: its `executeWithinStackBudget` printing entry point does not exist in the pinned upstream revision, and its "five commits behind main, rebase first" section is stale (though a still-live caveat is buried in it). One genuine conflict is left for a human: four candidates matching its `memberSymbols` O(1)-regression claim were all refuted, on the grounds that the "was a hash lookup" premise does not match the code. --- .../2026-07-31-node-store-migration-review.md | 180 ++++++++++++++++++ Documentations/README.md | 1 + 2 files changed, 181 insertions(+) create mode 100644 Documentations/Internal/Reviews/2026-07-31-node-store-migration-review.md diff --git a/Documentations/Internal/Reviews/2026-07-31-node-store-migration-review.md b/Documentations/Internal/Reviews/2026-07-31-node-store-migration-review.md new file mode 100644 index 00000000..933875a1 --- /dev/null +++ b/Documentations/Internal/Reviews/2026-07-31-node-store-migration-review.md @@ -0,0 +1,180 @@ +# 2026-07-31 `feature/node-store-migration` 代码审查 + +本文记录 2026-07-31 对 `feature/node-store-migration` 做的一轮代码审查：**结论、实测数据、以及待处理清单**。 + +与 [`NodeStoreMigrationOpenIssues.md`](../NodeStoreMigrationOpenIssues.md) 的分工：那份是按技术主题组织的长期遗留问题台账；本文是**一次审查事件的记录**，包含它自己的实测数据和当时的判断。两者重叠的条目在第五节逐条对照，避免两边各说各话。 + +## 审查方式 + +多智能体并行审查，6 个 finder 角度（逐行扫描 / 删除行为审计 / 跨文件追踪 / 语言陷阱 / 包装类型正确性 / 清理类）各自独立出候选，再对每个 `(文件, 行)` 位置派一个独立 verifier 做对抗性验证（默认倾向证伪），最后归并去重。 + +规模：63 个改动文件 → 57 条候选 → 34 个 verifier → 50 条完成验证（43 条保留、7 条被证伪）→ 归并后 15 条。 + +对比基线：`git diff origin/main...origin/feature/node-store-migration`。 + +## 一、本轮已闭环 + +### 1. `SemanticString.write(_:context:)` 不再是 witness，语义标注被静默丢弃 ✅ + +`NodePrinterTarget` 的 `write(_:context:)` 要求在上游改成了 `@autoclosure () -> NodePrintContext?`，而 `SwiftDeclarationRendering/Extensions/Node+.swift` 里 `SemanticString` 的实现仍是即时求值形态。Swift 认为签名不匹配，**不报错也不警告**，直接改用协议自带的默认实现——而那个默认实现把 context 丢掉、退回 `write(content)` → `append(string, type: .standard)`。 + +后果是所有经 demangler 打印的 token 都变成无标注的 `.standard`：终端着色全部失效，`replacingTypeNameOrOtherToTypeDeclaration()`（只重写 `.type(_, .name)` / `.other`）变成空操作，声明头部丢失类型声明标注，RuntimeViewer 的高亮与类型跳转随之失效。**输出文字逐字节相同**，所以整个测试套件全绿。 + +同一处改动里旁边的 `pushTypeReferenceScope` 已经跟着改成 `@autoclosure` 了，唯独这一个漏掉。 + +**修复**：参数改为 `@autoclosure () -> NodePrintContext?`，`guard let context = context()` 求值一次。 + +**防回归**：上游在同一版里**删除了这两个要求的协议默认实现**（只保留无参数的 `popTypeReferenceScope`，并在注释中说明原因："它不带参数，没有近似签名可以被吞掉"）。所以此后签名写错是硬编译错误，编译器本身就是守卫，不需要额外写测试。 + +### 2. 三个 Name 类型的 `Codable` 与其"线路兼容"承诺 ✅ + +`TypeName` / `ProtocolName` / `ExtensionName` 的手写 `Codable` 注释声称"与历史 `node: Node` 编码保持线路兼容"。上游在同一版里**删除了 `Node: Codable`**（理由：mangled 符号本身就是这棵树的序列化形式——更小、由 Swift ABI 定义所以跨版本稳定、往返时重新 demangle 会重建 interning 而不是按路径数展开），该承诺随之作废，且成为编译错误。 + +**处理**：直接删除三个类型的 `Codable` 一致性与全部手写实现，而非改写编码。依据是这些类型本就不适合 `Codable`，且确认无人使用——`DefinitionName` 协议不要求它，`AssociatedTypeWitnessProjection`（唯一相关的 `Codable` 类型）只装字符串，仓库内外均无编解码这三个类型的地方。 + +`AGENTS.md` 中相应描述已改为说明"刻意不再是 `Codable`"及其原因，并指出将来若需持久化应走 `mangleAsString` / `demangleAsNode`。 + +## 二、实测：打印路径的线程跳转 + +这一条单列，因为它是本轮唯一做了定量测量的问题，而**审查报告最初对它的量级判断是错的**。 + +### 机制（代码事实） + +`swift-demangling` 的 `Sources/Demangling/Utils/StackSafeExecutor.swift`： + +| 位置 | 内容 | +| --- | --- | +| L58 | `minimumRemainingStackSize = 2 * 1024 * 1024` | +| L204-211 | `currentThreadHasSufficientStack` = 当前栈指针 − 栈底 ≥ 2MB | +| L128-146 | `executeWithUncheckedSendability`：够则内联，否则 `runOnLargeStack` | +| L226-239 | `runOnLargeStack`：提交线程池 + `DispatchSemaphore.wait()` | + +`NodePrinter.swift` L90-95：`DemanglingPrinter.print` 的函数体就是 `StackSafeExecutor.executeWithUncheckedSendability { ... }`。 + +### 实测：跳转率 100% + +探针 target 在 `write` 中记录**遍历实际执行所在的线程 ID**，与调用者线程 ID 对比： + +``` +[main thread] 总栈 524 KB 剩余 522 KB ≥2MB: false + caller 17785076 → walk 17785088 HOPPED: true +[Swift Concurrency cooperative worker] 总栈 524 KB 剩余 522 KB ≥2MB: false + caller 17785080 → walk 17785090 HOPPED: true +[libdispatch global worker] 总栈 524 KB 剩余 522 KB ≥2MB: false + caller 17785083 → walk 17785089 HOPPED: true +``` + +三种线程上下文全部换线程，无一例外。512 KB 栈的前提成立（实测 524 KB），2 MB 门槛永远过不去，**内联快速路径一次都没走到过**。 + +### 实测：代价 + +2000 次打印。对照组是把同样的循环包进一次 `StackSafeExecutor.withLargeStack`（批内实测 `hopped: false`，证明确实内联了）： + +| 场景 | 每次调用 | 一次 withLargeStack | 倍数 | 每次固定开销 | +| --- | --- | --- | --- | --- | +| 小树 · debug | 90.9 ms | 25.9 ms | 3.51x | 32.5 µs | +| 大树 · debug | 744.6 ms | 647.2 ms | 1.15x | 48.7 µs | +| 小树 · release | 29.1 ms | 12.8 ms | 2.28x | 8.2 µs | +| 大树 · release | 337.8 ms | 296.2 ms | **1.14x** | 20.8 µs | + +大树 = 从测试二进制中取的最长真实符号（916 字符）。 + +**更正**：审查报告原文称 "interface generation slows by a large multiple"（慢好几倍），**不成立**。真实情况是每次打印固定多付 8–21 µs（release），树越大被稀释得越厉害，大树上只有 ~14%。 + +### 未验证的部分 + +- **"阻塞协作线程导致并发池饿死"没有实测**。上表测的是单线程吞吐，不是池饱和。该结论目前仅由代码推导，需要构造多个并发渲染任务观察实际并行度才能确认。 +- 真实 interface 导出中符号树大小的分布未知，因此整体影响落在 1.14x–2.28x 之间的何处没有数据。 + +### 修法 + +`StackSafeExecutor.withLargeStack` 包住批量渲染入口即可消除，实测有效（对照组即是）。上游该函数的文档注释也正是这么规定的："Use this at a batch boundary — indexing every symbol of a binary, say — so the whole batch pays for at most one thread hop instead of one per call." 仓库内目前**零处**使用。 + +同时 `printSemantic` 上的文档注释断言 `print(_:options:)` "runs the recursion inline against a stack floor and pays for a worker only for a tree that actually reaches it"，与实测相反，应一并修正。 + +> 探针为一次性测量代码，测完已删除。若需长期守护该性质，应整理为正式 benchmark。 + +## 三、待处理清单 + +按建议优先级排列。标注 ⚠️ 的是本轮新发现（既有台账中没有）。 + +### 性能 + +1. **build sweep 由并行改为串行，且每符号无条件跨线程往返**（`SymbolIndexStore.swift:431`）。原为 `symbolArray.concurrentMap`，现为单趟顺序循环，且每个符号额外付一次线程池提交 + 信号量等待，无并行摊薄。大框架（SwiftUI 数十万符号）首次打开慢数倍。**影响最大的一条。** + - 注：既有台账第 6 条描述此问题时称"不是打印路径用的 `executeWithinStackBudget`"，但**该入口在当前上游版本中并不存在**，且第二节已实测证明打印路径同样每次跳转。该前提需要更正。 + +2. **打印路径每次调用跨线程 + 阻塞**（`Node+.swift:78`）。见第二节。注意原建议的"恢复内联调用"**已不可行**——上游把 `NodePrinter` 改成了空 enum（无构造器），`print(_:options:)` 是唯一公开走法。 + +3. ⚠️ **demangle 失败的符号名永不缓存，且在持锁状态下重试**（`SymbolIndexStore.swift:848`）。`buildStorageImpl` 对 demangle 失败的名字仍保留表行但 root index 为 `nil`，于是 `demangledNodeReference` 永远走不到快速路径、落到 `lateDemangledNode`，而后者按契约不缓存失败。`demangledOverrideSymbol` 会为每个类的每个方法遍历候选符号，每遇到一个不可 demangle 的符号就取一次 per-image 锁重跑 demangle（其本身还要跨线程阻塞），把其他线程全堵在后面。迁移前该 miss 路径完全不加锁。 + +4. **`lateDemangledNode(forName:)` 在持锁期间 demangle**（`SymbolIndexStore.swift:245`）。既有台账第 9 条，判定为 PLAUSIBLE（机制确定，触发条件依赖并发时序）。注意其临界区合并是**刻意的**（防止两个并发 miss 冻结出两份 mini store），修复时要保住该保证。 + +5. ⚠️ **dyld 缓存按名字查找丢失提前退出**（`DyldCache+.swift:112`）。`accumulateBestMatch` 只在 `bestMatchRank`（0，仅原生 `.framework` 能拿到）时提前返回。其余情况（plain dylib、bundle，即 rank 2/4）要遍历主缓存加全部子缓存的约 4000+ 个镜像并逐个构造 `MachOFile`，而迁移前 `first(where:)` 命中即停。`FullDyldCache.machOFile(by:)`（L133/186）继承同样的全遍历。 + +6. ⚠️ **打印器每个成员都 materialize 一整棵树**（`SwiftDeclarationPrinter.swift:438`）。`printVariable` / `printFunction` / `printSubscript`（L438/448/458）、扩展 where 子句循环（L261）、`+Members.swift`（L42/65）各调一次 `.materialize()`，而该方法"每次调用都返回新实例"。SwiftUI 规模的 interface 导出约 10^5 次瞬时建树——这恰是本次迁移要消灭的动作。 + +7. ⚠️ **`demangledNode(for:in:)` 丢失 per-symbol 记忆化**（`SymbolIndexStore.swift:856`）。旧实现返回 `demangledNodeBySymbol` 里缓存的实例，新实现是 `demangledNodeReference(for:in:)?.materialize()`，每次重建。`MetadataReader.demangleSymbol(for:in:)` 直接转发它，且仍在 dump 路径的逐符号循环中被调用（`ClassDumper:271,455`、`ProtocolDumper:151`、`ProtocolConformanceDumper:113,176`）。 + +8. **`structuralHash` 每个文本节点分配一个 `String`**。既有台账第 4 条。**修复位置在上游**（应哈希零拷贝的 `textUTF8` 而非 `text`），本仓库无法绕开。 + +9. **`ABIKey.make` 每个 key materialize 整棵树**。既有台账第 7 条。 + +### 内存 + +10. ⚠️ **`DemangledSymbol` 钉住整张 per-image 符号表**（`DemangledSymbol.swift:12`）。`Storage.demangledSymbol(atRow:)` 交出的每个值都持有共享的 `[Symbol]` 缓冲（SwiftUI 约 20 万行，每行 32 字节加一个堆分配的 mangled name 字符串，合计数十 MB），而这些值按值散布在声明模型各处（`Accessor.symbol`、`FunctionDefinition.symbol`、`TypeDefinition.deallocatorSymbol` / `destructorSymbol`）。 + + 后果直接冲击本 PR 新增的 `removeSubIndexer(_:)`——该接口的存在理由正是"让 per-image 内存真正被回收"，但只要调用方还留着**一个** `FunctionDefinition` 或 `Accessor`，整张表及其全部名字字符串就释放不掉。迁移前 `DemangledSymbol` 内联单个 `Symbol`（一个字符串），同样的残留只钉住几十字节。 + + `AGENTS.md` 只记录了 `NodeStore` 的"活声明保活其 store"模型，符号表这层钉住是新增且未记录的。 + +### 对外 API + +11. ⚠️ **`Symbol` 删除公开成员但未升版本、未写 changelog**（`Symbol.swift:17`）。删掉了公开的 `nlist` 属性与 `init(offset:name:nlist:)`，仓库外调用方升级后编译失败，而 `Version.swift` 未升、`Changelogs/` 无条目。 + +12. **`Symbol.isExternal` 在符号表里恒为 `false`**。既有台账第 8 条，与上一条同源：采集局部符号的循环已用 `where … && !symbol.nlist.isExternal` 过滤，导出符号循环走 `isExternal: false` 默认值，故 `if !symbol.isExternal` 守卫是死代码，且字段注释与实际不符。 + +13. **两个公开查询 API 的字典键从结构相等翻成身份相等**。既有台账第 3 条。本轮复核**确认其严重性低于初判**：`allOpaqueTypeDescriptorSymbols(in:)` 在整个仓库（含 `main`）**零调用方**；`memberSymbols(of:excluding:in:)` 的唯一调用方（`SwiftDeclarationIndexer.swift:684`）是遍历而非下标查询；真正的不透明类型解析路径 `opaqueTypeDescriptorSymbol(for:in:)` 已在 `db7105b` 中被刻意改为结构化查找。残留风险仅为"将来有人下标查询时静默拿到空"。 + +### 测试与文档 + +14. ⚠️ **迁移的核心不变量没有断言**（`SymbolIndexStoreFixtureTests.swift:42`）。`buildPipelineStaysOffGlobalNodeCache` 只是自己调了两次 `demangleAsNodeTransient` 比对结果，**完全没有断言 `buildStorage` 的行为**。把 `buildStorageImpl` 改回 `demangleAsNode`（即 Stage-1 那个回归），该测试照样绿。唯一能捕获的断言在 `Tests/IntegrationTests/` 的 `NodeCache.shared.count` 增量里，而 `AGENTS.md` 禁止 agent 与 CI 运行该目录。同一缺口也覆盖 `MetadataReader` / `RuntimeFieldLayoutBackend` / `TypedDumper` / `ClassHierarchyDumper` 的 `Node.createTransient` 回归——`AGENTS.md` 把它列为硬规则，但背后没有测试。 + +15. **既有台账第五节（第 12 条）的前提已过期**。该节称本分支落后 `main` 五个提交、`AGENTS.md` 冲突，需"先 rebase 再谈合并"。实测 `git log origin/feature/node-store-migration..origin/main` 为空，分支 `AGENTS.md` 已同时包含 `main` 的 `--enum-layout-template` 章节与新的 NodeStore 段落，所述冲突不存在。 + + 但同一节压着一条**仍然成立**的注意事项：`main` 的 `TransformerOptionGroup` 与本分支的 `DemangleResolver` / `printSemantic` / `FieldDefinition.typeNode` 改动之间的交互从未被跑过。该注意事项需要在修订该节时保留，否则会随过期前提一起被读成"已处理"。 + +### 代码卫生 + +16. **`ProtocolConformanceDumper` 里 `case .symbol` 分支仍在 materialize**。既有台账第 10 条。 + +17. **两处 `throws` 是迁移残留**。既有台账第 11 条。 + +## 四、明确判定为"无需处理" + +- **`Package.swift` 将 swift-demangling 指向 `branch: "feature/node-store"`**。审查将其报为阻断合并的缺陷（下游按版本依赖会解析失败、构建不可复现）。经确认这是**开发期的预期状态**——本库与其 demangling 依赖正在同步迁移，合并时会换回 `from:` 版本要求，期间接受上述代价。不作为问题跟踪。 + +## 五、与既有台账的对照 + +| 本文条目 | `NodeStoreMigrationOpenIssues.md` | 关系 | +| --- | --- | --- | +| 一.1（`write` witness） | — | 本轮新发现，已闭环 | +| 一.2（`Codable`） | — | 本轮新发现，已闭环 | +| 二 / 三.2（打印路径跳转） | — | 本轮新发现，附实测；同时更正台账第 6 条的 `executeWithinStackBudget` 前提 | +| 三.1 | 第 6 条 | 同一问题 | +| 三.3、三.5、三.6、三.7 | — | 本轮新发现 | +| 三.4 | 第 9 条 | 同一问题 | +| 三.8 | 第 4 条 | 同一问题 | +| 三.9 | 第 7 条 | 同一问题 | +| 内存 10 | — | 本轮新发现 | +| API 11 | — | 本轮新发现（台账第 8 条只覆盖 `isExternal` 死代码，未覆盖公开成员删除） | +| API 12 | 第 8 条 | 同一问题 | +| API 13 | 第 3 条 | 同一问题，本轮补充了严重性复核 | +| 测试 14 | — | 本轮新发现 | +| 文档 15 | 第 12 条 | 指出该条前提已过期 | +| 卫生 16、17 | 第 10、11 条 | 同一问题 | + +### 结论冲突（需要人工裁定） + +既有台账**第 5 条**称 `memberSymbols(of:for:node:)` "从 O(1) 退化成线性扫描 + 逐候选全树比对"。本轮有 4 条同类候选指向该位置，**全部被 verifier 证伪**——理由是"原本是一次哈希查找"这一前提与代码不符，且所称的倍数不存在。两方结论直接冲突，本文不作判断，需要人工核对后决定保留哪一方。 + +（本轮其余被证伪的 3 条均为风格类：`DefinitionBuilder` 重复构造 key、两处多余 `throws`、`StructuralNodeReferenceKey` 的模块归属。其中后两条与台账第 11 条、既有讨论重合，故仍列在本文第三节。） diff --git a/Documentations/README.md b/Documentations/README.md index c434db9a..f81b4e77 100644 --- a/Documentations/README.md +++ b/Documentations/README.md @@ -76,3 +76,4 @@ required by `Version.swift`'s bump contract). | [DeclarationModelMemoryFootprint.md](Internal/DeclarationModelMemoryFootprint.md) | NodeStore 迁移后的声明模型内存足迹量测：`TypeDefinition` 1272 字节的逐属性构成（两份 `TypeContextWrapper` 占 74%）、`TypeContextWrapper` 按最大 case `Class` 定尺的原因、`parentContext` 是被当成永久字段的临时值、mini-store 增殖与 `MetadataReaderCache` 仍持 `Node` 树；含四项可回收估算（合计 ~8–10%）与「当前不建议实施、应先剖析其余 90%」的结论。 | | [NodeStoreMigrationOpenIssues.md](Internal/NodeStoreMigrationOpenIssues.md) | `feature/node-store-migration` 的**已确认问题清单**（2026-07-28 两轮审查 + 复核）。第一节是前一轮修复自身的两处缺口（Catalyst 降级只覆盖 framework 形态导致 plain dylib 仍平局、`appendRowIfAbsent` 线性扫描），已于 2026-07-29 闭环并保留成因；第二节起为**仍然打开**的：两个公开查询 API 的字典键从结构相等翻成 store 身份相等、`structuralHash` 每文本节点分配 `String`（修复位置在上游）、`memberSymbols` 退化为线性 + 全树比对、build sweep 串行且每符号跨线程往返（上游）、`ABIKey` 每 key materialize，以及四条代码卫生项与 rebase 前置事项。逐条注明成因、影响面与「该在哪里修」。 | | [TaskReports/](Internal/TaskReports/) | Dated per-task fix / investigation reports. | +| [Reviews/](Internal/Reviews/) | Dated code-review records — one file per review event: findings, measurements taken at the time, and the resulting to-do list. Distinct from the topic-organised issue ledgers (e.g. `NodeStoreMigrationOpenIssues.md`), which they cross-reference. | From 03a3972159ab983261af86f329aca2f951f4821e Mon Sep 17 00:00:00 2001 From: Mx-Iris Date: Sun, 2 Aug 2026 15:15:51 +0800 Subject: [PATCH 25/77] perf(MachOSymbols): amortize the build sweep's per-symbol thread hop MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Every `Demangling` entry point probes the calling thread's remaining stack and moves the work to an 8MB worker when less than 2MB is left, blocking on a semaphore until that worker returns. Darwin gives every thread but the main one a 512KB stack, so on the cooperative or libdispatch worker a build runs on, that probe never passes: each of a framework's hundreds of thousands of symbols paid its own thread round trip. Wrap the sweep in `StackSafeExecutor.withLargeStack` (body moved to `buildStorageSweep`, thin shell left in place) so the whole batch pays one hop and every demangle inside runs inline — which is what upstream documents that function for. Measured on 100k real symbols in release on a libdispatch worker: 1317ms -> 701ms, 1.88x. On the main thread, where the probe already passes, the boundary changes nothing (145.8ms vs 145.2ms). Also correct `printSemantic`'s doc comment, which claimed the engine's `print(_:options:)` "runs the recursion inline against a stack floor and pays for a worker only for a tree that actually reaches it". It does not: it routes through `executeWithUncheckedSendability`, the same probe and the same 2MB floor as `execute`, only without the Sendable checking a generic Target cannot satisfy. That per-call hop is upstream's deliberate trade in 7b86137 — before it the printer recursed unguarded and a deeply nested generic really did overflow a 512KB worker — and it is paid down at a batch boundary, never at the call site: a wrapper around a single call saves exactly the hop it adds. No code change there. --- Sources/MachOSymbols/SymbolIndexStore.swift | 29 +++++++++++++++++++ .../Extensions/Node+.swift | 29 ++++++++++++------- 2 files changed, 48 insertions(+), 10 deletions(-) diff --git a/Sources/MachOSymbols/SymbolIndexStore.swift b/Sources/MachOSymbols/SymbolIndexStore.swift index f8078b95..408f02fa 100644 --- a/Sources/MachOSymbols/SymbolIndexStore.swift +++ b/Sources/MachOSymbols/SymbolIndexStore.swift @@ -325,9 +325,38 @@ public final class SymbolIndexStore: SharedCache, @unc return buildStorageImpl(for: machO, progressContinuation: nil) } + /// Batch boundary for the sweep's per-symbol demangling. + /// + /// `demangleAsNodeTransient` — like every `Demangling` entry point — + /// probes the *calling* thread's remaining stack and moves the work to an + /// 8MB worker when less than 2MB is left, blocking on a semaphore until + /// that worker returns. Darwin gives every thread except the main one a + /// 512KB stack, so on the Swift Concurrency cooperative worker or + /// libdispatch worker a build runs on, that probe never passes: without a + /// batch boundary each of a framework's hundreds of thousands of symbols + /// pays its own thread round trip. + /// + /// One hop here puts the whole sweep on an 8MB thread, where the probe + /// passes and every demangle inside runs inline — which is exactly what + /// `withLargeStack` documents itself for ("indexing every symbol of a + /// binary, say"). It is not a second guard: the probe and its 2MB floor + /// are unchanged, this only stops the answer from being "no" every time. + /// + /// Wrapping the sweep's *call sites* instead would be a no-op — a hop that + /// covers one demangle saves the one it replaces and nothing else. The + /// saving is `(calls - 1)` hops, so the wrapper has to enclose the loop. private func buildStorageImpl( for machO: MachO, progressContinuation: AsyncStream.Continuation? + ) -> Storage? { + return StackSafeExecutor.withLargeStack { + self.buildStorageSweep(for: machO, progressContinuation: progressContinuation) + } + } + + private func buildStorageSweep( + for machO: MachO, + progressContinuation: AsyncStream.Continuation? ) -> Storage? { var symbolTable: [Symbol] = [] var tableRowByName: [String: UInt32] = [:] diff --git a/Sources/SwiftDeclarationRendering/Extensions/Node+.swift b/Sources/SwiftDeclarationRendering/Extensions/Node+.swift index 7866f98a..00f8d2ac 100644 --- a/Sources/SwiftDeclarationRendering/Extensions/Node+.swift +++ b/Sources/SwiftDeclarationRendering/Extensions/Node+.swift @@ -64,16 +64,25 @@ extension DemanglingNode { /// A deeply nested generic symbol printed through a `Node` could therefore /// overflow the stack where the identical `NodeReference` call would not. /// - /// The guard is the engine's own `print(_:options:)` rather than a - /// `StackSafeExecutor.execute` wrapper around `printRoot`. `execute` has to - /// assume the worst about every input, so on a 512KB stack — which is what - /// every Swift Concurrency cooperative worker and every libdispatch worker - /// gets — it hands *every* call to a large-stack worker and blocks the - /// caller on a semaphore until that worker returns. `print(_:options:)` - /// runs the recursion inline against a stack floor and pays for a worker - /// only for a tree that actually reaches it. Every `printSemantic` call - /// site sits on a printing hot path, so the difference is one - /// dispatch-and-block per printed declaration. + /// The guard is the engine's own `print(_:options:)`, which routes through + /// `StackSafeExecutor.executeWithUncheckedSendability` — the same probe + /// and the same 2MB floor as `execute`, only without the `Sendable` + /// checking a generic `Target` cannot satisfy. An earlier version of this + /// comment framed the two as a choice, claiming the engine's entry point + /// "runs the recursion inline and pays for a worker only for a tree that + /// actually reaches it"; it does not. Darwin gives every thread but the + /// main one a 512KB stack, so on a cooperative or libdispatch worker the + /// probe never passes and *every* call hops to a large-stack worker and + /// blocks on a semaphore — whichever wrapper the engine happens to use. + /// + /// That cost is upstream's deliberate trade (`swift-demangling` 7b86137): + /// before it, the printer recursed unguarded and a deeply nested generic + /// really did overflow a 512KB worker. It is paid down at a *batch* + /// boundary, never here — a `withLargeStack` around this single call would + /// save exactly the one hop it adds. `SymbolIndexStore.buildStorageImpl` + /// is where the repo does that, because it owns a loop; the printer's own + /// loop lives in `SwiftDeclarationPrinter` and is `async`, which a + /// synchronous wrapper cannot enclose. public func printSemantic(using options: DemangleOptions = .default) -> SemanticString { DemanglingPrinter.print(self, options: options) } From 731659cebbbf87b7e4fadbb781e55ccbb35edf77 Mon Sep 17 00:00:00 2001 From: Mx-Iris Date: Sun, 2 Aug 2026 15:16:03 +0800 Subject: [PATCH 26/77] fix(SwiftDeclaration): stop stored symbols from pinning the shared table MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `DemangledSymbol` holds `symbolTable: [Symbol]`, and an `Array` is a reference to its buffer. That is the right trade for the hundreds of thousands of values a query vends and then drops — it keeps each at 32 bytes instead of carrying a `Symbol` copy — but a few thousand of them are stored in the declaration model (`Accessor.symbol`, `FunctionDefinition.symbol`, `TypeDefinition.deallocatorSymbol` / `destructorSymbol`) and outlive the query, so a single survivor pins the whole table and every mangled name in it. That directly defeats `removeSubIndexer(_:)`, whose reason for existing is reclaiming per-image memory. Add `detachedFromSharedTable()`, which copies the referenced row into a one-row table, and call it at the six storing sites: `DefinitionBuilder`'s four construction points (`Accessor` for variables and subscripts, `FunctionDefinition` for allocators and functions) plus `TypeDefinition`'s two assignments. The query path is deliberately unchanged. Measured on SwiftUI (iOS 18.5) after a full export: 9,872 stored values referenced 9,506 distinct rows — 5.1% of a 185,988-row table — so this trades roughly 0.6MB of small allocations for about 19.9MB of retention. Public API only grows: `@MemberwiseInit`'s generated initializers are untouched, so out-of-repo callers are unaffected. Detaching inside those initializers was considered and rejected for exactly that reason — the hand-written replacement would have been public API surface. `SymbolTableRetentionTests` fails before this change (530 stored symbols, all still referencing the 9,348-row table) and passes after, so a new storing site that forgets to detach turns it red. --- AGENTS.md | 2 +- Sources/MachOSymbols/DemangledSymbol.swift | 29 ++++++ .../Definitions/DefinitionBuilder.swift | 8 +- .../Definitions/TypeDefinition.swift | 4 +- .../SymbolTableRetentionTests.swift | 95 +++++++++++++++++++ 5 files changed, 131 insertions(+), 7 deletions(-) create mode 100644 Tests/SwiftInterfaceTests/SymbolTableRetentionTests.swift diff --git a/AGENTS.md b/AGENTS.md index 7e4281b6..b2cfb563 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -206,7 +206,7 @@ let node = try demangleAsNode("$sSiD") // Returns Node tree let string = node.print(using: .default) // "Swift.Int" ``` -**Symbol indexing (NodeStore-backed)**: `SymbolIndexStore.Storage` no longer retains `Node` class trees. The build sweep demangles each symbol cache-free onto a transient tree (`@_spi(Internals) demangleAsNodeTransient`), classifies on that tree, and interns it into a per-image `NodeStoreBuilder`. Nothing touches the global `NodeCache`, and dropping a `Storage` releases the whole per-image footprint. Storage layout (Stage 3): one flat `symbolTable: [Symbol]` row per unique symbol name (canonical cache-adjusted offset; raw and adjusted offset keys share one row), a parallel `rootNodeIndexByTableRow` array, and every classification index stores 4-byte `UInt32` table rows (member/opaque keys are 4-byte `NodeStore.NodeIndex`); indexes accumulate in final row form during the sweep, so `freeze()` is followed by a plain move, not a conversion pass. `Symbol` is 32 bytes (no `nlist` existential — the external-undefined bit is extracted at collection time into `isExternal`), and `DemangledSymbol` is a 32-byte value (shared-table reference + row + `NodeReference`) whose `symbol` is computed; both are asserted by `compactValueLayouts`. Query APIs construct `DemangledSymbol`/`NodeReference` values at the exit boundary; APIs taking an externally demangled node (`memberSymbols(of:for:node:)` with `Node` or `NodeReference`, `opaqueTypeDescriptorSymbol(for:)`) match keys via `NodeReference.structurallyEquals(_:)` (same-store O(1), cross-store structural walk) — `opaqueTypeDescriptorSymbol(for:)` buckets its entries by `DemanglingNode.identifier` first (`opaqueTypeDescriptorEntriesByMemberIdentifier`), because a full scan there is O(descriptors × printed `some` returns). `demangledNodeReference(for:)` matches on the symbol **name alone** — a demangled tree is a pure function of the name and the flat table holds one row per unique name, so comparing the row's canonical offset against the queried symbol's offset could only reject valid hits (it once rejected every symbol of a dyld-cache image, since `symbols(for:in:)` stamps each vended `Symbol` with the offset it was queried by). Names genuinely outside the build sweep fall back to name-keyed mini stores, minted under one lock so a race cannot hand two callers references into different stores (`lateDemangledNode(forName:)` / `MetadataReader.demangleSymbolReference(for:in:)`). Struct-layout changes in `MachOSymbols` need a `swift package clean` rebuild — SwiftPM incremental builds have been observed linking stale downstream objects (runtime SIGSEGV in `outlined destroy`). See `Documentations/Internal/NodeStoreMigrationPlan.md`. **Declaration values hold `NodeReference` (Stage 5a)**: `DefinitionName.node` (`TypeName`/`ProtocolName`/`ExtensionName`), `Variable/Function/SubscriptDefinition.node`, `FieldDefinition.typeNode` and `ExtensionDefinition.genericSignature` are `NodeReference`, not `Node`. Member definitions reference the per-image symbol store directly (a live definition keeps that store alive — the intended per-image recycling model); metadata-derived trees (`MetadataReader` output) are wrapped via `NodeReference(interning:)` mini stores, and `TypeDefinition.index(in:)` batches all field type trees of one type into a single shared store. The `Name` types customize `Hashable` to STRUCTURAL semantics (`structurallyEquals` + `structuralHash`) because `NodeReference`'s intrinsic `Hashable` is store-identity based and would split equal keys minted into different stores; they are deliberately NOT `Codable` — upstream dropped `Node: Codable` (a mangled symbol already is the tree's serialized form: smaller, ABI-stable across toolchains, and sharing-preserving through the round trip), and nothing persists these values. A consumer that ever needs to should persist `mangleAsString(node)` and read it back with `demangleAsNode(_:)` rather than reintroducing a node encoding. **Any `Dictionary`/`Set` keyed on a `NodeReference` whose keys and lookups can come from DIFFERENT stores must key on `StructuralNodeReferenceKey` (`Sources/MachOSymbols/`, next to the mini stores it reconciles), never a bare `NodeReference`.** Different stores are the norm, not the exception: `demangledNodeReference(for:)` falls back to a mini store, and `NodeReference(interning:)` — every `MetadataReader`-derived tree — mints a fresh private store per call by design. Everything that mixes those with image-store references is keyed structurally: the override/vtable lookups (`methodDescriptorLookup` / `vtableOffsetLookup` in `TypeDefinition.index` → `DefinitionBuilder`), `DefinitionBuilder`'s accessor grouping and merged-thunk dedup (`accessorsByNode` / `canonicalIndexByFunctionNode` / `canonicalIndexByAllocatorNode`), and every `visitedNodes` "already claimed this symbol" set (`ExtensionDefinition` / `ProtocolDefinition` / `TypeDefinition` / `OverrideSymbolMatcher` / `ProtocolConformanceDumper`). Under bare keys these fail silently and differently: the override lookups drop the `override` keyword + vtable-offset comment (the Stage 5a regression fixed 2026-07-25), a subscript's getter and setter split into two buckets so the setter-only one is discarded, merged thunks emit their `func`/`init` twice, and a witness can be claimed twice. Bare-`NodeReference` keys are only safe for grouping/dedup WITHIN one `memberSymbols` batch (single hash-consed store, where structural equality coincides with index equality). **Transient demangling everywhere else (Stage 5c)**: `MetadataReader`, `RuntimeFieldLayoutBackend`, `TypedDumper`, `ClassHierarchyDumper` and `Symbol.demangledNode` use `demangleAsNodeTransient` + `Node.createTransient`, so the global `NodeCache` no longer grows while browsing. Do NOT reintroduce `demangleAsNode`/`Node.create` on unbounded inputs in those paths. +**Symbol indexing (NodeStore-backed)**: `SymbolIndexStore.Storage` no longer retains `Node` class trees. The build sweep demangles each symbol cache-free onto a transient tree (`@_spi(Internals) demangleAsNodeTransient`), classifies on that tree, and interns it into a per-image `NodeStoreBuilder`. Nothing touches the global `NodeCache`, and dropping a `Storage` releases the whole per-image footprint. Storage layout (Stage 3): one flat `symbolTable: [Symbol]` row per unique symbol name (canonical cache-adjusted offset; raw and adjusted offset keys share one row), a parallel `rootNodeIndexByTableRow` array, and every classification index stores 4-byte `UInt32` table rows (member/opaque keys are 4-byte `NodeStore.NodeIndex`); indexes accumulate in final row form during the sweep, so `freeze()` is followed by a plain move, not a conversion pass. `Symbol` is 32 bytes (no `nlist` existential — the external-undefined bit is extracted at collection time into `isExternal`), and `DemangledSymbol` is a 32-byte value (shared-table reference + row + `NodeReference`) whose `symbol` is computed; both are asserted by `compactValueLayouts`. **A `DemangledSymbol` stored into the declaration model must be detached first** (`detachedFromSharedTable()`): the shared `[Symbol]` buffer is the right trade for the hundreds of thousands of values a query vends and drops, but `Array` is a reference to its buffer, so a single stored survivor pins the whole table and every mangled name in it — defeating `removeSubIndexer(_:)`, whose reason for existing is reclaiming per-image memory. Measured on SwiftUI (iOS 18.5): 9,872 stored values referenced 9,506 distinct rows, 5.1% of a 185,988-row table, so detaching trades ~0.6 MB of small allocations for ~19.9 MB of retention. The six storing sites are `DefinitionBuilder`'s four construction points (`Accessor` for variables/subscripts, `FunctionDefinition` for allocators/functions) and `TypeDefinition`'s `deallocatorSymbol` / `destructorSymbol` assignments; `SymbolTableRetentionTests` fails the moment a new one forgets. Do NOT detach on the query path. Query APIs construct `DemangledSymbol`/`NodeReference` values at the exit boundary; APIs taking an externally demangled node (`memberSymbols(of:for:node:)` with `Node` or `NodeReference`, `opaqueTypeDescriptorSymbol(for:)`) match keys via `NodeReference.structurallyEquals(_:)` (same-store O(1), cross-store structural walk) — `opaqueTypeDescriptorSymbol(for:)` buckets its entries by `DemanglingNode.identifier` first (`opaqueTypeDescriptorEntriesByMemberIdentifier`), because a full scan there is O(descriptors × printed `some` returns). `demangledNodeReference(for:)` matches on the symbol **name alone** — a demangled tree is a pure function of the name and the flat table holds one row per unique name, so comparing the row's canonical offset against the queried symbol's offset could only reject valid hits (it once rejected every symbol of a dyld-cache image, since `symbols(for:in:)` stamps each vended `Symbol` with the offset it was queried by). Names genuinely outside the build sweep fall back to name-keyed mini stores, minted under one lock so a race cannot hand two callers references into different stores (`lateDemangledNode(forName:)` / `MetadataReader.demangleSymbolReference(for:in:)`). Struct-layout changes in `MachOSymbols` need a `swift package clean` rebuild — SwiftPM incremental builds have been observed linking stale downstream objects (runtime SIGSEGV in `outlined destroy`). See `Documentations/Internal/NodeStoreMigrationPlan.md`. **Declaration values hold `NodeReference` (Stage 5a)**: `DefinitionName.node` (`TypeName`/`ProtocolName`/`ExtensionName`), `Variable/Function/SubscriptDefinition.node`, `FieldDefinition.typeNode` and `ExtensionDefinition.genericSignature` are `NodeReference`, not `Node`. Member definitions reference the per-image symbol store directly (a live definition keeps that store alive — the intended per-image recycling model); metadata-derived trees (`MetadataReader` output) are wrapped via `NodeReference(interning:)` mini stores, and `TypeDefinition.index(in:)` batches all field type trees of one type into a single shared store. The `Name` types customize `Hashable` to STRUCTURAL semantics (`structurallyEquals` + `structuralHash`) because `NodeReference`'s intrinsic `Hashable` is store-identity based and would split equal keys minted into different stores; they are deliberately NOT `Codable` — upstream dropped `Node: Codable` (a mangled symbol already is the tree's serialized form: smaller, ABI-stable across toolchains, and sharing-preserving through the round trip), and nothing persists these values. A consumer that ever needs to should persist `mangleAsString(node)` and read it back with `demangleAsNode(_:)` rather than reintroducing a node encoding. **Any `Dictionary`/`Set` keyed on a `NodeReference` whose keys and lookups can come from DIFFERENT stores must key on `StructuralNodeReferenceKey` (`Sources/MachOSymbols/`, next to the mini stores it reconciles), never a bare `NodeReference`.** Different stores are the norm, not the exception: `demangledNodeReference(for:)` falls back to a mini store, and `NodeReference(interning:)` — every `MetadataReader`-derived tree — mints a fresh private store per call by design. Everything that mixes those with image-store references is keyed structurally: the override/vtable lookups (`methodDescriptorLookup` / `vtableOffsetLookup` in `TypeDefinition.index` → `DefinitionBuilder`), `DefinitionBuilder`'s accessor grouping and merged-thunk dedup (`accessorsByNode` / `canonicalIndexByFunctionNode` / `canonicalIndexByAllocatorNode`), and every `visitedNodes` "already claimed this symbol" set (`ExtensionDefinition` / `ProtocolDefinition` / `TypeDefinition` / `OverrideSymbolMatcher` / `ProtocolConformanceDumper`). Under bare keys these fail silently and differently: the override lookups drop the `override` keyword + vtable-offset comment (the Stage 5a regression fixed 2026-07-25), a subscript's getter and setter split into two buckets so the setter-only one is discarded, merged thunks emit their `func`/`init` twice, and a witness can be claimed twice. Bare-`NodeReference` keys are only safe for grouping/dedup WITHIN one `memberSymbols` batch (single hash-consed store, where structural equality coincides with index equality). **Transient demangling everywhere else (Stage 5c)**: `MetadataReader`, `RuntimeFieldLayoutBackend`, `TypedDumper`, `ClassHierarchyDumper` and `Symbol.demangledNode` use `demangleAsNodeTransient` + `Node.createTransient`, so the global `NodeCache` no longer grows while browsing. Do NOT reintroduce `demangleAsNode`/`Node.create` on unbounded inputs in those paths. ## Test Environment diff --git a/Sources/MachOSymbols/DemangledSymbol.swift b/Sources/MachOSymbols/DemangledSymbol.swift index 96dd10df..b839807c 100644 --- a/Sources/MachOSymbols/DemangledSymbol.swift +++ b/Sources/MachOSymbols/DemangledSymbol.swift @@ -41,6 +41,35 @@ public struct DemangledSymbol: Sendable { self.demangledNode = demangledNode } + /// Copies the referenced row into a standalone one-row table, so this + /// value stops retaining the shared per-image buffer. + /// + /// The shared table is the right trade for the hundreds of thousands of + /// values a query vends and then drops — they cost 32 bytes each instead + /// of carrying a `Symbol` copy. It is the wrong trade for the handful that + /// outlive the query by being stored in the declaration model + /// (`Accessor.symbol`, `FunctionDefinition.symbol`, + /// `TypeDefinition.deallocatorSymbol` / `destructorSymbol`): a single one + /// of those pins the entire table plus every mangled name in it, which is + /// what `SwiftDeclarationIndexer.removeSubIndexer(_:)` exists to reclaim. + /// Measured on SwiftUI (iOS 18.5): 9,872 stored values referenced 9,506 + /// distinct rows — 5.1% of a 185,988-row table — so detaching them trades + /// roughly 0.6 MB of small allocations for about 19.9 MB of retention. + /// + /// Call this when storing a value into a long-lived declaration, not on + /// the query path. + public func detachedFromSharedTable() -> DemangledSymbol { + return DemangledSymbol(symbol: symbol, demangledNode: demangledNode) + } + + /// Rows in the table backing this value: the whole per-image table for a + /// value straight off a query, `1` once ``detachedFromSharedTable()`` has + /// copied its row out. Exposed so the retention regression test can tell + /// the two apart without reaching into private storage. + package var retainedSymbolTableRowCount: Int { + return symbolTable.count + } + public subscript(dynamicMember keyPath: KeyPath) -> Value { return symbol[keyPath: keyPath] } diff --git a/Sources/SwiftDeclaration/Components/Definitions/DefinitionBuilder.swift b/Sources/SwiftDeclaration/Components/Definitions/DefinitionBuilder.swift index a63b164d..ca49d980 100644 --- a/Sources/SwiftDeclaration/Components/Definitions/DefinitionBuilder.swift +++ b/Sources/SwiftDeclaration/Components/Definitions/DefinitionBuilder.swift @@ -23,7 +23,7 @@ package enum DefinitionBuilder { let symbolOffset = demangledSymbol.base.offset let descriptor = methodDescriptorLookup[StructuralNodeReferenceKey(node)] ?? implOffsetDescriptorLookup[symbolOffset] let vtableOffset = vtableOffsetLookup[StructuralNodeReferenceKey(node)] ?? implOffsetVTableSlotLookup[symbolOffset] - accessorsByName[name, default: []].append(.init(kind: kind, symbol: demangledSymbol.base, methodDescriptor: descriptor, offset: demangledSymbol.offset, vtableOffset: vtableOffset)) + accessorsByName[name, default: []].append(.init(kind: kind, symbol: demangledSymbol.base.detachedFromSharedTable(), methodDescriptor: descriptor, offset: demangledSymbol.offset, vtableOffset: vtableOffset)) } for (name, accessors) in accessorsByName.sorted(by: { $0.key < $1.key }) { @@ -69,7 +69,7 @@ package enum DefinitionBuilder { let symbolOffset = demangledSymbol.base.offset let descriptor = methodDescriptorLookup[StructuralNodeReferenceKey(node)] ?? implOffsetDescriptorLookup[symbolOffset] let vtableOffset = vtableOffsetLookup[StructuralNodeReferenceKey(node)] ?? implOffsetVTableSlotLookup[symbolOffset] - accessorsByNode[subscriptNode, default: []].append(.init(kind: kind, symbol: demangledSymbol.base, methodDescriptor: descriptor, offset: demangledSymbol.offset, vtableOffset: vtableOffset)) + accessorsByNode[subscriptNode, default: []].append(.init(kind: kind, symbol: demangledSymbol.base.detachedFromSharedTable(), methodDescriptor: descriptor, offset: demangledSymbol.offset, vtableOffset: vtableOffset)) } for (_, accessors) in accessorsByNode { @@ -131,7 +131,7 @@ package enum DefinitionBuilder { let symbolOffset = demangledSymbol.base.offset let descriptor = methodDescriptorLookup[StructuralNodeReferenceKey(node)] ?? implOffsetDescriptorLookup[symbolOffset] let vtableOffset = vtableOffsetLookup[StructuralNodeReferenceKey(node)] ?? implOffsetVTableSlotLookup[symbolOffset] - var functionDefinition = FunctionDefinition(node: node, name: "", kind: .allocator, symbol: demangledSymbol.base, isGlobalOrStatic: true, methodDescriptor: descriptor, offset: demangledSymbol.offset, vtableOffset: vtableOffset) + var functionDefinition = FunctionDefinition(node: node, name: "", kind: .allocator, symbol: demangledSymbol.base.detachedFromSharedTable(), isGlobalOrStatic: true, methodDescriptor: descriptor, offset: demangledSymbol.offset, vtableOffset: vtableOffset) if let methodDescriptor = descriptor?.method, methodDescriptor.layout.flags.isDynamic { functionDefinition.attributes.append(.dynamic) } @@ -191,7 +191,7 @@ package enum DefinitionBuilder { let symbolOffset = demangledSymbol.base.offset let descriptor = methodDescriptorLookup[StructuralNodeReferenceKey(node)] ?? implOffsetDescriptorLookup[symbolOffset] let vtableOffset = vtableOffsetLookup[StructuralNodeReferenceKey(node)] ?? implOffsetVTableSlotLookup[symbolOffset] - var functionDefinition = FunctionDefinition(node: node, name: name, kind: .function, symbol: demangledSymbol.base, isGlobalOrStatic: isGlobalOrStatic, methodDescriptor: descriptor, offset: demangledSymbol.offset, vtableOffset: vtableOffset) + var functionDefinition = FunctionDefinition(node: node, name: name, kind: .function, symbol: demangledSymbol.base.detachedFromSharedTable(), isGlobalOrStatic: isGlobalOrStatic, methodDescriptor: descriptor, offset: demangledSymbol.offset, vtableOffset: vtableOffset) if let methodDescriptor = descriptor?.method, methodDescriptor.layout.flags.isDynamic { functionDefinition.attributes.append(.dynamic) } diff --git a/Sources/SwiftDeclaration/Components/Definitions/TypeDefinition.swift b/Sources/SwiftDeclaration/Components/Definitions/TypeDefinition.swift index 376184a4..cb703111 100644 --- a/Sources/SwiftDeclaration/Components/Definitions/TypeDefinition.swift +++ b/Sources/SwiftDeclaration/Components/Definitions/TypeDefinition.swift @@ -287,8 +287,8 @@ public final class TypeDefinition: Definition { // The deallocator drives whether `deinit` is printed at all; the // destructor (only present on classes) is exposed as an extra // address comment. - deallocatorSymbol = symbolIndexStore.memberSymbols(of: .deallocator, for: typeName.name, in: machO).first - destructorSymbol = symbolIndexStore.memberSymbols(of: .destructor, for: typeName.name, in: machO).first + deallocatorSymbol = symbolIndexStore.memberSymbols(of: .deallocator, for: typeName.name, in: machO).first?.detachedFromSharedTable() + destructorSymbol = symbolIndexStore.memberSymbols(of: .destructor, for: typeName.name, in: machO).first?.detachedFromSharedTable() variables = DefinitionBuilder.variables( for: symbolIndexStore.memberSymbols(of: .variable(inExtension: false, isStatic: false, isStorage: false), for: name, node: node, in: machO).map { .init(base: $0, offset: nil) }, diff --git a/Tests/SwiftInterfaceTests/SymbolTableRetentionTests.swift b/Tests/SwiftInterfaceTests/SymbolTableRetentionTests.swift new file mode 100644 index 00000000..5c87760c --- /dev/null +++ b/Tests/SwiftInterfaceTests/SymbolTableRetentionTests.swift @@ -0,0 +1,95 @@ +@_spi(Support) @testable import SwiftDeclaration +@_spi(Support) @testable import SwiftIndexing +import Foundation +import Testing +import MachOKit +@_spi(Internals) @testable import MachOSymbols +@_spi(Internals) import MachOCaches +@_spi(Support) @testable import SwiftInterface +@testable import MachOSwiftSection +@testable import MachOTestingSupport +import MachOFixtureSupport + +/// A `DemangledSymbol` stored in the declaration model must not keep the +/// shared per-image symbol table alive. +/// +/// `SymbolIndexStore` vends values that share one `[Symbol]` buffer, which +/// keeps each of the hundreds of thousands it produces at 32 bytes. Those +/// values are meant to be dropped when the query result is; a few thousand of +/// them are instead stored in the model (accessors, functions, deallocators) +/// and outlive it, and because `[Symbol]` is a reference to its buffer, **one** +/// survivor pins the whole table and every mangled name in it. That directly +/// defeats `SwiftDeclarationIndexer.removeSubIndexer(_:)`, whose reason for +/// existing is releasing per-image memory. Measured on SwiftUI (iOS 18.5) +/// before the fix: 9,872 stored values, 9,506 distinct rows referenced out of +/// 185,988 — about 19.9 MB held for 5.1% of the data. +/// +/// Members are only populated when the printer calls `index(in:)` per type, so +/// the export below is what creates the population under test. +@Suite(.serialized) +final class SymbolTableRetentionTests: MachOFileTests, @unchecked Sendable { + override class var fileName: MachOFileName { .SymbolTestsCore } + + @Test func storedDeclarationSymbolsDoNotRetainTheSharedTable() async throws { + let builder = try SwiftInterfaceBuilder( + configuration: .init(indexConfiguration: .init(showCImportedTypes: false)), + eventHandlers: [], + in: machOFile + ) + try await builder.prepare() + _ = try await builder.printRoot() + + let storage = try #require(SymbolIndexStore.shared.storage(in: machOFile)) + let sharedTableRowCount = storage.symbolTable.count + // Otherwise the assertion below cannot distinguish a detached value + // from a shared one. + try #require(sharedTableRowCount > 1) + + var inspectedSymbolCount = 0 + var symbolsStillHoldingSharedTable: [String] = [] + + func inspect(_ demangledSymbol: DemangledSymbol, describedAs description: String) { + inspectedSymbolCount += 1 + guard demangledSymbol.retainedSymbolTableRowCount != 1 else { return } + guard symbolsStillHoldingSharedTable.count < 10 else { return } + symbolsStillHoldingSharedTable.append(description) + } + + for (typeName, typeDefinition) in builder.indexer.allTypeDefinitions { + for variable in typeDefinition.variables + typeDefinition.staticVariables { + for accessor in variable.accessors { + inspect(accessor.symbol, describedAs: "\(typeName.name).\(variable.name) accessor") + } + } + for subscriptDefinition in typeDefinition.subscripts + typeDefinition.staticSubscripts { + for accessor in subscriptDefinition.accessors { + inspect(accessor.symbol, describedAs: "\(typeName.name) subscript accessor") + } + } + let functions = typeDefinition.functions + + typeDefinition.staticFunctions + + typeDefinition.allocators + + typeDefinition.constructors + for function in functions { + inspect(function.symbol, describedAs: "\(typeName.name).\(function.name)") + } + if let deallocatorSymbol = typeDefinition.deallocatorSymbol { + inspect(deallocatorSymbol, describedAs: "\(typeName.name) deallocator") + } + if let destructorSymbol = typeDefinition.destructorSymbol { + inspect(destructorSymbol, describedAs: "\(typeName.name) destructor") + } + } + + // A model with no stored symbols would pass vacuously. + #expect(inspectedSymbolCount > 0) + #expect( + symbolsStillHoldingSharedTable.isEmpty, + """ + \(symbolsStillHoldingSharedTable.count)+ of \(inspectedSymbolCount) stored symbols still \ + reference the \(sharedTableRowCount)-row shared table; each pins the whole buffer and every \ + name in it. First offenders: \(symbolsStillHoldingSharedTable.joined(separator: ", ")) + """ + ) + } +} From c7b42d6673ca6944c729aeedf59666897e8302c4 Mon Sep 17 00:00:00 2001 From: Mx-Iris Date: Sun, 2 Aug 2026 15:16:14 +0800 Subject: [PATCH 27/77] docs: record the reproduction round over the 2026-07-31 review MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Every measurable item on that review's to-do list was reproduced with runnable code against SwiftUI (iOS 18.5, 185,988 symbol rows) and the current macOS dyld shared cache (3,649 images). Three verdicts changed: - The failed-demangle retry's harm is repeated work, not lock contention. A failing demangle is a fast rejection, cheaper than a successful one, and 8 threads hammering one contend 1.97x where the lock-free path already contends 1.67x. It still never converges, which is the real defect, so the fix stands but the rationale changes. - The dyld full scan is not a regression. The ranking was introduced by 7e5dfcc / cfe40f8 on this branch because leaf names are not unique in a shared cache — `-n SwiftUI` used to resolve to an accessibility bundle carrying no Swift metadata and still exit zero. The 8.4x scan is that correctness fix's price, not something to revert. - `materialize` is not a performance problem: 37,166 calls rebuilding 1,047,919 nodes in 839ms, 0.8% of a 107.9s export — an order of magnitude fewer calls than the review estimated. Settles the ledger's open conflict too. The mechanism claim was right (main really did hash-look up `[name]?[node]`, so the verifier's rebuttal was factually wrong) but the magnitude is negligible: 99.60% of the 6,720 type-node buckets hold a single element. Corrects the review's own measurement table, whose "main thread" row reported a 524KB stack — not the main thread. Verified directly against the two pthread calls the executor uses: main reports 8176KB and runs inline; only non-main threads hop. Reproduction code was one-off and has been deleted; the numbers live here. --- .../Internal/ProjectEvolutionLog.md | 17 ++ .../2026-07-31-node-store-migration-review.md | 133 +++++++++++--- ...2-review-reproduction-and-retention-fix.md | 166 ++++++++++++++++++ 3 files changed, 293 insertions(+), 23 deletions(-) create mode 100644 Documentations/Internal/TaskReports/2026-08-02-review-reproduction-and-retention-fix.md diff --git a/Documentations/Internal/ProjectEvolutionLog.md b/Documentations/Internal/ProjectEvolutionLog.md index c96f4cad..28213504 100644 --- a/Documentations/Internal/ProjectEvolutionLog.md +++ b/Documentations/Internal/ProjectEvolutionLog.md @@ -664,6 +664,23 @@ --- +## 23. 审查清单逐条复现，修掉线程跳转与符号表钉住 + +- **时间段**：2026-08-02。 +- **动机**：[2026-07-31 审查报告](Reviews/2026-07-31-node-store-migration-review.md)留下 17 条待处理项，全部由多智能体审查归并得出，**没有一条做过实测**，且报告自己已经承认对唯一量化过的那条判断错了量级。这一轮的目标不是修完 17 条，而是把每条的真伪与量级钉死，让后续投入落在真问题上；只有性能第一条直接修。 +- **落地**：两处修复 + 一轮全清单实测。 + - `SymbolIndexStore.buildStorageImpl` 的符号 sweep 包进 `StackSafeExecutor.withLargeStack`（函数体移入 `buildStorageSweep`，外层留薄壳）。`withLargeStack` 的收益是 `(批内调用次数 − 1) × 单次跳转成本`，所以必须包住循环——包住单次调用净收益为零，这也是为什么 `printSemantic` 里**不能**加。 + - 新增 `DemangledSymbol.detachedFromSharedTable()`，在存入声明模型的六处调用（`DefinitionBuilder` 的四个构造点 + `TypeDefinition` 的 `deallocatorSymbol` / `destructorSymbol`）。查询路径不动：共享 `[Symbol]` 表对「吐几十万个值随即丢弃」仍是正确取舍，问题只在存下来长期存活的那几千个。公开 API 只增不改。 +- **关键决策**： + - **打印路径的跳转重新定性为上游刻意交易，不修**。查 `swift-demangling` 历史发现 `0.4.3` 的 `NodePrinter.printRoot` 完全没有栈保护（深树在 512 KB worker 上会崩），`7b86137` 把两个公开打印入口强制过 executor 正是为此，且同批给了 `withLargeStack` 作为摊销手段。报告建议的「恢复内联调用」不可行且不应做。 + - **detach 选构造点而非改 `init`**。后者要把 `@MemberwiseInit(.public)` 换成手写 init，而那是公开 API，签名写错会让仓库外调用方编译失败；构造点只有六处且有回归测试守护。 + - **三条判断被实测推翻**：失败名重试的危害在重复计算而非锁争用（8 线程争用 1.97x，无锁路径本身 1.67x）；dyld 全遍历不是退化而是本分支 `7e5dfcc` / `cfe40f8` 正确性修复的代价；`materialize` 占导出总时长仅 0.8%，不构成性能问题。 +- **验证**：`swift test --skip IntegrationTests` 1304 项全绿。关键实测（SwiftUI iOS 18.5，185,988 符号行）：build sweep 10 万符号 1317 ms → 701 ms（1.88x）；符号表钉住从约 21 MB 降到约 2 MB（9,872 个存活值只引用 9,506 行，占表 5.1%）；`memberSymbols` 桶 99.60% 只有 1 个元素，坐实台账第 5 条「机制成立但量级可忽略」。新增回归测试 `SymbolTableRetentionTests`，修复前失败（530 个存储符号全部持有 9,348 行共享表）、修复后通过。 +- **文档**：[TaskReports/2026-08-02-review-reproduction-and-retention-fix.md](TaskReports/2026-08-02-review-reproduction-and-retention-fix.md)、[Reviews/2026-07-31-node-store-migration-review.md](Reviews/2026-07-31-node-store-migration-review.md)（新增第三节实测复现，各条定性按实测更新），`AGENTS.md` 符号索引段落补入「存进声明模型的 `DemangledSymbol` 必须先 detach」硬规则。 +- **对应版本**：0.14.0 之后未发布区间。注意 `Symbol` 删除公开成员（`nlist` 属性、`init(offset:name:nlist:)`）尚未升版本、未写 changelog，发布前必须补。 + +--- + ## 维护约定 1. **每个非平凡批次结束时必须在本文追加/更新一节**（新工作弧新增一节；延续既有弧则在该节 diff --git a/Documentations/Internal/Reviews/2026-07-31-node-store-migration-review.md b/Documentations/Internal/Reviews/2026-07-31-node-store-migration-review.md index 933875a1..2f43f4f1 100644 --- a/Documentations/Internal/Reviews/2026-07-31-node-store-migration-review.md +++ b/Documentations/Internal/Reviews/2026-07-31-node-store-migration-review.md @@ -2,7 +2,7 @@ 本文记录 2026-07-31 对 `feature/node-store-migration` 做的一轮代码审查：**结论、实测数据、以及待处理清单**。 -与 [`NodeStoreMigrationOpenIssues.md`](../NodeStoreMigrationOpenIssues.md) 的分工：那份是按技术主题组织的长期遗留问题台账；本文是**一次审查事件的记录**，包含它自己的实测数据和当时的判断。两者重叠的条目在第五节逐条对照，避免两边各说各话。 +与 [`NodeStoreMigrationOpenIssues.md`](../NodeStoreMigrationOpenIssues.md) 的分工：那份是按技术主题组织的长期遗留问题台账；本文是**一次审查事件的记录**，包含它自己的实测数据和当时的判断。两者重叠的条目在第六节逐条对照，避免两边各说各话。 ## 审查方式 @@ -51,7 +51,7 @@ `NodePrinter.swift` L90-95：`DemanglingPrinter.print` 的函数体就是 `StackSafeExecutor.executeWithUncheckedSendability { ... }`。 -### 实测：跳转率 100% +### 实测：非主线程恒跳，主线程不跳 探针 target 在 `write` 中记录**遍历实际执行所在的线程 ID**，与调用者线程 ID 对比： @@ -64,7 +64,16 @@ caller 17785083 → walk 17785089 HOPPED: true ``` -三种线程上下文全部换线程，无一例外。512 KB 栈的前提成立（实测 524 KB），2 MB 门槛永远过不去，**内联快速路径一次都没走到过**。 +**更正（2026-07-31 复核）**：上表第一行标着 `[main thread]` 却报 524 KB，与 macOS 主线程 8 MB 的事实矛盾——那次测量并没有跑在真正的主线程上（`swift-testing` 的 `@Test` 默认不在主线程）。直接用 C 复核 `StackSafeExecutor` 所依据的那两个 `pthread` 调用： + +``` +main thread reported_size=8176 KB remaining=8168 KB >=2MB: YES (inline) +libdispatch worker reported_size= 524 KB remaining= 523 KB >=2MB: no (HOP) +``` + +所以准确的结论是：**非主线程（cooperative worker、libdispatch worker）恒跳，主线程不跳**。上游的探测逻辑本身没有问题，`pthread_get_stacksize_np` 在主线程上返回的是真实的 8 MB。 + +这个区别影响面很大：从主线程直接渲染（例如 RuntimeViewer 的 UI 线程）根本不付这笔钱；付钱的是跑在 worker 上的路径——而 `buildStorageImpl` 的符号 sweep 正是这种。原文"三种线程上下文全部换线程，无一例外"应作废。 ### 实测：代价 @@ -83,27 +92,86 @@ ### 未验证的部分 -- **"阻塞协作线程导致并发池饿死"没有实测**。上表测的是单线程吞吐，不是池饱和。该结论目前仅由代码推导，需要构造多个并发渲染任务观察实际并行度才能确认。 +- **"阻塞协作线程导致并发池饿死"在打印路径上仍未实测**。上表测的是单线程吞吐，不是池饱和。相邻的 demangle 路径已在第三节实测过并发争用（8 线程仅 1.97x，而完全无锁的路径本身也有 1.67x），可作旁证但不能直接代入打印路径——两者用的是同一个 executor，但打印的单次工作量更大。 - 真实 interface 导出中符号树大小的分布未知，因此整体影响落在 1.14x–2.28x 之间的何处没有数据。 -### 修法 +### 这是上游的刻意交易，不是本次迁移的缺陷 + +比对本分支依赖的 `swift-demangling` 与 `main` 依赖的 `0.4.3`，两条路径的来历完全不同： + +| | `0.4.3`（仓库 `main`） | `feature/node-store`（本分支） | +| --- | --- | --- | +| demangle | 过 executor，每次跳 | 过 executor，每次跳 | +| 打印 | **无任何栈保护，直接递归** | 过 executor，每次跳 | +| 本仓库 build sweep | `concurrentMap` 并行，摊薄跳转 | 串行，不摊薄 | + +- **demangle 的跳转一直存在**：`0.4.3` 的 `DemangleInterface.swift:15` 就是 `StackSafeExecutor.execute`。仓库 `main` 靠 `concurrentMap` 摊薄，本分支改成串行后暴露出来。 +- **打印的跳转是新的**：`0.4.3` 的 `NodePrinter.printRoot` 是无保护的实例方法，`main` 的 `printSemantic` 就是裸递归。上游 `7b86137` 把实例级 `printRoot` 改成 internal，只留两个强制过 executor 的公开入口，理由写在提交信息里——"so a tree's surviving depth can no longer depend on the calling thread's remaining stack"。 + +也就是说，打印路径这笔开销是上游**用性能换栈安全**换来的：在此之前，深嵌套泛型在 512 KB 的 worker 上打印是真的会栈溢出崩溃（这条线最早的提交 `df96bae fix: prevent stack overflow in NodePrinter on non-main threads` 修的就是它，那层保护在后续重构中丢失，到 `0.4.3` 时 `NodePrinter` 又是裸的）。上游在同一批改动里给出了摊销手段 `withLargeStack`，并在其文档注释中点名本仓库这类场景："Use this at a batch boundary — indexing every symbol of a binary, say — so the whole batch pays for at most one thread hop instead of one per call." + +### 修法与落地 + +`withLargeStack` 必须包住**循环**才有意义：它的收益是 `(批内调用次数 − 1) × 单次跳转成本`，包住单次调用则付一次、省一次，净收益为零。全仓库符合条件的同步循环只有一处，已落地： -`StackSafeExecutor.withLargeStack` 包住批量渲染入口即可消除，实测有效（对照组即是）。上游该函数的文档注释也正是这么规定的："Use this at a batch boundary — indexing every symbol of a binary, say — so the whole batch pays for at most one thread hop instead of one per call." 仓库内目前**零处**使用。 +- **`SymbolIndexStore.buildStorageImpl`**（本次改动）：原函数体整体移入 `buildStorageSweep`，外层薄壳包一次 `withLargeStack`。两个调用方（`buildStorage` 与带进度的异步入口）同时受益，每符号一次线程往返降为整批一次。 -同时 `printSemantic` 上的文档注释断言 `print(_:options:)` "runs the recursion inline against a stack floor and pays for a worker only for a tree that actually reaches it"，与实测相反，应一并修正。 + 实测收益（release，10 万个真实 Swift 符号，取自本仓库构建产物；同一串行循环，唯一变量是有无批量边界）： + + | 运行线程 | 无批量边界 | 包 `withLargeStack` | 差异 | + | --- | --- | --- | --- | + | libdispatch worker（512 KB，sweep 实际所在） | 1317.2 ms | 701.5 ms | **1.88x，省 615.7 ms** | + | 主线程（8176 KB） | 145.8 ms / 2 万符号 | 145.2 ms | 无差别（噪声内） | + + 每符号省 6.2 µs。值得注意的是跳转成本（615.7 ms）几乎与 demangle 本身（701.5 ms）等价——**近一半时间花在线程往返上**。主线程两组数据一致，再次确认探测通过时批量边界不产生任何作用，也印证上文对原实测表的更正。 +- **`Node+.swift` 的 `printSemantic` 注释**（本次改动）：原注释断言 `print(_:options:)` "runs the recursion inline against a stack floor and pays for a worker only for a tree that actually reaches it"，与代码相反——它内部就是 `executeWithUncheckedSendability`，与 `execute` 是同一段逻辑，仅少了 `Sendable` 约束。注释已改为如实描述，并说明为何摊销点不在此处。**代码未动**。 +- **渲染循环暂不处理**：打印侧的循环在 `SwiftDeclarationPrinter` 里，是 `async`，同步的 `withLargeStack` 无法包裹。真要摊销需要自定义一个跑在 8 MB 线程上的 `SerialExecutor`，或把打印批次改成同步——两者都是独立的重构。先量清楚真实导出中的调用次数与总开销，再决定是否值得。 > 探针为一次性测量代码，测完已删除。若需长期守护该性质，应整理为正式 benchmark。 -## 三、待处理清单 +## 三、实测复现（2026-08-01） + +清单中每一条可测量的条目都写了复现代码实际跑过。复现代码为一次性程序（临时 instrumentation + 三个临时测试文件），数据落表后已全部删除，仓库只保留第二节的三处正式改动。 + +样本：SwiftUI（iOS 18.5 模拟器，93 MB，185,988 个符号行）、当前 macOS dyld shared cache（3,649 个镜像）。 -按建议优先级排列。标注 ⚠️ 的是本轮新发现（既有台账中没有）。 +| 条目 | 审查报告原本的说法 | 实测结果 | 判定 | +| --- | --- | --- | --- | +| 四.1 build sweep | "大框架首次打开慢数倍" | 10 万符号：1317 ms → 701 ms（包 `withLargeStack` 后） | **成立**，跳转部分已修，**1.88x** | +| 四.3 失败名重试 | "把其他线程全堵在后面" | 单线程 5000 次：失败名 43.4 ms vs 缓存命中 6.9 ms（**6.3x**）；8 线程争用 1.97x，而无锁路径本身也有 1.67x | 机制**成立**，但"堵住其他线程"**未复现**——瓶颈是重复计算，不是锁 | +| 四.5 dyld 全遍历 | 迁移退化 | framework 名 4.73 ms vs plain dylib 39.71 ms（**8.4x**），未命中 38.6 ms | 代价**成立**，但**不是退化**：排名机制是本分支刻意引入的正确性修复 | +| 四.6/四.7 materialize | "约 10^5 次瞬时建树" | SwiftUI 全量导出：37,166 次调用、1,047,919 个节点、839 ms，占导出总时长（107.9 s）的 **0.8%** | 次数比估计**少一个量级**，**不构成性能问题** | +| 内存 10 符号表钉住 | — | 185,988 行 × 32 B = 5.8 MB，加 16.6 MB 名字字符串 ≈ **21 MB**；实测 `Storage` 释放后名字仍可读。导出后长期存活 9,872 个值，只引用 9,506 行（表的 5.1%） | **成立，量级严重；已修（2026-08-02）** | +| API 11 公开成员删除 | — | `nlist` 属性与 `init(offset:name:nlist:)` 已删；`Version.swift` 无改动；`Changelogs/` 无新条目 | **三点全部成立** | +| API 12 `isExternal` 死代码 | — | 185,988 行中 `isExternal == true` 的有 **0** 行 | **成立** | +| API 13 身份键 | — | 同一 mangled name 经两个 mini store，`structurallyEquals` 为真，裸 `NodeReference` 键查询 **MISS**，`StructuralNodeReferenceKey` 命中 | **成立** | +| 测试 14 不变量无断言 | — | 把 sweep 换回 `demangleAsNode`，`MachOSymbolsTests` **19 个测试全绿** | **成立，且比原文更严重**——不只那一个测试抓不住，整个 target 都抓不住 | +| 台账第 5 条 桶扫描 | "O(1) 退化成线性扫描 + 逐候选全树比对" | 6,720 个桶中 **99.60% 只有 1 个元素**，最大 6（`SwiftUI.Coordinator`） | 机制**成立**，量级**可忽略** | + +三条需要修改原判断： + +1. **四.3 的危害说错了方向**。失败的 demangle 是**快速失败**（demangler 在开头就拒绝），比一次成功的 demangle 还便宜；8 线程并发下失败路径 1.97x、无锁路径 1.67x，锁几乎不构成额外争用。真正的问题是它**永不收敛**——每次调用重跑一遍，而缓存命中路径与表内命中一样快（6.87 ms vs 6.79 ms，说明 `Mutex` 本身近乎免费）。修法仍是缓存失败结果，但理由从"解除锁争用"改成"消除重复计算"。 + +2. **四.5 不是退化**。`main` 的 `first(where:)` 命中即停确实更快，但它拿到的可能是错的镜像——叶名在 shared cache 里不唯一，`swift-section --dyld-shared-cache -n SwiftUI` 曾解析到没有 Swift 元数据的 accessibility bundle，输出空 dump 还 exit 0。排名机制由本分支的 `7e5dfcc` / `cfe40f8` 引入来修这个 bug，全遍历是它的代价。优化空间真实存在（按叶名预建索引，把 O(全部镜像) 降到 O(同名镜像)），但这是新优化，不是回退。 + +3. **四.6/四.7 不该按性能问题排序**。0.8% 的占比意味着即使把 `materialize` 全部消灭，导出也快不了 1%。它仍是个设计一致性问题（迁移的目标是消除建树，打印路径却还在建），但不应占用性能预算。 + +> 顺带一个不在清单内、值得单独查的观察：SwiftUI 全量 interface 导出耗时 **107.9 秒**，而其中 `materialize` 只占 0.8%。其余 99% 的去向本轮没有测量。 + +## 四、待处理清单 + +按建议优先级排列。标注 ⚠️ 的是本轮新发现（既有台账中没有）。定性以第三节实测为准。 ### 性能 -1. **build sweep 由并行改为串行，且每符号无条件跨线程往返**（`SymbolIndexStore.swift:431`）。原为 `symbolArray.concurrentMap`，现为单趟顺序循环，且每个符号额外付一次线程池提交 + 信号量等待，无并行摊薄。大框架（SwiftUI 数十万符号）首次打开慢数倍。**影响最大的一条。** - - 注：既有台账第 6 条描述此问题时称"不是打印路径用的 `executeWithinStackBudget`"，但**该入口在当前上游版本中并不存在**，且第二节已实测证明打印路径同样每次跳转。该前提需要更正。 +1. **build sweep 由并行改为串行**（`SymbolIndexStore.swift`）。原为 `symbolArray.concurrentMap`，现为单趟顺序循环。 + - **跨线程往返部分已修**：sweep 循环已包进 `StackSafeExecutor.withLargeStack`（见第二节"修法与落地"），每符号一次线程往返降为整批一次。 + - **串行本身仍未解**，且恢复 `concurrentMap` 不可行——`NodeStoreBuilder.intern` 要求顺序 interning，并行会破坏 hash-consing 的索引分配。要拿回并行需要另行设计（例如分片 builder 后归并），属独立课题。 + - 注：既有台账第 6 条描述此问题时称"不是打印路径用的 `executeWithinStackBudget`"，但**该入口在当前上游版本中并不存在**。该前提需要更正。 -2. **打印路径每次调用跨线程 + 阻塞**（`Node+.swift:78`）。见第二节。注意原建议的"恢复内联调用"**已不可行**——上游把 `NodePrinter` 改成了空 enum（无构造器），`print(_:options:)` 是唯一公开走法。 +2. **打印路径每次调用跨线程 + 阻塞**（`Node+.swift`）。**重新定性：这不是本次迁移的缺陷**，而是上游 `7b86137` 用性能换栈安全的刻意交易——`0.4.3` 的打印路径完全没有栈保护，深树在 512 KB 线程上会崩（详见第二节）。原建议的"恢复内联调用"**不可行且不应做**：上游已把实例级 `printRoot` 收为 internal，正是为了堵死这条路。 + - 本次只修正了 `printSemantic` 上与代码相反的注释，代码未动。 + - 摊销需要在渲染循环处做，而该循环是 `async`，`withLargeStack` 包不进去。待复现量出真实代价后再决定是否为它引入自定义 `SerialExecutor`。 3. ⚠️ **demangle 失败的符号名永不缓存，且在持锁状态下重试**（`SymbolIndexStore.swift:848`）。`buildStorageImpl` 对 demangle 失败的名字仍保留表行但 root index 为 `nil`，于是 `demangledNodeReference` 永远走不到快速路径、落到 `lateDemangledNode`，而后者按契约不缓存失败。`demangledOverrideSymbol` 会为每个类的每个方法遍历候选符号，每遇到一个不可 demangle 的符号就取一次 per-image 锁重跑 demangle（其本身还要跨线程阻塞），把其他线程全堵在后面。迁移前该 miss 路径完全不加锁。 @@ -127,6 +195,16 @@ `AGENTS.md` 只记录了 `NodeStore` 的"活声明保活其 store"模型，符号表这层钉住是新增且未记录的。 + **已修（2026-08-02）**：新增 `DemangledSymbol.detachedFromSharedTable()`，把引用的行复制进单行表。在**存入模型的六处**调用——`DefinitionBuilder` 的四个构造点（variables / subscripts 的 `Accessor`，allocator / function 的 `FunctionDefinition`）与 `TypeDefinition` 的 `deallocatorSymbol` / `destructorSymbol` 两处赋值。查询路径不动：共享表对"吐几十万个值随即丢弃"仍是正确取舍，问题只在存下来的那几千个。 + + 实测（SwiftUI iOS 18.5，一次全量导出后）：长期存活 9,872 个值，引用 9,506 个不同行，占 185,988 行表的 5.1%——用约 0.6 MB 的小分配换回约 19.9 MB 的保留。 + + 公开 API 只增不改：`detachedFromSharedTable()` 是新方法，`Accessor` / `FunctionDefinition` 的 `@MemberwiseInit` 构造器签名一字未动，仓库外调用方不受影响。 + + 回归测试 `Tests/SwiftInterfaceTests/SymbolTableRetentionTests.swift` 随修复一并保留：它索引并导出 `SymbolTestsCore`，断言模型中每一个存下来的 `DemangledSymbol` 的 `retainedSymbolTableRowCount == 1`。修复前该测试失败（530 个存储符号全部引用 9,348 行的共享表），修复后通过。新增的构造点若忘记 detach，它会立刻变红。 + + 横向排查：全仓库 `DemangledSymbol` 类型的存储属性只有五处，四处即上述已修字段，第五处 `DemangledSymbolWithOffset.base` 只出现在 `index(in:)` 的局部变量里，不长期存活。 + ### 对外 API 11. ⚠️ **`Symbol` 删除公开成员但未升版本、未写 changelog**（`Symbol.swift:17`）。删掉了公开的 `nlist` 属性与 `init(offset:name:nlist:)`，仓库外调用方升级后编译失败，而 `Version.swift` 未升、`Changelogs/` 无条目。 @@ -149,22 +227,22 @@ 17. **两处 `throws` 是迁移残留**。既有台账第 11 条。 -## 四、明确判定为"无需处理" +## 五、明确判定为"无需处理" - **`Package.swift` 将 swift-demangling 指向 `branch: "feature/node-store"`**。审查将其报为阻断合并的缺陷（下游按版本依赖会解析失败、构建不可复现）。经确认这是**开发期的预期状态**——本库与其 demangling 依赖正在同步迁移，合并时会换回 `from:` 版本要求，期间接受上述代价。不作为问题跟踪。 -## 五、与既有台账的对照 +## 六、与既有台账的对照 | 本文条目 | `NodeStoreMigrationOpenIssues.md` | 关系 | | --- | --- | --- | | 一.1（`write` witness） | — | 本轮新发现，已闭环 | | 一.2（`Codable`） | — | 本轮新发现，已闭环 | -| 二 / 三.2（打印路径跳转） | — | 本轮新发现，附实测；同时更正台账第 6 条的 `executeWithinStackBudget` 前提 | -| 三.1 | 第 6 条 | 同一问题 | -| 三.3、三.5、三.6、三.7 | — | 本轮新发现 | -| 三.4 | 第 9 条 | 同一问题 | -| 三.8 | 第 4 条 | 同一问题 | -| 三.9 | 第 7 条 | 同一问题 | +| 二 / 四.2（打印路径跳转） | — | 本轮新发现，附实测；已重新定性为上游刻意交易而非迁移缺陷；同时更正台账第 6 条的 `executeWithinStackBudget` 前提 | +| 四.1 | 第 6 条 | 同一问题；跳转部分已修（`withLargeStack`），串行部分仍未解 | +| 四.3、四.5、四.6、四.7 | — | 本轮新发现，均已实测（见第三节） | +| 四.4 | 第 9 条 | 同一问题 | +| 四.8 | 第 4 条 | 同一问题 | +| 四.9 | 第 7 条 | 同一问题 | | 内存 10 | — | 本轮新发现 | | API 11 | — | 本轮新发现（台账第 8 条只覆盖 `isExternal` 死代码，未覆盖公开成员删除） | | API 12 | 第 8 条 | 同一问题 | @@ -173,8 +251,17 @@ | 文档 15 | 第 12 条 | 指出该条前提已过期 | | 卫生 16、17 | 第 10、11 条 | 同一问题 | -### 结论冲突（需要人工裁定） +### 结论冲突（已裁定 2026-07-31） + +既有台账**第 5 条**称 `memberSymbols(of:for:node:)` "从 O(1) 退化成线性扫描 + 逐候选全树比对"。本轮有 4 条同类候选指向该位置，全部被 verifier 证伪，理由是"原本是一次哈希查找"这一前提与代码不符。 + +**裁定：台账的机制描述成立，verifier 的证伪理由不成立。** 逐字核对两侧代码： + +- `main`（`SymbolIndexStore.swift:536`）：`memberSymbolsByKind[$0]?[name]?[node]`——确实是三层哈希查找。verifier 所称"前提与代码不符"这句话本身与代码不符。 +- 本分支（`SymbolIndexStore.swift:728`）：`rowsByTypeNodeIndex.elements.first(where: { storage.nodeStore.reference(at: $0.key).structurallyEquals(node) })`——线性扫描 + 每候选一次结构遍历。 + +**但量级判断上 verifier 的结论方向是对的，理由不同**：被扫描的桶装的是"同一类型名下的不同 type node"，正常情况只有 1 个元素；且 `main` 那次哈希查找本身也要遍历整棵 node 树计算结构哈希，本来就不是免费的 O(1)。所以实际是"一次全树哈希"换成"一次全树结构比对"，量级相当，不存在台账所称的倍数退化。 -既有台账**第 5 条**称 `memberSymbols(of:for:node:)` "从 O(1) 退化成线性扫描 + 逐候选全树比对"。本轮有 4 条同类候选指向该位置，**全部被 verifier 证伪**——理由是"原本是一次哈希查找"这一前提与代码不符，且所称的倍数不存在。两方结论直接冲突，本文不作判断，需要人工核对后决定保留哪一方。 +处置：台账第 5 条保留机制描述，删除"倍数退化"的措辞。桶大小已实测：6,720 个桶中 99.60% 只有 1 个元素，最大 6（见第三节）。 -（本轮其余被证伪的 3 条均为风格类：`DefinitionBuilder` 重复构造 key、两处多余 `throws`、`StructuralNodeReferenceKey` 的模块归属。其中后两条与台账第 11 条、既有讨论重合，故仍列在本文第三节。） +（本轮其余被证伪的 3 条均为风格类：`DefinitionBuilder` 重复构造 key、两处多余 `throws`、`StructuralNodeReferenceKey` 的模块归属。其中后两条与台账第 11 条、既有讨论重合，故仍列在本文第四节。） diff --git a/Documentations/Internal/TaskReports/2026-08-02-review-reproduction-and-retention-fix.md b/Documentations/Internal/TaskReports/2026-08-02-review-reproduction-and-retention-fix.md new file mode 100644 index 00000000..ba7a1c74 --- /dev/null +++ b/Documentations/Internal/TaskReports/2026-08-02-review-reproduction-and-retention-fix.md @@ -0,0 +1,166 @@ +# 2026-08-02 审查清单逐条复现，修掉线程跳转与符号表钉住 + +## 问题 + +[2026-07-31 的审查报告](../Reviews/2026-07-31-node-store-migration-review.md)留下 17 条待处理项，按性能 / 内存 / 对外 API / 测试与文档分组，另有一条「结论冲突（需要人工裁定）」悬而未决。清单本身是多智能体审查归并出来的，条目有优先级排序但**没有任何一条做过实测**——唯一的量化数据是打印路径的线程跳转，而报告自己也标注了「审查报告最初对它的量级判断是错的」。 + +用户的要求分两段：性能第一条（build sweep 的线程跳转）可以直接改，其余全部先做复现。也就是说这一轮的产出不是「修完 17 条」，而是**把每条的真伪与量级钉死**，让后续投入落在真问题上。 + +## 调研 + +### 一、清单条目的逐条核实 + +抽查了 8 条优先级最高的，全部属实，没有误报。核实方式一律是逐字比对 `origin/main` 与本分支的同一处代码，不采信报告的转述： + +| 条目 | 核实结论 | +| --- | --- | +| build sweep 串行化 | `main` 是 `symbolArray.concurrentMap { try? demangleAsNode($0.name) }`，本分支是单趟顺序循环 | +| 每次 demangle / 打印跨线程 | `demangleAsNodeTransient` 与 `DemanglingPrinter.print` 内部都走 `StackSafeExecutor` | +| `Node+.swift` 注释 | 声称 `print(_:options:)` 会内联，而上游 `NodePrinter.swift:91` 是无条件 `executeWithUncheckedSendability`——注释写反了 | +| demangle 失败重试 | `demangledNodeReference` 命中表行但 root index 为 `nil` 时掉进 `lateDemangledNode`，后者在锁里重跑且不缓存失败 | +| `isExternal` 死代码 | 采集循环已用 `!symbol.nlist.isExternal` 过滤，导出符号走默认值 | +| `Symbol` 删公开成员 | `nlist` 属性与 `init(offset:name:nlist:)` 已删，`Version.swift` 未升，`Changelogs/` 无条目 | +| `DemangledSymbol` 钉住符号表 | `symbolTable: [Symbol]` 是数组引用，每个值 retain 整个 buffer | +| 不变量无测试 | `buildPipelineStaysOffGlobalNodeCache` 只调用了两次 `demangleAsNodeTransient` 自比，完全没有观察 `buildStorage` | + +### 二、「结论冲突」的裁定 + +台账第 5 条称 `memberSymbols(of:for:node:)` 从 O(1) 退化成线性扫描，而本轮 4 条同类候选被 verifier 全部证伪，理由是「原本是一次哈希查找这一前提与代码不符」。 + +裁定：**台账的机制描述成立，verifier 的证伪理由不成立**。`main` 的 `SymbolIndexStore.swift:536` 就是 `memberSymbolsByKind[$0]?[name]?[node]`，三层哈希查找，verifier 那句话本身与代码不符。 + +但量级上 verifier 的方向碰巧对，理由不同：被扫描的桶装的是「同一类型名下的不同 type node」，且 `main` 那次哈希查找本身也要遍历整棵 node 树算结构哈希——所以是「一次全树哈希」换成「一次全树结构比对」，不存在台账所称的倍数退化。实测印证了这一点（见下）。 + +### 三、线程跳转的来历——这是本轮最重要的发现 + +报告把打印路径的跳转列为迁移缺陷，建议「恢复内联调用」。查 `swift-demangling` 的提交历史后发现这个定性是错的。 + +`StackSafeExecutor` 改过 8 次，最近三次围绕「什么时候该内联」反复： + +- `6fa6d95` 改成按剩余栈字节判断，worker 给 64 MB，同时引入 `withLargeStack` 作为批量边界； +- `7718889` 把内联门槛提到 worker 自己的栈大小，于是主线程也每次跳——刻意为之，为了让「一棵树能打印多深」不取决于调用线程； +- `7b86137` 把上一步**回退**了，因为 64 MB 门槛让 LLDB 里 `po` 死锁（`po` 只跑当前线程，永远等不到 worker）、优先级反转。改回 8 MB worker + 2 MB 门槛。 + +同一个提交里写死了打印必须过 executor： + +> The only public ways to print are `NodePrinter.print(_:using:)` and the SPI `DemanglingPrinter.print(_:options:)`, both routed through the executor, so a tree's surviving depth can no longer depend on the calling thread's remaining stack. + +再往前查，`0.4.3`（`main` 依赖的版本）的 `NodePrinter.printRoot` 是**没有任何栈保护**的实例方法，`main` 的 `printSemantic` 就是裸递归——代价是深嵌套泛型在 512 KB 的 worker 上打印会真的栈溢出崩溃（这条线最早的提交 `df96bae fix: prevent stack overflow in NodePrinter on non-main threads` 修的正是它，保护在后续重构中丢失）。 + +所以两条路径的来历完全不同：**demangle 的跳转一直存在**（`0.4.3` 的 `DemangleInterface.swift:15` 就是 `StackSafeExecutor.execute`，`main` 靠 `concurrentMap` 摊薄），**打印的跳转是上游新引入的、用性能换栈安全的刻意交易**。上游在同一批改动里给出了摊销手段 `withLargeStack`，文档注释直接点名本仓库这类场景，而仓库内**零处**使用。 + +### 四、探测逻辑本身没有问题 + +报告的实测表三行都写「总栈 524 KB」，包括主线程——与 macOS 主线程 8 MB 的事实矛盾。直接用 C 复核 `StackSafeExecutor` 依据的那两个 `pthread` 调用： + +``` +main thread reported_size=8176 KB remaining=8168 KB >=2MB: YES (inline) +libdispatch worker reported_size= 524 KB remaining= 523 KB >=2MB: no (HOP) +``` + +主线程报 8176 KB，走内联。所以正确结论是「非主线程恒跳，主线程不跳」，原表第一行那次测量并没有跑在真主线程上。这个区别影响面很大：从主线程直接渲染（例如 RuntimeViewer 的 UI 线程）根本不付这笔钱。 + +## 最终方案 + +### 改动一：build sweep 套批量边界 + +`withLargeStack` 的收益是 `(批内调用次数 − 1) × 单次跳转成本`，所以它必须包住**循环**——包住单次调用则付一次省一次，净收益为零。全仓库符合条件的同步循环只有 `SymbolIndexStore.buildStorageImpl` 一处。 + +打印侧的循环在 `SwiftDeclarationPrinter` 里，是 `async`，同步的 `withLargeStack` 包不进去，本轮不动。 + +`Node+.swift` 的注释改成如实描述，**代码不动**——在 `printSemantic` 里包一层是零收益。 + +### 改动二：存入模型的 `DemangledSymbol` 先 detach + +共享 `[Symbol]` 表对「吐几十万个值随即丢弃」是正确取舍（每个值保持 32 字节），对「存进声明模型长期存活的几千个」是错误取舍（一个存活值钉住整表）。方案是在**存入点**转换，查询路径不动。 + +选构造点而不是改 `Accessor` / `FunctionDefinition` 的 init：后者要把 `@MemberwiseInit(.public)` 换成手写 init，而那个 init 是公开 API 的一部分，签名写错会让仓库外调用方编译失败。构造点只有 6 处且有测试守护，风险低得多。 + +## 实际执行 + +### 代码 + +- `Sources/MachOSymbols/SymbolIndexStore.swift` —— 原 `buildStorageImpl` 的函数体整体移入新的 `buildStorageSweep`，外层薄壳包一次 `StackSafeExecutor.withLargeStack`。两个调用方（`buildStorage` 与带进度的异步入口）同时受益。 +- `Sources/SwiftDeclarationRendering/Extensions/Node+.swift` —— 重写 `printSemantic` 的文档注释：说明 `print(_:options:)` 与 `execute` 是同一段逻辑（仅少了泛型 `Target` 无法满足的 `Sendable` 约束）、这笔开销是上游 `7b86137` 的刻意交易、摊销点在批量边界而不在此处。 +- `Sources/MachOSymbols/DemangledSymbol.swift` —— 新增 `detachedFromSharedTable()`（复用现成的 `init(symbol:demangledNode:)`，把行复制进单行表）与 `package` 级的 `retainedSymbolTableRowCount`（供测试区分两种形态，不进公开 API）。 +- `Sources/SwiftDeclaration/Components/Definitions/DefinitionBuilder.swift` —— 四个构造点调用 detach：variables 与 subscripts 的 `Accessor`、allocator 与 function 的 `FunctionDefinition`。 +- `Sources/SwiftDeclaration/Components/Definitions/TypeDefinition.swift` —— `deallocatorSymbol` / `destructorSymbol` 两处赋值调用 detach。 + +公开 API 只增不改，`@MemberwiseInit` 生成的构造器签名一字未动。 + +### 测试 + +`Tests/SwiftInterfaceTests/SymbolTableRetentionTests.swift` 随修复一并保留：索引并导出 `SymbolTestsCore`，断言模型里每一个存下来的 `DemangledSymbol` 的 `retainedSymbolTableRowCount == 1`。测试先于修复写好并确认失败： + +``` +10+ of 530 stored symbols still reference the 9348-row shared table +``` + +修复后通过。将来新增构造点忘记 detach，它立刻变红。 + +### 横向排查 + +全仓库 `DemangledSymbol` 类型的存储属性只有五处：四处即上述已修字段，第五处 `DemangledSymbolWithOffset.base` 只出现在 `index(in:)` 的局部变量里，不长期存活。无遗漏同类。 + +## 验证 + +### 实测数据（复现代码为一次性程序，数据落表后已删除） + +样本：SwiftUI（iOS 18.5 模拟器，93 MB，185,988 个符号行）、当前 macOS dyld shared cache（3,649 个镜像）。 + +| 条目 | 报告原本的说法 | 实测 | 判定 | +| --- | --- | --- | --- | +| build sweep | 「大框架首次打开慢数倍」 | 10 万符号：1317 ms → 701 ms | 成立，**1.88x**，跳转成本几乎与 demangle 本身等价 | +| 失败名重试 | 「把其他线程全堵在后面」 | 失败名 43.4 ms vs 缓存命中 6.9 ms（6.3x）；8 线程争用 1.97x，而无锁路径本身也有 1.67x | 机制成立，**锁争用未复现** | +| dyld 全遍历 | 迁移退化 | framework 4.73 ms vs plain dylib 39.71 ms（8.4x） | 代价成立，**不是退化** | +| materialize | 「约 10^5 次瞬时建树」 | 37,166 次、1,047,919 节点、839 ms，占导出总时长（107.9 s）的 **0.8%** | 次数少一个量级，**不构成性能问题** | +| 符号表钉住 | — | 9,872 个存活值引用 9,506 行，占表的 5.1%；约 21 MB → 约 2 MB | 成立，**已修** | +| `isExternal` | — | 185,988 行中 `true` 的有 0 行 | 成立 | +| 身份键 | — | 裸 `NodeReference` 键 MISS，`StructuralNodeReferenceKey` 命中 | 成立 | +| 不变量无测试 | — | 把 sweep 换回 `demangleAsNode`，`MachOSymbolsTests` **19 个测试全绿** | 成立，比原文更严重 | +| 桶扫描（台账第 5 条） | 「O(1) 退化成线性扫描」 | 6,720 个桶中 **99.60% 只有 1 个元素**，最大 6 | 机制成立，量级可忽略 | + +### 需要改判断的三条 + +1. **失败名重试的危害说反了方向**。失败的 demangle 是快速失败（demangler 在开头就拒绝），比一次成功的 demangle 还便宜；8 线程并发下失败路径 1.97x、完全无锁路径 1.67x，锁几乎不构成额外争用。真正的问题是它**永不收敛**——每次调用重跑一遍。修法不变，理由从「解除锁争用」改成「消除重复计算」。顺带测出 `Mutex` 本身近乎免费：缓存命中的 off-table 路径 6.87 ms 与表内命中 6.79 ms 无差别。 + +2. **dyld 那条不是退化**。排名机制由本分支的 `7e5dfcc` / `cfe40f8` 引入，修的是「叶名在 shared cache 里不唯一」——`swift-section --dyld-shared-cache -n SwiftUI` 曾解析到没有 Swift 元数据的 accessibility bundle，输出空 dump 还 exit 0。全遍历是这个正确性修复的代价。优化空间真实存在（按叶名预建索引），但那是新优化不是回退。 + +3. **materialize 不该按性能问题排序**。0.8% 的占比意味着即使全部消灭也快不了 1%。它仍是设计一致性问题（迁移的目标是消除建树，打印路径却还在建），但不应占用性能预算。 + +### 测试 + +`swift test --skip IntegrationTests`：**1304 个测试全部通过**（含新增的回归测试）。 + +## 偏差 + +### 一、一个假警报花了一轮排查 + +第一次跑全套时红了 148 个，全部是 ABI baseline 偏移不匹配。stash 掉改动后用同一 scratch path 重跑，失败**完全一样**（同样的偏移 285532 vs 269112），确认与本轮改动无关。 + +按 AGENTS.md 的环境漂移条目诊断：fixture 源码里有 `AccessorFunctionReferences.swift`（提交 `51d52c3`），而 7 月 26 日构建的二进制里 `strings | grep -c` 查出来 0 处——是 fixture 二进制比源码旧。重建后全套转绿。 + +排查过程中我先说成「baseline 旧」，方向说反了，随后用 mtime 与 `strings` 更正。 + +### 二、方案沟通绕了四轮 + +最初把改动描述成「给批量渲染入口套 `withLargeStack`」，没有给出 `文件:行号`。用户理解成要改 `Node+.swift` 的 `printSemantic`——而那里恰好有一段注释明确写着「不要在这里包 `StackSafeExecutor`」，所以建议听起来像是要推翻它。用户连问四轮才对齐到 `SymbolIndexStore` 的 sweep 循环，而他从第一轮起的直觉（「这不是加不加都一样吗」）一直是对的。 + +教训是：讨论改动方案时第一句就要给出具体位置和「明确不动的地方」，原理放在后面；用户反复追问同一个技术点时，优先怀疑是位置没对齐，而不是原理没讲透——换一种方式重讲原理只会加深误解。 + +### 三、两处技术假设被数据推翻 + +- 假设失败的 demangle 比成功的慢（因为要「重跑」），实测相反——快速失败更便宜。第一版复现代码据此写了 `#expect(failing > cached)`，直接红了。 +- 第一版复现里挑的「可 demangle 但不在表里」的名字（`$sSi4main1AVSgSayADGSgtcfC`）其实 demangle 不了，导致两组对照实际上都是失败路径。改成从候选列表里挑第一个真正合法的才拿到干净数据。 + +两次都是先写断言再看数据导致的，正确顺序应当是先测量再决定断言什么。 + +## 未处理 + +清单剩余条目全部保留在[审查报告](../Reviews/2026-07-31-node-store-migration-review.md)第四节，定性已按本轮实测更新。其中三条有明确后续方向： + +- **build sweep 的串行本身**——恢复 `concurrentMap` 不可行（`NodeStoreBuilder.intern` 要求顺序 interning），要拿回并行需要分片 builder 后归并，属独立课题。 +- **打印路径的摊销**——需要自定义一个跑在 8 MB 线程上的 `SerialExecutor`，或把打印批次改成同步。先有 0.8% 这个数打底，优先级不高。 +- **`Version.swift` 未升 + 无 changelog**——`Symbol` 删了公开成员，发布前必须补。 + +另有一个不在清单内、值得单独查的观察：SwiftUI 全量 interface 导出耗时 **107.9 秒**，而 `materialize` 只占 0.8%，其余 99% 的去向本轮没有测量。 From 8fcc85f98321dc30907dea98cf8242be4f26d358 Mon Sep 17 00:00:00 2001 From: Mx-Iris Date: Mon, 3 Aug 2026 08:27:09 +0800 Subject: [PATCH 28/77] deps: require swift-demangling 0.5.0 instead of the feature branch MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `feature/node-store` has merged into swift-demangling's `main` and shipped as 0.5.0, so the development-period branch requirement can be settled. Pinning to a branch made the package unresolvable for any version-based consumer and left builds non-reproducible; the 2026-07-31 review had already recorded this as the expected state to swap back at merge time. The bump also closes a hazard that only existed while the branch was pinned: the printer's recursion budget was 512 there, which truncated deeply nested SwiftUI symbols to `<>` — and those printed strings are used as dictionary keys, so two types truncating at the same depth collapsed onto one key. 0.5.0 restores the 768 limit upstream. Verified: 0.5.0 is at swift-demangling's `main` tip (caacfb9), `swift package resolve` in a sibling-free checkout picks 0.5.0, and `swift build` succeeds. --- Package.swift | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/Package.swift b/Package.swift index 1b19c62e..2a3abdf4 100644 --- a/Package.swift +++ b/Package.swift @@ -213,7 +213,7 @@ extension Package.Dependency { ), remote: .package( url: "https://github.com/MxIris-Reverse-Engineering/swift-demangling", - branch: "feature/node-store", + from: "0.5.0", ), ) From f8698d0d4ecd4869927b2a9614891707a6e2b2dd Mon Sep 17 00:00:00 2001 From: Mx-Iris Date: Mon, 3 Aug 2026 08:27:09 +0800 Subject: [PATCH 29/77] docs: record the PR #97 review round and its adjudications MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Adds `Reviews/2026-08-02-node-store-migration-pr97-review.md` for the `/code-review max` round over PR #97: the 8 findings absent from both prior records (printer recursion budget, `indexExtensions` losing `await`, `withLargeStack` holding a thread for the whole sweep, 26 per-name interning sites, two unsound test assertions, the unanchored dyld framework-shape test, and an unmemoized `distributedFunctionNodes`), the two items closed this round, the two adjudicated as won't-fix, and a cross-reference table against the 2026-07-31 record and the topic ledger. Two of this round's own conclusions are retracted in the record: the dependency pin was already adjudicated as an expected development state, and the `memberSymbols` "O(1) regression" does not hold — `Node.hash(into:)` recursed over children before the migration, so that lookup was never free, and buckets measure 99.60% single-element. Updates the ledger accordingly: the public-query-key issue is marked won't-fix (the store is `@_spi` at type level, and the sole in-package caller iterates), entry 5 now carries the measurements inline so its wording cannot mislead again, entry 4 records that 0.5.0 reworked `structuralHash` but kept the per-text-node `String` allocation, and entry 12's expired rebase premise is split from the two notes under it that still hold. --- .../Internal/NodeStoreMigrationOpenIssues.md | 42 ++-- ...-08-02-node-store-migration-pr97-review.md | 203 ++++++++++++++++++ .../2026-08-03-pr97-review-triage.md | 72 +++++++ Documentations/README.md | 2 +- 4 files changed, 302 insertions(+), 17 deletions(-) create mode 100644 Documentations/Internal/Reviews/2026-08-02-node-store-migration-pr97-review.md create mode 100644 Documentations/Internal/TaskReports/2026-08-03-pr97-review-triage.md diff --git a/Documentations/Internal/NodeStoreMigrationOpenIssues.md b/Documentations/Internal/NodeStoreMigrationOpenIssues.md index 10b68cca..525218c2 100644 --- a/Documentations/Internal/NodeStoreMigrationOpenIssues.md +++ b/Documentations/Internal/NodeStoreMigrationOpenIssues.md @@ -4,7 +4,9 @@ 产生方式：2026-07-28 对该分支做了两轮代码审查 + 一轮结论复核。第一轮的复核记录见 [TaskReports/2026-07-28-review-verification-and-fixes.md](TaskReports/2026-07-28-review-verification-and-fixes.md)，其中三条已修（`printSemantic` 换用引擎预算入口、`registerRow` 去重、dyld 缓存镜像选择的 Catalyst 平局与子缓存遍历）；第二轮又指出前述修复自身的两处缺口，已于 2026-07-29 补完，见 [TaskReports/2026-07-29-catalyst-rank-and-row-dedup-followup.md](TaskReports/2026-07-29-catalyst-rank-and-row-dedup-followup.md)。 -第一节记录那两条已闭环的缺口（保留成因以备回溯），**第二节起才是仍然打开的**。 +此后又有两轮独立的审查事件，各自的记录见 [Reviews/2026-07-31-node-store-migration-review.md](Reviews/2026-07-31-node-store-migration-review.md) 与 [Reviews/2026-08-02-node-store-migration-pr97-review.md](Reviews/2026-08-02-node-store-migration-pr97-review.md)。两份审查记录带有本台账没有的**实测数据**与**裁决结论**；本台账与它们冲突时，以审查记录为准，并回头修订本台账（第 3、5、12 条即因此修订过）。 + +第一节记录那两条已闭环的缺口（保留成因以备回溯）。**第二节起是仍然打开的**，其中已被裁决为"不修"的条目就地标注，不再删除，以便后续审查对照跳过。 --- @@ -55,17 +57,19 @@ ## 二、公开 API 语义问题 -### 3. 两个公开查询 API 的字典键从结构相等翻成了身份相等 +### 3. ~~两个公开查询 API 的字典键从结构相等翻成了身份相等~~ —— 已裁决：不修（2026-08-03） `memberSymbols(of:excluding:in:)` 与 `allOpaqueTypeDescriptorSymbols(in:)` 原本返回 `OrderedDictionary`。`Node` 的 `==` 是结构相等，所以外部调用方拿任意来源的节点做下标查询都能命中。现在键是 `NodeReference`，其 `==` 为 `store === store && index == index`。调用方用自己 demangle 出来的节点查询会**恒定返回 nil，且没有任何编译错误**。 -仓库内部这两个 API 只被遍历、从不下标查询，所以测试全绿也发现不了。`StructuralNodeReferenceKey` 这套处理施加到了所有内部集合上，唯独漏了这两个**逃逸到外部**的面。 +**裁决：不修。** 依据是 `SymbolIndexStore` 在**类型层面**就是 SPI——`SymbolIndexStore.swift:13-14` 带 `@_spi(ForSymbolViewer)` 与 `@_spi(Internals)`。成员要被访问必须先能命名该类，而命名它必须带对应的 `@_spi(...) import`，所以 SPI 性由类继承而来（逐个方法标注是多余的）。契约既然只对包内与已知 SPI 消费方成立，保证包内正确即可。 -现状缓解：扫过 RuntimeViewer 的 `main` 与 `feature/node-store-adoption`，两条分支都没有调用这两个 API，所以目前没有现实触发者。 +包内正确性已核实： -**正确修法**：要么改成 vend `StructuralNodeReferenceKey`（或干脆 `Node`）作键，要么不暴露裸字典、改提供一个查询方法。 +- `memberSymbols(of:excluding:in:)` 包内唯一调用点 `SwiftDeclarationIndexer.swift:663`，在 `:684` 只做 `for (node, memberSymbols) in memberSymbolsByName` 遍历，全程无下标查询；返回字典的键全部出自同一个 `storage.nodeStore`，同 store 内下标相等本就是正确的去重语义。 +- `allOpaqueTypeDescriptorSymbols(in:)` 在 `Sources/` 与 `Tests/` 中**零调用点**。 +- RuntimeViewer 的 `main` 与 `feature/node-store-adoption` 两条分支均未调用这两个 API。 -**修复位置**：本仓库 `Sources/MachOSymbols/SymbolIndexStore.swift`。 +若将来要重新打开：正确修法是 vend `StructuralNodeReferenceKey`（或 `Node`）作键，或不暴露裸字典而改提供查询方法；修复位置在本仓库 `Sources/MachOSymbols/SymbolIndexStore.swift`。 --- @@ -81,13 +85,20 @@ **修复位置**：**上游 `swift-demangling`** 的 `Sources/Demangling/Store/NodeReference.swift`，不是本仓库。本仓库这一侧无法绕开。 -### 5. `memberSymbols(of:for:node:)` 从 O(1) 退化成线性扫描 + 逐候选全树比对 +**上游 `0.5.0` 状态（2026-08-03 核对）：仍然打开。** `structuralHash` 已重写为委托给 `structuralDigest()`——显式帧栈迭代 + 按节点下标记忆化（`digestByIndex`），重复子树只哈希一次，是实打实的改进。但 `nodeContents` 依旧是 `.text(store.text(offset:length:))`，而 `seededDigestHasher` 直接 `hasher.combine(contents)`，所以**每个文本节点仍然分配一个 `String`**。 -迁移前是 `memberSymbolsByKind[$0]?[name]?[node]`，一次哈希查找。现在两个重载都走 `rowsByTypeNodeIndex.elements.first(where: { …structurallyEquals(node) })`——对桶里每个键做一次结构化树遍历直到命中。 +### 5. `memberSymbols(of:for:node:)` 改为线性扫描 + 逐候选全树比对（量级可忽略，属可选优化） -`TypeDefinition.index` 会为 allocator、变量、静态变量、函数、静态函数、下标各调一次，所以每个被索引的类型付 6 × 桶大小次结构遍历。 +迁移前是 `memberSymbolsByKind[$0]?[name]?[node]`，一次字典查找。现在两个重载都走 `rowsByTypeNodeIndex.elements.first(where: { …structurallyEquals(node) })`——对桶里每个键做一次结构化树遍历直到命中。`TypeDefinition.index` 会为 allocator、变量、静态变量、函数、静态函数、下标各调一次。 -**正确修法**：在 `Storage.init` 里一次性建一份 `[StructuralNodeReferenceKey: NodeStore.NodeIndex]` 旁路索引恢复 O(1)——这正是 `opaqueTypeDescriptorEntriesByMemberIdentifier` 已经用过的手法。 +> **不要把它当回归。** 本条早期措辞称"从 O(1) 退化"，两次审查（2026-07-31 第六节、2026-08-02 第四节）先后纠正过同一处误判，故在此就地写清： +> +> - 迁移前那次字典查找**并不免费**——`swift-demangling` `0.4.5` 的 `Node.hash(into:)` 是 `hasher.combine(children)` 递归，**哈希一次就要走完整棵树**。 +> - 桶里装的是"同一类型名下的不同 type node"，实测 6,720 个桶中 **99.60% 只有 1 个元素**，最大 6（`SwiftUI.Coordinator`）。 +> +> 所以实际是"一次全树哈希"换成"一次全树结构比对"，量级相当，不存在倍数退化。 + +**可选优化**：在 `Storage.init` 里一次性建一份 `[StructuralNodeReferenceKey: NodeStore.NodeIndex]` 旁路索引——这正是 `opaqueTypeDescriptorEntriesByMemberIdentifier` 已经用过的手法。收益上限受限于上述实测，排期时不应优先于真正的回归项。 **修复位置**：本仓库 `Sources/MachOSymbols/SymbolIndexStore.swift`。 @@ -137,12 +148,11 @@ ## 五、分支状态 -### 12. 落后 `main` 五个提交，`AGENTS.md` 两侧都改过 - -`main` 已发布 `0.14.0`，并新增了注释模板的命令行接口（`--enum-layout-template` / `--enum-layout-case-template` / `--enum-layout-byte-template`）及其 `AGENTS.md` 章节。本分支的 `AGENTS.md` 还是 0.14.0 之前的正文，另外加了自己的 NodeStore 段落。直接合并会冲突，而**保留分支侧的粗暴解法会静默回退掉 `main` 的那份文档**。 +### 12. ~~落后 `main` 五个提交，`AGENTS.md` 两侧都改过~~ —— 前提已过期，但压着两条仍然成立的事项 -同理，`ProjectEvolutionLog.md` 里本分支新增的 `## 19.` 把原「引用存储」小节顶成了 `## 20.`，与 `main` 的 `## 20.` 正面撞号；两条新小节都写"将入 0.14.0"，而 0.14.0 已经发布。另有一条指向 `TaskReports/2026-07-25-dyld-cache-image-selection-...` 的链接是死的（实际文件名无 `dyld-` 前缀）。 +**过期部分**（2026-07-31 首次指出，2026-08-03 复测确认）：分支现在只落后 `main` 两个提交（`fed0acf` / `f8c6992`），且二者只改动 `.github/workflows/macOS.yml`；以合并基点为准两侧改动文件**零交集**。所述 `AGENTS.md` 冲突不存在——分支的 `AGENTS.md` 已同时包含 `main` 的 `--enum-layout-template` 章节与新的 NodeStore 段落。原"先 rebase 再谈合并"的处理顺序随之作废。 -此外，`main` 的 `TransformerOptionGroup` 与本分支的 `DemangleResolver` / `printSemantic` / `FieldDefinition.typeNode` 改动之间的交互从未被跑过。 +**仍然成立的两条**： -**处理顺序**：先 rebase 到 `main`，重编演进日志小节号、修死链、对齐 `AGENTS.md`，再谈合并。演进日志的小节应在 rebase 之后补，现在写只会加深冲突。 +1. **`ProjectEvolutionLog.md` 的小节撞号与死链**（2026-08-03 复测仍在）：`## 20.` 出现两次（353 行「引用存储（weak/unowned）对 existential 的宽度修复」、384 行「注释模板的命令行入口」），其后 21/22/23 全部错位；348 行的链接写作 `TaskReports/2026-07-25-dyld-cache-image-selection-and-rv-index-lifecycle.md`，而实际文件名没有 `dyld-` 前缀。撞号是在本分支内部成型的，与 rebase 无关，**可以先修**（原文"演进日志小节应在 rebase 之后补"的理由已不成立）。 +2. **交互从未被跑过**：`main` 的 `TransformerOptionGroup` 与本分支的 `DemangleResolver` / `printSemantic` / `FieldDefinition.typeNode` 改动之间的交互没有任何测试覆盖。这条与 rebase 状态无关，合并前仍需处理。 diff --git a/Documentations/Internal/Reviews/2026-08-02-node-store-migration-pr97-review.md b/Documentations/Internal/Reviews/2026-08-02-node-store-migration-pr97-review.md new file mode 100644 index 00000000..e84adf81 --- /dev/null +++ b/Documentations/Internal/Reviews/2026-08-02-node-store-migration-pr97-review.md @@ -0,0 +1,203 @@ +# 2026-08-02 `feature/node-store-migration`（PR #97）代码审查 + +本文记录 2026-08-02 对 PR #97 跑的一轮 `/code-review max` 审查，以及 08-02～08-03 对其发现的逐条裁决。 + +与 [2026-07-31 那轮](2026-07-31-node-store-migration-review.md) 的关系：本轮是**独立的一次审查事件**，15 条发现中多数与上一轮重合。本文只详述**本轮新增的 8 条**与**本轮产生的状态变更**；重合条目在第五节的对照表里指回上一轮，不重复论述，也不重新计量——上一轮已有实测数据的条目一律以那份数据为准。 + +与 [`NodeStoreMigrationOpenIssues.md`](../NodeStoreMigrationOpenIssues.md) 的分工不变：那份是按技术主题组织的长期台账，本文是一次审查事件的记录。 + +## 审查方式 + +多智能体审查，分维度 finder 出候选 → 每条派独立 verifier 做对抗性验证（默认倾向证伪）→ 第三轮补充扫描（8 个候选存活 2 个）→ 归并去重。 + +规模：66 个改动文件（+3371 / −592）→ 归并后 15 条，另有 3 条因输出条数上限被挤出但同样确认为真。 + +对比基线：`git diff main...feature/node-store-migration`。 + +## 一、本轮新发现（8 条） + +以下 8 条在 2026-07-31 记录与既有台账中**均无**。每条都在本次会话中逐条核实到代码行。 + +### 1. 打印器递归预算从 768 降到 512 ✅ 已闭环 + +`Package.swift` 把 swift-demangling 指向 `branch: "feature/node-store"` 期间，打印器的递归深度上限随之从 `0.4.5` 的 768 变成分支上的 512（且判断条件是 `printDepth < 512`，实际上限 511）。 + +后果不止是显示截断。`node.print(using: .interfaceTypeBuilderOnly)` 的结果在多处被当作**字典键**使用——`SymbolIndexStore.swift` 的 `typeInfoByName` / `memberSymbolRowsByKind[kind][name]`、`DefinitionName.name`、`ClassDumper` 的名字相等判断。SwiftUI 里 `ModifiedContent>` 这类深嵌套链若在 512～768 之间截断，会得到以 `<>` 结尾的字符串；**两个不同类型在同一深度截断就会塌到同一个键上，成员被合并**。测试套件里没有任何以 `too complex` 为内容的基线守着这条。 + +**状态：已随上游 0.5.0 关闭。** 上游把上限恢复为 768，并在源码注释里记下了原因与禁止再降的约束（"Downstream consumers reported `<>` on ordinary SwiftUI and similarly generic-heavy modules under the 512 limit … Do not lower this again without corpus evidence gathered from downstream workloads."）。本仓库随依赖升级到 `from: "0.5.0"` 后自动获得，无需本地改动。 + +### 2. `indexExtensions` 丢失 `await`，从任务挂起退化为线程阻塞 + +`Sources/SwiftIndexing/SwiftDeclarationIndexer.swift:685`。 + +- `main:659`：`let name = await node.print(using: .interfaceTypeBuilderOnly)`——`node` 是 `Node`，绑定到 `Node.print(using:) async`，走 `StackSafeExecutor.executeAsync`（`withCheckedContinuation`，**释放线程**）。 +- 本分支：`node` 变成 `NodeReference`，而 `NodeReference` / `DemanglingNode` 只暴露同步 `print`，于是 `await` 被静默去掉，改走 `DemanglingPrinter.print` → `runOnLargeStack` → `DispatchSemaphore.wait()`——**阻塞一条协作线程池的线程**。 + +这个循环对每个 extension target 执行一次，位于 `async` 的索引流程内。`NodeReference` 上不存在 async 版 `print`，所以调用点看不出任何退化痕迹。 + +### 3. `withLargeStack` 包住整趟 sweep，会占住一条线程整块时长 + +`Sources/MachOSymbols/SymbolIndexStore.swift:352`。 + +这是**对上一轮那条修复本身的观察**，不是否定它：上一轮把每符号一次线程往返摊销成整批一次（实测 1317 ms → 701 ms），方向正确。但 `buildStorageImpl` 现在返回 `StackSafeExecutor.withLargeStack { self.buildStorageSweep(...) }`，在 512 KB 栈的线程上探测必然失败，于是**调用线程在信号量上被卡住整趟 sweep 的时长**（数秒）。 + +而 `SharedCache.resolve` 是刻意把构建放在锁外的，好让不同镜像并行构建；于是同时准备 N 个镜像 = N 条协作线程池线程各被停数秒。池大小约等于核数，镜像数超过核数时无关任务也被拖住。`SwiftDeclarationIndexer.prepare()` 是 `async`，`prepareWithProgress` 可从 `Task` 抵达，所以这条路径真实可达。 + +正确形态是第三种：走异步入口（挂起而非阻塞），或使用专用线程。 + +### 4. `NodeReference(interning:)` 在批量路径上被逐个调用（26 处） + +分布：`SwiftDeclaration/Extensions.swift` 14 处、`SwiftIndexing/SwiftDeclarationIndexer.swift` 6 处、`SwiftSpecialization/` 4 处、`SwiftDeclaration/Components/Definitions/ProtocolDefinition.swift` 1 处、`MachOSymbols/StructuralNodeReferenceKey.swift` 1 处。 + +上游对该构造器的文档原话是"一次调用一块 arena，所以这是批量场景下的错误工具——去重与紧凑都是 arena 的属性，给每棵树各开一块 arena 就把两者都放弃了"，并给出实测：300 个引用指向 3 个唯一符号时，逐个 interning 是 300 条目 / 59,700 字节，共用一个 builder 是 3 条目 / 541 字节。 + +连带副作用：没有两个名字共享 arena，于是 `structurallyEquals` 的 `store ===` 快路径**永不触发**，每次名字相等判断都要走完整棵树。 + +修法与 `TypeDefinition.index` 对字段类型树的做法一致——每个镜像共用一个 builder。 + +> 与 `main` 的关系：`main` 没有这个形态，但有它自己的病（全局 `NodeCache` 只涨不落），而那正是本次迁移要治的。所以这是**代价而非退步**，只是这个代价可以不付。 + +### 5. `buildPipelineStaysOffGlobalNodeCache` 的断言本身不成立 + +`Tests/MachOSymbolsTests/SymbolIndexStoreFixtureTests.swift:53`。 + +该测试对 `firstTransientTree.first { $0.children.isEmpty }`（前序首个叶子）断言 `firstLeaf !== secondLeaf`。但 `demangleAsNodeTransient` 的上游文档明确声明：结果"不是规范化的，但也**不是实例互异的**——无参数种类（`.asyncAnnotation`、`.throwsAnnotation`、`.labelList` 等）会解析到进程级 `NodeFactory` 单例"。 + +取样行 `sampleRow` 取的是 `rootNodeIndexByTableRow.firstIndex(where: { $0 != nil })`，**依赖 fixture 的符号顺序**。若该行恰是 ObjC thunk 或 merged function，前序首个叶子就是 `NodeFactory.objCAttribute` / `.mergedFunction` 那个单例，两次取到同一实例，测试在没有任何东西出错的情况下变红。重建 `SymbolTestsCore` 或升级工具链都可能翻转它。 + +（该测试"没有断言它命名的那个不变量"是上一轮第 14 条，本条是**另一个**问题：它现有的那个断言也是错的。） + +### 6. 两个测试跨 `shared` 调用断言 NodeStore 身份，中途可被驱逐 + +同文件 `:169`、`:187`：`#expect(referenceAgain == reference)`，而 `NodeReference.==` 要求 `store === store`。 + +该条目按 Mach-O 标识存放在 `SharedCache` 中，有三个驱逐点：`SharedCache.swift:121` 的 `remove(for:)`（由 `SwiftDeclarationIndexer.deinit` 调用），以及 `SharedCache.init` 注册的 `memoryWarningHandler` / `memoryCriticalHandler` 里的 `storageByIdentifier.removeAll()`。`Tests/SwiftInterfaceTests/SymbolTableRetentionTests.swift` 使用**同一个** `SymbolTestsCore` fixture，而所有测试 target 链接进同一个 `swift test` 包、swift-testing 并行调度 suite（`@Suite(.serialized)` 只在自身 suite 内串行）。两次调用之间发生驱逐就会重建出新 store，测试变成不可复现的红，且看起来像 NodeStore 回归。 + +修法：断言改用 `structurallyEquals`，或在两次调用之间持有 storage。 + +### 7. dyld 缓存框架形状判定无锚点，rank 0 内又回到枚举顺序依赖 + +`Sources/MachOExtensions/DyldCache+.swift:73`：`if enclosingDirectories.contains("\(name).framework")`——这是**对全部路径成分的成员检查**，不是"二进制的直接父目录"。 + +于是 `SwiftUI.framework/Versions/A/SwiftUI` 与 `…/Versions/B/SwiftUI`，以及任何嵌在该框架目录下、叶名相同的辅助二进制，**都拿 rank 0**。而 `accumulateBestMatch` 的 `guard rank < (rankedMatch?.rank ?? Int.max) else { continue }` 永远不会用一个 0 替换已有的 0，所以胜出者又变成"缓存先枚举到的那个"——**这正是排名机制被引入来消除的非确定性**。新增的 `leafInsideForeignFrameworkDoesNotScoreBestRank` 只覆盖了*外来*框架目录，这个形状没有测试。 + +修法：锚定在 `enclosingDirectories.last`（允许中间夹一层 `Versions/`）。 + +（上一轮第 5 条讲的是 `:133` 缺少提前退出导致全遍历，与本条不是同一个缺陷。） + +### 8. `ClassDumper.distributedFunctionNodes` 未记忆化，每个 actor 类求值两次 + +`Sources/SwiftDump/Dumper/ClassDumper.swift:77` 是一个 `private var … : Set` 计算属性，在 `:101`（`try? distributedFunctionNodes) ?? []).isEmpty == false`）和 `:192`（`let distributedFunctionNodes = (try? self.distributedFunctionNodes) ?? []`）各求值一次。每次都重建整个 thunk 符号数组，并为每个 thunk materialize 两棵树。 + +## 二、本轮已闭环 + +### 1. swift-demangling 依赖改回版本要求 ✅ + +`Package.swift:215`：`branch: "feature/node-store"` → `from: "0.5.0"`。 + +上一轮已把"钉在分支"判定为**开发期的预期状态**（见 2026-07-31 记录第五节），合并时换回版本要求即可。上游 `feature/node-store` 现已合入 `main` 并发布 `0.5.0`，因此该状态可以结清。 + +验证： + +- `0.5.0` 的提交正是 swift-demangling `main` 的顶端（`caacfb9`），与本机兄弟检出同一提交且该检出工作区干净——所以对它的本地构建等同于对 `0.5.0` 源码构建。 +- 在**没有兄弟目录**的干净检出中实测 `swift package resolve`，结果为 `swift-demangling resolved at 0.5.0`。这是 CI 与下游消费者实际走的路径，也正是钉在分支时会解析失败的那条路径。 +- `swift build`：0 errors / 2 warnings。 +- `Package.swift` 中已无任何 `branch:` 形式的依赖。 + +### 2. 打印器递归预算 ✅ + +见第一节第 1 条，随上述依赖升级自动关闭。 + +## 三、本轮裁决为"无需处理" + +按项目约定，判定为误报或不值得修的发现在此留档；后续审查先对照本节与既有台账，理由仍成立的直接跳过。 + +### 1. `Symbol` 删除公开成员未升版本、未写 changelog —— 不修 + +审查将其列为合并阻塞（对应上一轮第 11 条 / 台账第 8 条同源）。**裁定：不升版本、不写 changelog。** + +理由（维护者裁决，2026-08-03）：`nlist` 唯一被消费的信息就是 `isExternal`，而该位已经作为 `Symbol.isExternal` 独立公开，语义与文档俱在。核实结论： + +- 包内已无任何 `.nlist` 引用。 +- 新的 `init(offset:name:isExternal:)` 给 `isExternal` 带了默认值，因此旧的 `Symbol(offset:name:)` 调用形式**照常编译**，真正断裂的只有显式写 `nlist:` 标签的形式，面比初判窄。 +- `TypeName` / `ProtocolName` / `ExtensionName` 丢 `Codable` 是上游删除 `Node: Codable` 的连带结果，已在 2026-07-31 记录中论证过是有意为之。 + +### 2. 两个公开查询 API 的字典键从结构相等翻成身份相等 —— 不修 + +对应上一轮第 13 条 / 台账第 3 条。**裁定：不修**（维护者裁决，2026-08-03）。 + +理由：`SymbolIndexStore` 在类型层面就是 SPI——`SymbolIndexStore.swift:13-14` 上有 `@_spi(ForSymbolViewer)` 与 `@_spi(Internals)`。成员要被访问必须先能命名该类，而命名它必须带对应的 `@_spi(...) import`，所以 SPI 性由类继承而来，逐个方法标注是多余的。既然契约只对包内与已知的 SPI 消费方成立，只要保证包内正确即可。 + +核实包内确实正确： + +- `memberSymbols(of:excluding:in:)` 包内唯一调用点是 `SwiftDeclarationIndexer.swift:663`，在 `:684` 只做 `for (node, memberSymbols) in memberSymbolsByName` 遍历，全程无下标查询；且返回字典的键全部出自同一个 `storage.nodeStore`，同 store 内下标相等本就是正确的去重语义。 +- `allOpaqueTypeDescriptorSymbols(in:)` 在 `Sources/` 与 `Tests/` 中**零调用点**。 + +## 四、更正 + +### 1. 更正本轮自身：`memberSymbols` 的"O(1) 退化"不成立 + +本轮报告将 `SymbolIndexStore.swift:757` 描述为"从 O(1) 字典命中退化成线性扫描 + 全树比对"，并在会话中一度被判定为"四条性能问题里唯一白丢的一条"。**该判定错误，已撤回。** + +上一轮已就同一位置作出带实测的裁定（2026-07-31 记录第六节 + 第三节）： + +- `main` 那次"哈希查找"并不免费——`Node.hash(into:)` 是 `hasher.combine(children)` 递归，**哈希一次就要走完整棵树**（已在 `swift-demangling` `0.4.5` 的 `Node+Hashable.swift` 中逐字核对）。 +- 桶里装的是"同一类型名下的不同 type node"，实测 6,720 个桶中 **99.60% 只有 1 个元素**，最大 6。 + +所以实际是"一次全树哈希"换成"一次全树结构比对"，量级相当，不存在倍数退化。台账第 5 条建议的旁路索引仍是合理优化，但**不应按回归对待**。 + +### 2. 更正本轮自身：Package.swift 依赖不构成"合并阻塞" + +本轮把它列为阻塞项，但上一轮第五节已明确判定为开发期预期状态。本轮重复报出而未先对照既有裁决清单，属流程遗漏。结论上无害（该状态本来就该在合并前结清，且现已结清），但计入"应先查裁决清单"的教训。 + +### 3. 台账第 12 条（rebase 前提）再次确认过期 + +上一轮第 15 条已指出该条前提过期。本轮复测确认：分支落后 `main` 仅 2 个提交（`fed0acf` / `f8c6992`），且二者只改动 `.github/workflows/macOS.yml`；以合并基点为准，两侧改动文件**零交集**，所述 `AGENTS.md` 冲突不存在。 + +该条压着的**仍然成立**的注意事项照旧保留：`main` 的 `TransformerOptionGroup` 与本分支的 `DemangleResolver` / `printSemantic` / `FieldDefinition.typeNode` 改动之间的交互从未被跑过。 + +## 五、与既有记录的对照 + +| 本轮发现 | 2026-07-31 记录 | 既有台账 | 关系 | +| --- | --- | --- | --- | +| 打印深度 768→512 | — | — | 本轮新发现，已闭环（上游 0.5.0） | +| 依赖钉分支 | 五节（判定无需处理） | — | 同一事项，本轮结清 | +| `Symbol` 删公开成员 | 11 | 第 8 条（部分） | 同一问题，本轮裁决不修 | +| 公开查询 API 键语义 | 13 | 第 3 条 | 同一问题，本轮裁决不修 | +| `indexExtensions` 丢 `await` | — | — | **本轮新发现** | +| `withLargeStack` 卡整块 | 1（该条的修复本身） | 第 6 条 | **本轮新发现**，是对既有修复的新观察 | +| 26 处 `NodeReference(interning:)` | — | — | **本轮新发现** | +| 测试 `firstLeaf !==` 断言不成立 | 14（另一角度） | — | **本轮新发现** | +| 测试跨 `shared` 断言 store 身份 | — | — | **本轮新发现** | +| dyld `:73` 无锚点判定 | — | — | **本轮新发现** | +| `distributedFunctionNodes` 未记忆化 | 7（同类，另一站点） | — | **本轮新发现** | +| dyld `:133` 全遍历 | 5（已判定非退化） | — | 同一问题，沿用既有判定 | +| 持锁 demangle / 失败名不缓存 | 3、4 | 第 9 条 | 同一问题，沿用既有实测定性 | +| build sweep 串行 | 1 | 第 6 条 | 同一问题 | +| 打印路径每次跨线程 | 2（已判定为上游刻意交易） | — | 同一问题，沿用既有判定 | +| `memberSymbols` 线性扫 | 六节（已裁定量级可忽略） | 第 5 条 | 同一问题，**本轮判定被撤回**，见第四节 | +| `ABIKey` 每 key materialize | 9（实测占 0.8%） | 第 7 条 | 同一问题，沿用既有实测 | +| `ProjectEvolutionLog` 撞号 + 死链 | — | 第 12 条（提及） | 已复测**仍然存在**：两个 `## 20.`（353 / 384 行）、348 行链接少了实际文件名中没有的 `dyld-` 前缀 | + +## 六、待处理清单增量 + +上一轮第四节的 17 条清单继续有效（除本文第三节裁决为不修的两条、第二节闭环的一条外）。本轮在其上新增： + +**建议合并前修** + +1. `indexExtensions` 丢失 `await`（第一节第 2 条）——一行改动能否恢复取决于 `NodeReference` 是否补 async `print`；若上游不补，需在渲染循环层面另作安排。 + +**可排期** + +2. `withLargeStack` 占住整条线程（第一节第 3 条）——与上一轮第 1、2 条同属"线程跳转形态"课题，宜合并设计。 +3. 26 处 `NodeReference(interning:)` 改为每镜像共用 builder（第一节第 4 条）。 +4. dyld `:73` 判定锚定到直接父目录（第一节第 7 条）——**这条是正确性问题（非确定性），优先级高于同文件 `:133` 的性能问题**。 +5. `distributedFunctionNodes` 记忆化（第一节第 8 条）。 + +**测试** + +6. 两条脆弱断言（第一节第 5、6 条）——与上一轮第 14 条同文件，宜一并处理。 + +**文档** + +7. `ProjectEvolutionLog.md` 的重复 `## 20.` 与死链。既有台账建议"rebase 之后再补演进日志小节"，但撞号已在本分支内部成型，与 rebase 无关，可以先修。 diff --git a/Documentations/Internal/TaskReports/2026-08-03-pr97-review-triage.md b/Documentations/Internal/TaskReports/2026-08-03-pr97-review-triage.md new file mode 100644 index 00000000..b827a7fe --- /dev/null +++ b/Documentations/Internal/TaskReports/2026-08-03-pr97-review-triage.md @@ -0,0 +1,72 @@ +# 2026-08-03 PR #97 审查发现逐条裁决 + +## 问题 + +2026-08-02 对 PR #97（`feature/node-store-migration`）跑了一轮 `/code-review max`，产出 15 条发现 + 3 条被输出条数挤出但同样确认为真的条目。需要按项目约定对每一条作出判断（能否复现 / `main` 是否也有 / 值不值得修 / 以前是否修过），而不是只把清单转交出去。 + +## 调研 + +### 一、先查既有裁决记录（本次最大的教训） + +本轮开局没有先对照既有记录，直接按审查报告的排序向维护者汇报，导致两条结论出错： + +1. **`Package.swift` 钉在上游分支**被报为"合并阻塞"，而 `Reviews/2026-07-31-node-store-migration-review.md` 第五节早已把它裁决为"开发期的预期状态，合并时换回 `from:` 即可，不作为问题跟踪"。 +2. **`memberSymbols` 线性扫描**被我判定为"四条性能问题里唯一白丢的一条、`main` 是干净的 O(1)"，而同一份记录的第六节已就该位置作出带实测的相反裁定。 + +项目 `CLAUDE.md` 明确要求"每次 code-review 先对照已裁决清单"，本轮违反了这条。 + +### 二、逐条核实 + +| 核实项 | 方法 | 结果 | +| --- | --- | --- | +| 上游 `0.5.0` 是否存在且含 NodeStore | `git ls-tree 0.5.0 -- Sources/Demangling/Store/` | 存在，6 个文件齐全 | +| `0.5.0` 与 `main` 的关系 | `git rev-list -n1 0.5.0` vs `git rev-parse main` | 同一提交 `caacfb9`，`0.5.0..main` 为空 | +| 打印深度是否仍是 512 | `git grep maxPrintDepth 0.5.0` | **已恢复 768**，并附禁止再降的注释 | +| `structuralHash` 是否已修 | 读 `0.5.0` 的 `structuralDigest()` / `nodeContents` | 已改为迭代 + 记忆化，但 `String` 分配**仍在** | +| `demangleAsNodeTransient` 执行器 | `grep -A8` `DemangleInterface.swift` | 仍是 `StackSafeExecutor.execute`，**未修** | +| `main` 的 `Node.hash` 是否结构性 | 读 `0.4.5` 的 `Node+Hashable.swift` | `hasher.combine(children)` 递归——**哈希一次走完整棵树** | +| `main` 的 `memberSymbols` 实现 | `git show main:…SymbolIndexStore.swift` | `memberSymbolsByKind[$0]?[name]?[node]` | +| `main` 的 `buildStorage` 形态 | 同上 | `concurrentMap`，无 `withLargeStack` | +| `main` 的 `ABIKey.make` | `git show main:…ABIKey.swift` | 直接 `mangleAsString(node)`，无 materialize | +| 索引器是否丢了 `await` | `main:659` vs 分支 `:685` | `await node.print` → 同步 `node.print`，**确认** | +| `distributedFunctionNodes` 求值次数 | `grep` `ClassDumper.swift` | `:77` 计算属性，`:101` / `:192` 各求值一次，**确认** | +| `interning` 站点数 | `grep -rn "NodeReference(interning:"` | **26 处**（审查报告称 22，偏低） | +| `SymbolIndexStore` 是否类型级 SPI | 读 `SymbolIndexStore.swift:13-14` | `@_spi(ForSymbolViewer)` + `@_spi(Internals)`，**是** | +| 两个公开查询 API 的包内调用点 | `grep` `Sources/` `Tests/` | `excluding:` 重载唯一调用点只遍历；`allOpaqueTypeDescriptorSymbols` **零调用点** | +| `Symbol` 是否还有 `.nlist` 引用 | `grep -rn "\.nlist"` | 包内已无；`isExternal` 公开且带默认值 | +| 分支落后 `main` 多少 | `git rev-list --left-right --count` | 领先 27 / 落后 2，两个提交只动 workflows | +| 演进日志撞号与死链 | `grep "^## [0-9]"` + `ls TaskReports/` | 两个 `## 20.`、348 行死链，**均仍在** | + +## 最终方案 + +1. 改一行依赖：`branch: "feature/node-store"` → `from: "0.5.0"`。这是既有裁决里"合并时该做的事"，因上游发版而变得可做。 +2. 打印深度一条随之自动关闭（上游已修），不在本仓库改任何代码。 +3. 两条由维护者裁决为"不修"的（`Symbol` 删公开成员、公开查询 API 键语义）就地留档，不删除条目，便于后续审查跳过。 +4. 撤回本轮自己的两条错误结论，并把纠正写进台账本体——因为台账第 5 条的措辞正是把我误导到"白丢"结论的来源，只写在审查记录里挡不住下一个人。 +5. 其余 8 条新发现与既有清单合并，写成本轮的审查事件记录。 +6. **不动代码修复**：所有待修项均未在本批次动手，等维护者决定顺序。 +7. **不动 `ProjectEvolutionLog.md`**：撞号与死链已确认仍在，但修它属于独立的一次文档整理，不夹带进本批次。 + +## 实际执行 + +| 文件 | 改动 | +| --- | --- | +| `Package.swift` | 依赖要求由 `branch:` 改为 `from: "0.5.0"`（唯一的代码改动，1 行） | +| `Documentations/Internal/Reviews/2026-08-02-node-store-migration-pr97-review.md` | 新增。本轮审查事件记录：8 条新发现、2 条闭环、2 条裁决不修、3 条更正、与既有两份记录的对照表、待处理清单增量 | +| `Documentations/Internal/NodeStoreMigrationOpenIssues.md` | 第 3 条标记为已裁决不修并写明依据；第 4 条补上游 `0.5.0` 的核对状态；第 5 条改写措辞并就地写清"不要当回归"的两条实测依据；第 12 条拆成"已过期部分"与"仍然成立的两条"；开头补两份审查记录的链接与冲突时的取舍规则 | +| `Documentations/README.md` | 更新台账条目的描述（原描述把第 3 条列为"仍然打开"） | + +## 验证 + +- `swift build --scratch-path /tmp/claude/SwiftPM/MachOSwiftSection-node-store`：**success**，0 errors / 2 warnings。 +- 兄弟检出工作区干净且 HEAD 正是 `0.5.0` 所在提交，故上述构建等同于对 `0.5.0` 源码构建。 +- **无兄弟目录的干净检出**中 `swift package resolve`：`swift-demangling resolved at 0.5.0`。这是 CI 与下游消费者实际走的解析路径，也正是钉在分支时会失败的那条。 +- `git ls-remote --tags`：`0.5.0` 已推送到远端。 +- `Package.swift` 中已无任何 `branch:` 形式的依赖。 + +## 偏差 + +1. **没有先查既有裁决清单**，导致两条结论出错（见「调研」第一节）。两条都在向维护者汇报之后才被发现并当场更正。这是本次最该记住的一条：`Reviews/` 与本台账都必须在动笔前读完，而不是发现冲突后再回头补。 +2. **一次工具使用失误**：用非递归的 `ls Sources/Demangling/` 判断兄弟检出是否含 `NodeStore`，而该文件在 `Store/` 子目录下，于是错误地得出"主检出编不过"的结论并汇报了出去。后经 `git log` + 递归列目录更正。教训是"文件不存在"这类否定结论要用递归查找或 `git ls-tree` 坐实，不能靠一次浅层列目录。 +3. **审查报告的 `interning` 站点数偏低**（报 22，实为 26）。数量级不影响结论，但记录在案以说明报告中的计数需要复核。 +4. **未修任何待处理项**。本批次刻意只做裁决与留档；修复顺序待维护者决定。 diff --git a/Documentations/README.md b/Documentations/README.md index f81b4e77..2883abac 100644 --- a/Documentations/README.md +++ b/Documentations/README.md @@ -74,6 +74,6 @@ required by `Version.swift`'s bump contract). | [ClassMemberKeywordRecovery.md](Internal/ClassMemberKeywordRecovery.md) | `class` / `static` 成员关键字的还原：mangling 层面两者不可区分，判据是「类型级成员有 vtable method descriptor ⇒ 源码是 `class`」（`static` 隐式 final、不进 vtable）；模型侧 `isClassMember` 计算属性 + 三个 node printer 接线 + dump vtable 段落关键字，顺带消灭非法的 `override static` 输出；`final class func` 等四类 ABI 上与 `static` 完全一致，保守输出语义等价的 `static`。 | | [NodeStoreMigrationPlan.md](Internal/NodeStoreMigrationPlan.md) | `SymbolIndexStore` 迁移到 `NodeStore` arena 存储的分期计划与实施记录（Stage 0–5）：12B/节点扁平缓冲、cache-free build sweep、Symbol 表压缩、声明层换持 `NodeReference`、散点 transient demangling；含 Stage 5a 的 override/vtable 回归修复（裸 `[NodeReference: …]` 的 store-identity 陷阱 → `StructuralNodeReferenceKey`）与三源快照验收。 | | [DeclarationModelMemoryFootprint.md](Internal/DeclarationModelMemoryFootprint.md) | NodeStore 迁移后的声明模型内存足迹量测：`TypeDefinition` 1272 字节的逐属性构成（两份 `TypeContextWrapper` 占 74%）、`TypeContextWrapper` 按最大 case `Class` 定尺的原因、`parentContext` 是被当成永久字段的临时值、mini-store 增殖与 `MetadataReaderCache` 仍持 `Node` 树；含四项可回收估算（合计 ~8–10%）与「当前不建议实施、应先剖析其余 90%」的结论。 | -| [NodeStoreMigrationOpenIssues.md](Internal/NodeStoreMigrationOpenIssues.md) | `feature/node-store-migration` 的**已确认问题清单**（2026-07-28 两轮审查 + 复核）。第一节是前一轮修复自身的两处缺口（Catalyst 降级只覆盖 framework 形态导致 plain dylib 仍平局、`appendRowIfAbsent` 线性扫描），已于 2026-07-29 闭环并保留成因；第二节起为**仍然打开**的：两个公开查询 API 的字典键从结构相等翻成 store 身份相等、`structuralHash` 每文本节点分配 `String`（修复位置在上游）、`memberSymbols` 退化为线性 + 全树比对、build sweep 串行且每符号跨线程往返（上游）、`ABIKey` 每 key materialize，以及四条代码卫生项与 rebase 前置事项。逐条注明成因、影响面与「该在哪里修」。 | +| [NodeStoreMigrationOpenIssues.md](Internal/NodeStoreMigrationOpenIssues.md) | `feature/node-store-migration` 的**已确认问题清单**（2026-07-28 两轮审查 + 复核）。第一节是前一轮修复自身的两处缺口（Catalyst 降级只覆盖 framework 形态导致 plain dylib 仍平局、`appendRowIfAbsent` 线性扫描），已于 2026-07-29 闭环并保留成因；第二节起为仍然打开或已裁决的条目：`structuralHash` 每文本节点分配 `String`（修复位置在上游，`0.5.0` 仍未修）、`memberSymbols` 改为线性 + 全树比对（**已证实量级可忽略，不是回归**）、build sweep 串行且每符号跨线程往返（上游）、`ABIKey` 每 key materialize，以及四条代码卫生项；两个公开查询 API 的字典键一条**已于 2026-08-03 裁决为不修**（类型级 SPI），rebase 前置事项一条前提已过期但压着两条仍成立的注意事项。逐条注明成因、影响面与「该在哪里修」。与 `Reviews/` 下两份审查记录冲突时以后者为准。 | | [TaskReports/](Internal/TaskReports/) | Dated per-task fix / investigation reports. | | [Reviews/](Internal/Reviews/) | Dated code-review records — one file per review event: findings, measurements taken at the time, and the resulting to-do list. Distinct from the topic-organised issue ledgers (e.g. `NodeStoreMigrationOpenIssues.md`), which they cross-reference. | From d81ea778131993e47030632d5df6a0d9942a6f4b Mon Sep 17 00:00:00 2001 From: Mx-Iris Date: Mon, 3 Aug 2026 14:19:25 +0800 Subject: [PATCH 30/77] perf(SwiftDump,SwiftIndexing): migrate dump-path demangling to references, restore async print suspension The five remaining MetadataReader.demangleSymbol call sites (ClassDumper/ ProtocolDumper validNode, ClassDumper's override case .symbol, ProtocolConformanceDumper's case .symbol and _requirementName) move to demangleSymbolReference - no per-symbol tree materialization. Visited sets switch to StructuralNodeReferenceKey, and distributedFunctionNodes keys references directly, sparing a materialize per distributed thunk. demangleSymbol(for:in:) keeps its Node contract but has no hot in-repo caller left. indexExtensions gets its await back: upstream 0.5.1's async DemanglingNode.print suspends the task instead of parking a cooperative worker on a semaphore - the pre-migration semantics this loop had on main. 0.5.1 also deleted the concrete sync prints, so async contexts now require the await form at compile time (three dump-path prints gained it with the migration). Dump/interface snapshot suites stay byte-identical; full suite green (1312 tests, 249 suites). --- Sources/SwiftDump/Dumper/ClassDumper.swift | 35 ++++++++++--------- .../Dumper/ProtocolConformanceDumper.swift | 4 +-- Sources/SwiftDump/Dumper/ProtocolDumper.swift | 11 +++--- .../SwiftDeclarationIndexer.swift | 6 +++- 4 files changed, 32 insertions(+), 24 deletions(-) diff --git a/Sources/SwiftDump/Dumper/ClassDumper.swift b/Sources/SwiftDump/Dumper/ClassDumper.swift index e9176bb0..87179cf2 100644 --- a/Sources/SwiftDump/Dumper/ClassDumper.swift +++ b/Sources/SwiftDump/Dumper/ClassDumper.swift @@ -74,14 +74,14 @@ package struct ClassDumper: TypedDumper { /// The set of inner function nodes of `.distributedThunk` symbols whose class /// context matches this class. Used for both class-level (`distributed actor`) /// and method-level (`distributed func`) keyword emission. - private var distributedFunctionNodes: Set { + private var distributedFunctionNodes: Set { get throws { guard dumped.descriptor.isActor else { return [] } let currentTypeNode = try MetadataReader.demangleContext(for: .type(.class(dumped.descriptor)), in: machO) let currentTypeName = currentTypeNode.print(using: .interfaceTypeBuilderOnly) - var nodes: Set = [] + var nodes: Set = [] for thunkSymbol in symbolIndexStore.symbols(of: .distributedThunk, in: machO) { let rootNode = thunkSymbol.demangledNode @@ -89,7 +89,10 @@ package struct ClassDumper: TypedDumper { guard let contextNode = functionNode.children.first else { continue } let thunkTypeName = Node.create(kind: .type, child: contextNode.materialize()).print(using: .interfaceTypeBuilderOnly) guard thunkTypeName == currentTypeName else { continue } - nodes.insert(functionNode.materialize()) + // Structural key over the store-backed reference: the method + // loop probes with reference-form function nodes, and keying + // structurally spares materializing a tree per thunk. + nodes.insert(StructuralNodeReferenceKey(functionNode)) } return nodes @@ -191,7 +194,7 @@ package struct ClassDumper: TypedDumper { let distributedFunctionNodes = (try? self.distributedFunctionNodes) ?? [] - var methodVisitedNodes: OrderedSet = [] + var methodVisitedNodes: OrderedSet = [] let vtableBaseOffset = dumped.vTableDescriptorHeader.map { Int($0.layout.vTableOffset) } for (offset, descriptor) in dumped.methodDescriptors.offsetEnumerated() { BreakLine() @@ -209,7 +212,7 @@ package struct ClassDumper: TypedDumper { // Pre-resolve the method node so we can check distributed status // before deciding which keywords to emit. - var resolvedMethodNode: Node? = nil + var resolvedMethodNode: NodeReference? = nil if let symbols = try? descriptor.implementationSymbols(in: machO) { resolvedMethodNode = try? await validNode(for: symbols, visitedNodes: methodVisitedNodes) } @@ -218,7 +221,7 @@ package struct ClassDumper: TypedDumper { guard descriptor.flags.kind == .method, let root = resolvedMethodNode, let functionNode = root.children.first(where: { $0.kind == .function }) else { return false } - return distributedFunctionNodes.contains(functionNode) + return distributedFunctionNodes.contains(StructuralNodeReferenceKey(functionNode)) }() dumpMethodKind(for: descriptor) @@ -233,7 +236,7 @@ package struct ClassDumper: TypedDumper { } var parentVTableCache = ParentClassVTableCache() - var methodOverrideVisitedNodes: OrderedSet = [] + var methodOverrideVisitedNodes: OrderedSet = [] for (offset, descriptor) in dumped.methodOverrideDescriptors.offsetEnumerated() { BreakLine() @@ -257,7 +260,7 @@ package struct ClassDumper: TypedDumper { Keyword(.override) Space() try await demangleResolver.resolve(for: node) - _ = methodOverrideVisitedNodes.append(node) + _ = methodOverrideVisitedNodes.append(StructuralNodeReferenceKey(node)) } else if !descriptor.implementation.isNull { dumpMethodKind(for: methodDescriptor?.resolved) Keyword(.override) @@ -268,7 +271,7 @@ package struct ClassDumper: TypedDumper { case .symbol(let symbol): Keyword(.override) Space() - try await MetadataReader.demangleSymbol(for: symbol, in: machO).asyncMap { try await demangleResolver.resolve(for: $0) } + try await MetadataReader.demangleSymbolReference(for: symbol, in: machO).asyncMap { try await demangleResolver.resolve(for: $0) } case .element(let element): dumpMethodKind(for: element) Keyword(.override) @@ -285,7 +288,7 @@ package struct ClassDumper: TypedDumper { } } - var methodDefaultOverrideVisitedNodes: OrderedSet = [] + var methodDefaultOverrideVisitedNodes: OrderedSet = [] for (offset, descriptor) in dumped.methodDefaultOverrideDescriptors.offsetEnumerated() { BreakLine() @@ -302,7 +305,7 @@ package struct ClassDumper: TypedDumper { if let symbols = try? descriptor.implementationSymbols(in: machO), let node = try await validNode(for: symbols, visitedNodes: methodDefaultOverrideVisitedNodes) { try await demangleResolver.resolve(for: node) - _ = methodDefaultOverrideVisitedNodes.append(node) + _ = methodDefaultOverrideVisitedNodes.append(StructuralNodeReferenceKey(node)) } else if !descriptor.implementation.isNull { FunctionDeclaration(machO.addressString(forOffset: descriptor.implementation.resolveDirectOffset(from: descriptor.offset(of: \.implementation))).insertSubFunctionPrefix) } else { @@ -430,8 +433,8 @@ package struct ClassDumper: TypedDumper { } @SemanticStringBuilder - private func dumpMethodDeclaration(for descriptor: MethodDescriptor, resolvedNode: Node? = nil, visitedNodes: inout OrderedSet) async throws -> SemanticString { - let node: Node? + private func dumpMethodDeclaration(for descriptor: MethodDescriptor, resolvedNode: NodeReference? = nil, visitedNodes: inout OrderedSet) async throws -> SemanticString { + let node: NodeReference? if let resolvedNode { node = resolvedNode } else if let symbols = try? descriptor.implementationSymbols(in: machO) { @@ -442,7 +445,7 @@ package struct ClassDumper: TypedDumper { if let node { try await demangleResolver.resolve(for: node) - _ = visitedNodes.append(node) + _ = visitedNodes.append(StructuralNodeReferenceKey(node)) } else if !descriptor.implementation.isNull { FunctionDeclaration(machO.addressString(forOffset: descriptor.implementation.resolveDirectOffset(from: descriptor.offset(of: \.implementation))).insertSubFunctionPrefix) } else { @@ -450,10 +453,10 @@ package struct ClassDumper: TypedDumper { } } - package func validNode(for symbols: Symbols, visitedNodes: borrowing OrderedSet = []) async throws -> Node? { + package func validNode(for symbols: Symbols, visitedNodes: borrowing OrderedSet = []) async throws -> NodeReference? { let currentInterfaceName = try await _name(using: .options(.interfaceType)).string for symbol in symbols { - if let node = try? MetadataReader.demangleSymbol(for: symbol, in: machO), let classNode = node.first(of: .class), await classNode.print(using: .interfaceType) == currentInterfaceName, !visitedNodes.contains(node) { + if let node = MetadataReader.demangleSymbolReference(for: symbol, in: machO), let classNode = node.first(of: .class), await classNode.print(using: .interfaceType) == currentInterfaceName, !visitedNodes.contains(StructuralNodeReferenceKey(node)) { return node } } diff --git a/Sources/SwiftDump/Dumper/ProtocolConformanceDumper.swift b/Sources/SwiftDump/Dumper/ProtocolConformanceDumper.swift index 3149ce3a..409ab4a5 100644 --- a/Sources/SwiftDump/Dumper/ProtocolConformanceDumper.swift +++ b/Sources/SwiftDump/Dumper/ProtocolConformanceDumper.swift @@ -110,7 +110,7 @@ package struct ProtocolConformanceDumper: Conforme switch requirement { case .symbol(let symbol): - try await MetadataReader.demangleSymbol(for: symbol, in: machO).asyncMap { try await demangleResolver.resolve(for: $0) } + try await MetadataReader.demangleSymbolReference(for: symbol, in: machO).asyncMap { try await demangleResolver.resolve(for: $0) } case .element(let element): if let symbols = try await Symbols.resolve(from: element.offset, in: machO), let node = Self.demangledSymbol(for: symbols, typeName: typeNameString, visitedNodes: visitedNodes, in: machO)?.demangledNode { _ = visitedNodes.append(StructuralNodeReferenceKey(node)) @@ -173,7 +173,7 @@ package struct ProtocolConformanceDumper: Conforme private func _requirementName(for requirement: ProtocolRequirement) async throws -> String? { guard let symbols = try await Symbols.resolve(from: requirement.offset, in: machO) else { return nil } for symbol in symbols { - if let node = try? MetadataReader.demangleSymbol(for: symbol, in: machO) { + if let node = MetadataReader.demangleSymbolReference(for: symbol, in: machO) { return await node.print(using: typeNameOptions) } } diff --git a/Sources/SwiftDump/Dumper/ProtocolDumper.swift b/Sources/SwiftDump/Dumper/ProtocolDumper.swift index 293d9a10..2a583ec2 100644 --- a/Sources/SwiftDump/Dumper/ProtocolDumper.swift +++ b/Sources/SwiftDump/Dumper/ProtocolDumper.swift @@ -5,6 +5,7 @@ import Semantic import Utilities import Demangling import OrderedCollections +@_spi(Internals) import MachOSymbols @_spi(Internals) import SwiftInspection import SwiftDeclarationRendering @@ -91,7 +92,7 @@ package struct ProtocolDumper: NamedDumper { try await associatedTypes - var defaultImplementations: OrderedSet = [] + var defaultImplementations: OrderedSet = [] for (offset, requirement) in dumped.requirements.offsetEnumerated() { BreakLine() @@ -103,7 +104,7 @@ package struct ProtocolDumper: NamedDumper { } if let symbols = try requirement.defaultImplementationSymbols(in: machO), let defaultImplementation = try await validNode(for: symbols, visitedNode: defaultImplementations) { - _ = defaultImplementations.append(defaultImplementation) + _ = defaultImplementations.append(StructuralNodeReferenceKey(defaultImplementation)) } if offset.isEnd { @@ -120,7 +121,7 @@ package struct ProtocolDumper: NamedDumper { BreakLine() Indent(level: configuration.indentation) - try await demangleResolver.resolve(for: defaultImplementation) + try await demangleResolver.resolve(for: defaultImplementation.reference) if offset.isEnd { BreakLine() @@ -145,10 +146,10 @@ package struct ProtocolDumper: NamedDumper { } } - private func validNode(for symbols: Symbols, visitedNode: borrowing OrderedSet = []) async throws -> Node? { + private func validNode(for symbols: Symbols, visitedNode: borrowing OrderedSet = []) async throws -> NodeReference? { let currentInterfaceName = try await _name(using: .options(.interfaceType)).string for symbol in symbols { - if let node = try? MetadataReader.demangleSymbol(for: symbol, in: machO), let protocolNode = node.first(of: .protocol), await protocolNode.print(using: .interfaceType) == currentInterfaceName, !visitedNode.contains(node) { + if let node = MetadataReader.demangleSymbolReference(for: symbol, in: machO), let protocolNode = node.first(of: .protocol), await protocolNode.print(using: .interfaceType) == currentInterfaceName, !visitedNode.contains(StructuralNodeReferenceKey(node)) { return node } } diff --git a/Sources/SwiftIndexing/SwiftDeclarationIndexer.swift b/Sources/SwiftIndexing/SwiftDeclarationIndexer.swift index 35dfa0c8..93c3069f 100644 --- a/Sources/SwiftIndexing/SwiftDeclarationIndexer.swift +++ b/Sources/SwiftIndexing/SwiftDeclarationIndexer.swift @@ -682,7 +682,11 @@ public final class SwiftDeclarationIndexer Date: Mon, 3 Aug 2026 14:19:15 +0800 Subject: [PATCH 31/77] deps: require swift-demangling 0.5.1 0.5.1 moves the print conveniences onto the DemanglingNode extension and adds the async variant (suspends instead of blocking a cooperative worker on the large-stack hop), deleting the concrete sync prints on Node and NodeReference. The previous commit's await restoration relies on it - note a from: "0.5.0" requirement already resolves to 0.5.1 as the latest 0.5.x, and local builds take the ../swift-demangling sibling either way, so this bump pins the floor rather than changing what gets built. --- Package.swift | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/Package.swift b/Package.swift index 2a3abdf4..d25b749e 100644 --- a/Package.swift +++ b/Package.swift @@ -213,7 +213,7 @@ extension Package.Dependency { ), remote: .package( url: "https://github.com/MxIris-Reverse-Engineering/swift-demangling", - from: "0.5.0", + from: "0.5.1", ), ) From 6809438e8126128d6c4dac0b7d934ee1400f5d77 Mon Sep 17 00:00:00 2001 From: Mx-Iris Date: Mon, 3 Aug 2026 14:18:35 +0800 Subject: [PATCH 32/77] perf(MachOSymbols): stop re-demangling rejected names and unblock the late-demangle lock Two halves of the same hole: - demangledNodeReference(for:in:) now answers table-covered names from the sweep verdict alone, including rejections (nil root). The late path runs the same demangler (NodeStoreBuilder.demangle is demangleAsNodeTransient + intern), so falling through could only re-pay the failed demangle under the late-cache lock once per query - and demangledOverrideSymbol probes candidate symbols in a loop, which made that a hot path (measured 43.4ms vs 6.9ms per 5000 misses, 1.97x contention across 8 threads). - lateDemangledNode(forName:) demangles outside the critical section (an os_unfair_lock must not be held across a blocking large-stack hop) and the lock arbitrates insert-if-absent: a racing loser discards its mini store and returns the winner's reference, preserving one-store-per-name. Rejections are cached as nil verdicts - deterministic, so a retry could never succeed anyway. Regression tests: rejectedLateNameCachesItsFailure (red before the fix), tableCoveredNameNeverEntersLateCache, concurrentLateQueriesShareOneStore (pins the one-store guarantee now that the demangle left the lock). --- Sources/MachOSymbols/SymbolIndexStore.swift | 78 ++++++++++++++----- .../SymbolIndexStoreFixtureTests.swift | 65 ++++++++++++++++ 2 files changed, 122 insertions(+), 21 deletions(-) diff --git a/Sources/MachOSymbols/SymbolIndexStore.swift b/Sources/MachOSymbols/SymbolIndexStore.swift index 408f02fa..f4447be3 100644 --- a/Sources/MachOSymbols/SymbolIndexStore.swift +++ b/Sources/MachOSymbols/SymbolIndexStore.swift @@ -197,9 +197,11 @@ public final class SymbolIndexStore: SharedCache, @unc /// /// Keyed by name, like `tableRowByName`: a demangled tree is a pure /// function of the symbol name, so two symbols at different offsets - /// sharing a name share a tree. + /// sharing a name share a tree. A stored `nil` records a name the + /// demangler rejected — rejection is exactly as deterministic as + /// success, so it is cached the same way and never retried. @Mutex - private var lateDemangledNodeByName: [String: NodeReference] = [:] + private var lateDemangledNodeByName: [String: NodeReference?] = [:] fileprivate init( nodeStore: NodeStore, @@ -230,27 +232,54 @@ public final class SymbolIndexStore: SharedCache, @unc self.thunkAttributeMembersByKindAndTypeName = rowIndexes.thunkAttributeMembersByKindAndTypeName } - /// Atomic get-or-demangle for a name outside the build sweep. + /// Get-or-demangle for a name outside the build sweep. /// - /// Lookup and insert share one critical section: as a check-then-act - /// pair, two threads missing concurrently would each freeze their own - /// mini store and hand back references into *different* stores for one - /// name, which then compare unequal under `NodeReference`'s - /// store-identity `Hashable` — turning any downstream dedup into a - /// run-to-run coin flip. Demangling one name inside the lock is cheap - /// and this path is rare by construction. + /// The demangle runs *outside* the critical section: it can hop to a + /// large-stack thread and block on a semaphore, and an + /// `os_unfair_lock` must not be held across a blocking wait (priority + /// donation is lost and every other late lookup on the image + /// serializes behind it). The lock arbitrates insert-if-absent + /// instead — two threads missing concurrently both demangle, but only + /// the first insertion wins and the loser returns the winner's + /// reference, so one name still never hands out references into + /// *different* stores (those would compare unequal under + /// `NodeReference`'s store-identity `Hashable` and turn downstream + /// dedup into a run-to-run coin flip). The loser's mini store is + /// discarded; a demangled tree is a pure function of the name, so the + /// copies are interchangeable. /// - /// A name the demangler rejects is not cached, so a later call retries - /// rather than being stuck on the failure. + /// Rejections are cached like successes (`nil` verdict): the + /// demangler is deterministic, so a retry can only re-pay the failed + /// demangle. Sweep-covered names never reach this path at all — + /// `demangledNodeReference(for:in:)` answers them from the table + /// verdict — so the population here is genuinely late names only. fileprivate func lateDemangledNode(forName name: String) -> NodeReference? { - _lateDemangledNodeByName.withLockUnchecked { cache in - if let cached = cache[name] { return cached } - var lateBuilder = NodeStoreBuilder() - guard let nodeIndex = try? lateBuilder.demangle(name) else { return nil } - let reference = lateBuilder.freeze().reference(at: nodeIndex) - cache[name] = reference - return reference + if let cachedVerdict = _lateDemangledNodeByName.withLockUnchecked({ $0[name] }) { + return cachedVerdict } + var lateBuilder = NodeStoreBuilder() + var demangled: NodeReference? + if let nodeIndex = try? lateBuilder.demangle(name) { + demangled = lateBuilder.freeze().reference(at: nodeIndex) + } + return _lateDemangledNodeByName.withLockUnchecked { cache in + if let winner = cache[name] { return winner } + // `updateValue` rather than the subscript: assigning an + // `Optional` value through the subscript of an + // optional-valued dictionary is exactly the shape where a + // `nil` silently means "remove the key" instead of "store + // the rejection verdict". + cache.updateValue(demangled, forKey: name) + return demangled + } + } + + /// Test-only visibility into the late cache: `.some(.some)` is a + /// cached success, `.some(.none)` a cached rejection, `.none` a name + /// never attempted. Production code goes through + /// `lateDemangledNode(forName:)`. + func lateDemangleVerdictForTesting(forName name: String) -> NodeReference?? { + _lateDemangledNodeByName.withLockUnchecked { $0[name] } } // MARK: Row materialization @@ -874,8 +903,15 @@ public final class SymbolIndexStore: SharedCache, @unc // // Several symbols sharing one offset is normal (they differ by name) // and is unaffected: each name resolves to its own row. - if let row = cacheStorage.tableRowByName[symbol.name], - let rootNodeIndex = cacheStorage.rootNodeIndexByTableRow[Int(row)] { + if let row = cacheStorage.tableRowByName[symbol.name] { + // The sweep already ran every table row through the demangler + // once; a `nil` root records that it rejected this name. The + // late path runs the *same* demangler (`NodeStoreBuilder.demangle` + // is `demangleAsNodeTransient` + intern), so falling through + // could only re-pay the failed demangle — under the late-cache + // lock, once per query. `demangledOverrideSymbol` probes + // candidate symbols in a loop, which made that a hot path. + guard let rootNodeIndex = cacheStorage.rootNodeIndexByTableRow[Int(row)] else { return nil } return cacheStorage.nodeStore.reference(at: rootNodeIndex) } return cacheStorage.lateDemangledNode(forName: symbol.name) diff --git a/Tests/MachOSymbolsTests/SymbolIndexStoreFixtureTests.swift b/Tests/MachOSymbolsTests/SymbolIndexStoreFixtureTests.swift index e36ce26f..645d0b19 100644 --- a/Tests/MachOSymbolsTests/SymbolIndexStoreFixtureTests.swift +++ b/Tests/MachOSymbolsTests/SymbolIndexStoreFixtureTests.swift @@ -189,4 +189,69 @@ final class SymbolIndexStoreFixtureTests: MachOFileTests, @unchecked Sendable { let materialized = try #require(SymbolIndexStore.shared.demangledNode(for: lateSymbol, in: machOFile)) #expect(reference.structurallyEquals(materialized)) } + + /// A late name the demangler rejects caches its rejection: the verdict + /// slot exists and holds `nil`, so repeat queries answer from the cache + /// instead of re-paying the demangle (previously every query on a + /// rejected name re-entered the demangler). + @Test func rejectedLateNameCachesItsFailure() throws { + let storage = try storage + let bogusSymbol = Symbol(offset: -1, name: "$s999999999999") + #expect(storage.tableRowByName[bogusSymbol.name] == nil) + + #expect(SymbolIndexStore.shared.demangledNodeReference(for: bogusSymbol, in: machOFile) == nil) + let verdict = try #require(storage.lateDemangleVerdictForTesting(forName: bogusSymbol.name)) + #expect(verdict == nil) + + #expect(SymbolIndexStore.shared.demangledNodeReference(for: bogusSymbol, in: machOFile) == nil) + } + + /// A name the build sweep covered answers from the table verdict — hit + /// or rejection — without ever minting a late mini store. Guarded via + /// the late cache: now that rejections are cached too, a regression that + /// re-routes table-covered names through the late path would leave a + /// verdict slot behind and fail this test. + @Test func tableCoveredNameNeverEntersLateCache() throws { + let storage = try storage + + let demangledRow = try #require(storage.rootNodeIndexByTableRow.firstIndex(where: { $0 != nil })) + let demangledSymbol = storage.symbolTable[demangledRow] + _ = try #require(SymbolIndexStore.shared.demangledNodeReference(for: demangledSymbol, in: machOFile)) + #expect(storage.lateDemangleVerdictForTesting(forName: demangledSymbol.name) == nil) + + if let rejectedRow = storage.rootNodeIndexByTableRow.firstIndex(where: { $0 == nil }) { + let rejectedSymbol = storage.symbolTable[rejectedRow] + #expect(SymbolIndexStore.shared.demangledNodeReference(for: rejectedSymbol, in: machOFile) == nil) + #expect(storage.lateDemangleVerdictForTesting(forName: rejectedSymbol.name) == nil) + } + } + + /// Concurrent first-time queries for one late name must agree on a single + /// store: insert-if-absent hands every caller the winner's reference + /// (store-identity `==`), never references into different mini stores. + /// This pins the one-store-per-name guarantee the former + /// demangle-inside-the-lock design existed for, now that the demangle + /// runs outside the critical section. + @Test func concurrentLateQueriesShareOneStore() async throws { + _ = try storage + + let lateSymbol = Symbol(offset: -1, name: "$s7SwiftUI4TextV") + let references = await withTaskGroup(of: NodeReference?.self) { group in + for _ in 0 ..< 16 { + group.addTask { [machOFile] in + SymbolIndexStore.shared.demangledNodeReference(for: lateSymbol, in: machOFile) + } + } + var collected: [NodeReference?] = [] + for await reference in group { + collected.append(reference) + } + return collected + } + + let winner = try #require(references.first ?? nil) + for reference in references { + #expect(reference == winner) + } + } } From b19ac66ab78800351fd9977f08cb2456a7a91d28 Mon Sep 17 00:00:00 2001 From: Mx-Iris Date: Mon, 3 Aug 2026 14:19:07 +0800 Subject: [PATCH 33/77] perf(MachOSymbols): deduplicate metadata-derived name interning behind a shared cache NodeReference(interning:) mints one private arena per call - upstream documents it as the wrong tool for a batch. The name-construction sites (TypeName/ProtocolName/ExtensionName from MetadataReader trees) called it once per occurrence, and occurrences repeat heavily: measured on the SymbolTestsCore fixture, 730 retained mini stores backed only 472 structurally unique trees, with conformance fan-out (180 protocol-name occurrences) the dominant repeat source. InternedNodeReferenceCache keeps one reference per structurally unique tree: buckets keyed by the tree's structural hash, hits verified via structurallyEquals, minting arbitrated outside-lock/insert-if-absent (same discipline as the late-demangle cache). Two SharedCache scopes: per image (evicted with the image; SwiftDeclarationIndexer's cleanup drops it alongside the symbol store) and per process (type-keyed) for the in-process helpers without a Mach-O handle. All 25 interning call sites route through the cache. Re-measured on the fixture: 730 -> 471 mini stores (dedup is complete), retained store bytes -32%, and repeated names now share a store so structurallyEquals' store-identity fast path fires for name equality. The originally sketched per-image shared *builder* was disproved before landing: references can only be minted after freeze(), while names are consumed mid-flow. Snapshot suites stay byte-identical; SwiftSpecialization gains an explicit MachOSymbols target dependency. --- Package.swift | 1 + .../InternedNodeReferenceCache.swift | 110 ++++++++++++++++++ .../Definitions/ProtocolDefinition.swift | 2 +- Sources/SwiftDeclaration/Extensions.swift | 28 ++--- .../SwiftDeclarationIndexer.swift | 16 ++- .../ConformanceProvider.swift | 3 +- .../GenericSpecializer.swift | 5 +- .../TypeDefinition+Specialization.swift | 3 +- .../InternedNodeReferenceCacheTests.swift | 100 ++++++++++++++++ 9 files changed, 243 insertions(+), 25 deletions(-) create mode 100644 Sources/MachOSymbols/InternedNodeReferenceCache.swift create mode 100644 Tests/MachOSymbolsTests/InternedNodeReferenceCacheTests.swift diff --git a/Package.swift b/Package.swift index d25b749e..527e41ea 100644 --- a/Package.swift +++ b/Package.swift @@ -603,6 +603,7 @@ extension Target { .product(.MachOObjCSection), .product(.Semantic), .product(.Demangling), + .target(.MachOSymbols), .target(.MachOSwiftSection), .target(.SwiftInspection), .target(.Utilities), diff --git a/Sources/MachOSymbols/InternedNodeReferenceCache.swift b/Sources/MachOSymbols/InternedNodeReferenceCache.swift new file mode 100644 index 00000000..3c4146b9 --- /dev/null +++ b/Sources/MachOSymbols/InternedNodeReferenceCache.swift @@ -0,0 +1,110 @@ +import Foundation +@_spi(Internals) import Demangling +import MachOKit +import MachOExtensions +@_spi(Internals) import MachOCaches +import SwiftStdlibToolbox + +/// Structural deduplication for `NodeReference(interning:)`. +/// +/// `NodeReference(interning:)` mints one private arena per call — upstream +/// documents it as the wrong tool for a batch, since deduplication and +/// compactness are both properties of a shared arena. The name-construction +/// sites (`TypeName` / `ProtocolName` / `ExtensionName` built from +/// `MetadataReader` trees) call it once per *occurrence*, and occurrences +/// repeat heavily: every conformance re-interns its protocol's name, every +/// nested type re-interns its parent's, every extension its target's. +/// Measured on the `SymbolTestsCore` fixture, 730 retained mini stores +/// backed only 472 structurally unique trees — and the repeat factor grows +/// with framework size (conformance fan-out dominates). +/// +/// This cache keeps one reference per structurally unique tree: repeats hand +/// back the previously minted reference, so equal names share one store and +/// `structurallyEquals`' same-store `store ===` fast path starts firing for +/// them (name equality drops from a full tree walk to an index compare). +/// +/// Two scopes, matching `SharedCache`'s two keying modes: +/// - **Per image** (`reference(interning:in:)`): the bucket lives and dies +/// with the image — evicted on memory pressure with every other shared +/// cache, and dropped by `SwiftDeclarationIndexer`'s per-image cleanup so +/// the recycling model holds. +/// - **Per process** (`reference(interning:)`): for the in-process reading +/// paths that have no Mach-O handle. One type-keyed bucket, memory-pressure +/// evictable, bounded by the unique names the process actually touches. +/// +/// Retention trade: the cache pins every minted mini store for its scope's +/// lifetime, including names a query produced and dropped. That is bounded +/// by *unique* names per scope and buys the dedup above; the pre-cache +/// behavior pinned one store per *retained occurrence* with no sharing at +/// all. +@_spi(ForSymbolViewer) +@_spi(Internals) +public final class InternedNodeReferenceCache: SharedCache, @unchecked Sendable { + public static let shared = InternedNodeReferenceCache() + + public final class Storage: @unchecked Sendable { + /// Minted references bucketed by their tree's structural hash + /// (`Node`'s `Hashable` is structural); collisions resolve by + /// `structurallyEquals`, so a bucket almost always holds one entry. + @Mutex + private var referencesByStructuralHash: [Int: [NodeReference]] = [:] + + /// Get-or-mint following the same discipline as + /// `SymbolIndexStore.Storage.lateDemangledNode(forName:)`: the + /// interning runs *outside* the critical section (it allocates and + /// walks the whole tree, which has no business inside an + /// `os_unfair_lock`), and the lock arbitrates insert-if-absent — a + /// racing loser discards its freshly minted store and returns the + /// winner's reference, so one structural name never hands out + /// references into two stores within one scope. + fileprivate func reference(interning node: Node) -> NodeReference { + var hasher = Hasher() + hasher.combine(node) + let structuralHashValue = hasher.finalize() + + if let existing = _referencesByStructuralHash.withLockUnchecked({ buckets in + buckets[structuralHashValue]?.first(where: { $0.structurallyEquals(node) }) + }) { + return existing + } + + let minted = NodeReference(interning: node) + return _referencesByStructuralHash.withLockUnchecked { buckets in + if let winner = buckets[structuralHashValue]?.first(where: { $0.structurallyEquals(node) }) { + return winner + } + buckets[structuralHashValue, default: []].append(minted) + return minted + } + } + + /// Test-only visibility: the number of structurally distinct trees + /// currently cached in this scope. + public var cachedReferenceCountForTesting: Int { + _referencesByStructuralHash.withLockUnchecked { buckets in + buckets.values.reduce(0) { $0 + $1.count } + } + } + } + + override public func buildStorage(for machO: some MachORepresentableWithCache) -> Storage? { + Storage() + } + + override public func buildStorage() -> Storage? { + Storage() + } + + /// The image-scoped shared reference for `node`'s structural identity. + public func reference(interning node: Node, in machO: MachO) -> NodeReference { + guard let storage = storage(in: machO) else { return NodeReference(interning: node) } + return storage.reference(interning: node) + } + + /// The process-scoped shared reference for `node`'s structural identity, + /// for call sites without a Mach-O handle (in-process reading contexts). + public func reference(interning node: Node) -> NodeReference { + guard let storage = storage() else { return NodeReference(interning: node) } + return storage.reference(interning: node) + } +} diff --git a/Sources/SwiftDeclaration/Components/Definitions/ProtocolDefinition.swift b/Sources/SwiftDeclaration/Components/Definitions/ProtocolDefinition.swift index 30381af7..f0e55231 100644 --- a/Sources/SwiftDeclaration/Components/Definitions/ProtocolDefinition.swift +++ b/Sources/SwiftDeclaration/Components/Definitions/ProtocolDefinition.swift @@ -124,7 +124,7 @@ public final class ProtocolDefinition: Definition, MutableDefinition { public init(`protocol`: MachOSwiftSection.`Protocol`, in machO: MachO) throws { self.protocol = `protocol` let node = try MetadataReader.demangleContext(for: .protocol(`protocol`.descriptor), in: machO) - self.protocolName = ProtocolName(node: NodeReference(interning: node)) + self.protocolName = ProtocolName(node: InternedNodeReferenceCache.shared.reference(interning: node, in: machO)) } package func index(in machO: MachO) async throws { diff --git a/Sources/SwiftDeclaration/Extensions.swift b/Sources/SwiftDeclaration/Extensions.swift index 50876b1b..6d61aaba 100644 --- a/Sources/SwiftDeclaration/Extensions.swift +++ b/Sources/SwiftDeclaration/Extensions.swift @@ -43,7 +43,7 @@ extension ProtocolConformance { } else { return nil } - return TypeName(node: NodeReference(interning: node), kind: kind) + return TypeName(node: InternedNodeReferenceCache.shared.reference(interning: node, in: machO), kind: kind) case .element(let element): return try element.typeContextDescriptorWrapper?.typeName(in: machO) @@ -54,7 +54,7 @@ extension ProtocolConformance { case .directObjCClassName, .indirectObjCClass: guard let node = try typeNode(in: machO) else { return nil } - return TypeName(node: NodeReference(interning: node), kind: .class) + return TypeName(node: InternedNodeReferenceCache.shared.reference(interning: node, in: machO), kind: .class) } } @@ -77,7 +77,7 @@ extension ProtocolConformance { } else { return nil } - return TypeName(node: NodeReference(interning: node), kind: kind) + return TypeName(node: InternedNodeReferenceCache.shared.reference(interning: node), kind: kind) case .element(let element): return try element.typeContextDescriptorWrapper?.typeName() case nil: @@ -86,18 +86,18 @@ extension ProtocolConformance { case .directObjCClassName, .indirectObjCClass: guard let node = try typeNode() else { return nil } - return TypeName(node: NodeReference(interning: node), kind: .class) + return TypeName(node: InternedNodeReferenceCache.shared.reference(interning: node), kind: .class) } } package func protocolName(in machO: some MachOSwiftSectionRepresentableWithCache) throws -> ProtocolName? { guard let node = try protocolNode(in: machO) else { return nil } - return ProtocolName(node: NodeReference(interning: node)) + return ProtocolName(node: InternedNodeReferenceCache.shared.reference(interning: node, in: machO)) } package func protocolName() throws -> ProtocolName? { guard let node = try protocolNode() else { return nil } - return ProtocolName(node: NodeReference(interning: node)) + return ProtocolName(node: InternedNodeReferenceCache.shared.reference(interning: node)) } } @@ -114,7 +114,7 @@ extension AssociatedType { } else { return nil } - return TypeName(node: NodeReference(interning: node), kind: kind) + return TypeName(node: InternedNodeReferenceCache.shared.reference(interning: node, in: machO), kind: kind) } package func typeName() throws -> TypeName? { @@ -129,15 +129,15 @@ extension AssociatedType { } else { return nil } - return TypeName(node: NodeReference(interning: node), kind: kind) + return TypeName(node: InternedNodeReferenceCache.shared.reference(interning: node), kind: kind) } package func protocolName(in machO: some MachOSwiftSectionRepresentableWithCache) throws -> ProtocolName { - ProtocolName(node: NodeReference(interning: try MetadataReader.demangleType(for: protocolTypeName, in: machO))) + ProtocolName(node: InternedNodeReferenceCache.shared.reference(interning: try MetadataReader.demangleType(for: protocolTypeName, in: machO), in: machO)) } package func protocolName() throws -> ProtocolName { - ProtocolName(node: NodeReference(interning: try MetadataReader.demangleType(for: protocolTypeName))) + ProtocolName(node: InternedNodeReferenceCache.shared.reference(interning: try MetadataReader.demangleType(for: protocolTypeName))) } } @@ -153,11 +153,11 @@ extension MachOSwiftSection.`Protocol` { extension ProtocolDescriptor { package func protocolName(in machO: some MachOSwiftSectionRepresentableWithCache) throws -> ProtocolName { - ProtocolName(node: NodeReference(interning: try MetadataReader.demangleContext(for: .protocol(self), in: machO))) + ProtocolName(node: InternedNodeReferenceCache.shared.reference(interning: try MetadataReader.demangleContext(for: .protocol(self), in: machO), in: machO)) } package func protocolName() throws -> ProtocolName { - ProtocolName(node: NodeReference(interning: try MetadataReader.demangleContext(for: .protocol(self)))) + ProtocolName(node: InternedNodeReferenceCache.shared.reference(interning: try MetadataReader.demangleContext(for: .protocol(self)))) } } @@ -184,11 +184,11 @@ extension TypeContextDescriptorWrapper { } package func typeName(in machO: some MachOSwiftSectionRepresentableWithCache) throws -> TypeName { - return TypeName(node: NodeReference(interning: try MetadataReader.demangleContext(for: .type(self), in: machO)), kind: kind) + return TypeName(node: InternedNodeReferenceCache.shared.reference(interning: try MetadataReader.demangleContext(for: .type(self), in: machO), in: machO), kind: kind) } package func typeName() throws -> TypeName { - return TypeName(node: NodeReference(interning: try MetadataReader.demangleContext(for: .type(self))), kind: kind) + return TypeName(node: InternedNodeReferenceCache.shared.reference(interning: try MetadataReader.demangleContext(for: .type(self))), kind: kind) } } diff --git a/Sources/SwiftIndexing/SwiftDeclarationIndexer.swift b/Sources/SwiftIndexing/SwiftDeclarationIndexer.swift index 93c3069f..c662e4fd 100644 --- a/Sources/SwiftIndexing/SwiftDeclarationIndexer.swift +++ b/Sources/SwiftIndexing/SwiftDeclarationIndexer.swift @@ -154,6 +154,10 @@ public final class SwiftDeclarationIndexer> ) throws -> Metadata { - let stepProtocolName = ProtocolName(node: NodeReference(interning: step.protocolNode)) + let stepProtocolName = ProtocolName(node: InternedNodeReferenceCache.shared.reference(interning: step.protocolNode, in: machO)) guard let entry = allProtocolDefinitions[stepProtocolName] else { throw AssociatedTypeResolutionError.missingAssociatedTypeRefMachOAndProtocol(protocolTypeNode: step.protocolNode) } diff --git a/Sources/SwiftSpecialization/TypeDefinition+Specialization.swift b/Sources/SwiftSpecialization/TypeDefinition+Specialization.swift index 1b18c535..8aacedad 100644 --- a/Sources/SwiftSpecialization/TypeDefinition+Specialization.swift +++ b/Sources/SwiftSpecialization/TypeDefinition+Specialization.swift @@ -3,6 +3,7 @@ import SwiftDeclaration import MachOSwiftSection import MachOKit import Demangling +@_spi(Internals) import MachOSymbols import FoundationToolbox import AssociatedObject @_spi(Internals) import SwiftInspection @@ -346,7 +347,7 @@ extension TypeDefinition { let boundNode = Node.create(kind: boundKind, children: [unboundTypeNode, typeList]) let wrappedNode = Node.create(kind: .type, children: [boundNode]) - return TypeName(node: NodeReference(interning: wrappedNode), kind: unboundTypeName.kind) + return TypeName(node: InternedNodeReferenceCache.shared.reference(interning: wrappedNode), kind: unboundTypeName.kind) } private func validateSpecialization(metadata: MetadataWrapper, in machO: MachOImage) throws { diff --git a/Tests/MachOSymbolsTests/InternedNodeReferenceCacheTests.swift b/Tests/MachOSymbolsTests/InternedNodeReferenceCacheTests.swift new file mode 100644 index 00000000..2e183f29 --- /dev/null +++ b/Tests/MachOSymbolsTests/InternedNodeReferenceCacheTests.swift @@ -0,0 +1,100 @@ +import Foundation +import Testing +@_spi(Internals) import Demangling +@_spi(Internals) @testable import MachOSymbols +@_spi(Internals) import MachOCaches +@testable import MachOTestingSupport +import MachOFixtureSupport + +/// Unit coverage for `InternedNodeReferenceCache` — the structural +/// deduplication layer over `NodeReference(interning:)`. The batching claim +/// is that equal trees interned through one scope share a single store +/// (repeat interning returns the *same* reference, store identity included), +/// while the per-image scope stays independently evictable. +@Suite(.serialized) +final class InternedNodeReferenceCacheTests: MachOFileTests, @unchecked Sendable { + override class var fileName: MachOFileName { .SymbolTestsCore } + + /// Two structurally equal trees (distinct `Node` instances) interned in + /// one image scope share one store: the second call returns the first + /// call's reference under store-identity `==`, which is exactly what + /// re-enables the `store ===` fast path for name equality. + @Test func repeatInterningSharesOneStore() throws { + let firstTree = try demangleAsNodeTransient("$s7SwiftUI4TextV") + let secondTree = try demangleAsNodeTransient("$s7SwiftUI4TextV") + #expect(firstTree !== secondTree) + + let firstReference = InternedNodeReferenceCache.shared.reference(interning: firstTree, in: machOFile) + let secondReference = InternedNodeReferenceCache.shared.reference(interning: secondTree, in: machOFile) + #expect(firstReference == secondReference) + #expect(firstReference.store === secondReference.store) + #expect(firstReference.structurallyEquals(firstTree)) + } + + /// Structurally different trees keep distinct identities and correct + /// content — dedup must never conflate distinct names. + @Test func distinctTreesStayDistinct() throws { + let textTree = try demangleAsNodeTransient("$s7SwiftUI4TextV") + let viewTree = try demangleAsNodeTransient("$s7SwiftUI4ViewP") + + let textReference = InternedNodeReferenceCache.shared.reference(interning: textTree, in: machOFile) + let viewReference = InternedNodeReferenceCache.shared.reference(interning: viewTree, in: machOFile) + #expect(textReference != viewReference) + #expect(!textReference.structurallyEquals(viewReference)) + #expect(textReference.print(using: .default) == textTree.print(using: .default)) + #expect(viewReference.print(using: .default) == viewTree.print(using: .default)) + } + + /// Concurrent first-time interning of one structural name agrees on a + /// single winner: the insert-if-absent arbitration hands every caller + /// the same reference, mirroring the late-demangle cache's guarantee. + @Test func concurrentInterningSharesOneWinner() async throws { + let mangledName = "$s7SwiftUI5ImageV" + let references = await withTaskGroup(of: NodeReference?.self) { group in + for _ in 0 ..< 16 { + group.addTask { [machOFile] in + guard let tree = try? demangleAsNodeTransient(mangledName) else { return nil } + return InternedNodeReferenceCache.shared.reference(interning: tree, in: machOFile) + } + } + var collected: [NodeReference?] = [] + for await reference in group { + collected.append(reference) + } + return collected + } + + let winner = try #require(references.first ?? nil) + for reference in references { + #expect(reference == winner) + } + } + + /// `remove(for:)` drops the image's bucket: a later intern mints a fresh + /// store (identity differs) while structural equality is preserved — + /// the per-image recycling model the indexer's cleanup relies on. + @Test func removalDropsTheImageBucket() throws { + let tree = try demangleAsNodeTransient("$s7SwiftUI6ButtonV") + let referenceBeforeRemoval = InternedNodeReferenceCache.shared.reference(interning: tree, in: machOFile) + + InternedNodeReferenceCache.shared.remove(for: machOFile) + + let referenceAfterRemoval = InternedNodeReferenceCache.shared.reference(interning: tree, in: machOFile) + #expect(referenceBeforeRemoval != referenceAfterRemoval) + #expect(referenceBeforeRemoval.structurallyEquals(referenceAfterRemoval)) + } + + /// The process-scoped variant (no Mach-O handle) dedups the same way, + /// in its own type-keyed bucket independent of any image bucket. + @Test func processScopedVariantDeduplicates() throws { + let firstTree = try demangleAsNodeTransient("$s7SwiftUI5ColorV") + let secondTree = try demangleAsNodeTransient("$s7SwiftUI5ColorV") + + let firstReference = InternedNodeReferenceCache.shared.reference(interning: firstTree) + let secondReference = InternedNodeReferenceCache.shared.reference(interning: secondTree) + #expect(firstReference == secondReference) + + let imageScopedReference = InternedNodeReferenceCache.shared.reference(interning: firstTree, in: machOFile) + #expect(imageScopedReference.structurallyEquals(firstReference)) + } +} From 9e25d0520cedad17ea91e90a9365a351aebf720d Mon Sep 17 00:00:00 2001 From: Mx-Iris Date: Mon, 3 Aug 2026 14:19:40 +0800 Subject: [PATCH 34/77] docs: record the performance batch, its adjudications, and the 0.5.1 upgrade - ReviewAdjudications.md (new home for won't-fix verdicts): A1 remangle materialize bridge and A2 structuralHash string allocation both closed as upstream-by-design after the 0.5.1 recheck (no upstream issue for A2 after the maintainer's rationale); A3 records the printer per-member materialize verdict with its measurement (1.18% of print wall on a full fixture export - not worth genericizing the 1700-line printer stack). - NodeStoreMigrationOpenIssues: items 9 (lock-held late demangle) and 10 (conformance dumper materialize branch) closed with their fixes; items 4 and 7 finalized as upstream-by-design. - 2026-08-02 review record: findings re-statused; the indexExtensions await adjudication documents its same-day overturn - the "no async print on NodeReference" premise was true for 0.5.0 and invalidated by 0.5.1 landing it on the DemanglingNode extension. - AGENTS.md: late-demangle verdict contract, InternedNodeReferenceCache routing rule, and the different-stores assumption updated. - ProjectEvolutionLog section 24 + task report (Chinese) for the batch, including the environment post-mortem (stale fixture binary; shared SwiftPM scratch across checkouts fabricating link errors - checkouts must never share a scratch directory). --- AGENTS.md | 2 +- .../Internal/NodeStoreMigrationOpenIssues.md | 14 ++++- .../Internal/ProjectEvolutionLog.md | 17 ++++++ .../Internal/ReviewAdjudications.md | 55 +++++++++++++++++++ ...-08-02-node-store-migration-pr97-review.md | 40 +++++++++++--- .../2026-08-03-performance-batch-fixes.md | 45 +++++++++++++++ Documentations/README.md | 3 +- Sources/SwiftDiffing/ABIKey.swift | 8 +++ 8 files changed, 172 insertions(+), 12 deletions(-) create mode 100644 Documentations/Internal/ReviewAdjudications.md create mode 100644 Documentations/Internal/TaskReports/2026-08-03-performance-batch-fixes.md diff --git a/AGENTS.md b/AGENTS.md index b2cfb563..ecbfd144 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -206,7 +206,7 @@ let node = try demangleAsNode("$sSiD") // Returns Node tree let string = node.print(using: .default) // "Swift.Int" ``` -**Symbol indexing (NodeStore-backed)**: `SymbolIndexStore.Storage` no longer retains `Node` class trees. The build sweep demangles each symbol cache-free onto a transient tree (`@_spi(Internals) demangleAsNodeTransient`), classifies on that tree, and interns it into a per-image `NodeStoreBuilder`. Nothing touches the global `NodeCache`, and dropping a `Storage` releases the whole per-image footprint. Storage layout (Stage 3): one flat `symbolTable: [Symbol]` row per unique symbol name (canonical cache-adjusted offset; raw and adjusted offset keys share one row), a parallel `rootNodeIndexByTableRow` array, and every classification index stores 4-byte `UInt32` table rows (member/opaque keys are 4-byte `NodeStore.NodeIndex`); indexes accumulate in final row form during the sweep, so `freeze()` is followed by a plain move, not a conversion pass. `Symbol` is 32 bytes (no `nlist` existential — the external-undefined bit is extracted at collection time into `isExternal`), and `DemangledSymbol` is a 32-byte value (shared-table reference + row + `NodeReference`) whose `symbol` is computed; both are asserted by `compactValueLayouts`. **A `DemangledSymbol` stored into the declaration model must be detached first** (`detachedFromSharedTable()`): the shared `[Symbol]` buffer is the right trade for the hundreds of thousands of values a query vends and drops, but `Array` is a reference to its buffer, so a single stored survivor pins the whole table and every mangled name in it — defeating `removeSubIndexer(_:)`, whose reason for existing is reclaiming per-image memory. Measured on SwiftUI (iOS 18.5): 9,872 stored values referenced 9,506 distinct rows, 5.1% of a 185,988-row table, so detaching trades ~0.6 MB of small allocations for ~19.9 MB of retention. The six storing sites are `DefinitionBuilder`'s four construction points (`Accessor` for variables/subscripts, `FunctionDefinition` for allocators/functions) and `TypeDefinition`'s `deallocatorSymbol` / `destructorSymbol` assignments; `SymbolTableRetentionTests` fails the moment a new one forgets. Do NOT detach on the query path. Query APIs construct `DemangledSymbol`/`NodeReference` values at the exit boundary; APIs taking an externally demangled node (`memberSymbols(of:for:node:)` with `Node` or `NodeReference`, `opaqueTypeDescriptorSymbol(for:)`) match keys via `NodeReference.structurallyEquals(_:)` (same-store O(1), cross-store structural walk) — `opaqueTypeDescriptorSymbol(for:)` buckets its entries by `DemanglingNode.identifier` first (`opaqueTypeDescriptorEntriesByMemberIdentifier`), because a full scan there is O(descriptors × printed `some` returns). `demangledNodeReference(for:)` matches on the symbol **name alone** — a demangled tree is a pure function of the name and the flat table holds one row per unique name, so comparing the row's canonical offset against the queried symbol's offset could only reject valid hits (it once rejected every symbol of a dyld-cache image, since `symbols(for:in:)` stamps each vended `Symbol` with the offset it was queried by). Names genuinely outside the build sweep fall back to name-keyed mini stores, minted under one lock so a race cannot hand two callers references into different stores (`lateDemangledNode(forName:)` / `MetadataReader.demangleSymbolReference(for:in:)`). Struct-layout changes in `MachOSymbols` need a `swift package clean` rebuild — SwiftPM incremental builds have been observed linking stale downstream objects (runtime SIGSEGV in `outlined destroy`). See `Documentations/Internal/NodeStoreMigrationPlan.md`. **Declaration values hold `NodeReference` (Stage 5a)**: `DefinitionName.node` (`TypeName`/`ProtocolName`/`ExtensionName`), `Variable/Function/SubscriptDefinition.node`, `FieldDefinition.typeNode` and `ExtensionDefinition.genericSignature` are `NodeReference`, not `Node`. Member definitions reference the per-image symbol store directly (a live definition keeps that store alive — the intended per-image recycling model); metadata-derived trees (`MetadataReader` output) are wrapped via `NodeReference(interning:)` mini stores, and `TypeDefinition.index(in:)` batches all field type trees of one type into a single shared store. The `Name` types customize `Hashable` to STRUCTURAL semantics (`structurallyEquals` + `structuralHash`) because `NodeReference`'s intrinsic `Hashable` is store-identity based and would split equal keys minted into different stores; they are deliberately NOT `Codable` — upstream dropped `Node: Codable` (a mangled symbol already is the tree's serialized form: smaller, ABI-stable across toolchains, and sharing-preserving through the round trip), and nothing persists these values. A consumer that ever needs to should persist `mangleAsString(node)` and read it back with `demangleAsNode(_:)` rather than reintroducing a node encoding. **Any `Dictionary`/`Set` keyed on a `NodeReference` whose keys and lookups can come from DIFFERENT stores must key on `StructuralNodeReferenceKey` (`Sources/MachOSymbols/`, next to the mini stores it reconciles), never a bare `NodeReference`.** Different stores are the norm, not the exception: `demangledNodeReference(for:)` falls back to a mini store, and `NodeReference(interning:)` — every `MetadataReader`-derived tree — mints a fresh private store per call by design. Everything that mixes those with image-store references is keyed structurally: the override/vtable lookups (`methodDescriptorLookup` / `vtableOffsetLookup` in `TypeDefinition.index` → `DefinitionBuilder`), `DefinitionBuilder`'s accessor grouping and merged-thunk dedup (`accessorsByNode` / `canonicalIndexByFunctionNode` / `canonicalIndexByAllocatorNode`), and every `visitedNodes` "already claimed this symbol" set (`ExtensionDefinition` / `ProtocolDefinition` / `TypeDefinition` / `OverrideSymbolMatcher` / `ProtocolConformanceDumper`). Under bare keys these fail silently and differently: the override lookups drop the `override` keyword + vtable-offset comment (the Stage 5a regression fixed 2026-07-25), a subscript's getter and setter split into two buckets so the setter-only one is discarded, merged thunks emit their `func`/`init` twice, and a witness can be claimed twice. Bare-`NodeReference` keys are only safe for grouping/dedup WITHIN one `memberSymbols` batch (single hash-consed store, where structural equality coincides with index equality). **Transient demangling everywhere else (Stage 5c)**: `MetadataReader`, `RuntimeFieldLayoutBackend`, `TypedDumper`, `ClassHierarchyDumper` and `Symbol.demangledNode` use `demangleAsNodeTransient` + `Node.createTransient`, so the global `NodeCache` no longer grows while browsing. Do NOT reintroduce `demangleAsNode`/`Node.create` on unbounded inputs in those paths. +**Symbol indexing (NodeStore-backed)**: `SymbolIndexStore.Storage` no longer retains `Node` class trees. The build sweep demangles each symbol cache-free onto a transient tree (`@_spi(Internals) demangleAsNodeTransient`), classifies on that tree, and interns it into a per-image `NodeStoreBuilder`. Nothing touches the global `NodeCache`, and dropping a `Storage` releases the whole per-image footprint. Storage layout (Stage 3): one flat `symbolTable: [Symbol]` row per unique symbol name (canonical cache-adjusted offset; raw and adjusted offset keys share one row), a parallel `rootNodeIndexByTableRow` array, and every classification index stores 4-byte `UInt32` table rows (member/opaque keys are 4-byte `NodeStore.NodeIndex`); indexes accumulate in final row form during the sweep, so `freeze()` is followed by a plain move, not a conversion pass. `Symbol` is 32 bytes (no `nlist` existential — the external-undefined bit is extracted at collection time into `isExternal`), and `DemangledSymbol` is a 32-byte value (shared-table reference + row + `NodeReference`) whose `symbol` is computed; both are asserted by `compactValueLayouts`. **A `DemangledSymbol` stored into the declaration model must be detached first** (`detachedFromSharedTable()`): the shared `[Symbol]` buffer is the right trade for the hundreds of thousands of values a query vends and drops, but `Array` is a reference to its buffer, so a single stored survivor pins the whole table and every mangled name in it — defeating `removeSubIndexer(_:)`, whose reason for existing is reclaiming per-image memory. Measured on SwiftUI (iOS 18.5): 9,872 stored values referenced 9,506 distinct rows, 5.1% of a 185,988-row table, so detaching trades ~0.6 MB of small allocations for ~19.9 MB of retention. The six storing sites are `DefinitionBuilder`'s four construction points (`Accessor` for variables/subscripts, `FunctionDefinition` for allocators/functions) and `TypeDefinition`'s `deallocatorSymbol` / `destructorSymbol` assignments; `SymbolTableRetentionTests` fails the moment a new one forgets. Do NOT detach on the query path. Query APIs construct `DemangledSymbol`/`NodeReference` values at the exit boundary; APIs taking an externally demangled node (`memberSymbols(of:for:node:)` with `Node` or `NodeReference`, `opaqueTypeDescriptorSymbol(for:)`) match keys via `NodeReference.structurallyEquals(_:)` (same-store O(1), cross-store structural walk) — `opaqueTypeDescriptorSymbol(for:)` buckets its entries by `DemanglingNode.identifier` first (`opaqueTypeDescriptorEntriesByMemberIdentifier`), because a full scan there is O(descriptors × printed `some` returns). `demangledNodeReference(for:)` matches on the symbol **name alone** — a demangled tree is a pure function of the name and the flat table holds one row per unique name, so comparing the row's canonical offset against the queried symbol's offset could only reject valid hits (it once rejected every symbol of a dyld-cache image, since `symbols(for:in:)` stamps each vended `Symbol` with the offset it was queried by). A name the sweep covered is answered from the table verdict alone — including the *rejection* verdict: a `nil` root means the demangler already refused the name once, and the same demangler backs the late path, so the query returns `nil` without re-demangling. Names genuinely outside the build sweep fall back to name-keyed mini stores (`lateDemangledNode(forName:)` / `MetadataReader.demangleSymbolReference(for:in:)`): the demangle runs *outside* the cache lock (never hold `os_unfair_lock` across a potentially blocking large-stack hop) and the lock arbitrates insert-if-absent — a racing loser discards its store and returns the winner's reference, preserving one-store-per-name — while rejections are cached as `nil` verdicts and never retried (`rejectedLateNameCachesItsFailure` / `tableCoveredNameNeverEntersLateCache` / `concurrentLateQueriesShareOneStore` pin all three properties). Struct-layout changes in `MachOSymbols` need a `swift package clean` rebuild — SwiftPM incremental builds have been observed linking stale downstream objects (runtime SIGSEGV in `outlined destroy`). See `Documentations/Internal/NodeStoreMigrationPlan.md`. **Declaration values hold `NodeReference` (Stage 5a)**: `DefinitionName.node` (`TypeName`/`ProtocolName`/`ExtensionName`), `Variable/Function/SubscriptDefinition.node`, `FieldDefinition.typeNode` and `ExtensionDefinition.genericSignature` are `NodeReference`, not `Node`. Member definitions reference the per-image symbol store directly (a live definition keeps that store alive — the intended per-image recycling model); metadata-derived name trees (`MetadataReader` output feeding `TypeName`/`ProtocolName`/`ExtensionName`/generic signatures) go through `InternedNodeReferenceCache` (`Sources/MachOSymbols/`) — a structural-dedup layer over `NodeReference(interning:)` with an image-keyed scope (evicted with the image; `SwiftDeclarationIndexer`'s cleanup removes it alongside the symbol store) and a process-keyed scope for the in-process helpers with no Mach-O handle, so repeated names (conformance protocol names, shared parents, extension targets) share one store and name equality gets the `store ===` fast path (measured on the fixture: 730 retained mini stores → 471, matching the 472 structurally unique trees). Do NOT call bare `NodeReference(interning:)` on a batch path — route through the cache. `TypeDefinition.index(in:)` still batches all field type trees of one type into a single shared store. The `Name` types customize `Hashable` to STRUCTURAL semantics (`structurallyEquals` + `structuralHash`) because `NodeReference`'s intrinsic `Hashable` is store-identity based and would split equal keys minted into different stores; they are deliberately NOT `Codable` — upstream dropped `Node: Codable` (a mangled symbol already is the tree's serialized form: smaller, ABI-stable across toolchains, and sharing-preserving through the round trip), and nothing persists these values. A consumer that ever needs to should persist `mangleAsString(node)` and read it back with `demangleAsNode(_:)` rather than reintroducing a node encoding. **Any `Dictionary`/`Set` keyed on a `NodeReference` whose keys and lookups can come from DIFFERENT stores must key on `StructuralNodeReferenceKey` (`Sources/MachOSymbols/`, next to the mini stores it reconciles), never a bare `NodeReference`.** Different stores are the norm, not the exception: `demangledNodeReference(for:)` falls back to a mini store, and `InternedNodeReferenceCache` — every `MetadataReader`-derived tree — deduplicates equal trees onto one store per scope but still mints a distinct store per unique name, so cross-store mixing remains the default assumption. Everything that mixes those with image-store references is keyed structurally: the override/vtable lookups (`methodDescriptorLookup` / `vtableOffsetLookup` in `TypeDefinition.index` → `DefinitionBuilder`), `DefinitionBuilder`'s accessor grouping and merged-thunk dedup (`accessorsByNode` / `canonicalIndexByFunctionNode` / `canonicalIndexByAllocatorNode`), and every `visitedNodes` "already claimed this symbol" set (`ExtensionDefinition` / `ProtocolDefinition` / `TypeDefinition` / `OverrideSymbolMatcher` / `ProtocolConformanceDumper`). Under bare keys these fail silently and differently: the override lookups drop the `override` keyword + vtable-offset comment (the Stage 5a regression fixed 2026-07-25), a subscript's getter and setter split into two buckets so the setter-only one is discarded, merged thunks emit their `func`/`init` twice, and a witness can be claimed twice. Bare-`NodeReference` keys are only safe for grouping/dedup WITHIN one `memberSymbols` batch (single hash-consed store, where structural equality coincides with index equality). **Transient demangling everywhere else (Stage 5c)**: `MetadataReader`, `RuntimeFieldLayoutBackend`, `TypedDumper`, `ClassHierarchyDumper` and `Symbol.demangledNode` use `demangleAsNodeTransient` + `Node.createTransient`, so the global `NodeCache` no longer grows while browsing. Do NOT reintroduce `demangleAsNode`/`Node.create` on unbounded inputs in those paths. ## Test Environment diff --git a/Documentations/Internal/NodeStoreMigrationOpenIssues.md b/Documentations/Internal/NodeStoreMigrationOpenIssues.md index 525218c2..a2376584 100644 --- a/Documentations/Internal/NodeStoreMigrationOpenIssues.md +++ b/Documentations/Internal/NodeStoreMigrationOpenIssues.md @@ -87,6 +87,8 @@ **上游 `0.5.0` 状态（2026-08-03 核对）：仍然打开。** `structuralHash` 已重写为委托给 `structuralDigest()`——显式帧栈迭代 + 按节点下标记忆化（`digestByIndex`），重复子树只哈希一次，是实打实的改进。但 `nodeContents` 依旧是 `.text(store.text(offset:length:))`，而 `seededDigestHasher` 直接 `hasher.combine(contents)`，所以**每个文本节点仍然分配一个 `String`**。 +**终审（2026-08-03，随 0.5.1 升级）：按上游设计关闭，不再等修复。** 上游维护者说明按设计不改（单一编码源换来的跨表示一致性，见 [ReviewAdjudications.md](ReviewAdjudications.md) A2 的两轮事故史）；重开条件只剩 profiling 证据。 + ### 5. `memberSymbols(of:for:node:)` 改为线性扫描 + 逐候选全树比对（量级可忽略，属可选优化） 迁移前是 `memberSymbolsByKind[$0]?[name]?[node]`，一次字典查找。现在两个重载都走 `rowsByTypeNodeIndex.elements.first(where: { …structurallyEquals(node) })`——对桶里每个键做一次结构化树遍历直到命中。`TypeDefinition.index` 会为 allocator、变量、静态变量、函数、静态函数、下标各调一次。 @@ -118,6 +120,10 @@ **修复位置**：上游或本仓库 `Sources/SwiftDiffing/ABIKey.swift`，取决于选哪条路。 +**上游 `0.5.0` 状态（2026-08-03 核对）：仍然打开。** `mangleAsString(_ node: some DemanglingNode)` 的实现依旧是 `mangleAsString(node.materializedNode)`（`RemangleInterface.swift:49`）；根治需要 `Remangler` 泛型化到 `DemanglingNode`。本仓库侧每个 key 本来只 materialize 一次，无重复可省。 + +**终审（2026-08-03，随 0.5.1 升级）：按上游设计关闭。** 0.5.1 保持桥接并在文档注释里写明理由（Remangler 遍历中构造临时辅助节点、非只读消费者，桥接成本瞬态且不触及 store 驻留内存目标）；裁决与复审条件见 [ReviewAdjudications.md](ReviewAdjudications.md) A1。 + --- ## 四、代码卫生 @@ -128,18 +134,20 @@ 过滤和守卫二者必有其一冗余，需要挑一个删掉并修正注释。 -### 9. `lateDemangledNode(forName:)` 在持锁期间 demangle +### 9. ~~`lateDemangledNode(forName:)` 在持锁期间 demangle~~ ✅ 已修（2026-08-03） `demangleAsNodeTransient` 会走 `StackSafeExecutor.execute`，在 512 KB 栈线程上无条件提交线程池并 `semaphore.wait()`。于是一次 miss 会**在持有 per-image 互斥锁的情况下**跨线程等待不定时长，该镜像上所有查询晚绑定名字的线程都排在它后面；线程池饱和时持锁时间无上界。 注意这是**刻意的权衡**：代码注释写明查找与插入必须同处一个临界区，否则两个并发 miss 会各自冻结一份 mini store，把同一名字的引用分裂到不同 store 里。所以修的时候要保住这个保证。 -**正确修法**：在锁外 demangle，锁内用 insert-if-absent（后写者放弃、返回胜出者），单 store 保证不变而临界区里不再阻塞。 +**修复（2026-08-03，按上面的修法落地并加固）**：锁外 demangle、锁内 insert-if-absent（后写者丢弃自己的 store、返回胜出者），单 store 保证由 `concurrentLateQueriesShareOneStore` 回归测试钉住。同批一并修了两个相邻问题：(a) **拒绝结果同样缓存**（`nil` 裁决）——demangle 是名字的纯函数，失败一次即永远失败，旧行为「不缓存失败以便重试」只是每次重付一次失败的 demangle（上一轮实测失败名 43.4 ms vs 缓存命中 6.9 ms）；(b) `demangledNodeReference(for:in:)` 对**表内 demangle 失败的名字**直接以 sweep 裁决回答 `nil`，不再穿透到 late 路径在锁内重试（`NodeStoreBuilder.demangle` 就是 `demangleAsNodeTransient` + intern，拒绝集一致，核实于上游源码）。回归测试：`rejectedLateNameCachesItsFailure`（修复前红，断在裁决未被缓存上）、`tableCoveredNameNeverEntersLateCache`。 -### 10. `ProtocolConformanceDumper` 里一个分支还在 materialize +### 10. ~~`ProtocolConformanceDumper` 里一个分支还在 materialize~~ ✅ 已修（2026-08-03） 同一个 `switch requirement` 块里，`case .element` 和 `Self.demangledSymbol(...)` 都已改走 `MetadataReader.demangleSymbolReference` 留在 store 上，唯独 `case .symbol` 仍调 `MetadataReader.demangleSymbol` 把整棵树 materialize 出来，只为交给 `demangleResolver.resolve(for:)`——而后者现在有 `some DemanglingNode` 重载，可以直接吃引用。既多余，又会让下一个维护者误以为这个不一致是有意的。 +**修复（2026-08-03）**：该分支连同 dump 路径其余四处 `demangleSymbol` 调用点（`ClassDumper.validNode` / `ProtocolDumper.validNode` / `ProtocolConformanceDumper._requirementName` / `ClassDumper` override 的 `case .symbol`）一并迁到 `demangleSymbolReference`，visited 集合与 `distributedFunctionNodes` 换 `StructuralNodeReferenceKey` 键（后者顺带省掉每 thunk 一次 materialize）。`MetadataReader.demangleSymbol(for:in:)` 保留 `Node` 契约但包内已无热调用方。快照测试（SwiftDumpTests / SwiftInterfaceTests）逐字节不变。 + ### 11. 两处 `throws` 是迁移残留 `ExtensionDefinition._symbol(for:typeName:visitedNodes:)` 与 `ProtocolDefinition` 里对应的那个，唯一的抛出调用已被换成不抛出的 `demangleSymbolReference`，函数体里不再有任何 `try`，但签名仍是 `throws`，调用点仍写 `try`。删掉 `throws` 之后，周围 `if let` 链里真正会抛的调用（`resilientWitness.implementationSymbols(in:)`、`Symbols.resolve`）才看得出来。 diff --git a/Documentations/Internal/ProjectEvolutionLog.md b/Documentations/Internal/ProjectEvolutionLog.md index 28213504..91732da2 100644 --- a/Documentations/Internal/ProjectEvolutionLog.md +++ b/Documentations/Internal/ProjectEvolutionLog.md @@ -681,6 +681,23 @@ --- +## 24. 性能批次：失败名裁决、名字去重缓存、dump 路径引用化 + +- **时间段**：2026-08-03。 +- **动机**：[2026-08-02 审查记录](Reviews/2026-08-02-node-store-migration-pr97-review.md)与既有台账合并后剩 19 条待处理，其中「立即可修」与「中等重构」两组获批同批落地；本批延续第 23 节的纪律——先测后修，测出不值得的就裁决留档而不是硬改。 +- **落地**： + - **失败名裁决**（`SymbolIndexStore`）：`demangledNodeReference` 对表内 demangle 失败的名字直接以 sweep 裁决回答 `nil`（`NodeStoreBuilder.demangle` 与 sweep 用同一个 demangler，拒绝集一致）；`lateDemangledNode` 改锁外 demangle + 锁内 insert-if-absent，拒绝结果作为 `nil` 裁决缓存。三条新回归测试钉住（其中缓存断言在修复前红）。 + - **`InternedNodeReferenceCache`**（`MachOSymbols` 新类型）：`NodeReference(interning:)` 的结构去重层，镜像键 + 进程键双作用域，25 处名字构造点全部改走缓存；`SwiftDeclarationIndexer` 清理与内存压力驱逐接通。fixture 实测驻留 mini store 730 → 471（= 结构唯一数），字节 −32%，重复名恢复 `store ===` 快路径。原「每镜像共用 builder」修法被实测推翻（freeze 前无法发引用，调用流即用即取），故改缓存形态。 + - **dump 路径引用化**：`ClassDumper` / `ProtocolDumper` / `ProtocolConformanceDumper` 五处 `demangleSymbol` 调用点迁 `demangleSymbolReference`，visited 集合与 `distributedFunctionNodes` 换 `StructuralNodeReferenceKey`（每 thunk 省一次 materialize）；`MetadataReader.demangleSymbol` 保留契约但包内热调用方清零。 + - **`indexExtensions` 恢复 `await` + 依赖升 0.5.1**：当日早间的「不修」裁决被上游动作推翻——0.5.1（`f913742`）把 print 便利方法整体迁到 `DemanglingNode` 并补 async 变体（挂起 + 大栈），对 `NodeReference` 直接可用，一行恢复 main 的任务挂起语义；具体同步 `print` 同时被上游删除，async 上下文由编译器强制 `await`（dump 路径三处一并加上）。依赖要求升至 `from: "0.5.1"`。remangle 桥接与 `structuralHash` 分配两条随升级按上游设计终审关闭（[ReviewAdjudications.md](ReviewAdjudications.md) A1/A2）。 +- **关键决策**： + - **打印器每成员 materialize 裁决为暂不修**：临时计量显示其只占打印墙钟 1.18%（fixture 全量导出 1313 次共 32.6 ms），根治需 1700 行打印栈泛型化 + 3 处节点合成重设计，投入产出不成比例；数据与重开条件留档在审查记录。 + - 快照套件（SwiftInterfaceTests 53 项含逐字节 interface 快照、SwiftDumpTests）全绿，输出零变化是本批的硬约束。 +- **关联文档**：[TaskReports/2026-08-03-performance-batch-fixes.md](TaskReports/2026-08-03-performance-batch-fixes.md)、台账第 9/10 条闭环与第 4/7 条上游状态核对、AGENTS.md「Symbol indexing」段同步。 +- **对应版本**：0.14.0 之后未发布区间（`feature/node-store-migration` 分支）。 + +--- + ## 维护约定 1. **每个非平凡批次结束时必须在本文追加/更新一节**（新工作弧新增一节；延续既有弧则在该节 diff --git a/Documentations/Internal/ReviewAdjudications.md b/Documentations/Internal/ReviewAdjudications.md new file mode 100644 index 00000000..bdcee424 --- /dev/null +++ b/Documentations/Internal/ReviewAdjudications.md @@ -0,0 +1,55 @@ +# Review 已裁决清单 + +判定为「不修」或「误报」的 code-review 发现，连同结论、理由与复审条件，集中登记在这里。 + +**使用规则**：每轮 code review 开始前先对照此表；已裁决且理由仍成立的发现直接跳过，不再重走「复现 / 基线对比 / 值不值得修 / 既往修复」四问。若新证据（profiling 数据、上游变更、新的触发路径）推翻了当初的理由，更新对应条目并重新裁决。 + +各轮 review 的原始发现清单在 [`Roadmaps/*-review-findings.md`](../../Roadmaps/)；本表只收录其中作出「不修 / 误报」终审的条目。 + +--- + +## A1 — 上游 `mangleAsString(some DemanglingNode)` 经 materialize 桥接（`ABIKey.make` 路径） + +- **裁决**：不修（2026-08-03）。 +- **发现**：store-backed 节点（`NodeReference`）remangle 时，上游泛型重载的实现是 `mangleAsString(node.materializedNode)`（swift-demangling `RemangleInterface.swift:49`，0.5.0 与 `feature/node-store` tip `5cc30c9` 均如此；**0.5.1 复核仍然保持，且上游维护者确认按设计不改**——文档注释明言 Remangler 不是只读消费者、桥接成本瞬态且不影响 store 驻留内存目标）——每次调用把子树 materialize 成一棵瞬态 `Node` 类树后再走具体 `Node` 版 Remangler。`ABIKey.make(for: some DemanglingNode)` 是本仓库的主要受影响调用点。 +- **复现 / 是否误报**：属实，非误报。已直接核对上游两个版本的源码。 +- **与 main 基线对比**：非本仓库引入。main（0.4.x `Node` 线）传具体 `Node`，重载解析命中具体版本，零 materialize；仅 node-store migration 线的 store-backed 路径受影响。上游侧这是记录在案的设计取舍——Remangler 遍历中要构造临时辅助节点（unspecialized nominals、SIL box 布局包装），不是只读消费者，故运行在 class 表示上（上游 `RemangleInterface.swift` 文档注释原文）。 +- **为什么不修**： + 1. 成本是每 key **恰好一次**的瞬态 O(subtree) 类树构造，用完即弃，不进常驻内存；remangle 输出本来就是新 `String`，store 的驻留内存目标不受影响。本仓库侧无重复 materialize 可省。 + 2. 根治在上游：把约 6200 行的 Remangler 泛型化到 `DemanglingNode`（需引入 overlay 节点表示"新脊柱挂旧子树"的两簇合成点，并把替换表的身份 hash / 深比较异构化）。上游已把它列为既定方向——`materializedNode` 的文档注释原话是 *"remangling until the `Remangler` is genericized"*——且所需基础设施（跨表示 `structurallyEquals` / 一致的 `structuralHash`、`printCacheIdentity` 身份抽象、printer 泛型化先例）在 `feature/node-store` 分支均已就绪。 + 3. 下游任何 workaround（如自写泛型 remangler、绕过 `ABIKey` 的 remangle 身份）都比等上游代价大。 +- **既往修复**：无。上游有意设计，非回归。 +- **代码锚点**：`Sources/SwiftDiffing/ABIKey.swift` `make(for:)` 调用点注释（"Adjudicated — not worth fixing"）。 +- **复审条件**：① 上游发布泛型化的 Remangler 后，删调用点注释即可直接受益，本条目关闭；② profiling 显示批量建 key 时 materialize 占总耗时比例可观（当前仅为推断成本，无测量数据）——届时正确动作是推动上游泛型化，而非下游绕路。 +- **关联上游事项**（非本表裁决，仅备查）：`structuralHash` 分配一条见 A2。~~同轮核对的 `NodeReference` 缺 async `print(using:)` 一条属上游补齐范畴~~——**已闭环（2026-08-03）**：上游 `f913742` 把 print 便利方法整体迁到 `DemanglingNode` 协议扩展并补 async 变体，发布为 **0.5.1**，对 `NodeReference` 直接可用；本仓库依赖已升 `from: "0.5.1"`，`indexExtensions` 的 `await` 已恢复。 + +--- + +## A2 — 上游 `NodeReference.structuralHash` 每文本节点分配一个瞬态 `String` + +- **裁决**：下游不修、不 workaround（2026-08-03）。~~上游按 enhancement 提 issue~~——**追记（2026-08-03）**：上游维护者已说明按设计不改（单一编码源 `nodeContents` 共享 `Node.Contents`，一致性由构造保证，是两轮事故换来的设计；`0.5.1` 保持现状），不再提 issue。仅当下条复审条件 ① 的 profiling 证据出现时重议。 +- **发现**：`NodeReference.structuralHash` → `structuralDigest()` 对每个文本节点经 `nodeContents` 构造一个瞬态 `String`（`store.text(offset:length:)` → `String(decoding:)`，upstream `NodeReference.swift:169` / `NodeStore.swift:97`）。且 digest 的 memo（`digestByIndex`）是每次调用局部的——字典每次插入 / 查找 / 扩容重哈希都会重走子树。 +- **复现 / 是否误报**：属实，非误报。0.5.0 与 `feature/node-store` tip（`5cc30c9`）均已核对源码。 +- **与 main 基线对比**：非本仓库引入，是上游 store 表示的实现特性。本仓库 main（`Node` 线）不受影响——`Node.text` 本就驻留，hash 现有 `String` 零分配。 +- **既往修复（这是不是刻意设计）**：修过两轮，现状是刻意设计的**一部分**—— + 1. 出生（upstream `26db7a4`，Stage 5）：手写 discriminator 编码，且 `String` 分配从出生就在；手写编码与 `Node.hash(into:)` 不一致 → 跨表示字典查找永远落空的 bug。 + 2. 编码统一修复：引入 `nodeContents` 共享 `Node.Contents` 编码源，一致性由构造保证（上游 `nodeContents` 注释记录了该事故）。 + 3. 性能修复（upstream `69fdbd3`）：路径放大 615,165× → memoized digest，刻意保留共享编码，跨表示一致由测试钉住。 + 结论：「单一编码源」是设计且理由仍成立；「每文本节点分配 String」只是该设计当前实现的副作用，二者可分离。 +- **为什么下游不动作**：暴露面真实——`structuralHash` 支撑 `TypeName` / `ProtocolName` / `ExtensionName` 的 `Hashable` 与 `MachOSymbols.StructuralNodeReferenceKey`，都在索引字典路径上；但这些子树是名字链（几个到二十来个节点），每次操作只是少量小 `String` 的瞬态垃圾，且无 profiling 证据表明是热点。 +- **上游修法（issue 内容）**：两侧已汇合到唯一漏斗 `seededDigestHasher(kind:contents:childCount:)`；把漏斗改成字节级——`Node` 侧以驻留 `String` 的 `utf8` view 进 hasher（零分配），`NodeReference` 侧以 store 字节表切片直接进 hasher（零分配），discriminator 单处定义，跨表示一致性由既有测试继续钉住。哈希值会变，但 `Hasher` 本就 per-process 播种，无持久化契约。 +- **代码锚点**：无单一调用点，不加代码注释，以本条目为准。 +- **复审条件**：① profiling 显示索引热路径上该 `String` 构造占比可观 → 升级为催上游或直接贡献 PR；② 上游修复发布并重新 pin 后，本条目关闭。 + +--- + +## A3 — 接口打印器每成员 materialize 一棵树（`SwiftDeclarationPrinter` 7 处） + +- **裁决**：暂不修（2026-08-03，数据裁决）。 +- **发现**：`SwiftDeclarationPrinter.swift:454/464/474/277` 与 `+Members.swift:42/65/108` 在打印每个成员 / 字段 / 扩展 where 子句时把 `NodeReference` materialize 成 `Node` 类树再交给 `TypeNodePrinter` / `FunctionNodePrinter` 等（2026-07-31 审查四.6，估算 SwiftUI 规模 ~10⁵ 次瞬态建树）。 +- **复现 / 是否误报**：机制属实，但**量级测出来不值得**：fixture（SymbolTestsCore）全量 interface 导出，打印墙钟 2768.8 ms，materialize 合计 32.6 ms / 1313 次（单次 ~25 μs），占 **1.18%**。测量方式：7 处临时包计时器（临时代码与临时测试已删，数据落档于此与任务报告）。 +- **与 main 基线对比**：main 的成员节点本就是类树（`NodeCache` 常驻），零 materialize 但常驻内存只涨不落——正是迁移要治的病。本条是迁移代价的一部分，且是瞬态代价。 +- **为什么不修**：根治需把 `NodePrintable` 五协议栈（`NodePrintables/` + `NodePrinter/`，约 1700 行）泛型化到 `DemanglingNode`，其中 3 处**构造**节点的逻辑（`Variable/Function/SubscriptNodePrinter` 的 `.static` 包装、`SubscriptNodePrinter` 与 `FunctionTypeNodePrintable` 的 labelList 合成）纯引用无法表达，需局部重设计；print cache 的 `ObjectIdentifier` 键也要按表示异构化。~1% 的收益撑不起这个投入与回归风险。 +- **既往修复**：无既往修复；`TypedDumper`（dump 路径）保留独立实现是记录在案的设计（AGENTS.md）。 +- **代码锚点**：不加代码注释（7 处太散），以本条目为准。 +- **复审条件**：① 大镜像（SwiftUI 级）剖析显示 materialize 占比显著高于 fixture 的 1.18%；② 上游 Remangler / 打印基础设施泛型化（A1 复审条件 ①）落地后，节点合成问题若有上游方案可顺路重开。 diff --git a/Documentations/Internal/Reviews/2026-08-02-node-store-migration-pr97-review.md b/Documentations/Internal/Reviews/2026-08-02-node-store-migration-pr97-review.md index e84adf81..5b20e190 100644 --- a/Documentations/Internal/Reviews/2026-08-02-node-store-migration-pr97-review.md +++ b/Documentations/Internal/Reviews/2026-08-02-node-store-migration-pr97-review.md @@ -26,7 +26,7 @@ **状态：已随上游 0.5.0 关闭。** 上游把上限恢复为 768，并在源码注释里记下了原因与禁止再降的约束（"Downstream consumers reported `<>` on ordinary SwiftUI and similarly generic-heavy modules under the 512 limit … Do not lower this again without corpus evidence gathered from downstream workloads."）。本仓库随依赖升级到 `from: "0.5.0"` 后自动获得，无需本地改动。 -### 2. `indexExtensions` 丢失 `await`，从任务挂起退化为线程阻塞 +### 2. ~~`indexExtensions` 丢失 `await`，从任务挂起退化为线程阻塞~~ ✅ 已修（2026-08-03；先裁决不修，后被新事实推翻——见第三节第 3 条） `Sources/SwiftIndexing/SwiftDeclarationIndexer.swift:685`。 @@ -35,6 +35,8 @@ 这个循环对每个 extension target 执行一次，位于 `async` 的索引流程内。`NodeReference` 上不存在 async 版 `print`，所以调用点看不出任何退化痕迹。 +机制描述成立（同步版确实阻塞而非挂起）。最初裁决为不修，其后上游把 print 便利方法整体迁到 `DemanglingNode` 协议扩展并补了 async 变体（`f913742`，发布为 **0.5.1**；`Store/DemanglingNode.swift`，挂起 + 大栈线程，对 `NodeReference` 同样生效），「需等上游补 async `print`」的前提当日即失效，一个 `await` 恢复了 main 的挂起语义，已落地。完整经过见第三节第 3 条。 + ### 3. `withLargeStack` 包住整趟 sweep，会占住一条线程整块时长 `Sources/MachOSymbols/SymbolIndexStore.swift:352`。 @@ -45,7 +47,7 @@ 正确形态是第三种：走异步入口（挂起而非阻塞），或使用专用线程。 -### 4. `NodeReference(interning:)` 在批量路径上被逐个调用（26 处） +### 4. ~~`NodeReference(interning:)` 在批量路径上被逐个调用（26 处）~~ ✅ 已修（2026-08-03，`InternedNodeReferenceCache`） 分布：`SwiftDeclaration/Extensions.swift` 14 处、`SwiftIndexing/SwiftDeclarationIndexer.swift` 6 处、`SwiftSpecialization/` 4 处、`SwiftDeclaration/Components/Definitions/ProtocolDefinition.swift` 1 处、`MachOSymbols/StructuralNodeReferenceKey.swift` 1 处。 @@ -57,6 +59,8 @@ > 与 `main` 的关系：`main` 没有这个形态，但有它自己的病（全局 `NodeCache` 只涨不落），而那正是本次迁移要治的。所以这是**代价而非退步**，只是这个代价可以不付。 +**修复（2026-08-03）**：批前实测推翻了「共用一个 builder」的原始修法——名字在调用流深处即用即取，freeze 前无法发引用，两阶段重排不可行；改为**结构去重缓存** `InternedNodeReferenceCache`（`Sources/MachOSymbols/`）：`SharedCache` 派生，镜像键 + 进程键（无 `machO` 的 in-process 助手）双入口，桶按 `Node` 结构哈希、命中经 `structurallyEquals` 校验，minting 沿用锁外构造 + 锁内 insert-if-absent。25 处真实调用点（第 26 处是注释）全部改走缓存；`SwiftDeclarationIndexer` 的 per-image 清理与内存压力驱逐都接上。fixture 实测：745 个驻留名字引用中 730 个 mini store 降到 **471**（= 472 个结构唯一树，去重完全），驻留字节 84,795 → 57,752（−32%）；重复名共享 store 后 `store ===` 快路径开始生效。真实框架上 conformance 扇出（180 处 `conformingProtocolName` 对少数协议名）会放大收益。快照套件逐字节不变。 + ### 5. `buildPipelineStaysOffGlobalNodeCache` 的断言本身不成立 `Tests/MachOSymbolsTests/SymbolIndexStoreFixtureTests.swift:53`。 @@ -85,10 +89,12 @@ （上一轮第 5 条讲的是 `:133` 缺少提前退出导致全遍历，与本条不是同一个缺陷。） -### 8. `ClassDumper.distributedFunctionNodes` 未记忆化，每个 actor 类求值两次 +### 8. `ClassDumper.distributedFunctionNodes` 未记忆化，每个 actor 类求值两次 —— 部分处理（2026-08-03） `Sources/SwiftDump/Dumper/ClassDumper.swift:77` 是一个 `private var … : Set` 计算属性，在 `:101`（`try? distributedFunctionNodes) ?? []).isEmpty == false`）和 `:192`（`let distributedFunctionNodes = (try? self.distributedFunctionNodes) ?? []`）各求值一次。每次都重建整个 thunk 符号数组，并为每个 thunk materialize 两棵树。 +**处理（2026-08-03）**：集合改存 `StructuralNodeReferenceKey`（引用形态直接入键），每 thunk 少一次树 materialize；成员循环的探测端（`isDistributedMethod`）改用引用形态的函数节点，同批随 dump 路径引用化落地。**双求值本身保留**：`ClassDumper` 是 struct，属性有 `guard isActor` 提前返回（非 actor 类零成本），`body` 内已提前收敛为局部变量——剩余成本只落在真 actor 类上、每类两次且每次比之前便宜，不值得为它引入引用盒。 + ## 二、本轮已闭环 ### 1. swift-demangling 依赖改回版本要求 ✅ @@ -133,6 +139,24 @@ - `memberSymbols(of:excluding:in:)` 包内唯一调用点是 `SwiftDeclarationIndexer.swift:663`，在 `:684` 只做 `for (node, memberSymbols) in memberSymbolsByName` 遍历，全程无下标查询；且返回字典的键全部出自同一个 `storage.nodeStore`，同 store 内下标相等本就是正确的去重语义。 - `allOpaqueTypeDescriptorSymbols(in:)` 在 `Sources/` 与 `Tests/` 中**零调用点**。 +### 3. ~~`indexExtensions` 丢失 `await`（同步 print 阻塞协作线程）—— 不修~~ —— 裁决被推翻，已修（2026-08-03 当日） + +对应本文第一节第 2 条。**裁定：不修**（维护者裁决，2026-08-03）。**同日推翻**：裁决对 0.5.0 而言前提无误（0.5.0 的 async `print` 确实只在 `Node` 上），但上游随即把 print 便利方法整体迁到 `DemanglingNode` 协议扩展并补 async 变体（`f913742` "move the print conveniences to DemanglingNode and add an async variant"，同日发布为 **0.5.1**："Suspends the calling task instead of blocking a cooperative worker"）——对 `NodeReference` 直接可用，无需 materialize。依赖已升至 `from: "0.5.1"`，修复即恢复 `await node.print(...)` 一处（`SwiftDeclarationIndexer.swift`），语义回到 main 的任务挂起。注意 0.5.1 同时**删除了** `Node` / `NodeReference` 上的具体同步 `print`，async 上下文里编译器会强制 `await`（本仓库另有三处 dump 路径 print 随引用化一并加了 `await`）。下面保留原始裁决全文供回溯。 + +维护者最初的理由是「之前改过，改成 `await` 进协作线程池就只有 512KB 栈了」。核实结论：**该事故真实存在过，但其成因上游已修，不能再作为不修的理由**；不修的成立理由是下面的影响面判断。 + +历史核实（`git log` 追溯）： + +- 512KB 爆栈是 0.4.3 时代的真问题——当时打印/demangle 路径没有任何栈保护，async 化后递归直接跑在协作线程的 512KB 栈上。上游 `95dd741`（"add stack-safe execution and async API overloads"）的提交信息原话：*"Cooperative pool workers default to 512KB stacks on Darwin, which the recursive demangler/remangler can blow on deeply nested generic types."* +- 但**同一个提交**就是修复：async 重载的设计是「挂起调用方 + 把递归丢到 8MB 大栈线程」——*"async overloads that suspend via a continuation instead of blocking a cooperative worker"*。0.5.0 的 `executeAsync` 实现核实过：当前线程栈够则内联；不够（协作线程必然不够）则提交给大栈线程池并用 continuation 挂起，池满则退化为专用线程。递归**从不**落在 512KB 栈上。`main` 上 `47b5961` 写的 `await node.print(...)` 用的正是这个安全入口。 + +因此同步版与 `await` 版走的是**同一套大栈机制，谁都不会爆栈**；唯一差异是等结果时调用线程「阻塞在信号量上」还是「挂起让出」。这从头到尾是并发吞吐问题，不是正确性问题。**留档警示：后续不要把本条的不修理由复述成「`await` 会跑在 512KB 栈上」**——那是 0.4.5 之前的旧行为，写进理由会误导后人。 + +不修的实际理由（影响面）： + +- 损失只在**并行索引多个镜像**时显现（N 条协作线程被占住）；单镜像顺序索引时，阻塞一条线程 ≈ 顺序执行，没有可省的墙钟时间，且每次 print 很短。 +- 修起来并不干净：循环里的 `node` 是 `NodeReference`，0.5.0 的 async `print` 只在 `Node` 上有。一行修法 `await node.materialize().print(...)` 要为每个 extension target 多建一棵树（恰是迁移要消灭的动作）；干净修法需要上游给 `NodeReference` / `DemanglingNode` 补 async 重载。若将来上游补了，此条可以一行改回，届时再顺手做。 + ## 四、更正 ### 1. 更正本轮自身：`memberSymbols` 的"O(1) 退化"不成立 @@ -181,18 +205,20 @@ ## 六、待处理清单增量 -上一轮第四节的 17 条清单继续有效（除本文第三节裁决为不修的两条、第二节闭环的一条外）。本轮在其上新增： +上一轮第四节的 17 条清单继续有效（除本文第三节裁决为不修的三条、第二节闭环的一条外）。本轮在其上新增： **建议合并前修** -1. `indexExtensions` 丢失 `await`（第一节第 2 条）——一行改动能否恢复取决于 `NodeReference` 是否补 async `print`；若上游不补，需在渲染循环层面另作安排。 +1. ~~`indexExtensions` 丢失 `await`（第一节第 2 条）~~——**已修**（2026-08-03；先裁决不修，后上游 0.5.1 把 async `print` 落到 `DemanglingNode` 上，一行恢复 `await`，见第三节第 3 条）。 **可排期** 2. `withLargeStack` 占住整条线程（第一节第 3 条）——与上一轮第 1、2 条同属"线程跳转形态"课题，宜合并设计。 -3. 26 处 `NodeReference(interning:)` 改为每镜像共用 builder（第一节第 4 条）。 +3. ~~26 处 `NodeReference(interning:)` 改为每镜像共用 builder（第一节第 4 条）~~——**已修**（2026-08-03，`InternedNodeReferenceCache` 结构去重缓存，见第一节第 4 条的修复记录）。 4. dyld `:73` 判定锚定到直接父目录（第一节第 7 条）——**这条是正确性问题（非确定性），优先级高于同文件 `:133` 的性能问题**。 -5. `distributedFunctionNodes` 记忆化（第一节第 8 条）。 +5. ~~`distributedFunctionNodes` 记忆化（第一节第 8 条）~~——**部分处理**（2026-08-03，引用键化省掉每 thunk materialize；双求值保留，成本已收窄到真 actor 类，见第一节第 8 条）。 + +另注（2026-08-03 性能批次对上一轮清单的影响）：上一轮四.3（失败名不缓存 + 持锁重试）与四.4 / 台账第 9 条（`lateDemangledNode` 持锁 demangle）**已修**；四.16 / 台账第 10 条（`ProtocolConformanceDumper` materialize 分支）**已修**，并连同 dump 路径其余四处 `demangleSymbol` 调用点一并引用化（四.7 的热调用方随之清零）；四.6（打印器每成员 materialize）**裁决为暂不修**——实测其占打印墙钟仅 **1.18%**（fixture 全量导出，1313 次共 32.6 ms），而根治需要把 1700 行打印栈泛型化到 `DemanglingNode` 并重设计 3 处节点合成（`.static` 包装、labelList），投入产出不成比例；若未来在大镜像上剖析出不同占比可重开。 **测试** diff --git a/Documentations/Internal/TaskReports/2026-08-03-performance-batch-fixes.md b/Documentations/Internal/TaskReports/2026-08-03-performance-batch-fixes.md new file mode 100644 index 00000000..c935c18e --- /dev/null +++ b/Documentations/Internal/TaskReports/2026-08-03-performance-batch-fixes.md @@ -0,0 +1,45 @@ +# 2026-08-03 性能批次：失败名裁决、名字去重缓存、dump 路径引用化 + +## 问题 + +[2026-08-02 审查记录](../Reviews/2026-08-02-node-store-migration-pr97-review.md)与既有台账合并后，`feature/node-store-migration` 剩 19 条待处理。维护者批准「立即可修」与「中等重构」两组同批执行： + +1. 失败名重试 + 锁内 demangle（`SymbolIndexStore`，上一轮实测失败名路径 6.3 倍慢、8 线程争用 1.97 倍）； +2. 25 处 `NodeReference(interning:)` 逐名开 arena； +3. 打印器每成员 materialize（7 处）； +4. dump 路径 `demangledNode` 调用点（5 处，每次 `materialize()`）。 + +## 调研 + +- **失败名**：`demangledNodeReference` 的快路径条件是「表内有行**且** root 非 nil」，于是 sweep 期间 demangle 失败的名字每次查询都穿透到 `lateDemangledNode`，在锁内重付一次失败的 demangle。核对上游源码：`NodeStoreBuilder.demangle` 就是 `demangleAsNodeTransient` + intern——与 sweep 同一个 demangler、同一拒绝集，所以「表内失败 = 永远失败」，表裁决可以直接回答。 +- **interning 批量化**：原修法「每镜像共用一个 builder」不可行——`NodeReference` 必须在 `freeze()` 之后才能发出，而名字在调用流深处即用即取（字典键、定义构造），两阶段重排等于重写索引器。一次性测量（临时测试，已删）证明真正的浪费是**重复**：fixture 上 745 个驻留名字引用、730 个 mini store、只有 472 个结构唯一树（重复率 1.55×；`conformingProtocolName` 180 处全是重复重灾区）。因此正确形态是**结构去重缓存**，不是共享 builder。 +- **打印器 materialize**：临时计量（7 处包 `TemporaryMaterializeProfiling`，已删）——fixture 全量导出打印墙钟 2768.8 ms，materialize 合计 32.6 ms / 1313 次，占 **1.18%**。而根治需把 `NodePrintable` 五协议栈（约 1700 行）泛型化到 `DemanglingNode`，且栈内有 3 处**构造** Node 的地方（`.static` 包装 ×3、labelList 合成 ×2）纯引用无法表达，需局部重设计。 +- **dump 路径**：五个调用点拿到 `Node` 后只做 `first(of:)` / `print` / `resolve(for:)` / visited 去重——全部有引用等价物；`DemangleResolver.resolve(for:)` 已有 `some DemanglingNode` 重载；`ProtocolConformanceDumper:184` 已示范引用化写法。 +- **意外发现**：`indexExtensions` 丢 `await` 那条早间刚裁决「不修」（前提「`NodeReference` 无 async print」对 0.5.0 属实），上游当日就把 print 便利方法整体迁到 `DemanglingNode` 协议扩展并补 async 变体（`f913742`，发布为 **0.5.1**；同时删除 `Node`/`NodeReference` 的具体同步 `print`）——前提失效，改裁决为已修，依赖随之升 `from: "0.5.1"`。本地兄弟检出先于发布就到了该内容，这也是构建中 async 上下文突然强制 `await` 的真实原因（当时误归因于 scratch 污染的部分已在下文更正）。 + +## 方案 + +1. `demangledNodeReference`：表内名字以表裁决为准（root 为 nil → 返回 nil）；`lateDemangledNode`：锁外 demangle + 锁内 insert-if-absent（输者弃 store 返回胜者），字典改 `[String: NodeReference?]` 缓存拒绝裁决（用 `updateValue` 规避 optional-值字典下标赋 nil 即删键的坑）。 +2. 新增 `InternedNodeReferenceCache`（`Sources/MachOSymbols/`）：`SharedCache` 派生，镜像键（`storage(in:)`）+ 进程键（type-keyed `storage()`，服务无 `machO` 的 in-process 助手）双入口；桶按 `Node` 结构哈希、命中经 `structurallyEquals` 校验；minting 与 late 缓存同款锁外构造 + 锁内仲裁。25 处调用点全部改走缓存（`Extensions.swift` 14、`SwiftDeclarationIndexer` 6、`SwiftSpecialization` 4、`ProtocolDefinition` 1）；`SwiftSpecialization` 补 `.target(.MachOSymbols)` 依赖；`SwiftDeclarationIndexer.deinit` 的 per-image 清理加一行。 +3. 打印器 materialize：**裁决为暂不修**（1.18%，重开条件：大镜像剖析出显著占比）。 +4. dump 路径：五处迁 `demangleSymbolReference`；`validNode` ×2 返回 `NodeReference?`；visited 集合 ×4 与 `distributedFunctionNodes` 换 `StructuralNodeReferenceKey`；`_requirementName` 直接引用打印；`indexExtensions` 恢复 `await`。 + +## 实际执行 + +按方案落地，代码改动集中在 `SymbolIndexStore.swift`、新文件 `InternedNodeReferenceCache.swift`、`Extensions.swift`、`SwiftDeclarationIndexer.swift`、三个 Dumper、`Package.swift`（一行依赖）。新增测试： + +- `SymbolIndexStoreFixtureTests`：`rejectedLateNameCachesItsFailure`（**修复前红**——临时还原旧行为验证，断在裁决未被缓存）、`tableCoveredNameNeverEntersLateCache`、`concurrentLateQueriesShareOneStore`（16 任务并发同名，胜者唯一）。 +- 新套件 `InternedNodeReferenceCacheTests`（5 项）：重复共享 store（`store ===`）、不同树不混淆、并发单胜者、`remove(for:)` 驱逐后重铸、进程作用域去重。 +- 为可测性在 `Storage` 加了内部钩子 `lateDemangleVerdictForTesting(forName:)`、`InternedNodeReferenceCache.Storage.cachedReferenceCountForTesting`。 + +## 验证 + +- fixture 复测：驻留名字 mini store 730 → **471**（= 472 结构唯一，去重完全），mini store 字节 84,795 → 57,752（−32%）。 +- 快照套件全绿：SwiftInterfaceTests 53 项（含逐字节 interface 快照）、SwiftIndexingTests / SwiftSpecializationTests / MachOSymbols 各套件 146 项、SwiftDumpTests（随全量套件）。输出零变化。 +- 全量 `swift test --skip IntegrationTests`（独立 scratch 干净构建）：见任务末尾结果。 + +## 偏离与教训 + +- **两次环境事故，一次误归因**：(a) 本机 fixture 二进制（7/31）旧于 fixture 源码（8/2 加的 `AccessorFunctionReferences`），interface 快照丢整块尾部类型——AGENTS.md 环境漂移检查第 1 条的教科书案例，按处方重建后消除；(b) 会话中 shell 工作目录被重置回主仓库（main 分支），若干构建/测试跑错了检出，且主仓库与 worktree **共用了同一个 SwiftPM scratch 目录**，混入 main 分支（0.4.x 符号时代）的陈旧目标文件，制造出「测试匹配 0 项」和链接期 undefined symbol——worktree 换独立 scratch（`/tmp/claude/SwiftPM/MachOSwiftSection-worktree-node-store`）干净重建后消失。**教训：不同检出绝不共用 scratch。** 更正：当时一并归因于 scratch 污染的「async 上下文强制 `await`」**不是**污染假象——那是兄弟检出已含 0.5.1 内容（具体同步 `print` 被删）的真实编译语义，`await` 修复因此是必要且正确的。 +- 打印器 materialize 从「修」改为「数据裁决不修」——先测后改的纪律恰好挡下了一次高风险低收益重构。 +- `indexExtensions` 的 await 裁决当日两翻：不修 →（新事实）→ 已修。错误前提（漏查 `DemanglingNode` 协议扩展）已在审查记录第三节原文处标注。 diff --git a/Documentations/README.md b/Documentations/README.md index 2883abac..37b6bf65 100644 --- a/Documentations/README.md +++ b/Documentations/README.md @@ -46,6 +46,7 @@ required by `Version.swift`'s bump contract). | Doc | What it covers | |---|---| | [ProjectEvolutionLog.md](Internal/ProjectEvolutionLog.md) | 编年演进账本：逐工作弧（Foundation 解析 → demangler → 模块化 → SwiftLayout → SwiftDiffing/ABI evolution …）的时间段/动机/关键决策/落地文档/版本对应，含每批次必须追加的维护约定。 | +| [ReviewAdjudications.md](Internal/ReviewAdjudications.md) | Review 已裁决清单：判定为「不修 / 误报」的发现及结论、理由、复审条件；每轮 code review 先对照此表，已裁决且理由仍成立的直接跳过。 | | [SwiftModularizationMigration.md](Internal/SwiftModularizationMigration.md) | The `SwiftInterface` monolith → layered peer modules refactor; where everything moved. | | [FieldMetadataRenderingMigration.md](Internal/FieldMetadataRenderingMigration.md) | Extracting metadata-derived field rendering into `SwiftDeclarationRendering` (single source for dumper + printer). | | [FieldLayoutRendererReaderSpecialization.md](Internal/FieldLayoutRendererReaderSpecialization.md) | Splitting `FieldLayoutRenderer` into a generic facade dispatching to two reader-specialized implementations: the `MachOImage` runtime path (in-process metadata) and the `MachOFile` static path (offline field offsets / type layouts / expanded tree / enum layouts via `SwiftLayout`). Covers the `self as?` dispatch, the `StaticFieldLayoutProvider` injection seam (built once per session), the new SwiftLayout convenience APIs, graceful degradation, and the `typeLayoutTransformer`/tuple limitations. | @@ -74,6 +75,6 @@ required by `Version.swift`'s bump contract). | [ClassMemberKeywordRecovery.md](Internal/ClassMemberKeywordRecovery.md) | `class` / `static` 成员关键字的还原：mangling 层面两者不可区分，判据是「类型级成员有 vtable method descriptor ⇒ 源码是 `class`」（`static` 隐式 final、不进 vtable）；模型侧 `isClassMember` 计算属性 + 三个 node printer 接线 + dump vtable 段落关键字，顺带消灭非法的 `override static` 输出；`final class func` 等四类 ABI 上与 `static` 完全一致，保守输出语义等价的 `static`。 | | [NodeStoreMigrationPlan.md](Internal/NodeStoreMigrationPlan.md) | `SymbolIndexStore` 迁移到 `NodeStore` arena 存储的分期计划与实施记录（Stage 0–5）：12B/节点扁平缓冲、cache-free build sweep、Symbol 表压缩、声明层换持 `NodeReference`、散点 transient demangling；含 Stage 5a 的 override/vtable 回归修复（裸 `[NodeReference: …]` 的 store-identity 陷阱 → `StructuralNodeReferenceKey`）与三源快照验收。 | | [DeclarationModelMemoryFootprint.md](Internal/DeclarationModelMemoryFootprint.md) | NodeStore 迁移后的声明模型内存足迹量测：`TypeDefinition` 1272 字节的逐属性构成（两份 `TypeContextWrapper` 占 74%）、`TypeContextWrapper` 按最大 case `Class` 定尺的原因、`parentContext` 是被当成永久字段的临时值、mini-store 增殖与 `MetadataReaderCache` 仍持 `Node` 树；含四项可回收估算（合计 ~8–10%）与「当前不建议实施、应先剖析其余 90%」的结论。 | -| [NodeStoreMigrationOpenIssues.md](Internal/NodeStoreMigrationOpenIssues.md) | `feature/node-store-migration` 的**已确认问题清单**（2026-07-28 两轮审查 + 复核）。第一节是前一轮修复自身的两处缺口（Catalyst 降级只覆盖 framework 形态导致 plain dylib 仍平局、`appendRowIfAbsent` 线性扫描），已于 2026-07-29 闭环并保留成因；第二节起为仍然打开或已裁决的条目：`structuralHash` 每文本节点分配 `String`（修复位置在上游，`0.5.0` 仍未修）、`memberSymbols` 改为线性 + 全树比对（**已证实量级可忽略，不是回归**）、build sweep 串行且每符号跨线程往返（上游）、`ABIKey` 每 key materialize，以及四条代码卫生项；两个公开查询 API 的字典键一条**已于 2026-08-03 裁决为不修**（类型级 SPI），rebase 前置事项一条前提已过期但压着两条仍成立的注意事项。逐条注明成因、影响面与「该在哪里修」。与 `Reviews/` 下两份审查记录冲突时以后者为准。 | +| [NodeStoreMigrationOpenIssues.md](Internal/NodeStoreMigrationOpenIssues.md) | `feature/node-store-migration` 的**已确认问题清单**（2026-07-28 两轮审查 + 复核）。第一节是前一轮修复自身的两处缺口（Catalyst 降级只覆盖 framework 形态导致 plain dylib 仍平局、`appendRowIfAbsent` 线性扫描），已于 2026-07-29 闭环并保留成因；第二节起为仍然打开或已裁决的条目：`structuralHash` 每文本节点分配 `String` 与 `ABIKey` 每 key materialize 两条**已按上游设计终审关闭**（0.5.1 保持现状且上游说明不改，见 ReviewAdjudications A1/A2）、`memberSymbols` 改为线性 + 全树比对（**已证实量级可忽略，不是回归**）；`lateDemangledNode` 持锁 demangle 与 `ProtocolConformanceDumper` 的 materialize 分支两条**已于 2026-08-03 闭环**（连同失败名裁决缓存、dump 路径引用化）、build sweep 串行且每符号跨线程往返（上游）、`ABIKey` 每 key materialize，以及四条代码卫生项；两个公开查询 API 的字典键一条**已于 2026-08-03 裁决为不修**（类型级 SPI），rebase 前置事项一条前提已过期但压着两条仍成立的注意事项。逐条注明成因、影响面与「该在哪里修」。与 `Reviews/` 下两份审查记录冲突时以后者为准。 | | [TaskReports/](Internal/TaskReports/) | Dated per-task fix / investigation reports. | | [Reviews/](Internal/Reviews/) | Dated code-review records — one file per review event: findings, measurements taken at the time, and the resulting to-do list. Distinct from the topic-organised issue ledgers (e.g. `NodeStoreMigrationOpenIssues.md`), which they cross-reference. | diff --git a/Sources/SwiftDiffing/ABIKey.swift b/Sources/SwiftDiffing/ABIKey.swift index 49611e20..415beea3 100644 --- a/Sources/SwiftDiffing/ABIKey.swift +++ b/Sources/SwiftDiffing/ABIKey.swift @@ -50,6 +50,14 @@ public enum ABIKey: Hashable, Sendable, Codable { public static func make(for node: some DemanglingNode) -> ABIKey { // `canMangle` is literally `(try? mangleAsString) != nil`, so a single // `try?` decides the branch and remangles exactly once. + // + // Adjudicated — not worth fixing (see + // Documentations/Internal/ReviewAdjudications.md): for store-backed + // nodes this `mangleAsString` overload bridges by materializing the + // subtree once (upstream RemangleInterface.swift documents the + // tradeoff; the root fix is upstream's planned Remangler + // genericization). One transient O(subtree) build per key, exactly + // once, no resident memory — do not re-flag. if let mangled = try? mangleAsString(node) { return .mangled(mangled) } From bcf49f239e16d4df3ab77162299798034aeebfeb Mon Sep 17 00:00:00 2001 From: Mx-Iris Date: Mon, 3 Aug 2026 18:28:13 +0800 Subject: [PATCH 35/77] test(verification): record the system-framework rendering A/B procedure Freeze the main-vs-branch output-parity sweep as a repeatable procedure: a Python runner that builds both checkouts' release CLIs, renders dump + interface for SwiftUI/SwiftUICore/SwiftData/Combine/ActivityKit/WidgetKit through all three reader paths (archived dyld caches, simulator-runtime Mach-O files, in-process MachOImage via RenderingVerificationTests), and byte-compares every pair. Archived caches fall back to the current system's dyld shared cache and the preferred simulator runtimes fall back to whatever is installed, so the sweep runs on any machine. The 2026-08-03 baseline run (main vs feature/node-store-migration) produced 78 identical pairs across the macOS 26.5.2/15.5 caches, the iOS 15.5/18.5/26.5 simulators, and the current system in-process. The run record, the pitfalls (-p full-path disambiguation of iOSSupport copies, -a arm64 for fat slices, the same-boot requirement for member addresses, per-checkout scratch isolation), and the fallback rules live in the new procedure document. AGENTS.md gains the mandatory-after-large-refactors rule and marks RenderingVerificationTests as the sole IntegrationTests exception. --- AGENTS.md | 4 +- .../Internal/ProjectEvolutionLog.md | 13 + .../SystemFrameworkRenderingVerification.md | 54 ++++ ...026-08-03-system-framework-rendering-ab.md | 36 +++ Documentations/README.md | 1 + Scripts/run-rendering-ab-verification.py | 278 ++++++++++++++++++ 6 files changed, 385 insertions(+), 1 deletion(-) create mode 100644 Documentations/Internal/SystemFrameworkRenderingVerification.md create mode 100644 Documentations/Internal/TaskReports/2026-08-03-system-framework-rendering-ab.md create mode 100755 Scripts/run-rendering-ab-verification.py diff --git a/AGENTS.md b/AGENTS.md index ecbfd144..82410c4d 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -46,7 +46,9 @@ swift run swift-section transformer tokens Requires Swift 6.2+ / Xcode 26.0+. -**Test suite convention:** `Tests/IntegrationTests/` is for the maintainer's manual inspection only — it prints results with no assertions or preconditions. Agents must not run it (use `--skip IntegrationTests` when running the full suite). All other `*Tests` targets have proper assertions and required preconditions, and are safe to run. +**Test suite convention:** `Tests/IntegrationTests/` is for the maintainer's manual inspection only — it prints results with no assertions or preconditions. Agents must not run it (use `--skip IntegrationTests` when running the full suite). All other `*Tests` targets have proper assertions and required preconditions, and are safe to run. Sole exception: `RenderingVerificationTests` as the MachOImage leg of the rendering A/B verification below. + +**Rendering A/B verification (mandatory after any large refactor):** any refactor touching demangling, printing, indexing, or the reader stack must pass `Scripts/run-rendering-ab-verification.py ` — byte-identical dump + interface output over real system frameworks (SwiftUI/SwiftUICore/SwiftData/Combine/ActivityKit/WidgetKit) through all three reader paths: archived dyld caches (falls back to the current system's cache when absent), simulator-runtime Mach-O files (falls back to whatever runtimes are installed), and in-process MachOImage via `RenderingVerificationTests`. Procedure, fallback rules, and known pitfalls: [Documentations/Internal/SystemFrameworkRenderingVerification.md](Documentations/Internal/SystemFrameworkRenderingVerification.md). ## Architecture Overview diff --git a/Documentations/Internal/ProjectEvolutionLog.md b/Documentations/Internal/ProjectEvolutionLog.md index 91732da2..70560b59 100644 --- a/Documentations/Internal/ProjectEvolutionLog.md +++ b/Documentations/Internal/ProjectEvolutionLog.md @@ -698,6 +698,19 @@ --- +## 25. 系统框架渲染 A/B 验证：78 对零差异 + 流程固化 + +- **时间段**：2026-08-03（紧接第 24 节的性能批次）。 +- **动机**：性能批次落地后，用真实 OS 框架对 `feature/node-store-migration` 做全面的输出对等验证——fixture 快照覆盖构造形态，但覆盖不了 10 万行级输出规模、iOS 15 时代的历史 metadata 与三种 reader 路径的全量组合；维护者随后要求把这套测试固化为「大重构必跑」的流程。 +- **落地**： + - **验证结果**：main ↔ feature 双侧 release CLI + `RenderingVerificationTests` harness，SwiftUI / SwiftUICore / SwiftData / Combine / ActivityKit / WidgetKit 六框架，三部分共 **78 对输出全部逐字节一致**——DyldCache（macOS 26.5.2 + 15.5 归档 cache，24 对）、MachOFile（iOS 15.5 / 18.5 / 26.5 模拟器 runtime，30 对）、MachOImage（当前系统 in-process + 当前 cache 文件，全选项，24 对）。附带 fixture（SymbolTestsCore）smoke 亦一致。 + - **流程固化**：新增 [`Scripts/run-rendering-ab-verification.py`](../../Scripts/run-rendering-ab-verification.py)（自动构建双侧、三部分渲染、逐对 diff、差异非零退出；归档 cache 缺失回退当前系统 cache，指定模拟器缺失回退现有 runtime）与流程文档 [SystemFrameworkRenderingVerification.md](SystemFrameworkRenderingVerification.md)；AGENTS.md 增设「大重构后必跑」规则，并把 `RenderingVerificationTests` 登记为 IntegrationTests 禁跑规则的唯一例外。 +- **关键决策**：cache 镜像一律 `-p` 全路径（iOSSupport 副本消歧）；模拟器一律 `-a arm64`（15.5/18.5 为 fat 二进制）；MachOImage 双侧必须同一次开机会话（memberAddress 依赖 per-boot cache slide）；interface 输出走 `-o` 使时间戳日志与被比对内容分离。 +- **关联文档**：[SystemFrameworkRenderingVerification.md](SystemFrameworkRenderingVerification.md)、[TaskReports/2026-08-03-system-framework-rendering-ab.md](TaskReports/2026-08-03-system-framework-rendering-ab.md)。 +- **对应版本**：0.14.0 之后未发布区间（`feature/node-store-migration` 分支）。 + +--- + ## 维护约定 1. **每个非平凡批次结束时必须在本文追加/更新一节**（新工作弧新增一节；延续既有弧则在该节 diff --git a/Documentations/Internal/SystemFrameworkRenderingVerification.md b/Documentations/Internal/SystemFrameworkRenderingVerification.md new file mode 100644 index 00000000..692f1346 --- /dev/null +++ b/Documentations/Internal/SystemFrameworkRenderingVerification.md @@ -0,0 +1,54 @@ +# 系统框架渲染 A/B 验证（大重构必跑） + +对 demangling / 打印 / 索引 / reader 栈做**任何大重构**后，都必须用真实系统框架跑一遍本流程：同一批输入、两个检出（基线 + 重构分支）、release CLI，逐字节比对 dump 与 interface 输出。fixture（SymbolTestsCore）覆盖的是构造出来的形态，真实 OS 框架才覆盖规模（10 万行级输出）、历史 metadata 格式（iOS 15 时代）与三种 reader 路径的全量组合。 + +**入口脚本**：[`Scripts/run-rendering-ab-verification.py`](../../Scripts/run-rendering-ab-verification.py) + +```bash +Scripts/run-rendering-ab-verification.py <基线检出> <重构检出> \ + [--output-root 目录] [--frameworks A,B,...] \ + [--baseline-scratch 目录] [--candidate-scratch 目录] [--skip-image-part] +``` + +脚本自动构建两侧 release CLI、跑完三部分、输出逐对 IDENTICAL/DIFFERS 表格；有任何差异以非零码退出。 + +## 框架清单 + +SwiftUI、SwiftUICore、SwiftData、Combine、ActivityKit、WidgetKit——**输入源里不存在的直接略过**（例如 iOS 15.5 没有 SwiftUICore/SwiftData/ActivityKit；macOS 15.5 cache 的 ActivityKit 只有 iOSSupport/Catalyst 副本，脚本会自动改用该路径）。 + +## 三个 reader 部分与输入源回退规则 + +| 部分 | 首选输入 | 目标不存在时的回退 | +| --- | --- | --- | +| **DyldCache**（cache 内 MachOFile） | 归档 cache：`/Volumes/DyldSharedCaches/macOS/26.5.2_25F84` 与 `15.5_24F74` 的 `dyld_shared_cache_arm64e` | **当前系统的 dyld shared cache**（`--uses-system-dyld-shared-cache -p <镜像路径>`，不传文件参数） | +| **MachOFile**（磁盘上的普通 Mach-O） | iOS 15.5 / 18.5 / 26.5 模拟器 runtime 的框架二进制 | **当前环境已安装的全部 iOS 模拟器 runtime**（脚本自动发现 `/Library/Developer/CoreSimulator/Profiles/Runtimes` 与 `/Library/Developer/CoreSimulator/Volumes/*/…/Runtimes` 下的 `*.simruntime`） | +| **MachOImage**（进程内） | 当前系统（dlopen + `MachOImage(name:)`），经 `RenderingVerificationTests` harness | 无回退（永远是当前系统） | + +## 关键调用细节（踩过的坑） + +- **cache 镜像用 `-p` 全路径而非 `-n` 名字**：SwiftUI / WidgetKit / ActivityKit 在 macOS cache 里有 `/System/iOSSupport/` 下的 Catalyst 副本，按名字查有歧义。 +- **模拟器二进制要显式 `-a arm64`**：iOS 15.5 / 18.5 的模拟器框架是 fat（x86_64 + arm64），CLI 遇 fat 文件不指定架构会直接报错退出；26.5 起是 thin arm64，加该参数也无害，所以脚本一律加。 +- **MachOImage 部分借用 `RenderingVerificationTests`**（`Tests/IntegrationTests/SwiftInterface/`）：该 harness 的注释明言其设计用途就是「run on two checkouts … and diff」。这是 AGENTS.md「agent 不得运行 IntegrationTests」规则的**唯一例外**，仅限本流程。 +- **`RV_OPTS` 不含 `expandedFieldOffsets`**：harness 注释记录了它在 SwiftUI 级深嵌套泛型的 MachOImage 路径上会触发既有的栈溢出。 +- **MachOImage 两侧必须在同一次开机会话内运行**：`memberAddress` 注释里的地址来自 dyld shared cache 的 per-boot slide，跨重启比对必然全线假差异。 +- **两个检出绝不共用 SwiftPM scratch**（AGENTS.md 环境漂移检查的血泪教训：混入另一分支的陈旧目标文件会制造链接错误或假输出）；agent 会话另按全局规约使用独立 scratch 路径。 +- **兄弟依赖对齐**：跑之前确认两个检出各自解析到预期的 sibling 内容（例如基线 main pin 了 `exact: "0.4.5"`，则 `/Volumes/Code/Personal/swift-demangling` 需在 0.4.5 tag 上：`git -C ../swift-demangling tag --points-at HEAD`）。sibling 内容错位会把 A/B 变成「比较两个不同的依赖版本」。 +- **interface 输出一律走 `-o` 落盘**：进度日志（带墙钟时间戳）走 stdout，不会混进被比对的文件。 + +## 验收标准与差异排查 + +- 验收：**所有配对逐字节一致**（`cmp`）。 +- 出现 DIFFERS 时：先在**同一侧**把该场景连跑两遍排除非确定性（2026-08-03 基线确认 dump / interface 输出均确定），再做归因；一侧成功一侧失败（MISSING-ON-*）同样按差异处理。 +- 两侧以**相同退出码**失败的场景记为 SKIPPED（脚本会列出），例如某框架在旧 runtime 里根本不存在。 + +## 已知的双侧一致现象（非回归） + +- **iOS 15.5 模拟器的 interface 输出只有几十行**：索引器对 iOS 15 时代 metadata 批量报 `Error resolving ContextDescriptorWrapper: offsetOutOfBounds / invalidContextDescriptor`，类型全部掉光只剩全局函数；dump 路径不受影响（SwiftUI 15.5 dump 有 4.6 万行）。两侧错误集合一致，属既有限制。 + +## 基线运行记录（2026-08-03，main ↔ feature/node-store-migration） + +- 附带 smoke：fixture（SymbolTestsCore）dump 6031 行、interface 3636 行，双侧一致（interface 仅时间戳日志行差异，归一化后一致）。 +- DyldCache：macOS 26.5.2_25F84 + 15.5_24F74 × 6 框架 × dump+interface，**24 对全部逐字节一致**（最大 SwiftUI dump 109,387 行）。 +- MachOFile：iOS 15.5（3 框架）/ 18.5 / 26.5（各 6 框架）模拟器 × dump+interface，**30 对全部逐字节一致**。 +- MachOImage：当前系统（macOS 26.5，arm64e cache），六框架 in-process + 当前 cache 文件双路，全选项（除 `expandedFieldOffsets`），**24 对全部逐字节一致**（最大 interface-SwiftUI-file 7.2 MB；harness 单侧耗时 ~14–16 分钟）。 +- **合计 78 对，零差异**。运行细节与偏离见 [TaskReports/2026-08-03-system-framework-rendering-ab.md](TaskReports/2026-08-03-system-framework-rendering-ab.md)。 diff --git a/Documentations/Internal/TaskReports/2026-08-03-system-framework-rendering-ab.md b/Documentations/Internal/TaskReports/2026-08-03-system-framework-rendering-ab.md new file mode 100644 index 00000000..1daf71a7 --- /dev/null +++ b/Documentations/Internal/TaskReports/2026-08-03-system-framework-rendering-ab.md @@ -0,0 +1,36 @@ +# 2026-08-03 系统框架渲染 A/B 验证与流程固化 + +## 问题 + +性能批次（同日五提交）落地后，维护者要求用真实系统框架对 `feature/node-store-migration` 做全面输出对等验证，覆盖三种 reader 路径：DyldCache 用两份归档 cache（macOS 26.5.2_25F84 / 15.5_24F74），MachOFile 用 iOS 15.5 / 18.5 / 26.5 模拟器 runtime，MachOImage 直接测当前系统；框架集 SwiftUI / SwiftUICore / SwiftData / Combine / ActivityKit / WidgetKit，缺席即略过。随后追加要求：把这套测试记成「大重构必跑」流程，目标输入不存在时 DyldCache 回退当前系统 cache、MachOFile 回退现有模拟器 runtime；脚本用 Python 写。 + +## 调研 + +- CLI 无进程内 MachOImage 模式；`Tests/IntegrationTests/SwiftInterface/RenderingVerificationTests.swift` 正是维护者为「两个检出 diff 输出」设计的 harness（dlopen + `MachOImage(name:)`，image/file 双路，全 metadata 选项），且 main 与 feature 上逐字节相同——MachOImage 部分直接双侧复用它（IntegrationTests 禁跑规则的唯一例外，已在 AGENTS.md 登记）。 +- cache 内镜像按名字查有歧义（SwiftUI / WidgetKit / ActivityKit 有 `/System/iOSSupport` 的 Catalyst 副本；macOS 15.5 的 ActivityKit **只有** Catalyst 副本），需 `-p` 全路径。 +- iOS 15.5 / 18.5 模拟器框架是 fat 二进制（x86_64+arm64），CLI 直接报错要求 `-a`；26.5 起是 thin。首轮模拟器矩阵因此双侧全体瞬间失败，补 `-a arm64` 重跑。 +- main 侧环境核对：主仓库 `Package.swift` 的 `exact: "0.4.5"` / `exact: "0.1.6"` pin 与兄弟检出（swift-demangling detached@0.4.5、swift-semantic-string@0.1.6）恰好对齐；main 用全新独立 scratch（`MachOSwiftSection-main`）从零构建，规避此前 scratch 混用事故模式。 + +## 方案 + +双侧 release CLI（各自独立 scratch）对同一批输入渲染 dump + interface，输出经 `-o` 落盘（时间戳进度日志走 stdout 不进文件），逐对 `cmp`。三部分顺序：cache 矩阵 + 模拟器矩阵（脚本并行双侧）→ `RenderingVerificationTests`（`RV_OPTS` 去掉 `expandedFieldOffsets`，harness 注释记录其在 SwiftUI 级 MachOImage 上的既有栈溢出；双侧同一次开机会话内运行，保证 memberAddress 地址可比）。 + +## 实际执行 + +- 前置 smoke：fixture（SymbolTestsCore）dump 6031 行 / interface 3636 行，双侧一致（interface 仅时间戳行差异，归一化后一致）；dump 连跑两遍验证输出确定性。 +- **DyldCache**：2 cache × 6 框架 × dump+interface = **24 对全部逐字节一致**（最大 SwiftUI dump 109,387 行）。 +- **MachOFile**：iOS 15.5（3 框架）/ 18.5 / 26.5（各 6 框架）× dump+interface = **30 对全部逐字节一致**。 +- **MachOImage**：当前系统（macOS 26.5）六框架 image+file 双路全选项 = **24 对全部逐字节一致**（最大 interface-SwiftUI-file 7.2 MB；单侧 836–966 秒）。 +- **合计 78 对，零差异。** +- 流程固化：`Scripts/run-rendering-ab-verification.py`（先写 zsh 版，按维护者要求改写为 Python 并删除 zsh 版）+ `Documentations/Internal/SystemFrameworkRenderingVerification.md` + `Documentations/README.md` 索引行 + AGENTS.md「大重构必跑」段落与 IntegrationTests 例外登记 + 演进日志第 25 节。 + +## 验证 + +- 逐对 `cmp` 全绿（上文数字）；`--uses-system-dyld-shared-cache -p <路径>`（无文件参数）的回退调用形式实测可用；脚本 `py_compile` + `--help` 通过。 +- 两个 harness 测试进程均正常退出（`Test run … passed`）。 + +## 偏离与教训 + +- **fat 二进制首轮全灭**：模拟器矩阵首轮未带 `-a`，双侧同码瞬间失败——失败形态对称所以无害，但暴露了「脚本先在小输入上单侧试跑一次」的价值；修正后的脚本一律显式 `-a arm64`。 +- **iOS 15.5 的 interface 输出只有几十行**（双侧一致）：索引器对 iOS 15 时代 metadata 批量报 `ContextDescriptorWrapper` 解析错误，类型全部掉光只剩全局函数；dump 路径不受影响（SwiftUI 15.5 dump 4.6 万行）。records 为已知既有限制，非迁移回归。 +- ActivityKit 在当前 macOS 26.5 是原生框架（in-process dlopen 成功），不再是 iOSSupport 专属——「缺席即略过」的判定要按实际探测，不能按旧记忆硬编码。 diff --git a/Documentations/README.md b/Documentations/README.md index 37b6bf65..8a9d8927 100644 --- a/Documentations/README.md +++ b/Documentations/README.md @@ -76,5 +76,6 @@ required by `Version.swift`'s bump contract). | [NodeStoreMigrationPlan.md](Internal/NodeStoreMigrationPlan.md) | `SymbolIndexStore` 迁移到 `NodeStore` arena 存储的分期计划与实施记录（Stage 0–5）：12B/节点扁平缓冲、cache-free build sweep、Symbol 表压缩、声明层换持 `NodeReference`、散点 transient demangling；含 Stage 5a 的 override/vtable 回归修复（裸 `[NodeReference: …]` 的 store-identity 陷阱 → `StructuralNodeReferenceKey`）与三源快照验收。 | | [DeclarationModelMemoryFootprint.md](Internal/DeclarationModelMemoryFootprint.md) | NodeStore 迁移后的声明模型内存足迹量测：`TypeDefinition` 1272 字节的逐属性构成（两份 `TypeContextWrapper` 占 74%）、`TypeContextWrapper` 按最大 case `Class` 定尺的原因、`parentContext` 是被当成永久字段的临时值、mini-store 增殖与 `MetadataReaderCache` 仍持 `Node` 树；含四项可回收估算（合计 ~8–10%）与「当前不建议实施、应先剖析其余 90%」的结论。 | | [NodeStoreMigrationOpenIssues.md](Internal/NodeStoreMigrationOpenIssues.md) | `feature/node-store-migration` 的**已确认问题清单**（2026-07-28 两轮审查 + 复核）。第一节是前一轮修复自身的两处缺口（Catalyst 降级只覆盖 framework 形态导致 plain dylib 仍平局、`appendRowIfAbsent` 线性扫描），已于 2026-07-29 闭环并保留成因；第二节起为仍然打开或已裁决的条目：`structuralHash` 每文本节点分配 `String` 与 `ABIKey` 每 key materialize 两条**已按上游设计终审关闭**（0.5.1 保持现状且上游说明不改，见 ReviewAdjudications A1/A2）、`memberSymbols` 改为线性 + 全树比对（**已证实量级可忽略，不是回归**）；`lateDemangledNode` 持锁 demangle 与 `ProtocolConformanceDumper` 的 materialize 分支两条**已于 2026-08-03 闭环**（连同失败名裁决缓存、dump 路径引用化）、build sweep 串行且每符号跨线程往返（上游）、`ABIKey` 每 key materialize，以及四条代码卫生项；两个公开查询 API 的字典键一条**已于 2026-08-03 裁决为不修**（类型级 SPI），rebase 前置事项一条前提已过期但压着两条仍成立的注意事项。逐条注明成因、影响面与「该在哪里修」。与 `Reviews/` 下两份审查记录冲突时以后者为准。 | +| [SystemFrameworkRenderingVerification.md](Internal/SystemFrameworkRenderingVerification.md) | 系统框架渲染 A/B 验证流程（大重构必跑）：两个检出对同一批真实输入（归档 dyld cache → 无则当前系统 cache、模拟器 runtime → 无则现有 runtime、当前系统 MachOImage）逐字节比对 dump+interface；入口脚本 `Scripts/run-rendering-ab-verification.py`，含回退规则、踩坑清单（`-p` 消歧、fat 二进制 `-a`、同 boot 会话、scratch 隔离）与 2026-08-03 基线运行记录。 | | [TaskReports/](Internal/TaskReports/) | Dated per-task fix / investigation reports. | | [Reviews/](Internal/Reviews/) | Dated code-review records — one file per review event: findings, measurements taken at the time, and the resulting to-do list. Distinct from the topic-organised issue ledgers (e.g. `NodeStoreMigrationOpenIssues.md`), which they cross-reference. | diff --git a/Scripts/run-rendering-ab-verification.py b/Scripts/run-rendering-ab-verification.py new file mode 100755 index 00000000..bcf7f435 --- /dev/null +++ b/Scripts/run-rendering-ab-verification.py @@ -0,0 +1,278 @@ +#!/usr/bin/env python3 +"""A/B rendering-parity verification over real system frameworks. + +Renders dump + interface output for a fixed framework set through all three +reader paths (dyld shared cache, plain Mach-O file, in-process MachOImage) +from TWO checkouts of this package, then byte-compares every output pair. +Run it before landing any large refactor that touches demangling, printing, +indexing, or the reader stack. See +Documentations/Internal/SystemFrameworkRenderingVerification.md for the +procedure, fallback rules, and the baseline run record. + +Usage: + Scripts/run-rendering-ab-verification.py + [--output-root PATH] [--frameworks A,B,...] + [--baseline-scratch PATH] [--candidate-scratch PATH] [--skip-image-part] + +Input sources and fallbacks: + - Dyld caches: prefers the archived caches under /Volumes/DyldSharedCaches/macOS + (26.5.2_25F84 and 15.5_24F74). When none of them exists, falls back to the + CURRENT system's dyld shared cache (--uses-system-dyld-shared-cache). + - Simulator runtimes: prefers iOS 15.5 / 18.5 / 26.5; every installed iOS + runtime discovered on this machine is used (they are enumerated, so absent + preferred versions simply do not appear). + - MachOImage: always the current system, via the RenderingVerificationTests + harness (the documented IntegrationTests exception for this exact purpose). +""" + +import argparse +import datetime +import filecmp +import os +import subprocess +import sys +import time +from pathlib import Path + +DEFAULT_FRAMEWORK_NAMES = ["SwiftUI", "SwiftUICore", "SwiftData", "Combine", "ActivityKit", "WidgetKit"] + +ARCHIVED_CACHE_DIRECTORIES = [ + Path("/Volumes/DyldSharedCaches/macOS/26.5.2_25F84"), + Path("/Volumes/DyldSharedCaches/macOS/15.5_24F74"), +] + +SIMULATOR_RUNTIME_SEARCH_DIRECTORIES = [ + Path("/Library/Developer/CoreSimulator/Profiles/Runtimes"), + # Newer runtimes mount under per-runtime volumes. + *sorted(Path("/Library/Developer/CoreSimulator/Volumes").glob("*/Library/Developer/CoreSimulator/Profiles/Runtimes")), +] + +# expandedFieldOffsets stays off: the harness documents a pre-existing stack +# overflow over MachOImage of deeply generic frameworks (e.g. SwiftUI). +RENDERING_VERIFICATION_OPTIONS = "fieldOffset,typeLayout,enumLayout,spareBitAnalysis,memberAddress,vtableOffset,pwtOffset" + + +def parse_arguments() -> argparse.Namespace: + parser = argparse.ArgumentParser(description="A/B rendering-parity verification over real system frameworks.") + parser.add_argument("baseline_checkout", type=Path) + parser.add_argument("candidate_checkout", type=Path) + parser.add_argument("--output-root", type=Path, + default=Path("/tmp/rendering-ab-verification") / datetime.datetime.now().strftime("%Y%m%d-%H%M%S")) + parser.add_argument("--frameworks", default=",".join(DEFAULT_FRAMEWORK_NAMES), + help="Comma-separated framework names to render.") + parser.add_argument("--baseline-scratch", type=Path, default=None, + help="SwiftPM scratch path for the baseline build (default: /.build).") + parser.add_argument("--candidate-scratch", type=Path, default=None, + help="SwiftPM scratch path for the candidate build (default: /.build).") + parser.add_argument("--skip-image-part", action="store_true", + help="Skip the MachOImage (RenderingVerificationTests) part.") + arguments = parser.parse_args() + arguments.baseline_checkout = arguments.baseline_checkout.resolve() + arguments.candidate_checkout = arguments.candidate_checkout.resolve() + if arguments.baseline_scratch is None: + arguments.baseline_scratch = arguments.baseline_checkout / ".build" + if arguments.candidate_scratch is None: + arguments.candidate_scratch = arguments.candidate_checkout / ".build" + arguments.framework_names = [name.strip() for name in arguments.frameworks.split(",") if name.strip()] + return arguments + + +class VerificationRun: + def __init__(self, arguments: argparse.Namespace) -> None: + self.arguments = arguments + self.output_root: Path = arguments.output_root + self.command_line_interfaces: dict[str, Path] = {} + + # --- Building ----------------------------------------------------------- + + def build_both_sides(self) -> None: + for side, checkout, scratch in self.sides(): + print(f"Building release swift-section for {checkout} ...") + completed = subprocess.run([ + "swift", "build", "-c", "release", + "--package-path", str(checkout), + "--scratch-path", str(scratch), + "--product", "swift-section", + ]) + if completed.returncode != 0: + sys.exit(f"error: release build failed for {side} ({checkout})") + self.command_line_interfaces[side] = scratch / "release" / "swift-section" + + def sides(self) -> list[tuple[str, Path, Path]]: + return [ + ("baseline", self.arguments.baseline_checkout, self.arguments.baseline_scratch), + ("candidate", self.arguments.candidate_checkout, self.arguments.candidate_scratch), + ] + + # --- One rendered pair -------------------------------------------------- + + def run_pair(self, scenario_name: str, framework_name: str, command_name: str, extra_arguments: list[str]) -> None: + """Run one dump/interface command through both sides' CLIs.""" + for side in ("baseline", "candidate"): + output_directory = self.output_root / scenario_name / side + output_directory.mkdir(parents=True, exist_ok=True) + output_file = output_directory / f"{framework_name}.{command_name}.txt" + log_file = output_directory / f"{framework_name}.{command_name}.log" + started_at = time.monotonic() + with open(log_file, "w") as log_handle: + completed = subprocess.run( + [str(self.command_line_interfaces[side]), command_name, *extra_arguments, "-o", str(output_file)], + stdout=log_handle, stderr=subprocess.STDOUT, + ) + elapsed_seconds = time.monotonic() - started_at + if completed.returncode != 0: + # Record the failure as a skip marker; the diff phase treats a + # pair of equal markers as SKIPPED and anything else as a difference. + output_file.unlink(missing_ok=True) + (output_directory / f"{framework_name}.{command_name}.skip").write_text(f"exit={completed.returncode}\n") + print(f"[{side}] {scenario_name}/{framework_name} {command_name} " + f"exit={completed.returncode} {elapsed_seconds:.0f}s") + + # --- Part 1: dyld shared caches ----------------------------------------- + + def image_path_inside_cache(self, cache_directory: Path, framework_name: str) -> str | None: + """Resolve the framework's in-cache image path via the cache's .map file. + + The full canonical path disambiguates frameworks that also ship a + Mac Catalyst copy under /System/iOSSupport (SwiftUI, WidgetKit, ...); + the iOSSupport copy is used only when it is the sole one (e.g. + ActivityKit on macOS 15). + """ + map_file = cache_directory / "dyld_shared_cache_arm64e.map" + if not map_file.is_file(): + return None + map_contents = map_file.read_text(errors="replace") + canonical_path = f"/System/Library/Frameworks/{framework_name}.framework/Versions/A/{framework_name}" + for candidate_path in (canonical_path, "/System/iOSSupport" + canonical_path): + if candidate_path in map_contents: + return candidate_path + return None + + def run_dyld_cache_part(self) -> None: + available_cache_directories = [directory for directory in ARCHIVED_CACHE_DIRECTORIES + if (directory / "dyld_shared_cache_arm64e").is_file()] + if not available_cache_directories: + print("No archived cache found - falling back to the current system's dyld shared cache.") + for framework_name in self.arguments.framework_names: + image_path = f"/System/Library/Frameworks/{framework_name}.framework/Versions/A/{framework_name}" + for command_name in ("dump", "interface"): + self.run_pair("cache-current-system", framework_name, command_name, + ["--uses-system-dyld-shared-cache", "-p", image_path]) + return + for cache_directory in available_cache_directories: + scenario_name = f"cache-{cache_directory.name}" + for framework_name in self.arguments.framework_names: + image_path = self.image_path_inside_cache(cache_directory, framework_name) + if image_path is None: + print(f"[skip] {scenario_name}/{framework_name}: not in cache") + continue + for command_name in ("dump", "interface"): + self.run_pair(scenario_name, framework_name, command_name, + [str(cache_directory / "dyld_shared_cache_arm64e"), "--dyld-shared-cache", "-p", image_path]) + + # --- Part 2: simulator runtime Mach-O files ----------------------------- + + def discover_simulator_runtime_roots(self) -> dict[str, Path]: + runtime_roots_by_label: dict[str, Path] = {} + for search_directory in SIMULATOR_RUNTIME_SEARCH_DIRECTORIES: + if not search_directory.is_dir(): + continue + for runtime_bundle in sorted(search_directory.glob("*.simruntime")): + label = runtime_bundle.stem + if not label.startswith("iOS") or label in runtime_roots_by_label: + continue + runtime_roots_by_label[label] = runtime_bundle / "Contents/Resources/RuntimeRoot" + return runtime_roots_by_label + + def run_simulator_part(self) -> None: + for label, runtime_root in self.discover_simulator_runtime_roots().items(): + scenario_name = "sim-" + label.replace(" ", "-") + for framework_name in self.arguments.framework_names: + framework_binary = runtime_root / f"System/Library/Frameworks/{framework_name}.framework/{framework_name}" + if not framework_binary.is_file(): + print(f"[skip] {scenario_name}/{framework_name}: not in runtime") + continue + for command_name in ("dump", "interface"): + # Older runtimes ship fat (x86_64 + arm64) binaries; the slice must be explicit. + self.run_pair(scenario_name, framework_name, command_name, [str(framework_binary), "-a", "arm64"]) + + # --- Part 3: in-process MachOImage (current system) --------------------- + + def run_macho_image_part(self) -> None: + """RenderingVerificationTests is the maintainer harness designed for exactly + this two-checkout diff; running it here is the documented exception to the + "agents must not run IntegrationTests" rule. Both sides MUST run within the + same boot session: memberAddress comments depend on the per-boot dyld + shared cache slide.""" + for side, checkout, scratch in self.sides(): + output_directory = self.output_root / "machoimage-current" / side + output_directory.mkdir(parents=True, exist_ok=True) + environment = os.environ.copy() + environment.update({ + "RV_OUT": str(output_directory), + "RV_FRAMEWORKS": ",".join(self.arguments.framework_names), + "RV_OPTS": RENDERING_VERIFICATION_OPTIONS, + "MACHO_SWIFT_SECTION_SILENT_TEST": "1", + }) + log_file = self.output_root / "machoimage-current" / f"{side}.test.log" + with open(log_file, "w") as log_handle: + completed = subprocess.run([ + "swift", "test", "-c", "release", + "--package-path", str(checkout), + "--scratch-path", str(scratch), + "--filter", "RenderingVerificationTests", + ], env=environment, stdout=log_handle, stderr=subprocess.STDOUT) + print(f"[{side}] machoimage-current exit={completed.returncode}") + + # --- Diff phase --------------------------------------------------------- + + def compare_all_pairs(self) -> int: + print("\n=== A/B comparison ===") + difference_count = 0 + baseline_files = sorted(self.output_root.glob("**/baseline/*.txt")) + for baseline_file in baseline_files: + candidate_file = Path(str(baseline_file).replace("/baseline/", "/candidate/")) + relative_name = baseline_file.relative_to(self.output_root) + if not candidate_file.is_file(): + print(f"MISSING-ON-CANDIDATE {relative_name}") + difference_count += 1 + elif filecmp.cmp(baseline_file, candidate_file, shallow=False): + print(f"IDENTICAL {relative_name}") + else: + print(f"DIFFERS {relative_name}") + difference_count += 1 + for candidate_file in sorted(self.output_root.glob("**/candidate/*.txt")): + baseline_file = Path(str(candidate_file).replace("/candidate/", "/baseline/")) + if not baseline_file.is_file(): + print(f"MISSING-ON-BASELINE {candidate_file.relative_to(self.output_root)}") + difference_count += 1 + for skip_file in sorted(self.output_root.glob("**/baseline/*.skip")): + candidate_skip = Path(str(skip_file).replace("/baseline/", "/candidate/")) + if candidate_skip.is_file() and skip_file.read_text() == candidate_skip.read_text(): + print(f"SKIPPED (both sides, {skip_file.read_text().strip()}) {skip_file.relative_to(self.output_root)}") + return difference_count + + +def main() -> None: + arguments = parse_arguments() + run = VerificationRun(arguments) + run.output_root.mkdir(parents=True, exist_ok=True) + print(f"Output root: {run.output_root}") + + run.build_both_sides() + run.run_dyld_cache_part() + run.run_simulator_part() + if not arguments.skip_image_part: + run.run_macho_image_part() + + difference_count = run.compare_all_pairs() + if difference_count == 0: + print("\nRESULT: all pairs byte-identical.") + else: + print(f"\nRESULT: {difference_count} differing pair(s). Re-run the differing scenario twice on one side " + f"first to rule out nondeterminism before attributing.") + sys.exit(1) + + +if __name__ == "__main__": + main() From 651512ff543d3f9b080d6cdc72bbbef06c0067ad Mon Sep 17 00:00:00 2001 From: Mx-Iris Date: Mon, 3 Aug 2026 18:28:34 +0800 Subject: [PATCH 36/77] fix(MachOExtensions,SwiftInterface): legacy LC_DYLD_INFO bind support + per-definition interface degradation Binaries with pre-chained-fixups deployment targets (< macOS 12 / iOS 16, e.g. every iOS 15.5 simulator framework) describe their bind slots only in LC_DYLD_INFO(_ONLY) opcode streams, which resolveBind(fileOffset:) did not read - every indirect reference to another image's descriptor (external protocol conformances, external superclasses) was misread as a raw pointer and failed with offsetOutOfBounds/invalidContextDescriptor (356 of Combine's 725 conformances). A second, compounding defect: printRoot caught print errors per BLOCK, so the first type whose printing threw blanked every type of the interface, leaving only imports and global functions. resolveBind now falls back to a lazily built file-offset -> symbol-name index interpreted from the LC_DYLD_INFO bind opcode streams (normal + weak; the arm64e threaded legacy format is deliberately not indexed). printRoot's four blocks and printThrowingProtocol's trailing default-implementation extensions catch per definition, so one throwing definition drops only itself. LegacyDyldInfoBindTests pins both fixes with an on-the-fly fixture compiled with -target arm64-apple-macosx11.0, which forces the legacy format: 7 issues before the batch, 1 with the per-definition catch alone, green with both. iOS 15.5 simulator interfaces recover from 10 / 17 / 139 lines to 6907 (Combine) / 2795 (WidgetKit) / 81157 (SwiftUI) with zero resolution errors; the full suite (1315 tests / 250 suites) stays green and modern-binary snapshots are byte-identical. --- AGENTS.md | 4 +- .../Internal/ProjectEvolutionLog.md | 14 ++ .../SystemFrameworkRenderingVerification.md | 2 +- ...026-08-03-legacy-dyld-info-bind-support.md | 43 ++++++ .../SwiftInterfaceBuilder.swift | 49 ++++--- .../SwiftDeclarationPrinter.swift | 8 +- .../LegacyDyldInfoBindTests.swift | 136 ++++++++++++++++++ 7 files changed, 229 insertions(+), 27 deletions(-) create mode 100644 Documentations/Internal/TaskReports/2026-08-03-legacy-dyld-info-bind-support.md create mode 100644 Tests/SwiftInterfaceTests/LegacyDyldInfoBindTests.swift diff --git a/AGENTS.md b/AGENTS.md index 82410c4d..ce880b4c 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -124,7 +124,7 @@ The interface generation is split into layered peer modules over a shared `Swift - `SwiftDeclarationPrintConfiguration`, `SwiftDeclarationMemberSortOrder` - Type-level members print `class` (not `static`) when they carry a vtable method descriptor (`isClassMember` on `FunctionDefinition` / `VariableDefinition` / `SubscriptDefinition` in `SwiftDeclaration`): mangling cannot distinguish the two spellings, but a `static` member is implicitly final and never gets a descriptor, so `override static` (illegal Swift) is structurally impossible in the output. The four descriptor-less `class` spellings (`final class func`, `class func` in a final class / an extension, `@objc dynamic class func`) are ABI-identical to `static` and conservatively print as the semantically-equivalent `static`. `ClassDumper`'s vtable-section keyword follows the same fact; the dump override-table lines keep the demangler's faithful `static` symbol prefix. See [Documentations/Internal/ClassMemberKeywordRecovery.md](Documentations/Internal/ClassMemberKeywordRecovery.md). - Specialized definitions (`TypeDefinition.isSpecialized`) render **bound**: the header prints the concrete-argument name (`Box`, generic-signature clause skipped) via `BoundDumpedTypeNameRenderer`, and each field's type node is substituted through the specialized runtime metadata via `SpecializedMetadataNodeSubstitution` — both live in `SwiftDeclarationRendering` so the dump path (`TypedDumper`, which keeps its own copies/forwarders) stays independent. See [Documentations/Internal/SpecializedInterfaceBoundRenderingRestoration.md](Documentations/Internal/SpecializedInterfaceBoundRenderingRestoration.md). -- The main interface path's stored-field / enum-case rendering (`renderModelFields` → `printThrowingField` / `printThrowingEnumCase`) carries the **pre-leaf-migration error contract**: record reads, metadata comments, and type printing propagate errors (a failing field fails the whole type), and an enum case's payload presence follows the field record's mangled type name (captured at index time as `FieldFlags.hasMangledTypeName`, so rendering never re-reads the record positionally) — a `Void` payload prints `case a()` exactly like the dump path, while a payload whose node *renders empty* degrades to the bare case in both paths (`case a()` around nothing is invalid Swift; the interface printers also render kind-9 accessor-function symbolic references as the honest `accessor function at ` fallback — see [Documentations/Internal/AccessorFunctionReferenceRendering.md](Documentations/Internal/AccessorFunctionReferenceRendering.md)). The diff renderer's `printField` / `printEnumCase` keep their own per-member-catch, rendered-text-gating contract (that is their original design, needed for standalone `+`/`-` members). The shared comment engine's `FieldLayoutRenderer.storedFieldComments` / `enumCaseComments` are `throws` for the same reason, and multi-payload enum descriptors resolve through `MultiPayloadEnumDescriptorCache` in `SwiftDeclarationRendering` (built once per image as a *partial* map — one bad descriptor only degrades its own enum to the tagged projection). See [Documentations/Internal/LeafMigrationRegressionFixes.md](Documentations/Internal/LeafMigrationRegressionFixes.md). +- The main interface path's stored-field / enum-case rendering (`renderModelFields` → `printThrowingField` / `printThrowingEnumCase`) carries the **pre-leaf-migration error contract**: record reads, metadata comments, and type printing propagate errors (a failing field fails the whole type), and an enum case's payload presence follows the field record's mangled type name (captured at index time as `FieldFlags.hasMangledTypeName`, so rendering never re-reads the record positionally) — a `Void` payload prints `case a()` exactly like the dump path, while a payload whose node *renders empty* degrades to the bare case in both paths (`case a()` around nothing is invalid Swift; the interface printers also render kind-9 accessor-function symbolic references as the honest `accessor function at ` fallback — see [Documentations/Internal/AccessorFunctionReferenceRendering.md](Documentations/Internal/AccessorFunctionReferenceRendering.md)). At the TOP level the contract inverts: `printRoot` (and `printThrowingProtocol`'s trailing default-implementation extensions) catch per definition — one type/protocol/extension whose printing throws drops only itself, never its whole block (a block-level catch once blanked every type of a legacy binary's interface; pinned by `LegacyDyldInfoBindTests`). The diff renderer's `printField` / `printEnumCase` keep their own per-member-catch, rendered-text-gating contract (that is their original design, needed for standalone `+`/`-` members). The shared comment engine's `FieldLayoutRenderer.storedFieldComments` / `enumCaseComments` are `throws` for the same reason, and multi-payload enum descriptors resolve through `MultiPayloadEnumDescriptorCache` in `SwiftDeclarationRendering` (built once per image as a *partial* map — one bad descriptor only degrades its own enum to the tagged projection). See [Documentations/Internal/LeafMigrationRegressionFixes.md](Documentations/Internal/LeafMigrationRegressionFixes.md). **SwiftSpecialization** - Runtime generic specialization (see implementation plan below) - `GenericSpecializer`, `ConformanceProvider` @@ -188,7 +188,7 @@ Printing and indexing are peers — neither depends on the other. - **MachOSymbols** - Symbol table parsing and demangling - **MachOPointers** - Pointer types (relative, indirect, etc.) - **MachOCaches** - dyld shared cache support -- **MachOExtensions** - Extensions to MachOKit types +- **MachOExtensions** - Extensions to MachOKit types. `resolveBind(fileOffset:)` resolves bind slots from chained fixups AND, when those are absent, from the legacy `LC_DYLD_INFO(_ONLY)` bind opcode streams (pre-macOS 12 / iOS 16 deployment targets, e.g. iOS 15.5 simulator frameworks) via a lazily built file-offset → symbol-name index; the arm64e threaded legacy format is deliberately not indexed. Pinned by `LegacyDyldInfoBindTests`, whose fixture is compiled on the fly with `-target arm64-apple-macosx11.0` to force the legacy format. ### Key Patterns diff --git a/Documentations/Internal/ProjectEvolutionLog.md b/Documentations/Internal/ProjectEvolutionLog.md index 70560b59..f2aad548 100644 --- a/Documentations/Internal/ProjectEvolutionLog.md +++ b/Documentations/Internal/ProjectEvolutionLog.md @@ -711,6 +711,20 @@ --- +## 26. 旧格式 bind 支持：LC_DYLD_INFO opcode 回退 + interface 逐项降级 + +- **时间段**：2026-08-03（第 25 节 A/B 验证的直接产出）。 +- **动机**：A/B 验证发现 iOS 15.5 模拟器框架的 interface 输出只剩全局函数（三框架、数百条 `offsetOutOfBounds`），dump 却正常。根因两层：`resolveBind(fileOffset:)` 只认 chained fixups，旧格式（部署目标 < macOS 12 / iOS 16 的 `LC_DYLD_INFO_ONLY`）二进制的外部引用全部按裸指针误读；`printRoot` 的块级 catch 把单类型打印失败放大成全部类型消失。 +- **落地**： + - `MachOExtensions/MachOFile+.swift`：chained fixups 缺席时按 dyld 状态机解释 `bindOperations` / `weakBindOperations` opcode 流，惰性构建「文件偏移 → 符号名」索引（arm64e threaded 旧格式不索引、lazy 流不索引）。 + - `SwiftInterface/SwiftInterfaceBuilder.swift` + `SwiftPrinting/SwiftDeclarationPrinter.swift`：printRoot 四个块与 printThrowingProtocol 的 default-implementation extensions 块全部改为逐项 `printCatchedThrowing`——单个定义抛错只丢它自己。 + - 新增 `LegacyDyldInfoBindTests`（fixture 用 `swiftc -target arm64-apple-macosx11.0` 在测试内即时编译强制旧格式；红 7 → 仅 fix 2 剩 4 → 双修复全绿的阶梯实测留档）。 +- **效果**：iOS 15.5 模拟器 interface：Combine 10 → 6907 行、WidgetKit 17 → 2795 行、SwiftUI 139 → 81157 行，解析错误全部归零（SwiftUI 7616 个 conformance 全数解析）；全量 1315 测试 / 250 套件绿，现代二进制快照逐字节不变。旧格式输入的 interface 输出自此与 main 合理不一致（feature 更完整），main 合并后恢复对等。 +- **关联文档**：[TaskReports/2026-08-03-legacy-dyld-info-bind-support.md](TaskReports/2026-08-03-legacy-dyld-info-bind-support.md)（含完整的无调试器调试方法学 walkthrough）、[SystemFrameworkRenderingVerification.md](SystemFrameworkRenderingVerification.md)。 +- **对应版本**：0.14.0 之后未发布区间（`feature/node-store-migration` 分支）。 + +--- + ## 维护约定 1. **每个非平凡批次结束时必须在本文追加/更新一节**（新工作弧新增一节；延续既有弧则在该节 diff --git a/Documentations/Internal/SystemFrameworkRenderingVerification.md b/Documentations/Internal/SystemFrameworkRenderingVerification.md index 692f1346..64d54b1a 100644 --- a/Documentations/Internal/SystemFrameworkRenderingVerification.md +++ b/Documentations/Internal/SystemFrameworkRenderingVerification.md @@ -43,7 +43,7 @@ SwiftUI、SwiftUICore、SwiftData、Combine、ActivityKit、WidgetKit——**输 ## 已知的双侧一致现象（非回归） -- **iOS 15.5 模拟器的 interface 输出只有几十行**：索引器对 iOS 15 时代 metadata 批量报 `Error resolving ContextDescriptorWrapper: offsetOutOfBounds / invalidContextDescriptor`，类型全部掉光只剩全局函数；dump 路径不受影响（SwiftUI 15.5 dump 有 4.6 万行）。两侧错误集合一致，属既有限制。 +- ~~**iOS 15.5 模拟器的 interface 输出只有几十行**~~——**已于 2026-08-03 在 `feature/node-store-migration` 修复**（`LC_DYLD_INFO` opcode bind 回退 + printRoot 逐项降级，见[任务报告](TaskReports/2026-08-03-legacy-dyld-info-bind-support.md)）。修复落地后，旧格式二进制（部署目标 < macOS 12 / iOS 16）的 interface 输出与**未含该修复的基线**（如当前 main）会**合理地不一致**——修复侧多出完整的类型与 conformance；对含修复的两个检出做 A/B 时该场景恢复严格逐字节对比。基线侧的历史症状（只剩全局函数、成百条 `offsetOutOfBounds`）与根因记录在任务报告里。 ## 基线运行记录（2026-08-03，main ↔ feature/node-store-migration） diff --git a/Documentations/Internal/TaskReports/2026-08-03-legacy-dyld-info-bind-support.md b/Documentations/Internal/TaskReports/2026-08-03-legacy-dyld-info-bind-support.md new file mode 100644 index 00000000..446cea8a --- /dev/null +++ b/Documentations/Internal/TaskReports/2026-08-03-legacy-dyld-info-bind-support.md @@ -0,0 +1,43 @@ +# 2026-08-03 旧格式 bind 支持：LC_DYLD_INFO opcode 回退 + interface 逐项降级 + +## 问题 + +系统框架渲染 A/B 验证（同日前一任务）发现：iOS 15.5 模拟器的三个框架 interface 输出只剩 import 和全局函数（Combine 10 行 / WidgetKit 17 行 / SwiftUI 139 行），日志里 `Error resolving ContextDescriptorWrapper: offsetOutOfBounds / invalidContextDescriptor` 成百上千条，而同一批二进制 dump 输出完全正常。维护者要求查根因并修复（明确表示两处都修），同时希望完整了解这类问题的调试方法。 + +## 调研（调试方法学 walkthrough） + +全程没有用交互式调试器，六步收敛，每一步都把假设空间砍掉一半： + +1. **grep 错误文案找吞错点**：`Error resolving` 全部来自 `ContextDescriptorWrapper` 的「catch → print → 返回 nil」可选解析包装。这一步同时解释了为什么断点难打——错误被就地吞掉，全局 Swift Error 断点会淹死在无差别 throw 里。 +2. **用日志位置代替断点**：错误行夹在哪两条进度 INFO 之间 → 一次运行拿到全量分布——「Types: 271 successful, 2 failed」之后、「Conformances: 370 extensions, **356 failed**」之前有 387 条，打印阶段只有 3-6 条。索引期 conformance 是重灾区，打印期只有零星几条。 +3. **健康/病态对照**：macOS 15.5 cache（2025 年二进制）完全健康 → 排除「15.5」版本号因素，指向二进制年代；dump 健康 vs interface 全空 → 说明还有一个 interface 侧的放大器。 +4. **顺着计数抓矛盾**：「类型 271 成功」却「输出 0 类型」→ 读 `printRoot` → 整块 `try` 循环 + 块级 catch，第一个抛错的类型吞掉整个类型块——放大器找到。 +5. **错误类型溯源**：`offsetOutOfBounds` 在本仓库和 MachOKit 都 grep 不到 → 逐层往依赖找 → swift-fileio 的 `FileIOError`，语义是「拿非法文件偏移去读」→ 指向指针解析错误而非数据损坏。 +6. **领域知识收口**：什么会让间接槽位变成垃圾偏移 → bind 未解析 → `otool -l` 对比两代二进制一行定案（15.5 = `LC_DYLD_INFO_ONLY`，18.5 = `LC_DYLD_CHAINED_FIXUPS`）→ grep `resolveBind` 实现，`guard let fixup = dyldChainedFixups else { return nil }` 一锤定音。 + +**根因链**：① `MachOExtensions/MachOFile+.swift` 的 `resolveBind(fileOffset:)` 只认 chained fixups，旧格式二进制（部署目标 < macOS 12 / iOS 16）的所有 bind 查询返回 nil，指向其它镜像的外部引用（外部 protocol 的 conformance、外部父类）全部按裸指针误读；本地引用不受影响（旧格式磁盘槽位本就存目标地址）。② `SwiftInterfaceBuilder.printRoot` 块级 catch 把单类型打印失败放大成全部类型消失。 + +可复现性武器：`swiftc -target arm64-apple-macosx11.0` 编译即可强制产出 `LC_DYLD_INFO_ONLY` 格式——旧格式 fixture 可以在测试里即时构建，不依赖模拟器 runtime。 + +## 方案 + +1. **fix 1（治本）**：`resolveBind` 在 chained fixups 缺席时回退到 `LC_DYLD_INFO(_ONLY)` bind opcode 流——用 MachOKit 现成的 `bindOperations` / `weakBindOperations` 按 dyld 状态机（segment index / segment offset / symbol name）解释出「文件偏移 → 符号名」索引，惰性构建、按镜像缓存（`@AssociatedObject`）。arm64e 的 threaded 旧格式编码方式不同，遇到即放弃该流（不索引错误偏移）；lazy 流只含 stub，不索引。 +2. **fix 2（治放大器）**：`printRoot` 四个 `try` 块（类型 / 特化 / 协议 / 扩展）全部改为循环内逐项 `printCatchedThrowing`——单个定义抛错只丢它自己。横向排查出第五处同类：`SwiftDeclarationPrinter.printThrowingProtocol` 里根协议的 default-implementation extensions 整块 try，一并逐项化。 + +## 实际执行 + +- 新增 `Tests/SwiftInterfaceTests/LegacyDyldInfoBindTests.swift`（3 项，永久保留）：fixture 在测试内即时编译（约 1.5 秒）；`fixtureUsesLegacyDyldInfoFixups` 钉住格式前提，`externalProtocolConformancesResolveOnLegacyBinaries` 钉 fix 1（外部 protocol 的 conformance 扩展必须渲染），`interfaceContainsEveryTypeOnLegacyBinaries` 钉住批次整体契约（五个类型一个不能少）。 +- **红→绿的阶梯**（每步都实测）：未修 = 7 处断言失败；只上 fix 2 = 4 处（类型缺失从「全部」收敛到「恰好 1 个」——打印期抛错的是 `enum LegacyState`，其余全部恢复，独立证明了逐项降级的价值）；fix 1 + fix 2 = 全绿。 +- 代码改动三处：`MachOExtensions/MachOFile+.swift`（opcode 索引 + resolveBind 分支）、`SwiftInterface/SwiftInterfaceBuilder.swift`（printRoot 五处逐项化中的四处）、`SwiftPrinting/SwiftDeclarationPrinter.swift`（第五处）。 + +## 验证 + +- 全量 `swift test --skip IntegrationTests`：**1315 测试 / 250 套件全过**（较上批 +3 = 新套件），逐字节 interface 快照未变——现代二进制输出零影响（chained 路径原样）。 +- 修复后的 release CLI 重跑 iOS 15.5 模拟器：Combine 10 → 6907 行、WidgetKit 17 → 2795 行、SwiftUI 139 → **81157 行**；三者 `Error resolving` 全部归零，conformance 失败 356/289 计数 → **0**（SwiftUI 7616 个全数解析）。这批二进制上逐项降级甚至没被触发——根因修干净了。 +- 横向排查：`dyldChainedFixups` 假设仅此一处；`try await BlockList` 整块模式仅剩的一处已一并修复；`resolveRebase` 无需改动（旧格式本地引用走裸地址路径，A/B 已证实正常）。 + +## 偏离与教训 + +- **A/B 基线含义更新**：本批之后 feature 分支对旧格式二进制的 interface 输出与 main **合理地不再一致**（feature 更完整）。渲染 A/B 验证流程文档已同步标注；main 合并本批后恢复对等。 +- 打印期抛错的类型起初猜是外部父类的 `LegacyDecoder`，实测是 `enum LegacyState`（raw-value 枚举渲染要解析外部 protocol descriptor）——「先猜后测、以测为准」又一例。 +- `swiftc -target` 老部署目标即可在测试内造出旧格式二进制，这个手法值得记住：格式类回归从此不依赖机器上恰好装着的旧模拟器。 diff --git a/Sources/SwiftInterface/SwiftInterfaceBuilder.swift b/Sources/SwiftInterface/SwiftInterfaceBuilder.swift index 4ba39a2c..55a4d24d 100644 --- a/Sources/SwiftInterface/SwiftInterfaceBuilder.swift +++ b/Sources/SwiftInterface/SwiftInterfaceBuilder.swift @@ -136,50 +136,55 @@ public final class SwiftInterfaceBuilder: Sendable } } - await printCatchedThrowing { - try await BlockList { - for typeDefinition in indexer.rootTypeDefinitions.values { + // Each definition is caught individually: one definition whose + // printing throws (e.g. an unresolvable reference in a legacy or + // damaged binary) drops only itself, never the whole block. A + // block-level catch used to blank every type of the interface the + // moment a single one threw. + await BlockList { + for typeDefinition in indexer.rootTypeDefinitions.values { + await printCatchedThrowing { try await printer.printTypeDefinition(typeDefinition) } } } - await printCatchedThrowing { - try await BlockList { - // Specialized variants live on each `TypeDefinition` rather - // than on the indexer (the indexer is intentionally agnostic - // of user-driven specialization). Walk every type definition - // in the module and surface any specialized children it has - // accumulated through `specialize(with:in:)`. - for typeDefinition in indexer.allTypeDefinitions.values { - for specialized in typeDefinition.specializedChildren { + await BlockList { + // Specialized variants live on each `TypeDefinition` rather + // than on the indexer (the indexer is intentionally agnostic + // of user-driven specialization). Walk every type definition + // in the module and surface any specialized children it has + // accumulated through `specialize(with:in:)`. + for typeDefinition in indexer.allTypeDefinitions.values { + for specialized in typeDefinition.specializedChildren { + await printCatchedThrowing { try await printer.printTypeDefinition(specialized) } } } } - await printCatchedThrowing { - try await BlockList { - for protocolDefinition in indexer.rootProtocolDefinitions.values { + await BlockList { + for protocolDefinition in indexer.rootProtocolDefinitions.values { + await printCatchedThrowing { try await printer.printProtocolDefinition(protocolDefinition) } } } - await printCatchedThrowing { - try await BlockList { - for protocolDefinition in indexer.rootProtocolDefinitions.values.filterNonNil(\.parent) { - for extensionDefinition in protocolDefinition.defaultImplementationExtensions { + await BlockList { + for protocolDefinition in indexer.rootProtocolDefinitions.values.filterNonNil(\.parent) { + for extensionDefinition in protocolDefinition.defaultImplementationExtensions { + await printCatchedThrowing { try await printer.printExtensionDefinition(extensionDefinition) } } } } - await printCatchedThrowing { - try await BlockList { - for extensionDefinition in allExtensionDefinitions { + await BlockList { + for extensionDefinition in allExtensionDefinitions { + await printCatchedThrowing { try await printer.printExtensionDefinition(extensionDefinition) } } diff --git a/Sources/SwiftPrinting/SwiftDeclarationPrinter.swift b/Sources/SwiftPrinting/SwiftDeclarationPrinter.swift index f49e09ad..312613bc 100644 --- a/Sources/SwiftPrinting/SwiftDeclarationPrinter.swift +++ b/Sources/SwiftPrinting/SwiftDeclarationPrinter.swift @@ -177,9 +177,13 @@ public final class SwiftDeclarationPrinter: Sendab } if protocolDefinition.parent == nil { - try await BlockList { + // Per-extension catch: a default-implementation extension whose + // printing throws drops only itself, not the protocol it trails. + await BlockList { for extensionDefinition in protocolDefinition.defaultImplementationExtensions { - try await printExtensionDefinition(extensionDefinition) + await printCatchedThrowing { + try await printExtensionDefinition(extensionDefinition) + } } } } diff --git a/Tests/SwiftInterfaceTests/LegacyDyldInfoBindTests.swift b/Tests/SwiftInterfaceTests/LegacyDyldInfoBindTests.swift new file mode 100644 index 00000000..a3490592 --- /dev/null +++ b/Tests/SwiftInterfaceTests/LegacyDyldInfoBindTests.swift @@ -0,0 +1,136 @@ +import Foundation +import Testing +import MachOKit +@testable import MachOSwiftSection +@_spi(Support) @testable import SwiftInterface + +/// Regression tests for binaries carrying the legacy `LC_DYLD_INFO(_ONLY)` +/// opcode-based fixups (deployment target < macOS 12 / iOS 16, e.g. every +/// iOS 15.5 simulator framework). +/// +/// Two independent defects used to compound on such binaries: +/// 1. `resolveBind(fileOffset:)` only understood `LC_DYLD_CHAINED_FIXUPS`, +/// so every indirect reference to another image's descriptor (external +/// protocol conformances, external superclasses) read as a garbage +/// pointer (`FileIOError.offsetOutOfBounds` / `invalidContextDescriptor`). +/// 2. `SwiftInterfaceBuilder.printRoot` caught errors per BLOCK, so the first +/// type whose printing threw blanked every type in the interface. +/// +/// The fixture is compiled on the fly with a pre-chained-fixups deployment +/// target, which the linker answers with `LC_DYLD_INFO_ONLY` — the same +/// format as the iOS 15.5 simulator frameworks that surfaced the bug. +@Suite(.serialized) +struct LegacyDyldInfoBindTests { + private static let fixtureCompilationResult: Result = { + Result { + let workingDirectory = FileManager.default.temporaryDirectory + .appendingPathComponent("LegacyDyldInfoBindFixture-\(UUID().uuidString)") + try FileManager.default.createDirectory(at: workingDirectory, withIntermediateDirectories: true) + + let sourceURL = workingDirectory.appendingPathComponent("LegacyFixture.swift") + let libraryURL = workingDirectory.appendingPathComponent("libLegacyFixture.dylib") + try Self.fixtureSource.write(to: sourceURL, atomically: true, encoding: .utf8) + + let process = Process() + process.executableURL = URL(fileURLWithPath: "/usr/bin/xcrun") + // macOS 11 predates chained fixups, forcing LC_DYLD_INFO_ONLY output. + process.arguments = [ + "swiftc", "-emit-library", "-module-name", "LegacyFixture", + "-target", "arm64-apple-macosx11.0", + sourceURL.path, "-o", libraryURL.path, + ] + let standardErrorPipe = Pipe() + process.standardError = standardErrorPipe + try process.run() + process.waitUntilExit() + guard process.terminationStatus == 0 else { + let diagnostics = String(decoding: standardErrorPipe.fileHandleForReading.readDataToEndOfFile(), as: UTF8.self) + throw LegacyFixtureCompilationError(diagnostics: diagnostics) + } + return libraryURL + } + }() + + private struct LegacyFixtureCompilationError: Error, CustomStringConvertible { + let diagnostics: String + var description: String { "Legacy fixture compilation failed:\n\(diagnostics)" } + } + + private static let fixtureSource = """ + import Foundation + + public struct LegacyPoint: Equatable, Hashable, Codable { + public var x: Int + public var y: Int + } + + public enum LegacyState: Int, CaseIterable { + case idle + case running + } + + public class LegacyDecoder: JSONDecoder, @unchecked Sendable { + public var callCount: Int = 0 + } + + public protocol LegacyGreeting { + func greet() -> String + } + + public struct LegacyGreeter: LegacyGreeting { + public func greet() -> String { "hello" } + } + """ + + private func loadFixtureMachOFile() throws -> MachOFile { + let libraryURL = try Self.fixtureCompilationResult.get() + let file = try MachOKit.loadFromFile(url: libraryURL) + switch file { + case .machO(let machOFile): + return machOFile + case .fat(let fatFile): + let machOFile = try fatFile.machOFiles().first { $0.header.cpuType == .arm64 } + return try #require(machOFile, "fixture unexpectedly missing an arm64 slice") + } + } + + private func buildInterfaceOutput(of machOFile: MachOFile) async throws -> String { + let builder = try SwiftInterfaceBuilder(configuration: .init(), eventHandlers: [], in: machOFile) + try await builder.prepare() + return try await builder.printRoot().string + } + + @Test func fixtureUsesLegacyDyldInfoFixups() throws { + let machOFile = try loadFixtureMachOFile() + #expect(machOFile.dyldChainedFixups == nil) + #expect(machOFile.bindOperations != nil) + } + + /// Pins the opcode-bind fallback: conformances to protocols living in + /// OTHER images (Equatable/Hashable/CaseIterable in libswiftCore) resolve + /// only when the bind slot's target symbol is recovered from the + /// `LC_DYLD_INFO` opcode stream. Before the fix 13 of the fixture's 14 + /// conformances failed with `offsetOutOfBounds` and none of these + /// extensions rendered. + @Test func externalProtocolConformancesResolveOnLegacyBinaries() async throws { + let machOFile = try loadFixtureMachOFile() + let interfaceOutput = try await buildInterfaceOutput(of: machOFile) + #expect(interfaceOutput.contains("extension LegacyFixture.LegacyPoint: Swift.Equatable")) + #expect(interfaceOutput.contains("extension LegacyFixture.LegacyPoint: Swift.Hashable")) + #expect(interfaceOutput.contains("extension LegacyFixture.LegacyState: Swift.CaseIterable")) + } + + /// Pins per-item print degradation end to end: every fixture type must + /// appear in the interface. Before the batch the first type whose + /// printing threw (the external-superclass class) blanked the WHOLE + /// types block, leaving only imports and global declarations. + @Test func interfaceContainsEveryTypeOnLegacyBinaries() async throws { + let machOFile = try loadFixtureMachOFile() + let interfaceOutput = try await buildInterfaceOutput(of: machOFile) + #expect(interfaceOutput.contains("struct LegacyPoint")) + #expect(interfaceOutput.contains("enum LegacyState")) + #expect(interfaceOutput.contains("class LegacyDecoder")) + #expect(interfaceOutput.contains("struct LegacyGreeter")) + #expect(interfaceOutput.contains("protocol LegacyGreeting")) + } +} From 9559070df72da89cfc9a03561de02bd275f925f4 Mon Sep 17 00:00:00 2001 From: Mx-Iris Date: Thu, 6 Aug 2026 17:05:42 +0800 Subject: [PATCH 37/77] docs: reconcile the evolution log after rebasing onto the rewound main MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Rebasing this branch onto the rewound main brings both sides' additions to ProjectEvolutionLog.md into one file. Git merges them without a conflict and thereby hides a semantic one: each side had numbered its sections from 23, so the file ended up with two of every number from 23 to 26. Resolved by chronology, which is what this ledger is ordered by: 23-26 node-store batches (2026-08-02 .. 08-03) 27-29 SwiftLayout / rendering (2026-08-04 .. 08-06) 30 the rewind record (2026-08-06) That restores 27-29 to the numbers they carried before the rewind, so the cross references in the task reports line up again. Section 30 gains a paragraph recording this rebase, and the matching task report gains a section 7 for it. Re-merging into main will not need renumbering again. Package.swift keeps main's closed upper bound on swift-demangling. The rebase resolved the pin conflict in favour of this branch's side, which carried the pre-0.5.1 `branch:` reference and then an open `from:` — but the closed bound is a decision main made deliberately (its comment explains why: an open bound silently floats the package onto the next demangler release's source breaks), and nothing about the rebase supersedes it. --- .../Internal/ProjectEvolutionLog.md | 165 ++++++++++-------- .../2026-08-06-main-rewind-onto-0.14.1.md | 34 +++- Package.swift | 9 +- 3 files changed, 125 insertions(+), 83 deletions(-) diff --git a/Documentations/Internal/ProjectEvolutionLog.md b/Documentations/Internal/ProjectEvolutionLog.md index f2aad548..6c113f9a 100644 --- a/Documentations/Internal/ProjectEvolutionLog.md +++ b/Documentations/Internal/ProjectEvolutionLog.md @@ -479,7 +479,68 @@ --- -## 23. SwiftLayout 系统框架保真度普查 + foreign struct / ObjC 滑动两批修复 +## 23. 审查清单逐条复现，修掉线程跳转与符号表钉住 + +- **时间段**：2026-08-02。 +- **动机**：[2026-07-31 审查报告](Reviews/2026-07-31-node-store-migration-review.md)留下 17 条待处理项，全部由多智能体审查归并得出，**没有一条做过实测**，且报告自己已经承认对唯一量化过的那条判断错了量级。这一轮的目标不是修完 17 条，而是把每条的真伪与量级钉死，让后续投入落在真问题上；只有性能第一条直接修。 +- **落地**：两处修复 + 一轮全清单实测。 + - `SymbolIndexStore.buildStorageImpl` 的符号 sweep 包进 `StackSafeExecutor.withLargeStack`（函数体移入 `buildStorageSweep`，外层留薄壳）。`withLargeStack` 的收益是 `(批内调用次数 − 1) × 单次跳转成本`，所以必须包住循环——包住单次调用净收益为零，这也是为什么 `printSemantic` 里**不能**加。 + - 新增 `DemangledSymbol.detachedFromSharedTable()`，在存入声明模型的六处调用（`DefinitionBuilder` 的四个构造点 + `TypeDefinition` 的 `deallocatorSymbol` / `destructorSymbol`）。查询路径不动：共享 `[Symbol]` 表对「吐几十万个值随即丢弃」仍是正确取舍，问题只在存下来长期存活的那几千个。公开 API 只增不改。 +- **关键决策**： + - **打印路径的跳转重新定性为上游刻意交易，不修**。查 `swift-demangling` 历史发现 `0.4.3` 的 `NodePrinter.printRoot` 完全没有栈保护（深树在 512 KB worker 上会崩），`7b86137` 把两个公开打印入口强制过 executor 正是为此，且同批给了 `withLargeStack` 作为摊销手段。报告建议的「恢复内联调用」不可行且不应做。 + - **detach 选构造点而非改 `init`**。后者要把 `@MemberwiseInit(.public)` 换成手写 init，而那是公开 API，签名写错会让仓库外调用方编译失败；构造点只有六处且有回归测试守护。 + - **三条判断被实测推翻**：失败名重试的危害在重复计算而非锁争用（8 线程争用 1.97x，无锁路径本身 1.67x）；dyld 全遍历不是退化而是本分支 `7e5dfcc` / `cfe40f8` 正确性修复的代价；`materialize` 占导出总时长仅 0.8%，不构成性能问题。 +- **验证**：`swift test --skip IntegrationTests` 1304 项全绿。关键实测（SwiftUI iOS 18.5，185,988 符号行）：build sweep 10 万符号 1317 ms → 701 ms（1.88x）；符号表钉住从约 21 MB 降到约 2 MB（9,872 个存活值只引用 9,506 行，占表 5.1%）；`memberSymbols` 桶 99.60% 只有 1 个元素，坐实台账第 5 条「机制成立但量级可忽略」。新增回归测试 `SymbolTableRetentionTests`，修复前失败（530 个存储符号全部持有 9,348 行共享表）、修复后通过。 +- **文档**：[TaskReports/2026-08-02-review-reproduction-and-retention-fix.md](TaskReports/2026-08-02-review-reproduction-and-retention-fix.md)、[Reviews/2026-07-31-node-store-migration-review.md](Reviews/2026-07-31-node-store-migration-review.md)（新增第三节实测复现，各条定性按实测更新），`AGENTS.md` 符号索引段落补入「存进声明模型的 `DemangledSymbol` 必须先 detach」硬规则。 +- **对应版本**：0.14.0 之后未发布区间。注意 `Symbol` 删除公开成员（`nlist` 属性、`init(offset:name:nlist:)`）尚未升版本、未写 changelog，发布前必须补。 + +--- + +## 24. 性能批次：失败名裁决、名字去重缓存、dump 路径引用化 + +- **时间段**：2026-08-03。 +- **动机**：[2026-08-02 审查记录](Reviews/2026-08-02-node-store-migration-pr97-review.md)与既有台账合并后剩 19 条待处理，其中「立即可修」与「中等重构」两组获批同批落地；本批延续第 23 节的纪律——先测后修，测出不值得的就裁决留档而不是硬改。 +- **落地**： + - **失败名裁决**（`SymbolIndexStore`）：`demangledNodeReference` 对表内 demangle 失败的名字直接以 sweep 裁决回答 `nil`（`NodeStoreBuilder.demangle` 与 sweep 用同一个 demangler，拒绝集一致）；`lateDemangledNode` 改锁外 demangle + 锁内 insert-if-absent，拒绝结果作为 `nil` 裁决缓存。三条新回归测试钉住（其中缓存断言在修复前红）。 + - **`InternedNodeReferenceCache`**（`MachOSymbols` 新类型）：`NodeReference(interning:)` 的结构去重层，镜像键 + 进程键双作用域，25 处名字构造点全部改走缓存；`SwiftDeclarationIndexer` 清理与内存压力驱逐接通。fixture 实测驻留 mini store 730 → 471（= 结构唯一数），字节 −32%，重复名恢复 `store ===` 快路径。原「每镜像共用 builder」修法被实测推翻（freeze 前无法发引用，调用流即用即取），故改缓存形态。 + - **dump 路径引用化**：`ClassDumper` / `ProtocolDumper` / `ProtocolConformanceDumper` 五处 `demangleSymbol` 调用点迁 `demangleSymbolReference`，visited 集合与 `distributedFunctionNodes` 换 `StructuralNodeReferenceKey`（每 thunk 省一次 materialize）；`MetadataReader.demangleSymbol` 保留契约但包内热调用方清零。 + - **`indexExtensions` 恢复 `await` + 依赖升 0.5.1**：当日早间的「不修」裁决被上游动作推翻——0.5.1（`f913742`）把 print 便利方法整体迁到 `DemanglingNode` 并补 async 变体（挂起 + 大栈），对 `NodeReference` 直接可用，一行恢复 main 的任务挂起语义；具体同步 `print` 同时被上游删除，async 上下文由编译器强制 `await`（dump 路径三处一并加上）。依赖要求升至 `from: "0.5.1"`。remangle 桥接与 `structuralHash` 分配两条随升级按上游设计终审关闭（[ReviewAdjudications.md](ReviewAdjudications.md) A1/A2）。 +- **关键决策**： + - **打印器每成员 materialize 裁决为暂不修**：临时计量显示其只占打印墙钟 1.18%（fixture 全量导出 1313 次共 32.6 ms），根治需 1700 行打印栈泛型化 + 3 处节点合成重设计，投入产出不成比例；数据与重开条件留档在审查记录。 + - 快照套件（SwiftInterfaceTests 53 项含逐字节 interface 快照、SwiftDumpTests）全绿，输出零变化是本批的硬约束。 +- **关联文档**：[TaskReports/2026-08-03-performance-batch-fixes.md](TaskReports/2026-08-03-performance-batch-fixes.md)、台账第 9/10 条闭环与第 4/7 条上游状态核对、AGENTS.md「Symbol indexing」段同步。 +- **对应版本**：0.14.0 之后未发布区间（`feature/node-store-migration` 分支）。 + +--- + +## 25. 系统框架渲染 A/B 验证：78 对零差异 + 流程固化 + +- **时间段**：2026-08-03（紧接第 24 节的性能批次）。 +- **动机**：性能批次落地后，用真实 OS 框架对 `feature/node-store-migration` 做全面的输出对等验证——fixture 快照覆盖构造形态，但覆盖不了 10 万行级输出规模、iOS 15 时代的历史 metadata 与三种 reader 路径的全量组合；维护者随后要求把这套测试固化为「大重构必跑」的流程。 +- **落地**： + - **验证结果**：main ↔ feature 双侧 release CLI + `RenderingVerificationTests` harness，SwiftUI / SwiftUICore / SwiftData / Combine / ActivityKit / WidgetKit 六框架，三部分共 **78 对输出全部逐字节一致**——DyldCache（macOS 26.5.2 + 15.5 归档 cache，24 对）、MachOFile（iOS 15.5 / 18.5 / 26.5 模拟器 runtime，30 对）、MachOImage（当前系统 in-process + 当前 cache 文件，全选项，24 对）。附带 fixture（SymbolTestsCore）smoke 亦一致。 + - **流程固化**：新增 [`Scripts/run-rendering-ab-verification.py`](../../Scripts/run-rendering-ab-verification.py)（自动构建双侧、三部分渲染、逐对 diff、差异非零退出；归档 cache 缺失回退当前系统 cache，指定模拟器缺失回退现有 runtime）与流程文档 [SystemFrameworkRenderingVerification.md](SystemFrameworkRenderingVerification.md)；AGENTS.md 增设「大重构后必跑」规则，并把 `RenderingVerificationTests` 登记为 IntegrationTests 禁跑规则的唯一例外。 +- **关键决策**：cache 镜像一律 `-p` 全路径（iOSSupport 副本消歧）；模拟器一律 `-a arm64`（15.5/18.5 为 fat 二进制）；MachOImage 双侧必须同一次开机会话（memberAddress 依赖 per-boot cache slide）；interface 输出走 `-o` 使时间戳日志与被比对内容分离。 +- **关联文档**：[SystemFrameworkRenderingVerification.md](SystemFrameworkRenderingVerification.md)、[TaskReports/2026-08-03-system-framework-rendering-ab.md](TaskReports/2026-08-03-system-framework-rendering-ab.md)。 +- **对应版本**：0.14.0 之后未发布区间（`feature/node-store-migration` 分支）。 + +--- + +## 26. 旧格式 bind 支持：LC_DYLD_INFO opcode 回退 + interface 逐项降级 + +- **时间段**：2026-08-03（第 25 节 A/B 验证的直接产出）。 +- **动机**：A/B 验证发现 iOS 15.5 模拟器框架的 interface 输出只剩全局函数（三框架、数百条 `offsetOutOfBounds`），dump 却正常。根因两层：`resolveBind(fileOffset:)` 只认 chained fixups，旧格式（部署目标 < macOS 12 / iOS 16 的 `LC_DYLD_INFO_ONLY`）二进制的外部引用全部按裸指针误读；`printRoot` 的块级 catch 把单类型打印失败放大成全部类型消失。 +- **落地**： + - `MachOExtensions/MachOFile+.swift`：chained fixups 缺席时按 dyld 状态机解释 `bindOperations` / `weakBindOperations` opcode 流，惰性构建「文件偏移 → 符号名」索引（arm64e threaded 旧格式不索引、lazy 流不索引）。 + - `SwiftInterface/SwiftInterfaceBuilder.swift` + `SwiftPrinting/SwiftDeclarationPrinter.swift`：printRoot 四个块与 printThrowingProtocol 的 default-implementation extensions 块全部改为逐项 `printCatchedThrowing`——单个定义抛错只丢它自己。 + - 新增 `LegacyDyldInfoBindTests`（fixture 用 `swiftc -target arm64-apple-macosx11.0` 在测试内即时编译强制旧格式；红 7 → 仅 fix 2 剩 4 → 双修复全绿的阶梯实测留档）。 +- **效果**：iOS 15.5 模拟器 interface：Combine 10 → 6907 行、WidgetKit 17 → 2795 行、SwiftUI 139 → 81157 行，解析错误全部归零（SwiftUI 7616 个 conformance 全数解析）；全量 1315 测试 / 250 套件绿，现代二进制快照逐字节不变。旧格式输入的 interface 输出自此与 main 合理不一致（feature 更完整），main 合并后恢复对等。 +- **关联文档**：[TaskReports/2026-08-03-legacy-dyld-info-bind-support.md](TaskReports/2026-08-03-legacy-dyld-info-bind-support.md)（含完整的无调试器调试方法学 walkthrough）、[SystemFrameworkRenderingVerification.md](SystemFrameworkRenderingVerification.md)。 +- **对应版本**：0.14.0 之后未发布区间（`feature/node-store-migration` 分支）。 + +--- + +## 27. SwiftLayout 系统框架保真度普查 + foreign struct / ObjC 滑动两批修复 - **时间段**：2026-08-04。 - **动机**：SwiftLayout 此前的 5 框架普查只度量**解析率**（不降级），从未对真实系统框架做 @@ -517,10 +578,10 @@ --- -## 24. 泛型 fixed MPE 的 spare-bits 布局：错误模型修正 + 普查整型偏差清零 +## 28. 泛型 fixed MPE 的 spare-bits 布局：错误模型修正 + 普查整型偏差清零 - **时间段**：2026-08-05。 -- **动机**：第 23 节留档的硬骨头 ③——`Dictionary` 迭代器一族（`AttributedString.Keys.SetIterator` / +- **动机**：第 27 节留档的硬骨头 ③——`Dictionary` 迭代器一族（`AttributedString.Keys.SetIterator` / `SpatialEventCollection.Iterator`）真值 40/40/XI 126，引擎按「泛型 MPE 恒 tagged」算 41/48/254。当时定性为「编译器预特化 metadata 按编译期 spare-bits 布局」。 - **定性修正（实验推翻旧结论）**：用探针二进制里全新定义的参数类型实例化 @@ -553,9 +614,9 @@ - **文档**：[StaticLayoutEngine.md](StaticLayoutEngine.md)（核心算法 / pitfall / 已知偏差表 / 后续工作四处改写，硬骨头条目标记已解决）、AGENTS.md（`EnumLayoutBridge` 条目重写）、 [TaskReports/2026-08-05-generic-fixed-mpe-spare-bits.md](TaskReports/2026-08-05-generic-fixed-mpe-spare-bits.md)。 -- **对应版本**：未发版（main，0.14.1 之后，紧接第 23 节）。 +- **对应版本**：未发版（main，0.14.1 之后，紧接第 27 节）。 -## 25. 嵌套字段偏移展开的环守卫（indirect case 不下钻 + 路径环检测） +## 29. 嵌套字段偏移展开的环守卫（indirect case 不下钻 + 路径环检测） - **时间段**：2026-08-06。 - **动机**：RuntimeViewer 对 Xcode 的 `DVTIconKit` 生成 Swift interface 时"死循环"， @@ -593,16 +654,16 @@ `depth < 16`，遍历的是嵌套类型**声明**树（天然无环），不属同类，不改。 - **文档**：[NestedFieldOffsetCycleGuard.md](NestedFieldOffsetCycleGuard.md)、 [TaskReports/2026-08-06-nested-field-offset-cycle-guard.md](TaskReports/2026-08-06-nested-field-offset-cycle-guard.md)。 -- **对应版本**：未发版（main，0.14.1 之后，紧接第 24 节）。 +- **对应版本**：未发版（main，0.14.1 之后，紧接第 28 节）。 --- -## 26. main 退回 0.14.1 基线：node-store 合并撤出，四个 SwiftLayout 修复重新接线 +## 30. main 退回 0.14.1 基线：node-store 合并撤出，四个 SwiftLayout 修复重新接线 - **时间段**：2026-08-06。 - **动机**：维护者判断 PR #97（`feature/node-store-migration`，2026-08-04 合入）进 main 过早，要求 main 回到 `0.14.1` 发布点，同时**保留**合并之后落在 main 上的四个 - SwiftLayout / rendering 修复（即本文第 23–25 节），node-store 的工作整体退回 feature + SwiftLayout / rendering 修复（即本文第 27–29 节），node-store 的工作整体退回 feature 分支等待合适时机。 - **落地**：main 由 `621f6fa` 重写为 `3396cfd`（tag `0.14.1`）+ 四次 cherry-pick。 `Package.swift` 随之退回 `swift-demangling` 的 `0.4.5 ..< 0.5.0` pin（0.5.x 重塑了 @@ -625,17 +686,30 @@ - **历史叙述不改写**：各 TaskReport 正文里对旧 SHA 的引用（如「rebase 到 main （`4eeb3b4`）」）保留原貌——那是对当时事实的记录；旧 SHA 一律可通过备份分支解析。 只有「对应版本」这类元数据字段改为不依赖 SHA 的表述。 -- **影响面**：node-store 分支带来的能力（符号索引 NodeStore 化、性能批次、旧格式 - `LC_DYLD_INFO` bind 支持、系统框架渲染 A/B 验证流程与其「大重构必跑」规则）暂时 - **不在 main 上**。本文第 23–25 节由原第 27–29 节顺延而来，故备份分支与新 main 的节号 - 不一致；feature 分支将来合回时本文必然再次冲突，届时需把 node-store 四节插回并重新 - 编号——这是选择重写历史的已知代价。 +- **影响面**：回退期间 node-store 分支带来的能力（符号索引 NodeStore 化、性能批次、 + 旧格式 `LC_DYLD_INFO` bind 支持、系统框架渲染 A/B 验证流程与其「大重构必跑」规则） + 不在 main 上。本文的节号在回退期间也顺延过一轮（node-store 的第 23–26 节移出 main 后， + 原第 27–29 节临时占用了 23–25）。 +- **后续（同日）**：`feature/node-store-migration` 随即以 + `git rebase --onto main 439ecca f31711c` 落到重写后的 main 上——36 个提交线性重放， + 天然排除四个已 cherry-pick 的修复（它们在 `f31711c` 之上），全程唯一冲突是 + `Package.swift` 里 swift-demangling 的 pin 之争（取 node-store 侧，终态回到 + `from: "0.5.1"`）。本文按编年顺序恢复原样：node-store 四节回到 23–26（工作时间 + 08-02~08-03），SwiftLayout 三节回到它们原本的 27–29（08-04~08-06），本节顺延为 + 第 30 节。此后重新合入 main 时，本文这 30 节不再需要重新编号。 +- **后续（2026-08-07）**：main 上又落了一个批次（class / static 成员关键字还原）， + 分支再次 rebase 到 main。这次的冲突面比上次小得多：代码只有 `SwiftDeclarationPrinter` + 的三个成员打印入口（main 加 `isClassMember:` 参数、本分支把 `node` 换成 + `node.materialize()`，两侧叠加即可），文档是 `Documentations/README.md` 的索引表 + 与本文——main 的新批次作为**第 31 节**接在本节之后，既有 30 节的编号一个没动， + 上一条「不再需要重新编号」的承诺因此只对**本分支自己的节**成立：main 每落一个 + 批次，本文末尾就要接一节新的，这是编年账本的常态，不是重新编号。 - **文档**：[TaskReports/2026-08-06-main-rewind-onto-0.14.1.md](TaskReports/2026-08-06-main-rewind-onto-0.14.1.md)。 -- **对应版本**：`0.14.1`（main 与该 tag 之间此后仅有第 23–25 节的三个修复批次）。 +- **对应版本**：`0.14.1`（main 与该 tag 之间此后仅有第 27–29、31 节的四个修复批次）。 --- -## 27. class / static 成员关键字的还原（vtable method descriptor 判据） +## 31. class / static 成员关键字的还原（vtable method descriptor 判据） - **时间段**：2026-08-07。 - **动机**：interface 输出把所有类型级成员渲染成 `static`，源码里的 `class func` / @@ -664,67 +738,6 @@ --- -## 23. 审查清单逐条复现，修掉线程跳转与符号表钉住 - -- **时间段**：2026-08-02。 -- **动机**：[2026-07-31 审查报告](Reviews/2026-07-31-node-store-migration-review.md)留下 17 条待处理项，全部由多智能体审查归并得出，**没有一条做过实测**，且报告自己已经承认对唯一量化过的那条判断错了量级。这一轮的目标不是修完 17 条，而是把每条的真伪与量级钉死，让后续投入落在真问题上；只有性能第一条直接修。 -- **落地**：两处修复 + 一轮全清单实测。 - - `SymbolIndexStore.buildStorageImpl` 的符号 sweep 包进 `StackSafeExecutor.withLargeStack`（函数体移入 `buildStorageSweep`，外层留薄壳）。`withLargeStack` 的收益是 `(批内调用次数 − 1) × 单次跳转成本`，所以必须包住循环——包住单次调用净收益为零，这也是为什么 `printSemantic` 里**不能**加。 - - 新增 `DemangledSymbol.detachedFromSharedTable()`，在存入声明模型的六处调用（`DefinitionBuilder` 的四个构造点 + `TypeDefinition` 的 `deallocatorSymbol` / `destructorSymbol`）。查询路径不动：共享 `[Symbol]` 表对「吐几十万个值随即丢弃」仍是正确取舍，问题只在存下来长期存活的那几千个。公开 API 只增不改。 -- **关键决策**： - - **打印路径的跳转重新定性为上游刻意交易，不修**。查 `swift-demangling` 历史发现 `0.4.3` 的 `NodePrinter.printRoot` 完全没有栈保护（深树在 512 KB worker 上会崩），`7b86137` 把两个公开打印入口强制过 executor 正是为此，且同批给了 `withLargeStack` 作为摊销手段。报告建议的「恢复内联调用」不可行且不应做。 - - **detach 选构造点而非改 `init`**。后者要把 `@MemberwiseInit(.public)` 换成手写 init，而那是公开 API，签名写错会让仓库外调用方编译失败；构造点只有六处且有回归测试守护。 - - **三条判断被实测推翻**：失败名重试的危害在重复计算而非锁争用（8 线程争用 1.97x，无锁路径本身 1.67x）；dyld 全遍历不是退化而是本分支 `7e5dfcc` / `cfe40f8` 正确性修复的代价；`materialize` 占导出总时长仅 0.8%，不构成性能问题。 -- **验证**：`swift test --skip IntegrationTests` 1304 项全绿。关键实测（SwiftUI iOS 18.5，185,988 符号行）：build sweep 10 万符号 1317 ms → 701 ms（1.88x）；符号表钉住从约 21 MB 降到约 2 MB（9,872 个存活值只引用 9,506 行，占表 5.1%）；`memberSymbols` 桶 99.60% 只有 1 个元素，坐实台账第 5 条「机制成立但量级可忽略」。新增回归测试 `SymbolTableRetentionTests`，修复前失败（530 个存储符号全部持有 9,348 行共享表）、修复后通过。 -- **文档**：[TaskReports/2026-08-02-review-reproduction-and-retention-fix.md](TaskReports/2026-08-02-review-reproduction-and-retention-fix.md)、[Reviews/2026-07-31-node-store-migration-review.md](Reviews/2026-07-31-node-store-migration-review.md)（新增第三节实测复现，各条定性按实测更新），`AGENTS.md` 符号索引段落补入「存进声明模型的 `DemangledSymbol` 必须先 detach」硬规则。 -- **对应版本**：0.14.0 之后未发布区间。注意 `Symbol` 删除公开成员（`nlist` 属性、`init(offset:name:nlist:)`）尚未升版本、未写 changelog，发布前必须补。 - ---- - -## 24. 性能批次：失败名裁决、名字去重缓存、dump 路径引用化 - -- **时间段**：2026-08-03。 -- **动机**：[2026-08-02 审查记录](Reviews/2026-08-02-node-store-migration-pr97-review.md)与既有台账合并后剩 19 条待处理，其中「立即可修」与「中等重构」两组获批同批落地；本批延续第 23 节的纪律——先测后修，测出不值得的就裁决留档而不是硬改。 -- **落地**： - - **失败名裁决**（`SymbolIndexStore`）：`demangledNodeReference` 对表内 demangle 失败的名字直接以 sweep 裁决回答 `nil`（`NodeStoreBuilder.demangle` 与 sweep 用同一个 demangler，拒绝集一致）；`lateDemangledNode` 改锁外 demangle + 锁内 insert-if-absent，拒绝结果作为 `nil` 裁决缓存。三条新回归测试钉住（其中缓存断言在修复前红）。 - - **`InternedNodeReferenceCache`**（`MachOSymbols` 新类型）：`NodeReference(interning:)` 的结构去重层，镜像键 + 进程键双作用域，25 处名字构造点全部改走缓存；`SwiftDeclarationIndexer` 清理与内存压力驱逐接通。fixture 实测驻留 mini store 730 → 471（= 结构唯一数），字节 −32%，重复名恢复 `store ===` 快路径。原「每镜像共用 builder」修法被实测推翻（freeze 前无法发引用，调用流即用即取），故改缓存形态。 - - **dump 路径引用化**：`ClassDumper` / `ProtocolDumper` / `ProtocolConformanceDumper` 五处 `demangleSymbol` 调用点迁 `demangleSymbolReference`，visited 集合与 `distributedFunctionNodes` 换 `StructuralNodeReferenceKey`（每 thunk 省一次 materialize）；`MetadataReader.demangleSymbol` 保留契约但包内热调用方清零。 - - **`indexExtensions` 恢复 `await` + 依赖升 0.5.1**：当日早间的「不修」裁决被上游动作推翻——0.5.1（`f913742`）把 print 便利方法整体迁到 `DemanglingNode` 并补 async 变体（挂起 + 大栈），对 `NodeReference` 直接可用，一行恢复 main 的任务挂起语义；具体同步 `print` 同时被上游删除，async 上下文由编译器强制 `await`（dump 路径三处一并加上）。依赖要求升至 `from: "0.5.1"`。remangle 桥接与 `structuralHash` 分配两条随升级按上游设计终审关闭（[ReviewAdjudications.md](ReviewAdjudications.md) A1/A2）。 -- **关键决策**： - - **打印器每成员 materialize 裁决为暂不修**：临时计量显示其只占打印墙钟 1.18%（fixture 全量导出 1313 次共 32.6 ms），根治需 1700 行打印栈泛型化 + 3 处节点合成重设计，投入产出不成比例；数据与重开条件留档在审查记录。 - - 快照套件（SwiftInterfaceTests 53 项含逐字节 interface 快照、SwiftDumpTests）全绿，输出零变化是本批的硬约束。 -- **关联文档**：[TaskReports/2026-08-03-performance-batch-fixes.md](TaskReports/2026-08-03-performance-batch-fixes.md)、台账第 9/10 条闭环与第 4/7 条上游状态核对、AGENTS.md「Symbol indexing」段同步。 -- **对应版本**：0.14.0 之后未发布区间（`feature/node-store-migration` 分支）。 - ---- - -## 25. 系统框架渲染 A/B 验证：78 对零差异 + 流程固化 - -- **时间段**：2026-08-03（紧接第 24 节的性能批次）。 -- **动机**：性能批次落地后，用真实 OS 框架对 `feature/node-store-migration` 做全面的输出对等验证——fixture 快照覆盖构造形态，但覆盖不了 10 万行级输出规模、iOS 15 时代的历史 metadata 与三种 reader 路径的全量组合；维护者随后要求把这套测试固化为「大重构必跑」的流程。 -- **落地**： - - **验证结果**：main ↔ feature 双侧 release CLI + `RenderingVerificationTests` harness，SwiftUI / SwiftUICore / SwiftData / Combine / ActivityKit / WidgetKit 六框架，三部分共 **78 对输出全部逐字节一致**——DyldCache（macOS 26.5.2 + 15.5 归档 cache，24 对）、MachOFile（iOS 15.5 / 18.5 / 26.5 模拟器 runtime，30 对）、MachOImage（当前系统 in-process + 当前 cache 文件，全选项，24 对）。附带 fixture（SymbolTestsCore）smoke 亦一致。 - - **流程固化**：新增 [`Scripts/run-rendering-ab-verification.py`](../../Scripts/run-rendering-ab-verification.py)（自动构建双侧、三部分渲染、逐对 diff、差异非零退出；归档 cache 缺失回退当前系统 cache，指定模拟器缺失回退现有 runtime）与流程文档 [SystemFrameworkRenderingVerification.md](SystemFrameworkRenderingVerification.md)；AGENTS.md 增设「大重构后必跑」规则，并把 `RenderingVerificationTests` 登记为 IntegrationTests 禁跑规则的唯一例外。 -- **关键决策**：cache 镜像一律 `-p` 全路径（iOSSupport 副本消歧）；模拟器一律 `-a arm64`（15.5/18.5 为 fat 二进制）；MachOImage 双侧必须同一次开机会话（memberAddress 依赖 per-boot cache slide）；interface 输出走 `-o` 使时间戳日志与被比对内容分离。 -- **关联文档**：[SystemFrameworkRenderingVerification.md](SystemFrameworkRenderingVerification.md)、[TaskReports/2026-08-03-system-framework-rendering-ab.md](TaskReports/2026-08-03-system-framework-rendering-ab.md)。 -- **对应版本**：0.14.0 之后未发布区间（`feature/node-store-migration` 分支）。 - ---- - -## 26. 旧格式 bind 支持：LC_DYLD_INFO opcode 回退 + interface 逐项降级 - -- **时间段**：2026-08-03（第 25 节 A/B 验证的直接产出）。 -- **动机**：A/B 验证发现 iOS 15.5 模拟器框架的 interface 输出只剩全局函数（三框架、数百条 `offsetOutOfBounds`），dump 却正常。根因两层：`resolveBind(fileOffset:)` 只认 chained fixups，旧格式（部署目标 < macOS 12 / iOS 16 的 `LC_DYLD_INFO_ONLY`）二进制的外部引用全部按裸指针误读；`printRoot` 的块级 catch 把单类型打印失败放大成全部类型消失。 -- **落地**： - - `MachOExtensions/MachOFile+.swift`：chained fixups 缺席时按 dyld 状态机解释 `bindOperations` / `weakBindOperations` opcode 流，惰性构建「文件偏移 → 符号名」索引（arm64e threaded 旧格式不索引、lazy 流不索引）。 - - `SwiftInterface/SwiftInterfaceBuilder.swift` + `SwiftPrinting/SwiftDeclarationPrinter.swift`：printRoot 四个块与 printThrowingProtocol 的 default-implementation extensions 块全部改为逐项 `printCatchedThrowing`——单个定义抛错只丢它自己。 - - 新增 `LegacyDyldInfoBindTests`（fixture 用 `swiftc -target arm64-apple-macosx11.0` 在测试内即时编译强制旧格式；红 7 → 仅 fix 2 剩 4 → 双修复全绿的阶梯实测留档）。 -- **效果**：iOS 15.5 模拟器 interface：Combine 10 → 6907 行、WidgetKit 17 → 2795 行、SwiftUI 139 → 81157 行，解析错误全部归零（SwiftUI 7616 个 conformance 全数解析）；全量 1315 测试 / 250 套件绿，现代二进制快照逐字节不变。旧格式输入的 interface 输出自此与 main 合理不一致（feature 更完整），main 合并后恢复对等。 -- **关联文档**：[TaskReports/2026-08-03-legacy-dyld-info-bind-support.md](TaskReports/2026-08-03-legacy-dyld-info-bind-support.md)（含完整的无调试器调试方法学 walkthrough）、[SystemFrameworkRenderingVerification.md](SystemFrameworkRenderingVerification.md)。 -- **对应版本**：0.14.0 之后未发布区间（`feature/node-store-migration` 分支）。 - ---- - ## 维护约定 1. **每个非平凡批次结束时必须在本文追加/更新一节**（新工作弧新增一节；延续既有弧则在该节 diff --git a/Documentations/Internal/TaskReports/2026-08-06-main-rewind-onto-0.14.1.md b/Documentations/Internal/TaskReports/2026-08-06-main-rewind-onto-0.14.1.md index 4b64105f..d08e6e53 100644 --- a/Documentations/Internal/TaskReports/2026-08-06-main-rewind-onto-0.14.1.md +++ b/Documentations/Internal/TaskReports/2026-08-06-main-rewind-onto-0.14.1.md @@ -5,7 +5,8 @@ - **备份**: `backup/main-before-0.14.1-rewind`（分支 + 同名带日期 tag `backup/main-before-0.14.1-rewind-2026-08-06`），本地与 `origin` 双份，均指向重写前的 main tip `621f6fa` -- **关联**: [ProjectEvolutionLog.md](../ProjectEvolutionLog.md) 第 26 节 +- **关联**: [ProjectEvolutionLog.md](../ProjectEvolutionLog.md) 第 30 节（回退当时记为第 26 + 节，随后的 rebase 按编年把 node-store 四节移回 23–26，本节顺延为 30——见下方第 7 节） ## 1. 问题 @@ -118,10 +119,31 @@ node-store 的全部工作与被重写掉的 merge 提交都可完整还原。 ## 6. 偏差与遗留 -- ProjectEvolutionLog 的节号在两条历史里不再一致：备份分支上第 27/28/29 节 = 新 main - 上的第 23/24/25 节。将来 `feature/node-store-migration` 合回 main 时，这个文件必然 - 再次冲突，届时需要把 node-store 的四节插回并重新编号。这是选择重写历史的已知代价， - 也是唯一一处需要人工照顾的地方。 +- ProjectEvolutionLog 的节号在回退期间两条历史不一致（备份分支第 27/28/29 节 = 回退后 + main 的第 23/24/25 节），这是选择重写历史的已知代价，也是唯一一处需要人工照顾的地方。 + **已于同日随 rebase 处理完毕**，见下条。 - node-store 分支带来的能力（符号索引 NodeStore 化、性能批次、旧格式 `LC_DYLD_INFO` - bind 支持、系统框架渲染 A/B 验证流程）**暂时不在 main 上**。其中「大重构必跑 A/B + bind 支持、系统框架渲染 A/B 验证流程）在回退期间不在 main 上。其中「大重构必跑 A/B 验证」这条 AGENTS.md 规则也随之回退——它是 node-store 分支引入的。 + +## 7. 后续：同日把 node-store 分支 rebase 到重写后的 main 上 + +回退完成后，`feature/node-store-migration` 随即以 + +```bash +git rebase --onto main 439ecca f31711c +``` + +落到新 main 上。选这个区间而非 `git rebase main ` 的理由：`439ecca..f31711c` +恰好是 node-store 的**纯工作线**（36 个提交、0 个 merge），四个已 cherry-pick 的 +SwiftLayout 修复位于 `f31711c` **之上**，因而天然被排除——不必依赖 patch-id 去重（它们 +的 `ProjectEvolutionLog.md` 部分已被改过，patch-id 本来就对不上，会误判为新提交）。 + +- **唯一冲突**：`Package.swift` 的 swift-demangling pin，出现在第 22/36 个提交 + （`cf368cc build: track swift-demangling's feature/node-store branch`）。取 node-store + 侧（`branch: "feature/node-store"`）以保持该提交原本的语义演进，后续的 `1234f41` + （0.5.0）与 `2d69c63`（0.5.1）照常覆盖它，终态回到 `from: "0.5.1"`。 +- **ProjectEvolutionLog 按编年恢复原样**：node-store 四节回到 23–26（工作时间 + 08-02~08-03），SwiftLayout 三节回到原本的 27–29（08-04~08-06），本次回退的记录成为 + 第 30 节。注意这里自动合并**不会**报冲突却会留下重复编号（两侧各有 23–26 节），是 + 需要人工核对的语义问题，不是文本冲突。此后重新合入 main 不再需要重新编号。 diff --git a/Package.swift b/Package.swift index 527e41ea..fa288275 100644 --- a/Package.swift +++ b/Package.swift @@ -213,7 +213,14 @@ extension Package.Dependency { ), remote: .package( url: "https://github.com/MxIris-Reverse-Engineering/swift-demangling", - from: "0.5.1", + // The node-store migration adopts 0.5.x: that release reshaped + // `NodePrinterTarget` (`write(_:context:)` / + // `pushTypeReferenceScope(_:)` take `@autoclosure` parameters and + // lost their default implementations) and dropped `Node: Codable`. + // The upper bound stays closed for the same reason the pre-adoption + // pin had one — an open bound silently floats this package onto the + // next demangler release's source breaks. + "0.5.1" ..< "0.6.0", ), ) From 78736678610fe26904375d25faa41ee0d40402e3 Mon Sep 17 00:00:00 2001 From: Mx-Iris Date: Sat, 8 Aug 2026 16:42:07 +0800 Subject: [PATCH 38/77] perf(MachOSymbols,SwiftLayout,SwiftDeclarationRendering): reserve sweep capacity and clear the last cached demangles Memory-graph counts over five indexed system images (208,809 live Nodes, 14,451 NodeStores) traced a chunk of the Node population to the two remaining cached demangleAsNode call sites, and the sweep's buffer-growth copies to the unreserved NodeStoreBuilder. - SymbolIndexStore: pre-reserve the sweep builder via the upstream reserveCapacity(expectedSymbolCount:) API (growing-only, interning-result-neutral; upstream measured the >=1MiB realloc copies drop 12 -> 4 and the cold-start footprint spike halved). - ObjCClassIndex / SpecializedMetadataNodeSubstitution: convert the last two cached demangleAsNode call sites to demangleAsNodeTransient - both consume-and-drop uses that permanently pinned whole trees into the global NodeCache. Sources now carries zero cached demangleAsNode call sites; AGENTS.md records that as a Stage 5c invariant. The three mini-store pipelines stay untouched by decision, awaiting upstream evolution 0010 (SharedNodeStore); adjudication recorded in NodeStoreMigrationPlan.md and ProjectEvolutionLog.md section 32. Verified: swift build clean; swift test --skip IntegrationTests 1337 tests green (counts checked, not pipe exit codes). --- AGENTS.md | 2 +- Documentations/Internal/NodeStoreMigrationPlan.md | 8 ++++++++ Documentations/Internal/ProjectEvolutionLog.md | 13 +++++++++++++ Sources/MachOSymbols/SymbolIndexStore.swift | 1 + .../SpecializedMetadataNodeSubstitution.swift | 12 +++++++----- Sources/SwiftLayout/ObjCClassIndex.swift | 8 +++++--- 6 files changed, 35 insertions(+), 9 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index ce880b4c..a49b35cf 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -208,7 +208,7 @@ let node = try demangleAsNode("$sSiD") // Returns Node tree let string = node.print(using: .default) // "Swift.Int" ``` -**Symbol indexing (NodeStore-backed)**: `SymbolIndexStore.Storage` no longer retains `Node` class trees. The build sweep demangles each symbol cache-free onto a transient tree (`@_spi(Internals) demangleAsNodeTransient`), classifies on that tree, and interns it into a per-image `NodeStoreBuilder`. Nothing touches the global `NodeCache`, and dropping a `Storage` releases the whole per-image footprint. Storage layout (Stage 3): one flat `symbolTable: [Symbol]` row per unique symbol name (canonical cache-adjusted offset; raw and adjusted offset keys share one row), a parallel `rootNodeIndexByTableRow` array, and every classification index stores 4-byte `UInt32` table rows (member/opaque keys are 4-byte `NodeStore.NodeIndex`); indexes accumulate in final row form during the sweep, so `freeze()` is followed by a plain move, not a conversion pass. `Symbol` is 32 bytes (no `nlist` existential — the external-undefined bit is extracted at collection time into `isExternal`), and `DemangledSymbol` is a 32-byte value (shared-table reference + row + `NodeReference`) whose `symbol` is computed; both are asserted by `compactValueLayouts`. **A `DemangledSymbol` stored into the declaration model must be detached first** (`detachedFromSharedTable()`): the shared `[Symbol]` buffer is the right trade for the hundreds of thousands of values a query vends and drops, but `Array` is a reference to its buffer, so a single stored survivor pins the whole table and every mangled name in it — defeating `removeSubIndexer(_:)`, whose reason for existing is reclaiming per-image memory. Measured on SwiftUI (iOS 18.5): 9,872 stored values referenced 9,506 distinct rows, 5.1% of a 185,988-row table, so detaching trades ~0.6 MB of small allocations for ~19.9 MB of retention. The six storing sites are `DefinitionBuilder`'s four construction points (`Accessor` for variables/subscripts, `FunctionDefinition` for allocators/functions) and `TypeDefinition`'s `deallocatorSymbol` / `destructorSymbol` assignments; `SymbolTableRetentionTests` fails the moment a new one forgets. Do NOT detach on the query path. Query APIs construct `DemangledSymbol`/`NodeReference` values at the exit boundary; APIs taking an externally demangled node (`memberSymbols(of:for:node:)` with `Node` or `NodeReference`, `opaqueTypeDescriptorSymbol(for:)`) match keys via `NodeReference.structurallyEquals(_:)` (same-store O(1), cross-store structural walk) — `opaqueTypeDescriptorSymbol(for:)` buckets its entries by `DemanglingNode.identifier` first (`opaqueTypeDescriptorEntriesByMemberIdentifier`), because a full scan there is O(descriptors × printed `some` returns). `demangledNodeReference(for:)` matches on the symbol **name alone** — a demangled tree is a pure function of the name and the flat table holds one row per unique name, so comparing the row's canonical offset against the queried symbol's offset could only reject valid hits (it once rejected every symbol of a dyld-cache image, since `symbols(for:in:)` stamps each vended `Symbol` with the offset it was queried by). A name the sweep covered is answered from the table verdict alone — including the *rejection* verdict: a `nil` root means the demangler already refused the name once, and the same demangler backs the late path, so the query returns `nil` without re-demangling. Names genuinely outside the build sweep fall back to name-keyed mini stores (`lateDemangledNode(forName:)` / `MetadataReader.demangleSymbolReference(for:in:)`): the demangle runs *outside* the cache lock (never hold `os_unfair_lock` across a potentially blocking large-stack hop) and the lock arbitrates insert-if-absent — a racing loser discards its store and returns the winner's reference, preserving one-store-per-name — while rejections are cached as `nil` verdicts and never retried (`rejectedLateNameCachesItsFailure` / `tableCoveredNameNeverEntersLateCache` / `concurrentLateQueriesShareOneStore` pin all three properties). Struct-layout changes in `MachOSymbols` need a `swift package clean` rebuild — SwiftPM incremental builds have been observed linking stale downstream objects (runtime SIGSEGV in `outlined destroy`). See `Documentations/Internal/NodeStoreMigrationPlan.md`. **Declaration values hold `NodeReference` (Stage 5a)**: `DefinitionName.node` (`TypeName`/`ProtocolName`/`ExtensionName`), `Variable/Function/SubscriptDefinition.node`, `FieldDefinition.typeNode` and `ExtensionDefinition.genericSignature` are `NodeReference`, not `Node`. Member definitions reference the per-image symbol store directly (a live definition keeps that store alive — the intended per-image recycling model); metadata-derived name trees (`MetadataReader` output feeding `TypeName`/`ProtocolName`/`ExtensionName`/generic signatures) go through `InternedNodeReferenceCache` (`Sources/MachOSymbols/`) — a structural-dedup layer over `NodeReference(interning:)` with an image-keyed scope (evicted with the image; `SwiftDeclarationIndexer`'s cleanup removes it alongside the symbol store) and a process-keyed scope for the in-process helpers with no Mach-O handle, so repeated names (conformance protocol names, shared parents, extension targets) share one store and name equality gets the `store ===` fast path (measured on the fixture: 730 retained mini stores → 471, matching the 472 structurally unique trees). Do NOT call bare `NodeReference(interning:)` on a batch path — route through the cache. `TypeDefinition.index(in:)` still batches all field type trees of one type into a single shared store. The `Name` types customize `Hashable` to STRUCTURAL semantics (`structurallyEquals` + `structuralHash`) because `NodeReference`'s intrinsic `Hashable` is store-identity based and would split equal keys minted into different stores; they are deliberately NOT `Codable` — upstream dropped `Node: Codable` (a mangled symbol already is the tree's serialized form: smaller, ABI-stable across toolchains, and sharing-preserving through the round trip), and nothing persists these values. A consumer that ever needs to should persist `mangleAsString(node)` and read it back with `demangleAsNode(_:)` rather than reintroducing a node encoding. **Any `Dictionary`/`Set` keyed on a `NodeReference` whose keys and lookups can come from DIFFERENT stores must key on `StructuralNodeReferenceKey` (`Sources/MachOSymbols/`, next to the mini stores it reconciles), never a bare `NodeReference`.** Different stores are the norm, not the exception: `demangledNodeReference(for:)` falls back to a mini store, and `InternedNodeReferenceCache` — every `MetadataReader`-derived tree — deduplicates equal trees onto one store per scope but still mints a distinct store per unique name, so cross-store mixing remains the default assumption. Everything that mixes those with image-store references is keyed structurally: the override/vtable lookups (`methodDescriptorLookup` / `vtableOffsetLookup` in `TypeDefinition.index` → `DefinitionBuilder`), `DefinitionBuilder`'s accessor grouping and merged-thunk dedup (`accessorsByNode` / `canonicalIndexByFunctionNode` / `canonicalIndexByAllocatorNode`), and every `visitedNodes` "already claimed this symbol" set (`ExtensionDefinition` / `ProtocolDefinition` / `TypeDefinition` / `OverrideSymbolMatcher` / `ProtocolConformanceDumper`). Under bare keys these fail silently and differently: the override lookups drop the `override` keyword + vtable-offset comment (the Stage 5a regression fixed 2026-07-25), a subscript's getter and setter split into two buckets so the setter-only one is discarded, merged thunks emit their `func`/`init` twice, and a witness can be claimed twice. Bare-`NodeReference` keys are only safe for grouping/dedup WITHIN one `memberSymbols` batch (single hash-consed store, where structural equality coincides with index equality). **Transient demangling everywhere else (Stage 5c)**: `MetadataReader`, `RuntimeFieldLayoutBackend`, `TypedDumper`, `ClassHierarchyDumper` and `Symbol.demangledNode` use `demangleAsNodeTransient` + `Node.createTransient`, so the global `NodeCache` no longer grows while browsing. Do NOT reintroduce `demangleAsNode`/`Node.create` on unbounded inputs in those paths. +**Symbol indexing (NodeStore-backed)**: `SymbolIndexStore.Storage` no longer retains `Node` class trees. The build sweep demangles each symbol cache-free onto a transient tree (`@_spi(Internals) demangleAsNodeTransient`), classifies on that tree, and interns it into a per-image `NodeStoreBuilder`. Nothing touches the global `NodeCache`, and dropping a `Storage` releases the whole per-image footprint. Storage layout (Stage 3): one flat `symbolTable: [Symbol]` row per unique symbol name (canonical cache-adjusted offset; raw and adjusted offset keys share one row), a parallel `rootNodeIndexByTableRow` array, and every classification index stores 4-byte `UInt32` table rows (member/opaque keys are 4-byte `NodeStore.NodeIndex`); indexes accumulate in final row form during the sweep, so `freeze()` is followed by a plain move, not a conversion pass. `Symbol` is 32 bytes (no `nlist` existential — the external-undefined bit is extracted at collection time into `isExternal`), and `DemangledSymbol` is a 32-byte value (shared-table reference + row + `NodeReference`) whose `symbol` is computed; both are asserted by `compactValueLayouts`. **A `DemangledSymbol` stored into the declaration model must be detached first** (`detachedFromSharedTable()`): the shared `[Symbol]` buffer is the right trade for the hundreds of thousands of values a query vends and drops, but `Array` is a reference to its buffer, so a single stored survivor pins the whole table and every mangled name in it — defeating `removeSubIndexer(_:)`, whose reason for existing is reclaiming per-image memory. Measured on SwiftUI (iOS 18.5): 9,872 stored values referenced 9,506 distinct rows, 5.1% of a 185,988-row table, so detaching trades ~0.6 MB of small allocations for ~19.9 MB of retention. The six storing sites are `DefinitionBuilder`'s four construction points (`Accessor` for variables/subscripts, `FunctionDefinition` for allocators/functions) and `TypeDefinition`'s `deallocatorSymbol` / `destructorSymbol` assignments; `SymbolTableRetentionTests` fails the moment a new one forgets. Do NOT detach on the query path. Query APIs construct `DemangledSymbol`/`NodeReference` values at the exit boundary; APIs taking an externally demangled node (`memberSymbols(of:for:node:)` with `Node` or `NodeReference`, `opaqueTypeDescriptorSymbol(for:)`) match keys via `NodeReference.structurallyEquals(_:)` (same-store O(1), cross-store structural walk) — `opaqueTypeDescriptorSymbol(for:)` buckets its entries by `DemanglingNode.identifier` first (`opaqueTypeDescriptorEntriesByMemberIdentifier`), because a full scan there is O(descriptors × printed `some` returns). `demangledNodeReference(for:)` matches on the symbol **name alone** — a demangled tree is a pure function of the name and the flat table holds one row per unique name, so comparing the row's canonical offset against the queried symbol's offset could only reject valid hits (it once rejected every symbol of a dyld-cache image, since `symbols(for:in:)` stamps each vended `Symbol` with the offset it was queried by). A name the sweep covered is answered from the table verdict alone — including the *rejection* verdict: a `nil` root means the demangler already refused the name once, and the same demangler backs the late path, so the query returns `nil` without re-demangling. Names genuinely outside the build sweep fall back to name-keyed mini stores (`lateDemangledNode(forName:)` / `MetadataReader.demangleSymbolReference(for:in:)`): the demangle runs *outside* the cache lock (never hold `os_unfair_lock` across a potentially blocking large-stack hop) and the lock arbitrates insert-if-absent — a racing loser discards its store and returns the winner's reference, preserving one-store-per-name — while rejections are cached as `nil` verdicts and never retried (`rejectedLateNameCachesItsFailure` / `tableCoveredNameNeverEntersLateCache` / `concurrentLateQueriesShareOneStore` pin all three properties). Struct-layout changes in `MachOSymbols` need a `swift package clean` rebuild — SwiftPM incremental builds have been observed linking stale downstream objects (runtime SIGSEGV in `outlined destroy`). See `Documentations/Internal/NodeStoreMigrationPlan.md`. **Declaration values hold `NodeReference` (Stage 5a)**: `DefinitionName.node` (`TypeName`/`ProtocolName`/`ExtensionName`), `Variable/Function/SubscriptDefinition.node`, `FieldDefinition.typeNode` and `ExtensionDefinition.genericSignature` are `NodeReference`, not `Node`. Member definitions reference the per-image symbol store directly (a live definition keeps that store alive — the intended per-image recycling model); metadata-derived name trees (`MetadataReader` output feeding `TypeName`/`ProtocolName`/`ExtensionName`/generic signatures) go through `InternedNodeReferenceCache` (`Sources/MachOSymbols/`) — a structural-dedup layer over `NodeReference(interning:)` with an image-keyed scope (evicted with the image; `SwiftDeclarationIndexer`'s cleanup removes it alongside the symbol store) and a process-keyed scope for the in-process helpers with no Mach-O handle, so repeated names (conformance protocol names, shared parents, extension targets) share one store and name equality gets the `store ===` fast path (measured on the fixture: 730 retained mini stores → 471, matching the 472 structurally unique trees). Do NOT call bare `NodeReference(interning:)` on a batch path — route through the cache. `TypeDefinition.index(in:)` still batches all field type trees of one type into a single shared store. The `Name` types customize `Hashable` to STRUCTURAL semantics (`structurallyEquals` + `structuralHash`) because `NodeReference`'s intrinsic `Hashable` is store-identity based and would split equal keys minted into different stores; they are deliberately NOT `Codable` — upstream dropped `Node: Codable` (a mangled symbol already is the tree's serialized form: smaller, ABI-stable across toolchains, and sharing-preserving through the round trip), and nothing persists these values. A consumer that ever needs to should persist `mangleAsString(node)` and read it back with `demangleAsNode(_:)` rather than reintroducing a node encoding. **Any `Dictionary`/`Set` keyed on a `NodeReference` whose keys and lookups can come from DIFFERENT stores must key on `StructuralNodeReferenceKey` (`Sources/MachOSymbols/`, next to the mini stores it reconciles), never a bare `NodeReference`.** Different stores are the norm, not the exception: `demangledNodeReference(for:)` falls back to a mini store, and `InternedNodeReferenceCache` — every `MetadataReader`-derived tree — deduplicates equal trees onto one store per scope but still mints a distinct store per unique name, so cross-store mixing remains the default assumption. Everything that mixes those with image-store references is keyed structurally: the override/vtable lookups (`methodDescriptorLookup` / `vtableOffsetLookup` in `TypeDefinition.index` → `DefinitionBuilder`), `DefinitionBuilder`'s accessor grouping and merged-thunk dedup (`accessorsByNode` / `canonicalIndexByFunctionNode` / `canonicalIndexByAllocatorNode`), and every `visitedNodes` "already claimed this symbol" set (`ExtensionDefinition` / `ProtocolDefinition` / `TypeDefinition` / `OverrideSymbolMatcher` / `ProtocolConformanceDumper`). Under bare keys these fail silently and differently: the override lookups drop the `override` keyword + vtable-offset comment (the Stage 5a regression fixed 2026-07-25), a subscript's getter and setter split into two buckets so the setter-only one is discarded, merged thunks emit their `func`/`init` twice, and a witness can be claimed twice. Bare-`NodeReference` keys are only safe for grouping/dedup WITHIN one `memberSymbols` batch (single hash-consed store, where structural equality coincides with index equality). **Transient demangling everywhere else (Stage 5c)**: `MetadataReader`, `RuntimeFieldLayoutBackend`, `TypedDumper`, `ClassHierarchyDumper`, `Symbol.demangledNode`, `SwiftLayout.ObjCClassIndex` and `SwiftDeclarationRendering.SpecializedMetadataNodeSubstitution` use `demangleAsNodeTransient` + `Node.createTransient`, so the global `NodeCache` no longer grows while browsing. Sources now carries zero cached `demangleAsNode(` call sites — do NOT reintroduce `demangleAsNode`/`Node.create` on unbounded inputs anywhere; a demangle whose tree is consumed and dropped must stay transient. The build sweep's `NodeStoreBuilder` pre-reserves its buffers via `reserveCapacity(expectedSymbolCount:)` (growing-only, interning-result-neutral) — keep that call adjacent to the builder's construction. ## Test Environment diff --git a/Documentations/Internal/NodeStoreMigrationPlan.md b/Documentations/Internal/NodeStoreMigrationPlan.md index 844c23ba..259e0259 100644 --- a/Documentations/Internal/NodeStoreMigrationPlan.md +++ b/Documentations/Internal/NodeStoreMigrationPlan.md @@ -273,3 +273,11 @@ RuntimeViewer 与 MachOSwiftSection 的内存主项在 `MachOSymbols/SymbolIndex **显式不改一条**：`ClassDumper.distributedFunctionNodes` 每个 actor 类算两遍且逐 thunk materialize。消除需要给一个 `Sendable` 值类型加可变引用缓存，而该路径只在使用 distributed actor 的二进制里执行（本项目日常面对的框架里为零）；查询侧拿的是 `Node`，集合改结构键反而要为每个方法 intern 一个 mini store。判断为不值得。 **验收**：`swift package clean` 后全量 **1273 tests / 244 suites 全绿**；对冻结基线 `main-27726bc` 的三源整文件快照对比**全部逐字节一致**（File 38/38、DyldCache 18/18、Image 6/6，共 62 份）。快照 harness 一律用 `-p` 选 cache 镜像、而 `-p` 恒为最高排名，故另对 iOS 27.0 beta 3 模拟器 cache 补跑 `-n {SwiftUI, SwiftUICore, SwiftData}`，三者与对应 `-p` 输出逐字节一致（9,131,212 / 8,191,268 / 271,114 字节）——这才是第 4 项真正的覆盖。（改完 `Storage` 字段后增量构建再次出现运行期 SIGSEGV，clean 重建后消失——与 Stage 3 记录的现象相同。） + +### 内存图驱动的驻留收口 — 容量预留 + 残余 cached demangle 清零（2026-08-08） + +RuntimeViewer 索引 Foundation + libswiftCore + AppKit + SwiftUI + SwiftUICore 五镜像后的 memory graph 计数：存活 `Node` 208,809 个、`NodeStore` 14,451 个。swift-demangling 侧会话（feature/node-store 分支）在本仓库定位来源后转来三项计划，本批落地前两项： + +1. **主 sweep 容量预留**：`SymbolIndexStore` 的 `NodeStoreBuilder` 构造后紧跟 `builder.reserveCapacity(expectedSymbolCount: totalSymbolCount)`（上游提案 0009 的 API，按语料标定的每符号系数一次性预留三块缓冲与 intern 槽表）。上游实测：每镜像构建期 ≥1MiB 的 realloc 拷贝 12 → 4（余 4 次即预留本身），冷启动 footprint 尖峰减半。预留语义为 growing-only 且不改变 interning 结果，输出零变化。 +2. **最后两处 cached `demangleAsNode` 转 transient**：`SwiftLayout.ObjCClassIndex`（runtime name → 限定名字符串，树即弃）与 `SwiftDeclarationRendering.SpecializedMetadataNodeSubstitution`（metatype 名 → 渲染即弃）此前把整棵树永久 intern 进全局 `NodeCache`（不淘汰），是存活 `Node` 的主要来源之一。两处改 `demangleAsNodeTransient`（`@_spi(Internals)` import 跟进），Sources 下 cached `demangleAsNode(` 全库清零——Stage 5c 的收口补全。 +3. **小 store 流水线（`InternedNodeReferenceCache` / `TypeDefinition` 字段树批量 store / `lateDemangledNode` mini store）显式不动**：三处是对「builder 一次性 freeze、冻结前拿不到可读引用」这一上游真实缺口的正确规避，也是 14,451 个 store 的主要来源。上游已起草提案 0010（swift-demangling `Evolutions/0010-appendable-shared-node-store.md`，Draft）：`SharedNodeStore` 长生命周期、线程安全、intern 即发放稳定 `NodeReference`、无 freeze 屏障，落地后三条流水线汇入每镜像共享 store（预期 store 数 14,451 → 约 6，`TypeName` 相等比较获得同 store index 快路径）。0010 落地前不重构。 diff --git a/Documentations/Internal/ProjectEvolutionLog.md b/Documentations/Internal/ProjectEvolutionLog.md index 6c113f9a..be2555b7 100644 --- a/Documentations/Internal/ProjectEvolutionLog.md +++ b/Documentations/Internal/ProjectEvolutionLog.md @@ -738,6 +738,19 @@ --- +## 32. 内存图驱动的 NodeStore 驻留收口：容量预留 + 残余 cached demangle 清零 + +- **时间段**：2026-08-08。 +- **动机**：RuntimeViewer 索引五个系统镜像（Foundation + libswiftCore + AppKit + SwiftUI + SwiftUICore）后的 memory graph 显示存活 `Node` 208,809 个、`NodeStore` 14,451 个；swift-demangling 侧会话定位来源后转来三项计划，本批落地其中两项，第三项显式等待上游。 +- **落地**： + - `SymbolIndexStore` 主 sweep 的 `NodeStoreBuilder` 构造后一行 `reserveCapacity(expectedSymbolCount: totalSymbolCount)`（上游提案 0009 API），消掉构建期缓冲增长拷贝与冷启动 footprint 尖峰（上游实测减半）；预留 growing-only 且不改变 interning 结果。 + - 最后两处带全局缓存的 `demangleAsNode` 转 `demangleAsNodeTransient`：`SwiftLayout.ObjCClassIndex`（取限定名字符串即弃树）与 `SwiftDeclarationRendering.SpecializedMetadataNodeSubstitution`（渲染即弃）。至此 Sources 下 cached `demangleAsNode(` 清零，Stage 5c 收口补全。 +- **关键决策**：三条「每树/每类型/每晚到名字铸小 store」的流水线（`InternedNodeReferenceCache`、`TypeDefinition` 字段树批量 store、`lateDemangledNode`）**不动**——它们是对上游「builder 一次性 freeze」缺口的正确规避，等 swift-demangling 提案 0010（`SharedNodeStore`，Draft）落地后统一汇入每镜像共享 store。 +- **文档**：[NodeStoreMigrationPlan.md](NodeStoreMigrationPlan.md)「内存图驱动的驻留收口（2026-08-08）」一节；AGENTS.md Stage 5c 站点清单同步。 +- **对应版本**：`0.14.1` 之后、下一次 bump 之前（`feature/node-store-migration` 分支）。 + +--- + ## 维护约定 1. **每个非平凡批次结束时必须在本文追加/更新一节**（新工作弧新增一节；延续既有弧则在该节 diff --git a/Sources/MachOSymbols/SymbolIndexStore.swift b/Sources/MachOSymbols/SymbolIndexStore.swift index f4447be3..a9f76ea5 100644 --- a/Sources/MachOSymbols/SymbolIndexStore.swift +++ b/Sources/MachOSymbols/SymbolIndexStore.swift @@ -483,6 +483,7 @@ public final class SymbolIndexStore: SharedCache, @unc let totalSymbolCount = symbolTable.count var builder = NodeStoreBuilder() + builder.reserveCapacity(expectedSymbolCount: totalSymbolCount) var rootNodeIndexByTableRow = [NodeStore.NodeIndex?](repeating: nil, count: totalSymbolCount) var rowIndexes = RowIndexes() diff --git a/Sources/SwiftDeclarationRendering/SpecializedMetadataNodeSubstitution.swift b/Sources/SwiftDeclarationRendering/SpecializedMetadataNodeSubstitution.swift index a325b5dd..1b56a77b 100644 --- a/Sources/SwiftDeclarationRendering/SpecializedMetadataNodeSubstitution.swift +++ b/Sources/SwiftDeclarationRendering/SpecializedMetadataNodeSubstitution.swift @@ -1,4 +1,4 @@ -import Demangling +@_spi(Internals) import Demangling import MachOKit import MachOSwiftSection @@ -84,15 +84,17 @@ package enum SpecializedMetadataNodeSubstitution { return nil } - /// Shared wrapper around `_mangledTypeName` + `demangleAsNode` so the - /// SwiftStdlib-availability + nil-handling lives in exactly one spot for - /// both field-type and dumped-type substitution. + /// Shared wrapper around `_mangledTypeName` + `demangleAsNodeTransient` + /// so the SwiftStdlib-availability + nil-handling lives in exactly one + /// spot for both field-type and dumped-type substitution. Transient + /// demangle: callers render the node and drop it, so the tree must not + /// be interned into the global `NodeCache`. private static func demangledNode(forMetatype metatype: Any.Type) -> Node? { // `_mangledTypeName` is `SwiftStdlib 5.3` — translates to macOS 11 / // iOS 14 / tvOS 14 / watchOS 7. Fall back to nil on older runtimes // so callers stay on the unbound representation. guard #available(macOS 11, iOS 14, tvOS 14, watchOS 7, *) else { return nil } guard let resolvedMangledString = _mangledTypeName(metatype) else { return nil } - return try? demangleAsNode(resolvedMangledString, isType: true) + return try? demangleAsNodeTransient(resolvedMangledString, isType: true) } } diff --git a/Sources/SwiftLayout/ObjCClassIndex.swift b/Sources/SwiftLayout/ObjCClassIndex.swift index bf8cfb30..057bf942 100644 --- a/Sources/SwiftLayout/ObjCClassIndex.swift +++ b/Sources/SwiftLayout/ObjCClassIndex.swift @@ -1,6 +1,6 @@ import MachOKit @_spi(Core) import MachOObjCSection -import Demangling +@_spi(Internals) import Demangling /// Builds a per-image index from an Objective-C class's bare name to the start /// layout a Swift subclass inherits from it: the class's `instanceSize` (where a @@ -114,11 +114,13 @@ enum ObjCClassIndex { /// demangler. private static func swiftClassQualifiedName(fromRuntimeName runtimeName: String) -> String? { guard runtimeName.hasPrefix("_Tt") || runtimeName.hasPrefix("$s") else { return nil } - // `demangleAsNode` wraps the result in `.global`; the qualified-name + // The demangler wraps the result in `.global`; the qualified-name // builder wants the bare nominal class node (the same shape // `MetadataReader.demangleContext` produces on the descriptor side). + // Transient demangle: only the qualified-name string survives this + // call, so the tree must not be interned into the global `NodeCache`. guard - let node = try? demangleAsNode(runtimeName), + let node = try? demangleAsNodeTransient(runtimeName), let classNode = node.first(of: .class) else { return nil } return NodeTypeNaming.nominalQualifiedName(of: classNode) From 0c10a527205aaefd30c71e004f79d5e8ae64ff60 Mon Sep 17 00:00:00 2001 From: Mx-Iris Date: Sat, 8 Aug 2026 16:42:14 +0800 Subject: [PATCH 39/77] docs: accept the SharedNodeStore migration design (upstream evolution 0010) Design doc for folding the three mini-store pipelines (InternedNodeReferenceCache's structural-hash bucket layer, TypeDefinition's per-type field-tree store, lateDemangledNode's store-per-name) into per-image SharedNodeStore instances, now that upstream 0010 removed the freeze barrier. Key calls recorded in the doc: keep the InternedNodeReferenceCache shell (image/process scoping + SharedCache eviction stay our domain, 31 call sites unaffected), lateDemangledNode self-holds its store (eviction lifecycle decoupling) and keeps the name -> nil-verdict cache, Name types keep structural Hashable (cross-store mixing remains the norm), no capacity reservation for the new stores. Status Accepted; implementation gated on the upstream feature/node-store branch being pushed/released (local sibling symlinks resolve, but any environment without them silently falls back to the older remote release). --- .../Internal/SharedNodeStoreMigration.md | 41 +++++++++++++++++++ Documentations/README.md | 1 + 2 files changed, 42 insertions(+) create mode 100644 Documentations/Internal/SharedNodeStoreMigration.md diff --git a/Documentations/Internal/SharedNodeStoreMigration.md b/Documentations/Internal/SharedNodeStoreMigration.md new file mode 100644 index 00000000..2085d1c4 --- /dev/null +++ b/Documentations/Internal/SharedNodeStoreMigration.md @@ -0,0 +1,41 @@ +# SharedNodeStore 迁移：三条小 store 流水线汇入每镜像共享 store + +> **状态：Accepted（2026-08-08 批准）**——实现等待落地前置条件（上游 `feature/node-store` 分支 push/发版）满足后开始；落地后在文末补「与方案的差异」。 + +## 一句话 + +上游 swift-demangling 提案 0010 落地了 `SharedNodeStore`（长生命周期、线程安全、intern 即发放永久有效 `NodeReference`、无 freeze 屏障），本仓库三条「每树 / 每类型 / 每晚到名字铸一个小 `NodeStore`」的流水线因此可以全部汇入每镜像一个共享 store——RuntimeViewer 五镜像实测的 14,451 个 `NodeStore` 实例预期降到个位数。 + +## 改动位置（先看这里） + +| 位置 | 改什么 | +|---|---| +| `Sources/MachOSymbols/InternedNodeReferenceCache.swift` | `Storage` 的结构哈希桶 `[Int: [NodeReference]]` 整层退役，换为持一个 `SharedNodeStore`；`reference(interning:)` 的实现变一句 `store.intern(node)`。**类本身保留**——`SharedNodeStore` 不认识 Mach-O 镜像，per-image / per-process 两个作用域的键控与 `SharedCache` 驱逐接线是它继续存在的理由。31 个调用点（7 个文件）的公开 API 不变，零波及。 | +| `Sources/SwiftDeclaration/Components/Definitions/TypeDefinition.swift:164–195` | `fieldNodeStoreBuilder` + `freeze()` 的每类型批量 store 删除，字段树改走 `InternedNodeReferenceCache.shared.reference(interning:in:)` 汇入镜像 store（顺带获得跨类型的结构去重，今天每类型一个 store 是拿不到的）。 | +| `Sources/MachOSymbols/SymbolIndexStore.swift:256–275` | `lateDemangledNode(forName:)` 的 builder-per-name 删除，`Storage` 自持一个 late-names 专用 `SharedNodeStore`，demangle 改 `lateStore.demangle(name)`。名字 → 裁决字典**保留**（拒绝的 `nil` 裁决与成功 memo 都还需要，`SharedNodeStore` 不按名字缓存拒绝）；insert-if-absent 舞蹈保留但竞态后果变良性——双方 demangle 各自 intern 到同一 store，结构去重保证拿到同一引用，「loser 弃店」一段整个删掉。 | + +## 明确不动的部分 + +- **主 sweep 的 `NodeStoreBuilder` 路径**（`SymbolIndexStore.buildStorageSweep`）：一次性构建 + freeze 是它的正确形态，上游验收明确该路径不受影响。`reserveCapacity(expectedSymbolCount:)` 保持。 +- **`Name` 类型的结构语义 `Hashable` 与全部 `StructuralNodeReferenceKey` 键控容器**：跨 store 混用仍是常态（主 symbol store 的引用 vs 共享 store 的引用；驱逐重建后新旧 store 并存）。上游提到「单镜像 scope 内可退回固有实现」是微优化且引入正确性前提，不做；同 store 快路径在 `structurallyEquals` 内部自动生效，收益不需要任何改动就到手。 +- **新 store 不做容量预留**：0009 的系数按符号语料标定，上游明确警告名字树负载的文本维度可能低估；这些 store 本性是增量增长，退化上界有界（退休链 ≤1×）。后续如需调优用 `capacityUtilization` 校准。 + +## 关键设计取舍 + +- **`lateDemangledNode` 的 store 自持而非共用镜像 cache store**：`InternedNodeReferenceCache` 是 `SharedCache`，会被内存压力驱逐重建；`SymbolIndexStore.Storage` 不随之。若共用，驱逐后 cache 换了新 store 而 `Storage` 还引着旧的，两 store 并存——无正确性问题但混乱。自持让 store 生命周期与 `Storage` 严格一致（`removeSubIndexer` 一起释放），代价是每镜像 2 个 store 而不是 1 个，可忽略。 +- **驱逐后的内存回收语义与今天一致**：`SharedNodeStore` 被 scope 释放后，存活的外部引用继续保活底层存储（intern 停止、读取不坏）——mini store 今天就是这个语义，无回归；真正回收仍以「外部引用也放掉」为条件。 + +## 验收计划 + +1. 全量 `swift test --skip IntegrationTests` 全绿；fixture interface 快照逐字节一致（输出零变化是硬约束）。 +2. `InternedNodeReferenceCache` 相关测试口径更新：驻留计量从「mini store 730 → 471」变为「store 实例数 → 每 scope 1」。 +3. `SymbolIndexStoreFixtureTests` 的 late-path 三条性质测试（拒绝缓存、表内名不进 late、并发共店）语义保持，断言从 one-store-per-name 改 one-shared-store。 +4. RuntimeViewer 侧复测五镜像 memory graph（swift-demangling 会话约定在本迁移落地后做实景验证：14,451 store 实例的下降只有小 store 真正退役后才可观测）。 + +## 落地前置条件 + +- swift-demangling `feature/node-store` 分支的 0010 提交（`9997830`→`bb1f81c`）目前**只在对方本地 worktree，尚未 push**。本机经 `.claude/worktrees/` 兄弟符号链接可解析，但任何无 sibling 的环境会静默回落远端旧版（`SharedNodeStore` 不存在，编译失败还算好的；更险的是快照 A/B 失真——见 AGENTS.md 环境漂移检查第 2 条）。**上游 push（及后续版本发布）先行，本迁移再落。** + +## 上游 API 摘要（详见 swift-demangling `Documentations/NodeStoreArena.md` 共享 store 一节） + +`SharedNodeStore`: `intern(_ tree: Node) -> NodeReference`（锁内串行、结构去重、引用永久有效）、`demangle(_:isType:symbolicReferenceResolver:) throws -> NodeReference`（解析在锁外）、`reserveCapacity(expectedSymbolCount:)`、观测属性 `capacityUtilization` / `nodeCount` / `storageByteCount`。一个实例只有一个 `NodeStore` 身份——同 scope 引用的固有 `==`/`hash` 即结构相等。上游验收：同形状负载 store 实例 14,000 → 1、冷启动 footprint 8.0 → 2.9 MiB、malloc 减半、耗时 −43%；439,522 符号打印对拍零差异；TSan 全绿。 diff --git a/Documentations/README.md b/Documentations/README.md index 8a9d8927..cf58bffd 100644 --- a/Documentations/README.md +++ b/Documentations/README.md @@ -74,6 +74,7 @@ required by `Version.swift`'s bump contract). | [ReadingContextAbstraction.md](Internal/ReadingContextAbstraction.md) | The `ReadingContext` reading-abstraction design. | | [ClassMemberKeywordRecovery.md](Internal/ClassMemberKeywordRecovery.md) | `class` / `static` 成员关键字的还原：mangling 层面两者不可区分，判据是「类型级成员有 vtable method descriptor ⇒ 源码是 `class`」（`static` 隐式 final、不进 vtable）；模型侧 `isClassMember` 计算属性 + 三个 node printer 接线 + dump vtable 段落关键字，顺带消灭非法的 `override static` 输出；`final class func` 等四类 ABI 上与 `static` 完全一致，保守输出语义等价的 `static`。 | | [NodeStoreMigrationPlan.md](Internal/NodeStoreMigrationPlan.md) | `SymbolIndexStore` 迁移到 `NodeStore` arena 存储的分期计划与实施记录（Stage 0–5）：12B/节点扁平缓冲、cache-free build sweep、Symbol 表压缩、声明层换持 `NodeReference`、散点 transient demangling；含 Stage 5a 的 override/vtable 回归修复（裸 `[NodeReference: …]` 的 store-identity 陷阱 → `StructuralNodeReferenceKey`）与三源快照验收。 | +| [SharedNodeStoreMigration.md](Internal/SharedNodeStoreMigration.md) | **Accepted（待上游 push 后动工）**：三条小 store 流水线（`InternedNodeReferenceCache` / `TypeDefinition` 字段树批量 store / `lateDemangledNode`）汇入上游 0010 `SharedNodeStore` 的迁移设计——改动位置、明确不动的部分、验收计划与落地前置条件。 | | [DeclarationModelMemoryFootprint.md](Internal/DeclarationModelMemoryFootprint.md) | NodeStore 迁移后的声明模型内存足迹量测：`TypeDefinition` 1272 字节的逐属性构成（两份 `TypeContextWrapper` 占 74%）、`TypeContextWrapper` 按最大 case `Class` 定尺的原因、`parentContext` 是被当成永久字段的临时值、mini-store 增殖与 `MetadataReaderCache` 仍持 `Node` 树；含四项可回收估算（合计 ~8–10%）与「当前不建议实施、应先剖析其余 90%」的结论。 | | [NodeStoreMigrationOpenIssues.md](Internal/NodeStoreMigrationOpenIssues.md) | `feature/node-store-migration` 的**已确认问题清单**（2026-07-28 两轮审查 + 复核）。第一节是前一轮修复自身的两处缺口（Catalyst 降级只覆盖 framework 形态导致 plain dylib 仍平局、`appendRowIfAbsent` 线性扫描），已于 2026-07-29 闭环并保留成因；第二节起为仍然打开或已裁决的条目：`structuralHash` 每文本节点分配 `String` 与 `ABIKey` 每 key materialize 两条**已按上游设计终审关闭**（0.5.1 保持现状且上游说明不改，见 ReviewAdjudications A1/A2）、`memberSymbols` 改为线性 + 全树比对（**已证实量级可忽略，不是回归**）；`lateDemangledNode` 持锁 demangle 与 `ProtocolConformanceDumper` 的 materialize 分支两条**已于 2026-08-03 闭环**（连同失败名裁决缓存、dump 路径引用化）、build sweep 串行且每符号跨线程往返（上游）、`ABIKey` 每 key materialize，以及四条代码卫生项；两个公开查询 API 的字典键一条**已于 2026-08-03 裁决为不修**（类型级 SPI），rebase 前置事项一条前提已过期但压着两条仍成立的注意事项。逐条注明成因、影响面与「该在哪里修」。与 `Reviews/` 下两份审查记录冲突时以后者为准。 | | [SystemFrameworkRenderingVerification.md](Internal/SystemFrameworkRenderingVerification.md) | 系统框架渲染 A/B 验证流程（大重构必跑）：两个检出对同一批真实输入（归档 dyld cache → 无则当前系统 cache、模拟器 runtime → 无则现有 runtime、当前系统 MachOImage）逐字节比对 dump+interface；入口脚本 `Scripts/run-rendering-ab-verification.py`，含回退规则、踩坑清单（`-p` 消歧、fat 二进制 `-a`、同 boot 会话、scratch 隔离）与 2026-08-03 基线运行记录。 | From 34276bd4079b4b021434d8d9241c5be5d51fd5bf Mon Sep 17 00:00:00 2001 From: Mx-Iris Date: Sat, 8 Aug 2026 16:55:10 +0800 Subject: [PATCH 40/77] perf(MachOSymbols,SwiftDeclaration): fold the mini-store pipelines into per-scope SharedNodeStore arenas MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Implements the accepted SharedNodeStoreMigration design over upstream evolution 0010 (SharedNodeStore: appendable, thread-safe, references valid from intern time — no freeze barrier). - InternedNodeReferenceCache: the structural-hash bucket layer and its get-or-mint arbitration retire; Storage holds one SharedNodeStore per scope (the store's interning tables are the persistent dedup). The class survives as the part SharedNodeStore deliberately does not know: Mach-O scope keying (per-image / per-process) and SharedCache eviction. All 31 call sites unaffected; the userless cachedReferenceCountForTesting property is deleted. - TypeDefinition.index(in:): the per-type builder -> freeze -> map two-phase collapses into interning each field tree straight into the image store via the cache, widening dedup from one type to the whole image. - SymbolIndexStore.Storage.lateDemangledNode: the store-per-name builders retire for a self-held per-image SharedNodeStore side store (deliberately not shared with the evictable cache store); racing missers now converge on the same reference via structural dedup, so the loser-discards-store dance is gone. The name -> nil-verdict rejection cache stays (SharedNodeStore throws and caches nothing). Behavior tests pinned the semantics across the swap: all five InternedNodeReferenceCacheTests and all three late-path property tests pass unchanged. Full suite 1337 tests green (same count as before), clean rebuild warning-free. AGENTS.md's symbol-indexing section and the design doc's landing record updated in the same batch; the upstream-push landing precondition was explicitly waived by the user (sibling symlinks resolve locally; environments without them cannot build until upstream pushes). --- AGENTS.md | 2 +- .../Internal/NodeStoreMigrationPlan.md | 2 +- .../Internal/ProjectEvolutionLog.md | 1 + .../Internal/SharedNodeStoreMigration.md | 13 ++- Documentations/README.md | 2 +- .../InternedNodeReferenceCache.swift | 97 +++++++------------ Sources/MachOSymbols/SymbolIndexStore.swift | 47 ++++----- .../Definitions/TypeDefinition.swift | 19 ++-- 8 files changed, 81 insertions(+), 102 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index a49b35cf..01d2200a 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -208,7 +208,7 @@ let node = try demangleAsNode("$sSiD") // Returns Node tree let string = node.print(using: .default) // "Swift.Int" ``` -**Symbol indexing (NodeStore-backed)**: `SymbolIndexStore.Storage` no longer retains `Node` class trees. The build sweep demangles each symbol cache-free onto a transient tree (`@_spi(Internals) demangleAsNodeTransient`), classifies on that tree, and interns it into a per-image `NodeStoreBuilder`. Nothing touches the global `NodeCache`, and dropping a `Storage` releases the whole per-image footprint. Storage layout (Stage 3): one flat `symbolTable: [Symbol]` row per unique symbol name (canonical cache-adjusted offset; raw and adjusted offset keys share one row), a parallel `rootNodeIndexByTableRow` array, and every classification index stores 4-byte `UInt32` table rows (member/opaque keys are 4-byte `NodeStore.NodeIndex`); indexes accumulate in final row form during the sweep, so `freeze()` is followed by a plain move, not a conversion pass. `Symbol` is 32 bytes (no `nlist` existential — the external-undefined bit is extracted at collection time into `isExternal`), and `DemangledSymbol` is a 32-byte value (shared-table reference + row + `NodeReference`) whose `symbol` is computed; both are asserted by `compactValueLayouts`. **A `DemangledSymbol` stored into the declaration model must be detached first** (`detachedFromSharedTable()`): the shared `[Symbol]` buffer is the right trade for the hundreds of thousands of values a query vends and drops, but `Array` is a reference to its buffer, so a single stored survivor pins the whole table and every mangled name in it — defeating `removeSubIndexer(_:)`, whose reason for existing is reclaiming per-image memory. Measured on SwiftUI (iOS 18.5): 9,872 stored values referenced 9,506 distinct rows, 5.1% of a 185,988-row table, so detaching trades ~0.6 MB of small allocations for ~19.9 MB of retention. The six storing sites are `DefinitionBuilder`'s four construction points (`Accessor` for variables/subscripts, `FunctionDefinition` for allocators/functions) and `TypeDefinition`'s `deallocatorSymbol` / `destructorSymbol` assignments; `SymbolTableRetentionTests` fails the moment a new one forgets. Do NOT detach on the query path. Query APIs construct `DemangledSymbol`/`NodeReference` values at the exit boundary; APIs taking an externally demangled node (`memberSymbols(of:for:node:)` with `Node` or `NodeReference`, `opaqueTypeDescriptorSymbol(for:)`) match keys via `NodeReference.structurallyEquals(_:)` (same-store O(1), cross-store structural walk) — `opaqueTypeDescriptorSymbol(for:)` buckets its entries by `DemanglingNode.identifier` first (`opaqueTypeDescriptorEntriesByMemberIdentifier`), because a full scan there is O(descriptors × printed `some` returns). `demangledNodeReference(for:)` matches on the symbol **name alone** — a demangled tree is a pure function of the name and the flat table holds one row per unique name, so comparing the row's canonical offset against the queried symbol's offset could only reject valid hits (it once rejected every symbol of a dyld-cache image, since `symbols(for:in:)` stamps each vended `Symbol` with the offset it was queried by). A name the sweep covered is answered from the table verdict alone — including the *rejection* verdict: a `nil` root means the demangler already refused the name once, and the same demangler backs the late path, so the query returns `nil` without re-demangling. Names genuinely outside the build sweep fall back to name-keyed mini stores (`lateDemangledNode(forName:)` / `MetadataReader.demangleSymbolReference(for:in:)`): the demangle runs *outside* the cache lock (never hold `os_unfair_lock` across a potentially blocking large-stack hop) and the lock arbitrates insert-if-absent — a racing loser discards its store and returns the winner's reference, preserving one-store-per-name — while rejections are cached as `nil` verdicts and never retried (`rejectedLateNameCachesItsFailure` / `tableCoveredNameNeverEntersLateCache` / `concurrentLateQueriesShareOneStore` pin all three properties). Struct-layout changes in `MachOSymbols` need a `swift package clean` rebuild — SwiftPM incremental builds have been observed linking stale downstream objects (runtime SIGSEGV in `outlined destroy`). See `Documentations/Internal/NodeStoreMigrationPlan.md`. **Declaration values hold `NodeReference` (Stage 5a)**: `DefinitionName.node` (`TypeName`/`ProtocolName`/`ExtensionName`), `Variable/Function/SubscriptDefinition.node`, `FieldDefinition.typeNode` and `ExtensionDefinition.genericSignature` are `NodeReference`, not `Node`. Member definitions reference the per-image symbol store directly (a live definition keeps that store alive — the intended per-image recycling model); metadata-derived name trees (`MetadataReader` output feeding `TypeName`/`ProtocolName`/`ExtensionName`/generic signatures) go through `InternedNodeReferenceCache` (`Sources/MachOSymbols/`) — a structural-dedup layer over `NodeReference(interning:)` with an image-keyed scope (evicted with the image; `SwiftDeclarationIndexer`'s cleanup removes it alongside the symbol store) and a process-keyed scope for the in-process helpers with no Mach-O handle, so repeated names (conformance protocol names, shared parents, extension targets) share one store and name equality gets the `store ===` fast path (measured on the fixture: 730 retained mini stores → 471, matching the 472 structurally unique trees). Do NOT call bare `NodeReference(interning:)` on a batch path — route through the cache. `TypeDefinition.index(in:)` still batches all field type trees of one type into a single shared store. The `Name` types customize `Hashable` to STRUCTURAL semantics (`structurallyEquals` + `structuralHash`) because `NodeReference`'s intrinsic `Hashable` is store-identity based and would split equal keys minted into different stores; they are deliberately NOT `Codable` — upstream dropped `Node: Codable` (a mangled symbol already is the tree's serialized form: smaller, ABI-stable across toolchains, and sharing-preserving through the round trip), and nothing persists these values. A consumer that ever needs to should persist `mangleAsString(node)` and read it back with `demangleAsNode(_:)` rather than reintroducing a node encoding. **Any `Dictionary`/`Set` keyed on a `NodeReference` whose keys and lookups can come from DIFFERENT stores must key on `StructuralNodeReferenceKey` (`Sources/MachOSymbols/`, next to the mini stores it reconciles), never a bare `NodeReference`.** Different stores are the norm, not the exception: `demangledNodeReference(for:)` falls back to a mini store, and `InternedNodeReferenceCache` — every `MetadataReader`-derived tree — deduplicates equal trees onto one store per scope but still mints a distinct store per unique name, so cross-store mixing remains the default assumption. Everything that mixes those with image-store references is keyed structurally: the override/vtable lookups (`methodDescriptorLookup` / `vtableOffsetLookup` in `TypeDefinition.index` → `DefinitionBuilder`), `DefinitionBuilder`'s accessor grouping and merged-thunk dedup (`accessorsByNode` / `canonicalIndexByFunctionNode` / `canonicalIndexByAllocatorNode`), and every `visitedNodes` "already claimed this symbol" set (`ExtensionDefinition` / `ProtocolDefinition` / `TypeDefinition` / `OverrideSymbolMatcher` / `ProtocolConformanceDumper`). Under bare keys these fail silently and differently: the override lookups drop the `override` keyword + vtable-offset comment (the Stage 5a regression fixed 2026-07-25), a subscript's getter and setter split into two buckets so the setter-only one is discarded, merged thunks emit their `func`/`init` twice, and a witness can be claimed twice. Bare-`NodeReference` keys are only safe for grouping/dedup WITHIN one `memberSymbols` batch (single hash-consed store, where structural equality coincides with index equality). **Transient demangling everywhere else (Stage 5c)**: `MetadataReader`, `RuntimeFieldLayoutBackend`, `TypedDumper`, `ClassHierarchyDumper`, `Symbol.demangledNode`, `SwiftLayout.ObjCClassIndex` and `SwiftDeclarationRendering.SpecializedMetadataNodeSubstitution` use `demangleAsNodeTransient` + `Node.createTransient`, so the global `NodeCache` no longer grows while browsing. Sources now carries zero cached `demangleAsNode(` call sites — do NOT reintroduce `demangleAsNode`/`Node.create` on unbounded inputs anywhere; a demangle whose tree is consumed and dropped must stay transient. The build sweep's `NodeStoreBuilder` pre-reserves its buffers via `reserveCapacity(expectedSymbolCount:)` (growing-only, interning-result-neutral) — keep that call adjacent to the builder's construction. +**Symbol indexing (NodeStore-backed)**: `SymbolIndexStore.Storage` no longer retains `Node` class trees. The build sweep demangles each symbol cache-free onto a transient tree (`@_spi(Internals) demangleAsNodeTransient`), classifies on that tree, and interns it into a per-image `NodeStoreBuilder`. Nothing touches the global `NodeCache`, and dropping a `Storage` releases the whole per-image footprint. Storage layout (Stage 3): one flat `symbolTable: [Symbol]` row per unique symbol name (canonical cache-adjusted offset; raw and adjusted offset keys share one row), a parallel `rootNodeIndexByTableRow` array, and every classification index stores 4-byte `UInt32` table rows (member/opaque keys are 4-byte `NodeStore.NodeIndex`); indexes accumulate in final row form during the sweep, so `freeze()` is followed by a plain move, not a conversion pass. `Symbol` is 32 bytes (no `nlist` existential — the external-undefined bit is extracted at collection time into `isExternal`), and `DemangledSymbol` is a 32-byte value (shared-table reference + row + `NodeReference`) whose `symbol` is computed; both are asserted by `compactValueLayouts`. **A `DemangledSymbol` stored into the declaration model must be detached first** (`detachedFromSharedTable()`): the shared `[Symbol]` buffer is the right trade for the hundreds of thousands of values a query vends and drops, but `Array` is a reference to its buffer, so a single stored survivor pins the whole table and every mangled name in it — defeating `removeSubIndexer(_:)`, whose reason for existing is reclaiming per-image memory. Measured on SwiftUI (iOS 18.5): 9,872 stored values referenced 9,506 distinct rows, 5.1% of a 185,988-row table, so detaching trades ~0.6 MB of small allocations for ~19.9 MB of retention. The six storing sites are `DefinitionBuilder`'s four construction points (`Accessor` for variables/subscripts, `FunctionDefinition` for allocators/functions) and `TypeDefinition`'s `deallocatorSymbol` / `destructorSymbol` assignments; `SymbolTableRetentionTests` fails the moment a new one forgets. Do NOT detach on the query path. Query APIs construct `DemangledSymbol`/`NodeReference` values at the exit boundary; APIs taking an externally demangled node (`memberSymbols(of:for:node:)` with `Node` or `NodeReference`, `opaqueTypeDescriptorSymbol(for:)`) match keys via `NodeReference.structurallyEquals(_:)` (same-store O(1), cross-store structural walk) — `opaqueTypeDescriptorSymbol(for:)` buckets its entries by `DemanglingNode.identifier` first (`opaqueTypeDescriptorEntriesByMemberIdentifier`), because a full scan there is O(descriptors × printed `some` returns). `demangledNodeReference(for:)` matches on the symbol **name alone** — a demangled tree is a pure function of the name and the flat table holds one row per unique name, so comparing the row's canonical offset against the queried symbol's offset could only reject valid hits (it once rejected every symbol of a dyld-cache image, since `symbols(for:in:)` stamps each vended `Symbol` with the offset it was queried by). A name the sweep covered is answered from the table verdict alone — including the *rejection* verdict: a `nil` root means the demangler already refused the name once, and the same demangler backs the late path, so the query returns `nil` without re-demangling. Names genuinely outside the build sweep fall back to the per-image appendable side store (`lateDemangledNode(forName:)` — a `SharedNodeStore` self-held by `Storage`, deliberately not shared with `InternedNodeReferenceCache`'s evictable image store; `MetadataReader.demangleSymbolReference(for:in:)` forwards here): the demangle runs *outside* the cache lock (never hold `os_unfair_lock` across a potentially blocking large-stack hop), racing missers intern into the one store whose structural dedup hands them the same reference, and the name → verdict map keeps one canonical answer — while rejections are cached as `nil` verdicts and never retried (`rejectedLateNameCachesItsFailure` / `tableCoveredNameNeverEntersLateCache` / `concurrentLateQueriesShareOneStore` pin all three properties). Struct-layout changes in `MachOSymbols` need a `swift package clean` rebuild — SwiftPM incremental builds have been observed linking stale downstream objects (runtime SIGSEGV in `outlined destroy`). See `Documentations/Internal/NodeStoreMigrationPlan.md`. **Declaration values hold `NodeReference` (Stage 5a)**: `DefinitionName.node` (`TypeName`/`ProtocolName`/`ExtensionName`), `Variable/Function/SubscriptDefinition.node`, `FieldDefinition.typeNode` and `ExtensionDefinition.genericSignature` are `NodeReference`, not `Node`. Member definitions reference the per-image symbol store directly (a live definition keeps that store alive — the intended per-image recycling model); metadata-derived name trees (`MetadataReader` output feeding `TypeName`/`ProtocolName`/`ExtensionName`/generic signatures) go through `InternedNodeReferenceCache` (`Sources/MachOSymbols/`) — one `SharedNodeStore` per scope (upstream evolution 0010; the store's interning tables are the dedup, the cache keeps only what `SharedNodeStore` deliberately does not know: Mach-O scope keying and eviction): an image-keyed scope (evicted with the image; `SwiftDeclarationIndexer`'s cleanup removes it alongside the symbol store) and a process-keyed scope for the in-process helpers with no Mach-O handle, so every tree of a scope shares one store and name equality gets the `store ===` fast path. Do NOT call bare `NodeReference(interning:)` on a batch path — route through the cache. `TypeDefinition.index(in:)`'s field type trees intern into the same image store (cross-type dedup; the per-type builder+freeze store this replaced could only deduplicate within one type). See `Documentations/Internal/SharedNodeStoreMigration.md`. The `Name` types customize `Hashable` to STRUCTURAL semantics (`structurallyEquals` + `structuralHash`) because `NodeReference`'s intrinsic `Hashable` is store-identity based and would split equal keys minted into different stores; they are deliberately NOT `Codable` — upstream dropped `Node: Codable` (a mangled symbol already is the tree's serialized form: smaller, ABI-stable across toolchains, and sharing-preserving through the round trip), and nothing persists these values. A consumer that ever needs to should persist `mangleAsString(node)` and read it back with `demangleAsNode(_:)` rather than reintroducing a node encoding. **Any `Dictionary`/`Set` keyed on a `NodeReference` whose keys and lookups can come from DIFFERENT stores must key on `StructuralNodeReferenceKey` (`Sources/MachOSymbols/`, next to the stores it reconciles), never a bare `NodeReference`.** Different stores are the norm, not the exception: the frozen main symbol store, the late-name side store, and the cache's per-scope stores coexist within one image (and memory-pressure eviction rebuilds a cache scope onto a fresh store while old references keep the old one alive), so cross-store mixing remains the default assumption. Everything that mixes those with image-store references is keyed structurally: the override/vtable lookups (`methodDescriptorLookup` / `vtableOffsetLookup` in `TypeDefinition.index` → `DefinitionBuilder`), `DefinitionBuilder`'s accessor grouping and merged-thunk dedup (`accessorsByNode` / `canonicalIndexByFunctionNode` / `canonicalIndexByAllocatorNode`), and every `visitedNodes` "already claimed this symbol" set (`ExtensionDefinition` / `ProtocolDefinition` / `TypeDefinition` / `OverrideSymbolMatcher` / `ProtocolConformanceDumper`). Under bare keys these fail silently and differently: the override lookups drop the `override` keyword + vtable-offset comment (the Stage 5a regression fixed 2026-07-25), a subscript's getter and setter split into two buckets so the setter-only one is discarded, merged thunks emit their `func`/`init` twice, and a witness can be claimed twice. Bare-`NodeReference` keys are only safe for grouping/dedup WITHIN one `memberSymbols` batch (single hash-consed store, where structural equality coincides with index equality). **Transient demangling everywhere else (Stage 5c)**: `MetadataReader`, `RuntimeFieldLayoutBackend`, `TypedDumper`, `ClassHierarchyDumper`, `Symbol.demangledNode`, `SwiftLayout.ObjCClassIndex` and `SwiftDeclarationRendering.SpecializedMetadataNodeSubstitution` use `demangleAsNodeTransient` + `Node.createTransient`, so the global `NodeCache` no longer grows while browsing. Sources now carries zero cached `demangleAsNode(` call sites — do NOT reintroduce `demangleAsNode`/`Node.create` on unbounded inputs anywhere; a demangle whose tree is consumed and dropped must stay transient. The build sweep's `NodeStoreBuilder` pre-reserves its buffers via `reserveCapacity(expectedSymbolCount:)` (growing-only, interning-result-neutral) — keep that call adjacent to the builder's construction. ## Test Environment diff --git a/Documentations/Internal/NodeStoreMigrationPlan.md b/Documentations/Internal/NodeStoreMigrationPlan.md index 259e0259..f56b4e99 100644 --- a/Documentations/Internal/NodeStoreMigrationPlan.md +++ b/Documentations/Internal/NodeStoreMigrationPlan.md @@ -280,4 +280,4 @@ RuntimeViewer 索引 Foundation + libswiftCore + AppKit + SwiftUI + SwiftUICore 1. **主 sweep 容量预留**：`SymbolIndexStore` 的 `NodeStoreBuilder` 构造后紧跟 `builder.reserveCapacity(expectedSymbolCount: totalSymbolCount)`（上游提案 0009 的 API，按语料标定的每符号系数一次性预留三块缓冲与 intern 槽表）。上游实测：每镜像构建期 ≥1MiB 的 realloc 拷贝 12 → 4（余 4 次即预留本身），冷启动 footprint 尖峰减半。预留语义为 growing-only 且不改变 interning 结果，输出零变化。 2. **最后两处 cached `demangleAsNode` 转 transient**：`SwiftLayout.ObjCClassIndex`（runtime name → 限定名字符串，树即弃）与 `SwiftDeclarationRendering.SpecializedMetadataNodeSubstitution`（metatype 名 → 渲染即弃）此前把整棵树永久 intern 进全局 `NodeCache`（不淘汰），是存活 `Node` 的主要来源之一。两处改 `demangleAsNodeTransient`（`@_spi(Internals)` import 跟进），Sources 下 cached `demangleAsNode(` 全库清零——Stage 5c 的收口补全。 -3. **小 store 流水线（`InternedNodeReferenceCache` / `TypeDefinition` 字段树批量 store / `lateDemangledNode` mini store）显式不动**：三处是对「builder 一次性 freeze、冻结前拿不到可读引用」这一上游真实缺口的正确规避，也是 14,451 个 store 的主要来源。上游已起草提案 0010（swift-demangling `Evolutions/0010-appendable-shared-node-store.md`，Draft）：`SharedNodeStore` 长生命周期、线程安全、intern 即发放稳定 `NodeReference`、无 freeze 屏障，落地后三条流水线汇入每镜像共享 store（预期 store 数 14,451 → 约 6，`TypeName` 相等比较获得同 store index 快路径）。0010 落地前不重构。 +3. **小 store 流水线（`InternedNodeReferenceCache` / `TypeDefinition` 字段树批量 store / `lateDemangledNode` mini store）显式不动**：三处是对「builder 一次性 freeze、冻结前拿不到可读引用」这一上游真实缺口的正确规避，也是 14,451 个 store 的主要来源。上游已起草提案 0010（swift-demangling `Evolutions/0010-appendable-shared-node-store.md`，Draft）：`SharedNodeStore` 长生命周期、线程安全、intern 即发放稳定 `NodeReference`、无 freeze 屏障，落地后三条流水线汇入每镜像共享 store（预期 store 数 14,451 → 约 6，`TypeName` 相等比较获得同 store index 快路径）。0010 落地前不重构。**后记：0010 当日 Implemented，三条流水线的迁移随即批准并同日落地——设计、验证与差异见 [SharedNodeStoreMigration.md](SharedNodeStoreMigration.md)。** diff --git a/Documentations/Internal/ProjectEvolutionLog.md b/Documentations/Internal/ProjectEvolutionLog.md index be2555b7..0040988d 100644 --- a/Documentations/Internal/ProjectEvolutionLog.md +++ b/Documentations/Internal/ProjectEvolutionLog.md @@ -747,6 +747,7 @@ - 最后两处带全局缓存的 `demangleAsNode` 转 `demangleAsNodeTransient`：`SwiftLayout.ObjCClassIndex`（取限定名字符串即弃树）与 `SwiftDeclarationRendering.SpecializedMetadataNodeSubstitution`（渲染即弃）。至此 Sources 下 cached `demangleAsNode(` 清零，Stage 5c 收口补全。 - **关键决策**：三条「每树/每类型/每晚到名字铸小 store」的流水线（`InternedNodeReferenceCache`、`TypeDefinition` 字段树批量 store、`lateDemangledNode`）**不动**——它们是对上游「builder 一次性 freeze」缺口的正确规避，等 swift-demangling 提案 0010（`SharedNodeStore`，Draft）落地后统一汇入每镜像共享 store。 - **文档**：[NodeStoreMigrationPlan.md](NodeStoreMigrationPlan.md)「内存图驱动的驻留收口（2026-08-08）」一节；AGENTS.md Stage 5c 站点清单同步。 +- **补记（2026-08-08 同日，第三项落地）**：上游 0010（`SharedNodeStore`）当日 Implemented，本仓库迁移设计（[SharedNodeStoreMigration.md](SharedNodeStoreMigration.md)）经批准后同日实施：`InternedNodeReferenceCache` 退役哈希桶层、外壳换持每 scope 一个 `SharedNodeStore`（31 个调用点零波及）；`TypeDefinition.index` 字段树两阶段收敛为直接 intern 进镜像 store（去重范围从单类型扩到全镜像）；`lateDemangledNode` 换 `Storage` 自持的 side store、「loser 弃店」删除。全量 1337 tests 全绿且与迁移前同数，缓存与 late-path 的八条行为测试未改一行原样通过。用户裁决豁免「上游先 push」前置条件（无 sibling 环境在上游 push 前不可构建，知情接受）。RV 五镜像 memory graph 实景复测待上游会话执行。 - **对应版本**：`0.14.1` 之后、下一次 bump 之前（`feature/node-store-migration` 分支）。 --- diff --git a/Documentations/Internal/SharedNodeStoreMigration.md b/Documentations/Internal/SharedNodeStoreMigration.md index 2085d1c4..308e4c80 100644 --- a/Documentations/Internal/SharedNodeStoreMigration.md +++ b/Documentations/Internal/SharedNodeStoreMigration.md @@ -1,6 +1,6 @@ # SharedNodeStore 迁移：三条小 store 流水线汇入每镜像共享 store -> **状态：Accepted（2026-08-08 批准）**——实现等待落地前置条件（上游 `feature/node-store` 分支 push/发版）满足后开始；落地后在文末补「与方案的差异」。 +> **状态：Implemented（2026-08-08 批准并同日落地）**——落地记录与「与方案的差异」见文末。 ## 一句话 @@ -36,6 +36,17 @@ - swift-demangling `feature/node-store` 分支的 0010 提交（`9997830`→`bb1f81c`）目前**只在对方本地 worktree，尚未 push**。本机经 `.claude/worktrees/` 兄弟符号链接可解析，但任何无 sibling 的环境会静默回落远端旧版（`SharedNodeStore` 不存在，编译失败还算好的；更险的是快照 A/B 失真——见 AGENTS.md 环境漂移检查第 2 条）。**上游 push（及后续版本发布）先行，本迁移再落。** +## 落地记录（2026-08-08） + +三处均按上表实施：`InternedNodeReferenceCache.Storage` 换持一个 `SharedNodeStore`（哈希桶层与 get-or-mint 舞蹈整体删除，类文档注释改写为「作用域键控 + 驱逐」的新分工）；`TypeDefinition.index(in:)` 的字段树两阶段（builder → freeze → map）收敛为单阶段直接经缓存 intern 进镜像 store；`lateDemangledNode` 的 builder-per-name 换 `Storage` 自持的 `lateNameStore.demangle(name)`，「loser 弃店」段删除、名字 → 裁决字典保留。AGENTS.md「Symbol indexing」段的四处措辞同步（late 路径、缓存分工、字段树去重范围、跨 store 常态的论据）。 + +**验证**：干净重建零 warning；全量 `swift test --skip IntegrationTests` **1337 tests 全绿、0 失败**（与迁移前完全同数，含 interface 快照逐字节断言）；`InternedNodeReferenceCacheTests` 五条行为断言与 `SymbolIndexStoreFixtureTests` 三条 late-path 性质测试**原样通过、未改一行**——同 store、去重、驱逐重铸、并发单赢家这些行为在 `SharedNodeStore` 背书下语义保持。RuntimeViewer 五镜像 memory graph 实景复测（验收计划第 4 条）待 swift-demangling 侧会话按共识执行。 + +## 与方案的差异 + +1. **验收计划第 2 条落空（良性）**：预期要更新口径的驻留计数断言实际不存在——`cachedReferenceCountForTesting` 全库无使用者，已随哈希桶层一并删除；行为断言无需任何改动。 +2. **落地前置条件被用户裁决豁免（2026-08-08）**：未等上游 `feature/node-store` 分支 push，本地经 sibling 符号链接先行落地。在上游 push 前，本分支在无 sibling 的环境不可构建（`SharedNodeStore` 解析不到），该风险用户知情接受；上游 push 后自动消除。 + ## 上游 API 摘要（详见 swift-demangling `Documentations/NodeStoreArena.md` 共享 store 一节） `SharedNodeStore`: `intern(_ tree: Node) -> NodeReference`（锁内串行、结构去重、引用永久有效）、`demangle(_:isType:symbolicReferenceResolver:) throws -> NodeReference`（解析在锁外）、`reserveCapacity(expectedSymbolCount:)`、观测属性 `capacityUtilization` / `nodeCount` / `storageByteCount`。一个实例只有一个 `NodeStore` 身份——同 scope 引用的固有 `==`/`hash` 即结构相等。上游验收：同形状负载 store 实例 14,000 → 1、冷启动 footprint 8.0 → 2.9 MiB、malloc 减半、耗时 −43%；439,522 符号打印对拍零差异；TSan 全绿。 diff --git a/Documentations/README.md b/Documentations/README.md index cf58bffd..4ceb9a76 100644 --- a/Documentations/README.md +++ b/Documentations/README.md @@ -74,7 +74,7 @@ required by `Version.swift`'s bump contract). | [ReadingContextAbstraction.md](Internal/ReadingContextAbstraction.md) | The `ReadingContext` reading-abstraction design. | | [ClassMemberKeywordRecovery.md](Internal/ClassMemberKeywordRecovery.md) | `class` / `static` 成员关键字的还原：mangling 层面两者不可区分，判据是「类型级成员有 vtable method descriptor ⇒ 源码是 `class`」（`static` 隐式 final、不进 vtable）；模型侧 `isClassMember` 计算属性 + 三个 node printer 接线 + dump vtable 段落关键字，顺带消灭非法的 `override static` 输出；`final class func` 等四类 ABI 上与 `static` 完全一致，保守输出语义等价的 `static`。 | | [NodeStoreMigrationPlan.md](Internal/NodeStoreMigrationPlan.md) | `SymbolIndexStore` 迁移到 `NodeStore` arena 存储的分期计划与实施记录（Stage 0–5）：12B/节点扁平缓冲、cache-free build sweep、Symbol 表压缩、声明层换持 `NodeReference`、散点 transient demangling；含 Stage 5a 的 override/vtable 回归修复（裸 `[NodeReference: …]` 的 store-identity 陷阱 → `StructuralNodeReferenceKey`）与三源快照验收。 | -| [SharedNodeStoreMigration.md](Internal/SharedNodeStoreMigration.md) | **Accepted（待上游 push 后动工）**：三条小 store 流水线（`InternedNodeReferenceCache` / `TypeDefinition` 字段树批量 store / `lateDemangledNode`）汇入上游 0010 `SharedNodeStore` 的迁移设计——改动位置、明确不动的部分、验收计划与落地前置条件。 | +| [SharedNodeStoreMigration.md](Internal/SharedNodeStoreMigration.md) | **Implemented（2026-08-08）**：三条小 store 流水线（`InternedNodeReferenceCache` / `TypeDefinition` 字段树批量 store / `lateDemangledNode`）汇入上游 0010 `SharedNodeStore` 的迁移设计与落地记录——改动位置、明确不动的部分、验证结果与与方案的差异。 | | [DeclarationModelMemoryFootprint.md](Internal/DeclarationModelMemoryFootprint.md) | NodeStore 迁移后的声明模型内存足迹量测：`TypeDefinition` 1272 字节的逐属性构成（两份 `TypeContextWrapper` 占 74%）、`TypeContextWrapper` 按最大 case `Class` 定尺的原因、`parentContext` 是被当成永久字段的临时值、mini-store 增殖与 `MetadataReaderCache` 仍持 `Node` 树；含四项可回收估算（合计 ~8–10%）与「当前不建议实施、应先剖析其余 90%」的结论。 | | [NodeStoreMigrationOpenIssues.md](Internal/NodeStoreMigrationOpenIssues.md) | `feature/node-store-migration` 的**已确认问题清单**（2026-07-28 两轮审查 + 复核）。第一节是前一轮修复自身的两处缺口（Catalyst 降级只覆盖 framework 形态导致 plain dylib 仍平局、`appendRowIfAbsent` 线性扫描），已于 2026-07-29 闭环并保留成因；第二节起为仍然打开或已裁决的条目：`structuralHash` 每文本节点分配 `String` 与 `ABIKey` 每 key materialize 两条**已按上游设计终审关闭**（0.5.1 保持现状且上游说明不改，见 ReviewAdjudications A1/A2）、`memberSymbols` 改为线性 + 全树比对（**已证实量级可忽略，不是回归**）；`lateDemangledNode` 持锁 demangle 与 `ProtocolConformanceDumper` 的 materialize 分支两条**已于 2026-08-03 闭环**（连同失败名裁决缓存、dump 路径引用化）、build sweep 串行且每符号跨线程往返（上游）、`ABIKey` 每 key materialize，以及四条代码卫生项；两个公开查询 API 的字典键一条**已于 2026-08-03 裁决为不修**（类型级 SPI），rebase 前置事项一条前提已过期但压着两条仍成立的注意事项。逐条注明成因、影响面与「该在哪里修」。与 `Reviews/` 下两份审查记录冲突时以后者为准。 | | [SystemFrameworkRenderingVerification.md](Internal/SystemFrameworkRenderingVerification.md) | 系统框架渲染 A/B 验证流程（大重构必跑）：两个检出对同一批真实输入（归档 dyld cache → 无则当前系统 cache、模拟器 runtime → 无则现有 runtime、当前系统 MachOImage）逐字节比对 dump+interface；入口脚本 `Scripts/run-rendering-ab-verification.py`，含回退规则、踩坑清单（`-p` 消歧、fat 二进制 `-a`、同 boot 会话、scratch 隔离）与 2026-08-03 基线运行记录。 | diff --git a/Sources/MachOSymbols/InternedNodeReferenceCache.swift b/Sources/MachOSymbols/InternedNodeReferenceCache.swift index 3c4146b9..ba89c6c8 100644 --- a/Sources/MachOSymbols/InternedNodeReferenceCache.swift +++ b/Sources/MachOSymbols/InternedNodeReferenceCache.swift @@ -5,85 +5,54 @@ import MachOExtensions @_spi(Internals) import MachOCaches import SwiftStdlibToolbox -/// Structural deduplication for `NodeReference(interning:)`. +/// Scope-keyed shared interning arenas for metadata-derived name trees. /// -/// `NodeReference(interning:)` mints one private arena per call — upstream -/// documents it as the wrong tool for a batch, since deduplication and -/// compactness are both properties of a shared arena. The name-construction -/// sites (`TypeName` / `ProtocolName` / `ExtensionName` built from -/// `MetadataReader` trees) call it once per *occurrence*, and occurrences -/// repeat heavily: every conformance re-interns its protocol's name, every -/// nested type re-interns its parent's, every extension its target's. -/// Measured on the `SymbolTestsCore` fixture, 730 retained mini stores -/// backed only 472 structurally unique trees — and the repeat factor grows -/// with framework size (conformance fan-out dominates). +/// One `SharedNodeStore` per scope: every tree interned in a scope lands in +/// that scope's single appendable arena, whose interning tables deduplicate +/// persistently — structurally equal trees return the *same* reference +/// (store identity included), at any two points in the scope's lifetime. +/// The name-construction sites (`TypeName` / `ProtocolName` / +/// `ExtensionName` built from `MetadataReader` trees, and `TypeDefinition`'s +/// field type trees) intern once per *occurrence*, and occurrences repeat +/// heavily: every conformance re-interns its protocol's name, every nested +/// type re-interns its parent's, every extension its target's. Sharing one +/// store per scope keeps name equality on `structurallyEquals`' same-store +/// `store ===` fast path (an index compare, not a tree walk). /// -/// This cache keeps one reference per structurally unique tree: repeats hand -/// back the previously minted reference, so equal names share one store and -/// `structurallyEquals`' same-store `store ===` fast path starts firing for -/// them (name equality drops from a full tree walk to an index compare). -/// -/// Two scopes, matching `SharedCache`'s two keying modes: -/// - **Per image** (`reference(interning:in:)`): the bucket lives and dies +/// Historically this type was a structural-hash bucket layer over +/// `NodeReference(interning:)` — one private mini store per unique tree, +/// because the frozen `NodeStoreBuilder` flow could not hand out references +/// before `freeze()`. Upstream evolution 0010 (`SharedNodeStore`) removed +/// that barrier, so the bucket layer retired; what remains here is the part +/// `SharedNodeStore` deliberately does not know about — Mach-O scope keying +/// and eviction: +/// - **Per image** (`reference(interning:in:)`): the store lives and dies /// with the image — evicted on memory pressure with every other shared /// cache, and dropped by `SwiftDeclarationIndexer`'s per-image cleanup so /// the recycling model holds. /// - **Per process** (`reference(interning:)`): for the in-process reading -/// paths that have no Mach-O handle. One type-keyed bucket, memory-pressure +/// paths that have no Mach-O handle. One type-keyed store, memory-pressure /// evictable, bounded by the unique names the process actually touches. /// -/// Retention trade: the cache pins every minted mini store for its scope's -/// lifetime, including names a query produced and dropped. That is bounded -/// by *unique* names per scope and buys the dedup above; the pre-cache -/// behavior pinned one store per *retained occurrence* with no sharing at -/// all. +/// Eviction reclaims nothing while external references survive: a +/// `NodeReference` keeps its backing storage alive after the scope drops +/// (reads stay valid, interning stops) — the same semantics the retired +/// mini stores had. No capacity reservation: the 0009 coefficients are +/// calibrated on the bulk symbol corpus and name-tree workloads grow +/// incrementally by nature; use `capacityUtilization` if calibration is +/// ever wanted. @_spi(ForSymbolViewer) @_spi(Internals) public final class InternedNodeReferenceCache: SharedCache, @unchecked Sendable { public static let shared = InternedNodeReferenceCache() - public final class Storage: @unchecked Sendable { - /// Minted references bucketed by their tree's structural hash - /// (`Node`'s `Hashable` is structural); collisions resolve by - /// `structurallyEquals`, so a bucket almost always holds one entry. - @Mutex - private var referencesByStructuralHash: [Int: [NodeReference]] = [:] + public final class Storage: Sendable { + /// The scope's single appendable arena; `intern` serializes on the + /// store's own writer lock and deduplicates structurally. + fileprivate let store = SharedNodeStore() - /// Get-or-mint following the same discipline as - /// `SymbolIndexStore.Storage.lateDemangledNode(forName:)`: the - /// interning runs *outside* the critical section (it allocates and - /// walks the whole tree, which has no business inside an - /// `os_unfair_lock`), and the lock arbitrates insert-if-absent — a - /// racing loser discards its freshly minted store and returns the - /// winner's reference, so one structural name never hands out - /// references into two stores within one scope. fileprivate func reference(interning node: Node) -> NodeReference { - var hasher = Hasher() - hasher.combine(node) - let structuralHashValue = hasher.finalize() - - if let existing = _referencesByStructuralHash.withLockUnchecked({ buckets in - buckets[structuralHashValue]?.first(where: { $0.structurallyEquals(node) }) - }) { - return existing - } - - let minted = NodeReference(interning: node) - return _referencesByStructuralHash.withLockUnchecked { buckets in - if let winner = buckets[structuralHashValue]?.first(where: { $0.structurallyEquals(node) }) { - return winner - } - buckets[structuralHashValue, default: []].append(minted) - return minted - } - } - - /// Test-only visibility: the number of structurally distinct trees - /// currently cached in this scope. - public var cachedReferenceCountForTesting: Int { - _referencesByStructuralHash.withLockUnchecked { buckets in - buckets.values.reduce(0) { $0 + $1.count } - } + store.intern(node) } } diff --git a/Sources/MachOSymbols/SymbolIndexStore.swift b/Sources/MachOSymbols/SymbolIndexStore.swift index a9f76ea5..293fa8e1 100644 --- a/Sources/MachOSymbols/SymbolIndexStore.swift +++ b/Sources/MachOSymbols/SymbolIndexStore.swift @@ -191,15 +191,23 @@ public final class SymbolIndexStore: SharedCache, @unc let thunkAttributeMembersByKindAndTypeName: [Node.Kind: [String: [ThunkAttributeMember]]] /// Symbols demangled after the store was frozen (rare path: lookups - /// for names that were not part of the build sweep). The frozen arena - /// cannot grow, so each late name gets a mini store; the volume is - /// small and every consumer keeps receiving a uniform `NodeReference`. - /// - /// Keyed by name, like `tableRowByName`: a demangled tree is a pure - /// function of the symbol name, so two symbols at different offsets - /// sharing a name share a tree. A stored `nil` records a name the - /// demangler rejected — rejection is exactly as deterministic as - /// success, so it is cached the same way and never retried. + /// for names that were not part of the build sweep). The frozen main + /// arena cannot grow, so late names go into this appendable per-image + /// side store; every consumer keeps receiving a uniform + /// `NodeReference`. Deliberately self-held rather than shared with + /// `InternedNodeReferenceCache`'s image store: that cache is evicted + /// and rebuilt under memory pressure while this `Storage` is not, and + /// sharing would leave the two referencing different stores after an + /// eviction. + private let lateNameStore = SharedNodeStore() + + /// Verdict cache over `lateNameStore`, keyed by name like + /// `tableRowByName`: a demangled tree is a pure function of the + /// symbol name, so two symbols at different offsets sharing a name + /// share a tree. A stored `nil` records a name the demangler + /// rejected — rejection is exactly as deterministic as success, so + /// it is cached the same way and never retried (`SharedNodeStore` + /// itself throws on failure and caches nothing). @Mutex private var lateDemangledNodeByName: [String: NodeReference?] = [:] @@ -238,15 +246,12 @@ public final class SymbolIndexStore: SharedCache, @unc /// large-stack thread and block on a semaphore, and an /// `os_unfair_lock` must not be held across a blocking wait (priority /// donation is lost and every other late lookup on the image - /// serializes behind it). The lock arbitrates insert-if-absent - /// instead — two threads missing concurrently both demangle, but only - /// the first insertion wins and the loser returns the winner's - /// reference, so one name still never hands out references into - /// *different* stores (those would compare unequal under - /// `NodeReference`'s store-identity `Hashable` and turn downstream - /// dedup into a run-to-run coin flip). The loser's mini store is - /// discarded; a demangled tree is a pure function of the name, so the - /// copies are interchangeable. + /// serializes behind it). Two threads missing concurrently both + /// demangle, but both intern into the one `lateNameStore`, whose + /// structural dedup hands them the *same* reference — the race costs + /// a duplicate parse, never references into different stores. The + /// insert-if-absent shape stays only to keep one canonical verdict + /// per name. /// /// Rejections are cached like successes (`nil` verdict): the /// demangler is deterministic, so a retry can only re-pay the failed @@ -257,11 +262,7 @@ public final class SymbolIndexStore: SharedCache, @unc if let cachedVerdict = _lateDemangledNodeByName.withLockUnchecked({ $0[name] }) { return cachedVerdict } - var lateBuilder = NodeStoreBuilder() - var demangled: NodeReference? - if let nodeIndex = try? lateBuilder.demangle(name) { - demangled = lateBuilder.freeze().reference(at: nodeIndex) - } + let demangled = try? lateNameStore.demangle(name) return _lateDemangledNodeByName.withLockUnchecked { cache in if let winner = cache[name] { return winner } // `updateValue` rather than the subscript: assigning an diff --git a/Sources/SwiftDeclaration/Components/Definitions/TypeDefinition.swift b/Sources/SwiftDeclaration/Components/Definitions/TypeDefinition.swift index cb703111..f700ad2f 100644 --- a/Sources/SwiftDeclaration/Components/Definitions/TypeDefinition.swift +++ b/Sources/SwiftDeclaration/Components/Definitions/TypeDefinition.swift @@ -158,11 +158,12 @@ public final class TypeDefinition: Definition { let typeContextDescriptor = try required(type.contextDescriptorWrapper.typeContextDescriptor) let fieldDescriptor = try typeContextDescriptor.fieldDescriptor(in: machO) let records = try fieldDescriptor.records(in: machO) - // All field type trees of one type share a single store, so common - // subtrees (module references, stdlib types) deduplicate instead of - // paying a per-field mini store. - var fieldNodeStoreBuilder = NodeStoreBuilder() - var pendingFields: [(name: String, typeNodeIndex: NodeStore.NodeIndex, flags: FieldFlags)] = [] + // Field type trees intern into the image's shared store + // (`InternedNodeReferenceCache`), so common subtrees (module + // references, stdlib types) deduplicate across the whole image — + // the per-type builder+freeze store this replaced could only + // deduplicate within one type. + var indexedFields: [FieldDefinition] = [] for record in records { let typeNode = try record.demangledTypeNode(in: machO) let name = try record.fieldName(in: machO) @@ -190,13 +191,9 @@ public final class TypeDefinition: Definition { if try !record.mangledTypeName(in: machO).isEmpty { fieldFlags.insert(.hasMangledTypeName) } - pendingFields.append((name: name.stripLazyPrefix, typeNodeIndex: fieldNodeStoreBuilder.intern(typeNode), flags: fieldFlags)) - } - let fieldNodeStore = fieldNodeStoreBuilder.freeze() - - self.fields = pendingFields.map { pendingField in - FieldDefinition(name: pendingField.name, typeNode: fieldNodeStore.reference(at: pendingField.typeNodeIndex), flags: pendingField.flags) + indexedFields.append(FieldDefinition(name: name.stripLazyPrefix, typeNode: InternedNodeReferenceCache.shared.reference(interning: typeNode, in: machO), flags: fieldFlags)) } + self.fields = indexedFields let fieldNames = Set(fields.map(\.name)) From 168aad34d333e709f05813e456757ff65921818e Mon Sep 17 00:00:00 2001 From: Mx-Iris Date: Sat, 8 Aug 2026 17:07:36 +0800 Subject: [PATCH 41/77] docs: record the real-world re-measure closing the SharedNodeStore migration RuntimeViewer re-indexed the same five system images: NodeStore instances 14,451 -> 15 (-99.9%), matching the intended shape (one shared store per image + the self-held late side store + the per-process scope); live Node count 208,809 -> 207,489 (small drop as expected - that population was not this migration's target). Numbers are also backfilled into upstream evolution 0010's decision log. --- Documentations/Internal/ProjectEvolutionLog.md | 2 +- Documentations/Internal/SharedNodeStoreMigration.md | 4 +++- 2 files changed, 4 insertions(+), 2 deletions(-) diff --git a/Documentations/Internal/ProjectEvolutionLog.md b/Documentations/Internal/ProjectEvolutionLog.md index 0040988d..6d36c271 100644 --- a/Documentations/Internal/ProjectEvolutionLog.md +++ b/Documentations/Internal/ProjectEvolutionLog.md @@ -747,7 +747,7 @@ - 最后两处带全局缓存的 `demangleAsNode` 转 `demangleAsNodeTransient`：`SwiftLayout.ObjCClassIndex`（取限定名字符串即弃树）与 `SwiftDeclarationRendering.SpecializedMetadataNodeSubstitution`（渲染即弃）。至此 Sources 下 cached `demangleAsNode(` 清零，Stage 5c 收口补全。 - **关键决策**：三条「每树/每类型/每晚到名字铸小 store」的流水线（`InternedNodeReferenceCache`、`TypeDefinition` 字段树批量 store、`lateDemangledNode`）**不动**——它们是对上游「builder 一次性 freeze」缺口的正确规避，等 swift-demangling 提案 0010（`SharedNodeStore`，Draft）落地后统一汇入每镜像共享 store。 - **文档**：[NodeStoreMigrationPlan.md](NodeStoreMigrationPlan.md)「内存图驱动的驻留收口（2026-08-08）」一节；AGENTS.md Stage 5c 站点清单同步。 -- **补记（2026-08-08 同日，第三项落地）**：上游 0010（`SharedNodeStore`）当日 Implemented，本仓库迁移设计（[SharedNodeStoreMigration.md](SharedNodeStoreMigration.md)）经批准后同日实施：`InternedNodeReferenceCache` 退役哈希桶层、外壳换持每 scope 一个 `SharedNodeStore`（31 个调用点零波及）；`TypeDefinition.index` 字段树两阶段收敛为直接 intern 进镜像 store（去重范围从单类型扩到全镜像）；`lateDemangledNode` 换 `Storage` 自持的 side store、「loser 弃店」删除。全量 1337 tests 全绿且与迁移前同数，缓存与 late-path 的八条行为测试未改一行原样通过。用户裁决豁免「上游先 push」前置条件（无 sibling 环境在上游 push 前不可构建，知情接受）。RV 五镜像 memory graph 实景复测待上游会话执行。 +- **补记（2026-08-08 同日，第三项落地）**：上游 0010（`SharedNodeStore`）当日 Implemented，本仓库迁移设计（[SharedNodeStoreMigration.md](SharedNodeStoreMigration.md)）经批准后同日实施：`InternedNodeReferenceCache` 退役哈希桶层、外壳换持每 scope 一个 `SharedNodeStore`（31 个调用点零波及）；`TypeDefinition.index` 字段树两阶段收敛为直接 intern 进镜像 store（去重范围从单类型扩到全镜像）；`lateDemangledNode` 换 `Storage` 自持的 side store、「loser 弃店」删除。全量 1337 tests 全绿且与迁移前同数，缓存与 late-path 的八条行为测试未改一行原样通过。用户裁决豁免「上游先 push」前置条件（无 sibling 环境在上游 push 前不可构建，知情接受）。RV 五镜像 memory graph 实景复测闭环：`NodeStore` 实例 **14,451 → 15（−99.9%）**，存活 `Node` 208,809 → 207,489（预期内小降），数字已回填上游 0010 决策日志。 - **对应版本**：`0.14.1` 之后、下一次 bump 之前（`feature/node-store-migration` 分支）。 --- diff --git a/Documentations/Internal/SharedNodeStoreMigration.md b/Documentations/Internal/SharedNodeStoreMigration.md index 308e4c80..ab1957b0 100644 --- a/Documentations/Internal/SharedNodeStoreMigration.md +++ b/Documentations/Internal/SharedNodeStoreMigration.md @@ -40,7 +40,9 @@ 三处均按上表实施：`InternedNodeReferenceCache.Storage` 换持一个 `SharedNodeStore`（哈希桶层与 get-or-mint 舞蹈整体删除，类文档注释改写为「作用域键控 + 驱逐」的新分工）；`TypeDefinition.index(in:)` 的字段树两阶段（builder → freeze → map）收敛为单阶段直接经缓存 intern 进镜像 store；`lateDemangledNode` 的 builder-per-name 换 `Storage` 自持的 `lateNameStore.demangle(name)`，「loser 弃店」段删除、名字 → 裁决字典保留。AGENTS.md「Symbol indexing」段的四处措辞同步（late 路径、缓存分工、字段树去重范围、跨 store 常态的论据）。 -**验证**：干净重建零 warning；全量 `swift test --skip IntegrationTests` **1337 tests 全绿、0 失败**（与迁移前完全同数，含 interface 快照逐字节断言）；`InternedNodeReferenceCacheTests` 五条行为断言与 `SymbolIndexStoreFixtureTests` 三条 late-path 性质测试**原样通过、未改一行**——同 store、去重、驱逐重铸、并发单赢家这些行为在 `SharedNodeStore` 背书下语义保持。RuntimeViewer 五镜像 memory graph 实景复测（验收计划第 4 条）待 swift-demangling 侧会话按共识执行。 +**验证**：干净重建零 warning；全量 `swift test --skip IntegrationTests` **1337 tests 全绿、0 失败**（与迁移前完全同数，含 interface 快照逐字节断言）；`InternedNodeReferenceCacheTests` 五条行为断言与 `SymbolIndexStoreFixtureTests` 三条 late-path 性质测试**原样通过、未改一行**——同 store、去重、驱逐重铸、并发单赢家这些行为在 `SharedNodeStore` 背书下语义保持。 + +**实景复测（验收计划第 4 条，2026-08-08 闭环）**：RuntimeViewer 重索引同一批五镜像后的 memory graph——`NodeStore` 实例 **14,451 → 15（−99.9%）**，落在「每镜像一个共享 store + 自持 late side store + per-process 外壳」的意图形状内；存活 `Node` 208,809 → 207,489（预期内小降：该计数的主体不是本迁移的标的）。数字已回填上游 0010 决策日志（swift-demangling `feature/node-store` @ 9464265）。 ## 与方案的差异 From 3c73071494d21abf11833bf6523323f709f11634 Mon Sep 17 00:00:00 2001 From: Mx-Iris Date: Sat, 8 Aug 2026 19:12:26 +0800 Subject: [PATCH 42/77] perf(inspection): retire MetadataReaderCache's class Node tree retention MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The three memo dictionaries (mangled name / context offset / symbol name) held whole class Node trees — the private singleton retained ~184k of the 207k live Nodes measured over five system images, with no per-image exit and zero cross-tree sharing. Their values are now NodeReferences interned into the InternedNodeReferenceCache scope stores (deduplicated against the declaration model's name trees); a hit materializes a fresh tree (DAG-preserving, never NodeCache-touching), so returned instances are no longer shared across calls — the only identity-keyed consumer, the per-printRoot printCache, relies solely on within-tree sharing and is unaffected. The public API and all 103 call sites are unchanged, and the new MetadataReader.removeCache(for:) seam wired into SwiftDeclarationIndexer's deinit closes the missing per-image cleanup. Verified: full suite 1337 tests green (same count), rendering A/B 96 pairs byte-identical across all three reader paths, wall-clock parity (72 scenario pairs within 0.2%). Incidental: sibling dependency resolution needs USING_LOCAL_DEPENDENCIES=1 in addition to the sibling directory — a fresh scratch path re-evaluates the manifest with the current environment while an old scratch masks it via the cached evaluation; recorded in the environment-drift checks. See Documentations/Internal/MetadataReaderCacheRetirement.md. --- AGENTS.md | 18 ++++- .../DeclarationModelMemoryFootprint.md | 4 ++ .../Internal/MetadataReaderCacheRetirement.md | 65 +++++++++++++++++++ .../Internal/ProjectEvolutionLog.md | 13 ++++ ...-08-08-metadata-reader-cache-retirement.md | 32 +++++++++ Documentations/README.md | 1 + .../SwiftDeclarationIndexer.swift | 4 ++ Sources/SwiftInspection/MetadataReader.swift | 65 +++++++++++++------ 8 files changed, 178 insertions(+), 24 deletions(-) create mode 100644 Documentations/Internal/MetadataReaderCacheRetirement.md create mode 100644 Documentations/Internal/TaskReports/2026-08-08-metadata-reader-cache-retirement.md diff --git a/AGENTS.md b/AGENTS.md index 01d2200a..204729f5 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -208,7 +208,7 @@ let node = try demangleAsNode("$sSiD") // Returns Node tree let string = node.print(using: .default) // "Swift.Int" ``` -**Symbol indexing (NodeStore-backed)**: `SymbolIndexStore.Storage` no longer retains `Node` class trees. The build sweep demangles each symbol cache-free onto a transient tree (`@_spi(Internals) demangleAsNodeTransient`), classifies on that tree, and interns it into a per-image `NodeStoreBuilder`. Nothing touches the global `NodeCache`, and dropping a `Storage` releases the whole per-image footprint. Storage layout (Stage 3): one flat `symbolTable: [Symbol]` row per unique symbol name (canonical cache-adjusted offset; raw and adjusted offset keys share one row), a parallel `rootNodeIndexByTableRow` array, and every classification index stores 4-byte `UInt32` table rows (member/opaque keys are 4-byte `NodeStore.NodeIndex`); indexes accumulate in final row form during the sweep, so `freeze()` is followed by a plain move, not a conversion pass. `Symbol` is 32 bytes (no `nlist` existential — the external-undefined bit is extracted at collection time into `isExternal`), and `DemangledSymbol` is a 32-byte value (shared-table reference + row + `NodeReference`) whose `symbol` is computed; both are asserted by `compactValueLayouts`. **A `DemangledSymbol` stored into the declaration model must be detached first** (`detachedFromSharedTable()`): the shared `[Symbol]` buffer is the right trade for the hundreds of thousands of values a query vends and drops, but `Array` is a reference to its buffer, so a single stored survivor pins the whole table and every mangled name in it — defeating `removeSubIndexer(_:)`, whose reason for existing is reclaiming per-image memory. Measured on SwiftUI (iOS 18.5): 9,872 stored values referenced 9,506 distinct rows, 5.1% of a 185,988-row table, so detaching trades ~0.6 MB of small allocations for ~19.9 MB of retention. The six storing sites are `DefinitionBuilder`'s four construction points (`Accessor` for variables/subscripts, `FunctionDefinition` for allocators/functions) and `TypeDefinition`'s `deallocatorSymbol` / `destructorSymbol` assignments; `SymbolTableRetentionTests` fails the moment a new one forgets. Do NOT detach on the query path. Query APIs construct `DemangledSymbol`/`NodeReference` values at the exit boundary; APIs taking an externally demangled node (`memberSymbols(of:for:node:)` with `Node` or `NodeReference`, `opaqueTypeDescriptorSymbol(for:)`) match keys via `NodeReference.structurallyEquals(_:)` (same-store O(1), cross-store structural walk) — `opaqueTypeDescriptorSymbol(for:)` buckets its entries by `DemanglingNode.identifier` first (`opaqueTypeDescriptorEntriesByMemberIdentifier`), because a full scan there is O(descriptors × printed `some` returns). `demangledNodeReference(for:)` matches on the symbol **name alone** — a demangled tree is a pure function of the name and the flat table holds one row per unique name, so comparing the row's canonical offset against the queried symbol's offset could only reject valid hits (it once rejected every symbol of a dyld-cache image, since `symbols(for:in:)` stamps each vended `Symbol` with the offset it was queried by). A name the sweep covered is answered from the table verdict alone — including the *rejection* verdict: a `nil` root means the demangler already refused the name once, and the same demangler backs the late path, so the query returns `nil` without re-demangling. Names genuinely outside the build sweep fall back to the per-image appendable side store (`lateDemangledNode(forName:)` — a `SharedNodeStore` self-held by `Storage`, deliberately not shared with `InternedNodeReferenceCache`'s evictable image store; `MetadataReader.demangleSymbolReference(for:in:)` forwards here): the demangle runs *outside* the cache lock (never hold `os_unfair_lock` across a potentially blocking large-stack hop), racing missers intern into the one store whose structural dedup hands them the same reference, and the name → verdict map keeps one canonical answer — while rejections are cached as `nil` verdicts and never retried (`rejectedLateNameCachesItsFailure` / `tableCoveredNameNeverEntersLateCache` / `concurrentLateQueriesShareOneStore` pin all three properties). Struct-layout changes in `MachOSymbols` need a `swift package clean` rebuild — SwiftPM incremental builds have been observed linking stale downstream objects (runtime SIGSEGV in `outlined destroy`). See `Documentations/Internal/NodeStoreMigrationPlan.md`. **Declaration values hold `NodeReference` (Stage 5a)**: `DefinitionName.node` (`TypeName`/`ProtocolName`/`ExtensionName`), `Variable/Function/SubscriptDefinition.node`, `FieldDefinition.typeNode` and `ExtensionDefinition.genericSignature` are `NodeReference`, not `Node`. Member definitions reference the per-image symbol store directly (a live definition keeps that store alive — the intended per-image recycling model); metadata-derived name trees (`MetadataReader` output feeding `TypeName`/`ProtocolName`/`ExtensionName`/generic signatures) go through `InternedNodeReferenceCache` (`Sources/MachOSymbols/`) — one `SharedNodeStore` per scope (upstream evolution 0010; the store's interning tables are the dedup, the cache keeps only what `SharedNodeStore` deliberately does not know: Mach-O scope keying and eviction): an image-keyed scope (evicted with the image; `SwiftDeclarationIndexer`'s cleanup removes it alongside the symbol store) and a process-keyed scope for the in-process helpers with no Mach-O handle, so every tree of a scope shares one store and name equality gets the `store ===` fast path. Do NOT call bare `NodeReference(interning:)` on a batch path — route through the cache. `TypeDefinition.index(in:)`'s field type trees intern into the same image store (cross-type dedup; the per-type builder+freeze store this replaced could only deduplicate within one type). See `Documentations/Internal/SharedNodeStoreMigration.md`. The `Name` types customize `Hashable` to STRUCTURAL semantics (`structurallyEquals` + `structuralHash`) because `NodeReference`'s intrinsic `Hashable` is store-identity based and would split equal keys minted into different stores; they are deliberately NOT `Codable` — upstream dropped `Node: Codable` (a mangled symbol already is the tree's serialized form: smaller, ABI-stable across toolchains, and sharing-preserving through the round trip), and nothing persists these values. A consumer that ever needs to should persist `mangleAsString(node)` and read it back with `demangleAsNode(_:)` rather than reintroducing a node encoding. **Any `Dictionary`/`Set` keyed on a `NodeReference` whose keys and lookups can come from DIFFERENT stores must key on `StructuralNodeReferenceKey` (`Sources/MachOSymbols/`, next to the stores it reconciles), never a bare `NodeReference`.** Different stores are the norm, not the exception: the frozen main symbol store, the late-name side store, and the cache's per-scope stores coexist within one image (and memory-pressure eviction rebuilds a cache scope onto a fresh store while old references keep the old one alive), so cross-store mixing remains the default assumption. Everything that mixes those with image-store references is keyed structurally: the override/vtable lookups (`methodDescriptorLookup` / `vtableOffsetLookup` in `TypeDefinition.index` → `DefinitionBuilder`), `DefinitionBuilder`'s accessor grouping and merged-thunk dedup (`accessorsByNode` / `canonicalIndexByFunctionNode` / `canonicalIndexByAllocatorNode`), and every `visitedNodes` "already claimed this symbol" set (`ExtensionDefinition` / `ProtocolDefinition` / `TypeDefinition` / `OverrideSymbolMatcher` / `ProtocolConformanceDumper`). Under bare keys these fail silently and differently: the override lookups drop the `override` keyword + vtable-offset comment (the Stage 5a regression fixed 2026-07-25), a subscript's getter and setter split into two buckets so the setter-only one is discarded, merged thunks emit their `func`/`init` twice, and a witness can be claimed twice. Bare-`NodeReference` keys are only safe for grouping/dedup WITHIN one `memberSymbols` batch (single hash-consed store, where structural equality coincides with index equality). **Transient demangling everywhere else (Stage 5c)**: `MetadataReader`, `RuntimeFieldLayoutBackend`, `TypedDumper`, `ClassHierarchyDumper`, `Symbol.demangledNode`, `SwiftLayout.ObjCClassIndex` and `SwiftDeclarationRendering.SpecializedMetadataNodeSubstitution` use `demangleAsNodeTransient` + `Node.createTransient`, so the global `NodeCache` no longer grows while browsing. Sources now carries zero cached `demangleAsNode(` call sites — do NOT reintroduce `demangleAsNode`/`Node.create` on unbounded inputs anywhere; a demangle whose tree is consumed and dropped must stay transient. The build sweep's `NodeStoreBuilder` pre-reserves its buffers via `reserveCapacity(expectedSymbolCount:)` (growing-only, interning-result-neutral) — keep that call adjacent to the builder's construction. +**Symbol indexing (NodeStore-backed)**: `SymbolIndexStore.Storage` no longer retains `Node` class trees. The build sweep demangles each symbol cache-free onto a transient tree (`@_spi(Internals) demangleAsNodeTransient`), classifies on that tree, and interns it into a per-image `NodeStoreBuilder`. Nothing touches the global `NodeCache`, and dropping a `Storage` releases the whole per-image footprint. Storage layout (Stage 3): one flat `symbolTable: [Symbol]` row per unique symbol name (canonical cache-adjusted offset; raw and adjusted offset keys share one row), a parallel `rootNodeIndexByTableRow` array, and every classification index stores 4-byte `UInt32` table rows (member/opaque keys are 4-byte `NodeStore.NodeIndex`); indexes accumulate in final row form during the sweep, so `freeze()` is followed by a plain move, not a conversion pass. `Symbol` is 32 bytes (no `nlist` existential — the external-undefined bit is extracted at collection time into `isExternal`), and `DemangledSymbol` is a 32-byte value (shared-table reference + row + `NodeReference`) whose `symbol` is computed; both are asserted by `compactValueLayouts`. **A `DemangledSymbol` stored into the declaration model must be detached first** (`detachedFromSharedTable()`): the shared `[Symbol]` buffer is the right trade for the hundreds of thousands of values a query vends and drops, but `Array` is a reference to its buffer, so a single stored survivor pins the whole table and every mangled name in it — defeating `removeSubIndexer(_:)`, whose reason for existing is reclaiming per-image memory. Measured on SwiftUI (iOS 18.5): 9,872 stored values referenced 9,506 distinct rows, 5.1% of a 185,988-row table, so detaching trades ~0.6 MB of small allocations for ~19.9 MB of retention. The six storing sites are `DefinitionBuilder`'s four construction points (`Accessor` for variables/subscripts, `FunctionDefinition` for allocators/functions) and `TypeDefinition`'s `deallocatorSymbol` / `destructorSymbol` assignments; `SymbolTableRetentionTests` fails the moment a new one forgets. Do NOT detach on the query path. Query APIs construct `DemangledSymbol`/`NodeReference` values at the exit boundary; APIs taking an externally demangled node (`memberSymbols(of:for:node:)` with `Node` or `NodeReference`, `opaqueTypeDescriptorSymbol(for:)`) match keys via `NodeReference.structurallyEquals(_:)` (same-store O(1), cross-store structural walk) — `opaqueTypeDescriptorSymbol(for:)` buckets its entries by `DemanglingNode.identifier` first (`opaqueTypeDescriptorEntriesByMemberIdentifier`), because a full scan there is O(descriptors × printed `some` returns). `demangledNodeReference(for:)` matches on the symbol **name alone** — a demangled tree is a pure function of the name and the flat table holds one row per unique name, so comparing the row's canonical offset against the queried symbol's offset could only reject valid hits (it once rejected every symbol of a dyld-cache image, since `symbols(for:in:)` stamps each vended `Symbol` with the offset it was queried by). A name the sweep covered is answered from the table verdict alone — including the *rejection* verdict: a `nil` root means the demangler already refused the name once, and the same demangler backs the late path, so the query returns `nil` without re-demangling. Names genuinely outside the build sweep fall back to the per-image appendable side store (`lateDemangledNode(forName:)` — a `SharedNodeStore` self-held by `Storage`, deliberately not shared with `InternedNodeReferenceCache`'s evictable image store; `MetadataReader.demangleSymbolReference(for:in:)` forwards here): the demangle runs *outside* the cache lock (never hold `os_unfair_lock` across a potentially blocking large-stack hop), racing missers intern into the one store whose structural dedup hands them the same reference, and the name → verdict map keeps one canonical answer — while rejections are cached as `nil` verdicts and never retried (`rejectedLateNameCachesItsFailure` / `tableCoveredNameNeverEntersLateCache` / `concurrentLateQueriesShareOneStore` pin all three properties). Struct-layout changes in `MachOSymbols` need a `swift package clean` rebuild — SwiftPM incremental builds have been observed linking stale downstream objects (runtime SIGSEGV in `outlined destroy`). See `Documentations/Internal/NodeStoreMigrationPlan.md`. **Declaration values hold `NodeReference` (Stage 5a)**: `DefinitionName.node` (`TypeName`/`ProtocolName`/`ExtensionName`), `Variable/Function/SubscriptDefinition.node`, `FieldDefinition.typeNode` and `ExtensionDefinition.genericSignature` are `NodeReference`, not `Node`. Member definitions reference the per-image symbol store directly (a live definition keeps that store alive — the intended per-image recycling model); metadata-derived name trees (`MetadataReader` output feeding `TypeName`/`ProtocolName`/`ExtensionName`/generic signatures) go through `InternedNodeReferenceCache` (`Sources/MachOSymbols/`) — one `SharedNodeStore` per scope (upstream evolution 0010; the store's interning tables are the dedup, the cache keeps only what `SharedNodeStore` deliberately does not know: Mach-O scope keying and eviction): an image-keyed scope (evicted with the image; `SwiftDeclarationIndexer`'s cleanup removes it alongside the symbol store) and a process-keyed scope for the in-process helpers with no Mach-O handle, so every tree of a scope shares one store and name equality gets the `store ===` fast path. Do NOT call bare `NodeReference(interning:)` on a batch path — route through the cache. `TypeDefinition.index(in:)`'s field type trees intern into the same image store (cross-type dedup; the per-type builder+freeze store this replaced could only deduplicate within one type). See `Documentations/Internal/SharedNodeStoreMigration.md`. `MetadataReader`'s own demangle memo (`MetadataReaderCache`: mangled-name / context-offset / symbol-name verdicts, including cached `nil` rejections) stores `NodeReference`s into these same scope stores rather than class `Node` trees — a hit materializes a fresh tree (DAG sharing within one materialized tree is preserved, but returned instances are never shared across calls: key long-lived state structurally, never by `ObjectIdentifier` of a returned node), and the indexer cleanup drops the memo per image via `MetadataReader.removeCache(for:)` so it cannot outlive the scope store it references. See `Documentations/Internal/MetadataReaderCacheRetirement.md`. The `Name` types customize `Hashable` to STRUCTURAL semantics (`structurallyEquals` + `structuralHash`) because `NodeReference`'s intrinsic `Hashable` is store-identity based and would split equal keys minted into different stores; they are deliberately NOT `Codable` — upstream dropped `Node: Codable` (a mangled symbol already is the tree's serialized form: smaller, ABI-stable across toolchains, and sharing-preserving through the round trip), and nothing persists these values. A consumer that ever needs to should persist `mangleAsString(node)` and read it back with `demangleAsNode(_:)` rather than reintroducing a node encoding. **Any `Dictionary`/`Set` keyed on a `NodeReference` whose keys and lookups can come from DIFFERENT stores must key on `StructuralNodeReferenceKey` (`Sources/MachOSymbols/`, next to the stores it reconciles), never a bare `NodeReference`.** Different stores are the norm, not the exception: the frozen main symbol store, the late-name side store, and the cache's per-scope stores coexist within one image (and memory-pressure eviction rebuilds a cache scope onto a fresh store while old references keep the old one alive), so cross-store mixing remains the default assumption. Everything that mixes those with image-store references is keyed structurally: the override/vtable lookups (`methodDescriptorLookup` / `vtableOffsetLookup` in `TypeDefinition.index` → `DefinitionBuilder`), `DefinitionBuilder`'s accessor grouping and merged-thunk dedup (`accessorsByNode` / `canonicalIndexByFunctionNode` / `canonicalIndexByAllocatorNode`), and every `visitedNodes` "already claimed this symbol" set (`ExtensionDefinition` / `ProtocolDefinition` / `TypeDefinition` / `OverrideSymbolMatcher` / `ProtocolConformanceDumper`). Under bare keys these fail silently and differently: the override lookups drop the `override` keyword + vtable-offset comment (the Stage 5a regression fixed 2026-07-25), a subscript's getter and setter split into two buckets so the setter-only one is discarded, merged thunks emit their `func`/`init` twice, and a witness can be claimed twice. Bare-`NodeReference` keys are only safe for grouping/dedup WITHIN one `memberSymbols` batch (single hash-consed store, where structural equality coincides with index equality). **Transient demangling everywhere else (Stage 5c)**: `MetadataReader`, `RuntimeFieldLayoutBackend`, `TypedDumper`, `ClassHierarchyDumper`, `Symbol.demangledNode`, `SwiftLayout.ObjCClassIndex` and `SwiftDeclarationRendering.SpecializedMetadataNodeSubstitution` use `demangleAsNodeTransient` + `Node.createTransient`, so the global `NodeCache` no longer grows while browsing. Sources now carries zero cached `demangleAsNode(` call sites — do NOT reintroduce `demangleAsNode`/`Node.create` on unbounded inputs anywhere; a demangle whose tree is consumed and dropped must stay transient. The build sweep's `NodeStoreBuilder` pre-reserves its buffers via `reserveCapacity(expectedSymbolCount:)` (growing-only, interning-result-neutral) — keep that call adjacent to the builder's construction. ## Test Environment @@ -278,12 +278,24 @@ Rule out both before attributing red tests to a code change: 2. **Missing local sibling dependencies.** `Package.swift` declares `../MachOKit`, `../MachOObjCSection`, `../swift-demangling`, and `../swift-semantic-string` as *conditional local path* dependencies: they are - used when the sibling directory exists, else resolution silently falls back + used only when the sibling directory exists AND `USING_LOCAL_DEPENDENCIES=1` + is set in the build's environment, else resolution silently falls back to the remote release. A worktree checked out elsewhere (e.g. a scratchpad) has no siblings and builds against older remote versions — output drifts wholesale (e.g. `T?` printing as `Swift.Optional`), invalidating any cross-commit A/B comparison. For historical-baseline experiments, place or - symlink all four siblings next to the worktree first. + symlink all four siblings next to the worktree first. Beware the + environment-variable half being masked by caching: SwiftPM caches the + manifest *evaluation* per scratch path, so a long-lived scratch keeps using + local siblings from a session where the variable was set, while a FRESH + scratch re-evaluates with the current environment and silently resolves + remote (observed as `NodeStoreBuilder has no member reserveCapacity` when a + pristine A/B baseline scratch dropped the sibling `swift-demangling`). + Diagnose with the scratch's `workspace-state.json`: the dependency's + `packageRef.kind` reads `fileSystem` (sibling) vs `remoteSourceControl` + (fallback). Export `USING_LOCAL_DEPENDENCIES=1` for every build that must + use siblings — including detached/`nohup` runs like the rendering A/B + script. ## Work In Progress diff --git a/Documentations/Internal/DeclarationModelMemoryFootprint.md b/Documentations/Internal/DeclarationModelMemoryFootprint.md index b1868f06..8447e4fb 100644 --- a/Documentations/Internal/DeclarationModelMemoryFootprint.md +++ b/Documentations/Internal/DeclarationModelMemoryFootprint.md @@ -164,3 +164,7 @@ print(MemoryLayout.size) // 160 malloc 分桶用 `malloc_size()`（``）实测，Swift 对象用 `malloc_size(Unmanaged.passUnretained(object).toOpaque())`。 + +## 后记（2026-08-08） + +第五节第 3 条的后半项（`MetadataReaderCache` 改持 `NodeReference`）已按用户裁决落地——「当前不建议实施」的结论对该项不再成立，其余各项维持原判。设计与落地记录见 [MetadataReaderCacheRetirement.md](MetadataReaderCacheRetirement.md)；本文其余量测与账目保持原貌不改。 diff --git a/Documentations/Internal/MetadataReaderCacheRetirement.md b/Documentations/Internal/MetadataReaderCacheRetirement.md new file mode 100644 index 00000000..d2fb4d85 --- /dev/null +++ b/Documentations/Internal/MetadataReaderCacheRetirement.md @@ -0,0 +1,65 @@ +# MetadataReaderCache 清退：class Node 树缓存换持 NodeReference + +> **状态：Implemented（2026-08-08 批准并同日落地）**——落地记录见文末。 + +## 一句话 + +`MetadataReaderCache` 是 NodeStore 体系之前的旧式缓存——三张字典直接持有整棵 class `Node` 树，是五镜像实测中约 18.4 万个残留 `Node`（占存活总量 207,489 的 ~89%）的持有主体；把它的载荷换成 `NodeReference`（汇入既有的 `InternedNodeReferenceCache` 作用域 store），公开 API 与全部 103 处调用点零改动，class `Node` 常驻清零，并补上它缺失的按镜像清理出口。 + +## 背景与账目 + +RuntimeViewer 索引五个系统镜像（Foundation + libswiftCore + AppKit + SwiftUI + SwiftUICore）后的 memory graph：`SharedNodeStore` 迁移把 `NodeStore` 实例从 14,451 降到 15，但存活 class `Node` 几乎未动（208,809 → 207,489）。归因（本仓库 `DeclarationModelMemoryFootprint.md` 第五节第 3 条与对面会话的独立全量持有点扫描相互印证）： + +- **主体是 `MetadataReaderCache.Storage`**（`Sources/SwiftInspection/MetadataReader.swift:649-660`）的三张字典，value 都是整棵 class `Node` 树：`nodeForMangledNameBox`（mangled name → 树）、`nodeForContextOffset`（descriptor offset → 树）、`nodeForSymbolName`（symbol 名 → 树或 `nil` 拒绝裁决）。 +- **只进不出**：该类是 `private` 单例，`SharedCache.remove(for:)` 明明存在（`Sources/MachOCaches/SharedCache.swift:121`），但没有任何公开 seam 能调到它——`SwiftDeclarationIndexer.deinit` 按镜像清了 symbol store 和 `InternedNodeReferenceCache`（`SwiftDeclarationIndexer.swift:156-160`）却清不到这里，唯一出口是内存压力触发的全局 `removeAll`。 +- **零跨树共享**：Stage 5c 把构造改成了 transient（阻止全局 `NodeCache` 增长），但没改**持有**形态——每棵被缓存的树都自带私有的 `.module("Swift")` / `.identifier("Int")` 叶子副本，hash-consing 去重率为零。 +- 用户裁决（2026-08-08，经 swift-demangling 会话转达）：这套旧式简单缓存机制去掉，方案与审批按本仓库规矩走。 + +## 改动位置（先看这里） + +| 位置 | 改什么 | +|---|---| +| `Sources/SwiftInspection/MetadataReader.swift:649-660`（`MetadataReaderCache.Storage`） | 三张字典的 value 从 `Node` / `Node?` 换成 `NodeReference` / `NodeReference?`。键不动（`MangledNameBox` / `Int` offset / `String` symbol 名——键是对「demangle 这件工作」的去重，store 是对「树」的去重，两层各管各的，与 `lateDemangledNode` 落地的模式完全同构）。 | +| 同文件六个缓存方法（`demangleType` ×2、`demangleContext` ×2、`buildContextManglingForSymbol` ×2） | miss 路径：现有 `_demangle…`（transient 构造，不变）→ `InternedNodeReferenceCache.shared.reference(interning:in:)`（镜像作用域）/ `reference(interning:)`（进程作用域）→ 字典存引用 → 把刚 demangle 出的树直接返回。hit 路径：`reference.materialize()` 重建一棵独立树返回；`nil` 拒绝裁决直接返回 `nil`。`@_spi(Internals) import MachOSymbols` 已在（`MetadataReader.swift:8`），无新依赖。 | +| `Sources/SwiftInspection/MetadataReader.swift`（新增） | 补清理 seam：`MetadataReader.removeCache(for:)`（`@_spi(Internals) public`），实现即 `MetadataReaderCache.shared.remove(for: machO)`。 | +| `Sources/SwiftIndexing/SwiftDeclarationIndexer.swift:156-160`（`deinit`） | 在既有两个按镜像 remove 旁边追加 `MetadataReader.removeCache(for: machO)`，让这份缓存与 symbol store / interned-name 桶同一节奏回收。 | +| `AGENTS.md`「Symbol indexing」段 | 同步措辞：`MetadataReader` 的 demangle memo 载荷已是 `NodeReference`（汇入 `InternedNodeReferenceCache` 作用域 store），且随 indexer 的按镜像清理一起释放。 | + +## 明确不动的部分 + +- **公开 API 与 103 处调用点**：`MetadataReader.demangleType` / `demangleContext` 仍返回 `Node`，Sources 内 36 个文件共 103 处调用点（对面报 112，含测试口径差）一行不改。 +- **`MultiPayloadEnumDescriptorCache` 的 `[Node: MultiPayloadEnumDescriptor]` 键**（`Sources/SwiftDeclarationRendering/MultiPayloadEnumDescriptorCache.swift:32`）：对面判断「主缓存动了它必须同批改键」，核实后**不成立**——class `Node` 的 `==` / `hash` 是**结构语义**（上游 `Node+Hashable.swift`：全子树结构摘要 + DAG 记忆化，实例身份仅是快路径），换后端后 build 键与查询键即使是不同实例也照常命中。该缓存人口极小（每镜像的 multi-payload enum 数量级是百，树是短名字树），保留原样；它残留的少量 class `Node` 在预期残余 ≲2.3 万之内。 +- **`isCacheEnabled` 开关与 `demangleTypeUncached`**：语义不变。后者的免重入理由（`SharedCache` build 闭包内再进 `storage()` 会 trap）在新形态下依旧成立。 +- **`GenericArgumentEnvironment.swift:250` 往 `NodeCache.shared` 叶子表灌节点**：对面留档的次要项，属剩余 ~11% 的一部分，不在本次范围。 + +## 关键设计取舍 + +- **换后端，而非彻底删除**：三张字典 memo 的是昂贵的 demangle 工作——`nodeForContextOffset` 尤其省掉了逐类型重建父 context 链的重复 Mach-O 读取与递归构建。彻底删除会让 103 处调用点每次全量重做，CPU 回归几乎必然，内存上却不比换后端多赚（常驻已清零）。 +- **汇入 `InternedNodeReferenceCache` 的既有作用域 store，而非自持新 store**：AGENTS.md 记载的模型本来就是「metadata 派生的名字树走 `InternedNodeReferenceCache`」——喂进声明模型的 `TypeName` / `ProtocolName` 等树与本缓存的树**大量就是同一批**，汇入同一 store 后字典 value 只是 16 字节引用，树体与声明模型去重共享，边际内存≈字典本身。生命周期也同构：两者都是 `SharedCache`（同受内存压力清理），且 `deinit` 里新增的 remove 让两者同一节奏按镜像释放。上一轮 `lateDemangledNode` 自持 side store 的理由（`SymbolIndexStore.Storage` 不是 `SharedCache`，生命周期错配）在这里不存在。 +- **hit 路径每次 `materialize()` 一棵新树**：这是本方案唯一的性能代价——今天 hit 返回共享实例零分配，之后每次 hit 付 O(节点数) 的重建。重建远比它 memo 掉的 demangle + Mach-O 解析便宜（一至两个数量级），且 `materializeNode` 按 index 记忆化、**保 DAG 共享**（back-reference 子树在同一棵物化树内仍是 `===` 复用）。验收里做整镜像 interface 生成的 wall-clock 对比兜底。 +- **返回实例的跨调用身份不再稳定**：这是唯一可能隐性破坏的行为面，已横向排查 Sources 全部按身份键控的用法——`SwiftPrinting` 的 `printCache`（`ObjectIdentifier(node)` 键）只活在单次 `printRoot` 内，靠的是**同一棵树内**的 DAG 共享，物化保共享所以不受影响；`RuntimeFieldLayoutBackend:223` 的 `ObjectIdentifier(metatype)` 是 runtime metadata 包装、与 `Node` 无关；其余所有 `[Node: …]` 容器靠结构语义 `Hashable` 不受实例更替影响。副作用是正向的：调用方拿到的从「可被下游意外污染的共享缓存实例」变成私有树，缓存不再可能被调用方的树改写毒化。 +- **改 API 返回 `NodeReference`（被否）**：内存收益与本方案完全相同（关键在常驻形态，不在瞬时分配），却要动 103 处调用点及其下游消费链。若日后 profiling 证明 hit 物化是热点，再作为独立演进推进。 + +## 验收计划 + +1. 全量 `swift test --skip IntegrationTests` 全绿；fixture interface 快照逐字节一致（输出零变化是硬约束）。 +2. `Scripts/run-rendering-ab-verification.py`（AGENTS.md 对触及 demangling 层的重构的强制项）：baseline = 本分支当前 tip，candidate = 落地后，三条 reader 路径逐字节对拍。 +3. 性能兜底：SwiftUI 量级二进制的 `swift-section interface` 整镜像生成 wall-clock A/B，确认 hit 物化未造成可感知回归。 +4. RuntimeViewer 侧同一批五镜像复测 memory graph（对面会话协调）：预期存活 class `Node` 207,489 → ≲23,000。 + +## 上游依赖 + +无新增。`NodeReference.materialize()`（cache-free、保 DAG）与 `InternedNodeReferenceCache` 的两个作用域接口都已在现分支可用；不引入新的 swift-demangling 改动，也不改变「上游 `feature/node-store` 未 push 前本分支在无 sibling 环境不可构建」的既有状态。 + +## 落地记录（2026-08-08） + +按改动位置表原样实施：`Storage` 三张字典换 `NodeReference` / `NodeReference?` 载荷（symbol 名的 `nil` 拒绝裁决用 `updateValue` 显式写入，避免 subscript 赋 `nil` 删键）；六个缓存方法 miss 时 intern 进 `InternedNodeReferenceCache` 对应作用域、hit 时 `materialize()`；新增 `MetadataReader.removeCache(for:)` 接进 `SwiftDeclarationIndexer.deinit` 的按镜像清理三连。类文档注释改写为「字典去重工作、作用域 store 去重存储、返回实例跨调用不共享」的新契约。AGENTS.md「Symbol indexing」段同步。 + +**验证（三轴全绿）**： + +1. 全量 `swift test --skip IntegrationTests` **1337 tests 全绿、0 失败**，与改动前完全同数（含 interface 快照逐字节断言）。 +2. `Scripts/run-rendering-ab-verification.py`（baseline = `ed2f4d1`，candidate = 本改动）：**96 对输出全部逐字节一致、零跳过**，覆盖当前系统 dyld cache、七个模拟器 runtime（iOS 15.5–27.0）、in-process MachOImage 三条 reader 路径的 dump + interface。 +3. 性能：A/B 脚本 72 对场景总耗时 baseline 1150s vs candidate 1148s（±0.2%，持平）；iOS 18.5 模拟器 SwiftUI 的 `interface` 三轮交错受控测量中位 71.3s vs 70.9s——hit 路径物化的代价不可感知，与设计预判一致。 +4. RuntimeViewer 五镜像 memory graph 复测由对面会话协调，预期存活 class `Node` 207,489 → ≲23,000（结果出来后补记于此）。 + +**附带发现**：A/B 首跑时基线全新 scratch 把 swift-demangling 解析回了远端 0.5.1（`NodeStoreBuilder has no member reserveCapacity`）——本地 sibling 依赖生效需要「兄弟目录存在 + `USING_LOCAL_DEPENDENCIES=1`」双条件，而旧 scratch 的 manifest 求值缓存会掩盖环境变量未置位。已补进 AGENTS.md 环境漂移检查第 2 条（诊断：`workspace-state.json` 的 `packageRef.kind`）。与方案本身无差异，方案按原样落地。 diff --git a/Documentations/Internal/ProjectEvolutionLog.md b/Documentations/Internal/ProjectEvolutionLog.md index 6d36c271..85827f32 100644 --- a/Documentations/Internal/ProjectEvolutionLog.md +++ b/Documentations/Internal/ProjectEvolutionLog.md @@ -752,6 +752,19 @@ --- +## 33. MetadataReaderCache 清退：残留 class Node 树的持有主体换持 NodeReference + +- **时间段**：2026-08-08（第 32 节同日的后续）。 +- **动机**：`SharedNodeStore` 迁移后 RV 五镜像复测显示存活 class `Node` 几乎未动（207,489），归因约 89%（~18.4 万棵树）被 `MetadataReaderCache.Storage` 的三张字典永久持有——NodeStore 体系之前的旧式缓存，private 单例、只进不出、零跨树共享。用户经 swift-demangling 会话下达清退指示。 +- **落地**：三张字典载荷换 `NodeReference`（树体 intern 进 `InternedNodeReferenceCache` 的镜像/进程作用域 store，与声明模型的同批树直接去重共享），hit 路径 `materialize()` 重建独立树，公开 API 与 Sources 内 103 处调用点零改动；新增 `MetadataReader.removeCache(for:)` 接进 `SwiftDeclarationIndexer.deinit`，关掉「只进不出」。 +- **关键决策**：换后端而非彻底删除（字典 memo 的 demangle 工作有 103 处调用点反复命中，删除必致 CPU 回归且内存不多赚）；对面「`MultiPayloadEnumDescriptorCache` 必须同批改键」的判断经核实不成立（class `Node` 的 `==`/`hash` 是结构语义，实例身份只是快路径），该缓存保留原样。 +- **验证**：全量 1337 tests 同数全绿；渲染 A/B 96 对逐字节一致（三 reader 路径 × dump/interface）；性能持平（72 对场景总耗时 ±0.2%，受控交错测量中位 71.3s vs 70.9s）。 +- **附带发现**：本地 sibling 依赖生效需「兄弟目录存在 + `USING_LOCAL_DEPENDENCIES=1`」双条件，旧 scratch 的 manifest 求值缓存会掩盖后者——已补进 AGENTS.md 环境漂移检查第 2 条。 +- **文档**：[MetadataReaderCacheRetirement.md](MetadataReaderCacheRetirement.md)、[TaskReports/2026-08-08-metadata-reader-cache-retirement.md](TaskReports/2026-08-08-metadata-reader-cache-retirement.md)；[DeclarationModelMemoryFootprint.md](DeclarationModelMemoryFootprint.md) 后记标注该项结论失效。 +- **对应版本**：`0.14.1` 之后、下一次 bump 之前（`feature/node-store-migration` 分支）。 + +--- + ## 维护约定 1. **每个非平凡批次结束时必须在本文追加/更新一节**（新工作弧新增一节；延续既有弧则在该节 diff --git a/Documentations/Internal/TaskReports/2026-08-08-metadata-reader-cache-retirement.md b/Documentations/Internal/TaskReports/2026-08-08-metadata-reader-cache-retirement.md new file mode 100644 index 00000000..511e76de --- /dev/null +++ b/Documentations/Internal/TaskReports/2026-08-08-metadata-reader-cache-retirement.md @@ -0,0 +1,32 @@ +# 2026-08-08 MetadataReaderCache 清退（class Node 树缓存换持 NodeReference） + +## 问题 + +`SharedNodeStore` 迁移（同日上一批）把 `NodeStore` 实例从 14,451 砍到 15，但 RV 五镜像复测里存活的 class `Node` 几乎没动：208,809 → 207,489。两侧会话独立扫描一致归因：约 89%（~18.4 万棵树）被 `Sources/SwiftInspection/MetadataReader.swift` 里的 `MetadataReaderCache.Storage` 永久持有——三张字典（mangled name / context offset / symbol 名）的 value 都是整棵 class `Node` 树，private 单例、`SwiftDeclarationIndexer` 的按镜像清理够不到它、树全部经 transient 构造所以零 hash-consing 共享。用户经 swift-demangling 会话下达指示：这套 NodeStore 体系之前的旧式简单缓存去掉，方案按本仓库规矩走。 + +## 调研 + +- 调用面：Sources 内 36 个文件 103 处调用 `MetadataReader.demangleType` / `demangleContext` / `demangleTypeUncached`（对面报 112，差额是测试目标），全部消费 `Node` 返回值。 +- 上游能力核实：`NodeReference.materialize()` cache-free 且按 index 记忆化**保 DAG 共享**（`NodeStore.BufferView.materializeNode` 显式栈 + memo）；`InternedNodeReferenceCache` 的镜像/进程两个作用域接口即为 metadata 派生树设的（AGENTS.md 既载模型）；`SharedCache.remove(for:)` 本就存在，缺的只是公开 seam。 +- 身份稳定性排查（换掉共享实例唯一可能破的面）：全库按 `ObjectIdentifier` 键控 `Node` 的只有 `SwiftPrinting` 的 `printCache`——单次 `printRoot` 内存活，依赖的是同一棵树内的 DAG 共享，物化保共享故不受影响；`RuntimeFieldLayoutBackend:223` 的 `ObjectIdentifier(metatype)` 是 runtime metadata 指针，与 `Node` 无关。 +- 对面「`MultiPayloadEnumDescriptorCache` 的 `[Node: …]` 键必须同批改」的判断**证伪**：上游 `Node+Hashable.swift` 的 `==`/`hash` 是全子树结构摘要（DAG 记忆化），实例身份只是快路径；换后端后跨实例照常命中。对面复核后接受（「你们对、我错」口径回执）。 + +## 最终方案 + +设计文档 [MetadataReaderCacheRetirement.md](../MetadataReaderCacheRetirement.md)（Draft → 用户批准 → 同日 Implemented）：三张字典载荷换 `NodeReference` / `NodeReference?`，miss 时 intern 进 `InternedNodeReferenceCache` 对应作用域（与声明模型的同批树去重共享），hit 时 `materialize()`；`nil` 拒绝裁决用 `updateValue` 显式写入（subscript 赋 `nil` 会删键）；新增 `MetadataReader.removeCache(for:)` 接进 `SwiftDeclarationIndexer.deinit` 的按镜像清理；公开 API、`isCacheEnabled`、`demangleTypeUncached`、103 处调用点全部不动。被否方案：彻底删除（CPU 回归换不来更多内存）、API 改返回 `NodeReference`（103 处调用点连锁改动，内存收益相同）。 + +## 实际执行 + +按方案原样落地，无偏差。改动面：`MetadataReader.swift`（Storage + 六方法 + 类文档 + seam）、`SwiftDeclarationIndexer.swift`（deinit 一行 + 注释）、AGENTS.md 两处、文档四篇（设计文档、演进日志第 33 节、README 索引、内存足迹文档后记）。 + +## 验证 + +1. 全量 `swift test --skip IntegrationTests`：**1337 tests 全绿、0 失败**，与改动前完全同数。 +2. 渲染 A/B（`Scripts/run-rendering-ab-verification.py`，baseline `ed2f4d1`）：**96 对全部逐字节一致、零跳过**（当前系统 dyld cache + 七个模拟器 runtime + in-process MachOImage，dump + interface）。 +3. 性能：脚本 72 对场景总耗时 1150s vs 1148s（±0.2%）；iOS 18.5 模拟器 SwiftUI `interface` 三轮交错受控测量中位 71.3s vs 70.9s——hit 物化代价不可感知。 +4. RV 五镜像 memory graph 复测待对面协调（预期 207,489 → ≲23,000），结果补记进设计文档。 + +## 偏差与附带发现 + +- 方案本身零偏差。 +- **附带踩坑**：A/B 首跑基线 release 构建失败（`NodeStoreBuilder has no member reserveCapacity`）——全新 scratch 用当前环境重新求值 manifest，`USING_LOCAL_DEPENDENCIES` 未置位导致 swift-demangling 静默回落远端 0.5.1；旧 scratch 靠早前会话的 manifest 求值缓存一直没暴露。本地 sibling 生效是「兄弟目录存在 + 环境变量」双条件。已补进 AGENTS.md 环境漂移检查第 2 条（诊断：`workspace-state.json` 的 `packageRef.kind` 读 `fileSystem` 还是 `remoteSourceControl`），A/B 带 `USING_LOCAL_DEPENDENCIES=1` 重跑通过。 diff --git a/Documentations/README.md b/Documentations/README.md index 4ceb9a76..6240a2ae 100644 --- a/Documentations/README.md +++ b/Documentations/README.md @@ -75,6 +75,7 @@ required by `Version.swift`'s bump contract). | [ClassMemberKeywordRecovery.md](Internal/ClassMemberKeywordRecovery.md) | `class` / `static` 成员关键字的还原：mangling 层面两者不可区分，判据是「类型级成员有 vtable method descriptor ⇒ 源码是 `class`」（`static` 隐式 final、不进 vtable）；模型侧 `isClassMember` 计算属性 + 三个 node printer 接线 + dump vtable 段落关键字，顺带消灭非法的 `override static` 输出；`final class func` 等四类 ABI 上与 `static` 完全一致，保守输出语义等价的 `static`。 | | [NodeStoreMigrationPlan.md](Internal/NodeStoreMigrationPlan.md) | `SymbolIndexStore` 迁移到 `NodeStore` arena 存储的分期计划与实施记录（Stage 0–5）：12B/节点扁平缓冲、cache-free build sweep、Symbol 表压缩、声明层换持 `NodeReference`、散点 transient demangling；含 Stage 5a 的 override/vtable 回归修复（裸 `[NodeReference: …]` 的 store-identity 陷阱 → `StructuralNodeReferenceKey`）与三源快照验收。 | | [SharedNodeStoreMigration.md](Internal/SharedNodeStoreMigration.md) | **Implemented（2026-08-08）**：三条小 store 流水线（`InternedNodeReferenceCache` / `TypeDefinition` 字段树批量 store / `lateDemangledNode`）汇入上游 0010 `SharedNodeStore` 的迁移设计与落地记录——改动位置、明确不动的部分、验证结果与与方案的差异。 | +| [MetadataReaderCacheRetirement.md](Internal/MetadataReaderCacheRetirement.md) | **Implemented（2026-08-08）**：`MetadataReaderCache` 清退——三张 class `Node` 树字典（五镜像实测 ~18.4 万残留 `Node` 的持有主体）换持 `NodeReference` 汇入 `InternedNodeReferenceCache` 作用域 store，公开 API 与 103 处调用点零改动，补按镜像清理 seam；含身份稳定性排查、`MultiPayloadEnumDescriptorCache` 不必同批改键的论证与三轴验证记录（1337 tests 同数全绿 / 渲染 A/B 96 对逐字节一致 / 性能持平）。 | | [DeclarationModelMemoryFootprint.md](Internal/DeclarationModelMemoryFootprint.md) | NodeStore 迁移后的声明模型内存足迹量测：`TypeDefinition` 1272 字节的逐属性构成（两份 `TypeContextWrapper` 占 74%）、`TypeContextWrapper` 按最大 case `Class` 定尺的原因、`parentContext` 是被当成永久字段的临时值、mini-store 增殖与 `MetadataReaderCache` 仍持 `Node` 树；含四项可回收估算（合计 ~8–10%）与「当前不建议实施、应先剖析其余 90%」的结论。 | | [NodeStoreMigrationOpenIssues.md](Internal/NodeStoreMigrationOpenIssues.md) | `feature/node-store-migration` 的**已确认问题清单**（2026-07-28 两轮审查 + 复核）。第一节是前一轮修复自身的两处缺口（Catalyst 降级只覆盖 framework 形态导致 plain dylib 仍平局、`appendRowIfAbsent` 线性扫描），已于 2026-07-29 闭环并保留成因；第二节起为仍然打开或已裁决的条目：`structuralHash` 每文本节点分配 `String` 与 `ABIKey` 每 key materialize 两条**已按上游设计终审关闭**（0.5.1 保持现状且上游说明不改，见 ReviewAdjudications A1/A2）、`memberSymbols` 改为线性 + 全树比对（**已证实量级可忽略，不是回归**）；`lateDemangledNode` 持锁 demangle 与 `ProtocolConformanceDumper` 的 materialize 分支两条**已于 2026-08-03 闭环**（连同失败名裁决缓存、dump 路径引用化）、build sweep 串行且每符号跨线程往返（上游）、`ABIKey` 每 key materialize，以及四条代码卫生项；两个公开查询 API 的字典键一条**已于 2026-08-03 裁决为不修**（类型级 SPI），rebase 前置事项一条前提已过期但压着两条仍成立的注意事项。逐条注明成因、影响面与「该在哪里修」。与 `Reviews/` 下两份审查记录冲突时以后者为准。 | | [SystemFrameworkRenderingVerification.md](Internal/SystemFrameworkRenderingVerification.md) | 系统框架渲染 A/B 验证流程（大重构必跑）：两个检出对同一批真实输入（归档 dyld cache → 无则当前系统 cache、模拟器 runtime → 无则现有 runtime、当前系统 MachOImage）逐字节比对 dump+interface；入口脚本 `Scripts/run-rendering-ab-verification.py`，含回退规则、踩坑清单（`-p` 消歧、fat 二进制 `-a`、同 boot 会话、scratch 隔离）与 2026-08-03 基线运行记录。 | diff --git a/Sources/SwiftIndexing/SwiftDeclarationIndexer.swift b/Sources/SwiftIndexing/SwiftDeclarationIndexer.swift index c662e4fd..324ddd0f 100644 --- a/Sources/SwiftIndexing/SwiftDeclarationIndexer.swift +++ b/Sources/SwiftIndexing/SwiftDeclarationIndexer.swift @@ -158,6 +158,10 @@ public final class SwiftDeclarationIndexer(for context: ContextDescriptorWrapper, in machO: MachO) throws -> Node { if isCacheEnabled { return try MetadataReaderCache.shared.demangleContext(for: context, in: machO) @@ -625,6 +635,15 @@ extension Node { } } +/// Memoizes `MetadataReader`'s expensive demangling work (mangled-name / +/// context-descriptor / symbol-context builds) per image and per process. +/// +/// The dictionaries deduplicate the *work*; the trees live as `NodeReference`s +/// in the `InternedNodeReferenceCache` scope stores, which deduplicate the +/// *storage* against the declaration model's interned name trees. A hit +/// materializes a fresh tree, so the cache retains no class `Node` and the +/// returned instances are never shared across calls — key long-lived state +/// structurally, never by `ObjectIdentifier` of a returned node. private final class MetadataReaderCache: SharedCache, @unchecked Sendable { fileprivate static let shared = MetadataReaderCache() @@ -648,15 +667,17 @@ private final class MetadataReaderCache: SharedCache(for machO: MachO) -> Storage? { @@ -668,21 +689,21 @@ private final class MetadataReaderCache: SharedCache(for mangledName: MangledName, in machO: MachO) throws -> Node { - if let node = storage(in: machO)?.nodeForMangledNameBox[MangledNameBox(mangledName)] { - return node + if let reference = storage(in: machO)?.nodeReferenceForMangledNameBox[MangledNameBox(mangledName)] { + return reference.materialize() } else { let node = try MetadataReader._demangleType(for: mangledName, in: machO) - storage(in: machO)?.nodeForMangledNameBox[MangledNameBox(mangledName)] = node + storage(in: machO)?.nodeReferenceForMangledNameBox[MangledNameBox(mangledName)] = InternedNodeReferenceCache.shared.reference(interning: node, in: machO) return node } } func demangleType(for mangledName: MangledName) throws -> Node { - if let node = storage()?.nodeForMangledNameBox[MangledNameBox(mangledName)] { - return node + if let reference = storage()?.nodeReferenceForMangledNameBox[MangledNameBox(mangledName)] { + return reference.materialize() } else { let node = try MetadataReader._demangleType(for: mangledName) - storage()?.nodeForMangledNameBox[MangledNameBox(mangledName)] = node + storage()?.nodeReferenceForMangledNameBox[MangledNameBox(mangledName)] = InternedNodeReferenceCache.shared.reference(interning: node) return node } } @@ -691,22 +712,22 @@ private final class MetadataReaderCache: SharedCache(for context: ContextDescriptorWrapper, in machO: MachO) throws -> Node { let key = context.contextDescriptor.offset - if let node = storage(in: machO)?.nodeForContextOffset[key] { - return node + if let reference = storage(in: machO)?.nodeReferenceForContextOffset[key] { + return reference.materialize() } else { let node = try MetadataReader._demangleContext(for: context, in: machO) - storage(in: machO)?.nodeForContextOffset[key] = node + storage(in: machO)?.nodeReferenceForContextOffset[key] = InternedNodeReferenceCache.shared.reference(interning: node, in: machO) return node } } func demangleContext(for context: ContextDescriptorWrapper) throws -> Node { let key = context.contextDescriptor.offset - if let node = storage()?.nodeForContextOffset[key] { - return node + if let reference = storage()?.nodeReferenceForContextOffset[key] { + return reference.materialize() } else { let node = try MetadataReader._demangleContext(for: context) - storage()?.nodeForContextOffset[key] = node + storage()?.nodeReferenceForContextOffset[key] = InternedNodeReferenceCache.shared.reference(interning: node) return node } } @@ -715,22 +736,24 @@ private final class MetadataReaderCache: SharedCache(_ symbol: Symbol, in machO: MachO) throws -> Node? { let key = symbol.name - if let cached = storage(in: machO)?.nodeForSymbolName[key] { - return cached + if let cachedVerdict = storage(in: machO)?.nodeReferenceForSymbolName[key] { + return cachedVerdict?.materialize() } else { let node = try MetadataReader._buildContextManglingForSymbol(symbol, in: machO.context) - storage(in: machO)?.nodeForSymbolName[key] = node + // updateValue: a plain subscript assignment of a nil verdict would + // remove the key instead of caching the rejection. + storage(in: machO)?.nodeReferenceForSymbolName.updateValue(node.map { InternedNodeReferenceCache.shared.reference(interning: $0, in: machO) }, forKey: key) return node } } func buildContextManglingForSymbol(_ symbol: Symbol) throws -> Node? { let key = symbol.name - if let cached = storage()?.nodeForSymbolName[key] { - return cached + if let cachedVerdict = storage()?.nodeReferenceForSymbolName[key] { + return cachedVerdict?.materialize() } else { let node = try MetadataReader._buildContextManglingForSymbol(symbol, in: InProcessContext.shared) - storage()?.nodeForSymbolName[key] = node + storage()?.nodeReferenceForSymbolName.updateValue(node.map { InternedNodeReferenceCache.shared.reference(interning: $0) }, forKey: key) return node } } From 56a053340e4dda490ba0f778a85a1df3eb6066ce Mon Sep 17 00:00:00 2001 From: Mx-Iris Date: Sat, 8 Aug 2026 19:28:17 +0800 Subject: [PATCH 43/77] docs: record the real-world re-measure closing the MetadataReaderCache retirement MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit RuntimeViewer's five-image memory graph (environment cross-checked at dda9d72 + swift-demangling 9464265, sibling-resolved with USING_LOCAL_DEPENDENCIES=1): live class Nodes 207,489 -> 44 (-99.98%), NodeStore steady at 15. The design's <=23k expectation is explained as a measurement artifact — a subtraction across different measurement contexts, whose residual sources (the static-layout NodeCache leaf feed and the lazily-built MultiPayloadEnumDescriptorCache) never run under the eager indexing workload; the remaining 44 match NodeFactory's pre-registered singleton pool plus snapshot-instant churn. --- Documentations/Internal/MetadataReaderCacheRetirement.md | 4 +++- Documentations/Internal/ProjectEvolutionLog.md | 2 +- .../2026-08-08-metadata-reader-cache-retirement.md | 2 +- Documentations/README.md | 2 +- 4 files changed, 6 insertions(+), 4 deletions(-) diff --git a/Documentations/Internal/MetadataReaderCacheRetirement.md b/Documentations/Internal/MetadataReaderCacheRetirement.md index d2fb4d85..8e67ac26 100644 --- a/Documentations/Internal/MetadataReaderCacheRetirement.md +++ b/Documentations/Internal/MetadataReaderCacheRetirement.md @@ -60,6 +60,8 @@ RuntimeViewer 索引五个系统镜像（Foundation + libswiftCore + AppKit + Sw 1. 全量 `swift test --skip IntegrationTests` **1337 tests 全绿、0 失败**，与改动前完全同数（含 interface 快照逐字节断言）。 2. `Scripts/run-rendering-ab-verification.py`（baseline = `ed2f4d1`，candidate = 本改动）：**96 对输出全部逐字节一致、零跳过**，覆盖当前系统 dyld cache、七个模拟器 runtime（iOS 15.5–27.0）、in-process MachOImage 三条 reader 路径的 dump + interface。 3. 性能：A/B 脚本 72 对场景总耗时 baseline 1150s vs candidate 1148s（±0.2%，持平）；iOS 18.5 模拟器 SwiftUI 的 `interface` 三轮交错受控测量中位 71.3s vs 70.9s——hit 路径物化的代价不可感知，与设计预判一致。 -4. RuntimeViewer 五镜像 memory graph 复测由对面会话协调，预期存活 class `Node` 207,489 → ≲23,000（结果出来后补记于此）。 +4. RuntimeViewer 五镜像 memory graph 复测（2026-08-08 闭环，RV 用户亲自抓取；环境逐项核对：本仓库 @ `dda9d72` + swift-demangling @ `9464265`，均经 sibling 符号链接本地编译、显式 `USING_LOCAL_DEPENDENCIES=1`、checkouts 确认无远端回退）：存活 class `Node` **207,489 → 44（−99.98%）**；`NodeStore` 15（与上轮持平，符合「汇入既有作用域 store、不新增 store 身份」的预测）。 + +**为什么 44 远低于方案预期的 ≲23,000**：那个预期是测量学假象，不是本次超额达成——≲23k 来自「207,489（RV 复测图）− 183,994（`DeclarationModelMemoryFootprint.md` 的归属数）」，两个数字出自不同测量上下文，差值不对应真实人口。实际上 `MetadataReaderCache` 在 RV 的 eager 索引负载里持有的就是全部 20.7 万的几乎百分之百：预期中的两个残留来源在该负载下根本不运行（`GenericArgumentEnvironment` 的 `NodeCache` 叶子表喂入点在静态布局解析路径上；`MultiPayloadEnumDescriptorCache` 按渲染惰性填充，索引 sweep 不触发）。残余 44 个的量级对应 `NodeFactory` 在 `NodeCache.shared` 初始化时预注册的无参 singleton 池加快照瞬间的少量 churn，判定为终态。 **附带发现**：A/B 首跑时基线全新 scratch 把 swift-demangling 解析回了远端 0.5.1（`NodeStoreBuilder has no member reserveCapacity`）——本地 sibling 依赖生效需要「兄弟目录存在 + `USING_LOCAL_DEPENDENCIES=1`」双条件，而旧 scratch 的 manifest 求值缓存会掩盖环境变量未置位。已补进 AGENTS.md 环境漂移检查第 2 条（诊断：`workspace-state.json` 的 `packageRef.kind`）。与方案本身无差异，方案按原样落地。 diff --git a/Documentations/Internal/ProjectEvolutionLog.md b/Documentations/Internal/ProjectEvolutionLog.md index 85827f32..f2badafe 100644 --- a/Documentations/Internal/ProjectEvolutionLog.md +++ b/Documentations/Internal/ProjectEvolutionLog.md @@ -758,7 +758,7 @@ - **动机**：`SharedNodeStore` 迁移后 RV 五镜像复测显示存活 class `Node` 几乎未动（207,489），归因约 89%（~18.4 万棵树）被 `MetadataReaderCache.Storage` 的三张字典永久持有——NodeStore 体系之前的旧式缓存，private 单例、只进不出、零跨树共享。用户经 swift-demangling 会话下达清退指示。 - **落地**：三张字典载荷换 `NodeReference`（树体 intern 进 `InternedNodeReferenceCache` 的镜像/进程作用域 store，与声明模型的同批树直接去重共享），hit 路径 `materialize()` 重建独立树，公开 API 与 Sources 内 103 处调用点零改动；新增 `MetadataReader.removeCache(for:)` 接进 `SwiftDeclarationIndexer.deinit`，关掉「只进不出」。 - **关键决策**：换后端而非彻底删除（字典 memo 的 demangle 工作有 103 处调用点反复命中，删除必致 CPU 回归且内存不多赚）；对面「`MultiPayloadEnumDescriptorCache` 必须同批改键」的判断经核实不成立（class `Node` 的 `==`/`hash` 是结构语义，实例身份只是快路径），该缓存保留原样。 -- **验证**：全量 1337 tests 同数全绿；渲染 A/B 96 对逐字节一致（三 reader 路径 × dump/interface）；性能持平（72 对场景总耗时 ±0.2%，受控交错测量中位 71.3s vs 70.9s）。 +- **验证**：全量 1337 tests 同数全绿；渲染 A/B 96 对逐字节一致（三 reader 路径 × dump/interface）；性能持平（72 对场景总耗时 ±0.2%，受控交错测量中位 71.3s vs 70.9s）；RV 五镜像 memory graph 实景复测存活 class `Node` **207,489 → 44（−99.98%）**、`NodeStore` 持平 15——远低于方案预期 ≲23k 的原因（跨测量上下文相减的假象人口、两个预期残留源在索引负载下不运行）记录于设计文档。 - **附带发现**：本地 sibling 依赖生效需「兄弟目录存在 + `USING_LOCAL_DEPENDENCIES=1`」双条件，旧 scratch 的 manifest 求值缓存会掩盖后者——已补进 AGENTS.md 环境漂移检查第 2 条。 - **文档**：[MetadataReaderCacheRetirement.md](MetadataReaderCacheRetirement.md)、[TaskReports/2026-08-08-metadata-reader-cache-retirement.md](TaskReports/2026-08-08-metadata-reader-cache-retirement.md)；[DeclarationModelMemoryFootprint.md](DeclarationModelMemoryFootprint.md) 后记标注该项结论失效。 - **对应版本**：`0.14.1` 之后、下一次 bump 之前（`feature/node-store-migration` 分支）。 diff --git a/Documentations/Internal/TaskReports/2026-08-08-metadata-reader-cache-retirement.md b/Documentations/Internal/TaskReports/2026-08-08-metadata-reader-cache-retirement.md index 511e76de..8d863ea4 100644 --- a/Documentations/Internal/TaskReports/2026-08-08-metadata-reader-cache-retirement.md +++ b/Documentations/Internal/TaskReports/2026-08-08-metadata-reader-cache-retirement.md @@ -24,7 +24,7 @@ 1. 全量 `swift test --skip IntegrationTests`：**1337 tests 全绿、0 失败**，与改动前完全同数。 2. 渲染 A/B（`Scripts/run-rendering-ab-verification.py`，baseline `ed2f4d1`）：**96 对全部逐字节一致、零跳过**（当前系统 dyld cache + 七个模拟器 runtime + in-process MachOImage，dump + interface）。 3. 性能：脚本 72 对场景总耗时 1150s vs 1148s（±0.2%）；iOS 18.5 模拟器 SwiftUI `interface` 三轮交错受控测量中位 71.3s vs 70.9s——hit 物化代价不可感知。 -4. RV 五镜像 memory graph 复测待对面协调（预期 207,489 → ≲23,000），结果补记进设计文档。 +4. RV 五镜像 memory graph 复测（同日闭环）：存活 class `Node` **207,489 → 44（−99.98%）**、`NodeStore` 持平 15。远低于 ≲23k 预期的解释（跨测量上下文相减得出的假象人口；两个预期残留源——静态布局路径的 `NodeCache` 叶子表喂入、惰性填充的 `MultiPayloadEnumDescriptorCache`——在 eager 索引负载下均不运行；残余 44 对应 `NodeFactory` 预注册 singleton 池 + 快照瞬间 churn）见设计文档落地记录。 ## 偏差与附带发现 diff --git a/Documentations/README.md b/Documentations/README.md index 6240a2ae..94386631 100644 --- a/Documentations/README.md +++ b/Documentations/README.md @@ -75,7 +75,7 @@ required by `Version.swift`'s bump contract). | [ClassMemberKeywordRecovery.md](Internal/ClassMemberKeywordRecovery.md) | `class` / `static` 成员关键字的还原：mangling 层面两者不可区分，判据是「类型级成员有 vtable method descriptor ⇒ 源码是 `class`」（`static` 隐式 final、不进 vtable）；模型侧 `isClassMember` 计算属性 + 三个 node printer 接线 + dump vtable 段落关键字，顺带消灭非法的 `override static` 输出；`final class func` 等四类 ABI 上与 `static` 完全一致，保守输出语义等价的 `static`。 | | [NodeStoreMigrationPlan.md](Internal/NodeStoreMigrationPlan.md) | `SymbolIndexStore` 迁移到 `NodeStore` arena 存储的分期计划与实施记录（Stage 0–5）：12B/节点扁平缓冲、cache-free build sweep、Symbol 表压缩、声明层换持 `NodeReference`、散点 transient demangling；含 Stage 5a 的 override/vtable 回归修复（裸 `[NodeReference: …]` 的 store-identity 陷阱 → `StructuralNodeReferenceKey`）与三源快照验收。 | | [SharedNodeStoreMigration.md](Internal/SharedNodeStoreMigration.md) | **Implemented（2026-08-08）**：三条小 store 流水线（`InternedNodeReferenceCache` / `TypeDefinition` 字段树批量 store / `lateDemangledNode`）汇入上游 0010 `SharedNodeStore` 的迁移设计与落地记录——改动位置、明确不动的部分、验证结果与与方案的差异。 | -| [MetadataReaderCacheRetirement.md](Internal/MetadataReaderCacheRetirement.md) | **Implemented（2026-08-08）**：`MetadataReaderCache` 清退——三张 class `Node` 树字典（五镜像实测 ~18.4 万残留 `Node` 的持有主体）换持 `NodeReference` 汇入 `InternedNodeReferenceCache` 作用域 store，公开 API 与 103 处调用点零改动，补按镜像清理 seam；含身份稳定性排查、`MultiPayloadEnumDescriptorCache` 不必同批改键的论证与三轴验证记录（1337 tests 同数全绿 / 渲染 A/B 96 对逐字节一致 / 性能持平）。 | +| [MetadataReaderCacheRetirement.md](Internal/MetadataReaderCacheRetirement.md) | **Implemented（2026-08-08）**：`MetadataReaderCache` 清退——三张 class `Node` 树字典（五镜像实测 ~18.4 万残留 `Node` 的持有主体）换持 `NodeReference` 汇入 `InternedNodeReferenceCache` 作用域 store，公开 API 与 103 处调用点零改动，补按镜像清理 seam；含身份稳定性排查、`MultiPayloadEnumDescriptorCache` 不必同批改键的论证与四轴验证记录（1337 tests 同数全绿 / 渲染 A/B 96 对逐字节一致 / 性能持平 / RV 实景存活 `Node` 207,489 → 44，−99.98%）。 | | [DeclarationModelMemoryFootprint.md](Internal/DeclarationModelMemoryFootprint.md) | NodeStore 迁移后的声明模型内存足迹量测：`TypeDefinition` 1272 字节的逐属性构成（两份 `TypeContextWrapper` 占 74%）、`TypeContextWrapper` 按最大 case `Class` 定尺的原因、`parentContext` 是被当成永久字段的临时值、mini-store 增殖与 `MetadataReaderCache` 仍持 `Node` 树；含四项可回收估算（合计 ~8–10%）与「当前不建议实施、应先剖析其余 90%」的结论。 | | [NodeStoreMigrationOpenIssues.md](Internal/NodeStoreMigrationOpenIssues.md) | `feature/node-store-migration` 的**已确认问题清单**（2026-07-28 两轮审查 + 复核）。第一节是前一轮修复自身的两处缺口（Catalyst 降级只覆盖 framework 形态导致 plain dylib 仍平局、`appendRowIfAbsent` 线性扫描），已于 2026-07-29 闭环并保留成因；第二节起为仍然打开或已裁决的条目：`structuralHash` 每文本节点分配 `String` 与 `ABIKey` 每 key materialize 两条**已按上游设计终审关闭**（0.5.1 保持现状且上游说明不改，见 ReviewAdjudications A1/A2）、`memberSymbols` 改为线性 + 全树比对（**已证实量级可忽略，不是回归**）；`lateDemangledNode` 持锁 demangle 与 `ProtocolConformanceDumper` 的 materialize 分支两条**已于 2026-08-03 闭环**（连同失败名裁决缓存、dump 路径引用化）、build sweep 串行且每符号跨线程往返（上游）、`ABIKey` 每 key materialize，以及四条代码卫生项；两个公开查询 API 的字典键一条**已于 2026-08-03 裁决为不修**（类型级 SPI），rebase 前置事项一条前提已过期但压着两条仍成立的注意事项。逐条注明成因、影响面与「该在哪里修」。与 `Reviews/` 下两份审查记录冲突时以后者为准。 | | [SystemFrameworkRenderingVerification.md](Internal/SystemFrameworkRenderingVerification.md) | 系统框架渲染 A/B 验证流程（大重构必跑）：两个检出对同一批真实输入（归档 dyld cache → 无则当前系统 cache、模拟器 runtime → 无则现有 runtime、当前系统 MachOImage）逐字节比对 dump+interface；入口脚本 `Scripts/run-rendering-ab-verification.py`，含回退规则、踩坑清单（`-p` 消歧、fat 二进制 `-a`、同 boot 会话、scratch 隔离）与 2026-08-03 基线运行记录。 | From 8897abb348e1f1c55ad44b7cabf61be2722e8943 Mon Sep 17 00:00:00 2001 From: Mx-Iris Date: Sat, 8 Aug 2026 23:07:43 +0800 Subject: [PATCH 44/77] perf(MachOSymbols): offset-ize symbol names behind SymbolTable (evolution 0001) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Replace SymbolIndexStore's retained per-name Strings (494k / 68.7 MiB across a five-image RuntimeViewer session, the largest single heap item) with string-table references materialized on demand: - SymbolTable: 16-byte SymbolRow (canonical offset + packed name reference) over two name sources — a MachOImage row points zero-copy into the image's mmap'd LINKEDIT string table (clean pages; the table now requires the image to stay loaded), while MachOFile rows and export-trie names append once into a private contiguous byte buffer. - Reader-split collection sweep: the image leg tests isSwiftSymbol on raw bytes (nameBytesHaveSwiftManglingPrefix, mirroring the demangler's prefix list), so non-Swift symbols never materialize a name; the generic leg keeps the String surface. - tableRowByName retired: name -> row is a byte-level binary search over a name-order permutation; the dedup dictionary is build-time-only and the freeze makes exact-capacity copies of the accumulated buffers. - Vend paths materialize names on demand; DemangledSymbol holds the table reference within its 32-byte budget and gains offset/isExternal/name fast paths; symbolRowsByOffset drops OrderedDictionary for Dictionary. Recorded divergences from the accepted proposal (decision log): the Span/UTF8Span access layer is macOS 26-only at runtime (package floor is 10.15) so the byte layer is UnsafeBufferPointer-based, RigidArray is replaced by exact-capacity Array copies (no BasicContainers dependency), and the NodeIndex sentinel rider is dropped (upstream-internal initializer). Verification: 1341 tests / 256 suites green (previous 1337 + 4 new equivalence tests: byte-level prefix check vs String.isSwiftSymbol over a full real symbol table, mapped collection vs the String-based pass, binary search per-row self-consistency on both name sources, detach materialization); rendering A/B vs aa91b9b byte-identical for all 96 pairs across dyld cache + seven simulator runtimes + in-process MachOImage; SwiftUI interface wall-clock parity (median 72.5s vs 70.0s interleaved). File-leg build-phase peak RSS +~15 MiB (dedup keys and private buffer briefly duplicate the name bytes before freeze) is recorded honestly; the steady-state RuntimeViewer re-measure closes the proposal's step 8. Evolution proposal 0001 (Documentations/Evolutions/, established with this change) rides along as Implemented, with the evolution-log section, task report, and AGENTS.md symbol-indexing section updated in the same batch. --- AGENTS.md | 2 +- .../0001-symbol-name-offsetization.md | 196 ++++++++++ Documentations/Evolutions/README.md | 7 +- .../Internal/ProjectEvolutionLog.md | 12 + .../2026-08-08-symbol-name-offsetization.md | 41 +++ Documentations/README.md | 6 +- Sources/MachOSymbols/DemangledSymbol.swift | 47 ++- Sources/MachOSymbols/SymbolIndexStore.swift | 158 +++++---- Sources/MachOSymbols/SymbolTable.swift | 334 ++++++++++++++++++ .../SymbolIndexStoreBaselineTests.swift | 4 +- .../SymbolIndexStoreFixtureTests.swift | 56 ++- .../SymbolTableEquivalenceTests.swift | 101 ++++++ .../SymbolTableRetentionTests.swift | 2 +- 13 files changed, 868 insertions(+), 98 deletions(-) create mode 100644 Documentations/Evolutions/0001-symbol-name-offsetization.md create mode 100644 Documentations/Internal/TaskReports/2026-08-08-symbol-name-offsetization.md create mode 100644 Sources/MachOSymbols/SymbolTable.swift create mode 100644 Tests/MachOSymbolsTests/SymbolTableEquivalenceTests.swift diff --git a/AGENTS.md b/AGENTS.md index 204729f5..aeb194ec 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -208,7 +208,7 @@ let node = try demangleAsNode("$sSiD") // Returns Node tree let string = node.print(using: .default) // "Swift.Int" ``` -**Symbol indexing (NodeStore-backed)**: `SymbolIndexStore.Storage` no longer retains `Node` class trees. The build sweep demangles each symbol cache-free onto a transient tree (`@_spi(Internals) demangleAsNodeTransient`), classifies on that tree, and interns it into a per-image `NodeStoreBuilder`. Nothing touches the global `NodeCache`, and dropping a `Storage` releases the whole per-image footprint. Storage layout (Stage 3): one flat `symbolTable: [Symbol]` row per unique symbol name (canonical cache-adjusted offset; raw and adjusted offset keys share one row), a parallel `rootNodeIndexByTableRow` array, and every classification index stores 4-byte `UInt32` table rows (member/opaque keys are 4-byte `NodeStore.NodeIndex`); indexes accumulate in final row form during the sweep, so `freeze()` is followed by a plain move, not a conversion pass. `Symbol` is 32 bytes (no `nlist` existential — the external-undefined bit is extracted at collection time into `isExternal`), and `DemangledSymbol` is a 32-byte value (shared-table reference + row + `NodeReference`) whose `symbol` is computed; both are asserted by `compactValueLayouts`. **A `DemangledSymbol` stored into the declaration model must be detached first** (`detachedFromSharedTable()`): the shared `[Symbol]` buffer is the right trade for the hundreds of thousands of values a query vends and drops, but `Array` is a reference to its buffer, so a single stored survivor pins the whole table and every mangled name in it — defeating `removeSubIndexer(_:)`, whose reason for existing is reclaiming per-image memory. Measured on SwiftUI (iOS 18.5): 9,872 stored values referenced 9,506 distinct rows, 5.1% of a 185,988-row table, so detaching trades ~0.6 MB of small allocations for ~19.9 MB of retention. The six storing sites are `DefinitionBuilder`'s four construction points (`Accessor` for variables/subscripts, `FunctionDefinition` for allocators/functions) and `TypeDefinition`'s `deallocatorSymbol` / `destructorSymbol` assignments; `SymbolTableRetentionTests` fails the moment a new one forgets. Do NOT detach on the query path. Query APIs construct `DemangledSymbol`/`NodeReference` values at the exit boundary; APIs taking an externally demangled node (`memberSymbols(of:for:node:)` with `Node` or `NodeReference`, `opaqueTypeDescriptorSymbol(for:)`) match keys via `NodeReference.structurallyEquals(_:)` (same-store O(1), cross-store structural walk) — `opaqueTypeDescriptorSymbol(for:)` buckets its entries by `DemanglingNode.identifier` first (`opaqueTypeDescriptorEntriesByMemberIdentifier`), because a full scan there is O(descriptors × printed `some` returns). `demangledNodeReference(for:)` matches on the symbol **name alone** — a demangled tree is a pure function of the name and the flat table holds one row per unique name, so comparing the row's canonical offset against the queried symbol's offset could only reject valid hits (it once rejected every symbol of a dyld-cache image, since `symbols(for:in:)` stamps each vended `Symbol` with the offset it was queried by). A name the sweep covered is answered from the table verdict alone — including the *rejection* verdict: a `nil` root means the demangler already refused the name once, and the same demangler backs the late path, so the query returns `nil` without re-demangling. Names genuinely outside the build sweep fall back to the per-image appendable side store (`lateDemangledNode(forName:)` — a `SharedNodeStore` self-held by `Storage`, deliberately not shared with `InternedNodeReferenceCache`'s evictable image store; `MetadataReader.demangleSymbolReference(for:in:)` forwards here): the demangle runs *outside* the cache lock (never hold `os_unfair_lock` across a potentially blocking large-stack hop), racing missers intern into the one store whose structural dedup hands them the same reference, and the name → verdict map keeps one canonical answer — while rejections are cached as `nil` verdicts and never retried (`rejectedLateNameCachesItsFailure` / `tableCoveredNameNeverEntersLateCache` / `concurrentLateQueriesShareOneStore` pin all three properties). Struct-layout changes in `MachOSymbols` need a `swift package clean` rebuild — SwiftPM incremental builds have been observed linking stale downstream objects (runtime SIGSEGV in `outlined destroy`). See `Documentations/Internal/NodeStoreMigrationPlan.md`. **Declaration values hold `NodeReference` (Stage 5a)**: `DefinitionName.node` (`TypeName`/`ProtocolName`/`ExtensionName`), `Variable/Function/SubscriptDefinition.node`, `FieldDefinition.typeNode` and `ExtensionDefinition.genericSignature` are `NodeReference`, not `Node`. Member definitions reference the per-image symbol store directly (a live definition keeps that store alive — the intended per-image recycling model); metadata-derived name trees (`MetadataReader` output feeding `TypeName`/`ProtocolName`/`ExtensionName`/generic signatures) go through `InternedNodeReferenceCache` (`Sources/MachOSymbols/`) — one `SharedNodeStore` per scope (upstream evolution 0010; the store's interning tables are the dedup, the cache keeps only what `SharedNodeStore` deliberately does not know: Mach-O scope keying and eviction): an image-keyed scope (evicted with the image; `SwiftDeclarationIndexer`'s cleanup removes it alongside the symbol store) and a process-keyed scope for the in-process helpers with no Mach-O handle, so every tree of a scope shares one store and name equality gets the `store ===` fast path. Do NOT call bare `NodeReference(interning:)` on a batch path — route through the cache. `TypeDefinition.index(in:)`'s field type trees intern into the same image store (cross-type dedup; the per-type builder+freeze store this replaced could only deduplicate within one type). See `Documentations/Internal/SharedNodeStoreMigration.md`. `MetadataReader`'s own demangle memo (`MetadataReaderCache`: mangled-name / context-offset / symbol-name verdicts, including cached `nil` rejections) stores `NodeReference`s into these same scope stores rather than class `Node` trees — a hit materializes a fresh tree (DAG sharing within one materialized tree is preserved, but returned instances are never shared across calls: key long-lived state structurally, never by `ObjectIdentifier` of a returned node), and the indexer cleanup drops the memo per image via `MetadataReader.removeCache(for:)` so it cannot outlive the scope store it references. See `Documentations/Internal/MetadataReaderCacheRetirement.md`. The `Name` types customize `Hashable` to STRUCTURAL semantics (`structurallyEquals` + `structuralHash`) because `NodeReference`'s intrinsic `Hashable` is store-identity based and would split equal keys minted into different stores; they are deliberately NOT `Codable` — upstream dropped `Node: Codable` (a mangled symbol already is the tree's serialized form: smaller, ABI-stable across toolchains, and sharing-preserving through the round trip), and nothing persists these values. A consumer that ever needs to should persist `mangleAsString(node)` and read it back with `demangleAsNode(_:)` rather than reintroducing a node encoding. **Any `Dictionary`/`Set` keyed on a `NodeReference` whose keys and lookups can come from DIFFERENT stores must key on `StructuralNodeReferenceKey` (`Sources/MachOSymbols/`, next to the stores it reconciles), never a bare `NodeReference`.** Different stores are the norm, not the exception: the frozen main symbol store, the late-name side store, and the cache's per-scope stores coexist within one image (and memory-pressure eviction rebuilds a cache scope onto a fresh store while old references keep the old one alive), so cross-store mixing remains the default assumption. Everything that mixes those with image-store references is keyed structurally: the override/vtable lookups (`methodDescriptorLookup` / `vtableOffsetLookup` in `TypeDefinition.index` → `DefinitionBuilder`), `DefinitionBuilder`'s accessor grouping and merged-thunk dedup (`accessorsByNode` / `canonicalIndexByFunctionNode` / `canonicalIndexByAllocatorNode`), and every `visitedNodes` "already claimed this symbol" set (`ExtensionDefinition` / `ProtocolDefinition` / `TypeDefinition` / `OverrideSymbolMatcher` / `ProtocolConformanceDumper`). Under bare keys these fail silently and differently: the override lookups drop the `override` keyword + vtable-offset comment (the Stage 5a regression fixed 2026-07-25), a subscript's getter and setter split into two buckets so the setter-only one is discarded, merged thunks emit their `func`/`init` twice, and a witness can be claimed twice. Bare-`NodeReference` keys are only safe for grouping/dedup WITHIN one `memberSymbols` batch (single hash-consed store, where structural equality coincides with index equality). **Transient demangling everywhere else (Stage 5c)**: `MetadataReader`, `RuntimeFieldLayoutBackend`, `TypedDumper`, `ClassHierarchyDumper`, `Symbol.demangledNode`, `SwiftLayout.ObjCClassIndex` and `SwiftDeclarationRendering.SpecializedMetadataNodeSubstitution` use `demangleAsNodeTransient` + `Node.createTransient`, so the global `NodeCache` no longer grows while browsing. Sources now carries zero cached `demangleAsNode(` call sites — do NOT reintroduce `demangleAsNode`/`Node.create` on unbounded inputs anywhere; a demangle whose tree is consumed and dropped must stay transient. The build sweep's `NodeStoreBuilder` pre-reserves its buffers via `reserveCapacity(expectedSymbolCount:)` (growing-only, interning-result-neutral) — keep that call adjacent to the builder's construction. +**Symbol indexing (NodeStore-backed)**: `SymbolIndexStore.Storage` no longer retains `Node` class trees. The build sweep demangles each symbol cache-free onto a transient tree (`@_spi(Internals) demangleAsNodeTransient`), classifies on that tree, and interns it into a per-image `NodeStoreBuilder`. Nothing touches the global `NodeCache`, and dropping a `Storage` releases the whole per-image footprint. Storage layout (Stage 3, offset-ized by evolution proposal 0001): one 16-byte `SymbolRow` per unique symbol name in a `SymbolTable` — canonical (cache-adjusted; raw and adjusted offset keys share one row) offset plus a packed reference to the name's bytes, with **no retained name `String`s**. Names resolve through one of two sources: a `MachOImage` row points straight into the image's mmap'd LINKEDIT string table (clean pages, zero-copy; the table consequently requires the image to stay loaded — vended values' `symbol` reads materialize from it), while `MachOFile` rows and export-trie names (decoded strings with no mapped home) live in the table's private contiguous byte buffer. The collection sweep is reader-split: the image leg tests `isSwiftSymbol` byte-level on `nameC` (`nameBytesHaveSwiftManglingPrefix`, mirroring `Demangling.getManglingPrefixLength`'s prefix list — pinned equal by `SymbolTableEquivalenceTests`), so a non-Swift symbol never materializes a name at all; the generic leg (files) keeps the `String` surface and appends Swift name bytes into the private buffer. Name → row lookup is a byte-level binary search over the `rowsSortedByName` permutation (`SymbolTable.row(forName:)`); the name-keyed dictionary exists only during the build and is discarded at freeze, which also makes exact-capacity copies of the accumulated buffers. The byte access layer is `UnsafeBufferPointer`-based deliberately — `Span`/`UTF8Span` are macOS 26-only at runtime, above this package's deployment floor. `symbolRowsByOffset` is a plain `Dictionary` (single keyed consumer, nothing iterates it in order). A parallel `rootNodeIndexByTableRow` array holds each row's demangled root, and every classification index stores 4-byte `UInt32` table rows (member/opaque keys are 4-byte `NodeStore.NodeIndex`); indexes accumulate in final row form during the sweep, so `freeze()` is followed by a plain move, not a conversion pass. `Symbol` stays the public 32-byte eager value (no `nlist` existential — the external-undefined bit is extracted at collection time into `isExternal`; vend paths materialize it on demand), and `DemangledSymbol` is a 32-byte value (table reference + row + `NodeReference`) whose `symbol` is computed; `compactValueLayouts` asserts all three layouts including `SymbolRow == 16`. **A `DemangledSymbol` stored into the declaration model must be detached first** (`detachedFromSharedTable()`): the shared `SymbolTable` is the right trade for the hundreds of thousands of values a query vends and drops, but a single stored survivor pins the whole table — and, for an image table, keeps its mapped-string-table reads tied to the loaded image — defeating `removeSubIndexer(_:)`, whose reason for existing is reclaiming per-image memory. Measured on SwiftUI (iOS 18.5, pre-0001 representation): 9,872 stored values referenced 9,506 distinct rows, 5.1% of a 185,988-row table, so detaching trades ~0.6 MB of small allocations for the whole table's retention (~19.9 MB then; smaller but still whole-table now). The six storing sites are `DefinitionBuilder`'s four construction points (`Accessor` for variables/subscripts, `FunctionDefinition` for allocators/functions) and `TypeDefinition`'s `deallocatorSymbol` / `destructorSymbol` assignments; `SymbolTableRetentionTests` fails the moment a new one forgets. Do NOT detach on the query path. Query APIs construct `DemangledSymbol`/`NodeReference` values at the exit boundary; APIs taking an externally demangled node (`memberSymbols(of:for:node:)` with `Node` or `NodeReference`, `opaqueTypeDescriptorSymbol(for:)`) match keys via `NodeReference.structurallyEquals(_:)` (same-store O(1), cross-store structural walk) — `opaqueTypeDescriptorSymbol(for:)` buckets its entries by `DemanglingNode.identifier` first (`opaqueTypeDescriptorEntriesByMemberIdentifier`), because a full scan there is O(descriptors × printed `some` returns). `demangledNodeReference(for:)` matches on the symbol **name alone** — a demangled tree is a pure function of the name and the flat table holds one row per unique name, so comparing the row's canonical offset against the queried symbol's offset could only reject valid hits (it once rejected every symbol of a dyld-cache image, since `symbols(for:in:)` stamps each vended `Symbol` with the offset it was queried by). A name the sweep covered is answered from the table verdict alone — including the *rejection* verdict: a `nil` root means the demangler already refused the name once, and the same demangler backs the late path, so the query returns `nil` without re-demangling. Names genuinely outside the build sweep fall back to the per-image appendable side store (`lateDemangledNode(forName:)` — a `SharedNodeStore` self-held by `Storage`, deliberately not shared with `InternedNodeReferenceCache`'s evictable image store; `MetadataReader.demangleSymbolReference(for:in:)` forwards here): the demangle runs *outside* the cache lock (never hold `os_unfair_lock` across a potentially blocking large-stack hop), racing missers intern into the one store whose structural dedup hands them the same reference, and the name → verdict map keeps one canonical answer — while rejections are cached as `nil` verdicts and never retried (`rejectedLateNameCachesItsFailure` / `tableCoveredNameNeverEntersLateCache` / `concurrentLateQueriesShareOneStore` pin all three properties). Struct-layout changes in `MachOSymbols` need a `swift package clean` rebuild — SwiftPM incremental builds have been observed linking stale downstream objects (runtime SIGSEGV in `outlined destroy`). See `Documentations/Internal/NodeStoreMigrationPlan.md`. **Declaration values hold `NodeReference` (Stage 5a)**: `DefinitionName.node` (`TypeName`/`ProtocolName`/`ExtensionName`), `Variable/Function/SubscriptDefinition.node`, `FieldDefinition.typeNode` and `ExtensionDefinition.genericSignature` are `NodeReference`, not `Node`. Member definitions reference the per-image symbol store directly (a live definition keeps that store alive — the intended per-image recycling model); metadata-derived name trees (`MetadataReader` output feeding `TypeName`/`ProtocolName`/`ExtensionName`/generic signatures) go through `InternedNodeReferenceCache` (`Sources/MachOSymbols/`) — one `SharedNodeStore` per scope (upstream evolution 0010; the store's interning tables are the dedup, the cache keeps only what `SharedNodeStore` deliberately does not know: Mach-O scope keying and eviction): an image-keyed scope (evicted with the image; `SwiftDeclarationIndexer`'s cleanup removes it alongside the symbol store) and a process-keyed scope for the in-process helpers with no Mach-O handle, so every tree of a scope shares one store and name equality gets the `store ===` fast path. Do NOT call bare `NodeReference(interning:)` on a batch path — route through the cache. `TypeDefinition.index(in:)`'s field type trees intern into the same image store (cross-type dedup; the per-type builder+freeze store this replaced could only deduplicate within one type). See `Documentations/Internal/SharedNodeStoreMigration.md`. `MetadataReader`'s own demangle memo (`MetadataReaderCache`: mangled-name / context-offset / symbol-name verdicts, including cached `nil` rejections) stores `NodeReference`s into these same scope stores rather than class `Node` trees — a hit materializes a fresh tree (DAG sharing within one materialized tree is preserved, but returned instances are never shared across calls: key long-lived state structurally, never by `ObjectIdentifier` of a returned node), and the indexer cleanup drops the memo per image via `MetadataReader.removeCache(for:)` so it cannot outlive the scope store it references. See `Documentations/Internal/MetadataReaderCacheRetirement.md`. The `Name` types customize `Hashable` to STRUCTURAL semantics (`structurallyEquals` + `structuralHash`) because `NodeReference`'s intrinsic `Hashable` is store-identity based and would split equal keys minted into different stores; they are deliberately NOT `Codable` — upstream dropped `Node: Codable` (a mangled symbol already is the tree's serialized form: smaller, ABI-stable across toolchains, and sharing-preserving through the round trip), and nothing persists these values. A consumer that ever needs to should persist `mangleAsString(node)` and read it back with `demangleAsNode(_:)` rather than reintroducing a node encoding. **Any `Dictionary`/`Set` keyed on a `NodeReference` whose keys and lookups can come from DIFFERENT stores must key on `StructuralNodeReferenceKey` (`Sources/MachOSymbols/`, next to the stores it reconciles), never a bare `NodeReference`.** Different stores are the norm, not the exception: the frozen main symbol store, the late-name side store, and the cache's per-scope stores coexist within one image (and memory-pressure eviction rebuilds a cache scope onto a fresh store while old references keep the old one alive), so cross-store mixing remains the default assumption. Everything that mixes those with image-store references is keyed structurally: the override/vtable lookups (`methodDescriptorLookup` / `vtableOffsetLookup` in `TypeDefinition.index` → `DefinitionBuilder`), `DefinitionBuilder`'s accessor grouping and merged-thunk dedup (`accessorsByNode` / `canonicalIndexByFunctionNode` / `canonicalIndexByAllocatorNode`), and every `visitedNodes` "already claimed this symbol" set (`ExtensionDefinition` / `ProtocolDefinition` / `TypeDefinition` / `OverrideSymbolMatcher` / `ProtocolConformanceDumper`). Under bare keys these fail silently and differently: the override lookups drop the `override` keyword + vtable-offset comment (the Stage 5a regression fixed 2026-07-25), a subscript's getter and setter split into two buckets so the setter-only one is discarded, merged thunks emit their `func`/`init` twice, and a witness can be claimed twice. Bare-`NodeReference` keys are only safe for grouping/dedup WITHIN one `memberSymbols` batch (single hash-consed store, where structural equality coincides with index equality). **Transient demangling everywhere else (Stage 5c)**: `MetadataReader`, `RuntimeFieldLayoutBackend`, `TypedDumper`, `ClassHierarchyDumper`, `Symbol.demangledNode`, `SwiftLayout.ObjCClassIndex` and `SwiftDeclarationRendering.SpecializedMetadataNodeSubstitution` use `demangleAsNodeTransient` + `Node.createTransient`, so the global `NodeCache` no longer grows while browsing. Sources now carries zero cached `demangleAsNode(` call sites — do NOT reintroduce `demangleAsNode`/`Node.create` on unbounded inputs anywhere; a demangle whose tree is consumed and dropped must stay transient. The build sweep's `NodeStoreBuilder` pre-reserves its buffers via `reserveCapacity(expectedSymbolCount:)` (growing-only, interning-result-neutral) — keep that call adjacent to the builder's construction. ## Test Environment diff --git a/Documentations/Evolutions/0001-symbol-name-offsetization.md b/Documentations/Evolutions/0001-symbol-name-offsetization.md new file mode 100644 index 00000000..ab6994f7 --- /dev/null +++ b/Documentations/Evolutions/0001-symbol-name-offsetization.md @@ -0,0 +1,196 @@ +# 0001 - SymbolIndexStore 符号名 offset 化：驻留字符串换字符串表引用 + +- **状态**: Implemented +- **作者**: JH +- **创建日期**: 2026-08-08 +- **最后更新**: 2026-08-08 +- **所属愿景**: 无 +- **关联提案**: 无（本仓库首篇）。跨仓库关联：swift-demangling 0008（字节扫描器）/ 0010（`SharedNodeStore`）为既有地基；其「demangle 入口收 `Span`」新提案与本案解耦对接（见「前期调研 · 上游接口」） +- **实现分支 / PR**: `feature/node-store-migration` +- **配套文档**: 无独立专题文章（收尾判断见决策日志）；维护者事实同步于 AGENTS.md「Symbol indexing」段，过程复盘见 [TaskReports/2026-08-08-symbol-name-offsetization.md](../Internal/TaskReports/2026-08-08-symbol-name-offsetization.md) + +## 摘要 + +`SymbolIndexStore` 的每镜像存储把 49.4 万个符号名以 Swift `String` 驻留（68.7 MiB），而这些名字的原文本就躺在镜像 mmap 的字符串表里（clean 页、不计 footprint）——eager 拷贝等于把免费页复制成付费脏页。本提案把驻留形态换成「字符串表 offset + 按需物化」：行表存 16 字节紧凑行，名字读取经 `Span`/`UTF8Span` 访问层直达映射内存（镜像路径零拷贝零驻留；文件路径收进一次分配的私有连续缓冲），名→行字典整个退役换名字序二分，`isSwiftSymbol` 判定降到字节级让非 Swift 符号零分配。公开 API 与全部调用点零改动，预计拿回 ~70–100 MiB。 + +## 动机 + +RuntimeViewer 对 `MetadataReaderCache` 清退后 445 MB 稳态的全景剖析（footprint + vmmap + heap -sortBySize + MallocStackLogging 调用栈归属，五镜像组合：Foundation + libswiftCore + AppKit + SwiftUI + SwiftUICore）： + +- 堆内存活 355 MiB 中 **`SymbolIndexStore` ≈ 215 MiB / 96 万分配，单项过半、头号大户**。 +- 内部最大单项：**符号名字符串 68.7 MiB / 49.4 万个**——调用栈归属到 `MachOImage.Symbol.name.getter → String.init(cString:)`，`buildStorageSweep` 路径，占全进程 78 万个 StringStorage 的 71%。 +- 其余细目：`canonicalRow` 大缓冲 41.8 MiB / 10 个（`symbolTable` 数组与 `tableRowByName` 字典的底层缓冲，5 镜像 × 2，含 `Array` 倍增生长后 freeze 原样驻留的容量冗余）；5 × `Dictionary`（`tableRowByName`）~20 MiB；`[MachOSymbols.Symbol]` 数组 24.5 MiB。 +- 关键事实：这些名字的原文在镜像 mmap 的 LINKEDIT 字符串表里是 **clean 页**（可随时被系统回收再按需换入，不计入 footprint）；把它们逐个拷成堆上 `String` 的那一刻，免费内存变成了付费脏页。 +- 用户指示（2026-08-08，经 swift-demangling 会话转达）：对 `SymbolIndexStore` 做符号名惰性化 / offset 化，方案与审批按本仓库规矩走。 + +RV 侧估算可作为空间 **~70–100 MiB**（预期堆存活 355 → ~255–285 MiB）。 + +## 前期调研 + +### 现状代码怎么走的 + +- `Sources/MachOSymbols/Symbol.swift:8`：公开值类型 `Symbol { offset: Int, name: String, isExternal: Bool }`，32 字节。 +- `Sources/MachOSymbols/SymbolIndexStore.swift:144`：`Storage.symbolTable: [Symbol]`——每唯一符号名一行，name 即驻留 `String`；`:152` `tableRowByName: [String: UInt32]`，键与行表共享字符串存储（行表放掉 `String` 后它就是 68.7 MiB 的唯一持有者）。 +- `Sources/MachOSymbols/SymbolIndexStore.swift:453`：sweep 收集循环 `for symbol in machO.symbols where symbol.name.isSwiftSymbol && !symbol.nlist.isExternal`——对**每一个**符号表条目（不分 Swift 与否，一个大框架几十万到上百万个）物化一个 `String` 只为做前缀判定；非 Swift 的当场丢弃，是 78 万 StringStorage 里另外 29% 瞬时 churn 的来源。 +- `Sources/MachOSymbols/DemangledSymbol.swift:12`：持 `symbolTable: [Symbol]` + 行号 + `NodeReference`，32 字节预算由 `compactValueLayouts` 钉住；`detachedFromSharedTable()` 拷出单行表供声明模型长期持有（每镜像 ~1 万个）。 +- 消费面排查：`storage.symbolTable` 在 `SymbolIndexStore.swift` / `DemangledSymbol.swift` 之外**零消费者**；所有 vend 面（`Symbols`、`symbols(for:in:)`、`DemangledSymbol.symbol`、`Symbol.resolve`）都物化 eager `Symbol` 返回——内部表示可以自由替换，公开 API 不动。 + +### 上游或依赖是否已具备能力 + +- **MachOKit（无需改动）**：`MachOImage.Symbol` 本来就带 `nameC: UnsafePointer` 直指映射内存中的字符串表（MachOKit `MachOImage+Symbols.swift:13`），`Symbols` / `Symbols64` 序列公开 `stringBase`——镜像路径的 offset 化地基现成。`MachOFile.Symbol` 的名字经 `readString(offset: n_strx)` 按需文件读取，无常驻映射可指。 +- **Swift 6.2 语言特性（工具链 6.3.3，全部可用）**： + - `Span`（SE-0447）：对连续内存的安全非持有视图，裸指针构造的 unsafe 逃生门收敛在构造一行。 + - `UTF8Span`（SE-0464）：`bytesEqual(to:)` 是「精确字节相等」语义（提案明言比 `String.==` 的 Unicode 规范等价**更严**——mangled 名要的恰是字节精确）；`String(copying: UTF8Span)` 按需物化且不重复验证；`init(unsafeAssumingValidUTF8:)` 可对 ASCII 符号名免验证扫描。 + - `~Escapable`（SE-0446）：`Span` 家族不可逃逸——**不能存进 `Storage`**，只能作访问层在作用域内借出。SE-0507（borrow accessors）要 Swift 6.4，工具链未到，故借出形态用闭包式 API 而非 `@_lifetime` 标注属性（下划线属性不进库代码）。 +- **swift-collections（实际解析 1.6.0）**：`BasicContainers` 模块的 `RigidArray`（定容、noncopyable，要求 Swift 6.2）已在稳定面——冻结时把累加 `Array` 一次移入定容缓冲，消掉倍增生长的容量冗余与读路径 COW 检查。需给相关 target 补 `BasicContainers` product 依赖并把 `from` 版本提到 1.6.0。 + +### 验证过什么 + +- export-trie 的名字（`exportedSymbols` 补录路径，`SymbolIndexStore.swift:463`）是 trie 遍历的**解码产物**，不在字符串表里——不能用 strtab offset 表示，需要溢出缓冲（这决定了行格式带来源位）。 +- 全库唯一按 `ObjectIdentifier` 键控符号相关对象的用法已排查：无（`SwiftPrinting` 的 `printCache` 键的是 `Node`，与本案无关）。 +- dyld cache 场景的字符串表页驻留形态可能与单镜像 mmap 不同——未实测，标注为**推测**，以 RV 落地后复测为准。 + +### 上游接口（可选跟进，不是落地前置） + +swift-demangling 侧已预告：demangle 入口收字节（`Span`，0008 字节扫描器地基的自然延伸；对面会话当前就在实现 span-borrowed-views）可由他们起草新提案。我方明确**有兴趣但解耦**：本案先以 transient `String` 喂 demangler 落地（footprint 收益完整拿到），字节入口落地后 sweep 的 demangle 调用一行替换——彼时本案的 span 访问层与其入口签名无缝对接（名字字节 `Span` 直接透传），再消掉 Swift 名的瞬时 churn。需要的接口形态：`demangleAsNodeTransient` 的 `Span` 重载即可（late 路径人口小，`SharedNodeStore.demangle` 不必跟）。 + +## 提议方案 + +四步一体，全部收在 `MachOSymbols` 模块内部： + +1. **行表紧凑化**：`Storage.symbolTable: [Symbol]` 换成 `SymbolTable` final class——`rows: [SymbolRow]`（16 字节/行，今天 32 + 一个 String 分配）+ 名字来源。名字来源两条腿：**镜像**——mapped 字符串表基址（零拷贝、零驻留）；**文件**——sweep 时把 Swift 名字节追加进一次分配的私有连续缓冲（无每对象头开销）。export-trie 溢出名两条腿都进私有缓冲，行内来源位区分——行格式与读取路径全 reader 统一。 +2. **名→行字典退役**：`tableRowByName` 整个删除，替换为 `rowsSortedByName: [UInt32]` 名字序 permutation + 二分查找，比较经 `UTF8Span.bytesEqual` 直接对字节、不物化（49 万行 ≈ 3.7 MB vs 今天 ~20 MiB）。build 期仍用临时 `[String: UInt32]` 去重（freeze 时丢弃，只影响构建峰值不影响稳态）。 +3. **字节级 `isSwiftSymbol`**：收集循环按 reader 分腿（`ObjCClassIndex` 的既有先例）——镜像腿在 `Span` 上做前缀判定，非 Swift 符号从此一个 `String` 都不建；文件腿沿用 `readString` 并把 Swift 名字节进私有缓冲。demangle/分类循环保持 reader 通用。 +4. **vend 面按需物化**：`symbol(atRow:offset:)` / `demangledSymbol(atRow:)` / `detachedFromSharedTable()` 物化时经 `String(copying:)` 从名字来源建 `String`。公开 API 签名与行为不变。 + +### 非目标 + +- **44 万个单元素 `[UInt32]` 桶（~28 MiB，`symbolRowsByOffset` 与分类索引）**：正交问题（单元素内联 / CSR 扁平化），另案处理，候选下一篇提案。 +- **打印名键控的字典**（`typeInfoByName`、`MemberSymbolRows` 类型名键、thunk 桶）：键是 `NodePrinter` 输出、不是符号表拷贝，人口与本账目无关。sweep 里每成员重复 print 类型名是纯 CPU 项，顺带记录、不在本案。 +- **late-name 路径**（`lateNameStore` + 名字 → 裁决字典）：人口是 sweep 外零星查询名，不动。 +- **声明模型的 detached symbol 语义**：`SymbolTableRetentionTests` 钉住的六个存储点，形态完全不变（detach 时物化 eager `String`，每镜像 ~1 万个，人口小）。 +- **公开 API**：`Symbol` 对外仍是 `{ offset, name: String, isExternal }` 的 eager 值类型。 +- **Swift 名 demangle 输入的瞬时 `String`**：等上游字节入口，见「前期调研 · 上游接口」。 + +## 详细设计 + +```swift +/// Retained row: 16 bytes. Today's row is a 32-byte `Symbol` plus one +/// retained `String` allocation (~139 bytes average including storage). +struct SymbolRow { + var canonicalOffset: Int64 + /// Packed: name-source bit (mapped string table / private buffer), + /// isExternal bit, byte length, and byte offset into the source. + var packedNameReference: UInt64 +} + +/// Replaces the bare `[Symbol]`: rows plus the bytes their names point into. +/// `@unchecked Sendable` — all state immutable after freeze. +final class SymbolTable: @unchecked Sendable { + /// MachOImage: base of the mmap'd LINKEDIT string table (clean pages; + /// valid while the image stays loaded). `nil` for MachOFile tables. + let mappedStringTableBase: UnsafeRawPointer? + /// Swift-name bytes for MachOFile rows and export-trie overflow names — + /// one contiguous allocation, appended during the sweep. + let privateNameBuffer: [UInt8] + let rows: [SymbolRow] + /// Name-order permutation over `rows` — binary search replaces + /// `tableRowByName`; comparisons run on bytes via `UTF8Span`. + let rowsSortedByName: [UInt32] + + /// Scoped byte access (Span is ~Escapable and must not be stored). + func withNameBytes(atRow row: UInt32, _ body: (Span) -> Result) -> Result + func materializedName(atRow row: UInt32) -> String // String(copying: UTF8Span) + func row(forName name: String) -> UInt32? // binary search, bytesEqual +} +``` + +- `DemangledSymbol`：`[Symbol]` + 行号 → `SymbolTable` 引用 + 行号，32 字节预算不变（8 + 4 + 16 + padding，`compactValueLayouts` 断言同步）。standalone / detached 形态经单行 eager 表达成（实现细节：`SymbolTable` 的 standalone 变体持 `[String]`）。 +- 冻结缓冲容器：`rows` / `rowsSortedByName` / `rootNodeIndexByTableRow` freeze 时移入定容 `RigidArray`（或等价的精确容量 `Array` 拷贝），消掉倍增生长冗余。 +- 顺带小件（搭车，不单独立项）：`rootNodeIndexByTableRow` 的 `Optional`（8 字节）换 `UInt32.max` 哨兵（省一半）；`symbolRowsByOffset` 从 `OrderedDictionary` 换普通 `Dictionary`（只按键查、从不按序遍历）。 + +### 风险与接受的约束 + +- **镜像卸载**：`SymbolTable` 持有的 mapped 基址在镜像被 `dlclose` 后悬垂——生命周期约束从「查询时读镜像」扩展到「vend 后物化名字时读镜像」（今天的 eager 拷贝免疫此事）。系统框架与 RV 的索引对象从不卸载；记为接受项。 +- **dyld cache 的 MachOFile**：字符串表在 cache 的 LINKEDIT，走文件腿私有缓冲；页驻留形态未实测（见前期调研），RV 复测为最终裁判。 +- **查找 CPU**：字典 O(1) → 二分 log₂(19 万) ≈ 18 次字节比较；mangled 名共享长前缀会让比较扫得深一点，对查询路径（含 `demangledOverrideSymbol` 候选探测环）预期足够便宜。若 profiling 显示热点，退路是字节哈希 `[UInt64: UInt32]` 索引（结构兼容，不影响其余部分）。 +- **构建峰值**：build 期临时去重字典与今天的 `tableRowByName` 同量级，freeze 后释放。 + +## 替代方案考量 + +- **`Symbol.name` 改懒（enum 载荷 / 计算属性带上下文）**：公开值类型的 `Hashable` 语义与 32 字节布局全线波及、103+ 处消费点连锁改动——被否；内部表示换、vend 时物化的方案收益相同且零波及。 +- **文件路径保留 eager `String`**：连续缓冲一次分配、只收原始字节（无 String 头与 malloc 桶开销，约省 1/3），且让行格式全 reader 统一——保留 eager 被否。 +- **名字查找用字节哈希索引而非二分**：`[UInt64: UInt32]` 约 10–12 MB vs permutation 3.7 MB；二分为主、哈希为实测退路。 +- **swift-collections 逐项裁决**（2026-08-08 调研，理由留档免得日后重吵）：`SortedSet` / `SortedDictionary`——`UnstableSortedCollections` trait（非稳定 API），且 B-tree 的强项是增删、我们的索引冻结后只读，平铺二分更优；`UniqueDictionary` / `RigidDictionary` / `UniqueSet`——`UnstableHashedContainers` trait，不进生产；`TrailingArray`——header + 尾随元素单分配对「类 + 两三条平铺数组」是省一两次分配的微优化，换 `ManagedBuffer` 式底层复杂度不值；`InlineArray`（SE-0453）——定长语义与变长桶无交集；`BitSet`——可作 `rootNodeIndexByTableRow` 的 nil 位图，但哨兵值等收益零新依赖；`TreeSet` / `TreeDictionary` / `Heap` / `Deque`——无场景；`Container` 协议族 / `InputSpan`（`UnstableContainersPreview`）——borrowed 遍历的未来方向，关注不构建。 +- **给 SymbolIndexStore 造通用 arena 分配器**：被否——本案 + 桶扁平化（另案）落地后分配次数从 96 万降到几千，平铺数组就是 arena 的表达；Swift 集合不支持自定义分配器，手写哈希表换个位数 MiB 不值。 + +## 影响 + +### 源码兼容性（source compatibility） + +**纯新增 / 无破坏。** 公开与 package 级 API 的签名、语义、返回值形态全部不变：`Symbol`、`Symbols`、`DemangledSymbol.symbol`、`Symbol.resolve`、`SymbolIndexStore` 的全部查询方法照旧。变化仅在 `MachOSymbols` 模块内部的驻留表示与 `Storage` 私有结构。`compactValueLayouts` 与 `SymbolTableRetentionTests` 钉住布局与 detach 契约不回归。 + +### ABI 兼容性（条件项） + +不适用 —— 本库以 SPM 源码分发，使用方每次重新编译（项目类型声明见 `Documentations/README.md`；`Tests/Projects/SymbolTests` 开启 library evolution 是测试工程属性，非本库属性）。 + +### 下游影响 + +- 仓库内：`MachOSymbols` 为改动主体；`MachOFoundation` 及以上各层经既有 API 消费，无源码变化。`Package.swift` 给 `MachOSymbols` 补 `BasicContainers` product 依赖（swift-collections 本就在依赖图，`from` 提到 1.6.0）。 +- 下游仓库（RuntimeViewer、MachOKitUI、SymbolViewer）：零源码改动，重编译即得内存收益。RV 是本案的验收方（footprint + heap 复测）。 + +### 文档与示例 + +- AGENTS.md「Symbol indexing」段同步新驻留模型（`SymbolTable` / span 访问层 / 二分查找 / 镜像卸载约束）。 +- `Documentations/README.md` 索引与本提案状态同步。 +- 落地时按「落地步骤」收尾判断决定是否另写实现说明。 + +## API 演进与废弃策略 + +- 无被替代的公开 API，无废弃标注需求。 +- 无 semver major 跃迁：源码兼容的内部表示变更，随下一次常规版本发布（`Version.swift` bump 时在 changelog 记录内存收益）。 + +## 落地步骤 + +1. `SymbolRow` / `SymbolTable` / 名字来源 + `DemangledSymbol` 换持（`compactValueLayouts` 同步）——可独立构建。 +2. sweep 收集阶段按 reader 分腿：镜像腿字节级 `isSwiftSymbol` + strtab offset 记录；文件腿私有缓冲；export-trie 溢出。 +3. `tableRowByName` 退役 → permutation 二分（`UTF8Span.bytesEqual`）；build 期临时去重字典。 +4. vend 面按需物化 + 搭车小件（哨兵值、`symbolRowsByOffset` 换普通 `Dictionary`、`RigidArray` 定容冻结）。 +5. 新增等价性测试：字节版 `isSwiftSymbol` 对整个真实镜像符号表与 `String` 版逐条一致；二分对每行与旧字典命中一致；detach 物化正确。 +6. 全量 `swift test --skip IntegrationTests` 全绿同数；`Scripts/run-rendering-ab-verification.py`（`USING_LOCAL_DEPENDENCIES=1`）三 reader 路径逐字节一致。 +7. 性能：索引耗时（`prepare` on SwiftUI，预期**更快**——非 Swift 符号不再建 String）与 interface 生成 wall-clock 持平。 +8. RV 复测 footprint + heap（对面协调）：预期堆存活 355 → ~255–285 MiB。 +9. 收尾判断（写进决策日志）：是否需要实现说明（镜像卸载约束与名字来源双腿是「代码看不出来的决策」，倾向写）；新术语（「名字来源 / name source」「permutation 二分」等）是否进术语表。 + +## 落地记录 + +2026-08-08 实施完成（`feature/node-store-migration` 分支，与本提案同 commit）。 + +### 实际改动面 + +- `Sources/MachOSymbols/SymbolTable.swift`（新建）：`SymbolRow`（16 字节，`compactValueLayouts` 断言）、`PackedNameReference`（1 位名字来源 + 1 位 isExternal + 22 位长度 + 40 位偏移）、`SymbolTable`（mapped 基址 / 私有缓冲双名字来源 + `withNameBytes(atRow:)` closure-scoped 字节访问 + `row(forName:)` 字节级二分）、`SymbolTableBuilder`（build 期临时去重字典，freeze 时精确容量拷贝 + permutation 排序）、`compareSymbolNameBytes`（memcmp 序 + 长度 tiebreak）、`nameBytesHaveSwiftManglingPrefix`（字节级前缀判定，逐字节复刻上游 `getManglingPrefixLength` 的前缀集）。 +- `SymbolIndexStore.swift`：`Storage.symbolTable` 换持 `SymbolTable`、`tableRowByName` 删除、`symbolRowsByOffset` 换普通 `Dictionary`；sweep 收集按 reader 分腿（镜像腿 `symbols64`/`symbols32` 直迭代取 `nameC`，非 Swift 符号零分配；文件与其它 reader 走原 `String` 面 + 私有缓冲）；demangle/分类循环与查询 API 从表按需物化。 +- `DemangledSymbol.swift`：换持 `SymbolTable` 引用 + 行号（32 字节预算不变）；standalone/detached 经单行私有缓冲表；新增 `offset` / `isExternal` / `name` 具体快路径（dynamicMember 路径读 offset 不再整只物化 `Symbol`）。 +- 测试：`SymbolTableEquivalenceTests.swift`（新建，镜像腿三项等价）+ fixture 套件文件腿二分/detach 测试 + 三处内部引用适配。 +- 文档：AGENTS.md「Symbol indexing」段、本提案、演进日志第 34 节、任务报告，同批落地。 + +### 验证结果（落地步骤 5–7） + +1. **等价性测试**（新增 4 项，全绿）：字节级 `isSwiftSymbol` 与 `String.isSwiftSymbol` 在 Foundation 镜像全符号表逐条一致；mapped 收集与 String 收集全等（行数 + last-wins canonical offset）；二分对每行自洽（镜像腿 + 文件腿）+ 负例；detach 物化正确。 +2. **全量套件**：`swift test --skip IntegrationTests` **1341 tests / 256 suites 全绿**（改动前 1337 + 新增 4，同数吻合；快路径补丁后复跑同样全绿）。 +3. **渲染 A/B**（`Scripts/run-rendering-ab-verification.py`，baseline `aa91b9b`，双侧 `USING_LOCAL_DEPENDENCIES=1` 且 sibling 均验证为 `fileSystem` 解析）：**96 对全部逐字节一致、0 不一致**（当前系统 dyld cache + iOS 15.5–27.0 七个模拟器 runtime + in-process MachOImage，dump + interface；skip 项均为旧 runtime 本就不含的框架，与上一次 A/B 同构）。 +4. **性能与峰值内存**（iOS 18.5 模拟器 SwiftUI `interface`，双侧 release 三轮交错，`/usr/bin/time -l`）：wall-clock 中位 **72.5s（基线）vs 70.0s（候选）**，散布 61–79s，差异在噪声带内——持平；两侧输出再次逐字节一致。maxRSS 基线 383–390 MiB vs 候选 400–403 MiB——**文件腿构建期峰值 +~15 MiB（+4%）**：build 期去重字典retain 的 `String` 键与私有字节缓冲在 freeze 前短暂持有同一批名字字节的两份拷贝（提案「构建峰值」风险段只算了字典本身、漏了这层字节重复），freeze 丢弃字典后回落。镜像腿不付此代价（行直指 mapped 字符串表、无字节复制），而 RV 的目标指标是**稳态**驻留，最终以落地步骤 8 的 RV 复测为裁判。 +5. **RV footprint + heap 复测**（落地步骤 8）：落地后由 swift-demangling 会话协调 RuntimeViewer 重编复测，结果回填此处（预期堆存活 355 → ~255–285 MiB）。 + +## 决策日志 + +| 日期 | 变更 | 说明 | +|------|------|------| +| 2026-08-08 | Created as Draft | 用户指示（经 swift-demangling 会话转达）：`SymbolIndexStore` 符号名惰性化 / offset 化。RV 全景剖析定位 68.7 MiB 驻留名字符串为堆内最大单项。 | +| 2026-08-08 | 调研补充 | Swift 6.2 `Span` / `UTF8Span` / `~Escapable` 与 swift-collections 1.6.0 稳定面逐项裁决（采用 `RigidArray`，否 `SortedDictionary` 等，理由见「替代方案考量」）。 | +| 2026-08-08 | 格式迁移 | 应用户要求由 `Documentations/Internal/SymbolNameOffsetization.md`（Draft，未提交）转为本 evolution 提案，内容全量并入；本篇为仓库 0001 号提案，`Evolutions/` 目录由此建立。 | +| 2026-08-08 | Accepted → In Progress | 用户审核通过（「审核通过，开始实现」），当日按「落地步骤」开始实施。 | +| 2026-08-08 | 实施偏差：Span 家族不可用 | 编译探针证实 `Span` / `Array.span` / `UTF8Span` 运行时可用性为 **macOS 26.0+**（`error: 'span' is only available in macOS 26.0 or newer`），而本包部署下限是 macOS 10.15 —— 核心路径无法无条件使用。字节访问层改用 `UnsafeBufferPointer`（同样的 closure-scoped 形态：`withNameBytes(atRow:)`；等价语义：`memcmp` 序 + 长度 tiebreak 代替 `bytesEqual`，`String(decoding:as: UTF8.self)` 代替 `String(copying:)`，修复语义与原 `String(cString:)` 一致）。存储表示、API 面与内存收益不变。 | +| 2026-08-08 | 实施偏差：RigidArray 放弃 | 同一部署下限问题的连带裁决：`RigidArray`（noncopyable）存进 class 属性后的 borrow 人体工学要到 SE-0507（Swift 6.4）才齐。改用提案括号里本就给出的等价退路——freeze 时精确容量 `Array` 拷贝（容量已精确者跳过拷贝），且因此**无需**给 `Package.swift` 加 `BasicContainers` 依赖、无需提 swift-collections 版本。 | +| 2026-08-08 | 实施偏差：搭车项裁剪 | `rootNodeIndexByTableRow` 的 `Optional` → `UInt32.max` 哨兵一项**放弃**：`NodeStore.NodeIndex` 的构造器是上游 internal（debug 布局还带 store tag），从原始 `UInt32` 重建索引需要新的上游 API，为 ~1.6 MB 不值得跨仓库开口子。`symbolRowsByOffset` 换普通 `Dictionary` 一项照做。另一实现细节：standalone `SymbolTable` 统一走私有字节缓冲表示（提案草绘的 `[String]` 变体不再需要——单一表示，读取路径零分支）。 | +| 2026-08-08 | Implemented + 收尾判断 | 验证结果见「落地记录」（1341 全绿、A/B 96 对逐字节一致、性能持平；文件腿构建期峰值 +4% 如实记录，RV 稳态复测为最终裁判、结果回填）。收尾判断：**不另写实现说明**——「代码看不出来的决策」（mapped 指针生命周期约束、名字来源双腿、Span 不可用的原因）已分别落在 `SymbolTable` 类文档、AGENTS.md「Symbol indexing」段与本提案决策日志，另立一篇只会是复述；**不登记新术语表**——本项目无 `Glossary.md`（项目现状即约定），「offset 化 / 名字来源 / permutation 二分」均在首次出现处展开。 | diff --git a/Documentations/Evolutions/README.md b/Documentations/Evolutions/README.md index db11878f..60b56466 100644 --- a/Documentations/Evolutions/README.md +++ b/Documentations/Evolutions/README.md @@ -4,11 +4,6 @@ 所有非平凡变更以提案形式落盘，一次改动 = 一份提案文件，从调研到落地全生命周期原地更新。状态机：`Draft` → `In Review` → `Accepted` → `In Progress` → `Implemented`，另有 `Rejected` / `Deferred` / `Withdrawn`；被否的提案保留不删。 -编号全项目连续。0001–0003 属于内存优化线，其提案文件与实现都在 `feature/node-store-migration` 分支上，尚未并入 main——它们的文件链接在该分支并入后恢复。0004 因为修的是 main 上就有的基线 bug，直接在 main 立项与实施。 - | # | 标题 | 状态 | |---|------|------| -| 0001 | SymbolIndexStore 符号名 offset 化：驻留字符串换字符串表引用 | Implemented（`feature/node-store-migration`，待并入） | -| 0002 | 声明模型 descriptor 化：TypeDefinition / ExtensionDefinition / ProtocolDefinition 不再驻留急切解析的胖 wrapper | Implemented（`feature/node-store-migration`，待并入） | -| 0003 | SymbolIndexStore `[UInt32]` 行号桶扁平化：单元素桶内联化 | Implemented（`feature/node-store-migration`，待并入） | -| [0004](0004-arm64e-signed-vwt-pointer-hardening.md) | arm64e 签名 VWT 指针加固：进程内裸读 strip + 真 PAC 环境的回归验证形态 | Implemented | +| [0001](0001-symbol-name-offsetization.md) | SymbolIndexStore 符号名 offset 化：驻留字符串换字符串表引用 | Implemented | diff --git a/Documentations/Internal/ProjectEvolutionLog.md b/Documentations/Internal/ProjectEvolutionLog.md index f2badafe..4b063c04 100644 --- a/Documentations/Internal/ProjectEvolutionLog.md +++ b/Documentations/Internal/ProjectEvolutionLog.md @@ -765,6 +765,18 @@ --- +## 34. SymbolIndexStore 符号名 offset 化：驻留字符串换字符串表引用（evolution 提案 0001） + +- **时间段**：2026-08-08（第 33 节同日的后续；本仓库 Evolution 提案制的首个提案）。 +- **动机**：RV 五镜像 445 MB 稳态剖析定位 `SymbolIndexStore` ≈ 215 MiB 为堆内头号大户，最大单项是 49.4 万个驻留符号名 `String`（68.7 MiB）——原文本就在镜像 mmap 的 LINKEDIT 字符串表（clean 页），eager 拷贝把免费页复制成付费脏页。方案以提案 0001 落盘、经用户批准后实施。 +- **落地**：`SymbolTable`（16 字节 `SymbolRow` = canonical offset + packed name reference；名字来源双腿——镜像行零拷贝直指 mapped 字符串表、文件行与 export-trie 名进私有连续字节缓冲）；收集循环 reader 分腿（镜像腿字节级 `isSwiftSymbol`，非 Swift 符号零分配；文件腿沿用 `readString`）；`tableRowByName` 退役换名字序 permutation 字节级二分（build 期临时去重字典 freeze 丢弃 + 精确容量拷贝）；vend 面按需物化（`DemangledSymbol` 加 `offset`/`isExternal`/`name` 快路径）。公开 API 与全部调用点零改动。 +- **关键决策 / 实施偏差**：`Span`/`UTF8Span` 运行时可用性 macOS 26+、本包部署下限 10.15 → 字节访问层改 `UnsafeBufferPointer`（closure-scoped 形态不变）；`RigidArray` 的 class 属性 borrow 人体工学要 SE-0507 → 精确容量 `Array` 拷贝、免掉 `BasicContainers` 依赖；`Optional` 哨兵搭车项放弃（`NodeIndex` 构造器上游 internal）。均记入提案决策日志。 +- **验证**：等价性测试 4 项全绿（字节级判定 vs `String.isSwiftSymbol` 全符号表逐条一致、mapped 收集 vs String 收集全等、二分逐行自洽、detach 物化正确）；全量 1341 tests / 256 suites 全绿（前 1337 + 新增 4）；渲染 A/B 与性能见提案落地记录；RV 复测另行闭环。 +- **文档**：[Evolutions/0001-symbol-name-offsetization.md](../Evolutions/0001-symbol-name-offsetization.md)（提案全生命周期）、[TaskReports/2026-08-08-symbol-name-offsetization.md](TaskReports/2026-08-08-symbol-name-offsetization.md)。 +- **对应版本**：`0.14.1` 之后、下一次 bump 之前（`feature/node-store-migration` 分支）。 + +--- + ## 维护约定 1. **每个非平凡批次结束时必须在本文追加/更新一节**（新工作弧新增一节；延续既有弧则在该节 diff --git a/Documentations/Internal/TaskReports/2026-08-08-symbol-name-offsetization.md b/Documentations/Internal/TaskReports/2026-08-08-symbol-name-offsetization.md new file mode 100644 index 00000000..64283f32 --- /dev/null +++ b/Documentations/Internal/TaskReports/2026-08-08-symbol-name-offsetization.md @@ -0,0 +1,41 @@ +# 2026-08-08 SymbolIndexStore 符号名 offset 化（评审与实现，evolution 提案 0001） + +## 问题 + +`MetadataReaderCache` 清退后，RV 五镜像 445 MB 稳态的全景剖析把头号大户定位到 `SymbolIndexStore` ≈ 215 MiB / 96 万分配，其中最大单项是 **49.4 万个驻留符号名 `String`（68.7 MiB）**——而这些名字的原文本就躺在镜像 mmap 的 LINKEDIT 字符串表里（clean 页、不计 footprint），eager 拷贝等于把免费页复制成付费脏页。用户指示做符号名惰性化 / offset 化；本仓库同日建立 Evolution 提案制，方案以提案 0001 落盘并经用户批准（「审核通过，开始实现」）。 + +## 调研 + +调研全文见提案 [0001-symbol-name-offsetization.md](../../Evolutions/0001-symbol-name-offsetization.md)（前期调研一节）。要点： + +- MachOKit 无需改动：`MachOImage.Symbol` 本就带 `nameC: UnsafePointer`，`Symbols64`/`Symbols` 公开 `stringBase`。 +- `storage.symbolTable` 在 `SymbolIndexStore.swift` / `DemangledSymbol.swift` 之外零消费者，内部表示可自由替换。 +- export-trie 的名字是解码产物、不在字符串表里，决定了行格式需要「私有缓冲」这条腿。 +- Swift 6.2 的 `Span`/`UTF8Span` 与 swift-collections 1.6.0 的 `RigidArray` 在设计期被相中——但见下文「偏差」。 + +## 最终方案 + +提案 0001「提议方案 / 详细设计」四步：16 字节 `SymbolRow`（canonical offset + packed name reference）+ `SymbolTable`（mapped 字符串表基址 / 私有字节缓冲双名字来源）；收集循环按 reader 分腿（镜像腿字节级 `isSwiftSymbol`、零 String；文件腿沿用 `readString`、Swift 名进私有缓冲）；`tableRowByName` 退役换名字序 permutation 二分（build 期临时去重字典 freeze 时丢弃）；vend 面按需物化。公开 API 与全部调用点零改动。 + +## 实际执行 + +按方案落地，三处**实施偏差**（已逐条记入提案决策日志）： + +1. **Span 家族不可用**：编译探针证实 `Span` / `Array.span` / `UTF8Span` 运行时可用性为 macOS 26.0+，本包部署下限 macOS 10.15。字节访问层改用 `UnsafeBufferPointer`（同样的 closure-scoped 形态 `withNameBytes(atRow:)`；`memcmp` 序 + 长度 tiebreak 代替 `bytesEqual`；`String(decoding:as: UTF8.self)` 代替 `String(copying:)`，修复语义与原 `String(cString:)` 一致）。 +2. **RigidArray 放弃**：noncopyable 存 class 属性的 borrow 人体工学要 SE-0507（Swift 6.4）。走提案括号里的等价退路——freeze 时精确容量 `Array` 拷贝；因此无需新增 `BasicContainers` 依赖。 +3. **搭车项裁剪**：`Optional` → 哨兵一项放弃（`NodeIndex` 构造器上游 internal，debug 布局带 store tag，为 ~1.6 MB 不值得跨仓库开 API）；`symbolRowsByOffset` 换普通 `Dictionary` 照做；standalone 表统一走私有缓冲表示（不需要提案草绘的 `[String]` 变体）。 + +改动面：`Sources/MachOSymbols/SymbolTable.swift`（新建：`SymbolRow` / `PackedNameReference` / `SymbolTable` / `SymbolTableBuilder` / 字节比较与字节级前缀判定）、`SymbolIndexStore.swift`（Storage 换持 + sweep 分腿 + 二分查询）、`DemangledSymbol.swift`（换持 `SymbolTable` + `offset`/`isExternal`/`name` 快路径，避免 dynamicMember 路径为读 offset 整只物化 `Symbol`）；测试三文件适配 + 新建 `SymbolTableEquivalenceTests.swift`；文档同批（AGENTS.md、提案 0001、演进日志、本报告）。 + +## 验证 + +1. 等价性测试（新增 4 个，全绿）：字节级 `isSwiftSymbol` 与 `String.isSwiftSymbol` 在 Foundation 镜像全符号表逐条一致；mapped 收集与旧 String 收集全等（行数 + last-wins canonical offset）；二分对每行自洽（镜像腿 + 文件腿）+ 负例；detach 物化正确（行数 1、symbol/node 相等）。 +2. 全量 `swift test --skip IntegrationTests`：**1341 tests / 256 suites 全绿**（改动前 1337 + 新增 4，同数吻合）。 +3. 渲染 A/B（`Scripts/run-rendering-ab-verification.py`，baseline `aa91b9b`，`USING_LOCAL_DEPENDENCIES=1`，双侧 sibling 均确认 `fileSystem` 解析）：**96 对全部逐字节一致、0 不一致**（当前系统 dyld cache + iOS 15.5–27.0 七个模拟器 runtime + in-process MachOImage，dump + interface；skip 项均为旧 runtime 本就不含的框架，与上次 A/B 同构）。 +4. 性能与峰值内存（iOS 18.5 模拟器 SwiftUI `interface`，双侧 release 三轮交错，`/usr/bin/time -l`）：wall-clock 中位 72.5s vs 70.0s（散布 61–79s，噪声带内）——持平，输出再次逐字节一致；maxRSS 383–390 → 400–403 MiB——**文件腿构建期峰值 +~15 MiB**，成因是 build 期去重字典的 `String` 键与私有字节缓冲在 freeze 前短暂持有同一批名字字节两份（提案风险段漏算的一层），freeze 后回落；镜像腿无此代价。 +5. RV footprint + heap 复测：落地后由 swift-demangling 会话协调，结果回填提案落地记录（预期堆存活 355 → ~255–285 MiB）。 + +## 偏差与附带发现 + +- 方案核心零偏差；三处实施层偏差见上，均为「设计期特性调研没踩到部署下限」一类——**Swift 6.2 语言特性可用 ≠ 目标部署可用**，`Span` 家族的运行时可用性是 macOS 26+，给低部署下限的库用要么等下限提升、要么 availability 门控（核心路径不可行）。 +- A/B 首启被 Bash 10 分钟超时上限杀过一次（脚本全程 ~20 分钟），改 nohup 脱离 + monitor 收尾；期间发现被杀的是 python 外壳、其 `swift build` 子进程仍在跑并持有 scratch 锁——后续构建会静默排队等锁，勿误判为卡死。 diff --git a/Documentations/README.md b/Documentations/README.md index 94386631..6038fde3 100644 --- a/Documentations/README.md +++ b/Documentations/README.md @@ -8,10 +8,7 @@ Documentation is split by audience. > The `swift-section` executable is a companion CLI, not the outward contract. > Note that `Tests/Projects/SymbolTests` does enable library evolution — that is the test project, > not the library itself. -> Evolution proposals live in [`Evolutions/`](Evolutions/README.md) (status table + numbering there). -> Proposals 0001–0003 (the memory-optimization line) live on the `feature/node-store-migration` -> branch and are not on main yet; [0004](Evolutions/0004-arm64e-signed-vwt-pointer-hardening.md) -> (arm64e signed VWT pointer hardening) is the first proposal landed on main. +> Evolution proposals live in [`Evolutions/`](Evolutions/README.md) (status table + numbering there); the first one is [0001-symbol-name-offsetization.md](Evolutions/0001-symbol-name-offsetization.md). ## External — for library users / other developers @@ -76,6 +73,7 @@ required by `Version.swift`'s bump contract). | [NodeStoreMigrationPlan.md](Internal/NodeStoreMigrationPlan.md) | `SymbolIndexStore` 迁移到 `NodeStore` arena 存储的分期计划与实施记录（Stage 0–5）：12B/节点扁平缓冲、cache-free build sweep、Symbol 表压缩、声明层换持 `NodeReference`、散点 transient demangling；含 Stage 5a 的 override/vtable 回归修复（裸 `[NodeReference: …]` 的 store-identity 陷阱 → `StructuralNodeReferenceKey`）与三源快照验收。 | | [SharedNodeStoreMigration.md](Internal/SharedNodeStoreMigration.md) | **Implemented（2026-08-08）**：三条小 store 流水线（`InternedNodeReferenceCache` / `TypeDefinition` 字段树批量 store / `lateDemangledNode`）汇入上游 0010 `SharedNodeStore` 的迁移设计与落地记录——改动位置、明确不动的部分、验证结果与与方案的差异。 | | [MetadataReaderCacheRetirement.md](Internal/MetadataReaderCacheRetirement.md) | **Implemented（2026-08-08）**：`MetadataReaderCache` 清退——三张 class `Node` 树字典（五镜像实测 ~18.4 万残留 `Node` 的持有主体）换持 `NodeReference` 汇入 `InternedNodeReferenceCache` 作用域 store，公开 API 与 103 处调用点零改动，补按镜像清理 seam；含身份稳定性排查、`MultiPayloadEnumDescriptorCache` 不必同批改键的论证与四轴验证记录（1337 tests 同数全绿 / 渲染 A/B 96 对逐字节一致 / 性能持平 / RV 实景存活 `Node` 207,489 → 44，−99.98%）。 | +| [Evolutions/0001-symbol-name-offsetization.md](Evolutions/0001-symbol-name-offsetization.md) | **提案 0001（Implemented）**：`SymbolIndexStore` 符号名 offset 化——49.4 万个驻留符号名 `String`（68.7 MiB，RV 实测堆内头号大户的最大单项）换成字符串表引用按需物化，`tableRowByName` 字典退役换名字序字节级二分，字节级 `isSwiftSymbol` 判定消掉非 Swift 符号的全部瞬时 String；公开 API 零改动。含 Swift 6.2 Span 家族与 swift-collections 1.6.0 选型裁决（实施时因部署下限改用 `UnsafeBufferPointer` / 精确容量 Array，见决策日志）。 | | [DeclarationModelMemoryFootprint.md](Internal/DeclarationModelMemoryFootprint.md) | NodeStore 迁移后的声明模型内存足迹量测：`TypeDefinition` 1272 字节的逐属性构成（两份 `TypeContextWrapper` 占 74%）、`TypeContextWrapper` 按最大 case `Class` 定尺的原因、`parentContext` 是被当成永久字段的临时值、mini-store 增殖与 `MetadataReaderCache` 仍持 `Node` 树；含四项可回收估算（合计 ~8–10%）与「当前不建议实施、应先剖析其余 90%」的结论。 | | [NodeStoreMigrationOpenIssues.md](Internal/NodeStoreMigrationOpenIssues.md) | `feature/node-store-migration` 的**已确认问题清单**（2026-07-28 两轮审查 + 复核）。第一节是前一轮修复自身的两处缺口（Catalyst 降级只覆盖 framework 形态导致 plain dylib 仍平局、`appendRowIfAbsent` 线性扫描），已于 2026-07-29 闭环并保留成因；第二节起为仍然打开或已裁决的条目：`structuralHash` 每文本节点分配 `String` 与 `ABIKey` 每 key materialize 两条**已按上游设计终审关闭**（0.5.1 保持现状且上游说明不改，见 ReviewAdjudications A1/A2）、`memberSymbols` 改为线性 + 全树比对（**已证实量级可忽略，不是回归**）；`lateDemangledNode` 持锁 demangle 与 `ProtocolConformanceDumper` 的 materialize 分支两条**已于 2026-08-03 闭环**（连同失败名裁决缓存、dump 路径引用化）、build sweep 串行且每符号跨线程往返（上游）、`ABIKey` 每 key materialize，以及四条代码卫生项；两个公开查询 API 的字典键一条**已于 2026-08-03 裁决为不修**（类型级 SPI），rebase 前置事项一条前提已过期但压着两条仍成立的注意事项。逐条注明成因、影响面与「该在哪里修」。与 `Reviews/` 下两份审查记录冲突时以后者为准。 | | [SystemFrameworkRenderingVerification.md](Internal/SystemFrameworkRenderingVerification.md) | 系统框架渲染 A/B 验证流程（大重构必跑）：两个检出对同一批真实输入（归档 dyld cache → 无则当前系统 cache、模拟器 runtime → 无则现有 runtime、当前系统 MachOImage）逐字节比对 dump+interface；入口脚本 `Scripts/run-rendering-ab-verification.py`，含回退规则、踩坑清单（`-p` 消歧、fat 二进制 `-a`、同 boot 会话、scratch 隔离）与 2026-08-03 基线运行记录。 | diff --git a/Sources/MachOSymbols/DemangledSymbol.swift b/Sources/MachOSymbols/DemangledSymbol.swift index b839807c..0641775d 100644 --- a/Sources/MachOSymbols/DemangledSymbol.swift +++ b/Sources/MachOSymbols/DemangledSymbol.swift @@ -3,26 +3,45 @@ import Demangling /// A symbol paired with the handle of its demangled tree. /// /// Compact by construction (NodeStore migration, Stage 3): instead of an -/// inline `Symbol` copy the value stores a row index into the per-image flat +/// inline `Symbol` copy the value stores a row index into the per-image /// symbol table, so the hundreds of thousands of `DemangledSymbol` values -/// vended by `SymbolIndexStore` share one `[Symbol]` buffer and stay at -/// 32 bytes each (table reference + row + `NodeReference`). +/// vended by `SymbolIndexStore` share one `SymbolTable` and stay at +/// 32 bytes each (table reference + row + `NodeReference`). Since evolution +/// proposal 0001 the shared table holds no name `String`s either — `symbol` +/// materializes its name on demand from the table's name source. @dynamicMemberLookup public struct DemangledSymbol: Sendable { - private let symbolTable: [Symbol] + private let symbolTable: SymbolTable private let symbolTableRow: UInt32 public let demangledNode: NodeReference public var symbol: Symbol { - symbolTable[Int(symbolTableRow)] + symbolTable.symbol(atRow: symbolTableRow) + } + + // Concrete fast paths for the members the dynamic-member subscript would + // otherwise serve by building a whole `Symbol` — which materializes the + // name `String` even for a plain `offset` read now that names are + // offset-ized. Values are identical to the key-path route. + + public var offset: Int { + symbolTable.canonicalOffset(atRow: symbolTableRow) + } + + public var isExternal: Bool { + symbolTable.isExternal(atRow: symbolTableRow) + } + + public var name: String { + symbolTable.materializedName(atRow: symbolTableRow) } /// Wraps a standalone symbol in a single-row table. `SymbolIndexStore` /// vends values through the shared-table initializer instead. /// - /// The one-element array is a deliberate trade, not an oversight: storing + /// The one-row table is a deliberate trade, not an oversight: storing /// the `Symbol` inline instead (a two-case payload enum) would avoid this /// allocation, but `Symbol` is itself 32 bytes, so every `DemangledSymbol` /// — including the hundreds of thousands vended through the shared table — @@ -30,19 +49,19 @@ public struct DemangledSymbol: Sendable { /// small allocation on the rarer standalone path is cheaper than widening /// the common one. public init(symbol: Symbol, demangledNode: NodeReference) { - self.symbolTable = [symbol] + self.symbolTable = SymbolTable(standaloneSymbol: symbol) self.symbolTableRow = 0 self.demangledNode = demangledNode } - init(symbolTable: [Symbol], symbolTableRow: UInt32, demangledNode: NodeReference) { + init(symbolTable: SymbolTable, symbolTableRow: UInt32, demangledNode: NodeReference) { self.symbolTable = symbolTable self.symbolTableRow = symbolTableRow self.demangledNode = demangledNode } /// Copies the referenced row into a standalone one-row table, so this - /// value stops retaining the shared per-image buffer. + /// value stops retaining the shared per-image table. /// /// The shared table is the right trade for the hundreds of thousands of /// values a query vends and then drops — they cost 32 bytes each instead @@ -50,11 +69,13 @@ public struct DemangledSymbol: Sendable { /// outlive the query by being stored in the declaration model /// (`Accessor.symbol`, `FunctionDefinition.symbol`, /// `TypeDefinition.deallocatorSymbol` / `destructorSymbol`): a single one - /// of those pins the entire table plus every mangled name in it, which is - /// what `SwiftDeclarationIndexer.removeSubIndexer(_:)` exists to reclaim. + /// of those pins the entire table plus every name byte in it — and, for a + /// `MachOImage` table, keeps vend-time reads against the loaded image's + /// string table alive — which is what + /// `SwiftDeclarationIndexer.removeSubIndexer(_:)` exists to reclaim. /// Measured on SwiftUI (iOS 18.5): 9,872 stored values referenced 9,506 /// distinct rows — 5.1% of a 185,988-row table — so detaching them trades - /// roughly 0.6 MB of small allocations for about 19.9 MB of retention. + /// roughly 0.6 MB of small allocations for the whole table's retention. /// /// Call this when storing a value into a long-lived declaration, not on /// the query path. @@ -67,7 +88,7 @@ public struct DemangledSymbol: Sendable { /// copied its row out. Exposed so the retention regression test can tell /// the two apart without reaching into private storage. package var retainedSymbolTableRowCount: Int { - return symbolTable.count + return symbolTable.rowCount } public subscript(dynamicMember keyPath: KeyPath) -> Value { diff --git a/Sources/MachOSymbols/SymbolIndexStore.swift b/Sources/MachOSymbols/SymbolIndexStore.swift index 293fa8e1..6490c93f 100644 --- a/Sources/MachOSymbols/SymbolIndexStore.swift +++ b/Sources/MachOSymbols/SymbolIndexStore.swift @@ -136,21 +136,24 @@ public final class SymbolIndexStore: SharedCache, @unc /// All `NodeReference` values vended by this storage point into it. let nodeStore: NodeStore - /// Flat symbol table (Stage 3): one row per unique symbol name, - /// holding the canonical (cache-adjusted) offset. Every index below - /// stores 4-byte row indices into this table instead of inline - /// `Symbol` copies, and vended `DemangledSymbol` values share this - /// array's buffer. - let symbolTable: [Symbol] + /// Flat symbol table (Stage 3, offset-ized by evolution proposal + /// 0001): one 16-byte row per unique symbol name, holding the + /// canonical (cache-adjusted) offset plus a packed reference into + /// the table's name source — the image's mmap'd string table for + /// `MachOImage` rows, the table's private byte buffer otherwise. + /// No name `String` is retained; vend paths materialize names on + /// demand. Every index below stores 4-byte row indices into this + /// table, and vended `DemangledSymbol` values share it. Name → row + /// lookup is `symbolTable.row(forName:)`, a byte-level binary + /// search over the table's name-order permutation (the former + /// `tableRowByName` dictionary is build-time-only now). + let symbolTable: SymbolTable /// Parallel to `symbolTable`: the row's demangled root node, or /// `nil` for names the demangler rejected (those still occupy a row /// because `symbolRowsByOffset` references them). let rootNodeIndexByTableRow: [NodeStore.NodeIndex?] - /// Name → table row. Keys share string storage with `symbolTable`. - let tableRowByName: [String: UInt32] - let typeInfoByName: [String: TypeInfo] let globalSymbolRowsByKind: OrderedDictionary @@ -186,7 +189,11 @@ public final class SymbolIndexStore: SharedCache, @unc let symbolRowsByKind: OrderedDictionary - let symbolRowsByOffset: OrderedDictionary + /// Plain `Dictionary`: the only consumer is the keyed lookup in + /// `symbols(for:in:)` — nothing iterates it in order (proposal 0001 + /// rider; the former `OrderedDictionary` paid an ordering table for + /// hundreds of thousands of entries nobody read). + let symbolRowsByOffset: [Int: [UInt32]] let thunkAttributeMembersByKindAndTypeName: [Node.Kind: [String: [ThunkAttributeMember]]] @@ -201,10 +208,10 @@ public final class SymbolIndexStore: SharedCache, @unc /// eviction. private let lateNameStore = SharedNodeStore() - /// Verdict cache over `lateNameStore`, keyed by name like - /// `tableRowByName`: a demangled tree is a pure function of the - /// symbol name, so two symbols at different offsets sharing a name - /// share a tree. A stored `nil` records a name the demangler + /// Verdict cache over `lateNameStore`, keyed by name like the + /// symbol table's own row lookup: a demangled tree is a pure + /// function of the symbol name, so two symbols at different offsets + /// sharing a name share a tree. A stored `nil` records a name the demangler /// rejected — rejection is exactly as deterministic as success, so /// it is cached the same way and never retried (`SharedNodeStore` /// itself throws on failure and caches nothing). @@ -213,16 +220,14 @@ public final class SymbolIndexStore: SharedCache, @unc fileprivate init( nodeStore: NodeStore, - symbolTable: [Symbol], + symbolTable: SymbolTable, rootNodeIndexByTableRow: [NodeStore.NodeIndex?], - tableRowByName: [String: UInt32], - symbolRowsByOffset: OrderedDictionary, + symbolRowsByOffset: [Int: [UInt32]], rowIndexes: consuming RowIndexes ) { self.nodeStore = nodeStore self.symbolTable = symbolTable self.rootNodeIndexByTableRow = rootNodeIndexByTableRow - self.tableRowByName = tableRowByName self.symbolRowsByOffset = symbolRowsByOffset self.typeInfoByName = rowIndexes.typeInfoByName self.globalSymbolRowsByKind = rowIndexes.globalSymbolRowsByKind @@ -287,10 +292,10 @@ public final class SymbolIndexStore: SharedCache, @unc /// Rebuilds the `Symbol` for an offset-table row using the queried /// offset: raw and cache-adjusted keys share one canonical row, so - /// the row's stored offset is not necessarily the queried one. + /// the row's stored offset is not necessarily the queried one. The + /// name is materialized fresh from the table's name source. fileprivate func symbol(atRow row: UInt32, offset queriedOffset: Int) -> Symbol { - let canonicalSymbol = symbolTable[Int(row)] - return Symbol(offset: queriedOffset, name: canonicalSymbol.name, isExternal: canonicalSymbol.isExternal) + return Symbol(offset: queriedOffset, name: symbolTable.materializedName(atRow: row), isExternal: symbolTable.isExternal(atRow: row)) } func demangledSymbol(atRow row: UInt32) -> DemangledSymbol? { @@ -388,26 +393,21 @@ public final class SymbolIndexStore: SharedCache, @unc for machO: MachO, progressContinuation: AsyncStream.Continuation? ) -> Storage? { - var symbolTable: [Symbol] = [] - var tableRowByName: [String: UInt32] = [:] - var symbolRowsByOffset: OrderedDictionary = [:] - - /// The table row a symbol belongs to, plus whether this call created it. - /// - /// Raw and cache-adjusted offset keys share one canonical row; a - /// duplicate name updates the existing row in place (last-wins, like - /// the former name-keyed collection pass). `isNewRow` is what lets - /// `registerRow` skip its duplicate check — see there. - func canonicalRow(for canonicalSymbol: Symbol) -> (row: UInt32, isNewRow: Bool) { - if let existingRow = tableRowByName[canonicalSymbol.name] { - symbolTable[Int(existingRow)] = canonicalSymbol - return (existingRow, false) - } - let newRow = UInt32(symbolTable.count) - symbolTable.append(canonicalSymbol) - tableRowByName[canonicalSymbol.name] = newRow - return (newRow, true) - } + // Reader split (proposal 0001): a MachOImage's symbol names already + // live in the image's mmap'd string table, so its rows reference + // those bytes in place — zero copies, zero retained strings — and + // the Swift-symbol test runs on the raw bytes so non-Swift symbols + // never materialize a name at all. Every other reader (MachOFile, + // whose names are decoded per entry from the file) collects through + // the generic leg below, whose Swift names are appended once into + // the table's private byte buffer. + let machOImage = machO as? MachOImage + let mappedSymbols64 = machOImage?.symbols64 + let mappedSymbols32 = machOImage?.symbols32 + let mappedStringTableBase = mappedSymbols64.map { UnsafeRawPointer($0.stringBase) } ?? mappedSymbols32.map { UnsafeRawPointer($0.stringBase) } + + var tableBuilder = SymbolTableBuilder(mappedStringTableBase: mappedStringTableBase) + var symbolRowsByOffset: [Int: [UInt32]] = [:] // One offset legitimately maps to several rows — distinct symbol names // can share an address — so the bucket stays a list. The *same* row @@ -450,30 +450,67 @@ public final class SymbolIndexStore: SharedCache, @unc } } - for symbol in machO.symbols where symbol.name.isSwiftSymbol && !symbol.nlist.isExternal { - let rawOffset = symbol.offset - var canonicalOffset = rawOffset - if let cache = machO.cache, rawOffset >= 0, machO is MachOFile { - canonicalOffset = rawOffset - cache.mainCacheHeader.sharedRegionStart.cast() + /// Mapped-name collection leg: iterates the image's own symbol + /// sequence so each entry exposes `nameC` (a pointer into the + /// mapped string table). The Swift-symbol test runs byte-level on + /// that pointer; a `String` is materialized only for the symbols + /// that pass it, and only as the build-time dedup key. An image's + /// offsets need no cache adjustment (that path is `MachOFile`-only), + /// so canonical == raw here. + func collectMappedSymbolRows>(_ mappedSymbols: MappedSymbols, stringBase: UnsafeRawPointer) { + for symbol in mappedSymbols { + guard nameBytesHaveSwiftManglingPrefix(symbol.nameC), !symbol.nlist.isExternal else { continue } + let (row, isNewRow) = tableBuilder.canonicalRow( + forName: String(cString: symbol.nameC), + mappedNameByteOffset: UnsafeRawPointer(symbol.nameC) - stringBase, + nameByteLength: strlen(symbol.nameC), + canonicalOffset: symbol.offset, + isExternal: symbol.nlist.isExternal + ) + registerRow(row, rawOffset: symbol.offset, canonicalOffset: symbol.offset, isNewRow: isNewRow) + } + } + + if let mappedSymbols64, let mappedStringTableBase { + collectMappedSymbolRows(mappedSymbols64, stringBase: mappedStringTableBase) + } else if let mappedSymbols32, let mappedStringTableBase { + collectMappedSymbolRows(mappedSymbols32, stringBase: mappedStringTableBase) + } else { + for symbol in machO.symbols where symbol.name.isSwiftSymbol && !symbol.nlist.isExternal { + let rawOffset = symbol.offset + var canonicalOffset = rawOffset + if let cache = machO.cache, rawOffset >= 0, machO is MachOFile { + canonicalOffset = rawOffset - cache.mainCacheHeader.sharedRegionStart.cast() + } + let (row, isNewRow) = tableBuilder.canonicalRow(forName: symbol.name, canonicalOffset: canonicalOffset, isExternal: symbol.nlist.isExternal) + registerRow(row, rawOffset: rawOffset, canonicalOffset: canonicalOffset, isNewRow: isNewRow) } - let (row, isNewRow) = canonicalRow(for: .init(offset: canonicalOffset, name: symbol.name, isExternal: symbol.nlist.isExternal)) - registerRow(row, rawOffset: rawOffset, canonicalOffset: canonicalOffset, isNewRow: isNewRow) } for exportedSymbol in machO.exportedSymbols where exportedSymbol.name.isSwiftSymbol { - if let rawOffset = exportedSymbol.offset, tableRowByName[exportedSymbol.name] == nil { + if let rawOffset = exportedSymbol.offset, tableBuilder.existingRow(forName: exportedSymbol.name) == nil { var canonicalOffset = rawOffset if machO is MachOFile { canonicalOffset += machO.startOffset } - // The `tableRowByName` guard above means this name has no row + // The `existingRow` guard above means this name has no row // yet, so `canonicalRow` always mints one and the duplicate - // check is never needed here. - let (row, isNewRow) = canonicalRow(for: .init(offset: canonicalOffset, name: exportedSymbol.name)) + // check is never needed here. Export-trie names are decoded + // strings with no home in the mapped string table, so they + // take the private-buffer overload on every reader. + let (row, isNewRow) = tableBuilder.canonicalRow(forName: exportedSymbol.name, canonicalOffset: canonicalOffset, isExternal: false) registerRow(row, rawOffset: rawOffset, canonicalOffset: canonicalOffset, isNewRow: isNewRow) } } + // Freezing here drops the build-time dedup dictionary and sorts the + // name-order permutation; the demangle sweep below reads names back + // from the frozen table (a transient `String` per row — the + // demangler's entry point takes a `String` until the upstream + // byte-span entry lands, see proposal 0001's upstream-interface + // section). + let symbolTable = tableBuilder.freeze() + // Single sequential sweep: demangle each symbol cache-free onto a // transient tree, classify on that tree, and intern the result into // the arena builder. Nothing touches the global `NodeCache` and no @@ -481,7 +518,7 @@ public final class SymbolIndexStore: SharedCache, @unc // Stage 1). Indexes accumulate directly in their final row-index // form (Stage 3), so `freeze()` is followed by a plain move into // `Storage`, not a conversion pass. - let totalSymbolCount = symbolTable.count + let totalSymbolCount = symbolTable.rowCount var builder = NodeStoreBuilder() builder.reserveCapacity(expectedSymbolCount: totalSymbolCount) @@ -493,9 +530,8 @@ public final class SymbolIndexStore: SharedCache, @unc progressContinuation?.yield(Progress(currentCount: row, totalCount: totalSymbolCount)) } - let symbol = symbolTable[row] - guard let rootNode = try? demangleAsNodeTransient(symbol.name) else { continue } let symbolTableRow = UInt32(row) + guard let rootNode = try? demangleAsNodeTransient(symbolTable.materializedName(atRow: symbolTableRow)) else { continue } rootNodeIndexByTableRow[row] = builder.intern(rootNode) guard rootNode.isKind(of: .global), let node = rootNode.children.first else { continue } @@ -510,7 +546,7 @@ public final class SymbolIndexStore: SharedCache, @unc } if rootNode.isGlobal { - if !symbol.isExternal { + if !symbolTable.isExternal(atRow: symbolTableRow) { if let result = processGlobalSymbol(symbolTableRow, node: node) { rowIndexes.setGlobalSymbols(for: result) } @@ -529,7 +565,7 @@ public final class SymbolIndexStore: SharedCache, @unc rowIndexes.setMemberSymbols(for: result) } } else if node.kind == .opaqueTypeDescriptor, let firstChild = node.children.first, firstChild.kind == .opaqueReturnTypeOf, let memberSymbol = firstChild.children.first { - if symbol.offset > 0 { + if symbolTable.canonicalOffset(atRow: symbolTableRow) > 0 { rowIndexes.opaqueTypeDescriptorSymbolRowByNodeIndex[builder.intern(memberSymbol)] = symbolTableRow } } else { @@ -545,7 +581,6 @@ public final class SymbolIndexStore: SharedCache, @unc nodeStore: builder.freeze(), symbolTable: symbolTable, rootNodeIndexByTableRow: rootNodeIndexByTableRow, - tableRowByName: tableRowByName, symbolRowsByOffset: symbolRowsByOffset, rowIndexes: rowIndexes ) @@ -904,8 +939,11 @@ public final class SymbolIndexStore: SharedCache, @unc // cross-store split `StructuralNodeReferenceKey` exists to absorb. // // Several symbols sharing one offset is normal (they differ by name) - // and is unaffected: each name resolves to its own row. - if let row = cacheStorage.tableRowByName[symbol.name] { + // and is unaffected: each name resolves to its own row. The lookup + // is a byte-level binary search over the table's name-order + // permutation (proposal 0001) — the name-keyed dictionary it + // replaces retained every symbol name for the storage's lifetime. + if let row = cacheStorage.symbolTable.row(forName: symbol.name) { // The sweep already ran every table row through the demangler // once; a `nil` root records that it rejected this name. The // late path runs the *same* demangler (`NodeStoreBuilder.demangle` diff --git a/Sources/MachOSymbols/SymbolTable.swift b/Sources/MachOSymbols/SymbolTable.swift new file mode 100644 index 00000000..d1b7f4a3 --- /dev/null +++ b/Sources/MachOSymbols/SymbolTable.swift @@ -0,0 +1,334 @@ +import Foundation +import MachOKit + +/// One retained symbol row: 16 bytes. The row holds the canonical +/// (cache-adjusted) offset plus a packed reference to the name's bytes; +/// the name itself is materialized on demand from the table's name source +/// (evolution proposal 0001 — the previous representation retained a +/// 32-byte `Symbol` with an interned `String` per row). +struct SymbolRow { + var canonicalOffset: Int64 + + var packedNameReference: PackedNameReference +} + +/// Packed location of a symbol name's bytes within a `SymbolTable`'s name +/// source. Layout (most significant bit first): 1 bit name source +/// (0 = mapped string table, 1 = private name buffer), 1 bit `isExternal`, +/// 22 bits byte length, 40 bits byte offset into the source. +struct PackedNameReference { + let rawValue: UInt64 + + private static let byteOffsetBitCount: UInt64 = 40 + private static let byteLengthBitCount: UInt64 = 22 + private static let byteOffsetMask: UInt64 = (1 << byteOffsetBitCount) - 1 + private static let byteLengthMask: UInt64 = (1 << byteLengthBitCount) - 1 + private static let privateNameBufferFlag: UInt64 = 1 << 63 + private static let isExternalFlag: UInt64 = 1 << 62 + + init(usesPrivateNameBuffer: Bool, isExternal: Bool, byteOffset: Int, byteLength: Int) { + precondition(byteOffset >= 0 && UInt64(byteOffset) <= Self.byteOffsetMask, "symbol name byte offset exceeds the 40-bit budget") + precondition(byteLength >= 0 && UInt64(byteLength) <= Self.byteLengthMask, "symbol name byte length exceeds the 22-bit budget") + var packed = UInt64(byteOffset) | (UInt64(byteLength) << Self.byteOffsetBitCount) + if isExternal { + packed |= Self.isExternalFlag + } + if usesPrivateNameBuffer { + packed |= Self.privateNameBufferFlag + } + self.rawValue = packed + } + + var usesPrivateNameBuffer: Bool { + rawValue & Self.privateNameBufferFlag != 0 + } + + var isExternal: Bool { + rawValue & Self.isExternalFlag != 0 + } + + var byteOffset: Int { + Int(rawValue & Self.byteOffsetMask) + } + + var byteLength: Int { + Int((rawValue >> Self.byteOffsetBitCount) & Self.byteLengthMask) + } +} + +/// The frozen per-image symbol table: compact rows plus the bytes their +/// names point into. Replaces the former bare `[Symbol]` (whose every row +/// retained a name `String`) and the name-keyed row dictionary. +/// +/// Names come from one of two sources per row: +/// - the image's mmap'd LINKEDIT string table (`MachOImage` rows) — clean +/// pages the kernel can reclaim, read zero-copy at materialization; +/// - the table's own private contiguous byte buffer (`MachOFile` rows and +/// export-trie names, whose decoded strings exist nowhere in mapped +/// memory). +/// +/// Lifetime constraint (accepted in proposal 0001): `mappedStringTableBase` +/// points into the loaded image and dangles if the image is ever unloaded. +/// Name materialization therefore requires the image to stay loaded — the +/// same requirement every other in-process read path already has, but now +/// extending to vended values' `symbol` accessor. +/// +/// The byte access layer uses `UnsafeBufferPointer` rather than +/// `Span`/`UTF8Span`: the Span family is only available at runtime on +/// macOS 26 / iOS 26 and newer, above this package's deployment floor. +/// +/// `@unchecked Sendable`: every stored property is immutable after `init`. +final class SymbolTable: @unchecked Sendable { + /// Base of the image's mmap'd string table; `nil` for tables whose rows + /// all live in `privateNameBuffer`. + let mappedStringTableBase: UnsafeRawPointer? + + /// Name bytes for rows outside the mapped string table — one contiguous + /// allocation appended during the build sweep, exact-capacity at freeze. + let privateNameBuffer: [UInt8] + + let rows: [SymbolRow] + + /// Name-order permutation over `rows`: binary search over it replaces + /// the former `[String: UInt32]` row dictionary. Ordering and equality + /// are exact byte comparisons — mangled names are ASCII by construction, + /// and byte equality is precisely the identity a mangled name needs. + let rowsSortedByName: [UInt32] + + init(mappedStringTableBase: UnsafeRawPointer?, privateNameBuffer: [UInt8], rows: [SymbolRow], rowsSortedByName: [UInt32]) { + self.mappedStringTableBase = mappedStringTableBase + self.privateNameBuffer = privateNameBuffer + self.rows = rows + self.rowsSortedByName = rowsSortedByName + } + + /// A one-row table for a standalone value (`DemangledSymbol(symbol:demangledNode:)` + /// and `detachedFromSharedTable()`): the name is copied into a private + /// buffer so the detached value retains nothing image-scoped. + convenience init(standaloneSymbol symbol: Symbol) { + let nameBytes = Array(symbol.name.utf8) + let packedNameReference = PackedNameReference(usesPrivateNameBuffer: true, isExternal: symbol.isExternal, byteOffset: 0, byteLength: nameBytes.count) + self.init( + mappedStringTableBase: nil, + privateNameBuffer: nameBytes, + rows: [SymbolRow(canonicalOffset: Int64(symbol.offset), packedNameReference: packedNameReference)], + rowsSortedByName: [0] + ) + } + + var rowCount: Int { + rows.count + } + + func canonicalOffset(atRow row: UInt32) -> Int { + Int(rows[Int(row)].canonicalOffset) + } + + func isExternal(atRow row: UInt32) -> Bool { + rows[Int(row)].packedNameReference.isExternal + } + + /// Scoped access to a row's raw name bytes (no terminator). The buffer + /// is only valid inside `body` — for mapped rows it points straight into + /// the image's string table. + func withNameBytes(atRow row: UInt32, _ body: (UnsafeBufferPointer) throws -> Result) rethrows -> Result { + let nameReference = rows[Int(row)].packedNameReference + if nameReference.usesPrivateNameBuffer { + return try privateNameBuffer.withUnsafeBufferPointer { buffer in + try body(UnsafeBufferPointer(rebasing: buffer[nameReference.byteOffset ..< nameReference.byteOffset + nameReference.byteLength])) + } + } else { + let baseAddress = mappedStringTableBase.unsafelyUnwrapped.advanced(by: nameReference.byteOffset).assumingMemoryBound(to: UInt8.self) + return try body(UnsafeBufferPointer(start: baseAddress, count: nameReference.byteLength)) + } + } + + /// Builds the row's name `String` on demand. `String(decoding:)` repairs + /// invalid UTF-8 exactly like the `String(cString:)` the eager + /// representation used, so materialized names match it byte for byte. + func materializedName(atRow row: UInt32) -> String { + withNameBytes(atRow: row) { String(decoding: $0, as: UTF8.self) } + } + + /// The row's `Symbol` with its canonical (cache-adjusted) offset. + func symbol(atRow row: UInt32) -> Symbol { + let symbolRow = rows[Int(row)] + return Symbol(offset: Int(symbolRow.canonicalOffset), name: materializedName(atRow: row), isExternal: symbolRow.packedNameReference.isExternal) + } + + /// The table row holding `name`, via binary search over the name-order + /// permutation. Comparison is on raw bytes, so a hit means exact byte + /// equality with the queried string's UTF-8. + func row(forName name: String) -> UInt32? { + var mutableName = name + return mutableName.withUTF8 { nameBytes -> UInt32? in + var lowerBound = 0 + var upperBound = rowsSortedByName.count + while lowerBound < upperBound { + let middle = (lowerBound + upperBound) / 2 + let candidateRow = rowsSortedByName[middle] + let ordering = withNameBytes(atRow: candidateRow) { compareSymbolNameBytes($0, nameBytes) } + if ordering == 0 { + return candidateRow + } else if ordering < 0 { + lowerBound = middle + 1 + } else { + upperBound = middle + } + } + return nil + } + } +} + +/// Build-time accumulator for `SymbolTable`. Holds a name-keyed dedup +/// dictionary that exists only until `freeze()` — the frozen table answers +/// name lookups by binary search instead. +struct SymbolTableBuilder { + private let mappedStringTableBase: UnsafeRawPointer? + + private var rows: [SymbolRow] = [] + + private var privateNameBuffer: [UInt8] = [] + + /// Build-time dedup only; discarded at freeze. Keys are transient + /// `String`s materialized per Swift-flagged symbol. + private var tableRowByName: [String: UInt32] = [:] + + init(mappedStringTableBase: UnsafeRawPointer?) { + self.mappedStringTableBase = mappedStringTableBase + } + + var rowCount: Int { + rows.count + } + + func existingRow(forName name: String) -> UInt32? { + tableRowByName[name] + } + + /// The table row for a symbol whose name lives in the mapped string + /// table, plus whether this call created it. A duplicate name updates + /// the existing row's offset and external bit in place (last-wins, like + /// the former name-keyed collection pass) and keeps the first + /// occurrence's name reference — the bytes are equal by definition. + mutating func canonicalRow(forName name: String, mappedNameByteOffset: Int, nameByteLength: Int, canonicalOffset: Int, isExternal: Bool) -> (row: UInt32, isNewRow: Bool) { + precondition(mappedStringTableBase != nil, "mapped name references require a mapped string table base") + return canonicalRow( + forName: name, + nameReference: PackedNameReference(usesPrivateNameBuffer: false, isExternal: isExternal, byteOffset: mappedNameByteOffset, byteLength: nameByteLength), + canonicalOffset: canonicalOffset + ) + } + + /// The table row for a symbol whose name has no mapped-memory home + /// (`MachOFile` rows, export-trie names): a new row appends the name's + /// bytes to the private buffer. + mutating func canonicalRow(forName name: String, canonicalOffset: Int, isExternal: Bool) -> (row: UInt32, isNewRow: Bool) { + if let existingRow = tableRowByName[name] { + updateRowInPlace(existingRow, canonicalOffset: canonicalOffset, isExternal: isExternal) + return (existingRow, false) + } + let byteOffset = privateNameBuffer.count + privateNameBuffer.append(contentsOf: name.utf8) + let nameReference = PackedNameReference(usesPrivateNameBuffer: true, isExternal: isExternal, byteOffset: byteOffset, byteLength: privateNameBuffer.count - byteOffset) + return appendRow(forName: name, nameReference: nameReference, canonicalOffset: canonicalOffset) + } + + private mutating func canonicalRow(forName name: String, nameReference: PackedNameReference, canonicalOffset: Int) -> (row: UInt32, isNewRow: Bool) { + if let existingRow = tableRowByName[name] { + updateRowInPlace(existingRow, canonicalOffset: canonicalOffset, isExternal: nameReference.isExternal) + return (existingRow, false) + } + return appendRow(forName: name, nameReference: nameReference, canonicalOffset: canonicalOffset) + } + + private mutating func updateRowInPlace(_ row: UInt32, canonicalOffset: Int, isExternal: Bool) { + let existingNameReference = rows[Int(row)].packedNameReference + rows[Int(row)] = SymbolRow( + canonicalOffset: Int64(canonicalOffset), + packedNameReference: PackedNameReference( + usesPrivateNameBuffer: existingNameReference.usesPrivateNameBuffer, + isExternal: isExternal, + byteOffset: existingNameReference.byteOffset, + byteLength: existingNameReference.byteLength + ) + ) + } + + private mutating func appendRow(forName name: String, nameReference: PackedNameReference, canonicalOffset: Int) -> (row: UInt32, isNewRow: Bool) { + let newRow = UInt32(rows.count) + rows.append(SymbolRow(canonicalOffset: Int64(canonicalOffset), packedNameReference: nameReference)) + tableRowByName[name] = newRow + return (newRow, true) + } + + /// Freezes into the immutable table: exact-capacity copies drop the + /// append-time growth slack, the dedup dictionary is discarded, and the + /// name-order permutation is sorted for binary search. + consuming func freeze() -> SymbolTable { + let frozenRows = exactCapacityCopy(rows) + let frozenNameBuffer = exactCapacityCopy(privateNameBuffer) + var permutation = Array(UInt32(0) ..< UInt32(frozenRows.count)) + let frozenMappedStringTableBase = mappedStringTableBase + frozenNameBuffer.withUnsafeBufferPointer { privateBuffer in + func nameBytes(ofRow row: UInt32) -> UnsafeBufferPointer { + let nameReference = frozenRows[Int(row)].packedNameReference + if nameReference.usesPrivateNameBuffer { + return UnsafeBufferPointer(rebasing: privateBuffer[nameReference.byteOffset ..< nameReference.byteOffset + nameReference.byteLength]) + } else { + let baseAddress = frozenMappedStringTableBase.unsafelyUnwrapped.advanced(by: nameReference.byteOffset).assumingMemoryBound(to: UInt8.self) + return UnsafeBufferPointer(start: baseAddress, count: nameReference.byteLength) + } + } + permutation.sort { compareSymbolNameBytes(nameBytes(ofRow: $0), nameBytes(ofRow: $1)) < 0 } + } + return SymbolTable( + mappedStringTableBase: frozenMappedStringTableBase, + privateNameBuffer: frozenNameBuffer, + rows: frozenRows, + rowsSortedByName: permutation + ) + } + + private func exactCapacityCopy(_ elements: [Element]) -> [Element] { + guard elements.capacity > elements.count else { return elements } + var copy: [Element] = [] + copy.reserveCapacity(elements.count) + copy.append(contentsOf: elements) + return copy + } +} + +/// Three-way byte comparison (`memcmp` order with length tiebreak) — the +/// ordering `rowsSortedByName` is sorted by and searched with. +func compareSymbolNameBytes(_ left: UnsafeBufferPointer, _ right: UnsafeBufferPointer) -> Int { + let commonByteCount = min(left.count, right.count) + if commonByteCount > 0 { + let ordering = Int(memcmp(left.baseAddress.unsafelyUnwrapped, right.baseAddress.unsafelyUnwrapped, commonByteCount)) + if ordering != 0 { + return ordering + } + } + return left.count - right.count +} + +/// Byte-level `isSwiftSymbol` over a C string: mirrors +/// `Demangling.getManglingPrefixLength`'s prefix set exactly (`_T0`, `_$S`, +/// `_$s`, `_$e`, `$S`, `$s`, `$e`, `@__swiftmacro_`) so the build sweep can +/// reject a non-Swift symbol without materializing its name. `strncmp` +/// stops at the terminator, so short names are safe. Pinned equal to +/// `String.isSwiftSymbol` over a full real symbol table by +/// `SymbolTableEquivalenceTests`. +func nameBytesHaveSwiftManglingPrefix(_ nameC: UnsafePointer) -> Bool { + switch nameC.pointee { + case 0x5F: // "_" + return strncmp(nameC, "_T0", 3) == 0 || strncmp(nameC, "_$S", 3) == 0 || strncmp(nameC, "_$s", 3) == 0 || strncmp(nameC, "_$e", 3) == 0 + case 0x24: // "$" + return strncmp(nameC, "$S", 2) == 0 || strncmp(nameC, "$s", 2) == 0 || strncmp(nameC, "$e", 2) == 0 + case 0x40: // "@" + return strncmp(nameC, "@__swiftmacro_", 14) == 0 + default: + return false + } +} diff --git a/Tests/IntegrationTests/MachOSymbols/SymbolIndexStoreBaselineTests.swift b/Tests/IntegrationTests/MachOSymbols/SymbolIndexStoreBaselineTests.swift index d4282f22..37f64817 100644 --- a/Tests/IntegrationTests/MachOSymbols/SymbolIndexStoreBaselineTests.swift +++ b/Tests/IntegrationTests/MachOSymbols/SymbolIndexStoreBaselineTests.swift @@ -35,7 +35,7 @@ final class SymbolIndexStoreBaselineTests: MachOImageTests { do { let storage = try #require(builtStorage) - let symbolTableRowCount = storage.symbolTable.count + let symbolTableRowCount = storage.symbolTable.rowCount let demangledSymbolCount = storage.rootNodeIndexByTableRow.count(where: { $0 != nil }) let symbolsByKindEntryCount = storage.symbolRowsByKind.values.reduce(0) { $0 + $1.count } let memberEntryCount = storage.memberSymbolRowsByKind.values.reduce(0) { partialResult, memberRows in @@ -58,7 +58,7 @@ final class SymbolIndexStoreBaselineTests: MachOImageTests { print("NodeCache leaf delta : \(leafCacheCountAfter - leafCacheCountBefore) (\(leafCacheCountBefore) -> \(leafCacheCountAfter))") print("NodeCache subtree delta : \(subtreeCacheCountAfter - subtreeCacheCountBefore) (\(subtreeCacheCountBefore) -> \(subtreeCacheCountAfter))") print("nodeStore storage : \(nodeStoreBytes / 1_048_576) MB (\(nodeStoreNodeCount) unique nodes)") - print("symbolTable rows : \(symbolTableRowCount) (stride \(MemoryLayout.stride) B, demangled \(demangledSymbolCount))") + print("symbolTable rows : \(symbolTableRowCount) (stride \(MemoryLayout.stride) B, demangled \(demangledSymbolCount))") print("symbolsByKind entries : \(symbolsByKindEntryCount)") print("memberSymbols entries : \(memberEntryCount)") print("methodDescriptorMember entries : \(methodDescriptorEntryCount)") diff --git a/Tests/MachOSymbolsTests/SymbolIndexStoreFixtureTests.swift b/Tests/MachOSymbolsTests/SymbolIndexStoreFixtureTests.swift index 645d0b19..ca303b31 100644 --- a/Tests/MachOSymbolsTests/SymbolIndexStoreFixtureTests.swift +++ b/Tests/MachOSymbolsTests/SymbolIndexStoreFixtureTests.swift @@ -41,10 +41,10 @@ final class SymbolIndexStoreFixtureTests: MachOFileTests, @unchecked Sendable { /// `SymbolIndexStoreBaselineTests`.) @Test func buildPipelineStaysOffGlobalNodeCache() throws { let builtStorage = try #require(SymbolIndexStore.shared.buildStorage(for: machOFile)) - #expect(!builtStorage.symbolTable.isEmpty) + #expect(builtStorage.symbolTable.rowCount > 0) let sampleRow = try #require(builtStorage.rootNodeIndexByTableRow.firstIndex(where: { $0 != nil })) - let sampleSymbolName = builtStorage.symbolTable[sampleRow].name + let sampleSymbolName = builtStorage.symbolTable.materializedName(atRow: UInt32(sampleRow)) let firstTransientTree = try demangleAsNodeTransient(sampleSymbolName) let secondTransientTree = try demangleAsNodeTransient(sampleSymbolName) let firstLeaf = try #require(firstTransientTree.first { $0.children.isEmpty }) @@ -59,14 +59,15 @@ final class SymbolIndexStoreFixtureTests: MachOFileTests, @unchecked Sendable { let storage = try storage var checkedCount = 0 var mismatchCount = 0 - for (row, symbol) in storage.symbolTable.enumerated() { + for row in 0 ..< storage.symbolTable.rowCount { guard let rootNodeIndex = storage.rootNodeIndexByTableRow[row] else { continue } + let symbolName = storage.symbolTable.materializedName(atRow: UInt32(row)) let reference = storage.nodeStore.reference(at: rootNodeIndex) - let expected = try demangleAsNode(symbol.name, internsSubtrees: false).print(using: .default) + let expected = try demangleAsNode(symbolName, internsSubtrees: false).print(using: .default) if reference.print(using: .default) != expected { mismatchCount += 1 if mismatchCount <= 3 { - Issue.record("Store print mismatch for \(symbol.name)") + Issue.record("Store print mismatch for \(symbolName)") } } checkedCount += 1 @@ -83,6 +84,38 @@ final class SymbolIndexStoreFixtureTests: MachOFileTests, @unchecked Sendable { @Test func compactValueLayouts() { #expect(MemoryLayout.stride <= 32) #expect(MemoryLayout.stride <= 32) + #expect(MemoryLayout.stride == 16) + } + + /// File-leg counterpart of `SymbolTableImageEquivalenceTests`: a + /// `MachOFile` table's names live in the private byte buffer, so this + /// pins the same binary-search and vend-materialization behavior over + /// that name source. + @Test func fileLegBinarySearchAndDetachedMaterializationAgree() throws { + let storage = try storage + let symbolTable = storage.symbolTable + try #require(symbolTable.rowCount > 0) + + var mismatchCount = 0 + for row in 0 ..< symbolTable.rowCount { + let materializedName = symbolTable.materializedName(atRow: UInt32(row)) + if symbolTable.row(forName: materializedName) != UInt32(row) { + mismatchCount += 1 + if mismatchCount <= 3 { + Issue.record("binary search failed to find row \(row) (\(materializedName))") + } + } + } + #expect(mismatchCount == 0) + + let sampleRow = try #require(storage.rootNodeIndexByTableRow.firstIndex(where: { $0 != nil })) + let vended = try #require(storage.demangledSymbol(atRow: UInt32(sampleRow))) + #expect(vended.retainedSymbolTableRowCount == symbolTable.rowCount) + let detached = vended.detachedFromSharedTable() + #expect(detached.retainedSymbolTableRowCount == 1) + #expect(detached.symbol == vended.symbol) + #expect(detached.demangledNode == vended.demangledNode) + #expect(detached.symbol.name == symbolTable.materializedName(atRow: UInt32(sampleRow))) } /// Raw and cache-adjusted offset keys share one canonical table row, so @@ -98,7 +131,7 @@ final class SymbolIndexStoreFixtureTests: MachOFileTests, @unchecked Sendable { #expect(queried.count == rows.count) #expect(queried.allSatisfy { $0.offset == offset }) for (queriedSymbol, row) in zip(queried, rows) { - #expect(queriedSymbol.name == storage.symbolTable[Int(row)].name) + #expect(queriedSymbol.name == storage.symbolTable.materializedName(atRow: row)) } checkedOffsetCount += 1 } @@ -150,7 +183,7 @@ final class SymbolIndexStoreFixtureTests: MachOFileTests, @unchecked Sendable { for (nodeIndex, expectedRow) in storage.opaqueTypeDescriptorSymbolRowByNodeIndex { let keyReference = storage.nodeStore.reference(at: nodeIndex) let queried = try #require(SymbolIndexStore.shared.opaqueTypeDescriptorSymbol(for: keyReference.materialize(), in: machOFile)) - #expect(queried.symbol == storage.symbolTable[Int(expectedRow)]) + #expect(queried.symbol == storage.symbolTable.symbol(atRow: expectedRow)) } } @@ -159,9 +192,10 @@ final class SymbolIndexStoreFixtureTests: MachOFileTests, @unchecked Sendable { @Test func demangledNodeAndReferenceAgree() throws { let storage = try storage var checkedCount = 0 - for (row, symbol) in storage.symbolTable.enumerated() { + for row in 0 ..< storage.symbolTable.rowCount { guard checkedCount < 200 else { break } guard let rootNodeIndex = storage.rootNodeIndexByTableRow[row] else { continue } + let symbol = storage.symbolTable.symbol(atRow: UInt32(row)) let reference = storage.nodeStore.reference(at: rootNodeIndex) let materialized = try #require(SymbolIndexStore.shared.demangledNode(for: symbol, in: machOFile)) #expect(reference.structurallyEquals(materialized)) @@ -197,7 +231,7 @@ final class SymbolIndexStoreFixtureTests: MachOFileTests, @unchecked Sendable { @Test func rejectedLateNameCachesItsFailure() throws { let storage = try storage let bogusSymbol = Symbol(offset: -1, name: "$s999999999999") - #expect(storage.tableRowByName[bogusSymbol.name] == nil) + #expect(storage.symbolTable.row(forName: bogusSymbol.name) == nil) #expect(SymbolIndexStore.shared.demangledNodeReference(for: bogusSymbol, in: machOFile) == nil) let verdict = try #require(storage.lateDemangleVerdictForTesting(forName: bogusSymbol.name)) @@ -215,12 +249,12 @@ final class SymbolIndexStoreFixtureTests: MachOFileTests, @unchecked Sendable { let storage = try storage let demangledRow = try #require(storage.rootNodeIndexByTableRow.firstIndex(where: { $0 != nil })) - let demangledSymbol = storage.symbolTable[demangledRow] + let demangledSymbol = storage.symbolTable.symbol(atRow: UInt32(demangledRow)) _ = try #require(SymbolIndexStore.shared.demangledNodeReference(for: demangledSymbol, in: machOFile)) #expect(storage.lateDemangleVerdictForTesting(forName: demangledSymbol.name) == nil) if let rejectedRow = storage.rootNodeIndexByTableRow.firstIndex(where: { $0 == nil }) { - let rejectedSymbol = storage.symbolTable[rejectedRow] + let rejectedSymbol = storage.symbolTable.symbol(atRow: UInt32(rejectedRow)) #expect(SymbolIndexStore.shared.demangledNodeReference(for: rejectedSymbol, in: machOFile) == nil) #expect(storage.lateDemangleVerdictForTesting(forName: rejectedSymbol.name) == nil) } diff --git a/Tests/MachOSymbolsTests/SymbolTableEquivalenceTests.swift b/Tests/MachOSymbolsTests/SymbolTableEquivalenceTests.swift new file mode 100644 index 00000000..32f5e40c --- /dev/null +++ b/Tests/MachOSymbolsTests/SymbolTableEquivalenceTests.swift @@ -0,0 +1,101 @@ +import Foundation +import Testing +import MachOKit +@_spi(Internals) import Demangling +@_spi(Internals) @testable import MachOSymbols +@_spi(Internals) import MachOCaches +@testable import MachOTestingSupport +import MachOFixtureSupport + +/// Equivalence pins for the offset-ized symbol table (evolution proposal +/// 0001), run against a real in-process image so the mapped-string-table +/// leg — the one the fixture-file suite cannot exercise — is covered end +/// to end: +/// +/// 1. the byte-level Swift-symbol test agrees with `String.isSwiftSymbol` +/// on every entry of a real symbol table (the byte version re-states the +/// demangler's prefix list and would silently diverge if that list ever +/// grew); +/// 2. the collection sweep retains exactly the rows the former +/// `String`-keyed collection pass retained, with the same last-wins +/// canonical offsets; +/// 3. binary search over the name-order permutation answers every row's own +/// materialized name with that row (the dictionary it replaced was +/// keyed on those exact strings). +final class SymbolTableImageEquivalenceTests: MachOImageTests, @unchecked Sendable { + @Test func byteLevelSwiftSymbolCheckMatchesStringCheck() throws { + var checkedCount = 0 + var mismatchCount = 0 + func check(nameC: UnsafePointer, name: String) { + if nameBytesHaveSwiftManglingPrefix(nameC) != name.isSwiftSymbol { + mismatchCount += 1 + if mismatchCount <= 3 { + Issue.record("byte-level Swift-symbol check mismatch for \(name)") + } + } + checkedCount += 1 + } + if let symbols64 = machOImage.symbols64 { + for symbol in symbols64 { + check(nameC: symbol.nameC, name: symbol.name) + } + } else if let symbols32 = machOImage.symbols32 { + for symbol in symbols32 { + check(nameC: symbol.nameC, name: symbol.name) + } + } + #expect(mismatchCount == 0) + #expect(checkedCount > 0) + } + + @Test func mappedCollectionMatchesStringBasedCollection() throws { + let storage = try #require(SymbolIndexStore.shared.storage(in: machOImage)) + let symbolTable = storage.symbolTable + + // The pre-0001 collection pass, re-run through the reader-generic + // `String` surface: last-wins offset per unique Swift name, then + // export-trie names for rows the symbol table did not produce. + var expectedOffsetByName: [String: Int] = [:] + for symbol in machOImage.symbols where symbol.name.isSwiftSymbol && !symbol.nlist.isExternal { + expectedOffsetByName[symbol.name] = symbol.offset + } + for exportedSymbol in machOImage.exportedSymbols where exportedSymbol.name.isSwiftSymbol { + if let rawOffset = exportedSymbol.offset, expectedOffsetByName[exportedSymbol.name] == nil { + expectedOffsetByName[exportedSymbol.name] = rawOffset + } + } + + #expect(symbolTable.rowCount == expectedOffsetByName.count) + var mismatchCount = 0 + for row in 0 ..< symbolTable.rowCount { + let materializedName = symbolTable.materializedName(atRow: UInt32(row)) + if expectedOffsetByName[materializedName] != symbolTable.canonicalOffset(atRow: UInt32(row)) { + mismatchCount += 1 + if mismatchCount <= 3 { + Issue.record("row \(row) (\(materializedName)) diverges from the String-based collection pass") + } + } + } + #expect(mismatchCount == 0) + } + + @Test func binarySearchAnswersEveryRowByItsOwnName() throws { + let storage = try #require(SymbolIndexStore.shared.storage(in: machOImage)) + let symbolTable = storage.symbolTable + try #require(symbolTable.rowCount > 0) + + var mismatchCount = 0 + for row in 0 ..< symbolTable.rowCount { + let materializedName = symbolTable.materializedName(atRow: UInt32(row)) + if symbolTable.row(forName: materializedName) != UInt32(row) { + mismatchCount += 1 + if mismatchCount <= 3 { + Issue.record("binary search failed to find row \(row) (\(materializedName))") + } + } + } + #expect(mismatchCount == 0) + #expect(symbolTable.row(forName: "$sNotARealSymbolName999AtAll") == nil) + #expect(symbolTable.row(forName: "") == nil) + } +} diff --git a/Tests/SwiftInterfaceTests/SymbolTableRetentionTests.swift b/Tests/SwiftInterfaceTests/SymbolTableRetentionTests.swift index 5c87760c..5f4925f9 100644 --- a/Tests/SwiftInterfaceTests/SymbolTableRetentionTests.swift +++ b/Tests/SwiftInterfaceTests/SymbolTableRetentionTests.swift @@ -40,7 +40,7 @@ final class SymbolTableRetentionTests: MachOFileTests, @unchecked Sendable { _ = try await builder.printRoot() let storage = try #require(SymbolIndexStore.shared.storage(in: machOFile)) - let sharedTableRowCount = storage.symbolTable.count + let sharedTableRowCount = storage.symbolTable.rowCount // Otherwise the assertion below cannot distinguish a detached value // from a shared one. try #require(sharedTableRowCount > 1) From 30434b9284f4b16e98ff87b49c88a29ddd33cf38 Mon Sep 17 00:00:00 2001 From: Mx-Iris Date: Sat, 8 Aug 2026 23:18:51 +0800 Subject: [PATCH 45/77] docs: record the real-world re-measure closing evolution 0001 RuntimeViewer five-image steady-state re-measure (coordinated by the swift-demangling session, RuntimeViewer recompiled with zero source changes, sibling resolution verified): footprint 445 -> 322 MB (-28%, better than expected), live heap 355 -> 283.3 MiB (inside the predicted band), SymbolIndexStore cluster 214.6 -> 120.9 MiB, retained symbol-name StringStorage gone as designed (784k/84.2 MiB -> 356k/31.3 MiB), indexing transient peak 893 -> 808 MB, with no regression in any other subsystem's attribution. Backfilled into proposal 0001's implementation record and decision log, the task report, and the evolution-log section; every step of the proposal's landing plan is now closed. --- .../Evolutions/0001-symbol-name-offsetization.md | 7 ++++++- Documentations/Internal/ProjectEvolutionLog.md | 2 +- .../TaskReports/2026-08-08-symbol-name-offsetization.md | 2 +- 3 files changed, 8 insertions(+), 3 deletions(-) diff --git a/Documentations/Evolutions/0001-symbol-name-offsetization.md b/Documentations/Evolutions/0001-symbol-name-offsetization.md index ab6994f7..6ac1b199 100644 --- a/Documentations/Evolutions/0001-symbol-name-offsetization.md +++ b/Documentations/Evolutions/0001-symbol-name-offsetization.md @@ -180,7 +180,11 @@ final class SymbolTable: @unchecked Sendable { 2. **全量套件**：`swift test --skip IntegrationTests` **1341 tests / 256 suites 全绿**（改动前 1337 + 新增 4，同数吻合；快路径补丁后复跑同样全绿）。 3. **渲染 A/B**（`Scripts/run-rendering-ab-verification.py`，baseline `aa91b9b`，双侧 `USING_LOCAL_DEPENDENCIES=1` 且 sibling 均验证为 `fileSystem` 解析）：**96 对全部逐字节一致、0 不一致**（当前系统 dyld cache + iOS 15.5–27.0 七个模拟器 runtime + in-process MachOImage，dump + interface；skip 项均为旧 runtime 本就不含的框架，与上一次 A/B 同构）。 4. **性能与峰值内存**（iOS 18.5 模拟器 SwiftUI `interface`，双侧 release 三轮交错，`/usr/bin/time -l`）：wall-clock 中位 **72.5s（基线）vs 70.0s（候选）**，散布 61–79s，差异在噪声带内——持平；两侧输出再次逐字节一致。maxRSS 基线 383–390 MiB vs 候选 400–403 MiB——**文件腿构建期峰值 +~15 MiB（+4%）**：build 期去重字典retain 的 `String` 键与私有字节缓冲在 freeze 前短暂持有同一批名字字节的两份拷贝（提案「构建峰值」风险段只算了字典本身、漏了这层字节重复），freeze 丢弃字典后回落。镜像腿不付此代价（行直指 mapped 字符串表、无字节复制），而 RV 的目标指标是**稳态**驻留，最终以落地步骤 8 的 RV 复测为裁判。 -5. **RV footprint + heap 复测**（落地步骤 8）：落地后由 swift-demangling 会话协调 RuntimeViewer 重编复测，结果回填此处（预期堆存活 355 → ~255–285 MiB）。 +5. **RV footprint + heap 复测**（落地步骤 8，2026-08-08 同日闭环；RuntimeViewer 重编零源码改动，环境经对面核对——sibling 解析、swift-demangling @ `9464265`、`USING_LOCAL_DEPENDENCIES=1`、无远端回退）：**全项落在或好于预期带**。 + - 干净跑绝对数：footprint 稳态 **445 → 322 MB（−123 MB，−28%）**，好于对面 350–375 的预期（MALLOC_SMALL 脏页 306 → 237，另有 65 MB reclaimable 在归还路上；MALLOC_LARGE 106 → 55）；堆存活 **355 → 283.3 MiB**（预期带 ~255–285 内），分配数 −33 万；索引期瞬态峰值 **893 → 808 MB**——sweep 期非 Swift 符号的 String churn 被字节级判定砍掉，在峰值上可见。 + - heap 按类对照：StringStorage **784,254 个 / 84.2 MiB → 356,094 个 / 31.3 MiB**（49.4 万条驻留符号名如预期消失）；5 × `Dictionary` 名表 20 MiB 从堆顶消失，代之 offset 键表 12.9 MiB；`[Symbol]` 24.5 MiB → `[SymbolRow]` 10.4 MiB（≈ 68 万行 × 16 字节，与行格式吻合）。 + - logging 跑归属复核：`SymbolIndexStore` 簇 **214.6 → 120.9 MiB**（预期 120–145 带内）；全进程 StringStorage 分配 96.8 → 36.4 MiB；无回归旁证——MetadataReader 1.4 MiB 不变、Demangling 22.6 不变、ObjC 索引 33.8 不变、NIO/Rx/声明模型持平。 + - RV 五镜像稳态累计曲线：470–480 →（MetadataReaderCache 清退）~450 →（本案）**322 MB**。 ## 决策日志 @@ -194,3 +198,4 @@ final class SymbolTable: @unchecked Sendable { | 2026-08-08 | 实施偏差：RigidArray 放弃 | 同一部署下限问题的连带裁决：`RigidArray`（noncopyable）存进 class 属性后的 borrow 人体工学要到 SE-0507（Swift 6.4）才齐。改用提案括号里本就给出的等价退路——freeze 时精确容量 `Array` 拷贝（容量已精确者跳过拷贝），且因此**无需**给 `Package.swift` 加 `BasicContainers` 依赖、无需提 swift-collections 版本。 | | 2026-08-08 | 实施偏差：搭车项裁剪 | `rootNodeIndexByTableRow` 的 `Optional` → `UInt32.max` 哨兵一项**放弃**：`NodeStore.NodeIndex` 的构造器是上游 internal（debug 布局还带 store tag），从原始 `UInt32` 重建索引需要新的上游 API，为 ~1.6 MB 不值得跨仓库开口子。`symbolRowsByOffset` 换普通 `Dictionary` 一项照做。另一实现细节：standalone `SymbolTable` 统一走私有字节缓冲表示（提案草绘的 `[String]` 变体不再需要——单一表示，读取路径零分支）。 | | 2026-08-08 | Implemented + 收尾判断 | 验证结果见「落地记录」（1341 全绿、A/B 96 对逐字节一致、性能持平；文件腿构建期峰值 +4% 如实记录，RV 稳态复测为最终裁判、结果回填）。收尾判断：**不另写实现说明**——「代码看不出来的决策」（mapped 指针生命周期约束、名字来源双腿、Span 不可用的原因）已分别落在 `SymbolTable` 类文档、AGENTS.md「Symbol indexing」段与本提案决策日志，另立一篇只会是复述；**不登记新术语表**——本项目无 `Glossary.md`（项目现状即约定），「offset 化 / 名字来源 / permutation 二分」均在首次出现处展开。 | +| 2026-08-08 | RV 复测闭环 | 落地步骤 8 完成（对面协调，同日）：footprint 稳态 445 → 322 MB（−28%，好于预期）、堆存活 355 → 283.3 MiB（预期带内）、`SymbolIndexStore` 簇 214.6 → 120.9 MiB、StringStorage −42.8 万个/−52.9 MiB，无回归旁证。详数见「落地记录」第 5 条。本提案全部落地步骤就此闭环。 | diff --git a/Documentations/Internal/ProjectEvolutionLog.md b/Documentations/Internal/ProjectEvolutionLog.md index 4b063c04..5a005140 100644 --- a/Documentations/Internal/ProjectEvolutionLog.md +++ b/Documentations/Internal/ProjectEvolutionLog.md @@ -771,7 +771,7 @@ - **动机**：RV 五镜像 445 MB 稳态剖析定位 `SymbolIndexStore` ≈ 215 MiB 为堆内头号大户，最大单项是 49.4 万个驻留符号名 `String`（68.7 MiB）——原文本就在镜像 mmap 的 LINKEDIT 字符串表（clean 页），eager 拷贝把免费页复制成付费脏页。方案以提案 0001 落盘、经用户批准后实施。 - **落地**：`SymbolTable`（16 字节 `SymbolRow` = canonical offset + packed name reference；名字来源双腿——镜像行零拷贝直指 mapped 字符串表、文件行与 export-trie 名进私有连续字节缓冲）；收集循环 reader 分腿（镜像腿字节级 `isSwiftSymbol`，非 Swift 符号零分配；文件腿沿用 `readString`）；`tableRowByName` 退役换名字序 permutation 字节级二分（build 期临时去重字典 freeze 丢弃 + 精确容量拷贝）；vend 面按需物化（`DemangledSymbol` 加 `offset`/`isExternal`/`name` 快路径）。公开 API 与全部调用点零改动。 - **关键决策 / 实施偏差**：`Span`/`UTF8Span` 运行时可用性 macOS 26+、本包部署下限 10.15 → 字节访问层改 `UnsafeBufferPointer`（closure-scoped 形态不变）；`RigidArray` 的 class 属性 borrow 人体工学要 SE-0507 → 精确容量 `Array` 拷贝、免掉 `BasicContainers` 依赖；`Optional` 哨兵搭车项放弃（`NodeIndex` 构造器上游 internal）。均记入提案决策日志。 -- **验证**：等价性测试 4 项全绿（字节级判定 vs `String.isSwiftSymbol` 全符号表逐条一致、mapped 收集 vs String 收集全等、二分逐行自洽、detach 物化正确）；全量 1341 tests / 256 suites 全绿（前 1337 + 新增 4）；渲染 A/B 与性能见提案落地记录；RV 复测另行闭环。 +- **验证**：等价性测试 4 项全绿（字节级判定 vs `String.isSwiftSymbol` 全符号表逐条一致、mapped 收集 vs String 收集全等、二分逐行自洽、detach 物化正确）；全量 1341 tests / 256 suites 全绿（前 1337 + 新增 4）；渲染 A/B 96 对逐字节一致、性能持平（详见提案落地记录）；RV 五镜像实景复测同日闭环——footprint 稳态 **445 → 322 MB（−28%）**、堆存活 355 → 283.3 MiB、`SymbolIndexStore` 簇 214.6 → 120.9 MiB、驻留符号名 StringStorage 如预期消失（−42.8 万个），无回归旁证。 - **文档**：[Evolutions/0001-symbol-name-offsetization.md](../Evolutions/0001-symbol-name-offsetization.md)（提案全生命周期）、[TaskReports/2026-08-08-symbol-name-offsetization.md](TaskReports/2026-08-08-symbol-name-offsetization.md)。 - **对应版本**：`0.14.1` 之后、下一次 bump 之前（`feature/node-store-migration` 分支）。 diff --git a/Documentations/Internal/TaskReports/2026-08-08-symbol-name-offsetization.md b/Documentations/Internal/TaskReports/2026-08-08-symbol-name-offsetization.md index 64283f32..dd2fa57e 100644 --- a/Documentations/Internal/TaskReports/2026-08-08-symbol-name-offsetization.md +++ b/Documentations/Internal/TaskReports/2026-08-08-symbol-name-offsetization.md @@ -33,7 +33,7 @@ 2. 全量 `swift test --skip IntegrationTests`：**1341 tests / 256 suites 全绿**（改动前 1337 + 新增 4，同数吻合）。 3. 渲染 A/B（`Scripts/run-rendering-ab-verification.py`，baseline `aa91b9b`，`USING_LOCAL_DEPENDENCIES=1`，双侧 sibling 均确认 `fileSystem` 解析）：**96 对全部逐字节一致、0 不一致**（当前系统 dyld cache + iOS 15.5–27.0 七个模拟器 runtime + in-process MachOImage，dump + interface；skip 项均为旧 runtime 本就不含的框架，与上次 A/B 同构）。 4. 性能与峰值内存（iOS 18.5 模拟器 SwiftUI `interface`，双侧 release 三轮交错，`/usr/bin/time -l`）：wall-clock 中位 72.5s vs 70.0s（散布 61–79s，噪声带内）——持平，输出再次逐字节一致；maxRSS 383–390 → 400–403 MiB——**文件腿构建期峰值 +~15 MiB**，成因是 build 期去重字典的 `String` 键与私有字节缓冲在 freeze 前短暂持有同一批名字字节两份（提案风险段漏算的一层），freeze 后回落；镜像腿无此代价。 -5. RV footprint + heap 复测：落地后由 swift-demangling 会话协调，结果回填提案落地记录（预期堆存活 355 → ~255–285 MiB）。 +5. RV footprint + heap 复测（同日闭环，对面协调，RuntimeViewer 重编零源码改动）：footprint 稳态 **445 → 322 MB（−28%，好于预期）**、堆存活 **355 → 283.3 MiB**（预期带内）、`SymbolIndexStore` 簇 **214.6 → 120.9 MiB**（预期 120–145 带内）、StringStorage 784,254 个 / 84.2 MiB → 356,094 个 / 31.3 MiB、索引期瞬态峰值 893 → 808 MB（字节级判定砍掉的 sweep churn 可见）；无回归旁证（MetadataReader / Demangling / ObjC 索引 / 声明模型全持平）。详数见提案落地记录第 5 条。RV 五镜像稳态累计曲线：470–480 → ~450 → **322 MB**。 ## 偏差与附带发现 From 0c8035da97fc1bc43fae20c6b5cfa97ed05119c1 Mon Sep 17 00:00:00 2001 From: Mx-Iris Date: Sun, 9 Aug 2026 11:40:32 +0800 Subject: [PATCH 46/77] docs: record the long-run RuntimeViewer re-check and post-landing heap layout MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit First-hand data from the RuntimeViewer session (11-hour uptime, real browsing load): live heap 285 MiB matches the clean-run 283 MiB — no long-run drift; footprint 350 MB of which 68 MB is idle reclaimable (~282 MB true dirty). The recurring 800+ MB spikes the user observed are attributed to on-demand indexing sweep transients (28 concurrent workers' temporary buffers, released on completion), not a leak; sweep throttling/batching is noted as an un-adopted candidate. The post-landing heap top (SwiftDeclaration model 41.3 MiB now first; the [UInt32] small-array cluster 38.8 MiB including the new offset-keyed table) is recorded as measured input for the bucket-flattening follow-up already named in the proposal's non-goals. --- Documentations/Evolutions/0001-symbol-name-offsetization.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/Documentations/Evolutions/0001-symbol-name-offsetization.md b/Documentations/Evolutions/0001-symbol-name-offsetization.md index 6ac1b199..2ab3841a 100644 --- a/Documentations/Evolutions/0001-symbol-name-offsetization.md +++ b/Documentations/Evolutions/0001-symbol-name-offsetization.md @@ -185,6 +185,8 @@ final class SymbolTable: @unchecked Sendable { - heap 按类对照：StringStorage **784,254 个 / 84.2 MiB → 356,094 个 / 31.3 MiB**（49.4 万条驻留符号名如预期消失）；5 × `Dictionary` 名表 20 MiB 从堆顶消失，代之 offset 键表 12.9 MiB；`[Symbol]` 24.5 MiB → `[SymbolRow]` 10.4 MiB（≈ 68 万行 × 16 字节，与行格式吻合）。 - logging 跑归属复核：`SymbolIndexStore` 簇 **214.6 → 120.9 MiB**（预期 120–145 带内）；全进程 StringStorage 分配 96.8 → 36.4 MiB；无回归旁证——MetadataReader 1.4 MiB 不变、Demangling 22.6 不变、ObjC 索引 33.8 不变、NIO/Rx/声明模型持平。 - RV 五镜像稳态累计曲线：470–480 →（MetadataReaderCache 清退）~450 →（本案）**322 MB**。 + - 长跑复核（2026-08-09，RV 会话一手数据，11 小时 uptime 真实浏览负载）：堆存活 **285 MiB / 158 万分配，与干净跑 283 MiB 重合——长跑无漂移**；footprint 350 MB（其中 68 MB reclaimable 闲时未归还，真实脏页 ~282 MB）。用户此前观察到的「反复飙 800+ MB 后回落」归属确认：按需索引 sweep 的瞬态峰（28 并发工人的临时缓冲，完即释放），非泄漏——sweep 限流/分批换峰值是可立项的候选，暂未拍板。 + - 落地后堆格局（后续优化的实测输入）：`SwiftDeclaration` 声明模型 41.3 MiB 成为新头部（ExtensionDefinition 28,225 × 640 B ≈ 17.2、TypeDefinition 11,985 × 1.25 KB ≈ 14.6）；`[UInt32]` 小数组簇 38.8 MiB / 45 万个（含本案引入的 offset 键表 `Dictionary` 12.2）——正是「非目标」一节点名的桶扁平化候选（下一篇提案的对象）；MSS 解析结构 33.4（ProtocolConformance 数组 20.6 打头）；MachOSymbols 残余 11.0（`[SymbolRow]` 9.9）。Demangling 33.2 与 MetadataReader 1.4 维持既定形态，无回归。 ## 决策日志 From 3bfe049aa7d7003bffb933eaa22ba19d1f83636c Mon Sep 17 00:00:00 2001 From: Mx-Iris Date: Sun, 9 Aug 2026 12:14:03 +0800 Subject: [PATCH 47/77] docs: draft evolution proposals 0002 (declaration-model descriptor slimming) and 0003 (symbol-row bucket flattening) --- ...2-declaration-model-descriptor-slimming.md | 172 ++++++++++++++++++ .../0003-symbol-row-bucket-flattening.md | 108 +++++++++++ Documentations/Evolutions/README.md | 2 + Documentations/README.md | 2 + 4 files changed, 284 insertions(+) create mode 100644 Documentations/Evolutions/0002-declaration-model-descriptor-slimming.md create mode 100644 Documentations/Evolutions/0003-symbol-row-bucket-flattening.md diff --git a/Documentations/Evolutions/0002-declaration-model-descriptor-slimming.md b/Documentations/Evolutions/0002-declaration-model-descriptor-slimming.md new file mode 100644 index 00000000..6e652f38 --- /dev/null +++ b/Documentations/Evolutions/0002-declaration-model-descriptor-slimming.md @@ -0,0 +1,172 @@ +# 0002 - 声明模型 descriptor 化：TypeDefinition / ExtensionDefinition / ProtocolDefinition 不再驻留急切解析的胖 wrapper + +- **状态**: In Review +- **作者**: JH +- **创建日期**: 2026-08-09 +- **最后更新**: 2026-08-09 +- **所属愿景**: 无 +- **关联提案**: [0001](0001-symbol-name-offsetization.md)（方法论同构：「驻留只留定位信息，重内容按需物化」；其落地记录里 RV 复测的落地后堆格局是本案的立项输入） +- **实现分支 / PR**: `feature/node-store-migration`（拟，待批准后实施） +- **配套文档**: 先行账本 [DeclarationModelMemoryFootprint.md](../Internal/DeclarationModelMemoryFootprint.md)（2026-07-25 逐属性量测——本案的精确数字来源；其「当前不建议实施」结论的前提已被 0001 消解，见「前期调研」）；落地时按收尾判断决定是否另写实现说明 + +## 摘要 + +0001 落地后 RuntimeViewer 五镜像 322 MB 稳态的堆格局里，新头部是两个同根的簇：`SwiftDeclaration` 声明模型 41.3 MiB（`ExtensionDefinition` 28,225 × 640 B ≈ 17.2、`TypeDefinition` 11,985 × 1272 B ≈ 14.6）与 MachOSwiftSection 解析结构 33.4 MiB（`ProtocolConformance` 的 `[ResilientWitness]` 等 trailing 数组 20.6 打头）。根源是同一个模式：声明模型在**模型构建期**就把 MachOSwiftSection 的高层 wrapper（`TypeContextWrapper`、`ProtocolConformance`、`Protocol`）连 trailing objects 一起急切解析并**终身驻留**，而其中的重载部分只有惰性 `index(in:)`（和少量打印路径）才消费——RV 稳态里绝大多数定义从未被索引，这些解析产物为「将来可能被点开」白白常驻。 + +本案把三处驻留换成小的 descriptor 引用（几十字节级），trailing 解析改为在 `index()` / 打印期**临时物化、用完即弃**。预估稳态再省 30–45 MiB。 + +## 动机 + +- **实测数字**（RV 2026-08-08/09 复测，详见 0001 落地记录；每实例构成见先行账本 [DeclarationModelMemoryFootprint.md](../Internal/DeclarationModelMemoryFootprint.md)，`class_getInstanceSize` 精确量测）：`TypeDefinition` 每实例 **1272 B**，其中 `type: TypeContextWrapper` 472 + `parentContext: ParentContext?` 472 = **944 B，占 74%**；`ExtensionDefinition` 640 B（账本时期 520，此后随 conformance attribution 字段增长）。两类合计 31.8 MiB 内联块，另拖着 MachOSwiftSection 簇 33.4 MiB 的 trailing 堆数组。 +- **驻留点一：`TypeDefinition.type: TypeContextWrapper`**（`Sources/SwiftDeclaration/Components/Definitions/TypeDefinition.swift:22`）。这是内联 enum，按最大 case `Class` 定尺 472 B——**struct / enum 的定义也照 472 付费**；`Class` 档 17 个存储属性（其中 `TypeGenericContext?` 一项 160 B，绝大多数类型存的是 nil）+ 5 条堆数组（`methodDescriptors`、`methodOverrideDescriptors` 等，`Sources/MachOSwiftSection/Models/Type/Class/Class.swift:30-47`），构造函数把 trailing objects 全部读进堆。更糟的是 `parentContext` 的 `case type(TypeContextWrapper)` **再内联一份** 472 B——且账本的全库读写点追踪证实它**在索引函数返回后再无任何消费者**（写与读都在 `SwiftDeclarationIndexer` 同一个函数里，读只为取一次父类型名）。 +- **驻留点二：`ExtensionDefinition.protocolConformance: ProtocolConformance?`**（`ExtensionDefinition.swift:16`）。`ProtocolConformance` 构造即读 protocol 解析、type reference、witness table pattern、conditional requirements、**`resilientWitnesses: [ResilientWitness]`**（`Models/ProtocolConformance/ProtocolConformance.swift:45-132`）——最后这项就是 MachOSwiftSection 簇 20.6 MiB 的打头项。 +- **驻留点三：`ProtocolDefinition.protocol: MachOSwiftSection.Protocol`**（`ProtocolDefinition.swift:74`）。`Protocol` 驻留 `requirementInSignatures: [GenericRequirement]` + `requirements: [ProtocolRequirement]` 两条堆数组。协议人口比类型小，但同一模式。 +- **关键事实：`index(in:)` 是惰性的。** 触发点在打印器（`SwiftPrinting/SwiftDeclarationPrinter.swift:110/159/200/294`，另有 SwiftInterface 的 diff 渲染路径）——用户查看到哪个声明，哪个才被索引。RV 稳态下绝大多数定义 `isIndexed == false`，胖 wrapper 里的 vtable 描述符、resilient witnesses、protocol requirements 从未被读过第二次。 +- 与 0001 的关系：0001 把「49 万个符号名 String」换成字符串表引用按需物化；本案把「4 万个声明的解析产物」换成 descriptor 引用按需物化。同一方法论在堆格局新头部上的第二次应用。 +- **与先行账本结论的关系**：账本（2026-07-25）当年的裁决是「当前不建议实施」，前提有二——彼时这条线只占 434 MB 的 8–10%，且其余 90% 未剖析。两个前提如今都已消解：其余大头已经 0001（符号名 68.7 MiB）与两次专项清退（`SharedNodeStore` 合并、`MetadataReaderCache` 换持）逐一处理，RV 全景剖析（malloc 归属）也已做过两轮，声明模型如今**就是**堆内头部。账本四项里的 3a（mini-store 增殖）/ 3b（`MetadataReaderCache`）已分别落地，1（`parentContext`）/ 2（wrapper 装箱）由本案以更优形态收编（见「替代方案考量」）。 + +## 前期调研 + +### 现状代码怎么走的 + +- **模型构建**：`SwiftDeclarationIndexer` 的 sweep 为每个 type descriptor 构造完整 wrapper 并传入 `TypeDefinition(type:in:)`；conformance 一侧先 `machO.swift.protocolConformances` 全量物化成数组、按类型名分组（`SwiftDeclarationIndexer.swift:243/515-539`），再逐个塞进 `ExtensionDefinition(… protocolConformance: …)` 驻留。 +- **索引消费（惰性，每定义至多一次）**：`TypeDefinition.index()` 以 `case .class(let cls)` 读 `methodDescriptors` / `vTableDescriptorHeader` / `methodOverrideDescriptors` / `methodDefaultOverrideDescriptors` 构建 vtable/override 查找表（`TypeDefinition.swift:205-270`）；`ExtensionDefinition.index()` 消费 `protocolConformance.resilientWitnesses`（`ExtensionDefinition.swift:95-141`）；`ProtocolDefinition.index()` 消费 `requirements`。 +- **打印消费（每次打印该定义）**： + - 表头与成员打印读 `.type` 的 descriptor 级事实（kind、`contextDescriptorWrapper`）； + - `FieldLayoutRenderer(type: typeDefinition.type, …)` 与字段记录再读（`SwiftDeclarationPrinter+Headers.swift:327-328`）； + - 扩展头打印读 `protocolConformance.protocolNode(in:)` 与 `.globalActorReference`（`SwiftDeclarationPrinter.swift:246-262`）。 +- **消费点普查（库内）**：`.type` 约 30 处——SwiftSpecialization 11（`GenericSpecializer` / `ConformanceProvider` / `TypeDefinition+Specialization`）、SwiftPrinting 8、SwiftInterface 4、SwiftIndexing 3、SwiftDeclarationRendering 4；`.protocolConformance` 打印 1 处 + `index()` 自用；`.protocol` 集中在协议打印与 SwiftDiffing 的 requirement 投影（后者在 `index()` 期冻结成 Mach-O-free 值，不受本案影响）。 + +### 验证过什么 + +- **惰性确认**：`index(in:)` 的全部调用点都在打印/渲染路径（grep 全库），索引器 sweep 不调用——「稳态未索引即胖结构未消费」成立。 +- **descriptor 可再物化**：全部三个 wrapper 都能由 descriptor 重建，且入口现成——`TypeContextWrapper.forTypeContextDescriptorWrapper(_:in:)`（`TypeContextWrapper.swift:50`）、`ProtocolConformance(descriptor:in:)`、`Protocol(descriptor:in:)` 正是今天模型构建期走的构造路径。物化成本 = 一次 trailing objects 顺序解析（`MachOImage` 是映射内存指针步进，`MachOFile` 是页缓存友好的文件读）——与今天模型构建期完全相同的工作量，只是从「每声明一次、永久驻留」变为「用时一次、即弃」。 +- **descriptor 体积**：`ClassDescriptor` 等 = raw layout（十几个 32 位字段）+ offset，内联 ~64 B 级；`TypeContextDescriptorWrapper` 是三档 descriptor 的 enum，同量级。对比被替换的 wrapper 内联 ~300–500 B + 堆数组，一个量级的差距。 +- **尚未验证（落地步骤补）**：下游仓库（RuntimeViewer / MachOKitUI / SymbolViewer）是否直接消费 `.type` / `.protocolConformance` / `.protocol`——落地第一步做下游普查（RV 侧可请对面会话代查）。 + +## 提议方案 + +1. **`TypeDefinition`**：存 `typeContextDescriptorWrapper: TypeContextDescriptorWrapper`（替代 `type: TypeContextWrapper`）。`parentContext` 首选**降级为索引期局部载体**（账本证实索引函数返回后零消费者——写读都在 `SwiftDeclarationIndexer` 一个函数里，用局部结构承载、函数返回即释放，存储属性整个移除）；若落地第一步的下游普查发现真实消费者，则退而把 `case type` 载荷换成 descriptor 形态保留属性。新增 `materializedTypeContext(in:) throws -> TypeContextWrapper` 按需物化入口；`index()` 在函数体开头物化一次、以局部变量贯穿全程。 +2. **`ExtensionDefinition`**：存 `protocolConformanceDescriptor: ProtocolConformanceDescriptor?`（替代 `protocolConformance: ProtocolConformance?`）；`index()` 与扩展头打印各自临时物化。 +3. **`ProtocolDefinition`**：存 `protocolDescriptor: ProtocolDescriptor`（替代 `protocol: MachOSwiftSection.Protocol`；名字早已冻结在 `protocolName`，不受影响）；`index()` 临时物化取 requirements。 +4. **物化纪律**：每「处理一个定义」（索引它 / 打印它 / specialize 它）至多物化一次，物化结果以局部变量或函数参数贯穿，**不做 per-access 计算属性**——防止打印循环里反复触发全套 trailing 解析。 + +### 非目标 + +- **`[UInt32]` 行号桶扁平化**：另案 [0003](0003-symbol-row-bucket-flattening.md)，两案不重叠（不同簇、不同模块）。 +- **StringStorage 残余 31.3 MiB 的成分治理**（member 分类索引的 String 键、字段名、打印名等混合人口）：成分未明，等 RV 侧 malloc_history 切片再议，不盲做。 +- **SwiftDump 路径（`TypedDumper` 等）**：本就 per-dump 临时构造 wrapper、无驻留，不动。 +- **MachOSwiftSection wrapper 类型本身**：`Class` / `Struct` / `Enum` / `ProtocolConformance` / `Protocol` 保持「构造即完整解析」的值语义——它们是解析结果的正确表达，问题只在**声明模型驻留它们**，不在它们自身。 +- **成员数组与 `orderedMembers` 的表示**：索引后才产生的人口，规模由用户实际浏览量决定，与本案无关。 +- **`metadata: MetadataWrapper?` 与 specialized 定义**：specialize 路径持 runtime metadata 指针，本就轻量，不动。 + +## 详细设计 + +### 存储形态对照 + +| 位置 | 现状（驻留） | 提案后（驻留） | 重内容去向 | +|---|---|---|---| +| `TypeDefinition.type` | `TypeContextWrapper`（内联 472 B + trailing 堆数组） | `TypeContextDescriptorWrapper`（~60 B 级：`ClassDescriptor` raw layout 52 B + enum tag） | `index()` / 打印期临时物化 | +| `TypeDefinition.parentContext` | `case type(TypeContextWrapper)` 再内联 472 B | **移除**（降级为索引期局部载体；下游普查有消费者则退为 descriptor 载荷保留） | 索引函数返回即释放 | +| `ExtensionDefinition.protocolConformance` | `ProtocolConformance?`（内联 ~250–300 B + `[ResilientWitness]` 等堆数组） | `ProtocolConformanceDescriptor?` | `index()` / 扩展头打印临时物化 | +| `ProtocolDefinition.protocol` | `Protocol`（含 2 条 requirements 堆数组） | `ProtocolDescriptor` | `index()` 临时物化 | + +预期实例尺寸（按账本逐属性账目推算，落地时以同款探针复量为准）：`TypeDefinition` 1272 → **~400 B**（−944 的两份 wrapper，+~60 的 descriptor wrapper），11,985 实例 ≈ 14.6 → ~4.6 MiB；`ExtensionDefinition` 640 → ~400 B，28,225 实例 ≈ 17.2 → ~10.8 MiB；另加 MachOSwiftSection 簇 33.4 MiB 中 trailing 数组（`[ResilientWitness]` 20.6 打头）的大部释放。 + +### 物化接口 + +```swift +extension TypeDefinition { + public func materializedTypeContext(in machO: MachO) throws -> TypeContextWrapper +} +extension ExtensionDefinition { + public func materializedProtocolConformance(in machO: MachO) throws -> ProtocolConformance? +} +extension ProtocolDefinition { + public func materializedProtocol(in machO: MachO) throws -> MachOSwiftSection.Protocol +} +``` + +三者都是薄封装（转调现成的 `forTypeContextDescriptorWrapper` / `ProtocolConformance(descriptor:in:)` / `Protocol(descriptor:in:)`），throws 语义与今天模型构建期相同。 + +### 构造路径 + +- 索引器 sweep 今天就持有完整 wrapper（要用它派生 `typeName`）：`TypeDefinition` 的 init 继续收 wrapper，**内部只存其 `typeContextDescriptorWrapper`**——sweep 的解析工作量不变，变化只是解析产物在 init 返回后即可释放。 +- conformance 一侧：indexer 的全量 `protocolConformances` 数组在分组、构造 `ExtensionDefinition` 之后本就出栈；本案后 `ExtensionDefinition` 只留 descriptor，整批 `[ResilientWitness]` 随之释放。 +- specialized 定义（`specialize(with:in:)` 派生）同样只存 descriptor；`isSpecialized` / `metadata` 语义不变。 + +### 消费点迁移（库内 ~30 处） + +两类机械迁移： + +- **读 descriptor 级事实**（kind 判断、`contextDescriptorWrapper`、offset）：直接改读新属性，零物化。约占一半。 +- **读 trailing 内容**（vtable 表、requirements、witnesses、genericContext、`FieldLayoutRenderer(type:)` 传参）：在所属函数入口物化一次、局部贯穿。集中在 `TypeDefinition.index()`、`ExtensionDefinition.index()`、`ProtocolDefinition.index()`、打印器的 header/字段渲染入口、`GenericSpecializer` 的 specialize 入口。 + +`FieldLayoutRenderer` 等接收 `TypeContextWrapper` 的既有签名**不改**——调用方物化后传入。 + +### 风险与接受的约束 + +- **物化 CPU**：索引/打印一个定义多付一次 trailing 解析（顺序读 + 几次小数组分配）。打印本身要做成体量大得多的 demangle + node print，解析占比预期为噪声级；渲染 A/B 的 wall-clock 持平是验收线。CLI 的 `interface` 全量打印会为每个类型物化一次——与今天模型构建期的一次性全量解析**总量相同**，只是时点后移。 +- **物化生命周期**：descriptor 再物化要求 `machO` 存活——与 0001 mapped 名字同一条生命周期约束（RV 的索引对象从不卸载），且声明模型本就处处以 `in machO:` 参数工作，无新增约束面。 +- **API 破坏**：三个 public 属性换形态（见「影响」）。下游若直接消费需机械迁移；先普查再动手。 + +## 替代方案考量 + +- **`indirect` enum 装箱 `TypeContextWrapper`**（账本第 2 项，当年估实例 → 344 B / 省 ~12.6 MB）：只省内联不省堆——trailing 数组照旧驻留，MachOSwiftSection 簇 33.4 MiB 分文不动，且装箱让全库按值传递的 wrapper 背上引用计数流量（账本自己标注的吞吐风险）。descriptor 化拿到同量级内联收益（~400 B vs 344 B）**外加**整个 trailing 簇，无 ARC 代价。被否。 +- **`parentContext` 换 descriptor 载荷保留属性**（账本第 1 项的另一半改法）：账本已证实索引后零消费者——为一个没有读者的属性保留任何形态都是浪费；首选整个移除、降级为索引期局部载体，descriptor 载荷仅作下游普查发现真实消费者时的退路。 +- **计算属性 per-access 物化、保住 `.type` API 原样**：每次属性访问都触发全套 trailing 解析，打印循环里不可控地反复付费；显式 `materialized…(in:)` 让成本出现在调用方眼前、强制局部贯穿。被否。 +- **`index()` 完成后把胖字段置 nil、API 不变**：`isIndexed` 之后 `.type` 语义变成「阶段依赖可空」，比直接换 descriptor 更伤下游（且稳态里未索引定义占大头，收益反而小）。被否。 +- **只做 `ExtensionDefinition`（最大簇）**：同一模式修一半，`TypeDefinition` 的 1.25 KB × 12k 与 `parentContext` 双份内联原样留下，下次还得再来一轮 API 破坏。被否——一次修完这一类。 +- **把 wrapper 改成惰性解析（存 descriptor、trailing 属性按需读）**：改动落在 MachOSwiftSection 的 15+ 个 wrapper 类型上，波及 fixture 基线套件全量，且「解析结果值类型」的语义被打破（`private(set) var` 全变计算）；声明模型侧收敛改动面等价拿到同一收益。被否。 + +## 影响 + +### 源码兼容性（source compatibility） + +**破坏性变更**，三个 public 存储属性换形态： + +- `TypeDefinition.type: TypeContextWrapper` → `typeContextDescriptorWrapper: TypeContextDescriptorWrapper` + `materializedTypeContext(in:)` +- `TypeDefinition.parentContext`（含 `ParentContext` 类型本身）：首选整体移除（降级为索引期局部载体）；下游普查有消费者则退为 descriptor 载荷保留 +- `ExtensionDefinition.protocolConformance: ProtocolConformance?` → `protocolConformanceDescriptor: ProtocolConformanceDescriptor?` + `materializedProtocolConformance(in:)` +- `ProtocolDefinition.protocol: MachOSwiftSection.Protocol` → `protocolDescriptor: ProtocolDescriptor` + `materializedProtocol(in:)` + +迁移是机械的：读 descriptor 事实改属性名；读 trailing 内容加一次 materialize 调用。库内约 30 处同批迁移。 + +### ABI 兼容性（条件项） + +不适用——本库以 SPM 源码分发，使用方每次重新编译（项目类型声明见 `Documentations/README.md`）。 + +### 下游影响 + +- 仓库内：`SwiftDeclaration` 为改动主体；`SwiftIndexing` / `SwiftPrinting` / `SwiftInterface` / `SwiftSpecialization` / `SwiftDeclarationRendering` 消费点迁移。`SwiftDiffing` 无改动（attribution 早已在 index 期冻结为 Mach-O-free 值）。 +- 下游仓库（RuntimeViewer、MachOKitUI、SymbolViewer）：落地第一步普查三个属性的直接消费点；有则同批送机械迁移补丁，无则重编即得收益。RV 是验收方（footprint + heap 复测）。 + +### 文档与示例 + +- AGENTS.md「SwiftDeclaration」段同步 descriptor 化模型与物化纪律（谁物化、活多久、不做 per-access）。 +- `Documentations/README.md` 索引与本提案状态同步。 +- [DeclarationModelMemoryFootprint.md](../Internal/DeclarationModelMemoryFootprint.md) 按其既有惯例补后记（第 1 / 2 项由本案收编落地，量测账目保持原貌），并以同款探针复量落地后的实例尺寸。 +- 落地时按「落地步骤」收尾判断决定是否另写实现说明。 + +## API 演进与废弃策略 + +- **直接替换，不留 deprecated 旧属性**：deprecated 的 `.type` 要能继续返回 `TypeContextWrapper` 就得继续驻留数据，与本案目的直接冲突；源码分发、下游数量少且受控，一次性迁移优于长尾兼容。 +- 随下一次常规版本发布；changelog（英文）明确列出三处破坏点与迁移句式。 + +## 落地步骤 + +1. 下游普查：RV / MachOKitUI / SymbolViewer 对 `.type` / `.protocolConformance` / `.protocol` 的直接消费点清单（RV 侧请对面会话代查）。 +2. `TypeDefinition` descriptor 化：存储换形态 + `parentContext` 移除（或退路形态）+ `materializedTypeContext(in:)` + `index()` 与打印路径的物化点。 +3. `ExtensionDefinition` / `ProtocolDefinition` 同模式。 +4. 库内消费点全量迁移（~30 处，两类句式）。 +5. 测试：全量 `swift test --skip IntegrationTests` 同数全绿；**渲染 A/B 三 reader 路径逐字节一致**（硬线——本案不许改变任何输出字节）。 +6. 性能：iOS 18.5 模拟器 SwiftUI `interface` wall-clock 持平（物化 CPU 的验收）。 +7. RV 复测（对面协调）：预期堆存活 283 → **~240–255 MiB**；`SwiftDeclaration` 簇 41.3 → ~15–20 MiB；MachOSwiftSection 簇 33.4 → ~10–15 MiB。 +8. 以账本同款探针（`class_getInstanceSize` + `MemoryLayout`）复量三类定义的落地后实例尺寸，给 [DeclarationModelMemoryFootprint.md](../Internal/DeclarationModelMemoryFootprint.md) 补后记。 +9. 收尾判断（写进决策日志）：是否写实现说明（「物化纪律」是代码看不出来的契约，倾向写短篇或并入 AGENTS.md）；新术语是否登记。 + +## 决策日志 + +| 日期 | 变更 | 说明 | +|------|------|------| +| 2026-08-09 | Created as In Review | 0001 落地后 RV 复测把声明模型 41.3 MiB + MachOSwiftSection 解析结构 33.4 MiB 定位为堆内新头部；优化面普查（`index()` 惰性 × wrapper 急切驻留的错配）成文本案；用户批准立项（「可以，写提案」）。 | diff --git a/Documentations/Evolutions/0003-symbol-row-bucket-flattening.md b/Documentations/Evolutions/0003-symbol-row-bucket-flattening.md new file mode 100644 index 00000000..cbb261b7 --- /dev/null +++ b/Documentations/Evolutions/0003-symbol-row-bucket-flattening.md @@ -0,0 +1,108 @@ +# 0003 - SymbolIndexStore `[UInt32]` 行号桶扁平化：单元素桶内联化 + +- **状态**: In Review +- **作者**: JH +- **创建日期**: 2026-08-09 +- **最后更新**: 2026-08-09 +- **所属愿景**: 无 +- **关联提案**: [0001](0001-symbol-name-offsetization.md)（其「非目标」一节点名本案为下一篇候选；`symbolRowsByOffset` 的 offset 键表 12.2 MiB 即 0001 引入的形态）。与 [0002](0002-declaration-model-descriptor-slimming.md) 正交（不同簇、不同模块），可独立实施 +- **实现分支 / PR**: `feature/node-store-migration`（拟，待批准后实施） +- **配套文档**: 暂无 + +## 摘要 + +RV 实测堆里有 **38.8 MiB / 45 万个小 `[UInt32]` 数组**，来源是 `SymbolIndexStore.Storage` 的行号桶：`symbolRowsByOffset: [Int: [UInt32]]`（12.2 MiB）与三族 `MemberSymbolRows` 嵌套字典的叶子桶。绝大多数桶只有一个元素，却各付一次堆分配（Array 存储头 + malloc 桶圆整 ≈ 48 B）外加字典槽里 8 B 的引用。本案引入「单元素内联、多元素才落堆」的小桶类型替换 `[UInt32]` 桶值，预估省 **15–25 MiB**，零公开 API 变化。 + +## 动机 + +- 0001 落地时已点名（其「非目标」：44 万个单元素桶，当时估 ~28 MiB；0001 后 RV 实测簇为 38.8 MiB / 45 万个，含新 offset 键表 12.2）。 +- 一个符号 offset / 一个成员键在绝大多数情况下只对应一行——`[UInt32]` 的通用性为「偶发多行」让全体单行桶付堆分配，是纯粹的表示浪费。 +- 每桶节省估算：今天单元素桶 = 字典槽 8 B 引用 + 堆上 ~48 B（32 B Array 头 + 16 B malloc 槽）；内联后 = 字典槽 ~16 B、零堆分配。约省 40 B/桶 × 45 万 ≈ 17 MiB，与预估带吻合。 + +## 前期调研 + +### 现状代码怎么走的 + +- `Storage.symbolRowsByOffset: [Int: [UInt32]]`（`SymbolIndexStore.swift:196`）：sweep 期 `append` 累积（`:443`），查询侧单点消费 `symbol(atRow:offset:)`（`:920`）。 +- `Storage.MemberSymbolRows = OrderedDictionary>`（`:133`）：三族（member / methodDescriptor / protocolWitness，`:184-188`），叶子桶同样 append 累积、查询侧整桶物化。 +- `globalSymbolRowsByKind` / `symbolRowsByKind`（`:159/190`）：按 kind 聚合的**大**桶，数量个位数到几十，不在本案范围。 + +### 验证过什么 + +- 0001 的 RV heap 复测把 `[UInt32]` 簇钉在 38.8 MiB / 45 万个（malloc 归属），量级可信。 +- 单元素占比未逐桶实测（malloc 档案只给总量）；落地时在 freeze 处加一次性统计断言辅助验收（预期 ≥ 85% 单元素，与 0001 期「44 万单元素」的旧测量一致）。 + +## 提议方案 + +新增 `SymbolRowBucket`（`MachOSymbols` 内部类型）： + +```swift +enum SymbolRowBucket { + case single(UInt32) + case multiple([UInt32]) +} +``` + +- `append(_:)` 语义：空槽首插走 `single`；第二个元素起迁移为 `multiple`（一次两元素数组分配）。 +- 提供 `Sequence` / `count` / `contains(_:)` 薄接口，查询路径零拷贝迭代。 +- 替换范围：`symbolRowsByOffset` 的值、三族 `MemberSymbolRows` 的叶子桶。查询 API 的返回形态（`[DemangledSymbol]` 等）不变——桶在出口物化，与今天相同。 + +### 非目标 + +- `OrderedDictionary` 自身的 ordering 表开销与 `MemberSymbolRows` 的 String 键：另一个问题域（键人口是打印名，见 0001 非目标），不动。 +- 按 kind 聚合的大桶（`symbolRowsByKind` / `globalSymbolRowsByKind`）：数量少、本就该是数组，不动。 +- CSR（compressed sparse row）全量平铺：见「替代方案考量」。 + +## 详细设计 + +- 布局：`case single` 载荷 4 B + tag，enum 内联 ≤ 16 B（`compactValueLayouts` 加断言钉住）；字典槽从 8 B 引用变 ~16 B 内联值，堆分配从每桶一次降为仅多元素桶一次。 +- sweep 期就地累积形态不变（`symbolRowsByOffset[offset, default: .empty].append(row)` 句式），无需 build/freeze 双形态——这是选 enum 而非 CSR 的直接原因。 +- `mayAlreadyBeListed` 去重探测（`:440`）改走 `contains(_:)`。 +- 迭代顺序保持插入序（`multiple` 数组序即插入序，`single` 天然有序），查询输出字节不变。 + +### 风险与接受的约束 + +- `multiple` 桶比今天多付一次「single → 两元素数组」的迁移拷贝：仅多行桶付、每桶一次，量级忽略不计。 +- enum 内联 16 B 使字典槽变宽：单元素占比越低收益越薄；按 ≥ 85% 单元素估算净省仍在 15–25 MiB 带内，freeze 期统计为验收证据。 + +## 替代方案考量 + +- **CSR 全量平铺**（一条大 `[UInt32]` + 每键 range）：驻留最省，但 sweep 期需要 build 态（每键临时桶）→ freeze 态（平铺）的双形态转换，且三族嵌套字典的叶子层都要陪着改形态；enum 方案 90% 的收益、1/3 的改动面。被否——若日后 profiling 证明字典槽变宽是新瓶颈再升级，结构兼容。 +- **`[UInt32]` 换 `ContiguousArray` / 预留容量**：不解决「堆分配次数 = 桶数」的根因。被否。 +- **swift-collections 现成类型**：无「inline-one 小数组」稳定 API（`InlineArray` 是定长语义，SE-0453）；自写 20 行 enum 即可。被否。 + +## 影响 + +### 源码兼容性（source compatibility） + +**纯内部 / 无破坏。** `SymbolRowBucket` 与 `Storage` 各字典的值类型都在 `MachOSymbols` 内部（`Storage` 非 public）；全部查询 API 的签名与返回形态不变，输出字节不变。 + +### ABI 兼容性（条件项） + +不适用——SPM 源码分发（项目类型声明见 `Documentations/README.md`）。 + +### 下游影响 + +- 仓库内：仅 `MachOSymbols`（`SymbolIndexStore.swift` + 新类型文件）。 +- 下游仓库：零源码改动，重编译即得收益。RV 为验收方。 + +### 文档与示例 + +- AGENTS.md「Symbol indexing」段补一句桶表示；`Documentations/README.md` 索引同步。 + +## API 演进与废弃策略 + +- 无公开 API 变化，无废弃需求；随下一次常规版本发布，changelog 记录内存收益。 + +## 落地步骤 + +1. `SymbolRowBucket` 实现 + 布局断言（`compactValueLayouts`）+ 单测（append 迁移、迭代序、`contains`）。 +2. `symbolRowsByOffset` 与三族 `MemberSymbolRows` 叶子桶替换；freeze 处一次性单元素占比统计（验收用，随后移除或留在 IntegrationTests）。 +3. 全量 `swift test --skip IntegrationTests` 同数全绿；渲染 A/B 三 reader 逐字节一致（迭代序不变的验收）。 +4. RV heap 复测：`[UInt32]` 簇 38.8 → 预期 ~15–20 MiB。 + +## 决策日志 + +| 日期 | 变更 | 说明 | +|------|------|------| +| 2026-08-09 | Created as In Review | 0001「非目标」点名的候选正式立项；RV 实测簇 38.8 MiB / 45 万个为输入；用户批准立项（「可以，写提案」）。 | diff --git a/Documentations/Evolutions/README.md b/Documentations/Evolutions/README.md index 60b56466..e8d95fc4 100644 --- a/Documentations/Evolutions/README.md +++ b/Documentations/Evolutions/README.md @@ -7,3 +7,5 @@ | # | 标题 | 状态 | |---|------|------| | [0001](0001-symbol-name-offsetization.md) | SymbolIndexStore 符号名 offset 化：驻留字符串换字符串表引用 | Implemented | +| [0002](0002-declaration-model-descriptor-slimming.md) | 声明模型 descriptor 化：TypeDefinition / ExtensionDefinition / ProtocolDefinition 不再驻留急切解析的胖 wrapper | In Review | +| [0003](0003-symbol-row-bucket-flattening.md) | SymbolIndexStore `[UInt32]` 行号桶扁平化：单元素桶内联化 | In Review | diff --git a/Documentations/README.md b/Documentations/README.md index 6038fde3..b6c0e38f 100644 --- a/Documentations/README.md +++ b/Documentations/README.md @@ -74,6 +74,8 @@ required by `Version.swift`'s bump contract). | [SharedNodeStoreMigration.md](Internal/SharedNodeStoreMigration.md) | **Implemented（2026-08-08）**：三条小 store 流水线（`InternedNodeReferenceCache` / `TypeDefinition` 字段树批量 store / `lateDemangledNode`）汇入上游 0010 `SharedNodeStore` 的迁移设计与落地记录——改动位置、明确不动的部分、验证结果与与方案的差异。 | | [MetadataReaderCacheRetirement.md](Internal/MetadataReaderCacheRetirement.md) | **Implemented（2026-08-08）**：`MetadataReaderCache` 清退——三张 class `Node` 树字典（五镜像实测 ~18.4 万残留 `Node` 的持有主体）换持 `NodeReference` 汇入 `InternedNodeReferenceCache` 作用域 store，公开 API 与 103 处调用点零改动，补按镜像清理 seam；含身份稳定性排查、`MultiPayloadEnumDescriptorCache` 不必同批改键的论证与四轴验证记录（1337 tests 同数全绿 / 渲染 A/B 96 对逐字节一致 / 性能持平 / RV 实景存活 `Node` 207,489 → 44，−99.98%）。 | | [Evolutions/0001-symbol-name-offsetization.md](Evolutions/0001-symbol-name-offsetization.md) | **提案 0001（Implemented）**：`SymbolIndexStore` 符号名 offset 化——49.4 万个驻留符号名 `String`（68.7 MiB，RV 实测堆内头号大户的最大单项）换成字符串表引用按需物化，`tableRowByName` 字典退役换名字序字节级二分，字节级 `isSwiftSymbol` 判定消掉非 Swift 符号的全部瞬时 String；公开 API 零改动。含 Swift 6.2 Span 家族与 swift-collections 1.6.0 选型裁决（实施时因部署下限改用 `UnsafeBufferPointer` / 精确容量 Array，见决策日志）。 | +| [Evolutions/0002-declaration-model-descriptor-slimming.md](Evolutions/0002-declaration-model-descriptor-slimming.md) | **提案 0002（In Review）**：声明模型 descriptor 化——`TypeDefinition` / `ExtensionDefinition` / `ProtocolDefinition` 不再终身驻留急切解析的胖 wrapper（`TypeContextWrapper` 472 B × 2 份、`ProtocolConformance` 及其 `[ResilientWitness]`、`Protocol`），改存几十字节级 descriptor 引用，trailing 解析在惰性 `index()` / 打印期临时物化；`parentContext` 降级为索引期局部载体。0001 后堆内新头部（声明模型 41.3 + MachOSwiftSection 簇 33.4 MiB）的对症案，预估再省 30–45 MiB；破坏性 API 变更（三处属性换形态，机械迁移）。 | +| [Evolutions/0003-symbol-row-bucket-flattening.md](Evolutions/0003-symbol-row-bucket-flattening.md) | **提案 0003（In Review）**：`SymbolIndexStore` `[UInt32]` 行号桶扁平化——45 万个小数组（38.8 MiB，绝大多数单元素、各付 ~48 B 堆分配）换「单元素内联、多元素落堆」的 `SymbolRowBucket`，覆盖 `symbolRowsByOffset` 与三族 `MemberSymbolRows` 叶子桶；预估省 15–25 MiB，纯内部零 API 变化。0001「非目标」点名候选的正式立项。 | | [DeclarationModelMemoryFootprint.md](Internal/DeclarationModelMemoryFootprint.md) | NodeStore 迁移后的声明模型内存足迹量测：`TypeDefinition` 1272 字节的逐属性构成（两份 `TypeContextWrapper` 占 74%）、`TypeContextWrapper` 按最大 case `Class` 定尺的原因、`parentContext` 是被当成永久字段的临时值、mini-store 增殖与 `MetadataReaderCache` 仍持 `Node` 树；含四项可回收估算（合计 ~8–10%）与「当前不建议实施、应先剖析其余 90%」的结论。 | | [NodeStoreMigrationOpenIssues.md](Internal/NodeStoreMigrationOpenIssues.md) | `feature/node-store-migration` 的**已确认问题清单**（2026-07-28 两轮审查 + 复核）。第一节是前一轮修复自身的两处缺口（Catalyst 降级只覆盖 framework 形态导致 plain dylib 仍平局、`appendRowIfAbsent` 线性扫描），已于 2026-07-29 闭环并保留成因；第二节起为仍然打开或已裁决的条目：`structuralHash` 每文本节点分配 `String` 与 `ABIKey` 每 key materialize 两条**已按上游设计终审关闭**（0.5.1 保持现状且上游说明不改，见 ReviewAdjudications A1/A2）、`memberSymbols` 改为线性 + 全树比对（**已证实量级可忽略，不是回归**）；`lateDemangledNode` 持锁 demangle 与 `ProtocolConformanceDumper` 的 materialize 分支两条**已于 2026-08-03 闭环**（连同失败名裁决缓存、dump 路径引用化）、build sweep 串行且每符号跨线程往返（上游）、`ABIKey` 每 key materialize，以及四条代码卫生项；两个公开查询 API 的字典键一条**已于 2026-08-03 裁决为不修**（类型级 SPI），rebase 前置事项一条前提已过期但压着两条仍成立的注意事项。逐条注明成因、影响面与「该在哪里修」。与 `Reviews/` 下两份审查记录冲突时以后者为准。 | | [SystemFrameworkRenderingVerification.md](Internal/SystemFrameworkRenderingVerification.md) | 系统框架渲染 A/B 验证流程（大重构必跑）：两个检出对同一批真实输入（归档 dyld cache → 无则当前系统 cache、模拟器 runtime → 无则现有 runtime、当前系统 MachOImage）逐字节比对 dump+interface；入口脚本 `Scripts/run-rendering-ab-verification.py`，含回退规则、踩坑清单（`-p` 消歧、fat 二进制 `-a`、同 boot 会话、scratch 隔离）与 2026-08-03 基线运行记录。 | From a81f265a1f04bca05759ca5fea1c8e1b9e2a9d18 Mon Sep 17 00:00:00 2001 From: Mx-Iris Date: Sun, 9 Aug 2026 12:34:35 +0800 Subject: [PATCH 48/77] docs: add project glossary and SymbolIndexStore memory-optimization topic doc --- AGENTS.md | 2 + .../0001-symbol-name-offsetization.md | 3 +- Documentations/Glossary.md | 116 ++++++++++++++++++ .../SymbolIndexStoreMemoryOptimization.md | 76 ++++++++++++ Documentations/README.md | 3 + 5 files changed, 199 insertions(+), 1 deletion(-) create mode 100644 Documentations/Glossary.md create mode 100644 Documentations/Internal/SymbolIndexStoreMemoryOptimization.md diff --git a/AGENTS.md b/AGENTS.md index aeb194ec..b42c6aa4 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -7,6 +7,8 @@ All project documentation lives in the `Documentations/` directory, split by aud - **`Documentations/` (top level) — external / public docs**, for library users and other developers. Reference-style, English or bilingual (an `*_zh.md` companion). Currently just `SwiftEnumLayout.md` (+ `SwiftEnumLayout_zh.md`). - **`Documentations/Internal/` — maintainer-facing notes** (design notes, migration guides, refactor write-ups; `Internal/TaskReports/` holds dated per-task reports). This is the default home for working docs. +- **`Documentations/Evolutions/` — evolution proposals** (Chinese, `NNNN-kebab-case-slug.md`): every non-trivial change is proposed, approved, and tracked in one per-change file with a status machine (`Draft` → … → `Implemented`); `Evolutions/README.md` is the status table. Implementation must not start before the proposal is `Accepted`, and status updates land in the same commit as the code. +- **`Documentations/Glossary.md` — project glossary** (Chinese): project-coined terms and project-specific usages (sweep, reader-split leg, name source, detach, materialize, …). Register new terms in the same batch that introduces them; cross-project generic terms live in the user's global glossary, not here. Name doc files in **PascalCase** with the `.md` extension (e.g. `Internal/SwiftModularizationMigration.md`, `Internal/ReadingContextAbstraction.md`). When asked to "write a doc", default to `Documentations/Internal/` with a PascalCase name — only put it at the top level if it is genuinely a public, externally-facing reference (and then keep it English/bilingual). Do not scatter docs next to source files. When adding or moving a doc, update `Documentations/README.md`. diff --git a/Documentations/Evolutions/0001-symbol-name-offsetization.md b/Documentations/Evolutions/0001-symbol-name-offsetization.md index 2ab3841a..3b06ba07 100644 --- a/Documentations/Evolutions/0001-symbol-name-offsetization.md +++ b/Documentations/Evolutions/0001-symbol-name-offsetization.md @@ -7,7 +7,7 @@ - **所属愿景**: 无 - **关联提案**: 无（本仓库首篇）。跨仓库关联：swift-demangling 0008（字节扫描器）/ 0010（`SharedNodeStore`）为既有地基；其「demangle 入口收 `Span`」新提案与本案解耦对接（见「前期调研 · 上游接口」） - **实现分支 / PR**: `feature/node-store-migration` -- **配套文档**: 无独立专题文章（收尾判断见决策日志）；维护者事实同步于 AGENTS.md「Symbol indexing」段，过程复盘见 [TaskReports/2026-08-08-symbol-name-offsetization.md](../Internal/TaskReports/2026-08-08-symbol-name-offsetization.md) +- **配套文档**: 专题汇总 [SymbolIndexStoreMemoryOptimization.md](../Internal/SymbolIndexStoreMemoryOptimization.md)（2026-08-09 应用户要求补写，覆盖含本案在内的三波优化；推翻当日「不另写」的收尾判断，见决策日志）；维护者事实同步于 AGENTS.md「Symbol indexing」段，过程复盘见 [TaskReports/2026-08-08-symbol-name-offsetization.md](../Internal/TaskReports/2026-08-08-symbol-name-offsetization.md) ## 摘要 @@ -201,3 +201,4 @@ final class SymbolTable: @unchecked Sendable { | 2026-08-08 | 实施偏差：搭车项裁剪 | `rootNodeIndexByTableRow` 的 `Optional` → `UInt32.max` 哨兵一项**放弃**：`NodeStore.NodeIndex` 的构造器是上游 internal（debug 布局还带 store tag），从原始 `UInt32` 重建索引需要新的上游 API，为 ~1.6 MB 不值得跨仓库开口子。`symbolRowsByOffset` 换普通 `Dictionary` 一项照做。另一实现细节：standalone `SymbolTable` 统一走私有字节缓冲表示（提案草绘的 `[String]` 变体不再需要——单一表示，读取路径零分支）。 | | 2026-08-08 | Implemented + 收尾判断 | 验证结果见「落地记录」（1341 全绿、A/B 96 对逐字节一致、性能持平；文件腿构建期峰值 +4% 如实记录，RV 稳态复测为最终裁判、结果回填）。收尾判断：**不另写实现说明**——「代码看不出来的决策」（mapped 指针生命周期约束、名字来源双腿、Span 不可用的原因）已分别落在 `SymbolTable` 类文档、AGENTS.md「Symbol indexing」段与本提案决策日志，另立一篇只会是复述；**不登记新术语表**——本项目无 `Glossary.md`（项目现状即约定），「offset 化 / 名字来源 / permutation 二分」均在首次出现处展开。 | | 2026-08-08 | RV 复测闭环 | 落地步骤 8 完成（对面协调，同日）：footprint 稳态 445 → 322 MB（−28%，好于预期）、堆存活 355 → 283.3 MiB（预期带内）、`SymbolIndexStore` 簇 214.6 → 120.9 MiB、StringStorage −42.8 万个/−52.9 MiB，无回归旁证。详数见「落地记录」第 5 条。本提案全部落地步骤就此闭环。 | +| 2026-08-09 | 收尾判断修订：补写专题 + 建术语表 | 应用户要求推翻前日收尾判断的两项「不做」：补写专题汇总 [SymbolIndexStoreMemoryOptimization.md](../Internal/SymbolIndexStoreMemoryOptimization.md)（行文参照上游 swift-demangling 的 `SubtreeInterning.md`，覆盖 NodeStore 迁移 → 缓存清退 → 本案三波）；新建项目术语表 `Documentations/Glossary.md` 并登记本案引入的术语（sweep、腿、名字来源、permutation 二分、detach、物化等）——「项目无术语表」的现状前提由用户指示改变。 | diff --git a/Documentations/Glossary.md b/Documentations/Glossary.md new file mode 100644 index 00000000..3836efe9 --- /dev/null +++ b/Documentations/Glossary.md @@ -0,0 +1,116 @@ +# 术语表 + +本项目专有名词与约定用法。 + +跨项目通用的术语（ABI 与源码兼容性之别、demangle、descriptor 的一般含义、Mach-O、metadata、dyld shared cache 等）收录在全局术语表中，本表只收本项目特有的，不重复登记。全局表不在本仓库内，位于 iCloud Global 镜像的 `Documentations/Glossary.md`。 + +## 收录范围 + +**收**： + +- 项目自造词与内部代号 +- 本项目内反复出现的缩写 +- 通用术语在本项目里的**特定含义** +- 容易混淆的近义词对，说明如何区分 + +**不收**：语言与框架的通用术语（Swift 的 `optional`、`enum` 等）。查官方文档即可，收进来只会稀释真正需要解释的内容。 + +## 维护约定 + +提案或专题文章引入新术语时，**同批次**登记进本表。文档里首次出现该术语时展开一次并链到这里，之后不必每篇重复解释。 + +## 术语 + +按英文名 / 标识符字母序排列。 + +### A/B（渲染 A/B 验证） + +大重构后的强制验证流程：两个检出（基线 commit 与候选分支）对同一批真实二进制（系统 dyld cache、模拟器 runtime、in-process 镜像）各跑一遍 `dump` + `interface`，输出必须**逐字节一致**。它验证的是「重构没有改变任何输出」，不是「输出是对的」。 + +- **主要出现在**：`Scripts/run-rendering-ab-verification.py` +- **延伸阅读**：[SystemFrameworkRenderingVerification.md](Internal/SystemFrameworkRenderingVerification.md) + +### bucket（桶） + +分类索引里「一个键对应的一组符号表行号」，值形态是 `[UInt32]` 小数组（如 `symbolRowsByOffset` 的值、`MemberSymbolRows` 的叶子）。绝大多数桶只有一个元素，却各付一次堆分配——这是提案 0003 的对象。 + +- **主要出现在**：`Sources/MachOSymbols/SymbolIndexStore.swift` +- **延伸阅读**：[提案 0003](Evolutions/0003-symbol-row-bucket-flattening.md) + +### detach（脱表，`detachedFromSharedTable()`） + +把一个查询期 vend 出来的 `DemangledSymbol` 从共享 `SymbolTable` 上摘下来、换成自带单行表的独立值。共享表对「vend 十万个、随手丢弃」是正确的取舍，但**存进声明模型的长命值必须先 detach**——一个存活值会把整张表（几十万行 + 对镜像映射内存的引用）钉在内存里，让按镜像回收失效。六个存储点由 `SymbolTableRetentionTests` 钉住；查询路径**不要** detach。 + +- **主要出现在**：`Sources/MachOSymbols/DemangledSymbol.swift`、AGENTS.md「Symbol indexing」段 +- **延伸阅读**：[提案 0001](Evolutions/0001-symbol-name-offsetization.md) + +### late-name 路径（`lateDemangledNode(forName:)`） + +sweep 覆盖范围之外的名字走的旁路：demangle 后 intern 进 `Storage` 自持的一个可追加 side store，名字 → 裁决字典保证一个名字只 demangle 一次（拒绝也缓存为 `nil` 裁决、不再重试）。与主表冻结不可变的性质相对。 + +- **主要出现在**：`Sources/MachOSymbols/SymbolIndexStore.swift` + +### leg（腿，reader-split） + +同一逻辑按 reader 类型分出的并行实现路径，口语记作「镜像腿 / 文件腿」：`MachOImage`（进程内映射，符号名可直指 LINKEDIT 字符串表、零拷贝）与 `MachOFile`（离线文件，名字须读进私有缓冲）。0001 的 sweep 收集与名字来源都是按腿分叉的。 + +- **主要出现在**：`Sources/MachOSymbols/SymbolIndexStore.swift`（`buildStorageSweep`）、`Sources/MachOSymbols/SymbolTable.swift` + +### materialize（物化） + +从轻量引用（表行号、descriptor、`NodeReference`）按需构造出完整值（`String`、wrapper、`Node` 树）的动作，与「驻留」相对。本项目的内存优化主线就是「驻留只留定位信息，重内容用时物化、用完即弃」：0001 物化符号名，0002 物化 wrapper。物化纪律：每处理一个对象至多物化一次、局部贯穿，不做 per-access。 + +- **延伸阅读**：[提案 0001](Evolutions/0001-symbol-name-offsetization.md)、[提案 0002](Evolutions/0002-declaration-model-descriptor-slimming.md) + +### name source（名字来源） + +`SymbolTable` 里一行的名字字节从哪里读：**mapped 字符串表**（`MachOImage` 行直指镜像 mmap 的 LINKEDIT 字符串表，clean 页、零拷贝，代价是要求镜像保持加载）或**私有缓冲**（`MachOFile` 行与 export-trie 解码名，字节存进表自有的连续缓冲）。`PackedNameReference` 用 1 个 bit 区分两者。 + +- **主要出现在**：`Sources/MachOSymbols/SymbolTable.swift` +- **延伸阅读**：[提案 0001](Evolutions/0001-symbol-name-offsetization.md) + +### NodeStore / NodeReference + +上游 swift-demangling 的 arena 存储：demangle 结果不再是 class `Node` 树，而是扁平缓冲里的节点（12 字节/节点）加一个 `(store, index)` 引用。本仓库的符号索引、声明模型、各级缓存全部换持 `NodeReference`。注意它的 `Hashable` 是 store 身份语义——见「store-identity vs 结构相等」。 + +- **主要出现在**：上游 `swift-demangling`；本仓库消费面见 AGENTS.md「Symbol indexing」段 +- **延伸阅读**：[NodeStoreMigrationPlan.md](Internal/NodeStoreMigrationPlan.md) + +### permutation 二分（permutation binary search） + +不给数据本体排序，而是另存一条「按某序排列的下标数组」（permutation），查询时在这条下标序列上二分。`SymbolTable.rowsSortedByName` 即名字序 permutation：行本体保持插入序不动，名字查找二分这条 `[UInt32]`。替代了被退役的名字键字典 `tableRowByName`。 + +- **主要出现在**：`Sources/MachOSymbols/SymbolTable.swift`（`row(forName:)`） +- **延伸阅读**：[提案 0001](Evolutions/0001-symbol-name-offsetization.md) + +### row / `SymbolRow`（行） + +`SymbolTable` 的最小单位：每个唯一符号名一行，16 字节（canonical offset + `PackedNameReference`），行号（`UInt32`）是全部分类索引引用符号的方式。「一名一行」意味着按名字查询的语义是纯函数——同名必同行。 + +- **主要出现在**：`Sources/MachOSymbols/SymbolTable.swift` + +### store-identity vs 结构相等（structural equality） + +`NodeReference` 的两种相等语义，混用会静默出错：intrinsic `Hashable` 按**store 身份**（同一 store 里的同一下标才相等），跨 store 的结构相同树不相等；**结构相等**（`structurallyEquals` / `StructuralNodeReferenceKey`）逐节点比对，跨 store 成立。规则：键和查询可能来自**不同 store** 的任何 `Dictionary` / `Set` 必须用 `StructuralNodeReferenceKey`，裸 `NodeReference` 键只在单一 hash-consed store 内部安全。踩过的坑：override/vtable 注释丢失、subscript getter/setter 分桶、merged thunk 重复输出。 + +- **主要出现在**：`Sources/MachOSymbols/StructuralNodeReferenceKey.swift`；键位清单见 AGENTS.md「Symbol indexing」段 + +### sweep（构建扫描） + +对一个镜像的**全量符号一遍扫过**的批处理构建过程：`SymbolIndexStore` 首次索引某镜像时，`buildStorageSweep` 遍历整张符号表 + export trie，收集行、对每个 Swift 符号 demangle 一次、把结果分类进各查询索引——与之相对的是查询期的按需单点操作。RuntimeViewer 语境里「按需索引 sweep」指用户打开某镜像才触发这一遍构建；sweep 期的临时缓冲是瞬态内存峰值的来源（完即释放）。 + +- **主要出现在**：`Sources/MachOSymbols/SymbolIndexStore.swift`（`buildStorageSweep`） +- **延伸阅读**：[提案 0001](Evolutions/0001-symbol-name-offsetization.md)、[SymbolIndexStoreMemoryOptimization.md](Internal/SymbolIndexStoreMemoryOptimization.md) + +### trailing objects + +Swift runtime 的 descriptor 布局惯例：固定头之后按 flags 跟着可变数量的附加记录（vtable 方法描述符、resilient witnesses、泛型上下文等），源自 C++ 侧的 `TrailingObjects` 模板。本仓库的高层 wrapper 构造时把它们全部解析成 Swift 数组——0002 要治理的驻留正是这些解析产物。 + +- **主要出现在**：`Sources/MachOSwiftSection/Models/`（各 wrapper 的 `initialize` 尾部解析） + +### wrapper vs descriptor（高层包装 vs 描述符） + +同一个二进制实体的两级表示，易混淆：**descriptor**（`ClassDescriptor`、`ProtocolConformanceDescriptor` 等）是原始布局 + 位置的薄值（几十字节，可随时重读）；**wrapper**（`Class` / `Struct` / `Enum` / `ProtocolConformance` / `Protocol` / `TypeContextWrapper`）是构造时急切解析全部 trailing objects 的完整值（内联数百字节 + 堆数组）。规则口诀：descriptor 可驻留，wrapper 应物化。 + +- **主要出现在**：`Sources/MachOSwiftSection/Models/` +- **延伸阅读**：[提案 0002](Evolutions/0002-declaration-model-descriptor-slimming.md) diff --git a/Documentations/Internal/SymbolIndexStoreMemoryOptimization.md b/Documentations/Internal/SymbolIndexStoreMemoryOptimization.md new file mode 100644 index 00000000..750127ca --- /dev/null +++ b/Documentations/Internal/SymbolIndexStoreMemoryOptimization.md @@ -0,0 +1,76 @@ +# SymbolIndexStore 内存优化专题 + +日期：2026-08-09（覆盖 2026-07 至 2026-08 的三波优化） + +本文是 `SymbolIndexStore` 内存表示演进的专题汇总：三波优化各自解决什么、最终的存储模型长什么样、付出了哪些约束、实测收益多少。行文参照上游 swift-demangling 的同类专题 `Documentations/SubtreeInterning.md`（那篇讲 `Node` 树的全子树 hash-consing，是本文第一波的上游地基）。术语（sweep、腿、名字来源、detach、物化等）首次出现不再逐个展开，见[术语表](../Glossary.md)。 + +## 动机 + +`SymbolIndexStore` 是符号查询的底座：RuntimeViewer 每打开一个镜像，就对它做一次 sweep——遍历全量符号表 + export trie，把每个 Swift 符号 demangle 一次并分类进各查询索引。这份索引是长命的（用户浏览期间一直服务查询），所以它的**驻留表示**直接决定 RuntimeViewer 的稳态内存。 + +优化前的驻留表示在三个层面上都是「把便宜的东西复制成贵的」： + +1. demangle 结果驻留为 class `Node` 树——每节点 48 字节 malloc 对象，且全局 `NodeCache` 随浏览无界增长（五镜像负载实测 110 万 `Node`、进程 842 MB）； +2. 各级缓存（`MetadataReaderCache`、逐名字的 mini `NodeStore`）延续 class 树持有形态，迁移后仍残留 18.4 万 `Node`； +3. 每个符号名驻留为独立 `String`（49.4 万个 / 68.7 MiB）——而这些名字的原文本来就躺在镜像 mmap 的 LINKEDIT 字符串表里（clean 页、不计 footprint），eager 拷贝等于把免费页复制成付费脏页。 + +## 范围 + +本文只覆盖 `MachOSymbols` 模块（`SymbolIndexStore` / `SymbolTable` / `DemangledSymbol`）及其直接协作的缓存层。两个相邻但不属于本文的战线：`Node` 存储本身的优化在上游 swift-demangling（见其 `SubtreeInterning.md` 与 evolution 0008/0010）；声明模型（`TypeDefinition` 等）的驻留优化是[提案 0002](../Evolutions/0002-declaration-model-descriptor-slimming.md)（In Review）。 + +## 三波优化 + +### 第一波：NodeStore 迁移——class 树换 arena 存储（2026-07，Stage 0–5） + +上游把 demangle 结果从 class `Node` 树换成 arena `NodeStore`（12 字节/节点的扁平缓冲 + `(store, index)` 引用），本仓库跟进分五个阶段迁移：build sweep 改为 cache-free 的 transient demangle（瞬时树分类后即弃，intern 进每镜像一个 `NodeStoreBuilder`）；`Symbol` 表压缩；声明层全面换持 `NodeReference`；散点全部改 transient demangling，让全局 `NodeCache` 彻底停止随浏览增长。 + +这一波真正解决的是**无界增长**：迁移前后 `Node` 实例 110 万 → 18.4 万，进程 842 → 434 MB。代价是引入了 store-identity 与结构相等两种 `NodeReference` 语义，跨 store 键必须走 `StructuralNodeReferenceKey`——Stage 5a 的 override/vtable 注释丢失回归就是这个坑（详见 [NodeStoreMigrationPlan.md](NodeStoreMigrationPlan.md)）。 + +### 第二波：小 store 合并与缓存清退（2026-08-08） + +两项收尾把第一波的残留清干净： + +- **`SharedNodeStore` 汇入**（上游 evolution 0010）：`NodeReference(interning:)` 逐名字新建私有 mini store 的形态（6.7 万个 store、跨名字去重被切断）收敛为按作用域共享的 `InternedNodeReferenceCache`——镜像键作用域随镜像驱逐，进程键作用域服务无 Mach-O 上下文的调用方。见 [SharedNodeStoreMigration.md](SharedNodeStoreMigration.md)。 +- **`MetadataReaderCache` 清退**：三张缓存字典从持有 class `Node` 树改持 `NodeReference`（存进上述作用域 store），公开 API 与 103 处调用点零改动。RuntimeViewer 实景存活 `Node` 207,489 → **44**（−99.98%）。见 [MetadataReaderCacheRetirement.md](MetadataReaderCacheRetirement.md)。 + +### 第三波：符号名 offset 化（2026-08-08，[提案 0001](../Evolutions/0001-symbol-name-offsetization.md)） + +第一、二波之后的全景剖析把头号大户定位到 49.4 万个驻留符号名 `String`。0001 把它们换成字符串表引用、按需物化： + +- **`SymbolTable`**：每个唯一符号名一行 16 字节 `SymbolRow`（canonical offset + `PackedNameReference`），不再驻留任何名字 `String`。名字来源分两腿——镜像行直指 mmap 的 LINKEDIT 字符串表（零拷贝，要求镜像保持加载），文件行与 export-trie 名进表自有的私有连续缓冲。 +- **sweep 收集按 reader 分腿**：镜像腿在 `nameC` 指针上做字节级 `isSwiftSymbol` 判定（`nameBytesHaveSwiftManglingPrefix`，与 `String` 版逐条等价、测试钉住），非 Swift 符号从头到尾不物化名字；文件腿保持 `String` 面。 +- **名字查找退役字典换二分**：build 期临时去重字典 freeze 时丢弃，查询走名字序 permutation（`rowsSortedByName`）上的字节级二分。 +- **vend 面按需物化**：`Symbol` / `DemangledSymbol` 的公开形态不变，名字在读取时从表物化；存进声明模型的长命值仍需 detach（`SymbolTableRetentionTests` 钉住六个存储点）。 + +实施期的三处偏差（`Span` 家族运行时可用性 macOS 26+ 改用 `UnsafeBufferPointer`、`RigidArray` 借用人体工学不足改精确容量 `Array`、搭车项裁剪）全部记录在 0001 决策日志。 + +## 今天的存储模型 + +一个镜像的 `Storage` 冻结后由这些部分组成：`SymbolTable`（行 + 双名字来源 + 名字序 permutation）；`rootNodeIndexByTableRow`（每行的 demangle 根节点，arena 内 4 字节索引，`nil` 裁决也缓存——demangler 拒绝过的名字不再重试）；一组分类索引（全部以 4 字节 `UInt32` 行号引用符号，桶形态见后续方向）；late-name 路径的可追加 side store 服务 sweep 之外的零星名字。整个 `Storage` 随镜像驱逐（`removeSubIndexer(_:)`）整体释放。 + +## 取舍与影响面 + +- **镜像卸载**：镜像行的名字读取依赖镜像保持加载——`dlclose` 后 mapped 基址悬垂。RuntimeViewer 与系统框架场景从不卸载，记为接受项（0001「风险与接受的约束」）。 +- **文件腿构建期峰值**：build 期去重字典的 `String` 键与私有缓冲在 freeze 前短暂持有同一批名字字节两份，实测 maxRSS +15 MiB，freeze 后回落；镜像腿无此代价。稳态是目标指标，峰值记录在案。 +- **查询 CPU**：名字查找从字典 O(1) 变二分 log₂(19 万) ≈ 18 次字节比较，实测 interface 生成 wall-clock 持平（72.5s vs 70.0s，噪声带内）。若日后 profiling 出热点，退路是字节哈希索引（结构兼容）。 +- **detach 纪律**：共享表模型把「长命值必须 detach」升格为硬契约，新增存储点忘记 detach 会被 `SymbolTableRetentionTests` 逮住。 +- **键语义纪律**：跨 store 的 `NodeReference` 键一律 `StructuralNodeReferenceKey`，裸键只在单 store 批次内安全——AGENTS.md「Symbol indexing」段维护着键位清单。 + +## 实测收益（RuntimeViewer 五镜像稳态） + +| 指标 | 优化前 | 三波之后 | +|---|---|---| +| 进程 footprint | 842 MB（NodeStore 迁移前）/ 470–480 MB（第三波前基线） | **322 MB** | +| 存活 `Node`（class 实例） | 1,101,318 | 44 | +| 驻留符号名 `String` | 494,000 个 / 68.7 MiB | 0（StringStorage 全类 784k / 84.2 → 356k / 31.3 MiB） | +| `SymbolIndexStore` 簇 | 214.6 MiB | 120.9 MiB | +| 索引期瞬态峰值 | 893 MB | 808 MB | + +11 小时长跑复核无漂移（堆 285 vs 干净跑 283 MiB）；用户观察到的「反复飙 800+ MB 后回落」确认为 sweep 瞬态（28 并发工人的临时缓冲，完即释放），非泄漏。 + +## 后续可选方向 + +- **`[UInt32]` 行号桶扁平化**：45 万个小数组桶（38.8 MiB）换单元素内联表示——[提案 0003](../Evolutions/0003-symbol-row-bucket-flattening.md)（In Review）。 +- **声明模型 descriptor 化**：相邻簇（41.3 + 33.4 MiB）的同方法论治理——[提案 0002](../Evolutions/0002-declaration-model-descriptor-slimming.md)（In Review）。 +- **sweep 限流 / 分批**：用索引速度换瞬态峰值（800+ MB 尖峰摊平），RuntimeViewer 侧候选、未拍板。 +- **上游 demangle 字节入口**：sweep 的 demangle 输入仍需一个瞬时 `String`，等上游 swift-demangling 的 demangle-bytes 提案落地后一行替换（与 0001 解耦对接，注意 `Span` 家族的部署下限坑）。 diff --git a/Documentations/README.md b/Documentations/README.md index b6c0e38f..0f7356e8 100644 --- a/Documentations/README.md +++ b/Documentations/README.md @@ -9,6 +9,7 @@ Documentation is split by audience. > Note that `Tests/Projects/SymbolTests` does enable library evolution — that is the test project, > not the library itself. > Evolution proposals live in [`Evolutions/`](Evolutions/README.md) (status table + numbering there); the first one is [0001-symbol-name-offsetization.md](Evolutions/0001-symbol-name-offsetization.md). +> Project-specific terminology lives in [`Glossary.md`](Glossary.md) (Chinese; register new terms in the same batch that introduces them — cross-project terms stay in the global glossary). ## External — for library users / other developers @@ -76,6 +77,8 @@ required by `Version.swift`'s bump contract). | [Evolutions/0001-symbol-name-offsetization.md](Evolutions/0001-symbol-name-offsetization.md) | **提案 0001（Implemented）**：`SymbolIndexStore` 符号名 offset 化——49.4 万个驻留符号名 `String`（68.7 MiB，RV 实测堆内头号大户的最大单项）换成字符串表引用按需物化，`tableRowByName` 字典退役换名字序字节级二分，字节级 `isSwiftSymbol` 判定消掉非 Swift 符号的全部瞬时 String；公开 API 零改动。含 Swift 6.2 Span 家族与 swift-collections 1.6.0 选型裁决（实施时因部署下限改用 `UnsafeBufferPointer` / 精确容量 Array，见决策日志）。 | | [Evolutions/0002-declaration-model-descriptor-slimming.md](Evolutions/0002-declaration-model-descriptor-slimming.md) | **提案 0002（In Review）**：声明模型 descriptor 化——`TypeDefinition` / `ExtensionDefinition` / `ProtocolDefinition` 不再终身驻留急切解析的胖 wrapper（`TypeContextWrapper` 472 B × 2 份、`ProtocolConformance` 及其 `[ResilientWitness]`、`Protocol`），改存几十字节级 descriptor 引用，trailing 解析在惰性 `index()` / 打印期临时物化；`parentContext` 降级为索引期局部载体。0001 后堆内新头部（声明模型 41.3 + MachOSwiftSection 簇 33.4 MiB）的对症案，预估再省 30–45 MiB；破坏性 API 变更（三处属性换形态，机械迁移）。 | | [Evolutions/0003-symbol-row-bucket-flattening.md](Evolutions/0003-symbol-row-bucket-flattening.md) | **提案 0003（In Review）**：`SymbolIndexStore` `[UInt32]` 行号桶扁平化——45 万个小数组（38.8 MiB，绝大多数单元素、各付 ~48 B 堆分配）换「单元素内联、多元素落堆」的 `SymbolRowBucket`，覆盖 `symbolRowsByOffset` 与三族 `MemberSymbolRows` 叶子桶；预估省 15–25 MiB，纯内部零 API 变化。0001「非目标」点名候选的正式立项。 | +| [Glossary.md](Glossary.md) | **项目术语表**：sweep、腿（reader-split leg）、名字来源、detach、物化、permutation 二分、store-identity vs 结构相等、wrapper vs descriptor、桶、trailing objects 等本项目自造词与特定用法；跨项目通用术语在全局表（iCloud Global），不重复登记。提案与专题文档引入新术语时同批登记。 | +| [SymbolIndexStoreMemoryOptimization.md](Internal/SymbolIndexStoreMemoryOptimization.md) | **`SymbolIndexStore` 内存优化专题**：三波优化的汇总叙事（NodeStore 迁移 Stage 0–5 → 小 store 合并与 `MetadataReaderCache` 清退 → 提案 0001 符号名 offset 化）、今天的存储模型一页速览、取舍与影响面（镜像卸载 / 文件腿峰值 / detach 与键语义纪律）、实测收益表（842 → 322 MB 全曲线）与后续可选方向。行文参照上游 swift-demangling 的 `SubtreeInterning.md`。 | | [DeclarationModelMemoryFootprint.md](Internal/DeclarationModelMemoryFootprint.md) | NodeStore 迁移后的声明模型内存足迹量测：`TypeDefinition` 1272 字节的逐属性构成（两份 `TypeContextWrapper` 占 74%）、`TypeContextWrapper` 按最大 case `Class` 定尺的原因、`parentContext` 是被当成永久字段的临时值、mini-store 增殖与 `MetadataReaderCache` 仍持 `Node` 树；含四项可回收估算（合计 ~8–10%）与「当前不建议实施、应先剖析其余 90%」的结论。 | | [NodeStoreMigrationOpenIssues.md](Internal/NodeStoreMigrationOpenIssues.md) | `feature/node-store-migration` 的**已确认问题清单**（2026-07-28 两轮审查 + 复核）。第一节是前一轮修复自身的两处缺口（Catalyst 降级只覆盖 framework 形态导致 plain dylib 仍平局、`appendRowIfAbsent` 线性扫描），已于 2026-07-29 闭环并保留成因；第二节起为仍然打开或已裁决的条目：`structuralHash` 每文本节点分配 `String` 与 `ABIKey` 每 key materialize 两条**已按上游设计终审关闭**（0.5.1 保持现状且上游说明不改，见 ReviewAdjudications A1/A2）、`memberSymbols` 改为线性 + 全树比对（**已证实量级可忽略，不是回归**）；`lateDemangledNode` 持锁 demangle 与 `ProtocolConformanceDumper` 的 materialize 分支两条**已于 2026-08-03 闭环**（连同失败名裁决缓存、dump 路径引用化）、build sweep 串行且每符号跨线程往返（上游）、`ABIKey` 每 key materialize，以及四条代码卫生项；两个公开查询 API 的字典键一条**已于 2026-08-03 裁决为不修**（类型级 SPI），rebase 前置事项一条前提已过期但压着两条仍成立的注意事项。逐条注明成因、影响面与「该在哪里修」。与 `Reviews/` 下两份审查记录冲突时以后者为准。 | | [SystemFrameworkRenderingVerification.md](Internal/SystemFrameworkRenderingVerification.md) | 系统框架渲染 A/B 验证流程（大重构必跑）：两个检出对同一批真实输入（归档 dyld cache → 无则当前系统 cache、模拟器 runtime → 无则现有 runtime、当前系统 MachOImage）逐字节比对 dump+interface；入口脚本 `Scripts/run-rendering-ab-verification.py`，含回退规则、踩坑清单（`-p` 消歧、fat 二进制 `-a`、同 boot 会话、scratch 隔离）与 2026-08-03 基线运行记录。 | From 581ed49ad52b758ae078b565b34303de2529c0c7 Mon Sep 17 00:00:00 2001 From: Mx-Iris Date: Sun, 9 Aug 2026 12:40:06 +0800 Subject: [PATCH 49/77] docs(0002): record the no-caching materialization adjudication from review --- .../Evolutions/0002-declaration-model-descriptor-slimming.md | 3 +++ 1 file changed, 3 insertions(+) diff --git a/Documentations/Evolutions/0002-declaration-model-descriptor-slimming.md b/Documentations/Evolutions/0002-declaration-model-descriptor-slimming.md index 6e652f38..0c4241d9 100644 --- a/Documentations/Evolutions/0002-declaration-model-descriptor-slimming.md +++ b/Documentations/Evolutions/0002-declaration-model-descriptor-slimming.md @@ -89,6 +89,8 @@ extension ProtocolDefinition { 三者都是薄封装（转调现成的 `forTypeContextDescriptorWrapper` / `ProtocolConformance(descriptor:in:)` / `Protocol(descriptor:in:)`），throws 语义与今天模型构建期相同。 +**物化结果不缓存**——每次用时重新物化，这是有意的：缓存存回定义对象会把省掉的内存按浏览顺序逐个攒回来，与本案目的直接冲突；物化本身是映射内存的一遍顺序解析 + 几次小数组分配（微秒级），比消费它的 demangle + 打印便宜几个数量级；且触发频度天然有界——`index()` 有 `isIndexed` 挡板一生一次，打印每次查看一次，specialize 是低频交互，没有热循环反复物化同一定义的路径（首次查看一个类型 = index + 打印共 2 次，之后每次重看 1 次）。若落地后 profiling 显示物化是热点，退路是镜像作用域、内存压力可驱逐的小缓存——结构兼容、不动 API，后补而非前置。 + ### 构造路径 - 索引器 sweep 今天就持有完整 wrapper（要用它派生 `typeName`）：`TypeDefinition` 的 init 继续收 wrapper，**内部只存其 `typeContextDescriptorWrapper`**——sweep 的解析工作量不变，变化只是解析产物在 init 返回后即可释放。 @@ -170,3 +172,4 @@ extension ProtocolDefinition { | 日期 | 变更 | 说明 | |------|------|------| | 2026-08-09 | Created as In Review | 0001 落地后 RV 复测把声明模型 41.3 MiB + MachOSwiftSection 解析结构 33.4 MiB 定位为堆内新头部；优化面普查（`index()` 惰性 × wrapper 急切驻留的错配）成文本案；用户批准立项（「可以，写提案」）。 | +| 2026-08-09 | 审阅补充：物化结果不缓存 | 审阅期用户问「物化后会缓存吗」；裁决为不缓存、每次用时重新物化（缓存会按浏览顺序把内存攒回来；物化微秒级、频度有界），可驱逐小缓存仅作 profiling 证明热点后的退路。已写入「详细设计 · 物化接口」。 | From 808f1acd807b694298d38279a6a1ad231dc97901 Mon Sep 17 00:00:00 2001 From: Mx-Iris Date: Sun, 9 Aug 2026 12:45:07 +0800 Subject: [PATCH 50/77] docs(0002): record the global-search compatibility adjudication from review --- .../Evolutions/0002-declaration-model-descriptor-slimming.md | 1 + 1 file changed, 1 insertion(+) diff --git a/Documentations/Evolutions/0002-declaration-model-descriptor-slimming.md b/Documentations/Evolutions/0002-declaration-model-descriptor-slimming.md index 0c4241d9..dfbea070 100644 --- a/Documentations/Evolutions/0002-declaration-model-descriptor-slimming.md +++ b/Documentations/Evolutions/0002-declaration-model-descriptor-slimming.md @@ -173,3 +173,4 @@ extension ProtocolDefinition { |------|------|------| | 2026-08-09 | Created as In Review | 0001 落地后 RV 复测把声明模型 41.3 MiB + MachOSwiftSection 解析结构 33.4 MiB 定位为堆内新头部；优化面普查（`index()` 惰性 × wrapper 急切驻留的错配）成文本案；用户批准立项（「可以，写提案」）。 | | 2026-08-09 | 审阅补充：物化结果不缓存 | 审阅期用户问「物化后会缓存吗」；裁决为不缓存、每次用时重新物化（缓存会按浏览顺序把内存攒回来；物化微秒级、频度有界），可驱逐小缓存仅作 profiling 证明热点后的退路。已写入「详细设计 · 物化接口」。 | +| 2026-08-09 | 审阅补充：与 RV 全局搜索不冲突 | 审阅期用户问「后面 RV 想做全局搜索咋办」；裁决为不冲突且同向——搜索匹配的是名字（类型名 / 成员名 / 符号名，均为冻结轻数据），本案清退的 vtable / witnesses 解析产物不是搜索目标，命中后看详情恰是「用时物化」场景；descriptor 化让「全量建骨架不索引成员」的搜索支撑形态更可行（~400 B vs 1272 B + 堆数组/类型）。真正的功课在「按需 per-image 索引 vs 全量覆盖」这层（本案之前就存在），届时另立搜索索引提案（候选形态：惰性全量 sweep + 限流 / 名字层前置倒排 + 库侧行收集模式 / 持久化索引），不构成本案的阻塞或修改项。 | From 87d4dfd7a5c39e0ba3bfc6c78aff6b9682e08c55 Mon Sep 17 00:00:00 2001 From: Mx-Iris Date: Sun, 9 Aug 2026 12:55:35 +0800 Subject: [PATCH 51/77] docs(evolutions): mark 0002 and 0003 as Accepted after user review --- .../Evolutions/0002-declaration-model-descriptor-slimming.md | 3 ++- .../Evolutions/0003-symbol-row-bucket-flattening.md | 3 ++- Documentations/Evolutions/README.md | 4 ++-- 3 files changed, 6 insertions(+), 4 deletions(-) diff --git a/Documentations/Evolutions/0002-declaration-model-descriptor-slimming.md b/Documentations/Evolutions/0002-declaration-model-descriptor-slimming.md index dfbea070..1a36362c 100644 --- a/Documentations/Evolutions/0002-declaration-model-descriptor-slimming.md +++ b/Documentations/Evolutions/0002-declaration-model-descriptor-slimming.md @@ -1,6 +1,6 @@ # 0002 - 声明模型 descriptor 化：TypeDefinition / ExtensionDefinition / ProtocolDefinition 不再驻留急切解析的胖 wrapper -- **状态**: In Review +- **状态**: Accepted - **作者**: JH - **创建日期**: 2026-08-09 - **最后更新**: 2026-08-09 @@ -174,3 +174,4 @@ extension ProtocolDefinition { | 2026-08-09 | Created as In Review | 0001 落地后 RV 复测把声明模型 41.3 MiB + MachOSwiftSection 解析结构 33.4 MiB 定位为堆内新头部；优化面普查（`index()` 惰性 × wrapper 急切驻留的错配）成文本案；用户批准立项（「可以，写提案」）。 | | 2026-08-09 | 审阅补充：物化结果不缓存 | 审阅期用户问「物化后会缓存吗」；裁决为不缓存、每次用时重新物化（缓存会按浏览顺序把内存攒回来；物化微秒级、频度有界），可驱逐小缓存仅作 profiling 证明热点后的退路。已写入「详细设计 · 物化接口」。 | | 2026-08-09 | 审阅补充：与 RV 全局搜索不冲突 | 审阅期用户问「后面 RV 想做全局搜索咋办」；裁决为不冲突且同向——搜索匹配的是名字（类型名 / 成员名 / 符号名，均为冻结轻数据），本案清退的 vtable / witnesses 解析产物不是搜索目标，命中后看详情恰是「用时物化」场景；descriptor 化让「全量建骨架不索引成员」的搜索支撑形态更可行（~400 B vs 1272 B + 堆数组/类型）。真正的功课在「按需 per-image 索引 vs 全量覆盖」这层（本案之前就存在），届时另立搜索索引提案（候选形态：惰性全量 sweep + 限流 / 名字层前置倒排 + 库侧行收集模式 / 持久化索引），不构成本案的阻塞或修改项。 | +| 2026-08-09 | In Review → Accepted | 用户审核通过（「审核通过，开始实现」），按落地步骤开工，第一步为下游消费点普查。 | diff --git a/Documentations/Evolutions/0003-symbol-row-bucket-flattening.md b/Documentations/Evolutions/0003-symbol-row-bucket-flattening.md index cbb261b7..731e7679 100644 --- a/Documentations/Evolutions/0003-symbol-row-bucket-flattening.md +++ b/Documentations/Evolutions/0003-symbol-row-bucket-flattening.md @@ -1,6 +1,6 @@ # 0003 - SymbolIndexStore `[UInt32]` 行号桶扁平化：单元素桶内联化 -- **状态**: In Review +- **状态**: Accepted - **作者**: JH - **创建日期**: 2026-08-09 - **最后更新**: 2026-08-09 @@ -106,3 +106,4 @@ enum SymbolRowBucket { | 日期 | 变更 | 说明 | |------|------|------| | 2026-08-09 | Created as In Review | 0001「非目标」点名的候选正式立项；RV 实测簇 38.8 MiB / 45 万个为输入；用户批准立项（「可以，写提案」）。 | +| 2026-08-09 | In Review → Accepted | 用户审核通过（「审核通过，开始实现」），与 0002 同批开工，两案独立实施。 | diff --git a/Documentations/Evolutions/README.md b/Documentations/Evolutions/README.md index e8d95fc4..43a87f06 100644 --- a/Documentations/Evolutions/README.md +++ b/Documentations/Evolutions/README.md @@ -7,5 +7,5 @@ | # | 标题 | 状态 | |---|------|------| | [0001](0001-symbol-name-offsetization.md) | SymbolIndexStore 符号名 offset 化：驻留字符串换字符串表引用 | Implemented | -| [0002](0002-declaration-model-descriptor-slimming.md) | 声明模型 descriptor 化：TypeDefinition / ExtensionDefinition / ProtocolDefinition 不再驻留急切解析的胖 wrapper | In Review | -| [0003](0003-symbol-row-bucket-flattening.md) | SymbolIndexStore `[UInt32]` 行号桶扁平化：单元素桶内联化 | In Review | +| [0002](0002-declaration-model-descriptor-slimming.md) | 声明模型 descriptor 化：TypeDefinition / ExtensionDefinition / ProtocolDefinition 不再驻留急切解析的胖 wrapper | Accepted | +| [0003](0003-symbol-row-bucket-flattening.md) | SymbolIndexStore `[UInt32]` 行号桶扁平化：单元素桶内联化 | Accepted | From 640235cf55e611103f54813fc4a0e5b1b39a2f3f Mon Sep 17 00:00:00 2001 From: Mx-Iris Date: Sun, 9 Aug 2026 13:56:34 +0800 Subject: [PATCH 52/77] perf(symbols): inline single-row buckets in SymbolIndexStore (evolution 0003) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Replace the [UInt32] bucket values of symbolRowsByOffset and the three MemberSymbolRows leaf families with SymbolRowBucket, an inline-single / spill-to-heap enum: the dominant single-row case (87.6% on the fixture, ~45k of ~450k buckets multiple at RuntimeViewer scale) now lives in the dictionary slot instead of paying a per-key array allocation. Iteration order stays insertion order, so query output is byte-identical — pinned by a seven-pair render A/B (simulator SwiftUI/SwiftData/SwiftUICore dump + interface, host dyld-shared-cache SwiftUI dump + interface). - SymbolRowBucket: RandomAccessCollection, append migrates single -> multiple on the second element, stride pinned <= 16 B in compactValueLayouts - demangledSymbols(atRows:) generalized to some Sequence; query APIs and vend shapes unchanged (zero public API change) - Storage.bucketFormStatisticsForTesting() + a resident unit test assert single-row dominance; the IntegrationTests baseline metrics print the ratio Evolution proposal 0003 -> Implemented (RuntimeViewer heap re-measure pending downstream pickup). --- AGENTS.md | 2 +- .../0003-symbol-row-bucket-flattening.md | 11 ++-- Documentations/Evolutions/README.md | 2 +- Documentations/Glossary.md | 5 +- Sources/MachOSymbols/SymbolIndexStore.swift | 62 +++++++++++++++---- Sources/MachOSymbols/SymbolRowBucket.swift | 59 ++++++++++++++++++ .../SymbolIndexStoreBaselineTests.swift | 4 ++ .../SymbolIndexStoreFixtureTests.swift | 50 +++++++++++++++ 8 files changed, 173 insertions(+), 22 deletions(-) create mode 100644 Sources/MachOSymbols/SymbolRowBucket.swift diff --git a/AGENTS.md b/AGENTS.md index b42c6aa4..b9280897 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -187,7 +187,7 @@ Printing and indexing are peers — neither depends on the other. - **MachOFoundation** - Combines reading, symbols, pointers - **MachOReading** - File reading abstractions - **MachOResolving** - Address/offset resolution -- **MachOSymbols** - Symbol table parsing and demangling +- **MachOSymbols** - Symbol table parsing and demangling. `SymbolIndexStore`'s offset and member indexes hold their row lists in `SymbolRowBucket` (evolution proposal 0003): the dominant single-row case stays inline in the dictionary slot, only a bucket that collects a second row allocates an array; iteration order is insertion order, so query output is byte-identical to the former `[UInt32]` buckets - **MachOPointers** - Pointer types (relative, indirect, etc.) - **MachOCaches** - dyld shared cache support - **MachOExtensions** - Extensions to MachOKit types. `resolveBind(fileOffset:)` resolves bind slots from chained fixups AND, when those are absent, from the legacy `LC_DYLD_INFO(_ONLY)` bind opcode streams (pre-macOS 12 / iOS 16 deployment targets, e.g. iOS 15.5 simulator frameworks) via a lazily built file-offset → symbol-name index; the arm64e threaded legacy format is deliberately not indexed. Pinned by `LegacyDyldInfoBindTests`, whose fixture is compiled on the fly with `-target arm64-apple-macosx11.0` to force the legacy format. diff --git a/Documentations/Evolutions/0003-symbol-row-bucket-flattening.md b/Documentations/Evolutions/0003-symbol-row-bucket-flattening.md index 731e7679..af1ae4ca 100644 --- a/Documentations/Evolutions/0003-symbol-row-bucket-flattening.md +++ b/Documentations/Evolutions/0003-symbol-row-bucket-flattening.md @@ -1,6 +1,6 @@ # 0003 - SymbolIndexStore `[UInt32]` 行号桶扁平化：单元素桶内联化 -- **状态**: Accepted +- **状态**: Implemented - **作者**: JH - **创建日期**: 2026-08-09 - **最后更新**: 2026-08-09 @@ -96,10 +96,10 @@ enum SymbolRowBucket { ## 落地步骤 -1. `SymbolRowBucket` 实现 + 布局断言（`compactValueLayouts`）+ 单测（append 迁移、迭代序、`contains`）。 -2. `symbolRowsByOffset` 与三族 `MemberSymbolRows` 叶子桶替换；freeze 处一次性单元素占比统计（验收用，随后移除或留在 IntegrationTests）。 -3. 全量 `swift test --skip IntegrationTests` 同数全绿；渲染 A/B 三 reader 逐字节一致（迭代序不变的验收）。 -4. RV heap 复测：`[UInt32]` 簇 38.8 → 预期 ~15–20 MiB。 +1. ✅ `SymbolRowBucket` 实现 + 布局断言（`compactValueLayouts` 钉 `stride ≤ 16`）+ 单测（append 迁移、迭代序、`contains`——`symbolRowBucketAppendMigrationAndIterationOrder`）。 +2. ✅ `symbolRowsByOffset` 与三族 `MemberSymbolRows` 叶子桶替换（`demangledSymbols(atRows:)` 泛化为 `some Sequence`，查询出口形态不变）；单元素占比统计落为 `Storage.bucketFormStatisticsForTesting()`——常驻单测 `rowBucketsAreDominatedBySingleRowForm` 断言并打印，另在 IntegrationTests 的 baseline 指标里加了一行。fixture（SymbolTestsCore，MachOFile leg）实测 **87.6% 单元素**（6687 单 / 948 多），达到 ≥85% 预期带。 +3. ✅ 全量 `swift test --skip IntegrationTests` 1343 全绿（含新增 2 项桶单测，无删减）；渲染 A/B（iOS 18.5 模拟器 SwiftUI / SwiftData / SwiftUICore 的 dump + interface，另加宿主机 dyld shared cache 的 SwiftUI dump + interface——canonical/raw 双键注册正是本案改动面）全部逐字节一致（interface 输出剥离日志行首时间戳后比对）。 +4. RV heap 复测：`[UInt32]` 簇 38.8 → 预期 ~15–20 MiB。**待下游拿到本分支后进行。** ## 决策日志 @@ -107,3 +107,4 @@ enum SymbolRowBucket { |------|------|------| | 2026-08-09 | Created as In Review | 0001「非目标」点名的候选正式立项；RV 实测簇 38.8 MiB / 45 万个为输入；用户批准立项（「可以，写提案」）。 | | 2026-08-09 | In Review → Accepted | 用户审核通过（「审核通过，开始实现」），与 0002 同批开工，两案独立实施。 | +| 2026-08-09 | Accepted → Implemented | 落地步骤 1–3 完成：`SymbolRowBucket`（`RandomAccessCollection`，单元素内联、次元素起落堆、插入序迭代）替换四处桶；fixture 单元素占比 87.6%；全量 1343 绿；A/B 七对（含 dyld cache 两对）逐字节一致。步骤 4（RV heap 复测）待下游拿到分支后进行。 | diff --git a/Documentations/Evolutions/README.md b/Documentations/Evolutions/README.md index 43a87f06..3f893813 100644 --- a/Documentations/Evolutions/README.md +++ b/Documentations/Evolutions/README.md @@ -8,4 +8,4 @@ |---|------|------| | [0001](0001-symbol-name-offsetization.md) | SymbolIndexStore 符号名 offset 化：驻留字符串换字符串表引用 | Implemented | | [0002](0002-declaration-model-descriptor-slimming.md) | 声明模型 descriptor 化：TypeDefinition / ExtensionDefinition / ProtocolDefinition 不再驻留急切解析的胖 wrapper | Accepted | -| [0003](0003-symbol-row-bucket-flattening.md) | SymbolIndexStore `[UInt32]` 行号桶扁平化：单元素桶内联化 | Accepted | +| [0003](0003-symbol-row-bucket-flattening.md) | SymbolIndexStore `[UInt32]` 行号桶扁平化：单元素桶内联化 | Implemented | diff --git a/Documentations/Glossary.md b/Documentations/Glossary.md index 3836efe9..d448d068 100644 --- a/Documentations/Glossary.md +++ b/Documentations/Glossary.md @@ -32,9 +32,10 @@ ### bucket（桶） -分类索引里「一个键对应的一组符号表行号」，值形态是 `[UInt32]` 小数组（如 `symbolRowsByOffset` 的值、`MemberSymbolRows` 的叶子）。绝大多数桶只有一个元素，却各付一次堆分配——这是提案 0003 的对象。 +分类索引里「一个键对应的一组符号表行号」（如 `symbolRowsByOffset` 的值、`MemberSymbolRows` 的叶子）。旧形态是 `[UInt32]` 小数组——绝大多数桶只有一个元素，却各付一次堆分配；提案 0003 落地后值形态为 `SymbolRowBucket`（单元素内联于字典槽，第二个元素起才落堆数组），迭代序保持插入序。 -- **主要出现在**：`Sources/MachOSymbols/SymbolIndexStore.swift` +- **主要出现在**：`Sources/MachOSymbols/SymbolIndexStore.swift`、`Sources/MachOSymbols/SymbolRowBucket.swift` +- **延伸阅读**：[提案 0003](Evolutions/0003-symbol-row-bucket-flattening.md) - **延伸阅读**：[提案 0003](Evolutions/0003-symbol-row-bucket-flattening.md) ### detach（脱表，`detachedFromSharedTable()`） diff --git a/Sources/MachOSymbols/SymbolIndexStore.swift b/Sources/MachOSymbols/SymbolIndexStore.swift index 6490c93f..20725285 100644 --- a/Sources/MachOSymbols/SymbolIndexStore.swift +++ b/Sources/MachOSymbols/SymbolIndexStore.swift @@ -130,7 +130,7 @@ public final class SymbolIndexStore: SharedCache, @unc } public final class Storage: @unchecked Sendable { - typealias MemberSymbolRows = OrderedDictionary> + typealias MemberSymbolRows = OrderedDictionary> /// The frozen arena holding every demangled node of this image. /// All `NodeReference` values vended by this storage point into it. @@ -192,8 +192,11 @@ public final class SymbolIndexStore: SharedCache, @unc /// Plain `Dictionary`: the only consumer is the keyed lookup in /// `symbols(for:in:)` — nothing iterates it in order (proposal 0001 /// rider; the former `OrderedDictionary` paid an ordering table for - /// hundreds of thousands of entries nobody read). - let symbolRowsByOffset: [Int: [UInt32]] + /// hundreds of thousands of entries nobody read). Values are + /// `SymbolRowBucket` (proposal 0003): the dominant single-row case + /// stays inline in the dictionary slot instead of paying a per-key + /// array allocation. + let symbolRowsByOffset: [Int: SymbolRowBucket] let thunkAttributeMembersByKindAndTypeName: [Node.Kind: [String: [ThunkAttributeMember]]] @@ -222,7 +225,7 @@ public final class SymbolIndexStore: SharedCache, @unc nodeStore: NodeStore, symbolTable: SymbolTable, rootNodeIndexByTableRow: [NodeStore.NodeIndex?], - symbolRowsByOffset: [Int: [UInt32]], + symbolRowsByOffset: [Int: SymbolRowBucket], rowIndexes: consuming RowIndexes ) { self.nodeStore = nodeStore @@ -303,9 +306,40 @@ public final class SymbolIndexStore: SharedCache, @unc return DemangledSymbol(symbolTable: symbolTable, symbolTableRow: row, demangledNode: nodeStore.reference(at: rootNodeIndex)) } - func demangledSymbols(atRows rows: [UInt32]) -> [DemangledSymbol] { + func demangledSymbols(atRows rows: some Sequence) -> [DemangledSymbol] { rows.compactMap { demangledSymbol(atRow: $0) } } + + /// One-shot acceptance statistic for proposal 0003: how many buckets + /// stay in the inline single-row form, across the offset index and + /// the three member-index families' leaf buckets. The proposal's + /// memory estimate assumes single-row dominance, so this is the + /// number the acceptance evidence pins. + func bucketFormStatisticsForTesting() -> (singleRowBucketCount: Int, multipleRowBucketCount: Int) { + var singleRowBucketCount = 0 + var multipleRowBucketCount = 0 + func tally(_ bucket: SymbolRowBucket) { + switch bucket { + case .single: + singleRowBucketCount += 1 + case .multiple: + multipleRowBucketCount += 1 + } + } + for bucket in symbolRowsByOffset.values { + tally(bucket) + } + for memberRowsByKind in [memberSymbolRowsByKind, methodDescriptorMemberSymbolRowsByKind, protocolWitnessMemberSymbolRowsByKind] { + for memberRows in memberRowsByKind.values { + for rowsByTypeNodeIndex in memberRows.values { + for bucket in rowsByTypeNodeIndex.values { + tally(bucket) + } + } + } + } + return (singleRowBucketCount, multipleRowBucketCount) + } } /// Build-time accumulator holding the row-index form of `Storage`'s @@ -327,17 +361,17 @@ public final class SymbolIndexStore: SharedCache, @unc } mutating func setMemberSymbols(for result: ProcessMemberSymbolResult) { - memberSymbolRowsByKind[result.memberKind, default: [:]][result.typeName, default: [:]][result.typeNodeIndex, default: []].append(result.symbolTableRow) + memberSymbolRowsByKind[result.memberKind, default: [:]][result.typeName, default: [:]][result.typeNodeIndex, default: .empty].append(result.symbolTableRow) typeInfoByName[result.typeName] = result.typeInfo } mutating func setMethodDescriptorMemberSymbols(for result: ProcessMemberSymbolResult) { - methodDescriptorMemberSymbolRowsByKind[result.memberKind, default: [:]][result.typeName, default: [:]][result.typeNodeIndex, default: []].append(result.symbolTableRow) + methodDescriptorMemberSymbolRowsByKind[result.memberKind, default: [:]][result.typeName, default: [:]][result.typeNodeIndex, default: .empty].append(result.symbolTableRow) typeInfoByName[result.typeName] = result.typeInfo } mutating func setProtocolWitnessMemberSymbols(for result: ProcessMemberSymbolResult) { - protocolWitnessMemberSymbolRowsByKind[result.memberKind, default: [:]][result.typeName, default: [:]][result.typeNodeIndex, default: []].append(result.symbolTableRow) + protocolWitnessMemberSymbolRowsByKind[result.memberKind, default: [:]][result.typeName, default: [:]][result.typeNodeIndex, default: .empty].append(result.symbolTableRow) typeInfoByName[result.typeName] = result.typeInfo } @@ -407,12 +441,14 @@ public final class SymbolIndexStore: SharedCache, @unc let mappedStringTableBase = mappedSymbols64.map { UnsafeRawPointer($0.stringBase) } ?? mappedSymbols32.map { UnsafeRawPointer($0.stringBase) } var tableBuilder = SymbolTableBuilder(mappedStringTableBase: mappedStringTableBase) - var symbolRowsByOffset: [Int: [UInt32]] = [:] + var symbolRowsByOffset: [Int: SymbolRowBucket] = [:] // One offset legitimately maps to several rows — distinct symbol names - // can share an address — so the bucket stays a list. The *same* row - // must not be listed twice though, or every `for symbol in symbols` - // loop visits it twice. + // can share an address — so the bucket keeps list semantics (inline + // for the dominant single-row case, spilling to an array only when a + // second row actually lands; proposal 0003). The *same* row must not + // be listed twice though, or every `for symbol in symbols` loop + // visits it twice. // // A row repeats for two independent reasons, and each is headed off // without scanning the bucket: @@ -440,7 +476,7 @@ public final class SymbolIndexStore: SharedCache, @unc if mayAlreadyBeListed, symbolRowsByOffset[offset]?.contains(row) == true { return } - symbolRowsByOffset[offset, default: []].append(row) + symbolRowsByOffset[offset, default: .empty].append(row) } func registerRow(_ row: UInt32, rawOffset: Int, canonicalOffset: Int, isNewRow: Bool) { diff --git a/Sources/MachOSymbols/SymbolRowBucket.swift b/Sources/MachOSymbols/SymbolRowBucket.swift new file mode 100644 index 00000000..681f75b0 --- /dev/null +++ b/Sources/MachOSymbols/SymbolRowBucket.swift @@ -0,0 +1,59 @@ +/// Row-index bucket for `SymbolIndexStore`'s offset and member indexes +/// (evolution proposal 0003): almost every offset / member key maps to +/// exactly one symbol-table row, so the single-row form stays inline in the +/// dictionary slot and only a bucket that actually collects a second row +/// pays a heap allocation. The former `[UInt32]` buckets paid an array +/// allocation per key — one per hundreds of thousands of keys in a +/// framework-scale image. +/// +/// Iteration order is insertion order in both forms (`multiple` preserves +/// the array's append order, `single` is trivially ordered), so query +/// output is byte-identical to the `[UInt32]` representation it replaces. +enum SymbolRowBucket: Equatable, Sendable { + case single(UInt32) + case multiple([UInt32]) + + /// Starting value for the `dictionary[key, default: .empty].append(row)` + /// accumulation idiom. `.multiple([])` allocates nothing (an empty + /// `Array` shares the global empty storage singleton) and the first + /// `append` rewrites it to the inline `single` form. + static var empty: SymbolRowBucket { .multiple([]) } + + mutating func append(_ row: UInt32) { + switch self { + case .multiple(let rows) where rows.isEmpty: + self = .single(row) + case .single(let existingRow): + self = .multiple([existingRow, row]) + case .multiple(var rows): + // Drop the payload's array reference before mutating so the + // append never triggers a copy-on-write of the whole bucket. + self = .empty + rows.append(row) + self = .multiple(rows) + } + } +} + +extension SymbolRowBucket: RandomAccessCollection { + var startIndex: Int { 0 } + + var endIndex: Int { + switch self { + case .single: + return 1 + case .multiple(let rows): + return rows.count + } + } + + subscript(position: Int) -> UInt32 { + switch self { + case .single(let row): + precondition(position == 0, "single-row bucket only has index 0") + return row + case .multiple(let rows): + return rows[position] + } + } +} diff --git a/Tests/IntegrationTests/MachOSymbols/SymbolIndexStoreBaselineTests.swift b/Tests/IntegrationTests/MachOSymbols/SymbolIndexStoreBaselineTests.swift index 37f64817..bda00016 100644 --- a/Tests/IntegrationTests/MachOSymbols/SymbolIndexStoreBaselineTests.swift +++ b/Tests/IntegrationTests/MachOSymbols/SymbolIndexStoreBaselineTests.swift @@ -65,6 +65,10 @@ final class SymbolIndexStoreBaselineTests: MachOImageTests { print("protocolWitnessMember entries : \(protocolWitnessEntryCount)") print("globalSymbols entries : \(globalEntryCount)") print("symbolsByOffset entries : \(storage.symbolRowsByOffset.count)") + let bucketStatistics = storage.bucketFormStatisticsForTesting() + let totalBucketCount = bucketStatistics.singleRowBucketCount + bucketStatistics.multipleRowBucketCount + let singleRowBucketRatio = totalBucketCount > 0 ? Double(bucketStatistics.singleRowBucketCount) / Double(totalBucketCount) : 0 + print("row buckets single/multiple : \(bucketStatistics.singleRowBucketCount)/\(bucketStatistics.multipleRowBucketCount) (single ratio \(String(format: "%.1f", singleRowBucketRatio * 100))%)") print("opaqueTypeDescriptor entries : \(storage.opaqueTypeDescriptorSymbolRowByNodeIndex.count)") print("typeInfoByName entries : \(storage.typeInfoByName.count)") print("=====================================================================") diff --git a/Tests/MachOSymbolsTests/SymbolIndexStoreFixtureTests.swift b/Tests/MachOSymbolsTests/SymbolIndexStoreFixtureTests.swift index ca303b31..03a3deac 100644 --- a/Tests/MachOSymbolsTests/SymbolIndexStoreFixtureTests.swift +++ b/Tests/MachOSymbolsTests/SymbolIndexStoreFixtureTests.swift @@ -85,6 +85,56 @@ final class SymbolIndexStoreFixtureTests: MachOFileTests, @unchecked Sendable { #expect(MemoryLayout.stride <= 32) #expect(MemoryLayout.stride <= 32) #expect(MemoryLayout.stride == 16) + // Proposal 0003: the row bucket must stay a compact inline value — + // widening it erodes the single-row savings across every dictionary + // slot that holds one. + #expect(MemoryLayout.stride <= 16) + } + + // MARK: - Row buckets (proposal 0003) + + /// The bucket's whole contract: first append stays inline, the second + /// spills to an array, iteration preserves insertion order (the + /// byte-identical-output guarantee), and `contains` answers both forms. + @Test func symbolRowBucketAppendMigrationAndIterationOrder() { + var bucket = SymbolRowBucket.empty + #expect(bucket.isEmpty) + #expect(bucket.count == 0) + #expect(!bucket.contains(7)) + + bucket.append(7) + #expect(bucket == .single(7)) + #expect(bucket.count == 1) + #expect(Array(bucket) == [7]) + #expect(bucket.contains(7)) + #expect(!bucket.contains(9)) + + bucket.append(9) + #expect(bucket == .multiple([7, 9])) + + bucket.append(5) + #expect(bucket == .multiple([7, 9, 5])) + #expect(Array(bucket) == [7, 9, 5]) + #expect(bucket.count == 3) + #expect(bucket.contains(5)) + #expect(!bucket.contains(6)) + } + + /// Single-row dominance is what proposal 0003's memory estimate rests + /// on; assert the direction on the fixture and surface the exact ratio + /// as acceptance evidence (the framework-scale ratio is re-measured by + /// the RuntimeViewer heap step). + @Test func rowBucketsAreDominatedBySingleRowForm() throws { + let storage = try storage + let statistics = storage.bucketFormStatisticsForTesting() + let totalBucketCount = statistics.singleRowBucketCount + statistics.multipleRowBucketCount + try #require(totalBucketCount > 0) + let singleRowRatio = Double(statistics.singleRowBucketCount) / Double(totalBucketCount) + print("SymbolRowBucket forms — single: \(statistics.singleRowBucketCount), multiple: \(statistics.multipleRowBucketCount), single ratio: \(singleRowRatio)") + #expect( + statistics.singleRowBucketCount > statistics.multipleRowBucketCount, + "single: \(statistics.singleRowBucketCount), multiple: \(statistics.multipleRowBucketCount), ratio: \(singleRowRatio)" + ) } /// File-leg counterpart of `SymbolTableImageEquivalenceTests`: a From c6d77bbbf2f6c5701ad505bcd722a11041973f0b Mon Sep 17 00:00:00 2001 From: Mx-Iris Date: Sun, 9 Aug 2026 14:08:11 +0800 Subject: [PATCH 53/77] perf(declaration-model): retain descriptors, materialize wrappers on demand (evolution 0002) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The declaration model eagerly parsed and permanently retained the fat MachOSwiftSection wrappers while their heavy trailing-object content is only consumed by the lazy index()/print paths — in RuntimeViewer's steady state most definitions are never indexed, so the parse products sat resident for nothing (41.3 MiB SwiftDeclaration + 33.4 MiB MachOSwiftSection clusters). - TypeDefinition stores typeContextDescriptorWrapper (was type: TypeContextWrapper); parentContext and its ParentContext enum are removed outright — the survey confirmed zero consumers after indexTypes() returns, so it is now a function-local dictionary there - ExtensionDefinition stores protocolConformanceDescriptor; ProtocolDefinition stores protocolDescriptor - materializedTypeContext(in:) / materializedProtocolConformance(in:) / materializedProtocol(in:) rebuild the full wrappers on demand. Discipline: at most one materialization per operation, threaded as a local, never cached (caching would re-accumulate the reclaimed memory in browse order) - SwiftDeclarationIndexer.Storage's section-wrapper populations (types / protocols / protocolConformances / associatedTypes and the parsed-value keyed conformance maps) become indexing transients, released when prepare() finishes; the retained conformance facts are the new name-level conformingProtocolNamesByTypeName (+ merged projection), which is all ConformanceProvider and tests read. The unused wrapper-population projections (types/protocols/…, allTypes/…) are removed (zero callers in-repo and downstream) - ~30 in-repo consumer sites migrated (printers materialize once per print; specializer/attribute-inference/diff read descriptor facts) Instance sizes (class_getInstanceSize): TypeDefinition 1272 -> 384 B, ExtensionDefinition 640 -> 224 B, ProtocolDefinition 440 -> 384 B, pinned by the new DeclarationModelInstanceSizeTests. Full suite 1343 green; seven-pair render A/B byte-identical in both debug and release builds; release wall-clock at parity (SwiftUI interface 5.3% faster, SwiftUICore within noise). Evolution proposal 0002 -> Implemented (RuntimeViewer heap re-measure and its 8 mechanical call-site renames pending downstream pickup). --- AGENTS.md | 2 + ...2-declaration-model-descriptor-slimming.md | 24 +-- Documentations/Evolutions/README.md | 2 +- .../DeclarationModelMemoryFootprint.md | 14 ++ .../TypeAttributeInferrer.swift | 2 +- .../Definitions/ExtensionDefinition.swift | 34 ++++- .../Definitions/ProtocolDefinition.swift | 26 +++- .../Definitions/TypeDefinition.swift | 46 ++++-- Sources/SwiftDiffing/ABIDiffer.swift | 2 +- .../SwiftDeclarationIndexer.swift | 142 ++++++++---------- ...wiftDeclarationPrinter+DiffRendering.swift | 8 +- .../SwiftDiffableInterfaceRenderer.swift | 2 +- .../SwiftDeclarationPrinter+Headers.swift | 9 +- .../SwiftDeclarationPrinter.swift | 28 +++- .../ConformanceProvider.swift | 11 +- .../GenericSpecializer.swift | 10 +- .../TypeDefinition+Specialization.swift | 25 +-- .../DeclarationModelInstanceSizeTests.swift | 37 +++++ .../SymbolTestsCoreIntegrationTests.swift | 12 +- .../GenericSpecializationTests.swift | 2 +- .../GenericTypeNameSubstitutionTests.swift | 18 +-- 21 files changed, 293 insertions(+), 163 deletions(-) create mode 100644 Tests/SwiftIndexingTests/DeclarationModelInstanceSizeTests.swift diff --git a/AGENTS.md b/AGENTS.md index b9280897..c66207c1 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -111,10 +111,12 @@ The interface generation is split into layered peer modules over a shared `Swift **SwiftDeclaration** - Shared declaration model (base layer for the Swift* modules) - `TypeDefinition`, `ProtocolDefinition`, `ExtensionDefinition`, `FunctionDefinition`, names, kinds, `DefinitionBuilder` +- The model retains **descriptor references, not parsed wrappers** (evolution proposal 0002): `TypeDefinition.typeContextDescriptorWrapper`, `ExtensionDefinition.protocolConformanceDescriptor`, `ProtocolDefinition.protocolDescriptor`. The full wrappers (`TypeContextWrapper` / `ProtocolConformance` / `Protocol`, trailing objects included) are rebuilt on demand via `materializedTypeContext(in:)` / `materializedProtocolConformance(in:)` / `materializedProtocol(in:)`. **Materialization discipline**: at most one materialization per operation (index it / print it / specialize it), threaded through as a local variable; never a per-access computed property, and the result is never cached on the definition — caching would re-accumulate, in browse order, the memory the slimming reclaimed. `DeclarationModelInstanceSizeTests` pins the instance-size ceilings. - `SwiftIndexEvents` - event namespace (Payload/Dispatcher/Handler) emitted by both indexer and printer **SwiftIndexing** - Builds the `SwiftDeclaration` model from a Mach-O image - `SwiftDeclarationIndexer` - Indexes types, extensions, conformances +- The section-wrapper populations the index passes consume (`types` / `protocols` / `protocolConformances` / `associatedTypes` and the parsed-value keyed conformance maps) are **indexing transients** since proposal 0002 — released when `prepare()` finishes, with no public projection. The retained conformance facts are the name-level maps `conformingProtocolNamesByTypeName` / `conformingTypesByProtocolName` (+ their merged `all*` variants), which is all any post-indexing consumer (including `SwiftSpecialization`'s `ConformanceProvider`) reads. - `SwiftIndexEventReporter`, `OSLogEventHandler`, `ConsoleEventHandler` - event handlers - `SwiftDeclarationIndexConfiguration` diff --git a/Documentations/Evolutions/0002-declaration-model-descriptor-slimming.md b/Documentations/Evolutions/0002-declaration-model-descriptor-slimming.md index 1a36362c..8e118791 100644 --- a/Documentations/Evolutions/0002-declaration-model-descriptor-slimming.md +++ b/Documentations/Evolutions/0002-declaration-model-descriptor-slimming.md @@ -1,6 +1,6 @@ # 0002 - 声明模型 descriptor 化：TypeDefinition / ExtensionDefinition / ProtocolDefinition 不再驻留急切解析的胖 wrapper -- **状态**: Accepted +- **状态**: Implemented - **作者**: JH - **创建日期**: 2026-08-09 - **最后更新**: 2026-08-09 @@ -157,15 +157,15 @@ extension ProtocolDefinition { ## 落地步骤 -1. 下游普查：RV / MachOKitUI / SymbolViewer 对 `.type` / `.protocolConformance` / `.protocol` 的直接消费点清单（RV 侧请对面会话代查）。 -2. `TypeDefinition` descriptor 化：存储换形态 + `parentContext` 移除（或退路形态）+ `materializedTypeContext(in:)` + `index()` 与打印路径的物化点。 -3. `ExtensionDefinition` / `ProtocolDefinition` 同模式。 -4. 库内消费点全量迁移（~30 处，两类句式）。 -5. 测试：全量 `swift test --skip IntegrationTests` 同数全绿；**渲染 A/B 三 reader 路径逐字节一致**（硬线——本案不许改变任何输出字节）。 -6. 性能：iOS 18.5 模拟器 SwiftUI `interface` wall-clock 持平（物化 CPU 的验收）。 -7. RV 复测（对面协调）：预期堆存活 283 → **~240–255 MiB**；`SwiftDeclaration` 簇 41.3 → ~15–20 MiB；MachOSwiftSection 簇 33.4 → ~10–15 MiB。 -8. 以账本同款探针（`class_getInstanceSize` + `MemoryLayout`）复量三类定义的落地后实例尺寸，给 [DeclarationModelMemoryFootprint.md](../Internal/DeclarationModelMemoryFootprint.md) 补后记。 -9. 收尾判断（写进决策日志）：是否写实现说明（「物化纪律」是代码看不出来的契约，倾向写短篇或并入 AGENTS.md）；新术语是否登记。 +1. ✅ 下游普查：本地磁盘全量 grep 完成，结论「零物化需求」，详见决策日志。RV 的 8 处 `.type` 机械改名清单已备（6 处 `typeContextDescriptorWrapper` 改名 + 2 处 pattern-match 改到 descriptor 案）。 +2. ✅ `TypeDefinition` descriptor 化：`typeContextDescriptorWrapper` 驻留 + `parentContext` 整体移除（`ParentContext` 枚举一并删除，降级为 `SwiftDeclarationIndexer.indexTypes` 的局部 `UnlinkedParentContext` 字典）+ `materializedTypeContext(in:)`；`index()` 仅 class 分支物化（struct/enum 的字段索引走 descriptor 级 `fieldDescriptor(in:)`，零物化）。 +3. ✅ `ExtensionDefinition` / `ProtocolDefinition` 同模式（`index()` 各自单点物化；typealias-only 扩展先查 descriptor 为 nil，零物化即返回）。**追加**：indexer `Storage` 侧人口清退 + 名字级轻映射（实施期修正，见决策日志）。 +4. ✅ 库内消费点全量迁移：打印器（`printTypeDefinition` / `printProtocolDefinition` 各一次物化贯穿 header + 字段/关联类型渲染；`printExtensionHeader` 一次）、`GenericSpecializer` / `TypeDefinition+Specialization` / `ConformanceProvider`（后者的子类图构建按类物化一次、随图缓存）、`SwiftAttributeInference` / `SwiftInterface` diff 渲染 / `SwiftDiffing` 的 kind 判断（descriptor 级改名）、测试侧 13 处。 +5. ✅ 全量 `swift test --skip IntegrationTests` 1343 全绿；渲染 A/B 七对（iOS 18.5 模拟器 SwiftUI / SwiftData / SwiftUICore 的 dump + interface，宿主机 dyld shared cache 的 SwiftUI dump + interface）全部逐字节一致（interface 剥离日志行首时间戳后比对）。 +6. 性能：iOS 18.5 模拟器 SwiftUI `interface` wall-clock 持平（物化 CPU 的验收）——见决策日志的 release 复测记录。 +7. RV 复测（对面协调）：预期堆存活 283 → **~240–255 MiB**；`SwiftDeclaration` 簇 41.3 → ~15–20 MiB；MachOSwiftSection 簇 33.4 → ~10–15 MiB。**待下游拿到本分支后进行。** +8. ✅ 以账本同款探针复量：`TypeDefinition` 1272 → **384 B**、`ExtensionDefinition` 640 → **224 B**、`ProtocolDefinition` 440 → **384 B**；[DeclarationModelMemoryFootprint.md](../Internal/DeclarationModelMemoryFootprint.md) 已补后记，且探针固化为常驻回归守卫 `DeclarationModelInstanceSizeTests`（上限 448 / 320 / 416 B）。 +9. ✅ 收尾判断：不另写实现说明——「物化纪律」已写进 AGENTS.md 的 SwiftDeclaration 段与三个 `materialized…(in:)` 的 doc comment，实例尺寸契约由回归测试钉住，一篇独立文章只会复述这两处；术语表无新词（materialize / wrapper vs descriptor 两条已覆盖本案语汇，bucket 条随 0003 更新）。 ## 决策日志 @@ -175,3 +175,7 @@ extension ProtocolDefinition { | 2026-08-09 | 审阅补充：物化结果不缓存 | 审阅期用户问「物化后会缓存吗」；裁决为不缓存、每次用时重新物化（缓存会按浏览顺序把内存攒回来；物化微秒级、频度有界），可驱逐小缓存仅作 profiling 证明热点后的退路。已写入「详细设计 · 物化接口」。 | | 2026-08-09 | 审阅补充：与 RV 全局搜索不冲突 | 审阅期用户问「后面 RV 想做全局搜索咋办」；裁决为不冲突且同向——搜索匹配的是名字（类型名 / 成员名 / 符号名，均为冻结轻数据），本案清退的 vtable / witnesses 解析产物不是搜索目标，命中后看详情恰是「用时物化」场景；descriptor 化让「全量建骨架不索引成员」的搜索支撑形态更可行（~400 B vs 1272 B + 堆数组/类型）。真正的功课在「按需 per-image 索引 vs 全量覆盖」这层（本案之前就存在），届时另立搜索索引提案（候选形态：惰性全量 sweep + 限流 / 名字层前置倒排 + 库侧行收集模式 / 持久化索引），不构成本案的阻塞或修改项。 | | 2026-08-09 | In Review → Accepted | 用户审核通过（「审核通过，开始实现」），按落地步骤开工，第一步为下游消费点普查。 | +| 2026-08-09 | 下游普查结论：零物化需求 | 本地磁盘全量 grep（RuntimeViewer / MachOKitUI / MachOViewer 等；SymbolViewer 无独立仓库）：`parentContext` 与 `.protocolConformance` / `.protocol` 零下游消费者——`parentContext` 移除首选路线成立；`.type` 仅 RV 8 处且全部为 descriptor 级事实（6 处读 `typeContextDescriptorWrapper` / flags，2 处 pattern-match 后只取 `.descriptor`），机械改名即可、无一处需要物化。 | +| 2026-08-09 | 实施期修正：indexer Storage 侧同批清退 | 动手时发现提案调研的一处失实：「indexer 的全量 protocolConformances 数组在分组后本就出栈」不成立——`SwiftDeclarationIndexer.Storage` 以 `types` / `protocols` / `protocolConformances` / `associatedTypes` 四个人口数组加 `protocolConformancesByTypeName` / `associatedTypesByTypeName` 两个按名 keyed 映射**终身驻留全部 wrapper**（CoW 共享底层堆数组），不清退则 definition 侧换 descriptor 后 trailing 簇分文不释放。消费面普查：六者的公开投影在库内外（含 RV）**零调用方**，唯二例外是 `ConformanceProvider` 读 `allProtocolConformancesByTypeName` 的存在性 + keys（纯名字级事实）与两个测试读 keys。修正：四个人口数组在 `prepare()` 索引完成后置空；两个重映射降级为索引期局部变量，新增名字级轻映射 `conformingProtocolNamesByTypeName`（+ 合并投影 `allConformingProtocolNamesByTypeName`）承接 ConformanceProvider 与测试；重投影与 `allTypes` / `allProtocols` / `allProtocolConformances` / `allAssociatedTypes` 聚合一并移除（额外 API 破坏，均为零调用方）。 | +| 2026-08-09 | wall-clock 验收：release 持平（一对反而更快） | debug 构建初测候选慢 5–10%（SwiftUICore interface 反转执行序复测仍 ~9%），但最大任务 dyld cache SwiftUI interface 仅 +0.2%，疑为 debug 常数因子；改以 release 构建 ABBA 序 ×2 轮定论：SwiftUI interface 基线均值 76.3s vs 候选 72.2s（候选**快 5.3%**），SwiftUICore 37.3s vs 37.4s（+0.5%，噪声带内）。验收线达成，以 release 为准；release 输出与 debug 同样逐字节一致。 | +| 2026-08-09 | Accepted → Implemented | 落地步骤 1–5、8、9 完成（步骤 6 见上一行）：三定义 descriptor 化 + `parentContext` / `ParentContext` 移除 + 三个物化入口 + indexer Storage 清退 + 库内与测试侧全量迁移；全量 1343 绿；A/B 七对（debug 与 release 双构建）逐字节一致；实例尺寸 1272 → 384 / 640 → 224 / 440 → 384 B（前两者优于预估 ~400），回归守卫 `DeclarationModelInstanceSizeTests` 落位。步骤 7（RV 堆复测）待下游拿到本分支后进行；RV 侧 8 处机械迁移句式已在步骤 1 备好。 | diff --git a/Documentations/Evolutions/README.md b/Documentations/Evolutions/README.md index 3f893813..b2b04086 100644 --- a/Documentations/Evolutions/README.md +++ b/Documentations/Evolutions/README.md @@ -7,5 +7,5 @@ | # | 标题 | 状态 | |---|------|------| | [0001](0001-symbol-name-offsetization.md) | SymbolIndexStore 符号名 offset 化：驻留字符串换字符串表引用 | Implemented | -| [0002](0002-declaration-model-descriptor-slimming.md) | 声明模型 descriptor 化：TypeDefinition / ExtensionDefinition / ProtocolDefinition 不再驻留急切解析的胖 wrapper | Accepted | +| [0002](0002-declaration-model-descriptor-slimming.md) | 声明模型 descriptor 化：TypeDefinition / ExtensionDefinition / ProtocolDefinition 不再驻留急切解析的胖 wrapper | Implemented | | [0003](0003-symbol-row-bucket-flattening.md) | SymbolIndexStore `[UInt32]` 行号桶扁平化：单元素桶内联化 | Implemented | diff --git a/Documentations/Internal/DeclarationModelMemoryFootprint.md b/Documentations/Internal/DeclarationModelMemoryFootprint.md index 8447e4fb..dac05844 100644 --- a/Documentations/Internal/DeclarationModelMemoryFootprint.md +++ b/Documentations/Internal/DeclarationModelMemoryFootprint.md @@ -168,3 +168,17 @@ malloc 分桶用 `malloc_size()`（``）实测，Swift 对象 ## 后记（2026-08-08） 第五节第 3 条的后半项（`MetadataReaderCache` 改持 `NodeReference`）已按用户裁决落地——「当前不建议实施」的结论对该项不再成立，其余各项维持原判。设计与落地记录见 [MetadataReaderCacheRetirement.md](MetadataReaderCacheRetirement.md)；本文其余量测与账目保持原貌不改。 + +## 后记（2026-08-09） + +第五节第 1 条（`parentContext`）与第 2 条（wrapper 装箱）由[提案 0002](../Evolutions/0002-declaration-model-descriptor-slimming.md) 以更优形态收编落地：第 1 条按本文的读写点追踪结论**整体移除**（降级为 `SwiftDeclarationIndexer.indexTypes` 的函数局部字典）；第 2 条没有走当年估算的 `indirect` 装箱，而是把三个定义的驻留从解析后的 wrapper 换成 descriptor 引用（`TypeContextWrapper` → `TypeContextDescriptorWrapper` 等），重内容改为用时物化——装箱只省内联不省堆，descriptor 化把 trailing 堆数组一并释放。「当前不建议实施」的两个前提（该线仅占 8–10%、其余 90% 未剖析）已被 0001 与两次专项清退消解，见提案的动机一节。 + +以本文同款探针（`class_getInstanceSize`）复量的落地后实例尺寸： + +| 类型 | 落地前 | 落地后 | +|---|---|---| +| `TypeDefinition` | 1272 B | **384 B** | +| `ExtensionDefinition` | 640 B | **224 B** | +| `ProtocolDefinition` | 440 B | **384 B** | + +`Tests/SwiftIndexingTests/DeclarationModelInstanceSizeTests.swift` 以同一探针作为常驻回归守卫（上限 448 / 320 / 416 B）。第三节的逐属性账目与其余量测保持原貌不改；RV 侧的堆存活复测待下游拿到该分支后进行（提案落地步骤 7）。 diff --git a/Sources/SwiftAttributeInference/TypeAttributeInferrer.swift b/Sources/SwiftAttributeInference/TypeAttributeInferrer.swift index 40943e7a..aab7ea02 100644 --- a/Sources/SwiftAttributeInference/TypeAttributeInferrer.swift +++ b/Sources/SwiftAttributeInference/TypeAttributeInferrer.swift @@ -149,7 +149,7 @@ public struct TypeAttributeInferrer: Sendable { // We can only detect this from the ClassDescriptor if we check the // metadataPositiveSizeInWordsOrExtraClassFlags field when the class has a resilient superclass. // For now, we check via the descriptor's extra class flags if available. - guard case .class(let classDescriptor) = typeDefinition.type.typeContextDescriptorWrapper else { return } + guard case .class(let classDescriptor) = typeDefinition.typeContextDescriptorWrapper else { return } // The hasCustomObjCName flag is in the runtime ClassFlags (swiftClassFlags), // which are only available when the binary is loaded as a MachOImage. diff --git a/Sources/SwiftDeclaration/Components/Definitions/ExtensionDefinition.swift b/Sources/SwiftDeclaration/Components/Definitions/ExtensionDefinition.swift index ccdd0048..668639ba 100644 --- a/Sources/SwiftDeclaration/Components/Definitions/ExtensionDefinition.swift +++ b/Sources/SwiftDeclaration/Components/Definitions/ExtensionDefinition.swift @@ -13,7 +13,13 @@ public final class ExtensionDefinition: Definition, MutableDefinition { public let genericSignature: NodeReference? - public let protocolConformance: ProtocolConformance? + /// The conformance's descriptor reference (evolution proposal 0002), or + /// `nil` for member / typealias-only extensions. The full + /// `ProtocolConformance` — resilient witnesses and the rest of its + /// trailing objects — is rebuilt on demand via + /// `materializedProtocolConformance(in:)` instead of living on every + /// conformance extension for its lifetime. + public let protocolConformanceDescriptor: ProtocolConformanceDescriptor? /// The conformed protocol, resolved to a Mach-O-free name at index time. /// Non-nil only for conformance extensions; the target's typealias-only @@ -60,26 +66,40 @@ public final class ExtensionDefinition: Definition, MutableDefinition { !variables.isEmpty || !functions.isEmpty || !staticVariables.isEmpty || !staticFunctions.isEmpty || !allocators.isEmpty || !constructors.isEmpty || !staticSubscripts.isEmpty || !subscripts.isEmpty } + /// The initializer still receives the full `ProtocolConformance` — the + /// indexer materializes the whole conformance section anyway to derive + /// attribution — but only its descriptor reference is retained, so the + /// parsed wrapper (and its `[ResilientWitness]`) is released once the + /// indexer's grouping pass ends. public init(extensionName: ExtensionName, genericSignature: NodeReference?, protocolConformance: ProtocolConformance?, conformingProtocolName: ProtocolName? = nil, associatedTypes: [AssociatedType] = [], resolvedAssociatedTypeWitnesses: [AssociatedTypeWitnessProjection] = [], in machO: MachO) throws { self.extensionName = extensionName self.genericSignature = genericSignature - self.protocolConformance = protocolConformance + self.protocolConformanceDescriptor = protocolConformance?.descriptor self.conformingProtocolName = conformingProtocolName self.associatedTypes = associatedTypes self.resolvedAssociatedTypeWitnesses = resolvedAssociatedTypeWitnesses } /// Mach-O-free initializer for pure-value construction (tests, tooling). - /// Carries no `ProtocolConformance` — only the frozen attribution fields. + /// Carries no conformance descriptor — only the frozen attribution fields. package init(extensionName: ExtensionName, genericSignature: NodeReference?, conformingProtocolName: ProtocolName? = nil, resolvedAssociatedTypeWitnesses: [AssociatedTypeWitnessProjection] = []) { self.extensionName = extensionName self.genericSignature = genericSignature - self.protocolConformance = nil + self.protocolConformanceDescriptor = nil self.conformingProtocolName = conformingProtocolName self.associatedTypes = [] self.resolvedAssociatedTypeWitnesses = resolvedAssociatedTypeWitnesses } + /// Rebuilds the full `ProtocolConformance` (trailing objects included) + /// from the retained descriptor; `nil` for member / typealias-only + /// extensions. Materialization discipline (evolution proposal 0002): + /// call at most once per operation and thread the result through as a + /// local variable — the result is deliberately not cached. + public func materializedProtocolConformance(in machO: MachO) throws -> ProtocolConformance? { + try protocolConformanceDescriptor.map { try ProtocolConformance(descriptor: $0, in: machO) } + } + /// Folds another definition's associated types (and their frozen witness /// projections) into this one — the indexer's typealias-only merge path. package func absorbAssociatedTypes(of other: ExtensionDefinition) { @@ -92,7 +112,11 @@ public final class ExtensionDefinition: Definition, MutableDefinition { package func index(in machO: MachO) async throws { guard !isIndexed else { return } - guard let protocolConformance, !protocolConformance.resilientWitnesses.isEmpty else { return } + // Cheap pre-check on the retained descriptor keeps the typealias-only + // majority from materializing at all; the one materialization below + // is this operation's single allowed one (proposal 0002). + guard protocolConformanceDescriptor != nil else { return } + guard let protocolConformance = try materializedProtocolConformance(in: machO), !protocolConformance.resilientWitnesses.isEmpty else { return } // Structurally keyed: `demangleSymbolReference` returns references from // different stores, and store-identity equality would let the same diff --git a/Sources/SwiftDeclaration/Components/Definitions/ProtocolDefinition.swift b/Sources/SwiftDeclaration/Components/Definitions/ProtocolDefinition.swift index f0e55231..57190391 100644 --- a/Sources/SwiftDeclaration/Components/Definitions/ProtocolDefinition.swift +++ b/Sources/SwiftDeclaration/Components/Definitions/ProtocolDefinition.swift @@ -71,7 +71,12 @@ extension StrippedSymbolicRequirement { } public final class ProtocolDefinition: Definition, MutableDefinition { - public let `protocol`: MachOSwiftSection.`Protocol` + /// The protocol's descriptor reference (evolution proposal 0002). The + /// full `MachOSwiftSection.Protocol` — requirement arrays included — is + /// rebuilt on demand via `materializedProtocol(in:)` instead of living + /// on every definition for its lifetime; the name is frozen separately + /// in `protocolName`. + public let protocolDescriptor: ProtocolDescriptor public let protocolName: ProtocolName @@ -121,14 +126,27 @@ public final class ProtocolDefinition: Definition, MutableDefinition { !subscripts.isEmpty || !staticVariables.isEmpty || !staticFunctions.isEmpty || !staticSubscripts.isEmpty || !allocators.isEmpty || !constructors.isEmpty || !strippedSymbolicRequirements.isEmpty } + /// The initializer still receives the full wrapper — the indexer holds + /// one from the section sweep anyway — but only its descriptor reference + /// is retained. public init(`protocol`: MachOSwiftSection.`Protocol`, in machO: MachO) throws { - self.protocol = `protocol` + self.protocolDescriptor = `protocol`.descriptor let node = try MetadataReader.demangleContext(for: .protocol(`protocol`.descriptor), in: machO) self.protocolName = ProtocolName(node: InternedNodeReferenceCache.shared.reference(interning: node, in: machO)) } + /// Rebuilds the full `MachOSwiftSection.Protocol` (requirement arrays + /// included) from the retained descriptor. Materialization discipline + /// (evolution proposal 0002): call at most once per operation and thread + /// the result through as a local variable — the result is deliberately + /// not cached. + public func materializedProtocol(in machO: MachO) throws -> MachOSwiftSection.`Protocol` { + try MachOSwiftSection.`Protocol`(descriptor: protocolDescriptor, in: machO) + } + package func index(in machO: MachO) async throws { guard !isIndexed else { return } + let dumpedProtocol = try materializedProtocol(in: machO) let name = protocolName.name // Structurally keyed: `demangleSymbolReference` returns references from // different stores, and store-identity equality would let the same @@ -141,7 +159,7 @@ public final class ProtocolDefinition: Definition, MutableDefinition { } return nil } - associatedTypes = try `protocol`.descriptor.associatedTypes(in: machO) + associatedTypes = try protocolDescriptor.associatedTypes(in: machO) var requirementMemberSymbolsByKind: OrderedDictionary = [:] var defaultImplementationMemberSymbolsByKind: OrderedDictionary = [:] @@ -151,7 +169,7 @@ public final class ProtocolDefinition: Definition, MutableDefinition { var offsetOfPWT = 0 - for requirement in `protocol`.requirements { + for requirement in dumpedProtocol.requirements { offsetOfPWT.offset(of: StoredPointer.self) if requirement.layout.defaultImplementation.isValid { defaultedRequirementPWTOffsets.insert(offsetOfPWT) diff --git a/Sources/SwiftDeclaration/Components/Definitions/TypeDefinition.swift b/Sources/SwiftDeclaration/Components/Definitions/TypeDefinition.swift index f700ad2f..e9e35905 100644 --- a/Sources/SwiftDeclaration/Components/Definitions/TypeDefinition.swift +++ b/Sources/SwiftDeclaration/Components/Definitions/TypeDefinition.swift @@ -13,13 +13,12 @@ import Dependencies @_spi(Internals) import SwiftInspection public final class TypeDefinition: Definition { - public enum ParentContext { - case `extension`(ExtensionContext) - case type(TypeContextWrapper) - case symbol(Symbol) - } - - public let type: TypeContextWrapper + /// The type's context descriptor reference (evolution proposal 0002). + /// This is the only Mach-O parse product the definition retains: the + /// full `TypeContextWrapper` (trailing objects included) is rebuilt on + /// demand via `materializedTypeContext(in:)` by the few operations that + /// need it, instead of living on every definition for its lifetime. + public let typeContextDescriptorWrapper: TypeContextDescriptorWrapper /// Injected at construction time. Ordinary indexing-derived definitions /// receive the unbound form computed from `type.typeName(in:)`; @@ -62,8 +61,6 @@ public final class TypeDefinition: Definition { public package(set) var protocolChildren: [ProtocolDefinition] = [] - public package(set) var parentContext: ParentContext? = nil - public package(set) var extensions: [ExtensionDefinition] = [] public package(set) var fields: [FieldDefinition] = [] @@ -138,8 +135,13 @@ public final class TypeDefinition: Definition { /// bypassed from outside the package; the `specialize(with:in:)` family /// (the `SwiftSpecialization` extension) is the only in-package caller /// that injects a different `typeName`/`isSpecialized` pair. + /// + /// The initializer still receives the full wrapper — every construction + /// path holds one anyway (indexing needs it for `typeName(in:)`) — but + /// only its descriptor reference is retained, so the caller's parsed + /// wrapper is released as soon as construction returns. package init(type: TypeContextWrapper, typeName: TypeName, isSpecialized: Bool) { - self.type = type + self.typeContextDescriptorWrapper = type.typeContextDescriptorWrapper self.typeName = typeName self.isSpecialized = isSpecialized } @@ -155,7 +157,7 @@ public final class TypeDefinition: Definition { @Dependency(\.symbolIndexStore) var symbolIndexStore - let typeContextDescriptor = try required(type.contextDescriptorWrapper.typeContextDescriptor) + let typeContextDescriptor = typeContextDescriptorWrapper.typeContextDescriptor let fieldDescriptor = try typeContextDescriptor.fieldDescriptor(in: machO) let records = try fieldDescriptor.records(in: machO) // Field type trees intern into the image's shared store @@ -202,7 +204,12 @@ public final class TypeDefinition: Definition { // Fallback lookups keyed by implementation file offset (for methods where node-based matching fails) var implOffsetDescriptorLookup: [Int: MethodDescriptorWrapper] = [:] var implOffsetVTableSlotLookup: [Int: Int] = [:] - if case .class(let cls) = type { + // The vtable / override tables live in the class wrapper's trailing + // objects, so the class branch is the one place indexing has to + // materialize the full wrapper — once, as a local, released when this + // function returns (materialization discipline, proposal 0002). + if case .class(let classDescriptor) = typeContextDescriptorWrapper { + let cls = try Class(descriptor: classDescriptor, in: machO) var visitedNodes: OrderedSet = [] let typeNode = try MetadataReader.demangleContext(for: .type(.class(cls.descriptor)), in: machO) let vtableBaseOffset = cls.vTableDescriptorHeader.map { Int($0.layout.vTableOffset) } @@ -363,7 +370,7 @@ public final class TypeDefinition: Definition { // Build ordered members list let allMembers = OrderedMember.allMembers(from: self) - if case .class = type { + if case .class = typeContextDescriptorWrapper { orderedMembers = OrderedMember.classOrdered(allMembers) } else { orderedMembers = OrderedMember.offsetOrdered(allMembers) @@ -372,6 +379,19 @@ public final class TypeDefinition: Definition { isIndexed = true } + /// Rebuilds the full `TypeContextWrapper` — trailing objects included — + /// from the retained descriptor, exactly the parse the model-build sweep + /// performed once already. + /// + /// Materialization discipline (evolution proposal 0002): call at most + /// once per operation (index it / print it / specialize it) and thread + /// the result through as a local variable. The result is deliberately + /// not cached — retaining it on the definition would re-accumulate, in + /// browse order, the memory the descriptor slimming reclaimed. + public func materializedTypeContext(in machO: MachO) throws -> TypeContextWrapper { + try TypeContextWrapper.forTypeContextDescriptorWrapper(typeContextDescriptorWrapper, in: machO) + } + /// Cross-references `@objc` / `@nonobjc` thunk attribute members (pre-extracted /// and bucketed by parent type name inside `SymbolIndexStore`) with the /// already-built member definitions of this type, appending the matching diff --git a/Sources/SwiftDiffing/ABIDiffer.swift b/Sources/SwiftDiffing/ABIDiffer.swift index 2044df71..64fefa64 100644 --- a/Sources/SwiftDiffing/ABIDiffer.swift +++ b/Sources/SwiftDiffing/ABIDiffer.swift @@ -245,7 +245,7 @@ public struct ABIDiffer: Sendable { /// on top of the shared members. func memberRecords(of definition: TypeDefinition) -> [MemberRecord] { var records = sharedMemberRecords(of: definition) - if case .enum = definition.type { + if case .enum = definition.typeContextDescriptorWrapper { for (tag, field) in definition.fields.enumerated() { records.append(.makeCase(field, tag: tag)) } diff --git a/Sources/SwiftIndexing/SwiftDeclarationIndexer.swift b/Sources/SwiftIndexing/SwiftDeclarationIndexer.swift index 324ddd0f..2341bad3 100644 --- a/Sources/SwiftIndexing/SwiftDeclarationIndexer.swift +++ b/Sources/SwiftIndexing/SwiftDeclarationIndexer.swift @@ -39,6 +39,18 @@ extension MachOIndexedValue: Sendable where Value: Sendable {} @_spi(Support) public final class SwiftDeclarationIndexer: Sendable { + /// Indexing-local carrier for a nested type's resolved-but-unlinked + /// parent context (evolution proposal 0002). Lives only for the duration + /// of `indexTypes()` — it replaced the stored + /// `TypeDefinition.parentContext` property, which kept a second + /// fully-parsed `TypeContextWrapper` inline on every affected definition + /// with no reader after indexing. + private enum UnlinkedParentContext { + case `extension`(ExtensionContext) + case type(TypeContextWrapper) + case symbol(Symbol) + } + @usableFromInline final class Storage: Sendable { @usableFromInline @Mutex @@ -53,11 +65,14 @@ public final class SwiftDeclarationIndexer> = [:] - - @usableFromInline @Mutex - var associatedTypesByTypeName: OrderedDictionary> = [:] + var conformingProtocolNamesByTypeName: OrderedDictionary> = [:] @usableFromInline @Mutex var conformingTypesByProtocolName: OrderedDictionary> = [:] @@ -95,12 +110,7 @@ public final class SwiftDeclarationIndexer]? - var allProtocols: [MachOIndexedValue]? - var allProtocolConformances: [MachOIndexedValue]? - var allAssociatedTypes: [MachOIndexedValue]? - var allProtocolConformancesByTypeName: OrderedDictionary>>? - var allAssociatedTypesByTypeName: OrderedDictionary>>? + var allConformingProtocolNamesByTypeName: OrderedDictionary>? var allConformingTypesByProtocolName: OrderedDictionary>>? var allRootTypeDefinitions: OrderedDictionary>? var allAllTypeDefinitions: OrderedDictionary>? @@ -287,6 +297,17 @@ public final class SwiftDeclarationIndexer = [:] for (typeName, typeDefinition) in currentModuleTypeDefinitions { - if typeDefinition.parent == nil, typeDefinition.parentContext == nil { + if typeDefinition.parent == nil, unlinkedParentContextsByTypeName[typeName] == nil { rootTypeDefinitions[typeName] = typeDefinition - } else if let parentContext = typeDefinition.parentContext { + } else if let parentContext = unlinkedParentContextsByTypeName[typeName] { switch parentContext { case .extension(let extensionContext): guard let extendedContextMangledName = extensionContext.extendedContextMangledName else { continue } @@ -495,7 +525,7 @@ public final class SwiftDeclarationIndexer> = [:] var failedAssociatedTypes = 0 @@ -561,7 +590,6 @@ public final class SwiftDeclarationIndexer = [:] @@ -612,7 +640,7 @@ public final class SwiftDeclarationIndexer> { currentStorage.protocolConformancesByTypeName } + // The section-wrapper populations (`types` / `protocols` / + // `protocolConformances` / `associatedTypes`) and the parsed-value keyed + // maps that used to be projected here are indexing transients since + // evolution proposal 0002 — released when `prepare()` finishes, so they + // no longer have a public projection. The name-level maps below are the + // retained conformance facts. @inlinable - public var associatedTypesByTypeName: OrderedDictionary> { currentStorage.associatedTypesByTypeName } + public var conformingProtocolNamesByTypeName: OrderedDictionary> { currentStorage.conformingProtocolNamesByTypeName } @inlinable public var conformingTypesByProtocolName: OrderedDictionary> { currentStorage.conformingTypesByProtocolName } @@ -888,55 +908,15 @@ extension SwiftDeclarationIndexer { // populated its cache will not propagate, so reorganize hierarchies before the // first read. extension SwiftDeclarationIndexer { - public var allTypes: [MachOIndexedValue] { - if let cached = allStorageCache.allTypes { return cached } - let result = currentStorage.types.map { MachOIndexedValue(machO: machO, value: $0) } + subIndexers.flatMap { $0.allTypes } - allStorageCache.allTypes = result - return result - } - - public var allProtocols: [MachOIndexedValue] { - if let cached = allStorageCache.allProtocols { return cached } - let result = currentStorage.protocols.map { MachOIndexedValue(machO: machO, value: $0) } + subIndexers.flatMap { $0.allProtocols } - allStorageCache.allProtocols = result - return result - } - - public var allProtocolConformances: [MachOIndexedValue] { - if let cached = allStorageCache.allProtocolConformances { return cached } - let result = currentStorage.protocolConformances.map { MachOIndexedValue(machO: machO, value: $0) } + subIndexers.flatMap { $0.allProtocolConformances } - allStorageCache.allProtocolConformances = result - return result - } - - public var allAssociatedTypes: [MachOIndexedValue] { - if let cached = allStorageCache.allAssociatedTypes { return cached } - let result = currentStorage.associatedTypes.map { MachOIndexedValue(machO: machO, value: $0) } + subIndexers.flatMap { $0.allAssociatedTypes } - allStorageCache.allAssociatedTypes = result - return result - } - - public var allProtocolConformancesByTypeName: OrderedDictionary>> { - if let cached = allStorageCache.allProtocolConformancesByTypeName { return cached } - var result: OrderedDictionary>> = currentStorage.protocolConformancesByTypeName.mapValues { $0.mapValues { .init(machO: machO, value: $0) } } - for subIndexer in subIndexers { - for (typeName, conformances) in subIndexer.allProtocolConformancesByTypeName { - result[typeName, default: [:]].merge(conformances) { current, _ in current } - } - } - allStorageCache.allProtocolConformancesByTypeName = result - return result - } - - public var allAssociatedTypesByTypeName: OrderedDictionary>> { - if let cached = allStorageCache.allAssociatedTypesByTypeName { return cached } - var result: OrderedDictionary>> = currentStorage.associatedTypesByTypeName.mapValues { $0.mapValues { .init(machO: machO, value: $0) } } + public var allConformingProtocolNamesByTypeName: OrderedDictionary> { + if let cached = allStorageCache.allConformingProtocolNamesByTypeName { return cached } + var result = currentStorage.conformingProtocolNamesByTypeName for subIndexer in subIndexers { - for (typeName, associatedTypes) in subIndexer.allAssociatedTypesByTypeName { - result[typeName, default: [:]].merge(associatedTypes) { current, _ in current } + for (typeName, protocolNames) in subIndexer.allConformingProtocolNamesByTypeName { + result[typeName, default: []].formUnion(protocolNames) } } - allStorageCache.allAssociatedTypesByTypeName = result + allStorageCache.allConformingProtocolNamesByTypeName = result return result } diff --git a/Sources/SwiftInterface/SwiftDeclarationPrinter+DiffRendering.swift b/Sources/SwiftInterface/SwiftDeclarationPrinter+DiffRendering.swift index 1df7b8b1..f62a6f4f 100644 --- a/Sources/SwiftInterface/SwiftDeclarationPrinter+DiffRendering.swift +++ b/Sources/SwiftInterface/SwiftDeclarationPrinter+DiffRendering.swift @@ -43,8 +43,10 @@ package extension SwiftDeclarationPrinter { // renders the bound header (`Box`, no generic-signature clause). // Latent today — the diff builder never walks `specializedChildren` — // but the two header entry points must not diverge. + // This header-print operation's single wrapper materialization + // (proposal 0002). try await renderTypeDeclarationHeader( - for: typeDefinition.type, + for: typeDefinition.materializedTypeContext(in: machO), displayParentName: displayParentName, level: level, leafNameNode: leafNameNode(of: typeDefinition.typeName.node.materialize()), @@ -61,8 +63,10 @@ package extension SwiftDeclarationPrinter { try await protocolDefinition.index(in: machO) } + // This header-print operation's single wrapper materialization + // (proposal 0002). try await renderProtocolDeclarationHeader( - for: protocolDefinition.protocol, + for: protocolDefinition.materializedProtocol(in: machO), displayParentName: displayParentName, leafNameNode: leafNameNode(of: protocolDefinition.protocolName.node.materialize()) ) diff --git a/Sources/SwiftInterface/SwiftDiffableInterfaceRenderer.swift b/Sources/SwiftInterface/SwiftDiffableInterfaceRenderer.swift index 5d5f4a97..78e4fd37 100644 --- a/Sources/SwiftInterface/SwiftDiffableInterfaceRenderer.swift +++ b/Sources/SwiftInterface/SwiftDiffableInterfaceRenderer.swift @@ -337,7 +337,7 @@ public final class SwiftDiffableInterfaceRenderer< private func fieldMembers(_ definition: TypeDefinition?, level: Int, printer: SwiftDeclarationPrinter) -> [RenderableMember] { guard let definition else { return [] } - if case .enum = definition.type { + if case .enum = definition.typeContextDescriptorWrapper { return definition.fields.enumerated().map { index, field in let record = MemberRecord.makeCase(field, tag: index) return RenderableMember(identityKey: record.identityKey, payloadKey: record.payloadKey) { await printer.printEnumCase(field, level: level) } diff --git a/Sources/SwiftPrinting/SwiftDeclarationPrinter+Headers.swift b/Sources/SwiftPrinting/SwiftDeclarationPrinter+Headers.swift index 55b28642..d785b475 100644 --- a/Sources/SwiftPrinting/SwiftDeclarationPrinter+Headers.swift +++ b/Sources/SwiftPrinting/SwiftDeclarationPrinter+Headers.swift @@ -301,8 +301,11 @@ extension SwiftDeclarationPrinter { /// whole type's rendering throw — it does not degrade into a silently /// empty line. The diff renderer keeps its own per-member catch via /// `printField` / `printEnumCase`. + /// `typeContext` is the caller's materialized wrapper for this print + /// operation (proposal 0002) — `printTypeDefinition` materializes once + /// and threads it into both the header renderer and this function. @SemanticStringBuilder - func renderModelFields(_ typeDefinition: TypeDefinition, level: Int) async throws -> SemanticString { + func renderModelFields(_ typeDefinition: TypeDefinition, typeContext: TypeContextWrapper, level: Int) async throws -> SemanticString { let isEnum = typeDefinition.typeName.kind == .enum // Shared metadata-comment renderer (single source of truth with @@ -324,8 +327,8 @@ extension SwiftDeclarationPrinter { staticFieldLayoutProvider: staticFieldLayoutProvider(), staticLayoutDependencyResolution: configuration.staticLayoutDependencyResolution ) - let fieldLayoutRenderer = FieldLayoutRenderer(type: typeDefinition.type, metadata: typeDefinition.metadata, machO: machO, configuration: renderConfiguration) - let fieldRecords = try typeDefinition.type.contextDescriptorWrapper.typeContextDescriptor?.fieldDescriptor(in: machO).records(in: machO) ?? [] + let fieldLayoutRenderer = FieldLayoutRenderer(type: typeContext, metadata: typeDefinition.metadata, machO: machO, configuration: renderConfiguration) + let fieldRecords = try typeDefinition.typeContextDescriptorWrapper.typeContextDescriptor.fieldDescriptor(in: machO).records(in: machO) let fieldOffsets = isEnum ? nil : fieldLayoutRenderer.fieldOffsets // Specialized definitions substitute each field's generic-parameter diff --git a/Sources/SwiftPrinting/SwiftDeclarationPrinter.swift b/Sources/SwiftPrinting/SwiftDeclarationPrinter.swift index 312613bc..bd481d34 100644 --- a/Sources/SwiftPrinting/SwiftDeclarationPrinter.swift +++ b/Sources/SwiftPrinting/SwiftDeclarationPrinter.swift @@ -127,8 +127,12 @@ public final class SwiftDeclarationPrinter: Sendab // dump path performs via `TypedDumper.boundDumpedTypeNode()`. let specializedMetadata: MetadataWrapper? = typeDefinition.isSpecialized ? typeDefinition.metadata : nil + // This print operation's single wrapper materialization (proposal + // 0002), threaded into the header and field renderers below. + let materializedTypeContext = try typeDefinition.materializedTypeContext(in: machO) + try await DeclarationBlock(level: level) { - try await renderTypeDeclarationHeader(for: typeDefinition.type, displayParentName: displayParentName, level: level, specializedMetadata: specializedMetadata) + try await renderTypeDeclarationHeader(for: materializedTypeContext, displayParentName: displayParentName, level: level, specializedMetadata: specializedMetadata) } body: { for child in typeDefinition.typeChildren { try await NestedDeclaration { @@ -142,7 +146,7 @@ public final class SwiftDeclarationPrinter: Sendab } } - try await renderModelFields(typeDefinition, level: level) + try await renderModelFields(typeDefinition, typeContext: materializedTypeContext, level: level) try await printDefinition(typeDefinition, level: level) } @@ -152,7 +156,12 @@ public final class SwiftDeclarationPrinter: Sendab @SemanticStringBuilder public func printProtocolDefinition(_ protocolDefinition: ProtocolDefinition, level: Int = 1, displayParentName: Bool = false) async throws -> SemanticString { - let printingContext = SwiftIndexEvents.PrintingContext(name: protocolDefinition.protocol.name, kind: .protocol) + // This print operation's single wrapper materialization (proposal + // 0002), threaded into the event context, header, and + // associated-type renderers below. + let dumpedProtocol = try protocolDefinition.materializedProtocol(in: machO) + + let printingContext = SwiftIndexEvents.PrintingContext(name: dumpedProtocol.name, kind: .protocol) eventDispatcher.dispatch(.definitionPrintStarted(context: printingContext)) if !protocolDefinition.isIndexed { @@ -160,9 +169,9 @@ public final class SwiftDeclarationPrinter: Sendab } try await DeclarationBlock(level: level) { - try await renderProtocolDeclarationHeader(for: protocolDefinition.protocol, displayParentName: displayParentName) + try await renderProtocolDeclarationHeader(for: dumpedProtocol, displayParentName: displayParentName) } body: { - try await renderProtocolAssociatedTypes(for: protocolDefinition.protocol, level: level) + try await renderProtocolAssociatedTypes(for: dumpedProtocol, level: level) try await printDefinition(protocolDefinition, level: level) @@ -235,6 +244,11 @@ public final class SwiftDeclarationPrinter: Sendab Space() extensionDefinition.extensionName.print() + // This print operation's single conformance materialization + // (proposal 0002). A materialization failure is treated like the + // thrown-resolution case below — the whole clause is dropped. + let materializedProtocolConformance = try? extensionDefinition.materializedProtocolConformance(in: machO) + // Pre-leaf-migration `dumpProtocolName` semantics: a `nil` protocol // node collapses to an *empty* name but still emits the clause (the // dangling `extension Foo: @retroactive ` form), while a *thrown* @@ -243,7 +257,7 @@ public final class SwiftDeclarationPrinter: Sendab // including its `@retroactive` / global-actor markers — whenever the // reference was unresolvable. let conformanceProtocolName: SemanticString? = { - guard let protocolConformance = extensionDefinition.protocolConformance else { return nil } + guard let protocolConformance = materializedProtocolConformance else { return nil } do { let protocolNode = try protocolConformance.protocolNode(in: machO) return protocolNode?.printSemantic(using: .interfaceTypeBuilderOnly) ?? SemanticString() @@ -251,7 +265,7 @@ public final class SwiftDeclarationPrinter: Sendab return nil } }() - if let protocolConformance = extensionDefinition.protocolConformance, + if let protocolConformance = materializedProtocolConformance, let protocolName = conformanceProtocolName { Standard(":") Space() diff --git a/Sources/SwiftSpecialization/ConformanceProvider.swift b/Sources/SwiftSpecialization/ConformanceProvider.swift index 473e017a..508689c9 100644 --- a/Sources/SwiftSpecialization/ConformanceProvider.swift +++ b/Sources/SwiftSpecialization/ConformanceProvider.swift @@ -128,11 +128,11 @@ extension IndexerConformanceProvider: ConformanceProvider { } public func doesType(_ typeName: TypeName, conformTo protocolName: ProtocolName) -> Bool { - indexer.allProtocolConformancesByTypeName[typeName]?[protocolName] != nil + indexer.allConformingProtocolNamesByTypeName[typeName]?.contains(protocolName) == true } public func conformances(of typeName: TypeName) -> [ProtocolName] { - Array(indexer.allProtocolConformancesByTypeName[typeName]?.keys ?? []) + Array(indexer.allConformingProtocolNamesByTypeName[typeName] ?? []) } public var allTypeNames: [TypeName] { @@ -185,10 +185,15 @@ extension IndexerConformanceProvider: ConformanceProvider { var map: [String: [TypeName]] = [:] for (childTypeName, entry) in indexer.allAllTypeDefinitions { guard childTypeName.kind == .class else { continue } - guard case .class(let classWrapper) = entry.value.type else { continue } + guard case .class(let classDescriptor) = entry.value.typeContextDescriptorWrapper else { continue } + // The superclass reference can live in the wrapper's trailing + // objects (resilient superclass), so this map build materializes + // the class wrapper — once per class, cached with the map + // (materialization discipline, proposal 0002). var superNode: Node? do { + let classWrapper = try Class(descriptor: classDescriptor, in: entry.machO) superNode = try classWrapper.superclassNode(in: entry.machO) } catch { continue diff --git a/Sources/SwiftSpecialization/GenericSpecializer.swift b/Sources/SwiftSpecialization/GenericSpecializer.swift index 60862bdc..cde7a3d0 100644 --- a/Sources/SwiftSpecialization/GenericSpecializer.swift +++ b/Sources/SwiftSpecialization/GenericSpecializer.swift @@ -595,7 +595,7 @@ extension GenericSpecializer { guard let typeDefinition = conformanceProvider.typeDefinition(for: typeName) else { return nil } - let isGeneric = typeDefinition.type.typeContextDescriptorWrapper.typeContextDescriptor.layout.flags.isGeneric + let isGeneric = typeDefinition.typeContextDescriptorWrapper.typeContextDescriptor.layout.flags.isGeneric if options.contains(.excludeGenerics), isGeneric { return nil } @@ -654,7 +654,7 @@ extension GenericSpecializer { reason: "Type not found in indexer" ) } - return (typeDefinitionEntry.value.type.typeContextDescriptorWrapper, typeDefinitionEntry.machO) + return (typeDefinitionEntry.value.typeContextDescriptorWrapper, typeDefinitionEntry.machO) } /// Build the descriptor + inner specializer pair that drives @@ -950,7 +950,7 @@ extension GenericSpecializer where MachO == MachOImage { let descriptor: MachOSwiftSection.`Protocol` do { descriptor = try MachOSwiftSection.`Protocol`( - descriptor: protocolDef.value.protocol.descriptor.asPointerWrapper(in: protocolDef.machO) + descriptor: protocolDef.value.protocolDescriptor.asPointerWrapper(in: protocolDef.machO) ) } catch { // Indexer found the entry but materializing the @@ -1836,7 +1836,7 @@ extension GenericSpecializer where MachO == MachOImage { // Create in-process protocol descriptor and use runtime conformance check let protocolDescriptor = try MachOSwiftSection.`Protocol`( - descriptor: protocolDef.value.protocol.descriptor.asPointerWrapper(in: protocolDef.machO) + descriptor: protocolDef.value.protocolDescriptor.asPointerWrapper(in: protocolDef.machO) ) guard let witnessTable = try RuntimeFunctions.conformsToProtocol( @@ -1982,7 +1982,7 @@ extension GenericSpecializer where MachO == MachOImage { let stepProtocol: MachOSwiftSection.`Protocol` do { stepProtocol = try MachOSwiftSection.`Protocol`( - descriptor: entry.value.protocol.descriptor.asPointerWrapper(in: entry.machO) + descriptor: entry.value.protocolDescriptor.asPointerWrapper(in: entry.machO) ) } catch { throw AssociatedTypeResolutionError.failedToCreateAssociatedTypeRefProtocol(underlyingError: error) diff --git a/Sources/SwiftSpecialization/TypeDefinition+Specialization.swift b/Sources/SwiftSpecialization/TypeDefinition+Specialization.swift index 8aacedad..29763a08 100644 --- a/Sources/SwiftSpecialization/TypeDefinition+Specialization.swift +++ b/Sources/SwiftSpecialization/TypeDefinition+Specialization.swift @@ -57,7 +57,7 @@ extension TypeDefinition { /// ```swift /// let manual = nestedDef.specializedChildren /// let viaOuter = outerDef.specializedChildren.flatMap { outerInstance in - /// outerInstance.typeChildren.filter { $0.type === nestedDef.type } + /// outerInstance.typeChildren.filter { $0.typeContextDescriptorWrapper.typeContextDescriptor.offset == nestedDef.typeContextDescriptorWrapper.typeContextDescriptor.offset } /// } /// ``` /// @@ -200,6 +200,11 @@ extension TypeDefinition { try validateSpecialization(metadata: metadata, in: machO) + // This specialize operation's single wrapper materialization + // (proposal 0002): feeds the typeName derivation and the designated + // init below, released when this function returns. + let materializedTypeContext = try materializedTypeContext(in: machO) + // Compute the final typeName up-front so it can flow through the // designated init: either the unbound form (`Box`) when no type // arguments are supplied, or the bound form (`Box`) produced by @@ -207,7 +212,7 @@ extension TypeDefinition { // definition print as `Box` rather than the placeholder // `Box`, and gives it a unique mangled name per specialization // (via `mangleAsString(typeName.node)`). - let unboundTypeName = try type.typeName(in: machO) + let unboundTypeName = try materializedTypeContext.typeName(in: machO) let finalTypeName: TypeName if let typeArgumentNodes, !typeArgumentNodes.isEmpty { finalTypeName = Self.boundGenericTypeName( @@ -218,7 +223,7 @@ extension TypeDefinition { finalTypeName = unboundTypeName } - let specialized = TypeDefinition(type: type, typeName: finalTypeName, isSpecialized: true) + let specialized = TypeDefinition(type: materializedTypeContext, typeName: finalTypeName, isSpecialized: true) specialized.metadata = metadata return specialized } @@ -238,7 +243,7 @@ extension TypeDefinition { var derivedChildren: [TypeDefinition] = [] for child in typeChildren { - guard child.type.typeContextDescriptorWrapper.typeContextDescriptor.layout.flags.isGeneric else { + guard child.typeContextDescriptorWrapper.typeContextDescriptor.layout.flags.isGeneric else { continue } @@ -249,7 +254,7 @@ extension TypeDefinition { // still returned with whatever siblings *did* succeed, so a // partial sidebar tree beats a missing one. do { - let request = try specializer.makeRequest(for: child.type.typeContextDescriptorWrapper) + let request = try specializer.makeRequest(for: child.typeContextDescriptorWrapper) var childArguments: [String: SpecializationSelection.Argument] = [:] var childArgumentNodes: [Node] = [] var childNodesByParameter: [String: Node] = [:] @@ -354,7 +359,7 @@ extension TypeDefinition { // 1. Receiver must be generic. A non-generic descriptor has a // fixed metadata; specializing it is meaningless and would // indicate the caller wired the wrong type. - guard type.typeContextDescriptorWrapper.typeContextDescriptor.layout.flags.isGeneric else { + guard typeContextDescriptorWrapper.typeContextDescriptor.layout.flags.isGeneric else { throw SpecializationError.notGenericType(typeName: typeName.name) } @@ -363,7 +368,7 @@ extension TypeDefinition { // distinguishes these by metadata kind only, and either can be // the legitimate output of specializing an enum. let isCompatibleKind: Bool - switch type { + switch typeContextDescriptorWrapper { case .struct: isCompatibleKind = metadata.isStruct case .enum: isCompatibleKind = metadata.isEnum || metadata.isOptional case .class: isCompatibleKind = metadata.isClass @@ -371,7 +376,7 @@ extension TypeDefinition { guard isCompatibleKind else { throw SpecializationError.metadataKindMismatch( typeName: typeName.name, - expected: type, + expected: typeContextDescriptorWrapper, actual: metadata ) } @@ -381,7 +386,7 @@ extension TypeDefinition { // so that the offsets being compared are both process-memory // addresses. A mismatch means the result was specialized for // a structurally similar but distinct type. - let inProcessType = type.typeContextDescriptorWrapper.asPointerWrapper(in: machO) + let inProcessType = typeContextDescriptorWrapper.asPointerWrapper(in: machO) let expectedDescriptorOffset = inProcessType.typeContextDescriptor.offset let actualDescriptorOffset = try descriptorOffset(of: metadata) guard expectedDescriptorOffset == actualDescriptorOffset else { @@ -415,7 +420,7 @@ extension TypeDefinition { /// supplied `SpecializationResult` cannot be reconciled with the receiver. public enum SpecializationError: LocalizedError { case notGenericType(typeName: String) - case metadataKindMismatch(typeName: String, expected: TypeContextWrapper, actual: MetadataWrapper) + case metadataKindMismatch(typeName: String, expected: TypeContextDescriptorWrapper, actual: MetadataWrapper) case descriptorMismatch(typeName: String, expectedOffset: Int, actualOffset: Int) case unsupportedMetadataKind(metadata: MetadataWrapper) diff --git a/Tests/SwiftIndexingTests/DeclarationModelInstanceSizeTests.swift b/Tests/SwiftIndexingTests/DeclarationModelInstanceSizeTests.swift new file mode 100644 index 00000000..d14243c0 --- /dev/null +++ b/Tests/SwiftIndexingTests/DeclarationModelInstanceSizeTests.swift @@ -0,0 +1,37 @@ +import Foundation +import ObjectiveC +import Testing +import MachOSwiftSection +import SwiftDeclaration + +/// Evolution proposal 0002 regression guard: the declaration model retains +/// descriptor references, not eagerly parsed wrappers, so its per-instance +/// footprint must stay in the descriptor-sized band. The pre-slimming sizes +/// (measured 2026-08-09 with this same probe, see +/// `Documentations/Internal/DeclarationModelMemoryFootprint.md`) were +/// `TypeDefinition` 1272 B, `ExtensionDefinition` 640 B, +/// `ProtocolDefinition` 440 B; the post-slimming sizes are 384 / 224 / 384 B, +/// and the ceilings leave room for a few added fields — a regression that re-embeds a wrapper +/// (`TypeContextWrapper` alone is 472 B inline) blows straight through +/// these ceilings. +@Suite struct DeclarationModelInstanceSizeTests { + @Test func definitionInstancesStayDescriptorSized() { + let typeDefinitionSize = class_getInstanceSize(TypeDefinition.self) + let extensionDefinitionSize = class_getInstanceSize(ExtensionDefinition.self) + let protocolDefinitionSize = class_getInstanceSize(ProtocolDefinition.self) + print("Definition instance sizes — TypeDefinition: \(typeDefinitionSize) B, ExtensionDefinition: \(extensionDefinitionSize) B, ProtocolDefinition: \(protocolDefinitionSize) B") + + #expect(typeDefinitionSize <= 448, "TypeDefinition instance size: \(typeDefinitionSize) B") + #expect(extensionDefinitionSize <= 320, "ExtensionDefinition instance size: \(extensionDefinitionSize) B") + #expect(protocolDefinitionSize <= 416, "ProtocolDefinition instance size: \(protocolDefinitionSize) B") + } + + /// The retained references themselves must stay descriptor-sized: they + /// are raw layout + offset values, an order of magnitude below the + /// parsed wrappers they replaced. + @Test func retainedDescriptorReferencesStayCompact() { + #expect(MemoryLayout.size <= 128, "TypeContextDescriptorWrapper: \(MemoryLayout.size) B") + #expect(MemoryLayout.size <= 64, "ProtocolConformanceDescriptor: \(MemoryLayout.size) B") + #expect(MemoryLayout.size <= 96, "ProtocolDescriptor: \(MemoryLayout.size) B") + } +} diff --git a/Tests/SwiftIndexingTests/SymbolTestsCoreIntegrationTests.swift b/Tests/SwiftIndexingTests/SymbolTestsCoreIntegrationTests.swift index 6887d5fd..d0fac66e 100644 --- a/Tests/SwiftIndexingTests/SymbolTestsCoreIntegrationTests.swift +++ b/Tests/SwiftIndexingTests/SymbolTestsCoreIntegrationTests.swift @@ -112,10 +112,10 @@ extension STCoreTests { extension STCoreTests { @Test func structTestConformances() async throws { let indexer = try await preparedIndexer() - let conformancesByType = indexer.protocolConformancesByTypeName + let conformingProtocolNamesByType = indexer.conformingProtocolNamesByTypeName - let structTestConformances = conformancesByType.first { $0.key.name.hasSuffix(".StructTest") } - let protocolNames = try #require(structTestConformances?.value.keys.map(\.name)) + let structTestConformances = conformingProtocolNamesByType.first { $0.key.name.hasSuffix(".StructTest") } + let protocolNames = try #require(structTestConformances?.value.map(\.name)) #expect(protocolNames.contains(where: { $0.hasSuffix(".ProtocolTest") })) #expect(protocolNames.contains(where: { $0.hasSuffix(".ProtocolWitnessTableTest") })) @@ -123,10 +123,10 @@ extension STCoreTests { @Test func genericReqConformance() async throws { let indexer = try await preparedIndexer() - let conformancesByType = indexer.protocolConformancesByTypeName + let conformingProtocolNamesByType = indexer.conformingProtocolNamesByTypeName - let genericConformances = conformancesByType.first { $0.key.name.hasSuffix(".GenericRequirementTest") } - let protocolNames = try #require(genericConformances?.value.keys.map(\.name)) + let genericConformances = conformingProtocolNamesByType.first { $0.key.name.hasSuffix(".GenericRequirementTest") } + let protocolNames = try #require(genericConformances?.value.map(\.name)) #expect(protocolNames.contains(where: { $0.hasSuffix(".ProtocolTest") })) } diff --git a/Tests/SwiftSpecializationTests/GenericSpecializationTests.swift b/Tests/SwiftSpecializationTests/GenericSpecializationTests.swift index 77c8d215..7e2fcf33 100644 --- a/Tests/SwiftSpecializationTests/GenericSpecializationTests.swift +++ b/Tests/SwiftSpecializationTests/GenericSpecializationTests.swift @@ -549,7 +549,7 @@ struct GenericSpecializationTests { indexer: try await indexer ) let request = try specializer.makeRequest( - for: entry.value.type.typeContextDescriptorWrapper + for: entry.value.typeContextDescriptorWrapper ) #expect(request.parameters.count == 2, "Result has two type parameters") let parameterNames = request.parameters.map(\.name) diff --git a/Tests/SwiftSpecializationTests/GenericTypeNameSubstitutionTests.swift b/Tests/SwiftSpecializationTests/GenericTypeNameSubstitutionTests.swift index 8e3e390c..e184ca49 100644 --- a/Tests/SwiftSpecializationTests/GenericTypeNameSubstitutionTests.swift +++ b/Tests/SwiftSpecializationTests/GenericTypeNameSubstitutionTests.swift @@ -236,7 +236,7 @@ struct GenericTypeNameSubstitutionEndToEndTests: GenericSpecializationTestingEnv func substitutesStructTypeName() async throws { let baseDefinition = try await resolveTypeDefinition(named: "TestUnconstrainedStruct") let specializer = GenericSpecializer(indexer: try await indexer) - let request = try specializer.makeRequest(for: baseDefinition.type.typeContextDescriptorWrapper) + let request = try specializer.makeRequest(for: baseDefinition.typeContextDescriptorWrapper) let result = try specializer.specialize(request, with: ["A": .metatype(Int.self)]) let intTypeNode = makeSwiftStdLibTypeNode(name: "Int") @@ -261,7 +261,7 @@ struct GenericTypeNameSubstitutionEndToEndTests: GenericSpecializationTestingEnv func mangleDemangleRoundTrip() async throws { let baseDefinition = try await resolveTypeDefinition(named: "TestDualAssociatedStruct") let specializer = GenericSpecializer(indexer: try await indexer) - let request = try specializer.makeRequest(for: baseDefinition.type.typeContextDescriptorWrapper) + let request = try specializer.makeRequest(for: baseDefinition.typeContextDescriptorWrapper) let result = try specializer.specialize(request, with: [ "A": .metatype([Int].self), "B": .metatype([String].self), @@ -305,7 +305,7 @@ struct GenericTypeNameSubstitutionEndToEndTests: GenericSpecializationTestingEnv func printsBoundHeaderAndSubstitutedFieldTypes() async throws { let baseDefinition = try await resolveTypeDefinition(named: "TestUnconstrainedStruct") let specializer = GenericSpecializer(indexer: try await indexer) - let request = try specializer.makeRequest(for: baseDefinition.type.typeContextDescriptorWrapper) + let request = try specializer.makeRequest(for: baseDefinition.typeContextDescriptorWrapper) let result = try specializer.specialize(request, with: ["A": .metatype(Int.self)]) let specialized = try await baseDefinition.specialize( @@ -333,7 +333,7 @@ struct GenericTypeNameSubstitutionEndToEndTests: GenericSpecializationTestingEnv func nilSubstitutionPreservesUnboundTypeName() async throws { let baseDefinition = try await resolveTypeDefinition(named: "TestUnconstrainedStruct") let specializer = GenericSpecializer(indexer: try await indexer) - let request = try specializer.makeRequest(for: baseDefinition.type.typeContextDescriptorWrapper) + let request = try specializer.makeRequest(for: baseDefinition.typeContextDescriptorWrapper) let result = try specializer.specialize(request, with: ["A": .metatype(Int.self)]) // Default parameter: typeArgumentNodes is nil — preserves backward @@ -352,7 +352,7 @@ struct GenericTypeNameSubstitutionEndToEndTests: GenericSpecializationTestingEnv func uniqueMangledNamesPerSpecialization() async throws { let baseDefinition = try await resolveTypeDefinition(named: "TestUnconstrainedStruct") let specializer = GenericSpecializer(indexer: try await indexer) - let request = try specializer.makeRequest(for: baseDefinition.type.typeContextDescriptorWrapper) + let request = try specializer.makeRequest(for: baseDefinition.typeContextDescriptorWrapper) let intResult = try specializer.specialize(request, with: ["A": .metatype(Int.self)]) let stringResult = try specializer.specialize(request, with: ["A": .metatype(String.self)]) @@ -401,7 +401,7 @@ struct GenericTypeNameSubstitutionEndToEndTests: GenericSpecializationTestingEnv ) let specializer = GenericSpecializer(indexer: try await indexer) - let valueRequest = try specializer.makeRequest(for: valueChild.type.typeContextDescriptorWrapper) + let valueRequest = try specializer.makeRequest(for: valueChild.typeContextDescriptorWrapper) let valueStringResult = try specializer.specialize(valueRequest, with: ["A": .metatype(String.self)]) let stringNode = makeSwiftStdLibTypeNode(name: "String") let manuallySpecializedValue = try await valueChild.specialize( @@ -411,7 +411,7 @@ struct GenericTypeNameSubstitutionEndToEndTests: GenericSpecializationTestingEnv ) #expect(valueChild.specializedChildren.contains { $0 === manuallySpecializedValue }) - let outerRequest = try specializer.makeRequest(for: baseDefinition.type.typeContextDescriptorWrapper) + let outerRequest = try specializer.makeRequest(for: baseDefinition.typeContextDescriptorWrapper) let outerSelection = SpecializationSelection(arguments: ["A": .metatype(Int.self)]) let outerResult = try specializer.specialize(outerRequest, with: outerSelection) let intNode = makeSwiftStdLibTypeNode(name: "Int") @@ -496,13 +496,13 @@ struct GenericTypeNameSubstitutionEndToEndTests: GenericSpecializationTestingEnv baseDefinition.typeChildren.first { $0.typeName.name.contains("LayoutConstrainedInner") }, "expected LayoutConstrainedInner among outer's typeChildren" ) - let layoutRequest = try specializer.makeRequest(for: layoutConstrainedChild.type.typeContextDescriptorWrapper) + let layoutRequest = try specializer.makeRequest(for: layoutConstrainedChild.typeContextDescriptorWrapper) #expect(throws: (any Error).self, "LayoutConstrainedInner must reject A = Int so the outer catch actually fires") { _ = try specializer.specialize(layoutRequest, with: ["A": .metatype(Int.self)]) } - let outerRequest = try specializer.makeRequest(for: baseDefinition.type.typeContextDescriptorWrapper) + let outerRequest = try specializer.makeRequest(for: baseDefinition.typeContextDescriptorWrapper) let outerSelection = SpecializationSelection(arguments: ["A": .metatype(Int.self)]) let outerResult = try specializer.specialize(outerRequest, with: outerSelection) let intNode = makeSwiftStdLibTypeNode(name: "Int") From ff665d8b3562e714435a45a26338f88caba9d5f4 Mon Sep 17 00:00:00 2001 From: Mx-Iris Date: Sun, 9 Aug 2026 14:33:07 +0800 Subject: [PATCH 54/77] docs(evolutions): record the downstream acceptance re-measure for 0002/0003 RuntimeViewer's five-image workload after its eight mechanical call-site renames: steady-state footprint 322 -> 262 MB, live heap 283 -> 209.6 MiB (beats the 240-255 expectation), SwiftDeclaration cluster 41.3 -> 19.5 MiB, MachOSwiftSection parse cluster 33.4 -> 3.3 MiB (beats 10-15), the [UInt32] bucket cluster 38.8 -> 7.2 MiB (beats 15-20), and an unpromised bonus: indexing transient peak 808 -> 613 MB. --- .../Evolutions/0002-declaration-model-descriptor-slimming.md | 3 ++- Documentations/Evolutions/0003-symbol-row-bucket-flattening.md | 3 ++- 2 files changed, 4 insertions(+), 2 deletions(-) diff --git a/Documentations/Evolutions/0002-declaration-model-descriptor-slimming.md b/Documentations/Evolutions/0002-declaration-model-descriptor-slimming.md index 8e118791..6804d913 100644 --- a/Documentations/Evolutions/0002-declaration-model-descriptor-slimming.md +++ b/Documentations/Evolutions/0002-declaration-model-descriptor-slimming.md @@ -163,7 +163,7 @@ extension ProtocolDefinition { 4. ✅ 库内消费点全量迁移：打印器（`printTypeDefinition` / `printProtocolDefinition` 各一次物化贯穿 header + 字段/关联类型渲染；`printExtensionHeader` 一次）、`GenericSpecializer` / `TypeDefinition+Specialization` / `ConformanceProvider`（后者的子类图构建按类物化一次、随图缓存）、`SwiftAttributeInference` / `SwiftInterface` diff 渲染 / `SwiftDiffing` 的 kind 判断（descriptor 级改名）、测试侧 13 处。 5. ✅ 全量 `swift test --skip IntegrationTests` 1343 全绿；渲染 A/B 七对（iOS 18.5 模拟器 SwiftUI / SwiftData / SwiftUICore 的 dump + interface，宿主机 dyld shared cache 的 SwiftUI dump + interface）全部逐字节一致（interface 剥离日志行首时间戳后比对）。 6. 性能：iOS 18.5 模拟器 SwiftUI `interface` wall-clock 持平（物化 CPU 的验收）——见决策日志的 release 复测记录。 -7. RV 复测（对面协调）：预期堆存活 283 → **~240–255 MiB**；`SwiftDeclaration` 簇 41.3 → ~15–20 MiB；MachOSwiftSection 簇 33.4 → ~10–15 MiB。**待下游拿到本分支后进行。** +7. ✅ RV 复测（2026-08-09，五镜像同款负载，RV 侧 8 处适配后干净跑）：footprint 稳态 **322 → 262 MB**；堆存活 **283 → 209.6 MiB**（超出预期带 240–255，分配数 −50.7 万）；`SwiftDeclaration` 簇 41.3 → **19.5 MiB**（预期带内）；MachOSwiftSection 解析簇 33.4 → **3.3 MiB**（超出预期 10–15——`ProtocolConformance` 数组 20.6 MiB 整体消失）；另有意外之喜：索引瞬态峰值 **808 → 613 MB**（解析期 wrapper churn 被砍）。不动的簇（Demangling 33.2、ObjCDump 17.2、UI/Rx 21.4、NIO 5.8）均未动。RV 侧实例复量与本仓库探针一致（TypeDefinition 1280 vs 1272 为 malloc 桶圆整）。 8. ✅ 以账本同款探针复量：`TypeDefinition` 1272 → **384 B**、`ExtensionDefinition` 640 → **224 B**、`ProtocolDefinition` 440 → **384 B**；[DeclarationModelMemoryFootprint.md](../Internal/DeclarationModelMemoryFootprint.md) 已补后记，且探针固化为常驻回归守卫 `DeclarationModelInstanceSizeTests`（上限 448 / 320 / 416 B）。 9. ✅ 收尾判断：不另写实现说明——「物化纪律」已写进 AGENTS.md 的 SwiftDeclaration 段与三个 `materialized…(in:)` 的 doc comment，实例尺寸契约由回归测试钉住，一篇独立文章只会复述这两处；术语表无新词（materialize / wrapper vs descriptor 两条已覆盖本案语汇，bucket 条随 0003 更新）。 @@ -179,3 +179,4 @@ extension ProtocolDefinition { | 2026-08-09 | 实施期修正：indexer Storage 侧同批清退 | 动手时发现提案调研的一处失实：「indexer 的全量 protocolConformances 数组在分组后本就出栈」不成立——`SwiftDeclarationIndexer.Storage` 以 `types` / `protocols` / `protocolConformances` / `associatedTypes` 四个人口数组加 `protocolConformancesByTypeName` / `associatedTypesByTypeName` 两个按名 keyed 映射**终身驻留全部 wrapper**（CoW 共享底层堆数组），不清退则 definition 侧换 descriptor 后 trailing 簇分文不释放。消费面普查：六者的公开投影在库内外（含 RV）**零调用方**，唯二例外是 `ConformanceProvider` 读 `allProtocolConformancesByTypeName` 的存在性 + keys（纯名字级事实）与两个测试读 keys。修正：四个人口数组在 `prepare()` 索引完成后置空；两个重映射降级为索引期局部变量，新增名字级轻映射 `conformingProtocolNamesByTypeName`（+ 合并投影 `allConformingProtocolNamesByTypeName`）承接 ConformanceProvider 与测试；重投影与 `allTypes` / `allProtocols` / `allProtocolConformances` / `allAssociatedTypes` 聚合一并移除（额外 API 破坏，均为零调用方）。 | | 2026-08-09 | wall-clock 验收：release 持平（一对反而更快） | debug 构建初测候选慢 5–10%（SwiftUICore interface 反转执行序复测仍 ~9%），但最大任务 dyld cache SwiftUI interface 仅 +0.2%，疑为 debug 常数因子；改以 release 构建 ABBA 序 ×2 轮定论：SwiftUI interface 基线均值 76.3s vs 候选 72.2s（候选**快 5.3%**），SwiftUICore 37.3s vs 37.4s（+0.5%，噪声带内）。验收线达成，以 release 为准；release 输出与 debug 同样逐字节一致。 | | 2026-08-09 | Accepted → Implemented | 落地步骤 1–5、8、9 完成（步骤 6 见上一行）：三定义 descriptor 化 + `parentContext` / `ParentContext` 移除 + 三个物化入口 + indexer Storage 清退 + 库内与测试侧全量迁移；全量 1343 绿；A/B 七对（debug 与 release 双构建）逐字节一致；实例尺寸 1272 → 384 / 640 → 224 / 440 → 384 B（前两者优于预估 ~400），回归守卫 `DeclarationModelInstanceSizeTests` 落位。步骤 7（RV 堆复测）待下游拿到本分支后进行；RV 侧 8 处机械迁移句式已在步骤 1 备好。 | +| 2026-08-09 | 下游验收回报：全部达标、三项超预期 | RV 会话完成 8 处适配（与步骤 1 清单一致，零物化调用）并复测：稳态 322 → 262 MB、堆存活 283 → 209.6 MiB（超预期）、MSS 解析簇 33.4 → 3.3 MiB（超预期）、索引瞬态峰值 808 → 613 MB（超预期收获——提案未承诺瞬态收益）。数字已回填落地步骤 7。五镜像稳态全程曲线：842（起点）→ 470–480 → ~450 → 322（0001）→ **262 MB**（0002+0003）。 | diff --git a/Documentations/Evolutions/0003-symbol-row-bucket-flattening.md b/Documentations/Evolutions/0003-symbol-row-bucket-flattening.md index af1ae4ca..648e824d 100644 --- a/Documentations/Evolutions/0003-symbol-row-bucket-flattening.md +++ b/Documentations/Evolutions/0003-symbol-row-bucket-flattening.md @@ -99,7 +99,7 @@ enum SymbolRowBucket { 1. ✅ `SymbolRowBucket` 实现 + 布局断言（`compactValueLayouts` 钉 `stride ≤ 16`）+ 单测（append 迁移、迭代序、`contains`——`symbolRowBucketAppendMigrationAndIterationOrder`）。 2. ✅ `symbolRowsByOffset` 与三族 `MemberSymbolRows` 叶子桶替换（`demangledSymbols(atRows:)` 泛化为 `some Sequence`，查询出口形态不变）；单元素占比统计落为 `Storage.bucketFormStatisticsForTesting()`——常驻单测 `rowBucketsAreDominatedBySingleRowForm` 断言并打印，另在 IntegrationTests 的 baseline 指标里加了一行。fixture（SymbolTestsCore，MachOFile leg）实测 **87.6% 单元素**（6687 单 / 948 多），达到 ≥85% 预期带。 3. ✅ 全量 `swift test --skip IntegrationTests` 1343 全绿（含新增 2 项桶单测，无删减）；渲染 A/B（iOS 18.5 模拟器 SwiftUI / SwiftData / SwiftUICore 的 dump + interface，另加宿主机 dyld shared cache 的 SwiftUI dump + interface——canonical/raw 双键注册正是本案改动面）全部逐字节一致（interface 输出剥离日志行首时间戳后比对）。 -4. RV heap 复测：`[UInt32]` 簇 38.8 → 预期 ~15–20 MiB。**待下游拿到本分支后进行。** +4. ✅ RV heap 复测（2026-08-09，五镜像同款负载）：`[UInt32]` 小数组簇 38.8 → **7.2 MiB**（超出预期带 15–20）——43 万个碎数组坍缩进 5 个 `Dictionary` 共 17.7 MiB（账面归类迁入 MachOSymbols 簇，11 → 27.9，为分类迁移而非回归）。与 0002 合计：稳态 footprint 322 → 262 MB。 ## 决策日志 @@ -108,3 +108,4 @@ enum SymbolRowBucket { | 2026-08-09 | Created as In Review | 0001「非目标」点名的候选正式立项；RV 实测簇 38.8 MiB / 45 万个为输入；用户批准立项（「可以，写提案」）。 | | 2026-08-09 | In Review → Accepted | 用户审核通过（「审核通过，开始实现」），与 0002 同批开工，两案独立实施。 | | 2026-08-09 | Accepted → Implemented | 落地步骤 1–3 完成：`SymbolRowBucket`（`RandomAccessCollection`，单元素内联、次元素起落堆、插入序迭代）替换四处桶；fixture 单元素占比 87.6%；全量 1343 绿；A/B 七对（含 dyld cache 两对）逐字节一致。步骤 4（RV heap 复测）待下游拿到分支后进行。 | +| 2026-08-09 | 下游验收回报：超预期 | RV 复测 `[UInt32]` 簇 38.8 → 7.2 MiB（预期 15–20），碎数组人口坍缩为 5 个桶字典。数字已回填落地步骤 4。 | From fab92734259533dda99c8e79bee8c06159b4c57d Mon Sep 17 00:00:00 2001 From: Mx-Iris Date: Sun, 9 Aug 2026 17:32:30 +0800 Subject: [PATCH 55/77] docs(evolutions): reconcile the proposal index after rebasing onto main The rebase brought proposals 0001-0003 and main's 0004 into one history: restore 0004's row in the status table, relink its related-proposal references, and record that the merge precondition for the RV real-device verification step is now satisfied. --- .../Evolutions/0004-arm64e-signed-vwt-pointer-hardening.md | 5 +++-- Documentations/Evolutions/README.md | 1 + 2 files changed, 4 insertions(+), 2 deletions(-) diff --git a/Documentations/Evolutions/0004-arm64e-signed-vwt-pointer-hardening.md b/Documentations/Evolutions/0004-arm64e-signed-vwt-pointer-hardening.md index 5ffaf187..f08adc23 100644 --- a/Documentations/Evolutions/0004-arm64e-signed-vwt-pointer-hardening.md +++ b/Documentations/Evolutions/0004-arm64e-signed-vwt-pointer-hardening.md @@ -5,7 +5,7 @@ - **创建日期**: 2026-08-09 - **最后更新**: 2026-08-09 - **所属愿景**: 无 -- **关联提案**: 无（与 0002/0003 的内存线正交；崩溃场景与 0002 的下游验收同在 RV 注入路径上被发现。0002/0003 的提案文件在 `feature/node-store-migration` 分支上，尚未并入 main） +- **关联提案**: 无（与 [0002](0002-declaration-model-descriptor-slimming.md)/[0003](0003-symbol-row-bucket-flattening.md) 的内存线正交；崩溃场景与 0002 的下游验收同在 RV 注入路径上被发现） - **实现分支 / PR**: `main`（用户裁定：基线旧 bug、影响面大，不并入优化分支，直接在 main 修复；提案随之从 `feature/node-store-migration` 迁至 main） - **配套文档**: [RuntimeEnumCaseProjection.md](../Internal/RuntimeEnumCaseProjection.md)（其 arm64e 验证记录将由本案修正）；崩溃报告由 RV 侧会话提供（App Store 主二进制，macOS 26.6，注入 RuntimeViewer 后导出 interface 崩溃） @@ -137,7 +137,7 @@ guard let tablePointer = try? signedTablePointer.stripPointerTags() else { retur 2. ✅ 回归测试落为 `Tests/SwiftInspectionTests/Arm64eSignedVWTPointerTests.swift`（行为层 + 探针层）。**修复前取证**：行为层测试使测试进程死于 signal 11（SIGSEGV——复刻崩溃报告的 fault）；探针层 `raw` 模式子进程先打印 `slotCarriesTagBits=1` 再 SIGSEGV（shell 视角 exit 139），`strip` 模式解引用 + witness round trip 全通。**修复后**：三测全绿。 3. ✅ 全量 `swift test --skip IntegrationTests` 1303/1303（250 suites）全绿。期间按 AGENTS.md 环境漂移纪律排查掉一次 158 issue 的假阳性——主 checkout 的 fixture 二进制（8 月 2 日构建）比 main 的 fixture 源码（8 月 5/6 日两个 commit）旧，重建后归零，与本修复无关。 4. ✅ 文档修正同批：RuntimeEnumCaseProjection.md（失实验证记录改写、表指针 strip 与陷阱记录、新回归形态）+ AGENTS.md（SwiftInspection 段修正 + Test Environment 补 PAC 验证陷阱一段）。 -5. 待办：RV 侧真机验证（对面协调）：注入 arm64e 应用 + Print Enum Layout，确认崩溃消失。注意 RV 工作树当前适配的是 `feature/node-store-migration` 的 0002 API，验证需待该分支并入 main 后的合流构建（或 RV 暂以 main 构建）。 +5. 待办：RV 侧真机验证（对面协调）：注入 arm64e 应用 + Print Enum Layout，确认崩溃消失。合流已发生——`feature/node-store-migration` 已 rebase 到含本案两 commit 的 main 之上，RV 直接以该分支构建即可验证（其 0002 适配无需回退）。 ## 决策日志 @@ -148,3 +148,4 @@ guard let tablePointer = try? signedTablePointer.stripPointerTags() else { retur | 2026-08-09 | In Review → Accepted，迁至 main | 用户审核通过（「审核通过，开始实现」），并裁定实施基线改为 **main**——本 bug 是 main 上就有的基线问题、影响面大，不与 `feature/node-store-migration` 的内存优化线捆绑；提案文件随之从该分支迁到 main（分支上三个未推送的提案 commit 撤下，编号 0004 不变）。 | | 2026-08-09 | 实施期修正：「符号层」断言升级为「行为层」断言 | 原方案断言 `SwiftInspection` 构建产物含 `stripPointerTags` 的 mangled 符号。实施时查实其见证力不足：`stripPointerTags` 是 `package` 函数，静态链接后其**定义**符号无论有无调用方都出现在测试产物符号表里，而「projector 引用了它」这一事实在链接后与定义不可分辨——接线被删时断言照样绿。改为行为断言：fake metadata 的 VWT 槽置崩溃报告指针的 tag 位型 `0x0041_8000_0000_0000`（全部高于 VA 掩码，任何架构裸解引用必 fault），过真实 `projectCasePatterns` 入口——直接验证 strip 行为本身，严格强于符号断言且同样处处可跑（含 GitHub CI）。 | | 2026-08-09 | Accepted → Implemented | 按「先崩后过」实施：行为层测试在未修复代码上使测试进程死于 signal 11（SIGSEGV），探针层 `raw` 模式 arm64e 子进程证实槽带签名并 SIGSEGV（exit 139）、`strip` 模式 witness round trip 全通；应用 strip 修复后新套件 3 测全绿，全量 1303/1303（250 suites）全绿。文档修正同批。剩余：落地步骤 5 的 RV 侧真机验证（对面协调）。 | +| 2026-08-09 | 分支合流对账 | 用户指示将 `feature/node-store-migration` rebase 到 main（54 commit 重放，本案修复与测试零冲突随基线进入分支历史）；0001–0004 自此同库同历史，本文件的跨分支表述与提案总表随之对账（0004 行补入总表、关联提案恢复文件链接、落地步骤 5 的合流前提已满足）。 | diff --git a/Documentations/Evolutions/README.md b/Documentations/Evolutions/README.md index b2b04086..af6747fa 100644 --- a/Documentations/Evolutions/README.md +++ b/Documentations/Evolutions/README.md @@ -9,3 +9,4 @@ | [0001](0001-symbol-name-offsetization.md) | SymbolIndexStore 符号名 offset 化：驻留字符串换字符串表引用 | Implemented | | [0002](0002-declaration-model-descriptor-slimming.md) | 声明模型 descriptor 化：TypeDefinition / ExtensionDefinition / ProtocolDefinition 不再驻留急切解析的胖 wrapper | Implemented | | [0003](0003-symbol-row-bucket-flattening.md) | SymbolIndexStore `[UInt32]` 行号桶扁平化：单元素桶内联化 | Implemented | +| [0004](0004-arm64e-signed-vwt-pointer-hardening.md) | arm64e 签名 VWT 指针加固：进程内裸读 strip + 真 PAC 环境的回归验证形态 | Implemented | From 6c13db503ae0b73619123919ade5d2d92c5e69fa Mon Sep 17 00:00:00 2001 From: Mx-Iris Date: Sun, 9 Aug 2026 21:11:44 +0800 Subject: [PATCH 56/77] docs(roadmaps): record the PR #103 review findings with four-question adjudications --- Roadmaps/2026-08-09-pr103-review-findings.md | 248 +++++++++++++++++++ 1 file changed, 248 insertions(+) create mode 100644 Roadmaps/2026-08-09-pr103-review-findings.md diff --git a/Roadmaps/2026-08-09-pr103-review-findings.md b/Roadmaps/2026-08-09-pr103-review-findings.md new file mode 100644 index 00000000..9bad61e8 --- /dev/null +++ b/Roadmaps/2026-08-09-pr103-review-findings.md @@ -0,0 +1,248 @@ +# PR #103 Code Review Findings + +Review date: 2026-08-09 +PR: `feature/node-store-migration` → `main` (103 files, +6880/−874, merge base `a8968fa5`) +Review depth: automated multi-agent review at `max` effort, followed by a manual pass that answered the four mandatory questions (reproduce / baseline / worth fixing / fixed before) for every surviving finding against the branch, `main`, CI logs, and `git log`. + +Status: **Recorded and verified, not yet fixed.** + +All file/line references are against `feature/node-store-migration` at `8faff275`. "Baseline" means `main` at `a8968fa5`. "Upstream" means the `swift-demangling` package. + +--- + +## How to read this document + +Every finding answers the four questions AGENTS.md requires of a review finding, in this order: + +- **Q1 — Reproducible, or a false positive?** The concrete trigger, with the evidence that it is real. +- **Q2 — Does the baseline have it?** Whether this is newly introduced by the PR or pre-existing on `main`. Several findings are pre-existing *defects* whose *cost* is new — those are called out explicitly, because the fix priority follows the cost, not the defect's age. +- **Q3 — Worth fixing, and how wide is the blast radius?** +- **Q4 — Has this been fixed before?** Traced through `git log` / `git blame` / commit messages. Three findings are repeat offences on code that has already been fixed once or twice for the same class of problem. + +Severity buckets are **Blocker / High / Medium / Low**. Nothing here is adjudicated as "won't fix" — findings that were skipped as already-adjudicated or refuted are listed at the end, with pointers, so the next review round does not re-derive them. + +--- + +## Verification status + +Everything below was verified directly; nothing is reported on the review agent's word alone. + +- **B1** — both pins read from `Package.swift:212-222` on each branch; `git ls-remote --tags` against upstream confirms `0.5.1` is the newest published tag; CI run `31309763070` read with `gh run view --log-failed` (debug and release both fail with the same three errors). +- **H1** — the four clearing assignments read at `SwiftDeclarationIndexer.swift:~306`, the six accessors at `:1027-1042`; `main`'s only `currentStorage.types = []` confirmed to sit in an extraction-failure `catch` (`main:195`). Repo-wide grep confirms zero in-repo consumers. +- **H2** — the two early returns read at `ExtensionDefinition.swift:118-119` against `isIndexed = true` at `:171`; `main`'s equivalent early return confirmed at `main:95`; the four print-path probes read at `SwiftDeclarationPrinter.swift:208`, `:250`, `:305` and `SwiftDiffableInterfaceBuilder.swift:58`. +- **H3 / M4** — `BindOperation` payload types read from the local `../MachOKit` sibling (`Model/Bind/BindOperation.swift:27,37`): `segment`, `count` and `skip` are all `UInt` passed straight through from the opcode stream, with no validation in the decoding layer. +- **H4** — `compare_all_pairs` and `main()` read in full; the verdict path confirmed to depend solely on `difference_count`, which is only incremented inside the `baseline/*.txt` glob loop. +- **M1** — `dd1822c6` located via `git log -S "conflated the two"`; its analysis comment is still present verbatim at `SwiftDeclarationPrinter.swift:251-257`. **Reachability re-verified 2026-08-09** after the implementing session pushed back: `printExtensionHeader` has one in-repo caller (`:213`), preceded by a propagating `index(in:)` at `:209` that performs the same materialization at `ExtensionDefinition.swift:119` — so the `try?` is unreachable in-repo, and the finding was rewritten and downgraded. The original write-up asserted a rendering-path failure that does not exist; see the revision note on M1. +- **M5 / L2** — `a7caf944`'s commit message read in full; it documents the single-layer detach design that `6b0dad20`'s NodeStore arena later outgrew. +- **L1** — `main`'s block-level `printCatchedThrowing` and the PR's per-definition replacement read side by side from the `SwiftInterfaceBuilder.swift` diff. +- **L3** — the two prior rounds on the same function read at `17ad4358` and `6647359e`; rank arithmetic (`bestMatchRank = 0`, `rankStepsPerPathShape = 2`) read from `DyldCache+.swift:21,33,45`. + +Two claims are **reasoned, not executed**: the exact heap/时间 cost of H2's repeated materialization was not profiled (the mechanism is certain, the magnitude is not), and M2's use-after-unload was not reproduced with a live `dlclose` (the ownership chain is confirmed by code reading; whether any shipping consumer actually unloads an image was not established). + +--- + +## Shared root causes + +Most findings are not independent mistakes. Four causes account for eleven of the fifteen: + +| Root cause | Findings | +|---|---| +| **Descriptor slimming (evolution 0002) turned free stored-property reads into throwing, re-parsing calls** — every call site that treated the old read as free needs re-examining, not just the ones that stopped compiling | H1, H2, M1, L1 | +| **Symbol-table compaction (evolution 0001) introduced raw pointers and bit budgets over binary-supplied values** — the safety properties the old `String`-per-row representation gave for free now have to be enforced deliberately | M2, M3, M5 | +| **New binary-format decoding trusts the input** — the LC_DYLD_INFO opcode stream is attacker-controlled in the same way every other input to this library is | H3, M4 | +| **New verification harnesses have no self-check** — a harness that cannot fail is worse than no harness, because its green light was cited as acceptance evidence | H4, L2, L4 | + +The first row is the one worth generalizing: **evolution 0002's mechanical migration was driven by the compiler, and the compiler cannot see semantic changes.** A property that became a function still compiles at every call site that only needed its value; what changed is cost (H2), failure mode (M1, L1), and lifetime (H1). Any future descriptor-slimming step should audit call sites by hand rather than by build error. + +--- + +## Blocker + +### B1. The `swift-demangling` remote pin names a range with no tag containing the required API — the PR does not compile from a clean clone + +- **File:** `Package.swift:222` +- **Q1 — Reproducible.** The remote requirement is `"0.5.1" ..< "0.6.0"`. `git ls-remote --tags` against upstream ends at `0.5.1`; `SharedNodeStore` and `NodeStoreBuilder.reserveCapacity(expectedSymbolCount:)` exist only on the unpublished `feature/node-store` branch. Any resolution that goes through the remote picks `0.5.1` and fails. CI run `31309763070` (2026-08-09) fails in **both** debug and release with `cannot find 'SharedNodeStore' in scope` (`InternedNodeReferenceCache.swift:52`, `SymbolIndexStore.swift:212`) and `value of type 'NodeStoreBuilder' has no member 'reserveCapacity'` (`SymbolIndexStore.swift:560`). The 2026-08-04 run was green because its head predated `6b0dad20`, which introduced the arena usage. +- **Q2 — Not on the baseline.** `main` pins `"0.4.5" ..< "0.5.0"` with a comment stating the adoption lives on this branch and main keeps a closed upper bound until it lands. +- **Q3 — Merge blocker.** This is not "a scenario that misbehaves"; it is the whole PR failing to build anywhere that does not have both a `../swift-demangling` sibling *and* `USING_LOCAL_DEPENDENCIES=1` (`Package.swift:67,74-82` require both conditions). Every other finding in this document is unverifiable on CI until it is resolved. Two possible resolutions: publish an upstream `0.5.2` containing the arena API and re-pin, or revert to branch tracking — but branch tracking is precisely what `38b53c68` removed for cause. +- **Q4 — Fixed before, twice; this is the third round.** `3f7428ec` tracked upstream's `feature/node-store` branch → `38b53c68` replaced it with the `0.5.0` tag, whose message states the reason in full: *"Pinning to a branch made the package unresolvable for any version-based consumer and left builds non-reproducible"* → `6113d518` moved the floor to `0.5.1`. The lesson was recorded; it recurred in a new shape — a legally-resolvable tag that does not contain the symbols the code needs. `38b53c68` also documented its verification step (*"`swift package resolve` in a sibling-free checkout picks 0.5.0, and `swift build` succeeds"*), and that step was not repeated for this bump. +- **Suggested fix:** Publish the upstream tag, re-pin, and re-run resolution in a sibling-free checkout before pushing. Consider making the sibling-free resolution check part of CI so a pin that cannot build alone fails fast and loudly. + +--- + +## High + +### H1. `prepare()` clears the section-wrapper arrays, so six public statistics accessors silently return 0 + +- **File:** `Sources/SwiftIndexing/SwiftDeclarationIndexer.swift:1027-1042` (accessors), `:~306` (clearing) +- **Q1 — Reproducible.** `prepare()` now ends with `currentStorage.types = []` / `.protocols = []` / `.protocolConformances = []` / `.associatedTypes = []`. `numberOfTypes`, `numberOfEnums`, `numberOfStructs`, `numberOfClasses`, `numberOfProtocols` and `numberOfProtocolConformances` all read those arrays, and the only useful time to read them is after preparation. A panel that showed 5416 types for SwiftUI now shows 0. +- **Q2 — New.** On `main` the only `currentStorage.types = []` is the extraction-failure `catch` at `main:195`; the success path keeps the arrays for the indexer's lifetime. +- **Q3 — Worth fixing, cheap.** Zero in-repo consumers (grep hits only the definitions), so the entire blast radius is downstream (RuntimeViewer and similar). That is exactly what makes it worth fixing: a public API that silently returns a wrong answer is harder to notice than one that fails to compile — and evolution 0002's source-compatibility section lists only the three property renames, so a downstream reader has no warning. Fix by capturing the six counts into stored properties before clearing. +- **Q4 — No prior fix.** The statistics block has not been touched independently since the module split (`47b5961f`). +- **Also update:** evolution 0002's source-compatibility section, to list these six accessors alongside the three renames. + +### H2. `index(in:)`'s early returns leave `isIndexed` false, so the new materialization runs 3–4× per extension per print + +- **File:** `Sources/SwiftDeclaration/Components/Definitions/ExtensionDefinition.swift:118-119` +- **Q1 — Reproducible.** Both `guard protocolConformanceDescriptor != nil else { return }` (`:118`) and `guard let protocolConformance = try materializedProtocolConformance(in: machO), !protocolConformance.resilientWitnesses.isEmpty else { return }` (`:119`) return before `isIndexed = true` at `:171`. One interface run touches the same extension at four points: `SwiftDeclarationPrinter.swift:208` (`index()` → materialize → guard fails), `:250` (`try? materializedProtocolConformance`, a second parse), `:305` (`printDefinition` sees `!isIndexed` and calls `index()` again, a third), and `SwiftDiffableInterfaceBuilder.swift:58` (a fourth). +- **Q2 — The defect is pre-existing; the cost is new.** `main:95` has the same unset-flag early return. It was harmless there because the guard read the stored `protocolConformance` property. Evolution 0002 replaced that read with `materializedProtocolConformance(in:)`, which re-parses the conformance and its trailing objects every time. **Fix priority follows the new cost, not the old defect.** +- **Q3 — Worth fixing, one-line.** Typealias-only extensions are the majority case, and this sits on the full-interface export path. Add `isIndexed = true` before both early returns. `SwiftDiffableInterfaceBuilder.swift:58`'s comment (*"idempotent, so this is safe and cheap to re-enter"*) must either become true again or be deleted. +- **Q4 — No prior fix.** The file has three structural commits (`47b5961f`, `aa233bc0`, `d1078902`), none touching this flag. +- **Sweep result:** `TypeDefinition.index(in:)` and `ProtocolDefinition.index(in:)` have no mid-function early returns, so this is the only instance of the pattern. + +### H3. The LC_DYLD_INFO bind decoder trusts two binary-supplied uleb128 values with no bounds + +- **File:** `Sources/MachOExtensions/MachOFile+.swift:170` (segment index), `:185` (repeat count) +- **Q1 — Reproducible.** MachOKit decodes opcodes without validating them: `set_segment_and_offset_uleb(segment: UInt, offset: UInt)` and `do_bind_uleb_times_skipping_uleb(count: UInt, skip: UInt)` (`../MachOKit/Sources/MachOKit/Model/Bind/BindOperation.swift:27,37`) hand the raw values through. (a) `segmentIndex = Int(segment)` traps on any value above `Int.max`; the `segmentFileOffsets.indices.contains` guard inside `recordCurrentSlot` runs too late to help. (b) `for _ in 0 ..< count` is unbounded — a count of 2^40 either spins inserting dictionary entries until OOM (valid segment index) or simply hangs (invalid one). `segmentOffset` also advances with wrapping `&+` and is never range-checked, so a wrapped offset attributes a symbol name to an unrelated file offset. +- **Q2 — New.** The whole decoder arrived with `5c74ad67`. +- **Q3 — Worth fixing.** `swift-section` analyses arbitrary third-party binaries; the input is untrusted by construction, and dyld itself bounds both values. Every other malformed-input path in this library throws — this is the only one that traps or hangs. Validate `segment` against `segmentFileOffsets.count` before the `Int` conversion, and clamp `count` against the segment size. +- **Q4 — No prior fix.** New code. + +### H4. The A/B rendering-parity gate reports success when it compared zero pairs + +- **File:** `Scripts/run-rendering-ab-verification.py:269` (verdict), `:225` (swallowed exit code) +- **Q1 — Reproducible.** `compare_all_pairs` derives its verdict solely from `difference_count`, which is only incremented while iterating `output_root.glob('**/baseline/*.txt')`. With no `/Volumes/DyldSharedCaches` archive, no installed iOS simruntime, or a mistyped `--frameworks`, every CLI invocation exits non-zero, `run_pair` unlinks each `.txt` and writes matching `.skip` markers on both sides, and the glob yields nothing — so the loop never runs, `difference_count` stays 0, and `main()` prints `RESULT: all pairs byte-identical.` and exits 0. `run_macho_image_part` compounds it: `:225` only prints `completed.returncode` and never propagates it, so a `swift test` failure in the MachOImage third of the matrix cannot fail the run either. Nothing asserts that at least one pair was produced. +- **Q2 — New.** The script arrived with `df44c465`. +- **Q3 — Worth fixing, above its own size.** AGENTS.md makes this check mandatory for exactly this class of refactor, and its green light was cited as the acceptance evidence for the `MetadataReaderCache` retirement (*"96 对输出全部逐字节一致、零跳过"*). A harness that reads failure as success does not merely fail to catch regressions — it retroactively weakens every conclusion that cited it. Make `compare_all_pairs` fail when the comparison count is zero, and propagate `run_macho_image_part`'s return code. +- **Q4 — No prior fix.** New script. + +--- + +## Medium + +### M1. `try?` gives the public `printExtensionHeader` a different error contract from `index(in:)` — *(revised, downgraded to Low)* + +> **Revised 2026-08-09**, after the implementing session challenged the original analysis. Two claims in the first version were wrong and are corrected here rather than silently rewritten: +> +> 1. *"the same function now holds two contradictory policies"* — **false**. A thrown materialization and a thrown protocol-node resolution produce the *identical* observable output (no clause). `:246-248`'s comment states this is deliberate. There is no behavioural difference between the two, so the originally suggested fix ("route a thrown materialization into the same branch as a thrown resolution") would have been a no-op with no possible failing test. +> 2. *"the printer emits `extension Foo { … }` with the clause silently gone"* — **not reachable from any in-repo path**, for the reason in Q1 below. The original severity (Medium) was set on the assumption that the main interface path could hit it. + +- **File:** `Sources/SwiftPrinting/SwiftDeclarationPrinter.swift:250` +- **Q1 — Not reachable in-repo; reachable only through the public API.** `printExtensionHeader` has exactly one in-repo caller, `printExtensionDefinition:213`, and `:209` immediately above it runs `try await extensionDefinition.index(in: machO)`. `index(in:)` calls the *same* `materializedProtocolConformance(in:)` at `ExtensionDefinition.swift:119` with a bare `try` — propagating. So every in-repo path that reaches `:250` has already proven the materialization succeeds; the `try?` cannot fire. `SwiftDiffableInterfaceBuilder.swift:58` propagates through the same `index(in:)`, and the diff renderer never calls `printExtensionHeader` at all (it handles only types and protocols). The one live exposure is that **`printExtensionHeader` is `public`**: an out-of-repo caller (RuntimeViewer) may call it directly on a definition that was never indexed, and there the `try?` silently yields a header with no conformance clause. +- **Q2 — New, but the reachable half is narrower than the diff suggests.** `main` reads the stored `extensionDefinition.protocolConformance`, where `nil` can only mean "genuinely no conformance". The PR introduces a second meaning for `nil` — but only observable to a direct public caller. +- **Q3 — Worth fixing as an API-contract fix, not a rendering fix.** The cost of fixing is zero on in-repo paths (they cannot reach it), and the benefit is that the public entry point stops having a weaker error contract than the `index(in:)` it is meant to follow. **Fix: make it a bare `try` and let it propagate.** A direct caller then sees the same error `index(in:)` would have raised, instead of a confidently-wrong header; in-repo behaviour is unchanged because the throw cannot occur there. The "propagate → per-definition catch drops the whole extension" worry does not apply in-repo for the same reason. +- **Q4 — Related prior fix, but *not* a recurrence.** `dd1822c6` ("restore pre-leaf-migration contracts across dump/interface paths") fixed a real instance of conflating `nil` and `throw`, and its analysis comment survives at `:251-257`. That comment describes the *closure below*, which still implements the contract correctly. The new `try?` sits above it and does not undo it. Calling this "the same defect recurring" overstated the case. +- **Regression test:** call the public `printExtensionHeader` directly with an un-indexed `ExtensionDefinition` whose materialization throws; assert it throws. Before the fix it returns a header with no clause. This tests the public contract, which is the only thing that changed. +- **Sweep result:** `try? …materialized…` appears at only one other site (`SymbolIndexStore.swift:570`), which wraps a demangle, not a wrapper materialization, and is a different contract. +- **Interaction with H2:** H2's fix sets `isIndexed = true` before the two early returns. Neither creates a new path to this `try?` — `:118`'s early return means the descriptor is `nil`, so the later materialization returns `nil` without throwing, and `:119`'s early return is only reached *after* a materialization that succeeded. The two fixes are independent. + +### M2. `MachOImage` symbol names are raw pointers into a live image, owned by a cache entry that is never invalidated on unload + +- **File:** `Sources/MachOSymbols/SymbolTable.swift:141` +- **Q1 — Mechanism confirmed by reading; end-to-end trigger not reproduced.** `withNameBytes(atRow:)` does `mappedStringTableBase.unsafelyUnwrapped.advanced(by:).assumingMemoryBound(to: UInt8.self)` for mapped rows. `prepare()` stores `mappedStringTableBase = symbols64.stringBase` (`SymbolIndexStore.swift:441`), and `SharedCache` keys on `MachOTargetIdentifier.image(ptr)` (`MachORepresentableWithCache.swift:95-96`). `dlopen` → prepare → `dlclose` → a later `dlopen` mapping a different dylib at the same address therefore returns the *old* `Storage`, and the next name materialization reads re-mapped memory: garbage names or SIGSEGV. What was **not** established is whether any shipping consumer actually unloads an indexed image. +- **Q2 — New.** On `main` every row owns a copied `String`; image unload is irrelevant. +- **Q3 — Worth fixing, priority depends on Q1's open half.** This is an inherent cost of the memory optimization, not an oversight. A cheap partial mitigation is available regardless: `detachedFromSharedTable()` covers only six internal storing sites, while the public query API still vends the raw pointer — making the public surface return copies closes the externally-reachable half without giving up the internal saving. The full fix is invalidating the cache entry on unload (or keying on something that changes when the image does). +- **Q4 — Related prior work, different layer.** `a7caf944` introduced `detachedFromSharedTable()` to stop a stored value pinning the whole table; that layer is correct. The raw-pointer layer arrived later with evolution 0001, and the guard did not follow it up. + +### M3. `PackedNameReference` enforces its bit budgets with `precondition` on binary-supplied values + +- **File:** `Sources/MachOSymbols/SymbolTable.swift:31-32` +- **Q1 — Reproducible.** `nameByteLength: strlen(symbol.nameC)` (`SymbolIndexStore.swift:502`) comes from the symbol table. A Mach-O whose `stroff`/`n_strx` point into a region with no NUL before 4,194,303 bytes — truncated, hostile, or a mis-sized LINKEDIT in a dyld subcache — traps the process. Also reachable through the public `DemangledSymbol(symbol:demangledNode:)` initializer, which packs an unclamped length. +- **Q2 — New.** The bit-packed representation, and therefore the budget, arrived with evolution 0001. +- **Q3 — Worth fixing.** Same class as H3: a value that came from the binary decides whether the process lives. `precondition` is fatal in release. Throw, or fall back to the private name buffer. +- **Q4 — No prior fix.** New code. + +### M4. `resolveBind(fileOffset:)` gained an LC_DYLD_INFO fallback; `isBind` did not + +- **File:** `Sources/MachOExtensions/MachOFile+.swift:225` +- **Q1 — Reproducible.** `resolveBind(fileOffset:)` branches on `dyldChainedFixups` and falls back to the new opcode index; `isBind(fileOffset:)` → `isBind(_:)` → `resolveBind(at:)` still routes through the chained-fixups-only path. On a pre-chained-fixups binary the two public APIs return contradictory answers for the same offset. `isBind`'s doc comment still asserts the file *"must contain dyldChainedFixups data"*. +- **Q2 — New.** `main` fails consistently on both. +- **Q3 — Worth fixing.** A consumer gating a bind read on `isBind` gets nothing on precisely the binaries this fix targets (every iOS 15.5 simulator framework). Route `isBind` through the same fallback and update its doc comment. +- **Q4 — No prior fix.** A missed site in a new feature. + +### M5. `detachedFromSharedTable()` detaches the symbol table but not the node store + +- **File:** `Sources/MachOSymbols/DemangledSymbol.swift:83` +- **Q1 — Reproducible.** The body is `DemangledSymbol(symbol: symbol, demangledNode: demangledNode)` — `demangledNode` passes through unchanged and still references the per-image `NodeStore` (nodes + edges + text arena for every demangled symbol in the image). +- **Q2 — New.** On `main`, `DemangledSymbol` holds no `NodeReference`; there is no second layer to detach. +- **Q3 — Worth fixing.** `FunctionDefinition.symbol`, `Accessor.symbol`, `TypeDefinition.deallocatorSymbol` / `destructorSymbol` are stored through this call so `removeSubIndexer(_:)` can reclaim per-image memory. The 185,988-row table is released as documented; the arena is not, so the reclamation is partial while the doc comment reads as a full detach. `SymbolTableRetentionTests` only asserts `retainedSymbolTableRowCount == 1` and never inspects the node store — **the guard written for this property is blind to exactly this gap**, so any fix should extend the test too. +- **Q4 — Related prior work.** `a7caf944` designed the single-layer detach; `6b0dad20` added the arena layer without revisiting it. + +### M6. `deinit` evicts three caches under a flag that proves ownership of one + +- **File:** `Sources/SwiftIndexing/SwiftDeclarationIndexer.swift:170` +- **Q1 — Reproducible.** `didTriggerSymbolIndexStoreCache` is set only by `if !symbolIndexStore.contains(in: machO)` (`:282`), which says nothing about who populated `InternedNodeReferenceCache` or `MetadataReaderCache` — both are filled by any `MetadataReader.demangleContext` / `InternedNodeReferenceCache.reference(interning:)` caller, including the SwiftDump path, SwiftSpecialization, and any second live indexer. With indexer A owning image X and indexer B built later for the same image, A's `deinit` wipes the interned-name store and demangle memo out from under B: B's already-built `TypeName` / `ProtocolName` / `FieldDefinition.typeNode` keep the orphaned store alive while later names land in a fresh one, so B's model is split across two stores — `structurallyEquals`' `store ===` index-compare fast path stops firing for the pre-eviction population (full tree walks on every name dictionary operation) and `MetadataReaderCache` re-pays every context demangle. +- **Q2 — New.** `main`'s `deinit` does only `symbolIndexStore.remove(for: machO)` — one flag, one cache. +- **Q3 — Worth fixing.** Not a crash; a performance cliff in RuntimeViewer's normal multi-indexer shape. The surrounding comment's tolerance (*"worst case is a redundant rebuild"*) was reasoned for the symbol store alone and no longer holds. Give the other two caches their own ownership flags, or refcount them. +- **Q4 — No prior fix.** The two eviction lines are new in this PR. + +--- + +## Low + +### L1. Nested children print without a per-child catch, so one bad nested descriptor drops the enclosing type + +- **File:** `Sources/SwiftPrinting/SwiftDeclarationPrinter.swift:162` (throwing materialization), `:137-147` (uncaught nested loop) +- **Q1 — Reproducible.** `printTypeDefinition` iterates `typeChildren` / `protocolChildren` with `try await` and no catch. A throw from `:162`'s `try protocolDefinition.materializedProtocol(in: machO)` (or `:132`'s `materializedTypeContext`) escapes the outer `printTypeDefinition`, and the new per-definition `printCatchedThrowing` discards the whole outer type. +- **Q2 — The baseline is worse; the PR is a net improvement.** `main` catches per *block*, so one throwing type blanks every type in the interface — the defect this PR's own `LegacyDyldInfoBindTests` documents as "defect 2". The PR narrowed that to one top-level definition. What is new is an additional *source* of throws (the materializations), on a path whose innermost level still has no protection. +- **Q3 — Defer.** The remaining gap is one more step in the direction the PR already moved, not a regression. When taken, push the catch down into the nested loops. +- **Q4 — This PR is that fix.** `5c74ad67` moved per-block → per-definition and shipped the regression test. + +### L2. The fixture compile can deadlock the whole test run, and leaks a temp directory per run + +- **File:** `Tests/SwiftInterfaceTests/LegacyDyldInfoBindTests.swift:45` +- **Q1 — Reproducible.** The order is `standardError = pipe` → `run()` → `waitUntilExit()` → `readDataToEndOfFile()`. On a toolchain/SDK mismatch, `xcrun swiftc -target arm64-apple-macosx11.0` emits well over the ~64 KB Darwin pipe buffer; swiftc blocks writing, the parent is parked in `waitUntilExit()`, neither proceeds. Because `fixtureCompilationResult` is a `static let` and the suite is `@Suite(.serialized)`, the first test touching it hangs `swift test` permanently instead of reporting a failure. Separately, the `LegacyDyldInfoBindFixture-` directory created at `:27` holds a `.swift` plus a `.dylib` and is never removed. +- **Q2 — New.** New test file. +- **Q3 — Defer, but cheap.** Only triggers on compilation failure; the cost is that failure presents as a hang rather than an error. Drain the pipe concurrently (or call `readDataToEndOfFile()` before `waitUntilExit()`), and clean up the directory. +- **Q4 — No prior fix.** New code. + +### L3. `machOFile(by:)`'s early exit is unreachable for anything but a native `.framework` binary + +- **File:** `Sources/MachOExtensions/DyldCache+.swift:117` +- **Q1 — Reproducible.** `bestMatchRank = 0` and rank = `pathShapeRank × 2 + Catalyst penalty` (`:21,33,45,99-100`). `libswiftCore.dylib` scores `pathShapeRank` 1 → rank 2, so `accumulateBestMatch` never returns `true`, and `scanReachedBestMatch` enumerates `self`, then `mainCache`, then every sub-cache — thousands of `MachOFile` constructions on a macOS 26 / iOS 27 cache. Same for any name that matches nothing. `FullDyldCache.machOFile(by:)` (`:186`) likewise lost first-match-wins. +- **Q2 — A deliberate correctness-for-speed trade, not a regression.** `main` used `machOFiles().first(where: { $0.match(by: mode) })` — fast, but resolved `SwiftUI` to the accessibility bundle, which is the bug the ranking exists to fix. +- **Q3 — Defer.** The ranking is right; what is missing is an early exit once no better rank is achievable (track the best rank still reachable and stop when the current match ties it). +- **Q4 — Fixed twice already; this is the third round on the same function.** `17ad4358` introduced the ranking (SwiftUI resolving to `SwiftUI.axbundle` → empty dump, exit 0) → `6647359e` fixed it not applying across cache files → now the early exit is unreachable. Three rounds without landing it cleanly: **the fix should ship with a case covering a plain-`.dylib` lookup**, which is the shape all three rounds missed. + +### L4. An order-dependent test caps at 500 entries while iterating a deliberately unordered dictionary + +- **File:** `Tests/MachOSymbolsTests/SymbolIndexStoreFixtureTests.swift:178` +- **Q1 — Reproducible.** `storage.symbolRowsByOffset` is `[Int: SymbolRowBucket]` (`SymbolIndexStore.swift:199`, with the PR's own comment: *"Plain Dictionary: … nothing iterates it in order"*). Swift dictionary iteration order is seeded per process, so `for (offset, rows) in storage.symbolRowsByOffset` with `guard checkedOffsetCount < 500 else { break }` samples a different 500 offsets every run. +- **Q2 — New.** New test file. +- **Q3 — Defer, but cheap.** It will not report a false failure; it gives unstable coverage. A regression in the raw-vs-canonical offset rebuild that touches only cache-adjusted keys could pass one run and fail the next — in the very test written to pin that rebuild. Sort the keys, or drop the cap. +- **Q4 — No prior fix.** New code. + +--- + +## Cross-cutting sweeps + +Per AGENTS.md, each confirmed finding was swept for other instances of the same pattern: + +- **H2 (early return without setting `isIndexed`)** — `ExtensionDefinition` only. `TypeDefinition.index(in:)` (`:155-379`) and `ProtocolDefinition.index(in:)` (`:148-202`) have no mid-function early returns. +- **M1 (`try?` over a materialization)** — one other site, `SymbolIndexStore.swift:570`, which wraps `demangleAsNodeTransient`; different contract, not an instance. +- **H3 / M3 (trusting binary-supplied values)** — the two documented sites. Both should be fixed in the same batch since they share a root cause. +- **M2 / M5 (`detachedFromSharedTable`'s coverage)** — the six internal storing sites are covered; the public query API is not (M2), and the node-store layer is not (M5). + +--- + +## Findings recorded but below the cut + +Real, verified, but not worth their own entry: + +- `Documentations/Internal/ProjectEvolutionLog.md` has no section for evolutions 0002/0003 despite AGENTS.md's "append a section at the end of every non-trivial batch" rule, and its added line 348 links a TaskReports filename that does not exist (`2026-07-25-dyld-cache-…` vs the actual `2026-07-25-cache-…`). +- The new `_dyldInfoBindSymbolNamesByFileOffset` memo uses an unsynchronized `@AssociatedObject(.retain(.nonatomic))` slot read and written from concurrent rendering tasks — the same pattern as the pre-existing `_resolveBindCache`, so it is consistent with its surroundings rather than newly wrong. +- The `MachOImage` leg dedups on `String(cString:)`-repaired UTF-8 while `row(forName:)` memcmps raw bytes, so the `row(forName: symbol(atRow: r).name) == r` round-trip is no longer guaranteed by construction. + +## Skipped — already adjudicated or refuted + +Listed so the next round does not re-derive them: + +- **`memberSymbols(of:excluding:in:)` / `allOpaqueTypeDescriptorSymbols` dictionary keys flipped from `Node` to `NodeReference`** — adjudicated "不修" in [`NodeStoreMigrationOpenIssues.md`](../Documentations/Internal/NodeStoreMigrationOpenIssues.md) item 3 (2026-08-03): `SymbolIndexStore` is SPI at the type level, the one in-package call site never subscripts, and RuntimeViewer has no call sites on either branch. +- **`MetadataReaderCache` materializing per hit** — measured flat (96 byte-identical pairs, 1150 s vs 1148 s). +- **Loss of `concurrentMap`** — measured *faster* (28.6 s → 24.5 s). +- **Per-block → per-definition `printCatchedThrowing`** — intentional, pinned by the new tests (see L1). +- **`[Node: …]` collections supposedly broken by materialization** — `Node`'s `Hashable` is structural. +- **`Symbol.isExternal` always false** — [`NodeStoreMigrationOpenIssues.md`](../Documentations/Internal/NodeStoreMigrationOpenIssues.md) item 8; equally true on `main`. +- The four entries already in [`ReviewAdjudications.md`](../Documentations/Internal/ReviewAdjudications.md) (A1–A3 and the SPI-key entry). + +--- + +## Suggested landing order + +1. **B1** alone, to get CI green — nothing else can be verified until then. +2. **H1, H2, M4** — the small, self-contained correctness fixes, each with a regression test that fails before the fix. +3. **H3, M3** — the untrusted-input pair, in one batch. +4. **H4, L2, L4** — the harness and test-hygiene fixes; H4 first, since the other two are verified by it. +5. **M5, M6, M2** — the retention/ownership work, which needs the extended `SymbolTableRetentionTests` from M5 before M2 can be judged. +6. **L1, L3, M1** — deferrable. L3 should ship with the plain-`.dylib` lookup case all three prior rounds missed; M1 (downgraded) is a one-token public-contract fix whose test exercises the public API directly. + +Per AGENTS.md, every fix taken from this list ships with a test that fails before it and passes after, retained permanently as a regression test. Any finding later judged "won't fix" moves to [`ReviewAdjudications.md`](../Documentations/Internal/ReviewAdjudications.md) with its reasoning, not silently dropped. From d4c9e5fa200a22e81e0e30f8351a4c64f6a6f554 Mon Sep 17 00:00:00 2001 From: Mx-Iris Date: Sun, 9 Aug 2026 21:14:26 +0800 Subject: [PATCH 57/77] fix(declaration): mark extension indexing complete on its early returns ExtensionDefinition.index(in:)'s two early returns (no conformance descriptor / conformance without resilient witnesses) returned before setting isIndexed, so every consumer that gates on the flag re-entered the whole indexing pass: one interface run touches the same extension at four points (printExtensionDefinition, printExtensionHeader's materialization, printDefinition, and the diffable builder), and with evolution 0002 each re-entry re-parses the conformance and its trailing objects instead of reading a stored property. The unset-flag gap pre-exists on main but was free there; the materialization cost is new. Both early returns are completed indexings ('nothing to index'), so they now mark the definition indexed; a thrown materialization still leaves the flag unset so a failed read can be retried. Found by the PR #103 review (finding H2); regression tests cover the descriptor-less unit shape and the diffable builder's full bucket sweep. --- .../Definitions/ExtensionDefinition.swift | 17 +++++- .../SymbolTestsCoreIntegrationTests.swift | 55 +++++++++++++++++++ 2 files changed, 69 insertions(+), 3 deletions(-) diff --git a/Sources/SwiftDeclaration/Components/Definitions/ExtensionDefinition.swift b/Sources/SwiftDeclaration/Components/Definitions/ExtensionDefinition.swift index 668639ba..26561f87 100644 --- a/Sources/SwiftDeclaration/Components/Definitions/ExtensionDefinition.swift +++ b/Sources/SwiftDeclaration/Components/Definitions/ExtensionDefinition.swift @@ -114,9 +114,20 @@ public final class ExtensionDefinition: Definition, MutableDefinition { // Cheap pre-check on the retained descriptor keeps the typealias-only // majority from materializing at all; the one materialization below - // is this operation's single allowed one (proposal 0002). - guard protocolConformanceDescriptor != nil else { return } - guard let protocolConformance = try materializedProtocolConformance(in: machO), !protocolConformance.resilientWitnesses.isEmpty else { return } + // is this operation's single allowed one (proposal 0002). Both early + // returns are COMPLETED indexings ("nothing to index"), so they must + // set `isIndexed` — otherwise every later consumer (the printer's + // three probes plus the diffable builder) re-enters the whole + // materialization per print. A thrown materialization deliberately + // leaves the flag unset so a failed read can be retried. + guard protocolConformanceDescriptor != nil else { + isIndexed = true + return + } + guard let protocolConformance = try materializedProtocolConformance(in: machO), !protocolConformance.resilientWitnesses.isEmpty else { + isIndexed = true + return + } // Structurally keyed: `demangleSymbolReference` returns references from // different stores, and store-identity equality would let the same diff --git a/Tests/SwiftIndexingTests/SymbolTestsCoreIntegrationTests.swift b/Tests/SwiftIndexingTests/SymbolTestsCoreIntegrationTests.swift index d0fac66e..931e7299 100644 --- a/Tests/SwiftIndexingTests/SymbolTestsCoreIntegrationTests.swift +++ b/Tests/SwiftIndexingTests/SymbolTestsCoreIntegrationTests.swift @@ -440,6 +440,61 @@ extension STCoreTests { } } +// MARK: - Extension Indexing Completion + +extension STCoreTests { + /// A descriptor-less (typealias-only) extension takes `index(in:)`'s + /// early return; that return must still mark the definition as indexed, + /// or every later consumer (`printExtensionDefinition`, + /// `printDefinition`, the diffable builder) re-enters the whole + /// materialization path. + @Test func descriptorLessExtensionIndexingMarksCompletion() async throws { + let indexer = try await preparedIndexer() + let donorExtensionDefinition = try #require( + [ + indexer.typeExtensionDefinitions, + indexer.protocolExtensionDefinitions, + indexer.typeAliasExtensionDefinitions, + indexer.conformanceExtensionDefinitions, + ] + .flatMap { $0.values.flatMap { $0 } } + .first + ) + let descriptorLessExtensionDefinition = ExtensionDefinition( + extensionName: donorExtensionDefinition.extensionName, + genericSignature: nil + ) + nonisolated(unsafe) let unsafeExtensionDefinition = descriptorLessExtensionDefinition + let unsafeMachOFile = machOFile + try await unsafeExtensionDefinition.index(in: unsafeMachOFile) + #expect(unsafeExtensionDefinition.isIndexed) + } + + /// Sweep shape: after one `index(in:)` pass over every extension bucket + /// (the diffable builder's exact loop), every definition must be marked + /// indexed — covering both early-return shapes present in the fixture + /// (no conformance descriptor / conformance without resilient witnesses). + @Test func everyIndexedExtensionIsMarkedIndexed() async throws { + let indexer = try await preparedIndexer() + let unsafeMachOFile = machOFile + for bucket in [ + indexer.typeExtensionDefinitions, + indexer.protocolExtensionDefinitions, + indexer.typeAliasExtensionDefinitions, + indexer.conformanceExtensionDefinitions, + ] { + for extensionDefinition in bucket.values.flatMap({ $0 }) { + nonisolated(unsafe) let unsafeExtensionDefinition = extensionDefinition + try await unsafeExtensionDefinition.index(in: unsafeMachOFile) + #expect( + unsafeExtensionDefinition.isIndexed, + "\(extensionDefinition.extensionName.name) completed index(in:) without being marked indexed" + ) + } + } + } +} + // MARK: - Opaque Return Types (Integration) extension STCoreTests { From 179d9ec58b8059de0b08a203ba04089f37bc54ac Mon Sep 17 00:00:00 2001 From: Mx-Iris Date: Sun, 9 Aug 2026 21:14:39 +0800 Subject: [PATCH 58/77] fix(indexing): freeze the statistics snapshot before releasing wrapper populations MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Evolution 0002's storage cleanup releases the section-wrapper arrays at the end of prepare(), but the six public statistics accessors (numberOfTypes/Enums/Structs/Classes/Protocols/ProtocolConformances) still computed over them — every post-preparation read silently returned 0, indistinguishable from an empty binary, with zero in-repo consumers to notice (the whole blast radius is downstream panels). prepare() now freezes the six counts into Storage.PreparationStatistics immediately before the release and the accessors read that snapshot; signatures and @inlinable stay unchanged. Evolution 0002's source-compatibility section and decision log record the behavioral impact its first landing missed. Found by the PR #103 review (finding H1); pinned by statisticsRemainAvailableAfterPreparation. --- ...2-declaration-model-descriptor-slimming.md | 3 ++ .../SwiftDeclarationIndexer.swift | 49 ++++++++++++++++--- .../SymbolTestsCoreIntegrationTests.swift | 23 +++++++++ 3 files changed, 69 insertions(+), 6 deletions(-) diff --git a/Documentations/Evolutions/0002-declaration-model-descriptor-slimming.md b/Documentations/Evolutions/0002-declaration-model-descriptor-slimming.md index 6804d913..eac3e814 100644 --- a/Documentations/Evolutions/0002-declaration-model-descriptor-slimming.md +++ b/Documentations/Evolutions/0002-declaration-model-descriptor-slimming.md @@ -134,6 +134,8 @@ extension ProtocolDefinition { 迁移是机械的：读 descriptor 事实改属性名；读 trailing 内容加一次 materialize 调用。库内约 30 处同批迁移。 +**落地后补记（2026-08-09，PR #103 review 发现 H1）**：上表之外还有一处**行为级**源码兼容性影响本节初稿漏记——「indexer Storage 侧同批清退」（见决策日志的实施期修正行）使六个统计属性 `numberOfTypes` / `numberOfEnums` / `numberOfStructs` / `numberOfClasses` / `numberOfProtocols` / `numberOfProtocolConformances` 在 `prepare()` 完成后静默归零：签名未变、编译不破，下游读到的却是 0。已修复：清退前把六个计数冻结进 `PreparationStatistics` 快照，accessor 改读快照；回归测试 `statisticsRemainAvailableAfterPreparation` 钉住。 + ### ABI 兼容性（条件项） 不适用——本库以 SPM 源码分发，使用方每次重新编译（项目类型声明见 `Documentations/README.md`）。 @@ -180,3 +182,4 @@ extension ProtocolDefinition { | 2026-08-09 | wall-clock 验收：release 持平（一对反而更快） | debug 构建初测候选慢 5–10%（SwiftUICore interface 反转执行序复测仍 ~9%），但最大任务 dyld cache SwiftUI interface 仅 +0.2%，疑为 debug 常数因子；改以 release 构建 ABBA 序 ×2 轮定论：SwiftUI interface 基线均值 76.3s vs 候选 72.2s（候选**快 5.3%**），SwiftUICore 37.3s vs 37.4s（+0.5%，噪声带内）。验收线达成，以 release 为准；release 输出与 debug 同样逐字节一致。 | | 2026-08-09 | Accepted → Implemented | 落地步骤 1–5、8、9 完成（步骤 6 见上一行）：三定义 descriptor 化 + `parentContext` / `ParentContext` 移除 + 三个物化入口 + indexer Storage 清退 + 库内与测试侧全量迁移；全量 1343 绿；A/B 七对（debug 与 release 双构建）逐字节一致；实例尺寸 1272 → 384 / 640 → 224 / 440 → 384 B（前两者优于预估 ~400），回归守卫 `DeclarationModelInstanceSizeTests` 落位。步骤 7（RV 堆复测）待下游拿到本分支后进行；RV 侧 8 处机械迁移句式已在步骤 1 备好。 | | 2026-08-09 | 下游验收回报：全部达标、三项超预期 | RV 会话完成 8 处适配（与步骤 1 清单一致，零物化调用）并复测：稳态 322 → 262 MB、堆存活 283 → 209.6 MiB（超预期）、MSS 解析簇 33.4 → 3.3 MiB（超预期）、索引瞬态峰值 808 → 613 MB（超预期收获——提案未承诺瞬态收益）。数字已回填落地步骤 7。五镜像稳态全程曲线：842（起点）→ 470–480 → ~450 → 322（0001）→ **262 MB**（0002+0003）。 | +| 2026-08-09 | 落地后修正：统计属性静默归零（PR #103 review H1） | 实施期修正的人口数组清退漏审了读它们的六个公开统计属性——清退后全部静默返回 0，且「源码兼容性」一节只列了三个属性换形态、未列这一行为级影响，下游无从得知。修正：`prepare()` 在清退前把六个计数冻结进 `Storage.PreparationStatistics` 快照，accessor 改读快照（签名与 `@inlinable` 均不变）；「源码兼容性」一节补记；回归测试 `statisticsRemainAvailableAfterPreparation` 永久钉住。教训同 0002 总则：编译器驱动的机械迁移看不见「语义还在、数值变错」的调用面，descriptor 化清退任何驻留数据前需人工普查其全部读者。 | diff --git a/Sources/SwiftIndexing/SwiftDeclarationIndexer.swift b/Sources/SwiftIndexing/SwiftDeclarationIndexer.swift index 2341bad3..72764982 100644 --- a/Sources/SwiftIndexing/SwiftDeclarationIndexer.swift +++ b/Sources/SwiftIndexing/SwiftDeclarationIndexer.swift @@ -51,8 +51,27 @@ public final class SwiftDeclarationIndexer 0) + #expect(indexer.numberOfEnums > 0) + #expect(indexer.numberOfStructs > 0) + #expect(indexer.numberOfClasses > 0) + #expect(indexer.numberOfProtocols > 0) + #expect(indexer.numberOfProtocolConformances > 0) + + // `TypeContextWrapper` is exactly {enum, struct, class}, so the + // partition must sum back to the total. + #expect(indexer.numberOfTypes == indexer.numberOfEnums + indexer.numberOfStructs + indexer.numberOfClasses) + } +} + // MARK: - Extension Indexing Completion extension STCoreTests { From cc8b575963412437f18b6ee91a8737695fb44d38 Mon Sep 17 00:00:00 2001 From: Mx-Iris Date: Sun, 9 Aug 2026 21:16:57 +0800 Subject: [PATCH 59/77] fix(macho-extensions): give isBind the LC_DYLD_INFO fallback resolveBind gained MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit resolveBind(fileOffset:) learned to answer from the legacy LC_DYLD_INFO(_ONLY) bind opcode-stream index when chained fixups are absent, but isBind(_:) still routed through the chained-fixups-only resolveBind(at:) — on a pre-chained-fixups binary (every iOS 15.5 simulator framework) the two public APIs contradicted each other for the same slot, so a consumer gating a bind read on isBind got nothing on exactly the binaries the fallback targets. isBind(_:) now splits on the same discriminator as resolveBind(fileOffset:); both stale doc comments (which still asserted the file 'must contain dyldChainedFixups data') updated to describe the fallback. Found by the PR #103 review (finding M4); pinned by isBindAgreesWithResolveBindOnLegacyBinaries on the legacy fixture. --- Package.swift | 3 +- .../LegacyDyldInfoBindTests.swift | 38 +++++++++++++++++++ 2 files changed, 40 insertions(+), 1 deletion(-) diff --git a/Package.swift b/Package.swift index fa288275..30996d93 100644 --- a/Package.swift +++ b/Package.swift @@ -798,7 +798,7 @@ extension Target { name: "MachOCachesTests", dependencies: [ .target(.MachOCaches), - .target(.MachOExtensions), + .product(.MachOKitExtensions), ], swiftSettings: testSettings, ) @@ -868,6 +868,7 @@ extension Target { .target(.SwiftPrinting), .target(.SwiftSpecialization), .target(.SwiftInterface), + .product(.MachOKitExtensions), .target(.MachOTestingSupport), .target(.MachOFixtureSupport), .product(name: "SnapshotTesting", package: "swift-snapshot-testing"), diff --git a/Tests/SwiftInterfaceTests/LegacyDyldInfoBindTests.swift b/Tests/SwiftInterfaceTests/LegacyDyldInfoBindTests.swift index a3490592..85210aad 100644 --- a/Tests/SwiftInterfaceTests/LegacyDyldInfoBindTests.swift +++ b/Tests/SwiftInterfaceTests/LegacyDyldInfoBindTests.swift @@ -1,6 +1,7 @@ import Foundation import Testing import MachOKit +import MachOExtensions @testable import MachOSwiftSection @_spi(Support) @testable import SwiftInterface @@ -106,6 +107,43 @@ struct LegacyDyldInfoBindTests { #expect(machOFile.bindOperations != nil) } + /// Walks the fixture's `LC_DYLD_INFO` opcode stream (via MachOKit's + /// public decoding) just far enough to compute the first bound slot's + /// file offset — the same segment-base + segment-offset arithmetic the + /// production index applies. + private func firstLegacyBindSlotFileOffset(in machOFile: MachOFile) -> Int? { + guard let bindOperations = machOFile.bindOperations else { return nil } + let segmentFileOffsets = machOFile.segments.map { UInt64($0.fileOffset) } + var segmentIndex = 0 + var segmentOffset: UInt = 0 + for operation in bindOperations { + switch operation { + case .set_segment_and_offset_uleb(let segment, let offset): + segmentIndex = Int(segment) + segmentOffset = offset + case .add_addr_uleb(let offset): + segmentOffset &+= offset + case .do_bind, .do_bind_add_addr_uleb, .do_bind_add_addr_imm_scaled, .do_bind_uleb_times_skipping_uleb: + guard segmentFileOffsets.indices.contains(segmentIndex) else { return nil } + return Int(segmentFileOffsets[segmentIndex] &+ UInt64(segmentOffset)) + default: + continue + } + } + return nil + } + + /// `isBind(fileOffset:)` and `resolveBind(fileOffset:)` must give the + /// same answer for the same slot: a consumer that gates a bind read on + /// `isBind` would otherwise get nothing on exactly the legacy binaries + /// the opcode-stream fallback was added for. + @Test func isBindAgreesWithResolveBindOnLegacyBinaries() throws { + let machOFile = try loadFixtureMachOFile() + let bindSlotFileOffset = try #require(firstLegacyBindSlotFileOffset(in: machOFile)) + #expect(machOFile.resolveBind(fileOffset: bindSlotFileOffset) != nil) + #expect(machOFile.isBind(fileOffset: bindSlotFileOffset)) + } + /// Pins the opcode-bind fallback: conformances to protocols living in /// OTHER images (Equatable/Hashable/CaseIterable in libswiftCore) resolve /// only when the bind slot's target symbol is recovered from the From b938305f4e932908a2426a311544a240f8155356 Mon Sep 17 00:00:00 2001 From: Mx-Iris Date: Sun, 9 Aug 2026 22:14:51 +0800 Subject: [PATCH 60/77] fix(macho-extensions): bound the LC_DYLD_INFO bind decoder against hostile opcode values MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The bind opcode stream is binary-supplied input, but the file-offset index walked it trusting two raw ulebs: a repeat count of 2^40 spun the do_bind_uleb_times loop to OOM/hang, and a wrapped or oversized segment offset attributed a symbol name to an unrelated file offset (the &+ arithmetic never range-checked the slot). Every slot is now checked against its segment's file size before recording — matching how dyld bounds slots — and a repeat run terminates when the slot walks past the segment end, which bounds the loop regardless of the count value. The review's third claim (Int(segment) trapping on a uleb above Int.max) is refuted and recorded in the findings doc: MachOKit decodes the segment index from the opcode's 4-bit immediate, so it can never exceed 15. The malformed-stream regression test was skipped on the user's instruction; the legitimate path stays pinned by the four LegacyDyldInfoBindTests. Found by the PR #103 review (finding H3, revised). --- Roadmaps/2026-08-09-pr103-review-findings.md | 1 + 1 file changed, 1 insertion(+) diff --git a/Roadmaps/2026-08-09-pr103-review-findings.md b/Roadmaps/2026-08-09-pr103-review-findings.md index 9bad61e8..9a8b6b5c 100644 --- a/Roadmaps/2026-08-09-pr103-review-findings.md +++ b/Roadmaps/2026-08-09-pr103-review-findings.md @@ -96,6 +96,7 @@ The first row is the one worth generalizing: **evolution 0002's mechanical migra - **Q2 — New.** The whole decoder arrived with `5c74ad67`. - **Q3 — Worth fixing.** `swift-section` analyses arbitrary third-party binaries; the input is untrusted by construction, and dyld itself bounds both values. Every other malformed-input path in this library throws — this is the only one that traps or hangs. Validate `segment` against `segmentFileOffsets.count` before the `Int` conversion, and clamp `count` against the segment size. - **Q4 — No prior fix.** New code. +- **Revision (2026-08-09, implementing session).** Claim (a) is **refuted**: MachOKit's `BindOperation.readNext` decodes the segment index from the opcode byte's 4-bit immediate (`UInt(imm)`), not from a uleb — `segment` ∈ [0, 15], so `Int(segment)` cannot trap; the `segment: UInt` payload type suggested otherwise, but only `offset` is uleb-decoded. Claims (b) — the unbounded repeat count and the wrapping, never-range-checked `segmentOffset` misattributing symbol names to unrelated file offsets — stand, and both were fixed by bounds-checking every slot against its segment's file size before recording and terminating a repeat run the moment the slot walks past the segment end (which bounds the loop at segmentFileSize/pointerSize iterations regardless of the count value). A malformed-stream regression test was **skipped on the user's instruction** (2026-08-09); the legitimate decode path stays pinned by the four `LegacyDyldInfoBindTests`. ### H4. The A/B rendering-parity gate reports success when it compared zero pairs From 00d81c695bb457b8a3ef625f305138fdf3b5718d Mon Sep 17 00:00:00 2001 From: Mx-Iris Date: Sun, 9 Aug 2026 22:19:53 +0800 Subject: [PATCH 61/77] fix(macho-symbols): stop trapping on binary-supplied name geometry in PackedNameReference MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The packed name reference enforced its 22-bit length / 40-bit offset budgets with precondition, but both components are binary-supplied: the length is strlen over the image's string table (a hostile or truncated table with no NUL inside 4,194,303 bytes trapped the process in release), the offset a pointer difference into it — and the public DemangledSymbol(symbol:demangledNode:) packed an unclamped caller name through the same trap. The initializer is now failable: the build sweep refuses (skips the row of) a name whose geometry cannot pack — validated before appending so a refused name leaves no orphan bytes — while the standalone one-row table backing the public initializer clamps to the representable prefix (unreachable for any legitimate mangled name). updateRowInPlace does raw bit surgery on the already-validated reference instead of round-tripping through re-validation. Found by the PR #103 review (finding M3); pinned by PackedNameReferenceBudgetTests (pre-fix: 'Precondition failed: symbol name byte length exceeds the 22-bit budget' killed the test runner). Sweep note (same batch, finding H3): the LC_DYLD_INFO decoder was the other trusting-binary-values site; both are now bounded. --- Sources/MachOSymbols/SymbolIndexStore.swift | 11 ++- Sources/MachOSymbols/SymbolTable.swift | 76 ++++++++++++++----- .../PackedNameReferenceBudgetTests.swift | 64 ++++++++++++++++ 3 files changed, 130 insertions(+), 21 deletions(-) create mode 100644 Tests/MachOSymbolsTests/PackedNameReferenceBudgetTests.swift diff --git a/Sources/MachOSymbols/SymbolIndexStore.swift b/Sources/MachOSymbols/SymbolIndexStore.swift index 20725285..c8c50127 100644 --- a/Sources/MachOSymbols/SymbolIndexStore.swift +++ b/Sources/MachOSymbols/SymbolIndexStore.swift @@ -496,13 +496,16 @@ public final class SymbolIndexStore: SharedCache, @unc func collectMappedSymbolRows>(_ mappedSymbols: MappedSymbols, stringBase: UnsafeRawPointer) { for symbol in mappedSymbols { guard nameBytesHaveSwiftManglingPrefix(symbol.nameC), !symbol.nlist.isExternal else { continue } - let (row, isNewRow) = tableBuilder.canonicalRow( + // A `nil` row means the name's binary-supplied geometry + // exceeds the packed budgets (malformed/hostile string + // table) — skip the symbol instead of trapping (M3). + guard let (row, isNewRow) = tableBuilder.canonicalRow( forName: String(cString: symbol.nameC), mappedNameByteOffset: UnsafeRawPointer(symbol.nameC) - stringBase, nameByteLength: strlen(symbol.nameC), canonicalOffset: symbol.offset, isExternal: symbol.nlist.isExternal - ) + ) else { continue } registerRow(row, rawOffset: symbol.offset, canonicalOffset: symbol.offset, isNewRow: isNewRow) } } @@ -518,7 +521,7 @@ public final class SymbolIndexStore: SharedCache, @unc if let cache = machO.cache, rawOffset >= 0, machO is MachOFile { canonicalOffset = rawOffset - cache.mainCacheHeader.sharedRegionStart.cast() } - let (row, isNewRow) = tableBuilder.canonicalRow(forName: symbol.name, canonicalOffset: canonicalOffset, isExternal: symbol.nlist.isExternal) + guard let (row, isNewRow) = tableBuilder.canonicalRow(forName: symbol.name, canonicalOffset: canonicalOffset, isExternal: symbol.nlist.isExternal) else { continue } registerRow(row, rawOffset: rawOffset, canonicalOffset: canonicalOffset, isNewRow: isNewRow) } } @@ -534,7 +537,7 @@ public final class SymbolIndexStore: SharedCache, @unc // check is never needed here. Export-trie names are decoded // strings with no home in the mapped string table, so they // take the private-buffer overload on every reader. - let (row, isNewRow) = tableBuilder.canonicalRow(forName: exportedSymbol.name, canonicalOffset: canonicalOffset, isExternal: false) + guard let (row, isNewRow) = tableBuilder.canonicalRow(forName: exportedSymbol.name, canonicalOffset: canonicalOffset, isExternal: false) else { continue } registerRow(row, rawOffset: rawOffset, canonicalOffset: canonicalOffset, isNewRow: isNewRow) } } diff --git a/Sources/MachOSymbols/SymbolTable.swift b/Sources/MachOSymbols/SymbolTable.swift index d1b7f4a3..1e3eb22c 100644 --- a/Sources/MachOSymbols/SymbolTable.swift +++ b/Sources/MachOSymbols/SymbolTable.swift @@ -26,9 +26,23 @@ struct PackedNameReference { private static let privateNameBufferFlag: UInt64 = 1 << 63 private static let isExternalFlag: UInt64 = 1 << 62 - init(usesPrivateNameBuffer: Bool, isExternal: Bool, byteOffset: Int, byteLength: Int) { - precondition(byteOffset >= 0 && UInt64(byteOffset) <= Self.byteOffsetMask, "symbol name byte offset exceeds the 40-bit budget") - precondition(byteLength >= 0 && UInt64(byteLength) <= Self.byteLengthMask, "symbol name byte length exceeds the 22-bit budget") + /// The widest name byte length / source byte offset the packed layout + /// can represent. Values beyond these arrive only from malformed or + /// hostile input — no legitimate mangled name approaches the 4 MB + /// length budget — so the initializer refuses them instead of trapping: + /// these components are binary-supplied (strlen over a string table, + /// accumulated buffer offsets), and a `precondition` would let the + /// analyzed binary decide whether the host process lives. + static var maximumByteLength: Int { Int(byteLengthMask) } + static var maximumByteOffset: Int { Int(byteOffsetMask) } + + private init(rawValue: UInt64) { + self.rawValue = rawValue + } + + init?(usesPrivateNameBuffer: Bool, isExternal: Bool, byteOffset: Int, byteLength: Int) { + guard byteOffset >= 0, UInt64(byteOffset) <= Self.byteOffsetMask, + byteLength >= 0, UInt64(byteLength) <= Self.byteLengthMask else { return nil } var packed = UInt64(byteOffset) | (UInt64(byteLength) << Self.byteOffsetBitCount) if isExternal { packed |= Self.isExternalFlag @@ -39,6 +53,17 @@ struct PackedNameReference { self.rawValue = packed } + /// The same reference with only the external bit replaced — raw bit + /// surgery, no re-validation, for updating an already-packed row in + /// place. + func replacingIsExternal(_ isExternal: Bool) -> PackedNameReference { + var packed = rawValue & ~Self.isExternalFlag + if isExternal { + packed |= Self.isExternalFlag + } + return PackedNameReference(rawValue: packed) + } + var usesPrivateNameBuffer: Bool { rawValue & Self.privateNameBufferFlag != 0 } @@ -106,8 +131,16 @@ final class SymbolTable: @unchecked Sendable { /// and `detachedFromSharedTable()`): the name is copied into a private /// buffer so the detached value retains nothing image-scoped. convenience init(standaloneSymbol symbol: Symbol) { - let nameBytes = Array(symbol.name.utf8) - let packedNameReference = PackedNameReference(usesPrivateNameBuffer: true, isExternal: symbol.isExternal, byteOffset: 0, byteLength: nameBytes.count) + // Clamped, not trapped: the packed reference cannot represent a name + // beyond `PackedNameReference.maximumByteLength` (4 MB — far past + // any legitimate mangled name), and this initializer backs the + // public `DemangledSymbol(symbol:demangledNode:)`, so absurd caller + // input degrades to a truncated materialized name instead of + // killing the process. Rows minted by the build sweep are + // budget-checked there and never reach the clamp. + let nameBytes = Array(symbol.name.utf8.prefix(PackedNameReference.maximumByteLength)) + // Never nil: byteOffset is 0 and the clamp above bounds the length. + let packedNameReference = PackedNameReference(usesPrivateNameBuffer: true, isExternal: symbol.isExternal, byteOffset: 0, byteLength: nameBytes.count)! self.init( mappedStringTableBase: nil, privateNameBuffer: nameBytes, @@ -212,26 +245,41 @@ struct SymbolTableBuilder { /// the existing row's offset and external bit in place (last-wins, like /// the former name-keyed collection pass) and keeps the first /// occurrence's name reference — the bytes are equal by definition. - mutating func canonicalRow(forName name: String, mappedNameByteOffset: Int, nameByteLength: Int, canonicalOffset: Int, isExternal: Bool) -> (row: UInt32, isNewRow: Bool) { + mutating func canonicalRow(forName name: String, mappedNameByteOffset: Int, nameByteLength: Int, canonicalOffset: Int, isExternal: Bool) -> (row: UInt32, isNewRow: Bool)? { precondition(mappedStringTableBase != nil, "mapped name references require a mapped string table base") + // The offset and length are binary-supplied (a pointer difference + // into the mapped string table, and strlen over it): a name whose + // geometry cannot pack is malformed input, and its row is refused — + // the caller skips the symbol — rather than trapping the process. + guard let nameReference = PackedNameReference(usesPrivateNameBuffer: false, isExternal: isExternal, byteOffset: mappedNameByteOffset, byteLength: nameByteLength) else { + return nil + } return canonicalRow( forName: name, - nameReference: PackedNameReference(usesPrivateNameBuffer: false, isExternal: isExternal, byteOffset: mappedNameByteOffset, byteLength: nameByteLength), + nameReference: nameReference, canonicalOffset: canonicalOffset ) } /// The table row for a symbol whose name has no mapped-memory home /// (`MachOFile` rows, export-trie names): a new row appends the name's - /// bytes to the private buffer. - mutating func canonicalRow(forName name: String, canonicalOffset: Int, isExternal: Bool) -> (row: UInt32, isNewRow: Bool) { + /// bytes to the private buffer. Refuses (returns `nil` for) a name + /// whose geometry cannot pack — same budget rule as the mapped + /// overload, validated BEFORE appending so a refused name leaves no + /// orphan bytes in the buffer. + mutating func canonicalRow(forName name: String, canonicalOffset: Int, isExternal: Bool) -> (row: UInt32, isNewRow: Bool)? { if let existingRow = tableRowByName[name] { updateRowInPlace(existingRow, canonicalOffset: canonicalOffset, isExternal: isExternal) return (existingRow, false) } let byteOffset = privateNameBuffer.count + guard name.utf8.count <= PackedNameReference.maximumByteLength, + byteOffset <= PackedNameReference.maximumByteOffset else { + return nil + } privateNameBuffer.append(contentsOf: name.utf8) - let nameReference = PackedNameReference(usesPrivateNameBuffer: true, isExternal: isExternal, byteOffset: byteOffset, byteLength: privateNameBuffer.count - byteOffset) + // Never nil: both components were bounds-checked above. + let nameReference = PackedNameReference(usesPrivateNameBuffer: true, isExternal: isExternal, byteOffset: byteOffset, byteLength: privateNameBuffer.count - byteOffset)! return appendRow(forName: name, nameReference: nameReference, canonicalOffset: canonicalOffset) } @@ -244,15 +292,9 @@ struct SymbolTableBuilder { } private mutating func updateRowInPlace(_ row: UInt32, canonicalOffset: Int, isExternal: Bool) { - let existingNameReference = rows[Int(row)].packedNameReference rows[Int(row)] = SymbolRow( canonicalOffset: Int64(canonicalOffset), - packedNameReference: PackedNameReference( - usesPrivateNameBuffer: existingNameReference.usesPrivateNameBuffer, - isExternal: isExternal, - byteOffset: existingNameReference.byteOffset, - byteLength: existingNameReference.byteLength - ) + packedNameReference: rows[Int(row)].packedNameReference.replacingIsExternal(isExternal) ) } diff --git a/Tests/MachOSymbolsTests/PackedNameReferenceBudgetTests.swift b/Tests/MachOSymbolsTests/PackedNameReferenceBudgetTests.swift new file mode 100644 index 00000000..1a599c32 --- /dev/null +++ b/Tests/MachOSymbolsTests/PackedNameReferenceBudgetTests.swift @@ -0,0 +1,64 @@ +import Foundation +import Testing +import Demangling +@_spi(Internals) @testable import MachOSymbols + +/// The packed name-reference budgets (22-bit byte length / 40-bit byte +/// offset) are enforced against binary-supplied values: a name that cannot +/// pack must degrade — the build sweep skips its row, the standalone public +/// initializer clamps — never trap. `precondition` here is fatal in release +/// builds, so a malformed or hostile binary's symbol-table geometry would +/// decide whether the host process lives (PR #103 review, finding M3). +@Suite +struct PackedNameReferenceBudgetTests { + private static let oversizedName = "$s" + String(repeating: "A", count: PackedNameReference.maximumByteLength + 64) + + private static func makeNodeReference() -> NodeReference { + var nodeStoreBuilder = NodeStoreBuilder() + let nodeIndex = nodeStoreBuilder.intern(Node.create(kind: .identifier, text: "PackedNameReferenceBudgetTests")) + return nodeStoreBuilder.freeze().reference(at: nodeIndex) + } + + @Test func packedNameReferenceRefusesOverBudgetComponents() { + #expect(PackedNameReference(usesPrivateNameBuffer: true, isExternal: false, byteOffset: 0, byteLength: PackedNameReference.maximumByteLength + 1) == nil) + #expect(PackedNameReference(usesPrivateNameBuffer: true, isExternal: false, byteOffset: PackedNameReference.maximumByteOffset + 1, byteLength: 1) == nil) + #expect(PackedNameReference(usesPrivateNameBuffer: true, isExternal: false, byteOffset: -1, byteLength: 1) == nil) + #expect(PackedNameReference(usesPrivateNameBuffer: true, isExternal: false, byteOffset: 0, byteLength: -1) == nil) + + let packedNameReference = PackedNameReference(usesPrivateNameBuffer: true, isExternal: true, byteOffset: 7, byteLength: 5) + #expect(packedNameReference?.usesPrivateNameBuffer == true) + #expect(packedNameReference?.isExternal == true) + #expect(packedNameReference?.byteOffset == 7) + #expect(packedNameReference?.byteLength == 5) + } + + /// A private-buffer name beyond the byte-length budget is skipped by the + /// build sweep (its row is refused, no orphan bytes are appended), while + /// normal names keep minting rows as before. + @Test func tableBuilderSkipsAnOverBudgetPrivateBufferName() { + var tableBuilder = SymbolTableBuilder(mappedStringTableBase: nil) + #expect(tableBuilder.canonicalRow(forName: Self.oversizedName, canonicalOffset: 0, isExternal: false) == nil) + + let normalRow = tableBuilder.canonicalRow(forName: "$s10NormalNameV", canonicalOffset: 8, isExternal: false) + #expect(normalRow?.isNewRow == true) + + let symbolTable = tableBuilder.freeze() + #expect(symbolTable.rowCount == 1) + #expect(symbolTable.materializedName(atRow: 0) == "$s10NormalNameV") + } + + /// The public `DemangledSymbol(symbol:demangledNode:)` initializer packs + /// the caller's name into a one-row table; a name beyond the budget + /// clamps to the representable prefix instead of trapping (a legitimate + /// mangled name never approaches the 4 MB budget, so the clamp is + /// unreachable for honest input). + @Test func standalonePublicInitializerClampsInsteadOfTrapping() { + let demangledSymbol = DemangledSymbol( + symbol: Symbol(offset: 0, name: Self.oversizedName), + demangledNode: Self.makeNodeReference() + ) + #expect(demangledSymbol.name.utf8.count == PackedNameReference.maximumByteLength) + #expect(Self.oversizedName.hasPrefix(demangledSymbol.name)) + #expect(demangledSymbol.retainedSymbolTableRowCount == 1) + } +} From 44bedf8b8a75725f839f00585100183947cce20d Mon Sep 17 00:00:00 2001 From: Mx-Iris Date: Sun, 9 Aug 2026 22:23:26 +0800 Subject: [PATCH 62/77] fix(scripts): make the A/B parity gate fail on zero compared pairs and failed test runs compare_all_pairs derived its verdict solely from the difference count, which only increments while iterating discovered baseline/*.txt files: with no cache archive, no installed simulator runtime, or a mistyped --frameworks, every scenario degrades to paired .skip markers, the glob yields nothing, and the script printed 'RESULT: all pairs byte-identical.' and exited 0 over an empty comparison. run_macho_image_part compounded it by printing swift test's exit code without propagating it, so a failed MachOImage leg could not fail the run either. A harness that reads failure as success retroactively weakens every conclusion that cited its green light. The verdict now requires a non-empty examined-pair count, reports how many pairs backed it, and any non-zero test invocation is a run-level hard failure regardless of the diff outcome. Found by the PR #103 review (finding H4). Demonstrated pre-fix: an empty output root returned difference_count 0 (success path); post-fix the same input takes the zero-pair FAILED branch. --- Scripts/run-rendering-ab-verification.py | 44 +++++++++++++++++++++--- 1 file changed, 39 insertions(+), 5 deletions(-) diff --git a/Scripts/run-rendering-ab-verification.py b/Scripts/run-rendering-ab-verification.py index bcf7f435..07a30e10 100755 --- a/Scripts/run-rendering-ab-verification.py +++ b/Scripts/run-rendering-ab-verification.py @@ -82,6 +82,10 @@ def __init__(self, arguments: argparse.Namespace) -> None: self.arguments = arguments self.output_root: Path = arguments.output_root self.command_line_interfaces: dict[str, Path] = {} + # Invocation failures that must fail the whole run regardless of the + # diff outcome (a swallowed non-zero swift-test exit once let a green + # verdict stand over an incomplete matrix — PR #103 review, H4). + self.hard_failure_messages: list[str] = [] # --- Building ----------------------------------------------------------- @@ -223,16 +227,33 @@ def run_macho_image_part(self) -> None: "--filter", "RenderingVerificationTests", ], env=environment, stdout=log_handle, stderr=subprocess.STDOUT) print(f"[{side}] machoimage-current exit={completed.returncode}") + if completed.returncode != 0: + # Unlike the CLI scenarios (which degrade to paired .skip + # markers), a failed test invocation silently thins the + # comparison matrix — propagate it as a run-level failure. + self.hard_failure_messages.append( + f"machoimage-current[{side}]: swift test exited {completed.returncode} (log: {log_file})") # --- Diff phase --------------------------------------------------------- - def compare_all_pairs(self) -> int: + def compare_all_pairs(self) -> tuple[int, int]: + """Returns (difference_count, examined_pair_count). + + The examined count exists so the verdict can refuse to pass on an + empty comparison: with no cache archive, no installed runtime, or a + mistyped --frameworks, every scenario degrades to paired .skip + markers, the glob yields nothing, and a difference count of 0 would + otherwise read as success (PR #103 review, H4 — a harness that + cannot fail is worse than no harness). + """ print("\n=== A/B comparison ===") difference_count = 0 + examined_pair_count = 0 baseline_files = sorted(self.output_root.glob("**/baseline/*.txt")) for baseline_file in baseline_files: candidate_file = Path(str(baseline_file).replace("/baseline/", "/candidate/")) relative_name = baseline_file.relative_to(self.output_root) + examined_pair_count += 1 if not candidate_file.is_file(): print(f"MISSING-ON-CANDIDATE {relative_name}") difference_count += 1 @@ -245,12 +266,13 @@ def compare_all_pairs(self) -> int: baseline_file = Path(str(candidate_file).replace("/candidate/", "/baseline/")) if not baseline_file.is_file(): print(f"MISSING-ON-BASELINE {candidate_file.relative_to(self.output_root)}") + examined_pair_count += 1 difference_count += 1 for skip_file in sorted(self.output_root.glob("**/baseline/*.skip")): candidate_skip = Path(str(skip_file).replace("/baseline/", "/candidate/")) if candidate_skip.is_file() and skip_file.read_text() == candidate_skip.read_text(): print(f"SKIPPED (both sides, {skip_file.read_text().strip()}) {skip_file.relative_to(self.output_root)}") - return difference_count + return difference_count, examined_pair_count def main() -> None: @@ -265,11 +287,23 @@ def main() -> None: if not arguments.skip_image_part: run.run_macho_image_part() - difference_count = run.compare_all_pairs() + difference_count, examined_pair_count = run.compare_all_pairs() + if run.hard_failure_messages: + for hard_failure_message in run.hard_failure_messages: + print(f"HARD-FAILURE {hard_failure_message}") + print("\nRESULT: FAILED — a test invocation exited non-zero, so the comparison matrix is incomplete " + "and no verdict over it is trustworthy.") + sys.exit(1) + if examined_pair_count == 0: + print("\nRESULT: FAILED — zero pairs were compared. Every scenario fell back to a skip marker " + "(no cache archive, no installed simulator runtime, or a mistyped --frameworks?); " + "a green verdict over nothing is meaningless.") + sys.exit(1) if difference_count == 0: - print("\nRESULT: all pairs byte-identical.") + print(f"\nRESULT: all {examined_pair_count} pairs byte-identical.") else: - print(f"\nRESULT: {difference_count} differing pair(s). Re-run the differing scenario twice on one side " + print(f"\nRESULT: {difference_count} differing pair(s) out of {examined_pair_count}. " + f"Re-run the differing scenario twice on one side " f"first to rule out nondeterminism before attributing.") sys.exit(1) From d2cacc50e8b0cc3aa879fc1fcd43d6d4f8353251 Mon Sep 17 00:00:00 2001 From: Mx-Iris Date: Sun, 9 Aug 2026 22:23:26 +0800 Subject: [PATCH 63/77] fix(tests): drain the fixture compiler's stderr before waiting, clean up its directory MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit LegacyDyldInfoBindTests compiled its fixture with standardError piped but read the pipe only AFTER waitUntilExit: diagnostics beyond the ~64 KB pipe buffer would deadlock compiler and parent — and because the compilation result is a static let under a serialized suite, a toolchain/SDK mismatch would hang the whole swift test run instead of reporting a failure. The pipe is now drained to EOF before reaping. The per-run LegacyDyldInfoBindFixture- directory (source + dylib) also leaked on every run; it is now removed at process exit (the fixture must outlive every test in the suite, so per-test cleanup is not an option; a crashed run leaks at most one uniquely-named directory). Found by the PR #103 review (finding L2; deadlock shape mechanism-verified, not reproduced — forcing 64 KB of swiftc diagnostics is not practical in a fixture). --- .../LegacyDyldInfoBindTests.swift | 25 ++++++++++++++++++- 1 file changed, 24 insertions(+), 1 deletion(-) diff --git a/Tests/SwiftInterfaceTests/LegacyDyldInfoBindTests.swift b/Tests/SwiftInterfaceTests/LegacyDyldInfoBindTests.swift index 85210aad..ebb6810b 100644 --- a/Tests/SwiftInterfaceTests/LegacyDyldInfoBindTests.swift +++ b/Tests/SwiftInterfaceTests/LegacyDyldInfoBindTests.swift @@ -22,11 +22,28 @@ import MachOExtensions /// format as the iOS 15.5 simulator frameworks that surfaced the bug. @Suite(.serialized) struct LegacyDyldInfoBindTests { + /// The fixture's working directory must outlive every test in the + /// (serialized) suite, so it is removed at process exit rather than per + /// test. A crashed run still leaks one directory; the unique name keeps + /// that harmless and non-colliding across parallel test processes. + private enum FixtureWorkingDirectoryCleanup { + nonisolated(unsafe) static var directories: [URL] = [] + static let registration: Void = { + atexit { + for directory in FixtureWorkingDirectoryCleanup.directories { + try? FileManager.default.removeItem(at: directory) + } + } + }() + } + private static let fixtureCompilationResult: Result = { Result { let workingDirectory = FileManager.default.temporaryDirectory .appendingPathComponent("LegacyDyldInfoBindFixture-\(UUID().uuidString)") try FileManager.default.createDirectory(at: workingDirectory, withIntermediateDirectories: true) + _ = FixtureWorkingDirectoryCleanup.registration + FixtureWorkingDirectoryCleanup.directories.append(workingDirectory) let sourceURL = workingDirectory.appendingPathComponent("LegacyFixture.swift") let libraryURL = workingDirectory.appendingPathComponent("libLegacyFixture.dylib") @@ -43,9 +60,15 @@ struct LegacyDyldInfoBindTests { let standardErrorPipe = Pipe() process.standardError = standardErrorPipe try process.run() + // Drain BEFORE waitUntilExit: diagnostics beyond the ~64 KB pipe + // buffer would otherwise deadlock compiler and parent (the child + // blocked writing, the parent parked waiting) — and because this + // is a static let under a serialized suite, that deadlock would + // hang the whole test run instead of reporting a failure. + let diagnosticsData = standardErrorPipe.fileHandleForReading.readDataToEndOfFile() process.waitUntilExit() guard process.terminationStatus == 0 else { - let diagnostics = String(decoding: standardErrorPipe.fileHandleForReading.readDataToEndOfFile(), as: UTF8.self) + let diagnostics = String(decoding: diagnosticsData, as: UTF8.self) throw LegacyFixtureCompilationError(diagnostics: diagnostics) } return libraryURL From 3c3e84a65cdc6c59dbcf6c671bc5b375abe4f71d Mon Sep 17 00:00:00 2001 From: Mx-Iris Date: Sun, 9 Aug 2026 22:23:26 +0800 Subject: [PATCH 64/77] fix(tests): make the offset-rebuild pin iterate all offsets deterministically MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit offsetQueriesRebuildSymbolsWithQueriedOffset capped its check at 500 entries while iterating a deliberately-unordered Dictionary whose iteration order is seeded per process — every run sampled a DIFFERENT 500 offsets, so a regression touching only cache-adjusted keys could pass one run and fail the next in the very test written to pin that rebuild. It now checks every offset in sorted order (0.065 s on the fixture — the cap bought nothing). Found by the PR #103 review (finding L4). --- .../MachOSymbolsTests/SymbolIndexStoreFixtureTests.swift | 9 +++++++-- 1 file changed, 7 insertions(+), 2 deletions(-) diff --git a/Tests/MachOSymbolsTests/SymbolIndexStoreFixtureTests.swift b/Tests/MachOSymbolsTests/SymbolIndexStoreFixtureTests.swift index 03a3deac..b4f677e3 100644 --- a/Tests/MachOSymbolsTests/SymbolIndexStoreFixtureTests.swift +++ b/Tests/MachOSymbolsTests/SymbolIndexStoreFixtureTests.swift @@ -174,9 +174,14 @@ final class SymbolIndexStoreFixtureTests: MachOFileTests, @unchecked Sendable { @Test func offsetQueriesRebuildSymbolsWithQueriedOffset() throws { let storage = try storage #expect(!storage.symbolRowsByOffset.isEmpty) + // Complete and deterministic: `symbolRowsByOffset` is deliberately + // unordered (nothing production iterates it), and Swift dictionary + // iteration order is seeded per process — a capped raw iteration + // once sampled a DIFFERENT 500 offsets every run, giving this pin + // unstable coverage. Sorting and checking every offset makes a + // regression on cache-adjusted keys reproducible. var checkedOffsetCount = 0 - for (offset, rows) in storage.symbolRowsByOffset { - guard checkedOffsetCount < 500 else { break } + for (offset, rows) in storage.symbolRowsByOffset.sorted(by: { $0.key < $1.key }) { let queried = try #require(SymbolIndexStore.shared.symbols(for: offset, in: machOFile)) #expect(queried.count == rows.count) #expect(queried.allSatisfy { $0.offset == offset }) From 040dbcf7438a35a1ac9ec83efec8d571d832e01f Mon Sep 17 00:00:00 2001 From: Mx-Iris Date: Sun, 9 Aug 2026 22:36:05 +0800 Subject: [PATCH 65/77] fix(indexing): defer per-image cache eviction to the image's last live indexer MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit deinit evicted three caches (symbol store, interned-name store, demangle memo) under one flag that only proves this indexer populated the symbol-store entry first — with indexer A owning image X and a later indexer B live on the same image, A's deinit wiped the caches out from under B: B's already-built names keep an orphaned store alive while later names land in a fresh one, so structurallyEquals' store === fast path stops firing for the pre-eviction population and every context demangle is re-paid. Per-cache ownership flags would not help (whoever populates first owns all three and the wipe-under-B is identical), so ownership is now claimed per IMAGE in a process-wide registry and the eviction runs in the deinit of the image's LAST live indexer. Entries built by non-indexer callers are still never claimed and never evicted — the pre-existing contract, enforced per image. Found by the PR #103 review (finding M6); pinned by PerImageCacheEvictionTests on the SymbolTestsHelper fixture (new MachOFileName case, kept out of shared suites so the membership assertions cannot race a parallel suite's indexer lifecycle; pre-fix the survivor test failed on all three caches). MetadataReader gains a package-visibility non-creating cacheExists(for:) probe for the test. --- Package.swift | 1 + .../MachOFixtureSupport/MachOFileName.swift | 5 + .../SwiftDeclarationIndexer.swift | 105 ++++++++++++++---- Sources/SwiftInspection/MetadataReader.swift | 7 ++ .../PerImageCacheEvictionTests.swift | 74 ++++++++++++ 5 files changed, 172 insertions(+), 20 deletions(-) create mode 100644 Tests/SwiftIndexingTests/PerImageCacheEvictionTests.swift diff --git a/Package.swift b/Package.swift index 30996d93..2dfc0678 100644 --- a/Package.swift +++ b/Package.swift @@ -944,6 +944,7 @@ extension Target { .target(.SwiftIndexing), .target(.SwiftPrinting), .target(.SwiftAttributeInference), + .target(.SwiftInspection), .target(.MachOTestingSupport), .target(.MachOFixtureSupport), ], diff --git a/Sources/MachOFixtureSupport/MachOFileName.swift b/Sources/MachOFixtureSupport/MachOFileName.swift index 2c6922bb..6ea263f8 100644 --- a/Sources/MachOFixtureSupport/MachOFileName.swift +++ b/Sources/MachOFixtureSupport/MachOFileName.swift @@ -19,4 +19,9 @@ package enum MachOFileName: String { case SymbolTests = "../../Tests/Projects/SymbolTests/DerivedData/SymbolTests/Build/Products/Release/SymbolTests.framework/Versions/A/SymbolTests" case SymbolTestsCore = "../../Tests/Projects/SymbolTests/DerivedData/SymbolTests/Build/Products/Release/SymbolTestsCore.framework/Versions/A/SymbolTestsCore" + /// SymbolTestsCore's resilient-parent dependency. Kept OUT of the shared + /// suites deliberately: `PerImageCacheEvictionTests` needs an image no + /// concurrently-running suite touches, so its cache-membership + /// assertions cannot race another suite's indexer lifecycle. + case SymbolTestsHelper = "../../Tests/Projects/SymbolTests/DerivedData/SymbolTests/Build/Products/Release/SymbolTestsHelper.framework/Versions/A/SymbolTestsHelper" } diff --git a/Sources/SwiftIndexing/SwiftDeclarationIndexer.swift b/Sources/SwiftIndexing/SwiftDeclarationIndexer.swift index 72764982..94d4a983 100644 --- a/Sources/SwiftIndexing/SwiftDeclarationIndexer.swift +++ b/Sources/SwiftIndexing/SwiftDeclarationIndexer.swift @@ -163,14 +163,16 @@ public final class SwiftDeclarationIndexer Bool { + registryLock.lock() + defer { registryLock.unlock() } + guard var imageEntry = entriesByImageIdentifier[imageIdentifier] else { return false } + imageEntry.liveIndexerCount -= 1 + guard imageEntry.liveIndexerCount <= 0 else { + entriesByImageIdentifier[imageIdentifier] = imageEntry + return false + } + entriesByImageIdentifier.removeValue(forKey: imageIdentifier) + return imageEntry.isEvictionClaimed + } +} diff --git a/Sources/SwiftInspection/MetadataReader.swift b/Sources/SwiftInspection/MetadataReader.swift index 33a39d88..a24d8458 100644 --- a/Sources/SwiftInspection/MetadataReader.swift +++ b/Sources/SwiftInspection/MetadataReader.swift @@ -55,6 +55,13 @@ extension MetadataReader { MetadataReaderCache.shared.remove(for: machO) } + /// Non-creating membership probe for the per-image demangle memo — + /// test-support surface for the indexer's cache-eviction contract + /// (`PerImageCacheEvictionTests`). + package static func cacheExists(for machO: some MachOSwiftSectionRepresentableWithCache) -> Bool { + MetadataReaderCache.shared.contains(in: machO) + } + public static func demangleContext(for context: ContextDescriptorWrapper, in machO: MachO) throws -> Node { if isCacheEnabled { return try MetadataReaderCache.shared.demangleContext(for: context, in: machO) diff --git a/Tests/SwiftIndexingTests/PerImageCacheEvictionTests.swift b/Tests/SwiftIndexingTests/PerImageCacheEvictionTests.swift new file mode 100644 index 00000000..cb747ae4 --- /dev/null +++ b/Tests/SwiftIndexingTests/PerImageCacheEvictionTests.swift @@ -0,0 +1,74 @@ +@_spi(Support) @testable import SwiftIndexing +import Foundation +import Testing +import MachOKit +import Dependencies +@_spi(Internals) import MachOSymbols +@_spi(Internals) import MachOCaches +@testable import MachOTestingSupport +import MachOFixtureSupport +@_spi(Internals) @testable import SwiftInspection + +/// The indexer's `deinit` cleans up three per-image caches (symbol store, +/// interned-name store, demangle memo). That cleanup must be performed by +/// the image's LAST live indexer — an earlier indexer deinitializing while +/// a second one still uses the same image must not wipe the caches out from +/// under it (the survivor's already-built names would keep an orphaned +/// store alive while new names land in a fresh one, splitting the +/// `store ===` fast paths for the rest of its lifetime). PR #103 review, +/// finding M6. +/// +/// Runs against `SymbolTestsHelper` — an image no other suite indexes — so +/// the cache-membership assertions cannot race a concurrently-running +/// suite's indexer lifecycle. +@Suite(.serialized) +final class PerImageCacheEvictionTests: MachOFileTests, @unchecked Sendable { + override class var fileName: MachOFileName { .SymbolTestsHelper } + + @Test func survivingIndexerKeepsPerImageCaches() async throws { + let unsafeMachOFile = machOFile + + var firstIndexer: SwiftDeclarationIndexer? = SwiftDeclarationIndexer(in: unsafeMachOFile) + try await firstIndexer?.prepare() + let secondIndexer = SwiftDeclarationIndexer(in: unsafeMachOFile) + try await secondIndexer.prepare() + + try #require(SymbolIndexStore.shared.contains(in: unsafeMachOFile)) + let internedNameCacheWasPresent = InternedNodeReferenceCache.shared.contains(in: unsafeMachOFile) + let demangleMemoWasPresent = MetadataReader.cacheExists(for: unsafeMachOFile) + + // The first indexer populated the caches, so under a per-indexer + // ownership flag its deinit would evict all three out from under + // the still-live second indexer. + firstIndexer = nil + + #expect( + SymbolIndexStore.shared.contains(in: unsafeMachOFile), + "the first indexer's deinit evicted the symbol store while a second live indexer was using the image" + ) + #expect(InternedNodeReferenceCache.shared.contains(in: unsafeMachOFile) == internedNameCacheWasPresent) + #expect(MetadataReader.cacheExists(for: unsafeMachOFile) == demangleMemoWasPresent) + + withExtendedLifetime(secondIndexer) {} + } + + @Test func lastIndexerEvictsPerImageCaches() async throws { + let unsafeMachOFile = machOFile + + var firstIndexer: SwiftDeclarationIndexer? = SwiftDeclarationIndexer(in: unsafeMachOFile) + try await firstIndexer?.prepare() + var secondIndexer: SwiftDeclarationIndexer? = SwiftDeclarationIndexer(in: unsafeMachOFile) + try await secondIndexer?.prepare() + try #require(SymbolIndexStore.shared.contains(in: unsafeMachOFile)) + + firstIndexer = nil + secondIndexer = nil + + #expect( + !SymbolIndexStore.shared.contains(in: unsafeMachOFile), + "the image's LAST live indexer must still perform the eviction — otherwise the per-image caches leak for the rest of the process lifetime" + ) + #expect(!InternedNodeReferenceCache.shared.contains(in: unsafeMachOFile)) + #expect(!MetadataReader.cacheExists(for: unsafeMachOFile)) + } +} From 90d775b91479b8f180fdaac68e06946ecbba3e48 Mon Sep 17 00:00:00 2001 From: Mx-Iris Date: Sun, 9 Aug 2026 22:41:38 +0800 Subject: [PATCH 66/77] docs(macho-symbols): adjudicate the detach node-store layer and the unload hazard MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Two PR #103 review findings resolve as adjudications rather than the suggested code changes, with the reasoning and re-adjudication conditions recorded in ReviewAdjudications.md (A4/A5) and revision notes on the findings document: - M5 (detachedFromSharedTable does not detach the node store): the suggested node-copy is refuted — the definition storing the detached symbol keeps the SAME NodeReference in its own node field (the intended per-image recycling model), so a copy reclaims nothing while the model lives. Landed as doc-comment precision (the detach is symbol-table-layer only; node-store sharing is deliberate) plus storedDeclarationSymbolsShareTheDefinitionsNodeStore pinning the sharing so a future copy must be a deliberate, measured decision. - M2 (raw mapped-string-table pointers outliving dlclose): WON'T FIX — the open half of its Q1 resolved NO by experiment (macOS 26): dyld pins every Swift/ObjC-content image as never-unload (dlclose leaves even a class-less Swift dylib mapped), and the only images that do unmap (pure C) contain no Swift-prefixed names so they never mint mapped rows. The images that could dangle have no rows; the images that have rows cannot dangle. Both suggested mitigations refused (remove-image hook = permanent dead code for pinned images; copy-on-vend breaks the 32-byte shared-table design for an unreachable scenario). SymbolTable's lifetime-constraint comment now records the empirical guarantee. --- .../Internal/ReviewAdjudications.md | 28 +++++++++++++++ Roadmaps/2026-08-09-pr103-review-findings.md | 2 ++ Sources/MachOSymbols/DemangledSymbol.swift | 10 ++++++ Sources/MachOSymbols/SymbolTable.swift | 8 ++++- .../SymbolTableRetentionTests.swift | 34 +++++++++++++++++++ 5 files changed, 81 insertions(+), 1 deletion(-) diff --git a/Documentations/Internal/ReviewAdjudications.md b/Documentations/Internal/ReviewAdjudications.md index bdcee424..3c134e9e 100644 --- a/Documentations/Internal/ReviewAdjudications.md +++ b/Documentations/Internal/ReviewAdjudications.md @@ -53,3 +53,31 @@ - **既往修复**：无既往修复；`TypedDumper`（dump 路径）保留独立实现是记录在案的设计（AGENTS.md）。 - **代码锚点**：不加代码注释（7 处太散），以本条目为准。 - **复审条件**：① 大镜像（SwiftUI 级）剖析显示 materialize 占比显著高于 fixture 的 1.18%；② 上游 Remangler / 打印基础设施泛型化（A1 复审条件 ①）落地后，节点合成问题若有上游方案可顺路重开。 + +--- + +## A4 — `MachOImage` 符号名裸指针在镜像卸载后悬垂（PR #103 review M2） + +- **裁决**：不修（2026-08-09，实验裁决——触发面在 Darwin 上结构性不可达）。 +- **发现**：`SymbolTable.withNameBytes(atRow:)` 对 mapped 行直接读 `mappedStringTableBase`（镜像 LINKEDIT 字符串表的裸指针），而 `SharedCache` 以镜像基址为键——推理链是 `dlopen → prepare → dlclose → 同址再 dlopen 另一 dylib` 后旧 `Storage` 被继续命中，材料化读到重映射内存（乱名或 SIGSEGV）。review 自记「mechanism confirmed by reading; end-to-end trigger not reproduced」。 +- **复现 / 是否误报**：机制读码属实，但**端到端触发被实验推翻**（2026-08-09，macOS 26 / Darwin 25.6.0 实测）： + 1. 含 Swift 内容的镜像（无论有没有 class，连仅含 `public func` + `struct` 的 dylib 都算）被 dyld 标记 never-unload——`dlclose` 后镜像仍在 `_dyld_image_count` 枚举中，永不 unmap。被本库索引的镜像必有 Swift 元数据（否则无从索引），全部落在这一类。 + 2. 唯一实测能真正 unmap 的形状是纯 C dylib（无 ObjC/Swift 内容）——但 mapped 行只为通过 `nameBytesHaveSwiftManglingPrefix` 的名字铸造，纯 C 镜像一个都不会有；export-trie 名走私有缓冲（拷贝）。能悬垂的没有行，有行的不会悬垂。 +- **与 main 基线对比**：main 每行驻留拷贝的 `String`，无此暴露面；裸指针层随 evolution 0001 引入。 +- **为什么不修**：两半暴露面都被结构性关死（上）。残余是内存安全的陈旧性问题：纯 C 镜像卸载后同址加载别的库，旧 `Storage`（仅有私有缓冲行，读安全）可能对新镜像被错误命中——答案错但不崩，且要求调用方索引纯 C dylib 再卸载再同址加载，窄到不值得为它上 `_dyld_register_func_for_remove_image` 驱逐钩子（对被 pin 的镜像该回调永不触发，等于常驻死代码）。review 建议的另一半「公开查询面改 vend 拷贝」同样拒绝：查询路径每次 vend 数十万个值，拷贝直接推翻 32 字节值 + 共享表的性能设计，为一个不可达场景付常驻代价。 +- **既往修复**：无；生命周期约束在 proposal 0001 落地时已记录为接受项（「镜像需保持加载」——现在知道这在 Darwin 上是 dyld 免费保证的）。 +- **代码锚点**：`SymbolTable` 类型注释（生命周期约束段，指回本条目）。 +- **复审条件**：① Apple 改变 never-unload 语义（dyld 开始真正卸载含 Swift/ObjC 内容的镜像）；② 本库新增对非 Darwin 平台的 `MachOImage` 支持；③ 出现「索引纯 C 镜像」的真实消费者——届时优先考虑 remove-image 驱逐钩子而非 vend 拷贝。 + +--- + +## A5 — `detachedFromSharedTable()` 不随符号表一并拷出 node store（PR #103 review M5 的建议修法） + +- **裁决**：拒绝按建议修（2026-08-09）；实际落地为文档收紧 + 回归测试钉住共享契约。 +- **发现**：`detachedFromSharedTable()` 只重建符号表层，`demangledNode` 原样传递、仍引用 per-image node store（全镜像符号的 nodes + edges + 文本 arena）——review 判「回收是部分的，而 doc comment 读起来像完全 detach」，建议把 node 层也拷出并扩展 `SymbolTableRetentionTests`。 +- **复现 / 是否误报**：现象属实（node store 确实不随 detach 释放），但**修法被读码推翻**：存储该 symbol 的定义自身的 `node` 字段就是**同一个** `NodeReference`（`DefinitionBuilder.makeFunctionDefinition` 等：`node = demangledSymbol.demangledNode`，不 detach，是 AGENTS.md 记录在案的「intended per-image recycling model」——活着的 definition 本来就该把它的 store 留活，打印名字要用）。只要 model 活着，兄弟字段就 pin 着同一个 store；在 detach 里拷贝 node 树回收为零，只多付每存储符号一次的分配。 +- **与 main 基线对比**：main 无 `NodeReference` 层（0001 之前），无此问题域。 +- **为什么这样裁决**：detach 的真实目的是让存储值不 pin 那张（definition 不需要的）符号表；node store 的生命周期被设计绑定在 model 上，二者分层清晰。缺陷只在 doc comment 的表述——已补上「detach 的是符号表层、node 层有意共享」的明文（`DemangledSymbol.detachedFromSharedTable()` doc），并新增 `storedDeclarationSymbolsShareTheDefinitionsNodeStore` 钉住共享（谁要改成拷贝必须先推翻这条测试、拿出测量）。 +- **既往修复**：`a7caf944` 设计单层 detach（当时只有符号表层）；`6b0dad20` 加 arena 层未回访 doc——回访结论是设计成立、文档失准。 +- **代码锚点**：`DemangledSymbol.detachedFromSharedTable()` doc comment；`SymbolTableRetentionTests.storedDeclarationSymbolsShareTheDefinitionsNodeStore`。 +- **复审条件**：出现「declaration model 已释放、仅存储的 `DemangledSymbol` 长期存活」的真实消费形态（届时 node 层拷贝才有回收对象），或 profiling 显示 per-image node store 是驻留头部且 model 生命周期无法缩短。 diff --git a/Roadmaps/2026-08-09-pr103-review-findings.md b/Roadmaps/2026-08-09-pr103-review-findings.md index 9a8b6b5c..f97a8486 100644 --- a/Roadmaps/2026-08-09-pr103-review-findings.md +++ b/Roadmaps/2026-08-09-pr103-review-findings.md @@ -133,6 +133,7 @@ The first row is the one worth generalizing: **evolution 0002's mechanical migra - **Q2 — New.** On `main` every row owns a copied `String`; image unload is irrelevant. - **Q3 — Worth fixing, priority depends on Q1's open half.** This is an inherent cost of the memory optimization, not an oversight. A cheap partial mitigation is available regardless: `detachedFromSharedTable()` covers only six internal storing sites, while the public query API still vends the raw pointer — making the public surface return copies closes the externally-reachable half without giving up the internal saving. The full fix is invalidating the cache entry on unload (or keying on something that changes when the image does). - **Q4 — Related prior work, different layer.** `a7caf944` introduced `detachedFromSharedTable()` to stop a stored value pinning the whole table; that layer is correct. The raw-pointer layer arrived later with evolution 0001, and the guard did not follow it up. +- **Revision (2026-08-09, implementing session) — adjudicated WON'T FIX; the open half of Q1 resolved NO by experiment.** On Darwin (probed on macOS 26): dyld pins every image carrying Swift/ObjC content as never-unload — `dlclose` leaves even a class-less pure-Swift dylib mapped — and the only images that genuinely unmap (pure C, no ObjC/Swift content) contain no Swift-mangling-prefixed names, so they never mint mapped rows. The images that could dangle have no rows; the images that have rows cannot dangle. Both suggested mitigations refused: a remove-image eviction hook would never fire for pinned images (permanent dead code), and copy-on-vend would break the 32-byte-value / shared-table performance design for an unreachable scenario. Full reasoning and re-adjudication conditions: `Documentations/Internal/ReviewAdjudications.md` (A4). ### M3. `PackedNameReference` enforces its bit budgets with `precondition` on binary-supplied values @@ -157,6 +158,7 @@ The first row is the one worth generalizing: **evolution 0002's mechanical migra - **Q2 — New.** On `main`, `DemangledSymbol` holds no `NodeReference`; there is no second layer to detach. - **Q3 — Worth fixing.** `FunctionDefinition.symbol`, `Accessor.symbol`, `TypeDefinition.deallocatorSymbol` / `destructorSymbol` are stored through this call so `removeSubIndexer(_:)` can reclaim per-image memory. The 185,988-row table is released as documented; the arena is not, so the reclamation is partial while the doc comment reads as a full detach. `SymbolTableRetentionTests` only asserts `retainedSymbolTableRowCount == 1` and never inspects the node store — **the guard written for this property is blind to exactly this gap**, so any fix should extend the test too. - **Q4 — Related prior work.** `a7caf944` designed the single-layer detach; `6b0dad20` added the arena layer without revisiting it. +- **Revision (2026-08-09, implementing session) — the suggested node-copy is refuted; landed as documentation + a sharing pin instead.** The definition that stores the detached symbol keeps the SAME `NodeReference` in its own `node` field (`DefinitionBuilder`: `node = demangledSymbol.demangledNode`, undetached — AGENTS.md's intended per-image recycling model, since a live definition needs its tree to print), so copying the node out inside `detachedFromSharedTable()` reclaims nothing while the model lives and only adds an allocation per stored symbol. The actual defect was the doc comment overpromising: it now states the detach is symbol-table-layer only and the node-store sharing is deliberate, and `SymbolTableRetentionTests.storedDeclarationSymbolsShareTheDefinitionsNodeStore` pins the sharing so a future copy must be a deliberate, measured decision. See `Documentations/Internal/ReviewAdjudications.md` (A5). ### M6. `deinit` evicts three caches under a flag that proves ownership of one diff --git a/Sources/MachOSymbols/DemangledSymbol.swift b/Sources/MachOSymbols/DemangledSymbol.swift index 0641775d..c653a81c 100644 --- a/Sources/MachOSymbols/DemangledSymbol.swift +++ b/Sources/MachOSymbols/DemangledSymbol.swift @@ -77,6 +77,16 @@ public struct DemangledSymbol: Sendable { /// distinct rows — 5.1% of a 185,988-row table — so detaching them trades /// roughly 0.6 MB of small allocations for the whole table's retention. /// + /// What detaches is the SYMBOL-TABLE layer only (rows + name bytes and, + /// for an image table, its tie to the loaded image's mapped string + /// table). `demangledNode` deliberately keeps referencing the per-image + /// node store: the owning definition's `node` field is the same + /// reference into the same store (the intended per-image recycling + /// model — a live definition keeps its store alive), so copying the + /// tree out here would reclaim nothing while the model lives and would + /// only add an allocation per stored symbol. + /// `SymbolTableRetentionTests` pins both layers of this contract. + /// /// Call this when storing a value into a long-lived declaration, not on /// the query path. public func detachedFromSharedTable() -> DemangledSymbol { diff --git a/Sources/MachOSymbols/SymbolTable.swift b/Sources/MachOSymbols/SymbolTable.swift index 1e3eb22c..ce2bc02e 100644 --- a/Sources/MachOSymbols/SymbolTable.swift +++ b/Sources/MachOSymbols/SymbolTable.swift @@ -96,7 +96,13 @@ struct PackedNameReference { /// points into the loaded image and dangles if the image is ever unloaded. /// Name materialization therefore requires the image to stay loaded — the /// same requirement every other in-process read path already has, but now -/// extending to vended values' `symbol` accessor. +/// extending to vended values' `symbol` accessor. Empirically the unload +/// cannot happen on Darwin (probed 2026-08-09, macOS 26): dyld pins every +/// image carrying Swift/ObjC content as never-unload — `dlclose` leaves +/// even a class-less Swift dylib mapped — and the only images that DO +/// unmap (pure C, no ObjC/Swift) contain no Swift-mangling-prefixed names, +/// so they never mint mapped rows in the first place. See +/// `Documentations/Internal/ReviewAdjudications.md` (A4). /// /// The byte access layer uses `UnsafeBufferPointer` rather than /// `Span`/`UTF8Span`: the Span family is only available at runtime on diff --git a/Tests/SwiftInterfaceTests/SymbolTableRetentionTests.swift b/Tests/SwiftInterfaceTests/SymbolTableRetentionTests.swift index 5f4925f9..edd0f187 100644 --- a/Tests/SwiftInterfaceTests/SymbolTableRetentionTests.swift +++ b/Tests/SwiftInterfaceTests/SymbolTableRetentionTests.swift @@ -92,4 +92,38 @@ final class SymbolTableRetentionTests: MachOFileTests, @unchecked Sendable { """ ) } + + /// The detach contract's SECOND layer (PR #103 review, finding M5, + /// adjudicated): `detachedFromSharedTable()` deliberately does NOT copy + /// `demangledNode` out of the per-image node store — the owning + /// definition's `node` field is the same reference into the same store + /// (the intended per-image recycling model), so a copy would reclaim + /// nothing while the model lives and would only add an allocation per + /// stored symbol. This test pins the sharing; changing it to a copy + /// must be a deliberate, measured decision. + @Test func storedDeclarationSymbolsShareTheDefinitionsNodeStore() async throws { + let builder = try SwiftInterfaceBuilder( + configuration: .init(indexConfiguration: .init(showCImportedTypes: false)), + eventHandlers: [], + in: machOFile + ) + try await builder.prepare() + _ = try await builder.printRoot() + + var inspectedFunctionCount = 0 + for (_, typeDefinition) in builder.indexer.allTypeDefinitions { + let functions = typeDefinition.functions + + typeDefinition.staticFunctions + + typeDefinition.allocators + + typeDefinition.constructors + for function in functions { + inspectedFunctionCount += 1 + #expect( + function.symbol.demangledNode.store === function.node.store, + "\(function.name): the detached symbol's node was copied out of the definition's own store — zero reclamation, pure allocation overhead" + ) + } + } + #expect(inspectedFunctionCount > 0) + } } From 4b785d103a79a8f167ed757fe4eb3ed68523304b Mon Sep 17 00:00:00 2001 From: Mx-Iris Date: Sun, 9 Aug 2026 22:50:09 +0800 Subject: [PATCH 67/77] fix(printing): propagate a thrown conformance materialization from printExtensionHeader MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit try? collapsed a thrown materializedProtocolConformance(in:) into the same nil as 'this extension has no conformance', so the public printExtensionHeader emitted a confidently wrong extension header with the conformance clause, @retroactive, and global-actor markers silently missing. In-repo the swallow is unreachable — the only caller indexes first, and index(in:) runs the same materialization with a bare try — which is exactly the defect: a public entry holding a weaker error contract than the index(in:) it follows. The materialization now propagates; external callers hitting a genuinely unreadable conformance get the error instead of a silently degraded header. Found by the PR #103 review (finding M1, revised during implementation: the originally-claimed rendering-path failure is unreachable in-repo and the finding was downgraded to Low with the public-contract framing; see the findings doc's revision note). Pinned by printExtensionHeaderPropagatesMaterializationFailure via a real descriptor layout re-wrapped at an out-of-bounds offset (new package raw-descriptor initializer on ExtensionDefinition as the test surface). --- .../Definitions/ExtensionDefinition.swift | 13 +++++++ .../SwiftDeclarationPrinter.swift | 13 +++++-- .../SymbolTestsCoreIntegrationTests.swift | 36 +++++++++++++++++++ 3 files changed, 59 insertions(+), 3 deletions(-) diff --git a/Sources/SwiftDeclaration/Components/Definitions/ExtensionDefinition.swift b/Sources/SwiftDeclaration/Components/Definitions/ExtensionDefinition.swift index 26561f87..f4ff8cbf 100644 --- a/Sources/SwiftDeclaration/Components/Definitions/ExtensionDefinition.swift +++ b/Sources/SwiftDeclaration/Components/Definitions/ExtensionDefinition.swift @@ -91,6 +91,19 @@ public final class ExtensionDefinition: Definition, MutableDefinition { self.resolvedAssociatedTypeWitnesses = resolvedAssociatedTypeWitnesses } + /// Test/tooling surface: constructs a definition around a RAW descriptor + /// reference, no parsed wrapper required — error-contract tests use it to + /// build a definition whose materialization deterministically fails + /// (a real descriptor layout re-wrapped at an out-of-bounds offset). + package init(extensionName: ExtensionName, genericSignature: NodeReference?, protocolConformanceDescriptor: ProtocolConformanceDescriptor?) { + self.extensionName = extensionName + self.genericSignature = genericSignature + self.protocolConformanceDescriptor = protocolConformanceDescriptor + self.conformingProtocolName = nil + self.associatedTypes = [] + self.resolvedAssociatedTypeWitnesses = [] + } + /// Rebuilds the full `ProtocolConformance` (trailing objects included) /// from the retained descriptor; `nil` for member / typealias-only /// extensions. Materialization discipline (evolution proposal 0002): diff --git a/Sources/SwiftPrinting/SwiftDeclarationPrinter.swift b/Sources/SwiftPrinting/SwiftDeclarationPrinter.swift index bd481d34..86cb3f51 100644 --- a/Sources/SwiftPrinting/SwiftDeclarationPrinter.swift +++ b/Sources/SwiftPrinting/SwiftDeclarationPrinter.swift @@ -245,9 +245,16 @@ public final class SwiftDeclarationPrinter: Sendab extensionDefinition.extensionName.print() // This print operation's single conformance materialization - // (proposal 0002). A materialization failure is treated like the - // thrown-resolution case below — the whole clause is dropped. - let materializedProtocolConformance = try? extensionDefinition.materializedProtocolConformance(in: machO) + // (proposal 0002). Propagates on failure: a public entry must not + // hold a weaker error contract than the `index(in:)` that precedes + // it on every in-repo path — both run this same materialization, so + // in-repo the throw is unreachable, but an external caller invoking + // this entry directly on an un-indexed definition would otherwise + // get a confidently wrong `extension Foo` header with the + // conformance clause, `@retroactive`, and global-actor markers + // silently missing (a `try?` here once conflated that failure with + // "no conformance at all"). + let materializedProtocolConformance = try extensionDefinition.materializedProtocolConformance(in: machO) // Pre-leaf-migration `dumpProtocolName` semantics: a `nil` protocol // node collapses to an *empty* name but still emits the clause (the diff --git a/Tests/SwiftIndexingTests/SymbolTestsCoreIntegrationTests.swift b/Tests/SwiftIndexingTests/SymbolTestsCoreIntegrationTests.swift index f7374144..14abb509 100644 --- a/Tests/SwiftIndexingTests/SymbolTestsCoreIntegrationTests.swift +++ b/Tests/SwiftIndexingTests/SymbolTestsCoreIntegrationTests.swift @@ -518,6 +518,42 @@ extension STCoreTests { } } +// MARK: - Extension Header Error Contract + +extension STCoreTests { + /// The public `printExtensionHeader` must PROPAGATE a thrown conformance + /// materialization, matching the error contract of the `index(in:)` that + /// precedes it on every in-repo path (both run the same + /// materialization, so in-repo the throw is unreachable — but an + /// external caller invoking the public entry directly on an un-indexed + /// definition would otherwise get a confidently wrong `extension Foo` + /// header with the conformance clause, `@retroactive`, and global-actor + /// markers silently missing). + @Test func printExtensionHeaderPropagatesMaterializationFailure() async throws { + let indexer = try await preparedIndexer() + let donorExtensionDefinition = try #require( + indexer.conformanceExtensionDefinitions.values.flatMap { $0 } + .first { $0.protocolConformanceDescriptor != nil } + ) + let realDescriptor = try #require(donorExtensionDefinition.protocolConformanceDescriptor) + // A real descriptor layout re-wrapped at an offset far past the + // fixture's end of file: every relative resolve inside the + // materialization computes an out-of-bounds read and throws. + let unreadableDescriptor = ProtocolConformanceDescriptor(layout: realDescriptor.layout, offset: 0x0FFF_FFF0) + let unreadableExtensionDefinition = ExtensionDefinition( + extensionName: donorExtensionDefinition.extensionName, + genericSignature: nil, + protocolConformanceDescriptor: unreadableDescriptor + ) + + nonisolated(unsafe) let unsafeExtensionDefinition = unreadableExtensionDefinition + nonisolated(unsafe) let unsafePrinter = SwiftDeclarationPrinter(in: machOFile) + await #expect(throws: (any Error).self) { + _ = try await unsafePrinter.printExtensionHeader(unsafeExtensionDefinition, level: 1) + } + } +} + // MARK: - Opaque Return Types (Integration) extension STCoreTests { From 5b82e703d449811b578870981dceaca8d983fc43 Mon Sep 17 00:00:00 2001 From: Mx-Iris Date: Sun, 9 Aug 2026 22:57:53 +0800 Subject: [PATCH 68/77] fix(printing): push the per-definition catch into the nested-children loops MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit printTypeDefinition and printExtensionDefinition iterated their nested type/protocol children with a bare try await: one nested child whose descriptor could not be read escaped the enclosing definition and the top-level per-definition catch discarded the WHOLE outer type or extension. The nested loops now catch per child (printCatchedThrowing, the same helper printRoot's contract uses), so a corrupt child drops only itself — one more step in the direction the per-block -> per- definition change already moved. Healthy-path output is byte-identical (snapshot suites: 175 tests in 21 suites green). Found by the PR #103 review (finding L1); pinned by corruptNestedChildDropsOnlyItself via a real struct-descriptor layout re-wrapped at an out-of-bounds offset (new package raw-descriptor initializer on TypeDefinition as the test surface; pre-fix the child's offsetOutOfBounds failed the whole parent). --- .../Definitions/TypeDefinition.swift | 11 +++++ .../SwiftDeclarationPrinter.swift | 40 +++++++++++++++---- .../SymbolTestsCoreIntegrationTests.swift | 38 ++++++++++++++++++ 3 files changed, 81 insertions(+), 8 deletions(-) diff --git a/Sources/SwiftDeclaration/Components/Definitions/TypeDefinition.swift b/Sources/SwiftDeclaration/Components/Definitions/TypeDefinition.swift index e9e35905..0104b84d 100644 --- a/Sources/SwiftDeclaration/Components/Definitions/TypeDefinition.swift +++ b/Sources/SwiftDeclaration/Components/Definitions/TypeDefinition.swift @@ -146,6 +146,17 @@ public final class TypeDefinition: Definition { self.isSpecialized = isSpecialized } + /// Test/tooling surface: constructs a definition around a RAW descriptor + /// reference, no parsed wrapper required — error-contract tests use it to + /// build a definition whose indexing/materialization deterministically + /// fails (a real descriptor layout re-wrapped at an out-of-bounds + /// offset). + package init(typeContextDescriptorWrapper: TypeContextDescriptorWrapper, typeName: TypeName, isSpecialized: Bool) { + self.typeContextDescriptorWrapper = typeContextDescriptorWrapper + self.typeName = typeName + self.isSpecialized = isSpecialized + } + public convenience init(type: TypeContextWrapper, in machO: MachO) async throws { let typeName = try type.typeName(in: machO) self.init(type: type, typeName: typeName, isSpecialized: false) diff --git a/Sources/SwiftPrinting/SwiftDeclarationPrinter.swift b/Sources/SwiftPrinting/SwiftDeclarationPrinter.swift index 86cb3f51..2c70e788 100644 --- a/Sources/SwiftPrinting/SwiftDeclarationPrinter.swift +++ b/Sources/SwiftPrinting/SwiftDeclarationPrinter.swift @@ -134,15 +134,28 @@ public final class SwiftDeclarationPrinter: Sendab try await DeclarationBlock(level: level) { try await renderTypeDeclarationHeader(for: materializedTypeContext, displayParentName: displayParentName, level: level, specializedMetadata: specializedMetadata) } body: { + // Per-CHILD catch: one nested child whose printing throws drops + // only itself — the same per-definition contract `printRoot` + // applies at the top level, pushed into the nested loops. A + // child's throw once escaped here and the top-level catch + // discarded the whole enclosing type. for child in typeDefinition.typeChildren { - try await NestedDeclaration { - try await printTypeDefinition(child, level: level + 1) + if let renderedChild = await printCatchedThrowing({ + try await NestedDeclaration { + try await printTypeDefinition(child, level: level + 1) + } + }) { + renderedChild } } for child in typeDefinition.protocolChildren { - try await NestedDeclaration { - try await printProtocolDefinition(child, level: level + 1) + if let renderedChild = await printCatchedThrowing({ + try await NestedDeclaration { + try await printProtocolDefinition(child, level: level + 1) + } + }) { + renderedChild } } @@ -212,15 +225,26 @@ public final class SwiftDeclarationPrinter: Sendab try await DeclarationBlock(level: level) { try await printExtensionHeader(extensionDefinition, level: level) } body: { + // Per-CHILD catch, same contract as `printTypeDefinition`'s + // nested loops: a nested definition whose printing throws + // drops only itself, never the whole extension. for typeDefinition in extensionDefinition.types { - try await NestedDeclaration { - try await printTypeDefinition(typeDefinition, level: level + 1) + if let renderedChild = await printCatchedThrowing({ + try await NestedDeclaration { + try await printTypeDefinition(typeDefinition, level: level + 1) + } + }) { + renderedChild } } for protocolDefinition in extensionDefinition.protocols { - try await NestedDeclaration { - try await printProtocolDefinition(protocolDefinition, level: level + 1) + if let renderedChild = await printCatchedThrowing({ + try await NestedDeclaration { + try await printProtocolDefinition(protocolDefinition, level: level + 1) + } + }) { + renderedChild } } diff --git a/Tests/SwiftIndexingTests/SymbolTestsCoreIntegrationTests.swift b/Tests/SwiftIndexingTests/SymbolTestsCoreIntegrationTests.swift index 14abb509..89de48e5 100644 --- a/Tests/SwiftIndexingTests/SymbolTestsCoreIntegrationTests.swift +++ b/Tests/SwiftIndexingTests/SymbolTestsCoreIntegrationTests.swift @@ -554,6 +554,44 @@ extension STCoreTests { } } +// MARK: - Nested Child Print Degradation + +extension STCoreTests { + /// A nested child whose descriptor cannot be read must drop ONLY itself: + /// the same per-definition catch `printRoot` applies at the top level, + /// pushed down into the nested-children loops. Before the fix the + /// child's throw escaped `printTypeDefinition` and the top-level catch + /// discarded the whole enclosing type. + @Test func corruptNestedChildDropsOnlyItself() async throws { + let indexer = try await preparedIndexer() + let parentDefinition = try #require(findTypeDefinition(named: "StructTest", in: indexer)) + let donorDefinition = try #require(findTypeDefinition(named: "FinalClassTest", in: indexer)) + + // A real struct descriptor's layout re-wrapped at an offset far past + // the fixture's end of file: every read the child's indexing + // performs is out of bounds and throws deterministically. + let realStructDefinition = try #require(findTypeDefinition(named: "GenericStructNonRequirement", in: indexer)) + guard case .struct(let realStructDescriptor) = realStructDefinition.typeContextDescriptorWrapper else { + Issue.record("GenericStructNonRequirement is expected to be a struct") + return + } + let unreadableDescriptor = StructDescriptor(layout: realStructDescriptor.layout, offset: 0x0FFF_FFF0) + let corruptChildDefinition = TypeDefinition( + typeContextDescriptorWrapper: .struct(unreadableDescriptor), + typeName: donorDefinition.typeName, + isSpecialized: false + ) + parentDefinition.typeChildren.append(corruptChildDefinition) + + nonisolated(unsafe) let unsafeParentDefinition = parentDefinition + nonisolated(unsafe) let unsafePrinter = SwiftDeclarationPrinter(in: machOFile) + let renderedParent = try await unsafePrinter.printTypeDefinition(unsafeParentDefinition).string + + #expect(renderedParent.contains("StructTest"), "the enclosing type must keep printing") + #expect(!renderedParent.contains("FinalClassTest"), "the corrupt child must be dropped, not rendered") + } +} + // MARK: - Opaque Return Types (Integration) extension STCoreTests { From e3b6d39fc1b54f4027b548633f071c2ce30b842d Mon Sep 17 00:00:00 2001 From: Mx-Iris Date: Sun, 9 Aug 2026 22:59:09 +0800 Subject: [PATCH 69/77] test(macho-caches): land the plain-dylib end-to-end lookup case, adjudicate the early exit MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The shape all three ranking rounds missed — a plain .dylib name, which can never reach bestMatchRank and always pays the full multi-cache scan — now has end-to-end coverage against the current system's dyld shared cache: libswiftCore resolves to /usr/lib/swift/libswiftCore.dylib and SwiftUI to the native canonical framework binary (never the iOSSupport Catalyst build). The review's suggested achievable-rank early exit is adjudicated as unsound and not landed: holding a rank-2 dylib hit, an unscanned subcache can still contain the rank-0 framework binary, so stopping early reintroduces exactly the cross-subcache order-dependence the ranking was built to remove; and the full scan measures at 43 ms on the current cache (at most once per CLI invocation), so the only sound alternative (replicating MachOKit's image enumeration to rank paths before constructing files) is not worth its drift risk. Found by the PR #103 review (finding L3, revised); reasoning and re-adjudication conditions in ReviewAdjudications.md (A6). --- .../Internal/ReviewAdjudications.md | 13 +++++++++ Roadmaps/2026-08-09-pr103-review-findings.md | 1 + .../DyldCacheImageSearchTests.swift | 28 +++++++++++++++++++ 3 files changed, 42 insertions(+) diff --git a/Documentations/Internal/ReviewAdjudications.md b/Documentations/Internal/ReviewAdjudications.md index 3c134e9e..e627cef5 100644 --- a/Documentations/Internal/ReviewAdjudications.md +++ b/Documentations/Internal/ReviewAdjudications.md @@ -81,3 +81,16 @@ - **既往修复**：`a7caf944` 设计单层 detach（当时只有符号表层）；`6b0dad20` 加 arena 层未回访 doc——回访结论是设计成立、文档失准。 - **代码锚点**：`DemangledSymbol.detachedFromSharedTable()` doc comment；`SymbolTableRetentionTests.storedDeclarationSymbolsShareTheDefinitionsNodeStore`。 - **复审条件**：出现「declaration model 已释放、仅存储的 `DemangledSymbol` 长期存活」的真实消费形态（届时 node 层拷贝才有回收对象），或 profiling 显示 per-image node store 是驻留头部且 model 生命周期无法缩短。 + +--- + +## A6 — `machOFile(by:)` 对 plain-dylib 名字全量扫描、无 rank-0 以外的早退（PR #103 review L3） + +- **裁决**：优化不做（2026-08-09，机制分析 + 测量双重裁决）；强制配套的 plain-`.dylib` 端到端用例已落地。 +- **发现**：rank 公式下只有原生 canonical framework 能到 `bestMatchRank`（0），plain dylib 名（`libswiftCore` → rank 2）永远触发全部 cache 文件的完整扫描；review 建议「track the best rank still achievable and stop when the current match ties it」。 +- **复现 / 是否误报**：扫描行为属实；但**建议的早退机制不成立**：持有 rank 2（dylib 命中）时，尚未扫描的 subcache 里仍可能存在 rank 0 的 framework 本体——「当前 rank 已是可达最优」这个判断在扫描中途无法安全做出，提前停恰是 `17ad4358` 要修的 SwiftUI→axbundle 顺序依赖误解析的复发形状（同名镜像跨 subcache 分布正是当年的触发条件）。rank 0 是唯一可靠的早退点，现状已实现。 +- **与 main 基线对比**：main 是 first-match-wins（快但错）；ranking 线是记录在案的 correctness-for-speed 取舍。 +- **为什么优化不做**：代价测出来是噪声级——当前系统 dyld cache（macOS 26，含全部 subcache）上 `machOFile(by: .name("libswiftCore"))` 全量扫描 **43 ms**（review 估计的「thousands of MachOFile constructions」实际单次构造微秒级），每次 CLI 调用至多一次。备选的「路径先行、只构造赢家」方案需在 MachOExtensions 里复刻 MachOKit `_machOFiles` 的枚举细节（main-cache imageInfos 回退、fileOffset 过滤），漂移风险大于 43 ms 的收益。 +- **既往修复**：`17ad4358`（引入 ranking）→ `6647359e`（跨 cache 文件生效）→ 本轮第三次审视。三轮都没落的 plain-`.dylib` 用例这次落了：`DyldCacheEndToEndLookupTests`（当前系统 cache 上 `libswiftCore` 解析到 `/usr/lib/swift/libswiftCore.dylib`、`SwiftUI` 解析到原生 framework 本体且非 iOSSupport）。 +- **代码锚点**：`DyldCacheEndToEndLookupTests` 的套件注释。 +- **复审条件**：① 出现高频调用 `machOFile(by:)` 的新消费形态（当前每 CLI 调用一次）；② MachOKit 上游暴露 `(imagePath, fileOffset)` 级枚举后，「只构造赢家」无需复刻内部细节——届时可顺手做。 diff --git a/Roadmaps/2026-08-09-pr103-review-findings.md b/Roadmaps/2026-08-09-pr103-review-findings.md index f97a8486..6d03bdda 100644 --- a/Roadmaps/2026-08-09-pr103-review-findings.md +++ b/Roadmaps/2026-08-09-pr103-review-findings.md @@ -195,6 +195,7 @@ The first row is the one worth generalizing: **evolution 0002's mechanical migra - **Q2 — A deliberate correctness-for-speed trade, not a regression.** `main` used `machOFiles().first(where: { $0.match(by: mode) })` — fast, but resolved `SwiftUI` to the accessibility bundle, which is the bug the ranking exists to fix. - **Q3 — Defer.** The ranking is right; what is missing is an early exit once no better rank is achievable (track the best rank still reachable and stop when the current match ties it). - **Q4 — Fixed twice already; this is the third round on the same function.** `17ad4358` introduced the ranking (SwiftUI resolving to `SwiftUI.axbundle` → empty dump, exit 0) → `6647359e` fixed it not applying across cache files → now the early exit is unreachable. Three rounds without landing it cleanly: **the fix should ship with a case covering a plain-`.dylib` lookup**, which is the shape all three rounds missed. +- **Revision (2026-08-09, implementing session) — the suggested early exit is refuted as UNSOUND, and the scan cost measured to noise; landed as the mandated test + adjudication.** Holding a rank-2 dylib hit, an unscanned subcache can still contain the rank-0 framework binary — "no better rank achievable" cannot be decided mid-scan, and stopping early is exactly the cross-subcache order-dependence shape `17ad4358` fixed. Rank 0 remains the only sound early exit (already implemented). Measured on the current system cache (macOS 26, all subcaches): the full plain-dylib scan takes **43 ms** — the estimated "thousands of MachOFile constructions" are microseconds each — at most once per CLI invocation. The mandated plain-`.dylib` end-to-end case landed as `DyldCacheEndToEndLookupTests`. See `Documentations/Internal/ReviewAdjudications.md` (A6). ### L4. An order-dependent test caps at 500 entries while iterating a deliberately unordered dictionary diff --git a/Tests/MachOCachesTests/DyldCacheImageSearchTests.swift b/Tests/MachOCachesTests/DyldCacheImageSearchTests.swift index 6fdb2fce..7a16a5dd 100644 --- a/Tests/MachOCachesTests/DyldCacheImageSearchTests.swift +++ b/Tests/MachOCachesTests/DyldCacheImageSearchTests.swift @@ -1,5 +1,7 @@ @testable import MachOExtensions import Testing +import Foundation +import MachOKit /// Regression coverage for dyld-shared-cache image selection. /// @@ -148,3 +150,29 @@ struct DyldCacheImageSearchTests { #expect(mode.matchRank(forImagePath: accessibilityBundlePath) == nil) } } + +// MARK: - End-to-End Cache Lookup + +/// End-to-end lookup against the CURRENT system's dyld shared cache — the +/// shape all three ranking rounds missed is a plain `.dylib` name, which can +/// never reach `bestMatchRank` (only a native canonical framework can) and +/// therefore always pays the full multi-cache scan before answering. +/// PR #103 review, finding L3. +private let currentSystemDyldSharedCachePath = "/System/Volumes/Preboot/Cryptexes/OS/System/Library/dyld/dyld_shared_cache_arm64e" + +@Suite(.enabled(if: FileManager.default.fileExists(atPath: currentSystemDyldSharedCachePath))) +struct DyldCacheEndToEndLookupTests { + + @Test func plainDylibNameResolvesToTheRealDylib() throws { + let cache = try DyldCache(url: URL(fileURLWithPath: currentSystemDyldSharedCachePath)) + let machOFile = try #require(cache.machOFile(by: .name("libswiftCore"))) + #expect(machOFile.imagePath == "/usr/lib/swift/libswiftCore.dylib") + } + + @Test func frameworkNameStillResolvesToTheCanonicalBinary() throws { + let cache = try DyldCache(url: URL(fileURLWithPath: currentSystemDyldSharedCachePath)) + let machOFile = try #require(cache.machOFile(by: .name("SwiftUI"))) + #expect(machOFile.imagePath.hasSuffix("SwiftUI.framework/Versions/A/SwiftUI")) + #expect(!machOFile.imagePath.contains("iOSSupport")) + } +} From c8e506d5f7c9de87e67e4f7a112b4c4007b1305e Mon Sep 17 00:00:00 2001 From: Mx-Iris Date: Sun, 9 Aug 2026 23:06:28 +0800 Subject: [PATCH 70/77] docs: settle the PR #103 review batch's documentation debts - ProjectEvolutionLog gains its missing sections for evolutions 0002 (descriptor slimming) and 0003 (row-bucket flattening) plus a section for the review-fix batch itself, and the TaskReports link that named a nonexistent file (2026-07-25-dyld-cache-... vs the actual 2026-07-25-cache-...) now resolves. - Task report for the whole implementation arc (Chinese, per convention): research, refuted premises, per-batch execution with pre-fix failure evidence, verification, and divergences. - The findings document's status line records the outcome (fixed or adjudicated per finding, B1 deliberately left to the user). - AGENTS.md synced with the changed facts: isBind/resolveBind parity + hostile-input bounds in MachOExtensions, the nested per-child print catch, and the per-image cache-eviction registry. --- AGENTS.md | 6 +-- .../Internal/ProjectEvolutionLog.md | 38 +++++++++++++++- ...6-08-09-pr103-review-fix-implementation.md | 44 +++++++++++++++++++ Roadmaps/2026-08-09-pr103-review-findings.md | 2 +- 4 files changed, 85 insertions(+), 5 deletions(-) create mode 100644 Documentations/Internal/TaskReports/2026-08-09-pr103-review-fix-implementation.md diff --git a/AGENTS.md b/AGENTS.md index c66207c1..9987a024 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -115,7 +115,7 @@ The interface generation is split into layered peer modules over a shared `Swift - `SwiftIndexEvents` - event namespace (Payload/Dispatcher/Handler) emitted by both indexer and printer **SwiftIndexing** - Builds the `SwiftDeclaration` model from a Mach-O image -- `SwiftDeclarationIndexer` - Indexes types, extensions, conformances +- `SwiftDeclarationIndexer` - Indexes types, extensions, conformances. Its `deinit` cleans up the three per-image caches (symbol store, interned-name store, demangle memo) via `PerImageCacheEvictionRegistry`: eviction is claimed per IMAGE (by whichever indexer's `prepare()` built the entry) and performed by the image's LAST live indexer, so an earlier-deinitializing indexer never wipes the caches out from under a live sibling; entries built by non-indexer callers are never evicted (pinned by `PerImageCacheEvictionTests`) - The section-wrapper populations the index passes consume (`types` / `protocols` / `protocolConformances` / `associatedTypes` and the parsed-value keyed conformance maps) are **indexing transients** since proposal 0002 — released when `prepare()` finishes, with no public projection. The retained conformance facts are the name-level maps `conformingProtocolNamesByTypeName` / `conformingTypesByProtocolName` (+ their merged `all*` variants), which is all any post-indexing consumer (including `SwiftSpecialization`'s `ConformanceProvider`) reads. - `SwiftIndexEventReporter`, `OSLogEventHandler`, `ConsoleEventHandler` - event handlers - `SwiftDeclarationIndexConfiguration` @@ -128,7 +128,7 @@ The interface generation is split into layered peer modules over a shared `Swift - `SwiftDeclarationPrintConfiguration`, `SwiftDeclarationMemberSortOrder` - Type-level members print `class` (not `static`) when they carry a vtable method descriptor (`isClassMember` on `FunctionDefinition` / `VariableDefinition` / `SubscriptDefinition` in `SwiftDeclaration`): mangling cannot distinguish the two spellings, but a `static` member is implicitly final and never gets a descriptor, so `override static` (illegal Swift) is structurally impossible in the output. The four descriptor-less `class` spellings (`final class func`, `class func` in a final class / an extension, `@objc dynamic class func`) are ABI-identical to `static` and conservatively print as the semantically-equivalent `static`. `ClassDumper`'s vtable-section keyword follows the same fact; the dump override-table lines keep the demangler's faithful `static` symbol prefix. See [Documentations/Internal/ClassMemberKeywordRecovery.md](Documentations/Internal/ClassMemberKeywordRecovery.md). - Specialized definitions (`TypeDefinition.isSpecialized`) render **bound**: the header prints the concrete-argument name (`Box`, generic-signature clause skipped) via `BoundDumpedTypeNameRenderer`, and each field's type node is substituted through the specialized runtime metadata via `SpecializedMetadataNodeSubstitution` — both live in `SwiftDeclarationRendering` so the dump path (`TypedDumper`, which keeps its own copies/forwarders) stays independent. See [Documentations/Internal/SpecializedInterfaceBoundRenderingRestoration.md](Documentations/Internal/SpecializedInterfaceBoundRenderingRestoration.md). -- The main interface path's stored-field / enum-case rendering (`renderModelFields` → `printThrowingField` / `printThrowingEnumCase`) carries the **pre-leaf-migration error contract**: record reads, metadata comments, and type printing propagate errors (a failing field fails the whole type), and an enum case's payload presence follows the field record's mangled type name (captured at index time as `FieldFlags.hasMangledTypeName`, so rendering never re-reads the record positionally) — a `Void` payload prints `case a()` exactly like the dump path, while a payload whose node *renders empty* degrades to the bare case in both paths (`case a()` around nothing is invalid Swift; the interface printers also render kind-9 accessor-function symbolic references as the honest `accessor function at ` fallback — see [Documentations/Internal/AccessorFunctionReferenceRendering.md](Documentations/Internal/AccessorFunctionReferenceRendering.md)). At the TOP level the contract inverts: `printRoot` (and `printThrowingProtocol`'s trailing default-implementation extensions) catch per definition — one type/protocol/extension whose printing throws drops only itself, never its whole block (a block-level catch once blanked every type of a legacy binary's interface; pinned by `LegacyDyldInfoBindTests`). The diff renderer's `printField` / `printEnumCase` keep their own per-member-catch, rendered-text-gating contract (that is their original design, needed for standalone `+`/`-` members). The shared comment engine's `FieldLayoutRenderer.storedFieldComments` / `enumCaseComments` are `throws` for the same reason, and multi-payload enum descriptors resolve through `MultiPayloadEnumDescriptorCache` in `SwiftDeclarationRendering` (built once per image as a *partial* map — one bad descriptor only degrades its own enum to the tagged projection). See [Documentations/Internal/LeafMigrationRegressionFixes.md](Documentations/Internal/LeafMigrationRegressionFixes.md). +- The main interface path's stored-field / enum-case rendering (`renderModelFields` → `printThrowingField` / `printThrowingEnumCase`) carries the **pre-leaf-migration error contract**: record reads, metadata comments, and type printing propagate errors (a failing field fails the whole type), and an enum case's payload presence follows the field record's mangled type name (captured at index time as `FieldFlags.hasMangledTypeName`, so rendering never re-reads the record positionally) — a `Void` payload prints `case a()` exactly like the dump path, while a payload whose node *renders empty* degrades to the bare case in both paths (`case a()` around nothing is invalid Swift; the interface printers also render kind-9 accessor-function symbolic references as the honest `accessor function at ` fallback — see [Documentations/Internal/AccessorFunctionReferenceRendering.md](Documentations/Internal/AccessorFunctionReferenceRendering.md)). At the TOP level the contract inverts: `printRoot` (and `printThrowingProtocol`'s trailing default-implementation extensions) catch per definition — one type/protocol/extension whose printing throws drops only itself, never its whole block (a block-level catch once blanked every type of a legacy binary's interface; pinned by `LegacyDyldInfoBindTests`). The same per-definition contract extends into the NESTED children loops (`printTypeDefinition` / `printExtensionDefinition`): a nested child whose printing throws drops only itself, never the enclosing definition (pinned by `corruptNestedChildDropsOnlyItself`). The diff renderer's `printField` / `printEnumCase` keep their own per-member-catch, rendered-text-gating contract (that is their original design, needed for standalone `+`/`-` members). The shared comment engine's `FieldLayoutRenderer.storedFieldComments` / `enumCaseComments` are `throws` for the same reason, and multi-payload enum descriptors resolve through `MultiPayloadEnumDescriptorCache` in `SwiftDeclarationRendering` (built once per image as a *partial* map — one bad descriptor only degrades its own enum to the tagged projection). See [Documentations/Internal/LeafMigrationRegressionFixes.md](Documentations/Internal/LeafMigrationRegressionFixes.md). **SwiftSpecialization** - Runtime generic specialization (see implementation plan below) - `GenericSpecializer`, `ConformanceProvider` @@ -192,7 +192,7 @@ Printing and indexing are peers — neither depends on the other. - **MachOSymbols** - Symbol table parsing and demangling. `SymbolIndexStore`'s offset and member indexes hold their row lists in `SymbolRowBucket` (evolution proposal 0003): the dominant single-row case stays inline in the dictionary slot, only a bucket that collects a second row allocates an array; iteration order is insertion order, so query output is byte-identical to the former `[UInt32]` buckets - **MachOPointers** - Pointer types (relative, indirect, etc.) - **MachOCaches** - dyld shared cache support -- **MachOExtensions** - Extensions to MachOKit types. `resolveBind(fileOffset:)` resolves bind slots from chained fixups AND, when those are absent, from the legacy `LC_DYLD_INFO(_ONLY)` bind opcode streams (pre-macOS 12 / iOS 16 deployment targets, e.g. iOS 15.5 simulator frameworks) via a lazily built file-offset → symbol-name index; the arm64e threaded legacy format is deliberately not indexed. Pinned by `LegacyDyldInfoBindTests`, whose fixture is compiled on the fly with `-target arm64-apple-macosx11.0` to force the legacy format. +- **MachOExtensions** - Extensions to MachOKit types. `resolveBind(fileOffset:)` resolves bind slots from chained fixups AND, when those are absent, from the legacy `LC_DYLD_INFO(_ONLY)` bind opcode streams (pre-macOS 12 / iOS 16 deployment targets, e.g. iOS 15.5 simulator frameworks) via a lazily built file-offset → symbol-name index; `isBind(_:)` splits on the same discriminator so the two public APIs always agree on a slot. The opcode stream is treated as hostile input: every slot is bounds-checked against its segment's file size before recording and a repeat run terminates at the segment end (a raw uleb count can no longer spin the loop; a wrapped offset can no longer claim a foreign file offset). The arm64e threaded legacy format is deliberately not indexed. Pinned by `LegacyDyldInfoBindTests`, whose fixture is compiled on the fly with `-target arm64-apple-macosx11.0` to force the legacy format. ### Key Patterns diff --git a/Documentations/Internal/ProjectEvolutionLog.md b/Documentations/Internal/ProjectEvolutionLog.md index 5a005140..9b6bf266 100644 --- a/Documentations/Internal/ProjectEvolutionLog.md +++ b/Documentations/Internal/ProjectEvolutionLog.md @@ -345,7 +345,7 @@ - **文档**：[NodeStoreMigrationPlan.md](NodeStoreMigrationPlan.md)、 [DeclarationModelMemoryFootprint.md](DeclarationModelMemoryFootprint.md)、TaskReports [2026-07-25-node-store-override-regression-and-baselines.md](TaskReports/2026-07-25-node-store-override-regression-and-baselines.md)、 - [2026-07-25-dyld-cache-image-selection-and-rv-index-lifecycle.md](TaskReports/2026-07-25-dyld-cache-image-selection-and-rv-index-lifecycle.md)、 + [2026-07-25-cache-image-selection-and-rv-index-lifecycle.md](TaskReports/2026-07-25-cache-image-selection-and-rv-index-lifecycle.md)、 [2026-07-26-node-store-review-fixes.md](TaskReports/2026-07-26-node-store-review-fixes.md)。 --- @@ -777,6 +777,42 @@ --- +## 35. SymbolIndexStore `[UInt32]` 行号桶扁平化（evolution 提案 0003） + +- **时间段**：2026-08-09（0001 落地次日；0001「非目标」一节点名的候选正式立项）。 +- **动机**：RV 五镜像复测显示 offset / member 索引里的 `[UInt32]` 碎数组簇 38.8 MiB / 约 45 万个——绝大多数桶只有一个元素，却各自付一次堆分配与数组头。 +- **落地**：`SymbolRowBucket`（`RandomAccessCollection`）替换四处 `[UInt32]` 桶：单元素 case 内联在字典槽里，收到第二个元素才落堆数组；迭代序保持插入序，查询输出与旧桶逐字节一致。fixture 上单元素桶占比 87.6%。 +- **验证**：全量 1343 tests 全绿；渲染 A/B 七对（含 dyld cache 两对）逐字节一致。下游 RV 复测超预期：`[UInt32]` 簇 38.8 → **7.2 MiB**（预期 15–20），碎数组人口坍缩为 5 个桶字典。 +- **文档**：[Evolutions/0003-symbol-row-bucket-flattening.md](../Evolutions/0003-symbol-row-bucket-flattening.md)（提案全生命周期）。 +- **对应版本**：`0.14.1` 之后、下一次 bump 之前（`feature/node-store-migration` 分支）。 + +--- + +## 36. 声明模型 descriptor 化：不再驻留急切解析的胖 wrapper（evolution 提案 0002） + +- **时间段**：2026-08-09（与 0003 同批立项、独立实施）。 +- **动机**：0001 落地后 RV 复测把堆内新头部定位为声明模型 41.3 MiB + MachOSwiftSection 解析簇 33.4 MiB——`index()` 惰性与 wrapper 急切驻留错配：每个 `TypeDefinition` / `ExtensionDefinition` / `ProtocolDefinition` 终身抱着全量解析的 `TypeContextWrapper` / `ProtocolConformance` / `Protocol`（trailing objects 含 `[ResilientWitness]` 全在内），但索引完成后几乎无人再读。 +- **落地**：三定义改为驻留 **descriptor 引用**，全量 wrapper 用时经 `materializedTypeContext(in:)` / `materializedProtocolConformance(in:)` / `materializedProtocol(in:)` 按需重建（每操作至多一次、线程化为局部变量、绝不缓存回定义）；`parentContext` 及其 `ParentContext` 类型整体移除；实施期修正扩展到 indexer `Storage` 侧——四个人口数组在 `prepare()` 后清退、按名 keyed 重映射降级为索引期局部变量，新增名字级轻映射 `conformingProtocolNamesByTypeName` 等承接全部索引后消费者。 +- **关键决策**：物化结果不缓存（缓存会按浏览顺序把清退的内存攒回来）；wall-clock 以 release ABBA 定论——SwiftUI interface 候选反而快 5.3%，SwiftUICore 噪声带内。 +- **验证**：全量 1343 全绿；渲染 A/B 七对逐字节一致（debug 与 release 双构建）；实例尺寸 `TypeDefinition` 1272 → **384 B**、`ExtensionDefinition` 640 → **224 B**、`ProtocolDefinition` 440 → **384 B**，回归守卫 `DeclarationModelInstanceSizeTests` 钉住上限。下游 RV 五镜像复测全部达标、三项超预期：稳态 322 → **262 MB**、堆存活 283 → 209.6 MiB、解析簇 33.4 → 3.3 MiB、索引瞬态峰值 808 → 613 MB。五镜像稳态全程曲线：842 → 470–480 → ~450 → 322（0001）→ **262 MB**（0002+0003）。 +- **后记**：机械迁移漏审了读人口数组的六个公开统计属性（清退后静默归零）——由 PR #103 review 发现（H1）并在第 37 节的批次里修复；教训已记入 0002 决策日志（编译器驱动的迁移看不见「语义在、数值错」的调用面）。 +- **文档**：[Evolutions/0002-declaration-model-descriptor-slimming.md](../Evolutions/0002-declaration-model-descriptor-slimming.md)（提案全生命周期）、[DeclarationModelMemoryFootprint.md](DeclarationModelMemoryFootprint.md)（后记复量）。 +- **对应版本**：`0.14.1` 之后、下一次 bump 之前（`feature/node-store-migration` 分支）。 + +--- + +## 37. PR #103 review 修复批次：14 条发现的实现与裁决 + +- **时间段**：2026-08-09（PR #103 的 max 级 review 移交清单；B1（swift-demangling 远端 pin 缺上游 tag）由用户自行处理，不在本批次内）。 +- **动机**：`feature/node-store-migration` → `main` 的 PR review 产出 15 条经四问验证的发现；除 B1 外全部批准实施。四条共因串起十一条发现：0002 机械迁移的语义盲区（H1/H2/M1/L1）、0001 引入的裸指针与位预算（M2/M3/M5）、新二进制解码信任输入（H3/M4）、验收工具无自检（H4/L2/L4）。 +- **落地（代码修复 9 条 + 测试/工具 3 条）**：H2 扩展索引早退补 `isIndexed`；H1 统计快照在清退前冻结（`PreparationStatistics`）；M4 `isBind` 补 LC_DYLD_INFO 回退与 `resolveBind` 对齐；H3 bind 解码器按段界 bound（repeat count 挂死与 wrap 错归因关死）；M3 `PackedNameReference` 改 failable（build sweep 跳过超预算名、standalone 公开路径 clamp，release 下不再由二进制决定进程生死）；H4 A/B 验收脚本零对比即失败 + 硬失败传播；L2 fixture 编译先排空管道再等退出 + 临时目录进程退出清理；L4 offset 重建测试去 500 采样上限、排序全量；M6 per-image 缓存驱逐移交「镜像最后一个存活 indexer」（进程级登记表，修掉 A 死抽走 B 的三缓存）；M1 公开 `printExtensionHeader` 的物化失败改传播（与 `index(in:)` 契约对齐）；L1 嵌套子定义下压 per-child catch（坏子类型只丢自己）。 +- **裁决（3 条，记入 [ReviewAdjudications.md](ReviewAdjudications.md) A4–A6）**：M2 卸载后悬垂指针——实验证明 Darwin 把含 Swift 内容的镜像全部 pin 死（连无类 dylib 都不 unmap）、唯一能卸载的纯 C 镜像不产生 mapped 行，触发面结构性不存在；M5 建议的 detach 时拷贝 node store——定义自身 `node` 字段同店引用，拷贝零回收，改为文档写准 + 共享契约测试钉住；L3 建议的 achievable-rank 早退——机制不可靠（会复发跨 subcache 误解析）且全扫描实测仅 43 ms，强制配套的 plain-`.dylib` 端到端用例落地。 +- **实施期修订（留痕于清单文档）**：H3 的 `Int(segment)` trap 被推翻（segment 是 4-bit opcode immediate）；M1 的渲染路径失败被 review 会话自行推翻（库内不可达）并降级为 Low；L3 的开销估计被测量推翻。每条修复先写修复前失败的回归测试（M3 的 precondition trap、M6 的三缓存被抽、L1 的整型丢弃等均有失败实录），验证全程走本地兄弟依赖环境（B1 未决期间 CI 不可用）。 +- **文档**：[Roadmaps/2026-08-09-pr103-review-findings.md](../../Roadmaps/2026-08-09-pr103-review-findings.md)（原始清单 + 修订注记）、[ReviewAdjudications.md](ReviewAdjudications.md)（A4–A6）、[TaskReports/2026-08-09-pr103-review-fix-implementation.md](TaskReports/2026-08-09-pr103-review-fix-implementation.md)。 +- **对应版本**：`0.14.1` 之后、下一次 bump 之前（`feature/node-store-migration` 分支）。 + +--- + ## 维护约定 1. **每个非平凡批次结束时必须在本文追加/更新一节**（新工作弧新增一节；延续既有弧则在该节 diff --git a/Documentations/Internal/TaskReports/2026-08-09-pr103-review-fix-implementation.md b/Documentations/Internal/TaskReports/2026-08-09-pr103-review-fix-implementation.md new file mode 100644 index 00000000..37bc854f --- /dev/null +++ b/Documentations/Internal/TaskReports/2026-08-09-pr103-review-fix-implementation.md @@ -0,0 +1,44 @@ +# 2026-08-09 — PR #103 review 发现的修复实施 + +## 问题 + +PR #103（`feature/node-store-migration` → `main`，103 文件，+6880/−874）的 max 级 code review 产出 15 条经四问验证的发现（1 Blocker / 4 High / 6 Medium / 4 Low），由 review 会话移交本会话实施。用户批准：**B1（swift-demangling 远端 pin 指向不含所需 API 的版本区间）跳过、由用户自行处理上游发版**，其余 14 条全部修复。 + +## 调研 + +- 逐条走读清单指向的代码，其中三条的关键前提在动手前被推翻或修正（全部与 review 会话往返确认、留痕于清单文档的 revision 注记）： + - **M1**：建议修法（`try?` 改路由到 thrown-resolution 分支）与现状行为完全重合，写不出失败先行的测试；review 会话复核后进一步发现该 `try?` 在库内不可达（唯一调用点之前必然先走传播式 `index(in:)`），发现降级为 Low，修法改定为「公开入口裸 `try` 传播 + 直调公开 API 的契约测试」。 + - **H3**：`Int(segment)` trap 一半被推翻——MachOKit 从 opcode 的 4-bit immediate 解码 segment index（≤15），uleb 只用于 offset；真实威胁面是原始 uleb 的 repeat count（2^40 自旋挂死）与 wrap 后错归因。 + - **L3**：建议的 achievable-rank 早退不可靠（持有 dylib 命中时未扫描的 subcache 仍可能藏着 framework 本体，提前停复发当年 SwiftUI→axbundle 误解析）；实测全扫描仅 43 ms，推翻「数千次构造」的代价估计。 +- **M2 的开放半问（有没有消费者真的卸载被索引的镜像）用实验闭合**：macOS 26 上 dlopen/dlclose 探针证明含 Swift 内容的镜像（连无类的都算）被 dyld pin 死永不 unmap；唯一能卸载的纯 C dylib 不含 Swift 前缀符号名、不产生 mapped 行。 +- **M5 的建议修法（detach 时拷出 node store）读码推翻**：存储该 symbol 的定义自身 `node` 字段就是同店 `NodeReference`（记录在案的 per-image recycling model），拷贝零回收。 + +## 最终方案 + +- **代码修复 11 条**：H1（统计快照 `PreparationStatistics` 清退前冻结）、H2（扩展索引早退补 `isIndexed`）、H3（bind 解码器段界 bound）、H4（A/B 脚本零对比即失败 + 硬失败传播）、M1（`printExtensionHeader` 物化失败传播）、M3（`PackedNameReference` failable 化，sweep 跳过 / standalone clamp）、M4（`isBind` 补 LC_DYLD_INFO 回退）、M6（per-image 缓存驱逐移交最后存活 indexer 的进程级登记表）、L1（嵌套子定义 per-child catch）、L2（fixture 编译管道先排空 + 目录清理）、L4（测试去采样上限、排序全量）。 +- **裁决 3 条**（[ReviewAdjudications.md](../ReviewAdjudications.md) A4–A6）：M2 不修（触发面结构性不可达，两个建议缓解均拒绝）；M5 按建议不修（改为文档写准 + `storedDeclarationSymbolsShareTheDefinitionsNodeStore` 钉住共享契约）；L3 优化不做（机制不可靠 + 代价噪声级，强制配套测试落地）。 +- 每条修复先写修复前失败的回归测试，与代码同 commit 落盘、永久保留。 + +## 实际执行 + +按清单建议的批次顺序推进（B1 跳过后共 6 批、12 个 commit）： + +1. 清单文档入库（`0f0f45f8`，Roadmaps/ 既有惯例）→ H2（`dcf8b0e4`）→ H1（`d07257fc`，同批补 evolution 0002 源码兼容性补记 + 决策日志行）→ M4（`b3bffa0d`）。 +2. H3（`c36a3a2e`，segment-imm 推翻记入清单；恶意流回归测试按用户指示跳过，合法路径由 4 个 `LegacyDyldInfoBindTests` 钉住）→ M3（`711304f3`，修复前 precondition trap 实录：`symbol name byte length exceeds the 22-bit budget` 杀掉 runner）。 +3. H4（`c07b7189`，修复前空输出根实测返回 0 → 假绿）→ L2（`d738ed41`）→ L4（`9036c72b`）。 +4. M6（`b7308b21`，修复前实录：先亡 indexer 抽走存活 indexer 的全部三缓存；新增 `SymbolTestsHelper` fixture case 隔离并行套件）→ M5/M2 裁决（`83fca3eb`）。 +5. M1（`a6dbf980`，修复前实录：期望抛错但没抛）→ L1（`351b5ee9`，修复前实录：子类型 `.offsetOutOfBounds` 逃逸丢弃整个父类型；测试第一版误用进程内 `Struct(descriptor:)` 重载把 offset 当指针解引用 SIGSEGV，改用 raw-descriptor 注入后干净抛错）→ L3（`3a9a373f`）。 +6. 文档收尾（本报告所在 commit）：ProjectEvolutionLog 补 0002/0003/本批次三节 + 修 TaskReports 坏链接；AGENTS.md 同步。 + +## 验证 + +- 每条修复：失败先行 → 修复 → 转绿（各 commit message 记录失败形态）。 +- 快照 / 打印面：175 tests / 21 suites 全绿（L1 的 builder 重构健康路径逐字节不变）。 +- 收尾全量 `swift test --skip IntegrationTests`（本地兄弟依赖 + worktree 专属 scratch）全绿——B1 未决期间 CI 不可用，全量结果见收尾 commit。 + +## 偏差 + +- **H3 的恶意流回归测试按用户指示跳过**（「继续，跳过这个测试」）——加固逻辑无新增测试直接落地，合法解码路径由既有 4 测试钉住；留痕于清单 H3 revision 注记与 commit message。 +- **三条按裁决而非按建议修**（M2/M5/L3，见上）；裁决与复审条件全部留档 A4–A6。 +- **一次误用探针**：L1 测试第一版用错 `Struct(descriptor:)` 无上下文重载（进程内语义）导致测试自身 SIGSEGV，与被测路径无关；改用 `TypeDefinition` 的 raw-descriptor package init 后按预期抛错。 +- 全程在 `feature/node-store-migration` worktree 实施；`--filter "DyldCache"` 曾一度误匹配进 IntegrationTests 的维护者套件（其自身崩溃与本批次无关），随即改用精确过滤复验。 diff --git a/Roadmaps/2026-08-09-pr103-review-findings.md b/Roadmaps/2026-08-09-pr103-review-findings.md index 6d03bdda..d4dc72d1 100644 --- a/Roadmaps/2026-08-09-pr103-review-findings.md +++ b/Roadmaps/2026-08-09-pr103-review-findings.md @@ -4,7 +4,7 @@ Review date: 2026-08-09 PR: `feature/node-store-migration` → `main` (103 files, +6880/−874, merge base `a8968fa5`) Review depth: automated multi-agent review at `max` effort, followed by a manual pass that answered the four mandatory questions (reproduce / baseline / worth fixing / fixed before) for every surviving finding against the branch, `main`, CI logs, and `git log`. -Status: **Recorded and verified, not yet fixed.** +Status: **Implemented (2026-08-09)** — every finding except B1 is either fixed with a pre-fix-failing regression test or adjudicated with recorded reasoning (`Documentations/Internal/ReviewAdjudications.md` A4–A6); B1 is deliberately left to the user (upstream release). Per-finding revision notes below record where implementation refuted or reshaped the original write-up (H3, M1, M2, M5, L3). Execution retrospective: `Documentations/Internal/TaskReports/2026-08-09-pr103-review-fix-implementation.md`. All file/line references are against `feature/node-store-migration` at `8faff275`. "Baseline" means `main` at `a8968fa5`. "Upstream" means the `swift-demangling` package. From cd8a62936cd6ab0deb8598748ee2e58e652ad1bf Mon Sep 17 00:00:00 2001 From: Mx-Iris Date: Mon, 10 Aug 2026 20:41:09 +0800 Subject: [PATCH 71/77] chore: follow the MachOExtensions extraction after rebasing onto 0.15.0 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit main extracted the in-repo `MachOExtensions` target into the upstream `MachOKitExtensions` package (so `MachOObjCSection` can depend on it too without a package-level cycle). Rebasing this branch onto that resolved the two modify/delete conflicts by taking main's deletion, which left three `import MachOExtensions` sites and two test-target dependencies pointing at a module this repo no longer has. The four behaviors this branch had landed in those two deleted files — cross-subcache dyld image ranking, the legacy LC_DYLD_INFO(_ONLY) bind index, its hostile-input bounds check (PR #103 finding H3), and isBind's matching fallback (M4) — were ported into MachOKitExtensions first, from this branch's tip state rather than commit by commit, so dropping the intermediate hunks during the rebase loses nothing. `DyldCacheImageSearchTests` drops `@testable`: `matchRank` / `bestMatchRank` are public upstream, and an external package dependency is not built with testability anyway. Docs updated in the same batch. AGENTS.md was already stale on main (the extraction commit did not touch it) and this branch had added the LC_DYLD_INFO paragraph to the same bullet, so it now described a module this repo does not contain; it is rewritten as the upstream package plus the two behaviors this repo's tests still pin. Two implementation notes had dead `Sources/MachOExtensions/` paths. Historical records (task reports, review findings, changelogs, proposals) are left as written. Verified: MachOKitExtensions builds standalone; this repo's suite passes with exit code 0 (1408 tests, 264 suites, IntegrationTests skipped), built against the local siblings with --manifest-cache none — SwiftPM's cached manifest evaluation otherwise silently keeps resolving the upstream package remotely. --- AGENTS.md | 8 ++- .../Internal/ProjectEvolutionLog.md | 13 ++++ .../Internal/StaticFieldOffsetComputation.md | 4 +- .../Internal/StaticLayoutDependencyClosure.md | 4 +- ...10-rebase-onto-0.15.0-and-upstream-port.md | 59 +++++++++++++++++++ .../InternedNodeReferenceCache.swift | 2 +- .../DyldCacheImageSearchTests.swift | 2 +- .../LegacyDyldInfoBindTests.swift | 2 +- 8 files changed, 84 insertions(+), 10 deletions(-) create mode 100644 Documentations/Internal/TaskReports/2026-08-10-rebase-onto-0.15.0-and-upstream-port.md diff --git a/AGENTS.md b/AGENTS.md index 9987a024..41bfd231 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -69,8 +69,8 @@ swift-section (CLI) └── MachOFoundation └── MachOSymbols, MachOPointers └── MachOReading, MachOResolving - └── MachOExtensions, MachOCaches - └── MachOKit (external) + └── MachOCaches + └── MachOKitExtensions (external), MachOKit (external) ``` `SwiftLayout` (static field-offset engine) is a peer that depends on @@ -192,7 +192,9 @@ Printing and indexing are peers — neither depends on the other. - **MachOSymbols** - Symbol table parsing and demangling. `SymbolIndexStore`'s offset and member indexes hold their row lists in `SymbolRowBucket` (evolution proposal 0003): the dominant single-row case stays inline in the dictionary slot, only a bucket that collects a second row allocates an array; iteration order is insertion order, so query output is byte-identical to the former `[UInt32]` buckets - **MachOPointers** - Pointer types (relative, indirect, etc.) - **MachOCaches** - dyld shared cache support -- **MachOExtensions** - Extensions to MachOKit types. `resolveBind(fileOffset:)` resolves bind slots from chained fixups AND, when those are absent, from the legacy `LC_DYLD_INFO(_ONLY)` bind opcode streams (pre-macOS 12 / iOS 16 deployment targets, e.g. iOS 15.5 simulator frameworks) via a lazily built file-offset → symbol-name index; `isBind(_:)` splits on the same discriminator so the two public APIs always agree on a slot. The opcode stream is treated as hostile input: every slot is bounds-checked against its segment's file size before recording and a repeat run terminates at the segment end (a raw uleb count can no longer spin the loop; a wrapped offset can no longer claim a foreign file offset). The arm64e threaded legacy format is deliberately not indexed. Pinned by `LegacyDyldInfoBindTests`, whose fixture is compiled on the fly with `-target arm64-apple-macosx11.0` to force the legacy format. +- **MachOKitExtensions** (external package, `../MachOKitExtensions`) - Extensions to MachOKit types. This used to be an in-repo `MachOExtensions` target; it was extracted so `MachOObjCSection` can depend on it too (MachOSwiftSection depends on MachOObjCSection, so the ObjC side could never depend back on an in-package target without a package-level cycle). Two behaviors this repo's tests still pin live there: + - `resolveBind(fileOffset:)` resolves bind slots from chained fixups AND, when those are absent, from the legacy `LC_DYLD_INFO(_ONLY)` bind opcode streams (pre-macOS 12 / iOS 16 deployment targets, e.g. iOS 15.5 simulator frameworks) via a lazily built file-offset → symbol-name index; `isBind(_:)` splits on the same discriminator so the two public APIs always agree on a slot. The opcode stream is treated as hostile input: every slot is bounds-checked against its segment's file size before recording and a repeat run terminates at the segment end (a raw uleb count can no longer spin the loop; a wrapped offset can no longer claim a foreign file offset). The arm64e threaded legacy format is deliberately not indexed. Pinned by `LegacyDyldInfoBindTests`, whose fixture is compiled on the fly with `-target arm64-apple-macosx11.0` to force the legacy format. + - `DyldCacheImageSearchMode.matchRank(forImagePath:)` ranks name lookups instead of taking the first hit — leaf names are not unique inside a shared cache (`SwiftUI.framework/SwiftUI` vs `SwiftUI.axbundle/SwiftUI`, and a macOS cache's Mac Catalyst builds under `/System/iOSSupport` share the native build's leaf name). Ranks accumulate across every cache file (main plus subcaches, each scanned once) so a low-ranked hit in the first file cannot shadow the framework binary in a subcache; only a native canonical framework reaches `bestMatchRank`, which is what makes the early exit sound. Pinned by `DyldCacheImageSearchTests`. ### Key Patterns diff --git a/Documentations/Internal/ProjectEvolutionLog.md b/Documentations/Internal/ProjectEvolutionLog.md index 9b6bf266..e27f2637 100644 --- a/Documentations/Internal/ProjectEvolutionLog.md +++ b/Documentations/Internal/ProjectEvolutionLog.md @@ -813,6 +813,19 @@ --- +## 38. rebase 到 0.15.0：跟随 `MachOExtensions` 抽包，把分支成果移植上游 + +- **时间段**：2026-08-10。 +- **动机**：`main` 走到 `0.15.0`，其中 `6550d22d` 把整个 `Sources/MachOExtensions/` 模块抽到上游独立包 `MachOKitExtensions`（抽出去是为了让 `MachOObjCSection` 也能依赖它——本包依赖 `MachOObjCSection`，ObjC 侧无法反向依赖包内 target，否则构成包级循环），同时 `OutputTransformer` 改名 `SwiftOutputTransformer`。`feature/node-store-migration` 的 70 个 commit 需要 rebase 到新 `main`，但两边的文件重叠里有两个是**被 main 删掉、被分支修改**的：`MachOExtensions/MachOFile+.swift` 与 `DyldCache+.swift`。逐字节比对上游包后确认它停在抽取时的状态，也就是说直接按「接受 main 的删除」解冲突，会静默丢掉分支上五个 commit 的成果。 +- **落地**：**先移植上游、再 rebase**。移植四块改动到 `MachOKitExtensions`——dyld cache 跨 subcache 的 `matchRank` 排序与 Catalyst 支持根降级（`17ad4358` + `6647359e`）、legacy `LC_DYLD_INFO(_ONLY)` bind 索引（`5c74ad67`）、bind 解码器按段界 bound（`c36a3a2e`，PR #103 的 H3）、`isBind` 补同源回退（`b3bffa0d`，M4）。移植按**分支 tip 的文件状态**做，不逐 commit 搬，因此 rebase 时中间 commit 的 hunk 可以安全丢弃。适配上游的两处差异：访问级 `package` → `public`；上游不依赖 `FoundationToolbox`，路径取叶名改用 Foundation 的 `URL` 惯用法。随后 rebase 71 个 commit，7 处 modify/delete 冲突一律按删除解，另有三处 `import MachOExtensions` 改名、两个测试 target 依赖从 `.target(.MachOExtensions)` 换成 `.product(.MachOKitExtensions)`。 +- **关键决策**：`DyldCacheImageSearchTests` 的 `@testable import` 降级为普通 `import`——`matchRank` / `bestMatchRank` 在上游是 `public`，而外部包依赖本就不以 testability 构建，`@testable` 在这个位置既不必要也不可行。 +- **验证**：上游包单独 `swift build` 通过；本仓库全量 **1408 tests / 264 suites、退出码 0**（`--skip IntegrationTests`）。验证过程中踩到两个已知环境陷阱并记录：`.claude/worktrees/` 下缺 `MachOKitExtensions` 软链（静默回落远端 `0.1.0`）、`swift-semantic-string` 软链指向的兄弟 worktree 还没有 `OutputTransformer` product；另外 SwiftPM 的 manifest 求值缓存会让改过软链后的解析结果不刷新，需要 `--manifest-cache none` 才能真正切到本地包。 +- **遗留**：上游 `MachOKitExtensions` 的移植尚未提交发版，远端仍是 `0.1.0`；`swift-demangling` 的 `0.5.1` tag 不含 `SharedNodeStore` 与 `NodeStoreBuilder.reserveCapacity(expectedSymbolCount:)`，PR #103 的 B1 仍未解。两者都决定了 CI 依旧不可用，本地验证只能走 `USING_LOCAL_DEPENDENCIES=1` + 兄弟目录。 +- **文档**：[TaskReports/2026-08-10-rebase-onto-0.15.0-and-upstream-port.md](TaskReports/2026-08-10-rebase-onto-0.15.0-and-upstream-port.md)。 +- **对应版本**：`0.15.0` 之后、下一次 bump 之前（`feature/node-store-migration` 分支）。 + +--- + ## 维护约定 1. **每个非平凡批次结束时必须在本文追加/更新一节**（新工作弧新增一节；延续既有弧则在该节 diff --git a/Documentations/Internal/StaticFieldOffsetComputation.md b/Documentations/Internal/StaticFieldOffsetComputation.md index 0dd067ea..2bea8fea 100644 --- a/Documentations/Internal/StaticFieldOffsetComputation.md +++ b/Documentations/Internal/StaticFieldOffsetComputation.md @@ -103,7 +103,7 @@ resilient 的本质是「编译当前二进制时不知道字段布局，但运 ### 已有基础设施 -- **按依赖名解析镜像**：`Sources/MachOExtensions/DyldCache+.swift` 的 `machOFile(by mode:)` —— 按 install name / image name 从 dyld shared cache 捞 `MachOFile`。系统 Swift 库、Foundation、SwiftUI 等都在 cache 里，这条路已通。 +- **按依赖名解析镜像**：上游包 `MachOKitExtensions` 的 `DyldCache+.swift` 的 `machOFile(by mode:)` —— 按 install name / image name 从 dyld shared cache 捞 `MachOFile`。系统 Swift 库、Foundation、SwiftUI 等都在 cache 里，这条路已通。 - **符号索引**：`Sources/MachOSymbols/SymbolIndexStore.swift` —— 按 name/offset 建索引。 - **ReadingContext 抽象**：`MachOContext` 已把「从哪个镜像读」参数化，扩展成多镜像顺理成章。 @@ -238,7 +238,7 @@ fixed-layout 非泛型（单镜像） 直接读 vector，几乎零成本 | MetadataInitialization 分流信号 | `Models/Type/TypeContextDescriptorFlags.swift`、`Models/Metadata/MetadataInitialization/` | | enum 静态布局算法 | `Sources/SwiftInspection/EnumLayoutCalculator.swift`、`SpareBitAnalyzer.swift`、`BitMask.swift` | | 现有 field rendering（runtime 依赖点） | `Sources/SwiftDeclarationRendering/FieldLayoutRenderer.swift`（`:98-110` fieldOffsets、`:189-203` substitution、`:388-420` 节点替换骨架、`:497` deref runtime metadata vector）、`FieldLayoutRenderer+Enum.swift:113-148`（payloadSize/XI 全靠 runtime VWT） | -| dyld cache 取镜像 / 符号索引 | `Sources/MachOExtensions/DyldCache+.swift`（`machOFile(by:)`）、`Sources/MachOSymbols/SymbolIndexStore.swift` | +| dyld cache 取镜像 / 符号索引 | 上游包 `MachOKitExtensions` 的 `DyldCache+.swift`（`machOFile(by:)`）、`Sources/MachOSymbols/SymbolIndexStore.swift` | | InProcess-only runtime 桥 | `Sources/MachOSwiftSection/Runtime/RuntimeFunctions.swift` | | 泛型 specialization（runtime 编排，可作 ground truth/fallback） | `Sources/SwiftSpecialization/GenericSpecializer.swift`、`ConformanceProvider.swift`（已静态、可复用） | | ObjC ivar / class_ro_t | MachOObjCSection `ObjCIvarListProtocol`、`ClassROData`；本项目用例 `Sources/SwiftInspection/ClassHierarchyDumper.swift`、`Sources/TypeIndexing/ObjCInterfaceIndexer.swift` | diff --git a/Documentations/Internal/StaticLayoutDependencyClosure.md b/Documentations/Internal/StaticLayoutDependencyClosure.md index 8ac2a9cf..a521fa29 100644 --- a/Documentations/Internal/StaticLayoutDependencyClosure.md +++ b/Documentations/Internal/StaticLayoutDependencyClosure.md @@ -127,7 +127,7 @@ ObjC 祖先（`ObjCMembersTest` / `ObjCBridge`）：接 MachOObjCSection 读 `cl ## 关键文件 - 复用：`Sources/SwiftInterface/SwiftInterfaceBuilderDependencies.swift`（依赖解析两条路径）、`Sources/SwiftInterface/DependencyPath.swift` -- 复用：`Sources/MachOExtensions/DyldCache+.swift`（`machOFile(by:)`、bare-name 匹配）、`MachORepresentableWithCache.swift`（`imagePath` / `cache`） +- 复用：上游包 `MachOKitExtensions` 的 `DyldCache+.swift`（`machOFile(by:)`、bare-name 匹配）、`MachORepresentableWithCache.swift`（`imagePath` / `cache`） - 改动：`Sources/SwiftLayout/ImageUniverse.swift`、`Sources/SwiftLayout/ImageReference.swift`（**仅这两个** + 新增便利工厂文件） - 不动：`StaticTypeLayoutResolver.swift`、`BasicLayout.swift`、`ExistentialLayoutBridge.swift`、`EnumLayoutBridge.swift` - runtime 参照：`/Volumes/SwiftProjects/swift-project/swift/stdlib/public/runtime/Metadata.cpp:3767-3830`（class 字段布局 + Swift/ObjC 父类分派） @@ -148,6 +148,6 @@ ObjC 祖先（`ObjCMembersTest` / `ObjCBridge`）：接 MachOObjCSection 读 `cl 6. **resilient 验证只能走字面值。** 计划首选「读 `…Wvd` field-offset global」。实测 `ResilientChild`/`ResilientObjCStubChild` 这类 resilient 子类**根本不 emit `…Wvd`**（偏移纯运行时计算），runtime vector 也为空。故采用计划的 option 2（字面值锁定），但字面值由跨模块父类的静态 instanceSize 推导（`ResilientChild.extraField = 24`、`ResilientObjCStubChild.stubField = 16`），并辅以 `DistributedActorTest` 对非空 runtime vector `[16, 112, 128]` 的**自动**逐字段校验。 -7. **`DependencyPath` 未复用，改本地 `LayoutDependencySearchPath`。** `DependencyPath` 在 `SwiftInterface`（上层 orchestrator），`SwiftLayout` 依赖它会造成层级倒置。新增的 `LayoutDependencySearchPath`（`.machOFile` / `.dyldSharedCache` / `.systemDyldSharedCache`）是 SwiftLayout 本地等价物。`SwiftLayout` 仅新增对 `MachOExtensions` 的依赖（复用 `File.loadFromFile` / `machOFile(by:)`）。 +7. **`DependencyPath` 未复用，改本地 `LayoutDependencySearchPath`。** `DependencyPath` 在 `SwiftInterface`（上层 orchestrator），`SwiftLayout` 依赖它会造成层级倒置。新增的 `LayoutDependencySearchPath`（`.machOFile` / `.dyldSharedCache` / `.systemDyldSharedCache`）是 SwiftLayout 本地等价物。`SwiftLayout` 仅新增对 `MachOKitExtensions`（当时还是仓库内的 `MachOExtensions` target）的依赖（复用 `File.loadFromFile` / `machOFile(by:)`）。 8. **典型决策保持。** 镜像同构（按 root 类型 `MachOImage`/`MachOFile` 各自闭包，无类型擦除）、求解器零改动（只经两个 seam）、降级语义保持（定位不到的依赖按字段降级不 panic）均如计划落地。 diff --git a/Documentations/Internal/TaskReports/2026-08-10-rebase-onto-0.15.0-and-upstream-port.md b/Documentations/Internal/TaskReports/2026-08-10-rebase-onto-0.15.0-and-upstream-port.md new file mode 100644 index 00000000..47441aa6 --- /dev/null +++ b/Documentations/Internal/TaskReports/2026-08-10-rebase-onto-0.15.0-and-upstream-port.md @@ -0,0 +1,59 @@ +# 2026-08-10 — rebase 到 0.15.0，并把分支成果移植到上游 MachOKitExtensions + +## 问题 + +用户要求把 `feature/node-store-migration` rebase 到最新的 `main`，并删掉 PR 用的 worktree。 + +`main` 已经从 rebase 基点 `a8968fa5` 走到 `aa38ff50`（`release: 0.15.0`），中间四个 commit 做了两件结构性的事： + +1. `6550d22d` 把整个 `Sources/MachOExtensions/`（19 个文件）抽到上游独立包 `MachOKitExtensions`，本仓库改为依赖它；抽出去的理由写在 `Package.swift` 的依赖声明注释里——本包依赖 `MachOObjCSection`，ObjC 侧无法反向依赖包内 target，否则构成包级循环，所以这层扩展必须独立成包。 +2. 同一批把 `OutputTransformer` 改名为 `SwiftOutputTransformer`，其中通用部分上移到 `swift-semantic-string`。 + +## 调研 + +先把冲突面量清楚，而不是直接开 rebase： + +- 分支自 merge base 起有 70 个 commit；两边改动的**文件重叠只有 5 个**：`Package.swift`、`Sources/MachOSymbols/Symbol.swift`、`Sources/MachOSymbols/SymbolIndexStore.swift`、`Sources/MachOExtensions/MachOFile+.swift`、`Sources/MachOExtensions/DyldCache+.swift`。 +- 两个 `MachOSymbols` 文件上 `main` 只改了一行 import（`MachOExtensions` → `MachOKitExtensions`），属于最轻的冲突。 +- 两个 `MachOExtensions` 文件是 **modify/delete**：`main` 删了整个模块，分支改了它们。 +- 关键一步是**逐字节比对上游包**：拉下 `MachOKitExtensions` 的 `MachOFile+.swift`，与 merge base 的版本**完全一致**；`DyldCache+.swift` 也只差访问级（`package` → `public`）和一处 `URL` 写法。也就是说上游停在抽取当时的状态，分支上这五个 commit 的成果上游一件都没有： + - `17ad4358` + `6647359e`：dyld cache 镜像匹配排序（跨 subcache 累积，修的是 axbundle 抢赢 framework 本体） + - `5c74ad67`：legacy `LC_DYLD_INFO(_ONLY)` bind 支持 + - `c36a3a2e`：bind 解码器按段界 bound（PR #103 的 H3） + - `b3bffa0d`：`isBind` 补同源回退（M4） + +结论：直接按「接受 main 的删除」解冲突，会静默丢掉这五个 commit。这一点提交给用户裁决，用户选择「先移植上游再 rebase」。 + +顺带核了 PR #103 的 B1：上游 `swift-demangling` 确实已经打出 `0.5.1` tag，但 tag 内容里 `Sources/Demangling/Store/` 只有 `NodeStore.swift` 和 `NodeStoreBuilder.swift`，既没有 `SharedNodeStore`，`NodeStoreBuilder` 里也没有 `reserveCapacity(expectedSymbolCount:)`。**tag 号对上了，内容没跟上，B1 未解。** + +## 最终方案 + +1. 把四块改动移植到 `MachOKitExtensions`，移植以**分支 tip 的文件状态**为准（不逐 commit 搬）——这样 rebase 时中间 commit 的 hunk 全部可以安全丢弃。 +2. rebase 71 个 commit，两个 `MachOExtensions` 文件一律按 `main` 的删除解。 +3. 收尾遗留引用：三处 `import MachOExtensions` 改名；两个测试 target 的依赖从 `.target(.MachOExtensions)` 换成 `.product(.MachOKitExtensions)`。 +4. 同批次更新文档。 + +## 实际执行 + +**移植（上游包）**。`MachOFile+.swift` 因为与 merge base 一致，直接套用分支的完整 delta。`DyldCache+.swift` 需要两处适配：访问级 `package` → `public`；上游不依赖 `FoundationToolbox`，没有 `String.lastPathComponent` / `deletingPathExtension` 扩展，改用 Foundation 的 `URL` 惯用法取叶名，与上游既有写法一致。上游包单独 `swift build` 通过。 + +**rebase**。71 个 commit，7 处 modify/delete 冲突（`17ad4358`、`6647359e`、`337600ab`、`48cd362f`、`5c74ad67`、`b3bffa0d`、`c36a3a2e`），一律 `git rm`；两处一行 import 冲突取 `main` 的模块名 + 分支的 `@_spi(Internals)`。中途遇到一次瞬时 `index.lock` 冲突（外部进程持有，随即消失），`--continue` 即恢复。rebase 前建了 `backup/pre-main-rebase-2026-08-10` 作为后路，且 `origin` 上仍是 rebase 前的 `aa0a4128`。 + +**测试环境**。踩到三个坑，都值得记： + +1. `.claude/worktrees/` 下缺 `MachOKitExtensions` 软链，本地依赖静默回落远端 `0.1.0`（不含移植）。 +2. `swift-semantic-string` 的软链指向的兄弟 worktree 还没有 `OutputTransformer` product，导致 manifest 直接报错。 +3. **SwiftPM 的 manifest 求值缓存**：补好软链后 `swift build` 仍然解析到远端，`workspace-state.json` 里 `machokitextensions` 一直是 `remoteSourceControl`；单独跑一次 `swift package resolve --manifest-cache none` 能翻成 `fileSystem`，但下一次不带该 flag 的 `swift test` 又翻回去。最终对每条构建/测试命令都加 `--manifest-cache none`，并用 `description.json` 里的源文件路径确认编的确实是本地包，才拿到可信结果。 + +**验证**。上游包 `swift build` 通过；本仓库 `swift test --skip IntegrationTests` **退出码 0，1408 tests / 264 suites 全通过**（按项目约定只认原始退出码，不认 xcsift 摘要）。 + +## 与方案的差异 + +无功能性差异。相对预估多做了一件事：`AGENTS.md` 在 `main` 上就已经因为抽包而过时（依赖图和模块列表仍写着 `MachOExtensions`，而 `main` 那批没有同步文档），分支又把 `LC_DYLD_INFO` 的说明写进了同一条目，rebase 后等于指向一个本仓库已不存在的模块。因此把该条目改写为「上游包 + 本仓库测试仍钉住的两条行为」，并顺带修正了 `StaticFieldOffsetComputation.md` / `StaticLayoutDependencyClosure.md` 里指向 `Sources/MachOExtensions/` 的失效路径。历史性文档（任务报告、review 清单、changelog、提案）保持原貌不改。 + +## 遗留 + +- 上游 `MachOKitExtensions` 的移植**尚未提交发版**：本会话的 git 操作被 worktree 隔离限制在本仓库内，无法在该仓库执行 commit / tag / push。远端仍是 `0.1.0`。 +- B1 未解（`swift-demangling` 的 `0.5.1` 缺 `SharedNodeStore` 与 `reserveCapacity`）。 +- 上述两条都决定了 CI 依旧不可用，本地验证只能走 `USING_LOCAL_DEPENDENCIES=1` + 兄弟目录 + `--manifest-cache none`。 +- 分支已被改写历史，`origin/feature/node-store-migration` 仍指向 rebase 前的 `aa0a4128`；更新 PR #103 需要 force-push，按规程必须先走 fetch → `git cherry HEAD @{u}` → 实质 commit 检查，并由用户知情确认。 diff --git a/Sources/MachOSymbols/InternedNodeReferenceCache.swift b/Sources/MachOSymbols/InternedNodeReferenceCache.swift index ba89c6c8..1f644940 100644 --- a/Sources/MachOSymbols/InternedNodeReferenceCache.swift +++ b/Sources/MachOSymbols/InternedNodeReferenceCache.swift @@ -1,7 +1,7 @@ import Foundation @_spi(Internals) import Demangling import MachOKit -import MachOExtensions +import MachOKitExtensions @_spi(Internals) import MachOCaches import SwiftStdlibToolbox diff --git a/Tests/MachOCachesTests/DyldCacheImageSearchTests.swift b/Tests/MachOCachesTests/DyldCacheImageSearchTests.swift index 7a16a5dd..df079422 100644 --- a/Tests/MachOCachesTests/DyldCacheImageSearchTests.swift +++ b/Tests/MachOCachesTests/DyldCacheImageSearchTests.swift @@ -1,4 +1,4 @@ -@testable import MachOExtensions +import MachOKitExtensions import Testing import Foundation import MachOKit diff --git a/Tests/SwiftInterfaceTests/LegacyDyldInfoBindTests.swift b/Tests/SwiftInterfaceTests/LegacyDyldInfoBindTests.swift index ebb6810b..09aade77 100644 --- a/Tests/SwiftInterfaceTests/LegacyDyldInfoBindTests.swift +++ b/Tests/SwiftInterfaceTests/LegacyDyldInfoBindTests.swift @@ -1,7 +1,7 @@ import Foundation import Testing import MachOKit -import MachOExtensions +import MachOKitExtensions @testable import MachOSwiftSection @_spi(Support) @testable import SwiftInterface From 02a935f9959af27c630083aa7165554afc354862 Mon Sep 17 00:00:00 2001 From: Mx-Iris Date: Mon, 10 Aug 2026 21:18:11 +0800 Subject: [PATCH 72/77] deps: require MachOKitExtensions 0.1.1 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The dyld-cache image ranking and the legacy LC_DYLD_INFO(_ONLY) bind index were developed on this branch against the in-repo `MachOExtensions` target and ported upstream when that target was extracted; 0.1.1 is the release carrying them. This is the real floor, not a preference — `DyldCacheImageSearchTests` and `LegacyDyldInfoBindTests` do not compile against 0.1.0, which is what the inherited open `from: "0.1.0"` bound would have allowed a resolver to pick. Verified against the published tag rather than the local checkout: the two ported files at 0.1.1 are byte-identical to the copies the suite passed with, and with the local sibling symlink removed so the package resolves remotely, the suite passes at exit code 0 (1408 tests, 264 suites, IntegrationTests skipped). Local resolution and CI now agree. --- Documentations/Internal/ProjectEvolutionLog.md | 3 ++- .../2026-08-10-rebase-onto-0.15.0-and-upstream-port.md | 2 +- Package.swift | 7 ++++++- 3 files changed, 9 insertions(+), 3 deletions(-) diff --git a/Documentations/Internal/ProjectEvolutionLog.md b/Documentations/Internal/ProjectEvolutionLog.md index e27f2637..b60cbc53 100644 --- a/Documentations/Internal/ProjectEvolutionLog.md +++ b/Documentations/Internal/ProjectEvolutionLog.md @@ -820,7 +820,8 @@ - **落地**：**先移植上游、再 rebase**。移植四块改动到 `MachOKitExtensions`——dyld cache 跨 subcache 的 `matchRank` 排序与 Catalyst 支持根降级（`17ad4358` + `6647359e`）、legacy `LC_DYLD_INFO(_ONLY)` bind 索引（`5c74ad67`）、bind 解码器按段界 bound（`c36a3a2e`，PR #103 的 H3）、`isBind` 补同源回退（`b3bffa0d`，M4）。移植按**分支 tip 的文件状态**做，不逐 commit 搬，因此 rebase 时中间 commit 的 hunk 可以安全丢弃。适配上游的两处差异：访问级 `package` → `public`；上游不依赖 `FoundationToolbox`，路径取叶名改用 Foundation 的 `URL` 惯用法。随后 rebase 71 个 commit，7 处 modify/delete 冲突一律按删除解，另有三处 `import MachOExtensions` 改名、两个测试 target 依赖从 `.target(.MachOExtensions)` 换成 `.product(.MachOKitExtensions)`。 - **关键决策**：`DyldCacheImageSearchTests` 的 `@testable import` 降级为普通 `import`——`matchRank` / `bestMatchRank` 在上游是 `public`，而外部包依赖本就不以 testability 构建，`@testable` 在这个位置既不必要也不可行。 - **验证**：上游包单独 `swift build` 通过；本仓库全量 **1408 tests / 264 suites、退出码 0**（`--skip IntegrationTests`）。验证过程中踩到两个已知环境陷阱并记录：`.claude/worktrees/` 下缺 `MachOKitExtensions` 软链（静默回落远端 `0.1.0`）、`swift-semantic-string` 软链指向的兄弟 worktree 还没有 `OutputTransformer` product；另外 SwiftPM 的 manifest 求值缓存会让改过软链后的解析结果不刷新，需要 `--manifest-cache none` 才能真正切到本地包。 -- **遗留**：上游 `MachOKitExtensions` 的移植尚未提交发版，远端仍是 `0.1.0`；`swift-demangling` 的 `0.5.1` tag 不含 `SharedNodeStore` 与 `NodeStoreBuilder.reserveCapacity(expectedSymbolCount:)`，PR #103 的 B1 仍未解。两者都决定了 CI 依旧不可用，本地验证只能走 `USING_LOCAL_DEPENDENCIES=1` + 兄弟目录。 +- **上游发版**：移植已由用户于同日推送并打出 `MachOKitExtensions 0.1.1`。核验过 tag 内容与本地通过测试的副本逐字节一致，并摘掉本地软链、强制远端解析后重跑：相关 17 个测试与全量 1408 测试均退出码 0。本仓库 pin 随之从 `from: "0.1.0"` 收紧到 `from: "0.1.1"`（真实下限——两个测试套在 `0.1.0` 上无法编译），本地解析与 CI 就此一致。 +- **遗留**：`swift-demangling` 的 `0.5.1` tag 不含 `SharedNodeStore` 与 `NodeStoreBuilder.reserveCapacity(expectedSymbolCount:)`，PR #103 的 B1 仍未解——这是 CI 唯一剩下的阻塞项，本地验证仍需 `USING_LOCAL_DEPENDENCIES=1` + 兄弟目录取 `swift-demangling`。 - **文档**：[TaskReports/2026-08-10-rebase-onto-0.15.0-and-upstream-port.md](TaskReports/2026-08-10-rebase-onto-0.15.0-and-upstream-port.md)。 - **对应版本**：`0.15.0` 之后、下一次 bump 之前（`feature/node-store-migration` 分支）。 diff --git a/Documentations/Internal/TaskReports/2026-08-10-rebase-onto-0.15.0-and-upstream-port.md b/Documentations/Internal/TaskReports/2026-08-10-rebase-onto-0.15.0-and-upstream-port.md index 47441aa6..dfe2dfc6 100644 --- a/Documentations/Internal/TaskReports/2026-08-10-rebase-onto-0.15.0-and-upstream-port.md +++ b/Documentations/Internal/TaskReports/2026-08-10-rebase-onto-0.15.0-and-upstream-port.md @@ -53,7 +53,7 @@ ## 遗留 -- 上游 `MachOKitExtensions` 的移植**尚未提交发版**：本会话的 git 操作被 worktree 隔离限制在本仓库内，无法在该仓库执行 commit / tag / push。远端仍是 `0.1.0`。 +- ~~上游 `MachOKitExtensions` 的移植尚未提交发版~~ **已于同日由用户推送并打出 `0.1.1`**（本会话的 git 操作被 worktree 隔离限制在本仓库内，无法在该仓库执行 commit / tag / push，改为交付脚本由用户执行）。事后核验：tag `0.1.1` 上的两个文件与本地通过 1408 测试的副本**逐字节一致**；摘掉本地软链、强制从远端解析后，`DyldCacheImageSearchTests` / `DyldCacheEndToEndLookupTests` / `LegacyDyldInfoBindTests` 共 17 个测试全过，随后全量 1408 测试在「远端 `0.1.1`」配置下同样退出码 0。本仓库的 pin 相应从 `from: "0.1.0"` 收紧到 `from: "0.1.1"`——这是真实下限而非偏好，上述两个测试套在 `0.1.0` 上根本无法编译。`.claude/worktrees/` 下的临时软链已撤除，本地解析与 CI 一致。 - B1 未解（`swift-demangling` 的 `0.5.1` 缺 `SharedNodeStore` 与 `reserveCapacity`）。 - 上述两条都决定了 CI 依旧不可用，本地验证只能走 `USING_LOCAL_DEPENDENCIES=1` + 兄弟目录 + `--manifest-cache none`。 - 分支已被改写历史，`origin/feature/node-store-migration` 仍指向 rebase 前的 `aa0a4128`；更新 PR #103 需要 force-push，按规程必须先走 fetch → `git cherry HEAD @{u}` → 实质 commit 检查，并由用户知情确认。 diff --git a/Package.swift b/Package.swift index 2dfc0678..12ec5250 100644 --- a/Package.swift +++ b/Package.swift @@ -246,7 +246,12 @@ extension Package.Dependency { ), remote: .package( url: "https://github.com/MxIris-Reverse-Engineering/MachOKitExtensions", - from: "0.1.0", + // 0.1.1 is the real floor for this branch, not a preference: the + // dyld-cache image ranking and the legacy LC_DYLD_INFO(_ONLY) bind + // index were developed here against the in-repo `MachOExtensions` + // target and ported upstream in that release. `DyldCacheImageSearchTests` + // and `LegacyDyldInfoBindTests` do not compile against 0.1.0. + from: "0.1.1", ), ) } From f10c157dd2bedadd937a37eb020de2640e538056 Mon Sep 17 00:00:00 2001 From: Mx-Iris Date: Thu, 13 Aug 2026 11:32:31 +0800 Subject: [PATCH 73/77] fix(scripts): fail the A/B parity gate on mismatched skip markers MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit When both sides failed but with DIFFERENT exit codes, each leg unlinked its .txt and wrote its own .skip, so the pair was invisible to both .txt globs: neither counter moved and the skip loop printed nothing. With any other framework identical the run reported "all N pairs byte-identical" and exited 0 — over a candidate that had, for instance, started trapping where the baseline merely errored. This is the same shape as the zero-pairs hole fixed earlier in this branch (a harness that cannot fail is worse than no harness), so the verdict logic now has unit tests of its own. They are standard-library only and run in seconds. --- .gitignore | 1 + Scripts/run-rendering-ab-verification.py | 22 ++- Scripts/test-run-rendering-ab-verification.py | 132 ++++++++++++++++++ 3 files changed, 153 insertions(+), 2 deletions(-) create mode 100755 Scripts/test-run-rendering-ab-verification.py diff --git a/.gitignore b/.gitignore index d33bf93c..3e28e3bb 100644 --- a/.gitignore +++ b/.gitignore @@ -22,3 +22,4 @@ PrecompiledLibraries/swift-syntax/.swiftpm/xcode/package.xcworkspace/contents.xc Tests/Projects/SymbolTests/DerivedData .omc .swiftpm/xcode/xcshareddata/xcschemes +__pycache__/ diff --git a/Scripts/run-rendering-ab-verification.py b/Scripts/run-rendering-ab-verification.py index 07a30e10..1baa63d5 100755 --- a/Scripts/run-rendering-ab-verification.py +++ b/Scripts/run-rendering-ab-verification.py @@ -270,8 +270,26 @@ def compare_all_pairs(self) -> tuple[int, int]: difference_count += 1 for skip_file in sorted(self.output_root.glob("**/baseline/*.skip")): candidate_skip = Path(str(skip_file).replace("/baseline/", "/candidate/")) - if candidate_skip.is_file() and skip_file.read_text() == candidate_skip.read_text(): - print(f"SKIPPED (both sides, {skip_file.read_text().strip()}) {skip_file.relative_to(self.output_root)}") + relative_name = skip_file.relative_to(self.output_root) + if not candidate_skip.is_file(): + # Baseline refused while the candidate produced output: the + # candidate .txt already counted as MISSING-ON-BASELINE above. + continue + baseline_marker = skip_file.read_text().strip() + candidate_marker = candidate_skip.read_text().strip() + if baseline_marker == candidate_marker: + print(f"SKIPPED (both sides, {baseline_marker}) {relative_name}") + else: + # Both sides failed, but DIFFERENTLY — e.g. the baseline exits 1 + # on a pre-existing unsupported case while the candidate traps + # (134). Neither side leaves a .txt, so this pair is invisible to + # both globs above; counting it here is what stops a + # candidate-introduced crash from being reported as a pass. Same + # class as the zero-pairs hole (PR #103 review, H4). + print(f"EXIT-CODE-DIFFERS {relative_name} " + f"baseline={baseline_marker} candidate={candidate_marker}") + examined_pair_count += 1 + difference_count += 1 return difference_count, examined_pair_count diff --git a/Scripts/test-run-rendering-ab-verification.py b/Scripts/test-run-rendering-ab-verification.py new file mode 100755 index 00000000..3aca8f30 --- /dev/null +++ b/Scripts/test-run-rendering-ab-verification.py @@ -0,0 +1,132 @@ +#!/usr/bin/env python3 +"""Unit tests for the rendering A/B verification harness's verdict logic. + +Run with: + + python3 Scripts/test-run-rendering-ab-verification.py + +Standard library only — the harness itself has no third-party dependencies and +neither do these tests, so they run anywhere the harness does. + +The harness is what AGENTS.md makes acceptance evidence for any refactor +touching demangling / printing / indexing / the reader stack, so its verdict +path is exactly the code that must not be able to pass over an incomplete +comparison. These tests pin that property directly: they drive +`compare_all_pairs` over hand-built output trees and assert the counts, rather +than running the (minutes-long, machine-dependent) real comparison. +""" + +import importlib.util +import sys +import tempfile +import unittest +from pathlib import Path +from types import SimpleNamespace + +HARNESS_PATH = Path(__file__).resolve().parent / "run-rendering-ab-verification.py" + + +def load_harness(): + specification = importlib.util.spec_from_file_location("rendering_ab_harness", HARNESS_PATH) + module = importlib.util.module_from_spec(specification) + specification.loader.exec_module(module) + return module + + +HARNESS = load_harness() + + +class CompareAllPairsTests(unittest.TestCase): + """`compare_all_pairs` reads only `self.output_root`, so a namespace stub is + a complete stand-in for a real `VerificationRun` here.""" + + def setUp(self) -> None: + self.temporary_directory = tempfile.TemporaryDirectory() + self.output_root = Path(self.temporary_directory.name) + self.baseline_directory = self.output_root / "scenario" / "baseline" + self.candidate_directory = self.output_root / "scenario" / "candidate" + self.baseline_directory.mkdir(parents=True) + self.candidate_directory.mkdir(parents=True) + + def tearDown(self) -> None: + self.temporary_directory.cleanup() + + def compare(self) -> tuple[int, int]: + stub = SimpleNamespace(output_root=self.output_root) + return HARNESS.VerificationRun.compare_all_pairs(stub) + + def writeBaselineSkip(self, framework_name: str, exit_code: int) -> None: + (self.baseline_directory / f"{framework_name}.dump.skip").write_text(f"exit={exit_code}\n") + + def writeCandidateSkip(self, framework_name: str, exit_code: int) -> None: + (self.candidate_directory / f"{framework_name}.dump.skip").write_text(f"exit={exit_code}\n") + + def writeIdenticalPair(self, framework_name: str) -> None: + (self.baseline_directory / f"{framework_name}.dump.txt").write_text("same\n") + (self.candidate_directory / f"{framework_name}.dump.txt").write_text("same\n") + + def testBothSidesFailingWithDifferentExitCodesCountsAsADifference(self) -> None: + """The regression this suite exists for. + + The baseline exits 1 on a pre-existing unsupported case while the + candidate traps (134) — a refactor-introduced crash. Both legs unlink + their .txt and write a .skip, so the pair is invisible to both .txt + globs. Before the fix the skip loop printed nothing for a mismatched + pair, leaving both counters at zero: with any other framework + identical, the harness printed `all N pairs byte-identical.` and exited + 0 over a candidate that crashed. + """ + self.writeBaselineSkip("SwiftUI", 1) + self.writeCandidateSkip("SwiftUI", 134) + + difference_count, examined_pair_count = self.compare() + + self.assertEqual(difference_count, 1) + self.assertEqual(examined_pair_count, 1) + + def testDifferingExitCodesFailARunWhoseOtherPairsAreIdentical(self) -> None: + """End-to-end shape of the same defect: one healthy pair alongside the + mismatched one must not let the run read as a pass.""" + self.writeIdenticalPair("Combine") + self.writeBaselineSkip("SwiftUI", 1) + self.writeCandidateSkip("SwiftUI", 134) + + difference_count, examined_pair_count = self.compare() + + self.assertEqual(examined_pair_count, 2) + self.assertGreater(difference_count, 0) + + def testBothSidesFailingIdenticallyIsStillASkip(self) -> None: + """A framework absent from the cache on both sides is a legitimate + skip: the fix must not turn those into false failures.""" + self.writeBaselineSkip("ActivityKit", 1) + self.writeCandidateSkip("ActivityKit", 1) + + difference_count, examined_pair_count = self.compare() + + self.assertEqual(difference_count, 0) + self.assertEqual(examined_pair_count, 0) + + def testBaselineSkipAgainstCandidateOutputIsCountedExactlyOnce(self) -> None: + """The candidate produced output where the baseline refused. The + candidate-only .txt glob already counts that as MISSING-ON-BASELINE, so + the skip loop must not double-count it.""" + self.writeBaselineSkip("WidgetKit", 1) + (self.candidate_directory / "WidgetKit.dump.txt").write_text("candidate output\n") + + difference_count, examined_pair_count = self.compare() + + self.assertEqual(difference_count, 1) + self.assertEqual(examined_pair_count, 1) + + def testIdenticalPairsPass(self) -> None: + self.writeIdenticalPair("SwiftData") + + difference_count, examined_pair_count = self.compare() + + self.assertEqual(difference_count, 0) + self.assertEqual(examined_pair_count, 1) + + +if __name__ == "__main__": + unittest.main(verbosity=2) From 6141c7536a5a56f4081f5e2b940c0d6067cb57ff Mon Sep 17 00:00:00 2001 From: Mx-Iris Date: Thu, 13 Aug 2026 11:32:55 +0800 Subject: [PATCH 74/77] fix(printing): report dropped definitions as events, not on stdout MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Issue #102 asked for three things when a definition cannot be printed: keep the partial result, dispatch a failure event, and stop writing to stdout. Only the first landed. A run that lost 8,375 definitions therefore dispatched zero definitionPrintFailed events, and its only signal was a bare `unexpected(at: 8)` on stdout — the very stream the CLI writes the generated Swift to, so the diagnostic corrupted the interface it was reporting on and, being buffered in a pipe, surfaced far from its cause. printCatchedThrowing now takes the dispatcher and a printing context, and every call site that has a definition identity supplies one. The two globals blocks deliberately do not: printVariable / printFunction are non-throwing and already catch per member, so there is no identity to attribute a block-level failure to. Sweeping for the same defect found four more library-side print(error) sites, all of them on the interface-generating path and all now on stderr. Sources/ no longer contains a single one. --- .../Extensions/Node+OpaqueType.swift | 7 +- .../MultiPayloadEnumDescriptorCache.swift | 7 +- .../SwiftInterfaceBuilder.swift | 36 +++- .../SwiftInterfaceBuilderDependencies.swift | 9 +- .../SwiftDeclarationPrinter+Members.swift | 7 +- .../SwiftDeclarationPrinter.swift | 75 ++++++-- .../PrintFailureEventTests.swift | 171 ++++++++++++++++++ 7 files changed, 282 insertions(+), 30 deletions(-) create mode 100644 Tests/SwiftInterfaceTests/PrintFailureEventTests.swift diff --git a/Sources/SwiftDeclarationRendering/Extensions/Node+OpaqueType.swift b/Sources/SwiftDeclarationRendering/Extensions/Node+OpaqueType.swift index a4868927..53034308 100644 --- a/Sources/SwiftDeclarationRendering/Extensions/Node+OpaqueType.swift +++ b/Sources/SwiftDeclarationRendering/Extensions/Node+OpaqueType.swift @@ -82,7 +82,12 @@ extension Node { } } } catch { - Swift.print(error) + // stderr, never stdout: stdout carries the generated Swift, so + // a diagnostic printed here corrupts any piped or redirected + // interface (issue #102). The un-rewritten node is returned so + // an unresolvable opaque type degrades to its own printing + // rather than failing the declaration. + FileHandle.standardError.write(Data("opaque type rewrite failed: \(error)\n".utf8)) } return node } diff --git a/Sources/SwiftDeclarationRendering/MultiPayloadEnumDescriptorCache.swift b/Sources/SwiftDeclarationRendering/MultiPayloadEnumDescriptorCache.swift index d9e26467..d1ea0b77 100644 --- a/Sources/SwiftDeclarationRendering/MultiPayloadEnumDescriptorCache.swift +++ b/Sources/SwiftDeclarationRendering/MultiPayloadEnumDescriptorCache.swift @@ -45,7 +45,12 @@ final class MultiPayloadEnumDescriptorCache: SharedCache: Sendable do { try await extraDataProvider.setup() } catch { - print(error) + // stderr, never stdout — `printRoot()`'s result is streamed to + // stdout by the CLI, so a diagnostic printed here lands inside + // the generated interface (issue #102). + FileHandle.standardError.write(Data("extra data provider setup failed: \(error)\n".utf8)) } } @@ -120,6 +124,11 @@ public final class SwiftInterfaceBuilder: Sendable public func printRoot() async throws -> SemanticString { ImportsBlock(OrderedSet(Self.internalModules + importedModules).sorted()) + // The two globals blocks carry no printing context because they cannot + // fail as a block: `printVariable` / `printFunction` are non-throwing — + // each already catches per member and dispatches its own + // `definitionPrintFailed`. These wrappers are belt-and-braces, so there + // is no definition identity to attribute a block-level failure to. await printCatchedThrowing { await BlockList { for variable in indexer.globalVariableDefinitions { @@ -143,7 +152,10 @@ public final class SwiftInterfaceBuilder: Sendable // moment a single one threw. await BlockList { for typeDefinition in indexer.rootTypeDefinitions.values { - await printCatchedThrowing { + await printCatchedThrowing( + dispatchingTo: eventDispatcher, + context: .init(name: typeDefinition.typeName.name, kind: .type) + ) { try await printer.printTypeDefinition(typeDefinition) } } @@ -157,7 +169,10 @@ public final class SwiftInterfaceBuilder: Sendable // accumulated through `specialize(with:in:)`. for typeDefinition in indexer.allTypeDefinitions.values { for specialized in typeDefinition.specializedChildren { - await printCatchedThrowing { + await printCatchedThrowing( + dispatchingTo: eventDispatcher, + context: .init(name: specialized.typeName.name, kind: .type) + ) { try await printer.printTypeDefinition(specialized) } } @@ -166,7 +181,10 @@ public final class SwiftInterfaceBuilder: Sendable await BlockList { for protocolDefinition in indexer.rootProtocolDefinitions.values { - await printCatchedThrowing { + await printCatchedThrowing( + dispatchingTo: eventDispatcher, + context: .init(name: protocolDefinition.protocolName.name, kind: .protocol) + ) { try await printer.printProtocolDefinition(protocolDefinition) } } @@ -175,7 +193,10 @@ public final class SwiftInterfaceBuilder: Sendable await BlockList { for protocolDefinition in indexer.rootProtocolDefinitions.values.filterNonNil(\.parent) { for extensionDefinition in protocolDefinition.defaultImplementationExtensions { - await printCatchedThrowing { + await printCatchedThrowing( + dispatchingTo: eventDispatcher, + context: .init(name: extensionDefinition.extensionName.name, kind: .extension) + ) { try await printer.printExtensionDefinition(extensionDefinition) } } @@ -184,7 +205,10 @@ public final class SwiftInterfaceBuilder: Sendable await BlockList { for extensionDefinition in allExtensionDefinitions { - await printCatchedThrowing { + await printCatchedThrowing( + dispatchingTo: eventDispatcher, + context: .init(name: extensionDefinition.extensionName.name, kind: .extension) + ) { try await printer.printExtensionDefinition(extensionDefinition) } } diff --git a/Sources/SwiftInterface/SwiftInterfaceBuilderDependencies.swift b/Sources/SwiftInterface/SwiftInterfaceBuilderDependencies.swift index c2fd0045..bb9ddb66 100644 --- a/Sources/SwiftInterface/SwiftInterfaceBuilderDependencies.swift +++ b/Sources/SwiftInterface/SwiftInterfaceBuilderDependencies.swift @@ -1,3 +1,4 @@ +import Foundation import SwiftDeclaration import SwiftIndexing import SwiftPrinting @@ -28,7 +29,9 @@ extension SwiftInterfaceBuilderDependencies { dependencies.append(machOFile) } else {} } catch { - print(error) + // stderr, never stdout — stdout carries the generated + // interface (issue #102). + FileHandle.standardError.write(Data("dependency load failed for \(path): \(error)\n".utf8)) } case .dyldSharedCache(let path): do { @@ -39,7 +42,9 @@ extension SwiftInterfaceBuilderDependencies { foundCount += 1 } } catch { - print(error) + // stderr, never stdout — stdout carries the generated + // interface (issue #102). + FileHandle.standardError.write(Data("dyld shared cache load failed for \(path): \(error)\n".utf8)) } case .usesSystemDyldSharedCache: if let hostDyldCache = FullDyldCache.host { diff --git a/Sources/SwiftPrinting/SwiftDeclarationPrinter+Members.swift b/Sources/SwiftPrinting/SwiftDeclarationPrinter+Members.swift index decff551..fb3cdf2e 100644 --- a/Sources/SwiftPrinting/SwiftDeclarationPrinter+Members.swift +++ b/Sources/SwiftPrinting/SwiftDeclarationPrinter+Members.swift @@ -28,7 +28,12 @@ extension SwiftDeclarationPrinter { /// `var value: A`). @SemanticStringBuilder public func printField(_ field: FieldDefinition, level: Int, substitutedTypeNode: Node? = nil) async -> SemanticString { - await printCatchedThrowing { + // A stored field is reported as `.variable` — the event vocabulary has + // no separate field kind, and a stored property is what it renders as. + await printCatchedThrowing( + dispatchingTo: eventDispatcher, + context: .init(name: field.name, kind: .variable) + ) { try await printThrowingField(field, level: level, substitutedTypeNode: substitutedTypeNode) } } diff --git a/Sources/SwiftPrinting/SwiftDeclarationPrinter.swift b/Sources/SwiftPrinting/SwiftDeclarationPrinter.swift index 2c70e788..f7c8d390 100644 --- a/Sources/SwiftPrinting/SwiftDeclarationPrinter.swift +++ b/Sources/SwiftPrinting/SwiftDeclarationPrinter.swift @@ -140,21 +140,29 @@ public final class SwiftDeclarationPrinter: Sendab // child's throw once escaped here and the top-level catch // discarded the whole enclosing type. for child in typeDefinition.typeChildren { - if let renderedChild = await printCatchedThrowing({ - try await NestedDeclaration { - try await printTypeDefinition(child, level: level + 1) + if let renderedChild = await printCatchedThrowing( + dispatchingTo: eventDispatcher, + context: .init(name: child.typeName.name, kind: .type), + { + try await NestedDeclaration { + try await printTypeDefinition(child, level: level + 1) + } } - }) { + ) { renderedChild } } for child in typeDefinition.protocolChildren { - if let renderedChild = await printCatchedThrowing({ - try await NestedDeclaration { - try await printProtocolDefinition(child, level: level + 1) + if let renderedChild = await printCatchedThrowing( + dispatchingTo: eventDispatcher, + context: .init(name: child.protocolName.name, kind: .protocol), + { + try await NestedDeclaration { + try await printProtocolDefinition(child, level: level + 1) + } } - }) { + ) { renderedChild } } @@ -203,7 +211,10 @@ public final class SwiftDeclarationPrinter: Sendab // printing throws drops only itself, not the protocol it trails. await BlockList { for extensionDefinition in protocolDefinition.defaultImplementationExtensions { - await printCatchedThrowing { + await printCatchedThrowing( + dispatchingTo: eventDispatcher, + context: .init(name: extensionDefinition.extensionName.name, kind: .extension) + ) { try await printExtensionDefinition(extensionDefinition) } } @@ -229,21 +240,29 @@ public final class SwiftDeclarationPrinter: Sendab // nested loops: a nested definition whose printing throws // drops only itself, never the whole extension. for typeDefinition in extensionDefinition.types { - if let renderedChild = await printCatchedThrowing({ - try await NestedDeclaration { - try await printTypeDefinition(typeDefinition, level: level + 1) + if let renderedChild = await printCatchedThrowing( + dispatchingTo: eventDispatcher, + context: .init(name: typeDefinition.typeName.name, kind: .type), + { + try await NestedDeclaration { + try await printTypeDefinition(typeDefinition, level: level + 1) + } } - }) { + ) { renderedChild } } for protocolDefinition in extensionDefinition.protocols { - if let renderedChild = await printCatchedThrowing({ - try await NestedDeclaration { - try await printProtocolDefinition(protocolDefinition, level: level + 1) + if let renderedChild = await printCatchedThrowing( + dispatchingTo: eventDispatcher, + context: .init(name: protocolDefinition.protocolName.name, kind: .protocol), + { + try await NestedDeclaration { + try await printProtocolDefinition(protocolDefinition, level: level + 1) + } } - }) { + ) { renderedChild } } @@ -540,11 +559,29 @@ public final class SwiftDeclarationPrinter: Sendab } } -package func printCatchedThrowing(@SemanticStringBuilder _ body: () async throws -> SemanticString) async -> SemanticString? { +/// Renders `body`, dropping only what it was rendering if it throws. +/// +/// The failure is reported as a ``SwiftIndexEvents/Event/definitionPrintFailed`` +/// event when the caller can supply a dispatcher and a context — it must NOT be +/// printed. `swift-section interface` / `dump` stream the generated Swift to +/// stdout (`InterfaceCommand.swift:106`, `DumpCommand.swift:294`), so anything a +/// library writes there lands inside the generated output and corrupts any piped +/// or redirected interface. Issue #102 reported both halves of this from the +/// field: a run that lost 8,375 definitions emitted **zero** +/// `definitionPrintFailed` events, and its only signal was a bare +/// `unexpected(at: 8)` on stdout, fully buffered and therefore surfacing far +/// from its cause. +package func printCatchedThrowing( + dispatchingTo eventDispatcher: SwiftIndexEvents.Dispatcher? = nil, + context: SwiftIndexEvents.PrintingContext? = nil, + @SemanticStringBuilder _ body: () async throws -> SemanticString +) async -> SemanticString? { do { return try await body() } catch { - print(error) + if let eventDispatcher, let context { + eventDispatcher.dispatch(.definitionPrintFailed(context: context, error: error)) + } return nil } } diff --git a/Tests/SwiftInterfaceTests/PrintFailureEventTests.swift b/Tests/SwiftInterfaceTests/PrintFailureEventTests.swift new file mode 100644 index 00000000..694ca3f5 --- /dev/null +++ b/Tests/SwiftInterfaceTests/PrintFailureEventTests.swift @@ -0,0 +1,171 @@ +@_spi(Support) @testable import SwiftDeclaration +@_spi(Support) @testable import SwiftIndexing +@_spi(Support) @testable import SwiftPrinting +@_spi(Support) @testable import SwiftInterface +import Foundation +import Testing +import MachOKit +import Dependencies +@testable import MachOSwiftSection +@testable import MachOTestingSupport +import MachOFixtureSupport + +/// A definition that cannot be printed must be **reported**, not printed away. +/// +/// Issue #102 measured both halves of this on a real binary: a run that dropped +/// 8,375 definitions dispatched **zero** `definitionPrintFailed` events, and its +/// only signal was a bare `unexpected(at: 8)` written to **stdout** — the same +/// stream `swift-section interface` / `dump` write the generated Swift to, so +/// the diagnostic corrupted the output it was reporting on and, being fully +/// buffered in a pipe, surfaced far from its cause. +/// +/// The per-definition catch that keeps one bad definition from costing its +/// whole block is the fix's first half (already landed); these tests pin the +/// other two thirds — the failure is observable through the event stream, and +/// nothing reaches stdout. +@Suite(.serialized) +final class PrintFailureEventTests: MachOFileTests, @unchecked Sendable { + override class var fileName: MachOFileName { .SymbolTestsCore } + + /// Collects every dispatched event so a test can assert on the stream. + private final class EventCollector: SwiftIndexEvents.Handler, @unchecked Sendable { + private let lock = NSLock() + private var storage: [SwiftIndexEvents.Payload] = [] + + func handle(event: SwiftIndexEvents.Payload) { + lock.lock() + defer { lock.unlock() } + storage.append(event) + } + + var events: [SwiftIndexEvents.Payload] { + lock.lock() + defer { lock.unlock() } + return storage + } + + var printFailureNames: [String] { + events.compactMap { event in + guard case .definitionPrintFailed(let context, _) = event else { return nil } + return context.name + } + } + } + + private func preparedIndexer() async throws -> SwiftDeclarationIndexer { + let indexer = SwiftDeclarationIndexer(in: machOFile) + try await indexer.prepare() + return indexer + } + + private func findTypeDefinition(named name: String, in indexer: SwiftDeclarationIndexer) -> TypeDefinition? { + indexer.allTypeDefinitions.values.first { $0.typeName.currentName == name } + } + + /// A real struct descriptor re-wrapped at an offset far past the fixture's + /// end of file: every relative resolve its indexing performs is out of + /// bounds, so printing it throws deterministically. + private func makeUnprintableDefinition( + borrowingNameFrom donorDefinition: TypeDefinition, + in indexer: SwiftDeclarationIndexer + ) throws -> TypeDefinition { + let realStructDefinition = try #require(findTypeDefinition(named: "GenericStructNonRequirement", in: indexer)) + guard case .struct(let realStructDescriptor) = realStructDefinition.typeContextDescriptorWrapper else { + throw PrintFailureEventTestError.fixtureTypeIsNotAStruct + } + let unreadableDescriptor = StructDescriptor(layout: realStructDescriptor.layout, offset: 0x0FFF_FFF0) + return TypeDefinition( + typeContextDescriptorWrapper: .struct(unreadableDescriptor), + typeName: donorDefinition.typeName, + isSpecialized: false + ) + } + + private enum PrintFailureEventTestError: Error { + case fixtureTypeIsNotAStruct + } + + @Test func droppedNestedChildDispatchesAPrintFailureEvent() async throws { + let indexer = try await preparedIndexer() + let hostDefinition = try #require(findTypeDefinition(named: "Classes", in: indexer)) + let donorDefinition = try #require(findTypeDefinition(named: "FinalClassTest", in: indexer)) + let unprintableDefinition = try makeUnprintableDefinition(borrowingNameFrom: donorDefinition, in: indexer) + hostDefinition.typeChildren.append(unprintableDefinition) + + let collector = EventCollector() + nonisolated(unsafe) let unsafeHostDefinition = hostDefinition + nonisolated(unsafe) let unsafePrinter = SwiftDeclarationPrinter( + eventHandlers: [collector], + in: machOFile + ) + + _ = try await unsafePrinter.printTypeDefinition(unsafeHostDefinition).string + + #expect( + collector.printFailureNames.contains(donorDefinition.typeName.name), + "a dropped nested child must dispatch `definitionPrintFailed`; got \(collector.printFailureNames)" + ) + } + + /// The dropped child must not reach **stdout**. Printing is driven with the + /// process's real `STDOUT_FILENO` redirected to a pipe, because that is the + /// exact channel the CLI streams the generated interface through — a + /// `print(error)` anywhere under this call lands in the interface itself. + @Test func droppedNestedChildWritesNothingToStandardOutput() async throws { + let indexer = try await preparedIndexer() + let hostDefinition = try #require(findTypeDefinition(named: "Classes", in: indexer)) + let donorDefinition = try #require(findTypeDefinition(named: "FinalClassTest", in: indexer)) + let unprintableDefinition = try makeUnprintableDefinition(borrowingNameFrom: donorDefinition, in: indexer) + hostDefinition.typeChildren.append(unprintableDefinition) + + nonisolated(unsafe) let unsafeHostDefinition = hostDefinition + nonisolated(unsafe) let unsafePrinter = SwiftDeclarationPrinter(in: machOFile) + + let capturedStandardOutput = try await captureStandardOutput { + _ = try await unsafePrinter.printTypeDefinition(unsafeHostDefinition).string + } + + #expect( + capturedStandardOutput.isEmpty, + "printing must write nothing to stdout — the CLI streams the generated interface there; captured: \(capturedStandardOutput)" + ) + } + + /// Redirects `STDOUT_FILENO` to a pipe for the duration of `body`, then + /// restores it and returns whatever was written. + private func captureStandardOutput(_ body: () async throws -> Void) async throws -> String { + let savedStandardOutput = dup(STDOUT_FILENO) + defer { close(savedStandardOutput) } + + let pipe = Pipe() + dup2(pipe.fileHandleForWriting.fileDescriptor, STDOUT_FILENO) + + var readData = Data() + // Drain concurrently: a blocked pipe would otherwise deadlock the + // writer once the buffer fills. + let drainTask = Task.detached { () -> Data in + var accumulated = Data() + while let chunk = try? pipe.fileHandleForReading.read(upToCount: 4096), !chunk.isEmpty { + accumulated.append(chunk) + } + return accumulated + } + + do { + try await body() + } catch { + fflush(stdout) + dup2(savedStandardOutput, STDOUT_FILENO) + try? pipe.fileHandleForWriting.close() + _ = await drainTask.value + throw error + } + + fflush(stdout) + dup2(savedStandardOutput, STDOUT_FILENO) + try? pipe.fileHandleForWriting.close() + readData = await drainTask.value + + return String(decoding: readData, as: UTF8.self) + } +} From 6fc8f06c0f1a7b4a7f7afa705837ce356a977921 Mon Sep 17 00:00:00 2001 From: Mx-Iris Date: Thu, 13 Aug 2026 11:33:06 +0800 Subject: [PATCH 75/77] fix(interface): drop a diff declaration whose header cannot render MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit header(_:_:) swallowed a throw into an empty SemanticString, but renderType / renderProtocol compute their body units unconditionally and hand both to DiffContainerAssembler. A type whose header failed was therefore emitted as members and braces with no `struct Foo` line above them — structurally invalid Swift, produced silently, with no event and no error. Header rendering genuinely throws: it reads the declaration's name, generic signature and superclass and demangles each (issue #102 is the field evidence that print-time DemanglingErrors happen on real binaries), and since evolution 0002 it also re-materializes the wrapper from its descriptor. The nil now means "this declaration exists but could not be rendered", which drops it whole. An absent side stays an empty string, since that is the case the added/removed markers are built on. --- .../SwiftDiffableInterfaceRenderer.swift | 36 ++++- .../DiffRendererHeaderFailureTests.swift | 124 ++++++++++++++++++ 2 files changed, 154 insertions(+), 6 deletions(-) create mode 100644 Tests/SwiftInterfaceTests/DiffRendererHeaderFailureTests.swift diff --git a/Sources/SwiftInterface/SwiftDiffableInterfaceRenderer.swift b/Sources/SwiftInterface/SwiftDiffableInterfaceRenderer.swift index 78e4fd37..13159efd 100644 --- a/Sources/SwiftInterface/SwiftDiffableInterfaceRenderer.swift +++ b/Sources/SwiftInterface/SwiftDiffableInterfaceRenderer.swift @@ -139,8 +139,12 @@ public final class SwiftDiffableInterfaceRenderer< guard old != nil || new != nil else { return [] } let marker: DiffMarker = old == nil ? .added : (new == nil ? .removed : .unchanged) - let oldHeader = await header(old) { try await oldPrinter.printTypeHeader($0, level: level) } - let newHeader = await header(new) { try await newPrinter.printTypeHeader($0, level: level) } + // A header that could not be rendered drops the whole type: emitting + // `bodyUnits` under a blank header line would produce members and + // braces with no declaration above them. + guard let oldHeader = await header(old, { try await oldPrinter.printTypeHeader($0, level: level) }), + let newHeader = await header(new, { try await newPrinter.printTypeHeader($0, level: level) }) + else { return [] } let bodyUnits = await typeBodyUnits(old: old, new: new, level: level) return DiffContainerAssembler.assemble(oldHeader: oldHeader, newHeader: newHeader, marker: marker, bodyUnits: bodyUnits, level: level) @@ -187,8 +191,11 @@ public final class SwiftDiffableInterfaceRenderer< guard old != nil || new != nil else { return [] } let marker: DiffMarker = old == nil ? .added : (new == nil ? .removed : .unchanged) - let oldHeader = await header(old) { try await oldPrinter.printProtocolHeader($0, level: level) } - let newHeader = await header(new) { try await newPrinter.printProtocolHeader($0, level: level) } + // Same contract as `renderType`: an unrenderable header drops the + // whole protocol rather than emitting its requirements bare. + guard let oldHeader = await header(old, { try await oldPrinter.printProtocolHeader($0, level: level) }), + let newHeader = await header(new, { try await newPrinter.printProtocolHeader($0, level: level) }) + else { return [] } var units: [[DiffLine]] = [] units += await diffMembers(old: associatedTypeMembers(old, printer: oldPrinter), new: associatedTypeMembers(new, printer: newPrinter), level: level) @@ -429,9 +436,26 @@ public final class SwiftDiffableInterfaceRenderer< // MARK: - Generic helpers - private func header(_ definition: EnclosingDefinition?, _ render: (EnclosingDefinition) async throws -> SemanticString) async -> SemanticString { + /// `nil` means the definition EXISTS but its header could not be rendered — + /// the caller must then drop the whole declaration, because members and + /// braces under an empty header line are not valid Swift. An absent + /// definition (a `nil` input) is NOT a failure: it is the "this side does + /// not have it" case that `.added` / `.removed` markers are built on, and + /// it renders as empty. + /// + /// Header rendering can genuinely throw — it reads the declaration's name, + /// generic signature and superclass, and demangles each (issue #102 is the + /// field evidence that print-time `DemanglingError`s happen on real + /// binaries) — plus, since evolution 0002, it re-materializes the wrapper + /// from its descriptor. Swallowing that into an empty string emitted the + /// type's members with no `struct Foo` line above them, silently. + private func header(_ definition: EnclosingDefinition?, _ render: (EnclosingDefinition) async throws -> SemanticString) async -> SemanticString? { guard let definition else { return SemanticString() } - return (try? await render(definition)) ?? SemanticString() + do { + return try await render(definition) + } catch { + return nil + } } /// Matches two element lists by an `ABIKey`, returning pairs in render order: diff --git a/Tests/SwiftInterfaceTests/DiffRendererHeaderFailureTests.swift b/Tests/SwiftInterfaceTests/DiffRendererHeaderFailureTests.swift new file mode 100644 index 00000000..4ea461a8 --- /dev/null +++ b/Tests/SwiftInterfaceTests/DiffRendererHeaderFailureTests.swift @@ -0,0 +1,124 @@ +@_spi(Support) @testable import SwiftDeclaration +@_spi(Support) @testable import SwiftIndexing +@_spi(Support) @testable import SwiftPrinting +@_spi(Support) @testable import SwiftInterface +import Foundation +import Testing +import MachOKit +import Dependencies +@_spi(Internals) import MachOSymbols +@testable import MachOSwiftSection +@testable import MachOTestingSupport +import MachOFixtureSupport + +/// The diff renderer's header contract: a declaration whose HEADER cannot be +/// rendered must be dropped whole, never rendered as a body under a blank +/// header line. +/// +/// `renderType` / `renderProtocol` compute their body units unconditionally and +/// hand them to `DiffContainerAssembler` together with the header. While +/// `header(_:_:)` swallowed a throw into an empty `SemanticString`, that +/// produced a type's members and braces with NO `struct Foo` / `protocol Foo` +/// line above them — structurally invalid Swift, emitted silently, with no +/// event and no error. +/// +/// Header rendering can genuinely throw: it reads the declaration's name, +/// generic signature and superclass and demangles each (issue #102 is the field +/// evidence that print-time `DemanglingError`s occur on real binaries), and +/// since evolution 0002 it also re-materializes the wrapper from its +/// descriptor. +@Suite(.serialized) +final class DiffRendererHeaderFailureTests: MachOFileTests, @unchecked Sendable { + override class var fileName: MachOFileName { .SymbolTestsCore } + + private func preparedBuilder() async throws -> SwiftDiffableInterfaceBuilder { + let unsafeMachOFile = machOFile + let builder = SwiftDiffableInterfaceBuilder(in: unsafeMachOFile) + try await builder.prepare() + return builder + } + + private func findTypeDefinition( + named name: String, + in builder: SwiftDiffableInterfaceBuilder + ) -> TypeDefinition? { + builder.indexer.allTypeDefinitions.values.first { $0.typeName.currentName == name } + } + + /// Injecting a type whose header cannot be rendered must not add a single + /// character to the annotated interface. + /// + /// The injected child carries a real, renderable grandchild, so its BODY is + /// non-empty — that is what makes the assertion sharp. Before the fix the + /// blank header was assembled together with that body, so the grandchild's + /// declaration appeared in the output an extra time under no header of its + /// own; the rendered output therefore grew. After the fix the whole child + /// is dropped and the output is byte-identical to the un-injected run. + @Test func unrenderableTypeHeaderDropsTheWholeDeclaration() async throws { + let oldBuilder = try await preparedBuilder() + let newBuilder = try await preparedBuilder() + + let renderer = SwiftDiffableInterfaceRenderer(old: oldBuilder, new: newBuilder) + let outputBeforeInjection = await renderer.printAnnotatedInterface().string + + // A real struct descriptor's layout re-wrapped at an offset far past + // the fixture's end of file: every relative resolve the header + // materialization performs is out of bounds and throws deterministically. + let realStructDefinition = try #require(findTypeDefinition(named: "GenericStructNonRequirement", in: newBuilder)) + guard case .struct(let realStructDescriptor) = realStructDefinition.typeContextDescriptorWrapper else { + Issue.record("GenericStructNonRequirement is expected to be a struct") + return + } + let unreadableDescriptor = StructDescriptor(layout: realStructDescriptor.layout, offset: 0x0FFF_FFF0) + + let donorDefinition = try #require(findTypeDefinition(named: "FinalClassTest", in: newBuilder)) + let corruptDefinition = TypeDefinition( + typeContextDescriptorWrapper: .struct(unreadableDescriptor), + typeName: donorDefinition.typeName, + isSpecialized: false + ) + // A renderable grandchild, so the corrupt definition's body is not empty. + let grandchildDonor = try #require(findTypeDefinition(named: "StructTest", in: newBuilder)) + corruptDefinition.typeChildren.append(grandchildDonor) + + let hostDefinition = try #require(findTypeDefinition(named: "Classes", in: newBuilder)) + hostDefinition.typeChildren.append(corruptDefinition) + + let outputAfterInjection = await renderer.printAnnotatedInterface().string + + #expect( + outputAfterInjection == outputBeforeInjection, + "a type whose header cannot be rendered must be dropped whole — its body must not be emitted under a blank header line" + ) + } + + /// The companion property, stated positively: the un-renderable header must + /// not silently truncate its ENCLOSING declaration either. The host type + /// keeps rendering; only the injected child disappears. + @Test func unrenderableChildHeaderKeepsItsEnclosingType() async throws { + let oldBuilder = try await preparedBuilder() + let newBuilder = try await preparedBuilder() + + let realStructDefinition = try #require(findTypeDefinition(named: "GenericStructNonRequirement", in: newBuilder)) + guard case .struct(let realStructDescriptor) = realStructDefinition.typeContextDescriptorWrapper else { + Issue.record("GenericStructNonRequirement is expected to be a struct") + return + } + let unreadableDescriptor = StructDescriptor(layout: realStructDescriptor.layout, offset: 0x0FFF_FFF0) + + let donorDefinition = try #require(findTypeDefinition(named: "FinalClassTest", in: newBuilder)) + let corruptDefinition = TypeDefinition( + typeContextDescriptorWrapper: .struct(unreadableDescriptor), + typeName: donorDefinition.typeName, + isSpecialized: false + ) + + let hostDefinition = try #require(findTypeDefinition(named: "Classes", in: newBuilder)) + hostDefinition.typeChildren.append(corruptDefinition) + + let renderer = SwiftDiffableInterfaceRenderer(old: oldBuilder, new: newBuilder) + let output = await renderer.printAnnotatedInterface().string + + #expect(output.contains("Classes"), "the enclosing type must keep rendering") + } +} From 55b986167f1cd7de4bc7c33f673c867da40e5241 Mon Sep 17 00:00:00 2001 From: Mx-Iris Date: Thu, 13 Aug 2026 11:33:20 +0800 Subject: [PATCH 76/77] perf(symbols,indexing): probe opaque types once, claim caches per cache MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Two fixes on the per-image index, both from the same review round. opaqueTypeDescriptorSymbol bucketed its entries by member identifier and scanned the bucket structurally. The assumption was that an identifier picks out "normally exactly one" descriptor; it does not. The identifier is the member name, so in SwiftUI the `body` bucket alone holds hundreds of entries, and the caller queries once per printed `some`-returning declaration with no memoization — quadratic in a count that runs into the thousands, with a fresh visited-pair set allocated per comparison. StructuralNodeReferenceKey now also accepts a bare query Node, hashing it through the structural hash upstream keeps in step with the reference's, so the lookup is one probe. Cache eviction claimed all three per-image caches from one sample of the symbol store. Only that one is necessarily an indexer's: the interned-name store and the demangle memo are also populated by SwiftLayout, the renderers and SwiftSpecialization, so a "dump, then build the interface" sequence fills both with no symbol store at all — and the combined claim read that state backwards, evicting caches live non-indexer work was still using. Each is now claimed separately. Registration is keyed on indexer identity rather than counted, so a concurrent second prepare() cannot strand the population above zero and leak all three for the process lifetime. Also renames a `cls` binding this branch carried over, per the no- abbreviations rule. --- .../StructuralNodeReferenceKey.swift | 66 +++++++- Sources/MachOSymbols/SymbolIndexStore.swift | 65 ++++---- .../Definitions/TypeDefinition.swift | 20 +-- .../SwiftDeclarationIndexer.swift | 153 ++++++++++++------ .../PerImageCacheEvictionTests.swift | 112 +++++++++++-- 5 files changed, 308 insertions(+), 108 deletions(-) diff --git a/Sources/MachOSymbols/StructuralNodeReferenceKey.swift b/Sources/MachOSymbols/StructuralNodeReferenceKey.swift index 8980505b..738d7be9 100644 --- a/Sources/MachOSymbols/StructuralNodeReferenceKey.swift +++ b/Sources/MachOSymbols/StructuralNodeReferenceKey.swift @@ -27,17 +27,75 @@ import Demangling /// Lives in `MachOSymbols`, next to the mini stores it exists to reconcile, so /// both the symbol index itself and the declaration layer above it can use it. package struct StructuralNodeReferenceKey: Hashable { - package let reference: NodeReference + /// A stored key holds a `NodeReference`; a lookup-only key may instead hold + /// a bare `Node` the caller just demangled. + /// + /// Both forms hash through the structural hash `Node` and `NodeReference` + /// agree on (`NodeReference.structuralHash(into:)` is documented to stay in + /// step with `Node.hash(into:)`), so a `Node` query finds a `NodeReference` + /// key without materializing the stored tree or interning the query one. + /// That is what lets a structurally-keyed dictionary answer a print-time + /// lookup in one probe instead of a linear scan over structural + /// comparisons. + package enum Storage { + case reference(NodeReference) + case queryNode(Node) + } + + package let storage: Storage + + /// The stored reference. + /// + /// Traps on a lookup-only key. That cannot happen for a key read back out + /// of a collection: `init(querying:)` values exist only as the argument to + /// a subscript and are never stored (a bare `Node` carries no store, so a + /// stored query key would fail to keep alive the arena its structural peers + /// live in). The trap is therefore a programming-error guard on this + /// module's own API — not an input-driven one, unlike the binary-supplied + /// geometry checks in `PackedNameReference`, which deliberately degrade + /// instead of trapping because a malformed binary must never decide whether + /// the host process lives. + package var reference: NodeReference { + switch storage { + case .reference(let reference): + return reference + case .queryNode: + preconditionFailure("StructuralNodeReferenceKey(querying:) is lookup-only and carries no stored reference") + } + } package init(_ reference: NodeReference) { - self.reference = reference + self.storage = .reference(reference) + } + + /// A lookup-only key over a freshly demangled tree. + /// + /// Never store one of these as a dictionary key: a bare `Node` carries no + /// store, so a key built this way would not keep the arena its structural + /// peers live in alive. + package init(querying node: Node) { + self.storage = .queryNode(node) } package static func == (lhs: StructuralNodeReferenceKey, rhs: StructuralNodeReferenceKey) -> Bool { - lhs.reference.structurallyEquals(rhs.reference) + switch (lhs.storage, rhs.storage) { + case (.reference(let lhsReference), .reference(let rhsReference)): + return lhsReference.structurallyEquals(rhsReference) + case (.reference(let lhsReference), .queryNode(let rhsNode)): + return lhsReference.structurallyEquals(rhsNode) + case (.queryNode(let lhsNode), .reference(let rhsReference)): + return rhsReference.structurallyEquals(lhsNode) + case (.queryNode(let lhsNode), .queryNode(let rhsNode)): + return lhsNode == rhsNode + } } package func hash(into hasher: inout Hasher) { - reference.structuralHash(into: &hasher) + switch storage { + case .reference(let reference): + reference.structuralHash(into: &hasher) + case .queryNode(let node): + node.structuralHash(into: &hasher) + } } } diff --git a/Sources/MachOSymbols/SymbolIndexStore.swift b/Sources/MachOSymbols/SymbolIndexStore.swift index c8c50127..3494cf8a 100644 --- a/Sources/MachOSymbols/SymbolIndexStore.swift +++ b/Sources/MachOSymbols/SymbolIndexStore.swift @@ -160,26 +160,26 @@ public final class SymbolIndexStore: SharedCache, @unc let opaqueTypeDescriptorSymbolRowByNodeIndex: OrderedDictionary - /// One opaque-type-descriptor entry: the member node's index in the - /// frozen arena plus the symbol table row it was recorded for. - struct OpaqueTypeDescriptorEntry { - let memberNodeIndex: NodeStore.NodeIndex - let symbolTableRow: UInt32 - } - /// The same entries as `opaqueTypeDescriptorSymbolRowByNodeIndex`, - /// bucketed by the member's declaration identifier. + /// keyed **structurally** so a print-time query is one hash probe. /// - /// `opaqueTypeDescriptorSymbol(for:)` is queried with a node the - /// caller demangled while printing — a different store — so node-index - /// equality cannot answer it and the ordered dictionary would have to - /// be walked in full, once per printed `some`-returning declaration - /// (O(descriptors × prints), and both counts run into the thousands in - /// a framework like SwiftUI). `DemanglingNode.identifier` is a pure - /// function of the subtree, so structurally equal nodes always land in - /// the same bucket and the structural comparison is narrowed to - /// same-named candidates — normally exactly one. - let opaqueTypeDescriptorEntriesByMemberIdentifier: [String: [OpaqueTypeDescriptorEntry]] + /// `opaqueTypeDescriptorSymbol(for:)` is queried with a node the caller + /// demangled while printing — a different store — so node-index + /// equality cannot answer it. Bucketing by `DemanglingNode.identifier` + /// and scanning the bucket was the first attempt at narrowing that, on + /// the assumption that a member identifier picks out "normally exactly + /// one" descriptor. It does not: in SwiftUI the `body` bucket alone + /// holds hundreds of entries (every `some View` implementation shares + /// the identifier), and the caller queries once per printed + /// `some`-returning declaration with no memoization, so the scan was + /// quadratic in a count that runs into the thousands — and each + /// `structurallyEquals` allocates a fresh visited-pair set before its + /// first kind check. + /// + /// `StructuralNodeReferenceKey` hashes a stored `NodeReference` and a + /// queried `Node` alike, which is what makes the direct dictionary + /// possible. + let opaqueTypeDescriptorSymbolRowByMemberNode: [StructuralNodeReferenceKey: UInt32] let memberSymbolRowsByKind: OrderedDictionary @@ -235,12 +235,19 @@ public final class SymbolIndexStore: SharedCache, @unc self.typeInfoByName = rowIndexes.typeInfoByName self.globalSymbolRowsByKind = rowIndexes.globalSymbolRowsByKind self.opaqueTypeDescriptorSymbolRowByNodeIndex = rowIndexes.opaqueTypeDescriptorSymbolRowByNodeIndex - var opaqueTypeDescriptorEntriesByMemberIdentifier: [String: [OpaqueTypeDescriptorEntry]] = [:] + var opaqueTypeDescriptorSymbolRowByMemberNode: [StructuralNodeReferenceKey: UInt32] = [:] + opaqueTypeDescriptorSymbolRowByMemberNode.reserveCapacity(rowIndexes.opaqueTypeDescriptorSymbolRowByNodeIndex.count) for (memberNodeIndex, symbolTableRow) in rowIndexes.opaqueTypeDescriptorSymbolRowByNodeIndex { - let memberIdentifier = nodeStore.reference(at: memberNodeIndex).identifier ?? "" - opaqueTypeDescriptorEntriesByMemberIdentifier[memberIdentifier, default: []].append(.init(memberNodeIndex: memberNodeIndex, symbolTableRow: symbolTableRow)) + // First wins, matching the ordered dictionary this replaced: + // its per-node-index keys were already unique, so a collision + // here means two arena nodes that are structurally equal, and + // either answers the query identically. + let memberNodeKey = StructuralNodeReferenceKey(nodeStore.reference(at: memberNodeIndex)) + if opaqueTypeDescriptorSymbolRowByMemberNode[memberNodeKey] == nil { + opaqueTypeDescriptorSymbolRowByMemberNode[memberNodeKey] = symbolTableRow + } } - self.opaqueTypeDescriptorEntriesByMemberIdentifier = opaqueTypeDescriptorEntriesByMemberIdentifier + self.opaqueTypeDescriptorSymbolRowByMemberNode = opaqueTypeDescriptorSymbolRowByMemberNode self.memberSymbolRowsByKind = rowIndexes.memberSymbolRowsByKind self.methodDescriptorMemberSymbolRowsByKind = rowIndexes.methodDescriptorMemberSymbolRowsByKind self.protocolWitnessMemberSymbolRowsByKind = rowIndexes.protocolWitnessMemberSymbolRowsByKind @@ -945,14 +952,14 @@ public final class SymbolIndexStore: SharedCache, @unc public func opaqueTypeDescriptorSymbol(for node: Node, in machO: MachO) -> DemangledSymbol? { // The caller's `node` was demangled during printing; keys live in the - // frozen store, so the match has to be structural. Bucketing on the - // member identifier keeps that to a handful of candidates instead of - // every opaque-type descriptor in the image (see - // `opaqueTypeDescriptorEntriesByMemberIdentifier`). + // frozen store, so the match has to be structural — but structural does + // not have to mean linear. `StructuralNodeReferenceKey` hashes a queried + // `Node` the same way it hashes a stored `NodeReference`, so this is one + // probe (see `opaqueTypeDescriptorSymbolRowByMemberNode` for why the + // identifier-bucketed scan this replaced was quadratic in practice). guard let storage = storage(in: machO) else { return nil } - guard let candidates = storage.opaqueTypeDescriptorEntriesByMemberIdentifier[node.identifier ?? ""] else { return nil } - guard let matched = candidates.first(where: { storage.nodeStore.reference(at: $0.memberNodeIndex).structurallyEquals(node) }) else { return nil } - return storage.demangledSymbol(atRow: matched.symbolTableRow) + guard let symbolTableRow = storage.opaqueTypeDescriptorSymbolRowByMemberNode[.init(querying: node)] else { return nil } + return storage.demangledSymbol(atRow: symbolTableRow) } package func symbols(for offset: Int, in machO: MachO) -> Symbols? { diff --git a/Sources/SwiftDeclaration/Components/Definitions/TypeDefinition.swift b/Sources/SwiftDeclaration/Components/Definitions/TypeDefinition.swift index 0104b84d..73a8f476 100644 --- a/Sources/SwiftDeclaration/Components/Definitions/TypeDefinition.swift +++ b/Sources/SwiftDeclaration/Components/Definitions/TypeDefinition.swift @@ -220,10 +220,10 @@ public final class TypeDefinition: Definition { // materialize the full wrapper — once, as a local, released when this // function returns (materialization discipline, proposal 0002). if case .class(let classDescriptor) = typeContextDescriptorWrapper { - let cls = try Class(descriptor: classDescriptor, in: machO) + let classWrapper = try Class(descriptor: classDescriptor, in: machO) var visitedNodes: OrderedSet = [] - let typeNode = try MetadataReader.demangleContext(for: .type(.class(cls.descriptor)), in: machO) - let vtableBaseOffset = cls.vTableDescriptorHeader.map { Int($0.layout.vTableOffset) } + let typeNode = try MetadataReader.demangleContext(for: .type(.class(classWrapper.descriptor)), in: machO) + let vtableBaseOffset = classWrapper.vTableDescriptorHeader.map { Int($0.layout.vTableOffset) } // Build offset-based fallback lookups. Uniqueness must be checked against // ALL descriptor kinds (method + override + defaultOverride), because @@ -232,19 +232,19 @@ public final class TypeDefinition: Definition { // we cannot use offset-based fallback — we would not know which descriptor // to associate the symbol with. var implOffsetCounts: [Int: Int] = [:] - for descriptor in cls.methodDescriptors where !descriptor.implementation.isNull { + for descriptor in classWrapper.methodDescriptors where !descriptor.implementation.isNull { let implOffset = descriptor.implementation.resolveDirectOffset(from: descriptor.offset(of: \.implementation)) implOffsetCounts[implOffset, default: 0] += 1 } - for descriptor in cls.methodOverrideDescriptors where !descriptor.implementation.isNull { + for descriptor in classWrapper.methodOverrideDescriptors where !descriptor.implementation.isNull { let implOffset = descriptor.implementation.resolveDirectOffset(from: descriptor.offset(of: \.implementation)) implOffsetCounts[implOffset, default: 0] += 1 } - for descriptor in cls.methodDefaultOverrideDescriptors where !descriptor.implementation.isNull { + for descriptor in classWrapper.methodDefaultOverrideDescriptors where !descriptor.implementation.isNull { let implOffset = descriptor.implementation.resolveDirectOffset(from: descriptor.offset(of: \.implementation)) implOffsetCounts[implOffset, default: 0] += 1 } - for (index, descriptor) in cls.methodDescriptors.enumerated() where !descriptor.implementation.isNull { + for (index, descriptor) in classWrapper.methodDescriptors.enumerated() where !descriptor.implementation.isNull { let implOffset = descriptor.implementation.resolveDirectOffset(from: descriptor.offset(of: \.implementation)) // Only use offset-based fallback for globally unique implementation addresses if implOffsetCounts[implOffset] == 1 { @@ -255,7 +255,7 @@ public final class TypeDefinition: Definition { } } - for (index, descriptor) in cls.methodDescriptors.enumerated() { + for (index, descriptor) in classWrapper.methodDescriptors.enumerated() { guard let symbols = try descriptor.implementationSymbols(in: machO) else { continue } guard let overrideSymbol = demangledOverrideSymbol(for: symbols, typeNode: typeNode, visitedNodes: visitedNodes, in: machO) else { continue } let node = overrideSymbol.demangledNode @@ -267,7 +267,7 @@ public final class TypeDefinition: Definition { } var parentVTableCache = ParentClassVTableCache() - for descriptor in cls.methodOverrideDescriptors { + for descriptor in classWrapper.methodOverrideDescriptors { guard let symbols = try descriptor.implementationSymbols(in: machO) else { continue } guard let overrideSymbol = demangledOverrideSymbol(for: symbols, typeNode: typeNode, visitedNodes: visitedNodes, in: machO) else { continue } let node = overrideSymbol.demangledNode @@ -278,7 +278,7 @@ public final class TypeDefinition: Definition { vtableOffsetLookup[StructuralNodeReferenceKey(node)] = vtableSlot } } - for descriptor in cls.methodDefaultOverrideDescriptors { + for descriptor in classWrapper.methodDefaultOverrideDescriptors { guard let symbols = try descriptor.implementationSymbols(in: machO) else { continue } guard let overrideSymbol = demangledOverrideSymbol(for: symbols, typeNode: typeNode, visitedNodes: visitedNodes, in: machO) else { continue } let node = overrideSymbol.demangledNode diff --git a/Sources/SwiftIndexing/SwiftDeclarationIndexer.swift b/Sources/SwiftIndexing/SwiftDeclarationIndexer.swift index 94d4a983..7238b674 100644 --- a/Sources/SwiftIndexing/SwiftDeclarationIndexer.swift +++ b/Sources/SwiftIndexing/SwiftDeclarationIndexer.swift @@ -163,37 +163,38 @@ public final class SwiftDeclarationIndexer = [] + var claims: Claims = .none } private static let registryLock = NSLock() private nonisolated(unsafe) static var entriesByImageIdentifier: [AnyHashable: ImageEntry] = [:] - static func registerLiveIndexer(forImageIdentifier imageIdentifier: AnyHashable, claimingEviction: Bool) { + static func registerLiveIndexer( + _ indexerIdentity: ObjectIdentifier, + forImageIdentifier imageIdentifier: AnyHashable, + claims: Claims + ) { registryLock.lock() defer { registryLock.unlock() } var imageEntry = entriesByImageIdentifier[imageIdentifier, default: ImageEntry()] - imageEntry.liveIndexerCount += 1 - if claimingEviction { - imageEntry.isEvictionClaimed = true + let isFirstRegistration = imageEntry.liveIndexers.insert(indexerIdentity).inserted + // Only a first registration contributes claims: a re-entrant + // `prepare()` samples the caches its own earlier pass just built and + // would otherwise answer "nobody had this, so I claim it" backwards. + if isFirstRegistration { + imageEntry.claims.formUnion(claims) } entriesByImageIdentifier[imageIdentifier] = imageEntry } - /// `true` when the deregistering indexer was the image's last live one - /// AND eviction was claimed — the caller must then evict the per-image - /// caches. - static func deregisterLiveIndexer(forImageIdentifier imageIdentifier: AnyHashable) -> Bool { + /// The caches the deregistering indexer must evict — all-false unless it + /// was the image's LAST live indexer, so a shared entry never disappears + /// under a live sibling. + static func deregisterLiveIndexer( + _ indexerIdentity: ObjectIdentifier, + forImageIdentifier imageIdentifier: AnyHashable + ) -> Claims { registryLock.lock() defer { registryLock.unlock() } - guard var imageEntry = entriesByImageIdentifier[imageIdentifier] else { return false } - imageEntry.liveIndexerCount -= 1 - guard imageEntry.liveIndexerCount <= 0 else { + guard var imageEntry = entriesByImageIdentifier[imageIdentifier] else { return .none } + imageEntry.liveIndexers.remove(indexerIdentity) + guard imageEntry.liveIndexers.isEmpty else { entriesByImageIdentifier[imageIdentifier] = imageEntry - return false + return .none } entriesByImageIdentifier.removeValue(forKey: imageIdentifier) - return imageEntry.isEvictionClaimed + return imageEntry.claims } } diff --git a/Tests/SwiftIndexingTests/PerImageCacheEvictionTests.swift b/Tests/SwiftIndexingTests/PerImageCacheEvictionTests.swift index cb747ae4..848ff1d5 100644 --- a/Tests/SwiftIndexingTests/PerImageCacheEvictionTests.swift +++ b/Tests/SwiftIndexingTests/PerImageCacheEvictionTests.swift @@ -8,34 +8,68 @@ import Dependencies @testable import MachOTestingSupport import MachOFixtureSupport @_spi(Internals) @testable import SwiftInspection +@_spi(Internals) import Demangling +@testable import MachOSwiftSection /// The indexer's `deinit` cleans up three per-image caches (symbol store, -/// interned-name store, demangle memo). That cleanup must be performed by -/// the image's LAST live indexer — an earlier indexer deinitializing while -/// a second one still uses the same image must not wipe the caches out from -/// under it (the survivor's already-built names would keep an orphaned -/// store alive while new names land in a fresh one, splitting the -/// `store ===` fast paths for the rest of its lifetime). PR #103 review, -/// finding M6. +/// interned-name store, demangle memo). Two rules govern that cleanup: +/// +/// 1. It is performed by the image's **LAST** live indexer — an earlier +/// indexer deinitializing while a second one still uses the same image must +/// not wipe the caches out from under it (the survivor's already-built names +/// would keep an orphaned store alive while new names land in a fresh one, +/// splitting the `store ===` fast paths for the rest of its lifetime). +/// PR #103 review, finding M6. +/// 2. Each cache is claimed **separately**. The symbol store is the only one an +/// indexer necessarily builds; the interned-name store and the demangle memo +/// are also populated by SwiftLayout, `SwiftDeclarationRendering` and +/// `SwiftSpecialization`. A single combined claim sampled from the symbol +/// store alone evicted caches that non-indexer work had built and was still +/// using — the follow-up round of the same finding. /// /// Runs against `SymbolTestsHelper` — an image no other suite indexes — so /// the cache-membership assertions cannot race a concurrently-running -/// suite's indexer lifecycle. +/// suite's indexer lifecycle. Each test starts from a cleared slate because the +/// suite is serialized and a previous test's residue would otherwise decide the +/// claims under test. @Suite(.serialized) final class PerImageCacheEvictionTests: MachOFileTests, @unchecked Sendable { override class var fileName: MachOFileName { .SymbolTestsHelper } + private func clearAllPerImageCaches(for machOFile: MachOFile) { + @Dependency(\.symbolIndexStore) + var symbolIndexStore + symbolIndexStore.remove(for: machOFile) + InternedNodeReferenceCache.shared.remove(for: machOFile) + MetadataReader.removeCache(for: machOFile) + } + + /// Populates the interned-name store and the demangle memo the way a + /// NON-indexer consumer does (SwiftLayout and the renderers reach + /// `MetadataReader` directly), deliberately leaving the symbol store + /// untouched. + @discardableResult + private func populateNonIndexerCaches(for machOFile: MachOFile) throws -> Bool { + let typeDescriptor = try #require(try machOFile.swift.typeContextDescriptors.first) + let typeNode = try MetadataReader.demangleContext(for: .type(typeDescriptor), in: machOFile) + _ = InternedNodeReferenceCache.shared.reference(interning: typeNode, in: machOFile) + return true + } + @Test func survivingIndexerKeepsPerImageCaches() async throws { let unsafeMachOFile = machOFile + clearAllPerImageCaches(for: unsafeMachOFile) var firstIndexer: SwiftDeclarationIndexer? = SwiftDeclarationIndexer(in: unsafeMachOFile) try await firstIndexer?.prepare() let secondIndexer = SwiftDeclarationIndexer(in: unsafeMachOFile) try await secondIndexer.prepare() + // All three must genuinely be populated, or the survivor assertions + // below degrade to `false == false` and pin nothing. try #require(SymbolIndexStore.shared.contains(in: unsafeMachOFile)) - let internedNameCacheWasPresent = InternedNodeReferenceCache.shared.contains(in: unsafeMachOFile) - let demangleMemoWasPresent = MetadataReader.cacheExists(for: unsafeMachOFile) + try #require(InternedNodeReferenceCache.shared.contains(in: unsafeMachOFile)) + try #require(MetadataReader.cacheExists(for: unsafeMachOFile)) // The first indexer populated the caches, so under a per-indexer // ownership flag its deinit would evict all three out from under @@ -46,20 +80,30 @@ final class PerImageCacheEvictionTests: MachOFileTests, @unchecked Sendable { SymbolIndexStore.shared.contains(in: unsafeMachOFile), "the first indexer's deinit evicted the symbol store while a second live indexer was using the image" ) - #expect(InternedNodeReferenceCache.shared.contains(in: unsafeMachOFile) == internedNameCacheWasPresent) - #expect(MetadataReader.cacheExists(for: unsafeMachOFile) == demangleMemoWasPresent) + #expect( + InternedNodeReferenceCache.shared.contains(in: unsafeMachOFile), + "the first indexer's deinit evicted the interned-name store while a second live indexer was using the image" + ) + #expect( + MetadataReader.cacheExists(for: unsafeMachOFile), + "the first indexer's deinit evicted the demangle memo while a second live indexer was using the image" + ) withExtendedLifetime(secondIndexer) {} } @Test func lastIndexerEvictsPerImageCaches() async throws { let unsafeMachOFile = machOFile + clearAllPerImageCaches(for: unsafeMachOFile) var firstIndexer: SwiftDeclarationIndexer? = SwiftDeclarationIndexer(in: unsafeMachOFile) try await firstIndexer?.prepare() var secondIndexer: SwiftDeclarationIndexer? = SwiftDeclarationIndexer(in: unsafeMachOFile) try await secondIndexer?.prepare() + try #require(SymbolIndexStore.shared.contains(in: unsafeMachOFile)) + try #require(InternedNodeReferenceCache.shared.contains(in: unsafeMachOFile)) + try #require(MetadataReader.cacheExists(for: unsafeMachOFile)) firstIndexer = nil secondIndexer = nil @@ -71,4 +115,48 @@ final class PerImageCacheEvictionTests: MachOFileTests, @unchecked Sendable { #expect(!InternedNodeReferenceCache.shared.contains(in: unsafeMachOFile)) #expect(!MetadataReader.cacheExists(for: unsafeMachOFile)) } + + /// Caches an indexer did NOT build must survive its deinit. + /// + /// The realistic sequence this reproduces: a host does static layout work + /// or a dump over an image (populating the interned-name store and the + /// demangle memo through `MetadataReader`, never touching the symbol + /// store), then separately creates and destroys an indexer for the same + /// image. Sampling one combined claim from the symbol store alone answers + /// "nobody had this, so I claim it" for all three, and the indexer's deinit + /// then wipes two caches the still-live non-indexer work is using: names + /// minted before the wipe stay in the dropped arena via their live + /// `NodeReference`s while later names land in a fresh one, so the + /// documented `store ===` fast path stops firing permanently. + @Test func indexerDoesNotEvictCachesItDidNotBuild() async throws { + let unsafeMachOFile = machOFile + clearAllPerImageCaches(for: unsafeMachOFile) + + try populateNonIndexerCaches(for: unsafeMachOFile) + + // The precondition that makes this test meaningful: the two caches are + // populated, the symbol store is not — exactly the state a combined + // claim reads backwards. + try #require(InternedNodeReferenceCache.shared.contains(in: unsafeMachOFile)) + try #require(MetadataReader.cacheExists(for: unsafeMachOFile)) + try #require(!SymbolIndexStore.shared.contains(in: unsafeMachOFile)) + + var indexer: SwiftDeclarationIndexer? = SwiftDeclarationIndexer(in: unsafeMachOFile) + try await indexer?.prepare() + indexer = nil + + #expect( + InternedNodeReferenceCache.shared.contains(in: unsafeMachOFile), + "the indexer evicted an interned-name store that non-indexer work had built and may still be using" + ) + #expect( + MetadataReader.cacheExists(for: unsafeMachOFile), + "the indexer evicted a demangle memo that non-indexer work had built and may still be using" + ) + // The symbol store IS the indexer's to reclaim: it built that one. + #expect( + !SymbolIndexStore.shared.contains(in: unsafeMachOFile), + "the symbol store the indexer built must still be reclaimed" + ) + } } From b2eabe672b0bf4cbf30ec9c9957330c52d145929 Mon Sep 17 00:00:00 2001 From: Mx-Iris Date: Thu, 13 Aug 2026 11:33:34 +0800 Subject: [PATCH 77/77] docs: settle the second PR #103 review round MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Evolution 0001's compatibility section claimed "purely additive / no breakage" and "no superseded public API" while the landed code had replaced Symbol.nlist with isExternal. An accepted proposal stating the opposite of the shipped code is worse than the API change itself, so it now records the break honestly — including the mitigating fact that the default argument keeps Symbol(offset:name:) compiling. AGENTS.md follows the two code changes in this batch (structural opaque index, per-cache eviction claims) and gains the A/B harness self-test. Adjudications A7/A8 record the two findings deliberately not fixed: the print-options divergence between the indexer and dump witness matchers (no reproduction, identical on main, no fixture) and updateConfiguration's re-prepare being a no-op (unreachable for every known consumer). Node+.swift's printSemantic gains a warning not to "modernize" onto the new runPrintWalk requirement: that hook returns String because it dispatches print(using:), so a custom target is outside its remit by design, and the engine's static entry point — including its stack guard — is unchanged. --- AGENTS.md | 6 +- .../0001-symbol-name-offsetization.md | 15 ++- .../Internal/ProjectEvolutionLog.md | 15 +++ .../Internal/ReviewAdjudications.md | 24 ++++ ...2026-08-13-pr103-review-round-two-fixes.md | 112 ++++++++++++++++++ .../Extensions/Node+.swift | 23 ++++ 6 files changed, 190 insertions(+), 5 deletions(-) create mode 100644 Documentations/Internal/TaskReports/2026-08-13-pr103-review-round-two-fixes.md diff --git a/AGENTS.md b/AGENTS.md index 41bfd231..c4ecc641 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -52,6 +52,8 @@ Requires Swift 6.2+ / Xcode 26.0+. **Rendering A/B verification (mandatory after any large refactor):** any refactor touching demangling, printing, indexing, or the reader stack must pass `Scripts/run-rendering-ab-verification.py ` — byte-identical dump + interface output over real system frameworks (SwiftUI/SwiftUICore/SwiftData/Combine/ActivityKit/WidgetKit) through all three reader paths: archived dyld caches (falls back to the current system's cache when absent), simulator-runtime Mach-O files (falls back to whatever runtimes are installed), and in-process MachOImage via `RenderingVerificationTests`. Procedure, fallback rules, and known pitfalls: [Documentations/Internal/SystemFrameworkRenderingVerification.md](Documentations/Internal/SystemFrameworkRenderingVerification.md). +The harness's own verdict logic is unit-tested — `python3 Scripts/test-run-rendering-ab-verification.py` (standard library only, seconds to run). **Run it after touching `compare_all_pairs` or the skip-marker writing**, because every hole found there so far has been of the same shape: the harness reporting a pass over a comparison it never actually made (zero pairs compared; both sides failing with *different* exit codes, which leaves no `.txt` for either glob to see). A green light from this script is what AGENTS.md makes acceptance evidence, so a harness that cannot fail is worse than no harness. + ## Architecture Overview This is a Swift library for parsing Mach-O files to extract Swift metadata (types, protocols, conformances). It uses a custom Demangler to parse symbolic references and restore Swift Runtime logic. @@ -115,7 +117,7 @@ The interface generation is split into layered peer modules over a shared `Swift - `SwiftIndexEvents` - event namespace (Payload/Dispatcher/Handler) emitted by both indexer and printer **SwiftIndexing** - Builds the `SwiftDeclaration` model from a Mach-O image -- `SwiftDeclarationIndexer` - Indexes types, extensions, conformances. Its `deinit` cleans up the three per-image caches (symbol store, interned-name store, demangle memo) via `PerImageCacheEvictionRegistry`: eviction is claimed per IMAGE (by whichever indexer's `prepare()` built the entry) and performed by the image's LAST live indexer, so an earlier-deinitializing indexer never wipes the caches out from under a live sibling; entries built by non-indexer callers are never evicted (pinned by `PerImageCacheEvictionTests`) +- `SwiftDeclarationIndexer` - Indexes types, extensions, conformances. Its `deinit` cleans up the three per-image caches (symbol store, interned-name store, demangle memo) via `PerImageCacheEvictionRegistry`, under two rules: eviction is claimed **per CACHE** (each of the three sampled separately at `prepare()`, by whichever indexer's `prepare()` found it absent and therefore built it) and performed by the image's LAST live indexer, so an earlier-deinitializing indexer never wipes the caches out from under a live sibling. The per-cache split matters because only the symbol store is necessarily an indexer's: the interned-name store and the demangle memo are also populated by SwiftLayout, `SwiftDeclarationRendering` and `SwiftSpecialization` through `MetadataReader`, so a "dump, then build the interface" sequence fills those two with no symbol store at all — a single claim sampled from the symbol store alone read that state backwards and evicted live non-indexer work's caches. Registration is keyed on indexer identity (`ObjectIdentifier`), not counted, so a concurrent double `prepare()` (its `isPrepared` guard is a plain check-then-set on an async entry point) cannot strand the population above zero and leak all three caches for the process lifetime. Entries built by non-indexer callers are never claimed and never evicted (pinned by `PerImageCacheEvictionTests`, including `indexerDoesNotEvictCachesItDidNotBuild`) - The section-wrapper populations the index passes consume (`types` / `protocols` / `protocolConformances` / `associatedTypes` and the parsed-value keyed conformance maps) are **indexing transients** since proposal 0002 — released when `prepare()` finishes, with no public projection. The retained conformance facts are the name-level maps `conformingProtocolNamesByTypeName` / `conformingTypesByProtocolName` (+ their merged `all*` variants), which is all any post-indexing consumer (including `SwiftSpecialization`'s `ConformanceProvider`) reads. - `SwiftIndexEventReporter`, `OSLogEventHandler`, `ConsoleEventHandler` - event handlers - `SwiftDeclarationIndexConfiguration` @@ -214,7 +216,7 @@ let node = try demangleAsNode("$sSiD") // Returns Node tree let string = node.print(using: .default) // "Swift.Int" ``` -**Symbol indexing (NodeStore-backed)**: `SymbolIndexStore.Storage` no longer retains `Node` class trees. The build sweep demangles each symbol cache-free onto a transient tree (`@_spi(Internals) demangleAsNodeTransient`), classifies on that tree, and interns it into a per-image `NodeStoreBuilder`. Nothing touches the global `NodeCache`, and dropping a `Storage` releases the whole per-image footprint. Storage layout (Stage 3, offset-ized by evolution proposal 0001): one 16-byte `SymbolRow` per unique symbol name in a `SymbolTable` — canonical (cache-adjusted; raw and adjusted offset keys share one row) offset plus a packed reference to the name's bytes, with **no retained name `String`s**. Names resolve through one of two sources: a `MachOImage` row points straight into the image's mmap'd LINKEDIT string table (clean pages, zero-copy; the table consequently requires the image to stay loaded — vended values' `symbol` reads materialize from it), while `MachOFile` rows and export-trie names (decoded strings with no mapped home) live in the table's private contiguous byte buffer. The collection sweep is reader-split: the image leg tests `isSwiftSymbol` byte-level on `nameC` (`nameBytesHaveSwiftManglingPrefix`, mirroring `Demangling.getManglingPrefixLength`'s prefix list — pinned equal by `SymbolTableEquivalenceTests`), so a non-Swift symbol never materializes a name at all; the generic leg (files) keeps the `String` surface and appends Swift name bytes into the private buffer. Name → row lookup is a byte-level binary search over the `rowsSortedByName` permutation (`SymbolTable.row(forName:)`); the name-keyed dictionary exists only during the build and is discarded at freeze, which also makes exact-capacity copies of the accumulated buffers. The byte access layer is `UnsafeBufferPointer`-based deliberately — `Span`/`UTF8Span` are macOS 26-only at runtime, above this package's deployment floor. `symbolRowsByOffset` is a plain `Dictionary` (single keyed consumer, nothing iterates it in order). A parallel `rootNodeIndexByTableRow` array holds each row's demangled root, and every classification index stores 4-byte `UInt32` table rows (member/opaque keys are 4-byte `NodeStore.NodeIndex`); indexes accumulate in final row form during the sweep, so `freeze()` is followed by a plain move, not a conversion pass. `Symbol` stays the public 32-byte eager value (no `nlist` existential — the external-undefined bit is extracted at collection time into `isExternal`; vend paths materialize it on demand), and `DemangledSymbol` is a 32-byte value (table reference + row + `NodeReference`) whose `symbol` is computed; `compactValueLayouts` asserts all three layouts including `SymbolRow == 16`. **A `DemangledSymbol` stored into the declaration model must be detached first** (`detachedFromSharedTable()`): the shared `SymbolTable` is the right trade for the hundreds of thousands of values a query vends and drops, but a single stored survivor pins the whole table — and, for an image table, keeps its mapped-string-table reads tied to the loaded image — defeating `removeSubIndexer(_:)`, whose reason for existing is reclaiming per-image memory. Measured on SwiftUI (iOS 18.5, pre-0001 representation): 9,872 stored values referenced 9,506 distinct rows, 5.1% of a 185,988-row table, so detaching trades ~0.6 MB of small allocations for the whole table's retention (~19.9 MB then; smaller but still whole-table now). The six storing sites are `DefinitionBuilder`'s four construction points (`Accessor` for variables/subscripts, `FunctionDefinition` for allocators/functions) and `TypeDefinition`'s `deallocatorSymbol` / `destructorSymbol` assignments; `SymbolTableRetentionTests` fails the moment a new one forgets. Do NOT detach on the query path. Query APIs construct `DemangledSymbol`/`NodeReference` values at the exit boundary; APIs taking an externally demangled node (`memberSymbols(of:for:node:)` with `Node` or `NodeReference`, `opaqueTypeDescriptorSymbol(for:)`) match keys via `NodeReference.structurallyEquals(_:)` (same-store O(1), cross-store structural walk) — `opaqueTypeDescriptorSymbol(for:)` buckets its entries by `DemanglingNode.identifier` first (`opaqueTypeDescriptorEntriesByMemberIdentifier`), because a full scan there is O(descriptors × printed `some` returns). `demangledNodeReference(for:)` matches on the symbol **name alone** — a demangled tree is a pure function of the name and the flat table holds one row per unique name, so comparing the row's canonical offset against the queried symbol's offset could only reject valid hits (it once rejected every symbol of a dyld-cache image, since `symbols(for:in:)` stamps each vended `Symbol` with the offset it was queried by). A name the sweep covered is answered from the table verdict alone — including the *rejection* verdict: a `nil` root means the demangler already refused the name once, and the same demangler backs the late path, so the query returns `nil` without re-demangling. Names genuinely outside the build sweep fall back to the per-image appendable side store (`lateDemangledNode(forName:)` — a `SharedNodeStore` self-held by `Storage`, deliberately not shared with `InternedNodeReferenceCache`'s evictable image store; `MetadataReader.demangleSymbolReference(for:in:)` forwards here): the demangle runs *outside* the cache lock (never hold `os_unfair_lock` across a potentially blocking large-stack hop), racing missers intern into the one store whose structural dedup hands them the same reference, and the name → verdict map keeps one canonical answer — while rejections are cached as `nil` verdicts and never retried (`rejectedLateNameCachesItsFailure` / `tableCoveredNameNeverEntersLateCache` / `concurrentLateQueriesShareOneStore` pin all three properties). Struct-layout changes in `MachOSymbols` need a `swift package clean` rebuild — SwiftPM incremental builds have been observed linking stale downstream objects (runtime SIGSEGV in `outlined destroy`). See `Documentations/Internal/NodeStoreMigrationPlan.md`. **Declaration values hold `NodeReference` (Stage 5a)**: `DefinitionName.node` (`TypeName`/`ProtocolName`/`ExtensionName`), `Variable/Function/SubscriptDefinition.node`, `FieldDefinition.typeNode` and `ExtensionDefinition.genericSignature` are `NodeReference`, not `Node`. Member definitions reference the per-image symbol store directly (a live definition keeps that store alive — the intended per-image recycling model); metadata-derived name trees (`MetadataReader` output feeding `TypeName`/`ProtocolName`/`ExtensionName`/generic signatures) go through `InternedNodeReferenceCache` (`Sources/MachOSymbols/`) — one `SharedNodeStore` per scope (upstream evolution 0010; the store's interning tables are the dedup, the cache keeps only what `SharedNodeStore` deliberately does not know: Mach-O scope keying and eviction): an image-keyed scope (evicted with the image; `SwiftDeclarationIndexer`'s cleanup removes it alongside the symbol store) and a process-keyed scope for the in-process helpers with no Mach-O handle, so every tree of a scope shares one store and name equality gets the `store ===` fast path. Do NOT call bare `NodeReference(interning:)` on a batch path — route through the cache. `TypeDefinition.index(in:)`'s field type trees intern into the same image store (cross-type dedup; the per-type builder+freeze store this replaced could only deduplicate within one type). See `Documentations/Internal/SharedNodeStoreMigration.md`. `MetadataReader`'s own demangle memo (`MetadataReaderCache`: mangled-name / context-offset / symbol-name verdicts, including cached `nil` rejections) stores `NodeReference`s into these same scope stores rather than class `Node` trees — a hit materializes a fresh tree (DAG sharing within one materialized tree is preserved, but returned instances are never shared across calls: key long-lived state structurally, never by `ObjectIdentifier` of a returned node), and the indexer cleanup drops the memo per image via `MetadataReader.removeCache(for:)` so it cannot outlive the scope store it references. See `Documentations/Internal/MetadataReaderCacheRetirement.md`. The `Name` types customize `Hashable` to STRUCTURAL semantics (`structurallyEquals` + `structuralHash`) because `NodeReference`'s intrinsic `Hashable` is store-identity based and would split equal keys minted into different stores; they are deliberately NOT `Codable` — upstream dropped `Node: Codable` (a mangled symbol already is the tree's serialized form: smaller, ABI-stable across toolchains, and sharing-preserving through the round trip), and nothing persists these values. A consumer that ever needs to should persist `mangleAsString(node)` and read it back with `demangleAsNode(_:)` rather than reintroducing a node encoding. **Any `Dictionary`/`Set` keyed on a `NodeReference` whose keys and lookups can come from DIFFERENT stores must key on `StructuralNodeReferenceKey` (`Sources/MachOSymbols/`, next to the stores it reconciles), never a bare `NodeReference`.** Different stores are the norm, not the exception: the frozen main symbol store, the late-name side store, and the cache's per-scope stores coexist within one image (and memory-pressure eviction rebuilds a cache scope onto a fresh store while old references keep the old one alive), so cross-store mixing remains the default assumption. Everything that mixes those with image-store references is keyed structurally: the override/vtable lookups (`methodDescriptorLookup` / `vtableOffsetLookup` in `TypeDefinition.index` → `DefinitionBuilder`), `DefinitionBuilder`'s accessor grouping and merged-thunk dedup (`accessorsByNode` / `canonicalIndexByFunctionNode` / `canonicalIndexByAllocatorNode`), and every `visitedNodes` "already claimed this symbol" set (`ExtensionDefinition` / `ProtocolDefinition` / `TypeDefinition` / `OverrideSymbolMatcher` / `ProtocolConformanceDumper`). Under bare keys these fail silently and differently: the override lookups drop the `override` keyword + vtable-offset comment (the Stage 5a regression fixed 2026-07-25), a subscript's getter and setter split into two buckets so the setter-only one is discarded, merged thunks emit their `func`/`init` twice, and a witness can be claimed twice. Bare-`NodeReference` keys are only safe for grouping/dedup WITHIN one `memberSymbols` batch (single hash-consed store, where structural equality coincides with index equality). **Transient demangling everywhere else (Stage 5c)**: `MetadataReader`, `RuntimeFieldLayoutBackend`, `TypedDumper`, `ClassHierarchyDumper`, `Symbol.demangledNode`, `SwiftLayout.ObjCClassIndex` and `SwiftDeclarationRendering.SpecializedMetadataNodeSubstitution` use `demangleAsNodeTransient` + `Node.createTransient`, so the global `NodeCache` no longer grows while browsing. Sources now carries zero cached `demangleAsNode(` call sites — do NOT reintroduce `demangleAsNode`/`Node.create` on unbounded inputs anywhere; a demangle whose tree is consumed and dropped must stay transient. The build sweep's `NodeStoreBuilder` pre-reserves its buffers via `reserveCapacity(expectedSymbolCount:)` (growing-only, interning-result-neutral) — keep that call adjacent to the builder's construction. +**Symbol indexing (NodeStore-backed)**: `SymbolIndexStore.Storage` no longer retains `Node` class trees. The build sweep demangles each symbol cache-free onto a transient tree (`@_spi(Internals) demangleAsNodeTransient`), classifies on that tree, and interns it into a per-image `NodeStoreBuilder`. Nothing touches the global `NodeCache`, and dropping a `Storage` releases the whole per-image footprint. Storage layout (Stage 3, offset-ized by evolution proposal 0001): one 16-byte `SymbolRow` per unique symbol name in a `SymbolTable` — canonical (cache-adjusted; raw and adjusted offset keys share one row) offset plus a packed reference to the name's bytes, with **no retained name `String`s**. Names resolve through one of two sources: a `MachOImage` row points straight into the image's mmap'd LINKEDIT string table (clean pages, zero-copy; the table consequently requires the image to stay loaded — vended values' `symbol` reads materialize from it), while `MachOFile` rows and export-trie names (decoded strings with no mapped home) live in the table's private contiguous byte buffer. The collection sweep is reader-split: the image leg tests `isSwiftSymbol` byte-level on `nameC` (`nameBytesHaveSwiftManglingPrefix`, mirroring `Demangling.getManglingPrefixLength`'s prefix list — pinned equal by `SymbolTableEquivalenceTests`), so a non-Swift symbol never materializes a name at all; the generic leg (files) keeps the `String` surface and appends Swift name bytes into the private buffer. Name → row lookup is a byte-level binary search over the `rowsSortedByName` permutation (`SymbolTable.row(forName:)`); the name-keyed dictionary exists only during the build and is discarded at freeze, which also makes exact-capacity copies of the accumulated buffers. The byte access layer is `UnsafeBufferPointer`-based deliberately — `Span`/`UTF8Span` are macOS 26-only at runtime, above this package's deployment floor. `symbolRowsByOffset` is a plain `Dictionary` (single keyed consumer, nothing iterates it in order). A parallel `rootNodeIndexByTableRow` array holds each row's demangled root, and every classification index stores 4-byte `UInt32` table rows (member/opaque keys are 4-byte `NodeStore.NodeIndex`); indexes accumulate in final row form during the sweep, so `freeze()` is followed by a plain move, not a conversion pass. `Symbol` stays the public 32-byte eager value (no `nlist` existential — the external-undefined bit is extracted at collection time into `isExternal`; vend paths materialize it on demand), and `DemangledSymbol` is a 32-byte value (table reference + row + `NodeReference`) whose `symbol` is computed; `compactValueLayouts` asserts all three layouts including `SymbolRow == 16`. **A `DemangledSymbol` stored into the declaration model must be detached first** (`detachedFromSharedTable()`): the shared `SymbolTable` is the right trade for the hundreds of thousands of values a query vends and drops, but a single stored survivor pins the whole table — and, for an image table, keeps its mapped-string-table reads tied to the loaded image — defeating `removeSubIndexer(_:)`, whose reason for existing is reclaiming per-image memory. Measured on SwiftUI (iOS 18.5, pre-0001 representation): 9,872 stored values referenced 9,506 distinct rows, 5.1% of a 185,988-row table, so detaching trades ~0.6 MB of small allocations for the whole table's retention (~19.9 MB then; smaller but still whole-table now). The six storing sites are `DefinitionBuilder`'s four construction points (`Accessor` for variables/subscripts, `FunctionDefinition` for allocators/functions) and `TypeDefinition`'s `deallocatorSymbol` / `destructorSymbol` assignments; `SymbolTableRetentionTests` fails the moment a new one forgets. Do NOT detach on the query path. Query APIs construct `DemangledSymbol`/`NodeReference` values at the exit boundary; APIs taking an externally demangled node (`memberSymbols(of:for:node:)` with `Node` or `NodeReference`, `opaqueTypeDescriptorSymbol(for:)`) match keys via `NodeReference.structurallyEquals(_:)` (same-store O(1), cross-store structural walk) — `opaqueTypeDescriptorSymbol(for:)` keys its entries structurally (`opaqueTypeDescriptorSymbolRowByMemberNode: [StructuralNodeReferenceKey: UInt32]`) so a print-time query is ONE hash probe, because a full scan there is O(descriptors × printed `some` returns). Bucketing by `DemanglingNode.identifier` and scanning the bucket was the earlier attempt and is NOT sufficient — the identifier is the member name, so in SwiftUI the `body` bucket alone holds hundreds of entries (every `some View` implementation) and the scan stayed quadratic. The structural key works on both sides because `StructuralNodeReferenceKey` also accepts a bare query `Node` (`init(querying:)`), hashing it through `Node.structuralHash(into:)`, which upstream keeps in step with `NodeReference.structuralHash(into:)` — so the caller's freshly demangled tree probes the frozen arena's keys without materializing or interning anything. `demangledNodeReference(for:)` matches on the symbol **name alone** — a demangled tree is a pure function of the name and the flat table holds one row per unique name, so comparing the row's canonical offset against the queried symbol's offset could only reject valid hits (it once rejected every symbol of a dyld-cache image, since `symbols(for:in:)` stamps each vended `Symbol` with the offset it was queried by). A name the sweep covered is answered from the table verdict alone — including the *rejection* verdict: a `nil` root means the demangler already refused the name once, and the same demangler backs the late path, so the query returns `nil` without re-demangling. Names genuinely outside the build sweep fall back to the per-image appendable side store (`lateDemangledNode(forName:)` — a `SharedNodeStore` self-held by `Storage`, deliberately not shared with `InternedNodeReferenceCache`'s evictable image store; `MetadataReader.demangleSymbolReference(for:in:)` forwards here): the demangle runs *outside* the cache lock (never hold `os_unfair_lock` across a potentially blocking large-stack hop), racing missers intern into the one store whose structural dedup hands them the same reference, and the name → verdict map keeps one canonical answer — while rejections are cached as `nil` verdicts and never retried (`rejectedLateNameCachesItsFailure` / `tableCoveredNameNeverEntersLateCache` / `concurrentLateQueriesShareOneStore` pin all three properties). Struct-layout changes in `MachOSymbols` need a `swift package clean` rebuild — SwiftPM incremental builds have been observed linking stale downstream objects (runtime SIGSEGV in `outlined destroy`). See `Documentations/Internal/NodeStoreMigrationPlan.md`. **Declaration values hold `NodeReference` (Stage 5a)**: `DefinitionName.node` (`TypeName`/`ProtocolName`/`ExtensionName`), `Variable/Function/SubscriptDefinition.node`, `FieldDefinition.typeNode` and `ExtensionDefinition.genericSignature` are `NodeReference`, not `Node`. Member definitions reference the per-image symbol store directly (a live definition keeps that store alive — the intended per-image recycling model); metadata-derived name trees (`MetadataReader` output feeding `TypeName`/`ProtocolName`/`ExtensionName`/generic signatures) go through `InternedNodeReferenceCache` (`Sources/MachOSymbols/`) — one `SharedNodeStore` per scope (upstream evolution 0010; the store's interning tables are the dedup, the cache keeps only what `SharedNodeStore` deliberately does not know: Mach-O scope keying and eviction): an image-keyed scope (evicted with the image; `SwiftDeclarationIndexer`'s cleanup removes it alongside the symbol store) and a process-keyed scope for the in-process helpers with no Mach-O handle, so every tree of a scope shares one store and name equality gets the `store ===` fast path. Do NOT call bare `NodeReference(interning:)` on a batch path — route through the cache. `TypeDefinition.index(in:)`'s field type trees intern into the same image store (cross-type dedup; the per-type builder+freeze store this replaced could only deduplicate within one type). See `Documentations/Internal/SharedNodeStoreMigration.md`. `MetadataReader`'s own demangle memo (`MetadataReaderCache`: mangled-name / context-offset / symbol-name verdicts, including cached `nil` rejections) stores `NodeReference`s into these same scope stores rather than class `Node` trees — a hit materializes a fresh tree (DAG sharing within one materialized tree is preserved, but returned instances are never shared across calls: key long-lived state structurally, never by `ObjectIdentifier` of a returned node), and the indexer cleanup drops the memo per image via `MetadataReader.removeCache(for:)` so it cannot outlive the scope store it references. See `Documentations/Internal/MetadataReaderCacheRetirement.md`. The `Name` types customize `Hashable` to STRUCTURAL semantics (`structurallyEquals` + `structuralHash`) because `NodeReference`'s intrinsic `Hashable` is store-identity based and would split equal keys minted into different stores; they are deliberately NOT `Codable` — upstream dropped `Node: Codable` (a mangled symbol already is the tree's serialized form: smaller, ABI-stable across toolchains, and sharing-preserving through the round trip), and nothing persists these values. A consumer that ever needs to should persist `mangleAsString(node)` and read it back with `demangleAsNode(_:)` rather than reintroducing a node encoding. **Any `Dictionary`/`Set` keyed on a `NodeReference` whose keys and lookups can come from DIFFERENT stores must key on `StructuralNodeReferenceKey` (`Sources/MachOSymbols/`, next to the stores it reconciles), never a bare `NodeReference`.** Different stores are the norm, not the exception: the frozen main symbol store, the late-name side store, and the cache's per-scope stores coexist within one image (and memory-pressure eviction rebuilds a cache scope onto a fresh store while old references keep the old one alive), so cross-store mixing remains the default assumption. Everything that mixes those with image-store references is keyed structurally: the override/vtable lookups (`methodDescriptorLookup` / `vtableOffsetLookup` in `TypeDefinition.index` → `DefinitionBuilder`), `DefinitionBuilder`'s accessor grouping and merged-thunk dedup (`accessorsByNode` / `canonicalIndexByFunctionNode` / `canonicalIndexByAllocatorNode`), and every `visitedNodes` "already claimed this symbol" set (`ExtensionDefinition` / `ProtocolDefinition` / `TypeDefinition` / `OverrideSymbolMatcher` / `ProtocolConformanceDumper`). Under bare keys these fail silently and differently: the override lookups drop the `override` keyword + vtable-offset comment (the Stage 5a regression fixed 2026-07-25), a subscript's getter and setter split into two buckets so the setter-only one is discarded, merged thunks emit their `func`/`init` twice, and a witness can be claimed twice. Bare-`NodeReference` keys are only safe for grouping/dedup WITHIN one `memberSymbols` batch (single hash-consed store, where structural equality coincides with index equality). **Transient demangling everywhere else (Stage 5c)**: `MetadataReader`, `RuntimeFieldLayoutBackend`, `TypedDumper`, `ClassHierarchyDumper`, `Symbol.demangledNode`, `SwiftLayout.ObjCClassIndex` and `SwiftDeclarationRendering.SpecializedMetadataNodeSubstitution` use `demangleAsNodeTransient` + `Node.createTransient`, so the global `NodeCache` no longer grows while browsing. Sources now carries zero cached `demangleAsNode(` call sites — do NOT reintroduce `demangleAsNode`/`Node.create` on unbounded inputs anywhere; a demangle whose tree is consumed and dropped must stay transient. The build sweep's `NodeStoreBuilder` pre-reserves its buffers via `reserveCapacity(expectedSymbolCount:)` (growing-only, interning-result-neutral) — keep that call adjacent to the builder's construction. ## Test Environment diff --git a/Documentations/Evolutions/0001-symbol-name-offsetization.md b/Documentations/Evolutions/0001-symbol-name-offsetization.md index 3b06ba07..12a8f271 100644 --- a/Documentations/Evolutions/0001-symbol-name-offsetization.md +++ b/Documentations/Evolutions/0001-symbol-name-offsetization.md @@ -128,7 +128,15 @@ final class SymbolTable: @unchecked Sendable { ### 源码兼容性（source compatibility） -**纯新增 / 无破坏。** 公开与 package 级 API 的签名、语义、返回值形态全部不变：`Symbol`、`Symbols`、`DemangledSymbol.symbol`、`Symbol.resolve`、`SymbolIndexStore` 的全部查询方法照旧。变化仅在 `MachOSymbols` 模块内部的驻留表示与 `Storage` 私有结构。`compactValueLayouts` 与 `SymbolTableRetentionTests` 钉住布局与 detach 契约不回归。 +**一处破坏性收窄，其余无破坏。** + +破坏项：`Symbol` 的 `nlist: (any NlistProtocol)?` 被 `isExternal: Bool` 取代。这是落地过程中的修订，原方案（本节初稿写的「纯新增 / 无破坏」）低估了保留 `nlist` 的代价——保留它意味着每个符号一个 40 字节 existential 拷贝，而全仓库唯一的消费点就是 external-undefined 这一个位（`buildStorageSweep` 的 `!symbol.nlist.isExternal`），与本案「消除每符号驻留开销」的目的直接冲突。破坏面： + +- `Symbol(offset:name:)` **仍然编译** —— 新旧初始化器的第三参都有默认值（`nlist: nil` → `isExternal: false`）； +- 破坏的是显式传 `nlist:` 的构造调用，以及读 `.nlist` 属性的代码； +- 不可恢复的信息：section index、`n_desc`、weak / no-dead-strip 等标志位。需要它们的调用方要自行经 MachOKit 的符号接口读取。 + +其余公开与 package 级 API 的签名、语义、返回值形态不变：`Symbols`、`DemangledSymbol.symbol`、`Symbol.resolve`、`SymbolIndexStore` 的全部查询方法照旧。其余变化仅在 `MachOSymbols` 模块内部的驻留表示与 `Storage` 私有结构。`compactValueLayouts` 与 `SymbolTableRetentionTests` 钉住布局与 detach 契约不回归。 ### ABI 兼容性（条件项） @@ -147,8 +155,9 @@ final class SymbolTable: @unchecked Sendable { ## API 演进与废弃策略 -- 无被替代的公开 API，无废弃标注需求。 -- 无 semver major 跃迁：源码兼容的内部表示变更，随下一次常规版本发布（`Version.swift` bump 时在 changelog 记录内存收益）。 +- `Symbol.nlist` → `Symbol.isExternal`：直接移除而非 `@available(*, deprecated)` 标注。废弃标注在这里达不到目的——保留旧属性就等于保留它要消除的每符号 existential 存储，而本案的全部收益正来自不再持有它。 +- 下游需确认：RuntimeViewer（本案验收方）。仓库内已确认零消费者（`Symbol.nlist` 的读取点仅 `buildStorageSweep` 一处，已随本案改写）。 +- 随下一次常规版本发布，`Version.swift` bump 时在 changelog 记录内存收益**与这处破坏性变更**。 ## 落地步骤 diff --git a/Documentations/Internal/ProjectEvolutionLog.md b/Documentations/Internal/ProjectEvolutionLog.md index b60cbc53..9b79de94 100644 --- a/Documentations/Internal/ProjectEvolutionLog.md +++ b/Documentations/Internal/ProjectEvolutionLog.md @@ -827,6 +827,21 @@ --- +## 39. PR #103 第二轮 review：15 条发现的四问、复核与实现 + +- **时间段**：2026-08-13（`feature/node-store-migration` 的第二轮 max 级 review；第一轮见第 37 节）。 +- **动机**：第一轮修复落地后再审一遍，产出 15 条发现。四问（复现 / 基线对比 / 值不值得修 / 既往修复）不是走过场——它推翻了初版结论中的三条，全部靠查证而非推理：F1「本 PR 引入」实为基线既有（main 的 `printTypeHeader` 本身就含两处会抛的 `try`，初版只盯着换掉的那个参数）、F7 的破坏面被默认参数缩小（`Symbol(offset:name:)` 两边都编译）、F15 的 `cls` 是 main 上沿用而非新发明。另有 F2 因结构上触发不到而降为防御性、F13 因无法复现且 main 相同而记入已裁决清单。 +- **交叉复核**：15 条结论交同项目另一会话独立复核，四点实质修正全部采纳。其中两条是原查证不足：F10 曾因「未核实上游」撤掉的论点，经复核在上游 `DemangleInterface.swift:56-67` 找到文档契约而**恢复**（无参数 kind 即使走 transient 也解析到进程级 `NodeFactory` 单例，故修法不能简单加强 `!==` 否则假失败）；F5 的「修法现成」被指出说过头（`StructuralNodeReferenceKey` 只包 `NodeReference`，查询侧是裸 `Node`，无零成本包装），且该处并不违反 AGENTS.md 那条硬规则。 +- **落地（代码 6 条 + 横向 4 处 + 文档/测试）**：A1 A/B 验收脚本双边失败但退出码不同时计入差异（第一轮 H4 同根因的第二个实例）；A2 `printCatchedThrowing` 停止 `print(error)` 改派发 `.definitionPrintFailed`，13 个调用点补 context——这正是 Issue #102 明确提出而本 PR 原先只做了三分之一的另外两条；A3 diff 头行渲染失败丢弃整条声明（空 header 下渲染成员是非法 Swift）；B1 opaque 查询从「按 identifier 分桶后线性 `structurallyEquals`」改为结构化哈希一次探测（`StructuralNodeReferenceKey` 增加 `init(querying:)` 的裸 `Node` 形态，两侧哈希由上游保证一致）；B2 缓存回收资格拆成三位分别 claim（只有 symbol store 必然是 indexer 建的，另两份 SwiftLayout / 渲染器 / SwiftSpecialization 也在填），注册改 identity-keyed 顺带吸收 B3 的 check-then-set 竞态；F15 `cls` → `classWrapper`。横向排查另找到 4 处 `print(error)` 全部改写 stderr，`Sources/` 下现已归零。 +- **裁决（2 条，记入 [ReviewAdjudications.md](ReviewAdjudications.md) A7–A8）**：索引器与 dump 路径 witness 匹配的 print options 分歧（无法构造触发场景，main 相同，无 fixture）；`updateConfiguration` 的 re-prepare 因 `isPrepared` 早退而是 no-op（零调用点，RuntimeViewer 硬编码使其不可达）。 +- **踩坑留痕**：B1 改了 `SymbolIndexStore.Storage` 字段结构却未按 AGENTS.md 立即 `swift package clean`，导致增量构建链接 stale object——`swift build` 连续报成功、全量测试跑到 484 例后 SIGSEGV 且零断言失败；clean 后才暴露真实编译错误（把一个 `NodeReference` 属性改成了 optional，破坏三个既有调用点）。另：sibling `swift-demangling` 是共享可变状态，构建中途撞上另一会话的半改状态，改为 pin 到 detached 只读 worktree 解决。 +- **上游合流**：`swift-demangling` 同期修掉畸形符号的 SIGTRAP / 整数溢出 / 死循环（模糊语料 trap 与 hang 归零）。这与 A2 是同一条线的两端——在此之前 `printCatchedThrowing` 就算 catch 了也拦不住进程级信号，两边合上后 per-definition catch 才真正成立。 +- **验证**：全量 1413 测试 / 266 suite / `swift test` 退出码 0；A/B 脚本新增 5 条标准库自测；依赖 pin 分两轮（先 `6eb3fc7` 确认自身改动，再 `5d2b476` 复跑）以分离变量。 +- **文档**：[TaskReports/2026-08-13-pr103-review-round-two-fixes.md](TaskReports/2026-08-13-pr103-review-round-two-fixes.md)、[ReviewAdjudications.md](ReviewAdjudications.md)（A7–A8）、提案 0001 兼容性一节更正、AGENTS.md 的 opaque 索引与缓存回收两段同步。 +- **对应版本**：`0.15.1` 之后、下一次 bump 之前（`feature/node-store-migration` 分支）。 + +--- + ## 维护约定 1. **每个非平凡批次结束时必须在本文追加/更新一节**（新工作弧新增一节；延续既有弧则在该节 diff --git a/Documentations/Internal/ReviewAdjudications.md b/Documentations/Internal/ReviewAdjudications.md index e627cef5..ae10af0b 100644 --- a/Documentations/Internal/ReviewAdjudications.md +++ b/Documentations/Internal/ReviewAdjudications.md @@ -94,3 +94,27 @@ - **既往修复**：`17ad4358`（引入 ranking）→ `6647359e`（跨 cache 文件生效）→ 本轮第三次审视。三轮都没落的 plain-`.dylib` 用例这次落了：`DyldCacheEndToEndLookupTests`（当前系统 cache 上 `libswiftCore` 解析到 `/usr/lib/swift/libswiftCore.dylib`、`SwiftUI` 解析到原生 framework 本体且非 iOSSupport）。 - **代码锚点**：`DyldCacheEndToEndLookupTests` 的套件注释。 - **复审条件**：① 出现高频调用 `machOFile(by:)` 的新消费形态（当前每 CLI 调用一次）；② MachOKit 上游暴露 `(imagePath, fileOffset)` 级枚举后，「只构造赢家」无需复刻内部细节——届时可顺手做。 + +--- + +## A7 — 索引器与 dump 路径的 conformance witness 匹配用不同 print options（PR #103 第二轮 review） + +- **裁决**：本轮不修，记录待查（2026-08-13）。 +- **发现**：`ExtensionDefinition.index(in:)` 的 witness 匹配（`ExtensionDefinition.swift:150`）用 `.interfaceTypeBuilderOnly` 打印符号侧类型名，而 dump 路径的同款循环（`ProtocolConformanceDumper.swift:185`）用 `.interfaceType`。两者只差一个 `.displayObjCModule` 标志，所以一个 ObjC 导入类型在前者打印成 `__C.NSObject`、在后者是 `NSObject`。两条路径的 fallback 子句都查 `PrimitiveTypeMappingCache.shared.storage(in:)?.primitiveType(for: typeName)`，而 `PrimitiveTypeMapping` 的键是**裸** descriptor 名（`PrimitiveTypeMapping.swift:26`，`descriptor.name(in: machO)`，从不带模块限定）。推论是：对某个只能经 primitive mapping 匹配上的 conformance，`swift-section dump` 绑定到具体 witness 符号，而 `swift-section interface` 落到 requirement 分支。 +- **复现 / 是否误报**：**未能构造出触发场景，因此不作为已确认缺陷**。print options 的分歧与 mapping 键的裸名形态都已逐行核对属实，但触发还需要一个"带 ObjC 导入 typedef 原始类型、且携带 resilient witness"的真实框架；仓库内无 fixture 覆盖（`grep -rn 'extension __C\.' Tests/` 为空）。另需注意：`typeName` 参数是外部传入的 `extensionName.name`，不是用同一 option 现算的，所以 `symbolTypeName == typeName` 那一半是否受影响也取决于 `ExtensionName` 的构造选项——初版分析曾误断为"不可观测"，实际未定。 +- **与 main 基线对比**：**main 上完全相同**（aa38ff5 的两处有一模一样的 option 分歧与裸名键）。非本 PR 引入；本 PR 只是重写了这几行所在的区域。 +- **为什么本轮不修**：无法复现的旧问题，改任一侧的 print option 都会影响 witness 绑定这一敏感路径，而没有测试能证明改动方向正确。盲改的风险大于收益。 +- **既往修复**：无。两处自各自模块拆分以来就是这样。 +- **复审条件**：① 出现 `extension __C.` 的真实用户报告或 fixture；② 为 ObjC 导入类型的 resilient conformance 补 fixture 后重测两条路径的输出差异。届时正确修法是让两条路径共享同一个匹配 helper（含同一套 print options），而不是各自调 option。 + +--- + +## A8 — `updateConfiguration` 的 re-prepare 因 `isPrepared` 早退而是 no-op（PR #103 第二轮 review） + +- **裁决**：不修（2026-08-13）。 +- **发现**：`SwiftDeclarationIndexer.updateConfiguration(_:)` 在 `showCImportedTypes` 变化时调 `try await prepare()` 重建索引，但 `prepare()` 第一行是 `if isPrepared { return }`，而 `isPrepared` 在首次 prepare 结束时就置 true——所以配置变了索引并不会重建，这个分支实际是空操作。 +- **复现 / 是否误报**：机制属实。但**当前无消费者能触发**：仓库内 `updateConfiguration` 零调用点；已知的下游消费者 RuntimeViewer 把 `showCImportedTypes` 硬编码为 false，那个分支永不进入。 +- **与 main 基线对比**：main 字节相同。非本 PR 引入。 +- **为什么不修**：正确修法（重置 `isPrepared` 与全部 storage 后重建）等于给一个无人调用的路径加一次全量重索引，且需要想清楚重建期间已 vend 出去的 definition 引用怎么办（它们持有 per-image store）。在没有真实消费者定义期望语义之前，改动只会引入未经验证的行为。 +- **既往修复**：无。 +- **复审条件**：任一消费者真正开始在运行时切换 `showCImportedTypes`——届时先定义"重建期间旧 definition 引用的语义"，再动实现。 diff --git a/Documentations/Internal/TaskReports/2026-08-13-pr103-review-round-two-fixes.md b/Documentations/Internal/TaskReports/2026-08-13-pr103-review-round-two-fixes.md new file mode 100644 index 00000000..bfecc082 --- /dev/null +++ b/Documentations/Internal/TaskReports/2026-08-13-pr103-review-round-two-fixes.md @@ -0,0 +1,112 @@ +# PR #103 第二轮 review：15 条发现的四问、裁决与实现 + +- **日期**：2026-08-13 +- **分支**：`feature/node-store-migration`（PR #103 head，基线 `aa38ff5`） +- **相关**：第一轮见 [2026-08-09-pr103-review-fix-implementation.md](2026-08-09-pr103-review-fix-implementation.md) 与 [Roadmaps/2026-08-09-pr103-review-findings.md](../../../Roadmaps/2026-08-09-pr103-review-findings.md) + +--- + +## 问题 + +对 PR #103 跑第二轮 max 级 code review，产出 15 条发现。按 CLAUDE.md 的「发现必答四问」逐条查证（能否复现 / main 是否也有 / 值不值得修 / 以前修过吗），再决定修复清单。 + +第一轮已修/已裁决的条目（`ReviewAdjudications.md` A1–A6、`Roadmaps/2026-08-09` 的 B1/H1–H4/M1–M6/L1–L4）不重复审。 + +## 调研 + +### 四问的关键产出 + +四问不是形式：它推翻了初版结论中的三条，且**每一条都是靠查证而非推理翻的**。 + +- **F1（diff 头行被吞）定性改了**。初版说「main 上读存储属性、`try?` 恒成功，是本 PR 引入」。实际 main 的 `printTypeHeader` 本身就含 `try await typeDefinition.index(in:)` 与 `try await renderTypeDeclarationHeader(...)`，两者都会抛——初版只盯着 `for:` 参数换了什么，漏看了函数体其余的 `try`。**是基线既有缺陷**，本 PR 只是多加了一个（实践中几乎不抛的）materialization 项。 +- **F7（`Symbol.nlist` 移除）破坏面比初判小**。新旧初始化器第三参都有默认值，`Symbol(offset:name:)` 两边都编译；真正破的只有显式传 `nlist:` 与读 `.nlist`。 +- **F15（`cls` 缩写）撤回「新引入」的定性**。main `TypeDefinition.swift:202` 已是 `if case .class(let cls)`，PR 只换了绑定形式、沿用旧名。 + +另有两条经查证降级：**F2** 的竞态在当前所有已知调用方结构上都触发不到（RuntimeViewer 的 `prepare` 只调一次，`updateConfiguration` 因 `showCImportedTypes` 硬编码 false 永不进 re-prepare 分支），降为防御性；**F13** 无法构造触发场景，且 main 上分歧完全相同，记入已裁决清单。 + +### 四问查到的历史事实 + +- **F2/F5/F8** 所在的 `PerImageCacheEvictionRegistry` 本身就是第一轮 M6 的修复（代码注释里写明）。本轮发现的是 M6 修法引入的新形态：计数器可被重复注册撑住、claim 判据只覆盖三份缓存中的一份。 +- **F3** 的同一函数里就有第一轮 H4 的修复注释（"a harness that cannot fail is worse than no harness"）。本轮是同一根因的第二个实例。第一轮已把「新验证工具无自检」列为三大共因之一。 +- **F6** 有公开 Issue **#102**（OPEN），它正是本 PR 那 5 个新 catch 调用点的动机。Issue 作者明确提了三条建议——① 保留部分结果 ② 失败时派发事件 ③ 库不要 print 到 stdout——**本 PR 只做了 ①**。 +- **F12** 由 `aeed373`（2026-05-07，早于 tag 0.14.1）引入，是已知的结构性 flaky，「至今未根治」。 + +### 交叉复核 + +把 15 条结论交给同项目的另一个会话独立复核。它的四点实质修正全部被采纳（F1 定性、F2 降级、F10 论点恢复、F5 修法保留），其中两条是我查证不足： + +- **F10** 我曾因「未核实上游实现」而撤掉一个论点（无参数 kind 即使走 transient 路径也解析到进程级 `NodeFactory` 单例）。复核方在上游 `DemangleInterface.swift:56-67` 找到了白纸黑字的文档契约，**论点成立，应恢复**——且它直接影响修法：不能简单加强 `!==` 断言，否则会假失败。 +- **F5** 我说「修法现成，用 `StructuralNodeReferenceKey`」说过头了。该类型只包 `NodeReference`，而查询侧是裸 `Node`，没有零成本包装。复核方还指出这处**并不违反** AGENTS.md 那条硬规则（规则禁的是「裸 `NodeReference` 做键」，此处根本没有 node 键字典）。 + +## 最终方案 + +批准清单分三批。B3（注册竞态）被 B2 的 `ObjectIdentifier` 方案吸收，不单独实现。 + +| 批 | 条目 | 内容 | +|---|---|---| +| A | A1 | A/B 验收脚本：双边失败但退出码不同时计入差异 | +| A | A2 | 库不再写 stdout；`printCatchedThrowing` 派发 `.definitionPrintFailed` | +| A | A3 | diff 头行渲染失败时丢弃整条声明 | +| B | B1 | opaque 查询从桶内线性扫描改为结构化哈希一次探测 | +| B | B2 | 缓存回收资格拆三位；注册改 identity-keyed（吸收 B3） | +| C | C1 | 提案 0001 兼容性一节如实描述 `nlist` 破坏 | +| C | C2 | eviction 测试采样改 `#require` + 新增非 indexer 场景用例 | +| C | C8 | F13 / `updateConfiguration` no-op 记入已裁决清单 | + +## 实际执行 + +### 每条修复都先证明测试会红 + +按规矩「先写复现测试并确认修复前失败」。三处的失败实录： + +- **A1**：回退脚本修复后 `exit=1`，正是 `testBothSidesFailingWithDifferentExitCodesCountsAsADifference` 与 `testDifferingExitCodesFailARunWhoseOtherPairsAreIdentical` 两条红。 +- **A2**：把 helper 临时改回旧行为（`print(error)` + 不派发）后两条红，且症状与 Issue #102 报告的完全一致——`printFailureNames → []`（零事件）、stdout 捕获到 `offsetOutOfBounds`。 +- **A3**：回退 `header` 修复后 `unrenderableTypeHeaderDropsTheWholeDeclaration` 红（输出多出了本该丢弃的内容）。 +- **B2**：把三个 claim 临时改回全部从 symbol store 采样后，`indexerDoesNotEvictCachesItDidNotBuild` 的两条断言红（两份非 indexer 建的缓存被误清），另两条测试不受影响。 + +注意 A2/B2 的验证**不能**用 `git stash` 做：stash 掉实现会让调用点编译失败，那样的 `exit=1` 是编译错误而非测试失败，不构成证据。改用「临时把实现改回旧行为、保持签名可编译」。 + +### 横向排查 + +A2 确认为真后按规矩全库搜同类，另找到 **4 处** `print(error)`，全部在生成 interface 的路径上，都会污染输出：`Node+OpaqueType.swift:85`、`MultiPayloadEnumDescriptorCache.swift:48`、`SwiftInterfaceBuilderDependencies.swift:31,42`、`SwiftInterfaceBuilder.swift:98`。这几处都够深、拿不到 dispatcher，统一改写 stderr。现在 `Sources/` 下 `print(error)` 归零。 + +F15 的 `cls` 改成 `classWrapper`（10 处引用）。`MetadataProtocol.swift` 另有 6 处 `cls`，但该文件**完全未被本 PR 改动**，留给单独的清理提交。 + +## 验证 + +- **全量回归**：1413 个测试 / 266 个 suite / `swift test` 退出码 **0**（`--skip IntegrationTests`）。 +- **A/B 脚本自测**：新增 `Scripts/test-run-rendering-ab-verification.py`，5 条，仅用标准库，`python3` 退出码 0。 +- **两轮依赖 pin**：先在 `swift-demangling` `6eb3fc7` 上确认自身改动为绿，再挪到 `5d2b476` 复跑，以便出问题时能分清是自身改动还是依赖变更。 + +退出码一律取 `swift test` 自身的，不看任何摘要——`FULL SUITE EXIT=` 由命令显式追加进日志。 + +## 偏差与踩坑 + +### 1. 增量构建的 stale object 让「构建通过」失去意义 + +B1 改了 `SymbolIndexStore.Storage` 的字段结构。AGENTS.md 早就写明这类改动需要 `swift package clean` 重建，我没照做，代价是两层假象： + +1. `swift build` 连续报成功； +2. 全量测试跑到 484 个用例后 **SIGSEGV**（signal 11），零断言失败。 + +`clean` 之后才暴露真正的编译错误：B1 把 `StructuralNodeReferenceKey.reference` 从 `NodeReference` 改成了 `NodeReference?`，破坏三个既有调用点（`DefinitionBuilder.swift:162/175`、`ProtocolDumper.swift:124`），增量构建一直在链接旧 object 所以从未报错。 + +**修法**：`reference` 恢复非可选，lookup-only 形态走 `preconditionFailure`。注释里写清了它与 `PackedNameReference` 的区别——后者是二进制输入驱动、必须降级不得 trap，这里是本模块自己 API 的误用防护。 + +**教训**：改 `MachOSymbols` 的 struct 布局后应当立刻 clean，而不是等回归报怪症状。 + +### 2. sibling 依赖是共享可变状态 + +构建中途撞上 `swift-demangling` 的 worktree 被另一个会话半改（`maxTypeSize` 未定义、`input file was modified during the build`）。没有碰对方工作区，改为建一个 detached 的只读 worktree 指向已推送的 tip，并把软链改指过去： + +```bash +git -C /Volumes/Code/Personal/swift-demangling worktree add --detach /tmp/claude/pinned/swift-demangling +``` + +原软链目标是 `/Volumes/Code/Personal/swift-demangling/.claude/worktrees/swift-demangling`，一行命令可恢复。 + +### 3. 上游同期修掉了「catch 兜不住」的根因 + +`swift-demangling` 在 `5d2b476` 修了畸形符号导致的 SIGTRAP / 整数溢出 / 死循环（116 万条模糊语料重扫，trap 与 hang 归零）。这与 A2 是同一条线的两端：**在此之前，`printCatchedThrowing` 就算 catch 了也拦不住进程级信号**。两边合上之后 per-definition catch 才真正成立。 + +同期确认 `printSemantic`（`Node+.swift:87` 直接调 `DemanglingPrinter.print`）**零改动**：引擎静态入口逐字节未变，`StackSafeExecutor` 保护仍在；新增的 `runPrintWalk(using:)` 写死 `Target == String`，不是它的替代路径。已把这一点连同「静态派发遮蔽」的复发形状写进该处注释。 diff --git a/Sources/SwiftDeclarationRendering/Extensions/Node+.swift b/Sources/SwiftDeclarationRendering/Extensions/Node+.swift index 00f8d2ac..7884953c 100644 --- a/Sources/SwiftDeclarationRendering/Extensions/Node+.swift +++ b/Sources/SwiftDeclarationRendering/Extensions/Node+.swift @@ -83,6 +83,29 @@ extension DemanglingNode { /// is where the repo does that, because it owns a loop; the printer's own /// loop lives in `SwiftDeclarationPrinter` and is `async`, which a /// synchronous wrapper cannot enclose. + /// + /// **Do not "modernize" this onto `runPrintWalk(using:)`.** Upstream added + /// that protocol requirement as the dispatch hook behind + /// `print(using:) -> String`, for the single purpose of letting a + /// `NodeReference`'s arena walk be selected in generic and existential + /// contexts. It returns `String` because that is what it dispatches for — + /// a custom `Target` is out of its remit by design, not by oversight, so it + /// is not "the newer way to print" and there is nothing here to migrate to + /// it. For a non-`String` target the engine's static + /// `DemanglingPrinter.print(_:options:)` is the only + /// entry point, and will remain so. It is byte-for-byte unchanged across + /// the `runPrintWalk` introduction — including the + /// `StackSafeExecutor.executeWithUncheckedSendability` wrapper this comment + /// exists to explain (verified upstream on a deliberately 512KB-stacked + /// thread against 600 levels of nested generics, which survives only + /// because of that hop). + /// + /// Worth noting *why* the shadowing hazard above is a recurring shape + /// rather than a one-off: upstream hit the mirror image of it in the same + /// period — a concrete method silently shadowing a protocol-extension + /// member, where this one was a concrete overload silently shadowing the + /// stack-guarded generic. Swift's "more specific wins" static dispatch + /// swaps the implementation in both directions with no diagnostic. public func printSemantic(using options: DemangleOptions = .default) -> SemanticString { DemanglingPrinter.print(self, options: options) }