fix(plugins): exclude transitive leveldbjni-all from :crypto#6738
fix(plugins): exclude transitive leveldbjni-all from :crypto#6738barbatos2011 wants to merge 6 commits intotronprotocol:developfrom
Conversation
Toolkit.jar built on x86_64 throws NoSuchMethodError on org.iq80.leveldb.Options.maxBatchSize(I) at runtime when running `db archive` / `--rebuild-manifest`. Root cause: the :crypto module added in PR tronprotocol#6637 brings in :common -> :platform transitively, which pulls in org.fusesource.leveldbjni:leveldbjni-all:1.8. The direct :platform dependency on x86 already excludes leveldbjni-all (so the io.github.tronprotocol fork at 1.18.2 can be used instead), but the transitive path through :crypto does not. Both jars contain org/iq80/leveldb/Options.class; in the fat jar the duplicate-entry write order leaves the 1.8 copy, which lacks Options.maxBatchSize(int). Why tests didn't catch it: - :plugins:test classpath has both jars side-by-side; the JVM classloader returns the first match, which is 1.18.2 (declared directly), so tests pass. - The fat jar binaryRelease task uses zipTree to merge classes; duplicate entries are overwritten by later writes, so the surviving Options.class can be the 1.8 copy. Same classpath, opposite winner. - ARM64 path declares only :platform directly (no 1.18.2), so there is no conflict; Archive tests are also excluded on ARM64. Fix: exclude org.fusesource.leveldbjni:leveldbjni-all from the :crypto implementation, mirroring the existing exclusion on the direct :platform dependency. After this change the x86 runtimeClasspath contains only io.github.tronprotocol:leveldbjni-all:1.18.2, and the fat jar's Options.class always exposes maxBatchSize(int). Verified on x86_64 + JDK 8 (eclipse-temurin:8-jdk container): - :plugins:dependencies no longer lists leveldbjni-all 1.8 - java -jar Toolkit.jar db archive -d <empty> exits 0 with the expected "directory does not contain any database" message - :plugins:test passes
After `./gradlew clean build` finishes, run the freshly built Toolkit.jar through a few picocli subcommands so that runtime class-loading errors in the fat jar are caught by CI rather than slipping into a release. Without this step, dependency-conflict bugs that survive `:plugins:test` can still ship in the fat jar, because: - the test classpath keeps source jars separate (classloader returns the first match), but - the fat jar merges class entries with last-write-wins, so a different copy of a duplicated class can end up in the artifact. The recent NoSuchMethodError on Options.maxBatchSize is exactly this shape of bug: tests passed, fat jar broke. The smoke test catches it. Smoke commands run on every platform build (macos / ubuntu-arm / rockylinux-x86 / debian11-x86): - `java -jar Toolkit.jar help` — exercises the top-level command tree - `java -jar Toolkit.jar db --help` — loads all db subcommands - `java -jar Toolkit.jar db archive -h` — directly exercises the leveldb Options path that previously broke - `java -jar Toolkit.jar keystore --help` — exercises the :crypto module path added by the recent keystore migration Each step adds ~10-15s; jobs run in parallel so PR critical path increases by at most ~15s.
| // declared below on x86. Both jars contain org/iq80/leveldb/Options.class; | ||
| // in the fat jar the duplicate-entry write order can leave the 1.8 copy, | ||
| // which lacks Options.maxBatchSize(int) and breaks `db archive` at runtime. | ||
| exclude group: 'org.fusesource.leveldbjni', module: 'leveldbjni-all' |
There was a problem hiding this comment.
The fix itself is correct, but the excludes aren't aligned.
Two ways to make this consistent — pick one:
- A. Add the other two excludes on
:crypto— ifpluginsshould NOT bundlezksnark/commons-io. - B. Remove the other two excludes on
:platform— ifpluginsactually NEEDSzksnark/commons-io.
There was a problem hiding this comment.
Good catch — verified via :plugins:dependencyInsight that both zksnark-java-sdk and commons-io were indeed leaking through :crypto -> :common -> :platform. Went with option A to preserve the existing :platform exclusion intent. Pushed 69d7f2156 aligning all three excludes on :crypto. Re-verified on x86_64 + JDK 8: runtimeClasspath no longer contains any of the three, smoke still passes.
There was a problem hiding this comment.
Update: option A worked for leveldbjni-all but broke :plugins:test on x86_64 (rockylinux CI just failed). DbLiteTest boots a Spring context that loads ZksnarkInitService, which actually pulls in classes from both commons-io (FileUtils) and zksnark-java-sdk (LibrustzcashWrapper) at runtime. Once both :platform and :crypto excluded those two, x86 testRuntimeClasspath lost them entirely (ARM64 was fine because the ARM branch declares :platform without excludes).
So the :platform-side excludes for zksnark and commons-io are deduplication, not a 'kept out of plugins' intent — the artifacts must keep arriving via :crypto -> :common -> :platform. Reverted those two in 320da62c3 and added a comment explaining why the asymmetry is intentional. leveldbjni-all remains mirrored because that one is the actual classpath conflict. :plugins:test passes again on x86 docker and smoke is still green.
There was a problem hiding this comment.
The binaryRelease task in plugins/build.gradle declares
dependsOn (project(':protocol').jar, project(':platform').jar)but its body consumes outputs from :crypto:jar and :common:jar via runtimeClasspath. Gradle prints an implicit_dependency warning and disables execution optimizations to compensate; under parallel / incremental / partial-build scenarios this can lead to the fat jar being assembled before :crypto:jar or :common:jar is up-to-date, producing broken artifacts or build failures. Future Gradle versions will likely promote this to a hard error.
Recommend adding project(':crypto').jar and project(':common').jar to the dependsOn list.
The log
> Task :plugins:buildToolkitJar
Execution optimizations have been disabled for task ':plugins:buildToolkitJar' to ensure correctness due to the following reasons:
- Gradle detected a problem with the following location: '/Users/boson/IdeaProjects/java-tron/crypto/build/libs/crypto-1.0.0.jar'. Reason: Task ':plugins:buildToolkitJar' uses this output of task ':crypto:jar' without declaring an explicit or implicit dependency. This can lead to incorrect results being produced, depending on what order the tasks are executed. Please refer to https://docs.gradle.org/7.6.4/userguide/validation_problems.html#implicit_dependency for more details about this problem.
- Gradle detected a problem with the following location: '/Users/boson/IdeaProjects/java-tron/common/build/libs/common-1.0.0.jar'. Reason: Task ':plugins:buildToolkitJar' uses this output of task ':common:jar' without declaring an explicit or implicit dependency. This can lead to incorrect results being produced, depending on what order the tasks are executed. Please refer to https://docs.gradle.org/7.6.4/userguide/validation_problems.html#implicit_dependency for more details about this problem.
There was a problem hiding this comment.
Confirmed locally — ./gradlew clean && ./gradlew :plugins:buildToolkitJar immediately fails with Cannot expand ZIP '.../crypto/build/libs/crypto-1.0.0.jar', exactly as you predicted. Pushed 0aebaffb3 adding :crypto:jar and :common:jar to the dependsOn list. After the fix the same command builds clean with --warning-mode all and no implicit_dependency warning. Thanks for catching this.
Per review on PR tronprotocol#6738: the direct :platform dependency on x86 already excludes leveldbjni-all, zksnark-java-sdk, and commons-io, but the transitive :crypto -> :common -> :platform path was only excluding leveldbjni-all. Verified via :plugins:dependencyInsight that zksnark-java-sdk and commons-io were leaking back into x86 runtimeClasspath through :crypto. Mirror the remaining two excludes on :crypto so both paths to :platform agree on what is kept out of the plugins fat jar. This preserves the intent of the existing :platform excludes rather than relaxing them. Re-verified on x86_64 + JDK 8: - runtimeClasspath no longer contains leveldbjni-all 1.8, zksnark-java-sdk, or commons-io:commons-io - java -jar Toolkit.jar db archive -d <empty> still exits 0 - :plugins:test, checkstyleMain, checkstyleTest all pass
…Release Per review on PR tronprotocol#6738 (halibobo1205): the binaryRelease Jar task zips up runtimeClasspath contents, which on this branch includes :crypto:jar and :common:jar (introduced by tronprotocol#6637), but the task only declared dependsOn on :protocol:jar and :platform:jar. Gradle was emitting an implicit_dependency warning and disabling execution optimizations to compensate. Reproduced locally: $ ./gradlew clean && ./gradlew :plugins:buildToolkitJar > Cannot expand ZIP '.../crypto/build/libs/crypto-1.0.0.jar' as it does not exist. Add :crypto:jar and :common:jar to dependsOn so partial / parallel / incremental builds can no longer race against missing dependency jars. Verified: - ./gradlew clean :plugins:buildToolkitJar passes with --warning-mode all and no implicit_dependency warning - Smoke (db archive -d <empty>) still exits 0 on x86 docker - :plugins:test, checkstyleMain, checkstyleTest all pass
CI on rockylinux (x86_64 + JDK 8) failed under 0aebaff: org.tron.plugins.rocksdb.DbLiteRocksDbTest > testToolsWithRocksDB FAILED org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'zksnarkInitService' ... NoClassDefFoundError: org/apache/commons/io/FileUtils After commit 69d7f21 mirrored the :platform excludes onto :crypto (reviewer's option A in tronprotocol#6738), x86 testRuntimeClasspath lost commons-io:commons-io and io.github.tronprotocol:zksnark-java-sdk entirely: both arrived only via :crypto -> :common -> :platform, which is now also excluding them. ARM64 was unaffected because the ARM64 branch declares :platform without exclusions. Diagnosis: the DbLiteTest Spring boot path loads org.tron.core.zen.ZksnarkInitService, which references both org.apache.commons.io.FileUtils and org.tron.common.zksnark.LibrustzcashWrapper at runtime. Reproduced on x86_64 docker: - removing only commons-io exclude -> FAILED at LibrustzcashWrapper - removing both exclusions -> testToolsWithRocksDB PASSED The :platform-side excludes for these two artifacts are therefore deduplication only, not a "kept out of plugins" intent. The leveldbjni-all exclude is the only one that must be mirrored, because that one is the actual classpath conflict. This commit drops the zksnark-java-sdk and commons-io excludes from :crypto and adds a comment recording why they are intentionally asymmetric with :platform. Verified on x86_64 + JDK 8 (eclipse-temurin:8-jdk): - :plugins:test passes (was 4 failed under 0aebaff) - runtimeClasspath still does NOT contain leveldbjni-all 1.8 - runtimeClasspath now contains commons-io and zksnark-java-sdk - Toolkit.jar smoke (db archive -d <empty>) still exits 0
| // jars carry org/iq80/leveldb/Options.class; the duplicate-entry write | ||
| // order in the fat jar can leave the 1.8 copy, which lacks | ||
| // Options.maxBatchSize(int) and breaks `db archive` at runtime. | ||
| exclude group: 'org.fusesource.leveldbjni', module: 'leveldbjni-all' |
There was a problem hiding this comment.
Maybe exclude just for x86_64?
pre :platform
if (rootProject.archInfo.isArm64) {
testRuntimeOnly group: 'org.fusesource.hawtjni', name: 'hawtjni-runtime', version: '1.18' // for test
implementation project(":platform")
} else {
implementation project(":platform"), {
exclude(group: 'org.fusesource.leveldbjni', module: 'leveldbjni-all')
exclude(group: 'io.github.tronprotocol', module: 'zksnark-java-sdk')
exclude(group: 'commons-io', module: 'commons-io')
}
implementation 'io.github.tronprotocol:leveldbjni-all:1.18.2'
implementation 'io.github.tronprotocol:leveldb:1.18.2'
}There was a problem hiding this comment.
Good call — applied in 26f463b72. Wrapped the leveldbjni-all exclude on :crypto in if (!rootProject.archInfo.isArm64) to mirror the existing arch split on the :platform-direct declaration. Behavior is preserved (ARM64 still resolves leveldbjni-all 1.8 via :platform direct; x86 still resolves only the io.github.tronprotocol 1.18.2 fork) and the comment now reads as 'this fix is x86-specific' rather than a generic policy. Verified :plugins:test and the Toolkit smoke on both arches.
Per follow-up review on PR tronprotocol#6738 (halibobo1205): the :platform-direct dependency in plugins/build.gradle is already split by arch (`if (rootProject.archInfo.isArm64) ... else ...`). The :crypto-side leveldbjni-all exclude is also only meaningful on x86 — on ARM64 the only leveldbjni-all on the runtime classpath comes via the direct :platform declaration at version 1.8, with no second copy to conflict with — so the unconditional exclude was a no-op on ARM64. Wrap the exclude in the same `if (!isArm64)` gate to match the existing arch pattern in this file and to make intent explicit (the exclusion is a fix for the x86-only fat-jar duplicate-class collision, not a defensive policy on every architecture). Behaviour-preserving on both arches: - ARM64: runtimeClasspath still has org.fusesource.leveldbjni:leveldbjni-all:1.8 via :platform direct (Gradle dedups the :crypto path); :plugins:test passes locally - x86: runtimeClasspath still has only io.github.tronprotocol:leveldbjni-all:1.18.2; :plugins:test passes and `db archive -d <empty>` smoke exits 0
Summary
x86_64 builds of
Toolkit.jarsince #6637 throwNoSuchMethodErroronorg.iq80.leveldb.Options.maxBatchSize(I)when running commands such asdb archive/--rebuild-manifest:This PR fixes the root cause and adds CI smoke tests so the same class of bug cannot reach a release artifact again.
Why
#6637 introduced
implementation(project(":crypto"))to thepluginsmodule without excludingorg.fusesource.leveldbjni:leveldbjni-all, which:cryptopulls in transitively via:common -> :platform. The direct:platformdependency inplugins/build.gradlealready excludesleveldbjni-all(so the TRON-forkio.github.tronprotocol:leveldbjni-all:1.18.2can be used instead), but the new transitive path through:cryptodoes not. The result on x86 is two jars carryingorg/iq80/leveldb/Options.classon the runtime classpath:org.fusesource.leveldbjni:leveldbjni-all:1.8— does NOT haveOptions.maxBatchSize(int)io.github.tronprotocol:leveldbjni-all:1.18.2— DOES haveOptions.maxBatchSize(int):plugins:dependencyInsightconfirms the regression path:Why tests didn't catch it
:plugins:testkeeps source jars separate; the JVM classloader returns the first match in classpath order, which is1.18.2(declared directly byplugins/build.gradle). Tests pass.binaryReleasetask merges class entries with last-write-wins (Gradle Zip defaultINCLUDE), so the survivingOptions.classcan be the1.8copy. Same classpath, opposite winner.:platformdirectly (no1.18.2fork), so there is no conflict; ARM64 also excludes Archive tests.Fix
plugins/build.gradle— excludeorg.fusesource.leveldbjni:leveldbjni-allfrom the:cryptodependency, mirroring the existing exclusion on the direct:platformdependency. This collapses the x86 runtimeClasspath to a singleleveldbjni-all(1.18.2)..github/workflows/pr-build.yml— after each platform build (macos / ubuntu-arm / rockylinux-x86 / debian11-x86), run a short smoke test against the freshly builtToolkit.jar:Verification
Reproduced on x86_64 + JDK 8 in
eclipse-temurin:8-jdk(clean build):java -jar Toolkit.jar db archive -d <empty>→NoSuchMethodErrorjava -jar Toolkit.jar db archive -d <empty>→ exits 0, reports "directory does not contain any database":plugins:dependencies --configuration runtimeClasspathno longer listsorg.fusesource.leveldbjni:leveldbjni-all:1.8./gradlew checkstyleMain checkstyleTestpasses./gradlew :plugins:testpasses (KeystoreUpdate, DbLite, RocksDb tests)Test plan
java -jar Toolkit.jar db archive -d <db>works on x86 with this branch's artifactNotes
plugins/build.gradle(+6 lines) and.github/workflows/pr-build.yml(+36 lines).:platformdirect dependency on ARM64 has no exclusion (it never had the conflict), and the new:cryptoexclusion is a no-op there.org/iq80/leveldb/Options.classentries that still exist in the fat jar (fromleveldbjni-all:1.18.2uber andleveldb-api:1.18.2standalone) are pre-existing and harmless because both are the same1.18.2version withmaxBatchSizeavailable.