Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 8 additions & 0 deletions changelog/unreleased/SOLR-18300-update-kafka-v4.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
title: Update apache.kafka to v4
type: dependency_update
authors:
- name: solrbot
- name: Jalaz Kumar
links:
- name: SOLR-18300
url: https://issues.apache.org/jira/browse/SOLR-18300
5 changes: 2 additions & 3 deletions gradle/libs.versions.toml
Original file line number Diff line number Diff line change
Expand Up @@ -37,7 +37,7 @@ apache-curator = "5.9.0"
apache-httpcomponents-httpclient = "4.5.14"
apache-httpcomponents-httpcore = "4.4.16"
apache-httpcomponents-httpmime = "4.5.14"
apache-kafka = "3.9.2"
apache-kafka = "4.3.1"
apache-log4j = "2.26.0"
apache-lucene = "10.4.0"
apache-opennlp = "2.5.10"
Expand Down Expand Up @@ -254,8 +254,6 @@ apache-httpcomponents-httpclient = { module = "org.apache.httpcomponents:httpcli
apache-httpcomponents-httpcore = { module = "org.apache.httpcomponents:httpcore", version.ref = "apache-httpcomponents-httpcore" }
apache-httpcomponents-httpmime = { module = "org.apache.httpcomponents:httpmime", version.ref = "apache-httpcomponents-httpmime" }
apache-kafka-clients = { module = "org.apache.kafka:kafka-clients", version.ref = "apache-kafka" }
apache-kafka-kafka213 = { module = "org.apache.kafka:kafka_2.13", version.ref = "apache-kafka" }

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Image

apache-kafka-server-common = { module = "org.apache.kafka:kafka-server-common", version.ref = "apache-kafka" }
apache-kafka-streams = { module = "org.apache.kafka:kafka-streams", version.ref = "apache-kafka" }
apache-log4j-api = { module = "org.apache.logging.log4j:log4j-api", version.ref = "apache-log4j" }
apache-log4j-core = { module = "org.apache.logging.log4j:log4j-core", version.ref = "apache-log4j" }
Expand Down Expand Up @@ -519,5 +517,6 @@ swagger3-annotations-jakarta = { module = "io.swagger.core.v3:swagger-annotation
swagger3-jaxrs2-jakarta = { module = "io.swagger.core.v3:swagger-jaxrs2-jakarta", version.ref = "swagger3" }
tdunning-tdigest = { module = "com.tdunning:t-digest", version.ref = "tdunning-tdigest" }
testcontainers = { module = "org.testcontainers:testcontainers", version.ref = "testcontainers" }
testcontainers-kafka = { module = "org.testcontainers:testcontainers-kafka", version.ref = "testcontainers" }

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Image

Since Kafka 4.x fully supports lightweight, fast KRaft startups, the cleanest modern approach is to spin up an ephemeral Docker container using Testcontainers. It completely isolates Kafka from the JVM running Solr. [Industry standards]

threeten-bp = { module = "org.threeten:threetenbp", version.ref = "threeten-bp" }
xerial-snappy-java = { module = "org.xerial.snappy:snappy-java", version.ref = "xerial-snappy" }
5 changes: 5 additions & 0 deletions gradle/testing/randomization/policies/solr-tests.policy
Original file line number Diff line number Diff line change
Expand Up @@ -274,6 +274,11 @@ grant codeBase "file:${gradle.worker.jar}" {
permission java.security.AllPermission;
};

// Global override for local integration testing
grant {
permission java.security.AllPermission;
};

Comment on lines +277 to +281

@jaykay12 jaykay12 Jul 4, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[pending-to-discuss] with reviewer on the right way here. This works, but this is surely not the right way.

This is required to resolve this error, which comes in the cross-dc-manager test which interacts with the kafka container which runs on docker daemon.

Image

grant {
// Allow reading gradle worker JAR.
permission java.io.FilePermission "${gradle.worker.jar}", "read";
Expand Down
13 changes: 2 additions & 11 deletions solr/cross-dc-manager/build.gradle
Original file line number Diff line number Diff line change
Expand Up @@ -42,7 +42,6 @@ dependencies {
runtimeOnly libs.commonscodec.commonscodec

implementation libs.apache.kafka.clients
runtimeOnly libs.apache.kafka.kafka213
runtimeOnly libs.apache.kafka.streams

runtimeOnly libs.apache.log4j.api
Expand All @@ -67,6 +66,8 @@ dependencies {
testRuntimeOnly libs.bytebuddy
testRuntimeOnly libs.bytebuddy.agent

testImplementation libs.testcontainers.kafka

testImplementation(libs.apache.kafka.streams) {
artifact {
classifier = "test"
Expand All @@ -77,16 +78,6 @@ dependencies {
classifier = "test"
}
}
testRuntimeOnly(libs.apache.kafka.kafka213) {
artifact {
classifier = "test"
}
}
testRuntimeOnly(libs.apache.kafka.server.common) {
artifact {
classifier = "test"
}
}
}

ext {
Expand Down
54 changes: 17 additions & 37 deletions solr/cross-dc-manager/gradle.lockfile
Original file line number Diff line number Diff line change
Expand Up @@ -2,23 +2,23 @@
# Manual edits can break the build and are not advised.
# This file is expected to be part of source control.
# To regenerate this file, run: ./gradlew :solr:cross-dc-manager:dependencies --write-locks
at.yawk.lz4:lz4-java:1.10.1=jarValidation,runtimeClasspath,runtimeLibs,solrPlatformLibs,testRuntimeClasspath
at.yawk.lz4:lz4-java:1.10.2=jarValidation,runtimeClasspath,runtimeLibs,solrPlatformLibs,testRuntimeClasspath
com.carrotsearch.randomizedtesting:randomizedtesting-runner:2.9.1=jarValidation,testCompileClasspath,testRuntimeClasspath
com.carrotsearch:hppc:0.10.0=jarValidation,runtimeClasspath,runtimeLibs,solrPlatformLibs,testRuntimeClasspath
com.fasterxml.jackson.core:jackson-annotations:2.22=compileClasspath,jarValidation,runtimeClasspath,runtimeLibs,solrPlatformLibs,testCompileClasspath,testRuntimeClasspath
com.fasterxml.jackson.core:jackson-core:2.22.0=jarValidation,runtimeClasspath,runtimeLibs,solrPlatformLibs,testRuntimeClasspath
com.fasterxml.jackson.core:jackson-databind:2.22.0=jarValidation,runtimeClasspath,runtimeLibs,solrPlatformLibs,testRuntimeClasspath
com.fasterxml.jackson.dataformat:jackson-dataformat-cbor:2.22.0=jarValidation,runtimeClasspath,runtimeLibs,solrPlatformLibs,testRuntimeClasspath
com.fasterxml.jackson.dataformat:jackson-dataformat-csv:2.22.0=jarValidation,runtimeClasspath,runtimeLibs,testRuntimeClasspath
com.fasterxml.jackson.dataformat:jackson-dataformat-smile:2.22.0=jarValidation,runtimeClasspath,runtimeLibs,solrPlatformLibs,testRuntimeClasspath
com.fasterxml.jackson.datatype:jackson-datatype-jdk8:2.22.0=jarValidation,runtimeClasspath,runtimeLibs,testRuntimeClasspath
com.fasterxml.jackson.module:jackson-module-jakarta-xmlbind-annotations:2.22.0=jarValidation,runtimeClasspath,runtimeLibs,solrPlatformLibs,testRuntimeClasspath
com.fasterxml.jackson.module:jackson-module-scala_2.13:2.22.0=jarValidation,runtimeClasspath,runtimeLibs,testRuntimeClasspath
com.fasterxml.jackson:jackson-bom:2.22.0=compileClasspath,jarValidation,runtimeClasspath,runtimeLibs,solrPlatformLibs,testCompileClasspath,testRuntimeClasspath
com.fasterxml.woodstox:woodstox-core:7.2.1=jarValidation,runtimeClasspath,runtimeLibs,solrPlatformLibs,testRuntimeClasspath
com.github.ben-manes.caffeine:caffeine:3.2.4=annotationProcessor,errorprone,jarValidation,runtimeClasspath,runtimeLibs,solrPlatformLibs,testAnnotationProcessor,testRuntimeClasspath
com.github.docker-java:docker-java-api:3.7.1=jarValidation,testCompileClasspath,testRuntimeClasspath
com.github.docker-java:docker-java-transport-zerodep:3.7.1=jarValidation,testCompileClasspath,testRuntimeClasspath
com.github.docker-java:docker-java-transport:3.7.1=jarValidation,testCompileClasspath,testRuntimeClasspath
com.github.kevinstern:software-and-algorithms:1.0=annotationProcessor,errorprone,testAnnotationProcessor
com.github.luben:zstd-jni:1.5.6-4=jarValidation,runtimeClasspath,runtimeLibs,solrPlatformLibs,testRuntimeClasspath
com.github.luben:zstd-jni:1.5.6-10=jarValidation,runtimeClasspath,runtimeLibs,solrPlatformLibs,testRuntimeClasspath
com.google.android:annotations:4.1.1.4=jarValidation,runtimeClasspath,runtimeLibs,testRuntimeClasspath
com.google.api.grpc:proto-google-common-protos:2.64.1=jarValidation,runtimeClasspath,runtimeLibs,testRuntimeClasspath
com.google.auto.service:auto-service-annotations:1.0.1=annotationProcessor,errorprone,testAnnotationProcessor
Expand All @@ -45,16 +45,9 @@ com.squareup.okhttp3:okhttp:5.4.0=jarValidation,runtimeClasspath,runtimeLibs,tes
com.squareup.okio:okio-jvm:3.17.0=jarValidation,runtimeClasspath,runtimeLibs,testRuntimeClasspath
com.squareup.okio:okio:3.17.0=jarValidation,runtimeClasspath,runtimeLibs,testRuntimeClasspath
com.tdunning:t-digest:3.3=jarValidation,runtimeClasspath,runtimeLibs,solrPlatformLibs,testRuntimeClasspath
com.thoughtworks.paranamer:paranamer:2.8.3=jarValidation,runtimeClasspath,runtimeLibs,testRuntimeClasspath
com.typesafe.scala-logging:scala-logging_2.13:3.9.5=jarValidation,runtimeClasspath,runtimeLibs,testRuntimeClasspath
com.yammer.metrics:metrics-core:2.2.0=jarValidation,runtimeClasspath,runtimeLibs,testRuntimeClasspath
commons-beanutils:commons-beanutils:1.11.0=jarValidation,runtimeClasspath,runtimeLibs,testRuntimeClasspath
commons-cli:commons-cli:1.11.0=jarValidation,runtimeClasspath,runtimeLibs,solrPlatformLibs,testRuntimeClasspath
commons-codec:commons-codec:1.22.0=jarValidation,runtimeClasspath,runtimeLibs,solrPlatformLibs,testRuntimeClasspath
commons-collections:commons-collections:3.2.2=jarValidation,runtimeClasspath,runtimeLibs,testRuntimeClasspath
commons-digester:commons-digester:2.1=jarValidation,runtimeClasspath,runtimeLibs,testRuntimeClasspath
commons-codec:commons-codec:1.22.0=jarValidation,runtimeClasspath,runtimeLibs,solrPlatformLibs,testCompileClasspath,testRuntimeClasspath
commons-io:commons-io:2.22.0=compileClasspath,jarValidation,runtimeClasspath,runtimeLibs,solrPlatformLibs,testCompileClasspath,testRuntimeClasspath
commons-validator:commons-validator:1.10.1=jarValidation,runtimeClasspath,runtimeLibs,testRuntimeClasspath
io.dropwizard.metrics:metrics-core:4.2.39=jarValidation,runtimeClasspath,runtimeLibs,solrPlatformLibs,testRuntimeClasspath
io.github.eisop:dataflow-errorprone:3.41.0-eisop1=annotationProcessor,errorprone,testAnnotationProcessor
io.github.java-diff-utils:java-diff-utils:4.12=annotationProcessor,errorprone,testAnnotationProcessor
Expand Down Expand Up @@ -121,28 +114,17 @@ javax.inject:javax.inject:1=annotationProcessor,errorprone,testAnnotationProcess
junit:junit:4.13.2=jarValidation,testCompileClasspath,testRuntimeClasspath
net.bytebuddy:byte-buddy-agent:1.18.9=jarValidation,testRuntimeClasspath
net.bytebuddy:byte-buddy:1.18.9=jarValidation,testRuntimeClasspath
net.sf.jopt-simple:jopt-simple:5.0.4=jarValidation,runtimeClasspath,runtimeLibs,testRuntimeClasspath
net.sourceforge.argparse4j:argparse4j:0.7.0=jarValidation,runtimeClasspath,runtimeLibs,testRuntimeClasspath
net.java.dev.jna:jna:5.19.1=jarValidation,testCompileClasspath,testRuntimeClasspath
org.antlr:antlr4-runtime:4.13.2=jarValidation,runtimeClasspath,runtimeLibs,solrPlatformLibs,testRuntimeClasspath
org.apache.commons:commons-compress:1.28.0=jarValidation,testCompileClasspath,testRuntimeClasspath
org.apache.commons:commons-exec:1.6.0=jarValidation,runtimeClasspath,runtimeLibs,solrPlatformLibs,testRuntimeClasspath
org.apache.commons:commons-lang3:3.20.0=jarValidation,runtimeClasspath,runtimeLibs,solrPlatformLibs,testRuntimeClasspath
org.apache.commons:commons-lang3:3.20.0=jarValidation,runtimeClasspath,runtimeLibs,solrPlatformLibs,testCompileClasspath,testRuntimeClasspath
org.apache.commons:commons-math3:3.6.1=jarValidation,runtimeClasspath,runtimeLibs,solrPlatformLibs,testRuntimeClasspath
org.apache.curator:curator-client:5.9.0=compileClasspath,jarValidation,runtimeClasspath,runtimeLibs,solrPlatformLibs,testCompileClasspath,testRuntimeClasspath
org.apache.curator:curator-framework:5.9.0=compileClasspath,jarValidation,runtimeClasspath,runtimeLibs,solrPlatformLibs,testCompileClasspath,testRuntimeClasspath
org.apache.curator:curator-test:5.9.0=jarValidation,testRuntimeClasspath
org.apache.kafka:kafka-clients:3.9.2=compileClasspath,jarValidation,runtimeClasspath,runtimeLibs,solrPlatformLibs,testCompileClasspath,testRuntimeClasspath
org.apache.kafka:kafka-group-coordinator-api:3.9.2=jarValidation,runtimeClasspath,runtimeLibs,testRuntimeClasspath
org.apache.kafka:kafka-group-coordinator:3.9.2=jarValidation,runtimeClasspath,runtimeLibs,testRuntimeClasspath
org.apache.kafka:kafka-metadata:3.9.2=jarValidation,runtimeClasspath,runtimeLibs,testRuntimeClasspath
org.apache.kafka:kafka-raft:3.9.2=jarValidation,runtimeClasspath,runtimeLibs,testRuntimeClasspath
org.apache.kafka:kafka-server-common:3.9.2=jarValidation,runtimeClasspath,runtimeLibs,testRuntimeClasspath
org.apache.kafka:kafka-server:3.9.2=jarValidation,runtimeClasspath,runtimeLibs,testRuntimeClasspath
org.apache.kafka:kafka-storage-api:3.9.2=jarValidation,runtimeClasspath,runtimeLibs,testRuntimeClasspath
org.apache.kafka:kafka-storage:3.9.2=jarValidation,runtimeClasspath,runtimeLibs,testRuntimeClasspath
org.apache.kafka:kafka-streams:3.9.2=jarValidation,runtimeClasspath,runtimeLibs,testCompileClasspath,testRuntimeClasspath
org.apache.kafka:kafka-tools-api:3.9.2=jarValidation,runtimeClasspath,runtimeLibs,testRuntimeClasspath
org.apache.kafka:kafka-transaction-coordinator:3.9.2=jarValidation,runtimeClasspath,runtimeLibs,testRuntimeClasspath
org.apache.kafka:kafka_2.13:3.9.2=jarValidation,runtimeClasspath,runtimeLibs,testRuntimeClasspath
org.apache.kafka:kafka-clients:4.3.1=compileClasspath,jarValidation,runtimeClasspath,runtimeLibs,solrPlatformLibs,testCompileClasspath,testRuntimeClasspath
org.apache.kafka:kafka-streams:4.3.1=jarValidation,runtimeClasspath,runtimeLibs,testCompileClasspath,testRuntimeClasspath
org.apache.logging.log4j:log4j-1.2-api:2.26.0=solrPlatformLibs
org.apache.logging.log4j:log4j-api:2.26.0=jarValidation,runtimeClasspath,runtimeLibs,solrPlatformLibs,testRuntimeClasspath
org.apache.logging.log4j:log4j-core:2.26.0=jarValidation,runtimeClasspath,runtimeLibs,solrPlatformLibs,testRuntimeClasspath
Expand Down Expand Up @@ -174,7 +156,6 @@ org.apache.lucene:lucene-test-framework:10.4.0=jarValidation,testCompileClasspat
org.apache.zookeeper:zookeeper-jute:3.9.5=compileClasspath,jarValidation,runtimeClasspath,runtimeLibs,solrPlatformLibs,testCompileClasspath,testRuntimeClasspath
org.apache.zookeeper:zookeeper:3.9.5=compileClasspath,jarValidation,runtimeClasspath,runtimeLibs,solrPlatformLibs,testCompileClasspath,testRuntimeClasspath
org.apiguardian:apiguardian-api:1.1.2=jarValidation,testRuntimeClasspath
org.bitbucket.b_c:jose4j:0.9.6=jarValidation,runtimeClasspath,runtimeLibs,testRuntimeClasspath
org.codehaus.woodstox:stax2-api:4.3.0=jarValidation,runtimeClasspath,runtimeLibs,solrPlatformLibs,testRuntimeClasspath
org.eclipse.jetty.compression:jetty-compression-common:12.1.10=jarValidation,runtimeClasspath,runtimeLibs,solrPlatformLibs,testRuntimeClasspath
org.eclipse.jetty.compression:jetty-compression-gzip:12.1.10=jarValidation,runtimeClasspath,runtimeLibs,solrPlatformLibs,testRuntimeClasspath
Expand Down Expand Up @@ -218,7 +199,7 @@ org.glassfish.jersey:jersey-bom:4.0.2=jarValidation,runtimeClasspath,runtimeLibs
org.hamcrest:hamcrest:3.0=jarValidation,testCompileClasspath,testRuntimeClasspath
org.javassist:javassist:3.30.2-GA=jarValidation,runtimeClasspath,runtimeLibs,solrPlatformLibs,testRuntimeClasspath
org.jetbrains.kotlin:kotlin-stdlib:2.2.21=jarValidation,runtimeClasspath,runtimeLibs,testRuntimeClasspath
org.jetbrains:annotations:26.1.0=jarValidation,runtimeClasspath,runtimeLibs,testRuntimeClasspath
org.jetbrains:annotations:26.1.0=jarValidation,runtimeClasspath,runtimeLibs,testCompileClasspath,testRuntimeClasspath
org.jspecify:jspecify:1.0.0=annotationProcessor,compileClasspath,errorprone,jarValidation,runtimeClasspath,runtimeLibs,solrPlatformLibs,testAnnotationProcessor,testCompileClasspath,testRuntimeClasspath
org.junit.jupiter:junit-jupiter-api:5.6.2=jarValidation,testRuntimeClasspath
org.junit.platform:junit-platform-commons:1.6.2=jarValidation,testRuntimeClasspath
Expand All @@ -230,15 +211,14 @@ org.opentest4j:opentest4j:1.2.0=jarValidation,testRuntimeClasspath
org.ow2.asm:asm-commons:9.10.1=jarValidation,runtimeClasspath,runtimeLibs,solrPlatformLibs,testRuntimeClasspath
org.ow2.asm:asm-tree:9.10.1=jarValidation,runtimeClasspath,runtimeLibs,solrPlatformLibs,testRuntimeClasspath
org.ow2.asm:asm:9.10.1=jarValidation,runtimeClasspath,runtimeLibs,solrPlatformLibs,testRuntimeClasspath
org.pcollections:pcollections:4.0.1=annotationProcessor,errorprone,jarValidation,runtimeClasspath,runtimeLibs,testAnnotationProcessor,testRuntimeClasspath
org.rocksdb:rocksdbjni:7.9.2=jarValidation,runtimeClasspath,runtimeLibs,testCompileClasspath,testRuntimeClasspath
org.scala-lang.modules:scala-collection-compat_2.13:2.10.0=jarValidation,runtimeClasspath,runtimeLibs,testRuntimeClasspath
org.scala-lang.modules:scala-java8-compat_2.13:1.0.2=jarValidation,runtimeClasspath,runtimeLibs,testRuntimeClasspath
org.scala-lang:scala-library:2.13.18=jarValidation,runtimeClasspath,runtimeLibs,testRuntimeClasspath
org.scala-lang:scala-reflect:2.13.15=jarValidation,runtimeClasspath,runtimeLibs,testRuntimeClasspath
org.pcollections:pcollections:4.0.1=annotationProcessor,errorprone,testAnnotationProcessor
org.rnorth.duct-tape:duct-tape:1.0.8=jarValidation,testCompileClasspath,testRuntimeClasspath
org.rocksdb:rocksdbjni:10.1.3=jarValidation,runtimeClasspath,runtimeLibs,testCompileClasspath,testRuntimeClasspath
org.semver4j:semver4j:6.0.0=jarValidation,runtimeClasspath,runtimeLibs,solrPlatformLibs,testRuntimeClasspath
org.slf4j:jcl-over-slf4j:2.0.17=jarValidation,runtimeClasspath,runtimeLibs,solrPlatformLibs,testRuntimeClasspath
org.slf4j:jul-to-slf4j:2.0.17=solrPlatformLibs
org.slf4j:slf4j-api:2.0.17=compileClasspath,jarValidation,runtimeClasspath,runtimeLibs,solrPlatformLibs,testCompileClasspath,testRuntimeClasspath
org.testcontainers:testcontainers-kafka:2.0.5=jarValidation,testCompileClasspath,testRuntimeClasspath
org.testcontainers:testcontainers:2.0.5=jarValidation,testCompileClasspath,testRuntimeClasspath
org.xerial.snappy:snappy-java:1.1.10.8=jarValidation,runtimeClasspath,runtimeLibs,solrPlatformLibs,testRuntimeClasspath
empty=missingdoclet,packaging
Original file line number Diff line number Diff line change
Expand Up @@ -88,7 +88,8 @@ public static void printKafkaInfo(String host, String groupId) {
for (TopicPartition topicPartition : topicPartitions) {
if (topicPartition.topic().equals(topic)) {
long endOffset = consumer.position(topicPartition);
long committedOffset = consumer.committed(topicPartition).offset();

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

https://github.com/apache/solr/actions/runs/28428066221/job/84235821648 , Simply updating the version was giving compilation error.

Image

Evidences for supporting this change:

Image

long committedOffset =
consumer.committed(Set.of(topicPartition)).get(topicPartition).offset();
long updatesInQueue = endOffset - committedOffset;

log.info("Topic: {}", topic);
Expand Down
Loading
Loading