feat(broker): INIT_PRODUCER_ID + LIST_GROUPS fixes + standalone Helm chart for smoke tests#142
Open
kamir wants to merge 3 commits into
Open
feat(broker): INIT_PRODUCER_ID + LIST_GROUPS fixes + standalone Helm chart for smoke tests#142kamir wants to merge 3 commits into
kamir wants to merge 3 commits into
Conversation
Minimal chart that deploys a single kafscale-broker Pod pointing at external etcd + S3 (MinIO or cloud). Intended for quick smoke tests and blueprint convergence on KIND — NOT a replacement for the full operator-based chart at deploy/helm/kafscale/. Used by scalytics-all-in-one bp-001 Ops Foundation smoke suite: COMP-kafscale-01 (pod Ready) + COMP-kafscale-02 (Kafka TCP reachable). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Resolves OPS-005 item #2. The Java admin client (kafka-consumer-groups, AdminClient.listConsumerGroups, Schema Registry) negotiates LIST_GROUPS in range [0,4]. Advertising a narrow 5-5 window caused: UnsupportedVersionException: Error listing groups ... The broker does not support LIST_GROUPS with version in range [0,4]. The supported range is [5,5]. The underlying h.coordinator.ListGroups handler is version-agnostic; the encoder handles v0 just as well as v5. The fix is one line — widen the advertised range. Verified: kafka-consumer-groups --bootstrap-server kafscale-broker:9092 --list now exits 0 cleanly (from UnsupportedVersionException pre-fix). The remaining OPS-005 items (INIT_PRODUCER_ID, transactional APIs, Schema Registry NPE on verifySchemaTopic) are substantive broker-engineering work and are not addressed here. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Adds a minimum-viable implementation of the Kafka INIT_PRODUCER_ID API
(API key 22). Allocates a monotonically-increasing producer ID with epoch 0;
does not yet track sequence numbers or deduplicate on replay. Sufficient to
unblock Java AdminClient default producers, franz-go idempotent producers,
and Schema Registry's producer-init probe.
Changes:
- pkg/protocol/api.go: add APIKeyInitProducerID = 22
- cmd/broker/main.go:
* handler gains nextProducerID int64 (atomic allocator)
* dispatch case for *kmsg.InitProducerIDRequest returns pid + epoch=0
* apiVersions: InitProducerID moves from unsupported to {0, 4}
* import sync/atomic
Verified:
- `kafka-console-producer --producer-property enable.idempotence=true`
now succeeds (was: UnsupportedVersionException).
- kaf-mirror (franz-go) replicates primary→standby end-to-end:
PRIMARY offsets = STANDBY offsets, measured lag <1s.
- SCEN-bp002-06_Replication scenario test flipped SKIP → PASS.
Known limitations (production correctness gap, tracked in OPS-005):
- no sequence-number tracking: duplicate-on-retry semantics not enforced
- no epoch management: fencing of stale producers on rebalance not implemented
- PID allocator is process-local, not persisted across broker restart
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Three small, independent changes that together unblock smoke-test convergence for the scalytics-all-in-one (SAO) bp-001 Ops Foundation scenarios and resolve OPS-005 items #1 and #2.
feat(broker): INIT_PRODUCER_ID stub handler (OPS-005 #1)UnsupportedVersionException.fix(broker): advertise LIST_GROUPS versions 0-5 (was 5-5)kafka-consumer-groups/ Schema Registry negotiate[0,4]— narrow5-5window causedUnsupportedVersionException: ... supported range is [5,5]. The underlying handler is version-agnostic.feat(deploy): add kafscale-broker-standalone Helm chartdeploy/helm/kafscale-broker-standalone/— single broker Pod pointing at external etcd + S3 (MinIO or cloud).deploy/helm/kafscale/. Not a replacement for the operator chart — explicit smoke-test scope, documented in commit message.Scope
deploy/helm/kafscale/Verification
kafka-console-producer --producer-property enable.idempotence=truenow succeeds; kaf-mirror (franz-go) replicates primary→standby with offsets matching and lag <1s; SCEN-bp002-06_Replication flipped SKIP → PASS.kafka-consumer-groups --bootstrap-server kafscale-broker:9092 --listexits 0 cleanly (wasUnsupportedVersionException).Known limitations (for INIT_PRODUCER_ID)
Tracked in OPS-005 — not addressed in this PR:
Remaining OPS-005 items (transactional APIs, Schema Registry NPE on
verifySchemaTopic) are substantive broker-engineering work and remain open.Test plan
make testpasses (Go test suite)helm lint deploy/helm/kafscale-broker-standalonepasses (Helm Lint CI):9092Relationship to PR #139
This branch was originally co-mingled with the security fixes on
fix/s3-bucket-takeover-cve. The branches have been separated cleanly:mainmainThe two PRs are independent and can be reviewed/merged in any order.
🤖 Generated with Claude Code