Hi. I'm curious how the adoption of these open table formats is going - who's using it in prod, who in dev, who's just curious. If anyone is willing to share where their Kafka cluster stands on the matter, I think it'll make for an interesting discussion!

We have certainly seen a plethora of vendors support OTF integration. One thing that's unclear to me is how useful those are, and which tradeoffs matter (e.g low ingestion lag, low management overhead, etc)

Just shipped Jikkou 1.0: the Resource-as-Code framework for Apache Kafka 🎉.

Substantial new capabilities anchor the release:

🧊 Apache Iceberg provider : manage namespaces, tables, and views declaratively. Schema evolution runs in two passes so renames + type promotions land as a single safe change instead of drop-and-add. Works with REST, Hive, JDBC, AWS Glue, and Nessie catalogs.

🌍 Multi-cluster orchestration : group your providers and roll changes across a whole fleet in one command: jikkou apply --provider-group production --continue-on-error. Fleet-wide diffs in one shot too.

🔐 Confluent Cloud RBAC : RoleBinding resources to manage Confluent Cloud role bindings as code, next to your topics and ACLs.

📦 Plus: resource dependency ordering, JSON Schema export, provider-grouped get commands, hardened deps

Release note : https://www.jikkou.io/docs/releases/release-v1.0.0/

Happy to answer questions, hear what you'd want from the 1.x line, or just discuss how you're managing Kafka resources today 🙂.

TL;DR: During blue-green broker deployments with 50K+ partition movements, Cruise Control moves all replicas over X hours, then executes all 10K+ leadership changes in a concentrated burst at the end, causing client latency spikes. Looking for ways to spread leadership movements throughout the rebalance.

Background scenario:

We run a 9-broker Kafka cluster and do blue-green deployments where we add 9 new brokers and rebalance the entire cluster. Our typical rebalance involves ~55,000 partition movements.

The execution is sequential:

1. Move ALL replicas (X hours)
2. Then move ALL leadership (concentrated burst)

This causes a "leadership storm" at the end where thousands of leadership changes happen rapidly, leading to client connection disruptions and request timeouts.

Questions:

1. Is this sequential execution (replicas → leadership) fundamental to CC's architecture, or are we missing a config option?
2. Has anyone else dealt with this during large rebalances or blue-green deployments?

Genuinely curious how teams here approach this.

For context, I've been spending a lot of time on the audit side of Kafka — SOC 2, PCI-DSS, ISO 27001 — and the recurring pain seems to be:

1. Inventory: nobody's quite sure how many topics, clusters, or principals exist

2. ACL audit: someone granted User:* during an incident a year ago and nobody undid it

3. Inter-broker TLS: enabled on the dev cluster, mysteriously not on prod

4. Audit logs: enabled, but no retention policy, so the auditor's "who consumed from this topic last quarter" question can't be answered

Some questions I'd love to hear answers to from this community:

- Do you run a pre-audit checklist? If yes, manual or automated?

- How do you prove inter-broker is encrypted, in writing, to an auditor?

- What's your strategy for ACL drift? (Periodic review? Diff against IaC?)

- Has anyone tied control evidence to CI/CD — i.e., the build won't merge if compliance breaks?

I work on an open-source project in this space (KafkaGuard) but I'm asking because the *questions* keep coming up identically across teams and I'd like to know what's working in the wild — tools, scripts, processes, anything.

Will share aggregated patterns from the replies if there's enough discussion.

Reposting my colleague Juha's blog all about the evolving language we use to describe Kafka, it's an interesting read :) He digs into how to way we describe Kafka has changed and the issues introduced by new innovations.