Autoscaling Kafka consumers with KEDA

Plain HPA scales on CPU and memory, which are poor proxies for whether a Kafka consumer is keeping up. KEDA scales on the signal you actually care about: consumer-group lag — how many messages behind a group is. It can also scale to zero when there’s no work and activate on the first new message, which HPA can’t do at all.

Watch on YouTube ↗

Tip

On OpenShift, Red Hat ships a supported build of KEDA as the Custom Metrics Autoscaler Operator. For production, install that from OperatorHub rather than the community Helm chart — functionally it’s KEDA, with the Red Hat support story.

The core object: a ScaledObject

A ScaledObject points at your consumer Deployment and defines the trigger:

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: consumer-scaler
  namespace: app
spec:
  scaleTargetRef:
    name: order-consumer        # your consumer Deployment
  minReplicaCount: 1
  maxReplicaCount: 6
  cooldownPeriod: 300
  pollingInterval: 20
  triggers:
    - type: kafka
      metadata:
        bootstrapServers: my-cluster-kafka-bootstrap.kafka:9092
        consumerGroup: order-processors
        topic: orders
        lagThreshold: "100"       # target messages of lag per replica

KEDA creates and drives an HPA under the hood: as lag climbs past the threshold, replicas scale up toward maxReplicaCount; as lag drains, they scale back down after cooldownPeriod.

Three things that will shape your config

Partition count is your hard ceiling. A Kafka consumer group can have at most one active consumer per partition — extra consumers just sit idle. So your effective maxReplicaCount is bounded by the topic’s partition count. Size partitions with your target scale in mind; you can’t scale past them.

Rebalancing churn. Every scale event triggers a consumer-group rebalance, which briefly pauses consumption while partitions are reassigned. Tune pollingInterval, cooldownPeriod and lagThreshold so you’re reacting to real trends, not thrashing on every spike.

Scale-to-zero has a cold-start cost. Setting minReplicaCount: 0 saves resources when idle, but the first message after idle waits for a pod to start and join the group. Weigh that activation latency against the savings for your workload.

Caution

KEDA scales workloads, not brokers. If what you actually want is to grow the Kafka cluster itself, that’s Strimzi’s job (with Cruise Control for rebalancing), not KEDA’s. Be clear which side you’re scaling — KEDA is for the apps reading from and writing to Kafka.

Watching it work

# the HPA KEDA created for you
oc get hpa -n app

# consumer-group lag from the Kafka side
oc exec -n kafka my-cluster-kafka-0 -- bin/kafka-consumer-groups.sh \
  --bootstrap-server localhost:9092 --describe --group order-processors

Push load onto the topic and you’ll watch lag climb, the HPA spin replicas up toward the max, and everything settle back to the minimum once lag drains. KEDA + Strimzi is a well-trodden pairing — if you’re already running Strimzi, this slots straight in.