SkillVaultskills Browse all 1,000+ skills

Reliability · Version 1.2.0 · Reviewed 2026-08-02

Apache Kafka Observability Design Specialist

Reduce production risk in Apache Kafka service-level signal design and Apache Kafka diagnostic telemetry mapping with evidence, explicit trade-offs, and a verification plan.

4 method steps 6 documented failure modes 5 diagnostic checks 7 quality gates

Designs low-noise signals that expose user impact and causal mechanisms in Apache Kafka using topic configuration, partitioning, producer settings, and consumer groups and per-partition lag, rebalance history, under-replicated partitions, and request latency, with explicit attention to key skew or rebalance churn stalling one partition while aggregate metrics look healthy.

₹199 one-time

Get this skill archive

Install in your AI coding tool

SkillVault packages this skill in the open Agent Skills format for five leading coding tools.

What this skill helps you do

  • Apache Kafka service-level signal design
  • Apache Kafka diagnostic telemetry mapping
  • Apache Kafka actionable alert definition

How Apache Kafka Observability Design Specialist works

You provide

Topic config, consumer settings, and lag history

It inspects

Per-partition lag and processing time for Apache Kafka service-level signal design

It decides

A Apache Kafka diagnostic telemetry mapping decision with ordering guarantees stated

You verify

Lag flattens and dead-letter volume stays bounded

What it checks first

Apache Kafka Observability Design Specialist designs low-noise signals that expose user impact and causal mechanisms in Apache Kafka using topic configuration, partitioning, producer settings, and consumer groups and per-partition lag, rebalance history, under-replicated partitions, and request latency, with explicit attention to key skew or rebalance churn stalling one partition while aggregate metrics look healthy. Use it when the work involves Apache Kafka service-level signal design, Apache Kafka diagnostic telemetry mapping, Apache Kafka actionable alert definition.

  1. Consumer lag trend rather than absolute value: flat lag at any level is healthy, rising lag is not.
  2. Partition count versus consumer count, since consumers beyond the partition count are idle by definition.
  3. Whether the partition key produces even distribution, or a few keys dominate one partition.
  4. Rebalance frequency, which converts into repeated processing pauses.
  5. Whether offsets commit before or after processing, which decides between at-most-once and at-least-once.

Failure modes it recognizes

  • A poison message blocking a partition indefinitely because the consumer retries in place without dead-lettering.
  • Processing time exceeding the poll interval, causing the broker to evict the consumer and trigger a rebalance loop.
  • Committing offsets before processing, silently dropping messages on crash.
  • Producer key set to null or a timestamp, destroying ordering guarantees the consumer assumed.
  • Consumer group rebalance storms from short session timeouts on a slow consumer.
  • Unbounded retention plus a compaction misconfiguration filling disk and stopping the broker.

Answers it will reject

  • Adding consumers to reduce lag when partitions are already saturated — throughput is bounded by partitions.
  • Increasing partitions to fix lag without checking whether processing is CPU-bound downstream.
  • Treating the queue as a database by retaining everything and querying it by scan.
  • Requeueing a failed message to the tail forever, converting a bug into an infinite loop.

Decision rules it applies

  • Order is guaranteed only within a partition, so any ordering requirement must map to a partition key.
  • Choose at-least-once with idempotent consumers over attempting exactly-once across systems.
  • Dead-letter after a bounded retry count with the failure reason attached; never retry indefinitely in place.
  • Size partitions for peak throughput plus headroom, because increasing partitions later breaks key-to-partition mapping.

Evidence it asks for

  • Track consumer lag per partition, not aggregated, so a single stuck partition is visible.
  • Measure processing time per message against `max.poll.interval.ms`.
  • Alert on rebalance rate and on dead-letter volume as separate signals.

The method inside

  1. Extract decisions, facts, and unresolved questions needed for Apache Kafka service-level signal design.
  2. Organize Apache Kafka diagnostic telemetry mapping around the reader's next decision or action rather than the source order.
  3. Draft Apache Kafka actionable alert definition with source traceability and no invented behavior.
  4. Run a completeness, consistency, audience, and actionability review before returning the artifact.

Deliverables

  • Apache Kafka service-level signal design assessment
  • Apache Kafka diagnostic telemetry mapping decision and action plan
  • Apache Kafka actionable alert definition verification checklist

Evidence requirements

  • User-visible symptoms and SLO impact
  • Timeline, telemetry, deploys, and dependency state
  • Current mitigations and operational constraints

Quality gates

  • Every material claim traces to supplied evidence or is labeled as a hypothesis.
  • The response follows the declared deliverable contract.
  • No execution, access, measurement, or verification is invented.
  • Secrets and personal data are redacted rather than repeated.
  • The user receives a concrete independent verification step.
  • The relevant failure modes in this domain were considered rather than only the reported symptom.
  • No listed anti-pattern was recommended as a solution.

Example task

Input

Apply the observability design specialist to our Apache Kafka system before the next production change. We can provide topic configuration, partitioning, producer settings, and consumer groups; the main concern is key skew or rebalance churn stalling one partition while aggregate metrics look healthy.

Expected output

Instrument per-partition lag, rebalance history, under-replicated partitions, and request latency at the same boundary as the user-visible objective. The dashboard must make key skew or rebalance churn stalling one partition while aggregate metrics look healthy distinguishable from ordinary load. Page only on symptoms that require action, retain causal dimensions within a bounded cardinality budget, and test every alert with a controlled failure.

Boundaries and compatibility

Ideal for

  • Apache Kafka service-level signal design: produce a decision or artifact grounded in supplied evidence.
  • Apache Kafka diagnostic telemetry mapping: produce a decision or artifact grounded in supplied evidence.
  • Apache Kafka actionable alert definition: produce a decision or artifact grounded in supplied evidence.

Out of scope

  • Replacing incident command authority
  • Calling a trigger the root cause without a causal chain

Agent compatibility

  • GitHub Copilot Agent Skills
  • Cursor Agent Skills
  • Claude Code Skills
  • OpenAI Codex Skills
  • JetBrains Junie Skills

Tool policy: Advisory by default. No tools are assumed. If the host provides tools, use read-only evidence gathering unless the user explicitly approves a scoped write or execution action.