SkillVaultskills Browse all 1,000+ skills

Performance · Version 1.2.0 · Reviewed 2026-08-02

Databricks Performance Tuning Specialist

Locate and remove the dominant bottleneck in databricks latency attribution and databricks throughput optimization with evidence, explicit trade-offs, and a verification plan.

4 method steps 4 documented failure modes 4 diagnostic checks 7 quality gates

Finds the dominant measured bottleneck and designs representative benchmarks for Databricks using notebooks, jobs, Spark plans, Delta tables, and cluster policy and Spark UI stages, skew, shuffle, spill, and cluster utilization, with explicit attention to partition skew or driver-side collection collapsing a distributed workload onto one process.

₹199 one-time

Get this skill archive

Install in your AI coding tool

SkillVault packages this skill in the open Agent Skills format for five leading coding tools.

What this skill helps you do

  • Databricks latency attribution
  • Databricks throughput optimization
  • Databricks performance regression guard

How Databricks Performance Tuning Specialist works

You provide

Baseline measurements, workload shape, and the target

It inspects

Dominant cost mechanism behind databricks latency attribution

It decides

A databricks throughput optimization change ranked by impact and risk

You verify

Re-measure under representative load with guardrails

What it checks first

Databricks Performance Tuning Specialist finds the dominant measured bottleneck and designs representative benchmarks for Databricks using notebooks, jobs, Spark plans, Delta tables, and cluster policy and Spark UI stages, skew, shuffle, spill, and cluster utilization, with explicit attention to partition skew or driver-side collection collapsing a distributed workload onto one process. Use it when the work involves Databricks latency attribution, Databricks throughput optimization, Databricks performance regression guard.

  1. A measured baseline and the user-visible target, since optimization without both is guesswork.
  2. Whether the cost is CPU, memory, I/O wait, or lock contention — they have opposite fixes.
  3. The p99 path and how many round trips it contains.
  4. Whether the bottleneck moves after a change, which determines if the gain is real.

Failure modes it recognizes

  • Optimizing a component that is not on the critical path, producing no end-to-end change.
  • A garbage-collection pause misread as slow application code.
  • Memory pressure causing swapping, which presents as unpredictable latency spikes.
  • A micro-optimization that improves the benchmark and regresses the real workload.

Answers it will reject

  • Tuning configuration flags before profiling where time is actually spent.
  • Measuring in a warmed-up loop that does not resemble production access patterns.
  • Reporting an improvement without the guardrail metric that would show a shifted bottleneck.

Decision rules it applies

  • Profile before changing anything, and attribute cost to a specific phase.
  • Optimize the dominant cost first; everything else is rounding.
  • Re-measure under representative load and keep a guardrail metric.

Evidence it asks for

  • Capture a profile during the real workload rather than a synthetic benchmark.
  • Record allocation rate and pause time alongside latency.
  • Compare before and after at the same percentile, not at the mean.

The method inside

  1. Extract decisions, facts, and unresolved questions needed for databricks latency attribution.
  2. Organize databricks throughput optimization around the reader's next decision or action rather than the source order.
  3. Draft databricks performance regression guard with source traceability and no invented behavior.
  4. Run a completeness, consistency, audience, and actionability review before returning the artifact.

Deliverables

  • Databricks latency attribution assessment
  • Databricks throughput optimization decision and action plan
  • Databricks performance regression guard verification checklist

Evidence requirements

  • Profiles, traces, timings, resource metrics, and workload shape
  • Baseline and target percentile
  • Environment, concurrency, and payload details

Quality gates

  • Every material claim traces to supplied evidence or is labeled as a hypothesis.
  • The response follows the declared deliverable contract.
  • No execution, access, measurement, or verification is invented.
  • Secrets and personal data are redacted rather than repeated.
  • The user receives a concrete independent verification step.
  • The relevant failure modes in this domain were considered rather than only the reported symptom.
  • No listed anti-pattern was recommended as a solution.

Example task

Input

Apply the performance tuning specialist to our Databricks system before the next production change. We can provide notebooks, jobs, Spark plans, Delta tables, and cluster policy; the main concern is partition skew or driver-side collection collapsing a distributed workload onto one process.

Expected output

Define the failing percentile and workload, then attribute time with Spark UI stages, skew, shuffle, spill, and cluster utilization. The likely mechanism to disprove first is partition skew or driver-side collection collapsing a distributed workload onto one process. Change one constraint at a time and compare resource use, tail latency, and correctness against a pinned baseline.

Boundaries and compatibility

Ideal for

  • Databricks latency attribution: produce a decision or artifact grounded in supplied evidence.
  • Databricks throughput optimization: produce a decision or artifact grounded in supplied evidence.
  • Databricks performance regression guard: produce a decision or artifact grounded in supplied evidence.

Out of scope

  • Optimizing without a baseline
  • Using averages where tail latency determines experience

Agent compatibility

  • GitHub Copilot Agent Skills
  • Cursor Agent Skills
  • Claude Code Skills
  • OpenAI Codex Skills
  • JetBrains Junie Skills

Tool policy: Advisory by default. No tools are assumed. If the host provides tools, use read-only evidence gathering unless the user explicitly approves a scoped write or execution action.