AI Performance Testing Guardrails Every Tester Needs

Performance testing isn't my primary discipline. I get pulled into it occasionally, enough that I wanted a real read on where AI actually helps in that space and where it's just hype, before the next time it comes up. That's what took me to Kaushal Dalvi's StarEast 2026 session, "Beyond the Dev Box: Claude Code Across the Entire Performance Testing Lifecycle." Dalvi leads a performance engineering team and spent the session making a specific argument: performance engineering has always carried a hidden tax, hours spent operating tools instead of doing the actual engineering judgment. Better observability tooling cut that tax once already, without removing the need for performance engineers. His claim is that Claude Code and other AI coding agents are the next wave of the same pattern, provided you constrain them with the right guardrails.

What made that argument land for me was a real incident, not a hypothetical demo.

The AI-Generated Script That Let a Critical Performance Issue Slip Into Production

An intern on Dalvi's team, new to a project that needed performance testing, used Claude Code to generate a K6 script for an API endpoint. The script was built in 30 minutes, running within the hour, and the results were published within half a day. A senior engineer on the team, with 20 to 25 years of performance engineering experience, estimated the same work would have taken him closer to a full day by hand. The results looked great too: 100% success rate, solid response times.

Days later, the application went live and fell over almost immediately, even under low load.

The root cause, found only after the fact, was a try/catch block wrapped around a key transaction in the generated script, silently swallowing every error. That's a silent failure in the truest sense: the tool reported 100% success because it genuinely never saw one. The errors were being caught and discarded before K6 could record them.

Below is an example of what this might have looked like, but wasn't shown during the lecture.

import http from 'k6/http';
import { check } from 'k6';

export const options = {
  vus: 50,
  duration: '5m',
};

export default function () {
  try {
    const res = http.post('https://shop-demo.io/api/checkout', payload());

    if (res.status !== 200) {
      throw new Error(`Checkout failed: ${res.status}`);
    }

    check(res, { 'checkout succeeded': (r) => r.status === 200 });
  } catch (err) {
    // Every real failure lands here and goes nowhere.
    // No check() runs, no error metric increments, nothing is recorded.
  }
}

Dalvi's read on why the model did this is the sharpest line from the whole talk: the model was trying to prevent errors because it thought that was the helpful thing to do, and in doing so it lost sight of what the script was actually for. A performance test exists to surface failures under load. A model optimizing for looking correct will do the opposite of that unless something stops it.

That's a general problem with AI-generated code, not a performance-testing-specific one. It just happened to show up here in a form with real production consequences.

Determinism vs. Non-Determinism

The thread running through the rest of the session was a distinction Dalvi kept returning to: determinism versus non-determinism, and how you put guardrails around the second one.

He illustrated it with a log analysis demo. He fed Claude a raw access log, hundreds of thousands of lines, the kind of file his team used to hand-parse before modern observability tooling existed. He was upfront that this was a contrived setup (an audience member correctly pointed out that no real organization works from a raw log with no retention policy), but the point he wanted to make didn't depend on the example being realistic.

The failure mode he was demonstrating: dump a huge log into a chat window and ask for analysis, and you'll get a plausible-looking answer that's different every time you ask. That's next-token prediction doing what it does. The fix wasn't to trust the model's summary less. It was to change what he asked the model to do:

1. Don't read the whole file into context. Inspect it first, using head, tail, random sampling, and time-range counts to learn its shape.
2. Write scripts that do the actual analysis, rather than reasoning over the raw data directly.
3. Run those scripts. The output is deterministic and repeatable because it came from code, not from a language model's recollection of what it read.
4. Cite every number back to the script or file it came from, so the results can be checked rather than taken on faith.

The scripts turned hundreds of thousands of log lines into a workload-model.md with request-per-second figures, concurrent user estimates using Little's Law, and endpoint hit ratios, none of which the model could have reliably held in its head from a single pass over the raw file.

Dalvi didn't share the file itself (there's no code repo for this session). What follows is a reconstruction matching the shape he described, including the part that mattered most to him: every number traceable back to the script that produced it.

# Workload Model: access.log analysis

**Source:** access.log (438,201 lines, 2026-05-01 to 2026-05-07)
**Generated by:** analyze_endpoints.py, compute_percentiles.py, estimate_concurrency.py

## Endpoint Hit Ratios
| Endpoint | Requests | % of Total |
|---|---|---|
| GET /api/products | 182,340 | 41.6% |
| POST /api/cart | 96,112 | 21.9% |
| GET /api/search | 74,558 | 17.0% |
| POST /api/checkout | 41,209 | 9.4% |
| other | 43,982 | 10.1% |

*Source: analyze_endpoints.py*

## Latency Percentiles (ms)
| Endpoint | p50 | p95 | p99 |
|---|---|---|---|
| GET /api/products | 82 | 310 | 640 |
| POST /api/cart | 110 | 420 | 810 |
| POST /api/checkout | 145 | 560 | 1,020 |

*Source: compute_percentiles.py*

## Concurrency Estimate (Little's Law)
- Session arrival rate (λ): 0.92 sessions/sec, derived from unique session IDs per minute
- Average session duration (W): 3.8 minutes (228 seconds), derived from first-to-last timestamp per session ID
- Estimated concurrent users (L = λ × W): 0.92 × 228 ≈ 210

*Source: estimate_concurrency.py*

That citation line under each section is what lets a reviewer walk back to compute_percentiles.py and check the p95 figure against the raw log instead of taking the model's summary on faith. The same discipline (inspect, script, run, cite) is what would have caught the K6 script's silent try/catch before it shipped: a reviewer working from a script's actual behavior, rather than the tool's self-reported summary, would have seen the error handling directly.

A Guardrails Checklist for the Next Time I Get Pulled In

The rest of the session built out a stack of guardrails in increasing order of formality. I'm not going to pretend I've used all of these in production the way Dalvi's team has, but this is the checklist I'd actually reach for the next time I'm asked to help with AI-assisted performance testing:

• Ground it in a spec. Feed the model an OpenAPI or Swagger spec before asking it to generate test assets. Dalvi said that without this, the model invented its own ad hoc control flow, if/else and for loops inside the test scripts, instead of using the load-testing tool's built-in features, in every example his team tried before adding this guardrail.
• Write a rules.md. A short, team-authored file listing what not to do (no swallowed errors, no logic embedded in test scripts) plus house conventions, handed to the model alongside the spec every time.
• Package repeated context as a skill. A skill is a folder of instructions, scripts, and resources that Claude can load automatically or on request, useful for anything you'd otherwise re-explain every session, like how to call an internal CLI to pull metrics or restart a service.
• Split large tasks across subagents. Dalvi noted that output quality tends to decline once a context window is 40 to 50% full. His team's fix was markdown-defined specialist subagents, each with isolated context. Given a 20-endpoint OpenAPI spec, he ran four K6 script-generator subagents in parallel, each following the same house rules, instead of one long session trying to hold all of it at once.
• Distribute the guardrails as a plugin once they're stable. Dalvi's central performance engineering team hosts a plugin in a GitHub repository that any of their 80-plus distributed development teams can install with a single command. When the central team fixes a flaw in their guidance, every consuming team gets the fix on their next plugin update, instead of the correction living in one team's head.
• Write separate reporting rules per audience. His team keeps different rules.md files for executive summaries (bullet points, no jargon, business impact only), engineering deep-dives (line numbers, timestamps, APM correlations), and release management (a mix of both, framed around go or no-go readiness). In one demo, Claude generated all three reports in a single pass from the same underlying analysis, swapping only which rules.md grounded each one.

Dalvi didn't share his team's actual rules.md files, but here's roughly the shape of what those three might look like for the same underlying finding:

# Executive Summary Rules

- Bullet points only, no paragraphs
- No technical jargon: no APM tool names, no percentile terms
- Lead with business impact: revenue risk, customer impact, launch readiness
- Five bullets maximum

# Engineering Deep-Dive Rules

- Cite every claim: file name, line number, or timestamp
- Include APM correlation IDs where available
- Recommend the specific code or config change, not just the symptom
- Do not summarize away technical detail for brevity

# Release Readiness Rules

- Lead with status: green, yellow, or red
- List open risks with severity and owner
- State a go or no-go recommendation explicitly, no hedging
- One paragraph maximum per section, mixing technical and business framing

None of this removes the review step. It just means the review is checking cited, reproducible work instead of a black-box claim.

Hiring a Second AI Agent to Check the First

The guardrail I found most transferable outside performance testing came from Dalvi's consulting background. When a company hires a consulting firm to build something, he said, it's common to hire a different firm to validate the work, because a firm validating its own output has a natural incentive to protect its own conclusions.

He applies the same idea to agents. Rather than asking one Claude session for a diagnosis and trusting it, he asks it to do the correlation work (build a timeline, cite artifacts, show its reasoning), then spins up a second, independent agent whose only job is to find flaws in the first agent's conclusion. In a bottleneck-analysis demo, he handed Claude seven artifacts from a load test where he'd deliberately injected a fault: a K6 summary, application logs, garbage collection logs, thread dumps, and slow-query exports from the application performance monitoring platform. The prompt explicitly told it to be skeptical of the first apparent cause, to flag suspicious or misleading readings, and to surface any conflicting evidence rather than pick a side silently. It correctly traced the fault to connection pool saturation at the four-minute mark, matching what Dalvi had actually injected, with citations back to the source artifact for every claim.

His summary of the approach: the judgment stays his, the grunt work doesn't.

The Same Failure Pattern I Already Knew From Test Automation

The pattern Dalvi described, an AI swallowing errors inside a try/catch, is one I've caught underskilled test automation consultants doing by hand: wrapping every test in a try/catch so nothing ever throws. I think it comes from the same instinct, avoiding execution failures, and it's the wrong instinct either way. Tests should fail when the software under test has changed behavior. There's a real difference between making a test reliable and making it so self-healing and overly resilient that it hides defects instead of reporting them.

Dalvi's K6 story is that same failure mode wearing performance-testing clothes. The model wasn't malicious or even wrong to want the script to run cleanly. It just optimized for the wrong signal, and nothing was in place to catch it before production did.

What I'm Taking Back to My Own Work

The determinism guardrail is the one I'll use most, even outside performance testing: when I'm asking Claude to make sense of a large, messy input (logs, a big CSV export, a pile of test results), I'd rather it write and run a script against the raw data and cite the output than summarize the data directly from a single read. That's a cheap habit to adopt and it applies well beyond any one testing discipline.

The adversarial-agent pattern is the other one I want to start using deliberately. A second agent whose only job is to argue with the first agent's conclusion is a lightweight way to catch the kind of confident, plausible-sounding mistake that's easy to miss when you're the one who asked the question in the first place.

Performance testing is still not my daily work. But the next time I'm handed a load test and an AI assistant to help build it, I have a specific list of questions to ask before I trust the results: what is it citing this number from, what did I tell it not to do, and who's checking its work besides me.

For more on the AI conference sessions I attended around this one: AI vision testing and Playwright MCP, hands-on AI tooling and evals, cost-efficient Playwright testing with AI, and prompt engineering techniques for testers.