[{"data":1,"prerenderedAt":5264},["ShallowReactive",2],{"content:\u002Fsoftware-testing\u002Ftest-automation\u002Freuse-playwright-tests-for-load-testing":3,"category:\u002Fsoftware-testing\u002Ftest-automation\u002Freuse-playwright-tests-for-load-testing":6,"read-next:\u002Fsoftware-testing\u002Ftest-automation\u002Fai-performance-testing-guardrails,\u002Fsoftware-testing\u002Ftest-automation\u002Fstareast-2026-playwright-ai-cost-efficient-testing,\u002Fsoftware-testing\u002Ftest-automation\u002Fai-test-automation-pitfalls-vs-user-error,\u002Fsoftware-testing\u002Ftest-automation\u002Fautomated-api-testing-with-schemathesis":224},{"id":4,"title":5,"bmcUsername":6,"body":7,"cover":213,"date":214,"description":215,"draft":216,"extension":217,"features":6,"githubRepo":6,"headline":6,"highlight":6,"icon":6,"meta":218,"navigation":219,"npmPackage":6,"order":6,"path":220,"seo":221,"stem":222,"__hash__":223},"content\u002Fsoftware-testing\u002Ftest-automation\u002Freuse-playwright-tests-for-load-testing.md","Reuse Your Playwright Tests for Load Testing",null,{"type":8,"value":9,"toc":201},"minimark",[10,20,25,28,36,44,47,51,54,57,60,63,66,70,73,79,94,101,105,108,111,129,132,135,139,142,145,153,156,159,163,166,169,177,180,184,187,190,194,197],[11,12,13,14,19],"p",{},"I'd already reused our existing Playwright functional tests to drive load testing with Artillery.io before I ever heard ",[15,16],"external-link",{"href":17,"text":18},"https:\u002F\u002Fwww.linkedin.com\u002Fin\u002Fandrew-clegg-ncsu\u002F","Andrew Clegg"," talk about the same idea at StarEast 2026. I hit a wall along the way and landed on an imperfect compromise to get past it. Clegg, a performance engineer at SAS Institute, spent his session \"Streamlined Performance Testing with Test Containers\" describing an enterprise team that took the same basic idea further, and hit the same kind of wall for the same underlying reasons. Two different tool stacks, two unrelated teams, the same conclusion.",[21,22,24],"h2",{"id":23},"reduce-reuse-recycle-reusing-functional-tests-for-load-testing","Reduce, Reuse, Recycle: Reusing Functional Tests for Load Testing",[11,26,27],{},"Clegg's team learned this lesson the hard way. For about ten years at SAS, performance testing meant LoadRunner: a centralized team of eight engineers, each owning different products, re-recording API workflows before every new build to catch whatever had changed underneath them. It was arduous, API-only, and didn't scale, eight people couldn't keep up with re-recording across all of SAS's heavy-frontend product lines, and some products just didn't get the attention they needed. That re-recording cycle looks to me like the direct cost of drift, though it's not language Clegg used himself: every new build could change which APIs got called, or how, and the old recording quietly went stale the moment it did. Retiring the recording step entirely, in favor of tests QA was already maintaining to support functional testing, removed that tax by design instead of chasing it build after build.",[11,29,30,31,35],{},"Then SAS reorganized. The centralized team got split apart and embedded into individual product teams. Clegg compares the moment to winning live chickens as a prize on ",[32,33,34],"em",{},"Survivor",": technically valuable, useless without the tools to actually process them. Rather than start from scratch and rebuild his own testing infrastructure from zero, he looked at what his new product's QA engineers had already built: hundreds of mature Playwright tests that already knew how to drive the UI. He treated those as something he could buy at the grocery store instead of raising himself.",[11,37,38,39,43],{},"He frames the whole approach as ",[40,41,42],"strong",{},"reduce, reuse, recycle",", with recycle deliberately last: rebuilding something to repurpose it is the most expensive option of the three, reuse is what you should reach for first.",[11,45,46],{},"My own reasoning got to the same place from a different direction. I wasn't reorganized into a new team; I was tasked with capturing what real users actually experience moving through complicated workflows that span several pages and screens, and I'd already concluded that simulating that with direct API calls was the wrong foundation. Hand-built API load scripts drift silently from what the product actually does over time, as pages get updated and start calling different endpoints or static resources than whatever the script assumed when it was written. Reusing the real functional tests avoids that by construction: if the UI changes what it calls, the test that drives the UI changes with it. Reduce duplicated effort was Clegg's reason to reuse existing tests. Avoiding silent drift was mine. We ended up doing the same thing anyway.",[21,48,50],{"id":49},"why-playwright-works-for-load-testing","Why Playwright Works for Load Testing",[11,52,53],{},"Clegg's case for Playwright as the reuse vehicle: it's open source, its post-failure trace files are unusually rich for diagnosing what actually happened, and its Chrome DevTools Protocol access lets him separate container-level resource usage from the browser's own actual CPU and memory use, rather than inferring browser behavior indirectly from the container's numbers. His concrete example is SAS Studio's flow-builder canvas, a canvas element that's a black box to plain Playwright, no accessible DOM nodes, nothing in a browser snapshot. QA's existing tests already had custom JavaScript hooks built to reach into it. By reusing those tests instead of writing his own, he inherited that hard-won access for free, and just as importantly, inherited QA's ongoing maintenance of it. When the UI changes and something breaks, QA fixes it as part of their own test maintenance instead of the burden falling solely on him.",[11,55,56],{},"My own reasoning for choosing Playwright pointed the same direction, though it started from a different problem. We were already transitioning to Playwright for functional testing, and in doing so had solved a set of proprietary automation gotchas specific to our own application, reliably waiting for ExtJS events and working around non-standard UI-blocking patterns that don't behave like a normal DOM. Any other tool would have meant solving those same problems over again from scratch, the same \"don't rebuild what's already been fought through\" logic Clegg applied to QA's canvas-object hooks. On top of that, Microsoft's continued investment in Playwright's CLI and MCP tooling fit squarely into our own push toward an AI-native SDLC, independent of any performance-testing use case.",[11,58,59],{},"Clegg didn't mention this specifically, but this is my own take on why API-only tests are fundamentally the wrong approach if you are trying to collect accurate front-end performance metrics. On a modern web app, a request finishing and a user seeing something useful aren't the same event. A page can download its HTML and initial payload quickly, then have JavaScript execute, discover it needs more data or assets, fetch those too, and only then let the browser finish laying out and painting what the user actually sees, the kind of cascading, browser-side work that a metric like Largest Contentful Paint (when the page's biggest visible element actually renders) exists to measure. Adding up API response times, or even resource-download times, tells you almost nothing about when a real user saw a usable page. That gap barely existed on the simpler, largely static HTML sites performance testing tools were originally built around. This record and playback or API testing mindset seems to be a holdover from simpler times, and the tools and techniques we use need to catch up.",[11,61,62],{},"Beyond being more technically accurate, front-end metrics are also simply more legible to non-technical stakeholders than backend ones. \"The home page loaded in 4.2 seconds\" is easy for anyone to understand. \"The inventory endpoint sustained 1,000 requests per minute\" doesn't without more context. Reusing UI-driven tests for performance work doesn't just produce more accurate data, it produces self-explanatory data.",[11,64,65],{},"That said, there's still a real case for testing API calls directly, to confirm your backend can sustain a given level of traffic at a given response time. Those tests matter for measuring services your front end ultimately depends on. My point is they shouldn't be asked to double as a proxy for front-end performance.",[21,67,69],{"id":68},"where-functional-and-performance-testing-goals-conflict","Where Functional and Performance Testing Goals Conflict",[11,71,72],{},"Clegg's most actionable slide was the one titled, plainly, \"Functional vs Performance tests: Different objectives and expectations.\" Two problems came up repeatedly:",[11,74,75,78],{},[40,76,77],{},"Hardcoded wait times."," A functional test might wait a fixed five or ten seconds for something to finish, harmless under normal conditions. Under load, that same fixed wait can be too short (the action now legitimately takes longer, and the test moves on and fails) or wastefully long (waiting ten seconds for something that finished in a tenth of a second). The deeper problem for performance work specifically: a hardcoded wait sitting inside a measured transaction block corrupts the recorded transaction time itself, you're no longer measuring how long the real operation took, you're measuring your own guess.",[11,80,81,84,85,89,90,93],{},[40,82,83],{},"Hardcoded resource names."," Test files like ",[86,87,88],"code",{},"cars.csv"," or ",[86,91,92],{},"program_baseline.sas"," work fine for a single functional run and collide immediately once multiple simulated users hit the same resource name concurrently. Clegg's team's fix, still in progress at the time of the talk, is timestamping or otherwise uniquifying these values wherever they're found.",[11,95,96,97,100],{},"My team never hit either problem, though not because we anticipated performance testing when we wrote the tests. Our Playwright tests already used data randomization for the test data they created, purely because our test environments have limited ability to tear data down between runs. And we already used Playwright's built in polling explicit waits with large max timeouts instead of fixed sleeps, for ordinary functional-testing reasons (e.g. ",[86,98,99],{},"await page.locator('.success-alert').waitFor({ state: 'visible', timeout: 60000 });","). Both decisions turned out to be exactly the guardrails a performance-testing reuse strategy needs. That's worth taking seriously as a general point: writing disciplined functional tests, the kind that avoid brittle fixed waits and brittle fixed test data as a matter of course, is what makes this whole approach viable later, whether or not performance testing is on anyone's mind when those tests get written.",[21,102,104],{"id":103},"bugs-that-ui-driven-load-testing-catches","Bugs That UI-Driven Load Testing Catches",[11,106,107],{},"Clegg's two bug stories are the strongest evidence for why any of this matters, and both were found specifically because testing drove the real UI rather than a simulated API contract.",[11,109,110],{},"The first came out of a two-day endurance test. A single transaction, saving a file, grew steadily from about six seconds to about twenty-four seconds over the course of the run. The cause: the save operation checked for filename collisions by fetching every file in the folder and comparing names one by one on the client side, instead of asking the server a direct \"does this name already exist?\" question. As the folder accumulated files over two days, that linear scan got proportionally slower. An audience member asked the obvious follow-up: shouldn't API-only testing have caught this too? Clegg's honest answer was that it's not that UI-driven testing finds things API testing structurally can't, it's that the client-side pre-check itself was client logic, invisible to a test that only exercises the underlying endpoints directly.",[11,112,113,114,117,118,121,122,125,126],{},"The second is the sharper one. During a six-hour test window, a burst of failures hit over about twenty minutes. A simulated user scrolled through a paginated file list and never found the file they were looking for. The trace told the story: two \"healthy\" requests, ",[86,115,116],{},"offset=0"," and ",[86,119,120],{},"offset=100",", both correctly reported 2,300 total items with a clean HTTP 200. A third request, ",[86,123,124],{},"offset=200",", also came back HTTP 200, but reported zero items. The real cause, found by cross-referencing other dashboards, was that the underlying file service had been OOM-killed and was mid-restart. The frontend's request chain hit a connection-refused error against the dead service, and instead of surfacing that failure to the browser, the API silently swallowed it and reported \"no files here\" with a healthy status code. As Clegg put it: ",[32,127,128],{},"\"That's really something we should be passing to the browser so it can then decide what to do. Instead we just tell it, 'Hey, there are no files in that location.' Which to me doesn't seem like a good failure mode.\"",[11,130,131],{},"That's functionally the same danger category as a system reporting false-healthy status because a real failure got silently absorbed somewhere in the stack, discoverable here specifically because the test was exercising what a user actually experiences (scrolling, finding nothing) rather than checking an API contract in isolation. It's the same argument I'd make for why simulating load at the API layer risks missing things: the client-observable behavior and the API's technical response can quietly diverge, and only one of them is what your customer actually lives with.",[11,133,134],{},"A third, more incidental finding is worth a mention: because Clegg's team pools test containers from many product teams and runs them together over long windows, they also surfaced a RabbitMQ service leak that only showed up under sustained multi-product load, something no single team's isolated testing would have caught, since some contributing products don't even have a dedicated performance engineer. Pooling containers from many teams buys you high-quality, long-duration integration testing as a side effect of the reuse strategy itself.",[21,136,138],{"id":137},"where-playwright-load-testing-stops-scaling","Where Playwright Load Testing Stops Scaling",[11,140,141],{},"Neither of us pretends this approach scales indefinitely, and the honesty on both sides is worth taking at face value.",[11,143,144],{},"Clegg's practical ceiling is around 600 concurrent users, and he's explicit that this is an Azure cost and resource constraint, not a limitation of the technique itself. SAS Studio's real usage pattern is backend-heavy with comparatively few concurrent UI sessions, so it's an appropriate ceiling for his product, not evidence the approach can't go further with more budget.",[11,146,147,148,152],{},"My own ceiling arrived much sooner. Running the ",[15,149],{"href":150,"text":151},"https:\u002F\u002Fwww.artillery.io","Artillery.io"," trial from a single workstation, concurrency became constrained by my own laptop, not the server, at somewhere around 10 to 12 concurrent users. Push past that and the test runner's own resource contention started polluting the timing data itself, not just capping how much load I could generate. The actual fix for that ceiling exists: Artillery supports distributed load generation across multiple machines in its licensed tier. Unfortunately we found our environment's network configuration requirements made adopting it problematic compared to our other options. Had we pursued the distributed architecture we probably would have hit a very similar load limit as Clegg.",[11,154,155],{},"Different scales, different specific causes, same underlying wall: browser-driven load generation is resource-expensive, and every team running it eventually meets a ceiling that has nothing to do with whether the approach is sound.",[11,157,158],{},"Even at 10-12 users, Artillery and Playwright surfaced real hotspots worth investigating.",[21,160,162],{"id":161},"two-paths-to-the-same-hybrid-load-testing-approach","Two Paths to the Same Hybrid Load Testing Approach",[11,164,165],{},"Clegg's explicit recommendation for teams that need higher concurrency than pure UI-driven testing can support is a hybrid: drive bulk load through API tests, and layer a smaller number of these UI-based sessions on top to get real client-side visibility while the system is under that load.",[11,167,168],{},"Interestingly, I landed on the same architecture, but not by choosing it. With Artillery's distributed scaling blocked, we moved to K6 instead, which meant giving something up: unlike Artillery, K6 can't run our existing Playwright scripts directly. K6 has its own browser-based test engine with similar-looking syntax, but a fundamentally different approach. Instead of generating load through the UI tests themselves, K6 uses API-driven load to scale up concurrency, then runs browser-based tests to sample the actual front-end experience while the server sits under that API-generated load. It sidesteps the browser-based resource ceiling entirely, at the cost of some realism, the load itself isn't browser-driven, only the sampling is.",[11,170,171,172,176],{},"Grafana’s own K6 documentation backs this up directly: their ",[15,173],{"href":174,"text":175},"https:\u002F\u002Fgrafana.com\u002Fdocs\u002Fk6\u002Flatest\u002Fusing-k6-browser\u002Frecommended-practices\u002Fhybrid-approach-to-performance\u002F","recommended hybrid approach"," describes combining \"a small number of virtual users for a browser test with a large number of virtual users for a protocol-level test\" as an alternative to full browser-based load testing that’s \"much less resource-intensive.\"",[11,178,179],{},"Clegg reached this design as a considered recommendation for other teams. I reached it as a forced compromise after my preferred path got blocked. It's the same architecture either way, arrived at through two unrelated tool stacks by two people who'd never heard of each other. That's a stronger argument for the hybrid model than either of us could have made alone.",[21,181,183],{"id":182},"limitations-of-reusing-playwright-tests-for-load-testing","Limitations of Reusing Playwright Tests for Load Testing",[11,185,186],{},"Clegg's mentioned in closing this isn't free: real effort goes into instrumenting containers and building the supporting infrastructure around them. Quality is mixed once you're borrowing tests from teams outside your own; his immediate team's tests are great, some neighboring teams' are usable, and further out across SAS's org it's \"a mixed bag.\" And he's explicit that this is not true performance tooling: it's a repurposed functional-test tool, and dedicated API-based load tools remain the right choice for genuinely high concurrency, thousands of users, not hundreds.",[11,188,189],{},"My own honest limitation is the one I already described: once we moved to K6, we lost the direct reuse benefit that made this appealing in the first place. We're back to maintaining a second set of test logic, just one that shares a language and a syntax style with our functional tests rather than the tests themselves. That said, we're now using AI to streamline the conversion process, which has made maintaining two sets of test logic less painful than it used to be, even though it's still fundamentally two sets of tests.",[21,191,193],{"id":192},"takeaway-reuse-playwright-tests-but-know-when-to-go-hybrid","Takeaway: Reuse Playwright Tests, But Know When to Go Hybrid",[11,195,196],{},"Pure API-simulated load is fast to build and easy to scale, but it drifts silently from what your product actually does as the UI changes underneath it, and it can't see client-side logic at all. Pure UI-driven load testing is accurate and reuses work your QA team already did, but it doesn't scale on ordinary hardware, or even on cloud infrastructure, without real cost. The honest answer for most teams sits in between: reuse your functional tests for accuracy where you can, and accept an API-driven hybrid once concurrency demands outgrow what browser-driven load can realistically generate. Two teams that had never heard of each other landed on exactly that answer, from opposite directions, within the same year.",[198,199],"read-next",{":items":200},"[\"\u002Fsoftware-testing\u002Ftest-automation\u002Fai-performance-testing-guardrails\",\"\u002Fsoftware-testing\u002Ftest-automation\u002Fstareast-2026-playwright-ai-cost-efficient-testing\",\"\u002Fsoftware-testing\u002Ftest-automation\u002Fai-test-automation-pitfalls-vs-user-error\",\"\u002Fsoftware-testing\u002Ftest-automation\u002Fautomated-api-testing-with-schemathesis\"]",{"title":202,"searchDepth":203,"depth":203,"links":204},"",2,[205,206,207,208,209,210,211,212],{"id":23,"depth":203,"text":24},{"id":49,"depth":203,"text":50},{"id":68,"depth":203,"text":69},{"id":103,"depth":203,"text":104},{"id":137,"depth":203,"text":138},{"id":161,"depth":203,"text":162},{"id":182,"depth":203,"text":183},{"id":192,"depth":203,"text":193},"\u002Fimages\u002Fposts\u002Freuse-playwright-tests-for-load-testing\u002Freuse-playwright-tests-for-load-testing-cover.webp","2026-08-30","Reuse existing Playwright functional tests for load testing instead of building a parallel suite. My own experience, validated by a StarEast 2026 talk.",false,"md",{},true,"\u002Fsoftware-testing\u002Ftest-automation\u002Freuse-playwright-tests-for-load-testing",{"title":5,"description":215},"software-testing\u002Ftest-automation\u002Freuse-playwright-tests-for-load-testing","Y4H6EgnXgiRY2xiiq8_rbZQiZFCsMEk1yjt39yecWNs",[225,1156,2325,3133],{"id":226,"title":227,"bmcUsername":6,"body":228,"cover":1148,"date":1149,"description":1150,"draft":216,"extension":217,"features":6,"githubRepo":6,"headline":6,"highlight":6,"icon":6,"meta":1151,"navigation":219,"npmPackage":6,"order":6,"path":1152,"seo":1153,"stem":1154,"__hash__":1155},"content\u002Fsoftware-testing\u002Ftest-automation\u002Fai-performance-testing-guardrails.md","AI Performance Testing Guardrails Every Tester Needs",{"type":8,"value":229,"toc":1140},[230,238,241,245,248,251,258,261,643,646,649,653,656,659,662,686,693,704,707,871,881,885,888,948,951,1059,1062,1066,1073,1076,1079,1083,1096,1099,1103,1106,1109,1112,1133,1136],[11,231,232,233,237],{},"Performance testing isn't my primary discipline. I get pulled into it occasionally, enough that I wanted a real read on where AI actually helps in that space and where it's just hype, before the next time it comes up. That's what took me to ",[15,234],{"href":235,"text":236},"https:\u002F\u002Fkaushaldalvi.com\u002F","Kaushal Dalvi","'s StarEast 2026 session, \"Beyond the Dev Box: Claude Code Across the Entire Performance Testing Lifecycle.\" Dalvi leads a performance engineering team and spent the session making a specific argument: performance engineering has always carried a hidden tax, hours spent operating tools instead of doing the actual engineering judgment. Better observability tooling cut that tax once already, without removing the need for performance engineers. His claim is that Claude Code and other AI coding agents are the next wave of the same pattern, provided you constrain them with the right guardrails.",[11,239,240],{},"What made that argument land for me was a real incident, not a hypothetical demo.",[21,242,244],{"id":243},"the-ai-generated-script-that-let-a-critical-performance-issue-slip-into-production","The AI-Generated Script That Let a Critical Performance Issue Slip Into Production",[11,246,247],{},"An intern on Dalvi's team, new to a project that needed performance testing, used Claude Code to generate a K6 script for an API endpoint. The script was built in 30 minutes, running within the hour, and the results were published within half a day. A senior engineer on the team, with 20 to 25 years of performance engineering experience, estimated the same work would have taken him closer to a full day by hand. The results looked great too: 100% success rate, solid response times.",[11,249,250],{},"Days later, the application went live and fell over almost immediately, even under low load.",[11,252,253,254,257],{},"The root cause, found only after the fact, was a ",[86,255,256],{},"try\u002Fcatch"," block wrapped around a key transaction in the generated script, silently swallowing every error. That's a silent failure in the truest sense: the tool reported 100% success because it genuinely never saw one. The errors were being caught and discarded before K6 could record them.",[11,259,260],{},"Below is an example of what this might have looked like, but wasn't shown during the lecture.",[262,263,268],"pre",{"className":264,"code":265,"filename":266,"language":267,"meta":202,"style":202},"language-js shiki shiki-themes material-theme-lighter github-light-high-contrast github-dark-high-contrast","import http from 'k6\u002Fhttp';\nimport { check } from 'k6';\n\nexport const options = {\n  vus: 50,\n  duration: '5m',\n};\n\nexport default function () {\n  try {\n    const res = http.post('https:\u002F\u002Fshop-demo.io\u002Fapi\u002Fcheckout', payload());\n\n    if (res.status !== 200) {\n      throw new Error(`Checkout failed: ${res.status}`);\n    }\n\n    check(res, { 'checkout succeeded': (r) => r.status === 200 });\n  } catch (err) {\n    \u002F\u002F Every real failure lands here and goes nowhere.\n    \u002F\u002F No check() runs, no error metric increments, nothing is recorded.\n  }\n}\n","checkout-load-test.js","js",[86,269,270,301,326,332,353,370,387,393,398,415,423,465,470,499,536,542,547,600,618,625,631,637],{"__ignoreMap":202},[271,272,275,279,283,286,290,294,297],"span",{"class":273,"line":274},"line",1,[271,276,278],{"class":277},"sZTni","import",[271,280,282],{"class":281},"sZ-rw"," http ",[271,284,285],{"class":277},"from",[271,287,289],{"class":288},"sZi47"," '",[271,291,293],{"class":292},"srGNg","k6\u002Fhttp",[271,295,296],{"class":288},"'",[271,298,300],{"class":299},"sPJuK",";\n",[271,302,303,305,308,311,314,317,319,322,324],{"class":273,"line":203},[271,304,278],{"class":277},[271,306,307],{"class":299}," {",[271,309,310],{"class":281}," check",[271,312,313],{"class":299}," }",[271,315,316],{"class":277}," from",[271,318,289],{"class":288},[271,320,321],{"class":292},"k6",[271,323,296],{"class":288},[271,325,300],{"class":299},[271,327,329],{"class":273,"line":328},3,[271,330,331],{"emptyLinePlaceholder":219},"\n",[271,333,335,338,342,346,350],{"class":273,"line":334},4,[271,336,337],{"class":277},"export",[271,339,341],{"class":340},"stWsX"," const",[271,343,345],{"class":344},"sQ79N"," options",[271,347,349],{"class":348},"sE6rD"," =",[271,351,352],{"class":299}," {\n",[271,354,356,360,363,367],{"class":273,"line":355},5,[271,357,359],{"class":358},"sq0XF","  vus",[271,361,362],{"class":299},":",[271,364,366],{"class":365},"s6g51"," 50",[271,368,369],{"class":299},",\n",[271,371,373,376,378,380,383,385],{"class":273,"line":372},6,[271,374,375],{"class":358},"  duration",[271,377,362],{"class":299},[271,379,289],{"class":288},[271,381,382],{"class":292},"5m",[271,384,296],{"class":288},[271,386,369],{"class":299},[271,388,390],{"class":273,"line":389},7,[271,391,392],{"class":299},"};\n",[271,394,396],{"class":273,"line":395},8,[271,397,331],{"emptyLinePlaceholder":219},[271,399,401,403,406,409,413],{"class":273,"line":400},9,[271,402,337],{"class":277},[271,404,405],{"class":277}," default",[271,407,408],{"class":340}," function",[271,410,412],{"class":411},"sCRTB"," ()",[271,414,352],{"class":299},[271,416,418,421],{"class":273,"line":417},10,[271,419,420],{"class":277},"  try",[271,422,352],{"class":299},[271,424,426,429,432,434,437,440,444,447,449,452,454,457,460,463],{"class":273,"line":425},11,[271,427,428],{"class":340},"    const",[271,430,431],{"class":344}," res",[271,433,349],{"class":348},[271,435,436],{"class":281}," http",[271,438,439],{"class":299},".",[271,441,443],{"class":442},"sb1SK","post",[271,445,446],{"class":358},"(",[271,448,296],{"class":288},[271,450,451],{"class":292},"https:\u002F\u002Fshop-demo.io\u002Fapi\u002Fcheckout",[271,453,296],{"class":288},[271,455,456],{"class":299},",",[271,458,459],{"class":442}," payload",[271,461,462],{"class":358},"())",[271,464,300],{"class":299},[271,466,468],{"class":273,"line":467},12,[271,469,331],{"emptyLinePlaceholder":219},[271,471,473,476,479,482,484,487,490,493,496],{"class":273,"line":472},13,[271,474,475],{"class":277},"    if",[271,477,478],{"class":358}," (",[271,480,481],{"class":281},"res",[271,483,439],{"class":299},[271,485,486],{"class":281},"status",[271,488,489],{"class":348}," !==",[271,491,492],{"class":365}," 200",[271,494,495],{"class":358},") ",[271,497,498],{"class":299},"{\n",[271,500,502,505,508,511,513,516,519,522,524,526,528,531,534],{"class":273,"line":501},14,[271,503,504],{"class":277},"      throw",[271,506,507],{"class":348}," new",[271,509,510],{"class":442}," Error",[271,512,446],{"class":358},[271,514,515],{"class":288},"`",[271,517,518],{"class":292},"Checkout failed: ",[271,520,521],{"class":288},"${",[271,523,481],{"class":281},[271,525,439],{"class":288},[271,527,486],{"class":281},[271,529,530],{"class":288},"}`",[271,532,533],{"class":358},")",[271,535,300],{"class":299},[271,537,539],{"class":273,"line":538},15,[271,540,541],{"class":299},"    }\n",[271,543,545],{"class":273,"line":544},16,[271,546,331],{"emptyLinePlaceholder":219},[271,548,550,553,555,557,559,561,563,567,569,571,573,577,579,582,585,587,589,592,594,596,598],{"class":273,"line":549},17,[271,551,552],{"class":442},"    check",[271,554,446],{"class":358},[271,556,481],{"class":281},[271,558,456],{"class":299},[271,560,307],{"class":299},[271,562,289],{"class":288},[271,564,566],{"class":565},"sqmHM","checkout succeeded",[271,568,296],{"class":288},[271,570,362],{"class":299},[271,572,478],{"class":299},[271,574,576],{"class":575},"s2xgV","r",[271,578,533],{"class":299},[271,580,581],{"class":340}," =>",[271,583,584],{"class":281}," r",[271,586,439],{"class":299},[271,588,486],{"class":281},[271,590,591],{"class":348}," ===",[271,593,492],{"class":365},[271,595,313],{"class":299},[271,597,533],{"class":358},[271,599,300],{"class":299},[271,601,603,606,609,611,614,616],{"class":273,"line":602},18,[271,604,605],{"class":299},"  }",[271,607,608],{"class":277}," catch",[271,610,478],{"class":358},[271,612,613],{"class":281},"err",[271,615,495],{"class":358},[271,617,498],{"class":299},[271,619,621],{"class":273,"line":620},19,[271,622,624],{"class":623},"s_gjE","    \u002F\u002F Every real failure lands here and goes nowhere.\n",[271,626,628],{"class":273,"line":627},20,[271,629,630],{"class":623},"    \u002F\u002F No check() runs, no error metric increments, nothing is recorded.\n",[271,632,634],{"class":273,"line":633},21,[271,635,636],{"class":299},"  }\n",[271,638,640],{"class":273,"line":639},22,[271,641,642],{"class":299},"}\n",[11,644,645],{},"Dalvi's read on why the model did this is the sharpest line from the whole talk: the model was trying to prevent errors because it thought that was the helpful thing to do, and in doing so it lost sight of what the script was actually for. A performance test exists to surface failures under load. A model optimizing for looking correct will do the opposite of that unless something stops it.",[11,647,648],{},"That's a general problem with AI-generated code, not a performance-testing-specific one. It just happened to show up here in a form with real production consequences.",[21,650,652],{"id":651},"determinism-vs-non-determinism","Determinism vs. Non-Determinism",[11,654,655],{},"The thread running through the rest of the session was a distinction Dalvi kept returning to: determinism versus non-determinism, and how you put guardrails around the second one.",[11,657,658],{},"He illustrated it with a log analysis demo. He fed Claude a raw access log, hundreds of thousands of lines, the kind of file his team used to hand-parse before modern observability tooling existed. He was upfront that this was a contrived setup (an audience member correctly pointed out that no real organization works from a raw log with no retention policy), but the point he wanted to make didn't depend on the example being realistic.",[11,660,661],{},"The failure mode he was demonstrating: dump a huge log into a chat window and ask for analysis, and you'll get a plausible-looking answer that's different every time you ask. That's next-token prediction doing what it does. The fix wasn't to trust the model's summary less. It was to change what he asked the model to do:",[663,664,665,677,680,683],"ol",{},[666,667,668,669,672,673,676],"li",{},"Don't read the whole file into context. Inspect it first, using ",[86,670,671],{},"head",", ",[86,674,675],{},"tail",", random sampling, and time-range counts to learn its shape.",[666,678,679],{},"Write scripts that do the actual analysis, rather than reasoning over the raw data directly.",[666,681,682],{},"Run those scripts. The output is deterministic and repeatable because it came from code, not from a language model's recollection of what it read.",[666,684,685],{},"Cite every number back to the script or file it came from, so the results can be checked rather than taken on faith.",[11,687,688,689,692],{},"The scripts turned hundreds of thousands of log lines into a ",[86,690,691],{},"workload-model.md"," with request-per-second figures, concurrent user estimates using Little's Law, and endpoint hit ratios, none of which the model could have reliably held in its head from a single pass over the raw file.",[694,695,698],"callout",{"color":696,"icon":697},"info","i-lucide-info",[11,699,700,703],{},[40,701,702],{},"Little's Law",", the average number of things in a stable system (L) equals how often new ones arrive (λ) multiplied by how long each one stays (W), or L = λW. Applied to a workload model, concurrent users equal how many sessions start per second multiplied by the average session length. In other words, if customers walk into a coffee shop every 2 minutes and each one lingers for 10 minutes, there are usually 5 people in the shop at any given moment. Same math, just users and sessions instead of customers and minutes.",[11,705,706],{},"Dalvi didn't share the file itself (there's no code repo for this session). What follows is a reconstruction matching the shape he described, including the part that mattered most to him: every number traceable back to the script that produced it.",[262,708,712],{"className":709,"code":710,"filename":691,"language":711,"meta":202,"style":202},"language-markdown shiki shiki-themes material-theme-lighter github-light-high-contrast github-dark-high-contrast","# Workload Model: access.log analysis\n\n**Source:** access.log (438,201 lines, 2026-05-01 to 2026-05-07)\n**Generated by:** analyze_endpoints.py, compute_percentiles.py, estimate_concurrency.py\n\n## Endpoint Hit Ratios\n| Endpoint | Requests | % of Total |\n|---|---|---|\n| GET \u002Fapi\u002Fproducts | 182,340 | 41.6% |\n| POST \u002Fapi\u002Fcart | 96,112 | 21.9% |\n| GET \u002Fapi\u002Fsearch | 74,558 | 17.0% |\n| POST \u002Fapi\u002Fcheckout | 41,209 | 9.4% |\n| other | 43,982 | 10.1% |\n\n*Source: analyze_endpoints.py*\n\n## Latency Percentiles (ms)\n| Endpoint | p50 | p95 | p99 |\n|---|---|---|---|\n| GET \u002Fapi\u002Fproducts | 82 | 310 | 640 |\n| POST \u002Fapi\u002Fcart | 110 | 420 | 810 |\n| POST \u002Fapi\u002Fcheckout | 145 | 560 | 1,020 |\n\n*Source: compute_percentiles.py*\n\n## Concurrency Estimate (Little's Law)\n- Session arrival rate (λ): 0.92 sessions\u002Fsec, derived from unique session IDs per minute\n- Average session duration (W): 3.8 minutes (228 seconds), derived from first-to-last timestamp per session ID\n- Estimated concurrent users (L = λ × W): 0.92 × 228 ≈ 210\n\n*Source: estimate_concurrency.py*\n","markdown",[86,713,714,719,723,728,733,737,742,747,752,757,762,767,772,777,781,786,790,795,800,805,810,815,820,825,831,836,842,848,854,860,865],{"__ignoreMap":202},[271,715,716],{"class":273,"line":274},[271,717,718],{},"# Workload Model: access.log analysis\n",[271,720,721],{"class":273,"line":203},[271,722,331],{"emptyLinePlaceholder":219},[271,724,725],{"class":273,"line":328},[271,726,727],{},"**Source:** access.log (438,201 lines, 2026-05-01 to 2026-05-07)\n",[271,729,730],{"class":273,"line":334},[271,731,732],{},"**Generated by:** analyze_endpoints.py, compute_percentiles.py, estimate_concurrency.py\n",[271,734,735],{"class":273,"line":355},[271,736,331],{"emptyLinePlaceholder":219},[271,738,739],{"class":273,"line":372},[271,740,741],{},"## Endpoint Hit Ratios\n",[271,743,744],{"class":273,"line":389},[271,745,746],{},"| Endpoint | Requests | % of Total |\n",[271,748,749],{"class":273,"line":395},[271,750,751],{},"|---|---|---|\n",[271,753,754],{"class":273,"line":400},[271,755,756],{},"| GET \u002Fapi\u002Fproducts | 182,340 | 41.6% |\n",[271,758,759],{"class":273,"line":417},[271,760,761],{},"| POST \u002Fapi\u002Fcart | 96,112 | 21.9% |\n",[271,763,764],{"class":273,"line":425},[271,765,766],{},"| GET \u002Fapi\u002Fsearch | 74,558 | 17.0% |\n",[271,768,769],{"class":273,"line":467},[271,770,771],{},"| POST \u002Fapi\u002Fcheckout | 41,209 | 9.4% |\n",[271,773,774],{"class":273,"line":472},[271,775,776],{},"| other | 43,982 | 10.1% |\n",[271,778,779],{"class":273,"line":501},[271,780,331],{"emptyLinePlaceholder":219},[271,782,783],{"class":273,"line":538},[271,784,785],{},"*Source: analyze_endpoints.py*\n",[271,787,788],{"class":273,"line":544},[271,789,331],{"emptyLinePlaceholder":219},[271,791,792],{"class":273,"line":549},[271,793,794],{},"## Latency Percentiles (ms)\n",[271,796,797],{"class":273,"line":602},[271,798,799],{},"| Endpoint | p50 | p95 | p99 |\n",[271,801,802],{"class":273,"line":620},[271,803,804],{},"|---|---|---|---|\n",[271,806,807],{"class":273,"line":627},[271,808,809],{},"| GET \u002Fapi\u002Fproducts | 82 | 310 | 640 |\n",[271,811,812],{"class":273,"line":633},[271,813,814],{},"| POST \u002Fapi\u002Fcart | 110 | 420 | 810 |\n",[271,816,817],{"class":273,"line":639},[271,818,819],{},"| POST \u002Fapi\u002Fcheckout | 145 | 560 | 1,020 |\n",[271,821,823],{"class":273,"line":822},23,[271,824,331],{"emptyLinePlaceholder":219},[271,826,828],{"class":273,"line":827},24,[271,829,830],{},"*Source: compute_percentiles.py*\n",[271,832,834],{"class":273,"line":833},25,[271,835,331],{"emptyLinePlaceholder":219},[271,837,839],{"class":273,"line":838},26,[271,840,841],{},"## Concurrency Estimate (Little's Law)\n",[271,843,845],{"class":273,"line":844},27,[271,846,847],{},"- Session arrival rate (λ): 0.92 sessions\u002Fsec, derived from unique session IDs per minute\n",[271,849,851],{"class":273,"line":850},28,[271,852,853],{},"- Average session duration (W): 3.8 minutes (228 seconds), derived from first-to-last timestamp per session ID\n",[271,855,857],{"class":273,"line":856},29,[271,858,859],{},"- Estimated concurrent users (L = λ × W): 0.92 × 228 ≈ 210\n",[271,861,863],{"class":273,"line":862},30,[271,864,331],{"emptyLinePlaceholder":219},[271,866,868],{"class":273,"line":867},31,[271,869,870],{},"*Source: estimate_concurrency.py*\n",[11,872,873,874,877,878,880],{},"That citation line under each section is what lets a reviewer walk back to ",[86,875,876],{},"compute_percentiles.py"," and check the p95 figure against the raw log instead of taking the model's summary on faith. The same discipline (inspect, script, run, cite) is what would have caught the K6 script's silent ",[86,879,256],{}," before it shipped: a reviewer working from a script's actual behavior, rather than the tool's self-reported summary, would have seen the error handling directly.",[21,882,884],{"id":883},"a-guardrails-checklist-for-the-next-time-i-get-pulled-in","A Guardrails Checklist for the Next Time I Get Pulled In",[11,886,887],{},"The rest of the session built out a stack of guardrails in increasing order of formality. I'm not going to pretend I've used all of these in production the way Dalvi's team has, but this is the checklist I'd actually reach for the next time I'm asked to help with AI-assisted performance testing:",[889,890,891,904,913,919,925,936],"ul",{},[666,892,893,896,897,117,900,903],{},[40,894,895],{},"Ground it in a spec."," Feed the model an OpenAPI or Swagger spec before asking it to generate test assets. Dalvi said that without this, the model invented its own ad hoc control flow, ",[86,898,899],{},"if\u002Felse",[86,901,902],{},"for"," loops inside the test scripts, instead of using the load-testing tool's built-in features, in every example his team tried before adding this guardrail.",[666,905,906,912],{},[40,907,908,909,439],{},"Write a ",[86,910,911],{},"rules.md"," A short, team-authored file listing what not to do (no swallowed errors, no logic embedded in test scripts) plus house conventions, handed to the model alongside the spec every time.",[666,914,915,918],{},[40,916,917],{},"Package repeated context as a skill."," A skill is a folder of instructions, scripts, and resources that Claude can load automatically or on request, useful for anything you'd otherwise re-explain every session, like how to call an internal CLI to pull metrics or restart a service.",[666,920,921,924],{},[40,922,923],{},"Split large tasks across subagents."," Dalvi noted that output quality tends to decline once a context window is 40 to 50% full. His team's fix was markdown-defined specialist subagents, each with isolated context. Given a 20-endpoint OpenAPI spec, he ran four K6 script-generator subagents in parallel, each following the same house rules, instead of one long session trying to hold all of it at once.",[666,926,927,930,931,935],{},[40,928,929],{},"Distribute the guardrails as a plugin once they're stable."," Dalvi's central performance engineering team hosts a ",[15,932],{"href":933,"text":934},"https:\u002F\u002Fcode.claude.com\u002Fdocs\u002Fen\u002Fplugins","plugin"," in a GitHub repository that any of their 80-plus distributed development teams can install with a single command. When the central team fixes a flaw in their guidance, every consuming team gets the fix on their next plugin update, instead of the correction living in one team's head.",[666,937,938,941,942,944,945,947],{},[40,939,940],{},"Write separate reporting rules per audience."," His team keeps different ",[86,943,911],{}," files for executive summaries (bullet points, no jargon, business impact only), engineering deep-dives (line numbers, timestamps, APM correlations), and release management (a mix of both, framed around go or no-go readiness). In one demo, Claude generated all three reports in a single pass from the same underlying analysis, swapping only which ",[86,946,911],{}," grounded each one.",[11,949,950],{},"Dalvi didn't share his team's actual rules.md files, but here's roughly the shape of what those three might look like for the same underlying finding:",[952,953,954,989,1024],"code-group",{},[262,955,958],{"className":709,"code":956,"filename":957,"language":711,"meta":202,"style":202},"# Executive Summary Rules\n\n- Bullet points only, no paragraphs\n- No technical jargon: no APM tool names, no percentile terms\n- Lead with business impact: revenue risk, customer impact, launch readiness\n- Five bullets maximum\n","executive-summary-rules.md",[86,959,960,965,969,974,979,984],{"__ignoreMap":202},[271,961,962],{"class":273,"line":274},[271,963,964],{},"# Executive Summary Rules\n",[271,966,967],{"class":273,"line":203},[271,968,331],{"emptyLinePlaceholder":219},[271,970,971],{"class":273,"line":328},[271,972,973],{},"- Bullet points only, no paragraphs\n",[271,975,976],{"class":273,"line":334},[271,977,978],{},"- No technical jargon: no APM tool names, no percentile terms\n",[271,980,981],{"class":273,"line":355},[271,982,983],{},"- Lead with business impact: revenue risk, customer impact, launch readiness\n",[271,985,986],{"class":273,"line":372},[271,987,988],{},"- Five bullets maximum\n",[262,990,993],{"className":709,"code":991,"filename":992,"language":711,"meta":202,"style":202},"# Engineering Deep-Dive Rules\n\n- Cite every claim: file name, line number, or timestamp\n- Include APM correlation IDs where available\n- Recommend the specific code or config change, not just the symptom\n- Do not summarize away technical detail for brevity\n","engineering-deep-dive-rules.md",[86,994,995,1000,1004,1009,1014,1019],{"__ignoreMap":202},[271,996,997],{"class":273,"line":274},[271,998,999],{},"# Engineering Deep-Dive Rules\n",[271,1001,1002],{"class":273,"line":203},[271,1003,331],{"emptyLinePlaceholder":219},[271,1005,1006],{"class":273,"line":328},[271,1007,1008],{},"- Cite every claim: file name, line number, or timestamp\n",[271,1010,1011],{"class":273,"line":334},[271,1012,1013],{},"- Include APM correlation IDs where available\n",[271,1015,1016],{"class":273,"line":355},[271,1017,1018],{},"- Recommend the specific code or config change, not just the symptom\n",[271,1020,1021],{"class":273,"line":372},[271,1022,1023],{},"- Do not summarize away technical detail for brevity\n",[262,1025,1028],{"className":709,"code":1026,"filename":1027,"language":711,"meta":202,"style":202},"# Release Readiness Rules\n\n- Lead with status: green, yellow, or red\n- List open risks with severity and owner\n- State a go or no-go recommendation explicitly, no hedging\n- One paragraph maximum per section, mixing technical and business framing\n","release-readiness-rules.md",[86,1029,1030,1035,1039,1044,1049,1054],{"__ignoreMap":202},[271,1031,1032],{"class":273,"line":274},[271,1033,1034],{},"# Release Readiness Rules\n",[271,1036,1037],{"class":273,"line":203},[271,1038,331],{"emptyLinePlaceholder":219},[271,1040,1041],{"class":273,"line":328},[271,1042,1043],{},"- Lead with status: green, yellow, or red\n",[271,1045,1046],{"class":273,"line":334},[271,1047,1048],{},"- List open risks with severity and owner\n",[271,1050,1051],{"class":273,"line":355},[271,1052,1053],{},"- State a go or no-go recommendation explicitly, no hedging\n",[271,1055,1056],{"class":273,"line":372},[271,1057,1058],{},"- One paragraph maximum per section, mixing technical and business framing\n",[11,1060,1061],{},"None of this removes the review step. It just means the review is checking cited, reproducible work instead of a black-box claim.",[21,1063,1065],{"id":1064},"hiring-a-second-ai-agent-to-check-the-first","Hiring a Second AI Agent to Check the First",[11,1067,1068,1069,1072],{},"The guardrail I found most transferable outside performance testing came from Dalvi's consulting background. When a company hires a consulting firm to build something, he said, it's common to hire a ",[32,1070,1071],{},"different"," firm to validate the work, because a firm validating its own output has a natural incentive to protect its own conclusions.",[11,1074,1075],{},"He applies the same idea to agents. Rather than asking one Claude session for a diagnosis and trusting it, he asks it to do the correlation work (build a timeline, cite artifacts, show its reasoning), then spins up a second, independent agent whose only job is to find flaws in the first agent's conclusion. In a bottleneck-analysis demo, he handed Claude seven artifacts from a load test where he'd deliberately injected a fault: a K6 summary, application logs, garbage collection logs, thread dumps, and slow-query exports from the application performance monitoring platform. The prompt explicitly told it to be skeptical of the first apparent cause, to flag suspicious or misleading readings, and to surface any conflicting evidence rather than pick a side silently. It correctly traced the fault to connection pool saturation at the four-minute mark, matching what Dalvi had actually injected, with citations back to the source artifact for every claim.",[11,1077,1078],{},"His summary of the approach: the judgment stays his, the grunt work doesn't.",[21,1080,1082],{"id":1081},"the-same-failure-pattern-i-already-knew-from-test-automation","The Same Failure Pattern I Already Knew From Test Automation",[11,1084,1085,1086,1088,1089,1091,1092,1095],{},"The pattern Dalvi described, an AI swallowing errors inside a ",[86,1087,256],{},", is one I've caught underskilled test automation consultants doing by hand: wrapping every test in a ",[86,1090,256],{}," so nothing ever throws. I think it comes from the same instinct, avoiding execution failures, and it's the wrong instinct either way. Tests ",[32,1093,1094],{},"should"," fail when the software under test has changed behavior. There's a real difference between making a test reliable and making it so self-healing and overly resilient that it hides defects instead of reporting them.",[11,1097,1098],{},"Dalvi's K6 story is that same failure mode wearing performance-testing clothes. The model wasn't malicious or even wrong to want the script to run cleanly. It just optimized for the wrong signal, and nothing was in place to catch it before production did.",[21,1100,1102],{"id":1101},"what-im-taking-back-to-my-own-work","What I'm Taking Back to My Own Work",[11,1104,1105],{},"The determinism guardrail is the one I'll use most, even outside performance testing: when I'm asking Claude to make sense of a large, messy input (logs, a big CSV export, a pile of test results), I'd rather it write and run a script against the raw data and cite the output than summarize the data directly from a single read. That's a cheap habit to adopt and it applies well beyond any one testing discipline.",[11,1107,1108],{},"The adversarial-agent pattern is the other one I want to start using deliberately. A second agent whose only job is to argue with the first agent's conclusion is a lightweight way to catch the kind of confident, plausible-sounding mistake that's easy to miss when you're the one who asked the question in the first place.",[11,1110,1111],{},"Performance testing is still not my daily work. But the next time I'm handed a load test and an AI assistant to help build it, I have a specific list of questions to ask before I trust the results: what is it citing this number from, what did I tell it not to do, and who's checking its work besides me.",[11,1113,1114,1115,672,1120,672,1124,1128,1129,439],{},"For more on the AI conference sessions I attended around this one: ",[1116,1117,1119],"a",{"href":1118},"\u002Fsoftware-testing\u002Ftest-automation\u002Fstareast-2026-getting-started-ai-driven-automation","AI vision testing and Playwright MCP",[1116,1121,1123],{"href":1122},"\u002Fsoftware-testing\u002Ftest-automation\u002Fstareast-2026-getting-dirty-ai-testing","hands-on AI tooling and evals",[1116,1125,1127],{"href":1126},"\u002Fsoftware-testing\u002Ftest-automation\u002Fstareast-2026-playwright-ai-cost-efficient-testing","cost-efficient Playwright testing with AI",", and ",[1116,1130,1132],{"href":1131},"\u002Fsoftware-testing\u002Ftest-automation\u002Fstareast-2026-prompt-engineering-techniques","prompt engineering techniques for testers",[198,1134],{":items":1135},"[\"\u002Fsoftware-testing\u002Ftest-automation\u002Fai-test-automation-pitfalls-vs-user-error\",\"\u002Fsoftware-testing\u002Ftest-automation\u002Fstareast-2026-prompt-engineering-techniques\",\"\u002Fsoftware-testing\u002Ftest-automation\u002Fstareast-2026-playwright-ai-cost-efficient-testing\",\"\u002Fsoftware-testing\u002Ftest-automation\u002Fhow-to-test-ai-chatbots-and-agents\"]",[1137,1138,1139],"style",{},"html pre.shiki code .sZTni,html code.shiki .sZTni{--shiki-light:#39ADB5;--shiki-light-font-style:italic;--shiki-default:#A0111F;--shiki-default-font-style:inherit;--shiki-dark:#FF9492;--shiki-dark-font-style:inherit}html pre.shiki code .sZ-rw,html code.shiki .sZ-rw{--shiki-light:#90A4AE;--shiki-default:#0E1116;--shiki-dark:#F0F3F6}html pre.shiki code .sZi47,html code.shiki .sZi47{--shiki-light:#39ADB5;--shiki-default:#032563;--shiki-dark:#ADDCFF}html pre.shiki code .srGNg,html code.shiki .srGNg{--shiki-light:#91B859;--shiki-default:#032563;--shiki-dark:#ADDCFF}html pre.shiki code .sPJuK,html code.shiki .sPJuK{--shiki-light:#39ADB5;--shiki-default:#0E1116;--shiki-dark:#F0F3F6}html pre.shiki code .stWsX,html code.shiki .stWsX{--shiki-light:#9C3EDA;--shiki-default:#A0111F;--shiki-dark:#FF9492}html pre.shiki code .sQ79N,html code.shiki .sQ79N{--shiki-light:#90A4AE;--shiki-default:#023B95;--shiki-dark:#91CBFF}html pre.shiki code .sE6rD,html code.shiki .sE6rD{--shiki-light:#39ADB5;--shiki-default:#A0111F;--shiki-dark:#FF9492}html pre.shiki code .sq0XF,html code.shiki .sq0XF{--shiki-light:#E53935;--shiki-default:#0E1116;--shiki-dark:#F0F3F6}html pre.shiki code .s6g51,html code.shiki .s6g51{--shiki-light:#F76D47;--shiki-default:#023B95;--shiki-dark:#91CBFF}html pre.shiki code .sCRTB,html code.shiki .sCRTB{--shiki-light:#39ADB5;--shiki-default:#702C00;--shiki-dark:#FFB757}html pre.shiki code .sb1SK,html code.shiki .sb1SK{--shiki-light:#6182B8;--shiki-default:#622CBC;--shiki-dark:#DBB7FF}html pre.shiki code .sqmHM,html code.shiki .sqmHM{--shiki-light:#E53935;--shiki-default:#032563;--shiki-dark:#ADDCFF}html pre.shiki code .s2xgV,html code.shiki .s2xgV{--shiki-light:#90A4AE;--shiki-light-font-style:italic;--shiki-default:#702C00;--shiki-default-font-style:inherit;--shiki-dark:#FFB757;--shiki-dark-font-style:inherit}html pre.shiki code .s_gjE,html code.shiki .s_gjE{--shiki-light:#90A4AE;--shiki-light-font-style:italic;--shiki-default:#66707B;--shiki-default-font-style:inherit;--shiki-dark:#BDC4CC;--shiki-dark-font-style:inherit}html .light .shiki span{color:var(--shiki-light);background:var(--shiki-light-bg);font-style:var(--shiki-light-font-style);font-weight:var(--shiki-light-font-weight);text-decoration:var(--shiki-light-text-decoration)}html.light .shiki span{color:var(--shiki-light);background:var(--shiki-light-bg);font-style:var(--shiki-light-font-style);font-weight:var(--shiki-light-font-weight);text-decoration:var(--shiki-light-text-decoration)}html .default .shiki span{color:var(--shiki-default);background:var(--shiki-default-bg);font-style:var(--shiki-default-font-style);font-weight:var(--shiki-default-font-weight);text-decoration:var(--shiki-default-text-decoration)}html .shiki span{color:var(--shiki-default);background:var(--shiki-default-bg);font-style:var(--shiki-default-font-style);font-weight:var(--shiki-default-font-weight);text-decoration:var(--shiki-default-text-decoration)}html .dark .shiki span{color:var(--shiki-dark);background:var(--shiki-dark-bg);font-style:var(--shiki-dark-font-style);font-weight:var(--shiki-dark-font-weight);text-decoration:var(--shiki-dark-text-decoration)}html.dark .shiki span{color:var(--shiki-dark);background:var(--shiki-dark-bg);font-style:var(--shiki-dark-font-style);font-weight:var(--shiki-dark-font-weight);text-decoration:var(--shiki-dark-text-decoration)}",{"title":202,"searchDepth":203,"depth":203,"links":1141},[1142,1143,1144,1145,1146,1147],{"id":243,"depth":203,"text":244},{"id":651,"depth":203,"text":652},{"id":883,"depth":203,"text":884},{"id":1064,"depth":203,"text":1065},{"id":1081,"depth":203,"text":1082},{"id":1101,"depth":203,"text":1102},"\u002Fimages\u002Fposts\u002Fai-performance-testing-guardrails\u002Fai-performance-testing-guardrails-cover.webp","2026-08-29","An AI-generated performance test missed a real bug, and the app broke in production. StarEast 2026 lessons on the guardrails that would have caught it.",{},"\u002Fsoftware-testing\u002Ftest-automation\u002Fai-performance-testing-guardrails",{"title":227,"description":1150},"software-testing\u002Ftest-automation\u002Fai-performance-testing-guardrails","r9dtt9EAy3QYEFk2tCph0o5I0SQmCUowpUA_KaYNrBI",{"id":1157,"title":1158,"bmcUsername":6,"body":1159,"cover":2318,"date":2319,"description":2320,"draft":216,"extension":217,"features":6,"githubRepo":6,"headline":6,"highlight":6,"icon":6,"meta":2321,"navigation":219,"npmPackage":6,"order":6,"path":1126,"seo":2322,"stem":2323,"__hash__":2324},"content\u002Fsoftware-testing\u002Ftest-automation\u002Fstareast-2026-playwright-ai-cost-efficient-testing.md","Playwright AI Testing on a Budget: Locators vs. Computer Vision — StarEast 2026",{"type":8,"value":1160,"toc":2310},[1161,1164,1183,1186,1193,1197,1200,1203,1209,1212,1215,1220,1227,1231,1238,1244,1251,1254,1999,2014,2018,2021,2024,2037,2042,2045,2051,2056,2088,2099,2102,2105,2125,2131,2138,2145,2150,2153,2185,2188,2193,2196,2200,2207,2212,2215,2220,2223,2228,2235,2239,2242,2247,2250,2253,2260,2263,2267,2270,2296,2304,2307],[11,1162,1163],{},"Andy Knight's half-day StarEast 2026 tutorial, officially titled \"Top-Notch Web Testing with Playwright and AI,\" was billed as a hands-on walkthrough, and for most of its four hours, that's exactly what it was. Two claims kept it from being just another how-to for me. Playwright's MCP server can burn through an AI testing budget fast enough to matter (one joke about a junior developer's $5,000 month illustrated that point), and computer vision based testing, despite what a different StarEast tutorial argued the day before, is unlikely to replace locator-based Playwright tests anytime soon.",[11,1165,1166,1167,1171,1172,117,1175,1178,1179,1182],{},"Knight, who goes by Pandy or Automation Panda depending on which corner of the testing internet you found him in, is an actual ",[15,1168],{"href":1169,"text":1170},"https:\u002F\u002Fautomationpanda.com","Playwright Ambassador",". His session was the third of four StarEast 2026 tutorials I attended over two days, the first two are their own write-ups, on ",[1116,1173,1174],{"href":1118},"getting started with AI-driven automation and AI vision testing",[1116,1176,1177],{"href":1122},"evals, vibe coding, and prompt engineering",". Knight acknowledged near the end that the class hadn't gotten through the whole tutorial repository live, \"we only got through about half of what's in the tutorial repository.\" Part of that had a funny explanation: Knight assumed most of the class had simply ignored the prerequisite machine setup instructions he'd sent out ahead of time. It turned out the StarEast organizers never actually emailed those instructions to anyone. So the room spent a chunk of class scrambling to install several hundred megabytes of Playwright's browser dependencies over the now-saturated conference Wi-Fi. The organizers only figured out what happened when they noticed the network anomaly and mentioned it to Knight, at which point I felt vindicated, I'd been ",[32,1180,1181],{},"certain"," no such instructions were ever sent and had assumed I'd just failed to do my homework.",[11,1184,1185],{},"Everything below is what we actually built and discussed in the room, plus what I read in his written tutorial chapters afterward to fill in gaps.",[11,1187,1188],{},[1189,1190],"img",{"alt":1191,"src":1192},"Andy Knight presenting his Playwright and AI tutorial at StarEast 2026","\u002Fimages\u002Fposts\u002Fstareast-2026-playwright-ai-cost-efficient-testing\u002Fandy-knight-stareast-2026.webp",[21,1194,1196],{"id":1195},"playwright-vs-selenium-what-actually-got-fixed","Playwright vs. Selenium: What Actually Got Fixed",[11,1198,1199],{},"Knight opened by asking the room what makes test automation hard, and the answers came fast: tests are slow, brittle, flaky, don't make sense when you read them back, don't make money (a real line, \"we're not shipping tests to customers\"), and force a context switch every time you flip from building a feature to testing it.",[11,1201,1202],{},"The classic fix for this was the Testing Pyramid, lots of cheap unit tests at the base, fewer expensive UI tests at the top, because UI tests were \"big, slow, and expensive.\" Knight's pushback wasn't that the pyramid's diagnosis was wrong. It was that the diagnosis got blamed on the wrong cause:",[1204,1205,1206],"blockquote",{},[11,1207,1208],{},"\"End-to-end tests can be very valuable. Unfortunately, the Testing Pyramid labeled them as 'difficult' and 'bad' primarily due to poor practices and tool shortcomings.\"",[11,1210,1211],{},"He had a punchier name for what should replace pyramid-style thinking (\"we don't build pyramids anymore, we build skyscrapers\"). We'll revisit that line in a later section because I don't think it holds up quite as cleanly as it sounded in the room at the time.",[11,1213,1214],{},"What does hold up is the tooling argument. Playwright's actual fix for \"UI tests are slow and flaky\" is architectural: one browser instance per worker, with each test pulling its own isolated browser context out of that instance (\"akin to an incognito session, or a mini container in your browser\"), and each context holding one or more pages. Spinning up a context is nearly instant, which is the opposite of Selenium's per-test full-browser-relaunch model. Knight's own story below, about discovering this, resonated with me because I had a similar reaction when using Playwright for the first time.",[1204,1216,1217],{},[11,1218,1219],{},"\"I remember the first time I used Playwright, this was back in late 2021... I quickly bang out about a dozen tests or so... I go to the terminal, I'm like npx Playwright test, run it, hit it, and then within a second it comes back and it says 12 tests passed. And I'm like, no, no, no, no, no, it didn't find the tests, it didn't run the tests, it skipped it, something went wrong... then I run it in headed mode, and it was so fast... I was expecting each test to take about a minute, because I came from Selenium, but it's like when I say it's freaky fast man, it is, it screams.\"",[11,1221,1222,1223,1226],{},"Playwright avoids the behavior that gives Selenium its flaky reputation by, among other things, polling automatically: locators and assertions keep rechecking until they succeed or time out, instead of failing the instant they're called, if misaligned. Selenium does the opposite by default, checking once, so a test that forgets to include explicit waits fails the moment the page hasn't caught up yet. Playwright's defaults give that polling a generous window: locator actions retry for 30 seconds, ",[86,1224,1225],{},"expect"," assertions for 5, enough slack to absorb a slower page load between runs without anyone configuring a thing. Knight was fair to say, \"Selenium itself is not flaky, it's the tests that people write with it.\" Playwright's real contribution is removing a specific set of execution-speed and tooling-friction problems that made E2E testing painful for the last decade, not inventing testing concepts from scratch.",[21,1228,1230],{"id":1229},"from-codegen-to-a-real-test","From Codegen to a Real Test",[11,1232,1233,1234,1237],{},"The hands-on portion started with ",[86,1235,1236],{},"npx playwright codegen"," against a local Trello-style Kanban app (a clone built by Filip Hric, used with permission). Codegen records your clicks and fills into a script, and the output is rough on purpose, Knight's framing: \"there's a difference between a script and a test case... we can use this to ruthlessly refine it into a better test case.\"",[11,1239,1240],{},[1189,1241],{"alt":1242,"src":1243},"Trello app being tested","\u002Fimages\u002Fposts\u002Fstareast-2026-playwright-ai-cost-efficient-testing\u002Ftrello-app-under-test-listview.webp",[11,1245,1246,1247,1250],{},"Refining it meant three things: trimming the clicks codegen over-records (you don't need to click an input before typing into it), picking stable locators (",[86,1248,1249],{},"data-testid"," attributes if you control the app, \"these are very nice test hooks to have\"), and adding the assertions codegen never gives you, since codegen only captures interactions, not verifications.",[11,1252,1253],{},"We iterated from the raw click events through refining the flow so it could be run repeatedly by adding things like pre and post test hooks to ensure the test launches in the correct state and doesn't leave behind past entries that would cause different state between runs. Here's my own rough version of that test, written live in the room.:",[262,1255,1260],{"className":1256,"code":1257,"filename":1258,"language":1259,"meta":202,"style":202},"language-typescript shiki shiki-themes material-theme-lighter github-light-high-contrast github-dark-high-contrast","import { test, expect } from '@playwright\u002Ftest';\n\ntest.beforeEach(async ({ page, request }) => {\n  \u002F\u002F Added this reset endpoint to erase the board and then naivate to the app at the start of each test run\n  await request.post('http:\u002F\u002Flocalhost:3000\u002Fapi\u002Freset');\n  await page.goto('http:\u002F\u002Flocalhost:3000\u002F');\n});\n\ntest.afterEach(async ({ request }) => {\n  \u002F\u002F Added this explicit reset after each test to erase the board (belt and suspenders with the beforeEach's erase)\n  await request.post('http:\u002F\u002Flocalhost:3000\u002Fapi\u002Freset');\n});\n\ntest.afterAll(async ({ browser }) => {\n  \u002F\u002F Added to close down the browser after all the tests complete\n  await browser.close();\n});\n\ntest('Create a new board with list and cards', async ({ page }) => {\n  \u002F\u002F You'll notice the selector repetition and lack of page objects which we didn't get to during the session \u002F wasn't a primary focus\n  await page.getByTestId('first-board').click();\n  await page.getByTestId('first-board').fill('chores');\n  await page.getByTestId('first-board').press('Enter');\n\n  expect(page.getByTestId('first-board')).toHaveValue('chores');\n\n  await page.getByTestId('add-list-input').click();\n  await page.getByTestId('add-list-input').fill('todo');\n  await page.getByRole('button', { name: 'Add list' }).click();\n  await page.getByTestId('new-card').click();\n  await page.getByTestId('new-card-input').fill('walk the dog');\n  await page.getByTestId('new-card-input').click();\n  await page.getByTestId('new-card-input').fill('mow the lawn');\n  await page.getByTestId('home').click();\n\n  \u002F\u002F Didn't have a chance to add more assertions, was helping classmates with setup.\n});\n","trello.spec.ts","typescript",[86,1261,1262,1289,1293,1326,1331,1355,1379,1388,1392,1415,1420,1442,1450,1454,1478,1483,1499,1507,1511,1539,1544,1575,1613,1651,1655,1697,1701,1730,1767,1815,1844,1882,1911,1949,1979,1984,1990],{"__ignoreMap":202},[271,1263,1264,1266,1268,1271,1273,1276,1278,1280,1282,1285,1287],{"class":273,"line":274},[271,1265,278],{"class":277},[271,1267,307],{"class":299},[271,1269,1270],{"class":281}," test",[271,1272,456],{"class":299},[271,1274,1275],{"class":281}," expect",[271,1277,313],{"class":299},[271,1279,316],{"class":277},[271,1281,289],{"class":288},[271,1283,1284],{"class":292},"@playwright\u002Ftest",[271,1286,296],{"class":288},[271,1288,300],{"class":299},[271,1290,1291],{"class":273,"line":203},[271,1292,331],{"emptyLinePlaceholder":219},[271,1294,1295,1298,1300,1303,1305,1308,1311,1314,1316,1319,1322,1324],{"class":273,"line":328},[271,1296,1297],{"class":281},"test",[271,1299,439],{"class":299},[271,1301,1302],{"class":442},"beforeEach",[271,1304,446],{"class":281},[271,1306,1307],{"class":340},"async",[271,1309,1310],{"class":299}," ({",[271,1312,1313],{"class":575}," page",[271,1315,456],{"class":299},[271,1317,1318],{"class":575}," request",[271,1320,1321],{"class":299}," })",[271,1323,581],{"class":340},[271,1325,352],{"class":299},[271,1327,1328],{"class":273,"line":334},[271,1329,1330],{"class":623},"  \u002F\u002F Added this reset endpoint to erase the board and then naivate to the app at the start of each test run\n",[271,1332,1333,1336,1338,1340,1342,1344,1346,1349,1351,1353],{"class":273,"line":355},[271,1334,1335],{"class":277},"  await",[271,1337,1318],{"class":281},[271,1339,439],{"class":299},[271,1341,443],{"class":442},[271,1343,446],{"class":358},[271,1345,296],{"class":288},[271,1347,1348],{"class":292},"http:\u002F\u002Flocalhost:3000\u002Fapi\u002Freset",[271,1350,296],{"class":288},[271,1352,533],{"class":358},[271,1354,300],{"class":299},[271,1356,1357,1359,1361,1363,1366,1368,1370,1373,1375,1377],{"class":273,"line":372},[271,1358,1335],{"class":277},[271,1360,1313],{"class":281},[271,1362,439],{"class":299},[271,1364,1365],{"class":442},"goto",[271,1367,446],{"class":358},[271,1369,296],{"class":288},[271,1371,1372],{"class":292},"http:\u002F\u002Flocalhost:3000\u002F",[271,1374,296],{"class":288},[271,1376,533],{"class":358},[271,1378,300],{"class":299},[271,1380,1381,1384,1386],{"class":273,"line":389},[271,1382,1383],{"class":299},"}",[271,1385,533],{"class":281},[271,1387,300],{"class":299},[271,1389,1390],{"class":273,"line":395},[271,1391,331],{"emptyLinePlaceholder":219},[271,1393,1394,1396,1398,1401,1403,1405,1407,1409,1411,1413],{"class":273,"line":400},[271,1395,1297],{"class":281},[271,1397,439],{"class":299},[271,1399,1400],{"class":442},"afterEach",[271,1402,446],{"class":281},[271,1404,1307],{"class":340},[271,1406,1310],{"class":299},[271,1408,1318],{"class":575},[271,1410,1321],{"class":299},[271,1412,581],{"class":340},[271,1414,352],{"class":299},[271,1416,1417],{"class":273,"line":417},[271,1418,1419],{"class":623},"  \u002F\u002F Added this explicit reset after each test to erase the board (belt and suspenders with the beforeEach's erase)\n",[271,1421,1422,1424,1426,1428,1430,1432,1434,1436,1438,1440],{"class":273,"line":425},[271,1423,1335],{"class":277},[271,1425,1318],{"class":281},[271,1427,439],{"class":299},[271,1429,443],{"class":442},[271,1431,446],{"class":358},[271,1433,296],{"class":288},[271,1435,1348],{"class":292},[271,1437,296],{"class":288},[271,1439,533],{"class":358},[271,1441,300],{"class":299},[271,1443,1444,1446,1448],{"class":273,"line":467},[271,1445,1383],{"class":299},[271,1447,533],{"class":281},[271,1449,300],{"class":299},[271,1451,1452],{"class":273,"line":472},[271,1453,331],{"emptyLinePlaceholder":219},[271,1455,1456,1458,1460,1463,1465,1467,1469,1472,1474,1476],{"class":273,"line":501},[271,1457,1297],{"class":281},[271,1459,439],{"class":299},[271,1461,1462],{"class":442},"afterAll",[271,1464,446],{"class":281},[271,1466,1307],{"class":340},[271,1468,1310],{"class":299},[271,1470,1471],{"class":575}," browser",[271,1473,1321],{"class":299},[271,1475,581],{"class":340},[271,1477,352],{"class":299},[271,1479,1480],{"class":273,"line":538},[271,1481,1482],{"class":623},"  \u002F\u002F Added to close down the browser after all the tests complete\n",[271,1484,1485,1487,1489,1491,1494,1497],{"class":273,"line":544},[271,1486,1335],{"class":277},[271,1488,1471],{"class":281},[271,1490,439],{"class":299},[271,1492,1493],{"class":442},"close",[271,1495,1496],{"class":358},"()",[271,1498,300],{"class":299},[271,1500,1501,1503,1505],{"class":273,"line":549},[271,1502,1383],{"class":299},[271,1504,533],{"class":281},[271,1506,300],{"class":299},[271,1508,1509],{"class":273,"line":602},[271,1510,331],{"emptyLinePlaceholder":219},[271,1512,1513,1515,1517,1519,1522,1524,1526,1529,1531,1533,1535,1537],{"class":273,"line":620},[271,1514,1297],{"class":442},[271,1516,446],{"class":281},[271,1518,296],{"class":288},[271,1520,1521],{"class":292},"Create a new board with list and cards",[271,1523,296],{"class":288},[271,1525,456],{"class":299},[271,1527,1528],{"class":340}," async",[271,1530,1310],{"class":299},[271,1532,1313],{"class":575},[271,1534,1321],{"class":299},[271,1536,581],{"class":340},[271,1538,352],{"class":299},[271,1540,1541],{"class":273,"line":627},[271,1542,1543],{"class":623},"  \u002F\u002F You'll notice the selector repetition and lack of page objects which we didn't get to during the session \u002F wasn't a primary focus\n",[271,1545,1546,1548,1550,1552,1555,1557,1559,1562,1564,1566,1568,1571,1573],{"class":273,"line":633},[271,1547,1335],{"class":277},[271,1549,1313],{"class":281},[271,1551,439],{"class":299},[271,1553,1554],{"class":442},"getByTestId",[271,1556,446],{"class":358},[271,1558,296],{"class":288},[271,1560,1561],{"class":292},"first-board",[271,1563,296],{"class":288},[271,1565,533],{"class":358},[271,1567,439],{"class":299},[271,1569,1570],{"class":442},"click",[271,1572,1496],{"class":358},[271,1574,300],{"class":299},[271,1576,1577,1579,1581,1583,1585,1587,1589,1591,1593,1595,1597,1600,1602,1604,1607,1609,1611],{"class":273,"line":639},[271,1578,1335],{"class":277},[271,1580,1313],{"class":281},[271,1582,439],{"class":299},[271,1584,1554],{"class":442},[271,1586,446],{"class":358},[271,1588,296],{"class":288},[271,1590,1561],{"class":292},[271,1592,296],{"class":288},[271,1594,533],{"class":358},[271,1596,439],{"class":299},[271,1598,1599],{"class":442},"fill",[271,1601,446],{"class":358},[271,1603,296],{"class":288},[271,1605,1606],{"class":292},"chores",[271,1608,296],{"class":288},[271,1610,533],{"class":358},[271,1612,300],{"class":299},[271,1614,1615,1617,1619,1621,1623,1625,1627,1629,1631,1633,1635,1638,1640,1642,1645,1647,1649],{"class":273,"line":822},[271,1616,1335],{"class":277},[271,1618,1313],{"class":281},[271,1620,439],{"class":299},[271,1622,1554],{"class":442},[271,1624,446],{"class":358},[271,1626,296],{"class":288},[271,1628,1561],{"class":292},[271,1630,296],{"class":288},[271,1632,533],{"class":358},[271,1634,439],{"class":299},[271,1636,1637],{"class":442},"press",[271,1639,446],{"class":358},[271,1641,296],{"class":288},[271,1643,1644],{"class":292},"Enter",[271,1646,296],{"class":288},[271,1648,533],{"class":358},[271,1650,300],{"class":299},[271,1652,1653],{"class":273,"line":827},[271,1654,331],{"emptyLinePlaceholder":219},[271,1656,1657,1660,1662,1665,1667,1669,1671,1673,1675,1677,1680,1682,1685,1687,1689,1691,1693,1695],{"class":273,"line":833},[271,1658,1659],{"class":442},"  expect",[271,1661,446],{"class":358},[271,1663,1664],{"class":281},"page",[271,1666,439],{"class":299},[271,1668,1554],{"class":442},[271,1670,446],{"class":358},[271,1672,296],{"class":288},[271,1674,1561],{"class":292},[271,1676,296],{"class":288},[271,1678,1679],{"class":358},"))",[271,1681,439],{"class":299},[271,1683,1684],{"class":442},"toHaveValue",[271,1686,446],{"class":358},[271,1688,296],{"class":288},[271,1690,1606],{"class":292},[271,1692,296],{"class":288},[271,1694,533],{"class":358},[271,1696,300],{"class":299},[271,1698,1699],{"class":273,"line":838},[271,1700,331],{"emptyLinePlaceholder":219},[271,1702,1703,1705,1707,1709,1711,1713,1715,1718,1720,1722,1724,1726,1728],{"class":273,"line":844},[271,1704,1335],{"class":277},[271,1706,1313],{"class":281},[271,1708,439],{"class":299},[271,1710,1554],{"class":442},[271,1712,446],{"class":358},[271,1714,296],{"class":288},[271,1716,1717],{"class":292},"add-list-input",[271,1719,296],{"class":288},[271,1721,533],{"class":358},[271,1723,439],{"class":299},[271,1725,1570],{"class":442},[271,1727,1496],{"class":358},[271,1729,300],{"class":299},[271,1731,1732,1734,1736,1738,1740,1742,1744,1746,1748,1750,1752,1754,1756,1758,1761,1763,1765],{"class":273,"line":850},[271,1733,1335],{"class":277},[271,1735,1313],{"class":281},[271,1737,439],{"class":299},[271,1739,1554],{"class":442},[271,1741,446],{"class":358},[271,1743,296],{"class":288},[271,1745,1717],{"class":292},[271,1747,296],{"class":288},[271,1749,533],{"class":358},[271,1751,439],{"class":299},[271,1753,1599],{"class":442},[271,1755,446],{"class":358},[271,1757,296],{"class":288},[271,1759,1760],{"class":292},"todo",[271,1762,296],{"class":288},[271,1764,533],{"class":358},[271,1766,300],{"class":299},[271,1768,1769,1771,1773,1775,1778,1780,1782,1785,1787,1789,1791,1794,1796,1798,1801,1803,1805,1807,1809,1811,1813],{"class":273,"line":856},[271,1770,1335],{"class":277},[271,1772,1313],{"class":281},[271,1774,439],{"class":299},[271,1776,1777],{"class":442},"getByRole",[271,1779,446],{"class":358},[271,1781,296],{"class":288},[271,1783,1784],{"class":292},"button",[271,1786,296],{"class":288},[271,1788,456],{"class":299},[271,1790,307],{"class":299},[271,1792,1793],{"class":358}," name",[271,1795,362],{"class":299},[271,1797,289],{"class":288},[271,1799,1800],{"class":292},"Add list",[271,1802,296],{"class":288},[271,1804,313],{"class":299},[271,1806,533],{"class":358},[271,1808,439],{"class":299},[271,1810,1570],{"class":442},[271,1812,1496],{"class":358},[271,1814,300],{"class":299},[271,1816,1817,1819,1821,1823,1825,1827,1829,1832,1834,1836,1838,1840,1842],{"class":273,"line":862},[271,1818,1335],{"class":277},[271,1820,1313],{"class":281},[271,1822,439],{"class":299},[271,1824,1554],{"class":442},[271,1826,446],{"class":358},[271,1828,296],{"class":288},[271,1830,1831],{"class":292},"new-card",[271,1833,296],{"class":288},[271,1835,533],{"class":358},[271,1837,439],{"class":299},[271,1839,1570],{"class":442},[271,1841,1496],{"class":358},[271,1843,300],{"class":299},[271,1845,1846,1848,1850,1852,1854,1856,1858,1861,1863,1865,1867,1869,1871,1873,1876,1878,1880],{"class":273,"line":867},[271,1847,1335],{"class":277},[271,1849,1313],{"class":281},[271,1851,439],{"class":299},[271,1853,1554],{"class":442},[271,1855,446],{"class":358},[271,1857,296],{"class":288},[271,1859,1860],{"class":292},"new-card-input",[271,1862,296],{"class":288},[271,1864,533],{"class":358},[271,1866,439],{"class":299},[271,1868,1599],{"class":442},[271,1870,446],{"class":358},[271,1872,296],{"class":288},[271,1874,1875],{"class":292},"walk the dog",[271,1877,296],{"class":288},[271,1879,533],{"class":358},[271,1881,300],{"class":299},[271,1883,1885,1887,1889,1891,1893,1895,1897,1899,1901,1903,1905,1907,1909],{"class":273,"line":1884},32,[271,1886,1335],{"class":277},[271,1888,1313],{"class":281},[271,1890,439],{"class":299},[271,1892,1554],{"class":442},[271,1894,446],{"class":358},[271,1896,296],{"class":288},[271,1898,1860],{"class":292},[271,1900,296],{"class":288},[271,1902,533],{"class":358},[271,1904,439],{"class":299},[271,1906,1570],{"class":442},[271,1908,1496],{"class":358},[271,1910,300],{"class":299},[271,1912,1914,1916,1918,1920,1922,1924,1926,1928,1930,1932,1934,1936,1938,1940,1943,1945,1947],{"class":273,"line":1913},33,[271,1915,1335],{"class":277},[271,1917,1313],{"class":281},[271,1919,439],{"class":299},[271,1921,1554],{"class":442},[271,1923,446],{"class":358},[271,1925,296],{"class":288},[271,1927,1860],{"class":292},[271,1929,296],{"class":288},[271,1931,533],{"class":358},[271,1933,439],{"class":299},[271,1935,1599],{"class":442},[271,1937,446],{"class":358},[271,1939,296],{"class":288},[271,1941,1942],{"class":292},"mow the lawn",[271,1944,296],{"class":288},[271,1946,533],{"class":358},[271,1948,300],{"class":299},[271,1950,1952,1954,1956,1958,1960,1962,1964,1967,1969,1971,1973,1975,1977],{"class":273,"line":1951},34,[271,1953,1335],{"class":277},[271,1955,1313],{"class":281},[271,1957,439],{"class":299},[271,1959,1554],{"class":442},[271,1961,446],{"class":358},[271,1963,296],{"class":288},[271,1965,1966],{"class":292},"home",[271,1968,296],{"class":288},[271,1970,533],{"class":358},[271,1972,439],{"class":299},[271,1974,1570],{"class":442},[271,1976,1496],{"class":358},[271,1978,300],{"class":299},[271,1980,1982],{"class":273,"line":1981},35,[271,1983,331],{"emptyLinePlaceholder":219},[271,1985,1987],{"class":273,"line":1986},36,[271,1988,1989],{"class":623},"  \u002F\u002F Didn't have a chance to add more assertions, was helping classmates with setup.\n",[271,1991,1993,1995,1997],{"class":273,"line":1992},37,[271,1994,1383],{"class":299},[271,1996,533],{"class":281},[271,1998,300],{"class":299},[11,2000,2001,2002,2005,2006,2009,2010,2013],{},"Test data was the other rough edge. The app resets its entire backend through a ",[86,2003,2004],{},"\u002Fapi\u002Freset"," endpoint, called via Playwright's ",[86,2007,2008],{},"request"," fixture, and Knight was explicit that this was a deliberate, temporary shortcut: \"Remember, this is a tutorial, friends. Don't do this for real... Do not say automation panda told me to drop my whole database as test setup. No, he did not.\" The honest cost of that shortcut showed up immediately: resetting the whole database before every test means tests can't run in parallel, so the class was capped at ",[86,2011,2012],{},"--workers 1"," for the rest of the session. Fixing that properly (per-test data instead of a global wipe) is exactly the kind of thing that's covered in the tutorial's later, unreached chapters, more on that near the end of this article.",[21,2015,2017],{"id":2016},"the-efficient-ai-workflow-playwright-cli-vs-mcp","The Efficient AI Workflow: Playwright CLI vs. MCP",[11,2019,2020],{},"Coming into this session, I'd already absorbed the soundbite that Playwright's CLI is more token-efficient than its MCP server, but nobody had explained why, and I had a more basic confusion sitting underneath that one: the CLI is just terminal commands, so in what sense is that even \"AI\"? Knight's session got me most of the way to an answer. It didn't fully click until I went and read more on my own afterward.",[11,2022,2023],{},"Once the manual test was working, Knight pivoted to AI, with an important framing up front: \"Playwright doesn't bring its own model, it doesn't bring its own magic. Basically what it does is it brings tooling to integrate into existing AI coding agents.\" You still need Claude, Cursor, Copilot, or Codex. Playwright gives that agent two different ways to actually drive a browser.",[11,2025,2026,2029,2030,117,2033,2036],{},[40,2027,2028],{},"MCP"," (Model Context Protocol) exposes structured tools like ",[86,2031,2032],{},"browser_navigate",[86,2034,2035],{},"browser_snapshot"," to your coding agent. It works well, and it's expensive. Knight's framing of why, in full:",[1204,2038,2039],{},[11,2040,2041],{},"\"There's a problem with MCP. Does anybody know the problem with MCP? Burns a lot of tokens. It burns a heckin' ton of tokens... Intelligence is a utility. You pay a power bill, you pay a water bill. Guess what we're all paying for next? An intelligence bill.\"",[11,2043,2044],{},"The joke that opened this article followed directly: a junior developer who ran up a $5,000 month using MCP without understanding the cost. The mechanism, explained later in the session, isn't about which model you use, it's that MCP's tool schemas and structured page snapshots eat far more context window per step than a plain terminal command does, which forces more turns, which burns more tokens.",[11,2046,2047,2050],{},[40,2048,2049],{},"Playwright's CLI"," does the same browser-driving job as MCP, as plain terminal commands instead of structured tool calls, and according to Knight, \"uses a tenth of the tokens.\" His actual decision rule, given directly in response to \"why would you ever use MCP if the CLI is so much cheaper\":",[1204,2052,2053],{},[11,2054,2055],{},"\"The CLI is really good if you are doing the workflow that we are doing, for test developers, for grinding out some code, with coding agents CLI is better. But let's say that you wanted a more agentic workflow that wasn't you coding. Let's say you had to use Playwright as a browser automation tool in some way, writing a web scraper or web browser. In those cases the MCP is going to be better than the CLI. Because the MCP can be hosted on a network that you can reach out to it back and forth. CLI is all local to your machine.\"",[11,2057,2058,2059,2062,2063,478,2066,672,2068,2071,2072,2075,2076,2079,2080,2083,2084,2087],{},"Here's the part that actually answered both of my questions, the AI-or-not question and the why-tokens question, together. Both MCP and the CLI are AI-driven, in both cases the coding agent itself is deciding what to do and reading the result back. ",[40,2060,2061],{},"The difference is just what vocabulary it uses to act."," MCP issues ",[40,2064,2065],{},"structured tool calls",[86,2067,2032],{},[86,2069,2070],{},"browser_click",") over a protocol built on JSON-RPC, so the call and its ",[40,2073,2074],{},"full response travel through the model's context every time",". The ",[40,2077,2078],{},"CLI"," has the agent run ",[40,2081,2082],{},"literal shell commands"," against itself, something like ",[86,2085,2086],{},"playwright-cli click e21",", the same way it would run any other terminal command in a coding session.",[11,2089,2090,2091,2094,2095,2098],{},"That's also where the token savings actually come from. ",[40,2092,2093],{},"MCP has to keep the page's structure resident in the session's context for as long as the agent is working with it."," The CLI's skills are markdown files sitting on disk, ",[40,2096,2097],{},"read in only when something needs them",", then left there. One holds everything it might need in memory the whole time. The other fetches what it needs and sets it back down.",[11,2100,2101],{},"That also sharpens Knight's own rule (local machine versus network-hosted) into something more concrete. The CLI needs a real terminal, a filesystem, and the ability to spawn its own processes, exactly what you have during local development, and exactly what you don't have everywhere else. MCP doesn't need any of that, which is why it's the better fit in more locked-down or remote contexts: AI-assisted CI failure triage running inside a pipeline with no terminal session attached, for instance, or a low-code product where an agent runs server-side and a non-technical user just describes a test case in plain English, with no shell ever exposed to that agent at all.",[11,2103,2104],{},"Three more habits from the session genuinely earn their place under an efficiency banner, each backed by Knight's own stated reasoning rather than just a vibe:",[889,2106,2107,2113,2119],{},[666,2108,2109,2112],{},[40,2110,2111],{},"Skills over re-explaining."," Installing CLI skills (markdown files that teach the agent what commands exist) means you're not \"pasting huge help text into every prompt.\" It's explicitly part of why the CLI uses fewer tokens than MCP in his own comparison, skills are loaded only when needed instead of being baked into every tool call.",[666,2114,2115,2118],{},[40,2116,2117],{},"Save state to markdown instead of letting it evaporate."," When Knight had the agent save a generated test plan to a file rather than leaving it in chat, his reasoning doubled as a genuinely good explanation of why: \"Your context window is only so big... if I didn't save my test plan in this markdown file, I'd have to make it regenerate the test plan again. That sucks.\" He compared it to saving progress in an old Super Nintendo game before your context window (or your save file) gets wiped.",[666,2120,2121,2124],{},[40,2122,2123],{},"Inside-out test generation."," Rather than guessing a locator, running the test, watching it crash, and correcting, Playwright's CLI and MCP tooling let the agent build a session step by step, discovering real locators as it goes. \"That usually leads to very short loops, not having to repeat a lot of loops.\" It's a real efficiency argument and it's specific to how Playwright's own tooling is built, not a generic prompting tip.",[11,2126,2127,2128,2130],{},"Knight also argued that AI-assisted test generation cuts maintenance cost, since a broken locator can trigger \"a little bit of agentic maintenance... a healing loop, commit that fix back in.\" I think that may be oversold, or at least dependent on your engineering practices. Maybe this has more of an ROI on pages undergoing rapid prototyping or constant redesigns, but outside of those scenarios, I find locators remain relatively stable once they're set up in a page object model, assuming you're using ID attributes (if they aren't randomly generated) or something like ",[86,2129,1249],{},". Playwright also has modern locator strategies that preclude a lot of the problems people used to get themselves into with XPath or text-based locators.",[11,2132,2133,2134,2137],{},"The live demos backed up the rest. One had the agent open the app, create a board, add a list, and invent three plausible user stories from a single plain-English prompt, no locators, no Playwright code written by hand. Another had it explore the app, propose a test plan, save that plan to a markdown file, and then generate full ",[86,2135,2136],{},"*.spec.ts"," files from it, self-healing failures as it ran, ending at 74 passed and 1 skipped. Knight's own retrospective on that second demo is worth keeping, because it's a caution about scope, not about cost: \"I would not recommend doing what I showed here, big asks. I would recommend many small asks.\" Review the output like a teammate's pull request, not like a vending machine.",[11,2139,2140,2141,2144],{},"I liked that Knight acknowledged the reality of the quality of test you get straight from AI with a prompt like this. The generated code was unoptimized and raw, similar to what the earlier codegen example created when we recorded our manual steps through the application to build a test case. You ",[32,2142,2143],{},"would not"," want to use these tests in your final test suite as-is:",[1204,2146,2147],{},[11,2148,2149],{},"\"There's no page objects here. There's no real library abstraction... these names aren't great.\"",[11,2151,2152],{},"Here's the clean version of the prompt he used, taken from his tutorial notes rather than transcribed live:",[262,2154,2158],{"className":2155,"code":2156,"language":2157,"meta":202,"style":202},"language-txt shiki shiki-themes material-theme-lighter github-light-high-contrast github-dark-high-contrast","Using playwright-cli, open http:\u002F\u002Flocalhost:3000\u002F, reset data if needed via API, then walk through\nthe \"create board → add list → add cards → go home\" flow. Use snapshots to pick stable locators.\nThen add a new Playwright TypeScript test under `tests\u002F` that matches our existing style:\n`test.beforeAll` or `beforeEach` for \u002Fapi\u002Freset, clear test name, getByRole\u002FgetByPlaceholder,\nand expect assertions. Reuse patterns from our existing trello spec if present.\n","txt",[86,2159,2160,2165,2170,2175,2180],{"__ignoreMap":202},[271,2161,2162],{"class":273,"line":274},[271,2163,2164],{},"Using playwright-cli, open http:\u002F\u002Flocalhost:3000\u002F, reset data if needed via API, then walk through\n",[271,2166,2167],{"class":273,"line":203},[271,2168,2169],{},"the \"create board → add list → add cards → go home\" flow. Use snapshots to pick stable locators.\n",[271,2171,2172],{"class":273,"line":328},[271,2173,2174],{},"Then add a new Playwright TypeScript test under `tests\u002F` that matches our existing style:\n",[271,2176,2177],{"class":273,"line":334},[271,2178,2179],{},"`test.beforeAll` or `beforeEach` for \u002Fapi\u002Freset, clear test name, getByRole\u002FgetByPlaceholder,\n",[271,2181,2182],{"class":273,"line":355},[271,2183,2184],{},"and expect assertions. Reuse patterns from our existing trello spec if present.\n",[11,2186,2187],{},"With more deliberate prompt engineering, Claude could have produced a cleaner first draft. But the rawer version is what actually demonstrated the accelerated-scaffolding benefit, and it set up a natural case for why prompt engineering matters in the first place:",[1204,2189,2190],{},[11,2191,2192],{},"\"If I were to do full context engineering, I would have my rules for Playwright tests, and I would say things like, use page object model.\"",[11,2194,2195],{},"Left on its own, a prompt like this gets you a fast, working first draft, not a finished one. The written version of this tutorial has a fair name for that tradeoff: \"accelerated scaffolding, not a substitute for judgment.\" Same deal as raw codegen output earlier in this piece, a working draft far faster than typing it by hand, just not something you'd commit as-is.",[21,2197,2199],{"id":2198},"why-locators-still-beat-computer-vision","Why Locators Still Beat Computer Vision",[11,2201,2202,2203,2206],{},"The day before Knight's session, Dionny Santiago's StarEast 2026 tutorial made close to the opposite argument about how AI should interact with a web page. I wrote about ",[1116,2204,2205],{"href":1118},"his case for AI vision testing over brittle CSS and XPath selectors"," in more detail, but the short version is direct: \"Computer vision is the evolution of the CSS selectors and the XPath selectors,\" reading a page the way a person does instead of hunting for a class name or test ID. Knight never mentioned Santiago's session, and might not have even been aware of it. An audience member raised a version of it anyway, describing tools that skip \"element work\" entirely in favor of a vision-based approach, and Knight disagreed without hesitating:",[1204,2208,2209],{},[11,2210,2211],{},"\"I disagree with that. Because even with AI superpowers, image matching is still going to be expensive. Whereas locators are very cheap and quick.\"",[11,2213,2214],{},"He built a full historical case for why, the kind of argument worth quoting at length because it's the most fully-reasoned claim in the entire session. The short version: programming has only ever moved toward higher abstraction (assembly to Fortran and C to Java, Python, and TypeScript), because each higher layer let us trust the layer below it without reading it. His extension of that idea to AI:",[1204,2216,2217],{},[11,2218,2219],{},"\"AI is the new compiler. Source code in TypeScript and Java and Python is the new assembly code... It will not be much longer that we still have to dance down at those levels because it's going to get so good. We still have to today because it's not as good yet.\"",[11,2221,2222],{},"Then the part that actually settles the locators-versus-vision question, mapping compiled-versus-interpreted execution onto test automation directly:",[1204,2224,2225],{},[11,2226,2227],{},"\"What I showed you before with, hey, let's just explore the app with Playwright CLI and just let it go and not record anything, that was equivalent to an interpreter. That's very slow. That's token heavy. Your image matching thing when it comes to test execution is also going to be inherently slow. Always, because if you're looking at something, you have to image match in the moment... that grinding can never not be done in that kind of model. So that's why I don't think the image matching of locators is ever really going to happen.\"",[11,2229,2230,2231,2234],{},"The distinction matters for accuracy: this is about test ",[32,2232,2233],{},"execution",", how the automation decides where to click while a test runs, not about visual regression tools that diff screenshots to catch rendering bugs. Knight never argues against that second category at all. Within the category he's actually addressing, his case is the more convincing one between these two tutorials. Generating a locator-based script costs tokens once. Running it costs almost nothing, over and over. Vision-based execution pays the image-matching cost every single run, forever, no matter how good the underlying model gets. That's a structural cost difference, not a current-capability gap that better models eventually close.",[21,2236,2238],{"id":2237},"test-pyramids-skyscrapers-and-the-gap-nobody-closed","Test Pyramids, Skyscrapers, and the Gap Nobody Closed",[11,2240,2241],{},"Back to the line I deferred earlier. Here's Knight's full skyscraper pivot, verbatim:",[1204,2243,2244],{},[11,2245,2246],{},"\"Today we don't build pyramids anymore. We build skyscrapers. Look up to testing skyscrapers. We need to reframe what we think of for testing in modern times because the world has changed since that previous mental model was created.\"",[11,2248,2249],{},"It's a good line, and it's worth being precise about what it actually claims. Knight never says UI tests are better than unit tests, the literal claim is narrower: \"UI tests are not bad. All tests are good because they mitigate different kinds of risks.\" That's an argument against rigid proportions, not a reordering of the hierarchy. He also never builds out the metaphor itself, there's no mapping of \"floors\" to test types anywhere in the session, the slides, or the written tutorial chapters. The skyscraper is a mood, not a blueprint.",[11,2251,2252],{},"His actual defense for ditching the pyramid's bias against UI tests is the tooling argument from earlier in this piece: Playwright's architecture fixed the execution speed and flakiness problems that gave UI tests their bad reputation. That's a real, demonstrated improvement. What it doesn't touch is the part of the pyramid's logic that was never about execution speed at all. A unit test calling a function in-process will always be faster than even the fastest browser context, that's a difference in kind, not in tooling. Unit tests also stay directly traceable to source lines and branches in a way browser-driven tests can't. Neither Playwright nor the AI tooling covered in this session does anything about that gap, and it never came up once in either half of the tutorial.",[11,2254,2255,2256,2259],{},"The AI-assisted authoring material from the previous section actually extends Knight's case further than he extended it himself, just not far enough to close that gap. If AI assistance genuinely lowers the cost of writing and maintaining E2E tests (and the token and time savings shown live back that up, even if the maintenance claim is softer), that addresses the ",[32,2257,2258],{},"other"," half of the pyramid's original justification, the cost of producing and keeping E2E tests working, which his own tooling argument never reached. So the fuller, more honest position: the case for de-emphasizing strict pyramid proportions is stronger than Knight made it sound, once you add AI-assisted authoring on top of Playwright's execution-speed fix. It's still not a full rebuttal of the pyramid, because the one gap that was never about tooling in the first place is still sitting there untouched.",[11,2261,2262],{},"Page objects, splitting one big test into independent behavior tests, and the parallel-safe test data strategy that actually fixes the \"drop the whole database\" shortcut from earlier in this piece are all covered in Knight's written tutorial chapters, just not in the room.",[21,2264,2266],{"id":2265},"my-takeaways-on-playwright-and-ai-testing","My Takeaways on Playwright and AI Testing",[11,2268,2269],{},"A few things I'm taking back with me:",[889,2271,2272,2278,2284,2290],{},[666,2273,2274,2277],{},[40,2275,2276],{},"Default to the CLI over MCP for routine test-development work."," Reach for MCP only when local terminal and filesystem access isn't an option in the first place, not just because it feels more capable.",[666,2279,2280,2283],{},[40,2281,2282],{},"Treat AI-generated tests as scaffolding, not a finished product."," The first draft comes out raw, the same as old-school codegen output, so the cleanup step (page objects, naming, structure) isn't optional, it's the rest of the job.",[666,2285,2286,2289],{},[40,2287,2288],{},"Locator-based testing wins for driving test execution, and I don't expect that to change as models improve."," The cost gap is structural, not a capability gap that better models eventually close. (Visual regression testing is a different problem, and a fair use case for vision-based tools.)",[666,2291,2292,2295],{},[40,2293,2294],{},"Playwright and AI assistance narrow the case for the old Testing Pyramid, but they don't close it."," Knight's argument only ever answered the execution-speed half of the pyramid's old bias against UI tests; AI-assisted authoring answers some of the authoring-cost half too. Neither touches the one gap that was never about tooling: a unit test will always run faster and trace more directly to source than any browser-driven test.",[11,2297,2298,2299,2303],{},"If testing AI systems themselves (not just using AI to write tests) is more your focus right now, ",[1116,2300,2302],{"href":2301},"\u002Fsoftware-testing\u002Ftest-automation\u002Fhow-to-test-ai-chatbots-and-agents","how I approached evals on a real agentic chatbot engagement"," is a related read you may find useful.",[198,2305],{":items":2306},"[\"\u002Fsoftware-testing\u002Ftest-automation\u002Fstareast-2026-getting-started-ai-driven-automation\",\"\u002Fsoftware-testing\u002Ftest-automation\u002Fstareast-2026-getting-dirty-ai-testing\",\"\u002Fsoftware-testing\u002Ftest-automation\u002Fplaywright-accessibility-testing-axe-lighthouse-limitations\",\"\u002Fsoftware-testing\u002Ftest-automation\u002Fstareast-2026-prompt-engineering-techniques\"]",[1137,2308,2309],{},"html pre.shiki code .sZTni,html code.shiki .sZTni{--shiki-light:#39ADB5;--shiki-light-font-style:italic;--shiki-default:#A0111F;--shiki-default-font-style:inherit;--shiki-dark:#FF9492;--shiki-dark-font-style:inherit}html pre.shiki code .sPJuK,html code.shiki .sPJuK{--shiki-light:#39ADB5;--shiki-default:#0E1116;--shiki-dark:#F0F3F6}html pre.shiki code .sZ-rw,html code.shiki .sZ-rw{--shiki-light:#90A4AE;--shiki-default:#0E1116;--shiki-dark:#F0F3F6}html pre.shiki code .sZi47,html code.shiki .sZi47{--shiki-light:#39ADB5;--shiki-default:#032563;--shiki-dark:#ADDCFF}html pre.shiki code .srGNg,html code.shiki .srGNg{--shiki-light:#91B859;--shiki-default:#032563;--shiki-dark:#ADDCFF}html pre.shiki code .sb1SK,html code.shiki .sb1SK{--shiki-light:#6182B8;--shiki-default:#622CBC;--shiki-dark:#DBB7FF}html pre.shiki code .stWsX,html code.shiki .stWsX{--shiki-light:#9C3EDA;--shiki-default:#A0111F;--shiki-dark:#FF9492}html pre.shiki code .s2xgV,html code.shiki .s2xgV{--shiki-light:#90A4AE;--shiki-light-font-style:italic;--shiki-default:#702C00;--shiki-default-font-style:inherit;--shiki-dark:#FFB757;--shiki-dark-font-style:inherit}html pre.shiki code .s_gjE,html code.shiki .s_gjE{--shiki-light:#90A4AE;--shiki-light-font-style:italic;--shiki-default:#66707B;--shiki-default-font-style:inherit;--shiki-dark:#BDC4CC;--shiki-dark-font-style:inherit}html pre.shiki code .sq0XF,html code.shiki .sq0XF{--shiki-light:#E53935;--shiki-default:#0E1116;--shiki-dark:#F0F3F6}html .light .shiki span{color:var(--shiki-light);background:var(--shiki-light-bg);font-style:var(--shiki-light-font-style);font-weight:var(--shiki-light-font-weight);text-decoration:var(--shiki-light-text-decoration)}html.light .shiki span{color:var(--shiki-light);background:var(--shiki-light-bg);font-style:var(--shiki-light-font-style);font-weight:var(--shiki-light-font-weight);text-decoration:var(--shiki-light-text-decoration)}html .default .shiki span{color:var(--shiki-default);background:var(--shiki-default-bg);font-style:var(--shiki-default-font-style);font-weight:var(--shiki-default-font-weight);text-decoration:var(--shiki-default-text-decoration)}html .shiki span{color:var(--shiki-default);background:var(--shiki-default-bg);font-style:var(--shiki-default-font-style);font-weight:var(--shiki-default-font-weight);text-decoration:var(--shiki-default-text-decoration)}html .dark .shiki span{color:var(--shiki-dark);background:var(--shiki-dark-bg);font-style:var(--shiki-dark-font-style);font-weight:var(--shiki-dark-font-weight);text-decoration:var(--shiki-dark-text-decoration)}html.dark .shiki span{color:var(--shiki-dark);background:var(--shiki-dark-bg);font-style:var(--shiki-dark-font-style);font-weight:var(--shiki-dark-font-weight);text-decoration:var(--shiki-dark-text-decoration)}",{"title":202,"searchDepth":203,"depth":203,"links":2311},[2312,2313,2314,2315,2316,2317],{"id":1195,"depth":203,"text":1196},{"id":1229,"depth":203,"text":1230},{"id":2016,"depth":203,"text":2017},{"id":2198,"depth":203,"text":2199},{"id":2237,"depth":203,"text":2238},{"id":2265,"depth":203,"text":2266},"\u002Fimages\u002Fposts\u002Fstareast-2026-playwright-ai-cost-efficient-testing\u002Fstareast-2026-playwright-ai-cost-efficient-testing-cover.webp","2026-06-22","Playwright's MCP server can burn your AI budget fast. StarEast 2026 lessons on efficient AI testing, and why locators still beat computer vision.",{},{"title":1158,"description":2320},"software-testing\u002Ftest-automation\u002Fstareast-2026-playwright-ai-cost-efficient-testing","joMPpMIfdSO7ZR8LztFrojUyxQeGStMo6mi6RIG0EI4",{"id":2326,"title":2327,"bmcUsername":6,"body":2328,"cover":3125,"date":3126,"description":3127,"draft":216,"extension":217,"features":6,"githubRepo":6,"headline":6,"highlight":6,"icon":6,"meta":3128,"navigation":219,"npmPackage":6,"order":6,"path":3129,"seo":3130,"stem":3131,"__hash__":3132},"content\u002Fsoftware-testing\u002Ftest-automation\u002Fai-test-automation-pitfalls-vs-user-error.md","AI in Test Automation: Real Limitations vs. User Error",{"type":8,"value":2329,"toc":3113},[2330,2333,2336,2339,2343,2347,2354,2357,2375,2378,2383,2397,2400,2405,2420,2423,2532,2626,2633,2709,2712,2715,2726,2729,2811,2814,2816,2820,2823,2833,2836,2846,2852,2872,2874,2878,2881,2884,2887,2898,2901,2904,2918,2920,2924,2927,2930,2933,2960,2963,2965,2969,2971,2974,2977,2997,3000,3003,3005,3009,3011,3014,3017,3025,3028,3033,3044,3049,3060,3062,3066,3068,3071,3074,3081,3092,3095,3097,3101,3104,3107,3110],[11,2331,2332],{},"Spend enough time in testing communities and you'll hear the same AI complaints on repeat. Some of them are legitimate. Some of them are skill gaps dressed up as tool limitations. And some land somewhere in the middle — real friction, but solvable with the right setup.",[11,2334,2335],{},"This article goes through the most common ones and gives each a verdict — with practical mitigations for the ones that hold up.",[2337,2338],"hr",{},[21,2340,2342],{"id":2341},"ai-will-rewrite-your-assertions-to-make-tests-pass","\"AI Will Rewrite Your Assertions to Make Tests Pass\"",[2344,2345],"verdict-badge",{"type":2346},"true",[11,2348,2349,2350,2353],{},"This is the most dangerous pitfall in AI-assisted testing and the one that gets the least attention in vendor demos. AI models are trained to produce working code. A test that passes ",[32,2351,2352],{},"is"," working code to the model — so when a test is failing, the path of least resistance is to make it pass, not to understand why it was failing.",[11,2355,2356],{},"In practice this means AI will silently:",[889,2358,2359,2369,2372],{},[666,2360,2361,2362,2365,2366,533],{},"Weaken assertions (",[86,2363,2364],{},"toEqual"," → ",[86,2367,2368],{},"toBeTruthy",[666,2370,2371],{},"Add conditional logic that bypasses the failing case",[666,2373,2374],{},"Change expected values to match whatever the app currently returns",[11,2376,2377],{},"The result is a green test suite that no longer tests what it was supposed to test. And because the change looks like a reasonable fix in isolation, it often gets through code review unnoticed.",[11,2379,2380],{},[40,2381,2382],{},"Mitigation:",[889,2384,2385,2388,2391,2394],{},[666,2386,2387],{},"Never let AI resolve a failing test without human review of the assertion specifically",[666,2389,2390],{},"Treat assertion changes in AI-generated diffs with extra scrutiny — they're the most likely place the model took a shortcut",[666,2392,2393],{},"Understand what a good assertion looks like before using AI to write them. If you can't evaluate the output, you can't catch when it's wrong",[666,2395,2396],{},"Explicitly tell the AI to expect that it may find defects and what it should do when one is detected (e.g. \"Leave the test failing, log a defect, and set the @Disabled annotation with comment tying back to the defect ticket\")",[11,2398,2399],{},"This is the clearest argument for why AI in testing raises the ceiling for experienced practitioners and lowers it for those without fundamentals. The model will confidently write you a useless test without human-in-the-loop auditing and supervision.",[2401,2402,2404],"h3",{"id":2403},"characterization-tests-and-the-case-of-the-silent-rewrite","Characterization Tests and the Case of the Silent Rewrite",[11,2406,2407,2408,2419],{},"When using Claude to modernize legacy functions, I asked it to establish test coverage for the existing code before beginning the refactor — a characterization test workflow. Without explicit instruction, it would start reasoning out loud in the terminal: ",[32,2409,2410,2411,2414,2415,2418],{},"\"The test is failing — I just need to change the expected value from ",[86,2412,2413],{},"foo"," to ",[86,2416,2417],{},"bar",".\""," Left unchecked, it would have written tests that codified the defect as the expected output. The fix was explicit instruction: assume the legacy code may have bugs, leave failing tests in a failed state, and we'll rerun them after the refactor to validate the improvements.",[11,2421,2422],{},"Here's a representative example of what that assertion rewrite looks like in practice:",[262,2424,2427],{"className":1256,"code":2425,"filename":2426,"language":1259,"meta":202,"style":202},"\u002F\u002F Legacy function — bug: discount applies to subtotal before tax (should be after)\nfunction calculateOrderTotal(subtotal, taxRate, discountPct) {\n  const discount = subtotal * (discountPct \u002F 100);\n  const tax = subtotal * taxRate;\n  return subtotal + tax - discount;\n}\n","shoppingCart.ts",[86,2428,2429,2434,2461,2492,2509,2528],{"__ignoreMap":202},[271,2430,2431],{"class":273,"line":274},[271,2432,2433],{"class":623},"\u002F\u002F Legacy function — bug: discount applies to subtotal before tax (should be after)\n",[271,2435,2436,2439,2442,2444,2447,2449,2452,2454,2457,2459],{"class":273,"line":203},[271,2437,2438],{"class":340},"function",[271,2440,2441],{"class":442}," calculateOrderTotal",[271,2443,446],{"class":299},[271,2445,2446],{"class":575},"subtotal",[271,2448,456],{"class":299},[271,2450,2451],{"class":575}," taxRate",[271,2453,456],{"class":299},[271,2455,2456],{"class":575}," discountPct",[271,2458,533],{"class":299},[271,2460,352],{"class":299},[271,2462,2463,2466,2469,2471,2474,2477,2479,2482,2485,2488,2490],{"class":273,"line":328},[271,2464,2465],{"class":340},"  const",[271,2467,2468],{"class":344}," discount",[271,2470,349],{"class":348},[271,2472,2473],{"class":281}," subtotal",[271,2475,2476],{"class":348}," *",[271,2478,478],{"class":358},[271,2480,2481],{"class":281},"discountPct",[271,2483,2484],{"class":348}," \u002F",[271,2486,2487],{"class":365}," 100",[271,2489,533],{"class":358},[271,2491,300],{"class":299},[271,2493,2494,2496,2499,2501,2503,2505,2507],{"class":273,"line":334},[271,2495,2465],{"class":340},[271,2497,2498],{"class":344}," tax",[271,2500,349],{"class":348},[271,2502,2473],{"class":281},[271,2504,2476],{"class":348},[271,2506,2451],{"class":281},[271,2508,300],{"class":299},[271,2510,2511,2514,2516,2519,2521,2524,2526],{"class":273,"line":355},[271,2512,2513],{"class":277},"  return",[271,2515,2473],{"class":281},[271,2517,2518],{"class":348}," +",[271,2520,2498],{"class":281},[271,2522,2523],{"class":348}," -",[271,2525,2468],{"class":281},[271,2527,300],{"class":299},[271,2529,2530],{"class":273,"line":372},[271,2531,642],{"class":299},[262,2533,2536],{"className":1256,"code":2534,"filename":2535,"language":1259,"meta":202,"style":202},"\u002F\u002F Test written to document correct behavior before refactor\ntest('discount should apply to post-tax total', () => {\n  \u002F\u002F subtotal: 100, 10% tax, 10% discount\n  \u002F\u002F correct: (100 + 10) * 0.90 = 99\n  expect(calculateOrderTotal(100, 0.10, 10)).toBe(99); \u002F\u002F FAILS — returns 100\n});\n","shoppingCart.test.ts",[86,2537,2538,2543,2564,2569,2574,2618],{"__ignoreMap":202},[271,2539,2540],{"class":273,"line":274},[271,2541,2542],{"class":623},"\u002F\u002F Test written to document correct behavior before refactor\n",[271,2544,2545,2547,2549,2551,2554,2556,2558,2560,2562],{"class":273,"line":203},[271,2546,1297],{"class":442},[271,2548,446],{"class":281},[271,2550,296],{"class":288},[271,2552,2553],{"class":292},"discount should apply to post-tax total",[271,2555,296],{"class":288},[271,2557,456],{"class":299},[271,2559,412],{"class":299},[271,2561,581],{"class":340},[271,2563,352],{"class":299},[271,2565,2566],{"class":273,"line":328},[271,2567,2568],{"class":623},"  \u002F\u002F subtotal: 100, 10% tax, 10% discount\n",[271,2570,2571],{"class":273,"line":334},[271,2572,2573],{"class":623},"  \u002F\u002F correct: (100 + 10) * 0.90 = 99\n",[271,2575,2576,2578,2580,2583,2585,2588,2590,2593,2595,2598,2600,2602,2605,2607,2610,2612,2615],{"class":273,"line":355},[271,2577,1659],{"class":442},[271,2579,446],{"class":358},[271,2581,2582],{"class":442},"calculateOrderTotal",[271,2584,446],{"class":358},[271,2586,2587],{"class":365},"100",[271,2589,456],{"class":299},[271,2591,2592],{"class":365}," 0.10",[271,2594,456],{"class":299},[271,2596,2597],{"class":365}," 10",[271,2599,1679],{"class":358},[271,2601,439],{"class":299},[271,2603,2604],{"class":442},"toBe",[271,2606,446],{"class":358},[271,2608,2609],{"class":365},"99",[271,2611,533],{"class":358},[271,2613,2614],{"class":299},";",[271,2616,2617],{"class":623}," \u002F\u002F FAILS — returns 100\n",[271,2619,2620,2622,2624],{"class":273,"line":372},[271,2621,1383],{"class":299},[271,2623,533],{"class":281},[271,2625,300],{"class":299},[11,2627,2628,2629,2632],{},"AI sees the failure and silently \"fixes\" it — updating both the assertion ",[32,2630,2631],{},"and"," the test name to stay internally consistent:",[262,2634,2636],{"className":1256,"code":2635,"filename":2535,"language":1259,"meta":202,"style":202},"\u002F\u002F What AI changed it to — test now passes, bug is invisible\ntest('discount should apply to pre-tax total', () => {\n  expect(calculateOrderTotal(100, 0.10, 10)).toBe(100); \u002F\u002F ← matches buggy output\n});\n",[86,2637,2638,2643,2664,2701],{"__ignoreMap":202},[271,2639,2640],{"class":273,"line":274},[271,2641,2642],{"class":623},"\u002F\u002F What AI changed it to — test now passes, bug is invisible\n",[271,2644,2645,2647,2649,2651,2654,2656,2658,2660,2662],{"class":273,"line":203},[271,2646,1297],{"class":442},[271,2648,446],{"class":281},[271,2650,296],{"class":288},[271,2652,2653],{"class":292},"discount should apply to pre-tax total",[271,2655,296],{"class":288},[271,2657,456],{"class":299},[271,2659,412],{"class":299},[271,2661,581],{"class":340},[271,2663,352],{"class":299},[271,2665,2666,2668,2670,2672,2674,2676,2678,2680,2682,2684,2686,2688,2690,2692,2694,2696,2698],{"class":273,"line":328},[271,2667,1659],{"class":442},[271,2669,446],{"class":358},[271,2671,2582],{"class":442},[271,2673,446],{"class":358},[271,2675,2587],{"class":365},[271,2677,456],{"class":299},[271,2679,2592],{"class":365},[271,2681,456],{"class":299},[271,2683,2597],{"class":365},[271,2685,1679],{"class":358},[271,2687,439],{"class":299},[271,2689,2604],{"class":442},[271,2691,446],{"class":358},[271,2693,2587],{"class":365},[271,2695,533],{"class":358},[271,2697,2614],{"class":299},[271,2699,2700],{"class":623}," \u002F\u002F ← matches buggy output\n",[271,2702,2703,2705,2707],{"class":273,"line":334},[271,2704,1383],{"class":299},[271,2706,533],{"class":281},[271,2708,300],{"class":299},[11,2710,2711],{},"The model updates both the assertion and the test name to stay internally consistent, so the result looks like a deliberate design decision, not a shortcut. Nothing in the diff signals that a bug just became the spec.",[11,2713,2714],{},"The mitigation was giving Claude explicit rules for exactly this scenario before it started writing any tests:",[1204,2716,2717],{},[11,2718,2719],{},[32,2720,2721,2722,2725],{},"\"These functions have no existing test coverage, so we may discover bugs as we add characterization tests. If a test fails, do not rewrite it to pass. Instead: leave the assertion as-is documenting the expected correct behavior, disable the test with ",[86,2723,2724],{},"test.skip",", and add a comment with the defect ticket ID and a TODO to re-enable once it's resolved.\"",[11,2727,2728],{},"With that instruction in place, the same failing scenario produces this instead:",[262,2730,2732],{"className":1256,"code":2731,"filename":2535,"language":1259,"meta":202,"style":202},"\u002F\u002F TODO: Re-enable once resolved — see WEB-1234\n\u002F\u002F BUG: Discount is applied to pre-tax subtotal instead of post-tax total\ntest.skip('discount should apply to post-tax total', () => {\n  expect(calculateOrderTotal(100, 0.10, 10)).toBe(99);\n});\n",[86,2733,2734,2739,2744,2769,2803],{"__ignoreMap":202},[271,2735,2736],{"class":273,"line":274},[271,2737,2738],{"class":623},"\u002F\u002F TODO: Re-enable once resolved — see WEB-1234\n",[271,2740,2741],{"class":273,"line":203},[271,2742,2743],{"class":623},"\u002F\u002F BUG: Discount is applied to pre-tax subtotal instead of post-tax total\n",[271,2745,2746,2748,2750,2753,2755,2757,2759,2761,2763,2765,2767],{"class":273,"line":328},[271,2747,1297],{"class":281},[271,2749,439],{"class":299},[271,2751,2752],{"class":442},"skip",[271,2754,446],{"class":281},[271,2756,296],{"class":288},[271,2758,2553],{"class":292},[271,2760,296],{"class":288},[271,2762,456],{"class":299},[271,2764,412],{"class":299},[271,2766,581],{"class":340},[271,2768,352],{"class":299},[271,2770,2771,2773,2775,2777,2779,2781,2783,2785,2787,2789,2791,2793,2795,2797,2799,2801],{"class":273,"line":334},[271,2772,1659],{"class":442},[271,2774,446],{"class":358},[271,2776,2582],{"class":442},[271,2778,446],{"class":358},[271,2780,2587],{"class":365},[271,2782,456],{"class":299},[271,2784,2592],{"class":365},[271,2786,456],{"class":299},[271,2788,2597],{"class":365},[271,2790,1679],{"class":358},[271,2792,439],{"class":299},[271,2794,2604],{"class":442},[271,2796,446],{"class":358},[271,2798,2609],{"class":365},[271,2800,533],{"class":358},[271,2802,300],{"class":299},[271,2804,2805,2807,2809],{"class":273,"line":355},[271,2806,1383],{"class":299},[271,2808,533],{"class":281},[271,2810,300],{"class":299},[11,2812,2813],{},"The test documents the intended behavior, the skip keeps the suite green without hiding the problem, and the ticket reference means it isn't silently forgotten — which is the whole point of a characterization test suite.",[2337,2815],{},[21,2817,2819],{"id":2818},"ai-testing-tools-cost-too-much-to-run-at-scale","\"AI Testing Tools Cost Too Much to Run at Scale\"",[2344,2821],{"type":2822},"partially-true",[11,2824,2825,2826,117,2829,2832],{},"I was recently talking with a peer evaluating AI tooling for spec-driven testing — writing tests close to acceptance criteria that non-technical stakeholders could read and contribute to. That's exactly the use case platforms like ",[40,2827,2828],{},"testRigor",[40,2830,2831],{},"Momentic"," are built for. The appeal is real: tests read like plain English, the barrier to authoring drops, and product and QA can collaborate on coverage.",[11,2834,2835],{},"The cost concern is real too, and it's specific to how these tools work. Tests are written in natural language and an LLM interprets and executes each step against the live application at runtime. Every test step triggers an API call — and at scale that compounds fast. If you've vendor-locked into one of these platforms, the cost and inefficiency complaints are justified.",[11,2837,2838,2839,117,2842,2845],{},"This isn't a new problem space though — tools like ",[40,2840,2841],{},"SpecFlow",[40,2843,2844],{},"Cucumber"," solved spec-driven testing before AI by generating the translation layer once at authoring time as coded step definitions. The difference with AI execution platforms is that translation happens at runtime on every run.",[11,2847,2848,2851],{},[40,2849,2850],{},"Mitigation — the best of both worlds:"," Use AI to generate Playwright step definitions from your Gherkin scenarios. You get the plain-English spec, stakeholder-readable coverage, and deterministic execution without the per-run API cost or vendor lock-in. The translation layer is authored once, not re-interpreted on every CI run.",[889,2853,2854,2860,2866],{},[666,2855,2856,2859],{},[40,2857,2858],{},"AI for authoring"," (Claude Code, Copilot) — tokens consumed once at write time, tests run deterministically forever after",[666,2861,2862,2865],{},[40,2863,2864],{},"AI for execution"," (testRigor, Momentic) — per-run API cost that scales with suite size and CI frequency; vendor lock-in compounds the risk",[666,2867,2868,2871],{},[40,2869,2870],{},"AI-generated Gherkin + Playwright step definitions"," — spec-driven workflow, one-time authoring cost, deterministic execution",[2337,2873],{},[21,2875,2877],{"id":2876},"self-healing-tests-are-a-game-changer","\"Self-Healing Tests Are a Game Changer\"",[2344,2879],{"type":2880},"false",[11,2882,2883],{},"Self-healing sounds appealing until you watch it in practice. The mechanism is: test fails → AI tries alternative selectors → updates the test if one works. The problem is what this hides.",[11,2885,2886],{},"A test that needs to heal constantly is a signal:",[889,2888,2889,2892,2895],{},[666,2890,2891],{},"The selectors are brittle to begin with",[666,2893,2894],{},"The application's DOM structure is changing in ways that aren't intentional",[666,2896,2897],{},"Nobody is reviewing what the \"heal\" actually changed",[11,2899,2900],{},"Self-healing burns tokens in retry loops, produces increasingly complex scripts to work around what should be a simple locator fix, and obscures whether the application itself changed in a meaningful way.",[11,2902,2903],{},"More importantly, a breaking test on a well-written selector is a useful signal — something changed and deserves attention. An accidental commit, a feature flag flipped in the wrong environment, a UI change pushed without notice that would have gone through untested. Self-healing silently absorbs that signal on your behalf. You avoid the maintenance burden, but you also lose the bump in the road that was trying to tell you something.",[11,2905,2906,2909,2910,672,2912,1128,2915,2917],{},[40,2907,2908],{},"The real fix:"," write resilient selectors from the start. Prefer ",[86,2911,1777],{},[86,2913,2914],{},"getByLabel",[86,2916,1554],{}," over XPath or CSS chains. If a selector breaks, fix it — don't automate around it. The discipline to write good selectors upfront costs less than the ongoing overhead of managing a self-healing test suite.",[2337,2919],{},[21,2921,2923],{"id":2922},"ai-agents-cant-get-past-corporate-sso","\"AI Agents Can't Get Past Corporate SSO\"",[2344,2925],{"type":2926},"user-error",[11,2928,2929],{},"Google Auth and enterprise SSO do actively block automated agents because the same techniques are used by bad actors. But treating this as an insurmountable AI limitation misses the point — this same friction exists with traditional automation too.",[11,2931,2932],{},"The solutions are engineering problems, not AI problems:",[889,2934,2935,2941,2950],{},[666,2936,2937,2940],{},[40,2938,2939],{},"Feature-flag a test auth bypass"," — a password-based test login path, never exposed near production, that bypasses SSO for automation",[666,2942,2943,2946,2947],{},[40,2944,2945],{},"Cookie injection"," — capture an authenticated session and inject the cookies into your Playwright context via ",[86,2948,2949],{},"storageState",[666,2951,2952,2955,2956,2959],{},[40,2953,2954],{},"Pre-authenticated session state"," — Playwright's built-in ",[86,2957,2958],{},"playwright\u002Ftest"," supports saving and reusing auth state across tests",[11,2961,2962],{},"The teams calling SSO an AI blocker are usually the same teams that haven't solved it for traditional automation either. That's an org prioritization problem, not a tool limitation.",[2337,2964],{},[21,2966,2968],{"id":2967},"ai-cant-understand-why-a-test-is-failing","\"AI Can't Understand Why a Test Is Failing\"",[2344,2970],{"type":2822},[11,2972,2973],{},"AI can only reason about what you give it. Feed it raw logs and it burns tokens guessing. Feed it a screenshot of the failure, the relevant DOM snapshot, network traffic, and the test output together and it's significantly more useful.",[11,2975,2976],{},"Playwright gives you everything you need:",[889,2978,2979,2985,2991],{},[666,2980,2981,2984],{},[40,2982,2983],{},"Trace viewer"," — full timeline of actions, screenshots, and network calls",[666,2986,2987,2990],{},[40,2988,2989],{},"HAR files"," — captured network traffic for the failing scenario",[666,2992,2993,2996],{},[40,2994,2995],{},"Console logs"," — surfaced alongside test output",[11,2998,2999],{},"The failure is usually in how failure information is surfaced to the model, not the model's ability to interpret it. MCP servers that expose the running app state close this gap further — giving the agent direct access to the DOM rather than a static snapshot.",[11,3001,3002],{},"If your AI debugging workflow is \"paste the error message and ask what's wrong,\" you're leaving most of the capability on the table.",[2337,3004],{},[21,3006,3008],{"id":3007},"just-point-ai-at-a-user-story-and-itll-write-good-tests","\"Just Point AI at a User Story and It'll Write Good Tests\"",[2344,3010],{"type":2822},[11,3012,3013],{},"If you feed acceptance criteria directly to AI and expect meaningful test coverage, you'll get tests that mirror the criteria without covering edge cases, negative paths, or real-world usage patterns. The model faithfully tests what the story says — which is a problem when the story is incomplete, ambiguous, or just wrong.",[11,3015,3016],{},"Garbage in, garbage out. AI doesn't save you from bad requirements. Testing the requirements before writing any code or tests is the most important step to prevent defects and rework later:",[889,3018,3019,3022],{},[666,3020,3021],{},"Apply shift-left techniques to poke holes in the story before anyone writes a line of code",[666,3023,3024],{},"Use AI as another voice in the room — ask it to surface ambiguities, missing edge cases, and unstated assumptions in the acceptance criteria",[11,3026,3027],{},"The better framing is to treat AI output as a first draft from a capable junior tester. It covers the happy path, follows the instructions it was given, and misses the things an experienced tester would catch. Your job is to review it with that lens — not to treat it as done.",[11,3029,3030],{},[40,3031,3032],{},"What makes the workflow salvageable:",[889,3034,3035,3038,3041],{},[666,3036,3037],{},"Encode your testing conventions and selector strategy in a reusable Skill so the model isn't inventing its own patterns each time",[666,3039,3040],{},"Give the model the page source alongside the spec so it's working from real structure, not assumptions",[666,3042,3043],{},"Plan before writing — have the agent outline which scenarios it intends to cover before generating any code, so you can catch misunderstandings early",[11,3045,3046],{},[40,3047,3048],{},"How to validate what AI produced:",[889,3050,3051,3054,3057],{},[666,3052,3053],{},"Be explicit about the coverage type you want — statement, branch, or line — and verify with your coverage tool's report rather than trusting the AI's output",[666,3055,3056],{},"Check that tests cover edges, not just the happy path — AI-generated tests tend to be written just easy enough to pass",[666,3058,3059],{},"Run mutation testing to validate test quality — tools like Stryker introduce small code changes to verify your tests actually catch them; if a mutant survives, the test isn't doing its job",[2337,3061],{},[21,3063,3065],{"id":3064},"ai-has-no-place-in-test-execution","\"AI Has No Place in Test Execution\"",[2344,3067],{"type":2822},[11,3069,3070],{},"AI belongs in authoring and maintenance, not runtime execution — your regression suite needs to be deterministic and cost-predictable. Non-deterministic tests erode trust fast: a flaky red build gets ignored, and an ignored build stops being a safety net. AI in the execution loop introduces variability in both behavior and cost that undermines both. Debugging failures becomes an exercise in chasing ghosts — different inputs or different test paths on each run mean you can't reliably reproduce what actually failed.",[11,3072,3073],{},"Write deterministic Playwright tests and deploy them to CI. When selectors break or frameworks change, AI is a useful tool for resolving those failures — but that's a deliberate maintenance task triggered by a human, not an automated self-healing loop running on every failure.",[11,3075,3076,3077,3080],{},"Where AI at runtime ",[32,3078,3079],{},"does"," make sense:",[889,3082,3083,3086,3089],{},[666,3084,3085],{},"Exploratory accessibility scans — running axe-core across a surface and having AI triage and prioritize findings",[666,3087,3088],{},"One-off audit workflows run by a human — not scheduled CI jobs",[666,3090,3091],{},"Failure investigation — giving an agent access to a failing test's trace to diagnose the root cause",[11,3093,3094],{},"If it runs in CI on every commit, it should be deterministic. If it's a human-driven investigative workflow, AI at runtime can be useful.",[2337,3096],{},[21,3098,3100],{"id":3099},"separating-ai-testing-hype-from-legitimate-limitation","Separating AI Testing Hype from Legitimate Limitation",[11,3102,3103],{},"The complaints that are legitimate — assertion rewriting, execution cost — are the ones you almost never hear in vendor demos. The complaints that turn out to be user error — SSO, AI can't understand failures — are solvable with the right setup.",[11,3105,3106],{},"The tooling is only part of the equation. AI will confidently write passing, useless tests if you let it — and most of the mitigations in this article come down to the same thing: staying in the loop, knowing what good looks like, and not outsourcing your judgment to the model.",[198,3108],{":items":3109},"[\"\u002Fsoftware-testing\u002Ftest-automation\u002Fai-in-testing-2026-state-of-the-industry\",\"\u002Fsoftware-testing\u002Ftest-automation\u002Fautomated-api-testing-with-schemathesis\"]",[1137,3111,3112],{},"html pre.shiki code .s_gjE,html code.shiki .s_gjE{--shiki-light:#90A4AE;--shiki-light-font-style:italic;--shiki-default:#66707B;--shiki-default-font-style:inherit;--shiki-dark:#BDC4CC;--shiki-dark-font-style:inherit}html pre.shiki code .stWsX,html code.shiki .stWsX{--shiki-light:#9C3EDA;--shiki-default:#A0111F;--shiki-dark:#FF9492}html pre.shiki code .sb1SK,html code.shiki .sb1SK{--shiki-light:#6182B8;--shiki-default:#622CBC;--shiki-dark:#DBB7FF}html pre.shiki code .sPJuK,html code.shiki .sPJuK{--shiki-light:#39ADB5;--shiki-default:#0E1116;--shiki-dark:#F0F3F6}html pre.shiki code .s2xgV,html code.shiki .s2xgV{--shiki-light:#90A4AE;--shiki-light-font-style:italic;--shiki-default:#702C00;--shiki-default-font-style:inherit;--shiki-dark:#FFB757;--shiki-dark-font-style:inherit}html pre.shiki code .sQ79N,html code.shiki .sQ79N{--shiki-light:#90A4AE;--shiki-default:#023B95;--shiki-dark:#91CBFF}html pre.shiki code .sE6rD,html code.shiki .sE6rD{--shiki-light:#39ADB5;--shiki-default:#A0111F;--shiki-dark:#FF9492}html pre.shiki code .sZ-rw,html code.shiki .sZ-rw{--shiki-light:#90A4AE;--shiki-default:#0E1116;--shiki-dark:#F0F3F6}html pre.shiki code .sq0XF,html code.shiki .sq0XF{--shiki-light:#E53935;--shiki-default:#0E1116;--shiki-dark:#F0F3F6}html pre.shiki code .s6g51,html code.shiki .s6g51{--shiki-light:#F76D47;--shiki-default:#023B95;--shiki-dark:#91CBFF}html pre.shiki code .sZTni,html code.shiki .sZTni{--shiki-light:#39ADB5;--shiki-light-font-style:italic;--shiki-default:#A0111F;--shiki-default-font-style:inherit;--shiki-dark:#FF9492;--shiki-dark-font-style:inherit}html .light .shiki span{color:var(--shiki-light);background:var(--shiki-light-bg);font-style:var(--shiki-light-font-style);font-weight:var(--shiki-light-font-weight);text-decoration:var(--shiki-light-text-decoration)}html.light .shiki span{color:var(--shiki-light);background:var(--shiki-light-bg);font-style:var(--shiki-light-font-style);font-weight:var(--shiki-light-font-weight);text-decoration:var(--shiki-light-text-decoration)}html .default .shiki span{color:var(--shiki-default);background:var(--shiki-default-bg);font-style:var(--shiki-default-font-style);font-weight:var(--shiki-default-font-weight);text-decoration:var(--shiki-default-text-decoration)}html .shiki span{color:var(--shiki-default);background:var(--shiki-default-bg);font-style:var(--shiki-default-font-style);font-weight:var(--shiki-default-font-weight);text-decoration:var(--shiki-default-text-decoration)}html .dark .shiki span{color:var(--shiki-dark);background:var(--shiki-dark-bg);font-style:var(--shiki-dark-font-style);font-weight:var(--shiki-dark-font-weight);text-decoration:var(--shiki-dark-text-decoration)}html.dark .shiki span{color:var(--shiki-dark);background:var(--shiki-dark-bg);font-style:var(--shiki-dark-font-style);font-weight:var(--shiki-dark-font-weight);text-decoration:var(--shiki-dark-text-decoration)}html pre.shiki code .sZi47,html code.shiki .sZi47{--shiki-light:#39ADB5;--shiki-default:#032563;--shiki-dark:#ADDCFF}html pre.shiki code .srGNg,html code.shiki .srGNg{--shiki-light:#91B859;--shiki-default:#032563;--shiki-dark:#ADDCFF}",{"title":202,"searchDepth":203,"depth":203,"links":3114},[3115,3118,3119,3120,3121,3122,3123,3124],{"id":2341,"depth":203,"text":2342,"children":3116},[3117],{"id":2403,"depth":328,"text":2404},{"id":2818,"depth":203,"text":2819},{"id":2876,"depth":203,"text":2877},{"id":2922,"depth":203,"text":2923},{"id":2967,"depth":203,"text":2968},{"id":3007,"depth":203,"text":3008},{"id":3064,"depth":203,"text":3065},{"id":3099,"depth":203,"text":3100},"\u002Fimages\u002Fposts\u002Fai-test-automation-pitfalls-vs-user-error\u002Fai-testing-limits-v-skills-cover.webp","2026-03-24","Not every AI testing complaint is a real tool limitation. Some are skill gaps dressed up as criticism. This article goes through the most common ones and gives each a verdict — with practical mitigations for the ones that are legitimate.",{},"\u002Fsoftware-testing\u002Ftest-automation\u002Fai-test-automation-pitfalls-vs-user-error",{"title":2327,"description":3127},"software-testing\u002Ftest-automation\u002Fai-test-automation-pitfalls-vs-user-error","FJCtOw8QEmpNSJnUFAmNsJF3XPEDe08EVVeVlCHI-kY",{"id":3134,"title":3135,"bmcUsername":6,"body":3136,"cover":5256,"date":5257,"description":5258,"draft":216,"extension":217,"features":6,"githubRepo":6,"headline":6,"highlight":6,"icon":6,"meta":5259,"navigation":219,"npmPackage":6,"order":6,"path":5260,"seo":5261,"stem":5262,"__hash__":5263},"content\u002Fsoftware-testing\u002Ftest-automation\u002Fautomated-api-testing-with-schemathesis.md","Automated API Testing with Schemathesis: Let Your OpenAPI Spec Do the Work",{"type":8,"value":3137,"toc":5235},[3138,3141,3144,3147,3151,3154,3168,3171,3175,3178,3181,3184,3188,3191,3341,3345,3348,3351,3362,3365,3412,3415,3419,3430,3441,3449,3464,3468,3471,3499,3502,3505,3650,3653,3714,3717,3864,3868,3871,3923,3935,3986,3989,3992,3999,4158,4165,4182,4187,4194,4213,4216,4234,4373,4376,4476,4550,4625,4629,4632,4638,4644,4650,4653,4657,4674,4737,4756,4760,4770,4790,4839,4842,4846,4850,4861,4895,4908,4912,4923,4932,5078,5085,5116,5119,5156,5158,5162,5165,5176,5180,5197,5201,5212,5216,5219,5227,5230,5232],[11,3139,3140],{},"You've written the OpenAPI spec. You've documented every endpoint, every request body, every response schema. And then you open your test file and start writing the same information all over again — this time as test cases. Endpoint by endpoint. Parameter by parameter. It's tedious, it's repetitive, and the coverage is always incomplete because there are only so many edge cases a human will think to test manually.",[11,3142,3143],{},"At this point you might reach for Postman, REST Assured, or Karate — solid tools, but ones that don't change the fundamental problem. You're still the one writing every test case. Or maybe you think to hand it off to AI: ask ChatGPT or Copilot to generate the test cases for you. That helps with the tedium, but it introduces a different problem. AI generates tests based on what it thinks your API should do — not what your spec actually says it does. The output needs reviewing, it can hallucinate edge cases that don't apply to your schema, and when your spec changes, the AI-generated tests don't automatically follow.",[11,3145,3146],{},"The spec you already have sits unused as a test asset in both approaches. There's a better way, and most QA engineers haven't heard of it yet.",[21,3148,3150],{"id":3149},"the-real-world-challenge-that-started-this-search","The Real-World Challenge That Started This Search",[11,3152,3153],{},"Recently a team transitioning to a more AI-forward approach was working with a largely untested API that did have a Swagger\u002FOpenAPI spec. Their plan was to have Claude generate the roughly 1,700 API test cases needed to cover it. I was asked to weigh in and didn't like where that was headed. It would have meant:",[889,3155,3156,3159,3162,3165],{},[666,3157,3158],{},"A major maintenance burden over time — AI-generated tests aren't tied to the spec, so every API change requires regenerating or manually updating them",[666,3160,3161],{},"Unnecessary overhead at that scale — Playwright is optimized for browser automation; running 1,700 pure API tests through it adds tooling overhead that purpose-built API testing tools don't have",[666,3163,3164],{},"Defaults to a happy-path bias — without explicit prompting to think adversarially, AI-generated tests tend to confirm what the spec says should work rather than probe where it breaks, creating false confidence in coverage",[666,3166,3167],{},"A wasteful use of AI for a problem that a purpose-built tool could solve more reliably",[11,3169,3170],{},"I suspected there had to be open-source tooling that could generate test cases directly from the OpenAPI spec. After researching the options — Dredd, Portman, CATS, and others — Schemathesis came out the clear winner. Here's why.",[21,3172,3174],{"id":3173},"your-openapi-spec-is-already-a-test-asset","Your OpenAPI Spec Is Already a Test Asset",[11,3176,3177],{},"The OpenAPI specification describes everything a testing tool needs: the endpoints, the HTTP methods, the request parameters, the request body schemas, the expected response codes. It's a complete contract — and Schemathesis treats it as one.",[11,3179,3180],{},"Rather than running fixed, hand-authored test cases, Schemathesis reads your spec and generates a large variety of inputs automatically, sends them to your running API, and verifies that every response conforms to what the spec says it should be. Coverage that would take days to write manually runs in minutes, and it updates automatically as your spec evolves.",[11,3182,3183],{},"I feel it's the most capable open-source tool in this space, and genuinely underused relative to how good it is. Most QA engineers I polled hadn't heard of it — which is exactly why it's worth understanding before the rest of the industry catches up.",[21,3185,3187],{"id":3186},"schemathesis-vs-other-api-testing-tools","Schemathesis vs Other API Testing Tools",[11,3189,3190],{},"Here's how the API tools I evaluated rank across the criteria that mattered the most to me for this use case.",[3192,3193,3194,3216],"table",{},[3195,3196,3197],"thead",{},[3198,3199,3200,3204,3207,3210,3213],"tr",{},[3201,3202,3203],"th",{},"Tool",[3201,3205,3206],{},"Approach",[3201,3208,3209],{},"Generates edge cases",[3201,3211,3212],{},"Open source",[3201,3214,3215],{},"Actively maintained",[3217,3218,3219,3237,3251,3265,3281,3295,3309,3323],"tbody",{},[3198,3220,3221,3225,3228,3231,3234],{},[3222,3223,3224],"td",{},"Postman \u002F Newman",[3222,3226,3227],{},"Manual test authoring",[3222,3229,3230],{},"❌ No",[3222,3232,3233],{},"⚠️ Partially",[3222,3235,3236],{},"✅ Yes",[3198,3238,3239,3242,3245,3247,3249],{},[3222,3240,3241],{},"REST Assured",[3222,3243,3244],{},"Manual test authoring (Java)",[3222,3246,3230],{},[3222,3248,3236],{},[3222,3250,3236],{},[3198,3252,3253,3256,3259,3261,3263],{},[3222,3254,3255],{},"Karate",[3222,3257,3258],{},"Manual test authoring (DSL)",[3222,3260,3230],{},[3222,3262,3236],{},[3222,3264,3236],{},[3198,3266,3267,3270,3273,3276,3278],{},[3222,3268,3269],{},"Dredd",[3222,3271,3272],{},"Spec-driven contract testing",[3222,3274,3275],{},"⚠️ Limited",[3222,3277,3236],{},[3222,3279,3280],{},"💀 Largely abandoned",[3198,3282,3283,3286,3289,3291,3293],{},[3222,3284,3285],{},"Portman",[3222,3287,3288],{},"OpenAPI → Postman collection",[3222,3290,3230],{},[3222,3292,3236],{},[3222,3294,3236],{},[3198,3296,3297,3300,3303,3305,3307],{},[3222,3298,3299],{},"CATS",[3222,3301,3302],{},"Spec-driven fuzzing",[3222,3304,3236],{},[3222,3306,3236],{},[3222,3308,3275],{},[3198,3310,3311,3314,3317,3319,3321],{},[3222,3312,3313],{},"42Crunch",[3222,3315,3316],{},"Spec-driven security testing",[3222,3318,3236],{},[3222,3320,3230],{},[3222,3322,3236],{},[3198,3324,3325,3330,3335,3337,3339],{},[3222,3326,3327],{},[40,3328,3329],{},"Schemathesis",[3222,3331,3332],{},[40,3333,3334],{},"Property-based testing from spec",[3222,3336,3236],{},[3222,3338,3236],{},[3222,3340,3236],{},[21,3342,3344],{"id":3343},"what-schemathesis-does-differently","What Schemathesis Does Differently",[11,3346,3347],{},"Schemathesis is an open-source API testing tool that performs property-based testing against your OpenAPI (or GraphQL) spec. Rather than running a fixed set of hand-written test cases, it uses your spec as a contract to generate use and misuse cases using techniques like input fuzzing.",[11,3349,3350],{},"Types of testing Schemathesis performs:",[889,3352,3353,3356,3359],{},[666,3354,3355],{},"Boundary validation",[666,3357,3358],{},"Property input fuzzing (use and misuse cases \u002F error handling)",[666,3360,3361],{},"Resource state changes",[11,3363,3364],{},"Defect categories Schemathesis catches:",[889,3366,3367,3384,3401],{},[666,3368,3369,3370],{},"Schema-related\n",[889,3371,3372,3375,3378,3381],{},[666,3373,3374],{},"API responses not matching schema definition",[666,3376,3377],{},"Status codes not documented in the spec",[666,3379,3380],{},"Missing headers",[666,3382,3383],{},"Wrong content type returned",[666,3385,3386,3387],{},"Implementation bugs\n",[889,3388,3389,3392,3395,3398],{},[666,3390,3391],{},"Unhandled exceptions\u002Fcrashes (5xx level errors)",[666,3393,3394],{},"Rejection of valid inputs according to the spec",[666,3396,3397],{},"Header issues",[666,3399,3400],{},"Authentication bypasses",[666,3402,3403,3404],{},"Stateful tests\n",[889,3405,3406,3409],{},[666,3407,3408],{},"Getting deleted resource e.g. POST → DELETE → GET (deleted item)",[666,3410,3411],{},"Getting created resource e.g. POST → GET (created item)",[11,3413,3414],{},"Some of these classes of bugs are easy to miss with manual test writing: unhandled parameter combinations, missing input validation, responses that don't match the documented schema, and server errors triggered by unexpected but technically valid inputs. It can be difficult to think of all the different input combinations which is where leveraging a tool like this can cover more with less effort and time.",[21,3416,3418],{"id":3417},"how-to-install-and-run-schemathesis","How to Install and Run Schemathesis",[11,3420,3421,3422,3425,3426,3429],{},"Schemathesis can be run as a CLI tool or as a Python library. To first evaluate Schemathesis we will use ",[32,3423,3424],{},"The Petstore API"," which is a popular OpenAPI testing playground. After, we'll use ",[32,3427,3428],{},"RESTful Booker"," for a more complicated example. RESTful Booker has:",[889,3431,3432,3435,3438],{},[666,3433,3434],{},"Realistic CRUD endpoints (create, update, delete)",[666,3436,3437],{},"Authentication",[666,3439,3440],{},"Intentionally buggy behavior",[11,3442,3443,3444,3448],{},"I have a section listing more practice API endpoints on ",[1116,3445,3447],{"href":3446},"\u002Fsoftware-testing\u002Ftest-automation\u002Fbest-websites-for-practicing-test-automation","The Best Websites for Practicing Test Automation"," if you want to try some others after.",[11,3450,3451,3452,3456,3457,3459,3460,3463],{},"The recommended way to run Schemathesis is via ",[15,3453],{"href":3454,"text":3455},"https:\u002F\u002Fdocs.astral.sh\u002Fuv\u002F","uv",", a fast Python package manager. If you don't have it installed, grab it first — it's a one-liner and handles everything including Python if needed. Once you have ",[86,3458,3455],{},", no separate install step is required. ",[86,3461,3462],{},"uvx"," runs Schemathesis in an isolated environment automatically.",[21,3465,3467],{"id":3466},"petstore-api-unauthenticated-endpoints","PetStore API (unauthenticated endpoints)",[11,3469,3470],{},"The Petstore API has a publicly available OpenAPI spec and no authentication requirements, making it the simplest way to see Schemathesis in action:",[262,3472,3476],{"className":3473,"code":3474,"language":3475,"meta":202,"style":202},"language-bash shiki shiki-themes material-theme-lighter github-light-high-contrast github-dark-high-contrast","uvx schemathesis run https:\u002F\u002Fpetstore.swagger.io\u002Fv2\u002Fswagger.json --checks all\n","bash",[86,3477,3478],{"__ignoreMap":202},[271,3479,3480,3483,3486,3489,3492,3496],{"class":273,"line":274},[271,3481,3462],{"class":3482},"sA8fK",[271,3484,3485],{"class":292}," schemathesis",[271,3487,3488],{"class":292}," run",[271,3490,3491],{"class":292}," https:\u002F\u002Fpetstore.swagger.io\u002Fv2\u002Fswagger.json",[271,3493,3495],{"class":3494},"spMcu"," --checks",[271,3497,3498],{"class":292}," all\n",[11,3500,3501],{},"That's it — Schemathesis reads the spec, generates test cases for every endpoint, and reports any violations or server errors it finds.",[11,3503,3504],{},"Here's what the run looks like as it executes:",[262,3506,3511],{"className":3507,"code":3508,"filename":3509,"language":3510,"meta":202,"style":202},"language-sh shiki shiki-themes material-theme-lighter github-light-high-contrast github-dark-high-contrast","Schemathesis v4.12.0\n━━━━━━━━━━━━━━━━━━━━\n\n ✅  Loaded specification from https:\u002F\u002Fpetstore.swagger.io\u002Fv2\u002Fswagger.json (in 1.03s)\n\n     Base URL:         https:\u002F\u002Fpetstore.swagger.io\u002Fv2\n     Specification:    Open API 2.0\n     Operations:       20 selected \u002F 20 total\n\n ✅  API capabilities:\n\n     Supports NULL byte in headers:    ✘\n\n ❌  Examples (in 1.17s)\n\n     ❌  2 failed  ⏭  18 skipped\n\n ❌  Coverage (in 10.33s)\n\n     ❌ 20 failed\n\n ❌  Fuzzing (in 3.97s)\n\n     ✅  1 passed  ❌ 19 failed\n\n 🕓  Stateful\n\n     0:00:22 119 scenarios  •  12 covered \u002F 28 selected \u002F 28 total (28 inferred)\n\n     ✅ 99 passed  ❌ 20 failed\n","Schemathesis Output","sh",[86,3512,3513,3518,3523,3527,3532,3536,3541,3546,3551,3555,3560,3564,3569,3573,3578,3582,3587,3591,3596,3600,3605,3609,3614,3618,3623,3627,3632,3636,3641,3645],{"__ignoreMap":202},[271,3514,3515],{"class":273,"line":274},[271,3516,3517],{},"Schemathesis v4.12.0\n",[271,3519,3520],{"class":273,"line":203},[271,3521,3522],{},"━━━━━━━━━━━━━━━━━━━━\n",[271,3524,3525],{"class":273,"line":328},[271,3526,331],{"emptyLinePlaceholder":219},[271,3528,3529],{"class":273,"line":334},[271,3530,3531],{}," ✅  Loaded specification from https:\u002F\u002Fpetstore.swagger.io\u002Fv2\u002Fswagger.json (in 1.03s)\n",[271,3533,3534],{"class":273,"line":355},[271,3535,331],{"emptyLinePlaceholder":219},[271,3537,3538],{"class":273,"line":372},[271,3539,3540],{},"     Base URL:         https:\u002F\u002Fpetstore.swagger.io\u002Fv2\n",[271,3542,3543],{"class":273,"line":389},[271,3544,3545],{},"     Specification:    Open API 2.0\n",[271,3547,3548],{"class":273,"line":395},[271,3549,3550],{},"     Operations:       20 selected \u002F 20 total\n",[271,3552,3553],{"class":273,"line":400},[271,3554,331],{"emptyLinePlaceholder":219},[271,3556,3557],{"class":273,"line":417},[271,3558,3559],{}," ✅  API capabilities:\n",[271,3561,3562],{"class":273,"line":425},[271,3563,331],{"emptyLinePlaceholder":219},[271,3565,3566],{"class":273,"line":467},[271,3567,3568],{},"     Supports NULL byte in headers:    ✘\n",[271,3570,3571],{"class":273,"line":472},[271,3572,331],{"emptyLinePlaceholder":219},[271,3574,3575],{"class":273,"line":501},[271,3576,3577],{}," ❌  Examples (in 1.17s)\n",[271,3579,3580],{"class":273,"line":538},[271,3581,331],{"emptyLinePlaceholder":219},[271,3583,3584],{"class":273,"line":544},[271,3585,3586],{},"     ❌  2 failed  ⏭  18 skipped\n",[271,3588,3589],{"class":273,"line":549},[271,3590,331],{"emptyLinePlaceholder":219},[271,3592,3593],{"class":273,"line":602},[271,3594,3595],{}," ❌  Coverage (in 10.33s)\n",[271,3597,3598],{"class":273,"line":620},[271,3599,331],{"emptyLinePlaceholder":219},[271,3601,3602],{"class":273,"line":627},[271,3603,3604],{},"     ❌ 20 failed\n",[271,3606,3607],{"class":273,"line":633},[271,3608,331],{"emptyLinePlaceholder":219},[271,3610,3611],{"class":273,"line":639},[271,3612,3613],{}," ❌  Fuzzing (in 3.97s)\n",[271,3615,3616],{"class":273,"line":822},[271,3617,331],{"emptyLinePlaceholder":219},[271,3619,3620],{"class":273,"line":827},[271,3621,3622],{},"     ✅  1 passed  ❌ 19 failed\n",[271,3624,3625],{"class":273,"line":833},[271,3626,331],{"emptyLinePlaceholder":219},[271,3628,3629],{"class":273,"line":838},[271,3630,3631],{}," 🕓  Stateful\n",[271,3633,3634],{"class":273,"line":844},[271,3635,331],{"emptyLinePlaceholder":219},[271,3637,3638],{"class":273,"line":850},[271,3639,3640],{},"     0:00:22 119 scenarios  •  12 covered \u002F 28 selected \u002F 28 total (28 inferred)\n",[271,3642,3643],{"class":273,"line":856},[271,3644,331],{"emptyLinePlaceholder":219},[271,3646,3647],{"class":273,"line":862},[271,3648,3649],{},"     ✅ 99 passed  ❌ 20 failed\n",[11,3651,3652],{},"One of the failures Schemathesis surfaced:",[262,3654,3657],{"className":3507,"code":3655,"filename":3656,"language":3510,"meta":202,"style":202},"________________________________________________ POST \u002Fuser\u002FcreateWithList ________________________________________________\n1. Test Case ID: z6UISM\n\n- Server error\n\n[500] Internal Server Error:\n\n    `{\"code\":500,\"type\":\"unknown\",\"message\":\"something bad happened\"}`\n\nReproduce with:\n\n    curl -X POST -H 'Content-Type: application\u002Fjson' -d false https:\u002F\u002Fpetstore.swagger.io\u002Fv2\u002Fuser\u002FcreateWithList\n","Failure Detail",[86,3658,3659,3664,3669,3673,3678,3682,3687,3691,3696,3700,3705,3709],{"__ignoreMap":202},[271,3660,3661],{"class":273,"line":274},[271,3662,3663],{},"________________________________________________ POST \u002Fuser\u002FcreateWithList ________________________________________________\n",[271,3665,3666],{"class":273,"line":203},[271,3667,3668],{},"1. Test Case ID: z6UISM\n",[271,3670,3671],{"class":273,"line":328},[271,3672,331],{"emptyLinePlaceholder":219},[271,3674,3675],{"class":273,"line":334},[271,3676,3677],{},"- Server error\n",[271,3679,3680],{"class":273,"line":355},[271,3681,331],{"emptyLinePlaceholder":219},[271,3683,3684],{"class":273,"line":372},[271,3685,3686],{},"[500] Internal Server Error:\n",[271,3688,3689],{"class":273,"line":389},[271,3690,331],{"emptyLinePlaceholder":219},[271,3692,3693],{"class":273,"line":395},[271,3694,3695],{},"    `{\"code\":500,\"type\":\"unknown\",\"message\":\"something bad happened\"}`\n",[271,3697,3698],{"class":273,"line":400},[271,3699,331],{"emptyLinePlaceholder":219},[271,3701,3702],{"class":273,"line":417},[271,3703,3704],{},"Reproduce with:\n",[271,3706,3707],{"class":273,"line":425},[271,3708,331],{"emptyLinePlaceholder":219},[271,3710,3711],{"class":273,"line":467},[271,3712,3713],{},"    curl -X POST -H 'Content-Type: application\u002Fjson' -d false https:\u002F\u002Fpetstore.swagger.io\u002Fv2\u002Fuser\u002FcreateWithList\n",[11,3715,3716],{},"And the final summary:",[262,3718,3721],{"className":3507,"code":3719,"filename":3720,"language":3510,"meta":202,"style":202},"========================================================= SUMMARY =========================================================\n\nAPI Operations:\n  Selected: 20\u002F20\n  Tested: 20\n\nTest Phases:\n  ❌ Examples\n  ❌ Coverage\n  ❌ Fuzzing\n  ❌ Stateful\n\nFailures:\n  ❌ API accepts invalid authentication: 1\n  ❌ API accepts requests without authentication: 1\n  ❌ Server error: 7\n  ❌ Use after free: 1\n  ❌ Response header does not conform to the schema: 1\n  ❌ Response violates schema: 8\n  ❌ API accepted schema-violating request: 9\n  ❌ API rejected schema-compliant request: 3\n  ❌ Missing Content-Type header: 3\n  ❌ Missing header not rejected: 2\n  ❌ Undocumented Content-Type: 3\n  ❌ Undocumented HTTP status code: 15\n  ❌ Unsupported methods: 17\n\nTest cases:\n  2860 generated, 59 found 71 unique failures\n","Summary",[86,3722,3723,3728,3732,3737,3742,3747,3751,3756,3761,3766,3771,3776,3780,3785,3790,3795,3800,3805,3810,3815,3820,3825,3830,3835,3840,3845,3850,3854,3859],{"__ignoreMap":202},[271,3724,3725],{"class":273,"line":274},[271,3726,3727],{},"========================================================= SUMMARY =========================================================\n",[271,3729,3730],{"class":273,"line":203},[271,3731,331],{"emptyLinePlaceholder":219},[271,3733,3734],{"class":273,"line":328},[271,3735,3736],{},"API Operations:\n",[271,3738,3739],{"class":273,"line":334},[271,3740,3741],{},"  Selected: 20\u002F20\n",[271,3743,3744],{"class":273,"line":355},[271,3745,3746],{},"  Tested: 20\n",[271,3748,3749],{"class":273,"line":372},[271,3750,331],{"emptyLinePlaceholder":219},[271,3752,3753],{"class":273,"line":389},[271,3754,3755],{},"Test Phases:\n",[271,3757,3758],{"class":273,"line":395},[271,3759,3760],{},"  ❌ Examples\n",[271,3762,3763],{"class":273,"line":400},[271,3764,3765],{},"  ❌ Coverage\n",[271,3767,3768],{"class":273,"line":417},[271,3769,3770],{},"  ❌ Fuzzing\n",[271,3772,3773],{"class":273,"line":425},[271,3774,3775],{},"  ❌ Stateful\n",[271,3777,3778],{"class":273,"line":467},[271,3779,331],{"emptyLinePlaceholder":219},[271,3781,3782],{"class":273,"line":472},[271,3783,3784],{},"Failures:\n",[271,3786,3787],{"class":273,"line":501},[271,3788,3789],{},"  ❌ API accepts invalid authentication: 1\n",[271,3791,3792],{"class":273,"line":538},[271,3793,3794],{},"  ❌ API accepts requests without authentication: 1\n",[271,3796,3797],{"class":273,"line":544},[271,3798,3799],{},"  ❌ Server error: 7\n",[271,3801,3802],{"class":273,"line":549},[271,3803,3804],{},"  ❌ Use after free: 1\n",[271,3806,3807],{"class":273,"line":602},[271,3808,3809],{},"  ❌ Response header does not conform to the schema: 1\n",[271,3811,3812],{"class":273,"line":620},[271,3813,3814],{},"  ❌ Response violates schema: 8\n",[271,3816,3817],{"class":273,"line":627},[271,3818,3819],{},"  ❌ API accepted schema-violating request: 9\n",[271,3821,3822],{"class":273,"line":633},[271,3823,3824],{},"  ❌ API rejected schema-compliant request: 3\n",[271,3826,3827],{"class":273,"line":639},[271,3828,3829],{},"  ❌ Missing Content-Type header: 3\n",[271,3831,3832],{"class":273,"line":822},[271,3833,3834],{},"  ❌ Missing header not rejected: 2\n",[271,3836,3837],{"class":273,"line":827},[271,3838,3839],{},"  ❌ Undocumented Content-Type: 3\n",[271,3841,3842],{"class":273,"line":833},[271,3843,3844],{},"  ❌ Undocumented HTTP status code: 15\n",[271,3846,3847],{"class":273,"line":838},[271,3848,3849],{},"  ❌ Unsupported methods: 17\n",[271,3851,3852],{"class":273,"line":844},[271,3853,331],{"emptyLinePlaceholder":219},[271,3855,3856],{"class":273,"line":850},[271,3857,3858],{},"Test cases:\n",[271,3860,3861],{"class":273,"line":856},[271,3862,3863],{},"  2860 generated, 59 found 71 unique failures\n",[21,3865,3867],{"id":3866},"restful-booker-authenticated-endpoints","RESTFul Booker (authenticated endpoints)",[11,3869,3870],{},"RESTful Booker has protected endpoints that require authentication. This is why I chose to demo it. All the APIs I test at work have some form of authentication — multiple kinds. Schemathesis will work with basic auth, apiKey, bearer tokens, and custom third party, but you have to configure it (to use authentication). If you skip this step, Schemathesis will hit those endpoints and receive 403 responses, cluttering your results with a bunch of false positives. So, get a token first:",[262,3872,3874],{"className":3473,"code":3873,"language":3475,"meta":202,"style":202},"curl -X POST https:\u002F\u002Frestful-booker.herokuapp.com\u002Fauth \\\n  -H \"Content-Type: application\u002Fjson\" \\\n  -d '{\"username\": \"admin\", \"password\": \"password123\"}'\n",[86,3875,3876,3894,3910],{"__ignoreMap":202},[271,3877,3878,3881,3884,3887,3890],{"class":273,"line":274},[271,3879,3880],{"class":3482},"curl",[271,3882,3883],{"class":3494}," -X",[271,3885,3886],{"class":292}," POST",[271,3888,3889],{"class":292}," https:\u002F\u002Frestful-booker.herokuapp.com\u002Fauth",[271,3891,3893],{"class":3892},"sVYJ7"," \\\n",[271,3895,3896,3899,3902,3905,3908],{"class":273,"line":203},[271,3897,3898],{"class":3494},"  -H",[271,3900,3901],{"class":288}," \"",[271,3903,3904],{"class":292},"Content-Type: application\u002Fjson",[271,3906,3907],{"class":288},"\"",[271,3909,3893],{"class":3892},[271,3911,3912,3915,3917,3920],{"class":273,"line":328},[271,3913,3914],{"class":3494},"  -d",[271,3916,289],{"class":288},[271,3918,3919],{"class":292},"{\"username\": \"admin\", \"password\": \"password123\"}",[271,3921,3922],{"class":288},"'\n",[11,3924,3925,3926,3930,3931,3934],{},"That returns a token you pass as a cookie on the Schemathesis run. RESTful Booker doesn't publish an official OpenAPI spec — we're using a ",[15,3927],{"href":3928,"text":3929},"https:\u002F\u002Fgithub.com\u002Ftexttest\u002Frestful-booker","community-maintained spec"," hosted here for reliability. The ",[86,3932,3933],{},"--base-url"," flag tells Schemathesis where to send the actual API requests, independent of where the spec is hosted:",[262,3936,3938],{"className":3473,"code":3937,"language":3475,"meta":202,"style":202},"uvx schemathesis run https:\u002F\u002Fwww.davidmello.com\u002Fspecs\u002Frestful-booker.swagger.json \\\n  --base-url https:\u002F\u002Frestful-booker.herokuapp.com \\\n  --checks all \\\n  --header \"Cookie: token=\u003Cyour-token>\"\n",[86,3939,3940,3953,3963,3973],{"__ignoreMap":202},[271,3941,3942,3944,3946,3948,3951],{"class":273,"line":274},[271,3943,3462],{"class":3482},[271,3945,3485],{"class":292},[271,3947,3488],{"class":292},[271,3949,3950],{"class":292}," https:\u002F\u002Fwww.davidmello.com\u002Fspecs\u002Frestful-booker.swagger.json",[271,3952,3893],{"class":3892},[271,3954,3955,3958,3961],{"class":273,"line":203},[271,3956,3957],{"class":3494},"  --base-url",[271,3959,3960],{"class":292}," https:\u002F\u002Frestful-booker.herokuapp.com",[271,3962,3893],{"class":3892},[271,3964,3965,3968,3971],{"class":273,"line":328},[271,3966,3967],{"class":3494},"  --checks",[271,3969,3970],{"class":292}," all",[271,3972,3893],{"class":3892},[271,3974,3975,3978,3980,3983],{"class":273,"line":334},[271,3976,3977],{"class":3494},"  --header",[271,3979,3901],{"class":288},[271,3981,3982],{"class":292},"Cookie: token=\u003Cyour-token>",[271,3984,3985],{"class":288},"\"\n",[11,3987,3988],{},"That single command reads the spec, generates test cases for every endpoint — authenticated and unauthenticated — sends them to the live API, and reports any spec violations or server errors it finds.",[11,3990,3991],{},"You, like me, might be thinking at this point, \"That seems annoyingly manual to require a separate command to get the token then copy it to the header collection in the Schemathesis command.\" Or, \"What if my tokens have a short shelf life and it expires before my test completes?\" Fortunately, Schemathesis has an answer for both of these built in.",[11,3993,3994,3995,3998],{},"Create a ",[86,3996,3997],{},"schemathesis_hooks.py"," file in the same directory you'll run Schemathesis from:",[262,4000,4004],{"className":4001,"code":4002,"filename":3997,"language":4003,"meta":202,"style":202},"language-python shiki shiki-themes material-theme-lighter github-light-high-contrast github-dark-high-contrast","import requests\nimport schemathesis\n\nTOKEN = None\n\n@schemathesis.hook\ndef before_call(context, case, request):\n    global TOKEN\n    \n    if TOKEN is None:\n        # Restful Booker requires Content-Type: application\u002Fjson\n        response = requests.post(\n            \"https:\u002F\u002Frestful-booker.herokuapp.com\u002Fauth\",\n            headers={\"Content-Type\": \"application\u002Fjson\"},\n            json={\"username\": \"admin\", \"password\": \"password123\"}\n        )\n        \n        if response.status_code == 200:\n            data = response.json()\n            TOKEN = data.get(\"token\")\n            if TOKEN:\n                print(f\"\\n✅ Auth Successful! Token: {TOKEN}\")\n            else:\n                # Sometimes it returns {\"reason\": \"Bad credentials\"} with 200 OK\n                print(f\"\\n❌ Auth Failed. Response: {data}\")\n        else:\n            print(f\"\\n❌ Auth HTTP Error {response.status_code}: {response.text}\")\n\n    if TOKEN:\n        # Inject into the case headers for Schemathesis to use\n        case.headers[\"Cookie\"] = f\"token={TOKEN}\"\n","python",[86,4005,4006,4011,4016,4020,4025,4029,4034,4039,4044,4049,4054,4059,4064,4069,4074,4079,4084,4089,4094,4099,4104,4109,4114,4119,4124,4129,4134,4139,4143,4148,4153],{"__ignoreMap":202},[271,4007,4008],{"class":273,"line":274},[271,4009,4010],{},"import requests\n",[271,4012,4013],{"class":273,"line":203},[271,4014,4015],{},"import schemathesis\n",[271,4017,4018],{"class":273,"line":328},[271,4019,331],{"emptyLinePlaceholder":219},[271,4021,4022],{"class":273,"line":334},[271,4023,4024],{},"TOKEN = None\n",[271,4026,4027],{"class":273,"line":355},[271,4028,331],{"emptyLinePlaceholder":219},[271,4030,4031],{"class":273,"line":372},[271,4032,4033],{},"@schemathesis.hook\n",[271,4035,4036],{"class":273,"line":389},[271,4037,4038],{},"def before_call(context, case, request):\n",[271,4040,4041],{"class":273,"line":395},[271,4042,4043],{},"    global TOKEN\n",[271,4045,4046],{"class":273,"line":400},[271,4047,4048],{},"    \n",[271,4050,4051],{"class":273,"line":417},[271,4052,4053],{},"    if TOKEN is None:\n",[271,4055,4056],{"class":273,"line":425},[271,4057,4058],{},"        # Restful Booker requires Content-Type: application\u002Fjson\n",[271,4060,4061],{"class":273,"line":467},[271,4062,4063],{},"        response = requests.post(\n",[271,4065,4066],{"class":273,"line":472},[271,4067,4068],{},"            \"https:\u002F\u002Frestful-booker.herokuapp.com\u002Fauth\",\n",[271,4070,4071],{"class":273,"line":501},[271,4072,4073],{},"            headers={\"Content-Type\": \"application\u002Fjson\"},\n",[271,4075,4076],{"class":273,"line":538},[271,4077,4078],{},"            json={\"username\": \"admin\", \"password\": \"password123\"}\n",[271,4080,4081],{"class":273,"line":544},[271,4082,4083],{},"        )\n",[271,4085,4086],{"class":273,"line":549},[271,4087,4088],{},"        \n",[271,4090,4091],{"class":273,"line":602},[271,4092,4093],{},"        if response.status_code == 200:\n",[271,4095,4096],{"class":273,"line":620},[271,4097,4098],{},"            data = response.json()\n",[271,4100,4101],{"class":273,"line":627},[271,4102,4103],{},"            TOKEN = data.get(\"token\")\n",[271,4105,4106],{"class":273,"line":633},[271,4107,4108],{},"            if TOKEN:\n",[271,4110,4111],{"class":273,"line":639},[271,4112,4113],{},"                print(f\"\\n✅ Auth Successful! Token: {TOKEN}\")\n",[271,4115,4116],{"class":273,"line":822},[271,4117,4118],{},"            else:\n",[271,4120,4121],{"class":273,"line":827},[271,4122,4123],{},"                # Sometimes it returns {\"reason\": \"Bad credentials\"} with 200 OK\n",[271,4125,4126],{"class":273,"line":833},[271,4127,4128],{},"                print(f\"\\n❌ Auth Failed. Response: {data}\")\n",[271,4130,4131],{"class":273,"line":838},[271,4132,4133],{},"        else:\n",[271,4135,4136],{"class":273,"line":844},[271,4137,4138],{},"            print(f\"\\n❌ Auth HTTP Error {response.status_code}: {response.text}\")\n",[271,4140,4141],{"class":273,"line":850},[271,4142,331],{"emptyLinePlaceholder":219},[271,4144,4145],{"class":273,"line":856},[271,4146,4147],{},"    if TOKEN:\n",[271,4149,4150],{"class":273,"line":862},[271,4151,4152],{},"        # Inject into the case headers for Schemathesis to use\n",[271,4154,4155],{"class":273,"line":867},[271,4156,4157],{},"        case.headers[\"Cookie\"] = f\"token={TOKEN}\"\n",[11,4159,4160,4161,4164],{},"Then create a ",[86,4162,4163],{},"schemathesis.toml"," in the same directory to wire the hook in automatically:",[262,4166,4170],{"className":4167,"code":4168,"filename":4163,"language":4169,"meta":202,"style":202},"language-toml shiki shiki-themes material-theme-lighter github-light-high-contrast github-dark-high-contrast","hooks = \"schemathesis_hooks\"\nbase-url = \"https:\u002F\u002Frestful-booker.herokuapp.com\"\n","toml",[86,4171,4172,4177],{"__ignoreMap":202},[271,4173,4174],{"class":273,"line":274},[271,4175,4176],{},"hooks = \"schemathesis_hooks\"\n",[271,4178,4179],{"class":273,"line":203},[271,4180,4181],{},"base-url = \"https:\u002F\u002Frestful-booker.herokuapp.com\"\n",[11,4183,4184],{},[32,4185,4186],{},"Note we are also leveraging the base-url setting in the toml file so we don't need to send that in the command line either now",[11,4188,4189,4190,4193],{},"With both files in place, drop the manual ",[86,4191,4192],{},"--header"," flag — the hook handles authentication for every request automatically:",[262,4195,4197],{"className":3473,"code":4196,"language":3475,"meta":202,"style":202},"uvx schemathesis run https:\u002F\u002Fwww.davidmello.com\u002Fspecs\u002Frestful-booker.swagger.json --checks all\n",[86,4198,4199],{"__ignoreMap":202},[271,4200,4201,4203,4205,4207,4209,4211],{"class":273,"line":274},[271,4202,3462],{"class":3482},[271,4204,3485],{"class":292},[271,4206,3488],{"class":292},[271,4208,3950],{"class":292},[271,4210,3495],{"class":3494},[271,4212,3498],{"class":292},[11,4214,4215],{},"Before running a full test, it's worth validating the spec first to catch any issues before sending a full request set:",[262,4217,4219],{"className":3473,"code":4218,"language":3475,"meta":202,"style":202},"uvx schemathesis run https:\u002F\u002Fwww.davidmello.com\u002Fspecs\u002Frestful-booker.swagger.json --max-examples=1\n",[86,4220,4221],{"__ignoreMap":202},[271,4222,4223,4225,4227,4229,4231],{"class":273,"line":274},[271,4224,3462],{"class":3482},[271,4226,3485],{"class":292},[271,4228,3488],{"class":292},[271,4230,3950],{"class":292},[271,4232,4233],{"class":3494}," --max-examples=1\n",[262,4235,4238],{"className":3507,"code":4236,"filename":4237,"language":3510,"meta":202,"style":202},"Schemathesis v4.12.0\n━━━━━━━━━━━━━━━━━━━━\n\n ✅  Loaded specification from restful-booker.swagger.json (in 0.13s)\n\n     Base URL:         https:\u002F\u002Frestful-booker.herokuapp.com\n     Specification:    Open API 2.0\n     Operations:       8 selected \u002F 8 total\n     Configuration:    schemathesis.toml\n\n ✅  API capabilities:\n\n     Supports NULL byte in headers:    ✘\n\n✅ Auth Successful! Token: ec3389fa87e2c05\n\n ❌  Coverage (in 4.55s)\n\n     ❌ 8 failed\n\n ❌  Fuzzing (in 0.84s)\n\n     ✅ 4 passed  ❌ 4 failed\n\n ❌  Stateful (in 0.80s)\n\n     Scenarios:    4\n     API Links:    0 covered \u002F 16 selected \u002F 16 total (16 inferred)\n\n     ✅ 2 passed  ❌ 2 failed\n","RESTful Booker constrained run",[86,4239,4240,4244,4248,4252,4257,4261,4266,4270,4275,4280,4284,4288,4292,4296,4300,4305,4309,4314,4318,4323,4327,4332,4336,4341,4345,4350,4354,4359,4364,4368],{"__ignoreMap":202},[271,4241,4242],{"class":273,"line":274},[271,4243,3517],{},[271,4245,4246],{"class":273,"line":203},[271,4247,3522],{},[271,4249,4250],{"class":273,"line":328},[271,4251,331],{"emptyLinePlaceholder":219},[271,4253,4254],{"class":273,"line":334},[271,4255,4256],{}," ✅  Loaded specification from restful-booker.swagger.json (in 0.13s)\n",[271,4258,4259],{"class":273,"line":355},[271,4260,331],{"emptyLinePlaceholder":219},[271,4262,4263],{"class":273,"line":372},[271,4264,4265],{},"     Base URL:         https:\u002F\u002Frestful-booker.herokuapp.com\n",[271,4267,4268],{"class":273,"line":389},[271,4269,3545],{},[271,4271,4272],{"class":273,"line":395},[271,4273,4274],{},"     Operations:       8 selected \u002F 8 total\n",[271,4276,4277],{"class":273,"line":400},[271,4278,4279],{},"     Configuration:    schemathesis.toml\n",[271,4281,4282],{"class":273,"line":417},[271,4283,331],{"emptyLinePlaceholder":219},[271,4285,4286],{"class":273,"line":425},[271,4287,3559],{},[271,4289,4290],{"class":273,"line":467},[271,4291,331],{"emptyLinePlaceholder":219},[271,4293,4294],{"class":273,"line":472},[271,4295,3568],{},[271,4297,4298],{"class":273,"line":501},[271,4299,331],{"emptyLinePlaceholder":219},[271,4301,4302],{"class":273,"line":538},[271,4303,4304],{},"✅ Auth Successful! Token: ec3389fa87e2c05\n",[271,4306,4307],{"class":273,"line":544},[271,4308,331],{"emptyLinePlaceholder":219},[271,4310,4311],{"class":273,"line":549},[271,4312,4313],{}," ❌  Coverage (in 4.55s)\n",[271,4315,4316],{"class":273,"line":602},[271,4317,331],{"emptyLinePlaceholder":219},[271,4319,4320],{"class":273,"line":620},[271,4321,4322],{},"     ❌ 8 failed\n",[271,4324,4325],{"class":273,"line":627},[271,4326,331],{"emptyLinePlaceholder":219},[271,4328,4329],{"class":273,"line":633},[271,4330,4331],{}," ❌  Fuzzing (in 0.84s)\n",[271,4333,4334],{"class":273,"line":639},[271,4335,331],{"emptyLinePlaceholder":219},[271,4337,4338],{"class":273,"line":822},[271,4339,4340],{},"     ✅ 4 passed  ❌ 4 failed\n",[271,4342,4343],{"class":273,"line":827},[271,4344,331],{"emptyLinePlaceholder":219},[271,4346,4347],{"class":273,"line":833},[271,4348,4349],{}," ❌  Stateful (in 0.80s)\n",[271,4351,4352],{"class":273,"line":838},[271,4353,331],{"emptyLinePlaceholder":219},[271,4355,4356],{"class":273,"line":844},[271,4357,4358],{},"     Scenarios:    4\n",[271,4360,4361],{"class":273,"line":850},[271,4362,4363],{},"     API Links:    0 covered \u002F 16 selected \u002F 16 total (16 inferred)\n",[271,4365,4366],{"class":273,"line":856},[271,4367,331],{"emptyLinePlaceholder":219},[271,4369,4370],{"class":273,"line":862},[271,4371,4372],{},"     ✅ 2 passed  ❌ 2 failed\n",[11,4374,4375],{},"A couple of notable failures from the run:",[262,4377,4380],{"className":3507,"code":4378,"filename":4379,"language":3510,"meta":202,"style":202},"__________________________________________________________________________________________________ GET \u002Fbooking __________________________________________________________________________________________________\n1. Test Case ID: myEjmx\n\n- Server error\n- Undocumented Content-Type\n\n    Received: text\u002Fplain; charset=utf-8\n    Documented: application\u002Fjson\n\n- Undocumented HTTP status code\n\n    Received: 500\n    Documented: 200, 400\n\n[500] Internal Server Error:\n\n    `Internal Server Error`\n\nReproduce with:\n\n    curl -X GET -H 'Cookie: [Filtered]' 'https:\u002F\u002Frestful-booker.herokuapp.com\u002Fbooking?firstname=null&checkin=null&checkout=null'\n","Failure Detail — Server Error",[86,4381,4382,4387,4392,4396,4400,4405,4409,4414,4419,4423,4428,4432,4437,4442,4446,4450,4454,4459,4463,4467,4471],{"__ignoreMap":202},[271,4383,4384],{"class":273,"line":274},[271,4385,4386],{},"__________________________________________________________________________________________________ GET \u002Fbooking __________________________________________________________________________________________________\n",[271,4388,4389],{"class":273,"line":203},[271,4390,4391],{},"1. Test Case ID: myEjmx\n",[271,4393,4394],{"class":273,"line":328},[271,4395,331],{"emptyLinePlaceholder":219},[271,4397,4398],{"class":273,"line":334},[271,4399,3677],{},[271,4401,4402],{"class":273,"line":355},[271,4403,4404],{},"- Undocumented Content-Type\n",[271,4406,4407],{"class":273,"line":372},[271,4408,331],{"emptyLinePlaceholder":219},[271,4410,4411],{"class":273,"line":389},[271,4412,4413],{},"    Received: text\u002Fplain; charset=utf-8\n",[271,4415,4416],{"class":273,"line":395},[271,4417,4418],{},"    Documented: application\u002Fjson\n",[271,4420,4421],{"class":273,"line":400},[271,4422,331],{"emptyLinePlaceholder":219},[271,4424,4425],{"class":273,"line":417},[271,4426,4427],{},"- Undocumented HTTP status code\n",[271,4429,4430],{"class":273,"line":425},[271,4431,331],{"emptyLinePlaceholder":219},[271,4433,4434],{"class":273,"line":467},[271,4435,4436],{},"    Received: 500\n",[271,4438,4439],{"class":273,"line":472},[271,4440,4441],{},"    Documented: 200, 400\n",[271,4443,4444],{"class":273,"line":501},[271,4445,331],{"emptyLinePlaceholder":219},[271,4447,4448],{"class":273,"line":538},[271,4449,3686],{},[271,4451,4452],{"class":273,"line":544},[271,4453,331],{"emptyLinePlaceholder":219},[271,4455,4456],{"class":273,"line":549},[271,4457,4458],{},"    `Internal Server Error`\n",[271,4460,4461],{"class":273,"line":602},[271,4462,331],{"emptyLinePlaceholder":219},[271,4464,4465],{"class":273,"line":620},[271,4466,3704],{},[271,4468,4469],{"class":273,"line":627},[271,4470,331],{"emptyLinePlaceholder":219},[271,4472,4473],{"class":273,"line":633},[271,4474,4475],{},"    curl -X GET -H 'Cookie: [Filtered]' 'https:\u002F\u002Frestful-booker.herokuapp.com\u002Fbooking?firstname=null&checkin=null&checkout=null'\n",[262,4477,4480],{"className":3507,"code":4478,"filename":4479,"language":3510,"meta":202,"style":202},"_______________________________________________________________________________________________ PUT \u002Fbooking\u002F{id} ________________________________________________________________________________________________\n1. Test Case ID: yJU461\n\n- API rejected schema-compliant request\n\n    Valid data should have been accepted\n    Expected: 2xx, 401, 403, 404, 409, 5xx\n\n[400] Bad Request:\n\n    `Bad Request`\n\nReproduce with:\n\n    curl -X PUT -H 'Authorization: [Filtered]' -H 'Cookie: [Filtered]' -H 'Content-Type: application\u002Fjson' -d '{}' https:\u002F\u002Frestful-booker.herokuapp.com\u002Fbooking\u002F1\n","Failure Detail — Schema Violation",[86,4481,4482,4487,4492,4496,4501,4505,4510,4515,4519,4524,4528,4533,4537,4541,4545],{"__ignoreMap":202},[271,4483,4484],{"class":273,"line":274},[271,4485,4486],{},"_______________________________________________________________________________________________ PUT \u002Fbooking\u002F{id} ________________________________________________________________________________________________\n",[271,4488,4489],{"class":273,"line":203},[271,4490,4491],{},"1. Test Case ID: yJU461\n",[271,4493,4494],{"class":273,"line":328},[271,4495,331],{"emptyLinePlaceholder":219},[271,4497,4498],{"class":273,"line":334},[271,4499,4500],{},"- API rejected schema-compliant request\n",[271,4502,4503],{"class":273,"line":355},[271,4504,331],{"emptyLinePlaceholder":219},[271,4506,4507],{"class":273,"line":372},[271,4508,4509],{},"    Valid data should have been accepted\n",[271,4511,4512],{"class":273,"line":389},[271,4513,4514],{},"    Expected: 2xx, 401, 403, 404, 409, 5xx\n",[271,4516,4517],{"class":273,"line":395},[271,4518,331],{"emptyLinePlaceholder":219},[271,4520,4521],{"class":273,"line":400},[271,4522,4523],{},"[400] Bad Request:\n",[271,4525,4526],{"class":273,"line":417},[271,4527,331],{"emptyLinePlaceholder":219},[271,4529,4530],{"class":273,"line":425},[271,4531,4532],{},"    `Bad Request`\n",[271,4534,4535],{"class":273,"line":467},[271,4536,331],{"emptyLinePlaceholder":219},[271,4538,4539],{"class":273,"line":472},[271,4540,3704],{},[271,4542,4543],{"class":273,"line":501},[271,4544,331],{"emptyLinePlaceholder":219},[271,4546,4547],{"class":273,"line":538},[271,4548,4549],{},"    curl -X PUT -H 'Authorization: [Filtered]' -H 'Cookie: [Filtered]' -H 'Content-Type: application\u002Fjson' -d '{}' https:\u002F\u002Frestful-booker.herokuapp.com\u002Fbooking\u002F1\n",[262,4551,4553],{"className":3507,"code":4552,"filename":3720,"language":3510,"meta":202,"style":202},"API Operations:\n  Selected: 8\u002F8\n  Tested: 8\n\nFailures:\n  ❌ Server error: 2\n  ❌ API accepted schema-violating request: 1\n  ❌ API rejected schema-compliant request: 2\n  ❌ Missing header not rejected: 1\n  ❌ Undocumented Content-Type: 5\n  ❌ Undocumented HTTP status code: 4\n  ❌ Unsupported methods: 6\n\nTest cases:\n  27 generated, 12 found 21 unique failures in 16.74s\n",[86,4554,4555,4559,4564,4569,4573,4577,4582,4587,4592,4597,4602,4607,4612,4616,4620],{"__ignoreMap":202},[271,4556,4557],{"class":273,"line":274},[271,4558,3736],{},[271,4560,4561],{"class":273,"line":203},[271,4562,4563],{},"  Selected: 8\u002F8\n",[271,4565,4566],{"class":273,"line":328},[271,4567,4568],{},"  Tested: 8\n",[271,4570,4571],{"class":273,"line":334},[271,4572,331],{"emptyLinePlaceholder":219},[271,4574,4575],{"class":273,"line":355},[271,4576,3784],{},[271,4578,4579],{"class":273,"line":372},[271,4580,4581],{},"  ❌ Server error: 2\n",[271,4583,4584],{"class":273,"line":389},[271,4585,4586],{},"  ❌ API accepted schema-violating request: 1\n",[271,4588,4589],{"class":273,"line":395},[271,4590,4591],{},"  ❌ API rejected schema-compliant request: 2\n",[271,4593,4594],{"class":273,"line":400},[271,4595,4596],{},"  ❌ Missing header not rejected: 1\n",[271,4598,4599],{"class":273,"line":417},[271,4600,4601],{},"  ❌ Undocumented Content-Type: 5\n",[271,4603,4604],{"class":273,"line":425},[271,4605,4606],{},"  ❌ Undocumented HTTP status code: 4\n",[271,4608,4609],{"class":273,"line":467},[271,4610,4611],{},"  ❌ Unsupported methods: 6\n",[271,4613,4614],{"class":273,"line":472},[271,4615,331],{"emptyLinePlaceholder":219},[271,4617,4618],{"class":273,"line":501},[271,4619,3858],{},[271,4621,4622],{"class":273,"line":538},[271,4623,4624],{},"  27 generated, 12 found 21 unique failures in 16.74s\n",[21,4626,4628],{"id":4627},"what-it-found-that-i-wouldnt-have-tested","What It Found That I Wouldn't Have Tested",[11,4630,4631],{},"Where Schemathesis shines is the coverage it delivers without you having to think about what to test. A few findings from the Petstore run stood out:",[11,4633,4634,4637],{},[40,4635,4636],{},"15 undocumented HTTP status codes"," — across 20 endpoints. Manually writing tests to catch undocumented response codes would require exhaustively hitting every endpoint with every possible input variation. Nobody does that. Schemathesis does it automatically.",[11,4639,4640,4643],{},[40,4641,4642],{},"\"Use after free\""," — this is actually a good example of a scenario many people don't look for, but it's something I check and teach others to do. In API testing terms it means Schemathesis successfully retrieved or interacted with a resource after it had been deleted — a POST to create, DELETE to remove, then a GET that still returned data. If it's deleted, you shouldn't be able to GET it. Often, people forget to do stateful testing like this, but that's where the really fun bugs live. Further, many of the competing spec testing tools omit stateful testing altogether. It's also worth noting this is an area where spec-driven testing tools like Schemathesis have a clear edge over hand-written test suites.",[11,4645,4646,4649],{},[40,4647,4648],{},"9 instances of the API accepting schema-violating requests"," — the API happily accepted inputs that its own spec said it should reject. This is the kind of silent validation gap that can lead to corrupt data or unexpected behavior in production — or user confusion. Without property-based testing generating invalid inputs systematically, these would be invisible.",[11,4651,4652],{},"In a real-world engagement with limited time I might have applied a risk-based approach and never gotten to these. Schemathesis found all of them in under 20 seconds and the full petstore run in under 1 minute.",[21,4654,4656],{"id":4655},"what-i-liked","What I Liked",[889,4658,4659,4662,4668,4671],{},[666,4660,4661],{},"Against RESTful Booker I was able to cover the entire swagger file running 623 scenarios in under 2 minutes, finding 26 defects.",[666,4663,4664,4665,4667],{},"Very readable numbered test failures that include a title, description, expected vs. actual response — and a ",[86,4666,3880],{}," command to reproduce the exact failure, making it easy to hand off directly to a developer.",[666,4669,4670],{},"Ability to export results to jUnit, HAR, VCR, and NDJSON — making it easy to plug into CI pipelines or share with developers.",[666,4672,4673],{},"This paired nicely with Claude and an ADO MCP server to parse and deduplicate the failures by category and automatically file defect reports in Azure DevOps.",[262,4675,4678],{"className":3507,"code":4676,"filename":4677,"language":3510,"meta":202,"style":202},"2. Test Case ID: WjVfwK\n\n- API rejected schema-compliant request\n\n    Valid data should have been accepted\n    Expected: 2xx, 401, 403, 404, 409, 5xx\n\n[400] Bad Request:\n\n    `Bad Request`\n\nReproduce with:\n\n    curl -X PUT -H 'Authorization: [Filtered]' -H 'Cookie: [Filtered]' -H 'Content-Type: application\u002Fjson' -d '{}' https:\u002F\u002Frestful-booker.herokuapp.com\u002Fbooking\u002F1\n","Specific test failure snippet",[86,4679,4680,4685,4689,4693,4697,4701,4705,4709,4713,4717,4721,4725,4729,4733],{"__ignoreMap":202},[271,4681,4682],{"class":273,"line":274},[271,4683,4684],{},"2. Test Case ID: WjVfwK\n",[271,4686,4687],{"class":273,"line":203},[271,4688,331],{"emptyLinePlaceholder":219},[271,4690,4691],{"class":273,"line":328},[271,4692,4500],{},[271,4694,4695],{"class":273,"line":334},[271,4696,331],{"emptyLinePlaceholder":219},[271,4698,4699],{"class":273,"line":355},[271,4700,4509],{},[271,4702,4703],{"class":273,"line":372},[271,4704,4514],{},[271,4706,4707],{"class":273,"line":389},[271,4708,331],{"emptyLinePlaceholder":219},[271,4710,4711],{"class":273,"line":395},[271,4712,4523],{},[271,4714,4715],{"class":273,"line":400},[271,4716,331],{"emptyLinePlaceholder":219},[271,4718,4719],{"class":273,"line":417},[271,4720,4532],{},[271,4722,4723],{"class":273,"line":425},[271,4724,331],{"emptyLinePlaceholder":219},[271,4726,4727],{"class":273,"line":467},[271,4728,3704],{},[271,4730,4731],{"class":273,"line":472},[271,4732,331],{"emptyLinePlaceholder":219},[271,4734,4735],{"class":273,"line":501},[271,4736,4549],{},[11,4738,4739,4740,4743,4744,4747,4748,4751,4752,4755],{},"For example, in the above failure, the test indicates we received a ",[86,4741,4742],{},"400 Bad Request"," instead of a ",[86,4745,4746],{},"200"," OK\u002FAccepted level response and, per the schema, the value sent, ",[86,4749,4750],{},"{}",", should have been an acceptable PUT option. The title ",[86,4753,4754],{},"API rejected schema-compliant request"," immediately conveys that.",[21,4757,4759],{"id":4758},"what-surprised-me-or-needs-work","What Surprised Me or Needs Work",[11,4761,4762,4763,89,4766,4769],{},"I noticed repeated back-to-back test runs would generate a different number of test cases, which was confusing at first. Schemathesis uses random seeding to generate different inputs from run to run (you can lock this down for reproducible runs via ",[86,4764,4765],{},"--seed",[86,4767,4768],{},"--generation-deterministic"," for even stronger determinism), and it chains requests during stateful testing based on what the API returns — so if the API responds differently, it may explore a different set of scenarios.",[11,4771,4772,4773,117,4775,4777,4778,4781,4782,4785,4786,4789],{},"There are two similar settings to help with the determinism if needed; ",[86,4774,4765],{},[86,4776,4768],{},". ",[86,4779,4780],{},"seed"," lets you rerun the tests under the same seed e.g. ",[86,4783,4784],{},"--seed 42"," which helps run with the same randomness between repeated runs. ",[86,4787,4788],{},"generation-deterministic"," encompasses the seed and also ensures environment consistency and phase control. When to use either:",[3192,4791,4792,4805],{},[3195,4793,4794],{},[3198,4795,4796,4801],{},[3201,4797,4798],{},[86,4799,4800],{},"--seed INT",[3201,4802,4803],{},[86,4804,4768],{},[3217,4806,4807,4815,4823,4831],{},[3198,4808,4809,4812],{},[3222,4810,4811],{},"Debugging specific test runs",[3222,4813,4814],{},"Better for CI runs (ensures same test scenarios)",[3198,4816,4817,4820],{},[3222,4818,4819],{},"Consistent on same machine\u002Fworkstation",[3222,4821,4822],{},"Consistent on any machine\u002Fworkstation",[3198,4824,4825,4828],{},[3222,4826,4827],{},"Replays tests with same values for a given seed",[3222,4829,4830],{},"Forces fully deterministic logic across the board",[3198,4832,4833,4836],{},[3222,4834,4835],{},"Use when rerunning to find a specific bug found under a given seed",[3222,4837,4838],{},"Use in CI for stable builds",[11,4840,4841],{},"On the needs-work side — the documentation is overall pretty good, but finding some things was unintuitive without using their site search and some sections could use deeper content.",[21,4843,4845],{"id":4844},"fixing-common-schemathesis-usage-errors","Fixing Common Schemathesis Usage Errors",[2401,4847,4849],{"id":4848},"loading-a-password-protected-openapi-swagger-spec","Loading a Password Protected OpenAPI \u002F Swagger Spec",[11,4851,4852,4853,4856,4857,4860],{},"If your OpenAPI spec is hosted on a staging or internal environment protected by HTTP basic authentication, Schemathesis will fail to load it without credentials. The ",[86,4854,4855],{},"--auth"," flag passes a ",[86,4858,4859],{},"username:password"," pair to authenticate the spec fetch:",[262,4862,4864],{"className":3473,"code":4863,"language":3475,"meta":202,"style":202},"uvx schemathesis run https:\u002F\u002Fyour-internal-api.com\u002Fopenapi.json \\\n  --auth user:pass \\\n  --checks all\n",[86,4865,4866,4879,4889],{"__ignoreMap":202},[271,4867,4868,4870,4872,4874,4877],{"class":273,"line":274},[271,4869,3462],{"class":3482},[271,4871,3485],{"class":292},[271,4873,3488],{"class":292},[271,4875,4876],{"class":292}," https:\u002F\u002Fyour-internal-api.com\u002Fopenapi.json",[271,4878,3893],{"class":3892},[271,4880,4881,4884,4887],{"class":273,"line":203},[271,4882,4883],{"class":3494},"  --auth",[271,4885,4886],{"class":292}," user:pass",[271,4888,3893],{"class":3892},[271,4890,4891,4893],{"class":273,"line":328},[271,4892,3967],{"class":3494},[271,4894,3498],{"class":292},[11,4896,4897,4898,4900,4901,4904,4905,4907],{},"Note that ",[86,4899,4855],{}," authenticates access to the ",[40,4902,4903],{},"spec URL itself"," — it is separate from any authentication your API endpoints require. If your API endpoints also need authentication, you will still need to configure that separately via ",[86,4906,4192],{}," or a hook as covered in the RESTful Booker example above.",[2401,4909,4911],{"id":4910},"schema-loading-error-api-returning-xml-instead-of-json","Schema Loading Error: API Returning XML Instead of JSON",[11,4913,4914,4915,4918,4919,4922],{},"While testing against an internal API, I kept hitting a ",[86,4916,4917],{},"Schema Loading Error"," even though the spec URL loaded fine in the browser. The cause turned out to be a missing ",[86,4920,4921],{},"Accept"," header — without it, the server defaulted to its fallback content type and returned XML instead of JSON, producing a response Schemathesis couldn't parse.",[11,4924,4925,4926,4928,4929,362],{},"In this case you may see an error message ",[86,4927,4917],{}," with reason ",[86,4930,4931],{},"API schema does not appear syntactically valid",[262,4933,4935],{"className":3473,"code":4934,"language":3475,"meta":202,"style":202},"Schemathesis v4.15.2\n━━━━━━━━━━━━━━━━━━━━\n\n ❌  Failed to load specification from https:\u002F\u002Fyour-site\u002Fv2\u002Fapi-docs after 3.00s                                                                                                                                                                                                                               \n                                                                                                                                                                                                                                                                                                                                           \n Schema Loading Error                                                                                                                                                                                                                                                                                                                      \n                                                                                                                                                                                                                                                                                                                                           \n API schema does not appear syntactically valid                                                                                                                                                                                                                                                                                            \n\n     mapping values are not allowed in this context                                                                                                                                                                                                                                                                                        \n       in \"\u003Cunicode string>\", line 1, column 51031 \n",[86,4936,4937,4944,4948,4952,4983,4988,5001,5005,5031,5035,5063],{"__ignoreMap":202},[271,4938,4939,4941],{"class":273,"line":274},[271,4940,3329],{"class":3482},[271,4942,4943],{"class":292}," v4.15.2\n",[271,4945,4946],{"class":273,"line":203},[271,4947,3522],{"class":3482},[271,4949,4950],{"class":273,"line":328},[271,4951,331],{"emptyLinePlaceholder":219},[271,4953,4954,4957,4960,4963,4966,4969,4971,4974,4977,4980],{"class":273,"line":334},[271,4955,4956],{"class":3482}," ❌",[271,4958,4959],{"class":292},"  Failed",[271,4961,4962],{"class":292}," to",[271,4964,4965],{"class":292}," load",[271,4967,4968],{"class":292}," specification",[271,4970,316],{"class":292},[271,4972,4973],{"class":292}," https:\u002F\u002Fyour-site\u002Fv2\u002Fapi-docs",[271,4975,4976],{"class":292}," after",[271,4978,4979],{"class":292}," 3.00s",[271,4981,4982],{"class":281},"                                                                                                                                                                                                                               \n",[271,4984,4985],{"class":273,"line":355},[271,4986,4987],{"class":281},"                                                                                                                                                                                                                                                                                                                                           \n",[271,4989,4990,4993,4996,4998],{"class":273,"line":372},[271,4991,4992],{"class":3482}," Schema",[271,4994,4995],{"class":292}," Loading",[271,4997,510],{"class":292},[271,4999,5000],{"class":281},"                                                                                                                                                                                                                                                                                                                      \n",[271,5002,5003],{"class":273,"line":389},[271,5004,4987],{"class":281},[271,5006,5007,5010,5013,5016,5019,5022,5025,5028],{"class":273,"line":395},[271,5008,5009],{"class":3482}," API",[271,5011,5012],{"class":292}," schema",[271,5014,5015],{"class":292}," does",[271,5017,5018],{"class":292}," not",[271,5020,5021],{"class":292}," appear",[271,5023,5024],{"class":292}," syntactically",[271,5026,5027],{"class":292}," valid",[271,5029,5030],{"class":281},"                                                                                                                                                                                                                                                                                            \n",[271,5032,5033],{"class":273,"line":400},[271,5034,331],{"emptyLinePlaceholder":219},[271,5036,5037,5040,5043,5046,5048,5051,5054,5057,5060],{"class":273,"line":417},[271,5038,5039],{"class":3482},"     mapping",[271,5041,5042],{"class":292}," values",[271,5044,5045],{"class":292}," are",[271,5047,5018],{"class":292},[271,5049,5050],{"class":292}," allowed",[271,5052,5053],{"class":292}," in",[271,5055,5056],{"class":292}," this",[271,5058,5059],{"class":292}," context",[271,5061,5062],{"class":281},"                                                                                                                                                                                                                                                                                        \n",[271,5064,5065,5068,5070,5073,5075],{"class":273,"line":425},[271,5066,5067],{"class":281},"       in ",[271,5069,3907],{"class":288},[271,5071,5072],{"class":292},"\u003Cunicode string>",[271,5074,3907],{"class":288},[271,5076,5077],{"class":281},", line 1, column 51031\n",[11,5079,5080,5081,5084],{},"If you examine the schema in the browser you'll see the root cause where the JSON schema is wrapped in an XML-style ",[86,5082,5083],{},"\u003CJSON>"," tag instead of actual JSON which causes the parsing error.",[262,5086,5090],{"className":5087,"code":5088,"language":5089,"meta":202,"style":202},"language-html shiki shiki-themes material-theme-lighter github-light-high-contrast github-dark-high-contrast","This XML file does not appear to have any style information associated with it. The document tree is shown below.\n\n\u003CJson>{\"swagger\":\"2.0\",\"info\":{\"description\":\"Api Documentation\",\"version\":\"1.0\",\"title\":\"Api Documentation\",\"termsOfService\":\"urn:tos\",\"contact\":{},\"license\":{\"name\":\"Apache 2.0\",\"url\":\"http:\u002F\u002Fwww.apache.org\u002Flicenses\u002FLICENSE-2.0\"}}...\n","html",[86,5091,5092,5097,5101],{"__ignoreMap":202},[271,5093,5094],{"class":273,"line":274},[271,5095,5096],{"class":281},"This XML file does not appear to have any style information associated with it. The document tree is shown below.\n",[271,5098,5099],{"class":273,"line":203},[271,5100,331],{"emptyLinePlaceholder":219},[271,5102,5103,5106,5110,5113],{"class":273,"line":328},[271,5104,5105],{"class":299},"\u003C",[271,5107,5109],{"class":5108},"sb2_1","Json",[271,5111,5112],{"class":299},">",[271,5114,5115],{"class":281},"{\"swagger\":\"2.0\",\"info\":{\"description\":\"Api Documentation\",\"version\":\"1.0\",\"title\":\"Api Documentation\",\"termsOfService\":\"urn:tos\",\"contact\":{},\"license\":{\"name\":\"Apache 2.0\",\"url\":\"http:\u002F\u002Fwww.apache.org\u002Flicenses\u002FLICENSE-2.0\"}}...\n",[11,5117,5118],{},"Passing the header explicitly resolves it:",[262,5120,5122],{"className":3473,"code":5121,"language":3475,"meta":202,"style":202},"uvx schemathesis run https:\u002F\u002Fyour-api.com\u002Fopenapi.json \\\n  -H \"Accept: application\u002Fjson\" \\\n  --checks all\n",[86,5123,5124,5137,5150],{"__ignoreMap":202},[271,5125,5126,5128,5130,5132,5135],{"class":273,"line":274},[271,5127,3462],{"class":3482},[271,5129,3485],{"class":292},[271,5131,3488],{"class":292},[271,5133,5134],{"class":292}," https:\u002F\u002Fyour-api.com\u002Fopenapi.json",[271,5136,3893],{"class":3892},[271,5138,5139,5141,5143,5146,5148],{"class":273,"line":203},[271,5140,3898],{"class":3494},[271,5142,3901],{"class":288},[271,5144,5145],{"class":292},"Accept: application\u002Fjson",[271,5147,3907],{"class":288},[271,5149,3893],{"class":3892},[271,5151,5152,5154],{"class":273,"line":328},[271,5153,3967],{"class":3494},[271,5155,3498],{"class":292},[2337,5157],{},[21,5159,5161],{"id":5160},"pairing-schemathesis-with-ai","Pairing Schemathesis with AI",[11,5163,5164],{},"There are at least two ways to leverage AI to turbocharge Schemathesis that I've thought of and used so far.",[663,5166,5167,5170],{},[666,5168,5169],{},"Parse the output to bucketize errors, create summaries of findings, and automatically file defects — as covered above with Claude and ADO MCP.",[666,5171,5172,5173,5175],{},"Have the AI extract all the ",[86,5174,3880],{}," commands from the output and add them to your regression test suite, or parse the VCR or HAR export to do the same — turning a single Schemathesis run into a reusable, deterministic test suite.",[21,5177,5179],{"id":5178},"who-should-use-this","Who Should Use This",[889,5181,5182,5185,5188,5191,5194],{},[666,5183,5184],{},"Test engineers or developers who have OpenAPI\u002FSwagger or GraphQL APIs with little or no test coverage — get high coverage in minutes without hand-writing a single test case.",[666,5186,5187],{},"Teams in rapid API prototyping phases where the spec is evolving — Schemathesis regenerates coverage automatically as the spec changes, with no test maintenance required.",[666,5189,5190],{},"Teams doing regression testing or pre-release certification — it provides fast coverage and surfaces things you may have missed.",[666,5192,5193],{},"Teams performing a bug bash — those activities don't have the luxury of constructing or wiring up many API tests, and Schemathesis handles that work for you.",[666,5195,5196],{},"Fits naturally into a CI\u002FCD pipeline.",[2401,5198,5200],{"id":5199},"where-it-isnt-a-good-fit","Where it isn't a good fit",[889,5202,5203,5206],{},[666,5204,5205],{},"APIs without OpenAPI specs",[666,5207,5208,5209,5211],{},"Teams that need fully deterministic tests by default (though ",[86,5210,4768],{}," addresses this for CI use cases)",[21,5213,5215],{"id":5214},"conclusion","Conclusion",[11,5217,5218],{},"I've been genuinely impressed by the speed at which Schemathesis covers an API and the quality of defects it surfaces. The stateful testing in particular is a differentiator — it's where the most critical bugs tend to hide, and most competing tools skip it entirely.",[11,5220,5221,5222,5226],{},"There's more to explore in the ",[15,5223],{"href":5224,"text":5225},"https:\u002F\u002Fschemathesis.readthedocs.io\u002Fen\u002Fstable\u002F","official docs"," — advanced filtering, custom checks, and deeper CI integration among them. My next steps include trialing it on larger production-scale APIs and finding more creative ways to pair the output with AI tooling to automate the defect reporting pipeline further.",[11,5228,5229],{},"It only takes a few minutes to set up a run. Point it at one of your APIs and see what it finds — you might be surprised.",[198,5231],{"to":3446},[1137,5233,5234],{},"html .light .shiki span{color:var(--shiki-light);background:var(--shiki-light-bg);font-style:var(--shiki-light-font-style);font-weight:var(--shiki-light-font-weight);text-decoration:var(--shiki-light-text-decoration)}html.light .shiki span{color:var(--shiki-light);background:var(--shiki-light-bg);font-style:var(--shiki-light-font-style);font-weight:var(--shiki-light-font-weight);text-decoration:var(--shiki-light-text-decoration)}html .default .shiki span{color:var(--shiki-default);background:var(--shiki-default-bg);font-style:var(--shiki-default-font-style);font-weight:var(--shiki-default-font-weight);text-decoration:var(--shiki-default-text-decoration)}html .shiki span{color:var(--shiki-default);background:var(--shiki-default-bg);font-style:var(--shiki-default-font-style);font-weight:var(--shiki-default-font-weight);text-decoration:var(--shiki-default-text-decoration)}html .dark .shiki span{color:var(--shiki-dark);background:var(--shiki-dark-bg);font-style:var(--shiki-dark-font-style);font-weight:var(--shiki-dark-font-weight);text-decoration:var(--shiki-dark-text-decoration)}html.dark .shiki span{color:var(--shiki-dark);background:var(--shiki-dark-bg);font-style:var(--shiki-dark-font-style);font-weight:var(--shiki-dark-font-weight);text-decoration:var(--shiki-dark-text-decoration)}html pre.shiki code .sA8fK,html code.shiki .sA8fK{--shiki-light:#E2931D;--shiki-default:#702C00;--shiki-dark:#FFB757}html pre.shiki code .srGNg,html code.shiki .srGNg{--shiki-light:#91B859;--shiki-default:#032563;--shiki-dark:#ADDCFF}html pre.shiki code .spMcu,html code.shiki .spMcu{--shiki-light:#91B859;--shiki-default:#023B95;--shiki-dark:#91CBFF}html pre.shiki code .sVYJ7,html code.shiki .sVYJ7{--shiki-light:#90A4AE;--shiki-default:#A0111F;--shiki-dark:#FF9492}html pre.shiki code .sZi47,html code.shiki .sZi47{--shiki-light:#39ADB5;--shiki-default:#032563;--shiki-dark:#ADDCFF}html pre.shiki code .sZ-rw,html code.shiki .sZ-rw{--shiki-light:#90A4AE;--shiki-default:#0E1116;--shiki-dark:#F0F3F6}html pre.shiki code .sPJuK,html code.shiki .sPJuK{--shiki-light:#39ADB5;--shiki-default:#0E1116;--shiki-dark:#F0F3F6}html pre.shiki code .sb2_1,html code.shiki .sb2_1{--shiki-light:#E53935;--shiki-light-font-style:inherit;--shiki-default:#6E011A;--shiki-default-font-style:italic;--shiki-dark:#FFB1AF;--shiki-dark-font-style:italic}",{"title":202,"searchDepth":203,"depth":203,"links":5236},[5237,5238,5239,5240,5241,5242,5243,5244,5245,5246,5247,5251,5252,5255],{"id":3149,"depth":203,"text":3150},{"id":3173,"depth":203,"text":3174},{"id":3186,"depth":203,"text":3187},{"id":3343,"depth":203,"text":3344},{"id":3417,"depth":203,"text":3418},{"id":3466,"depth":203,"text":3467},{"id":3866,"depth":203,"text":3867},{"id":4627,"depth":203,"text":4628},{"id":4655,"depth":203,"text":4656},{"id":4758,"depth":203,"text":4759},{"id":4844,"depth":203,"text":4845,"children":5248},[5249,5250],{"id":4848,"depth":328,"text":4849},{"id":4910,"depth":328,"text":4911},{"id":5160,"depth":203,"text":5161},{"id":5178,"depth":203,"text":5179,"children":5253},[5254],{"id":5199,"depth":328,"text":5200},{"id":5214,"depth":203,"text":5215},"\u002Fimages\u002Fposts\u002Fautomated-api-testing-with-schemathesis\u002Fautomated-api-testing-with-schemathesis-cover.webp","2026-04-15","Schemathesis automates API testing by generating test cases directly from your OpenAPI spec — covering edge cases, schema violations, and server errors you'd never think of or have the time to write manually.",{},"\u002Fsoftware-testing\u002Ftest-automation\u002Fautomated-api-testing-with-schemathesis",{"title":3135,"description":5258},"software-testing\u002Ftest-automation\u002Fautomated-api-testing-with-schemathesis","Z2tRGHXSfFju1R1FS7L5IaFjkP-zhxYWy2j0jfjkS2I",1788055403707]