[{"data":1,"prerenderedAt":3272},["ShallowReactive",2],{"content:\u002Fsoftware-testing\u002Ftest-automation\u002Frun-fewer-tests-catch-the-same-bugs":3,"category:\u002Fsoftware-testing\u002Ftest-automation\u002Frun-fewer-tests-catch-the-same-bugs":6,"read-next:\u002Fsoftware-testing\u002Ftest-automation\u002Freuse-playwright-tests-for-load-testing,\u002Fsoftware-testing\u002Ftest-automation\u002Fai-performance-testing-guardrails,\u002Fsoftware-testing\u002Ftest-automation\u002Fwhat-would-you-stop-doing-when-ui-tests-are-flaky,\u002Fsoftware-testing\u002Ftest-automation\u002Fstareast-2026-playwright-ai-cost-efficient-testing":637},{"id":4,"title":5,"bmcUsername":6,"body":7,"cover":626,"date":627,"description":628,"draft":629,"extension":630,"features":6,"githubRepo":6,"headline":6,"highlight":6,"icon":6,"meta":631,"navigation":632,"npmPackage":6,"order":6,"path":633,"seo":634,"stem":635,"__hash__":636},"content\u002Fsoftware-testing\u002Ftest-automation\u002Frun-fewer-tests-catch-the-same-bugs.md","Test Selection: Run Fewer Tests, Catch the Same Bugs",null,{"type":8,"value":9,"toc":614},"minimark",[10,14,19,22,31,34,38,41,46,53,59,63,66,84,87,178,181,185,190,195,200,205,208,212,215,218,226,229,233,236,243,250,253,260,263,267,270,273,276,279,287,406,530,536,544,554,557,560,566,577,580,584,587,590,603,606,610],[11,12,13],"p",{},"An 8+ hour CI feedback loop is the kind of pain that makes any promise of faster tests sound appealing, and at a past job the company adopted a black-box predictive test-selection tool that trained on historical pass and fail data to guess which tests a given change was likely to break. While seemingly effective initially, it surfaced a real problem almost immediately, just not the one we were hoping to fix: our UI suite had hidden dependencies, tests relying on shared, warmed-up page state or on data another test had mutated earlier in the run. Predictive test selection, by not always running things in the suite's original order, introduced flakiness that proved difficult to troubleshoot due to different tests running each build. We were never able to fully resolve it, which never allowed us to realize the benefits of predictive test selection.",[15,16,18],"h2",{"id":17},"an-ai-test-selection-framework-that-shows-its-work","An AI Test Selection Framework That Shows Its Work",[11,20,21],{},"That experience made me skeptical of any tool promising to safely trim a test suite, especially one that couldn't explain itself. So when I saw a StarEast 2026 session titled \"Dear AI, Which Tests Should We Run Now?\" promising to go deeper into the actual mechanics behind test selection, I made a point of attending. I wanted to know whether the field had advanced since my own bad experience, and whether I could actually learn the theory behind these approaches instead of trusting a vendor's black box to get it right.",[11,23,24,25,30],{},"The session was presented by ",[26,27],"external-link",{"href":28,"text":29},"https:\u002F\u002Fwww.linkedin.com\u002Fin\u002Felmar-juergens\u002F","Dr. Elmar Jürgens",", co-founder of CQSE GmbH, makers of the TeamScale software-quality-analysis platform, who holds a PhD in software quality analysis himself. His talk covers four distinct approaches to picking a smaller, faster subset of a large test suite that still catches nearly as many bugs as running everything, each with its tradeoffs measured and stated plainly rather than buried inside a product.",[11,32,33],{},"What made this talk earn my trust, in a way my prior experience with an opaque tool never did, is that he shows his work. Every approach comes with a measured tradeoff, a named limitation, and an honest account of when it doesn't apply, instead of a black box you're asked to take on faith. He's also explicit that he doesn't care whether a given approach technically counts as AI: \"I care about whether the approaches find more bugs more quickly.\" That's the right question and I appreciated him being up front about it since speakers at these events tend to just rename their presentation from a past year with this year's trending keywords.",[15,35,37],{"id":36},"four-test-selection-approaches","Four Test Selection Approaches",[11,39,40],{},"Jürgens builds up to four approaches by crossing two questions. The first is which situation you're in.",[42,43,45],"h3",{"id":44},"question-1-do-you-know-what-changed","Question 1: Do You Know What Changed?",[11,47,48,52],{},[49,50,51],"strong",{},"Quality Gate"," means you have a large suite and a small time budget, but you don't know what changed in the software since you last ran tests. One of Jürgens' customers, a German pension fund, ran a 20-hour test suite only on weekends because there wasn't time to run it more often. By the time a build reached that gate, days of accumulated commits sat behind it, no single diff to point to, just a build that needed a yes-or-no answer in a fraction of the time the full suite would take.",[11,54,55,58],{},[49,56,57],{},"Continuous Integration",", by contrast, means you already know exactly what changed, because you have a diff or a pull request in hand. Dolby, another of Jürgens' customers, dealt with a combinatorial explosion of audio bitrate and sample-rate combinations to test and wanted CI feedback on every pull request within 10 minutes, even though their full suite took 3 to 12 hours to run. Each PR came with a specific, known diff to work from, a very different starting point than the pension fund's accumulated pile of weekend changes.",[42,60,62],{"id":61},"question-2-coverage-based-or-content-based-test-selection","Question 2: Coverage-Based or Content-Based Test Selection?",[11,64,65],{},"The second question is how much you're willing to invest in setup to get a better number. Coverage-based approaches require measuring test-case-specific code coverage first, real upfront effort, but they pay it back with a better result. Content-based approaches skip that setup entirely and work directly from the test and change content, in exchange for a slightly worse number. Jürgens presents this as a real choice inside each situation, not a fixed rule: for Quality Gate, coverage-based Pareto Optimization versus content-based AI Test Clustering; for Continuous Integration, coverage-based Test Impact Analysis versus content-based Similarity Scoring. Two situations, two ways to solve each one.",[67,68,69,76],"figure",{},[11,70,71],{},[72,73],"img",{"alt":74,"src":75},"Dr. Elmar Jürgens presenting his four test-selection approaches at StarEast 2026","\u002Fimages\u002Fposts\u002Frun-fewer-tests-catch-the-same-bugs\u002Felmar-juergens-stareast-2026-test-selection-presentation.webp",[77,78,83],"figcaption",{"className":79},[80,81,82],"text-sm","text-muted","mt-2","Jürgens' summary slide: all four approaches, and what percentage of full suite runtime it takes each one to find 90% of the bugs the full suite would find.",[11,85,86],{},"Cross those two questions and you get the four approaches, each measured the same way: what percentage of full suite runtime it takes to find 90% of the bugs the full suite would find. The quick reference, then what each one actually does:",[88,89,90,112],"table",{},[91,92,93],"thead",{},[94,95,96,100,103,106,109],"tr",{},[97,98,99],"th",{},"Approach",[97,101,102],{},"Category",[97,104,105],{},"Use Case",[97,107,108],{},"Runtime Required to Find 90% of Bugs",[97,110,111],{},"Setup Effort",[113,114,115,132,148,163],"tbody",{},[94,116,117,121,124,126,129],{},[118,119,120],"td",{},"Test Impact Analysis",[118,122,123],{},"Coverage-based",[118,125,57],{},[118,127,128],{},"⭐⭐⭐⭐⭐ (2%)",[118,130,131],{},"🕐🕐 about half a year",[94,133,134,137,140,142,145],{},[118,135,136],{},"Similarity Scoring",[118,138,139],{},"Content-based",[118,141,57],{},[118,143,144],{},"⭐⭐⭐⭐☆ (4%)",[118,146,147],{},"🕐 about a day",[94,149,150,153,155,157,160],{},[118,151,152],{},"Pareto Optimization",[118,154,123],{},[118,156,51],{},[118,158,159],{},"⭐⭐⭐☆☆ (11%)",[118,161,162],{},"🕐🕐🕐🕐 about a year",[94,164,165,168,170,172,175],{},[118,166,167],{},"AI Test Clustering",[118,169,139],{},[118,171,51],{},[118,173,174],{},"⭐⭐☆☆☆ (13%)",[118,176,177],{},"🕐 about two days",[11,179,180],{},"Test Impact Analysis (TIA) is the most speed optimized, but Similarity Scoring gets you close in about a day instead of half a year.",[42,182,184],{"id":183},"predictive-test-selection-approaches-explained","Predictive Test Selection Approaches Explained",[11,186,187,189],{},[49,188,152],{}," measures per-test code coverage for the whole suite, then greedily orders tests by how much previously-uncovered code each one covers per second of execution time, cutting off once you hit your time budget. The clearest illustration of why this beats running tests in whatever order they were written came from PixelitOr, an open-source paint program Jürgens uses as a fully visualizable example. Running four UI tests in sequence, Gaussian Blur, Motion Blur, Lens Blur, Smart Blur, each new test lights up progressively less new code and re-covers more of what the earlier blur tests already exercised. His team checked whether that redundant coverage was actually worthless by tracking real and injected bugs: if a bug lives in that repeated code, typically either all of those blur tests find it or none of them do. Running all four buys you almost nothing over running one, and a greedy, coverage-per-second reordering naturally spreads the budget across dissimilar tests instead of exhausting it inside one redundant cluster.",[11,191,192,194],{},[49,193,167],{}," gets you a similar diversity-driven selection without needing any coverage data at all. Every test gets represented as a point in a high-dimensional vector space, generated from a large language model's embedding of the test's actual content, its code if it's automated, or even a plain-English Given\u002FWhen\u002FThen description if it's a manual test case. Tests that exercise similar functionality land close together in that space; tests that do genuinely different things land far apart. The selection algorithm then greedily picks whichever remaining test is furthest from everything already chosen, which naturally avoids getting stuck resampling one redundant cluster the way running tests in file order would. Jürgens showed this working on a 3D projection of around 2,000 real customer test cases: the tests visibly clustered into dense groups, and the selected subset landed inside essentially every visible cluster rather than missing whole regions.",[11,196,197,199],{},[49,198,120],{}," exploits something the other two approaches don't have access to: you already know exactly what changed, because you have a diff or a PR in hand. Combine that with coverage data recorded from prior test runs, and you can directly compute which historically-recorded tests actually execute the changed lines, then skip everything that provably can't be affected by this specific change. On a real change to Jürgens' own team's codebase, out of 5,000 automated tests, only 4 actually executed the changed code. The other 4,996 categorically could not have caught a bug introduced by that change, because they never touch it.",[11,201,202,204],{},[49,203,136],{}," gets a comparable result to Test Impact Analysis without needing coverage data, by treating test selection like a search engine query. Every test's content gets indexed into a document database, the way a search engine indexes web pages. A code change then becomes the search query, built from the changed identifiers plus their surrounding context, not just the raw diff lines. Retrieval works conceptually like TF-IDF (term frequency, inverse document frequency), the same scoring method search engines use: a test scores higher the more it mentions terms that appear in the change, weighted down for terms so common across the whole suite that they don't tell you much. In Jürgens' worked example, a change to a bank transfer method matched correctly against a Cucumber test, a Robot Framework test, and even a plain manual test description, three completely different formats, all identified as relevant through shared identifiers alone.",[11,206,207],{},"Counterintuitively, both AI Test Clustering and Similarity Scoring need more suite runtime than their coverage-based counterparts to find 90% of the defects a full run would find. He recommends starting with them anyway, since coverage-based approaches can take months to a year to set up on a large industrial system, while the content-based ones can be running in a day or two.",[42,209,211],{"id":210},"how-test-gap-analysis-complements-predictive-test-selection","How Test Gap Analysis Complements Predictive Test Selection",[11,213,214],{},"All four predictive test selection approaches assume somewhere in your suite a test already exists that's capable of catching a bug in the code being changed. You can't optimize tests that don't exist yet. Jürgens described a related technique, Test Gap Analysis. It works by overlaying two maps: which code changed recently, and which code has ever been executed by any test, unit, integration, manual, anything at all. Wherever those two maps don't overlap, you have changed code with missing test coverage.",[11,216,217],{},"He mentioned at one large software company, running this the night before a scheduled release revealed entire multi-year components that had never been touched by a single test. The release was postponed three weeks.",[11,219,220,221,225],{},"Initially I thought, \"Isn't this what our SonarQube pull request Quality Gates do for us when we set coverage floors?\" Jürgens' version has a broader scope and combines the coverage data from JaCoCo, Istanbul, Jest, etc. with manual and exploratory coverage data as well to get a full picture. Beyond that though, this lets one see that there are changes in code where your testing has blind spots. ",[222,223,224],"em",{},"This is where you would want to focus on adding coverage"," because these are areas that may or may not have been working before, but now they've been modified. Without tests you don't know if this introduced a regression in this area or not.",[11,227,228],{},"Since uncovered areas may be large, Jürgens recommends a risk-based prioritization pass, tackling the gaps sitting in your highest-domain-risk code first. I think this is a great technique, but getting that full picture is a prerequisite and test-case-specific coverage recorded and persisted across every kind of testing you do is realistically its own months-to-a-year project.",[15,230,232],{"id":231},"the-one-prerequisite-hidden-test-dependencies","The One Prerequisite: Hidden Test Dependencies",[11,234,235],{},"I attended the session because this is something we actually tried with a commercial product, Launchable, years ago at a previous employer with inconclusive results. I wanted to see what Jürgens' experience was and what has changed in the space since, especially with advances in AI.",[11,237,238,239,242],{},"The problem we were trying to solve by trialing Launchable was our extremely long feedback loop. It took 8-10 hours to see our test results after the code was committed. As a result, if a developer introduced a breaking change during their workday they wouldn't know before they left for the day. Launchable predictively selected tests based on our code coverage, drastically reducing the number of tests required to run, but caused the test suite to become unreliable. Tests were ",[222,240,241],{},"supposed"," to be written without dependencies on other tests, but over the years they naturally formed dependencies due to innocent mistakes like test engineers assuming certain data was pre-seeded in our baseline where in reality the data was inserted by upstream tests.",[11,244,245,246,249],{},"With Launchable running different tests each build it would expose these issues. We'd end up fixing the issue, but since the next run may not execute that test again it was hard to confirm the fix organically. Worse, the next run might surface other tests with the same issue. This led to expensive delays trying to get a ",[222,247,248],{},"green"," build.",[11,251,252],{},"During the Q&A, I described my experience. Jürgens' answer generalized my specific pain point and gave it a name, the test-dependency-chain problem, and it applies to all four approaches to predictive test selection.",[11,254,255,256,259],{},"If your tests form a hidden sequential chain, test A sets up state that test B needs, which a batch job then depends on as a prerequisite for test C, none of these approaches can safely select \"test C\" in isolation. Jürgens uses the German term ",[222,257,258],{},"Testkappen"," for this pattern. His rule is direct: the entire chain has to be treated and selected as a single atomic unit. If your whole system is effectively one long chain end to end, none of these approaches are useful at all, no matter how good the underlying algorithm is.",[11,261,262],{},"So the takeaway is you need to verify how interwoven your tests are, and how well they'd handle being reordered or omitted altogether from build to build, before adopting any of these approaches, whether through a commercial tool or your own implementation of what Jürgens described.",[15,264,266],{"id":265},"how-id-implement-similarity-scoring","How I'd Implement Similarity Scoring",[11,268,269],{},"I started to research which of these techniques would be the most feasible at my current employer and narrowed it down to content-based approaches: Similarity Scoring or AI Test Clustering. While we have extensive code coverage from our unit tests for some parts of the codebase, large sections are XML-driven and tested through special tooling where we don't have the coverage data we'd need to get started quickly with coverage-based approaches. Mapping the UI tests back through Test Impact Analysis, as mentioned earlier, would take a longer time investment relative to the content-based approaches.",[11,271,272],{},"I thought using Claude would quickly get us to AI Test Clustering, but initial research showed Claude doesn't provide the vector embeddings this approach depends on directly, Anthropic points people to third-party providers like Voyage AI for that instead.",[11,274,275],{},"Similarity Scoring, by contrast, needs no embedding infrastructure to get started. Its crudest version, literal keyword overlap between a test's content and the identifiers in a change, is plain text matching I could build with a basic search index. It's also easier to inspect and explain than a black box that would trade one set of problems for another, which I consider valuable for a toe-dip investigation. Weighing my options, Similarity Scoring is what I'd start with.",[11,277,278],{},"It comes down to two decisions.",[11,280,281,282,286],{},"First, I'd need to decide what counts as a \"test document,\" the thing Similarity Scoring compares a change against. It's not just the test's own code, it's every identifier that test touches: selectors, page-object property names, the API endpoints or service methods it exercises, even indirectly through a fixture or helper. Say a Playwright spec calls a page-object method that, several layers down, calls a service method named ",[283,284,285],"code",{},"ProcessRefund",":",[288,289,295],"pre",{"className":290,"code":291,"filename":292,"language":293,"meta":294,"style":294},"language-ts shiki shiki-themes material-theme-lighter github-light-high-contrast github-dark-high-contrast","test('customer can request a refund', async ({ page }) => {\n  const checkoutPage = new CheckoutPage(page);\n  await checkoutPage.submitRefundRequest();\n});\n","refund.spec.ts","ts","",[283,296,297,344,376,396],{"__ignoreMap":294},[298,299,302,306,310,314,318,320,324,328,331,335,338,341],"span",{"class":300,"line":301},"line",1,[298,303,305],{"class":304},"sb1SK","test",[298,307,309],{"class":308},"sZ-rw","(",[298,311,313],{"class":312},"sZi47","'",[298,315,317],{"class":316},"srGNg","customer can request a refund",[298,319,313],{"class":312},[298,321,323],{"class":322},"sPJuK",",",[298,325,327],{"class":326},"stWsX"," async",[298,329,330],{"class":322}," ({",[298,332,334],{"class":333},"s2xgV"," page",[298,336,337],{"class":322}," })",[298,339,340],{"class":326}," =>",[298,342,343],{"class":322}," {\n",[298,345,347,350,354,358,361,364,367,370,373],{"class":300,"line":346},2,[298,348,349],{"class":326},"  const",[298,351,353],{"class":352},"sQ79N"," checkoutPage",[298,355,357],{"class":356},"sE6rD"," =",[298,359,360],{"class":356}," new",[298,362,363],{"class":304}," CheckoutPage",[298,365,309],{"class":366},"sq0XF",[298,368,369],{"class":308},"page",[298,371,372],{"class":366},")",[298,374,375],{"class":322},";\n",[298,377,379,383,385,388,391,394],{"class":300,"line":378},3,[298,380,382],{"class":381},"sZTni","  await",[298,384,353],{"class":308},[298,386,387],{"class":322},".",[298,389,390],{"class":304},"submitRefundRequest",[298,392,393],{"class":366},"()",[298,395,375],{"class":322},[298,397,399,402,404],{"class":300,"line":398},4,[298,400,401],{"class":322},"}",[298,403,372],{"class":308},[298,405,375],{"class":322},[288,407,410],{"className":290,"code":408,"filename":409,"language":293,"meta":294,"style":294},"export class CheckoutPage {\n  async submitRefundRequest() {\n    await this.page.click('#refund-button');\n    await this.api.post('\u002Frefunds', { handler: 'ProcessRefund' });\n  }\n}\n","checkout-page.ts",[283,411,412,425,438,469,518,524],{"__ignoreMap":294},[298,413,414,417,420,423],{"class":300,"line":301},[298,415,416],{"class":381},"export",[298,418,419],{"class":326}," class",[298,421,363],{"class":422},"sA8fK",[298,424,343],{"class":322},[298,426,427,430,434,436],{"class":300,"line":346},[298,428,429],{"class":326},"  async",[298,431,433],{"class":432},"stzDA"," submitRefundRequest",[298,435,393],{"class":322},[298,437,343],{"class":322},[298,439,440,443,447,449,451,453,456,458,460,463,465,467],{"class":300,"line":378},[298,441,442],{"class":381},"    await",[298,444,446],{"class":445},"sPxkN"," this",[298,448,387],{"class":322},[298,450,369],{"class":308},[298,452,387],{"class":322},[298,454,455],{"class":304},"click",[298,457,309],{"class":366},[298,459,313],{"class":312},[298,461,462],{"class":316},"#refund-button",[298,464,313],{"class":312},[298,466,372],{"class":366},[298,468,375],{"class":322},[298,470,471,473,475,477,480,482,485,487,489,492,494,496,499,502,504,507,509,511,514,516],{"class":300,"line":398},[298,472,442],{"class":381},[298,474,446],{"class":445},[298,476,387],{"class":322},[298,478,479],{"class":308},"api",[298,481,387],{"class":322},[298,483,484],{"class":304},"post",[298,486,309],{"class":366},[298,488,313],{"class":312},[298,490,491],{"class":316},"\u002Frefunds",[298,493,313],{"class":312},[298,495,323],{"class":322},[298,497,498],{"class":322}," {",[298,500,501],{"class":366}," handler",[298,503,286],{"class":322},[298,505,506],{"class":312}," '",[298,508,285],{"class":316},[298,510,313],{"class":312},[298,512,513],{"class":322}," }",[298,515,372],{"class":366},[298,517,375],{"class":322},[298,519,521],{"class":300,"line":520},5,[298,522,523],{"class":322},"  }\n",[298,525,527],{"class":300,"line":526},6,[298,528,529],{"class":322},"}\n",[11,531,532,533,535],{},"That spec's test document would include ",[283,534,285],{},", even though the name never appears in the spec file itself, because the spec's actual behavior depends on it:",[288,537,542],{"className":538,"code":540,"language":541},[539],"language-text","test_document[\"refund.spec.ts\"] = [\n  \"customer can request a refund\",\n  \"CheckoutPage\",\n  \"submitRefundRequest\",\n  \"#refund-button\",\n  \"\u002Frefunds\",\n  \"ProcessRefund\"\n]\n","text",[283,543,540],{"__ignoreMap":294},[11,545,546,547,549,550,553],{},"Second, I'd need to decide what counts as a \"query,\" the thing built from an incoming change to search those test documents against. That's one combined query per pull request, built from the diff plus the surrounding context of whatever the changed code calls or is called by, not just the raw changed lines. If a PR renames ",[283,548,285],{}," to ",[283,551,552],{},"ProcessRefundRequest",", a query built from only the two literal diff lines would miss every test that reaches the old name through a helper several layers away. A query built from the diff's surrounding context catches those too.",[11,555,556],{},"With both of those defined, the mechanism connecting them is just comparison, not anything more exotic: score every test document against the query, rank the results, and take the highest-scoring tests, or everything above some threshold. The only real decision left is what that scoring function actually is, and that's also where I'd start simple.",[11,558,559],{},"As a first pass, before trying TF-IDF (term frequency, inverse document frequency) weighting, I'd try literal keyword overlap to match on, in other words, does the test document contain the changed identifier strings. This would be a quick-to-implement form of Similarity Scoring:",[288,561,564],{"className":562,"code":563,"language":541},[539],"score(test) = count of identifiers shared between test.document and query.identifiers\n\nselected_tests = tests\n    .where(score(test) > 0)\n    .order_by(score, descending)\n",[283,565,563],{"__ignoreMap":294},[11,567,568,569,572,573,576],{},"That ranked list still has to turn into something a CI job can actually run. Each test document maps back to a real Playwright spec file, so ",[283,570,571],{},"selected_tests"," becomes a list of file paths handed straight to ",[283,574,575],{},"npx playwright test",", the same command any CI job already invokes, just pointed at a smaller, targeted list instead of the whole suite.",[11,578,579],{},"Before trusting any of this in a real CI gate, I'd validate it the way Jürgens' own team did: pull a handful of recent pull requests that caused a regression someone caught later, and check whether my selected subset would have included the test that actually caught it. I'd also heavily leverage mutation testing to see how well this approach holds up before trusting it, and follow Jürgens' example of always running test suites that cover critical business functionality.",[15,581,583],{"id":582},"takeaway-what-test-selection-actually-requires","Takeaway: What Test Selection Actually Requires",[11,585,586],{},"After attending, I learned that my own experience, predictive test selection working in theory but introducing too much flakiness into the pipeline, wasn't unique to me.",[11,588,589],{},"Predictive test selection is not just something that is an academic exercise, but requires:",[591,592,593,597,600],"ol",{},[594,595,596],"li",{},"A healthy test suite without hidden interdependencies between tests",[594,598,599],{},"Coverage instrumentation, if you're going the coverage-based route, content-based approaches skip this entirely",[594,601,602],{},"Time to invest in building out the ecosystem (or funding to buy it)",[11,604,605],{},"Jürgens' session was valuable because it laid out the real pros and cons of each approach, letting you match one to your own tech stack and appetite for investment. Versus a black box, he lays out their weaknesses and trade-offs up front, rather than leaving you to discover where they fall apart mid-evaluation.",[607,608],"read-next",{":items":609},"[\"\u002Fsoftware-testing\u002Ftest-automation\u002Freuse-playwright-tests-for-load-testing\",\"\u002Fsoftware-testing\u002Ftest-automation\u002Fai-performance-testing-guardrails\",\"\u002Fsoftware-testing\u002Ftest-automation\u002Fwhat-would-you-stop-doing-when-ui-tests-are-flaky\",\"\u002Fsoftware-testing\u002Ftest-automation\u002Fstareast-2026-playwright-ai-cost-efficient-testing\"]",[611,612,613],"style",{},"html pre.shiki code .sb1SK,html code.shiki .sb1SK{--shiki-light:#6182B8;--shiki-default:#622CBC;--shiki-dark:#DBB7FF}html pre.shiki code .sZ-rw,html code.shiki .sZ-rw{--shiki-light:#90A4AE;--shiki-default:#0E1116;--shiki-dark:#F0F3F6}html pre.shiki code .sZi47,html code.shiki .sZi47{--shiki-light:#39ADB5;--shiki-default:#032563;--shiki-dark:#ADDCFF}html pre.shiki code .srGNg,html code.shiki .srGNg{--shiki-light:#91B859;--shiki-default:#032563;--shiki-dark:#ADDCFF}html pre.shiki code .sPJuK,html code.shiki .sPJuK{--shiki-light:#39ADB5;--shiki-default:#0E1116;--shiki-dark:#F0F3F6}html pre.shiki code .stWsX,html code.shiki .stWsX{--shiki-light:#9C3EDA;--shiki-default:#A0111F;--shiki-dark:#FF9492}html pre.shiki code .s2xgV,html code.shiki .s2xgV{--shiki-light:#90A4AE;--shiki-light-font-style:italic;--shiki-default:#702C00;--shiki-default-font-style:inherit;--shiki-dark:#FFB757;--shiki-dark-font-style:inherit}html pre.shiki code .sQ79N,html code.shiki .sQ79N{--shiki-light:#90A4AE;--shiki-default:#023B95;--shiki-dark:#91CBFF}html pre.shiki code .sE6rD,html code.shiki .sE6rD{--shiki-light:#39ADB5;--shiki-default:#A0111F;--shiki-dark:#FF9492}html pre.shiki code .sq0XF,html code.shiki .sq0XF{--shiki-light:#E53935;--shiki-default:#0E1116;--shiki-dark:#F0F3F6}html pre.shiki code .sZTni,html code.shiki .sZTni{--shiki-light:#39ADB5;--shiki-light-font-style:italic;--shiki-default:#A0111F;--shiki-default-font-style:inherit;--shiki-dark:#FF9492;--shiki-dark-font-style:inherit}html .light .shiki span{color:var(--shiki-light);background:var(--shiki-light-bg);font-style:var(--shiki-light-font-style);font-weight:var(--shiki-light-font-weight);text-decoration:var(--shiki-light-text-decoration)}html.light .shiki span{color:var(--shiki-light);background:var(--shiki-light-bg);font-style:var(--shiki-light-font-style);font-weight:var(--shiki-light-font-weight);text-decoration:var(--shiki-light-text-decoration)}html .default .shiki span{color:var(--shiki-default);background:var(--shiki-default-bg);font-style:var(--shiki-default-font-style);font-weight:var(--shiki-default-font-weight);text-decoration:var(--shiki-default-text-decoration)}html .shiki span{color:var(--shiki-default);background:var(--shiki-default-bg);font-style:var(--shiki-default-font-style);font-weight:var(--shiki-default-font-weight);text-decoration:var(--shiki-default-text-decoration)}html .dark .shiki span{color:var(--shiki-dark);background:var(--shiki-dark-bg);font-style:var(--shiki-dark-font-style);font-weight:var(--shiki-dark-font-weight);text-decoration:var(--shiki-dark-text-decoration)}html.dark .shiki span{color:var(--shiki-dark);background:var(--shiki-dark-bg);font-style:var(--shiki-dark-font-style);font-weight:var(--shiki-dark-font-weight);text-decoration:var(--shiki-dark-text-decoration)}html pre.shiki code .sA8fK,html code.shiki .sA8fK{--shiki-light:#E2931D;--shiki-default:#702C00;--shiki-dark:#FFB757}html pre.shiki code .stzDA,html code.shiki .stzDA{--shiki-light:#E53935;--shiki-default:#622CBC;--shiki-dark:#DBB7FF}html pre.shiki code .sPxkN,html code.shiki .sPxkN{--shiki-light:#39ADB5;--shiki-default:#023B95;--shiki-dark:#91CBFF}",{"title":294,"searchDepth":346,"depth":346,"links":615},[616,617,623,624,625],{"id":17,"depth":346,"text":18},{"id":36,"depth":346,"text":37,"children":618},[619,620,621,622],{"id":44,"depth":378,"text":45},{"id":61,"depth":378,"text":62},{"id":183,"depth":378,"text":184},{"id":210,"depth":378,"text":211},{"id":231,"depth":346,"text":232},{"id":265,"depth":346,"text":266},{"id":582,"depth":346,"text":583},"\u002Fimages\u002Fposts\u002Frun-fewer-tests-catch-the-same-bugs\u002Frun-fewer-tests-catch-the-same-bugs-cover.webp","2026-09-05","A framework for running fewer tests without missing real bugs, and the one prerequisite that has to be true first. StarEast 2026 lessons.",false,"md",{},true,"\u002Fsoftware-testing\u002Ftest-automation\u002Frun-fewer-tests-catch-the-same-bugs",{"title":5,"description":628},"software-testing\u002Ftest-automation\u002Frun-fewer-tests-catch-the-same-bugs","dx_k1CWyKJGJnlLB7rFIeHbcWe6GZ2YCaZlDXBjGkz8",[638,845,1736,2115],{"id":639,"title":640,"bmcUsername":6,"body":641,"cover":837,"date":838,"description":839,"draft":629,"extension":630,"features":6,"githubRepo":6,"headline":6,"highlight":6,"icon":6,"meta":840,"navigation":632,"npmPackage":6,"order":6,"path":841,"seo":842,"stem":843,"__hash__":844},"content\u002Fsoftware-testing\u002Ftest-automation\u002Freuse-playwright-tests-for-load-testing.md","Reuse Your Playwright Tests for Load Testing",{"type":8,"value":642,"toc":827},[643,651,655,658,665,672,675,679,682,685,688,691,694,698,701,707,721,728,732,735,738,756,759,762,766,769,772,780,783,786,790,793,796,804,807,811,814,817,821,824],[11,644,645,646,650],{},"I'd already reused our existing Playwright functional tests to drive load testing with Artillery.io before I ever heard ",[26,647],{"href":648,"text":649},"https:\u002F\u002Fwww.linkedin.com\u002Fin\u002Fandrew-clegg-ncsu\u002F","Andrew Clegg"," talk about the same idea at StarEast 2026. I hit a wall along the way and landed on an imperfect compromise to get past it. Clegg, a performance engineer at SAS Institute, spent his session \"Streamlined Performance Testing with Test Containers\" describing an enterprise team that took the same basic idea further, and hit the same kind of wall for the same underlying reasons. Two different tool stacks, two unrelated teams, the same conclusion.",[15,652,654],{"id":653},"reduce-reuse-recycle-reusing-functional-tests-for-load-testing","Reduce, Reuse, Recycle: Reusing Functional Tests for Load Testing",[11,656,657],{},"Clegg's team learned this lesson the hard way. For about ten years at SAS, performance testing meant LoadRunner: a centralized team of eight engineers, each owning different products, re-recording API workflows before every new build to catch whatever had changed underneath them. It was arduous, API-only, and didn't scale, eight people couldn't keep up with re-recording across all of SAS's heavy-frontend product lines, and some products just didn't get the attention they needed. That re-recording cycle looks to me like the direct cost of drift, though it's not language Clegg used himself: every new build could change which APIs got called, or how, and the old recording quietly went stale the moment it did. Retiring the recording step entirely, in favor of tests QA was already maintaining to support functional testing, removed that tax by design instead of chasing it build after build.",[11,659,660,661,664],{},"Then SAS reorganized. The centralized team got split apart and embedded into individual product teams. Clegg compares the moment to winning live chickens as a prize on ",[222,662,663],{},"Survivor",": technically valuable, useless without the tools to actually process them. Rather than start from scratch and rebuild his own testing infrastructure from zero, he looked at what his new product's QA engineers had already built: hundreds of mature Playwright tests that already knew how to drive the UI. He treated those as something he could buy at the grocery store instead of raising himself.",[11,666,667,668,671],{},"He frames the whole approach as ",[49,669,670],{},"reduce, reuse, recycle",", with recycle deliberately last: rebuilding something to repurpose it is the most expensive option of the three, reuse is what you should reach for first.",[11,673,674],{},"My own reasoning got to the same place from a different direction. I wasn't reorganized into a new team; I was tasked with capturing what real users actually experience moving through complicated workflows that span several pages and screens, and I'd already concluded that simulating that with direct API calls was the wrong foundation. Hand-built API load scripts drift silently from what the product actually does over time, as pages get updated and start calling different endpoints or static resources than whatever the script assumed when it was written. Reusing the real functional tests avoids that by construction: if the UI changes what it calls, the test that drives the UI changes with it. Reduce duplicated effort was Clegg's reason to reuse existing tests. Avoiding silent drift was mine. We ended up doing the same thing anyway.",[15,676,678],{"id":677},"why-playwright-works-for-load-testing","Why Playwright Works for Load Testing",[11,680,681],{},"Clegg's case for Playwright as the reuse vehicle: it's open source, its post-failure trace files are unusually rich for diagnosing what actually happened, and its Chrome DevTools Protocol access lets him separate container-level resource usage from the browser's own actual CPU and memory use, rather than inferring browser behavior indirectly from the container's numbers. His concrete example is SAS Studio's flow-builder canvas, a canvas element that's a black box to plain Playwright, no accessible DOM nodes, nothing in a browser snapshot. QA's existing tests already had custom JavaScript hooks built to reach into it. By reusing those tests instead of writing his own, he inherited that hard-won access for free, and just as importantly, inherited QA's ongoing maintenance of it. When the UI changes and something breaks, QA fixes it as part of their own test maintenance instead of the burden falling solely on him.",[11,683,684],{},"My own reasoning for choosing Playwright pointed the same direction, though it started from a different problem. We were already transitioning to Playwright for functional testing, and in doing so had solved a set of proprietary automation gotchas specific to our own application, reliably waiting for ExtJS events and working around non-standard UI-blocking patterns that don't behave like a normal DOM. Any other tool would have meant solving those same problems over again from scratch, the same \"don't rebuild what's already been fought through\" logic Clegg applied to QA's canvas-object hooks. On top of that, Microsoft's continued investment in Playwright's CLI and MCP tooling fit squarely into our own push toward an AI-native SDLC, independent of any performance-testing use case.",[11,686,687],{},"Clegg didn't mention this specifically, but this is my own take on why API-only tests are fundamentally the wrong approach if you are trying to collect accurate front-end performance metrics. On a modern web app, a request finishing and a user seeing something useful aren't the same event. A page can download its HTML and initial payload quickly, then have JavaScript execute, discover it needs more data or assets, fetch those too, and only then let the browser finish laying out and painting what the user actually sees, the kind of cascading, browser-side work that a metric like Largest Contentful Paint (when the page's biggest visible element actually renders) exists to measure. Adding up API response times, or even resource-download times, tells you almost nothing about when a real user saw a usable page. That gap barely existed on the simpler, largely static HTML sites performance testing tools were originally built around. This record and playback or API testing mindset seems to be a holdover from simpler times, and the tools and techniques we use need to catch up.",[11,689,690],{},"Beyond being more technically accurate, front-end metrics are also simply more legible to non-technical stakeholders than backend ones. \"The home page loaded in 4.2 seconds\" is easy for anyone to understand. \"The inventory endpoint sustained 1,000 requests per minute\" doesn't without more context. Reusing UI-driven tests for performance work doesn't just produce more accurate data, it produces self-explanatory data.",[11,692,693],{},"That said, there's still a real case for testing API calls directly, to confirm your backend can sustain a given level of traffic at a given response time. Those tests matter for measuring services your front end ultimately depends on. My point is they shouldn't be asked to double as a proxy for front-end performance.",[15,695,697],{"id":696},"where-functional-and-performance-testing-goals-conflict","Where Functional and Performance Testing Goals Conflict",[11,699,700],{},"Clegg's most actionable slide was the one titled, plainly, \"Functional vs Performance tests: Different objectives and expectations.\" Two problems came up repeatedly:",[11,702,703,706],{},[49,704,705],{},"Hardcoded wait times."," A functional test might wait a fixed five or ten seconds for something to finish, harmless under normal conditions. Under load, that same fixed wait can be too short (the action now legitimately takes longer, and the test moves on and fails) or wastefully long (waiting ten seconds for something that finished in a tenth of a second). The deeper problem for performance work specifically: a hardcoded wait sitting inside a measured transaction block corrupts the recorded transaction time itself, you're no longer measuring how long the real operation took, you're measuring your own guess.",[11,708,709,712,713,716,717,720],{},[49,710,711],{},"Hardcoded resource names."," Test files like ",[283,714,715],{},"cars.csv"," or ",[283,718,719],{},"program_baseline.sas"," work fine for a single functional run and collide immediately once multiple simulated users hit the same resource name concurrently. Clegg's team's fix, still in progress at the time of the talk, is timestamping or otherwise uniquifying these values wherever they're found.",[11,722,723,724,727],{},"My team never hit either problem, though not because we anticipated performance testing when we wrote the tests. Our Playwright tests already used data randomization for the test data they created, purely because our test environments have limited ability to tear data down between runs. And we already used Playwright's built in polling explicit waits with large max timeouts instead of fixed sleeps, for ordinary functional-testing reasons (e.g. ",[283,725,726],{},"await page.locator('.success-alert').waitFor({ state: 'visible', timeout: 60000 });","). Both decisions turned out to be exactly the guardrails a performance-testing reuse strategy needs. That's worth taking seriously as a general point: writing disciplined functional tests, the kind that avoid brittle fixed waits and brittle fixed test data as a matter of course, is what makes this whole approach viable later, whether or not performance testing is on anyone's mind when those tests get written.",[15,729,731],{"id":730},"bugs-that-ui-driven-load-testing-catches","Bugs That UI-Driven Load Testing Catches",[11,733,734],{},"Clegg's two bug stories are the strongest evidence for why any of this matters, and both were found specifically because testing drove the real UI rather than a simulated API contract.",[11,736,737],{},"The first came out of a two-day endurance test. A single transaction, saving a file, grew steadily from about six seconds to about twenty-four seconds over the course of the run. The cause: the save operation checked for filename collisions by fetching every file in the folder and comparing names one by one on the client side, instead of asking the server a direct \"does this name already exist?\" question. As the folder accumulated files over two days, that linear scan got proportionally slower. An audience member asked the obvious follow-up: shouldn't API-only testing have caught this too? Clegg's honest answer was that it's not that UI-driven testing finds things API testing structurally can't, it's that the client-side pre-check itself was client logic, invisible to a test that only exercises the underlying endpoints directly.",[11,739,740,741,744,745,748,749,752,753],{},"The second is the sharper one. During a six-hour test window, a burst of failures hit over about twenty minutes. A simulated user scrolled through a paginated file list and never found the file they were looking for. The trace told the story: two \"healthy\" requests, ",[283,742,743],{},"offset=0"," and ",[283,746,747],{},"offset=100",", both correctly reported 2,300 total items with a clean HTTP 200. A third request, ",[283,750,751],{},"offset=200",", also came back HTTP 200, but reported zero items. The real cause, found by cross-referencing other dashboards, was that the underlying file service had been OOM-killed and was mid-restart. The frontend's request chain hit a connection-refused error against the dead service, and instead of surfacing that failure to the browser, the API silently swallowed it and reported \"no files here\" with a healthy status code. As Clegg put it: ",[222,754,755],{},"\"That's really something we should be passing to the browser so it can then decide what to do. Instead we just tell it, 'Hey, there are no files in that location.' Which to me doesn't seem like a good failure mode.\"",[11,757,758],{},"That's functionally the same danger category as a system reporting false-healthy status because a real failure got silently absorbed somewhere in the stack, discoverable here specifically because the test was exercising what a user actually experiences (scrolling, finding nothing) rather than checking an API contract in isolation. It's the same argument I'd make for why simulating load at the API layer risks missing things: the client-observable behavior and the API's technical response can quietly diverge, and only one of them is what your customer actually lives with.",[11,760,761],{},"A third, more incidental finding is worth a mention: because Clegg's team pools test containers from many product teams and runs them together over long windows, they also surfaced a RabbitMQ service leak that only showed up under sustained multi-product load, something no single team's isolated testing would have caught, since some contributing products don't even have a dedicated performance engineer. Pooling containers from many teams buys you high-quality, long-duration integration testing as a side effect of the reuse strategy itself.",[15,763,765],{"id":764},"where-playwright-load-testing-stops-scaling","Where Playwright Load Testing Stops Scaling",[11,767,768],{},"Neither of us pretends this approach scales indefinitely, and the honesty on both sides is worth taking at face value.",[11,770,771],{},"Clegg's practical ceiling is around 600 concurrent users, and he's explicit that this is an Azure cost and resource constraint, not a limitation of the technique itself. SAS Studio's real usage pattern is backend-heavy with comparatively few concurrent UI sessions, so it's an appropriate ceiling for his product, not evidence the approach can't go further with more budget.",[11,773,774,775,779],{},"My own ceiling arrived much sooner. Running the ",[26,776],{"href":777,"text":778},"https:\u002F\u002Fwww.artillery.io","Artillery.io"," trial from a single workstation, concurrency became constrained by my own laptop, not the server, at somewhere around 10 to 12 concurrent users. Push past that and the test runner's own resource contention started polluting the timing data itself, not just capping how much load I could generate. The actual fix for that ceiling exists: Artillery supports distributed load generation across multiple machines in its licensed tier. Unfortunately we found our environment's network configuration requirements made adopting it problematic compared to our other options. Had we pursued the distributed architecture we probably would have hit a very similar load limit as Clegg.",[11,781,782],{},"Different scales, different specific causes, same underlying wall: browser-driven load generation is resource-expensive, and every team running it eventually meets a ceiling that has nothing to do with whether the approach is sound.",[11,784,785],{},"Even at 10-12 users, Artillery and Playwright surfaced real hotspots worth investigating.",[15,787,789],{"id":788},"two-paths-to-the-same-hybrid-load-testing-approach","Two Paths to the Same Hybrid Load Testing Approach",[11,791,792],{},"Clegg's explicit recommendation for teams that need higher concurrency than pure UI-driven testing can support is a hybrid: drive bulk load through API tests, and layer a smaller number of these UI-based sessions on top to get real client-side visibility while the system is under that load.",[11,794,795],{},"Interestingly, I landed on the same architecture, but not by choosing it. With Artillery's distributed scaling blocked, we moved to K6 instead, which meant giving something up: unlike Artillery, K6 can't run our existing Playwright scripts directly. K6 has its own browser-based test engine with similar-looking syntax, but a fundamentally different approach. Instead of generating load through the UI tests themselves, K6 uses API-driven load to scale up concurrency, then runs browser-based tests to sample the actual front-end experience while the server sits under that API-generated load. It sidesteps the browser-based resource ceiling entirely, at the cost of some realism, the load itself isn't browser-driven, only the sampling is.",[11,797,798,799,803],{},"Grafana’s own K6 documentation backs this up directly: their ",[26,800],{"href":801,"text":802},"https:\u002F\u002Fgrafana.com\u002Fdocs\u002Fk6\u002Flatest\u002Fusing-k6-browser\u002Frecommended-practices\u002Fhybrid-approach-to-performance\u002F","recommended hybrid approach"," describes combining \"a small number of virtual users for a browser test with a large number of virtual users for a protocol-level test\" as an alternative to full browser-based load testing that’s \"much less resource-intensive.\"",[11,805,806],{},"Clegg reached this design as a considered recommendation for other teams. I reached it as a forced compromise after my preferred path got blocked. It's the same architecture either way, arrived at through two unrelated tool stacks by two people who'd never heard of each other. That's a stronger argument for the hybrid model than either of us could have made alone.",[15,808,810],{"id":809},"limitations-of-reusing-playwright-tests-for-load-testing","Limitations of Reusing Playwright Tests for Load Testing",[11,812,813],{},"Clegg's mentioned in closing this isn't free: real effort goes into instrumenting containers and building the supporting infrastructure around them. Quality is mixed once you're borrowing tests from teams outside your own; his immediate team's tests are great, some neighboring teams' are usable, and further out across SAS's org it's \"a mixed bag.\" And he's explicit that this is not true performance tooling: it's a repurposed functional-test tool, and dedicated API-based load tools remain the right choice for genuinely high concurrency, thousands of users, not hundreds.",[11,815,816],{},"My own honest limitation is the one I already described: once we moved to K6, we lost the direct reuse benefit that made this appealing in the first place. We're back to maintaining a second set of test logic, just one that shares a language and a syntax style with our functional tests rather than the tests themselves. That said, we're now using AI to streamline the conversion process, which has made maintaining two sets of test logic less painful than it used to be, even though it's still fundamentally two sets of tests.",[15,818,820],{"id":819},"takeaway-reuse-playwright-tests-but-know-when-to-go-hybrid","Takeaway: Reuse Playwright Tests, But Know When to Go Hybrid",[11,822,823],{},"Pure API-simulated load is fast to build and easy to scale, but it drifts silently from what your product actually does as the UI changes underneath it, and it can't see client-side logic at all. Pure UI-driven load testing is accurate and reuses work your QA team already did, but it doesn't scale on ordinary hardware, or even on cloud infrastructure, without real cost. The honest answer for most teams sits in between: reuse your functional tests for accuracy where you can, and accept an API-driven hybrid once concurrency demands outgrow what browser-driven load can realistically generate. Two teams that had never heard of each other landed on exactly that answer, from opposite directions, within the same year.",[607,825],{":items":826},"[\"\u002Fsoftware-testing\u002Ftest-automation\u002Fai-performance-testing-guardrails\",\"\u002Fsoftware-testing\u002Ftest-automation\u002Fstareast-2026-playwright-ai-cost-efficient-testing\",\"\u002Fsoftware-testing\u002Ftest-automation\u002Fai-test-automation-pitfalls-vs-user-error\",\"\u002Fsoftware-testing\u002Ftest-automation\u002Fautomated-api-testing-with-schemathesis\"]",{"title":294,"searchDepth":346,"depth":346,"links":828},[829,830,831,832,833,834,835,836],{"id":653,"depth":346,"text":654},{"id":677,"depth":346,"text":678},{"id":696,"depth":346,"text":697},{"id":730,"depth":346,"text":731},{"id":764,"depth":346,"text":765},{"id":788,"depth":346,"text":789},{"id":809,"depth":346,"text":810},{"id":819,"depth":346,"text":820},"\u002Fimages\u002Fposts\u002Freuse-playwright-tests-for-load-testing\u002Freuse-playwright-tests-for-load-testing-cover.webp","2026-08-30","Reuse existing Playwright functional tests for load testing instead of building a parallel suite. My own experience, validated by a StarEast 2026 talk.",{},"\u002Fsoftware-testing\u002Ftest-automation\u002Freuse-playwright-tests-for-load-testing",{"title":640,"description":839},"software-testing\u002Ftest-automation\u002Freuse-playwright-tests-for-load-testing","Y4H6EgnXgiRY2xiiq8_rbZQiZFCsMEk1yjt39yecWNs",{"id":846,"title":847,"bmcUsername":6,"body":848,"cover":1728,"date":1729,"description":1730,"draft":629,"extension":630,"features":6,"githubRepo":6,"headline":6,"highlight":6,"icon":6,"meta":1731,"navigation":632,"npmPackage":6,"order":6,"path":1732,"seo":1733,"stem":1734,"__hash__":1735},"content\u002Fsoftware-testing\u002Ftest-automation\u002Fai-performance-testing-guardrails.md","AI Performance Testing Guardrails Every Tester Needs",{"type":8,"value":849,"toc":1720},[850,858,861,865,868,871,878,881,1226,1229,1232,1236,1239,1242,1245,1267,1274,1285,1288,1452,1462,1466,1469,1529,1532,1640,1643,1647,1654,1657,1660,1664,1677,1680,1684,1687,1690,1693,1714,1717],[11,851,852,853,857],{},"Performance testing isn't my primary discipline. I get pulled into it occasionally, enough that I wanted a real read on where AI actually helps in that space and where it's just hype, before the next time it comes up. That's what took me to ",[26,854],{"href":855,"text":856},"https:\u002F\u002Fkaushaldalvi.com\u002F","Kaushal Dalvi","'s StarEast 2026 session, \"Beyond the Dev Box: Claude Code Across the Entire Performance Testing Lifecycle.\" Dalvi leads a performance engineering team and spent the session making a specific argument: performance engineering has always carried a hidden tax, hours spent operating tools instead of doing the actual engineering judgment. Better observability tooling cut that tax once already, without removing the need for performance engineers. His claim is that Claude Code and other AI coding agents are the next wave of the same pattern, provided you constrain them with the right guardrails.",[11,859,860],{},"What made that argument land for me was a real incident, not a hypothetical demo.",[15,862,864],{"id":863},"the-ai-generated-script-that-let-a-critical-performance-issue-slip-into-production","The AI-Generated Script That Let a Critical Performance Issue Slip Into Production",[11,866,867],{},"An intern on Dalvi's team, new to a project that needed performance testing, used Claude Code to generate a K6 script for an API endpoint. The script was built in 30 minutes, running within the hour, and the results were published within half a day. A senior engineer on the team, with 20 to 25 years of performance engineering experience, estimated the same work would have taken him closer to a full day by hand. The results looked great too: 100% success rate, solid response times.",[11,869,870],{},"Days later, the application went live and fell over almost immediately, even under low load.",[11,872,873,874,877],{},"The root cause, found only after the fact, was a ",[283,875,876],{},"try\u002Fcatch"," block wrapped around a key transaction in the generated script, silently swallowing every error. That's a silent failure in the truest sense: the tool reported 100% success because it genuinely never saw one. The errors were being caught and discarded before K6 could record them.",[11,879,880],{},"Below is an example of what this might have looked like, but wasn't shown during the lecture.",[288,882,887],{"className":883,"code":884,"filename":885,"language":886,"meta":294,"style":294},"language-js shiki shiki-themes material-theme-lighter github-light-high-contrast github-dark-high-contrast","import http from 'k6\u002Fhttp';\nimport { check } from 'k6';\n\nexport const options = {\n  vus: 50,\n  duration: '5m',\n};\n\nexport default function () {\n  try {\n    const res = http.post('https:\u002F\u002Fshop-demo.io\u002Fapi\u002Fcheckout', payload());\n\n    if (res.status !== 200) {\n      throw new Error(`Checkout failed: ${res.status}`);\n    }\n\n    check(res, { 'checkout succeeded': (r) => r.status === 200 });\n  } catch (err) {\n    \u002F\u002F Every real failure lands here and goes nowhere.\n    \u002F\u002F No check() runs, no error metric increments, nothing is recorded.\n  }\n}\n","checkout-load-test.js","js",[283,888,889,909,932,937,951,965,981,987,992,1009,1017,1054,1059,1088,1123,1129,1134,1185,1203,1210,1216,1221],{"__ignoreMap":294},[298,890,891,894,897,900,902,905,907],{"class":300,"line":301},[298,892,893],{"class":381},"import",[298,895,896],{"class":308}," http ",[298,898,899],{"class":381},"from",[298,901,506],{"class":312},[298,903,904],{"class":316},"k6\u002Fhttp",[298,906,313],{"class":312},[298,908,375],{"class":322},[298,910,911,913,915,918,920,923,925,928,930],{"class":300,"line":346},[298,912,893],{"class":381},[298,914,498],{"class":322},[298,916,917],{"class":308}," check",[298,919,513],{"class":322},[298,921,922],{"class":381}," from",[298,924,506],{"class":312},[298,926,927],{"class":316},"k6",[298,929,313],{"class":312},[298,931,375],{"class":322},[298,933,934],{"class":300,"line":378},[298,935,936],{"emptyLinePlaceholder":632},"\n",[298,938,939,941,944,947,949],{"class":300,"line":398},[298,940,416],{"class":381},[298,942,943],{"class":326}," const",[298,945,946],{"class":352}," options",[298,948,357],{"class":356},[298,950,343],{"class":322},[298,952,953,956,958,962],{"class":300,"line":520},[298,954,955],{"class":366},"  vus",[298,957,286],{"class":322},[298,959,961],{"class":960},"s6g51"," 50",[298,963,964],{"class":322},",\n",[298,966,967,970,972,974,977,979],{"class":300,"line":526},[298,968,969],{"class":366},"  duration",[298,971,286],{"class":322},[298,973,506],{"class":312},[298,975,976],{"class":316},"5m",[298,978,313],{"class":312},[298,980,964],{"class":322},[298,982,984],{"class":300,"line":983},7,[298,985,986],{"class":322},"};\n",[298,988,990],{"class":300,"line":989},8,[298,991,936],{"emptyLinePlaceholder":632},[298,993,995,997,1000,1003,1007],{"class":300,"line":994},9,[298,996,416],{"class":381},[298,998,999],{"class":381}," default",[298,1001,1002],{"class":326}," function",[298,1004,1006],{"class":1005},"sCRTB"," ()",[298,1008,343],{"class":322},[298,1010,1012,1015],{"class":300,"line":1011},10,[298,1013,1014],{"class":381},"  try",[298,1016,343],{"class":322},[298,1018,1020,1023,1026,1028,1031,1033,1035,1037,1039,1042,1044,1046,1049,1052],{"class":300,"line":1019},11,[298,1021,1022],{"class":326},"    const",[298,1024,1025],{"class":352}," res",[298,1027,357],{"class":356},[298,1029,1030],{"class":308}," http",[298,1032,387],{"class":322},[298,1034,484],{"class":304},[298,1036,309],{"class":366},[298,1038,313],{"class":312},[298,1040,1041],{"class":316},"https:\u002F\u002Fshop-demo.io\u002Fapi\u002Fcheckout",[298,1043,313],{"class":312},[298,1045,323],{"class":322},[298,1047,1048],{"class":304}," payload",[298,1050,1051],{"class":366},"())",[298,1053,375],{"class":322},[298,1055,1057],{"class":300,"line":1056},12,[298,1058,936],{"emptyLinePlaceholder":632},[298,1060,1062,1065,1068,1071,1073,1076,1079,1082,1085],{"class":300,"line":1061},13,[298,1063,1064],{"class":381},"    if",[298,1066,1067],{"class":366}," (",[298,1069,1070],{"class":308},"res",[298,1072,387],{"class":322},[298,1074,1075],{"class":308},"status",[298,1077,1078],{"class":356}," !==",[298,1080,1081],{"class":960}," 200",[298,1083,1084],{"class":366},") ",[298,1086,1087],{"class":322},"{\n",[298,1089,1091,1094,1096,1099,1101,1104,1107,1110,1112,1114,1116,1119,1121],{"class":300,"line":1090},14,[298,1092,1093],{"class":381},"      throw",[298,1095,360],{"class":356},[298,1097,1098],{"class":304}," Error",[298,1100,309],{"class":366},[298,1102,1103],{"class":312},"`",[298,1105,1106],{"class":316},"Checkout failed: ",[298,1108,1109],{"class":312},"${",[298,1111,1070],{"class":308},[298,1113,387],{"class":312},[298,1115,1075],{"class":308},[298,1117,1118],{"class":312},"}`",[298,1120,372],{"class":366},[298,1122,375],{"class":322},[298,1124,1126],{"class":300,"line":1125},15,[298,1127,1128],{"class":322},"    }\n",[298,1130,1132],{"class":300,"line":1131},16,[298,1133,936],{"emptyLinePlaceholder":632},[298,1135,1137,1140,1142,1144,1146,1148,1150,1154,1156,1158,1160,1163,1165,1167,1170,1172,1174,1177,1179,1181,1183],{"class":300,"line":1136},17,[298,1138,1139],{"class":304},"    check",[298,1141,309],{"class":366},[298,1143,1070],{"class":308},[298,1145,323],{"class":322},[298,1147,498],{"class":322},[298,1149,506],{"class":312},[298,1151,1153],{"class":1152},"sqmHM","checkout succeeded",[298,1155,313],{"class":312},[298,1157,286],{"class":322},[298,1159,1067],{"class":322},[298,1161,1162],{"class":333},"r",[298,1164,372],{"class":322},[298,1166,340],{"class":326},[298,1168,1169],{"class":308}," r",[298,1171,387],{"class":322},[298,1173,1075],{"class":308},[298,1175,1176],{"class":356}," ===",[298,1178,1081],{"class":960},[298,1180,513],{"class":322},[298,1182,372],{"class":366},[298,1184,375],{"class":322},[298,1186,1188,1191,1194,1196,1199,1201],{"class":300,"line":1187},18,[298,1189,1190],{"class":322},"  }",[298,1192,1193],{"class":381}," catch",[298,1195,1067],{"class":366},[298,1197,1198],{"class":308},"err",[298,1200,1084],{"class":366},[298,1202,1087],{"class":322},[298,1204,1206],{"class":300,"line":1205},19,[298,1207,1209],{"class":1208},"s_gjE","    \u002F\u002F Every real failure lands here and goes nowhere.\n",[298,1211,1213],{"class":300,"line":1212},20,[298,1214,1215],{"class":1208},"    \u002F\u002F No check() runs, no error metric increments, nothing is recorded.\n",[298,1217,1219],{"class":300,"line":1218},21,[298,1220,523],{"class":322},[298,1222,1224],{"class":300,"line":1223},22,[298,1225,529],{"class":322},[11,1227,1228],{},"Dalvi's read on why the model did this is the sharpest line from the whole talk: the model was trying to prevent errors because it thought that was the helpful thing to do, and in doing so it lost sight of what the script was actually for. A performance test exists to surface failures under load. A model optimizing for looking correct will do the opposite of that unless something stops it.",[11,1230,1231],{},"That's a general problem with AI-generated code, not a performance-testing-specific one. It just happened to show up here in a form with real production consequences.",[15,1233,1235],{"id":1234},"determinism-vs-non-determinism","Determinism vs. Non-Determinism",[11,1237,1238],{},"The thread running through the rest of the session was a distinction Dalvi kept returning to: determinism versus non-determinism, and how you put guardrails around the second one.",[11,1240,1241],{},"He illustrated it with a log analysis demo. He fed Claude a raw access log, hundreds of thousands of lines, the kind of file his team used to hand-parse before modern observability tooling existed. He was upfront that this was a contrived setup (an audience member correctly pointed out that no real organization works from a raw log with no retention policy), but the point he wanted to make didn't depend on the example being realistic.",[11,1243,1244],{},"The failure mode he was demonstrating: dump a huge log into a chat window and ask for analysis, and you'll get a plausible-looking answer that's different every time you ask. That's next-token prediction doing what it does. The fix wasn't to trust the model's summary less. It was to change what he asked the model to do:",[591,1246,1247,1258,1261,1264],{},[594,1248,1249,1250,1253,1254,1257],{},"Don't read the whole file into context. Inspect it first, using ",[283,1251,1252],{},"head",", ",[283,1255,1256],{},"tail",", random sampling, and time-range counts to learn its shape.",[594,1259,1260],{},"Write scripts that do the actual analysis, rather than reasoning over the raw data directly.",[594,1262,1263],{},"Run those scripts. The output is deterministic and repeatable because it came from code, not from a language model's recollection of what it read.",[594,1265,1266],{},"Cite every number back to the script or file it came from, so the results can be checked rather than taken on faith.",[11,1268,1269,1270,1273],{},"The scripts turned hundreds of thousands of log lines into a ",[283,1271,1272],{},"workload-model.md"," with request-per-second figures, concurrent user estimates using Little's Law, and endpoint hit ratios, none of which the model could have reliably held in its head from a single pass over the raw file.",[1275,1276,1279],"callout",{"color":1277,"icon":1278},"info","i-lucide-info",[11,1280,1281,1284],{},[49,1282,1283],{},"Little's Law",", the average number of things in a stable system (L) equals how often new ones arrive (λ) multiplied by how long each one stays (W), or L = λW. Applied to a workload model, concurrent users equal how many sessions start per second multiplied by the average session length. In other words, if customers walk into a coffee shop every 2 minutes and each one lingers for 10 minutes, there are usually 5 people in the shop at any given moment. Same math, just users and sessions instead of customers and minutes.",[11,1286,1287],{},"Dalvi didn't share the file itself (there's no code repo for this session). What follows is a reconstruction matching the shape he described, including the part that mattered most to him: every number traceable back to the script that produced it.",[288,1289,1293],{"className":1290,"code":1291,"filename":1272,"language":1292,"meta":294,"style":294},"language-markdown shiki shiki-themes material-theme-lighter github-light-high-contrast github-dark-high-contrast","# Workload Model: access.log analysis\n\n**Source:** access.log (438,201 lines, 2026-05-01 to 2026-05-07)\n**Generated by:** analyze_endpoints.py, compute_percentiles.py, estimate_concurrency.py\n\n## Endpoint Hit Ratios\n| Endpoint | Requests | % of Total |\n|---|---|---|\n| GET \u002Fapi\u002Fproducts | 182,340 | 41.6% |\n| POST \u002Fapi\u002Fcart | 96,112 | 21.9% |\n| GET \u002Fapi\u002Fsearch | 74,558 | 17.0% |\n| POST \u002Fapi\u002Fcheckout | 41,209 | 9.4% |\n| other | 43,982 | 10.1% |\n\n*Source: analyze_endpoints.py*\n\n## Latency Percentiles (ms)\n| Endpoint | p50 | p95 | p99 |\n|---|---|---|---|\n| GET \u002Fapi\u002Fproducts | 82 | 310 | 640 |\n| POST \u002Fapi\u002Fcart | 110 | 420 | 810 |\n| POST \u002Fapi\u002Fcheckout | 145 | 560 | 1,020 |\n\n*Source: compute_percentiles.py*\n\n## Concurrency Estimate (Little's Law)\n- Session arrival rate (λ): 0.92 sessions\u002Fsec, derived from unique session IDs per minute\n- Average session duration (W): 3.8 minutes (228 seconds), derived from first-to-last timestamp per session ID\n- Estimated concurrent users (L = λ × W): 0.92 × 228 ≈ 210\n\n*Source: estimate_concurrency.py*\n","markdown",[283,1294,1295,1300,1304,1309,1314,1318,1323,1328,1333,1338,1343,1348,1353,1358,1362,1367,1371,1376,1381,1386,1391,1396,1401,1406,1412,1417,1423,1429,1435,1441,1446],{"__ignoreMap":294},[298,1296,1297],{"class":300,"line":301},[298,1298,1299],{},"# Workload Model: access.log analysis\n",[298,1301,1302],{"class":300,"line":346},[298,1303,936],{"emptyLinePlaceholder":632},[298,1305,1306],{"class":300,"line":378},[298,1307,1308],{},"**Source:** access.log (438,201 lines, 2026-05-01 to 2026-05-07)\n",[298,1310,1311],{"class":300,"line":398},[298,1312,1313],{},"**Generated by:** analyze_endpoints.py, compute_percentiles.py, estimate_concurrency.py\n",[298,1315,1316],{"class":300,"line":520},[298,1317,936],{"emptyLinePlaceholder":632},[298,1319,1320],{"class":300,"line":526},[298,1321,1322],{},"## Endpoint Hit Ratios\n",[298,1324,1325],{"class":300,"line":983},[298,1326,1327],{},"| Endpoint | Requests | % of Total |\n",[298,1329,1330],{"class":300,"line":989},[298,1331,1332],{},"|---|---|---|\n",[298,1334,1335],{"class":300,"line":994},[298,1336,1337],{},"| GET \u002Fapi\u002Fproducts | 182,340 | 41.6% |\n",[298,1339,1340],{"class":300,"line":1011},[298,1341,1342],{},"| POST \u002Fapi\u002Fcart | 96,112 | 21.9% |\n",[298,1344,1345],{"class":300,"line":1019},[298,1346,1347],{},"| GET \u002Fapi\u002Fsearch | 74,558 | 17.0% |\n",[298,1349,1350],{"class":300,"line":1056},[298,1351,1352],{},"| POST \u002Fapi\u002Fcheckout | 41,209 | 9.4% |\n",[298,1354,1355],{"class":300,"line":1061},[298,1356,1357],{},"| other | 43,982 | 10.1% |\n",[298,1359,1360],{"class":300,"line":1090},[298,1361,936],{"emptyLinePlaceholder":632},[298,1363,1364],{"class":300,"line":1125},[298,1365,1366],{},"*Source: analyze_endpoints.py*\n",[298,1368,1369],{"class":300,"line":1131},[298,1370,936],{"emptyLinePlaceholder":632},[298,1372,1373],{"class":300,"line":1136},[298,1374,1375],{},"## Latency Percentiles (ms)\n",[298,1377,1378],{"class":300,"line":1187},[298,1379,1380],{},"| Endpoint | p50 | p95 | p99 |\n",[298,1382,1383],{"class":300,"line":1205},[298,1384,1385],{},"|---|---|---|---|\n",[298,1387,1388],{"class":300,"line":1212},[298,1389,1390],{},"| GET \u002Fapi\u002Fproducts | 82 | 310 | 640 |\n",[298,1392,1393],{"class":300,"line":1218},[298,1394,1395],{},"| POST \u002Fapi\u002Fcart | 110 | 420 | 810 |\n",[298,1397,1398],{"class":300,"line":1223},[298,1399,1400],{},"| POST \u002Fapi\u002Fcheckout | 145 | 560 | 1,020 |\n",[298,1402,1404],{"class":300,"line":1403},23,[298,1405,936],{"emptyLinePlaceholder":632},[298,1407,1409],{"class":300,"line":1408},24,[298,1410,1411],{},"*Source: compute_percentiles.py*\n",[298,1413,1415],{"class":300,"line":1414},25,[298,1416,936],{"emptyLinePlaceholder":632},[298,1418,1420],{"class":300,"line":1419},26,[298,1421,1422],{},"## Concurrency Estimate (Little's Law)\n",[298,1424,1426],{"class":300,"line":1425},27,[298,1427,1428],{},"- Session arrival rate (λ): 0.92 sessions\u002Fsec, derived from unique session IDs per minute\n",[298,1430,1432],{"class":300,"line":1431},28,[298,1433,1434],{},"- Average session duration (W): 3.8 minutes (228 seconds), derived from first-to-last timestamp per session ID\n",[298,1436,1438],{"class":300,"line":1437},29,[298,1439,1440],{},"- Estimated concurrent users (L = λ × W): 0.92 × 228 ≈ 210\n",[298,1442,1444],{"class":300,"line":1443},30,[298,1445,936],{"emptyLinePlaceholder":632},[298,1447,1449],{"class":300,"line":1448},31,[298,1450,1451],{},"*Source: estimate_concurrency.py*\n",[11,1453,1454,1455,1458,1459,1461],{},"That citation line under each section is what lets a reviewer walk back to ",[283,1456,1457],{},"compute_percentiles.py"," and check the p95 figure against the raw log instead of taking the model's summary on faith. The same discipline (inspect, script, run, cite) is what would have caught the K6 script's silent ",[283,1460,876],{}," before it shipped: a reviewer working from a script's actual behavior, rather than the tool's self-reported summary, would have seen the error handling directly.",[15,1463,1465],{"id":1464},"a-guardrails-checklist-for-the-next-time-i-get-pulled-in","A Guardrails Checklist for the Next Time I Get Pulled In",[11,1467,1468],{},"The rest of the session built out a stack of guardrails in increasing order of formality. I'm not going to pretend I've used all of these in production the way Dalvi's team has, but this is the checklist I'd actually reach for the next time I'm asked to help with AI-assisted performance testing:",[1470,1471,1472,1485,1494,1500,1506,1517],"ul",{},[594,1473,1474,1477,1478,744,1481,1484],{},[49,1475,1476],{},"Ground it in a spec."," Feed the model an OpenAPI or Swagger spec before asking it to generate test assets. Dalvi said that without this, the model invented its own ad hoc control flow, ",[283,1479,1480],{},"if\u002Felse",[283,1482,1483],{},"for"," loops inside the test scripts, instead of using the load-testing tool's built-in features, in every example his team tried before adding this guardrail.",[594,1486,1487,1493],{},[49,1488,1489,1490,387],{},"Write a ",[283,1491,1492],{},"rules.md"," A short, team-authored file listing what not to do (no swallowed errors, no logic embedded in test scripts) plus house conventions, handed to the model alongside the spec every time.",[594,1495,1496,1499],{},[49,1497,1498],{},"Package repeated context as a skill."," A skill is a folder of instructions, scripts, and resources that Claude can load automatically or on request, useful for anything you'd otherwise re-explain every session, like how to call an internal CLI to pull metrics or restart a service.",[594,1501,1502,1505],{},[49,1503,1504],{},"Split large tasks across subagents."," Dalvi noted that output quality tends to decline once a context window is 40 to 50% full. His team's fix was markdown-defined specialist subagents, each with isolated context. Given a 20-endpoint OpenAPI spec, he ran four K6 script-generator subagents in parallel, each following the same house rules, instead of one long session trying to hold all of it at once.",[594,1507,1508,1511,1512,1516],{},[49,1509,1510],{},"Distribute the guardrails as a plugin once they're stable."," Dalvi's central performance engineering team hosts a ",[26,1513],{"href":1514,"text":1515},"https:\u002F\u002Fcode.claude.com\u002Fdocs\u002Fen\u002Fplugins","plugin"," in a GitHub repository that any of their 80-plus distributed development teams can install with a single command. When the central team fixes a flaw in their guidance, every consuming team gets the fix on their next plugin update, instead of the correction living in one team's head.",[594,1518,1519,1522,1523,1525,1526,1528],{},[49,1520,1521],{},"Write separate reporting rules per audience."," His team keeps different ",[283,1524,1492],{}," files for executive summaries (bullet points, no jargon, business impact only), engineering deep-dives (line numbers, timestamps, APM correlations), and release management (a mix of both, framed around go or no-go readiness). In one demo, Claude generated all three reports in a single pass from the same underlying analysis, swapping only which ",[283,1527,1492],{}," grounded each one.",[11,1530,1531],{},"Dalvi didn't share his team's actual rules.md files, but here's roughly the shape of what those three might look like for the same underlying finding:",[1533,1534,1535,1570,1605],"code-group",{},[288,1536,1539],{"className":1290,"code":1537,"filename":1538,"language":1292,"meta":294,"style":294},"# Executive Summary Rules\n\n- Bullet points only, no paragraphs\n- No technical jargon: no APM tool names, no percentile terms\n- Lead with business impact: revenue risk, customer impact, launch readiness\n- Five bullets maximum\n","executive-summary-rules.md",[283,1540,1541,1546,1550,1555,1560,1565],{"__ignoreMap":294},[298,1542,1543],{"class":300,"line":301},[298,1544,1545],{},"# Executive Summary Rules\n",[298,1547,1548],{"class":300,"line":346},[298,1549,936],{"emptyLinePlaceholder":632},[298,1551,1552],{"class":300,"line":378},[298,1553,1554],{},"- Bullet points only, no paragraphs\n",[298,1556,1557],{"class":300,"line":398},[298,1558,1559],{},"- No technical jargon: no APM tool names, no percentile terms\n",[298,1561,1562],{"class":300,"line":520},[298,1563,1564],{},"- Lead with business impact: revenue risk, customer impact, launch readiness\n",[298,1566,1567],{"class":300,"line":526},[298,1568,1569],{},"- Five bullets maximum\n",[288,1571,1574],{"className":1290,"code":1572,"filename":1573,"language":1292,"meta":294,"style":294},"# Engineering Deep-Dive Rules\n\n- Cite every claim: file name, line number, or timestamp\n- Include APM correlation IDs where available\n- Recommend the specific code or config change, not just the symptom\n- Do not summarize away technical detail for brevity\n","engineering-deep-dive-rules.md",[283,1575,1576,1581,1585,1590,1595,1600],{"__ignoreMap":294},[298,1577,1578],{"class":300,"line":301},[298,1579,1580],{},"# Engineering Deep-Dive Rules\n",[298,1582,1583],{"class":300,"line":346},[298,1584,936],{"emptyLinePlaceholder":632},[298,1586,1587],{"class":300,"line":378},[298,1588,1589],{},"- Cite every claim: file name, line number, or timestamp\n",[298,1591,1592],{"class":300,"line":398},[298,1593,1594],{},"- Include APM correlation IDs where available\n",[298,1596,1597],{"class":300,"line":520},[298,1598,1599],{},"- Recommend the specific code or config change, not just the symptom\n",[298,1601,1602],{"class":300,"line":526},[298,1603,1604],{},"- Do not summarize away technical detail for brevity\n",[288,1606,1609],{"className":1290,"code":1607,"filename":1608,"language":1292,"meta":294,"style":294},"# Release Readiness Rules\n\n- Lead with status: green, yellow, or red\n- List open risks with severity and owner\n- State a go or no-go recommendation explicitly, no hedging\n- One paragraph maximum per section, mixing technical and business framing\n","release-readiness-rules.md",[283,1610,1611,1616,1620,1625,1630,1635],{"__ignoreMap":294},[298,1612,1613],{"class":300,"line":301},[298,1614,1615],{},"# Release Readiness Rules\n",[298,1617,1618],{"class":300,"line":346},[298,1619,936],{"emptyLinePlaceholder":632},[298,1621,1622],{"class":300,"line":378},[298,1623,1624],{},"- Lead with status: green, yellow, or red\n",[298,1626,1627],{"class":300,"line":398},[298,1628,1629],{},"- List open risks with severity and owner\n",[298,1631,1632],{"class":300,"line":520},[298,1633,1634],{},"- State a go or no-go recommendation explicitly, no hedging\n",[298,1636,1637],{"class":300,"line":526},[298,1638,1639],{},"- One paragraph maximum per section, mixing technical and business framing\n",[11,1641,1642],{},"None of this removes the review step. It just means the review is checking cited, reproducible work instead of a black-box claim.",[15,1644,1646],{"id":1645},"hiring-a-second-ai-agent-to-check-the-first","Hiring a Second AI Agent to Check the First",[11,1648,1649,1650,1653],{},"The guardrail I found most transferable outside performance testing came from Dalvi's consulting background. When a company hires a consulting firm to build something, he said, it's common to hire a ",[222,1651,1652],{},"different"," firm to validate the work, because a firm validating its own output has a natural incentive to protect its own conclusions.",[11,1655,1656],{},"He applies the same idea to agents. Rather than asking one Claude session for a diagnosis and trusting it, he asks it to do the correlation work (build a timeline, cite artifacts, show its reasoning), then spins up a second, independent agent whose only job is to find flaws in the first agent's conclusion. In a bottleneck-analysis demo, he handed Claude seven artifacts from a load test where he'd deliberately injected a fault: a K6 summary, application logs, garbage collection logs, thread dumps, and slow-query exports from the application performance monitoring platform. The prompt explicitly told it to be skeptical of the first apparent cause, to flag suspicious or misleading readings, and to surface any conflicting evidence rather than pick a side silently. It correctly traced the fault to connection pool saturation at the four-minute mark, matching what Dalvi had actually injected, with citations back to the source artifact for every claim.",[11,1658,1659],{},"His summary of the approach: the judgment stays his, the grunt work doesn't.",[15,1661,1663],{"id":1662},"the-same-failure-pattern-i-already-knew-from-test-automation","The Same Failure Pattern I Already Knew From Test Automation",[11,1665,1666,1667,1669,1670,1672,1673,1676],{},"The pattern Dalvi described, an AI swallowing errors inside a ",[283,1668,876],{},", is one I've caught underskilled test automation consultants doing by hand: wrapping every test in a ",[283,1671,876],{}," so nothing ever throws. I think it comes from the same instinct, avoiding execution failures, and it's the wrong instinct either way. Tests ",[222,1674,1675],{},"should"," fail when the software under test has changed behavior. There's a real difference between making a test reliable and making it so self-healing and overly resilient that it hides defects instead of reporting them.",[11,1678,1679],{},"Dalvi's K6 story is that same failure mode wearing performance-testing clothes. The model wasn't malicious or even wrong to want the script to run cleanly. It just optimized for the wrong signal, and nothing was in place to catch it before production did.",[15,1681,1683],{"id":1682},"what-im-taking-back-to-my-own-work","What I'm Taking Back to My Own Work",[11,1685,1686],{},"The determinism guardrail is the one I'll use most, even outside performance testing: when I'm asking Claude to make sense of a large, messy input (logs, a big CSV export, a pile of test results), I'd rather it write and run a script against the raw data and cite the output than summarize the data directly from a single read. That's a cheap habit to adopt and it applies well beyond any one testing discipline.",[11,1688,1689],{},"The adversarial-agent pattern is the other one I want to start using deliberately. A second agent whose only job is to argue with the first agent's conclusion is a lightweight way to catch the kind of confident, plausible-sounding mistake that's easy to miss when you're the one who asked the question in the first place.",[11,1691,1692],{},"Performance testing is still not my daily work. But the next time I'm handed a load test and an AI assistant to help build it, I have a specific list of questions to ask before I trust the results: what is it citing this number from, what did I tell it not to do, and who's checking its work besides me.",[11,1694,1695,1696,1253,1701,1253,1705,1709,1710,387],{},"For more on the AI conference sessions I attended around this one: ",[1697,1698,1700],"a",{"href":1699},"\u002Fsoftware-testing\u002Ftest-automation\u002Fstareast-2026-getting-started-ai-driven-automation","AI vision testing and Playwright MCP",[1697,1702,1704],{"href":1703},"\u002Fsoftware-testing\u002Ftest-automation\u002Fstareast-2026-getting-dirty-ai-testing","hands-on AI tooling and evals",[1697,1706,1708],{"href":1707},"\u002Fsoftware-testing\u002Ftest-automation\u002Fstareast-2026-playwright-ai-cost-efficient-testing","cost-efficient Playwright testing with AI",", and ",[1697,1711,1713],{"href":1712},"\u002Fsoftware-testing\u002Ftest-automation\u002Fstareast-2026-prompt-engineering-techniques","prompt engineering techniques for testers",[607,1715],{":items":1716},"[\"\u002Fsoftware-testing\u002Ftest-automation\u002Fai-test-automation-pitfalls-vs-user-error\",\"\u002Fsoftware-testing\u002Ftest-automation\u002Fstareast-2026-prompt-engineering-techniques\",\"\u002Fsoftware-testing\u002Ftest-automation\u002Fstareast-2026-playwright-ai-cost-efficient-testing\",\"\u002Fsoftware-testing\u002Ftest-automation\u002Fhow-to-test-ai-chatbots-and-agents\"]",[611,1718,1719],{},"html pre.shiki code .sZTni,html code.shiki .sZTni{--shiki-light:#39ADB5;--shiki-light-font-style:italic;--shiki-default:#A0111F;--shiki-default-font-style:inherit;--shiki-dark:#FF9492;--shiki-dark-font-style:inherit}html pre.shiki code .sZ-rw,html code.shiki .sZ-rw{--shiki-light:#90A4AE;--shiki-default:#0E1116;--shiki-dark:#F0F3F6}html pre.shiki code .sZi47,html code.shiki .sZi47{--shiki-light:#39ADB5;--shiki-default:#032563;--shiki-dark:#ADDCFF}html pre.shiki code .srGNg,html code.shiki .srGNg{--shiki-light:#91B859;--shiki-default:#032563;--shiki-dark:#ADDCFF}html pre.shiki code .sPJuK,html code.shiki .sPJuK{--shiki-light:#39ADB5;--shiki-default:#0E1116;--shiki-dark:#F0F3F6}html pre.shiki code .stWsX,html code.shiki .stWsX{--shiki-light:#9C3EDA;--shiki-default:#A0111F;--shiki-dark:#FF9492}html pre.shiki code .sQ79N,html code.shiki .sQ79N{--shiki-light:#90A4AE;--shiki-default:#023B95;--shiki-dark:#91CBFF}html pre.shiki code .sE6rD,html code.shiki .sE6rD{--shiki-light:#39ADB5;--shiki-default:#A0111F;--shiki-dark:#FF9492}html pre.shiki code .sq0XF,html code.shiki .sq0XF{--shiki-light:#E53935;--shiki-default:#0E1116;--shiki-dark:#F0F3F6}html pre.shiki code .s6g51,html code.shiki .s6g51{--shiki-light:#F76D47;--shiki-default:#023B95;--shiki-dark:#91CBFF}html pre.shiki code .sCRTB,html code.shiki .sCRTB{--shiki-light:#39ADB5;--shiki-default:#702C00;--shiki-dark:#FFB757}html pre.shiki code .sb1SK,html code.shiki .sb1SK{--shiki-light:#6182B8;--shiki-default:#622CBC;--shiki-dark:#DBB7FF}html pre.shiki code .sqmHM,html code.shiki .sqmHM{--shiki-light:#E53935;--shiki-default:#032563;--shiki-dark:#ADDCFF}html pre.shiki code .s2xgV,html code.shiki .s2xgV{--shiki-light:#90A4AE;--shiki-light-font-style:italic;--shiki-default:#702C00;--shiki-default-font-style:inherit;--shiki-dark:#FFB757;--shiki-dark-font-style:inherit}html pre.shiki code .s_gjE,html code.shiki .s_gjE{--shiki-light:#90A4AE;--shiki-light-font-style:italic;--shiki-default:#66707B;--shiki-default-font-style:inherit;--shiki-dark:#BDC4CC;--shiki-dark-font-style:inherit}html .light .shiki span{color:var(--shiki-light);background:var(--shiki-light-bg);font-style:var(--shiki-light-font-style);font-weight:var(--shiki-light-font-weight);text-decoration:var(--shiki-light-text-decoration)}html.light .shiki span{color:var(--shiki-light);background:var(--shiki-light-bg);font-style:var(--shiki-light-font-style);font-weight:var(--shiki-light-font-weight);text-decoration:var(--shiki-light-text-decoration)}html .default .shiki span{color:var(--shiki-default);background:var(--shiki-default-bg);font-style:var(--shiki-default-font-style);font-weight:var(--shiki-default-font-weight);text-decoration:var(--shiki-default-text-decoration)}html .shiki span{color:var(--shiki-default);background:var(--shiki-default-bg);font-style:var(--shiki-default-font-style);font-weight:var(--shiki-default-font-weight);text-decoration:var(--shiki-default-text-decoration)}html .dark .shiki span{color:var(--shiki-dark);background:var(--shiki-dark-bg);font-style:var(--shiki-dark-font-style);font-weight:var(--shiki-dark-font-weight);text-decoration:var(--shiki-dark-text-decoration)}html.dark .shiki span{color:var(--shiki-dark);background:var(--shiki-dark-bg);font-style:var(--shiki-dark-font-style);font-weight:var(--shiki-dark-font-weight);text-decoration:var(--shiki-dark-text-decoration)}",{"title":294,"searchDepth":346,"depth":346,"links":1721},[1722,1723,1724,1725,1726,1727],{"id":863,"depth":346,"text":864},{"id":1234,"depth":346,"text":1235},{"id":1464,"depth":346,"text":1465},{"id":1645,"depth":346,"text":1646},{"id":1662,"depth":346,"text":1663},{"id":1682,"depth":346,"text":1683},"\u002Fimages\u002Fposts\u002Fai-performance-testing-guardrails\u002Fai-performance-testing-guardrails-cover.webp","2026-08-29","An AI-generated performance test missed a real bug, and the app broke in production. StarEast 2026 lessons on the guardrails that would have caught it.",{},"\u002Fsoftware-testing\u002Ftest-automation\u002Fai-performance-testing-guardrails",{"title":847,"description":1730},"software-testing\u002Ftest-automation\u002Fai-performance-testing-guardrails","r9dtt9EAy3QYEFk2tCph0o5I0SQmCUowpUA_KaYNrBI",{"id":1737,"title":1738,"bmcUsername":6,"body":1739,"cover":2107,"date":2108,"description":2109,"draft":629,"extension":630,"features":6,"githubRepo":6,"headline":6,"highlight":6,"icon":6,"meta":2110,"navigation":632,"npmPackage":6,"order":6,"path":2111,"seo":2112,"stem":2113,"__hash__":2114},"content\u002Fsoftware-testing\u002Ftest-automation\u002Fwhat-would-you-stop-doing-when-ui-tests-are-flaky.md","What Would You Stop Doing When UI Tests Are Flaky?",{"type":8,"value":1740,"toc":2094},[1741,1748,1754,1765,1768,1771,1774,1778,1782,1796,1803,1805,1809,1830,1845,1848,1855,1858,1860,1864,1867,1893,1896,1903,1905,1909,1916,1919,1923,1926,1929,1933,1936,1947,1958,1963,1967,1978,1981,1985,1988,1991,1994,1997,2000,2003,2007,2010,2017,2020,2034,2037,2041,2044,2047,2050,2052,2056,2059,2069,2079,2085,2091],[11,1742,1743,1744,1747],{},"This question ",[222,1745,1746],{},"about"," an interview question was recently posted in a QA forum, and the discussion it generated is more interesting than the question itself:",[1749,1750,1751],"blockquote",{},[11,1752,1753],{},"\"What would you stop doing when UI tests are flaky?\"",[11,1755,1756,1757,1760,1761,1764],{},"The phrasing trips people up. Most interview questions ask what you ",[222,1758,1759],{},"would do",", essentially what's your process, how do you handle it, what tools do you reach for. This one inverts it. It's asking about habits to ",[222,1762,1763],{},"eliminate",", which implies the interviewer already assumes you have them. It's also, perhaps intentionally, phrased awkwardly.",[11,1766,1767],{},"I've spent over 20 years in software testing across fintech, SaaS HCM, and insurtech and currently serve as the Director of Quality Engineering at my current employer. I haven't been asked this question in exactly this phrasing, but I've used similar ones from the other side of the table. I know what this type of question is designed to surface.",[11,1769,1770],{},"Before we get to the answer, let's look at what the QA community said. See if you can guess or click reveal to see all the survey responses.",[1772,1773],"hr",{},[15,1775,1777],{"id":1776},"survey-says-what-the-qa-community-answered-this-interview-question","Survey Says — What the QA Community Answered This Interview Question",[1779,1780],"flaky-test-survey",{":answers":1781},"[{\"text\":\"Stop using sleep() \u002F fix timing and waits\",\"votes\":25,\"keywords\":[\"sleep\",\"pause\",\"timing\",\"wait\",\"thread.sleep\",\"time.sleep\"]},{\"text\":\"Investigate root cause first\",\"votes\":11,\"keywords\":[\"investigate\",\"root cause\",\"diagnose\",\"why\",\"cause\",\"reason\"]},{\"text\":\"Quarantine tests from CI\",\"votes\":3,\"keywords\":[\"quarantine\",\"mute\",\"skip\",\"disable\",\"isolate\"]},{\"text\":\"Stop automating an unstable UI\",\"votes\":2,\"keywords\":[\"unstable\",\"automat\",\"flaky ui\",\"not ready\"]},{\"text\":\"Stop adding more tests\",\"votes\":2,\"keywords\":[\"adding\",\"add test\",\"more test\",\"new test\",\"expand\"]},{\"text\":\"Stop running tests in parallel\",\"votes\":1,\"keywords\":[\"parallel\",\"concurrent\",\"simultaneously\"]}]",[11,1783,1784,1785,1253,1788,1791,1792,1795],{},"The most popular community answers were technical and relatable — ",[222,1786,1787],{},"stop using sleep()",[222,1789,1790],{},"fix timing and waits"," — the instinctive responses from anyone who has spent time debugging intermittent failures. ",[222,1793,1794],{},"Investigate root cause first"," ranked lower by sheer volume but drew the most endorsement from people who paused to think about what was actually being asked.",[11,1797,1798,1799,1802],{},"I also ran a LinkedIn poll with the same question. It had 357 impressions and only 5 votes — low participation — but those 5 voters unanimously chose ",[222,1800,1801],{},"investigate root cause first",". The gap between the free-comment community vote pattern and the forced-choice poll result is itself telling: when people had to commit to one answer, they chose the diagnostic approach. When free-commenting, they led with the most relatable war story.",[1772,1804],{},[15,1806,1808],{"id":1807},"why-most-candidates-answer-the-wrong-question","Why Most Candidates Answer the Wrong Question",[11,1810,1811,1812,1253,1815,1253,1818,1821,1822,1825,1826,1829],{},"Here's what's worth pausing on: many of the most popular community answers — including ",[222,1813,1814],{},"quarantine tests from CI",[222,1816,1817],{},"add retry logic",[222,1819,1820],{},"report flakiness to the dev team"," — are valid responses to \"what would you ",[222,1823,1824],{},"do"," about flaky tests.\" They are not answers to \"what would you ",[222,1827,1828],{},"stop"," doing.\"",[11,1831,1832,1833,1836,1837,1840,1841,1844],{},"Quarantining is an action you ",[222,1834,1835],{},"add"," to your process. Retries are something you ",[222,1838,1839],{},"implement",". Reporting is something you ",[222,1842,1843],{},"start"," doing. None of these are things you stop.",[11,1846,1847],{},"The community's own discussion demonstrated the exact failure mode the question is designed to surface: answering a different question than the one being asked.",[11,1849,1850,1851,1854],{},"This is worth a conscious moment when you're in an interview seat. Before diving in, restate the question: ",[222,1852,1853],{},"\"So you're asking what habits I'd stop — not what I'd add to my process?\""," That one sentence signals precision under pressure, and precision matters.",[11,1856,1857],{},"When I'm conducting an interview, if a candidate is giving an answer that feels off, I'll ask them to repeat back their understanding of the question. Sometimes they're just wrong, but more often they didn't fully process it in the moment due to nerves, language barrier, or, in the case of remote interviews, dropped audio packets. The candidates who handle interviews best are the ones who preemptively restate their understanding before answering. It reads as both confident and careful (good qualities for testers and quality engineers).",[1772,1859],{},[15,1861,1863],{"id":1862},"what-this-flaky-test-interview-question-is-actually-testing","What This Flaky Test Interview Question Is Actually Testing",[11,1865,1866],{},"This question tests at least four things at once:",[591,1868,1869,1875,1881,1887],{},[594,1870,1871,1874],{},[49,1872,1873],{},"Technical knowledge"," — Do you know the common anti-patterns that cause flaky UI tests?",[594,1876,1877,1880],{},[49,1878,1879],{},"Diagnostic thinking"," — Can you reason about root causes rather than recite a fix list?",[594,1882,1883,1886],{},[49,1884,1885],{},"Listening comprehension"," — Did you actually process what was asked?",[594,1888,1889,1892],{},[49,1890,1891],{},"Confidence to challenge ambiguity"," — Will the candidate accept the awkwardly worded question or point that out and ask for clarification?",[11,1894,1895],{},"A junior answer names tactics: stop using sleep, fix your waits, add retries. Not wrong, but symptom-level.",[11,1897,1898,1899,1902],{},"An experienced answer narrates a ",[222,1900,1901],{},"thought process"," — how you'd identify what's causing the flakiness before deciding what to change. The \"stop doing\" framing is a clue. It's asking which habits you've already had to unlearn, implying you've operated at enough scale to have learned them the hard way.",[1772,1904],{},[15,1906,1908],{"id":1907},"what-to-stop-doing-when-ui-tests-are-flaky-the-full-answer","What to Stop Doing When UI Tests Are Flaky: The Full Answer",[11,1910,1911,1912,1915],{},"If asked this question in an interview, I'd clarify the framing first: ",[222,1913,1914],{},"\"Are you asking about common anti-patterns that lead to flakiness, or more about how I'd approach the investigation?\""," That distinction matters, and asking it signals diagnostic thinking before the answer even starts.",[11,1917,1918],{},"If they want the approach angle, this is how I'd answer.",[42,1920,1922],{"id":1921},"stop-adding-tests-to-an-unstable-suite","Stop Adding Tests to an Unstable Suite",[11,1924,1925],{},"This would be my first answer, and I'd lead with it.",[11,1927,1928],{},"Adding tests to a flaky suite compounds the problem. Every new test inherits the instability of the environment it runs in. Before expanding coverage, you need to stop the bleeding and understand whether the flakiness lives in the test code, the application behavior, or the infrastructure. That distinction determines the shape of your fix.",[42,1930,1932],{"id":1931},"stop-using-sleep-and-pause-statements","Stop Using sleep() and pause Statements",[11,1934,1935],{},"This is the answer that generates the most community agreement, and for good reason — it's the most widespread bad habit in UI test automation.",[11,1937,1938,744,1941,1944,1945,387],{},[283,1939,1940],{},"sleep()",[283,1942,1943],{},"pause"," are blunt instruments. They wait a fixed amount of time regardless of whether the condition they're waiting for became true a second in or never became true at all. They're slow, brittle, and mask the real problem: the test doesn't know what it's waiting ",[222,1946,1483],{},[11,1948,1949,1950,1953,1954,1957],{},"This is so well understood that Playwright formally marks ",[283,1951,1952],{},"page.waitForTimeout()"," as ",[222,1955,1956],{},"Discouraged"," in their own API docs:",[1749,1959,1960],{},[11,1961,1962],{},"\"Never wait for timeout in production. Tests that wait for time are inherently flaky. Use Locator actions and web assertions that wait automatically.\"",[26,1964],{"href":1965,"text":1966},"https:\u002F\u002Fplaywright.dev\u002Fdocs\u002Fapi\u002Fclass-page#page-wait-for-timeout","Playwright docs — page.waitForTimeout()",[11,1968,1969,1970,1973,1974,1977],{},"I've mandated the removal of pause statements from test suites I've managed and replaced them with explicit wait patterns — ",[283,1971,1972],{},"waitForElementPresent",", custom polling waits — anything that returns as soon as the condition is true rather than waiting out a fixed interval. I've added lint rules to prevent ",[283,1975,1976],{},".pause"," commands from being checked in at all. On one large serial suite, removing sleep and pause statements alone saved over an hour off the total test run time.",[11,1979,1980],{},"One practical detail: when setting a max wait timeout, I set it to roughly twice what I'd expect the worst case to be. CI environments consistently run slower than local development in ways that aren't always predictable. A wait that looks generous locally can time out under CI load.",[42,1982,1984],{"id":1983},"stop-assuming-the-problem-is-in-the-test-code","Stop Assuming the Problem Is in the Test Code",[11,1986,1987],{},"Some flakiness isn't in the test at all.",[11,1989,1990],{},"I had a test that failed intermittently depending on what time of day the build kicked off. After investigation, the root cause was a timezone mismatch between the server under test and the system running the tests. A validation rule in the application behaved differently at a specific hour because of this offset — the test was faithfully catching real behavior, but it looked like random flakiness until you looked closely enough. The initial investigation was tricky because it would pass during normal business hours when we tried to reproduce the failure in the first place!",[11,1992,1993],{},"The fix was a conditional branch in the test to account for the business rule at that magic hour. I generally avoid conditional branched logic in tests — it adds complexity and makes tests harder to reason about. But we couldn't time-travel or alter system clocks, and the conditional was the honest solution.",[11,1995,1996],{},"The point: before assuming the test is broken, determine whether you're dealing with test code, an application bug, or an infrastructure mismatch. The investigation approach is different for each.",[11,1998,1999],{},"It's also worth noting that some intermittent failures aren't flakiness at all — they're the test catching a real intermittent bug in the application. A test that fails once and passes on the next re-run looks identical to a flaky test on the surface. One is noise; the other is a signal you're about to dismiss. This is why every failure deserves investigation before it gets written off.",[11,2001,2002],{},"The goal is a suite trustworthy enough that the team's first instinct when a test fails is \"it found something\" — not \"ugh, it's flaky, just re-run it.\" The moment re-running becomes the default response, it becomes an annoying car alarm at 3 AM instead of a useful tool.",[42,2004,2006],{"id":2005},"stop-running-tests-in-parallel-without-isolating-shared-state","Stop Running Tests in Parallel Without Isolating Shared State",[11,2008,2009],{},"Parallelism is worth pursuing — the time savings on a large suite are significant, and it's one of the highest-leverage improvements you can make to CI feedback time. The problem isn't parallelism itself; it's running tests in parallel that were never designed for it.",[11,2011,2012,2013,2016],{},"Tests that share data, database state, or external resources become order-dependent and environment-dependent the moment you parallelize them. A suite that runs cleanly in serial can look deeply flaky in parallel for no obvious reason — because the flakiness is in the ",[222,2014,2015],{},"interaction"," between tests, not in any individual test.",[11,2018,2019],{},"The practical solution is to stop treating your suite as a single homogeneous run and start thinking in terms of what can safely run concurrently:",[1470,2021,2022,2028],{},[594,2023,2024,2027],{},[49,2025,2026],{},"Read-only tests"," — tests that only query state without mutating it — are natural candidates for parallel execution. They can't interfere with each other.",[594,2029,2030,2033],{},[49,2031,2032],{},"Write operations, state-dependent flows, and anything touching shared fixtures"," are better kept in a serial suite until you've isolated their data properly (unique test data per run, dedicated test accounts, isolated environments).",[11,2035,2036],{},"A combined approach — a parallel suite for safe tests and a serial suite for the rest — gets you most of the speed benefit while keeping the flakiness surface small. Once the serial tests are properly isolated with their own data, you can graduate them into the parallel suite over time.",[42,2038,2040],{"id":2039},"stop-treating-flakiness-as-normal","Stop Treating Flakiness as Normal",[11,2042,2043],{},"The most damaging thing a team can do with a flaky test is shrug and accept it.",[11,2045,2046],{},"Flakiness trains everyone to ignore failures. Once the build becomes a noise generator instead of a signal, real regressions slip through unchallenged. A test suite that cries wolf is functionally worse than no test suite, because it creates false confidence.",[11,2048,2049],{},"I've used flakiness scoring in both BitBucket and BrowserStack Test Analytics to identify and mute the worst offenders. Muting is not the same as deleting: the test still runs, it just doesn't fail the build while it's under investigation. That distinction matters — it preserves your ability to track whether improvements helped without letting the instability contaminate every build in the meantime.",[1772,2051],{},[15,2053,2055],{"id":2054},"how-to-answer-flaky-ui-test-interview-questions","How to Answer Flaky UI Test Interview Questions",[11,2057,2058],{},"A few framing notes regardless of how you structure your answer:",[11,2060,2061,2064,2065,2068],{},[49,2062,2063],{},"Restate first."," Before diving in, confirm you understood the question. ",[222,2066,2067],{},"\"So you're asking what habits I'd stop, not what I'd add to my process?\""," One sentence of confirmation demonstrates careful listening — which is arguably what the question is testing most.",[11,2070,2071,2074,2075,2078],{},[49,2072,2073],{},"Narrate, don't list."," A list of tactics sounds like you memorized a checklist. A thought process — ",[222,2076,2077],{},"\"I'd start by determining whether this is test code, application behavior, or environment, because the fix is different for each\""," — sounds like someone who has actually dealt with this at scale.",[11,2080,2081,2084],{},[49,2082,2083],{},"Distinguish the problem type."," Not all flakiness has the same root cause. Timing issues, shared state, environment inconsistency, and automating an unstable UI are four different problems with four different fixes. Showing you can distinguish them is what separates a good answer from a more experienced one.",[11,2086,2087,2090],{},[49,2088,2089],{},"Own a specific example."," The most memorable interview answers are concrete. If you've refactored a suite full of sleep statements, or tracked down a timezone mismatch that looked like random flakiness for weeks, say so. Specific experience is more credible than correct-sounding generalizations.",[607,2092],{":items":2093},"[\"\u002Fsoftware-testing\u002Ftest-automation\u002Fhow-to-handle-failing-tests-caused-by-known-bugs\",\"\u002Fsoftware-testing\u002Ftest-automation\u002Fai-in-testing-2026-state-of-the-industry\",\"\u002Fsoftware-testing\u002Ftest-automation\u002Frun-fewer-tests-catch-the-same-bugs\",\"\u002Fsoftware-testing\u002Ftest-automation\u002Fai-performance-testing-guardrails\"]",{"title":294,"searchDepth":346,"depth":346,"links":2095},[2096,2097,2098,2099,2106],{"id":1776,"depth":346,"text":1777},{"id":1807,"depth":346,"text":1808},{"id":1862,"depth":346,"text":1863},{"id":1907,"depth":346,"text":1908,"children":2100},[2101,2102,2103,2104,2105],{"id":1921,"depth":378,"text":1922},{"id":1931,"depth":378,"text":1932},{"id":1983,"depth":378,"text":1984},{"id":2005,"depth":378,"text":2006},{"id":2039,"depth":378,"text":2040},{"id":2054,"depth":346,"text":2055},"\u002Fimages\u002Fposts\u002Fwhat-would-you-stop-doing-when-ui-tests-are-flaky\u002Fwhat-would-you-stop-doing-when-ui-tests-are-flaky-cover.webp","2026-05-16","Most QA engineers answer this interview question confidently wrong. Here's what \"What would you stop doing when UI tests are flaky?\" is actually testing and what an experienced answer sounds like.",{},"\u002Fsoftware-testing\u002Ftest-automation\u002Fwhat-would-you-stop-doing-when-ui-tests-are-flaky",{"title":1738,"description":2109},"software-testing\u002Ftest-automation\u002Fwhat-would-you-stop-doing-when-ui-tests-are-flaky","J3cN721HjJWyvMOPlZArPiG3YREFE1qMDtp5Lcsm2n4",{"id":2116,"title":2117,"bmcUsername":6,"body":2118,"cover":3265,"date":3266,"description":3267,"draft":629,"extension":630,"features":6,"githubRepo":6,"headline":6,"highlight":6,"icon":6,"meta":3268,"navigation":632,"npmPackage":6,"order":6,"path":1707,"seo":3269,"stem":3270,"__hash__":3271},"content\u002Fsoftware-testing\u002Ftest-automation\u002Fstareast-2026-playwright-ai-cost-efficient-testing.md","Playwright AI Testing on a Budget: Locators vs. Computer Vision — StarEast 2026",{"type":8,"value":2119,"toc":3257},[2120,2123,2142,2145,2151,2155,2158,2161,2166,2169,2172,2177,2184,2188,2195,2201,2208,2211,2946,2961,2965,2968,2971,2984,2989,2992,2998,3003,3035,3046,3049,3052,3072,3078,3085,3092,3097,3100,3132,3135,3140,3143,3147,3154,3159,3162,3167,3170,3175,3182,3186,3189,3194,3197,3200,3207,3210,3214,3217,3243,3251,3254],[11,2121,2122],{},"Andy Knight's half-day StarEast 2026 tutorial, officially titled \"Top-Notch Web Testing with Playwright and AI,\" was billed as a hands-on walkthrough, and for most of its four hours, that's exactly what it was. Two claims kept it from being just another how-to for me. Playwright's MCP server can burn through an AI testing budget fast enough to matter (one joke about a junior developer's $5,000 month illustrated that point), and computer vision based testing, despite what a different StarEast tutorial argued the day before, is unlikely to replace locator-based Playwright tests anytime soon.",[11,2124,2125,2126,2130,2131,744,2134,2137,2138,2141],{},"Knight, who goes by Pandy or Automation Panda depending on which corner of the testing internet you found him in, is an actual ",[26,2127],{"href":2128,"text":2129},"https:\u002F\u002Fautomationpanda.com","Playwright Ambassador",". His session was the third of four StarEast 2026 tutorials I attended over two days, the first two are their own write-ups, on ",[1697,2132,2133],{"href":1699},"getting started with AI-driven automation and AI vision testing",[1697,2135,2136],{"href":1703},"evals, vibe coding, and prompt engineering",". Knight acknowledged near the end that the class hadn't gotten through the whole tutorial repository live, \"we only got through about half of what's in the tutorial repository.\" Part of that had a funny explanation: Knight assumed most of the class had simply ignored the prerequisite machine setup instructions he'd sent out ahead of time. It turned out the StarEast organizers never actually emailed those instructions to anyone. So the room spent a chunk of class scrambling to install several hundred megabytes of Playwright's browser dependencies over the now-saturated conference Wi-Fi. The organizers only figured out what happened when they noticed the network anomaly and mentioned it to Knight, at which point I felt vindicated, I'd been ",[222,2139,2140],{},"certain"," no such instructions were ever sent and had assumed I'd just failed to do my homework.",[11,2143,2144],{},"Everything below is what we actually built and discussed in the room, plus what I read in his written tutorial chapters afterward to fill in gaps.",[11,2146,2147],{},[72,2148],{"alt":2149,"src":2150},"Andy Knight presenting his Playwright and AI tutorial at StarEast 2026","\u002Fimages\u002Fposts\u002Fstareast-2026-playwright-ai-cost-efficient-testing\u002Fandy-knight-stareast-2026.webp",[15,2152,2154],{"id":2153},"playwright-vs-selenium-what-actually-got-fixed","Playwright vs. Selenium: What Actually Got Fixed",[11,2156,2157],{},"Knight opened by asking the room what makes test automation hard, and the answers came fast: tests are slow, brittle, flaky, don't make sense when you read them back, don't make money (a real line, \"we're not shipping tests to customers\"), and force a context switch every time you flip from building a feature to testing it.",[11,2159,2160],{},"The classic fix for this was the Testing Pyramid, lots of cheap unit tests at the base, fewer expensive UI tests at the top, because UI tests were \"big, slow, and expensive.\" Knight's pushback wasn't that the pyramid's diagnosis was wrong. It was that the diagnosis got blamed on the wrong cause:",[1749,2162,2163],{},[11,2164,2165],{},"\"End-to-end tests can be very valuable. Unfortunately, the Testing Pyramid labeled them as 'difficult' and 'bad' primarily due to poor practices and tool shortcomings.\"",[11,2167,2168],{},"He had a punchier name for what should replace pyramid-style thinking (\"we don't build pyramids anymore, we build skyscrapers\"). We'll revisit that line in a later section because I don't think it holds up quite as cleanly as it sounded in the room at the time.",[11,2170,2171],{},"What does hold up is the tooling argument. Playwright's actual fix for \"UI tests are slow and flaky\" is architectural: one browser instance per worker, with each test pulling its own isolated browser context out of that instance (\"akin to an incognito session, or a mini container in your browser\"), and each context holding one or more pages. Spinning up a context is nearly instant, which is the opposite of Selenium's per-test full-browser-relaunch model. Knight's own story below, about discovering this, resonated with me because I had a similar reaction when using Playwright for the first time.",[1749,2173,2174],{},[11,2175,2176],{},"\"I remember the first time I used Playwright, this was back in late 2021... I quickly bang out about a dozen tests or so... I go to the terminal, I'm like npx Playwright test, run it, hit it, and then within a second it comes back and it says 12 tests passed. And I'm like, no, no, no, no, no, it didn't find the tests, it didn't run the tests, it skipped it, something went wrong... then I run it in headed mode, and it was so fast... I was expecting each test to take about a minute, because I came from Selenium, but it's like when I say it's freaky fast man, it is, it screams.\"",[11,2178,2179,2180,2183],{},"Playwright avoids the behavior that gives Selenium its flaky reputation by, among other things, polling automatically: locators and assertions keep rechecking until they succeed or time out, instead of failing the instant they're called, if misaligned. Selenium does the opposite by default, checking once, so a test that forgets to include explicit waits fails the moment the page hasn't caught up yet. Playwright's defaults give that polling a generous window: locator actions retry for 30 seconds, ",[283,2181,2182],{},"expect"," assertions for 5, enough slack to absorb a slower page load between runs without anyone configuring a thing. Knight was fair to say, \"Selenium itself is not flaky, it's the tests that people write with it.\" Playwright's real contribution is removing a specific set of execution-speed and tooling-friction problems that made E2E testing painful for the last decade, not inventing testing concepts from scratch.",[15,2185,2187],{"id":2186},"from-codegen-to-a-real-test","From Codegen to a Real Test",[11,2189,2190,2191,2194],{},"The hands-on portion started with ",[283,2192,2193],{},"npx playwright codegen"," against a local Trello-style Kanban app (a clone built by Filip Hric, used with permission). Codegen records your clicks and fills into a script, and the output is rough on purpose, Knight's framing: \"there's a difference between a script and a test case... we can use this to ruthlessly refine it into a better test case.\"",[11,2196,2197],{},[72,2198],{"alt":2199,"src":2200},"Trello app being tested","\u002Fimages\u002Fposts\u002Fstareast-2026-playwright-ai-cost-efficient-testing\u002Ftrello-app-under-test-listview.webp",[11,2202,2203,2204,2207],{},"Refining it meant three things: trimming the clicks codegen over-records (you don't need to click an input before typing into it), picking stable locators (",[283,2205,2206],{},"data-testid"," attributes if you control the app, \"these are very nice test hooks to have\"), and adding the assertions codegen never gives you, since codegen only captures interactions, not verifications.",[11,2209,2210],{},"We iterated from the raw click events through refining the flow so it could be run repeatedly by adding things like pre and post test hooks to ensure the test launches in the correct state and doesn't leave behind past entries that would cause different state between runs. Here's my own rough version of that test, written live in the room.:",[288,2212,2217],{"className":2213,"code":2214,"filename":2215,"language":2216,"meta":294,"style":294},"language-typescript shiki shiki-themes material-theme-lighter github-light-high-contrast github-dark-high-contrast","import { test, expect } from '@playwright\u002Ftest';\n\ntest.beforeEach(async ({ page, request }) => {\n  \u002F\u002F Added this reset endpoint to erase the board and then naivate to the app at the start of each test run\n  await request.post('http:\u002F\u002Flocalhost:3000\u002Fapi\u002Freset');\n  await page.goto('http:\u002F\u002Flocalhost:3000\u002F');\n});\n\ntest.afterEach(async ({ request }) => {\n  \u002F\u002F Added this explicit reset after each test to erase the board (belt and suspenders with the beforeEach's erase)\n  await request.post('http:\u002F\u002Flocalhost:3000\u002Fapi\u002Freset');\n});\n\ntest.afterAll(async ({ browser }) => {\n  \u002F\u002F Added to close down the browser after all the tests complete\n  await browser.close();\n});\n\ntest('Create a new board with list and cards', async ({ page }) => {\n  \u002F\u002F You'll notice the selector repetition and lack of page objects which we didn't get to during the session \u002F wasn't a primary focus\n  await page.getByTestId('first-board').click();\n  await page.getByTestId('first-board').fill('chores');\n  await page.getByTestId('first-board').press('Enter');\n\n  expect(page.getByTestId('first-board')).toHaveValue('chores');\n\n  await page.getByTestId('add-list-input').click();\n  await page.getByTestId('add-list-input').fill('todo');\n  await page.getByRole('button', { name: 'Add list' }).click();\n  await page.getByTestId('new-card').click();\n  await page.getByTestId('new-card-input').fill('walk the dog');\n  await page.getByTestId('new-card-input').click();\n  await page.getByTestId('new-card-input').fill('mow the lawn');\n  await page.getByTestId('home').click();\n\n  \u002F\u002F Didn't have a chance to add more assertions, was helping classmates with setup.\n});\n","trello.spec.ts","typescript",[283,2218,2219,2246,2250,2279,2284,2307,2331,2339,2343,2366,2371,2393,2401,2405,2429,2434,2449,2457,2461,2488,2493,2523,2561,2599,2603,2644,2648,2677,2714,2762,2791,2829,2858,2896,2926,2931,2937],{"__ignoreMap":294},[298,2220,2221,2223,2225,2228,2230,2233,2235,2237,2239,2242,2244],{"class":300,"line":301},[298,2222,893],{"class":381},[298,2224,498],{"class":322},[298,2226,2227],{"class":308}," test",[298,2229,323],{"class":322},[298,2231,2232],{"class":308}," expect",[298,2234,513],{"class":322},[298,2236,922],{"class":381},[298,2238,506],{"class":312},[298,2240,2241],{"class":316},"@playwright\u002Ftest",[298,2243,313],{"class":312},[298,2245,375],{"class":322},[298,2247,2248],{"class":300,"line":346},[298,2249,936],{"emptyLinePlaceholder":632},[298,2251,2252,2254,2256,2259,2261,2264,2266,2268,2270,2273,2275,2277],{"class":300,"line":378},[298,2253,305],{"class":308},[298,2255,387],{"class":322},[298,2257,2258],{"class":304},"beforeEach",[298,2260,309],{"class":308},[298,2262,2263],{"class":326},"async",[298,2265,330],{"class":322},[298,2267,334],{"class":333},[298,2269,323],{"class":322},[298,2271,2272],{"class":333}," request",[298,2274,337],{"class":322},[298,2276,340],{"class":326},[298,2278,343],{"class":322},[298,2280,2281],{"class":300,"line":398},[298,2282,2283],{"class":1208},"  \u002F\u002F Added this reset endpoint to erase the board and then naivate to the app at the start of each test run\n",[298,2285,2286,2288,2290,2292,2294,2296,2298,2301,2303,2305],{"class":300,"line":520},[298,2287,382],{"class":381},[298,2289,2272],{"class":308},[298,2291,387],{"class":322},[298,2293,484],{"class":304},[298,2295,309],{"class":366},[298,2297,313],{"class":312},[298,2299,2300],{"class":316},"http:\u002F\u002Flocalhost:3000\u002Fapi\u002Freset",[298,2302,313],{"class":312},[298,2304,372],{"class":366},[298,2306,375],{"class":322},[298,2308,2309,2311,2313,2315,2318,2320,2322,2325,2327,2329],{"class":300,"line":526},[298,2310,382],{"class":381},[298,2312,334],{"class":308},[298,2314,387],{"class":322},[298,2316,2317],{"class":304},"goto",[298,2319,309],{"class":366},[298,2321,313],{"class":312},[298,2323,2324],{"class":316},"http:\u002F\u002Flocalhost:3000\u002F",[298,2326,313],{"class":312},[298,2328,372],{"class":366},[298,2330,375],{"class":322},[298,2332,2333,2335,2337],{"class":300,"line":983},[298,2334,401],{"class":322},[298,2336,372],{"class":308},[298,2338,375],{"class":322},[298,2340,2341],{"class":300,"line":989},[298,2342,936],{"emptyLinePlaceholder":632},[298,2344,2345,2347,2349,2352,2354,2356,2358,2360,2362,2364],{"class":300,"line":994},[298,2346,305],{"class":308},[298,2348,387],{"class":322},[298,2350,2351],{"class":304},"afterEach",[298,2353,309],{"class":308},[298,2355,2263],{"class":326},[298,2357,330],{"class":322},[298,2359,2272],{"class":333},[298,2361,337],{"class":322},[298,2363,340],{"class":326},[298,2365,343],{"class":322},[298,2367,2368],{"class":300,"line":1011},[298,2369,2370],{"class":1208},"  \u002F\u002F Added this explicit reset after each test to erase the board (belt and suspenders with the beforeEach's erase)\n",[298,2372,2373,2375,2377,2379,2381,2383,2385,2387,2389,2391],{"class":300,"line":1019},[298,2374,382],{"class":381},[298,2376,2272],{"class":308},[298,2378,387],{"class":322},[298,2380,484],{"class":304},[298,2382,309],{"class":366},[298,2384,313],{"class":312},[298,2386,2300],{"class":316},[298,2388,313],{"class":312},[298,2390,372],{"class":366},[298,2392,375],{"class":322},[298,2394,2395,2397,2399],{"class":300,"line":1056},[298,2396,401],{"class":322},[298,2398,372],{"class":308},[298,2400,375],{"class":322},[298,2402,2403],{"class":300,"line":1061},[298,2404,936],{"emptyLinePlaceholder":632},[298,2406,2407,2409,2411,2414,2416,2418,2420,2423,2425,2427],{"class":300,"line":1090},[298,2408,305],{"class":308},[298,2410,387],{"class":322},[298,2412,2413],{"class":304},"afterAll",[298,2415,309],{"class":308},[298,2417,2263],{"class":326},[298,2419,330],{"class":322},[298,2421,2422],{"class":333}," browser",[298,2424,337],{"class":322},[298,2426,340],{"class":326},[298,2428,343],{"class":322},[298,2430,2431],{"class":300,"line":1125},[298,2432,2433],{"class":1208},"  \u002F\u002F Added to close down the browser after all the tests complete\n",[298,2435,2436,2438,2440,2442,2445,2447],{"class":300,"line":1131},[298,2437,382],{"class":381},[298,2439,2422],{"class":308},[298,2441,387],{"class":322},[298,2443,2444],{"class":304},"close",[298,2446,393],{"class":366},[298,2448,375],{"class":322},[298,2450,2451,2453,2455],{"class":300,"line":1136},[298,2452,401],{"class":322},[298,2454,372],{"class":308},[298,2456,375],{"class":322},[298,2458,2459],{"class":300,"line":1187},[298,2460,936],{"emptyLinePlaceholder":632},[298,2462,2463,2465,2467,2469,2472,2474,2476,2478,2480,2482,2484,2486],{"class":300,"line":1205},[298,2464,305],{"class":304},[298,2466,309],{"class":308},[298,2468,313],{"class":312},[298,2470,2471],{"class":316},"Create a new board with list and cards",[298,2473,313],{"class":312},[298,2475,323],{"class":322},[298,2477,327],{"class":326},[298,2479,330],{"class":322},[298,2481,334],{"class":333},[298,2483,337],{"class":322},[298,2485,340],{"class":326},[298,2487,343],{"class":322},[298,2489,2490],{"class":300,"line":1212},[298,2491,2492],{"class":1208},"  \u002F\u002F You'll notice the selector repetition and lack of page objects which we didn't get to during the session \u002F wasn't a primary focus\n",[298,2494,2495,2497,2499,2501,2504,2506,2508,2511,2513,2515,2517,2519,2521],{"class":300,"line":1218},[298,2496,382],{"class":381},[298,2498,334],{"class":308},[298,2500,387],{"class":322},[298,2502,2503],{"class":304},"getByTestId",[298,2505,309],{"class":366},[298,2507,313],{"class":312},[298,2509,2510],{"class":316},"first-board",[298,2512,313],{"class":312},[298,2514,372],{"class":366},[298,2516,387],{"class":322},[298,2518,455],{"class":304},[298,2520,393],{"class":366},[298,2522,375],{"class":322},[298,2524,2525,2527,2529,2531,2533,2535,2537,2539,2541,2543,2545,2548,2550,2552,2555,2557,2559],{"class":300,"line":1223},[298,2526,382],{"class":381},[298,2528,334],{"class":308},[298,2530,387],{"class":322},[298,2532,2503],{"class":304},[298,2534,309],{"class":366},[298,2536,313],{"class":312},[298,2538,2510],{"class":316},[298,2540,313],{"class":312},[298,2542,372],{"class":366},[298,2544,387],{"class":322},[298,2546,2547],{"class":304},"fill",[298,2549,309],{"class":366},[298,2551,313],{"class":312},[298,2553,2554],{"class":316},"chores",[298,2556,313],{"class":312},[298,2558,372],{"class":366},[298,2560,375],{"class":322},[298,2562,2563,2565,2567,2569,2571,2573,2575,2577,2579,2581,2583,2586,2588,2590,2593,2595,2597],{"class":300,"line":1403},[298,2564,382],{"class":381},[298,2566,334],{"class":308},[298,2568,387],{"class":322},[298,2570,2503],{"class":304},[298,2572,309],{"class":366},[298,2574,313],{"class":312},[298,2576,2510],{"class":316},[298,2578,313],{"class":312},[298,2580,372],{"class":366},[298,2582,387],{"class":322},[298,2584,2585],{"class":304},"press",[298,2587,309],{"class":366},[298,2589,313],{"class":312},[298,2591,2592],{"class":316},"Enter",[298,2594,313],{"class":312},[298,2596,372],{"class":366},[298,2598,375],{"class":322},[298,2600,2601],{"class":300,"line":1408},[298,2602,936],{"emptyLinePlaceholder":632},[298,2604,2605,2608,2610,2612,2614,2616,2618,2620,2622,2624,2627,2629,2632,2634,2636,2638,2640,2642],{"class":300,"line":1414},[298,2606,2607],{"class":304},"  expect",[298,2609,309],{"class":366},[298,2611,369],{"class":308},[298,2613,387],{"class":322},[298,2615,2503],{"class":304},[298,2617,309],{"class":366},[298,2619,313],{"class":312},[298,2621,2510],{"class":316},[298,2623,313],{"class":312},[298,2625,2626],{"class":366},"))",[298,2628,387],{"class":322},[298,2630,2631],{"class":304},"toHaveValue",[298,2633,309],{"class":366},[298,2635,313],{"class":312},[298,2637,2554],{"class":316},[298,2639,313],{"class":312},[298,2641,372],{"class":366},[298,2643,375],{"class":322},[298,2645,2646],{"class":300,"line":1419},[298,2647,936],{"emptyLinePlaceholder":632},[298,2649,2650,2652,2654,2656,2658,2660,2662,2665,2667,2669,2671,2673,2675],{"class":300,"line":1425},[298,2651,382],{"class":381},[298,2653,334],{"class":308},[298,2655,387],{"class":322},[298,2657,2503],{"class":304},[298,2659,309],{"class":366},[298,2661,313],{"class":312},[298,2663,2664],{"class":316},"add-list-input",[298,2666,313],{"class":312},[298,2668,372],{"class":366},[298,2670,387],{"class":322},[298,2672,455],{"class":304},[298,2674,393],{"class":366},[298,2676,375],{"class":322},[298,2678,2679,2681,2683,2685,2687,2689,2691,2693,2695,2697,2699,2701,2703,2705,2708,2710,2712],{"class":300,"line":1431},[298,2680,382],{"class":381},[298,2682,334],{"class":308},[298,2684,387],{"class":322},[298,2686,2503],{"class":304},[298,2688,309],{"class":366},[298,2690,313],{"class":312},[298,2692,2664],{"class":316},[298,2694,313],{"class":312},[298,2696,372],{"class":366},[298,2698,387],{"class":322},[298,2700,2547],{"class":304},[298,2702,309],{"class":366},[298,2704,313],{"class":312},[298,2706,2707],{"class":316},"todo",[298,2709,313],{"class":312},[298,2711,372],{"class":366},[298,2713,375],{"class":322},[298,2715,2716,2718,2720,2722,2725,2727,2729,2732,2734,2736,2738,2741,2743,2745,2748,2750,2752,2754,2756,2758,2760],{"class":300,"line":1437},[298,2717,382],{"class":381},[298,2719,334],{"class":308},[298,2721,387],{"class":322},[298,2723,2724],{"class":304},"getByRole",[298,2726,309],{"class":366},[298,2728,313],{"class":312},[298,2730,2731],{"class":316},"button",[298,2733,313],{"class":312},[298,2735,323],{"class":322},[298,2737,498],{"class":322},[298,2739,2740],{"class":366}," name",[298,2742,286],{"class":322},[298,2744,506],{"class":312},[298,2746,2747],{"class":316},"Add list",[298,2749,313],{"class":312},[298,2751,513],{"class":322},[298,2753,372],{"class":366},[298,2755,387],{"class":322},[298,2757,455],{"class":304},[298,2759,393],{"class":366},[298,2761,375],{"class":322},[298,2763,2764,2766,2768,2770,2772,2774,2776,2779,2781,2783,2785,2787,2789],{"class":300,"line":1443},[298,2765,382],{"class":381},[298,2767,334],{"class":308},[298,2769,387],{"class":322},[298,2771,2503],{"class":304},[298,2773,309],{"class":366},[298,2775,313],{"class":312},[298,2777,2778],{"class":316},"new-card",[298,2780,313],{"class":312},[298,2782,372],{"class":366},[298,2784,387],{"class":322},[298,2786,455],{"class":304},[298,2788,393],{"class":366},[298,2790,375],{"class":322},[298,2792,2793,2795,2797,2799,2801,2803,2805,2808,2810,2812,2814,2816,2818,2820,2823,2825,2827],{"class":300,"line":1448},[298,2794,382],{"class":381},[298,2796,334],{"class":308},[298,2798,387],{"class":322},[298,2800,2503],{"class":304},[298,2802,309],{"class":366},[298,2804,313],{"class":312},[298,2806,2807],{"class":316},"new-card-input",[298,2809,313],{"class":312},[298,2811,372],{"class":366},[298,2813,387],{"class":322},[298,2815,2547],{"class":304},[298,2817,309],{"class":366},[298,2819,313],{"class":312},[298,2821,2822],{"class":316},"walk the dog",[298,2824,313],{"class":312},[298,2826,372],{"class":366},[298,2828,375],{"class":322},[298,2830,2832,2834,2836,2838,2840,2842,2844,2846,2848,2850,2852,2854,2856],{"class":300,"line":2831},32,[298,2833,382],{"class":381},[298,2835,334],{"class":308},[298,2837,387],{"class":322},[298,2839,2503],{"class":304},[298,2841,309],{"class":366},[298,2843,313],{"class":312},[298,2845,2807],{"class":316},[298,2847,313],{"class":312},[298,2849,372],{"class":366},[298,2851,387],{"class":322},[298,2853,455],{"class":304},[298,2855,393],{"class":366},[298,2857,375],{"class":322},[298,2859,2861,2863,2865,2867,2869,2871,2873,2875,2877,2879,2881,2883,2885,2887,2890,2892,2894],{"class":300,"line":2860},33,[298,2862,382],{"class":381},[298,2864,334],{"class":308},[298,2866,387],{"class":322},[298,2868,2503],{"class":304},[298,2870,309],{"class":366},[298,2872,313],{"class":312},[298,2874,2807],{"class":316},[298,2876,313],{"class":312},[298,2878,372],{"class":366},[298,2880,387],{"class":322},[298,2882,2547],{"class":304},[298,2884,309],{"class":366},[298,2886,313],{"class":312},[298,2888,2889],{"class":316},"mow the lawn",[298,2891,313],{"class":312},[298,2893,372],{"class":366},[298,2895,375],{"class":322},[298,2897,2899,2901,2903,2905,2907,2909,2911,2914,2916,2918,2920,2922,2924],{"class":300,"line":2898},34,[298,2900,382],{"class":381},[298,2902,334],{"class":308},[298,2904,387],{"class":322},[298,2906,2503],{"class":304},[298,2908,309],{"class":366},[298,2910,313],{"class":312},[298,2912,2913],{"class":316},"home",[298,2915,313],{"class":312},[298,2917,372],{"class":366},[298,2919,387],{"class":322},[298,2921,455],{"class":304},[298,2923,393],{"class":366},[298,2925,375],{"class":322},[298,2927,2929],{"class":300,"line":2928},35,[298,2930,936],{"emptyLinePlaceholder":632},[298,2932,2934],{"class":300,"line":2933},36,[298,2935,2936],{"class":1208},"  \u002F\u002F Didn't have a chance to add more assertions, was helping classmates with setup.\n",[298,2938,2940,2942,2944],{"class":300,"line":2939},37,[298,2941,401],{"class":322},[298,2943,372],{"class":308},[298,2945,375],{"class":322},[11,2947,2948,2949,2952,2953,2956,2957,2960],{},"Test data was the other rough edge. The app resets its entire backend through a ",[283,2950,2951],{},"\u002Fapi\u002Freset"," endpoint, called via Playwright's ",[283,2954,2955],{},"request"," fixture, and Knight was explicit that this was a deliberate, temporary shortcut: \"Remember, this is a tutorial, friends. Don't do this for real... Do not say automation panda told me to drop my whole database as test setup. No, he did not.\" The honest cost of that shortcut showed up immediately: resetting the whole database before every test means tests can't run in parallel, so the class was capped at ",[283,2958,2959],{},"--workers 1"," for the rest of the session. Fixing that properly (per-test data instead of a global wipe) is exactly the kind of thing that's covered in the tutorial's later, unreached chapters, more on that near the end of this article.",[15,2962,2964],{"id":2963},"the-efficient-ai-workflow-playwright-cli-vs-mcp","The Efficient AI Workflow: Playwright CLI vs. MCP",[11,2966,2967],{},"Coming into this session, I'd already absorbed the soundbite that Playwright's CLI is more token-efficient than its MCP server, but nobody had explained why, and I had a more basic confusion sitting underneath that one: the CLI is just terminal commands, so in what sense is that even \"AI\"? Knight's session got me most of the way to an answer. It didn't fully click until I went and read more on my own afterward.",[11,2969,2970],{},"Once the manual test was working, Knight pivoted to AI, with an important framing up front: \"Playwright doesn't bring its own model, it doesn't bring its own magic. Basically what it does is it brings tooling to integrate into existing AI coding agents.\" You still need Claude, Cursor, Copilot, or Codex. Playwright gives that agent two different ways to actually drive a browser.",[11,2972,2973,2976,2977,744,2980,2983],{},[49,2974,2975],{},"MCP"," (Model Context Protocol) exposes structured tools like ",[283,2978,2979],{},"browser_navigate",[283,2981,2982],{},"browser_snapshot"," to your coding agent. It works well, and it's expensive. Knight's framing of why, in full:",[1749,2985,2986],{},[11,2987,2988],{},"\"There's a problem with MCP. Does anybody know the problem with MCP? Burns a lot of tokens. It burns a heckin' ton of tokens... Intelligence is a utility. You pay a power bill, you pay a water bill. Guess what we're all paying for next? An intelligence bill.\"",[11,2990,2991],{},"The joke that opened this article followed directly: a junior developer who ran up a $5,000 month using MCP without understanding the cost. The mechanism, explained later in the session, isn't about which model you use, it's that MCP's tool schemas and structured page snapshots eat far more context window per step than a plain terminal command does, which forces more turns, which burns more tokens.",[11,2993,2994,2997],{},[49,2995,2996],{},"Playwright's CLI"," does the same browser-driving job as MCP, as plain terminal commands instead of structured tool calls, and according to Knight, \"uses a tenth of the tokens.\" His actual decision rule, given directly in response to \"why would you ever use MCP if the CLI is so much cheaper\":",[1749,2999,3000],{},[11,3001,3002],{},"\"The CLI is really good if you are doing the workflow that we are doing, for test developers, for grinding out some code, with coding agents CLI is better. But let's say that you wanted a more agentic workflow that wasn't you coding. Let's say you had to use Playwright as a browser automation tool in some way, writing a web scraper or web browser. In those cases the MCP is going to be better than the CLI. Because the MCP can be hosted on a network that you can reach out to it back and forth. CLI is all local to your machine.\"",[11,3004,3005,3006,3009,3010,1067,3013,1253,3015,3018,3019,3022,3023,3026,3027,3030,3031,3034],{},"Here's the part that actually answered both of my questions, the AI-or-not question and the why-tokens question, together. Both MCP and the CLI are AI-driven, in both cases the coding agent itself is deciding what to do and reading the result back. ",[49,3007,3008],{},"The difference is just what vocabulary it uses to act."," MCP issues ",[49,3011,3012],{},"structured tool calls",[283,3014,2979],{},[283,3016,3017],{},"browser_click",") over a protocol built on JSON-RPC, so the call and its ",[49,3020,3021],{},"full response travel through the model's context every time",". The ",[49,3024,3025],{},"CLI"," has the agent run ",[49,3028,3029],{},"literal shell commands"," against itself, something like ",[283,3032,3033],{},"playwright-cli click e21",", the same way it would run any other terminal command in a coding session.",[11,3036,3037,3038,3041,3042,3045],{},"That's also where the token savings actually come from. ",[49,3039,3040],{},"MCP has to keep the page's structure resident in the session's context for as long as the agent is working with it."," The CLI's skills are markdown files sitting on disk, ",[49,3043,3044],{},"read in only when something needs them",", then left there. One holds everything it might need in memory the whole time. The other fetches what it needs and sets it back down.",[11,3047,3048],{},"That also sharpens Knight's own rule (local machine versus network-hosted) into something more concrete. The CLI needs a real terminal, a filesystem, and the ability to spawn its own processes, exactly what you have during local development, and exactly what you don't have everywhere else. MCP doesn't need any of that, which is why it's the better fit in more locked-down or remote contexts: AI-assisted CI failure triage running inside a pipeline with no terminal session attached, for instance, or a low-code product where an agent runs server-side and a non-technical user just describes a test case in plain English, with no shell ever exposed to that agent at all.",[11,3050,3051],{},"Three more habits from the session genuinely earn their place under an efficiency banner, each backed by Knight's own stated reasoning rather than just a vibe:",[1470,3053,3054,3060,3066],{},[594,3055,3056,3059],{},[49,3057,3058],{},"Skills over re-explaining."," Installing CLI skills (markdown files that teach the agent what commands exist) means you're not \"pasting huge help text into every prompt.\" It's explicitly part of why the CLI uses fewer tokens than MCP in his own comparison, skills are loaded only when needed instead of being baked into every tool call.",[594,3061,3062,3065],{},[49,3063,3064],{},"Save state to markdown instead of letting it evaporate."," When Knight had the agent save a generated test plan to a file rather than leaving it in chat, his reasoning doubled as a genuinely good explanation of why: \"Your context window is only so big... if I didn't save my test plan in this markdown file, I'd have to make it regenerate the test plan again. That sucks.\" He compared it to saving progress in an old Super Nintendo game before your context window (or your save file) gets wiped.",[594,3067,3068,3071],{},[49,3069,3070],{},"Inside-out test generation."," Rather than guessing a locator, running the test, watching it crash, and correcting, Playwright's CLI and MCP tooling let the agent build a session step by step, discovering real locators as it goes. \"That usually leads to very short loops, not having to repeat a lot of loops.\" It's a real efficiency argument and it's specific to how Playwright's own tooling is built, not a generic prompting tip.",[11,3073,3074,3075,3077],{},"Knight also argued that AI-assisted test generation cuts maintenance cost, since a broken locator can trigger \"a little bit of agentic maintenance... a healing loop, commit that fix back in.\" I think that may be oversold, or at least dependent on your engineering practices. Maybe this has more of an ROI on pages undergoing rapid prototyping or constant redesigns, but outside of those scenarios, I find locators remain relatively stable once they're set up in a page object model, assuming you're using ID attributes (if they aren't randomly generated) or something like ",[283,3076,2206],{},". Playwright also has modern locator strategies that preclude a lot of the problems people used to get themselves into with XPath or text-based locators.",[11,3079,3080,3081,3084],{},"The live demos backed up the rest. One had the agent open the app, create a board, add a list, and invent three plausible user stories from a single plain-English prompt, no locators, no Playwright code written by hand. Another had it explore the app, propose a test plan, save that plan to a markdown file, and then generate full ",[283,3082,3083],{},"*.spec.ts"," files from it, self-healing failures as it ran, ending at 74 passed and 1 skipped. Knight's own retrospective on that second demo is worth keeping, because it's a caution about scope, not about cost: \"I would not recommend doing what I showed here, big asks. I would recommend many small asks.\" Review the output like a teammate's pull request, not like a vending machine.",[11,3086,3087,3088,3091],{},"I liked that Knight acknowledged the reality of the quality of test you get straight from AI with a prompt like this. The generated code was unoptimized and raw, similar to what the earlier codegen example created when we recorded our manual steps through the application to build a test case. You ",[222,3089,3090],{},"would not"," want to use these tests in your final test suite as-is:",[1749,3093,3094],{},[11,3095,3096],{},"\"There's no page objects here. There's no real library abstraction... these names aren't great.\"",[11,3098,3099],{},"Here's the clean version of the prompt he used, taken from his tutorial notes rather than transcribed live:",[288,3101,3105],{"className":3102,"code":3103,"language":3104,"meta":294,"style":294},"language-txt shiki shiki-themes material-theme-lighter github-light-high-contrast github-dark-high-contrast","Using playwright-cli, open http:\u002F\u002Flocalhost:3000\u002F, reset data if needed via API, then walk through\nthe \"create board → add list → add cards → go home\" flow. Use snapshots to pick stable locators.\nThen add a new Playwright TypeScript test under `tests\u002F` that matches our existing style:\n`test.beforeAll` or `beforeEach` for \u002Fapi\u002Freset, clear test name, getByRole\u002FgetByPlaceholder,\nand expect assertions. Reuse patterns from our existing trello spec if present.\n","txt",[283,3106,3107,3112,3117,3122,3127],{"__ignoreMap":294},[298,3108,3109],{"class":300,"line":301},[298,3110,3111],{},"Using playwright-cli, open http:\u002F\u002Flocalhost:3000\u002F, reset data if needed via API, then walk through\n",[298,3113,3114],{"class":300,"line":346},[298,3115,3116],{},"the \"create board → add list → add cards → go home\" flow. Use snapshots to pick stable locators.\n",[298,3118,3119],{"class":300,"line":378},[298,3120,3121],{},"Then add a new Playwright TypeScript test under `tests\u002F` that matches our existing style:\n",[298,3123,3124],{"class":300,"line":398},[298,3125,3126],{},"`test.beforeAll` or `beforeEach` for \u002Fapi\u002Freset, clear test name, getByRole\u002FgetByPlaceholder,\n",[298,3128,3129],{"class":300,"line":520},[298,3130,3131],{},"and expect assertions. Reuse patterns from our existing trello spec if present.\n",[11,3133,3134],{},"With more deliberate prompt engineering, Claude could have produced a cleaner first draft. But the rawer version is what actually demonstrated the accelerated-scaffolding benefit, and it set up a natural case for why prompt engineering matters in the first place:",[1749,3136,3137],{},[11,3138,3139],{},"\"If I were to do full context engineering, I would have my rules for Playwright tests, and I would say things like, use page object model.\"",[11,3141,3142],{},"Left on its own, a prompt like this gets you a fast, working first draft, not a finished one. The written version of this tutorial has a fair name for that tradeoff: \"accelerated scaffolding, not a substitute for judgment.\" Same deal as raw codegen output earlier in this piece, a working draft far faster than typing it by hand, just not something you'd commit as-is.",[15,3144,3146],{"id":3145},"why-locators-still-beat-computer-vision","Why Locators Still Beat Computer Vision",[11,3148,3149,3150,3153],{},"The day before Knight's session, Dionny Santiago's StarEast 2026 tutorial made close to the opposite argument about how AI should interact with a web page. I wrote about ",[1697,3151,3152],{"href":1699},"his case for AI vision testing over brittle CSS and XPath selectors"," in more detail, but the short version is direct: \"Computer vision is the evolution of the CSS selectors and the XPath selectors,\" reading a page the way a person does instead of hunting for a class name or test ID. Knight never mentioned Santiago's session, and might not have even been aware of it. An audience member raised a version of it anyway, describing tools that skip \"element work\" entirely in favor of a vision-based approach, and Knight disagreed without hesitating:",[1749,3155,3156],{},[11,3157,3158],{},"\"I disagree with that. Because even with AI superpowers, image matching is still going to be expensive. Whereas locators are very cheap and quick.\"",[11,3160,3161],{},"He built a full historical case for why, the kind of argument worth quoting at length because it's the most fully-reasoned claim in the entire session. The short version: programming has only ever moved toward higher abstraction (assembly to Fortran and C to Java, Python, and TypeScript), because each higher layer let us trust the layer below it without reading it. His extension of that idea to AI:",[1749,3163,3164],{},[11,3165,3166],{},"\"AI is the new compiler. Source code in TypeScript and Java and Python is the new assembly code... It will not be much longer that we still have to dance down at those levels because it's going to get so good. We still have to today because it's not as good yet.\"",[11,3168,3169],{},"Then the part that actually settles the locators-versus-vision question, mapping compiled-versus-interpreted execution onto test automation directly:",[1749,3171,3172],{},[11,3173,3174],{},"\"What I showed you before with, hey, let's just explore the app with Playwright CLI and just let it go and not record anything, that was equivalent to an interpreter. That's very slow. That's token heavy. Your image matching thing when it comes to test execution is also going to be inherently slow. Always, because if you're looking at something, you have to image match in the moment... that grinding can never not be done in that kind of model. So that's why I don't think the image matching of locators is ever really going to happen.\"",[11,3176,3177,3178,3181],{},"The distinction matters for accuracy: this is about test ",[222,3179,3180],{},"execution",", how the automation decides where to click while a test runs, not about visual regression tools that diff screenshots to catch rendering bugs. Knight never argues against that second category at all. Within the category he's actually addressing, his case is the more convincing one between these two tutorials. Generating a locator-based script costs tokens once. Running it costs almost nothing, over and over. Vision-based execution pays the image-matching cost every single run, forever, no matter how good the underlying model gets. That's a structural cost difference, not a current-capability gap that better models eventually close.",[15,3183,3185],{"id":3184},"test-pyramids-skyscrapers-and-the-gap-nobody-closed","Test Pyramids, Skyscrapers, and the Gap Nobody Closed",[11,3187,3188],{},"Back to the line I deferred earlier. Here's Knight's full skyscraper pivot, verbatim:",[1749,3190,3191],{},[11,3192,3193],{},"\"Today we don't build pyramids anymore. We build skyscrapers. Look up to testing skyscrapers. We need to reframe what we think of for testing in modern times because the world has changed since that previous mental model was created.\"",[11,3195,3196],{},"It's a good line, and it's worth being precise about what it actually claims. Knight never says UI tests are better than unit tests, the literal claim is narrower: \"UI tests are not bad. All tests are good because they mitigate different kinds of risks.\" That's an argument against rigid proportions, not a reordering of the hierarchy. He also never builds out the metaphor itself, there's no mapping of \"floors\" to test types anywhere in the session, the slides, or the written tutorial chapters. The skyscraper is a mood, not a blueprint.",[11,3198,3199],{},"His actual defense for ditching the pyramid's bias against UI tests is the tooling argument from earlier in this piece: Playwright's architecture fixed the execution speed and flakiness problems that gave UI tests their bad reputation. That's a real, demonstrated improvement. What it doesn't touch is the part of the pyramid's logic that was never about execution speed at all. A unit test calling a function in-process will always be faster than even the fastest browser context, that's a difference in kind, not in tooling. Unit tests also stay directly traceable to source lines and branches in a way browser-driven tests can't. Neither Playwright nor the AI tooling covered in this session does anything about that gap, and it never came up once in either half of the tutorial.",[11,3201,3202,3203,3206],{},"The AI-assisted authoring material from the previous section actually extends Knight's case further than he extended it himself, just not far enough to close that gap. If AI assistance genuinely lowers the cost of writing and maintaining E2E tests (and the token and time savings shown live back that up, even if the maintenance claim is softer), that addresses the ",[222,3204,3205],{},"other"," half of the pyramid's original justification, the cost of producing and keeping E2E tests working, which his own tooling argument never reached. So the fuller, more honest position: the case for de-emphasizing strict pyramid proportions is stronger than Knight made it sound, once you add AI-assisted authoring on top of Playwright's execution-speed fix. It's still not a full rebuttal of the pyramid, because the one gap that was never about tooling in the first place is still sitting there untouched.",[11,3208,3209],{},"Page objects, splitting one big test into independent behavior tests, and the parallel-safe test data strategy that actually fixes the \"drop the whole database\" shortcut from earlier in this piece are all covered in Knight's written tutorial chapters, just not in the room.",[15,3211,3213],{"id":3212},"my-takeaways-on-playwright-and-ai-testing","My Takeaways on Playwright and AI Testing",[11,3215,3216],{},"A few things I'm taking back with me:",[1470,3218,3219,3225,3231,3237],{},[594,3220,3221,3224],{},[49,3222,3223],{},"Default to the CLI over MCP for routine test-development work."," Reach for MCP only when local terminal and filesystem access isn't an option in the first place, not just because it feels more capable.",[594,3226,3227,3230],{},[49,3228,3229],{},"Treat AI-generated tests as scaffolding, not a finished product."," The first draft comes out raw, the same as old-school codegen output, so the cleanup step (page objects, naming, structure) isn't optional, it's the rest of the job.",[594,3232,3233,3236],{},[49,3234,3235],{},"Locator-based testing wins for driving test execution, and I don't expect that to change as models improve."," The cost gap is structural, not a capability gap that better models eventually close. (Visual regression testing is a different problem, and a fair use case for vision-based tools.)",[594,3238,3239,3242],{},[49,3240,3241],{},"Playwright and AI assistance narrow the case for the old Testing Pyramid, but they don't close it."," Knight's argument only ever answered the execution-speed half of the pyramid's old bias against UI tests; AI-assisted authoring answers some of the authoring-cost half too. Neither touches the one gap that was never about tooling: a unit test will always run faster and trace more directly to source than any browser-driven test.",[11,3244,3245,3246,3250],{},"If testing AI systems themselves (not just using AI to write tests) is more your focus right now, ",[1697,3247,3249],{"href":3248},"\u002Fsoftware-testing\u002Ftest-automation\u002Fhow-to-test-ai-chatbots-and-agents","how I approached evals on a real agentic chatbot engagement"," is a related read you may find useful.",[607,3252],{":items":3253},"[\"\u002Fsoftware-testing\u002Ftest-automation\u002Fstareast-2026-getting-started-ai-driven-automation\",\"\u002Fsoftware-testing\u002Ftest-automation\u002Fstareast-2026-getting-dirty-ai-testing\",\"\u002Fsoftware-testing\u002Ftest-automation\u002Fplaywright-accessibility-testing-axe-lighthouse-limitations\",\"\u002Fsoftware-testing\u002Ftest-automation\u002Fstareast-2026-prompt-engineering-techniques\"]",[611,3255,3256],{},"html pre.shiki code .sZTni,html code.shiki .sZTni{--shiki-light:#39ADB5;--shiki-light-font-style:italic;--shiki-default:#A0111F;--shiki-default-font-style:inherit;--shiki-dark:#FF9492;--shiki-dark-font-style:inherit}html pre.shiki code .sPJuK,html code.shiki .sPJuK{--shiki-light:#39ADB5;--shiki-default:#0E1116;--shiki-dark:#F0F3F6}html pre.shiki code .sZ-rw,html code.shiki .sZ-rw{--shiki-light:#90A4AE;--shiki-default:#0E1116;--shiki-dark:#F0F3F6}html pre.shiki code .sZi47,html code.shiki .sZi47{--shiki-light:#39ADB5;--shiki-default:#032563;--shiki-dark:#ADDCFF}html pre.shiki code .srGNg,html code.shiki .srGNg{--shiki-light:#91B859;--shiki-default:#032563;--shiki-dark:#ADDCFF}html pre.shiki code .sb1SK,html code.shiki .sb1SK{--shiki-light:#6182B8;--shiki-default:#622CBC;--shiki-dark:#DBB7FF}html pre.shiki code .stWsX,html code.shiki .stWsX{--shiki-light:#9C3EDA;--shiki-default:#A0111F;--shiki-dark:#FF9492}html pre.shiki code .s2xgV,html code.shiki .s2xgV{--shiki-light:#90A4AE;--shiki-light-font-style:italic;--shiki-default:#702C00;--shiki-default-font-style:inherit;--shiki-dark:#FFB757;--shiki-dark-font-style:inherit}html pre.shiki code .s_gjE,html code.shiki .s_gjE{--shiki-light:#90A4AE;--shiki-light-font-style:italic;--shiki-default:#66707B;--shiki-default-font-style:inherit;--shiki-dark:#BDC4CC;--shiki-dark-font-style:inherit}html pre.shiki code .sq0XF,html code.shiki .sq0XF{--shiki-light:#E53935;--shiki-default:#0E1116;--shiki-dark:#F0F3F6}html .light .shiki span{color:var(--shiki-light);background:var(--shiki-light-bg);font-style:var(--shiki-light-font-style);font-weight:var(--shiki-light-font-weight);text-decoration:var(--shiki-light-text-decoration)}html.light .shiki span{color:var(--shiki-light);background:var(--shiki-light-bg);font-style:var(--shiki-light-font-style);font-weight:var(--shiki-light-font-weight);text-decoration:var(--shiki-light-text-decoration)}html .default .shiki span{color:var(--shiki-default);background:var(--shiki-default-bg);font-style:var(--shiki-default-font-style);font-weight:var(--shiki-default-font-weight);text-decoration:var(--shiki-default-text-decoration)}html .shiki span{color:var(--shiki-default);background:var(--shiki-default-bg);font-style:var(--shiki-default-font-style);font-weight:var(--shiki-default-font-weight);text-decoration:var(--shiki-default-text-decoration)}html .dark .shiki span{color:var(--shiki-dark);background:var(--shiki-dark-bg);font-style:var(--shiki-dark-font-style);font-weight:var(--shiki-dark-font-weight);text-decoration:var(--shiki-dark-text-decoration)}html.dark .shiki span{color:var(--shiki-dark);background:var(--shiki-dark-bg);font-style:var(--shiki-dark-font-style);font-weight:var(--shiki-dark-font-weight);text-decoration:var(--shiki-dark-text-decoration)}",{"title":294,"searchDepth":346,"depth":346,"links":3258},[3259,3260,3261,3262,3263,3264],{"id":2153,"depth":346,"text":2154},{"id":2186,"depth":346,"text":2187},{"id":2963,"depth":346,"text":2964},{"id":3145,"depth":346,"text":3146},{"id":3184,"depth":346,"text":3185},{"id":3212,"depth":346,"text":3213},"\u002Fimages\u002Fposts\u002Fstareast-2026-playwright-ai-cost-efficient-testing\u002Fstareast-2026-playwright-ai-cost-efficient-testing-cover.webp","2026-06-22","Playwright's MCP server can burn your AI budget fast. StarEast 2026 lessons on efficient AI testing, and why locators still beat computer vision.",{},{"title":2117,"description":3267},"software-testing\u002Ftest-automation\u002Fstareast-2026-playwright-ai-cost-efficient-testing","joMPpMIfdSO7ZR8LztFrojUyxQeGStMo6mi6RIG0EI4",1788657218531]