Test Selection: Run Fewer Tests, Catch the Same Bugs

An 8+ hour CI feedback loop is the kind of pain that makes any promise of faster tests sound appealing, and at a past job the company adopted a black-box predictive test-selection tool that trained on historical pass and fail data to guess which tests a given change was likely to break. While seemingly effective initially, it surfaced a real problem almost immediately, just not the one we were hoping to fix: our UI suite had hidden dependencies, tests relying on shared, warmed-up page state or on data another test had mutated earlier in the run. Predictive test selection, by not always running things in the suite's original order, introduced flakiness that proved difficult to troubleshoot due to different tests running each build. We were never able to fully resolve it, which never allowed us to realize the benefits of predictive test selection.

An AI Test Selection Framework That Shows Its Work

That experience made me skeptical of any tool promising to safely trim a test suite, especially one that couldn't explain itself. So when I saw a StarEast 2026 session titled "Dear AI, Which Tests Should We Run Now?" promising to go deeper into the actual mechanics behind test selection, I made a point of attending. I wanted to know whether the field had advanced since my own bad experience, and whether I could actually learn the theory behind these approaches instead of trusting a vendor's black box to get it right.

The session was presented by Dr. Elmar Jürgens, co-founder of CQSE GmbH, makers of the TeamScale software-quality-analysis platform, who holds a PhD in software quality analysis himself. His talk covers four distinct approaches to picking a smaller, faster subset of a large test suite that still catches nearly as many bugs as running everything, each with its tradeoffs measured and stated plainly rather than buried inside a product.

What made this talk earn my trust, in a way my prior experience with an opaque tool never did, is that he shows his work. Every approach comes with a measured tradeoff, a named limitation, and an honest account of when it doesn't apply, instead of a black box you're asked to take on faith. He's also explicit that he doesn't care whether a given approach technically counts as AI: "I care about whether the approaches find more bugs more quickly." That's the right question and I appreciated him being up front about it since speakers at these events tend to just rename their presentation from a past year with this year's trending keywords.

Four Test Selection Approaches

Jürgens builds up to four approaches by crossing two questions. The first is which situation you're in.

Question 1: Do You Know What Changed?

Quality Gate means you have a large suite and a small time budget, but you don't know what changed in the software since you last ran tests. One of Jürgens' customers, a German pension fund, ran a 20-hour test suite only on weekends because there wasn't time to run it more often. By the time a build reached that gate, days of accumulated commits sat behind it, no single diff to point to, just a build that needed a yes-or-no answer in a fraction of the time the full suite would take.

Continuous Integration, by contrast, means you already know exactly what changed, because you have a diff or a pull request in hand. Dolby, another of Jürgens' customers, dealt with a combinatorial explosion of audio bitrate and sample-rate combinations to test and wanted CI feedback on every pull request within 10 minutes, even though their full suite took 3 to 12 hours to run. Each PR came with a specific, known diff to work from, a very different starting point than the pension fund's accumulated pile of weekend changes.

Question 2: Coverage-Based or Content-Based Test Selection?

The second question is how much you're willing to invest in setup to get a better number. Coverage-based approaches require measuring test-case-specific code coverage first, real upfront effort, but they pay it back with a better result. Content-based approaches skip that setup entirely and work directly from the test and change content, in exchange for a slightly worse number. Jürgens presents this as a real choice inside each situation, not a fixed rule: for Quality Gate, coverage-based Pareto Optimization versus content-based AI Test Clustering; for Continuous Integration, coverage-based Test Impact Analysis versus content-based Similarity Scoring. Two situations, two ways to solve each one.

Cross those two questions and you get the four approaches, each measured the same way: what percentage of full suite runtime it takes to find 90% of the bugs the full suite would find. The quick reference, then what each one actually does:

Test Impact Analysis (TIA) is the most speed optimized, but Similarity Scoring gets you close in about a day instead of half a year.

Predictive Test Selection Approaches Explained

Pareto Optimization measures per-test code coverage for the whole suite, then greedily orders tests by how much previously-uncovered code each one covers per second of execution time, cutting off once you hit your time budget. The clearest illustration of why this beats running tests in whatever order they were written came from PixelitOr, an open-source paint program Jürgens uses as a fully visualizable example. Running four UI tests in sequence, Gaussian Blur, Motion Blur, Lens Blur, Smart Blur, each new test lights up progressively less new code and re-covers more of what the earlier blur tests already exercised. His team checked whether that redundant coverage was actually worthless by tracking real and injected bugs: if a bug lives in that repeated code, typically either all of those blur tests find it or none of them do. Running all four buys you almost nothing over running one, and a greedy, coverage-per-second reordering naturally spreads the budget across dissimilar tests instead of exhausting it inside one redundant cluster.

AI Test Clustering gets you a similar diversity-driven selection without needing any coverage data at all. Every test gets represented as a point in a high-dimensional vector space, generated from a large language model's embedding of the test's actual content, its code if it's automated, or even a plain-English Given/When/Then description if it's a manual test case. Tests that exercise similar functionality land close together in that space; tests that do genuinely different things land far apart. The selection algorithm then greedily picks whichever remaining test is furthest from everything already chosen, which naturally avoids getting stuck resampling one redundant cluster the way running tests in file order would. Jürgens showed this working on a 3D projection of around 2,000 real customer test cases: the tests visibly clustered into dense groups, and the selected subset landed inside essentially every visible cluster rather than missing whole regions.

Test Impact Analysis exploits something the other two approaches don't have access to: you already know exactly what changed, because you have a diff or a PR in hand. Combine that with coverage data recorded from prior test runs, and you can directly compute which historically-recorded tests actually execute the changed lines, then skip everything that provably can't be affected by this specific change. On a real change to Jürgens' own team's codebase, out of 5,000 automated tests, only 4 actually executed the changed code. The other 4,996 categorically could not have caught a bug introduced by that change, because they never touch it.

Similarity Scoring gets a comparable result to Test Impact Analysis without needing coverage data, by treating test selection like a search engine query. Every test's content gets indexed into a document database, the way a search engine indexes web pages. A code change then becomes the search query, built from the changed identifiers plus their surrounding context, not just the raw diff lines. Retrieval works conceptually like TF-IDF (term frequency, inverse document frequency), the same scoring method search engines use: a test scores higher the more it mentions terms that appear in the change, weighted down for terms so common across the whole suite that they don't tell you much. In Jürgens' worked example, a change to a bank transfer method matched correctly against a Cucumber test, a Robot Framework test, and even a plain manual test description, three completely different formats, all identified as relevant through shared identifiers alone.

Counterintuitively, both AI Test Clustering and Similarity Scoring need more suite runtime than their coverage-based counterparts to find 90% of the defects a full run would find. He recommends starting with them anyway, since coverage-based approaches can take months to a year to set up on a large industrial system, while the content-based ones can be running in a day or two.

How Test Gap Analysis Complements Predictive Test Selection

All four predictive test selection approaches assume somewhere in your suite a test already exists that's capable of catching a bug in the code being changed. You can't optimize tests that don't exist yet. Jürgens described a related technique, Test Gap Analysis. It works by overlaying two maps: which code changed recently, and which code has ever been executed by any test, unit, integration, manual, anything at all. Wherever those two maps don't overlap, you have changed code with missing test coverage.

He mentioned at one large software company, running this the night before a scheduled release revealed entire multi-year components that had never been touched by a single test. The release was postponed three weeks.

Initially I thought, "Isn't this what our SonarQube pull request Quality Gates do for us when we set coverage floors?" Jürgens' version has a broader scope and combines the coverage data from JaCoCo, Istanbul, Jest, etc. with manual and exploratory coverage data as well to get a full picture. Beyond that though, this lets one see that there are changes in code where your testing has blind spots. This is where you would want to focus on adding coverage because these are areas that may or may not have been working before, but now they've been modified. Without tests you don't know if this introduced a regression in this area or not.

Since uncovered areas may be large, Jürgens recommends a risk-based prioritization pass, tackling the gaps sitting in your highest-domain-risk code first. I think this is a great technique, but getting that full picture is a prerequisite and test-case-specific coverage recorded and persisted across every kind of testing you do is realistically its own months-to-a-year project.

The One Prerequisite: Hidden Test Dependencies

I attended the session because this is something we actually tried with a commercial product, Launchable, years ago at a previous employer with inconclusive results. I wanted to see what Jürgens' experience was and what has changed in the space since, especially with advances in AI.

The problem we were trying to solve by trialing Launchable was our extremely long feedback loop. It took 8-10 hours to see our test results after the code was committed. As a result, if a developer introduced a breaking change during their workday they wouldn't know before they left for the day. Launchable predictively selected tests based on our code coverage, drastically reducing the number of tests required to run, but caused the test suite to become unreliable. Tests were supposed to be written without dependencies on other tests, but over the years they naturally formed dependencies due to innocent mistakes like test engineers assuming certain data was pre-seeded in our baseline where in reality the data was inserted by upstream tests.

With Launchable running different tests each build it would expose these issues. We'd end up fixing the issue, but since the next run may not execute that test again it was hard to confirm the fix organically. Worse, the next run might surface other tests with the same issue. This led to expensive delays trying to get a green build.

During the Q&A, I described my experience. Jürgens' answer generalized my specific pain point and gave it a name, the test-dependency-chain problem, and it applies to all four approaches to predictive test selection.

If your tests form a hidden sequential chain, test A sets up state that test B needs, which a batch job then depends on as a prerequisite for test C, none of these approaches can safely select "test C" in isolation. Jürgens uses the German term Testkappen for this pattern. His rule is direct: the entire chain has to be treated and selected as a single atomic unit. If your whole system is effectively one long chain end to end, none of these approaches are useful at all, no matter how good the underlying algorithm is.

So the takeaway is you need to verify how interwoven your tests are, and how well they'd handle being reordered or omitted altogether from build to build, before adopting any of these approaches, whether through a commercial tool or your own implementation of what Jürgens described.

How I'd Implement Similarity Scoring

I started to research which of these techniques would be the most feasible at my current employer and narrowed it down to content-based approaches: Similarity Scoring or AI Test Clustering. While we have extensive code coverage from our unit tests for some parts of the codebase, large sections are XML-driven and tested through special tooling where we don't have the coverage data we'd need to get started quickly with coverage-based approaches. Mapping the UI tests back through Test Impact Analysis, as mentioned earlier, would take a longer time investment relative to the content-based approaches.

I thought using Claude would quickly get us to AI Test Clustering, but initial research showed Claude doesn't provide the vector embeddings this approach depends on directly, Anthropic points people to third-party providers like Voyage AI for that instead.

Similarity Scoring, by contrast, needs no embedding infrastructure to get started. Its crudest version, literal keyword overlap between a test's content and the identifiers in a change, is plain text matching I could build with a basic search index. It's also easier to inspect and explain than a black box that would trade one set of problems for another, which I consider valuable for a toe-dip investigation. Weighing my options, Similarity Scoring is what I'd start with.

It comes down to two decisions.

First, I'd need to decide what counts as a "test document," the thing Similarity Scoring compares a change against. It's not just the test's own code, it's every identifier that test touches: selectors, page-object property names, the API endpoints or service methods it exercises, even indirectly through a fixture or helper. Say a Playwright spec calls a page-object method that, several layers down, calls a service method named ProcessRefund:

test('customer can request a refund', async ({ page }) => {
  const checkoutPage = new CheckoutPage(page);
  await checkoutPage.submitRefundRequest();
});

export class CheckoutPage {
  async submitRefundRequest() {
    await this.page.click('#refund-button');
    await this.api.post('/refunds', { handler: 'ProcessRefund' });
  }
}

That spec's test document would include ProcessRefund, even though the name never appears in the spec file itself, because the spec's actual behavior depends on it:

test_document["refund.spec.ts"] = [
  "customer can request a refund",
  "CheckoutPage",
  "submitRefundRequest",
  "#refund-button",
  "/refunds",
  "ProcessRefund"
]

Second, I'd need to decide what counts as a "query," the thing built from an incoming change to search those test documents against. That's one combined query per pull request, built from the diff plus the surrounding context of whatever the changed code calls or is called by, not just the raw changed lines. If a PR renames ProcessRefund to ProcessRefundRequest, a query built from only the two literal diff lines would miss every test that reaches the old name through a helper several layers away. A query built from the diff's surrounding context catches those too.

With both of those defined, the mechanism connecting them is just comparison, not anything more exotic: score every test document against the query, rank the results, and take the highest-scoring tests, or everything above some threshold. The only real decision left is what that scoring function actually is, and that's also where I'd start simple.

As a first pass, before trying TF-IDF (term frequency, inverse document frequency) weighting, I'd try literal keyword overlap to match on, in other words, does the test document contain the changed identifier strings. This would be a quick-to-implement form of Similarity Scoring:

score(test) = count of identifiers shared between test.document and query.identifiers

selected_tests = tests
    .where(score(test) > 0)
    .order_by(score, descending)

That ranked list still has to turn into something a CI job can actually run. Each test document maps back to a real Playwright spec file, so selected_tests becomes a list of file paths handed straight to npx playwright test, the same command any CI job already invokes, just pointed at a smaller, targeted list instead of the whole suite.

Before trusting any of this in a real CI gate, I'd validate it the way Jürgens' own team did: pull a handful of recent pull requests that caused a regression someone caught later, and check whether my selected subset would have included the test that actually caught it. I'd also heavily leverage mutation testing to see how well this approach holds up before trusting it, and follow Jürgens' example of always running test suites that cover critical business functionality.

Takeaway: What Test Selection Actually Requires

After attending, I learned that my own experience, predictive test selection working in theory but introducing too much flakiness into the pipeline, wasn't unique to me.

Predictive test selection is not just something that is an academic exercise, but requires:

1. A healthy test suite without hidden interdependencies between tests
2. Coverage instrumentation, if you're going the coverage-based route, content-based approaches skip this entirely
3. Time to invest in building out the ecosystem (or funding to buy it)

Jürgens' session was valuable because it laid out the real pros and cons of each approach, letting you match one to your own tech stack and appetite for investment. Versus a black box, he lays out their weaknesses and trade-offs up front, rather than leaving you to discover where they fall apart mid-evaluation.