---
title: "Pattern: Progressive Determinism for AI Tasks"
type: post
url: "https://robotpaper.ai/pattern-progressive-determinism-for-ai-tasks/"
canonical: "https://robotpaper.ai/pattern-progressive-determinism-for-ai-tasks/"
markdown_url: "https://robotpaper.ai/pattern-progressive-determinism-for-ai-tasks.md"
published: "2026-10-28T09:16:20.000+00:00"
updated: "2026-10-28T09:20:59.000+00:00"
tags: ["Pattern"]
author: "Roy Osherove"
author_links: ["https://robotpaper.ai", "https://x.com/royosherove", "https://www.linkedin.com/in/osherove/", "https://forms.gle/jQSx5KoXvrVA1JCK7", "https://robotpaper.ai/about-roy.md"]
description: "Turning AI to be incrementally more deterministic for specific tasks over time. Web Application Testing: From fully dynamic to full deterministic + failover * An agent can use playwright or playwright cli to dynamically find out what an application structure \"looks\" like and discover page structure, and do initial \"manual testing on it. * The agent can then generate deterministic playwright script to repeat the same test next time in a fully repeatable way. * On next run of the specific sc"
usage: "Licensed under CC BY 4.0 — reuse, quote and adapt with attribution (Roy Osherove, link to the original post)."
license: "https://creativecommons.org/licenses/by/4.0/"
---

# Pattern: Progressive Determinism for AI Tasks

Turning AI to be incrementally more deterministic for specific tasks over time.

## Web Application Testing: From fully dynamic to full deterministic + failover

-   An agent can use [playwright](https://playwright.dev/) or [playwright cli](https://playwright.dev/agent-cli/introduction) to dynamically find out what an application structure "looks" like and discover page structure, and do initial "manual testing on it.
-   The agent can then generate deterministic playwright script to repeat the same test next time in a fully repeatable way.
-   On next run of the specific scenario agent "finds out" that deterministic script already exists and prefers running it.
-   If deterministic script fails, agent falls back to "manual discovery mode", agent then discovers and auto-fixes deterministic script to be run next time
-   Repeat for future scenarios.

## Example

Imagine a to-do application. We ask an agent:

> Add “Buy milk” and verify that it appears in the task list.

For this example, the application starts each test with an empty list.

**First run: discover and save.**

The agent explores the page, finds the “New task” field and the “Add” button, and successfully completes the scenario. It then saves this Playwright test as `tests/add-task.spec.ts`, using role locators and an assertion for the expected result. ([playwright.dev](https://playwright.dev/docs/writing-tests))

```
import { test, expect } from '@playwright/test';

test('add a task', async ({ page }) => {
  await page.goto('http://localhost:3000');

  await page.getByRole('textbox', { name: 'New task' })
    .fill('Buy milk');

  await page.getByRole('button', { name: 'Add', exact: true })
    .click();

  await expect(page.getByRole('listitem'))
    .toHaveText('Buy milk');
});
```

The agent runs the saved test to verify that it reproduces the successful interaction.  
This new test can now be added to the continuous testing and verification pipeline.

**Assuming in the future we ask the agent to test the same flow again using a prompt: reuse the saved test.**

We ALWAYS give the agent a standing instruction:

> Before exploring the application, check whether a test already exists for this scenario. If it does, run it first. Enter discovery mode only when the test is missing or fails.

On the next request, it finds the file and runs this command: ([playwright.dev](https://playwright.dev/docs/running-tests))

```
npx playwright test tests/add-task.spec.ts
```

In this workflow, the browser interactions and expected result are already defined in code.

**After a page change: investigate and repair.**

Suppose the application renames “Add” to “Create task”. The saved test fails at the button lookup during CI.

The agent listens to failed CI/CD runs, and tries and detect what is the root cause of the test failure. Assuming it finds that the cause is that the test no longer fits the website, it returns to discovery mode, identifies the renamed button, and proposes this small repair, and applies it:

```
- await page.getByRole('button', { name: 'Add', exact: true })
+ await page.getByRole('button', { name: 'Create task', exact: true })
    .click();
```

It reruns the test from an empty list and retains the repair only after the original assertion passes. Future runs reuse the updated script.

The auto repair rule contains: "**preserve the expected behavior".**

## What if the feature isn't available anymore?

If the application no longer adds “Buy milk” to the list, it could mean one of of two things:

-   False positive: The feature is no longer needed or relevant, and so the test is not relevant. the agent should report the failure and change in functionality and given human input or llm as a judge with broader context, might have the authority to prune and remove tests that are no longer relevant.
-   True positive. The feature is missing and something is broken. In that case the test just fails, **as expected. The judge llm might just say that's a "true positive".**

## Conclusion

  
So, the test started as an almost "manual" test, fully dynamic, that quickly turned into a deterministic test. The test saves time and tokens, but we now have a powerful fallback: e2e test maintainability is a nightmare (used to be!) but we can now continuously update/fix/prune tests to fit UI changes of teh application.

### P.S

  
I discovered during the writing of this post that [Playwright actually has built in support for planner, generator and healer agents](https://playwright.dev/docs/test-agents) that help with this pattern. I've not tried them myself yet, so take it with a grain of salt - but worth testing this out (let me know [if you have feedback on those)](https://forms.gle/jQSx5KoXvrVA1JCK7)

## Read next

- [The New Bottlenecks After Coding](https://robotpaper.ai/the-new-bottlenecks-after-coding.md)
- [Pattern: Agent-Owned Accounts (Deploy Environments)](https://robotpaper.ai/agent-owned-accounts-deploy-environments.md)
- [Github Repo Forks Are a New Token-Saving Mechanism](https://robotpaper.ai/github-repo-forks-are-a-new-token-saving-mechanism.md)

---

## About the author

**Roy Osherove** - Senior Solutions Architect at AWS working on AI for production and AI-driven development; author of The Art of Unit Testing and Elastic Leadership, with about 20 years of consulting and training on unit testing, TDD, team leadership and continuous delivery.

### Sites

-   [RobotPaper](https://robotpaper.ai/): This blog: AI patterns and anti-patterns, AI and coding agents, and engineering leadership.
-   [osherove.com](https://osherove.com): Roy's personal site. Author of The Art of Unit Testing and Elastic Leadership; consulting and training on unit testing, TDD, team leadership and continuous delivery, plus about 20 years of blog posts.
-   [The Art of Unit Testing](https://www.artofunittesting.com): The Art of Unit Testing, 3rd Edition (Manning): readable, maintainable, trustworthy unit tests, plus TDD workshops in C#, JavaScript, Java and Kotlin.
-   [5whys.com](https://www.5whys.com): Roy's writing on software team leadership: self-organizing teams, surviving chaos, learning mode and the Elastic Leadership model.
-   [Pipeline Driven](https://pipelinedriven.org): Continuous delivery and DevOps practices: build patterns, pipeline-friendly testing, DevOps metrics and enterprise DevOps.
-   [Elastic Leadership](https://www.elasticleadership.com): Elastic Leadership, Roy's framework for software team leaders (survival, learning and self-organizing modes).

### Social

-   X: [https://x.com/royosherove](https://x.com/royosherove)
-   LinkedIn: [https://www.linkedin.com/in/osherove/](https://www.linkedin.com/in/osherove/)

### Contact

-   Contact form: [https://forms.gle/jQSx5KoXvrVA1JCK7](https://forms.gle/jQSx5KoXvrVA1JCK7)

### License

Content on robotpaper.ai is licensed under [CC BY 4.0](https://creativecommons.org/licenses/by/4.0/): reuse, quote and adapt with attribution (Roy Osherove, link to the original post).

More about Roy and his other sites and projects: [https://robotpaper.ai/about-roy.md](https://robotpaper.ai/about-roy.md)