DEV Community: Playwright end to end Testing

Playwright Agents: Planner, Generator, and Healer in Action

Debbie O'Brien — Mon, 06 Oct 2025 17:49:03 +0000

Playwright has introduced a powerful new feature: Playwright Agents, new in Version 1.56. These agents can generate test plans, create tests based on those plans, and even debug and fix failing tests automatically. This post demonstrates how Planner, Generator, and Healer agents work together using a movies application as an example.

🎭 Planner generates comprehensive test plans.
🎭 Generator produces runnable test files from the plans.
🎭 Healer debugs and fixes failing tests automatically.

Getting Started with Playwright Agents

To begin make sure you have the latest version of Playwright installed. Then, run the init-agents command with your preferred loop, vscode, claude, or opencode:

npx playwright init-agents --loop=vscode

This command generates the agents along with a seed file.

Seeding your tests

The seed file is used to seed tests and will be copied into generated tests by the agent. It can be left blank, but it is typically used to include fixtures or setup logic that must run before tests.

For example, in a movies application, a fixture can be defined to create a page containing a list of movies. This page becomes the starting point for the tests and is copied into each generated test file.

/* eslint-disable @typescript-eslint/no-unused-vars */
import { listTest as test } from '../helpers/list-test';
import { expect } from '@playwright/test';

test.describe('seed for logged in user', () => {
  test('seed using listPage fixture', async ({ listPage }) => {
    const page = listPage; // set the page to the listPage fixture
  });
});

If you don't have any setup logic you can leave the seed file empty or add a page.goto() to a starting URL.

Planner Agent – Creating a Test Plan

The first step is to use the Planner agent to generate a test plan for a specific feature, for example managing the movies list. The plan should be saved in a specs folder as it is possible you will have multiple plans.

In VS Code open chat mode and select the Playwright Planner Agent. If using the seed file, it should be added to the context before generating the plan.

Example Prompt

Generate a test plan for managing movies list and save as movies-list-plan.md in specs folder

The Playwright Planner Agent explores the site, analyzes the “managing lists” feature, and produces a structured test plan in markdown format. The generated plan can be reviewed and refined as needed.

Generator Agent – Creating the Tests

The Playwright Generator agent is used to generate test files from the test plan. As the test plan may include numerous testing scenarios you may want to choose to generate tests for a specific section of the plan.

In VS Code open the chat and select the Playwright Generator Agent.

Example Prompt

Generate tests for the "Adding a Movie" section of the movies-list-plan.md

The Playwright Generator Agent navigates through the site and executes each of the scenarios from the chosen section of the test plan.

The result is a set of generated test files, a file for each scenario. If you had any setup logic in the seed file, you will see it will have been copied into each test file.

Next step is to run the generated tests.

Healer Agent – Fixing a Failing Test

Sometimes the generated tests will all pass, but sometimes there might be some test failures. Instead of debugging manually you can leverage the Playwright Healer agent. Start a new chat and select the Playwright Healer agent, and ask it to help fix the failing test.

Example Prompt

Run and fix failing tes

The Playwright Healer Agent will run the tests in debug mode, check console logs, network requests and the page snapshots to identify the root cause of the failure. It will keep trying to fix the test until it passes or if the agent believes the functionality is broken it marks the test as skipped.

Conclusion

Playwright Agents enable AI-powered test generation, execution, and healing with minimal manual effort.

To get started, update to the latest version of Playwright (v1.56)

npm install -D @playwright/test@latest

Then initialise your agents and choose your preferred agent loop:

npx playwright agents --loop=vscode/claude/opencode

With Planner, Generator, and Healer working together, maintaining end-to-end tests becomes faster, smarter, and more reliable.

Check out the video to see it in action:

Happy testing with AI and Playwright Agents!

What’s New in Playwright v1.52 & v1.53: Fix with AI, Describable Locators, and More!

Debbie O'Brien — Tue, 17 Jun 2025 09:38:32 +0000

Here’s a quick tour of what’s new in Playwright v1.52 and v1.53! We’ve packed in some powerful features to make your testing experience smoother, especially around debugging and reporting:

🧠 Fix with AI in VS Code

Playwright now integrates seamlessly with GitHub Copilot to help you fix test failures directly from VS Code. When a test fails, just click the ✨ icon next to the error message or while hovering over the test name in the test explorer. Playwright gives Copilot enough context, so it can generate a targeted fix. You review, accept, and re-run. It's fast, contextual, and incredibly useful.

💡 Want to try it? Make sure you're using the Playwright VS Code extension.

🔎 `locator.describe()` – Better Trace and Report Visibility

You can now add a human-readable description to any locator using .describe():

const newTodo = page.getByPlaceholder('What needs to be done?').describe('new todo input');

await newTodo.fill('Buy milk');

These descriptions show up in:

The trace viewer
UI mode
HTML reports

This small change makes a big difference when debugging complex UIs or collaborating with teammates.

📊 Custom Titles in HTML Reports

Add more clarity to your test reports with a configurable title:

import { defineConfig } from '@playwright/test';

export default defineConfig({
  reporter: [['html', { title: 'Custom test run #1028' }]]
});

Great for team dashboards, CI output, or distinguishing between multiple test runs.

✅ New Assertion: `toContainClass`

Need to assert that an element contains a specific class? Use:

await expect(page.getByRole('listitem', { name: 'Ship v1.52' })).toContainClass('done');

It's ergonomic, precise, and especially useful for class based UI state.

🧪 Snapshot Enhancements: `children` and `url` Support

Aria snapshots (via toMatchAriaSnapshot) now support:

/children: equal – ensures child elements are included
/url: "https://playwright.dev" – match specific URLs in snapshots

await expect(locator).toMatchAriaSnapshot(`
  - list
    - /children: equal
    - listitem: Feature A
    - listitem:
      - link "Feature B":
        - /url: "https://playwright.dev"
`);

This makes UI snapshot testing more robust, especially for complex or dynamic components.

⚙️ Test Runner Updates

A few more helpful additions:

testProject.workers – Fine-tune concurrency per test project
failOnFlaky – Automatically fail test runs if flakiness is detected

🆙 How to update?

Install the latest Playwright version:

npm i -D @playwright/test@latest

Make sure your VS Code extension is also up-to-date to use the new AI features.

📺 Watch the Demo

Check out our latest YouTube video walking through all of these features in action:

For a detailed list check out the release notes

Happy testing!

Debugging Playwright Tests with AI: A Smarter, Faster Workflow

Debbie O'Brien — Tue, 10 Jun 2025 15:02:11 +0000

Debugging failing tests can be time-consuming and frustrating—especially when error messages are long or snapshot diffs are difficult to interpret. With the latest AI-powered tools in Playwright, resolving test failures has become significantly faster and more intuitive.

Consider a basic Playwright test that uses snapshot testing to verify that the content inside getByRole('main') matches a previously stored snapshot. The snapshot might contain two distinct headings, for example.

If the heading level in the application is changed—from <h1> to <h2>—the test fails, as expected, due to a mismatch with the stored snapshot. Traditionally, resolving this would involve reading the stack trace, analyzing the diff, and manually updating the test or snapshot.

One-Click AI Fixes

Playwright’s AI integration streamlines this process. When a test fails, developers can click an AI Fix button in the error message that automatically analyzes the context—such as the snapshot diff, the test code, and the full error message—and suggests a fix.

This suggestion can be reviewed and accepted directly in the development environment. Once applied, the test can be re-run, and in many cases, it passes without the need for manual debugging.

The AI Fix button is also available in the Test Explorer inside Visual Studio Code, offering a seamless way to resolve issues as they’re encountered during development.

Debugging in UI Mode

Playwright’s UI mode also supports AI-assisted debugging. Running npx playwright test --ui launches an interactive interface where tests can be run visually.

When a test fails, a Copy Prompt button appears alongside the error.

This prompt includes all the relevant context: the full page snapshot, the error message, and the corresponding test code. Developers can paste this prompt into Copilot or any preferred AI tool to receive targeted fix suggestions.

Once the AI suggests a fix, developers can click Apply in Editor, inspect the proposed changes, and confirm them within their editor. After re-running the test, it's often back to passing status.

Support Across Reports and CI

AI prompt generation isn’t limited to development tools. The Copy Prompt feature is also available in Playwright’s HTML reports. After a test run, developers can open the report, copy the AI prompt, and use it with their preferred AI assistant to debug the issue.

Even when debugging tests in CI environments, the Trace Viewer offers access to the same AI prompt feature. This ensures that developers can resolve issues intelligently and efficiently, even when working with remote test results.

Faster, Smarter Debugging with AI

AI-assisted debugging in Playwright enhances the developer experience by reducing the manual effort required to resolve test failures. Whether working locally, in UI mode, or reviewing CI failures, these AI-powered features help teams ship faster with greater confidence.

Have fun debugging your Playwright tests with AI! 🚀

Useful links

Catch Missing `await` Calls in Playwright Tests with ESLint

Debbie O'Brien — Wed, 16 Apr 2025 11:55:06 +0000

Have you ever run a Playwright test that just... didn’t behave as expected? Maybe it failed randomly. Maybe it passed locally but broke in CI. The culprit? Often—it’s just a missing await.

It’s one of the easiest mistakes to make in async code, and in testing, it can lead to race conditions and flaky results.

The Problem: A Missing `await` Can Break Your Test

Let’s look at a simple example. We click a button and expect a modal to appear:

await page.getByRole('button', { name: 'Submit' }).click();
expect(page.getByRole('dialog')).toBeVisible(); // ❌ Missing 'await'

The expect() call returns a promise, but we didn’t await it, so Playwright moves on before it finishes. Sometimes the modal appears in time. Sometimes it doesn't. Result? Flaky test.

What’s worse? VS Code won’t warn you about this by default.

The Fix: Let ESLint Catch Floating Promises

Luckily, we can set up a TypeScript ESLint rule to help us catch these mistakes automatically. The rule is called @typescript-eslint/no-floating-promises, and it highlights any promise that isn't properly handled.

Here’s how to set it up:

1. Install ESLint and TypeScript Support

If you haven’t already (ensure you have ESLint v9+ and TypeScript):

npm install --save-dev eslint typescript-eslint typescript

2. Create `tsconfig.json` (If Needed)

The @typescript-eslint/no-floating-promises rule requires type information. Ensure you have a tsconfig.json file in your project root. If not, create a basic one:

// tsconfig.json
{
  "compilerOptions": {
    "target": "ES2022",
    "module": "NodeNext",
    "moduleResolution": "NodeNext",
    "strict": true,
    "esModuleInterop": true,
    "skipLibCheck": true,
    "forceConsistentCasingInFileNames": true,
    "noEmit": true // Important for linting without compiling
    // Add other options relevant to your project (e.g., "jsx": "react-jsx")
  },
  "include": ["**/*.ts", "**/*.tsx", "**/*.js", "**/*.mjs"],
  "exclude": ["node_modules"]
}

3. Configure ESLint in `eslint.config.mjs`

Create or update eslint.config.mjs in your project root to enable type-aware linting:

// eslint.config.mjs
import eslint from '@eslint/js';
import tseslint from 'typescript-eslint';

export default tseslint.config(
  eslint.configs.recommended,
  // Use recommendedTypeChecked for rules requiring type information
  ...tseslint.configs.recommendedTypeChecked,
  {
    // Configure language options for type checking
    languageOptions: {
      parserOptions: {
        project: true, // Use the tsconfig.json in the root
      },
    },
    rules: {
      '@typescript-eslint/no-floating-promises': 'error',
      // Other rules can be added here
    },
  },
  {
    // Optionally, ignore specific files/folders
    ignores: ['dist/**', 'node_modules/**', 'playwright.config.ts'], // Adjust as needed
  }
);

ESLint v9+ uses a new configuration file format. Checkout the ESLint documentation for more details.

4. Install the ESLint VS Code Extension

To see the errors directly in your editor, make sure you have the official ESLint extension installed in VS Code. Search for dbaeumer.vscode-eslint in the Extensions view and click Install.

Tip: If ESLint warnings don't appear immediately after configuration, try reloading the VS Code window (Command Palette: Developer: Reload Window).

The Result: Instant Feedback in VS Code

Now, when you forget an await, VS Code will show a squiggly line under the issue before you even run the test.

Here’s the fixed version of the earlier code:

await page.getByRole('button', { name: 'Submit' }).click();
await expect(page.getByRole('dialog')).toBeVisible(); // ✅ Correct!

That one missing keyword can make a world of difference, and now you'll never miss it again.

Bonus: Add Type Checking to Your CI

Want an extra safety net? Add TypeScript checking to your CI pipeline:

npx eslint && tsc

It ensures your code compiles cleanly and flags async issues if you enable strict settings.

Say Goodbye to Flaky Tests

And that’s it! No more sneaky await issues tripping up your Playwright tests. Set up ESLint once, and future you will thank you every time a missing await gets caught before it becomes a flaky bug.

If this tip helped you out, drop a comment, and feel free to share it with your team. Happy testing!

Happy testing with Playwright!

Useful links

Handling Visibility in Playwright: getByText vs. getByRole

Debbie O'Brien — Tue, 18 Mar 2025 09:18:17 +0000

When automating web testing with Playwright, handling visibility properly is crucial, especially when elements share the same text. In this post, we'll explore how Playwright's locator methods handle visibility and how to ensure your tests interact with the right elements.

The Scenario: Multiple Buttons with the Same Text

Imagine you have a web page with two buttons: one is hidden, and one is visible. Both buttons contain the text "Submit":

<button style="display: none">Submit</button>
<button>Submit</button>

If you attempt to interact with the "Submit" button using getByText, you might run into an issue.

How `getByText` and `getByRole` Differ

When you use getByText to locate the button, Playwright may find multiple matching elements, including hidden ones:

await page.getByText('Submit').click();

Since getByText does not automatically filter out hidden elements, you might unintentionally target an invisible button.

On the other hand, getByRole behaves differently—it only finds visible elements by default:

await page.getByRole('button', { name: 'Submit' }).click();

This difference in behavior means that getByRole will not run into the same issue as getByText in cases where hidden elements are present.

Explicitly Filtering for Visibility

To make getByText and other locators behave like getByRole, you need to explicitly filter for visibility. Adding .filter({ visible: true }) ensures only visible elements are considered, avoiding unintended interactions:

await page.getByText('Submit').filter({ visible: true }).click();

This approach works not only for getByText but also for other locator methods like locator() that do not automatically filter out hidden elements.

Conclusion

Understanding how different Playwright locators handle visibility can help ensure that your tests interact with the right elements. getByRole inherently filters out hidden elements, whereas getByText and other locators do not, which can lead to unintended interactions. If you're using getByText or other locators, remember to explicitly filter for visibility when needed.

Happy testing with Playwright!

Useful links

Playwright Release 1.51: Smarter Debugging, Enhanced Reports & More!

Debbie O'Brien — Fri, 07 Mar 2025 12:12:38 +0000

Playwright 1.51 is here, bringing a host of new features to make debugging, reporting, and test automation even more powerful. Whether you're tackling test failures, improving test visibility, or streamlining authentication workflows, this release has something for you. Let’s dive into what’s new!

🚀 AI-Friendly Copy Prompt for Debugging

When a test fails, Playwright now provides a Copy Prompt feature that generates a structured debugging prompt. You can paste this directly into an AI tool to get assistance in understanding and fixing the issue. This makes troubleshooting faster and more efficient. The copy prompt button appears in the error tab of both the HTML reporter and UI mode.

If you want your prompt to be smart and include the diff, add this to your config:

export default defineConfig({
   captureGitInfo: { diff: true }
 });

🪪 Git Info in HTML Reports

Understanding test failures in context just got easier! Playwright 1.51 allows you to include Git commit details in your HTML reports, providing more traceability for when and why a test was executed. This is especially helpful for teams working in CI/CD environments where changes happen frequently.

Capture and Store Git Information in HTML Reports

If you don’t use GitHub, GitLab, or Jenkins, you need to specify this line:

export default defineConfig({
   captureGitInfo: { commit: true }
 });

🔥 Firebase Authentication Support

Testing applications that use Firebase authentication is now smoother with native support. This enhancement simplifies authentication flows in tests, making it easier to validate user access and interactions without complex setup.

Save and Restore IndexedDB Contents in Storage State – Firebase Authentication

To persist authentication state, including IndexedDB, you can use the following setup:

import { test as setup, expect } from '@playwright/test';
import path from 'path';

const authFile = path.join(__dirname, '../playwright/.auth/user.json');

setup('authenticate', async ({ page }) => {
  await page.goto('/');
  // ... perform authentication steps ...
  // make sure to save IndexedDB
  await page.context().storageState({ path: authFile, indexedDB: true });
});

👁️ Match Only Visible Locators

A major improvement in locator precision! Playwright now lets you filter locators to match only visible elements using filter: { visible: true }. This ensures tests interact only with elements users can see, reducing flakiness and improving reliability.

Example: Filtering Locators to Only Match Visible Elements

Consider the following HTML:

<button id="submit1" type="button" style="display: none">Submit</button>
<button id="submit2" type="button">Submit</button>

Using getByText('Submit') would cause a strict mode violation because multiple buttons match. To filter only visible locators, use:

import { test, expect } from '@playwright/test';

test('should only show visible submit button', async ({ page }) => {
  await page.goto('/');
  await expect(page.getByText('Submit').filter({
    visible: true
  })).toBeVisible();
});

🪜 Test Step Improvements: Timeouts, Skipping, and Attachments

Managing test steps is now even more flexible. Playwright 1.51 introduces:

Timeouts for test steps, allowing granular control over execution time.
Skipping steps conditionally, making workflows more adaptable.
Attachments for test steps, improving reporting and debugging visibility.

Add Timeout to a Test Step

Specify a maximum runtime for an individual test step:

import { test, expect } from '@playwright/test';

test('some test', async ({ page }) => {
  await test.step('a step', async () => {
    // This step can time out separately from the test
  }, { timeout: 1000 });
});

Conditionally Skip a Running Step

Abort the currently running step and mark it as skipped with an optional description:

import { test, expect } from '@playwright/test';

test('basic test', async ({ page, isMobile }) => {
  await test.step('check some behavior', async step => {
    step.skip(isMobile, 'not relevant on mobile layouts');
    // ... rest of the step code
  });
});

Attach Files or Values to a Test Step

Attach a value or a file from disk to the current test step:

import { test, expect } from '@playwright/test';

await test.step('Verify movie rating tooltip content', async (step) => {
  await firstMovieRating.hover();
  const tooltip = page.getByRole('tooltip');
  const tooltipText = await tooltip.textContent() ?? '';
  await expect(tooltip).toContainText(/average rating on/);

  // Attach the tooltip text content to the test report
  await step.attach('tooltip text content', { body: tooltipText, contentType: 'text/markdown' });
});

These enhancements make complex test scenarios more maintainable and easier to debug.

📷 ARIA Snapshot Updates

ARIA snapshots, which help in accessibility testing, now store snapshots in separate YAML files. This improves organization and makes it easier to track updates when snapshots change.

Example: Storing ARIA Snapshots in Separate Files

await expect(page.getByRole('main')).toMatchAriaSnapshot({ name: 'main.aria.yml' });

Updating Snapshots & Source Code

--update-snapshots updates all snapshots or only changed ones.
--update-source-method defines how source code is updated (overwrite, 3-way, or patch).

Example:

npx playwright test --update-snapshots=changed --update-source-method=3way

🎉 Upgrade to Playwright 1.51

With Playwright 1.51, debugging becomes smarter, reports get more context, and test automation becomes even more reliable. Ready to explore these new features? Upgrade now and take your Playwright tests to the next level! 🚀

npm install @playwright/test@latest

Check out the latest release video for more details and a live demo:

What’s your favorite feature in this release? Let us know in the comments! 👇

How to Reproduce CI Failures Locally in Playwright

Debbie O'Brien — Wed, 19 Feb 2025 18:55:55 +0000

Have you ever run a Playwright test locally, watched it pass with flying colors, and then pushed your code—only for it to fail on CI? Debugging these inconsistencies can be tricky, but there’s a simple trick that can help you figure out what’s going wrong.

The Key to Diagnosing Flaky Tests

When a test passes consistently on your local machine but fails on CI, the issue is often related to timing, race conditions, or resource constraints. To expose these hidden problems, you can use Playwright’s --repeat-each flag.

Step 1: Run Your Test Multiple Times

Use the following command to run your test 100 times:

npx playwright test file-with-failing-test.spec.ts:20 --repeat-each=100

This will execute the test repeatedly in multiple workers, increasing the chances of catching intermittent failures. If there’s a race condition or an unstable dependency, it’s more likely to show up during repeated runs.

Step 2: Stress Test with More Workers

Sometimes, running the test repeatedly isn’t enough, especially if the failure is triggered by CPU or memory constraints on CI. In this case, you can stress your system by maximizing the number of workers:

npx playwright test --repeat-each=100 --workers=10

If your local machine has 10 CPU cores, Playwright will utilize them to simulate the load your test might face in a CI environment. This can help uncover concurrency issues and race conditions that may not appear in single-run scenarios.

Step 3: Stop on First Failure

Running a test 100 times can take a while, but you don’t have to wait for all runs to complete. Use the -x flag to stop execution as soon as a failure occurs:

npx playwright test --repeat-each=100 -x

This will save you time by immediately identifying when a test starts failing under stress.

Final Check: Ensure Stability Before Pushing

Once you’ve identified and fixed the root cause of the flaky test, rerun the test with --repeat-each=100 to confirm stability. If all 100 tests pass, you can push your changes with confidence, knowing it’s far less likely to fail on CI.

Conclusion

Debugging flaky Playwright tests on CI doesn’t have to be a guessing game. By leveraging --repeat-each, --workers, and -x, you can reliably reproduce failures locally and fix them before they cause problems in your pipeline.

If you found this guide helpful, be sure to share it with your team and check out more Playwright testing tips. 🚀 Happy testing!

🔹 Want to see this in action? Check out our latest video:

Useful links

Playwright Assertions: Avoid Race Conditions with This Simple Fix!

Debbie O'Brien — Fri, 14 Feb 2025 15:26:42 +0000

Writing reliable and stable tests is crucial for ensuring a smooth CI/CD pipeline. However, one common mistake many testers make in Playwright is introducing race conditions in their assertions. A small difference in how we write assertions can go a long way in making our tests robust.

In this blog post, we’ll discuss how to properly assert text in Playwright to avoid race conditions and ensure consistent test results.

The Problem: Flaky Assertions

Consider the following Playwright assertion:

expect(await heading.textContent()).toBe('Action');

At first glance, this might seem like a valid assertion. However, this approach can lead to flakiness. Here’s why:

textContent() fetches the value immediately, meaning it does not wait for the element or page to update.

If the element or page hasn’t yet rendered or updated with the expected text, the test fails unnecessarily.

This can result in intermittent failures, making debugging and test maintenance more difficult.

The Solution: Playwright’s Auto-Retrying Assertions

Playwright provides a better way to assert text values:

await expect(heading).toHaveText('Action');

Why is this better?
✅ Automatic Waiting – Playwright waits for the condition to be met before proceeding.
✅ Handles Dynamic Content – Ensures the test does not fail due to minor timing issues.
✅ More Readable & Maintainable – Clearly expresses the intent.

How Playwright Handles Assertions Under the Hood

Playwright’s expect().toHaveText() is a polling assertion, meaning it repeatedly checks the condition until it passes or times out. By default, Playwright waits for up to 5 seconds, ensuring the text has had enough time to update before failing the test.

Conclusion

Avoiding race conditions in Playwright assertions is key to reliable test automation. Always use auto-retrying assertions like toHaveText() instead of textContent(). This simple adjustment can save hours of debugging and make your test suite robust and stable.

Check out the Video

🔹 Want to see this in action? Check out our latest video where we demonstrate the difference between these approaches:🚀

Useful Links

Playwright docs
Join our Discord server
Subscribe to the Playwright YouTube channel

Have you encountered flaky tests due to race conditions? Let’s discuss in the comments! 👇

Organizing Playwright Tests Effectively

Debbie O'Brien — Tue, 21 Jan 2025 13:28:30 +0000

Organizing Playwright Tests Effectively

When working with end-to-end (E2E) testing in Playwright, maintaining a clean and scalable test suite is crucial. A well-organized structure not only improves maintainability but also makes it easier to onboard new team members. In this post, we'll cover how to best organize your Playwright tests, from folder structures to using hooks, annotations and tags.

Structuring Your Test Folders

Playwright tests are typically organized within a tests folder. You can create multiple levels of sub folders within the tests folder to better organize your tests. When dealing with tests that require user authentication, separating tests into logged-in and logged-out states makes the test suite cleaner. Here's an example folder structure:

/tests
  /helpers
    - list-test.ts        # custom fixture for a page with list of movies
    - list-utilities.ts       # helper functions for creating lists of movies
  /logged-in
    - api.spec.ts    # API tests for logged-in users
    - login.setup.ts      # Tests for logging in
    - manage-lists.spec.ts  # Tests for managing lists of movies
  /logged-out
    - api.spec.ts # Tests API endpoints for logged-out users
    - auth.spec.ts     # Tests login flow
    - movie-search.spec.ts  # Tests for searching movies
    - sort-by.spec.ts       # Tests for sorting movies

Using Test Hooks

Playwright offers test hooks such as beforeEach and afterEach to handle common setup and teardown tasks for each test. These hooks are particularly useful for actions like logging in, initializing test data, or navigating to a specific page before each test.

test.beforeEach(async ({ page }) => {
  await page.goto('');
});

test('should edit an existing list', async ({ page }) => {
  // ...
});

test('should add and delete movies from a list', async ({ page }) => {
  //...
});

When you have common setup and teardown tasks across multiple tests, helper functions can help you avoid code duplication and keep your tests DRY (Don't Repeat Yourself). Below is an example of helper functions used to create a list of movies:

import { test, expect, Page } from '@playwright/test';

export async function createList(
  page: Page,
  listName: string,
  listDescription: string,
) {
  await test.step('create a new list', async () => {
    await page.getByLabel('User Profile').click();
    await page.getByRole('link', { name: 'Create New List' }).click();
    await page.getByLabel('Name').fill(listName);
    await page.getByLabel('Description').fill(listDescription);
    await page.getByRole('button', { name: 'Continue' }).click();
  });
}

export async function openLists(page: Page, name: string = 'My Lists') {
  //...
}

The helper functions can be used in the beforeEach hook in various test files ensuring each test starts with a list of movies and opens on the lists page:

import { test, expect } from '@playwright/test';
import { addMovie, createList, openLists } from '../helpers/list-utilities';

// Before each test, navigate to the base URL, create a list, and open the lists page
test.beforeEach(async ({ page }) => {
  await page.goto('');
  await createList(
    page,
    'my favorite movies',
    'here is a list of my favorite movies',
  );
  await openLists(page);
});

test('should edit an existing list', async ({ page }) => {
  await page.getByRole('link', { name: 'my favorite movies' }).click();
  await page.getByRole('link', { name: 'Edit' }).click();
  // ...
});

test('should add and delete movies from a list', async ({ page }) => {
  await page.getByRole('link', { name: 'my favorite movies' }).click();
  await page.getByRole('button', { name: 'Add/Remove Movies' }).click();
  //...
});

Fixtures

Fixtures can be used instead of a beforeEach hook and are a great way of creating a page context that can be shared across multiple tests. Fixtures can be defined in a separate file and imported into test files where they are needed. Here's an example of a fixture for a movie list page:

export const listTest = baseTest.extend<{ listPage: Page }>({
  listPage: async ({ context }, use) => {
    // fixture setup
    const page = await context.newPage();
    await page.goto('');
    await createList(page, 'my favorite movies', 'list of my favorite movies');

    await listTest.step('add movies to list', async () => {
      await addMovie(page, 'Twisters');
      await addMovie(page, 'The Garfield Movie');
      await addMovie(page, 'Bad Boys: Ride or Die');
    });
  //...
});

The fixture can be used in test files by importing it and passing it as an argument to the test function instead of passing in Playwright's built in page fixture. Each test that uses the custom fixture will start with a page that has a list of movies:

import { expect } from '@playwright/test';
import { listTest as test } from '../helpers/list-test';
import { addMovie } from '../helpers/list-utilities';

test('editing an existing list', async ({ listPage }) => {
  // set the page to the listPage fixture
  const page = listPage;

  await page.getByRole('link', { name: 'Edit' }).click();
  // ...
});

Splitting Tests into Steps with `test.step`

When you want to add more clarity to your tests, Playwright's test.step function is handy. It breaks down complex tests into more digestible steps, improving readability and reporting.

import { test, expect } from '@playwright/test';

test('should add and delete movies from a list', async ({ page }) => {
  const movieList = page.getByRole('listitem', { name: 'movie' });
  await page.getByRole('link', { name: 'my favorite movies' }).click();
  await page.getByRole('button', { name: 'Add/Remove Movies' }).click();

  await test.step('add and verify movies in the list', async () => {
    await addMovie(page, 'Twisters');
    await addMovie(page, 'Bad Boys: Ride or Die');
    await expect
      .soft(movieList)
      .toHaveText([/Twisters/, /Bad Boys: Ride or Die/]);
  });

  await test.step('remove and verify movies in the list', async () => {
    const movie1 = page.getByRole('listitem').filter({ hasText: 'Twisters' });
    await movie1.getByLabel('Remove').click();
    await expect.soft(movieList).toHaveText([/Bad Boys: Ride or Die/]);
  });
});

Using Built-in Annotations: `skip`, `fail`, and `fixme`

Annotations in Playwright help you mark tests for specific behaviors or conditions, such as skipping them under certain conditions, marking them as expected failures, or flagging them as needing attention. Let's dive into each one with examples:

1. `test.skip`: Skipping Tests Conditionally

The test.skip annotation is useful when you want to skip a test based on certain conditions or configuration. For example, you might want to skip tests on specific browsers, in particular environments, or when a feature is not available.

import { test, expect } from '@playwright/test';

test('hamburger menu in mobile', async ({ page, isMobile }) => {
  // Skip the test if viewport is mobile
  test.skip(!isMobile, 'Test is only relevant for desktop');

  await page.goto('/');
  //..
});

test('should not run on Safari', async ({ page, browserName }) => {
  // Skip the test on Safari due to known compatibility issues
  test.skip(browserName === 'webkit', 'Safari not supported for this feature');

  await page.goto('');
  //..
});

2. `test.fixme`: Marking Tests to Be Fixed

Use test.fixme to mark a test that you intend to fix later. This annotation is typically applied to tests that are not yet implemented correctly or tests that are incomplete. Playwright will automatically skip these tests and flag them in reports, serving as a reminder.

import { test, expect } from '@playwright/test';

// Mark this test as a "fixme" since it's not fully implemented yet
test.fixme('log user in and verify profile access', async ({ page }) => {
  await page.goto('');
  await page.getByRole('banner').getByLabel('Log In').click();
  await page
    .getByPlaceholder('you@example.com')
    .fill(process.env.MOVIES_USERNAME!);
  await page.getByPlaceholder('Password').fill(process.env.MOVIES_PASSWORD!);
  await page.getByRole('button', { name: 'login' }).click();
  await page.getByLabel('User Profile').click();
});

How These Annotations Help

test.skip is excellent for conditionally running tests based on criteria like environment, platform, or feature availability. This helps maintain a green test suite by skipping irrelevant tests instead of letting them fail.
test.fixme is useful when you have incomplete or not-yet-implemented features. These tests are automatically skipped and reported, keeping them in mind as future to-dos.

Summary of Annotations Usage

Using these built-in annotations can streamline your workflow, reduce false negatives in test results, and communicate known issues and unimplemented features clearly to your team.

Adding custom Annotations

In Playwright, you can add custom annotations to your tests. For example, a common practice is including a link to a related issue. These can be valuable in reports and are also visible in UI mode. Here's an example of how you can add issue links to your tests:

test(
  'should delete a list',
  {
    annotation: {
      type: 'issue',
      description: 'https://github.com/microsoft/demo.playwright.dev/issues/58',
    },
  },
  async ({ page }) => {
    await page.getByRole('link', { name: 'my favorite movies' }).click();
    await page.getByRole('link', { name: 'Edit' }).click();
    await page.getByRole('link', { name: 'Delete List' }).click();
  // ... remaining test steps
});

Using Tags to Filter and Organize Tests

Tags are a powerful way to categorize and filter tests. You can use tags to organize tests based on features, priorities, user roles, or release cycles. This allows you to easily filter and run specific sets of tests or generate targeted reports.

Adding Tags

In Playwright, you can add tags as part of the test definition's metadata. Here's an example of how to define tags for tests:

test('sort by with api mocking', { tag: '@mocking' }, async ({ page }) => {
  // Mock the API call for sorting by popularity
  await page.route('*/**/**sort_by=popularity.desc', async (route) => {
    await route.fulfill({
      path: path.join(__dirname, '../mocks/sort-by-popularity.json'),
    });
  });
});

In this example, we've tagged the test with a "@mocking" tag. This metadata will be used to filter tests in reports or command-line execution.

Running Tests by Tags

To run tests with specific tags, you can use the --grep flag followed by a tag name:

npx playwright test --grep @mocking

To exclude tests with a specific tag, use the --grep-invert flag:

npx playwright test --grep-invert @mocking

Filtering by Tags in HTML Reports

Tags are displayed in the HTML report, making it easy to identify and filter tests based on tags while reviewing results. This can be very helpful for debugging or focusing on a subset of tests related to a specific feature.

Summary

By following these guidelines, you can create a well-structured and maintainable Playwright test suite:

Organized Folder Structure: Separate tests based on their context (logged-in vs. logged-out) and per feature. For example, testing GitHub we could have tests/repos, tests/prs, tests/issues, etc.
Using Hooks and Describe blocks: Improve readability and set up common prerequisites.
Step Definition: Use test.step to break down complex test cases.
Leveraging Annotations and Tags: Use annotations and tags to mark failing or incomplete tests, link issues, and categorize your tests for better filtering and reporting.

With a thoughtful approach to organizing your tests, you'll be able to create a cleaner and more maintainable test suite that scales well with your application.

Happy testing! 🎭

Useful links

Playwright documentation
Subscribe to our YouTube channel
Star us on GitHub
Join our Discord server
[Check out our DEMO: Playwright movies APP(https://github.com/debs-obrien/playwright-movies-app)

Accelerate your Playwright test suite with Microsoft Playwright Testing

Vansh Vardhan Singh — Tue, 12 Nov 2024 18:10:27 +0000

Are you tired of waiting for your Playwright test suite to finish? Do you want to run tests across multiple operating systems and browsers without the hassle of managing complex infrastructure? With Microsoft Playwright Testing, you can speed up your test runs and improve test coverage by seamlessly executing tests across various OS-browser combinations.

In this post, we’ll dive into the cloud-hosted browsers feature offered by the service, explore how it works, and learn how to integrate it with your existing Playwright test suite.

Microsoft Playwright Testing service

Microsoft Playwright Testing is a managed service that leverages the cloud to allow much higher test parallelization across different OS-browser combinations. This means faster test runs with broader coverage, enabling quicker feature delivery without compromising quality.

As your web application grows, your test suite expands, leading to longer execution times that can bottleneck your CI pipeline and delay feature delivery. Modern web apps must work flawlessly across various browsers and operating systems, adding to test duration. Microsoft Playwright Testing eliminates these limitations by running your tests on cloud-hosted browsers, allowing you to:

Expedite test runs: Run more tests in parallel without worrying about local resource constraints.
Improve test coverage: Easily test across different browsers and operating systems, with infrastructure management handled by the service.
Test with confidence: Rely on a managed infrastructure for consistent results, reducing test flakiness with no additional overhead.

How does the service work?

The Playwright Testing service utilizes Playwright OSS’s native ability to run multiple tests in parallel. When you trigger a test run, Playwright spins up multiple worker processes, each responsible for launching its own browser instance.

When running tests on your dev machine, the number of browsers you can launch is restricted by your system's resources. This limitation becomes even more pronounced in CI pipelines, where large test suites need to be executed, often causing delays as teams wait for results to move forward.

With Microsoft Playwright Testing, however, the browsers are hosted and managed in the cloud. This eliminates local resource constraints, enabling you to run many more tests in parallel. Additionally, you can choose the operating system where the browsers are hosted, improving test coverage without increasing complexity.

How to Integrate Your Test Suite with Microsoft Playwright Testing

To get started with Microsoft Playwright Testing, follow these steps:

Prerequisites

A functioning Playwright test suite in JS/TS using the Playwright Test Runner. If you don't have one, create it by using the Playwright getting started documentation or use our Microsoft Playwright Testing sample project.
An Azure account with an active subscription. If you don't have an Azure subscription, create a free account before you begin.
Azure CLI installed download it here.

Steps to integrate

Create a Playwright Testing workspace:

Sign in to service portal using your Azure credentials.
Select your subscription, click New Workspace, and provide the following information:

Field	Description
Workspace name	Enter a unique name to identify your workspace. The name can only consist of alphanumerical characters and must be between 3 and 64 characters long.
Azure subscription	Select the Azure subscription that you want to use for this Microsoft Playwright Testing workspace.
Region	Select a geographic location to host your workspace. This is the location where the test run data is stored for the workspace.

Click Create workspace to create it within your subscription. Once the workspace is created, you will be taken to a page with instructions to get started.

Install the service package in your Playwright project:

In your IDE, navigate to Playwright project and run the following command. This will create a playwright.service.config.ts file alongside your Playwright config file, enabling tests to run on the service.
```
npm init @azure/microsoft-playwright-testing
```
Set up Authentication:

Microsoft Entra ID is the default and recommended authentication for the service. Run this command to sign in:
Note: Before running this command, make sure you have Azure CLI installed.
```
 az login
```
Configure region endpoint:

Copy the region endpoint from service portal and set it as 'PLAYWRIGHT_SERVICE_URL' environment variable in your test setup.
Run the tests on the service:

Run the following command. This will create 20 workers, each executing a test on the browsers hosted on the service.
```
npx playwright test --config=playwright.service.config.ts --workers=20
```

Next steps

Run tests across different operating systems:
- Modify the playwright.service.config.ts file to change the operating system. Learn more here.
Test locally hosted applications:
- To test locally hosted or private applications, set the exposeNetwork field in playwright.service.config.ts. Learn more here.
Integrate MPT with CI pipelines to expedite test execution:
- Integrating the service in CI pipelines is easy. Here’s the documentation.

Conclusion

Microsoft Playwright Testing streamlines your testing workflows, allowing you to run tests across multiple operating systems and browsers without the complexity of managing infrastructure. By integrating cloud-hosted browsers into your existing Playwright test suite, you can significantly reduce test execution times, improve test coverage, and enhance overall efficiency—leading to faster feature delivery and smoother releases.

Learn more

Get started with Microsoft Playwright Testing.
Explore Playwright Testing features.
Check out the pricing.
Join the mpt-service channel on our Discord community.
Playwright on YouTube.

Implementing Microsoft Entra Certificate-Based Authentication with Playwright

Max Schmitt — Wed, 18 Sep 2024 13:33:06 +0000

Introduction to Certificate-Based Authentication (CBA)

Microsoft recently announced (July 2024) support for certificate-based authentication (CBA) for Microsoft Entra. CBA is a phishing-resistant, passwordless, and convenient way to authenticate users with X.509 certificates without relying on passwords. With the recent v1.46 release, Playwright now supports the CBA method for authenticating with Entra.

TLS Client Certificates and Their Role in CBA

TLS client certificates are digital credentials that authenticate clients to servers in secure connections. They contain the client's public key, are signed by trusted authorities, and enable mutual authentication. These certificates enhance security by verifying client identities, preventing unauthorized access to sensitive resources in various IT environments.

If no client certificate is available, your browser will display a prompt like this:

Implementing CBA with Playwright

Step 1: Obtain the certificates

If the certificate files are not yet available on the client, we recommend defining base fixtures that will fetch the certificates in every worker process. The certificates will then remain in memory. See here for all available options for clientCertificates (e.g., if you are using PEM instead of PKCS#12).



import { DefaultAzureCredential } from '@azure/identity';
import { SecretClient } from '@azure/keyvault-secrets';
import { test as base } from '@playwright/test'

// DefaultAzureCredential automatically detects and uses the most appropriate authentication method,
// including environment variables, managed identities, and OIDC tokens from GitHub Actions.
// We recommend OIDC logins via e.g. azure/login:
// https://github.com/Azure/login/?tab=readme-ov-file#login-with-openid-connect-oidc-recommended
const credential = new DefaultAzureCredential();

const vaultName = '<YOUR KEYVAULT NAME>';
const KEYVAULT_URI = `https://${vaultName}.vault.azure.net`;
const secretName = '<YOUR SECRET>';

const secretClient = new SecretClient(KEYVAULT_URI, credential);

export const test = base.extend({
  clientCertificates: async ({ }, use) => {
    const certificateSecret = await secretClient.getSecret(secretName);
    await use([
      {
        origin: 'https://certauth.login.microsoftonline.com',

        // Alternatively, if you use e.g. PPE, it's: https://certauth.login.windows-ppe.net
        pfx: Buffer.from(certificateSecret.value!, 'base64'),

        // You might need to provide your passphrase here:
        // passphrase: process.env.SECRET_PASSPHRASE
      }
    ]);
  }
});

export { expect } from '@playwright/test'

Step 2: Update your test @playwright/test imports



-import { test, expect } from '@playwright/test';
+import { test, expect } from './baseTest';

Step 3: Log in inside a test



import { test, expect } from './baseTest';

test('is able to login', async ({ page }) => {
  await page.goto('https://your-application.com/login');
  // Your login logic here
  await page.getByRole('textbox', { name: 'Email' }).fill('example@foo.com');
  await page.getByRole('button', { name: 'Sign in' }).click();
});

You can also provide client certificates as a parameter of browser.newContext() and apiRequest.newContext().

Troubleshooting

To debug which network requests are being made, you can set the DEBUG=pw:client-certificates environment variable. This will print all the connections that are being established.

It's a known issue that the authentication does not work if the --disable-web-security argument is passed to Chromium.

Useful links

Iterate quickly using the new --only-changed option

Simon Knott — Wed, 21 Aug 2024 07:14:15 +0000

Playwright v1.46 ships with a nifty new feature for local development.
By specifying the --only-changed option, Playwright looks at uncommited changes and runs all affected test files.

If you've used Jest before, this is pretty similar to Jest's --onlyChanged flag.

Let's see it in action. If you prefer video, watch the Demo in the release video. In this example, we have a repository without uncommitted changes:

// utils.ts
export const inputPlaceholder = 'What needs to be done?'
export const todoItemTestID = 'todo-item'
export const todo = 'buy milk'

// test.spec.ts
import { test, expect } from '@playwright/test'
import { inputPlaceholder, todo, todoItemTestID } from './utils.ts'

test('adding a todo', async ({ page }) => {
    await page.goto('https://demo.playwright.dev/todomvc')

    const inputField = page.getByPlaceholder(inputPlaceholder)
    await inputField.fill(todo);
    await inputField.press('Enter');

    await expect(inputField).toBeEmpty();
    await expect(page.getByTestId(todoItemTestID)).toHaveText(todo);
})

If we run a test with --only-changed, we see that no tests are run:

$ npx playwright test --only-changed
Error: No tests found

Now let's make a change to one of the files, but not commit it yet:

// utils.ts
export const inputPlaceholder = 'What needs to be done?'
export const todoItemTestID = 'todo-item'
- export const todo = 'buy milk'
+ export const todo = 'make cappucino'

❯ npx playwright test

Running 1 test using 1 worker

  ✓  1 [chromium] › test.spec.ts:4:5 › adding a todo (426ms)

  1 passed (837ms)

Playwright found changes in utils.ts, so it executed all test files that depend on utils.ts. Wonderful!

By default, Playwright will compare against HEAD to determine changed files.
But you can also specify a different Git revision to compare against: Using --only-changed=main on a feature branch executes all tests that are affected by changes on the feature branch. And --only-changed=<commit sha> executes all changes between HEAD and the specified commit.

How do I use this?

--only-changed is another tool in the toolbelt to help you iterate quickly. Here's five ideas for how you can integrate it into your workflow:

1. Local development: Use --only-changed=main to quickly run affected tests as you make changes. This is especially useful when you're working on a feature and want to see the impact of your changes.

2. pre-commit hook: Running with --only-changed in a pre-commit hook prevents failing tests from being commited. Here's an example hook you can use:

npx playwright test --only-changed

3. pre-push hook: As the name says, a pre-push hook runs before pushing. This prevents failing tests from invoking a costly CI run. Here's an example implementation:

while read local_ref local_sha remote_ref remote_sha
do
    # deleting branch, nothing to check
    if [ "$local_sha" = '0000000000000000000000000000000000000000' ]; then continue; fi

    echo "Running Playwright for HEAD...$remote_sha"
    npx playwright test --only-changed="$remote_sha"
done

4. Faster feedback in CI: Prior to a full run, use --only-changed to get faster feedback on new failures. This is especially useful in large test suites, where running all tests can take a long time. Here's how that'd look in GitHub Actions:

...
    - name: Install Playwright Browsers
      run: npx playwright install --with-deps
+   - name: Run changed Playwright tests
+     run: npx playwright test --only-changed=${{ github.base_ref }}
    - name: Run Playwright tests
      run: npx playwright test
...

5. Bonus idea: If you're interested in --only-changed, you might also be interested in Playwright's Watch mode that's available in the UI and in the VS Code Extension.

That's it! Let us know what you think and how you're using --only-changed. Either in the comments here, in the Community Discord, or on LinkedIn! And have a wonderful day.

DEV Community: Playwright end to end Testing

Playwright Agents: Planner, Generator, and Healer in Action

Getting Started with Playwright Agents

Seeding your tests

Planner Agent – Creating a Test Plan

Example Prompt

Generator Agent – Creating the Tests

Example Prompt

Healer Agent – Fixing a Failing Test

Example Prompt

Conclusion

What’s New in Playwright v1.52 & v1.53: Fix with AI, Describable Locators, and More!

🧠 Fix with AI in VS Code

🔎 locator.describe() – Better Trace and Report Visibility

📊 Custom Titles in HTML Reports

✅ New Assertion: toContainClass

🧪 Snapshot Enhancements: children and url Support

⚙️ Test Runner Updates

🆙 How to update?

📺 Watch the Demo

Debugging Playwright Tests with AI: A Smarter, Faster Workflow

One-Click AI Fixes

Debugging in UI Mode

Support Across Reports and CI

Faster, Smarter Debugging with AI

Useful links

Catch Missing `await` Calls in Playwright Tests with ESLint

The Problem: A Missing await Can Break Your Test

The Fix: Let ESLint Catch Floating Promises

1. Install ESLint and TypeScript Support

2. Create tsconfig.json (If Needed)

3. Configure ESLint in eslint.config.mjs

4. Install the ESLint VS Code Extension

The Result: Instant Feedback in VS Code

Bonus: Add Type Checking to Your CI

Say Goodbye to Flaky Tests

Useful links

Handling Visibility in Playwright: getByText vs. getByRole

The Scenario: Multiple Buttons with the Same Text

How getByText and getByRole Differ

Explicitly Filtering for Visibility

Conclusion

Useful links

Playwright Release 1.51: Smarter Debugging, Enhanced Reports & More!

🚀 AI-Friendly Copy Prompt for Debugging

🪪 Git Info in HTML Reports

Capture and Store Git Information in HTML Reports

🔥 Firebase Authentication Support

Save and Restore IndexedDB Contents in Storage State – Firebase Authentication

👁️ Match Only Visible Locators

Example: Filtering Locators to Only Match Visible Elements

🪜 Test Step Improvements: Timeouts, Skipping, and Attachments

Add Timeout to a Test Step

Conditionally Skip a Running Step

Attach Files or Values to a Test Step

📷 ARIA Snapshot Updates

Example: Storing ARIA Snapshots in Separate Files

Updating Snapshots & Source Code

🎉 Upgrade to Playwright 1.51

How to Reproduce CI Failures Locally in Playwright

The Key to Diagnosing Flaky Tests

Final Check: Ensure Stability Before Pushing

Conclusion

Useful links

Playwright Assertions: Avoid Race Conditions with This Simple Fix!

The Problem: Flaky Assertions

The Solution: Playwright’s Auto-Retrying Assertions

How Playwright Handles Assertions Under the Hood

Conclusion

Check out the Video

Useful Links

Organizing Playwright Tests Effectively

Organizing Playwright Tests Effectively

Structuring Your Test Folders

Using Test Hooks

Fixtures

Splitting Tests into Steps with test.step

Using Built-in Annotations: skip, fail, and fixme

1. test.skip: Skipping Tests Conditionally

2. test.fixme: Marking Tests to Be Fixed

🔎 `locator.describe()` – Better Trace and Report Visibility

✅ New Assertion: `toContainClass`

🧪 Snapshot Enhancements: `children` and `url` Support

The Problem: A Missing `await` Can Break Your Test

2. Create `tsconfig.json` (If Needed)

3. Configure ESLint in `eslint.config.mjs`

How `getByText` and `getByRole` Differ

Splitting Tests into Steps with `test.step`

Using Built-in Annotations: `skip`, `fail`, and `fixme`

1. `test.skip`: Skipping Tests Conditionally

2. `test.fixme`: Marking Tests to Be Fixed