Go Back

Navigating, Waiting and Timeouts

September 30, 2021

Every useful script that we will write will almost certainly do three key things:

  • Navigating to some web page
  • Waiting for something
  • Possibly getting a timeout

Both frameworks handle these scenarios in very similar ways but Playwright explicitly differentiates itself from Puppeteer by having a “built in” waiting mechanism that covers a lot of common scenarios.

Initial navigation to any page is pretty much the same for both frameworks and can happen in multiple ways.

Whenever your code does a page.goto(), or a page.click() on a link, you explicitly trigger a navigation. The webpage you are on can also trigger a navigation by executing location.href= 'https://example.com' or using the history.pushState() API.

In the example below we trigger two navigations:

  • The initial load of the page.
  • A navigation to the feeds by clicking a link
const puppeteer = require('puppeteer');

(async () => {
  const browser = await puppeteer.launch()
  const page = await browser.newPage()
  await page.goto('https://examplesite.com')
  await page.click('#loginpage')
  await browser.close()
})()

Run this example as follows:

$ node navigation.js

We also add await browser.close() to ensure that we are shutting down our browser before terminating the session. Without it, our execution would not return after the page has loaded, keeping our test hanging indefinitely.

Waiting

In your code, you have a range of options to wait for different things to happen in your browser session. There are a couple that are very important:

page.waitForSelector()

This method waits for an element to appear in the page. This is your bread and butter and should be used whenever something needs to be loaded after clicking, hovering, navigating etc. You can pass it an object with a timeout attribute to override the default 30 seconds.

In the example below, we type an email address into an input field on a login modal.

const puppeteer = require('puppeteer');

(async () => {
  const browser = await puppeteer.launch()
  const page = await browser.newPage()
  await page.goto('https://examplesite.com/')
  await page.click('#login')
  await page.waitForSelector('#email')
  await page.type('#email', '[email protected]')
  await browser.close()
})()

Run this example as follows:

$ node waiting.js

This works exactly the same for the page.waitForXpath() function is you are using XPath selectors instead of CSS selectors.

page.waitForNavigation()

In your scripts you can click on a link that triggers a navigation to a new page. You can use Puppeteer’s page.waitForNavigation() method here to explicitly wait for this event to happen and then continue your script. The accepted notation in Puppeteer’s case is by using the Promise.all() method to wait for the click to happen and the navigation to happen before continuing.

Timeouts

The page.waitForNavigation() method — but also similar methods like page.reload() and page.goBack() — all take some options that determine “how” it should wait and what the timeout limits are.

These options come in two flavors:

  1. Hard timeout

The time in milliseconds passed as the timeout property e.g. page.waitForNavigation({ timeout: 2000 }). We do not recommend using this if you do not explicitly need to.

2a. DOM event based

These two options are directly related to the events your browser emits when it has reached a certain loading stage. These events are not specific to Puppeteer and are used in almost all browsers.

load: This is the default and very strict: your whole page including all dependent resources, i.e. images, scripts, css etc. domcontentloaded: less strict: when your HTML has loaded.

Note: the load option is the default.

2b. Heuristic based

These two options are based on the heuristic that if (almost) all network connections your browser has are no longer active, your page has probably finished loading.

networkidle0: consider navigation to be finished when there are no more than 0 network connections for at least 500 ms. networkidle2: consider navigation to be finished when there are no more than 2 network connections for at least 500 ms.

Playwright has done away with the distinction between networkidle0 and networkidle2 and just has:

networkidle: consider navigation to be finished when there are no more than 0 network connections for at least 500 ms.

Both options 2a and 2b are passed using the waitUntil property, e.g. page.waitForNavigation({ waitUntil: 'networkidle2' })

Which of these options is useful to you depends on your situation:

Does your SPA need to be fully rendered and finish all XHR calls? Go with load You server render and load in some non-crucial element in a lazy fashion? go for one of the networkidle variant.