> ## Documentation index
> Fetch the complete documentation index at: https://upstash.com/docs/llms.txt
> Search it with GET https://upstash.com/docs/search?q=<query>.
> Use these to discover all available pages before exploring further.

# Playwright Web Scraping with a Cloud Browser

> Scrape JavaScript-heavy sites with the managed headless Chromium in Upstash Box. Extract structured data through the SDK, then connect Playwright over CDP.

Every box can come with its own browser. Create one with `browser: true` and you get a managed, headless Chromium that boots on first use. Nothing to install, no `apt-get`, no Chromium binary to keep up to date.

This guide scrapes a JavaScript-heavy site with that built-in browser. We read rendered pages through the SDK, extract structured data against a schema, then connect Playwright over CDP to harvest many pages deterministically.

<Note>
  The browser can only be provisioned when the box is created. It cannot be enabled on an existing box. See [Browser](/box/overall/browser/overview) for the full surface.
</Note>

---

## 1. Installation

```bash
npm install @upstash/box playwright-core zod
```

`playwright-core` is enough. You connect to the box's Chromium instead of launching one locally, so there are no browser binaries to download on your side either.

Set your environment variable:

```bash title=".env"
UPSTASH_BOX_API_KEY=box_xxxxxxxxxxxxxxxxxxxxxxxx
```

---

## 2. Create a box with a browser

```typescript title="scripts/scrape.ts"
import "dotenv/config"
import { Box } from "@upstash/box"

const box = await Box.create({
  runtime: "node",
  browser: true,
})

const tab = await box.browser.tab.create("https://books.toscrape.com", {
  waitUntil: "domcontentloaded",
})

console.log(`Box ready: ${box.id}`)
console.log(tab.id, tab.url, tab.title)
```

The first `tab.create` boots Chromium. The tab handle is addressed by its Chrome DevTools Protocol target id, so it stays valid across navigations and you can re-attach to it later with `box.browser.getTab(id)`.

---

## 3. Read the rendered page

`content()` returns the tab's current title, URL, visible text, and links from the real DOM, including anything JavaScript rendered after load. No model is involved, so this costs no tokens.

```typescript title="scripts/scrape.ts"
const page = await tab.content()

console.log(page.title)
console.log(page.text.slice(0, 200))

for (const link of page.links ?? []) {
  console.log(link.text, link.href)
}
```

This is already enough for a crawler: follow the links you care about with `tab.goto(url)`, read each page with `content()`, and chunk the text into a dataset.

<Note>
  Client-rendered pages can hydrate after `domcontentloaded`. If `page.text` comes back short, wait briefly and call `content()` again until it settles.
</Note>

---

## 4. Extract structured data with a schema

When you want typed fields instead of raw text, `extract()` hands the page to a DOM-aware agent inside the box and validates the result against a [Zod](https://zod.dev) schema:

```typescript title="scripts/scrape.ts"
import { z } from "zod"

const catalog = await tab.extract(
  "extract every book in the listing with its title and price",
  z.object({
    books: z.array(
      z.object({
        title: z.string(),
        price: z.string(),
      }),
    ),
  }),
)

console.table(catalog.books.slice(0, 5))
```

The result is parsed with your schema before it is returned, so a successful call always gives you the shape you asked for. Capture a screenshot next to it if you want provenance for what was on screen:

```typescript
import { writeFile } from "node:fs/promises"

await writeFile("catalog.png", await tab.screenshot({ fullPage: true }))
```

<Note>
  `extract` uses an LLM and is metered. It needs a model provider key on the box or your account. See [AI Actions](/box/overall/browser/ai-actions) for the model override and the full list of providers.
</Note>

---

## 5. Scrape many pages with Playwright over CDP

An AI call per page gets expensive fast. For repeatable multi-page work, drive the same browser with Playwright. `cdpUrl()` returns an authenticated WebSocket URL that `chromium.connectOverCDP` accepts directly:

```typescript title="scripts/scrape.ts"
import { chromium } from "playwright-core"

const browser = await chromium.connectOverCDP(await box.browser.cdpUrl())

try {
  const ctx = browser.contexts()[0] ?? (await browser.newContext())
  const p = await ctx.newPage()

  const books: { title: string | null; price: string | null }[] = []

  for (let n = 1; n <= 3; n++) {
    await p.goto(`https://books.toscrape.com/catalogue/page-${n}.html`, {
      waitUntil: "domcontentloaded",
    })

    books.push(
      ...(await p.$$eval("article.product_pod", (rows) =>
        rows.map((row) => ({
          title: row.querySelector("h3 a")?.getAttribute("title") ?? null,
          price: row.querySelector(".price_color")?.textContent?.trim() ?? null,
        })),
      )),
    )
  }

  console.log(`${books.length} books scraped`)
} finally {
  await browser.close()
}
```

This is an ordinary Playwright script. Migrating an existing one is usually a single line: `chromium.launch()` becomes `chromium.connectOverCDP(await box.browser.cdpUrl())`, and your selectors, actions, and assertions stay as they are.

<Warning>
  The CDP URL carries its auth token in the URL. Anyone who has it gets full control of the browser, so treat it as a secret.
</Warning>

CDP and the SDK drive the same browser and the same tabs. A page opened by Playwright shows up in `box.browser.listTabs()`, and a tab created by the SDK is visible to Playwright, so you can mix scripted steps with AI steps in one run.

---

## 6. Compile the selectors once, then scrape for free

The two previous sections combine into the pattern worth using in production: let the model read the layout **once** and emit selectors, then run every later scrape deterministically with no model tokens at all.

This is a file of its own rather than a continuation of the script above, so it opens its own box and browser:

```typescript title="scripts/compile-and-scrape.ts"
import "dotenv/config"
import { Box } from "@upstash/box"
import { chromium } from "playwright-core"
import { z } from "zod"

const PAGE = (n: number) => `https://books.toscrape.com/catalogue/page-${n}.html`
const RECIPE_PATH = "recipe-books.json"

const RecipeSchema = z.object({
  itemSelector: z.string(),
  title: z.object({ selector: z.string(), attr: z.string() }),
  price: z.object({ selector: z.string(), attr: z.string() }),
})
type Recipe = z.infer<typeof RecipeSchema>

const box = await Box.create({ runtime: "node", browser: true })

// Ask for the selectors, not the data. One metered call per layout.
async function compile(): Promise<Recipe> {
  const tab = await box.browser.tab.create(PAGE(1), { waitUntil: "domcontentloaded" })
  return await tab.extract(
    [
      "You are compiling a scraper for the repeating list of books.",
      "Return `itemSelector`: a CSS selector matching each book's container.",
      "Return `title` and `price` field specs: each with a CSS `selector`",
      'RELATIVE to the container and `attr` ("text" or an attribute name).',
      "If the visible text is truncated, prefer an attribute holding the",
      "full value (such as an anchor's title attribute).",
      "Selectors must be generic, no :nth-child tied to one item.",
    ].join(" "),
    RecipeSchema,
  )
}

// Deterministic harvest: same recipe, every page, zero model tokens.
async function scrape(recipe: Recipe, pages: number) {
  const browser = await chromium.connectOverCDP(await box.browser.cdpUrl())
  try {
    const ctx = browser.contexts()[0] ?? (await browser.newContext())
    const p = await ctx.newPage()
    const rows = []
    for (let n = 1; n <= pages; n++) {
      await p.goto(PAGE(n), { waitUntil: "domcontentloaded" })
      rows.push(
        ...(await p.$$eval(
          recipe.itemSelector,
          (items, fields) =>
            items.map((row) => {
              const read = (field: { selector: string; attr: string }) => {
                const el = field.selector ? row.querySelector(field.selector) : row
                if (!el) return null
                return field.attr === "text" ? el.textContent?.trim() ?? null : el.getAttribute(field.attr)
              }
              return { title: read(fields.title), price: read(fields.price) }
            }),
          { title: recipe.title, price: recipe.price },
        )),
      )
    }
    return rows
  } finally {
    await browser.close()
  }
}

// Reuse the recipe cached on the box, or compile and cache a fresh one.
let recipe: Recipe
try {
  recipe = RecipeSchema.parse(JSON.parse(await box.files.read(RECIPE_PATH)))
} catch {
  recipe = await compile()
  await box.files.write({ path: RECIPE_PATH, content: JSON.stringify(recipe, null, 2) })
}

const rows = await scrape(recipe, 3)
console.log(`${rows.length} books`, rows[0])
```

Leave that tab open. Closing the last tab shuts the browser down, and the CDP connection in `scrape()` then fails with `The browser is shutting down`.

Caching the recipe on the box filesystem means the box carries its own knowledge: the next run reuses it and calls no model at all.

Validate the harvest (row count, non-empty fields, price format) on every run. When validation fails, the site's layout changed: recompile the recipe with one `extract` call and cache the new one. That way the AI cost is paid once per layout, not once per page.

A full runnable version of this is the [AI-compiled scraper](https://github.com/upstash/box/blob/main/packages/sdk/examples/browser/retrieval/02-ai-compiled-scraper.ts) example.

---

## 7. Reuse the box across runs

A browser box has nothing to install, so there is no setup cost to snapshot away. What is worth keeping is the box itself: the cached recipe, the cookies, and a logged-in session.

Keep the box id and re-attach on the next run:

```typescript title="scripts/run-scrape-job.ts"
const box = process.env.SCRAPER_BOX_ID
  ? await Box.get(process.env.SCRAPER_BOX_ID)
  : await Box.create({ runtime: "node", browser: true })
```

The box pauses when idle and resumes on demand with its filesystem intact. It bills until you delete it, so delete it when you are done with the target site:

```typescript
await box.delete()
```

To log in once and have every later script reuse that session, see the [login once, reuse session](https://github.com/upstash/box/blob/main/packages/sdk/examples/browser/automation/04a-login-once-keep-alive.ts) example.

---

## Next steps

- [Reading pages](/box/overall/browser/reading-pages) for `content`, `screenshot`, and `extract`.
- [AI actions](/box/overall/browser/ai-actions) for `observe` and `act`, including replaying a resolved action with no LLM.
- [Live view](/box/overall/browser/live-view) and [recordings](/box/overall/browser/recordings) to watch or replay a scrape.
- [Browser cookbook](/box/overall/browser/cookbook) for runnable examples across crawling, automation, and testing.