Monitoring a Chrome Extension in Production: Building a 24/7 Playwright Watchdog

Browser extensions live inside someone else's DOM. Your extension can be perfectly engineered, fully covered by tests, and still break in production because the host app shipped a redesign overnight and renamed the CSS class your content script was anchoring to. No unit test catches that. No CI run against a saved fixture catches that. The only way to know is to open the real product, with the real extension, as a real user — and look.

This post walks through a monitoring system I built for exactly that problem: a 24/7 watchdog that loads the Everhour Chrome extension into Chromium, opens Asana pages every 10 minutes, verifies that 20+ injected UI controls are rendered, and sends a Slack alert with a pixel-diff screenshot when something disappears. The whole thing runs in Docker with no monitor attached.

The interesting parts are not the selectors. They are the reliability engineering around the checks: headed Chromium in a container, retries driven by error classification, a watchdog that watches the watcher, and an alerting pipeline designed to not cry wolf.

# Why headed Chromium, and why Xvfb

Chrome extensions only load in a persistent context — browser.newContext() will not cut it. And while Chromium's newer headless mode has made progress with extensions, the battle-tested route for a 24/7 system is still a headed browser: --load-extension behaves predictably, extension service workers start the same way they do for real users, and the page's anti-automation heuristics treat the session as normal.

A headed browser needs a display. In Docker there is none, so you give it a virtual one:

# entrypoint: start virtual display, then the app
Xvfb :99 -screen 0 1920x1080x24 &
export DISPLAY=:99
node dist/main.js

The browser itself is launched once and kept alive between check cycles:

const context = await chromium.launchPersistentContext(userDataDir, {
  headless: false,
  args: [
    `--load-extension=${extensionPath}`,
    `--disable-extensions-except=${extensionPath}`,
  ],
});

Two practical consequences:

• The profile persists. Login sessions survive restarts because userDataDir is a mounted volume. First run on an empty profile triggers an auto-login flow (including TOTP-based 2FA via otplib); after that, cookies do the work.
• Profiles are platform-specific. A profile created on a Windows dev machine will not work inside a Linux container. Don't fight this — let auto-login mint a fresh profile per environment.

# Checks as configuration, not code

Each monitored page is declared in YAML: a URL, a list of selectors with descriptions, and an optional chainAfter edge. Per-integration files (config/asana.yaml, config/everhour.yaml) are merged at startup and validated for id uniqueness:

checks:
  - id: asana-board-view-everhour
    name: "Board view"
    group: Asana
    url: https://app.asana.com/1/.../board/...
    chainAfter: asana-list-view-everhour
    waitAfterNavigationMs: 3000
    selectors:
      - selector: ".EverhourTimerButton"
        description: "Timer button on card"
        minCount: 1
      - selector: "text/Estimated time"
        description: "Estimate badge"

chainAfter exists because every check runs on one shared tab. Parallel navigation on a single page is a URL race; a sequential chain turns the whole cycle into a deterministic walk: extension popup → list view → board view → task details → home → my tasks. A failing check never blocks the chain — its dependents still run.

The check itself is dumb on purpose: navigate, wait, assert each selector exists (respecting minCount), record PASS/FAIL per control. All the intelligence lives in what happens around it.

# Error taxonomy drives retry policy

The single most important design decision: not every failure is retried the same way. Every exception is classified into one of seven types, and the type determines the recovery strategy:

Two rules deserve emphasis.

SessionLost does not consume a retry attempt. An expired session is not a check failure — it is an environmental condition. The retry loop calls an onSessionLost callback (which re-authenticates), then tries again on the same attempt number. A separate counter caps this at 3 re-logins per check so a broken login flow cannot spin forever.

BrowserCrash gets no retry. Retrying against a dead browser process is meaningless; the only correct response is a full browser restart, after which the next cycle proceeds normally.

The retry budget itself is small — 3 attempts, 30-second base delay, doubling per attempt for network-class errors. A check that recovers on attempt 2 or 3 is logged as flaky, which feeds into alerting (more on that below):

for (let attempt = 0; attempt <= cfg.maxAttempts; attempt++) {
  try {
    const result = await runCheck(page, check);
    if (result.result === 'PASS') {
      if (attempt > 0) logger.warn(`Check ${check.id} recovered after ${attempt} retry(s) (flaky)`);
      return { result, retryCount: attempt, /* ... */ };
    }
    // FAIL: fall through to retry logic
  } catch (err) {
    if (err instanceof MonitorError && err.type === 'SessionLost') {
      if (++sessionLostCount > SESSION_LOST_MAX_RETRIES) throw err;
      await onSessionLost();          // re-login
      continue;                        // same attempt number
    }
    if (err instanceof MonitorError && err.type === 'BrowserCrash') throw err; // no retry
    // ...
  }
  await sleep(calculateDelay(baseDelaySec, exponential, attempt));
}

There is also a proactive layer: every 5 minutes, registered auth providers check isAuthenticated() and re-login before the next check cycle if the session expired. Most session losses are healed before any check notices.

# Who watches the watcher

A monitoring system that hangs silently is worse than no monitoring — it gives you false confidence. Three mechanisms keep the monitor itself honest:

Heartbeat + watchdog process. The main process rewrites heartbeat.json after every cycle. A separate, deliberately tiny watchdog process reads it every 60 seconds; if the heartbeat is older than 90 seconds, it kills and respawns the main process. The watchdog has no browser, no network, no dependencies — it is too simple to hang the same way the main process can.

Memory guard. Long-lived Chromium leaks. The monitor polls the browser's RSS (tasklist on Windows, ps on Linux) and restarts Chrome when it crosses a configurable limit (2 GB by default).

Scheduled daily restart. Even below the memory limit, the browser gets a clean restart every night at 03:00. Extensions are re-injected via --load-extension on every launch, so a restart also picks up any extension files updated on the mounted volume.

The combination means the system recovers from hangs, leaks, and crashes without human involvement — and every recovery path is exercised regularly enough that you trust it when it matters.

# Alerting without alert fatigue

A monitor that pages you for every transient blip gets muted within a week. The alerting pipeline has three filters:

Flaky ≠ broken. A check that failed but recovered on retry is recorded as FLAKY in the dashboard but does not alert. Only stable failures — still failing after the full retry budget — trigger Slack.

Deduplication with a signature. Repeat alerts for the same failure are suppressed for 30 minutes. The dedup key is not just the check id: it is a signature encoding which checks failed and how badly (:crit for browser crashes, :skip for auth-skipped). If the failure picture changes — a second page type starts failing, or a FAIL escalates to a crash — the signature changes and the alert goes through immediately, because this is genuinely new information.

Quiet hours with a digest, not a black hole. Between 21:00 and 09:00 (configured timezone), Slack alerts are suppressed — but written to an alert_queue table in SQLite. In the morning you get a digest of everything that happened overnight. Suppression never means loss.

Every alert carries the failure context: which controls are missing, the error details, and — the best part — screenshots.

# Pixel-diff screenshots: show, don't tell

A text alert saying "Timer button not found on Board view" forces you to open the app and hunt. A screenshot shows you instantly. The system goes one step further and borrows from visual regression testing:

• On PASS, the highlighted screenshot is written to {checkId}-baseline.png. Every green run refreshes the last-known-good reference.
• On FAIL, the screenshot is written to a timestamped file, then pixel-diffed against the baseline using pixelmatch. Changed pixels are highlighted into {checkId}-diff.png.

// PASS → shot becomes the new last-known-good baseline
// FAIL → timestamped live shot + pixelmatch diff vs baseline
const sc = await takeHighlightedScreenshot(page, check.id, controls, mode);

The Slack alert then carries three links: 📸 live screenshot, 📋 baseline, 🔍 diff. When Asana ships a UI update and half the controls vanish, the diff shows you the redesigned toolbar in one glance — no dashboard visit required. When the diff is empty but selectors fail, you know the extension stopped injecting rather than the page changing. That distinction alone cuts triage time from minutes to seconds.

Everything is persisted in SQLite (check_history with per-control JSON detail, screenshot paths, retry counts, durations), with 30-day retention for detail rows and 90-day aggregates for trend reporting.

# Lessons from running it 24/7

A few things only production time teaches you:

1. Monitoring code is production code. It needs its own reliability engineering — watchdogs, rotation, retention, restarts. Budget for it.
2. Classify before you retry. A single "retry 3 times on error" policy would have masked session expirations and hammered a dead browser. The taxonomy is the feature.
3. ERR_ABORTED is not an error. Single-page apps abort navigations during redirects all the time. A safeGoto() wrapper that swallows this specific case eliminates an entire class of false alarms.
4. Extension service workers are flaky to detect. waitForEvent('serviceworker') occasionally times out while content scripts work perfectly. Don't gate checks on the SW; gate them on what the user sees.
5. Flaky tracking pays off. The retry-recovery log turned out to be an early-warning system: a control that goes flaky before going stable-fail is usually a host app A/B test rolling out.
6. Give alerts a digest mode. Quiet hours that queue instead of drop keep nights silent and mornings informative.

# What to do next

If you maintain a browser extension, an integration, or any UI that renders inside a third-party product:

1. Pick the 3–5 user-facing controls that define "the integration works."
2. Write one YAML-style check per page type with those selectors.
3. Classify your failures before writing a single retry.
4. Add a screenshot pipeline — first for failures, then a baseline/diff pair.
5. Put a heartbeat on the monitor itself from day one.

The full system — checker, retry engine, Slack alerts, dashboard, watchdog, Docker setup — is TypeScript with Playwright, Express, and SQLite, and the architecture is integration-agnostic: adding Jira or Trello monitoring means writing one auth provider and one YAML file.

Written by Evgeny Pershukov. Follow me on LinkedIn or GitHub for more notes on AI Test Engineering.