Skip to content

Crawl and audit#

The crawl is a read-only map of every non-blog page on the live site, broken down into its sections and modules. It needs no credentials. Every later decision cites it: the design rules, the generator's choices, the usage counts in the module catalog.

In Claude Code

The hubspot-audit skill of the bb-hubspot plugin covers the crawl, the design rules and the catalog. The crawler itself is a site tool: each site writes its own.

Run it#

python3 tools/crawl-site.py             # the site's own crawler: fetch the sitemap, cache every page, analyse
python3 tools/crawl-site.py --offline   # re-analyse the cache after a parser change

It writes reports/site-crawl.json (the dataset) and reports/site-crawl.md (the report). The page cache in reports/crawl-cache/ is git-ignored. Keep it, because the copy-doc exporter reads it too.

What it records#

For every page, and every section on it in order:

  • The band: the section's background class, plus the padding and background image set in the editor, read from the row CSS HubSpot generates.
  • Each module, identified by the root classes its module.html emits, with its theme, its layout classes, its style variables and its content counts (cards, logos, slides, services).
  • Column widths wherever modules share a row, so an 8/12 text panel next to a 4/12 stats rail shows up.
  • Header and footer variant, template, title, and whether the page has an H1.

The report groups pages by type. For each type it gives sections per page, which modules open and close the page, how often each module theme and band appears, and then every page section by section:

## listing-index (2 pages)

- sections per page: [3, 3]  ·  theme: [('main', 2)]  ·  header: [('light', 2)]
- modules: [('text-media', 4), ('listing-cards', 2)]
- module themes: [('light', 2), ('dark', 2)]  ·  section bgs: [('plain', 6)]
- opens with: [('text-media/light', 2)]  ·  closes with: [('text-media/dark', 2)]

Porting it to a new site#

Two parts of the crawler are specific to each site:

  1. The module matcher: the table that maps root CSS classes to module names. Build it from the theme's module.html files (the catalog generator records those same hooks).
  2. The page-type classifier: URL patterns plus the opening module. Expect to tune it until no page lands in other.

Lessons learned#

Check one measured value in a browser

The first crawl read HubSpot's mobile padding overrides instead of the desktop values, and every padding rule in the design rules was wrong until a draft review caught it. Before writing rules from the crawl, check one measured value against the rendered page.

  • Take screenshots alongside the crawl. Take one full-page screenshot per page type. The composition data shows what is on a page but not why it looks right. $BBH/scripts/shoot.py takes them in headless Chrome with its own temporary profile.
  • Rerun when a new archetype appears. The crawl is cheap. Rerun it whenever new page types go live, then update the rules.