The Problem
Web scraping means fighting every site individually: CAPTCHAs block automated browsers, markup noise buries the signal, and writing per-site selectors or regexes doesn't scale when you just need a specific slice of a page.
The Solution
A two-step natural-language scraper. Pages render inside Bright Data's Scraping Browser — remote Chromium that auto-solves CAPTCHAs over CDP before capture — then BeautifulSoup strips scripts and styles into clean text, chunked to fit a local model's context. The user describes what they want in one sentence; a LangChain LCEL chain runs each chunk through Ollama-hosted Llama 3 under a strict output contract that returns only matching data (empty string if nothing matches). Everything parses on-device: scraped content never leaves the machine, with zero per-request API cost.
How the workflow runs
01 · Scrape
The URL loads in Bright Data's remote browser; CAPTCHAs are detected and solved before the rendered DOM is captured.
02 · Clean
BeautifulSoup extracts the body and removes script/style tags, collapsing markup into line-normalised plain text.
03 · Chunk
Long pages split into ~6,000-character windows so each fits the local model's context without truncation.
04 · Parse
Each chunk flows through a strict extraction prompt describing the user's request; progress prints per batch.
05 · Join
Per-chunk results concatenate into one answer containing only the requested data — empty string when nothing matches.
AI layer
- Local LLM parsing
- Ollama + Llama 3 keeps inference on-device — sensitive scraped content never leaves the machine.
- Strict extraction prompt
- Output contract forbids commentary: only explicitly requested data, empty string as honest fallback.
- Chunk-aware chain
- LCEL template applied per window lets arbitrarily long pages be parsed coherently.
Automation layer
- CAPTCHA-proof intake
- Bright Data's Scraping Browser handles challenges, proxies, and rendering — no per-site workarounds.
- Zero-selector workflow
- Plain-English descriptions replace XPath/CSS maintenance across sites.
- Deterministic preprocessing
- Cleaning and chunking are pure functions, testable independently of browser and model.
Technical challenges
Modern sites block headless browsers with CAPTCHAs and bot detection before any HTML is served.
Delegated rendering to Bright Data's managed browser, waiting on Captcha.waitForSolve over CDP before reading page_source.
Raw HTML is mostly boilerplate — feeding full pages to an LLM wastes context on scripts and styles.
BeautifulSoup body-extraction plus tag stripping reduces pages to dense text before any model call.
Long pages exceed the local model's context window, truncating mid-data.
Fixed-size chunking (~6k chars) with sequential parsing and joined results keeps long-page extraction lossless.
Outcome
- Any public page becomes queryable in one sentence — no selectors, no regex, no per-site code.
- Scraped data stays fully local: browser-side rendering handled by Bright Data, parsing by an on-device model.
- The same three-stage pipeline (render → clean → parse) works unchanged across arbitrary sites.
Lessons learned
- “Most scraping complexity lives at the edge (CAPTCHAs, rendering) — delegating it turns scraping into a boring input function.”
- “LLMs excel at sifting, not fetching: clean deterministic preprocessing plus strict prompts beats clever prompting on raw HTML.”
- “Local models make scraping workflows viable for sensitive data — privacy and cost constraints shape architecture early.”