Setup & Installation
Or with the ClawHub CLI, for registry-managed skill folders outside a full OpenClaw workspace:
What This Skill Does
A 5-stage web scraping pipeline that detects news articles, extracts content using a cascade of strategies (static HTTP, Playwright, Scrapy), removes boilerplate, and pulls structured metadata. Handles JS-rendered pages, soft paywalls, bulk crawls, and encoding normalization. Optionally extracts named entities via LLM using clean text only, never raw HTML.
The cascade fallback (static HTTP first, Playwright only when needed) avoids headless browser overhead for simple pages while still handling JS-rendered SPAs that single-strategy scrapers fail on.
When to use it
- Extracting full article text and metadata from a news site
- Bulk crawling all articles on a domain into structured JSON
- Scraping content from a React or Next.js SPA that requires JS rendering
- Extracting people, organizations, and events mentioned in news articles
- Checking which parts of a paywalled article are publicly accessible
Example Workflow
Here's how your AI assistant might use this skill in practice.
User asks: Scrape this news article URL and extract all named entities
- 1Analyzes URL pattern and page HTML to confirm it is a news article (Stage 1)
- 2Fetches page with static HTTP; detects JS rendering need and escalates to Playwright (Stage 2)
- 3Removes boilerplate with trafilatura and normalizes encoding (Stage 3)
- 4Extracts structured metadata: title, author, publication date, paywall status (Stage 4)
- 5Sends clean text to LLM via OpenRouter and parses entity JSON response (Stage 5)
JSON file with article text, metadata fields, and structured entities grouped by people, organizations, locations, and events
Requirements
Accounts, API keys, or tools you or your AI assistant may need to set up while using this skill.