Scraper
Scraper
Scraper collects products from online stores into the Catalogue. You list the stores to collect from, run them by hand or on a schedule, and see what each run found (module scraper).
In the sidebar under Commerce → Scraper. Needs a fleet machine that can run the scrape engine (see Devices).
Scraper holds no product data itself. It keeps the stores, the runs and the problems each run met. The products go to the catalogue, so a crawl that goes wrong cannot damage what was already collected.
What you see
A list of sources, ten at a time. Each row shows:
- the store's name and address, with the last error under it if the last run failed;
- the platform the store is built on (Shopify, WooCommerce and so on), detected on the first run, or "not yet detected";
- the status: active, paused or failing;
- the last run: queued, running, failed or cancelled, or for a finished run how many products it extracted and how many were new, changed or failed.
The page refreshes itself every ten seconds, because a run finishes in the background rather than while you wait. With no sources yet, the page says to add a store and run it.
What you can do
Add a store
- Open Commerce → Scraper and choose Add a store.
- Name: what to call it, such as "Acme Supplies".
- Address: the storefront, a public
https://address. - Only addresses matching (optional): a part of the address that product pages share, such as
/product/. - Stop after: the most products to collect in one run. It starts at 500.
- Schedule (optional): a cron expression in UTC, such as
0 3 * * *. Leave it empty to run the store only by hand. - Needs a real browser: tick it for a shop that draws its products with JavaScript. It is slower, so it is off by default.
- Save. Adding a store does not start a scrape.
Edit on a row changes these settings later. Delete removes a store for good, with its run history.
Run a store
- Choose Run on the store's row. You need
scraper:run. - The run is queued on a fleet machine and the button reads Running…. A store with hundreds of products takes minutes.
- When the run finishes, its products appear in Catalogue and the row shows the counts.
Only one run per store can be in progress at a time. With no machine able to run the engine, the run is refused with a message saying what to install.
Fix a failing store
- A store that fails three runs in a row turns failing and its schedule stops starting it.
- Read the error under its name, fix the cause, and choose Edit source.
- Setting the status back to active (through the API or an agent) clears the failure count and lets the schedule run it again. Paused stops the schedule without deleting anything.
Stop waiting on a run
Cancelling marks a queued or running scrape cancelled so it no longer counts as in progress. It does not reach into the machine: a crawl already under way finishes there and its result is not recorded. Cancelling is done through the API or by an agent.
Choose where a scrape runs
A scrape runs on any fleet machine enrolled with the scrape engine. To pin a source to one machine, or to point it at an engine installed somewhere else, set its device, engine path or Python path through the API. The page does not offer these fields, and changing them needs scraper:engine.
What a run retires
A run marks products gone from the catalogue only when it saw the whole store: it was not capped by Stop after, met no problems and had nothing rejected. A partial crawl never removes products.
Who can do what
| Permission | Label | Lets you |
|---|---|---|
scraper:read | View sources and run history | Open Scraper and see sources, runs and their problems. Agents need it for scraper.sources and scraper.runs. |
scraper:write | Add and edit sources | Add, edit and delete stores. Agents need it for scraper.createSource, scraper.updateSource and scraper.deleteSource. |
scraper:engine | Choose the program a scrape runs, and the fleet machine it runs on | Set a source's device, engine path or Python path. scraper:write alone is refused for these. |
scraper:run | Start a scrape | Choose Run, cancel a run, and give a source a schedule, resume a scheduled source or change its address, because each of those starts scrapes. Agents need it for scraper.start and scraper.cancel. |
Roles and how permissions are granted are explained in People, roles and permissions.
What runs on its own
| Job | When | What it does |
|---|---|---|
collect | Every 2 minutes (*/2 * * * *) | Looks at runs waiting on a fleet machine. For each one that finished, it hands the products to the catalogue, records any problems, marks products gone when the run saw the whole store, and updates the source. A run whose job failed is marked failed and counts towards the source's failures. |
schedule | Every 15 minutes (*/15 * * * *) | Starts every active source that has a schedule and is due. A source is due when its schedule's interval has passed since its last run. Common shapes (every N minutes, every N hours, once a day) are read; anything else is treated as hourly. |
When a run ends, well or badly, the module announces it (scraper.run.finished) with the run, the source and the counts.
For agents
An agent can list stores and runs, start and cancel scrapes, and add, change or delete stores.
Read:
scraper.sources: the stores set up for scraping, with their platform and health.scraper.runs: recent runs and what each collected.
Change:
scraper.start: start a scrape of one store. It returns a run id at once; results reach the catalogue when the crawl finishes.scraper.cancel: mark a queued or running scrape cancelled.scraper.createSource: add a store. Giving it a schedule also needsscraper:run.scraper.updateSource: change a store. Setting a schedule, resuming a scheduled store or changing its address also needsscraper:run.scraper.deleteSource: delete a store for good, with its run history. This is a destructive act.
Starting a scrape is treated as an outside act, because it sends requests to someone else's store. Whether an agent's change goes ahead or waits for a person follows its approval policy (Agents and approvals).
Over MCP the same tools are named scraper__start and so on (API keys and MCP access).
Related
- Commerce
- Catalogue: where the products go.
- Devices: the fleet machines that run scrapes.
- Agents and approvals