Just Scrape

# just-scrape CLI

Search, scrape, crawl, extract structured JSON, and monitor page changes using the just-scrape CLI.

Run `just-scrape --help` or `just-scrape <command> --help` for full option details.

If the task is to integrate ScrapeGraph AI into application code, add `SGAI_API_KEY` to a project, or choose endpoint usage in product code, inspect the project first and use the ScrapeGraph AI SDK/API docs directly instead of this CLI skill.

## Prerequisites

Must be installed and authenticated. Check with `just-scrape validate` and `just-scrape credits`.

```bash command -v just-scrape >/dev/null 2>&1 || npm install -g just-scrape@latest just-scrape validate just-scrape credits ```

- **API key**: Set `SGAI_API_KEY`, use a `.env` file, use `~/.scrapegraphai/config.json`, or complete the interactive prompt. - **Credits**: Remaining ScrapeGraph AI credits. Each operation consumes credits.

Before doing real work, verify the setup with one small request:

```bash mkdir -p .just-scrape just-scrape scrape "https://example.com" --json > .just-scrape/install-check.json ```

```bash just-scrape search "query" --num-results 3 --json > .just-scrape/search-check.json ```

## Workflow

Follow this escalation pattern:

1. **Search** - No specific URL yet. Find pages, answer questions, discover sources. 2. **Scrape** - Have a URL. Extract markdown, html, screenshots, links, images, summaries, or branding. 3. **Extract** - Need structured JSON from a known URL with an AI prompt and optional schema. 4. **Crawl** - Need bulk content from an entire site section. 5. **Monitor** - Need scheduled page-change tracking with optional webhook notifications.

| Need | Command | When | | --------------------------- | ---------- | ------------------------------------------ | | Find pages on a topic | `search` | No specific URL yet | | Get a page's content | `scrape` | Have a URL, need one or more page formats | | AI-powered data extraction | `extract` | Need structured data from a known URL | | Bulk extract a site section | `crawl` | Need many pages or docs sections | | Track changes over time | `monitor` | Need recurring scraping and webhooks | | Inspect prior requests | `history` | Need past request IDs, status, or payloads | | Check credit balance | `credits` | Need remaining API credits | | Validate API setup | `validate` | Need health check and API key validation |

For detailed command reference, run `just-scrape <command> --help`.

**Scrape vs extract:**

- Use `scrape` for raw page formats: `markdown`, `html`, `screenshot`, `branding`, `links`, `images`, `summary`. - Use `scrape -f json -p "<prompt>"` or `extract -p "<prompt>"` for AI-structured output. - Use `extract` when the task is only structured data. Use `scrape` when mixed formats are needed in one call.

**Avoid redundant fetches:**

- `search -p` can extract structured data from search results. Do not re-scrape those URLs unless results are incomplete. - `crawl` already fetches per-page formats. Do not re-scrape every crawled URL unless a second pass is required. - Check `.just-scrape/` for existing data before fetching again.

## Commands

### Search

```bash just-scrape search "query" just-scrape search "query" --num-results 10 just-scrape search "query" -p "Extract provider names and prices" just-scrape search "query" -p "Extract provider names and prices" --schema '<json-schema>' just-scrape search "query" --format html just-scrape search "query" --country us just-scrape search "query" --time-range past_week ```

Time ranges: `past_hour`, `past_24_hours`, `past_week`, `past_month`, `past_year`.

### Scrape

```bash just-scrape scrape "<url>" just-scrape scrape "<url>" -f markdown just-scrape scrape "<url>" -f html just-scrape scrape "<url>" -f markdown,html,links --json just-scrape scrape "<url>" -f screenshot just-scrape scrape "<url>" -f branding just-scrape scrape "<url>" -f summary just-scrape scrape "<url>" -f json -p "Extract all products" just-scrape scrape "<url>" -f json -p "Extract all products" --schema '<json-schema>' just-scrape scrape "<url>" --html-mode reader just-scrape scrape "<url>" --mode js --stealth --scrolls 5 just-scrape scrape "<url>" --country DE ```

Formats: `markdown`, `html`, `screenshot`, `branding`, `links`, `images`, `summary`, `json`.

### Extract

```bash just-scrape extract "<url>" -p "Extract product names and prices" just-scrape extract "<url>" -p "Extract headlines and dates" --schema '<json-schema>' just-scrape extract "<url>" -p "Extract visible items" --scrolls 5 just-scrape extract "<url>" -p "Extract account stats" --cookies "{\"session\":\"$SESSION_COOKIE\"}" --stealth just-scrape extract "<url>" -p "Extract table rows" --headers "{\"Authorization\":\"Bearer $API_TOKEN\"}" just-scrape extract "<url>" -p "Extract article data" --html-mode re