POST/api/crawl

Start Crawl

Start a website crawl from a seed URL with configurable depth and page limits.

ScrapeGraphAI·Dynamic pricing

Dynamic pricing

0.01 + ((includesField(body.formats, 'type', 'markdown') ? 0.005 : 0) + (includesField(body.formats, 'type', 'html') ? 0.005 : 0) + (includesField(body.formats, 'type', 'links') ? 0.005 : 0) + (includesField(body.formats, 'type', 'images') ? 0.005 : 0) + (includesField(body.formats, 'type', 'summary') ? 0.005 : 0) + (includesField(body.formats, 'type', 'screenshot') ? 0.01 : 0) + (includesField(body.formats, 'type', 'json') ? 0.025 : 0) + (includesField(body.formats, 'type', 'branding') ? 0.125 : 0) + (get(body, 'fetchConfig.stealth', 0) ? 0.025 : 0)) * (body.maxPages ? body.maxPages : 50)

Example code

// Using @orth/sdk - The simplest way to call APIs
// Install: npm install @orth/sdk
import Orthogonal from "@orth/sdk";

const orthogonal = new Orthogonal({
  apiKey: process.env.ORTHOGONAL_API_KEY,
});

const output = await orthogonal.run({
  api: "scrapegraphai",
  path: "/api/crawl",
  body: {
    "url": "example"
  }
});

console.log(output);

Body parameters

urlrequired

string

Seed URL to crawl

formats

array

Output formats

maxDepth

number

Max crawl depth (default 2)

maxPages

number

Max pages (1-1000, default 50)

maxLinksPerPage

number

Max links per page (default 10)

allowExternal

boolean

Follow external links (default false)

includePatterns

array

URL patterns to include

excludePatterns

array

URL patterns to exclude

contentTypes

array

Allowed content types

fetchConfig

object

Fetch options

Response

string

pages

array

total

number

status

string

finished

number