GET/web/scrape/sitemap

Crawl website sitemap

Crawls the sitemap of the given domain and returns all discovered page URLs. Supports sitemap index files (recursive), parallel fetching with concurrency control, deduplication, and filters out non-page resources (images, PDFs, etc.).

Context.dev·$0.03 per call

Example code

// Using @orth/sdk - The simplest way to call APIs
// Install: npm install @orth/sdk
import Orthogonal from "@orth/sdk";

const orthogonal = new Orthogonal({
  apiKey: process.env.ORTHOGONAL_API_KEY,
});

const output = await orthogonal.run({
  api: "context-dev",
  path: "/web/scrape/sitemap",
  query: {
    "domain": "example"
  }
});

console.log(output);

Query parameters

domainrequired

string

Domain name to crawl sitemaps for (e.g., 'example.com'). The domain will be automatically normalized and validated.

maxLinks

integer

Maximum number of links to return from the sitemap crawl. Defaults to 10,000. Minimum is 1, maximum is 100,000.

Response

meta.errors

number

meta.sitemapsFetched

number

meta.sitemapsSkipped

number

meta.sitemapsDiscovered

number

urls.0

string

domain

string

success

boolean

key_metadata.credits_consumed

number

key_metadata.credits_remaining

number