Start Crawl
Starts a new crawl. You receive a `id` to track the progress. The operation may take 1-10 mins depending upon the site and depth and pages parameters.
Dynamic pricing
(body.max_pages) ? body.max_pages * 0.01 : (body.max_depth) ? body.max_depth * 0.10 : 0.01
Example code
// Using @orth/sdk - The simplest way to call APIs
// Install: npm install @orth/sdk
import Orthogonal from "@orth/sdk";
const orthogonal = new Orthogonal({
apiKey: process.env.ORTHOGONAL_API_KEY,
});
const output = await orthogonal.run({
api: "olostep",
path: "/v1/crawls",
body: {
"start_url": "example",
"max_pages": 123
}
});
console.log(output);Body parameters
The starting point of the crawl.
Maximum number of pages to crawl. Recommended for most use cases like crawling an entire website.
URL path patterns to include in the crawl using glob syntax. Defaults to `/**` which includes all URLs. Use patterns like `/blog/**` to crawl specific sections (e.g., only blog pages), `/products/*.html` for product pages, or multiple patterns for different sections. Supports standard glob features like * (any characters) and ** (recursive matching).
URL path names in glob pattern to exclude. For example: `/careers/**`. Excluded URLs will supersede included URLs.
Maximum depth of the crawl. Useful to extract only up to n-degree of links.
Crawl first-degree external links.
Include subdomains of the website. `false` by default.
An optional search query to find specific links and also sort the results by relevance.
An optional number to only crawl the top N most relevant links on every page as per search query.
An optional POST request endpoint called when this crawl is completed. The body of the request will be same as the response of this [`v1/crawls/{crawl_id}`](./info#response-created) endpoint.
End the crawl after n seconds with the pages completed until then. May take ~10s extra from provided timeout.
Response