Olostep + Apify Integration

Olostep is a Web search, scraping and crawling API — an API to search, extract and structure web data. This guide shows how to use Olostep with Apify Actors to build reliable web data pipelines end‑to‑end.

What you can build

Scrape Website

Extract content from any single URL in Markdown, HTML, JSON, or Text

Batch Scrape URLs

Process large lists of URLs in parallel with structured outputs

Create Crawl

Discover and scrape linked pages to build complete datasets

Create Map

Extract all URLs from a website (sitemap-like discovery)

AI-powered Answers

Ask questions and get structured JSON answers with sources

Quick start

1) Install Apify CLI

npm install -g apify-cli
apify --version

2) Get your Olostep API key

From the Olostep Dashboard → API Keys.

3) Run the Olostep Actor locally

cd olostep-tools/integrations/apify
apify run

Default local input file lives at: olostep-tools/integrations/apify/storage/key_value_stores/default/INPUT.json Example input:

{
  "operation": "scrape",
  "apiKey": "YOUR_OLostep_API_KEY",
  "url_to_scrape": "https://example.com",
  "formats": "markdown"
}

4) Deploy to Apify (cloud)

apify login
apify push

Then open Apify Console → Actors → run the actor with your desired input.

Run in Apify Console (step by step)

Open your Actor in Apify Console → Source → Input.
In the Manual tab you’ll see a visible “Olostep API Key” field. Paste your key from the Olostep Dashboard.
Choose an operation (defaults to “scrape”).
Fill the relevant fields (for “scrape”, set “URL to Scrape”).
Click Save → Start.
When the run finishes, open the Dataset tab to download results (JSON/CSV/Excel).

Notes:

For “URL to Scrape”, you can paste with or without scheme. If missing, the actor automatically prepends https://.
If a site is heavy in JavaScript and you see a timeout, set “Wait Before Scraping” to 2000–5000 ms and run again.

Available operations

Scrape Website

Extract content from a single URL. Great for page‑level automation.

constant

default:"scrape"

Must be “scrape”

string

required

Your Olostep API key (Bearer)

string

required

The URL to scrape (must include http:// or https://)

dropdown

default:"markdown"

One of: Markdown, HTML, JSON, Text

string

Optional country code (e.g., “US”, “GB”, “CA”)

integer

Optional wait time in ms for JavaScript rendering (0–10000)

string

Optional parser ID (e.g., “@olostep/amazon-product”)

Output fields:

id, url, status, formats
markdown_content / html_content / json_content / text_content
hosted URLs (if available), page metadata

Batch Scrape URLs

Process many URLs at once with consistent formatting and structure.

constant

default:"batch"

Must be “batch”

string

required

Your Olostep API key

text

required

JSON array of objects with url and optional custom_id
Example: [{"url":"https://example.com","custom_id":"site1"}]

dropdown

default:"markdown"

One of: Markdown, HTML, JSON, Text

string

Optional country code

integer

Optional wait time in ms for JS sites

string

Optional parser ID

Output fields:

batch_id, status, total_urls, created_at, formats, country, parser, urls[]

Create Crawl

Follow links and scrape multiple pages from a start URL.

constant

default:"crawl"

Must be “crawl”

string

required

Your Olostep API key

string

required

Starting URL for the crawl

integer

default:"10"

Max pages to crawl. Set to 1 to scrape only the start URL.

dropdown

default:"markdown"

One of: Markdown, HTML, JSON, Text

string

Optional country code

string

Optional parser ID

Output fields:

crawl_id, object, status, start_url, max_pages, created, formats

Create Map

Discover all URLs on a website and prepare for later batch scraping.

constant

default:"map"

Must be “map”

string

required

Your Olostep API key

string

required

The website to map

string

Optional query filter

integer

Limit number of URLs

string

Include glob(s), e.g. “/products/**”

string

Exclude glob(s), e.g. “/admin/**”

Output fields:

map_id, object, website_url, total_urls, urls[], search_query, top_n

Copy‑paste JSON examples (Console → Input → JSON)

Scrape

{
  "operation": "scrape",
  "apiKey": "YOUR_OLOSTEP_API_KEY",
  "url_to_scrape": "https://www.wikipedia.org",
  "formats": "markdown",
  "wait_before_scraping": 2000
}

Batch

{
  "operation": "batch",
  "apiKey": "YOUR_OLOSTEP_API_KEY",
  "batch_array": "[{\"url\":\"https://example.com\",\"custom_id\":\"site1\"},{\"url\":\"https://olostep.com\",\"custom_id\":\"site2\"}]",
  "formats": "json"
}

Crawl

{
  "operation": "crawl",
  "apiKey": "YOUR_OLOSTEP_API_KEY",
  "start_url": "https://docs.example.com",
  "max_pages": 50,
  "formats": "markdown"
}

Map

{
  "operation": "map",
  "apiKey": "YOUR_OLOSTEP_API_KEY",
  "website_url": "https://example.com",
  "include_patterns": "/blog/**",
  "top_n": 200
}

Answers

{
  "operation": "answers",
  "apiKey": "YOUR_OLOSTEP_API_KEY",
  "task": "What is the latest funding round of Olostep? Provide company, round, date, amount.",
  "json": "{\"company\":\"\",\"round\":\"\",\"date\":\"\",\"amount\":\"\"}"
}

Example workflows

Discover and Scrape Products

Create Map → include “/products/**”
Parse URLs → build batch array
Batch Scrape URLs → formats: JSON
Send to Google Sheets / Airtable

Daily Content Monitoring

Schedule actor (daily)
Scrape Website → formats: Markdown
Summarize with LLM
Notify on Slack

Competitor Knowledge Base

Create Crawl (blog/docs)
Store outputs in Notion
Refresh weekly with Schedule

Specialized parsers

Olostep supports parsers to structure data for popular sites.

Amazon Product

@olostep/amazon-product → title, price, rating, reviews, images, variants

Google Search

@olostep/google-search → results, titles, snippets, URLs

Google Maps

@olostep/google-maps → business info, reviews, ratings, location

More Parsers

Explore email extractors, social handle finders, calendar link extractors, and more

Best practices

Prefer batch for 3+ URLs

Faster, cheaper, easier to monitor and respect rate limits.

Use appropriate wait times

JS‑heavy sites: increase wait_before_scraping (e.g., 2000–5000ms).

Filter before scraping

Avoid unnecessary tasks — check changes first, keep deduplication state.

Store large content via hosted URLs

Use hosted outputs to bypass payload size limits in Apify flows.

Treat async operations as long‑running

Batch/Crawl/Map return IDs; retrieve later or chain with a delay.

Handle transient timeouts cleanly

If you see a 504 or transient timeout, the actor automatically retries once with a short wait time.
You can also set “Wait Before Scraping” to 2000–5000 ms for JS‑heavy pages.

Troubleshooting

Authentication failed

Check API key from dashboard
Remove trailing spaces
Re‑enter in Apify input form

Empty content

Increase wait time
Verify URL is public / not login‑gated
Try different output format

Rate limit exceeded

Space runs via schedule
Prefer batch for many URLs
Upgrade Olostep plan if needed

Blocked or dynamic sites

Try country parameter
Adjust wait and parser
Contact support for guidance

Pricing

Olostep charges by API usage (independent of Apify):

Scrapes → per scrape
Batches → per URL
Crawls → per page
Maps → per operation

See https://www.olostep.com/pricing.

Security

Your API key is sent as Bearer token at runtime.
Do not commit keys to version control; Apify stores inputs in Key‑Value Store.
In local development, keep keys in storage/key_value_stores/default/INPUT.json (gitignored).

Scrapes API

Extract LLM‑friendly Markdown, HTML, text or structured JSON from any URL.

Batches API

Process up to 10k URLs concurrently and retrieve results later.

Crawls API

Recursively discover and scrape a site’s content.

Maps API

Get all URLs on a website to prepare batch scrapes.

Support

Apify Website

Apify platform

Apify Docs

Apify platform & SDK docs

Documentation

Complete API docs

Support Email

info@olostep.com

​What you can build

Scrape Website

Batch Scrape URLs

Create Crawl

Create Map

AI-powered Answers

​Quick start

​1) Install Apify CLI

​2) Get your Olostep API key

​3) Run the Olostep Actor locally

​4) Deploy to Apify (cloud)

​Run in Apify Console (step by step)

​Available operations

​Scrape Website

​Batch Scrape URLs

​Create Crawl

​Create Map

​Copy‑paste JSON examples (Console → Input → JSON)

​Scrape

​Batch

​Crawl

​Map

​Answers

​Example workflows

​Specialized parsers

Amazon Product

Google Search

Google Maps

More Parsers

​Best practices

​Troubleshooting

​Pricing

​Security

​Related resources

Scrapes API

Batches API

Crawls API

Maps API

​Support

Apify Website

Apify Docs

Documentation

Support Email

What you can build

Quick start

1) Install Apify CLI

2) Get your Olostep API key

3) Run the Olostep Actor locally

4) Deploy to Apify (cloud)

Run in Apify Console (step by step)

Available operations

Scrape Website

Batch Scrape URLs

Create Crawl

Create Map

Copy‑paste JSON examples (Console → Input → JSON)

Scrape

Batch

Crawl

Map

Answers

Example workflows

Specialized parsers

Best practices

Troubleshooting

Pricing

Security

Related resources

Support