Algolia Crawler API

Name: Algolia Crawler API
Brand: Algolia Crawler API
Availability: InStock

★ Only Publicly Available OpenAPI DocumentDeveloper ToolsMonitoring Observabilitybasic20 Endpoints

Install Jentic One Beta

Connect the Algolia Crawler API to your agent

Jentic One is a self-hosted execution layer for AI agents. It lets your agent call the Algolia Crawler API, or any other public or private API you need. You set the rules, the agent never sees your credentials, and every call is logged.

Two steps, two machines. Install the instance in a safe environment, then register your agent from wherever it runs.

Step 1: Jentic One Host machine

# On the machine that will host your Jentic One instance:
curl -fsSL https://raw.githubusercontent.com/jentic/jentic-one/main/tools/install.sh | sh

Step 2: Agent machine

# On the machine where your agent runs (keep this separate from the instance):
curl -fsSL https://raw.githubusercontent.com/jentic/jentic-one/main/tools/install.sh | sh
jentic register       # connects your agent to your Jentic One instance

Jentic One is in public beta. The setup above keeps your agent separate from the instance, which is what you want before using real credentials: an agent running as the same OS user as Jentic One can read its stored keys directly. Just evaluating? A single local install is fine to start. See the secure deployment guide for the tiers.

Capabilities

What an agent can do with Algolia Crawler API.

Create a new Algolia crawler with POST /1/crawlers and update its configuration with PATCH /1/crawlers/{id}/config

Trigger a full reindex with POST /1/crawlers/{id}/reindex or pause and resume crawlers

Test crawl a single URL via POST /1/crawlers/{id}/test to validate extractors before a full run

GET STARTED

Start building with Algolia Crawler API

Explore with Jentic

View OpenAPI Document

For Agents

Drive Algolia's hosted web crawler programmatically: create crawlers, run and pause them, test URL extraction, and inspect crawl runs across 20 endpoints.

Use for: I need to create a new Algolia web crawler for a documentation site, Trigger a fresh reindex on an existing Algolia crawler, Pause an Algolia crawler before a major site migration, Test extraction on a single URL before kicking off a full crawl

Not supported: Does not serve search queries, ingest direct records, or manage A/B tests — use for managing Algolia hosted web crawlers, runs, and configurations only.

Jentic publishes the only available OpenAPI specification for the Algolia Crawler API, keeping it validated and agent-ready. The Crawler API drives Algolia's hosted web crawler — programmatically creating crawlers, updating their configuration, starting and pausing crawls, testing URL extraction, listing runs, downloading run logs, and managing registered domains. It is distinct from the standard Algolia search and ingestion APIs and uses its own credential pair. Authentication is HTTP basic auth with the Crawler user id and Crawler API key.

Use Cases

Patterns agents use Algolia Crawler API for, with concrete tasks.

★ Documentation site crawler bootstrapping

A docs platform team creates a new Algolia crawler for a freshly launched documentation portal by calling POST /1/crawlers with the start URLs and extraction config, validates extraction with POST /1/crawlers/{id}/test on a sample URL, and then kicks off the first full crawl with POST /1/crawlers/{id}/reindex. The same workflow can be re-run when site structure changes substantially.

POST /1/crawlers with the start URLs, POST /1/crawlers/{id}/test against a representative URL, then POST /1/crawlers/{id}/reindex

On-demand recrawl after a publish event

A CMS publishes new articles and triggers an Algolia recrawl by calling POST /1/crawlers/{id}/urls/crawl with the affected URLs. This avoids waiting for the next scheduled reindex and keeps Algolia search results fresh without re-crawling the entire site.

POST /1/crawlers/{id}/urls/crawl with the list of newly published URLs and check the response task id with GET /1/crawlers/{id}/tasks/{taskID}

Crawler health monitoring

An ops dashboard polls GET /1/crawlers/{id}/stats/urls and GET /1/crawlers/{id}/crawl_runs to display recent run durations, success rates, and URL counts. When a run fails, the dashboard fetches the log file via GET /1/crawlers/{id}/{logId}/download and surfaces the relevant error lines for triage.

GET /1/crawlers/{id}/crawl_runs, render the latest runs, and pull GET /1/crawlers/{id}/{logId}/download for any failing run id

Agent-driven crawler configuration through Jentic

A Jentic agent receives an intent like 'add /blog to the Algolia crawler' and chains GET /1/crawlers/{id}/config, PATCH /1/crawlers/{id}/config with the updated URL patterns, and then POST /1/crawlers/{id}/reindex. The agent never sees the basic auth credentials — Jentic injects them at execution time.

Search Jentic for 'update algolia crawler config', execute PATCH /1/crawlers/{id}/config with the new patterns, then POST /1/crawlers/{id}/reindex

Key Endpoints

20 endpoints — jentic publishes the only available openapi specification for the algolia crawler api, keeping it validated and agent-ready.

METHOD

PATH

DESCRIPTION

POST

/1/crawlers

Create a crawler

POST

/1/crawlers/{id}/reindex

Start a fresh crawl

POST

/1/crawlers/{id}/test

Test crawl a single URL

POST

/1/crawlers/{id}/urls/crawl

Crawl specific URLs on demand

PATCH

/1/crawlers/{id}/config

Update crawler configuration

GET

/1/crawlers/{id}/crawl_runs

List crawler runs

GET

/1/crawlers/{id}/{logId}/download

Download a crawler run log

POST

/1/crawlers

Create a crawler

POST

/1/crawlers/{id}/reindex

Start a fresh crawl

POST

/1/crawlers/{id}/test

Test crawl a single URL

POST

/1/crawlers/{id}/urls/crawl

Crawl specific URLs on demand

PATCH

/1/crawlers/{id}/config

Update crawler configuration

Why Jentic?

Three things that make agents converge on Jentic-routed access.

Credential isolation

Your Algolia Crawler basic credential is stored once, encrypted, by your own Jentic One instance and injected at execution time. It never enters the agent's prompt, logs, or context.

Intent-based discovery

Agents search Jentic by intent such as 'reindex a crawler' or 'crawl a specific set of URLs', and Jentic returns the matching Algolia Crawler operation with its input schema so the agent calls the right endpoint without browsing the reference docs.

Time to first call

FAQs

Specific to using Algolia Crawler API through Jentic.

Why is there no official OpenAPI spec for the Algolia Crawler API?

Algolia does not publish a downstream-stable OpenAPI specification for direct agent use. Jentic generates and maintains this spec from Algolia's source bundles so AI agents and developers can call the Crawler API via structured tooling. It is validated against the live API and kept up to date. Get started at https://app.jentic.com/sign-up.

What authentication does the Algolia Crawler API use?

The Crawler API uses HTTP basic auth with a Crawler user id and Crawler API key — distinct from the standard Algolia application id and api key. Jentic stores the credential pair encrypted and builds the Authorization header at execution time so the secret never enters the agent's prompt.

Can I trigger an on-demand recrawl of specific URLs?

Yes. POST /1/crawlers/{id}/urls/crawl with a list of URLs. The endpoint returns a task id you can poll with GET /1/crawlers/{id}/tasks/{taskID} until the recrawl completes.

What are the rate limits for the Algolia Crawler API?

The OpenAPI spec does not publish per-endpoint rate limits for the Crawler API. Algolia enforces them server-side; respect 4xx responses and back off when triggering rapid reindex or test calls in succession.

How do I update an Algolia crawler's configuration through Jentic?

Search Jentic for 'update algolia crawler config', load the schema for PATCH /1/crawlers/{id}/config, and execute it with the updated URL patterns or extractor settings. Then POST /1/crawlers/{id}/reindex to apply the change to live results.

Can I download the log for a specific crawler run?

Yes. GET /1/crawlers/{id}/{logId}/download returns the run log file. Pair this with GET /1/crawlers/{id}/crawl_runs to find the run id you need before downloading.

Algolia Crawler API

Connect the Algolia Crawler API to your agent

Capabilities

Start building with Algolia Crawler API

Use Cases

Key Endpoints

Why Jentic?

Related APIs

FAQs