Skip to main content
POST
web.robots.parse

Authorizations

Authorization
string
header
required

API key: sk_live_…

Headers

Idempotency-Key
string

Optional. A retry with the same key returns the stored result.

Maximum string length: 255
Pramaan-Max-Price-Micro
string

Optional. The most this request may be priced at, in micro-USD (a whole number above 0). It replaces the capability's own price ceiling for this request, lower or higher. Sources that could cost more are left out; when none is left the answer is 400 with the lowest limit that would work.

Pattern: ^[1-9]\d{0,14}$

Body

application/json
url
string
required

Address of the robots.

Response

The result

url
string
required

Url link.

rules
object[]

Rules.

totalAllowRules
number

Total allow rules: how many.

totalDisallowRules
number

Total disallow rules: how many.

totalRules
number

Total rules: how many.

blocksAllCrawlers
boolean

True when blocks all crawlers.

exists
boolean

True when exists.

sitemaps
string[]

Sitemaps.

userAgents
string[]

User agents.