API Documentation - Central Bot Database

Getting Started

The Central Bot Database API provides programmatic access to our comprehensive bot database. All endpoints return JSON data and require no authentication.

Base URL

https://central-bot-database.pages.dev/api

Rate Limits

Currently no rate limits. Please use responsibly.

Understanding the Data

What is "Operator"?

The operator field represents a bot's category or classification. It can be either:

Organization name - The company running the bot (e.g., "Google", "Amazon", "Ahrefs")
Functional category - What type of bot it is (e.g., "Search Engines", "Monitoring & Analytics", "AI Crawlers")

Data Sources Priority:

Cloudflare category (e.g., "Monitoring & Analytics")
ai.robots.txt function field
ai.robots.txt operator/company/owner fields
Defaults to "Other" if none specified

Examples:

Googlebot → operator: "Search Engines"
360Monitoring → operator: "Monitoring & Analytics"
AmazonBot → operator: "Amazon"

Response Data Structure

All endpoints that return bot data include these fields:

{
  "user_agent": "BotName/1.0",
  "operator": "Category or Company Name",
  "purpose": "AI-generated description",
  "impact_of_blocking": "What happens if blocked",
  "categories": {
    "ecommerce": "beneficial|neutral|harmful|not_applicable",
    ...
  },
  "sources": ["ai-robots-txt", "cloudflare-radar", "manual"],
  "raw_data": {
    "ip_ranges": ["192.0.2.0/24"],
    "asn": "AS64496",
    "cf_traffic_percentage": "0.123"
  },
  "last_updated": "2026-02-01T12:00:00Z"
}

Endpoints

GET Search Bots

/api/search

Search for bots by user agent, operator, or purpose.

Query Parameters

q string (required)

Search query - matches user agent, operator, or purpose

limit number (optional)

Maximum results to return (default: 10, max: 100)

Example Request

GET /api/search?q=googlebot&limit=5

Example Response

{
  "results": [
    {
      "user_agent": "Googlebot",
      "operator": "Search Engines",
      "purpose": "...",
      "sources": ["ai-robots-txt"]
    }
  ],
  "count": 1,
  "query": "googlebot"
}

GET Get Recommendations

/api/recommend

Get bot recommendations for a specific site category.

Query Parameters

category string (required)

Site category: ecommerce, news, media, blog, saas, corporate, documentation, social, portfolio, government

rating string (optional)

Filter by rating: beneficial, neutral, harmful, not_applicable

Example Request

GET /api/recommend?category=ecommerce&rating=beneficial

Example Response

{
  "category": "ecommerce",
  "rating": "beneficial",
  "bots": [...],
  "count": 42
}

GET Get Bot Details

/api/details

Get detailed information about a specific bot.

Query Parameters

bot string (required)

Bot user agent (case-insensitive partial match)

Example Request

GET /api/details?bot=Googlebot

Example Response

{
  "user_agent": "Googlebot",
  "operator": "Search Engines",
  "purpose": "...",
  "impact_of_blocking": "...",
  "categories": { ... },
  "sources": ["ai-robots-txt"],
  "raw_data": { ... },
  "last_updated": "2026-02-01T12:00:00Z"
}

GET Generate robots.txt

/api/robots

Generate a robots.txt file based on category recommendations.

Query Parameters

category string (required)

Site category

block string (optional)

Which bots to block: harmful (default), harmful+neutral

Example Request

GET /api/robots?category=ecommerce&block=harmful

Example Response

# Generated robots.txt for ecommerce sites
# Block harmful bots

User-agent: BadBot
Disallow: /

User-agent: ScraperBot
Disallow: /

GET Get Statistics

/api/stats

Get database statistics.

Example Request

GET /api/stats

Example Response

{
  "total_bots": 368,
  "operators": 45,
  "sources": {
    "ai-robots-txt": 108,
    "cloudflare-radar": 260,
    "manual": 15
  },
  "last_updated": "2026-02-01T12:00:00Z"
}

Client Libraries

Currently, no official client libraries are available. You can use any HTTP client to make requests.

JavaScript/TypeScript Example

// Search for bots
const response = await fetch('/api/search?q=googlebot');
const data = await response.json();
console.log(data.results);

// Get recommendations
const recs = await fetch('/api/recommend?category=ecommerce&rating=beneficial');
const bots = await recs.json();
console.log(bots.bots);

Python Example

import requests

# Search for bots
response = requests.get('https://central-bot-database.pages.dev/api/search?q=googlebot')
data = response.json()
print(data['results'])

# Get recommendations
response = requests.get('https://central-bot-database.pages.dev/api/recommend?category=ecommerce&rating=beneficial')
bots = response.json()
print(bots['bots'])