Infrastructure

# As a condition of accessing this website, you agree to abide by the
# following content signals:

# (a)  If a content signal = yes, you may collect content for the
#      corresponding use.
# (b)  If a content signal = no, you may not collect content for the
#      corresponding use.
# (c)  If the website operator does not include a content signal for a
#      corresponding use, the website operator neither grants nor restricts
#      permission via content signal with respect to the corresponding use.

# The content signals and their meanings are:

# search:   building a search index and providing search results (e.g.,
#           returning hyperlinks and short excerpts from this website's
#           contents). Search does not include providing AI-generated search
#           summaries.
# ai-input: inputting content into one or more AI models (e.g., retrieval
#           augmented generation, grounding, or other real-time taking of
#           content for generative AI search answers).
# ai-train: training or fine-tuning AI models.

# ANY RESTRICTIONS EXPRESSED VIA CONTENT SIGNALS ARE EXPRESS RESERVATIONS OF
# RIGHTS UNDER ARTICLE 4 OF THE EUROPEAN UNION DIRECTIVE 2019/790 ON COPYRIGHT
# AND RELATED RIGHTS IN THE DIGITAL SINGLE MARKET.

User-agent: *
# `ai-input=yes` is deliberate and not the Content Signals default: an assistant
# grounding an answer in our docs is how agents find the APIs, which is the same
# job llms.txt, auth.md and the api-catalog below are doing. Saying no here would
# tell those assistants to ignore the very pages we publish for them. Training is
# a separate question — it returns nothing to us, so `ai-train=no` reserves it.
Content-Signal: search=yes, ai-input=yes, ai-train=no
Disallow: /register?*
Disallow: /login?*
Disallow: /developers/brand-api?*
Disallow: /developers/logo-api?*
Disallow: /share/*
Disallow: /onboarding
Disallow: /brandspace/*

# Pages first, then the explore directory and the blogs — crawlers read every
# Sitemap line, so the split only affects how coverage is reported, not what
# gets discovered.
Sitemap: https://brandfetch.com/sitemap.xml
Sitemap: https://brandfetch.com/explore/sitemap.xml
Sitemap: https://brandfetch.com/blog/sitemap.xml
Sitemap: https://brandfetch.com/developers/blog/sitemap.xml

# Agent capability manifest (ARD): https://brandfetch.com/.well-known/ard.json
# Every API, the MCP server and the agent docs below, each with the questions it
# answers, so a registry can index them semantically. Also served at the
# predecessor path, /.well-known/ai-catalog.json.
Agentmap: https://brandfetch.com/.well-known/ard.json

# AI agents and LLMs: https://brandfetch.com/llms.txt
# Machine-readable overview of the Brandfetch APIs, MCP server, pricing and FAQ.

# API discovery (RFC 9727): https://brandfetch.com/.well-known/api-catalog
# Every public API with its OpenAPI description, documentation and status.

# OAuth discovery (RFC 9728): https://brandfetch.com/.well-known/oauth-protected-resource
# Which authorization server issues Brandfetch access tokens, and the scope it grants.

# Agent authentication: https://brandfetch.com/auth.md
# How agents get and use Brandfetch API credentials.

# Agent onboarding: https://brandfetch.com/agent-onboarding/SKILL.md
# Pick an API, get credentials, call it — for AI agents.

# Agent Skills index: https://brandfetch.com/.well-known/agent-skills/index.json
# Every skill we publish, each with a SHA-256 digest to verify it against.

# Paying without an account: https://docs.brandfetch.com/agents/overview
# x402 and MPP per request, and standing access bought with one payment. The
# OpenAPI description at https://brandfetch.com/openapi.json states the terms.

# Content usage preferences: https://contentsignals.org