Infrastructure
· 9 checks — DNS, redirects, IPv6, crawlability, URL variants, and domain intelligence rolled into one auditable list.DCrawlabilityActionrobots.txt present, no sitemapFIX
Disallow: / for all user-agents prevents search engines from indexing any page. This will remove the site from search results.
Disallow: / in robots.txt blocks every search crawler — the site becomes invisible in organic search.
Learn more ▾ ▴
Common deployment mistake: a staging robots.txt with `User-agent: * / Disallow: /` ships to prod. The site falls out of search results within days. Verify your robots.txt is the production-intended version. If this is intentional (private site), no action needed.
Source: Google Search Central
A sitemap helps search engines discover and index your pages more efficiently.
No sitemap.xml — Google relies on crawl-graph discovery alone, slowing indexing of deep or fresh URLs.
Learn more ▾ ▴
A sitemap accelerates Google's discovery of new and updated content. Most CMSes auto-generate one; static-site frameworks need a build-step plugin. Reference it from robots.txt and submit in Search Console to confirm Google can fetch it.
Source: sitemaps.org / Google Search Central
# If you would like to crawl CourtListener, please contact us. We also have an
# extensive REST API and provide bulk data.
# Google, AOL, Bing, Yahoo!, DuckDuckGo
# These support meta robots and x-robots-tag or are otherwise harmless (DDG)
# Crawl everything except a few explicit blocks
User-agent: Googlebot
# Blocking robots.txt makes it not get a search snippet in Google results.
Disallow: /robots.txt
Disallow: /assets/
Disallow: /api/rest/v1/
Disallow: /api/rest/v2/
Disallow: /api/rest/v3/
User-agent: bingbot
Disallow: /robots.txt
Disallow: /assets/
Disallow: /api/rest/v1/
Disallow: /api/rest/v2/
Disallow: /api/rest/v3/
User-agent: Slurp
Disallow: /robots.txt
Disallow: /assets/
Disallow: /api/rest/v1/
Disallow: /api/rest/v2/
Disallow: /api/rest/v3/
User-agent: DuckDuckBot
Disallow: /robots.txt
Disallow: /assets/
Disallow: /api/rest/v1/
Disallow: /api/rest/v2/
Disallow: /api/rest/v3/
# Needed for open graph crawling
User-agent: twitterbot
Disallow: /robots.txt
Disallow: /assets/
Disallow: /api/rest/v1/
Disallow: /api/rest/v2/
Disallow: /api/rest/v3/
User-agent: facebookexternalhit
Disallow: /robots.txt
Disallow: /assets/
Disallow: /api/rest/v1/
Disallow: /api/rest/v2/
Disallow: /api/rest/v3/
# Yandex, Ask
# These support meta robots, but not x-robots-tag
# Crawl everything except real files
User-agent: YandexBot
Disallow: /robots.txt
Disallow: /assets/
Disallow: /api/rest/v1/
Disallow: /api/rest/v2/
Disallow: /api/rest/v3/
Disallow: /pdf/
Disallow: /wpd/
Disallow: /txt/
Disallow: /doc/
User-agent: teoma
Disallow: /robots.txt
Disallow: /assets/
Disallow: /api/rest/v1/
Disallow: /api/rest/v2/
Disallow: /api/rest/v3/
Disallow: /pdf/
Disallow: /wpd/
Disallow: /txt/
Disallow: /doc/
User-agent: ia_archiver
Disallow: /robots.txt
Disallow: /assets/
Disallow: /api/rest/v1/
Disallow: /api/rest/v2/
Disallow: /api/rest/v3/
Disallow: /pdf/
Disallow: /wpd/
Disallow: /txt/
Disallow: /doc/
User-agent: MojeekBot
Disallow: /robots.txt
Disallow: /assets/
Disallow: /api/rest/v1/
Disallow: /api/rest/v2/
Disallow: /api/rest/v3/
Disallow: /pdf/
Disallow: /wpd/
Disallow: /txt/
Disallow: /doc/
# AI Bots
# Training/Scraping bots — crawl to build datasets for model training
User-agent: ClaudeBot
User-agent: GPTBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: meta-externalagent
User-agent: Bytespider
User-agent: CCBot
# Search/Indexing bots — crawl to power AI-assisted search results
User-agent: Claude-SearchBot
User-agent: OAI-SearchBot
User-agent: PerplexityBot
User-agent: Amazonbot
User-agent: Gemini-Deep-Research
# User-initiated agents — fetch pages on behalf of a real user in a conversation
User-agent: Claude-User
User-agent: ChatGPT-User
User-agent: Perplexity-User
User-agent: MistralAI-User
Allow: /robots.txt
Allow: /sitemap.xml
Allow: /help/
Allow: /faq/
Allow: /feeds/
Allow: /podcasts/
Allow: /contact/
Allow: /terms/
Allow: /privacy/
Allow: /removal/
Allow: /donate/
Disallow: /
# Baidu, Blekko, Others
# No support for robots meta tag nor x-robots-tag.
# Be conservative; Block everything.
User-agent: *
Disallow: /
Sitemap: https://www.courtlistener.com/sitemap.xml
No sitemap found
Adding a sitemap helps search engines discover your pages.
CIPv6 ReadinessActionNo IPv6 supportREVIEW
IPv6 support is increasingly important for global accessibility. About 40% of internet users have IPv6 connectivity.
No AAAA records — same impact as 'no IPv6 (AAAA) records'; IPv6-preferring clients pay extra latency falling back to IPv4.
Source: Google IPv6 stats
BTLS Certificate Expiry & Recommendations307 days until leaf cert expires — 3 issues to addressREVIEW
Certificate validity
Recommended actions
- Enable HSTS: Strict-Transport-Security: max-age=31536000; includeSubDomains
- Enable DNSSEC on your domain for DNS spoofing protection
- Enable OCSP stapling on your TLS server to remove a CA roundtrip and protect user privacy
A+DNS Records4 A records, 45 ms lookupPASS
| A | 108.157.98.22, 108.157.98.39, 108.157.98.55, 108.157.98.61 |
| AAAA | — |
| CNAME | — |
| NS | ns-432.awsdns-54.com, ns-1405.awsdns-47.org, ns-791.awsdns-34.net, ns-2043.awsdns-63.co.uk |
| MX | 1 aspmx.l.google.com 5 alt1.aspmx.l.google.com 5 alt2.aspmx.l.google.com 10 alt3.aspmx.l.google.com 10 alt4.aspmx.l.google.com |
| TXT | SPF v=spf1 a include:_spf.google.com ~all brave-ledger-verification=c9a3bd30e22e16786ca2a3e4971a1b521577e2e956c846ff30aa37... google-site-verification=E9l8i1T-eezoNaA3SRmfkIs8TflGMgaDDh6z5TokUno |
| CAA | Lookup not available with standard resolver |
CAA record lookup requires a specialized DNS resolver. This check will be available in a future update.
Informational: CAA (Certification Authority Authorization) records weren't checked in this scan.
ARedirect Chain1 redirect(s), 869 ms totalPASS
https://courtlistener.com
454 ms · HTTP/1.1
https://www.courtlistener.com:443/
416 ms · HTTP/1.1 FINAL
| # | URL | Status | Time | Protocol | Server |
|---|---|---|---|---|---|
| 1 | https://courtlistener.com | 301 | 454 ms | HTTP/1.1 | awselb/2.0 |
| 2 | https://www.courtlistener.com:443/ | 200 | 416 ms | HTTP/1.1 | uvicorn |
See the visual redirect chain in the HTTP Probe tab →
A+URL Variantswww/non-www, trailing slash, HTTP→HTTPSPASS
www / non-www
HTTP → HTTPS
Consistent
A+Domain Intelligencecourtlistener.com — via NameCheap, Inc., 16 years, 3 months old, hosted on AWSPASS
2046 days
March 23, 2032
307 days
Issued by Amazon
16 years, 3 months
Registered March 23, 2010
Not enabled
Protects against DNS spoofing
AWS
ASN AS16509
108.157.98.22
NameCheap, Inc.
Expiry timeline
Recommended actions
- Enable DNSSEC to protect visitors from DNS spoofing
- Enable registrar lock (clientTransferProhibited) to block unauthorized domain transfers
DNSSEC protects against DNS spoofing attacks. While not required, enabling DNSSEC adds an additional layer of security. Contact your DNS provider to enable it.
Without DNSSEC, an attacker who can poison your DNS can hijack your domain — and SSL certs alone don't stop them.
Learn more ▾ ▴
DNSSEC adds cryptographic signatures to DNS records, preventing forged responses from poisoning resolver caches. Without it, an attacker who controls the network path can redirect your domain to a malicious server before any HTTPS handshake happens. Most modern registrars (Cloudflare, Google Domains, Route 53) enable it with one toggle.
Source: ICANN / RFC 4033
The domain can be transferred without an unlock step. Enable registrar lock (clientTransferProhibited) in your registrar's control panel to protect against unauthorized or accidental transfers.
Without registrar lock, an attacker who phishes your registrar credentials can transfer the domain in minutes — total brand hijack.
Learn more ▾ ▴
Registrar lock (clientTransferProhibited, clientUpdateProhibited, clientDeleteProhibited) requires extra verification before any transfer/update/delete. Every major registrar offers it free. Combined with 2FA on your registrar account, it's the strongest defense against domain hijacking.
Source: ICANN / domain-security best practice