Infrastructure
· 17 checks — DNS, redirects, IPv6, crawlability, URL variants, and domain intelligence rolled into one auditable list.FCrawlabilityActionrobots.txt present, sitemap with 0 URLsFIX
Disallow: / for all user-agents prevents search engines from indexing any page. This will remove the site from search results.
Disallow: / in robots.txt blocks every search crawler — the site becomes invisible in organic search.
Learn more ▾ ▴
Common deployment mistake: a staging robots.txt with `User-agent: * / Disallow: /` ships to prod. The site falls out of search results within days. Verify your robots.txt is the production-intended version. If this is intentional (private site), no action needed.
Source: Google Search Central
Search engines may not be able to parse the sitemap. Fix XML validation errors.
An unparseable sitemap is silently ignored by Google — the URLs it advertises are never queued for crawl.
Learn more ▾ ▴
Google's sitemap parser is strict about XML validity. A single unescaped `&` or unclosed tag invalidates the whole file. Run your sitemap through a validator (Search Console's Sitemaps report flags it) and fix the offending entry. Most generators escape correctly; mistakes usually come from manually-written entries.
Source: sitemaps.org / Google Search Central
An empty sitemap provides no value. Add <url> entries for your pages.
An empty sitemap signals 'no content to index' to Google — actively harmful versus having no sitemap at all.
Learn more ▾ ▴
Google compares URLs in the sitemap against URLs it has crawled. An empty sitemap on a site with thousands of pages signals abandonment. Either populate it correctly (most CMSes auto-generate) or delete the file and let Google crawl normally.
Source: Google Search Central / sitemaps.org
# Google Search Engine Robot
# ==========================
User-agent: Googlebot
Allow: /*?s=
Allow: /*?t=
Allow: /*?ref_src=
Allow: /hashtag/*?src=
Allow: /i/api/
# Same length as the /*/likes and /*/media rules; RFC 9309 breaks the tie in
# favor of Allow, so /i/api/ fetches stay crawlable.
Allow: /i/api/*
Disallow: /*?lang=en-ss
Allow: /*?lang=
Disallow: /search?q=
Disallow: /search/realtime
Disallow: /search/users
Disallow: /search/*/grid
Disallow: /*/analytics
Disallow: /*/followers
Disallow: /*/following
Disallow: /*/verified_followers
Disallow: /account/deactivated
Disallow: /settings/deactivated
# Only `*` and `$` are metacharacters here (RFC 9309); regex classes match
# literally. /photo is anchored so /status/*/photo/1 media pages stay open.
Disallow: /*/status/*/likes
Disallow: /*/status/*/retweets
Disallow: /*/likes
Disallow: /*/likes?
Disallow: /*/media
Disallow: /*/media?
Disallow: /*/photo$
Disallow: /*/photo?
Allow: /*?
User-agent: Bingbot
Allow: /*?s=
Allow: /*?t=
Allow: /*?ref_src=
Allow: /hashtag/*?src=
Allow: /i/api/
Allow: /i/api/*
Disallow: /*?lang=en-ss
Allow: /*?lang=
Disallow: /search?q=
Disallow: /search/realtime
Disallow: /search/users
Disallow: /search/*/grid
Disallow: /*/analytics
Disallow: /*/followers
Disallow: /*/following
Disallow: /*/verified_followers
Disallow: /account/deactivated
Disallow: /settings/deactivated
Disallow: /*/status/*/likes
Disallow: /*/status/*/retweets
Disallow: /*/likes
Disallow: /*/likes?
Disallow: /*/media
Disallow: /*/media?
Disallow: /*/photo$
Disallow: /*/photo?
Allow: /*?
User-agent: facebookexternalhit
Allow: /*?lang=
Allow: /*?s=
Allow: /*?t=
Allow: /*?ref_src=
Allow: /hashtag/*?src=
Allow: /search?*cashtagRestId=
Allow: /i/api/
Allow: /i/api/*
Disallow: /search?q=
Disallow: /search/realtime
Disallow: /search/users
Disallow: /search/*/grid
Disallow: /*?
Disallow: /*/followers
Disallow: /*/following
Disallow: /*/verified_followers
Disallow: /account/deactivated
Disallow: /settings/deactivated
Disallow: /*/status/*/likes
Disallow: /*/status/*/retweets
Disallow: /*/likes
Disallow: /*/likes?
Disallow: /*/media
Disallow: /*/media?
Disallow: /*/photo$
Disallow: /*/photo?
User-Agent: Google-Extended
Disallow: *
User-Agent: FacebookBot
Disallow: *
User-agent: Discordbot
Disallow: *
# Every bot that might possibly read and respect this file
# ========================================================
User-agent: *
Disallow: /
# WHAT-4882 - Keep notification-email links (/i/u) out of search results.
# Named crawlers stay un-blocked on purpose: they must crawl /i/u to see its
# X-Robots-Tag noindex (the robots.txt Noindex directive died in 2019).
Disallow: /i/u
# Wait 1 second between successive requests. See ONBOARD-2698 for details.
Crawl-delay: 1
# Independent of user agent. Links in the sitemap are full URLs using https://
# and need to match the protocol of the sitemap.
Sitemap: https://x.com/sitemap.xml
BDNSSECUnsigned (DNSSEC not deployed)REVIEW
BCAA RecordsNo CAA records (any CA may issue certificates)REVIEW
CReverse DNSAction0/1 IPs match cert SANREVIEW
CIPv6 ReadinessActionNo IPv6 supportREVIEW
IPv6 support is increasingly important for global accessibility. About 40% of internet users have IPv6 connectivity.
No AAAA records — same impact as 'no IPv6 (AAAA) records'; IPv6-preferring clients pay extra latency falling back to IPv4.
Source: Google IPv6 stats
BURL Variantswww/non-www, trailing slash, HTTP→HTTPSREVIEW
www / non-www
Preferred variant: non-www
HTTP → HTTPS
HTTP version does not redirect to HTTPS
BTLS Certificate Expiry & Recommendations42 days until leaf cert expires — 3 issues to addressREVIEW
Certificate validity
Recommended actions
- Add the preload directive and submit to hstspreload.org once max-age + includeSubDomains are in place
- Enable DNSSEC on your domain for DNS spoofing protection
- Enable OCSP stapling on your TLS server to remove a CA roundtrip and protect user privacy
BOperational Status PageNo status page link detectedREVIEW
A+DNS Records1 A records, 176 ms lookupPASS
| A | 162.159.140.229 |
| AAAA | — |
| CNAME | — |
| NS | a.r06.twtrdns.net, a.u06.twtrdns.net, b.r06.twtrdns.net, b.u06.twtrdns.net, c.r06.twtrdns.net, c.u06.twtrdns.net, d.r06.twtrdns.net, d.u06.twtrdns.net |
| MX | 1 aspmx.l.google.com 5 alt1.aspmx.l.google.com 5 alt2.aspmx.l.google.com 10 alt3.aspmx.l.google.com 10 alt4.aspmx.l.google.com |
| TXT | 0a8c0fc6-bfa5-4ea7-b09b-87f2989022d6 MS=BEE202D20C326867290BDEFA2DDDF4594B5D6860 adobe-idp-site-verification=a2ff8fc40c434d1d6f02f68b0b1a683e400572ab8c1f2c180c71... apple-domain-verification=zd1iHoEO9LILEQIq atlassian-domain-verification=j6u0o1PTkobCXC84uEF/sWpIPtaZURBVYqKzmTvT8wugLcHT1v... bj6sbt5xqs9hw9jrfvz7hplrg0l680sb canva-site-verification=lMnZ3wMh7c1uqZqa-cxZTg google-site-verification=600dQ0pZYsH2xOFt4hYmf5f5NpjCbWE_qk5Y04dErYM google-site-verification=F2uUiLUsD6kQlpUVQzxUM3PHa0uPo5GBS84SCG8QwXI google-site-verification=P9-NRZ0gaRKRGNDOXOjct5XETPtr3P9D-XA8HnlbAy4 google-site-verification=TNhAkfLUeIbzzzSgPNxS5aEkKMf3aUcpPmCK1_kmIvU google-site-verification=h6dJIv0HXjLOkGAotLAWEzvoi9SxqP4vjpx98vrCvvQ google-site-verification=q1ghWjGLX9Ba-Gy_B4n_pAgC_mQYzWmQpOD8CMWl_Hw linear-domain-verification=t5iq7e7nbw5w loom-site-verification=638c6bc173b9458997f64d305bf42499 miro-verification=6e1ca9ad6d0c2cd2e4186141265f23ed618cfe37 mixpanel-domain-verify=164dda91-31f4-41e8-a816-0f59b38fea30 slack-domain-verification=9oO8P4Glf4252QJDOg4rHGs6KlSkBuI5ZVmWRO8d stripe-verification=46F7B88485621DC18923B43D12E90E6CDBCE232F2FEBCF084E6EFA91F6BA... traction-guest=6882b04e-4188-4ff9-8bb4-bff5a3d358e6 traction-guest=a4d0248d-fe01-4222-8fcc-33f68323e667 SPF v=spf1 ip4:199.16.156.0/22 ip4:199.59.148.0/22 ip4:8.25.194.0/23 ip4:8.25.196.0/... wrike-verification=MjU4MTA5MjoyN2UzNDc1MjU3MDZiZTY4NjBiNzliNDQ2OTUwNWY3NmM5NDgyM... |
| CAA | Lookup not available with standard resolver |
Multiple A records provide failover if one server goes down.
Single A record means a single point of failure — if that IP goes down, your site is unreachable until DNS TTL expires.
Learn more ▾ ▴
Add multiple A records for round-robin failover, or use a managed DNS provider with health-checked failover (Route 53, Cloudflare, NS1). Short TTL (60-300s) lets clients recover faster on outages.
Source: SRE practice / DNS architecture
A+Subdomain TakeoverNo subdomain takeover risk detectedPASS
A+Multi-Resolver DNS SpeedMean 1ms across 3 resolvers (spread 1ms)PASS
ARedirect Chain1 redirect(s), 437 ms totalPASS
https://twitter.com
221 ms · HTTP/1.1
https://x.com/
215 ms · HTTP/1.1 FINAL
| # | URL | Status | Time | Protocol | Server |
|---|---|---|---|---|---|
| 1 | https://twitter.com | 301 | 221 ms | HTTP/1.1 | cloudflare envoy |
| 2 | https://x.com/ | 200 | 215 ms | HTTP/1.1 | cloudflare envoy |
See the visual redirect chain in the HTTP Probe tab →
A+Domain Intelligencetwitter.com — via CSC Corporate Domains, Inc., 26 years, 10 months old, hosted on CloudflarePASS
193 days
January 21, 2027
42 days
Issued by Let's Encrypt
26 years, 10 months
Registered January 21, 2000
Not enabled
Protects against DNS spoofing
Cloudflare
ASN AS13335
162.159.140.229
CSC Corporate Domains, Inc.
Expiry timeline
Recommended actions
- Enable DNSSEC to protect visitors from DNS spoofing
- Enable registrar lock (clientTransferProhibited) to block unauthorized domain transfers
DNSSEC protects against DNS spoofing attacks. While not required, enabling DNSSEC adds an additional layer of security. Contact your DNS provider to enable it.
Without DNSSEC, an attacker who can poison your DNS can hijack your domain — and SSL certs alone don't stop them.
Learn more ▾ ▴
DNSSEC adds cryptographic signatures to DNS records, preventing forged responses from poisoning resolver caches. Without it, an attacker who controls the network path can redirect your domain to a malicious server before any HTTPS handshake happens. Most modern registrars (Cloudflare, Google Domains, Route 53) enable it with one toggle.
Source: ICANN / RFC 4033
The domain can be transferred without an unlock step. Enable registrar lock (clientTransferProhibited) in your registrar's control panel to protect against unauthorized or accidental transfers.
Without registrar lock, an attacker who phishes your registrar credentials can transfer the domain in minutes — total brand hijack.
Learn more ▾ ▴
Registrar lock (clientTransferProhibited, clientUpdateProhibited, clientDeleteProhibited) requires extra verification before any transfer/update/delete. Every major registrar offers it free. Combined with 2FA on your registrar account, it's the strongest defense against domain hijacking.
Source: ICANN / domain-security best practice