Infrastructure
· 9 checks — DNS, redirects, IPv6, crawlability, URL variants, and domain intelligence rolled into one auditable list.FIPv6 ReadinessActionIPv6 records exist but unreachableFIX
Having AAAA records but an unreachable server is worse than no AAAA — clients may experience delays before falling back to IPv4.
Advertising IPv6 (AAAA records) without a reachable server means IPv6-preferring clients silently fail every connection.
Learn more ▾ ▴
Modern browsers prefer IPv6 if AAAA exists (Happy Eyeballs algorithm). If the IPv6 server isn't reachable, browsers fall back to IPv4 — but with seconds of added latency per request. Either fix IPv6 reachability or remove the AAAA records.
Source: RFC 8305 (Happy Eyeballs)
FCrawlabilityActionrobots.txt present, sitemap with 0 URLsFIX
Disallow: / for all user-agents prevents search engines from indexing any page. This will remove the site from search results.
Disallow: / in robots.txt blocks every search crawler — the site becomes invisible in organic search.
Learn more ▾ ▴
Common deployment mistake: a staging robots.txt with `User-agent: * / Disallow: /` ships to prod. The site falls out of search results within days. Verify your robots.txt is the production-intended version. If this is intentional (private site), no action needed.
Source: Google Search Central
Search engines may not be able to parse the sitemap. Fix XML validation errors.
An unparseable sitemap is silently ignored by Google — the URLs it advertises are never queued for crawl.
Learn more ▾ ▴
Google's sitemap parser is strict about XML validity. A single unescaped `&` or unclosed tag invalidates the whole file. Run your sitemap through a validator (Search Console's Sitemaps report flags it) and fix the offending entry. Most generators escape correctly; mistakes usually come from manually-written entries.
Source: sitemaps.org / Google Search Central
An empty sitemap provides no value. Add <url> entries for your pages.
An empty sitemap signals 'no content to index' to Google — actively harmful versus having no sitemap at all.
Learn more ▾ ▴
Google compares URLs in the sitemap against URLs it has crawled. An empty sitemap on a site with thousands of pages signals abandonment. Either populate it correctly (most CMSes auto-generate) or delete the file and let Google crawl normally.
Source: Google Search Central / sitemaps.org
# PR https://github.com/lobsters/lobsters to be added to the list.
User-agent: Applebot
User-agent: BingBot
User-agent: DuckDuckBot
User-agent: GoogleBot
User-agent: ia_archiver
User-agent: Kagibot
User-agent: Slurp
Allow: /
Disallow: /search
Disallow: /page/
Disallow: /comments/page/
Content-Signal: ai-input=no, ai-train=no, search=yes
User-agent: *
Crawl-delay: 1
Disallow: /
# https://lobste.rs/c/kxd3ji Please stop "respecting" robots.txt and start
# honestly following the standard.
User-agent: Brave
Disallow: /
# https://developers.google.com/search/docs/crawling-indexing/robots/create-robots-txt
Sitemap: https://lobste.rs/sitemap.xml.gz
DCDN & DeliveryActionNo CDN detectedFIX
Consider using a CDN to improve global delivery speed and reduce origin load.
BHTTP Probe TimingTotal 821 ms — DNS, TCP, TLS, TTFB, content transfer breakdownREVIEW
Connection waterfall
BTLS Certificate Expiry & Recommendations77 days until leaf cert expires — 4 issues to addressREVIEW
Certificate validity
Recommended actions
- Add includeSubDomains to the HSTS directive
- Add the preload directive and submit to hstspreload.org once max-age + includeSubDomains are in place
- Enable DNSSEC on your domain for DNS spoofing protection
- Enable OCSP stapling on your TLS server to remove a CA roundtrip and protect user privacy
A+DNS Records1 A records, 12 ms lookupPASS
| A | 68.183.100.95 |
| AAAA | 2604:a880:400:d1:0:2:16bc:d001 |
| CNAME | — |
| NS | ns1.dnsimple.com, ns2.dnsimple.com, ns3.dnsimple.com, ns4.dnsimple.com |
| MX | 10 lobste.rs |
| TXT | SPF v=spf1 mx -all |
| CAA | Lookup not available with standard resolver |
Multiple A records provide failover if one server goes down.
Single A record means a single point of failure — if that IP goes down, your site is unreachable until DNS TTL expires.
Learn more ▾ ▴
Add multiple A records for round-robin failover, or use a managed DNS provider with health-checked failover (Route 53, Cloudflare, NS1). Short TTL (60-300s) lets clients recover faster on outages.
Source: SRE practice / DNS architecture
CAA record lookup requires a specialized DNS resolver. This check will be available in a future update.
Informational: CAA (Certification Authority Authorization) records weren't checked in this scan.
A+Redirect ChainNo redirects — direct accessPASS
https://lobste.rs
270 ms · HTTP/1.1 FINAL
| # | URL | Status | Time | Protocol | Server |
|---|---|---|---|---|---|
| 1 | https://lobste.rs | 200 | 270 ms | HTTP/1.1 |
AURL Variantswww/non-www, trailing slash, HTTP→HTTPSPASS
www / non-www
Preferred variant: non-www
HTTP → HTTPS
Use 301 (permanent) instead of 302 (temporary)
A+Domain Intelligencelobste.rs — via Mainstream Public Cloud Services d.o.o., hosted on DigitalOceanPASS
Unknown
77 days
Issued by Let's Encrypt
Unknown
Status unknown
Protects against DNS spoofing
DigitalOcean
ASN AS14061
68.183.100.95
Mainstream Public Cloud Services d.o.o.
Expiry timeline
Recommended actions
- Enable registrar lock (clientTransferProhibited) to block unauthorized domain transfers
The domain can be transferred without an unlock step. Enable registrar lock (clientTransferProhibited) in your registrar's control panel to protect against unauthorized or accidental transfers.
Without registrar lock, an attacker who phishes your registrar credentials can transfer the domain in minutes — total brand hijack.
Learn more ▾ ▴
Registrar lock (clientTransferProhibited, clientUpdateProhibited, clientDeleteProhibited) requires extra verification before any transfer/update/delete. Every major registrar offers it free. Combined with 2FA on your registrar account, it's the strongest defense against domain hijacking.
Source: ICANN / domain-security best practice