Infrastructure
· 9 checks — DNS, redirects, IPv6, crawlability, URL variants, and domain intelligence rolled into one auditable list.FCrawlabilityActionrobots.txt present, sitemap with 0 URLsFIX
Disallow: / for all user-agents prevents search engines from indexing any page. This will remove the site from search results.
Disallow: / in robots.txt blocks every search crawler — the site becomes invisible in organic search.
Learn more ▾ ▴
Common deployment mistake: a staging robots.txt with `User-agent: * / Disallow: /` ships to prod. The site falls out of search results within days. Verify your robots.txt is the production-intended version. If this is intentional (private site), no action needed.
Source: Google Search Central
Search engines may not be able to parse the sitemap. Fix XML validation errors.
An unparseable sitemap is silently ignored by Google — the URLs it advertises are never queued for crawl.
Learn more ▾ ▴
Google's sitemap parser is strict about XML validity. A single unescaped `&` or unclosed tag invalidates the whole file. Run your sitemap through a validator (Search Console's Sitemaps report flags it) and fix the offending entry. Most generators escape correctly; mistakes usually come from manually-written entries.
Source: sitemaps.org / Google Search Central
An empty sitemap provides no value. Add <url> entries for your pages.
An empty sitemap signals 'no content to index' to Google — actively harmful versus having no sitemap at all.
Learn more ▾ ▴
Google compares URLs in the sitemap against URLs it has crawled. An empty sitemap on a site with thousands of pages signals abandonment. Either populate it correctly (most CMSes auto-generate) or delete the file and let Google crawl normally.
Source: Google Search Central / sitemaps.org
# This is the robots.txt file for thesun.co.uk
# The Sun does not permit the unlicensed use of our content for large language models.
# News Group Newspaper's Terms of Use may be found at
# https://www.thesun.co.uk/terms-and-conditions/#website-and-services-terms-of-use
# Contact enquiries@newslicensing.co.uk for assistance.
User-agent: *
Disallow: /
# Sitemap archive
Sitemap: https://www.thesun.co.uk/sitemap.xml
# Sitemaps
Sitemap: https://www.thesun.co.uk/news-sitemap.xml
# Additional sitemaps
Sitemap: https://www.thesun.co.uk/sitemap-nypost-authors.xml
Sitemap: https://www.thesun.co.uk/sitemap-nypost-post_tag.xml
Sitemap: https://www.thesun.co.uk/sitemap-nypost-where.xml
Sitemap: https://www.thesun.co.uk/sitemap-nypost-who.xml
Sitemap: https://www.thesun.co.uk/sitemap-nypost-feeds.xml
Sitemap: https://www.thesun.co.uk/sitemap-nypost-category.xml
Sitemap: https://www.thesun.co.uk/sitemap-nypost-football-team-taxo.xml
Sitemap: https://www.thesun.co.uk/sitemap-nypost-vertical-video.xml
User-agent: Googlebot
User-agent: Googlebot-Image
User-agent: Googlebot-Video
User-agent: Googlebot-News
User-agent: Google-InspectionTool
User-agent: GoogleOther
User-agent: APIs-Google
User-agent: Mediapartners-Google
User-agent: DuckDuckBot
User-agent: Slurp
User-agent: Twitterbot
User-agent: Bingbot
User-agent: adidxbot
User-agent: MicrosoftPreview
User-agent: Applebot
User-agent: AdsBot-Google
User-agent: AmazonAdBot
User-agent: facebookexternalhit
User-agent: Yahoo Ad Monitoring
User-agent: parse.ly scraper
User-agent: Screaming Frog SEO Spider
User-agent: SemrushBot
User-agent: Botify
User-agent: OnCrawl
User-agent: Chrome-Lighthouse
User-agent: Slackbot-LinkExpanding
User-agent: Slack-ImgProxy
User-agent: Slackbot
User-agent: Snap URL Preview Service
User-agent: proximic
User-agent: GumGum-Bot
User-agent: TTD-Content
User-agent: Pubmatic
User-agent: ias_crawler
User-agent: outbrain
User-agent: Leikibot
User-agent: weborama-fetcher
User-agent: AHC/2.1
User-agent: SirdataBot
User-agent: peer39_crawler/1.0
User-agent: EchoboxBot/1.0
User-agent: CriteoBot/0.1
User-agent: AdDefend Site Context Crawler/1.0
User-agent: SmartologyBot
User-agent: Dianomi
User-agent: ChatGPT-User
User-agent: GPTBot
User-agent: OAI-SearchBot
User-agent: snews
Allow: /
Allow: /ads.txt
Allow: /wp-admin/admin-ajax.php
Disallow: /search/
Disallow: /horoscope/
Disallow: /*?x=
Disallow: /*?action=
Disallow: /*&action=
Disallow: /*?adid=
Disallow: /*&adid=
Disallow: /*?bitrate=
Disallow: /*&bitrate=
Disallow: /*?channel=
Disallow: /*&channel=
Disallow: /*?clipid=
Disallow: /*&clipid=
Disallow: /*?contentCollection=
Disallow: /*&contentCollection=
Disallow: /*?contentId=
Disallow: /*&contentId=
Disallow: /*?ei=
Disallow: /*&ei=
Disallow: /*?format=
Disallow: /*&format=
Disallow: /*?height=
Disallow: /*&height=
Disallow: /*?iframe=
Disallow: /*&iframe=
Disallow: /*?mediaId=
Disallow: /*&mediaId=
Disallow: /*?module=
Disallow: /*&module=
Disallow: /*?monthnum=
Disallow: /*&monthnum=
Disallow: /*?pgtype=
Disallow: /*&pgtype=
Disallow: /*?preview=
Disallow: /*&preview=
Disallow: /*?priority=
Disallow: /*&priority=
Disallow: /*?sa=
Disallow: /*&sa=
Disallow: /*?section=
Disallow: /*§ion=
Disallow: /*?share=
Disallow: /*&share=
Disallow: /*?TB_iframe=
Disallow: /*&TB_iframe=
Disallow: /*?TBL_Prod100Page=
Disallow: /*&TBL_Prod100Page=
Disallow: /*?theme_preview=
Disallow: /*&theme_preview=
Disallow: /*?usg=
Disallow: /*&usg=
Disallow: /*?ved=
Disallow: /*&ved=
Disallow: /*?version=
Disallow: /*&version=
Disallow: /*?VID=
Disallow: /*&VID=
Disallow: /*?vxBitrate=
Disallow: /*&vxBitrate=
Disallow: /*?vxChannel=
Disallow: /*&vxChannel=
Disallow: /*?vxClipId=
Disallow: /*&vxClipId=
Disallow: /*?vxSiteId=
Disallow: /*&vxSiteId=
Disallow: /*?width=
Disallow: /*&width=
Disallow: /*?year=
Disallow: /*&year=
Disallow: /*.xml?uuid=
Disallow: /*?post_type=feedback&p=
Disallow: /*?wptouch_preview_theme=
Disallow: /*>
Disallow: /?attachment_id=
Disallow: /affiliate/
Disallow: /ai_log/
Disallow: /*?*pg99.asia*
Disallow: /*&*pg99.asia*
Disallow: /*?*gg155.cn*
Disallow: /*&*gg155.cn*
Disallow: /*?*To66.Asia*
Disallow: /*&*To66.Asia*
Disallow: /wp-admin/
Disallow: /*horseracing/racecards/2019*
Disallow: /*horseracing/racecards/2020*
Disallow: /*horseracing/racecards/2021*
Disallow: /*horseracing/racecards/2022*
Disallow: /*horseracing/results/2019*
Disallow: /*horseracing/results/2020*
Disallow: /*horseracing/results/2021*
Disallow: /*horseracing/results/2022*
Disallow: /simwidgets/
Disallow: /player/brightcove/
Disallow: /*?s=*
Disallow: *&s=*
Disallow: /?p=*
Disallow: /video/
Disallow: /app/
Disallow: /sso/login/
Disallow: /wp-login.php
Disallow: /amp-tealium/
Disallow: /archives/
DCDN & DeliveryActionNo CDN detectedFIX
Consider using a CDN to improve global delivery speed and reduce origin load.
CIPv6 ReadinessActionNo IPv6 supportREVIEW
IPv6 support is increasingly important for global accessibility. About 40% of internet users have IPv6 connectivity.
No AAAA records — same impact as 'no IPv6 (AAAA) records'; IPv6-preferring clients pay extra latency falling back to IPv4.
Source: Google IPv6 stats
BURL Variantswww/non-www, trailing slash, HTTP→HTTPSREVIEW
www / non-www
Inconsistent — duplicate content risk
HTTP → HTTPS
Consistent
BHTTP Probe TimingTotal 846 ms — DNS, TCP, TLS, TTFB, content transfer breakdownREVIEW
Connection waterfall
BTLS Certificate Expiry & Recommendations32 days until leaf cert expires — 3 issues to addressREVIEW
Certificate validity
Recommended actions
- Submit your domain to hstspreload.org to be added to the Chrome preload list
- Enable DNSSEC on your domain for DNS spoofing protection
- Enable OCSP stapling on your TLS server to remove a CA roundtrip and protect user privacy
A+DNS Records1 A records, 102 ms lookupPASS
| A | 192.0.66.2 |
| AAAA | — |
| CNAME | — |
| NS | ns-1478.awsdns-56.org, ns-1970.awsdns-54.co.uk, ns-405.awsdns-50.com, ns-640.awsdns-16.net |
| MX | 1 mxa-00596a01.gslb.pphosted.com 10 mxb-00596a01.gslb.pphosted.com |
| TXT | 00d20000000cog5eao 6j2pfjlk25t2mbgjtqv5gq3hmchtrkwf MS=ms48418918 ValidationTokenValue=2a79563d-5a1e-4202-9cb0-a418f81c5166 ZOOM_verify_02A0JDVORdqHbeawnzucYg adobe-idp-site-verification=627d3f0b-a0d9-4a42-93c9-52d7e9b35602 atlassian-domain-verification=uNoIhBXurxzVlQa0FvK2t9Yld5byfvXbFRQaMToGvrieKjBdyl... figma-domain-verification=b411f1d2852c2c7e057a2d6d70fb22896f37ccc1412d1e1bb4f9da... google-site-verification=03Y2kKMdenvUPoEswcV75nBxRUfMR2F7nu4Ym2Bj9Cg google-site-verification=51m6lrrVo632ZuCqs4SqZZ6N5n6T9_goU9aaf40P0tE google-site-verification=8NQNb3fg0E9f3H0p4HnMV0qFgY9ZMPwignjAgYmCllI google-site-verification=B0ZYMWDxLa5XxtE_cbGUpwtB07xOSb3hRXsVU4rL_88 google-site-verification=D1F-vgfcPF7Tu6VDTyDxeeWVaHvOvw_HmDba3Iea-3g google-site-verification=S4cYKZ1Loesq-BgxfX_ddNqvD8vfhDneE_1tD8CE-YA google-site-verification=V3d6HoAVyohf82CQPLv2n7K3M-vZeKAa0O5viYf41uY google-site-verification=a576LGe3E0YA5QjAMY268jcuKWwPLIORxwlTMrjnfnE google-site-verification=eixRLq6ZFPfabL_W12if1QGJxwbBXFQhFO5J0Vc27EE google-site-verification=hc4Vrp8sPUek7njgfxNHum9CXmJ0OnmTjwXdlxRKxZE google-site-verification=ktbIDvT70_hbFNTapMeOkJ5BkjqY5hO7HRFtkB0yzHc google-site-verification=yrJcP_0hr0wOFdsxjyzctmwWP8_ZtQDxmg6oMhtjGRo knowbe4-site-verification=0694ce74005828dc4bb8b7299bfb6f61 miro-verification=876fd2e87e812aadd96eca532bf071be63109b0e sfv90rs4tcf2wkhcl2tnj311z1z5zn1g SPF v=spf1 include:%{ir}.%{v}.%{d}.spf.has.pphosted.com ~all |
| CAA | Lookup not available with standard resolver |
Multiple A records provide failover if one server goes down.
Single A record means a single point of failure — if that IP goes down, your site is unreachable until DNS TTL expires.
Learn more ▾ ▴
Add multiple A records for round-robin failover, or use a managed DNS provider with health-checked failover (Route 53, Cloudflare, NS1). Short TTL (60-300s) lets clients recover faster on outages.
Source: SRE practice / DNS architecture
CAA record lookup requires a specialized DNS resolver. This check will be available in a future update.
Informational: CAA (Certification Authority Authorization) records weren't checked in this scan.
ARedirect Chain1 redirect(s), 563 ms totalPASS
https://thesun.co.uk
9 ms · HTTP/1.1
https://www.thesun.co.uk/
554 ms · HTTP/1.1 FINAL
| # | URL | Status | Time | Protocol | Server |
|---|---|---|---|---|---|
| 1 | https://thesun.co.uk | 301 | 9 ms | HTTP/1.1 | nginx |
| 2 | https://www.thesun.co.uk/ | 200 | 554 ms | HTTP/1.1 | CloudFront |
See the visual redirect chain in the HTTP Probe tab →
A+Domain Intelligencethesun.co.uk — via Markmonitor Inc., 29 years, 4 months old, hosted on WordPress.com (Automattic)PASS
675 days
May 19, 2028
32 days
Issued by Let's Encrypt
29 years, 4 months
Registered May 19, 1997
Not enabled
Protects against DNS spoofing
WordPress.com (Automattic)
ASN AS2635
192.0.66.2
Markmonitor Inc.
Expiry timeline
Recommended actions
- Enable DNSSEC to protect visitors from DNS spoofing
- Enable registrar lock (clientTransferProhibited) to block unauthorized domain transfers
DNSSEC protects against DNS spoofing attacks. While not required, enabling DNSSEC adds an additional layer of security. Contact your DNS provider to enable it.
Without DNSSEC, an attacker who can poison your DNS can hijack your domain — and SSL certs alone don't stop them.
Learn more ▾ ▴
DNSSEC adds cryptographic signatures to DNS records, preventing forged responses from poisoning resolver caches. Without it, an attacker who controls the network path can redirect your domain to a malicious server before any HTTPS handshake happens. Most modern registrars (Cloudflare, Google Domains, Route 53) enable it with one toggle.
Source: ICANN / RFC 4033
The domain can be transferred without an unlock step. Enable registrar lock (clientTransferProhibited) in your registrar's control panel to protect against unauthorized or accidental transfers.
Without registrar lock, an attacker who phishes your registrar credentials can transfer the domain in minutes — total brand hijack.
Learn more ▾ ▴
Registrar lock (clientTransferProhibited, clientUpdateProhibited, clientDeleteProhibited) requires extra verification before any transfer/update/delete. Every major registrar offers it free. Combined with 2FA on your registrar account, it's the strongest defense against domain hijacking.
Source: ICANN / domain-security best practice