Infrastructure - https://afr.com BeaverCheck Audit

D

URL Variants

Action

www/non-www, trailing slash, HTTP→HTTPS

FIX

www/non-www, trailing slash, HTTP→HTTPS

Both www and non-www versions serve content

Got: Both variants return 200 Expected: One variant 301-redirects to the other

HTTP version does not redirect to HTTPS

Got: HTTP 308 Expected: 301 redirect to HTTPS

www / non-www

200https://www.afr.com/

200https://afr.com/

Inconsistent — duplicate content risk

HTTP → HTTPS

308http://afr.com/ → http://www.afr.com/

HTTP version does not redirect to HTTPS

C

IPv6 Readiness

Action

No IPv6 support

REVIEW

No IPv6 support

No IPv6 (AAAA) records found

IPv6 support is increasingly important for global accessibility. About 40% of internet users have IPv6 connectivity.

No IPv6 Support

About 40% of internet users have IPv6. Consider adding AAAA records.

IPv6 support is increasingly important for global accessibility. About 40% of internet users have IPv6 connectivity.

Why this matters

No AAAA records — same impact as 'no IPv6 (AAAA) records'; IPv6-preferring clients pay extra latency falling back to IPv4.

Source: Google IPv6 stats

B

TLS Certificate Expiry & Recommendations

88 days until leaf cert expires — 4 issues to address

REVIEW

Certificate validity

88

days left

0d 30d 60d 90d+

Recommended actions

Prefer TLS 1.3 — TLS 1.2 is acceptable but TLS 1.3 removes RSA key exchange and improves latency
Enable HSTS: Strict-Transport-Security: max-age=31536000; includeSubDomains
Enable DNSSEC on your domain for DNS spoofing protection
Enable OCSP stapling on your TLS server to remove a CA roundtrip and protect user privacy

A+

DNS Records

4 A records, 36 ms lookup

PASS

4 A records, 36 ms lookup

Resolves to 4 IPv4 address(es)

Got: 151.101.194.133, 151.101.130.133, 151.101.2.133, 151.101.66.133

No IPv6 (AAAA) records

4 nameserver(s) configured

Got: ns-1452.awsdns-53.org, ns-1910.awsdns-46.co.uk, ns-573.awsdns-07.net, ns-170.awsdns-21.com

5 mail exchanger(s) configured

CAA records not checked

CAA record lookup requires a specialized DNS resolver. This check will be available in a future update.

SPF record present in TXT

DNS resolution time: 36 ms

Got: 36 ms

A	151.101.194.133, 151.101.130.133, 151.101.2.133, 151.101.66.133
AAAA	—
CNAME	—
NS	ns-1452.awsdns-53.org, ns-1910.awsdns-46.co.uk, ns-573.awsdns-07.net, ns-170.awsdns-21.com
MX	1 aspmx.l.google.com 5 alt2.aspmx.l.google.com 5 alt1.aspmx.l.google.com 10 alt3.aspmx.l.google.com 10 alt4.aspmx.l.google.com
TXT	qqvlwj40n21sf63jxstdhxb3nb7rqjwz Validity-Domain-Verification=9d2h18sVUYaUN4ldkVFMw00gLtA= google-site-verification=AJ_gHWz8q5jBiihSn1yOlk50f6khww2RXN9LXlKSwtA tollbit-domain-verification=ddd6d45b3533e53b044803643b60bd48d88b09a49f6ac9edbd6d... google-site-verification=DnKEAnO6su4_Eoj6TWpxL7DFdO_nP_VkSsR0W23k718 miro-verification=e2736fab20b8db58132d7cb294330e593e27d1b7 facebook-domain-verification=oq5hte90vjalukaz521iflc61wqnck google-site-verification=ufM9cnrHvUIsNW_qUDK4UyBP4OP594UFgX1Hho6Ht90 SPF v=spf1 include:%{i}._ip.%{h}._ehlo.%{d}._spf.vali.email include:mail.zendesk.com...
CAA	Lookup not available with standard resolver

Resolved in 36 ms

CAA record lookup requires a specialized DNS resolver. This check will be available in a future update.

Why this matters

Informational: CAA (Certification Authority Authorization) records weren't checked in this scan.

A

Redirect Chain

1 redirect(s), 300 ms total

PASS

1 redirect(s), 300 ms total

Single redirect

Got: https://afr.com → https://www.afr.com/ (308)

WWW normalization redirect

https://afr.com

6 ms · HTTP/1.1

308

https://www.afr.com/

294 ms · HTTP/1.1 FINAL

#	URL	Status	Time	Protocol	Server
1	https://afr.com	308	6 ms	HTTP/1.1	Varnish
2	https://www.afr.com/	200	294 ms	HTTP/1.1

See the visual redirect chain in the HTTP Probe tab →

A+

Crawlability

robots.txt present, sitemap with 187 URLs

PASS

robots.txt present, sitemap with 187 URLs

robots.txt is present

Got: 4524 bytes

sitemap.xml is present

sitemap.xml is valid XML

sitemap.xml contains 187 entries

robots.txt references sitemap

robots.txt 200 OK

Size 4524 B Sitemaps referenced 3 User-agents meta-externalagent, echoboxbot, imagesiftbot, webzio-extended, OAI-SearchBot, awariosmartbot, bytespider, dataforseobot, omgilibot, scrapy, applebot-extended, *, OAISearch, claude-web, audigentadbot, ia_archiver, mycentralaiscraperbot, news-please, anthropic-ai, claudebot, cohere-ai, omgili, seznamhomepagecrawler, turnitinbot, magpie-crawler, Google-Extended, youbot, aliyunsecbot, blexbot, quillbot.com, taragroup intelligent bot, timpibot, Google-CloudVertexBot, meta-externalfetcher, poseidon research crawler, ChatGPT-User, perplexitybot, friendlycrawler, peer39_crawler, peer39_crawler/1.0, quora-bot, seekrbot, jetslide, amazonbot, awariorssbot, ccbot, diffbot, duckassistbot, newsnow, viennatinybot, GPTBot, facebookbot Blocking No — crawling allowed

# NINE ENTERTAINMENT CO. POLICY STATEMENT
# Nine Entertainment Co expressly prohibits the use of any Nine
# content or data, including associated metadata, for any machine
# learning and/or artificial intelligence including for the purposes
# of training or development of AI technology, tools and machine
# learning language models.
# View our terms of use - https://login.nine.com.au/terms?client_id=afr

# Sitemaps
Sitemap: https://www.afr.com/sitemaps/news/brands/afr
Sitemap: https://www.afr.com/sitemaps/afr-sitemaps-articles.xml
Sitemap: https://www.afr.com/sitemaps/afr-sitemaps-videos.xml


# -----------------------------------------------------------------
# 1. GENERAL CRAWLER RULES (Allowing standard search engines)
# -----------------------------------------------------------------

# All visitors
User-agent: *
Allow: /
Disallow: /afr-newsletter-btb
Disallow: /afr-newsletter-mw5
Disallow: /btb-newsletter-v1
Disallow: /btb-newsletter-v2
Disallow: /btb-newsletter-v3
Disallow: /lifestyle-newsletter-v1
Disallow: /lifestyle-newsletter-v2
Disallow: /mw5-newsletter-v1
Disallow: /mw5-newsletter-v2
Disallow: /myaccount
Disallow: /newsfeed
Disallow: /newsletter
Disallow: /search$
Disallow: /search?
Disallow: /smi-newsletter
Disallow: /street-talk-early-look
Disallow: /the-election-brief
Disallow: /word-from-the-editor-v1
Disallow: /word-from-the-editor-v2
Disallow: /webview
Disallow: *?app=*
Disallow: *?do=*
Disallow: *?ocid=*
Disallow: *?ref=*

# -----------------------------------------------------------------
# 2. SPECIFIC BLOCKS FOR AI, LLM, AND DATA-SCRAPING AGENTS
# -----------------------------------------------------------------


##########
# Google AI Agents (Allows standard Googlebot to continue crawling)
User-agent: Google-CloudVertexBot
Disallow: /
User-agent: Google-Extended
Disallow: /

##########
# OpenAI
User-agent: ChatGPT-User
Disallow: /
User-agent: GPTBot
Disallow: /
User-agent: OAI-SearchBot
Disallow: /
User-agent: OAISearch
Disallow: /

##########
# Anthropic
User-agent: anthropic-ai
Disallow: /
User-agent: claude-web
Disallow: /
User-agent: claudebot
Disallow: /

##########
# Meta (Facebook/LLaMA)
User-agent: facebookbot
Disallow: /
User-agent: meta-externalagent
Disallow: /
User-agent: meta-externalfetcher
Disallow: /

##########
# Apple
User-agent: applebot-extended
Disallow: /

##########
# Perplexity AI
User-agent: perplexitybot
Disallow: /

##########
# Cohere
User-agent: cohere-ai
Disallow: /

##########
# You.com
User-agent: youbot
Disallow: /

##########
# Amazon
User-agent: amazonbot
Disallow: /

##########
# Alibaba Cloud
User-agent: aliyunsecbot
Disallow: /

##########
# Audigent
User-agent: audigentadbot
Disallow: /

##########
# Awario
User-agent: awariorssbot
Disallow: /
User-agent: awariosmartbot
Disallow: /

##########
# BLEX AI
User-agent: blexbot
Disallow: /

##########
# ByteDance
User-agent: bytespider
Disallow: /

##########
# Common Crawl
User-agent: ccbot
Disallow: /

##########
# DataForSEO
User-agent: dataforseobot
Disallow: /

##########
# Diffbot
User-agent: diffbot
Disallow: /

##########
# DuckDuckGo
User-agent: duckassistbot
Disallow: /

##########
# Echobox
User-agent: echoboxbot
Disallow: /

##########
# Friendly Technologies
User-agent: friendlycrawler
Disallow: /

##########
# Internet Archive / "Wayback Machine"
User-agent: ia_archiver
Disallow: /

##########
# ImageSift
User-agent: imagesiftbot
Disallow: /

##########
# MyCentralAI
User-agent: mycentralaiscraperbot
Disallow: /

##########
# NewsNow
User-agent: newsnow
Disallow: /

##########
# News-Please (Open-source)
User-agent: news-please
Disallow: /

##########
# Omgili
User-agent: omgili
Disallow: /
User-agent: omgilibot
Disallow: /
User-agent: webzio-extended
Disallow: /

##########
# Peer39
User-agent: peer39_crawler
Disallow: /
User-agent: peer39_crawler/1.0
Disallow: /

##########
# QuillBot
User-agent: quillbot.com
Disallow: /

##########
# Quora
User-agent: quora-bot
Disallow: /

##########
# Scrapy (Open-source)
User-agent: scrapy
Disallow: /

##########
# Seekr
User-agent: seekrbot
Disallow: /

##########
# Seznam.cz
User-agent: seznamhomepagecrawler
Disallow: /

##########
# TaraGroup
User-agent: taragroup intelligent bot
Disallow: /

##########
# Timpi
User-agent: timpibot
Disallow: /

##########
# Turnitin
User-agent: turnitinbot
Disallow: /

##########
# Others
User-agent: viennatinybot
Disallow: /
User-agent: jetslide
Disallow: /
User-agent: magpie-crawler
Disallow: /
User-agent: poseidon research crawler
Disallow: /

sitemap.xml 200 OK

Type URL Set URLs 187 entries Valid XML Yes

Sample URLs:

A

Domain Intelligence

afr.com — via CSC Corporate Domains, Inc., 29 years, 3 months old, hosted on Fastly

PASS

afr.com — via CSC Corporate Domains, Inc., 29 years, 3 months old, hosted on Fastly

Domain expires in 41 days

Consider enabling auto-renewal to prevent accidental expiration.

Got: Expires Jun 2, 2026

DNSSEC is not enabled

DNSSEC protects against DNS spoofing attacks. While not required, enabling DNSSEC adds an additional layer of security. Contact your DNS provider to enable it.

Registrar: CSC Corporate Domains, Inc.

Registrar lock is NOT enabled

The domain can be transferred without an unlock step. Enable registrar lock (clientTransferProhibited) in your registrar's control panel to protect against unauthorized or accidental transfers.

Hosting: Fastly

Got: AS54113

Domain expiry

EXPIRED

June 2, 2026

SSL certificate

88 days

Issued by COMODO CA Limited

Domain age

29 years, 3 months

Registered June 3, 1997

DNSSEC

Not enabled

Protects against DNS spoofing

Hosting

Fastly

ASN AS54113

151.101.130.133

Registrar

CSC Corporate Domains, Inc.

Unlocked 4 NS records

Expiry timeline

Today

+1 year

Domain expiry SSL expiry Danger zone (≤30 days)

Recommended actions

Domain has EXPIRED — renew immediately to avoid total site outage
Enable DNSSEC to protect visitors from DNS spoofing
Enable registrar lock (clientTransferProhibited) to block unauthorized domain transfers

Registrar CSC Corporate Domains, Inc.

Created June 3, 1997 (29 years, 3 months ago)

Expires June 2, 2026 (1 months)

Last Updated May 29, 2025

Name Servers ns-1452.awsdns-53.org, ns-170.awsdns-21.com, ns-1910.awsdns-46.co.uk, ns-573.awsdns-07.net

DNSSEC Not enabled

Hosting

IP Address 151.101.130.133

ASN AS54113 (FASTLY - Fastly, Inc., US)

Provider Fastly

Data source: rdap (0.3s)

Consider enabling auto-renewal to prevent accidental expiration.

Why this matters

Domain expiry approaching — renew immediately and ensure auto-renew + alerting are configured.

Source: ICANN renewal policy

DNSSEC protects against DNS spoofing attacks. While not required, enabling DNSSEC adds an additional layer of security. Contact your DNS provider to enable it.

Why this matters

Without DNSSEC, an attacker who can poison your DNS can hijack your domain — and SSL certs alone don't stop them.

Learn more ▾

DNSSEC adds cryptographic signatures to DNS records, preventing forged responses from poisoning resolver caches. Without it, an attacker who controls the network path can redirect your domain to a malicious server before any HTTPS handshake happens. Most modern registrars (Cloudflare, Google Domains, Route 53) enable it with one toggle.

Source: ICANN / RFC 4033

The domain can be transferred without an unlock step. Enable registrar lock (clientTransferProhibited) in your registrar's control panel to protect against unauthorized or accidental transfers.

Why this matters

Without registrar lock, an attacker who phishes your registrar credentials can transfer the domain in minutes — total brand hijack.

Learn more ▾

Registrar lock (clientTransferProhibited, clientUpdateProhibited, clientDeleteProhibited) requires extra verification before any transfer/update/delete. Every major registrar offers it free. Combined with 2FA on your registrar account, it's the strongest defense against domain hijacking.

Source: ICANN / domain-security best practice

A+

HTTP Probe Timing

Total 42 ms — DNS, TCP, TLS, TTFB, content transfer breakdown

PASS

DNS Lookup

37 ms

TCP Connect

0 ms

TLS Handshake

5 ms

Time to First Byte

43 ms

Total Time

43 ms

Connection waterfall

DNS Lookup 37 ms TCP Connect 0 ms TLS Handshake 5 ms Server Processing 1 ms Content Transfer 0 ms

A+

CDN & Delivery

Fastly (HIT)

PASS

Fastly (HIT)

Site is served via Fastly CDN

Got: x-served-by: cache-toj-leto2350054-TOJ

CDN cache status: HIT

CDN Detected: Fastly

Provider Fastly Cache Status HIT Evidence x-served-by: cache-toj-leto2350054-TOJ