Skip to content
https://theage.com.au

Infrastructure

· 9 checks — DNS, redirects, IPv6, crawlability, URL variants, and domain intelligence rolled into one auditable list.
SCORE
83
GRADE
B
FIX
1
REVIEW
2
PASS
6
INFO
0
Probed from Santa Clara, United States
308 Permanent Redirect
Checks
9
6 PASS 2 REVIEW 1 FIX
D
URL Variants
Action
www/non-www, trailing slash, HTTP→HTTPS
FIX
www/non-www, trailing slash, HTTP→HTTPS
Critical::
Both www and non-www versions serve content
Got: Both variants return 200 Expected: One variant 301-redirects to the other
Critical::
HTTP version does not redirect to HTTPS
Got: HTTP 308 Expected: 301 redirect to HTTPS

www / non-www

200https://www.theage.com.au/
200https://theage.com.au/

Inconsistent — duplicate content risk

HTTP → HTTPS

308http://theage.com.au/ → http://www.theage.com.au/

HTTP version does not redirect to HTTPS

C
IPv6 Readiness
Action
No IPv6 support
REVIEW
No IPv6 support
Info::
No IPv6 (AAAA) records found
IPv6 support is increasingly important for global accessibility. About 40% of internet users have IPv6 connectivity.
No IPv6 Support
About 40% of internet users have IPv6. Consider adding AAAA records.

IPv6 support is increasingly important for global accessibility. About 40% of internet users have IPv6 connectivity.

Why this matters

No AAAA records — same impact as 'no IPv6 (AAAA) records'; IPv6-preferring clients pay extra latency falling back to IPv4.

Source: Google IPv6 stats

B
TLS Certificate Expiry & Recommendations
88 days until leaf cert expires — 4 issues to address
REVIEW

Certificate validity

88
days left
0d 30d 60d 90d+

Recommended actions

  • Prefer TLS 1.3 — TLS 1.2 is acceptable but TLS 1.3 removes RSA key exchange and improves latency
  • Enable HSTS: Strict-Transport-Security: max-age=31536000; includeSubDomains
  • Enable DNSSEC on your domain for DNS spoofing protection
  • Enable OCSP stapling on your TLS server to remove a CA roundtrip and protect user privacy
A+
DNS Records
4 A records, 143 ms lookup
PASS
4 A records, 143 ms lookup
Info::
Resolves to 4 IPv4 address(es)
Got: 151.101.194.133, 151.101.130.133, 151.101.2.133, 151.101.66.133
Info::
No IPv6 (AAAA) records
Info::
4 nameserver(s) configured
Got: ns-106.awsdns-13.com, ns-1391.awsdns-45.org, ns-1683.awsdns-18.co.uk, ns-938.awsdns-53.net
Info::
5 mail exchanger(s) configured
Info::
CAA records not checked
CAA record lookup requires a specialized DNS resolver. This check will be available in a future update.
Info::
SPF record present in TXT
Info::
DNS resolution time: 143 ms
Got: 143 ms
A151.101.194.133, 151.101.130.133, 151.101.2.133, 151.101.66.133
AAAA—
CNAME—
NSns-106.awsdns-13.com, ns-1391.awsdns-45.org, ns-1683.awsdns-18.co.uk, ns-938.awsdns-53.net
MX
1 aspmx.l.google.com
5 alt1.aspmx.l.google.com
5 alt2.aspmx.l.google.com
10 alt4.aspmx.l.google.com
10 alt3.aspmx.l.google.com
TXT
2302895
Validity-Domain-Verification=9d2h18sVUYaUN4ldkVFMw00gLtA=
_globalsign-domain-verification=7JmN6dazOT7LXjyHvi_YmaRlN4ZcwfGCyQ7CfElelu
cg5vp72m2fvf2z87hpnpwf2n3jldzbmn
google-site-verification=4_2xwltTFXpsts3VDUqAtnG_2Xzu4SuNQh8KGvLlfvs
google-site-verification=Ys4iwpVo9s_xd5K2DBynzSNVPwJ9InbqYNGdaEGmH6c
google-site-verification=xYibPFLOTw_tZVFicMI0mBmOueX7Kt2YmhD8gaDna1g
google-site-verification=zlJrhEySjwCMM6RQ4VlfUx_viHIB1mo3H41LJqV6ZDA
miro-verification=e2736fab20b8db58132d7cb294330e593e27d1b7
tollbit-domain-verification=7b557e873cf646a54516cf370c7105f5e28a0eed23c2ed1f6d41...
SPF v=spf1 include:%{i}._ip.%{h}._ehlo.%{d}._spf.vali.email ~all
CAALookup not available with standard resolver
Resolved in 143 ms

CAA record lookup requires a specialized DNS resolver. This check will be available in a future update.

Why this matters

Informational: CAA (Certification Authority Authorization) records weren't checked in this scan.

A
Redirect Chain
1 redirect(s), 51 ms total
PASS
1 redirect(s), 51 ms total
Info::
Single redirect
Got: https://theage.com.au → https://www.theage.com.au/ (308)
Info::
WWW normalization redirect

https://theage.com.au

37 ms · HTTP/1.1

308

https://www.theage.com.au/

14 ms · HTTP/1.1 FINAL

#URLStatusTimeProtocolServer
1https://theage.com.au30837 msHTTP/1.1Varnish
2https://www.theage.com.au/20014 msHTTP/1.1

See the visual redirect chain in the HTTP Probe tab →

A+
Crawlability
robots.txt present, sitemap with 56 URLs
PASS
robots.txt present, sitemap with 56 URLs
Info::
robots.txt is present
Got: 4074 bytes
Info::
sitemap.xml is present
Info::
sitemap.xml is valid XML
Info::
sitemap.xml contains 56 entries
Info::
robots.txt references sitemap
robots.txt 200 OK
Size 4074 B Sitemaps referenced 4 User-agents claude-web, blexbot, timpibot, OAISearch, claudebot, aliyunsecbot, audigentadbot, echoboxbot, friendlycrawler, peer39_crawler, seekrbot, Google-CloudVertexBot, facebookbot, youbot, dataforseobot, diffbot, newsnow, news-please, omgili, meta-externalfetcher, ccbot, mycentralaiscraperbot, webzio-extended, peer39_crawler/1.0, GPTBot, omgilibot, taragroup intelligent bot, turnitinbot, viennatinybot, jetslide, Google-Extended, applebot-extended, perplexitybot, amazonbot, ia_archiver, imagesiftbot, *, ChatGPT-User, cohere-ai, awariorssbot, awariosmartbot, bytespider, duckassistbot, quillbot.com, OAI-SearchBot, anthropic-ai, meta-externalagent, quora-bot, scrapy, seznamhomepagecrawler, magpie-crawler, poseidon research crawler Blocking No — crawling allowed
# NINE ENTERTAINMENT CO. POLICY STATEMENT
# Nine Entertainment Co expressly prohibits the use of any Nine
# content or data, including associated metadata, for any machine
# learning and/or artificial intelligence including for the purposes
# of training or development of AI technology, tools and machine
# learning language models.
# View our terms of use - https://login.nine.com.au/terms?client_id=theage

# Sitemaps
Sitemap: https://www.theage.com.au/sitemaps/news/brands/theage
Sitemap: https://www.theage.com.au/sitemaps/theage-sitemaps-videos.xml
Sitemap: https://www.theage.com.au/sitemaps/theage-sitemaps-articles.xml
Sitemap: https://www.theage.com.au/rss/feed.xml


# -----------------------------------------------------------------
# 1. GENERAL CRAWLER RULES (Allowing standard search engines)
# -----------------------------------------------------------------


# All visitors
User-agent: *
Allow: /
Disallow: /search?text=*
Disallow: *?app=*
Disallow: *?do=*
Disallow: *?ocid=*
Disallow: *?ref=*

# -----------------------------------------------------------------
# 2. SPECIFIC BLOCKS FOR AI, LLM, AND DATA-SCRAPING AGENTS
# -----------------------------------------------------------------


##########
# Google AI Agents (Allows standard Googlebot to continue crawling)
User-agent: Google-CloudVertexBot
Disallow: /
User-agent: Google-Extended
Disallow: /

##########
# OpenAI
User-agent: ChatGPT-User
Disallow: /
User-agent: GPTBot
Disallow: /
User-agent: OAI-SearchBot
Disallow: /
User-agent: OAISearch
Disallow: /

##########
# Anthropic
User-agent: anthropic-ai
Disallow: /
User-agent: claude-web
Disallow: /
User-agent: claudebot
Disallow: /

##########
# Meta (Facebook/LLaMA)
User-agent: facebookbot
Disallow: /
User-agent: meta-externalagent
Disallow: /
User-agent: meta-externalfetcher
Disallow: /

##########
# Apple
User-agent: applebot-extended
Disallow: /

##########
# Perplexity AI
User-agent: perplexitybot
Disallow: /

##########
# Cohere
User-agent: cohere-ai
Disallow: /

##########
# You.com
User-agent: youbot
Disallow: /

##########
# Amazon
User-agent: amazonbot
Disallow: /

##########
# Alibaba Cloud
User-agent: aliyunsecbot
Disallow: /

##########
# Audigent
User-agent: audigentadbot
Disallow: /

##########
# Awario
User-agent: awariorssbot
Disallow: /
User-agent: awariosmartbot
Disallow: /

##########
# BLEX AI
User-agent: blexbot
Disallow: /

##########
# ByteDance
User-agent: bytespider
Disallow: /

##########
# Common Crawl
User-agent: ccbot
Disallow: /

##########
# DataForSEO
User-agent: dataforseobot
Disallow: /

##########
# Diffbot
User-agent: diffbot
Disallow: /

##########
# DuckDuckGo
User-agent: duckassistbot
Disallow: /

##########
# Echobox
User-agent: echoboxbot
Disallow: /

##########
# Friendly Technologies
User-agent: friendlycrawler
Disallow: /

##########
# Internet Archive / "Wayback Machine"
User-agent: ia_archiver
Disallow: /

##########
# ImageSift
User-agent: imagesiftbot
Disallow: /

##########
# MyCentralAI
User-agent: mycentralaiscraperbot
Disallow: /

##########
# NewsNow
User-agent: newsnow
Disallow: /

##########
# News-Please (Open-source)
User-agent: news-please
Disallow: /

##########
# Omgili
User-agent: omgili
Disallow: /
User-agent: omgilibot
Disallow: /
User-agent: webzio-extended
Disallow: /

##########
# Peer39
User-agent: peer39_crawler
Disallow: /
User-agent: peer39_crawler/1.0
Disallow: /

##########
# QuillBot
User-agent: quillbot.com
Disallow: /

##########
# Quora
User-agent: quora-bot
Disallow: /

##########
# Scrapy (Open-source)
User-agent: scrapy
Disallow: /

##########
# Seekr
User-agent: seekrbot
Disallow: /

##########
# Seznam.cz
User-agent: seznamhomepagecrawler
Disallow: /

##########
# TaraGroup
User-agent: taragroup intelligent bot
Disallow: /

##########
# Timpi
User-agent: timpibot
Disallow: /

##########
# Turnitin
User-agent: turnitinbot
Disallow: /

##########
# Others
User-agent: viennatinybot
Disallow: /
User-agent: jetslide
Disallow: /
User-agent: magpie-crawler
Disallow: /
User-agent: poseidon research crawler
Disallow: /
sitemap.xml 200 OK
Type URL Set URLs 56 entries Valid XML Yes
Sample URLs:
A+
Domain Intelligence
theage.com.au — via Corporation Service Company (Aust) Pty Ltd, hosted on Fastly
PASS
theage.com.au — via Corporation Service Company (Aust) Pty Ltd, hosted on Fastly
Info::
DNSSEC is not enabled
DNSSEC protects against DNS spoofing attacks. While not required, enabling DNSSEC adds an additional layer of security. Contact your DNS provider to enable it.
Info::
Registrar: Corporation Service Company (Aust) Pty Ltd
Warning::
Registrar lock is NOT enabled
The domain can be transferred without an unlock step. Enable registrar lock (clientTransferProhibited) in your registrar's control panel to protect against unauthorized or accidental transfers.
Info::
Hosting: Fastly
Got: AS54113
Domain expiry

Unknown

SSL certificate

88 days

Issued by COMODO CA Limited

Domain age

Unknown

DNSSEC

Not enabled

Protects against DNS spoofing

Hosting

Fastly

ASN AS54113

151.101.194.133

Registrar

Corporation Service Company (Aust) Pty Ltd

Unlocked 4 NS records
Expiry timeline
Today
+1 year
SSL expiry Danger zone (≤30 days)
Recommended actions
  • Enable DNSSEC to protect visitors from DNS spoofing
  • Enable registrar lock (clientTransferProhibited) to block unauthorized domain transfers
Registrar Corporation Service Company (Aust) Pty Ltd
Last Updated August 19, 2025
Name Servers ns-106.awsdns-13.com, ns-1391.awsdns-45.org, ns-1683.awsdns-18.co.uk, ns-938.awsdns-53.net
DNSSEC Not enabled
Hosting
IP Address 151.101.194.133
ASN AS54113 (FASTLY - Fastly, Inc., US)
Provider Fastly
Data source: rdap (0.7s)

DNSSEC protects against DNS spoofing attacks. While not required, enabling DNSSEC adds an additional layer of security. Contact your DNS provider to enable it.

Why this matters

Without DNSSEC, an attacker who can poison your DNS can hijack your domain — and SSL certs alone don't stop them.

Learn more ▾

DNSSEC adds cryptographic signatures to DNS records, preventing forged responses from poisoning resolver caches. Without it, an attacker who controls the network path can redirect your domain to a malicious server before any HTTPS handshake happens. Most modern registrars (Cloudflare, Google Domains, Route 53) enable it with one toggle.

Source: ICANN / RFC 4033

The domain can be transferred without an unlock step. Enable registrar lock (clientTransferProhibited) in your registrar's control panel to protect against unauthorized or accidental transfers.

Why this matters

Without registrar lock, an attacker who phishes your registrar credentials can transfer the domain in minutes — total brand hijack.

Learn more ▾

Registrar lock (clientTransferProhibited, clientUpdateProhibited, clientDeleteProhibited) requires extra verification before any transfer/update/delete. Every major registrar offers it free. Combined with 2FA on your registrar account, it's the strongest defense against domain hijacking.

Source: ICANN / domain-security best practice

A+
HTTP Probe Timing
Total 155 ms — DNS, TCP, TLS, TTFB, content transfer breakdown
PASS
DNS Lookup DNS Lookup — time to resolve the domain name to an IP address.
145 ms
TCP Connect TCP Connect — time to establish a TCP connection to the server.
1 ms
TLS Handshake TLS Handshake — time to complete the HTTPS encryption handshake.
7 ms
Time to First Byte Time to First Byte — how long the server takes to respond with the first byte of data.
155 ms
Total Time Total request time from DNS lookup through full response.
155 ms

Connection waterfall

DNS Lookup 145 ms TCP Connect 1 ms TLS Handshake 7 ms Server Processing 2 ms Content Transfer 0 ms
A+
CDN & Delivery
Fastly (HIT)
PASS
Fastly (HIT)
Info::
Site is served via Fastly CDN
Got: x-served-by: cache-sjc1000108-SJC
Info::
CDN cache status: HIT
CDN Detected: Fastly
Provider Fastly Cache Status HIT Evidence x-served-by: cache-sjc1000108-SJC
All checks on this page are automated. Results are estimates - run targeted manual reviews when the score affects a release decision.

Send Feedback