Skip to content
https://refinery29.com

Infrastructure

· 9 checks — DNS, redirects, IPv6, crawlability, URL variants, and domain intelligence rolled into one auditable list.
SCORE
88
GRADE
B
FIX
0
REVIEW
3
PASS
6
INFO
0
Probed from Madrid, Spain
301 Moved Permanently
Checks
9
6 PASS 3 REVIEW
B
Redirect Chain
2 redirect(s), 390 ms total
REVIEW
2 redirect(s), 390 ms total
Warning::
2 redirects before reaching final URL
Each redirect adds latency. Try to minimize the chain to 1 hop.
Info::
WWW normalization redirect

https://refinery29.com

6 ms · HTTP/1.1

301

https://www.refinery29.com/

168 ms · HTTP/1.1

301

https://www.refinery29.com/fr-fr

216 ms · HTTP/1.1 FINAL

#URLStatusTimeProtocolServer
1https://refinery29.com3016 msHTTP/1.1
2https://www.refinery29.com/301168 msHTTP/1.1
3https://www.refinery29.com/fr-fr200216 msHTTP/1.1

See the visual redirect chain in the HTTP Probe tab →

Each redirect adds latency. Try to minimize the chain to 1 hop.

Why this matters

Redirect chain — each hop adds latency; combine into one redirect where possible.

Source: Google Search Central / web.dev

C
IPv6 Readiness
Action
No IPv6 support
REVIEW
No IPv6 support
Info::
No IPv6 (AAAA) records found
IPv6 support is increasingly important for global accessibility. About 40% of internet users have IPv6 connectivity.
No IPv6 Support
About 40% of internet users have IPv6. Consider adding AAAA records.

IPv6 support is increasingly important for global accessibility. About 40% of internet users have IPv6 connectivity.

Why this matters

No AAAA records — same impact as 'no IPv6 (AAAA) records'; IPv6-preferring clients pay extra latency falling back to IPv4.

Source: Google IPv6 stats

B
TLS Certificate Expiry & Recommendations
85 days until leaf cert expires — 3 issues to address
REVIEW

Certificate validity

85
days left
0d 30d 60d 90d+

Recommended actions

  • Add the preload directive and submit to hstspreload.org once max-age + includeSubDomains are in place
  • Enable DNSSEC on your domain for DNS spoofing protection
  • Enable OCSP stapling on your TLS server to remove a CA roundtrip and protect user privacy
A+
DNS Records
4 A records, 86 ms lookup
PASS
4 A records, 86 ms lookup
Info::
Resolves to 4 IPv4 address(es)
Got: 151.101.65.55, 151.101.129.55, 151.101.193.55, 151.101.1.55
Info::
No IPv6 (AAAA) records
Info::
4 nameserver(s) configured
Got: ns-497.awsdns-62.com, ns-1334.awsdns-38.org, ns-1547.awsdns-01.co.uk, ns-724.awsdns-26.net
Info::
2 mail exchanger(s) configured
Info::
CAA records not checked
CAA record lookup requires a specialized DNS resolver. This check will be available in a future update.
Info::
SPF record present in TXT
Info::
DNS resolution time: 86 ms
Got: 86 ms
A151.101.65.55, 151.101.129.55, 151.101.193.55, 151.101.1.55
AAAA—
CNAME—
NSns-497.awsdns-62.com, ns-1334.awsdns-38.org, ns-1547.awsdns-01.co.uk, ns-724.awsdns-26.net
MX
10 us-smtp-inbound-2.mimecast.com
10 us-smtp-inbound-1.mimecast.com
TXT
SPF v=spf1 include:us._netblocks.mimecast.com ~all
globalsign-domain-verification=pyR6ci6IB7uVAxLPZN5Z7_imdnvGJLhXCcmfs8v5RP
google-site-verification=cJ3FILEcdhMJjPMjwDBiIL04IbRZ6YlGcDyCsnyaB84
zone-ownership-verification-1c491587c899aa838c9c5747cf64e039a41b9c96a2e60de96cd3...
include:aspmx.sailthru.com include:spf.mtasv.net include:_netblocks.sparkpostmai...
airtable-verification=e558e6c5f5fcae3c9165a7b5d2245b92
MS=ms37368417
google-site-verification=G_g2a-9YzbrEJMsfZ550LV-oa_2bBZ_aSUbRB4ZpK6U
globalsign-domain-verification=Wq9iztGUzQcBBi1OLrEOlXtefwOzX7yfvhbsw-CVdo
yahoo-verification-key=ZLCMc3K+ejn56qa0hC6EGHSTod61wO6Pm23N+BB43N0=
f4jrbvle3p63lka71pennjvvqj
_globalsign-domain-verification=xfkrv3yRwA5GGm0E4l5RlcNKTqVD8KAYsYdCYTBMF0
google-site-verification=CTjE_9gj7lLpkqCpXk5rJxzIQlUt9KCKz6BPe8geiYk
google-site-verification=H_0MNFssZifazjHTV5b_ea2nusJ1xRVqEHUG75FJcFk
facebook-domain-verification=zq06gb4g3wy80zzzx6sza8h7beljak
docusign=3d92aff7-73a5-4ffa-b3ed-bcaef967bc07
google-site-verification=1kcxCrYA1m1KQqZE0cT7CBTEmsbRpGYT1Xg9aSxHwOw
google-site-verification=biRiI7k275QdekUiJLAezVeUHFRyML7Odwf3lktMVk8
pinterest-site-verification=dc5e879bef1414e51cc028deebc3028b
07.11.2025
google-site-verification=aQr9iqlPpPoWMVmPdJALeVk8SKU7vrpd2iW5AW-_f8s
CAALookup not available with standard resolver
Resolved in 86 ms

CAA record lookup requires a specialized DNS resolver. This check will be available in a future update.

Why this matters

Informational: CAA (Certification Authority Authorization) records weren't checked in this scan.

A+
Crawlability
robots.txt present, sitemap with 6792 URLs
PASS
robots.txt present, sitemap with 6792 URLs
Info::
robots.txt is present
Got: 6302 bytes
Info::
sitemap.xml is present
Info::
sitemap.xml is valid XML
Info::
sitemap.xml contains 6792 entries
Info::
Sitemap index with 6792 child sitemaps
Info::
robots.txt references sitemap
robots.txt 200 OK
Size 6302 B Sitemaps referenced 1 User-agents Operator, CloudVertexBot, Google-CloudVertexBot, TikTokSpider, LinkupBot, Andibot, ISSCyberRiskCrawler, quillbot.com, Amazonbot, Crawl4AI, FirecrawlAgent, AddSearchBot, atlassian-bot, MyCentralAIScraperBot, Bytespider, MistralAI-User/1.0, cohere-ai, LAIONDownloader, omgili, Cotoyogi, KlaviyoAIBot, ZanistaBot, Google-Firebase, panscient.com, Claude-Web, Devin, TerraCotta, YaK, GoogleOther-Video, Google-NotebookLM, Meta-ExternalFetcher, EchoboxBot, FriendlyCrawler, Poseidon Research Crawler, WRTNBot, YandexAdditionalBot, GPTBot, amazon-kendra, cohere-training-data-crawler, TavilyBot, AI2Bot, Claude-SearchBot, Gemini-Deep-Research, meta-webindexer, Amzn-User, PetalBot, iaskspider, LinerBot, NotebookLM, ClaudeBot, Bravebot, bigsur.ai, Echobot Bot, KunatoCrawler, Sidetrade indexer bot, Spider, PerplexityBot, kagi-fetcher, Scrapy, SemrushBot-SWA, ShapBot, *, Claude-User, MistralAI-User, DuckAssistBot, QualifiedBot, Thinkbot, meta-externalagent, FacebookBot, Perplexity-User, PhindBot, TwinAgent, img2dataset, GoogleOther-Image, meta-externalfetcher, facebookexternalhit, netEstate Imprint Crawler, aiHitBot, Kangaroo Bot, Ai2Bot-Dolma, SemrushBot-OCOB, iaskspider/2.0, AmazonBuyForMe, IbouBot, Factset_spyderbot, OAI-SearchBot, ChatGPT Agent, LCC, Linguee Bot, iAskBot, Anomura, imageSpider, Panscient, SBIntuitionsBot, OpenAI, DeepSeekBot, ChatGLM-Spider, Cloudflare-AutoRAG, Poggio-Citations, ChatGPT-User, Timpibot, WARDBot, Webzio-Extended, GoogleOther, Meta-ExternalAgent, Amzn-SearchBot, bedrockbot, YouBot, AI2Bot-DeepResearchEval, ICC-Crawler, Brightbot 1.0, anthropic-ai, GoogleAgent-Mariner, AzureAI-SearchBot, Diffbot, laion-huggingface-processor, omgilibot, Datenbank Crawler, VelenPublicWebCrawler, Applebot-Extended, NovaAct, Crawlspace, BuddyBot, QuillBot, wpbot, Google-Extended, CCBot, Channel3Bot, ImagesiftBot, PanguBot, Awario, webzio-extended, Manus-User, YandexAdditional Blocking No — crawling allowed
# ============================================================
# REFINERY29 ROBOTS.TXT
# Last updated: April 2026
# ============================================================

# ============================================================
# AI CRAWLERS FULL BLOCK
# Source: github.com/ai-robots-txt/ai.robots.txt
# ============================================================

# --- OPENAI ---
User-agent: GPTBot
Disallow: /
User-agent: ChatGPT-User
Disallow: /
User-agent: OAI-SearchBot
Disallow: /
User-agent: ChatGPT Agent
Disallow: /
User-agent: Operator
Disallow: /
User-agent: OpenAI
Disallow: /

# --- ANTHROPIC ---
User-agent: ClaudeBot
Disallow: /
User-agent: anthropic-ai
Disallow: /
User-agent: Claude-SearchBot
Disallow: /
User-agent: Claude-User
Disallow: /
User-agent: Claude-Web
Disallow: /

# --- GOOGLE AI ---
User-agent: Google-Extended
Disallow: /
User-agent: GoogleOther
Disallow: /
User-agent: GoogleOther-Image
Disallow: /
User-agent: GoogleOther-Video
Disallow: /
User-agent: Google-CloudVertexBot
Disallow: /
User-agent: CloudVertexBot
Disallow: /
User-agent: Google-NotebookLM
Disallow: /
User-agent: NotebookLM
Disallow: /
User-agent: GoogleAgent-Mariner
Disallow: /
User-agent: Gemini-Deep-Research
Disallow: /
User-agent: Google-Firebase
Disallow: /

# --- APPLE ---
User-agent: Applebot-Extended
Disallow: /

# --- META ---
User-agent: Meta-ExternalAgent
Disallow: /
User-agent: meta-externalagent
Disallow: /
User-agent: Meta-ExternalFetcher
Disallow: /
User-agent: meta-externalfetcher
Disallow: /
User-agent: meta-webindexer
Disallow: /
User-agent: FacebookBot
Disallow: /
User-agent: facebookexternalhit
Disallow: /

# --- BYTEDANCE ---
User-agent: Bytespider
Disallow: /
User-agent: TikTokSpider
Disallow: /

# --- AMAZON ---
User-agent: Amazonbot
Disallow: /
User-agent: Amzn-SearchBot
Disallow: /
User-agent: Amzn-User
Disallow: /
User-agent: AmazonBuyForMe
Disallow: /
User-agent: amazon-kendra
Disallow: /

# --- PERPLEXITY ---
User-agent: PerplexityBot
Disallow: /
User-agent: Perplexity-User
Disallow: /

# --- DEEPSEEK ---
User-agent: DeepSeekBot
Disallow: /

# --- MISTRAL ---
User-agent: MistralAI-User
Disallow: /
User-agent: MistralAI-User/1.0
Disallow: /

# --- COMMON CRAWL ---
User-agent: CCBot
Disallow: /

# --- COHERE ---
User-agent: cohere-ai
Disallow: /
User-agent: cohere-training-data-crawler
Disallow: /

# --- MICROSOFT / AZURE ---
User-agent: AzureAI-SearchBot
Disallow: /
User-agent: bedrockbot
Disallow: /

# --- OTHER AI SEARCH BOTS ---
User-agent: Bravebot
Disallow: /
User-agent: DuckAssistBot
Disallow: /
User-agent: PhindBot
Disallow: /
User-agent: YouBot
Disallow: /
User-agent: PetalBot
Disallow: /
User-agent: Diffbot
Disallow: /
User-agent: iAskBot
Disallow: /
User-agent: iaskspider
Disallow: /
User-agent: iaskspider/2.0
Disallow: /
User-agent: kagi-fetcher
Disallow: /
User-agent: LinerBot
Disallow: /
User-agent: LinkupBot
Disallow: /
User-agent: IbouBot
Disallow: /
User-agent: PanguBot
Disallow: /
User-agent: ChatGLM-Spider
Disallow: /

# --- AGENTIC TOOLS ---
User-agent: Manus-User
Disallow: /
User-agent: NovaAct
Disallow: /
User-agent: Devin
Disallow: /
User-agent: TwinAgent
Disallow: /

# --- SCRAPERS & DATA PIPELINES ---
User-agent: Scrapy
Disallow: /
User-agent: Crawl4AI
Disallow: /
User-agent: Crawlspace
Disallow: /
User-agent: FirecrawlAgent
Disallow: /
User-agent: img2dataset
Disallow: /
User-agent: LAIONDownloader
Disallow: /
User-agent: laion-huggingface-processor
Disallow: /
User-agent: Cloudflare-AutoRAG
Disallow: /
User-agent: TavilyBot
Disallow: /
User-agent: omgili
Disallow: /
User-agent: omgilibot
Disallow: /

# --- MISCELLANEOUS AI BOTS ---
User-agent: AddSearchBot
Disallow: /
User-agent: AI2Bot
Disallow: /
User-agent: AI2Bot-DeepResearchEval
Disallow: /
User-agent: Ai2Bot-Dolma
Disallow: /
User-agent: aiHitBot
Disallow: /
User-agent: Andibot
Disallow: /
User-agent: Anomura
Disallow: /
User-agent: atlassian-bot
Disallow: /
User-agent: Awario
Disallow: /
User-agent: bigsur.ai
Disallow: /
User-agent: Brightbot 1.0
Disallow: /
User-agent: BuddyBot
Disallow: /
User-agent: Channel3Bot
Disallow: /
User-agent: Cotoyogi
Disallow: /
User-agent: Datenbank Crawler
Disallow: /
User-agent: Echobot Bot
Disallow: /
User-agent: EchoboxBot
Disallow: /
User-agent: Factset_spyderbot
Disallow: /
User-agent: FriendlyCrawler
Disallow: /
User-agent: ICC-Crawler
Disallow: /
User-agent: ImagesiftBot
Disallow: /
User-agent: imageSpider
Disallow: /
User-agent: ISSCyberRiskCrawler
Disallow: /
User-agent: Kangaroo Bot
Disallow: /
User-agent: KlaviyoAIBot
Disallow: /
User-agent: KunatoCrawler
Disallow: /
User-agent: LCC
Disallow: /
User-agent: Linguee Bot
Disallow: /
User-agent: MyCentralAIScraperBot
Disallow: /
User-agent: netEstate Imprint Crawler
Disallow: /
User-agent: Panscient
Disallow: /
User-agent: panscient.com
Disallow: /
User-agent: Poggio-Citations
Disallow: /
User-agent: Poseidon Research Crawler
Disallow: /
User-agent: QualifiedBot
Disallow: /
User-agent: QuillBot
Disallow: /
User-agent: quillbot.com
Disallow: /
User-agent: SBIntuitionsBot
Disallow: /
User-agent: SemrushBot-OCOB
Disallow: /
User-agent: SemrushBot-SWA
Disallow: /
User-agent: ShapBot
Disallow: /
User-agent: Sidetrade indexer bot
Disallow: /
User-agent: Spider
Disallow: /
User-agent: TerraCotta
Disallow: /
User-agent: Thinkbot
Disallow: /
User-agent: Timpibot
Disallow: /
User-agent: VelenPublicWebCrawler
Disallow: /
User-agent: WARDBot
Disallow: /
User-agent: Webzio-Extended
Disallow: /
User-agent: webzio-extended
Disallow: /
User-agent: wpbot
Disallow: /
User-agent: WRTNBot
Disallow: /
User-agent: YaK
Disallow: /
User-agent: YandexAdditional
Disallow: /
User-agent: YandexAdditionalBot
Disallow: /
User-agent: ZanistaBot
Disallow: /

# ============================================================
# GENERAL CRAWLER RULES
# ============================================================
User-agent: *
Crawl-delay: 1
Disallow: /api/
Disallow: /shops/
Disallow: /sweeps/
Disallow: /rsvp/
Disallow: /search?*
Disallow: /en-us/search?*
Disallow: /en-gb/search?*
Disallow: /de-de/search?*
Disallow: /fr-fr/search?*
Disallow: /en-ca/search?*
Disallow: /en-au/search?*
Disallow: /*.php
Disallow: /*?bucketed*
Disallow: /*&bucketed*
Disallow: /history-cults-los-angeles

Sitemap: https://www.refinery29.com/sitemap.xml
sitemap.xml 200 OK
Type Sitemap Index URLs 6792 entries Valid XML Yes
Child Sitemaps:
A+
URL Variants
www/non-www, trailing slash, HTTP→HTTPS
PASS
www/non-www, trailing slash, HTTP→HTTPS
Info::
www/non-www redirect configured correctly (preferred: non-www)
Info::
HTTP correctly 301-redirects to HTTPS

www / non-www

301https://www.refinery29.com/
200https://refinery29.com/

Preferred variant: non-www

HTTP → HTTPS

301http://refinery29.com/ → https://refinery29.com/

Consistent

A+
Domain Intelligence
refinery29.com — via Gandi SAS, 21 years, 8 months old, hosted on Fastly
PASS
refinery29.com — via Gandi SAS, 21 years, 8 months old, hosted on Fastly
Info::
Domain registered until Nov 22, 2028 (2 years, 7 months remaining)
Info::
DNSSEC is not enabled
DNSSEC protects against DNS spoofing attacks. While not required, enabling DNSSEC adds an additional layer of security. Contact your DNS provider to enable it.
Info::
Registrar: Gandi SAS
Warning::
Registrar lock is NOT enabled
The domain can be transferred without an unlock step. Enable registrar lock (clientTransferProhibited) in your registrar's control panel to protect against unauthorized or accidental transfers.
Info::
Hosting: Fastly
Got: AS54113
Domain expiry

890 days

November 22, 2028

SSL certificate

85 days

Issued by Let's Encrypt

Domain age

21 years, 8 months

Registered November 22, 2004

DNSSEC

Not enabled

Protects against DNS spoofing

Hosting

Fastly

ASN AS54113

151.101.193.55

Registrar

Gandi SAS

Unlocked 4 NS records
Expiry timeline
Today
+1 year
Domain expiry SSL expiry Danger zone (≤30 days)
Recommended actions
  • Enable DNSSEC to protect visitors from DNS spoofing
  • Enable registrar lock (clientTransferProhibited) to block unauthorized domain transfers
Registrar Gandi SAS
Created November 22, 2004 (21 years, 8 months ago)
Expires November 22, 2028 (2 years, 7 months)
Last Updated September 23, 2025
Name Servers ns-1334.awsdns-38.org, ns-1547.awsdns-01.co.uk, ns-497.awsdns-62.com, ns-724.awsdns-26.net
DNSSEC Not enabled
Hosting
IP Address 151.101.193.55
ASN AS54113 (FASTLY - Fastly, Inc., US)
Provider Fastly
Data source: rdap (0.3s)

DNSSEC protects against DNS spoofing attacks. While not required, enabling DNSSEC adds an additional layer of security. Contact your DNS provider to enable it.

Why this matters

Without DNSSEC, an attacker who can poison your DNS can hijack your domain — and SSL certs alone don't stop them.

Learn more ▾

DNSSEC adds cryptographic signatures to DNS records, preventing forged responses from poisoning resolver caches. Without it, an attacker who controls the network path can redirect your domain to a malicious server before any HTTPS handshake happens. Most modern registrars (Cloudflare, Google Domains, Route 53) enable it with one toggle.

Source: ICANN / RFC 4033

The domain can be transferred without an unlock step. Enable registrar lock (clientTransferProhibited) in your registrar's control panel to protect against unauthorized or accidental transfers.

Why this matters

Without registrar lock, an attacker who phishes your registrar credentials can transfer the domain in minutes — total brand hijack.

Learn more ▾

Registrar lock (clientTransferProhibited, clientUpdateProhibited, clientDeleteProhibited) requires extra verification before any transfer/update/delete. Every major registrar offers it free. Combined with 2FA on your registrar account, it's the strongest defense against domain hijacking.

Source: ICANN / domain-security best practice

A+
HTTP Probe Timing
Total 41 ms — DNS, TCP, TLS, TTFB, content transfer breakdown
PASS
DNS Lookup DNS Lookup — time to resolve the domain name to an IP address.
37 ms
TCP Connect TCP Connect — time to establish a TCP connection to the server.
0 ms
TLS Handshake TLS Handshake — time to complete the HTTPS encryption handshake.
4 ms
Time to First Byte Time to First Byte — how long the server takes to respond with the first byte of data.
42 ms
Total Time Total request time from DNS lookup through full response.
42 ms

Connection waterfall

DNS Lookup 37 ms TCP Connect 0 ms TLS Handshake 4 ms Server Processing 1 ms Content Transfer 0 ms
A+
CDN & Delivery
Fastly (HIT)
PASS
Fastly (HIT)
Info::
Site is served via Fastly CDN
Got: x-served-by: cache-toj-leto2350032-TOJ
Info::
CDN cache status: HIT
CDN Detected: Fastly
Provider Fastly Cache Status HIT Evidence x-served-by: cache-toj-leto2350032-TOJ
All checks on this page are automated. Results are estimates - run targeted manual reviews when the score affects a release decision.

Send Feedback