Which AI Crawlers Actually Respect robots.txt in 2026? A Real IP/Proxy Test
A reproducible canary-URL test of how Claude, ChatGPT, Gemini, Grok, and Perplexity actually handle robots.txt, verified against each company's own current documentation.
Nadir Yaqoob is the founder of South Asia Digital, an SEO and AI search agency working with SaaS and Shopify brands since 2019. He leads strategy on technical SEO, content and AI visibility, helping brands get found in Google and recommended in ChatGPT, Perplexity and AI Overviews.
A reproducible canary-URL test of how Claude, ChatGPT, Gemini, Grok, and Perplexity actually handle robots.txt, verified against each company's own current documentation.
Google renders JavaScript before indexing. GPTBot, ClaudeBot, and PerplexityBot mostly don't. A reproducible audit for checking whether your product's price, variants, and availability survive that gap.
A real first-90-days playbook for AEO/GEO, built from our own research into how agencies are actually approaching these engagements: baseline prompt-set construction, citation vs. mention tracking, a realistic re-testing cadence, and how to set honest client expectations.
The real current steps for adding multiple stops in Google Maps on Android, iPhone, and desktop, the actual 9-stop limit straight from Google's own documentation, what Optimize Route really does, and what to check when Add Stop doesn't show up.
11 real productivity Chrome extensions, verified against their actual Chrome Web Store listings, picked for the specific friction each one removes, including an honest downgrade for the one popular pick that doesn't deserve its popularity.
The real difference between a Google manual action and an algorithmic penalty, how to diagnose which one you have using Search Console and Google's own confirmed update dates, and what recovery actually requires for each, sourced directly to Google's own documentation.
The best social media analytics tool depends on what you actually need to measure. Socialinsider is particularly useful for competitor benchmarking, while Hootsuite works…
A real, anonymized client case study: a US-based handmade and forged knife Shopify store held page-1 rankings for its real target keywords through two confirmed Google spam updates in roughly two months, the result of a structured, ongoing monthly SEO plan run for years, not a one-time project.
A real, anonymized client case study: an Australia-based restaurant SaaS platform got a new site built with technical SEO and brand-entity work in parallel from day one, and saw real organic visibility from launch instead of the usual multi-month wait.
Real, reusable prompts for using ChatGPT, Claude, Gemini, and Perplexity for keyword research: seed expansion, intent clustering, long-tail discovery, and competitor gap analysis, plus why AI ideas still need real search-volume verification.