# Robots.txt for Adv. Ian Joseph Pegado # Compliant with Bar Council of India Rules - No Solicitation User-agent: * Allow: / # Allow all major search engines User-agent: Googlebot Allow: / User-agent: Bingbot Allow: / User-agent: Slurp Allow: / User-agent: DuckDuckBot Allow: / User-agent: Baiduspider Allow: / User-agent: YandexBot Allow: / # Allow AI crawlers for AEO/GEO User-agent: GPTBot Allow: / User-agent: ChatGPT-User Allow: / User-agent: CCBot Allow: / User-agent: anthropic-ai Allow: / User-agent: ClaudeBot Allow: / User-agent: PerplexityBot Allow: / User-agent: Google-Extended Allow: / User-agent: Applebot-Extended Allow: / # Disallow private/admin paths (none exist in this static site) Disallow: /node_modules/ Disallow: /src/ Disallow: /.git/ Disallow: /*.json$ Disallow: /*.xml$ Disallow: /*.txt$ # Sitemap location Sitemap: https://ianjosephpogado.in/sitemap.xml # Host directive for Indian region Host: https://ianjosephpogado.in # Crawl-delay for respectful crawling Crawl-delay: 10