# Robots.txt for React India Website # https://reactindia.io # Optimized for AI Search Engines, SEO Tools & LLMs # ============================================ # UNIVERSAL ACCESS - Allow all crawlers # ============================================ User-agent: * Allow: / # ============================================ # SEARCH ENGINE BOTS # ============================================ User-agent: Googlebot Allow: / User-agent: Googlebot-Image Allow: / User-agent: Googlebot-Video Allow: / User-agent: Googlebot-News Allow: / User-agent: Google-InspectionTool Allow: / User-agent: Bingbot Allow: / User-agent: BingPreview Allow: / User-agent: Slurp Allow: / User-agent: DuckDuckBot Allow: / User-agent: Baiduspider Allow: / User-agent: YandexBot Allow: / User-agent: Sogou Allow: / User-agent: Exabot Allow: / # ============================================ # SEO & MARKETING TOOLS # ============================================ User-agent: AhrefsBot Allow: / User-agent: SemrushBot Allow: / User-agent: MJ12bot Allow: / User-agent: DotBot Allow: / User-agent: Screaming Frog SEO Spider Allow: / User-agent: rogerbot Allow: / User-agent: SiteAuditBot Allow: / User-agent: SEOkicks-Robot Allow: / User-agent: SEOkicks Allow: / User-agent: sistrix Allow: / User-agent: JEEVES Allow: / User-agent: Moz Allow: / User-agent: MozBot Allow: / # ============================================ # AI & LLM BOTS # ============================================ User-agent: GPTBot Allow: / User-agent: ChatGPT-User Allow: / User-agent: Claude-Web Allow: / User-agent: ClaudeBot Allow: / User-agent: anthropic-ai Allow: / User-agent: PerplexityBot Allow: / User-agent: Google-Extended Allow: / User-agent: cohere-ai Allow: / User-agent: Applebot-Extended Allow: / User-agent: CCBot Allow: / User-agent: DuckAssistBot Allow: / User-agent: Google-CloudVertexBot Allow: / User-agent: YouBot Allow: / User-agent: OAI-SearchBot Allow: / # ============================================ # SOCIAL MEDIA BOTS # ============================================ User-agent: Twitterbot Allow: / User-agent: facebookexternalhit Allow: / User-agent: FacebookBot Allow: / User-agent: Meta-ExternalAgent Allow: / User-agent: LinkedInBot Allow: / User-agent: Pinterest Allow: / User-agent: Slackbot Allow: / User-agent: WhatsApp Allow: / User-agent: TelegramBot Allow: / User-agent: Discordbot Allow: / # ============================================ # ADS & ANALYTICS BOTS # ============================================ User-agent: AdsBot-Google Allow: / User-agent: AdsBot-Google-Mobile Allow: / User-agent: Mediapartners-Google Allow: / User-agent: Amazonbot Allow: / User-agent: Quora-Bot Allow: / # ============================================ # ALLOWED PATHS - Main Content # ============================================ Allow: / Allow: /about Allow: /benefits Allow: /schedule Allow: /speakers Allow: /workshops Allow: /sponsors Allow: /partners Allow: /volunteer Allow: /cfp Allow: /jobs Allow: /articles Allow: /conferences Allow: /team Allow: /accessibility Allow: /scholarship-program Allow: /talks Allow: /meetups Allow: /guides Allow: /cities Allow: /topics Allow: /compare Allow: /past-speakers Allow: /contact # Allow crawling of static assets Allow: /favicon.ico Allow: /logo.svg Allow: /*.css Allow: /*.js Allow: /*.png Allow: /*.jpg Allow: /*.jpeg Allow: /*.gif Allow: /*.svg Allow: /*.webp Allow: /*.ico # Allow crawling of LLM context files Allow: /llms.txt Allow: /llms-full.txt # ============================================ # DISALLOWED PATHS - Private/Internal # ============================================ # API routes (except AI and OG endpoints) Allow: /api/og/* Allow: /api/ai/* Allow: /api/llm-sitemap.json Disallow: /api/ # Build artifacts Disallow: /_next/static/ Disallow: /_next/data/ # Admin pages Disallow: /admin/ # Legal pages (low SEO value, still accessible) Disallow: /privacy Disallow: /terms Disallow: /refund Disallow: /code-of-conduct # Live streaming (temporary content) Disallow: /live Disallow: /stage # ============================================ # CRAWL OPTIMIZATION # ============================================ # No crawl-delay for good bots (we welcome fast crawling) # Host directive for canonical domain Host: https://reactindia.io # LLM Context Documentation # For AI assistants and crawlers # Primary: /llms.txt (API docs) # Full context: Root llms.txt (comprehensive project docs) # Sitemap References - Enterprise Multi-Sitemap Strategy # Main sitemap index with all sitemap references Sitemap: https://reactindia.io/sitemap.xml # Individual sitemaps for specialized indexing Sitemap: https://reactindia.io/sitemap-pages.xml Sitemap: https://reactindia.io/sitemap-landing.xml Sitemap: https://reactindia.io/sitemap-speakers.xml Sitemap: https://reactindia.io/sitemap-articles.xml Sitemap: https://reactindia.io/sitemap-meetups.xml Sitemap: https://reactindia.io/sitemap-jobs.xml Sitemap: https://reactindia.io/sitemap-images.xml Sitemap: https://reactindia.io/sitemap-videos.xml # AI/LLM Structured Data Endpoints # For AI citation, RAG systems, and knowledge graphs # Full context: https://reactindia.io/llms-full.txt # Quick reference: https://reactindia.io/llms.txt # Facts API: https://reactindia.io/api/ai/facts # Context API: https://reactindia.io/api/ai/context # LLM Sitemap: https://reactindia.io/api/llm-sitemap.json