# Internal GSA appliance User-agent: gsa-crawler Disallow: User-agent: 008 Crawl-delay: 10 Disallow: / # Everyone else User-agent: * Disallow: /feedback/ Disallow: /readers/ Disallow: /api/ Disallow: /bugs/ Disallow: /rankit/ Disallow: /services/ Disallow: /spamscan/ Disallow: /testing/ Disallow: /m/ Disallow: /scrape/ Disallow: /comments/ Disallow: /search/index Disallow: /restaurants/search