User-agent: Yandex / Crawl-delay: 10 User-agent: BLEXBot / Crawl-delay: 10 User-agent: MegaIndex.ru/2.0 Disallow: / User-agent: Gluten Free Crawler/1.0 Disallow: / User-agent: SemrushBot Disallow: / User-agent: SEOkicks Disallow: / User-agent: sogou spider Disallow: / User-agent: dotbot Disallow: / User-agent: Nutch Disallow: / User-agent: Nutch CCBot: / User-agent: magpie-crawler Disallow: / User-agent: AhrefsBot Disallow: /checkout/ Crawl-delay: 30 User-agent: ADVANCEDHOSTERS-NET Disallow: /checkout/ Crawl-delay: 30 User-agent: * Crawl-delay: 5 User-agent: * Disallow: /404/ Disallow: /checkout/ Disallow: /checkout/standard/ Disallow: /find/ Disallow: /sitemap/ Disallow: /chmail/ Disallow: /microcart/ Disallow: /mvccont/ Disallow: /direct/ Disallow: /redirect/ Sitemap: http://www.span.com/sitemap.xml