User-agent: AhrefsBot User-agent: SemrushBot User-agent: MJ12bot User-agent: DotBot User-agent: ShapBot User-agent: DataForSeoBot User-agent: BLEXBot User-agent: PetalBot Crawl-delay: 30 Disallow: User-agent: * Disallow: