User-agent: SemrushBot Disallow: / User-agent: serpstatbot Disallow: / User-agent: DotBot Disallow: / User-agent: Baiduspider Disallow: / User-agent: Bytespider Disallow: / Crawl-delay : 864000 Comment: This crawler violates robots.txt rules. Stop crawling immediately. User-agent: Slurp Disallow: / User-agent: BLEXBot Disallow: / User-agent: MJ12bot Disallow: / User-agent: PetalBot Disallow: / User-agent: Amazonbot Disallow: / User-agent: AmazonBot Disallow: / User-agent: GPTBot Disallow: / User-agent: meta-externalagent Disallow: / User-agent: facebookexternalhit Disallow: / User-agent: DataForSeoBot Disallow: / User-agent: dataforseo-bot Disallow: / User-agent: * Disallow: /admin/ Crawl-delay: 10 # 필요한 경우에만 설정, 기본적으로 제거할 수 있음