Cloudflare has accused AI search startup Perplexity of deliberately evading website restrictions designed to block its web crawlers. The internet infrastructure company said in a report that Perplexity’s bots attempt to bypass protections after encountering blocks.
Cloudflare said customers complained that Perplexity’s bots continued to access their websites despite blocking instructions in robots.txt files and additional Web Application Firewall (WAF) rules. To investigate, Cloudflare created test domains with the same restrictions.
How is Perplexity Doing It?
According to the report, Perplexity initially identified itself with user agents such as “PerplexityBot” or “Perplexity-User.” When blocked, the startup allegedly switched tactics, disguising its bots as Google Chrome on macOS and rotating its IP addresses to avoid detection. Cloudflare also claimed that Perplexity changed its autonomous system networks (ASN) to sidestep access controls.
Cloudflare said the stealth crawling activity spanned “tens of thousands of domains and millions of requests per day.” The company responded by delisting Perplexity as a verified bot and introducing new methods to block the startup’s crawlers.
Perplexity’s Response
Perplexity spokesperson Jesse Dwyer dismissed Cloudflare’s claims as a “publicity stunt” and said the blog post included “a lot of misunderstandings.”
The dispute comes as AI companies face growing scrutiny over their data collection practices. Cloudflare CEO Matthew Prince has warned that unregulated AI scraping poses an “existential threat” to publishers. Last month, Cloudflare began blocking AI crawlers by default and offered website owners the option to require payment for access.