Which Pages Does Perplexity Crawl? It Skips Automatically Generated Pages
Published
This article was first published in Korean by Intrix Lab on August 10, 2026.
When we monitored two sites over the same 30 days, Perplexity skipped all the detail pages on one of them. It made 0 crawls of hospital detail pages on a medical information site, while visiting business detail pages on a restaurant information site 2.75 times per page (as of 2026-08-10).
Could it have been blocked?
No. We checked three things in sequence.
The robots.txt file explicitly allowed Perplexity's crawler, and nothing in front of the server was blocking it. Nor was it avoiding the site altogether. During the same period, it crawled the blog 146 times and treatment guides 82 times.
In other words, it filters by page type rather than excluding the site. Even within the same site, it retrieves human-written articles, while the tens of thousands of hospital detail pages automatically generated from public data receive 0 crawls. The business pages on the other site cover 687 places curated by people, and it crawls those actively.
Does this vary by bot?
Some bots show the opposite pattern. GPTBot crawled detail pages on the medical site 0.55 times per page, but only 0.12 times per page on the restaurant site. That is the complete opposite of Perplexity's pattern.
This is why "AI crawlers" should not be treated as a single group. The assessment of the same site reverses depending on which bot you use as the reference.
What does this mean in practice?
The assumption that creating more pages leads to more AI citations needs another look. Pages generated in bulk may be passed over for crawling. If they are, citation cannot even begin.
These are observations, however, not evidence of causation. The two sites differ in size (about 3,600 places versus 687), launch date, domain recognition, and topic. This data cannot distinguish whether the reason is the medical topic or automatic generation. The circumstantial evidence points a little more strongly toward the latter, because human-written articles on the same site are crawled. We do not guarantee citations or visibility.
Frequently asked questions (FAQ)
Q. Which sites did you monitor?
Kdermzip (kdermzip.com, dermatology information) and Loneat (loneat.kr, curated restaurant recommendations for drinking alone), both of which we operate ourselves. We did not use client data.
Q. Can sites of different sizes be compared?
We did not directly compare total visit counts. We compared values adjusted for differences in site size.