How Can We Test GEO's Effect? Starting a Randomized Controlled Trial of 1,991 Hospital Pages
Published
This article was first published in Korean by Intrix Lab on August 10, 2026.
To find out whether a GEO intervention works, we need to randomly split comparable pages into two groups, change only one group, and decide how to calculate the results before seeing them. A before-and-after comparison will not work. Weekly AI crawler visits fluctuate by as much as 66-fold, making it impossible to separate the intervention's effect from what would have happened that week anyway.
On August 10, 2026, at 2:04 p.m., we started an experiment by randomly dividing 1,991 hospital pages into two groups (998 control, 993 treatment). Results are due on October 5. We are writing this article before then.
Why is a before-and-after comparison insufficient?
AI crawlers do not visit in small, steady numbers each day. They go quiet for a while, then arrive in bursts. Weekly visits to our site ranged from 268 to 17,661 (measured from 2026-06-01 through 2026-08-10). The variation is even greater for individual bots. ClaudeBot made 8 visits 3–4 weeks ago, then 7,048 in the most recent 2 weeks.
With data like this, saying "visits increased after we made changes" tells us nothing. Visits might have increased without any changes. We need a group that remains unchanged over the same period so we can measure the intervention's effect by comparing the groups.
What did we change, and what are we measuring?
We added two things to the treatment pages: a block at the top of the body that answers the question first, and subheadings that end with question marks. We did not invent facts. We changed only the order and wording of information already on the pages. We left the control group untouched.
We are measuring AI crawler visits. We count four bots: GPTBot, ClaudeBot, OAI-SearchBot, and PerplexityBot.
How do we know the measurement was not contaminated?
A file change can itself signal crawlers to return. They may revisit simply because something changed, rather than because the content improved. We therefore checked in advance that the deployment did not independently create such a signal.
We also checked the calculation method in advance against a historical period with no intervention and confirmed that it returned 0 when there was no effect.
What if the experiment produces no result?
We will publish it as it is. That commitment is in the preregistration document.
Here is one example. Our free audit reports had recommended creating a file called llms.txt, describing it as a way to tell AI about a site's structure. But when we checked 75 days of logs for our site, we found 0 requests for that file across 1,622 visits by 10 types of AI crawlers (as of 2026-08-10). The same bots fetched robots.txt 413 times and sitemaps 285 times.
We therefore changed the report wording. We still provide a draft, but now state that its effectiveness has not been confirmed, include the numbers above, and explain that it is not a priority.
How far can these results be trusted?
The sample shrank to 13% of the original plan. Many URLs in the logs were pages that did not actually exist, and one planned intervention was already present on every page, making a comparison impossible. We made both corrections before seeing the results and documented their timing and reasons.
We are therefore making this commitment in advance: even if the difference is not statistically significant, we will not write that "there is no effect." We will report confidence intervals. If an effect of meaningful size falls within the interval, we will write that "this sample could not establish the effect." Insufficient statistical power and no effect are different things.
The experiment covers one site that we operate. There is no guarantee that other sites will produce the same results. We do not guarantee rankings, citations, or specific outcomes. This experiment can answer only how much one intervention changed crawler behavior on one site. We will publish the results regardless of statistical significance.
Frequently asked questions (FAQ)
Q. What is preregistration?
It is a process of documenting and publicly fixing the hypothesis, sample, calculation method, and conditions that would invalidate the experiment before seeing the results. It prevents choosing a favorable interpretation after seeing the results. The same approach is used in clinical trials.
Q. Why disclose the site used in the experiment?
Because the experiment involves only one site, others cannot verify it unless we identify that site. It is Kdermzip (kdermzip.com), a dermatology information site we operate directly. We did not use client data.
Q. When and where will the results be published?
Data collection ends on October 5, 2026. We will publish every metric on this Lab page, regardless of statistical significance.