US market research

Repeat the Same Questions or Ask More? A Reanalysis of 5,108 AI Responses

Published

This article was first published in Korean by Intrix Lab on October 7, 2026.

Asking more questions was more accurate. As of 2026-10-07, we reanalyzed 5,108 AI responses Intrix collected in August and September. When measuring AI visibility scores with the same number of calls, designs with more questions had less error than designs that repeated the same questions more often. The direction was consistent across all 4 studies and 11 model combinations. With 400 calls, the standard error was 3.26 percentage points for 40 questions asked 10 times each, versus 1.50 percentage points for 400 questions asked once each.

What did we set out to check?

Peec, which makes AI search analytics tools, recommended "tracking more questions rather than running the same question repeatedly," based on 120 million conversations. Its claim was that, for the same number of calls, broader question coverage produces less error, and a design with 5 repetitions has about 1.5 times the error. Intrix had also been conducting assessments by asking each question 10 times, so we checked whether our data showed the same pattern.

We made no new AI calls. We reread 4 studies collected between 2026-08-24 and 09-22. The industries were design support programs, personal rehabilitation proceedings, divorce, and branding. Each study had 36 or 50 questions, with each question asked 10 or 20 times. We excluded responses that ended in errors.

How did we compare the designs?

For each study, we used the visibility scores of the 10 most-cited domains as reference values. Within the same call budget, we varied the combination of question count and repetition count and compared how far the results deviated from those reference values. We fixed the analysis design before performing the calculations.

How much does error fall with fewer repetitions and more questions?

These results compare designs within a budget of 400 calls, using a study that asked 50 questions 20 times each.

Design (questions × repetitions)Average error (RMSE)Top 3 domain match rate
20 × 202.57 percentage points0.15
40 × 101.14 percentage points0.53
50 × 80.43 percentage points0.87

With the same 400 calls, increasing the number of questions from 20 to 50 reduced error to about one-sixth. The rate of correctly identifying which domains were in the top 3 also rose from 0.15 to 0.87.

Why is question count more effective than repetition?

Most of the variation in whether a domain was cited came from which question was asked. Asking the same question again did not change the result much, so repetition spent calls remeasuring something already measured.

Calculating under Peec's comparison conditions gave 1.61–1.87 times the error for the design with 5 repetitions. The direction matched Peec's claim of about 1.5 times, with a larger difference in our data.

How much does the current 10-repetition design cost in precision?

These are standard errors averaged across the top domains in 11 model combinations.

CallsDesign (questions × repetitions)Standard error
40040 × 103.26 percentage points
400100 × 42.26 percentage points
400200 × 21.79 percentage points
400400 × 11.50 percentage points
2,000200 × 101.46 percentage points

The precision of 2,000 calls asking 200 questions 10 times each was similar to that of 400 calls asking 400 questions once each. If the goal is to produce one accurate score, it is better to redirect calls from repetitions toward more questions.

This does not mean repetition is useless. Repetition is needed to assess the consistency of a single question, such as "Are we cited every time for this question?" To find out which questions make a difference in AI responses for your industry, you can start with a First Measurement.

How far can we trust these results?

The reference values come from our full available sample, not the true population. Each study had a fixed set of 36 or 50 questions, so figures for 100 or more questions are calculations that assume new questions of the same kind can continue to be drawn. If similar questions are included, the actual benefit will be smaller. We also did not account for the cost of creating and reviewing questions.

The analysis covers 4 Korean-language studies and only four industries. We could not test Peec's other claim that noise arising over time can be filtered out through trend analysis, because we had no records of asking the same questions again on different days. This analysis concerns measurement design and does not guarantee any particular ranking or citation.

Frequently asked questions (FAQ)

Q. Should we ask each question only once, then?
If the only goal is the accuracy of one score, yes. Within the same 400-call budget, asking 400 questions once each produced the smallest error. Repetition is needed to see whether you are cited every time for a particular question, so it is better to define the goal first and allocate calls accordingly.

Q. What if it is difficult to create many questions?
These calculations assume new questions do not overlap with existing ones. Adding questions that only change the wording means measuring the same thing again, reducing the benefit. It is better to cover a wider range of distinct situations that real customers ask about.

Q. Can we trust the top 3 domain rankings?
They fluctuate when the call count is low. In the 50-question study, even with 400 calls, the design using 20 questions had a top 3 match rate of 0.15. When reviewing rankings, first check whether enough questions were included.

← Back to insights