Can you trust an SEO ranking study? A business owner's checklist

An SEO study can be useful without revealing Google's formula. The problem starts when a pattern in search results is rewritten as a guaranteed action: top-ranking businesses have more reviews, therefore buying more reviews will move you up.

Before spending money on a study's conclusion, check whether the method actually answered the business question attached to it.

The six questions to ask

1. What exactly was measured?
2. Were like-for-like competitors compared?
3. Is the result a cause, an association or only a description?
4. What is the real sample size?
5. What did the study exclude or fail to measure?
6. Would the recommended action still make sense if the ranking claim is weaker than advertised?

1. What exactly was measured?

A label such as “competitor density” can sound clearer than the underlying data. In my Google Maps study, the measure was the number of different businesses returned across 25 search points. It was not a census of every business operating in the city.

That measure is still useful, but it changes the conclusion. The data can show that grids with more result turnover had smaller top-three footprints. It cannot prove that one new competitor caused another business's territory to shrink.

2. Were like-for-like competitors compared?

A Melbourne plumber and a Ballarat cafe are not fighting for the same customer or result. Pooling unrelated searches can manufacture a pattern.

A stronger comparison looks inside the same search point or the same industry and then checks whether the direction repeats. The Google Maps study compared reviews among businesses returned at the same point and checked the competitive-footprint relationship separately inside all 20 industries.

3. Is the result causal?

Businesses with more reviews may rank better because reviews help. They may also be older, better known, linked from more websites or located closer to demand. A search-result study observes all those things together.

“These things moved together” can guide an audit. It is not the same as “buy this and Google will move you up.”

4. What is the real sample size?

The Maps study recorded 231,907 business appearances, but those appearances came from 480 grids. Nearby points repeatedly show many of the same businesses. The honest headline sample is 480 grids.

A large row count does not create more independent evidence when the rows are repeated measurements of the same local market.

5. What was excluded?

Every useful headline has a boundary. The Maps study used businesses DataForSEO returned, so absent businesses were not counted in the distance bands. Its furthest top-three measure included businesses that won somewhere and did not create a value for businesses that never reached the top three.

Those facts do not invalidate the study. They stop the result being sold as a universal probability or radius.

6. Does the action still make business sense?

Review growth is commercially useful even if the ranking effect is smaller than hoped because reviews also affect trust and choice. A second office is different. If the location only makes sense under an unproven ranking-radius calculation, the business case is not strong enough.

Warning signs

• The study claims a universal Google weighting.
• A correlation is described as proof.
• The biggest number in the report is repeated observations rather than searches or markets.
• A complicated term is used without explaining the underlying count.
• The result produces an exact operational rule the data never tested.
• Limitations are hidden after the sales pitch.

What the first Maps write-up got wrong

The early public interpretation said competitors “decided” a business's catchment and converted a sampled furthest point into circles used to calculate how many branches could cover a city. The data did not show clean circles, so the branch-count calculation has been removed.

It also downplayed reviews. The final 480-grid sample found a positive review-count relationship in 17 of 20 industries. The honest correction is that review growth remains important, while the study cannot turn it into a guaranteed ranking weight.

What useful research should give an owner

• A practical question in ordinary language.
• An answer that stays inside what was measured.
• A clear reason the result affects leads, spend or location decisions.
• A small action or test rather than a guarantee.
• Enough method to check the claim.
• A visible list of what the study could not prove.

You can see that standard applied, including the corrected limits, in the Google Maps ranking study.