Case Study Methodology

The rules

1. Name the market. A result with no city cannot be checked.
2. Name the keywords. Aggregate growth with no query attached is not a finding.
3. State the window. Every published study reports a window of under 90 days.
4. Measure on a grid, not a point. See search grid methodology.
5. Rescan identically. Same points, same queries, same method.
6. Publish weak results with strong ones. The Brooklyn study (+129% across three keywords) and the Nashville study (+153%, including one keyword that moved +4%) are published alongside the San Jose study (+316%).
7. Label the instrument. These are first-party measurements by the agency that did the work.
8. Do not present visibility as revenue. Grid movement is a leading indicator.

The published set

Where this set is thinner than it looks

These are the same weaknesses the measurement standards grade this network on, where three of seven rows score “partly”. Publishing that assessment costs something and is the reason the rest of the figures can be taken seriously.

What a firm should ask about any case study

• Which city, which keywords, which dates?
• Measured from where, one point or a grid?
• What was the starting position, not just the change?
• Did anything else change at the same time?
• What happened to the clients not shown?

What each case study reports

Each published case study carries the same fields, and the uniformity is the point: a reader can compare them and can see when one is thinner than another.

Every entry names the city and state, so the market is identifiable and its competitiveness can be judged. Every entry lists the tracked keywords individually with their own movement, rather than a single aggregate that could conceal a set where one term carried everything. Every entry states the window, and all six fall under ninety days. Every entry reports a total grid gain measured by scanning a grid before the work and rescanning the identical grid after.

The keyword sets differ between cities because the statutory vocabulary differs. Cleveland is measured partly on OVI terms and Fort Worth partly on DWI terms, since those are what those states call the offence. A uniform national set would have measured terms nobody in those cities searches.

Whether the results would replicate

The question a reader should ask of any case study is whether the same work in a seventh city would produce a comparable figure, and the honest answer here is that nobody knows.

Six cities is a small set, all of them clients who engaged and stayed, which is a selected group rather than a sample. No unsuccessful engagements appear, and their absence is not evidence they do not exist. A set of published case studies is always the successes, on this network as everywhere else, and treating it as a success rate is a mistake the format invites.

What the set does establish is narrower and still useful: that the method was applied in six identifiable markets, that the movement was measured on a described instrument, and that the figures can be checked by anyone willing to rescan the same grids. That is a claim about what happened rather than a prediction, and it is the claim made.

Reading the spread across cities

The six published figures range widely, and the range is informative in a way a uniform set would not be.

Markets differ in competitive density. A city where several established defense firms have invested for years is harder ground than one where the incumbents have neglected their profiles, and identical work produces different curves. A set of case studies reporting near-identical gains across six unrelated cities would be the less believable result, because it would imply the market made no difference.

The starting position matters as much as the market. A firm beginning from near-invisibility has more room to move in percentage terms than one already holding pack positions, so a large figure can represent a smaller absolute change than a modest figure elsewhere. Reading the keyword-level movements alongside the totals is what separates the two, which is why each city lists its terms individually.

The parameters behind each scan

A before-and-after comparison is only meaningful where both scans are the same instrument, so five parameters are held constant between a baseline and its rescan.

The grid’s centre point stays fixed, since moving it toward the office flatters every later reading. The dimensions and spacing stay fixed, because a tighter grid over a smaller area concentrates on the ground a firm already holds. The query list stays fixed, which is where agency reporting most often drifts as hard terms quietly leave the set. The scan dates are recorded, since a figure without a date cannot be checked. And every point counts, including the ones where the firm did not appear, because averaging only the successful points discards the information about reach.

These are the parameters a firm should require of any agency presenting a grid comparison, and they are the ones this network publishes for its own figures. The method in full is at search grid methodology.

What these figures do not establish

Four limits apply to every figure published here, and they are stated with the figures rather than on a separate page.

They are first-party. The measurements were taken by the agency that performed the work, and no independent party audited them. That is disclosed in the source bibliography, where they are labelled first-party alongside everything else the publisher produced.

They measure visibility. A grid gain is a leading indicator of cases signed and not a count of them, and the client’s own intake numbers remain the measure that matters.

They are percentages against each firm’s own starting position, so a large gain from a low base is a smaller absolute movement than the number suggests. A firm going from nearly invisible to modestly visible can report a striking figure.

They describe six cities in a ninety-day window. They are not a forecast for a seventh city, and competitive density varies enough between markets that the same work produces different curves, which the published range itself shows.

Why publish the method at all

Most agency case studies report a number and stop, and the reason is that the number reads better without its conditions. Publishing the method makes the figures easier to attack, which is the argument for doing it.

A figure whose grid parameters, keyword set and window are published can be checked by rescanning, and it could be contradicted. A figure with none of those attached cannot be contradicted because there is nothing to test, and a claim that cannot be wrong is not evidence of anything. The standard this sets is the one the vendor quality guide asks readers to apply to any agency, including this network’s publisher.

See also: Search grid methodology · SEO measurement standards · Agency evaluation reference