How we measure
Every number in Pickory comes from real answers of AI assistants to real shopper questions. This page says where the questions come from, how often we ask, what the numbers mean and where they stop.
Where the questions come from
We start from your store: your products, categories and price range, read from your catalog and your own pages. Then we look for what people actually type: Google, Amazon and YouTube search suggestions in the US, and on paid plans Reddit and shopping forums. A question goes into your set when it asks for a product or a store to buy from, the way a shopper would ask ChatGPT. If we find no search signal for your category, we say so next to the questions instead of calling them “real”.
Your set stays the same from week to week (we swap a question only for a reason we show you), so this week can be compared with the last one.
How often we ask
- Free check: 10 questions in ChatGPT, once.
- Lite: 20 questions a week: ChatGPT twice each, Google AI Mode once each.
- Starter: 50 questions a week: ChatGPT twice each, Perplexity and Google AI Mode once each, and Claude once on a sample of 10 of them.
- Growth: 150 questions a week: ChatGPT three times each, Perplexity twice, Google AI Mode once, and Claude once on a sample of 30 of them.
Google AI Mode and Claude’s sample each have a row of their own in your report, with their own range; they are not added into the main number. On every paid plan we also search each question on Google once a week and note whether Google showed an AI Overview and whether it named you: one line of its own, never added to the numbers above.
We ask through each assistant’s official programming interface with web search turned on; for ChatGPT, on the model the free ChatGPT app uses. On Starter and Growth we also ask the most important questions on the paid ChatGPT model and show those answers apart. The answers are close to what a shopper sees in the app, not identical: the app also uses the shopper’s location, chat history and account. A failed answer is never counted as “not named”.
What “named in K of N answers” means
N is the number of real answers we got this week; K is how many of them recommended your store. We check every answer twice: a second AI reads it for the brands it recommends, and we also look for your exact name next to a link to your site. An answer that mentions you only to advise against you does not count as named.
Why every number has a likely range
Ask an AI the same question twice and you can get two different answers. So a single week is a sample, not a fact. Next to each number we show its likely range — a 95% confidence interval (the Wilson interval), which is the range where the true rate most likely is. With few answers the range is wide; it narrows as the weeks add up.
We call a change “up” or “down” only when this week’s range and the earlier one do not overlap. Anything smaller is shown as normal variation, so you never celebrate noise or worry about it.
More than a prompt
Asking ChatGPT one question tells you little: the answer changes from one ask to the next, and the question may not be one shoppers ask. Pickory adds four things: questions from real searches (above); a fixed set asked every week, with the likely range next to each number; a check through Shopify’s own entrance for AI shopping agents; and changes in your store that you approve, can undo, and see measured before and after.
How we check AI shopping agents
AI shopping agents reach Shopify stores through Shopify’s own agent entrance (the Universal Commerce Protocol). Once a day at most, we use it the way an agent does: we search your catalog with a shopper’s question for each of your main products, then put one option in a cart and cancel that cart at once — never a checkout, never an order. We compare what the agent sees with your store: is the product found by the question or only by its name, and are the price and stock the same. A failed try is shown as failed, never as “not found”.
Changes and “before → after”
Every change we make in your store (a product text, a fact saved as product data) waits for your approval and can be undone; Undo restores what was there. After a change we show the numbers for the questions it targets before and after, next to the same numbers for your other questions.
What we don’t claim
- We don’t promise a place in ChatGPT or any other assistant. Nobody can.
- When a change follows something you did (a new product text, say), we show the numbers before and after, and the same numbers for your other questions, but we don’t claim it was the cause: AI answers change for many reasons.
- Structured data and llms.txt help AI apps read your store; they are not shown to make AI recommend it, and we don’t sell them as if they did.
Questions about a number? Write to hello@getpickory.com — a real person answers.
See your own numbers