Setting Up a Holdout Group Test for On-Site AI Tools

Setting up a holdout group test for an on-site AI tool means separating purchases the tool caused from purchases it merely touched. For Kinect or a generic on-site chatbot, the defensible approach is a brand-controlled comparison against first-party orders. Measuring incremental value requires adequate group size and duration, control isolation, and a preselected metric.

Why standard A/B testing cannot answer the question a DTC brand actually needs to ask about its on-site AI tool

A shopper sees an AI assistant, asks a question, and buys a product, so the vendor credits the assistant with the order. Yet the shopper may already have intended to buy. High-intent shoppers disproportionately engage assistants. Engaged sessions would have converted at a higher rate even without the tool. Comparing those sessions with the rest of the audience therefore confuses self-selection with impact.

The dashboard compounds this problem when it reports only on sessions that engaged the assistant. AI sales reps built for the job, such as Kinect, have the same limits on what they can measure as generic on-site chatbots. An A/B split that compares engaged-order attribution against a brand’s baseline still relies on what the tool reports unless the brand controls suppression and checks results through its own transaction records.

Standard A/B testing compares variants, such as creative or targeting, among people who are exposed to the tool. It does not establish the tool’s contribution unless the comparison withholds the tool itself. Nor does a rise in overall conversion after launch establish causation, since a seasonal promotion, email send, or organic PR event could explain it. The test must establish how many purchases would disappear without the assistant.

What a holdout test measures and how it differs from the attribution models most brands already run

A holdout test answers that question by withholding the AI tool from a representative slice of the audience while the rest receives it as usual. For on-site AI, the exposed group consists of users who see the assistant. The holdout consists of users who visit without it, and Shopify order data or equivalent first-party records supplies the outcomes for comparison. Last-click, multi-touch, and view-through attribution instead give credit to touchpoints near a conversion without showing whether those touchpoints caused it. A tool can consequently report a large ROAS multiple even when many credited purchases would have happened anyway. Holdout experimentation supplies the causal foundation against which platform attribution and MMM can be calibrated. That distinction also makes the holdout useful beyond a one-time assessment, because it tests the baseline assumptions behind other reporting. However, withholding access creates a trustworthy counterfactual only when the groups remain comparable and the measurement is sufficiently precise.

Choosing the right holdout type for an on-site AI deployment

For most DTC brands testing an on-site AI tool, user-level holdout is the right starting design. Suppression can operate at the session or cookie level without geographic constraints, and Shopify order data maps cleanly to user sessions. More importantly, the question concerns a shopper’s conversion. That makes user-level testing the natural fit for a storefront deployment such as Kinect’s brand agent.

The other approaches answer different practical needs. A geo holdout assigns different treatments to matched regions and can suit cross-channel testing or cases where user-level suppression is impossible, although it needs meaningful monthly channel spend to avoid noisy results. A time-based on/off test runs the tool for one period and removes it for another, making it vulnerable to seasonality, email activity, and organic momentum. It is therefore a rough sanity check. An MMM-validated matched market constructs a statistical counterfactual from historical data, but its modeling demands make it unsuitable as a first test for most DTC operators.

Whatever test design the brand chooses, it must withhold the tool itself. Excluding a user from a report does not remove the user’s exposure. A shopper who sees the assistant even once has been exposed, whether or not that shopper engages it.

How to size the holdout group so the result is actually statistically reliable

Once the user-level structure is chosen, the next question is whether the withheld group will generate enough conversions to detect a useful difference. An undersized holdout produces a number that cannot be separated from random variation. An oversized group unnecessarily withholds a potentially useful tool from more shoppers. The target is therefore a calculation based on expected lift and conversion volume.

That calculation starts with weekly conversion volume and the minimum detectable effect. Brands with low conversion volume need to put a larger share of users in the holdout, run the test longer, or do both. A larger share can accumulate control-group events faster, while a smaller share reduces the immediate opportunity cost of withholding the assistant. Neither choice removes the need for enough events to support the planned comparison.

For brands running its agent, Kinect can assist with this calculation at setup. The starting recommendation is a holdout sized to detect the lift the tool is expected to produce. This prevents a common failure in which a team chooses a share by instinct, waits weeks, and receives a confidence interval too wide to act on. Duration must then follow from the required evidence.

How long to run the test, and what can contaminate a clean result

The test must run long enough to accumulate conversions and give both groups a comparable opportunity to purchase. Set a minimum of four weeks and extend the window to cover at least two full purchase cycles for the category. A skincare brand may have a six-to-twelve-week repurchase window, while furniture may involve a six-to-twelve-month consideration cycle.

A longer window also creates more opportunities for outside events to disturb the comparison. Email and SMS sends add separate conversion touchpoints, so how these are spread across the groups needs a close look. Competitor promotions can affect the groups unevenly when audience overlap differs, while platform algorithm changes can do the same when paid audiences overlap with the holdout definition. Organic PR or influencer traffic can distort results if the holdout over-indexes in the affected channel. Shared activity is not automatically contamination, but unequal effects undermine comparability.

Buyer-agent sessions require a separate decision before launch. Decide how sessions from ChatGPT, Gemini, and Perplexity will be identified, then exclude them from the primary analysis or treat them as a separate segment. This makes agent-readiness a measurement concern as well as a storefront concern.

Log every email, SMS, paid media, and promotional event by date throughout the window. Overlay that record against the test timeline before drawing conclusions, rather than trying to reconstruct potential confounders after a surprising result.

Selecting the primary metric before the test launches, and why "new customer conversion rate" is the strongest choice for an on-site AI tool

Isolation protects the comparison from outside influences, while selecting the primary metric in advance protects it from selective interpretation. Picking the winning metric after seeing the results turns the test into confirmation bias. For an on-site AI tool, new customer conversion rate is the strongest primary choice because it focuses the test on acquisition. The clearest incremental case is a buyer who would not have purchased without help resolving a hesitation or answering a sizing question.

Overall conversion rate is easy to measure, but returning customers can dilute that acquisition question. Their high intent can be especially misleading when reports give credit for assistant engagement instead of comparing conversion rates across the groups as assigned. Revenue per visitor shows changes in order value as well as conversion and is useful as a secondary metric, though the mix of returning customers still affects how to read it. Average order value among shoppers who buy can test upselling or cross-selling, but it leaves out conversion volume and so cannot be the primary measure.

Other secondary signals to track are session-to-cart rate, return rate, and customer satisfaction. Conversation topics show what product pages leave unexplained, but they do not measure causal performance. Through conversation logs and shopper intent signals, Kinect gives operators a view of what shoppers are asking. That information remains useful as a supplement to the metric chosen before launch, regardless of whether the primary conversion result shows lift.

Reading the result without overstating incremental lift

Once both groups have completed the test window, subtract the holdout conversion rate from the exposed conversion rate, divide the difference by the holdout rate, and express the result as a percentage. Use the preselected primary metric and Shopify order records or equivalent first-party transaction data. The assistant’s dashboard credits orders that include purchases belonging to the baseline, while incremental lift measures the difference above that baseline.

The result must meet the confidence level set before launch, usually in the high eighties to mid-nineties percent range. A result below that threshold is inconclusive. If a sufficiently precise comparison shows materially similar conversion rates, the test has found no incremental conversion benefit at the current configuration. That is different from an underpowered test that cannot distinguish a useful effect from noise.

Kinect provides engaged-shopper cohort data against a brand’s baseline through an A/B split, but engagement alone does not define a clean holdout. The causal comparison still requires actual tool suppression and first-party outcomes for the exposed and withheld audiences.

A null result is not a failed test. Before turning the tool off, examine conversation logs for product data gaps, unanswered policy questions, or friction in the checkout handoff.

Putting the test design together as a pre-launch sequence

First, choose the holdout type, usually user-level for an on-site AI deployment, and document why geo or time-based alternatives were ruled out. Next, work out weekly conversion volume, set the minimum detectable effect, and use those to size the holdout. Then set the start and end dates, allowing a minimum of four weeks and at least two full purchase cycles.

Before traffic begins splitting, record new customer conversion rate as the primary metric, the secondary measures, and the confidence threshold. Create a shared contamination log for email, SMS, paid media, and promotional events, and assign someone to update it weekly. Configure zero exposure for holdout users and verify the suppression logic before launch. Reduced exposure is not a control condition for testing the effect of removing the assistant.

Next, document how buyer-agent sessions will be identified and whether they will be excluded or analyzed separately. Confirm Shopify order records or equivalent first-party transactions as the source of truth, explicitly ruling out the tool’s dashboard as the primary input. Where the deployment uses Kinect, setup assistance can support the sizing calculation without changing the brand’s responsibility for the test design. Each decision needs to precede the result it governs.

After the test, read the lift number and then the conversation log. What shoppers asked, where the assistant could not answer confidently, and where sessions dropped after an interaction point toward specific product data gaps. Those gaps matter because measurement alone cannot make an assistant more useful. A well-designed holdout establishes whether the tool creates incremental value, and the conversation record shows what to fix before the next test.