The Boring Math Behind a Synthetic Panel You Can Trust

Companies have big incentives to make the calibration of synthetic panels with real data sound like magic. They talk about proprietary models, novel architectures, and complex scoring rubrics. These things can improve synthetic panel performance, but they are not the primary driver of quality.

Synthetic panels offer instant predictions about what real people will say or do in response to given stimuli. Behind all of the “magic,” the biggest determinant of whether a prediction is right is not the cleverness of the model, but rather the quality of the data used to calibrate the model. “Garbage in, garbage out.”

Flashpoint.AI takes a transparent and rigorous approach:

First, from both quality and ease-of-use perspectives, we built the best platform for collecting high-quality proprietary data. This includes traditional methods and a patented Market Experiments tool that directly addresses the selection bias and say-do gap problems that have plagued research for decades.

Second, we validate our synthetic panels with a method that any statistician would recognize, because we believe that it is not possible to trust results that you can’t explain.

AI models will become increasingly advanced. We built a platform to take advantage of the rising tide of AI capabilities, let you collect and train with uniquely accurate data, and calibrate your own panels with that data.

How we get calibration data

Flashpoint.AI supports several ways to get fresh, proprietary, first-party data:

• A world-class survey tool, with built-in data quality and panel integrations that let you reach tens of millions of respondents globally
• A patented Market Experiments tool that is a totally new way to do research, and addresses selection bias and the say-do gap
• AI-moderated qualitative interviews
• Support for uploading data that you may have gathered elsewhere

Our Market Experiments is a unique way to run behavioral experiments at scale, anywhere in the world. It enables you to reach hard-to-reach audiences like High Net-Worth Individuals, Mass Affluent, Emerging Markets, B2B, and more.

It's not a survey, not synth, not AI-moderated qual, not a behavioral browser extension, not a clever approach to recruiting or data quality — it's a completely new way to test what real people really do, not what they say they do.

The tool is in Early Access. Schedule a call to learn more. For now, we’ll continue our discussion of how to use this data as the ground truth to calibrate uniquely effective synth.

How we validate: K-fold validation

K-fold validation is a statistical method for evaluating how well a machine learning model will perform on data the model has not seen. It works like this:

• Divide the survey into k groups of questions. We typically use k between 5 and 10. With k = 5, each group holds roughly 20% of the questions.
• Run k rounds. In each round, one group is set aside. We calibrate the synth on the rest, then measure how well it predicts the group it never saw.
• Rotate. Every group takes a turn as the held-out set, so every question in the survey is tested exactly once as new data.
• Average. The error across all k rounds becomes a single number.

For short surveys, we go to leave-one-out: each round holds back a single question. Same idea, k equals the number of questions.

We show the averaged number. No single lucky or unlucky split can distort the validation result.

We use a metric that you can read without a PhD

We measure total variation distance (TVD) between the synth's predicted answer distribution and the real one. TVD is the largest gap between the two across all answer options, on a scale from 0 (identical) to 1 (no overlap). A TVD of 0.05 means that on questions it was never calibrated on, the synth's answer shares are at worst five points off from real respondents.

We compute this on the unseen questions in every fold and average across folds. The ground truth can be traditional survey data, Market Experiments data, or other data that you upload to our platform.

What you see

• Calibrated. Out-of-sample error is under the threshold. The synth generalizes to questions it was not tuned on in the topic areas covered by the training data.
• Training topics. The list of topics present in the training data, which the synth can answer questions about with confidence.
• Needs more data. Error is above the threshold. The model is not the problem; the synth simply has not seen enough real signal yet. Collect more, and we recalibrate.

We focus on the data

We believe the best fix to a suboptimal synth is more high-quality data, not more complex algorithms on top of (often compromised) data.

We use the well-trodden cross-validation method so that you have robust and explainable validation. This data-centric approach to synth also allows us to ride the rising tide of core foundation model development.

We focus our effort where it compounds: helping you collect proprietary data that no one else has, and giving you a clear assessment of what it tells you.