Summary
A live district poll costs tens of thousands of dollars and takes a week or more to collect responses. Because of this time and cost, campaigns ration them and make do with the staff's intuition to fill in the gaps.
A major U.S. political party wanted to know whether synthetic polling calibrated with real primary research data could substitute. We started with the campaign's own data, from their 25-question phone surveys, in competitive state legislative districts.
Uncalibrated, our synthetic panel matched the phone poll on 80.5% of answers. One calibration pass with real response data brought the average out-of-sample accuracy to 91.3%. A prototype of our next calibration approach, scored on the same benchmark, reached 96.1%.
The problem with polls
Voter sentiment can fluctuate with news events, and the lag of designing, fielding, weighting, and delivering phone polls leaves campaigns in a constant state of catch-up. The next poll is weeks away and costs tens of thousands more.
As a result, campaigns ask fewer questions than they have, and they ask as rarely as possible. Few people in the campaign have the opportunity to use this data to inform decisions. Speechwriting, counter-messages, the decision to move field staff, and other important decisions run on stale polls and instinct.
What Connected Synth is
A synthetic panel is a simulated population that takes your survey. Uncalibrated, it answers from the model's generic training data, which makes it “over-averaged.” This can be good enough for pretesting a questionnaire, but it's not sufficient for steering a campaign. In our case, the uncalibrated baseline had double digit variance from real survey responses.
Flashpoint.AI's platform made it easy to conduct primary research, whether through surveys or our patented Market Experiments tool. Then, our Connected Synth can use that data to calibrate synthetic panels, continuously “grounding” them in real data.
Our platform retrieves demographic data to define the characteristics of the population, creates a synthetic mirror of the population, and tests whether the synthetic panel can recreate the distribution of answers from the real population.
The benchmark
In this case, we created a benchmark with the party's own phone poll, which includes around 500 completes per district. In each of the six districts, the committee's questionnaire (ballot tests, favorability thermometers, vote motivation, issue and message batteries) runs through the synthetic panel three ways:
- Uncalibrated. No seed data. This is the baseline every calibration is measured against.
- Calibrated with the district phone poll, then run against the questionnaire again.
- Reset, then calibrated with Flashpoint.AI's own data — a statewide online survey and revealed-preference research — and run a third time.
Each calibration run starts from a fresh baseline, and is scored against the live phone poll.
The results
See below the accuracy relative to the live phone poll, across the full out-of-sample questionnaire:
Uncalibrated baseline
80.5%
First calibration pass (mean of three runs)
91.3%
Next-generation engine (in testing)
96.1%
The most important number in a campaign poll is the head-to-head ballot margin. Here is how far the synthetic panel's margin missed the live poll's:
Uncalibrated
Calibrated
District A
12.9 pts
3.9 pts
District B
19.9 pts
10.1 pts
One pass cut District A's ballot error from 12.9 points to 3.9 — within four points of the live result. District B fell from 19.9 to 10.1 on the first pass. The next-generation engine, scored on the same benchmark, reached 96.1% correlation and halved the remaining error.
Live benchmark: district phone polls of roughly 500 likely voters fielded by the client's pollster. Agreement is answer-level agreement across the full questionnaire. The client and state are withheld by agreement; districts are labeled A and B.
Polling as a utility
Today, a poll is a static snapshot: commissioned at the top and crystallized as a deck that tumbles down the org chart until it reaches the field office, where a staffer half-remembers the old data when making decisions. Connected Synth transforms polling from a static snapshot into an ongoing conversation with voters, available to anyone in the campaign:
- Comms tests the speech line Tuesday morning and the revision Tuesday afternoon.
- Leadership re-runs the ballot the day the news breaks, instead of arguing about what it did.
- Field re-ranks districts on current numbers and moves money and staff while it still matters.
- Any authorized team asks its own next question — no procurement cycle, no six-figure gatekeeping.
Staff don't need to go through procurement to ask new questions. And because the research and the panel live on one platform, every number traces back to the run that produced it.
Beyond the panel
There are bigger, structural issues with the traditional approach to polling, beyond speed and cost: the “say-do gap” and selection bias.
The “say-do gap” is the straightforward but significant observation that what people say does not always match what they do. Whether for conscious or subconscious reasons, the result is the same. We all say we like to eat healthy, but we impulse buy junk food more than we would like to admit. This is doubly true in politics, where real concerns and preferences can feel uncomfortable to admit to ourselves or others.
Selection bias here is the added complication that the people who answer phone polls are a low single-digit — and often non-representative — slice of the population. Most people don't answer surveys. The people who do answer are anomalies compared to the people who hang up; and weighting, no matter how advanced, struggles to overcome the structural selection bias at the heart of survey-based research.
Nevertheless, surveys still provide an important window to understand voters. But Flashpoint.AI also introduces a new approach to research that tackles the say-do gap and selection bias head-on. Market Experiments are live, blinded tests run over the digital ad networks that measure how real people react to messaging and other stimuli in the “wild” of the internet, rather than asking survey respondents what they would do.
In addition to capturing the observed behavior of real people, this capability uses advanced targeting to reach niche populations that will never respond to a survey, enabling our Connected Synth to model demographics that other platforms cannot reach.
The economics
The committee pays five figures for each live district poll and waits a week or more for the results. A calibrated synthetic run finishes in hours, at low cost, and gives the entire campaign grounded intelligence at their fingertips.
The budget for one snapshot now covers the benchmark plus unlimited follow-up — continuous polling at the price of occasional polling.
Run the same test
Flashpoint.AI can bring the power of Connected Synth to your organization. Bring your last live poll and we'll calibrate against your data, run your questionnaire, and score the results side by side.
Or skip the phone bank and source the calibration data on the platform itself: quantitative surveys for the baseline, and our patented Market Experiments for the people polls miss. Market Experiments run in days, read revealed preference instead of stated intent, and target by geography and fine-grained demographics — a single legislative district, or a niche population no panel can deliver.