Accuracy Studies
Why Your People Counter's Accuracy Claim Is Probably Wrong
A critical look at how vendors measure and report accuracy for people counting systems. What 'up to 99% accuracy' really means, and how to validate claims with on-site testing.
6 min read ·
Key Takeaways
- Vendor accuracy claims are typically based on controlled lab environments that don't reflect real-world retail conditions.
- The average gap between vendor-claimed and independently-verified accuracy is 6–12 percentage points for people counting software.
- "Up to 99% accuracy" usually means single-file entry in ideal lighting — conditions that represent less than 30% of real-world traffic.
- Running your own 2-week accuracy audit costs nothing but staff time and provides the most reliable performance data.
- For most retail applications, 93–96% real-world accuracy is achievable with AI systems and sufficient for reliable business decisions.
Every people counting vendor claims high accuracy. "Up to 99%." "Industry-leading precision." "Near-perfect counting." These claims aren't technically lies — but they're deeply misleading. The accuracy you see in a vendor's marketing materials bears little resemblance to what their system will deliver in your store, library, or facility. Here's why the accuracy gap exists, how vendors manipulate their numbers, and how you can determine the real accuracy you'll get before signing a contract.
The Accuracy Gap
Independent testing consistently reveals a significant gap between vendor-claimed accuracy and real-world performance. In our own accuracy study of five people counting systems, the average gap was 6.2 percentage points — meaning a system claiming 98% accuracy typically delivered 91.8% in real-world conditions. For some systems, the gap was much larger: one infrared system claiming "up to 95%" delivered just 83.2% in our comprehensive testing.
| System Type | Typical Vendor Claim | Typical Real-World Performance | Gap |
|---|---|---|---|
| Stereoscopic AI Video | 98–99% | 94–97% | 2–4 pts |
| Monocular AI Video | 97–98% | 90–94% | 4–7 pts |
| Thermal Sensor | 95–97% | 85–92% | 5–10 pts |
| Infrared Beam | 93–95% | 78–88% | 7–15 pts |
| WiFi/Bluetooth | 85–90% | 60–75% | 15–25 pts |
This gap matters because business decisions built on inaccurate foot traffic data compound errors. If your people counting software overcounts by 10%, your conversion rate appears 10% lower than reality — potentially driving unnecessary staffing increases or marketing spend. If it undercounts by 10%, you're underestimating demand and likely understaffing during peak periods.
How Vendors Measure Accuracy
Understanding how vendors derive their accuracy claims reveals why the numbers are so inflated. Most vendor testing is conducted in controlled environments — laboratory settings or carefully selected test locations that optimize for the system's strengths. Common characteristics of vendor testing environments include single-file entry corridors (1–1.2m wide), consistent overhead fluorescent lighting (200+ lux), minimal reflective surfaces, adult-only test subjects walking at moderate pace, and low traffic density (20–40 entries per hour).
Real-world environments are messier. Entrances are wider. Lighting varies throughout the day. Groups enter together. Children and strollers create height variations. Peak traffic periods create congestion. Weather conditions affect lighting and visitor behavior. Every one of these factors reduces accuracy from the lab-tested baseline.
Common Accuracy Tricks
Beyond favorable testing conditions, some vendors employ specific tactics to inflate their reported accuracy numbers. Being aware of these practices helps you evaluate claims more critically.
- Cherry-picking scenarios — Reporting accuracy only for single-entry scenarios while omitting group, children, or wide-entrance results.
- Excluding edge cases — Removing data points where the system clearly failed (e.g., large groups, strollers) as "anomalies" before calculating accuracy.
- Short test windows — Running accuracy tests only during low-traffic, ideal-condition periods rather than across a full week including peak hours.
- Favorable rounding — Reporting "up to 99%" when the best individual test scenario hit 98.6%, while the weighted average across all scenarios was 93%.
- Directional accuracy — Reporting entry accuracy separately from exit accuracy and showcasing whichever is higher. Entry counting is typically easier.
- Post-calibration numbers — Reporting accuracy after extensive manual calibration specific to the test environment — calibration that won't apply to your unique space.
Real-World vs. Lab Results
To illustrate the accuracy gap in practice, consider this breakdown of how a typical AI-based people counting system performs across different real-world conditions compared to its lab-tested accuracy of 98.5%.
Accuracy: Lab Conditions vs. Real-World Scenarios
- Single Entry — lab: 98.5, realWorld: 97.2
- Group Entry — lab: 98.5, realWorld: 91.4
- Children — lab: 98.5, realWorld: 89.8
- Wide Entrance — lab: 98.5, realWorld: 88.1
- Peak Hour — lab: 98.5, realWorld: 93.5
- Low Light — lab: 98.5, realWorld: 86.3
The lab number (98.5%) is accurate — for lab conditions. But the weighted real-world accuracy, accounting for the proportion of traffic in each scenario, comes to approximately 93.1%. That 5.4-point gap is the accuracy tax you pay for real-world deployment. Any vendor who doesn't acknowledge this gap is either uninformed about their own product's field performance or deliberately misleading buyers.
Run Your Own Audit
The most reliable way to determine accuracy is to test the system yourself. Here's a straightforward protocol that any retailer or facility manager can run during a trial period:
- Request a trial installation — Most vendors offer 14–30 day trials. Insist on your actual location, not a demo environment.
- Designate manual counters — Assign a staff member (or two, for verification) to manually count entries during 4 designated windows: a quiet morning, a busy afternoon, peak hour Saturday, and an evening session.
- Count in 15-minute intervals — Record manual counts per 15-minute block and cross-reference against the system's reported counts for the same intervals.
- Calculate accuracy per interval — Accuracy = (1 – |system count – manual count| / manual count) × 100. Average across all intervals for overall accuracy.
- Test challenging scenarios specifically — During your test, observe how the system handles groups (3+ people entering together), children, strollers, and anyone carrying large items.
- Compare weekday vs. weekend — Accuracy often varies between quiet weekdays and busy weekends. Both matter for your business decisions.
- Document and challenge — Present your findings to the vendor. If real-world accuracy falls significantly below their claims, negotiate pricing accordingly or evaluate alternatives.
What's "Good Enough"?
Not every application needs 99% accuracy. Understanding what accuracy level your use case actually requires prevents you from overpaying for precision you don't need — or accepting inaccuracy that undermines your data's usefulness.
| Use Case | Minimum Useful Accuracy | Why |
|---|---|---|
| Conversion rate optimization | 93%+ | Small counting errors significantly skew conversion calculations |
| Staffing optimization | 90%+ | Traffic trends matter more than exact counts; 90% captures patterns reliably |
| Marketing attribution | 93%+ | Campaign ROI calculations need reliable traffic lift measurement |
| Occupancy compliance (fire code) | 97%+ | Safety-critical application; undercounting creates legal liability |
| Library/museum visit counting | 88%+ | Funding justification needs directionally correct data; exact counts less critical |
| Trend analysis (year-over-year) | 85%+ | Consistent bias is acceptable if it's stable over time; trends remain valid |
For most retail applications, 93–96% real-world accuracy is the sweet spot — achievable with modern AI people counting software and sufficient for reliable business decision-making. Chasing the last few percentage points of accuracy typically requires disproportionate investment in premium hardware and narrow entrance configurations. Focus your evaluation on real-world accuracy in your specific environment, not lab-tested maximums.