Ensuring a Representative Sample

At Zencity, we are committed to achieving comprehensive representativeness across demographic categories and perspectives in our surveys. To do so, we have an evidence-based approach to produce representative data. 

Our primary tool for ensuring representativeness is a rigorous, multistep process for designing, distributing,
and processing survey data:

1) Identify the community

We work with government teams to identify the geographic or administrative areas (e.g., police district, city, county) that make up the community. We use national statistical data such as the U.S. Census or the UK Census and American Community Survey to establish the number of adults in each age, gender, and race/ethnicity group within the community. This information determines how our sample should be distributed across these groups. For instance, if the Census shows that 20 percent of a community’s residents are Hispanic, the survey should be based on a sample where approximately 20 percent of responses reflect the perspectives of Hispanic residents.
 

2) Determine sample size based on how the data will be used

Sample sizes include enough data on groups that government teams need to understand. This could include their community as a whole, geographic subdivisions, and key demographic groups. These samples produce results that are credible enough to be used for informed decisions. This step draws on statistical guidelines for survey sample sizes and is akin to determining how much blood needs to be drawn from a patientto test for a particular health condition. We then adjust the sample size to ensure we can produce reporting that matches the team’s needs. For instance, if a team needs representative data for particular subdivisions of the community, we will choose a sample size that can describe those areas credibly, exactly like making sure there’s enough blood drawn for all the tests a doctor would conduct.
 

3) Distribute the survey digitally and accessibly across many channels

Zencity recruits survey respondents where the vast majority of residents can be most readily reached: on any digital device. Respondents are recruited with the help of targeted ads on social media, mobile apps, and survey panels. Our survey application is WCAG 2.2 AA accessible, mobile-native, and supports multiple languages, so that residents are able to participate regardless oflanguage or ability. This allows us to more effectively engage hard-to-reach populations compared to traditional phone and mail surveys.
 

4) Real-time dynamic monitoring

As the survey is in the field recruiting across the population, we monitor and adjust how we invite respondents. This means if at any point one specific demographic is responding more on mobile apps, we will adjust the effort we spend to recruit from this specific channel for this demographic. This allows Zencity to recruit a diverse pool of residents that ensure we meet the sample targets that align with each community’s population profile as defined by the US Census.
 

5) Implement checks and other adjustments that safeguard response quality

After distribution, we apply checks that ensure the data are generated by residents and are high quality. For instance, we use device fingerprinting, a privacy-preserving process that analyzes respondents’ metadata - to prevent duplicate submissions. In some cases, we respond to these checks by removing low-quality data from the sample, so that erroneous responses don’t lead to inaccurate conclusions.

6) Produce weights to correct for slight deviations that arise during distribution

When we receive slightly more or slightly fewer responses than anticipated for some demographic groups, we use rake weighting, an industry-standard approach, to correct for these discrepancies. Weights align the data to the targets we established earlier by determining how much each response should count to maintain representativeness. If the sample should have 100 residents ages 18–34 but we obtain responses from only 95 such residents, the weights may count each of those responses for about1.053. With weights, the 95 responses would be counted as 100 responses (as 95 × 1.053 roughly equals 100). Ourstandard weighting takes age, gender, race, and ethnicity into account.

All of this allows us to hear opinions well beyond the STP: the Same Ten People, whose voices often get disproportionate prominence. Zencity’s adherence to best practices and continual attention to error reduction leads to survey results with coverage of the actual community that is as good, if not better than, traditional address-based surveys. 

Learn more about how we produce representative survey data here