Understanding Noteworthy Results in Zencity Surveys

What is a Noteworthy Change?

Noteworthiness is a way of highlighting unusual or meaningful changes in your survey data, without relying on traditional statistical significance tests, which don’t apply to Zencity's non-random sample design. Instead, we use a smart, data-driven method that compares your current results to a wide range of past data.

By flagging noteworthy changes, we help you focus on insights that truly matter for your community, ensuring your decisions are backed by meaningful data.

Why It Matters

In traditional survey methods, statistical significance helps identify important changes. However, since our data isn’t collected through random sampling, this kind of test doesn’t work. The Noteworthiness methodology provides a tailored alternative:

• It filters out background noise from small sample sizes.
  
  
• It uses historical data for contextual comparison.
  
  
• It allows you to focus on changes that are unusually large, not just any numerical difference.
  
  

This makes it easier to track trends, spot shifts, and support strategic decisions with clarity and confidence.

How It Works

When looking at how a score changes between two survey cycles, it’s often difficult to know if a shift is genuinely important or just a fluctuation. That’s where noteworthiness for over time changes comes in.

Zencity compares the magnitude of change from one survey cycle to the next against thousands of similar changes from past surveys across the Zencity database. If the change is unusually large for its sample size, we flag it.

Step-by-Step: How We Determine If a Change Is Noteworthy

1. Calculate the Change Between Cycles - We calculate the absolute change in score between two survey cycles. This could be for overall results or any demographic or geographic segment.
   
   
2. Account for Sample Size - We combine the sample sizes from both cycles to determine the total number of respondents. Results are sorted into bins based on sample size ranges (under 100, 100-199, 200-299, etc.).
   
   
3. Benchmark Against Historical Data - We then compare your result change to a database of historical changes from surveys with similar sample sizes. This tells us how common or rare your change is.
   
   
   Important: The benchmark is based solely on sample size bins across all surveys, not specific questions or demographic groups. All changes with similar sample sizes are compared against the same thresholds, regardless of which question, population group, or segment they involve.
   
   
   
4. Apply the Thresholds - Based on percentiles of past results from all types of questions and population segments, we classify the change into one of three tiers:
   
   

Example of a "Noteworthy" change label in the Zencity dashboard

Example of a "Somewhat Noteworthy" change label in the Zencity dashboard

Example

If the trust in local government score among female respondents in your community increased by 6%, and the combined sample size is around 300:

1. The system places this in the "300-399 responses" sample size bin
2. It compares the 6% change to all historical changes for surveys with similar sample sizes
3. It does not specifically compare only to "trust in local government" questions or only to "female respondent" subgroups
4. It does not create special thresholds for specific population segments or demographic groups
5. If most past changes across all questions and groups with this sample size were only 1–3%, this 6% jump would likely fall in the top 10% and be flagged as Noteworthy

Why This Matters

This approach allows you to:

• Avoid reacting to small, normal fluctuations
• Recognize when a real shift has likely occurred
• Prioritize what to communicate or act on in your reports and strategy

And because the methodology uses real data from many communities and is based primarily on sample size, it's tailored to how scores naturally vary in practice, not based on assumptions from randomized samples.

Important Notes

• Thresholds are determined solely by sample size bins, not by specific questions or population segments.
• All comparisons within the same sample size bin use the same universal thresholds.
• When we say a change is "noteworthy," we mean it's unusual compared to all historical changes of that sample size, not just changes in that specific question or demographic segment.