Deduplication Explained (A Deep Dive)

Shasa McCarogher
Shasa McCarogher
  • Updated

Deduplication in Engage is an option that Zencity clients can enable on various results and reporting screens to reduce the impact of participants’ attempts to ‘stuff the ballot box’ by submitting the same responses to an engagement multiple times.  It should be noted that even without using these administrative deduplication tools, the resident-facing engagement experience acts as the first line of defense by actively discouraging this behavior unless kiosk mode has been enabled.

Duplicates are identified by analyzing participant information over the entire set of an engagement’s submissions  The two pieces of information that are currently evaluated are the participant’s IP address and their digital fingerprint.  It should be noted that some clients choose to disable digital fingerprint collection entirely and others allow participants to opt-out of digital fingerprint collection, both of which can reduce the effectiveness of the deduplication functionality.  

An important note on privacy: the digital fingerprint and IP address are not stored in an identifiable fashion, but rather as an irreversible hash of letters and numbers that maintains comparability without storing anything personally identifiable. So an IP address like 10.15.190.123 might be stored as 5dtry38u8j9cru0we98rux118934ukr and a digital fingerprint comprised of many different properties of the participant’s computer hardware and software might be stored as 3tr123jiojf132jf23. 

Deduplication is executed as a ‘first-pass’ before any other filters are applied, reducing the list of overall submissions that the results or reporting system uses to only include the first submission by an identifiable device (digital fingerprint) or if the digital fingerprint is not available, the first submission by a particular internet connection (IP address).  This means that occasionally if the first submission from a specific device does not contain a response to a specific question, even if the subsequent submissions do, the result set will not contain a response to that question from that particular device.

Here is an example of submissions (in the order they were received) for an engagement run in Zencity Engage (you export this information as a CSV file for any engagement you run).  Note that if you have deduplication applied, only the bold submission rows (and the responses that are part of those submissions) will be included when looking using results visualization, clustering, analysis or reporting capabilities.  Submissions for the first instance of each unique combination of Hashed IP & Digital Fingerprint will be included, subsequent submissions for the same Hashed IP & Digital Fingerprint will not.  

 

Submission Date Hashed IP Digital Fingerprint Include
22 2023-09-06 17:09:06 d0568d562eb279ed0ea1f0dd48d0c dd8702fb7b45ad886ee1cf1086de2693 Yes
23 2023-09-06 17:12:16 82a0406aa2d969c61c62d030bcaf 7c94b01eca72cbda391ef255163c0a89 Yes
24 2023-09-06 17:13:07 d0568d562eb279ed0ea1f0dd48d0c dd8702fb7b45ad886ee1cf1086de2693 No
25 2023-09-06 17:13:22 82a0406aa2d969c61c62d030bcaf 7c94b01eca72cbda391ef255163c0a89 No
28 2023-09-06 22:41:01 a1ea0aeb88814dff62d642be f13f5127ae1cfb7b2e8cf3d8daa08eab Yes
29 2023-09-07 21:10:50 d0568d562eb279ed0ea1f0dd48d0c dd8702fb7b45ad886ee1cf1086de2693 No
31 2023-09-07 21:18:35 d0568d562eb279ed0ea1f0dd48d0c c028014397218331ea26c868397985ba Yes
41 2023-09-27 18:58:20 d0568d562eb279ed0ea1f0dd48d0c f13f5127ae1cfb7b2e8cf3d8daa08eab No
49 2023-10-27 22:24:56 d0568d562eb279ed0ea1f0dd48d0c dd8702fb7b45ad886ee1cf1086de2693 No
102 2023-11-01 20:42:59 706c9f4b8d70a50df7b079860eaaf 95ac31c515f5b6e6455b4ad273fce8fb Yes
103 2023-11-09 0:06:08 942505fa67acbf6ed5b22dd2e5 0beece094a5a6289bc70714f2b1c89d1 Yes
104 2023-11-09 0:08:06 942505fa67acbf6ed5b22dd2e5 0beece094a5a6289bc70714f2b1c89d1 No
105 2023-11-20 23:56:39 d0568d562eb279ed0ea1f0dd48d0c 04e48ad7ddabfeb645eb6a6b8cc93ae7 Yes
106 2023-11-20 23:57:58 d0568d562eb279ed0ea1f0dd48d0c 04e48ad7ddabfeb645eb6a6b8cc93ae7 No
107 2023-11-20 23:58:46 d0568d562eb279ed0ea1f0dd48d0c 04e48ad7ddabfeb645eb6a6b8cc93ae7 No
108 2023-11-21 0:04:06 d0568d562eb279ed0ea1f0dd48d0c 5ff8b4f17cd5f62d2a4ffb4159a18fd8 Yes
109 2023-11-21 6:02:15 942505fa67acbf6ed5b22dd2e5 0ec56ef326a629c2e9ae162f8f83c278 Yes
110 2023-11-21 15:27:12 82a0406aa2d969c61c62d030bcaf 9d1c9107a47fcb6258c7ddb822770488 Yes
111 2023-11-21 22:31:28 82a0406aa2d969c61c62d030bcaf   Yes
112 2023-11-21 22:58:03 22d05cb7ffb31f44c70e12d7d6e7   Yes
113 2023-11-21 22:58:51 22d05cb7ffb31f44c70e12d7d6e7   No
114 2023-11-21 23:00:35 fb7bda8db7174ca9cc04a62142   Yes
115 2023-11-21 23:03:03 fb7bda8db7174ca9cc04a62142   No

 

 

Note: Clients who have used Zencity Engage for many years may have used results filtering tools where IP address and digital fingerprint deduplication were available for use separately.  This approach is less effective at deduplication and has been removed from the results and analysis tools.

Was this article helpful?

/

Comments

0 comments

Article is closed for comments.