This post is about my software product Hyperion. You can read more about it here.
The Original Design
A Hyperion administrator creates a “PIT count” for a particular year:
A unique code is attached to each one, which an administrator distributes to his volunteers so they can view the PIT count survey and fill it out.
Additionally, the administrators can “disable” a PIT count, preventing volunteers from submitting surveys. If a volunteer submits a survey to a disabled PIT count, then the data isn’t sent to the server and volunteers see an error message saying, “The PIT count is disabled.”
I added this last feature because an admin was concerned that volunteers would be able to keep submitting data (fake data?) after the PIT count is over.
The Unforeseen Complication
Organization admins want to train their volunteers on the system prior to the live PIT count. Ideally, the volunteers would submit a mock survey just as they would on the day of the PIT count. I recommended that the admins create a “test” PIT count for that year and distribute one URL for testing and one for live data:
The Emergent Problem
Every organization using Hyperion (15 in 2020) used this training model. Many had hundreds of volunteers. The volunteer training was typically 1-2 hours long – sometimes virtual and sometimes in person. Despite the trainers best efforts, many volunteers didn’t listen when they were told to use a different URL for the actual PIT count and continued to use the test URL. This wasn’t a huge problem: it typically just required the administrators to comb through the test data submitted during the PIT count, grab the offending surveys, and add them to the live data export.
In a couple cases this problem was exacerbated because the administrators (reasonably) either disabled or deleted the test PIT counts. In these situations the data was still recoverable, but it involved a messy intervention (re-adding or re-enabling the PIT count and then asking the volunteers to re-open the volunteer portal so the surveys could be automatically re-submitted).
The Proposed Solution
You could make the argument that this is a training problem – and it is – but my goal when building software is to make it intuitive and useful enough that it doesn’t require training.
I’m planning on changing the behavior of the “disable” toggle. It will now toggle between a “test” mode and a “live” mode (starting in test mode). When a survey is submitted, it will be put into either a test or live bucket.
This will remove the need for administrators to distribute a separate URL just for testing. They’ll just have to click the toggle right before the PIT count starts.
The admins will be able to see the data in both buckets, and move individual surveys back and forth between them (including in bulk). That way they’ll be okay if they forget to click the toggle at the appropriate time. There’s already a concept of “invalid” surveys so this won’t add much conceptual complexity.