The City of Calgary
The City of Calgary was increasing the functionality of their Open Data Portal to include a Citizen Dashboard where users can navigate datasets and create data visualizations with the available data. Before launching the new platform, the Corporate Analytics and Innovation Division at the City of Calgary wanted to distill issues and ensure that this new product will offer optimal user experiences to citizens with a variability in age, levels of computer literacy and familiarity with data mining.
To evaluate the usability of the Open Data Portal, identify usability issues and advance user experience of the platform, our team had the opportunity to conduct usability testing of the portal.
Our high level goals were:
-To map usability issues to make the experience more intuitive and frictionless
-Understand user behaviour and ability to complete common tasks that the platform is designed to support
-Provide insights and recommendations to the Corporate Analytics and Innovation Division at the City of Calgary
It was important to have participants who varied in age, levels of computer literacy, and familiarity with data mining to reflect the Portal’s users. We recruited 9 participants between the ages of 20 – 64 who had never used the Portal but have previously accessed the City of Calgary’s website and have experience using websites that organize and categorize products.
Running Tests –
The tests were conducted in a private, quiet testing room with only the moderator and participant present. We followed a moderator protocol and used a data collection log to capture user data. The participants were asked to complete the tasks while continuously thinking out loud – verbalizing their thoughts as they move through the interface. This was done to be able to capture as much feedback and user insights as possible.
We developed four tasks that focused on the primary use of the Portal, which is locating and manipulating data sets. Tasks were presented to each participant in a different sequence as counter balancing controls for carry-over effects, including user fatigue and learnability.
Task 1: Visualizing Data
Starting from the Data Home Page, find the “Recreation Facilities” chart data set. Change ‘Plot Style’ to ‘Heat Map’ and apply
Task 2: Accessing Help Documents
Starting from the Open Performance Home Page, open the “Sorting Datasets” video.
Task 3: Finding Specific Data
Starting from the Data Home Page, find the “Calgary Public Library Catchment Areas” data set. Go to download the data set as a CSV file (end task before downloading it).
Task 4: Manipulating / Filtering Data Sets
Starting from the Data Home Page, find the data set “Calgary Seniors Directory Map”, filter out the data set for Bankview and ‘Save as’ the data set as “Bankview”.
Usability Metrics –
Time on Task
(Occurrence, Frequency, Type of Error)
Quantitative data were synthesized into graphs to display performance results, adn qualitative data, which includes verbal and non-verbal feedback, were summarized to identify Usability Issues and to provide recommendations for improving the user experience of the portal.
Unclear labelling of Help Documentation
We recommended improved labelling and that the aesthetics of the Help Documentation page match the Data Portal home page, as users took a long time to find the right page, and when they did, they were unsure as the page did not seem associated with the portal.
“The help documentation is located in what looks like a separate site; if one had to find help documentation starting from the Data Portal one would not be able to find it.” – Test Participant
Search bar that doesn’t search?
We recommended adding a universal search bar for the Portal or improving the function of the existing bar as multiple users were observed typing into the search bar with no displayed results.
“The search bar does not search the whole site, nor does it suggest or autocomplete similar words.” – Test Participant
Steep learning curve
We recommended including simple prompts on the toggles of the Portal to explain specific functions for first time users, as we saw a 66% failure rate in Manipulating or Filtering Data Sets.
“The portal is too complex for a general user and there is too much information about the detailed functions of portal compared to basic functions, which is overwhelming for first-time users.” – Test Participant
Thinking out loud?
I found that participants found it challenging to think aloud while they are navigating tasks. It’s an unusual activity and I found that I had to remind them to vocalize their thoughts and emotions.
I’m not testing YOU
I observed that participants felt like I was testing their abilities and skills when I gave them tasks. They became frustrated when they didn’t know how to do certain tasks and I believe that they were not behaving as they would in natural environments.