Project Overview
This study aimed to evaluate the usability and accessibility of the Seattle City Light website in providing information about incentive programs to small and medium-sized business owners. The usability testing specifically focused on users' experiences navigating to the "Small and Medium Business Solutions" page and understanding the associated "Commercial Retrofit Program Requirements Documentation."
We assessed whether the website allowed business owners to easily navigate, access information, and comprehend incentive program requirements. We set goals, developed test plans, recruited participants, conducted usability tests, analyzed data, and delivered findings and recommendations.
Research artifacts
Timeline
Project Motivation
Our clients from Seattle City Light and Seattle Office of Economic Development (OED) collaborated to bridge the gap for small and medium-sized businesses to effectively leverage Seattle City Light's incentive programs and achieve their energy goals.
However, it was found that the existing program materials were not easily understandable by most laypeople with a small business, which highlighted the need of enhancing the accessibility and clarity of these resources.
Turning Client Questions into Research Questions
To guide project planning and scoping, we broke down our clients' broad questions into specific, user-centered inquiries to inform the research process. Their goals centered on two aspects: website navigation (how users discover and access program information) and documentation comprehension (how well users understand the program requirements).
Methods
Cognitive Walkthrough & Heuristic Evaluation
To explore the potential problem space, we visualized different navigational paths and key screens, and used established design heuristics to evaluate the website.
Interaction map
Demo of heuristic evaluation
Task-based Usability Testing
Participants
Target audience
Seattle-based small or medium business owners with physical commercial spaces
Navigating recruitment challenges
At first, recruiting participants through client connections seemed viable, but we quickly encountered challenges with low response rates and time constraints. To overcome this, we took a proactive approach by visiting the University District, directly engaging business owners to participate in our usability testing. We conducted on-site sessions when possible, scheduled future appointments, or collected contacts for follow-ups. This hands-on strategy not only resolved our recruitment challenges but also reinforced the importance of contingency planning and creative problem-solving.
Participant overview
We recruited 5 local business owners: four from shops in the University District and one via the clients' network. Participants were aged 30-60 and consisted of three males and two females.
Study Design
Testing procedure
We conducted both remote moderated testing and in-person testing, with each session lasting about 30 minutes.
Task 1: Website navigation
The first task was to find relevant incentive programs for the participant’s business on the City of Seattle official website.
Testing goals
Assess how intuitively users can navigate to the incentive information on the website.
Identify any confusing or incorrect navigational choices.
Task 2: Documentation comprehension
The second task was to use the commercial retrofit program requirements documentation to determine the participant’s business eligibility.
Testing goals
Evaluate the clarity of the information in guiding users to their next steps.
Analyze how the document’s layout and language affect user comprehension.
Data Collection & Analysis
Data collection map
We created a table to present the data we gathered and how it aligns with our research questions.
Qualitative data affinity diagram
We utilized affinity diagrams to organize open-ended responses, think-aloud notes, and answers to post-task and post-test questions. This helped us categorize qualitative data into core themes and extract major usability findings.
Quantitative data analysis overview
We analyzed and visualized quantitative data, complementing qualitative insights for deeper interpretation.
Website navigation
Documentation comprehension
Website navigation (number of clicks attempted)
Findings & Recommendations
Severity Rating
We used a rating system developed by Dumas and Redish to evaluate and prioritize the severity of each usability issue, enabling our clients to address the most critical usability barriers first.
Level 1 - prevents completion of a task
Level 2 - creates significant delay and frustration
Level 3 - has a minor effect on usability
Level 4 - subtle problem, points to a future enhancement
Key Findings
Unclear Website Navigation and Information Structure
Relevant task
Website navigation
Description
Many participants struggled to locate the incentive program information. They suggested that the entry points could be more prominent on the homepage and within the navigation menu. In addition, the search functionality was problematic, with participants noting that the search bar frequently failed to return relevant results.
City of Seattle official website hompage navigation menu
Recommendation
Increase the visibility of the incentive programs by featuring them in the main navigation bar and homepage. This would create clearer entry points and make it easier for users to navigate.
Optimize the search functionality to handle typos and provide more relevant results, and make the search bar more noticeable.
Misaligned Audience and Confusing Technical Terms
Relevant task
Documentation comprehension
Description
Participants found the documentation overly technical and difficult to understand, with limited relevance to their expertise as business owners. All five participants indicated they would need a contractor or advisor to review the requirements documentation, regardless of their level of comprehension.
"As a regular business person, I wouldn't understand all the details in the documentation; I would definitely take this info to consultants or contractors."
Recommendation
Integrate an AI chatbot to assist business owners in navigating the documentation by providing clear, step-by-step explanations of program requirements, eligibility criteria, and next steps.
Since contractors appear to be the primary users of the documentation, further testing with them would help identify improvement areas and ensure alignment with actual user needs.
Lengthy and Time-Consuming Documentation
Relevant task
Documentation comprehension
Description
Participants noted that small businesses often rely on one person to manage all tasks, so the documentation needs to be clear and easy to navigate. They found the long documentation difficult to sift through and preferred preferred having direct links to key sections upfront.
“I didn’t know to look further down to what might qualify.”
Recommendation
Include a catalog or summary at the beginning of the documentation with a side navigation guide for easier access to key sections, such as the lighting and non-lighting requirements, and add a section with bulleted next steps.
Outcomes
Product/External
We identified major usability issues and provided evidence-based recommendations to enhance incentive program accessibility and clarity, which have driven updates to the Seattle City Light website to improve user experience.
For example, a table of contents was incorporated into the documentation based on our suggestions to improve navigation.
Before: No table of contents
Current: With table of contents
Process/Internal
The usability study revealed that contractors rather than business owners are the end users, and additional testing is needed to refine the website and documentation further.
Our experience indicated that participant recruitment should begin earlier and be strategically planned to ensure timely involvement for future research.
We showcased effective communication and collaboration practices, setting a strong precedent for how Seattle City can successfully partner with external teams to streamline workflows and improve outcomes.
Reflection
Conducting usability testing with a PDF document required adapting traditional approaches, treating the document as a long web page. This unique scenario challenged me to modify testing methods for unconventional formats.
Collaborating with multiple researchers and interview moderators required consistency in scripting, note-taking, and reporting processes. Composing clear interview scripts became paramount, especially when explaining key terms or guiding participants through various tasks.
Capturing comprehensive user interactions was essential for data analysis and generating accurate insights. Using multiple tools to record test sessions and pre-testing devices would ensure smoother test execution and documentation.
The research highlighted the value of methodological triangulation in revealing deeper insights and offering a more complete understanding than any single approach alone. Cognitive walkthroughs and heuristic evaluations identified initial usability issues, while usability testing exposed unexpected user behaviors and perspectives.