AM+A designed/tested improved navigation and screen design for Kaiser’s
online library viewable by all doctors and nurses in the US’s largest healthcare provider.
Project Description
Overview
In June of 2006, Kaiser Permanente (KP) engaged AM+A to analyze and test the usability of the Clinical Library (CL), a Web-based internal information resource. The Clinical Library makes a wide variety of clinical information available to physicians, clinicians, and nurses, both during actual patient care and at times when they are engaged in research or study. Through the Clinical Library, KP clinicians and nurses can find internally produced guidelines for medical conditions and procedures, find and print out patient handouts, reach other Kaiser Websites, access external resources such as journals and medical texts, and reach a wide variety of other resources of a operational or managerial nature.
Working with a team from CMI, AM+A drew up an interview script and interviewed sixteen clinicians and nurses in their offices in four KP regions: Colorado, Northern California, Northwest, and Southern California. Subjects were given a series of mostly medical questions, and asked to find answers using Clinical Library.
• The Usability Analysis report resulting from this study detailed both the powerful potential of the CL to serve its target audiences and numerous severe usability issues currently limiting its effectiveness, particularly during time-critical periods of actual patient care.
• The report detailed 31 major “observations” concerning usability issues, grouped in four general categories such as “Search Issues” and “Navigation Issues.” A fifth category detailed observations of the effectiveness of “Clinical Tools,” a partial restructuring of the CL UI that had been implemented in the Colorado region. A table of priorities was prepared based on comments by the interview subjects. AM+A made preliminary recommendations for how each issue could best be resolved.
In the fall of 2007, Kaiser asked AM+A to propose a usability analysis and design project to address some of the issues discovered in the 2006 report. The proposal divided work into two Phases:
• Phase 1 was a rapid, two-month project focused on a “proof of concept” redesign of the most time-critical parts of Clinical Library, with four rapid tasks including a paper user test conducted by phone.
• Phase 2 was a six-month complete redesign, including the development of an interactive prototype and three user tests in the field.
Redesign Phase 1
The rapid, preliminary Phase 1 consisted of four tasks:
• Consult with Kaiser Experts
AM+A prepared a questionnaire designed to solicit input from Regional Physician Leads, CL Content Managers, and other key stakeholders, asking their opinions about high-level usability issues and the importance of particular CL features. AM+A summarized their observations in a report, which was used as guidance in later tasks.
• Develop Conceptual Design
Based on the 2006 research and the prior consultation with stakeholders, AM+A prepared a series of annotated wireframe screens focused on the most important parts of the CL for users engaged in patient care.
• Develop Visual Design
After revisions based on consultations with the client, visual design details applied to the conceptual design wireframes, including the design of a set of icons. AM+A prepared a sequence of screens for use in testing.
• Conduct User Test
Phase 1 concluded with a test of a series of screenshots, conducted by phone with 27 doctors, nurses, physician’s assistants and clinical pharmacists from six Kaiser regions. The test included a comparative exercise, in which subjects were asked to state their preference among three different approaches to a central portion of the library. One approach was the Clinical Tools, developed internally by the client. The other two were variations of a “faceted navigation” technique proposed by AM+A.
The reaction of those users tested was highly encouraging, suggesting that the general design direction was well chosen. As always, user response was also very informative concerning which aspects of the design were successful and which required more work.
Redesign Phase 2
The full redesign phase began in June 2008 and concluded in December 2008. The general design approach for the core Clinical Library, chosen during Phase 1, was progressively refined, tested, and refined again. The redesign was expanded to cover the complete Clinical Library, including sections that had not been addressed during Phase 1. There were 11 Tasks in all, summarized as follows:
• Develop and Refine Taxonomy and Faceted Navigation
Working with a leading Kaiser physician and other client personnel, AM+A assisted in the refinement of a “taxonomy” logically organizing the different types of content in the CL. AM+A prepared a terminology questionnaire designed to elicit expert opinions from senior stakeholders concerning which of a set of alternative terms should be used as labels for key structural elements in the CL. The faceted navigation structure originated during Phase 1 was refined based on the results of the Phase 1 User Test and developments in the Taxonomy.
• Develop Conceptual Design in Three Stages
At three points during the six-month process, AM+A prepared a series of rough mock-up screens, with annotations noting changes that had been made since the previous version. Changes were occasioned by the results of user tests and ongoing discussions between AM+A and key representatives of KP, and regular large meetings including Physician Leads from each KP region and others representing interested KP departments.
• Develop Visual Design in Three Stages
After information architecture issues had been resolved during each Conceptual Design task, an AM+A Visual Designer prepared final versions of key screens, including the creation of icons and the final placement of key controls.
• Conduct First User Test
The evolving design was tested three times, with different groups of approximately 12 Clinicians and nurses for each test, who were visited in their offices in four Kaiser regions. In the first test, users were shown a series of screenshots and asked a series of questions to answer using the redesigned screens. AM+A prepared a report after each test, with annotated screenshots detailing the results and suggesting options for improvement.
• Second and Third User Tests
In the second and third user tests, users were asked to find answers to typical questions using a web-based interactive prototype of the redesigned system. These tests, which were recorded using the usability testing software Morae, had three parts. The first part focused on introducing the redesign, and testing specific points that had proved problematic in earlier tests to see if a solution had been found. In the second, users were asked to find answers to the same question twice: once using the prototype redesign, and once using the current system. Finally, users were given a short survey asking their overall impressions of the new system. The reports for these tests included graphs comparing how long users took to complete their tasks using the two systems, and graphs showing the results of the survey. AM+A also prepared a compilation of video excerpts from each User Test, showing the screens of users engaged in using the two systems (current and revised) while also displaying their faces and voices in a PIP (picture-in-picture) format.
• Write Requirements Document
Using a template provided by the Kaiser Information Technology (IT) Department, AM+A prepared a highly detailed list of requirements for implementation. These requirements included Business Objectives for the redesign project, User Requirements, Functional Requirements, Integration Points with other systems, and Non-Functional Requirements. After review by Kaiser IT, AM+A revised this document twice.
• Write Business-Case Document
Working in collaboration with Kaiser personnel, and using another client template, AM+A assisted in the writing of a Business Case based on the work completed in the several phases of the project and plans for implementation. The presentation included the charts created during one of the User Tests, and an assessment of the return-on-investment (ROI) of implementing the project in production.
Results
With each successive test, Kaiser clinicians and nurses became increasingly strong in their decisive preference for the redesigned system. As shown in the chart below, “Results of User Test 3 Comparative Exercise,” users took much less time finding answers using the redesigned system, despite relative unfamiliarity with it (most users had been using the Clinical Library for years). This positive result was confirmed in one of the survey questions, the results of which are displayed below in the chart “User Survey Result.” Asked to choose on a scale of 1 to 10 how strongly they agreed with the proposition “If the CL were redesigned along the lines of this prototype, I would probably be able to find the information I need more easily,” user responses all fell in the 7-10 range.
Home Page: Current
Home Page: Redesigned
Faceted Navigation Filters and Two-Column Link-Display
Results of User Tests 3: Comparative Exercise
User Survey Result
