Conducting a Set of Usability Tests: The Final (Pre-Launch) Tests

There were certain design decisions on both wireframes that led to better performance, and we used the best from both wireframes to produce a final wireframe.  The final wireframe then was used as the basis for developing a graphic prototype.  The graphic prototype, with images, colors, and a variety of type fonts for headers, was used for the final pre-launch usability test.

The participants were a mixture of federal and non-federal employees.  The majority of participants included the primary audience for this site, which were healthcare providers, public health professionals and consumers.  Again, these tests were conducted using Bailey’s Usability Testing Environment (UTE).

The pre-launch usability tests were divided into two parts: a pilot test and the final test.  The pilot test was conducted on Monday using 18 participants and 56 task scenarios that were divided into three categories: 24 FirstClick from the homepage, 24 FirstClick from one of the new second-level pages, and 9 ‘homepage to the actual content page’ (all clicks).  All participants saw all scenarios in one-hour in-person testing sessions. 

After the testing was complete, the data were summarized and analyzed, and a set of observations and recommendations prepared.  On Tuesday morning, the usability team met with the primary designers and conveyed the test results, with numerous recommendations for changes.  On Tuesday afternoon and evening, many changes were made to the homepage, and a few changes to the test itself.  The final in-person tests were conducted all-day on Wednesday using 56 slightly revised scenarios, and 19 different participants.

For purposes of this article, we will only discuss the findings related to the 9 ‘homepage to content page’ scenarios in the final in-person (pre-launch) tests.  In the ‘homepage to content page’ scenarios, users were allowed to navigate through the website to find information, exactly as they had done in the original Baseline test.  This provided us with an estimate of the percent improvement from the Baseline to the final pre-launch test.  The results showed a success rate of 78% that could be compared back to the original success rate of 62% for these same scenarios (an improvement of 26%).  In addition, the satisfaction score was 78, which could be compared with the original satisfaction score of 46 (an improvement of 70%).

The success rate, average time, average page view and satisfaction score improvements are shown below.  Note that all human performance and user satisfaction scores were significantly improved. Even so, it is also noteworthy to observe that it was very difficult to achieve these improvements. Our goal was to have perfect performance on all scenarios, but this was not to be.

As can be seen in the following figure, the changes made to the homepage for some of the scenarios, led to substantial increases in performance (Budget, ADHD and BMI).  Four others had moderate increases in performance.  Eight of these nine scenarios (89%) showed improved performance.  Looking back at the FirstClick data, two of these scenarios were shown to elicit good performance (BMI and FAS), while one of them still showed that it was a problem (Budget).  Obviously, useful changes were made to the website to bring the success rate for ‘Budget’ from 17% in the Baseline test, to 59% in the FirstClick test, to 91% in the Pre-Launch test.

Only one of these scenarios had a decrease in performance (NewRelease).  The ‘NewsRelease’ scenario shows how certain changes can result in no substantial changes to the success rate, while others actually can cause difficulties.  The success rate for ‘NewsRelease’ went from a low 67% in the Baseline, to an even lower 48% in the FirstClick, to a 45% in the Pre-Launch test.  None of these success rates were acceptable, and this task continued to pose a problem for users trying to find news information on the homepage.

In the CDC.gov revision of the homepage, and certain second level pages, we found the usability testing program to be very valuable.  The new homepage was substantially improved over the original page.  Making the correct set of changes is very, very difficult without the test results to guide the changes, and a follow-up test to show the results of the changes.  Based on our experiences, we find it hard to imagine how valid, useful and important changes could be made to improve a website without conducting a set of high-level usability tests to assist the change process.

The original CDC website homepage is shown below on the left, and the revised and substantially improved homepage is shown on the right.