Usability testing with 5 users challenged

Laura Faulkner wrote an interesting article “Beyond the five-user assumption: Benefits of increased sample sizes in usability testing.” I just heard about it at the UIDesigner blog, but the article has been around for awhile (published in Behavior Research Methods, Instruments, & Computers in August 2003).

In the article, Faulkner argues that “the risk of relying on any one set of 5 users was that nearly half of the identified problems could have been missed; however, each addition of users markedly increased the odds of finding the problems.” Here are some other interesting quotes from the article:

“the average percentage of problem areas found in 100 trials of 5 users was 85%” – this agrees with Nielsen’s claims of 5 users catching 85% of the problems. However…

“The percentage of problem areas found by any one set of 5 users ranged from 55% to nearly 100%. Thus, there was large variation between trials of small samples.” – wow! That’s one WIDE variation!

“groups of 5 found as few as 55% of the problems, whereas no group of 20 found fewer than 95%.” – the hint here is that more testers will find more problems.

Faulkner then goes on to discuss how her study still supports Nielsen’s claims, but that usability practitioners are incorrect about religiously using the 5-user test to always catch 85% of the problems.

The thing I found interesting was this: Faulkner’s article is based on using 5 users in single usability tests. But Nielsen doesn’t actually say to do that! In his article “Why You Only Need to Test With 5 Users” (I know, it sure SOUNDS like he recommends only using 5 users) he actually recommends doing “three tests with 5 users each” and correcting any errors found between each test. That’s VERY different than just testing 5 users, don’t you think?

I think Faulkner is REALLY talking about the perception that usability testers only do single tests, hoping to catch most problems, and then move on to other things. And it’s certainly possible that some testing is done this way. It’s just that, well… Nielsen didn’t suggest doing that (the basis of Faulkner’s article).

Here’s what Nielsen actually says to do:

  1. First test with 5 users: you’ll catch an average of 85% of usability problems
  3. Second test with 5 users: tests the corrections made from results of the first test, catches stuff your first 5 users might not have caught, and even better, “the second test will be able to probe deeper into the usability of the fundamental structure of the site, assessing issues like information architecture, task flow, and match with user needs.” So the second test fixes, tests, and probes much deeper than the first test.
  5. Third test with 5 users: you just fixed more problems, so you need to test those fixes out… hence the third test.

One other difference I noticed. Faulkner’s focus in usability testing is the goal of catching all usability problems with testing, but Nielsen’s usability testing goals are different. His goal is “to improve the design and not just to document its weaknesses.” It’s possible that in a life-or-death, mission-critical (egad, did I just write “mission-critical?”) product (like healthcare or flight equipment) with high regulatory standards to meet, documenting weaknesses and catching EVEYTHING would be important. But remember – I’m a library web manager! This isn’t a life-or-death situation, and I’m not a brain surgeon.

Is there a big difference between testing 20 users or multiple groups of 5 users? I’ll let the Research Methods people figure that one out. But it sure seems like three groups of 5 users for usability testing still works just fine, and catches most, if not all, web usability problems.

Ten things your web sites should be doing

This article discusses ten things websites should be doing. Here are the 10 things, rearranged a little (because of repetition) and with comments pertinent to library websites:

1. Offering regularly updated information (blogs, CMSs, etc.) and 2. Increased efficiency in news and information distribution (RSS, ATOM, etc.)
Blogs are great ways to quickly and painlessly update information on your website. CMS’s are another (think of them as fancy blogs that keep better track of your info). Or just start typing good ole’ HTML – at least you can have updated information on your site.

RSS, ATOM, etc. all make information distribution quick and easy. If you want frequently-updated information on your site, this is the way to go. Otherwise, you can talk your web developer into creating a custom version of a CMS (what we did). But it accomplishes the same thing – it allows our library staff to update information without messing with HTML, CSS, etc.

3. Alternative methods of information distribution (email newsletters, RSS,, etc.) and 4. Enhanced notification and announcement systems (pings, email alerts, etc.)
Again, RSS… In this case, think of the aggregator part of RSS – sorta like an e-newsletter. The point here is not so much with website design, but how to get new information out to your customers. And RSS and email are two great ways to do this. But remember – we’re librarians! There are other simple ways for us to alert our customers to new information – think printed bookmarks by the circ desk, flyers for library programs, and other more traditional methods. Use those just as often as you use RSS and email!

Pings? Anyone know how this helps? Let me know…

5. A place for your site’s users to offer feedback and input (blog comments, forums, etc.) and 9. Collaborative communication and documentation (Wikis, blogs, etc.) and 10. On-demand support feedback (user-driven FAQs, click-to-chat, etc.)
Besides the methods mentioned, use chat reference, email “ask us” links, or create an online comments form (go here to see ours in action). Just remember to update it if you creat one!

6. Improved performance and code optimization (CSS, XHTML, etc.) and 8. Intelligent system to system communication (XML, SOAP, etc.)
The geek stuff. Translation? Keep your website up-to-date with current coding practices, techniques, and language adjustments. This will keep the techies happy with your site in the short-term, and keep your site working on browsers in the long-term.

7. Multiple ways to access information (multi-faceted navigation, folksonomies, etc.)
Great suggestion, and one I hope us librarians remember. Make sure there is more than one way to access your great information! For example, if someone wants a new fiction book – provide multiple ways to access it – from the catalog, a fiction page, a staff book review page, a genre page, etc…

Have a wonderful Christmas, and here’s to better library websites in 2005!

Continuing Discussion of Blogging Ethics

Karen Schneider has posted about blogging ethics – very interesting reading, especially the two links to lists of potential “blogging ethics” that bloggers might want to follow. The whole “blogging ethics” thing is an interesting concept, though difficult at best to implement in any formal fashion, since anyone with a PC and a web connection can post whatever to their heart’s content. But it could be implemented in the seeting she’s interested in – that of the bloggers who plan to blog for PLA.

I’m going to comment on something Karen mentioned in the beginning of her post, and has mentioned in previous posts, because I think this concept is something that bloggers need to get their heads around before moving on to a discussion of blogging ethics. Karen says “Too many of us want to be considered serious citizen-journalists, when it suits us” … I even claimed this in a previous post .

Let me clarify a little bit (practicing the blog ethics concept of correcting misinformation :-). I don’t see my blog or other library blogs as replacing – they serve two very different purposes. Using my blog as an example, here’s my stated purpose (lame though it might be): “cool stuff about library web sites … and whatever else I decide to post.” Here’s what I do: I read other blogs, articles, books, etc., mostly focused on web technology. When I see something interesting and useful, something I think other library web techies would be interested in, I post it to my blog. I comment on it – try to take an idea further, stretch non-library web products into a library setting, etc. And I might post my own web-related thoughts when they occur and seem interesting, too. The goal could be stated as being in two places at one time – if you didn’t see something interesting, maybe I did. Two people can cover more ground in keeping up with web-related information than one person.

Is that being a citizen journalist? Well… yeah, kinda sorta. Googling the term “journalist” gives these two definitions, among others: 1. “one who keeps a journal” and 2. “The conductor of a public journal.” I fit both of those. But I think Karen is thinking more about the other definitions given – “one whose occupation is journalism” and “one whose business it is to write for a public journal … or other professional writer for a periodical.”

I don’t meet all those criteria. I’m not paid to blog, and it’s not my “business” (although Gary Price might fall into this category). But then again, my blog could be called a “public journal.” Could it be that this is a new medium of communication? What do you get when you mix a diary, niche information, a trade publication, notes, an email, an IM, and a converstion together, and then stick it on a website? Answer: a blog.

My point here? Blogs (library-related blogs, anyway) are just similar enough to traditional media (I’m thinking trade publications, or at least posts on trade publication websites) to be confused with actually being traditional media. But a blog is also very different from traditional media – for example, my blog doesn’t pass through any editorial channels. This is happening with other forms of media as well – I can post my own music on the web without having to pass through a record label; I can create a movie and post it on the web without having to pass through a movie studio; I can write a book and post it to the web (like Lawrence Lessig did with his latest book). Etc. This is a great thing – other professionals can access my blog, and “subscribe” to it if they find it useful. Sure, having no form of editorial board could also be a bad thing – but I’m not posting about brain surgery, either. If someone doesn’t find my blog useful, they can ignore it. Any discussion of what should be done when bad information is posted to a blog should really cover that concept for the whole web – not blogs alone.

Here are two other examples. LISNews is great at posting interesting library-related news. So that’s sorta traditional (except that you don’t have to wait for a monthly print publication to keep up-to-date on library news). But TeleRead isn’t very traditional at all: that blog is very focused on niche information (e-books). TeleRead includes information on new e-book products, comments on those products, observations on ebooks, etc. No traditional media is going to be that focused. At the least, it wouldn’t be profitable enough to sustain.

So… where does that leave us? Am I a citizen-journalist? Not really. I’m participating in a new medium of communication that resembles some aspects of aspects of traditional journalism, but is very different in many ways, too. How’s that sound? Am I going in the right direction with this? Feedback is always good :-)

Looking back at the early 1900s, I wonder if someone debated whether or not letter writing and phone calling were similar or different?

SJCPL Blogs Mentioned in the New York Times

How cool is this? Michael Stephens library was mentioned in the New York Times (login is required)… no, not just his library, but the BLOGS his library has set up!

The article goes on to mention lots of other cool, on-the-edge types of library technology. I thought it’d be fun to list them all out and do a comparison of who is using what the article mentioned… so here we go:

1. ebooks (we have them through netLibrary)
2. Downloadable ebooks (not yet, although we’ve talked about it)
3. Blogs (we have them)
4. Created a “my library account” feature – Richmond Public Library is mentioned, with ways to track favorite books for individuals… wow. (not really – our ILS does some of this)
5. receiving book passages by email and holding discussions on them… (partially through the BookLetters service) (We do use Chapter-A-Day to email book excerpts… does that count?)
6. Chat services (QuestionPoint and were mentioned, although they called them IM tools, which they’re not)(we have chat reference)
7. Networked video games (no. But an extremely cool idea.)
8. Wireless hot spots (got em)

OK – so what’s missing from this list? WiKi’s? True IM? Federated searching? What else? Let me know, and I’ll include it.

Web-based Instant Messaging

Aaron over at walking paper mentioned a web-based version of AIM – AIM Express. The web-based version allows people to do IM without having to use the AIM Messenger program – instead, you use a web-based version. Nothing to download!

Yahoo is testing a similar product (click the link, go to the bottom of the page, click Launch Web Messenger).

We’re going to test these out to make sure they work in our locked-down public PC setting…