Help in Question Writing for Usability Tests

I just read this – DonnaM’s Writing memorable scenarios for usability testing. Good stuff!

To sum it up… when you do a usability test, you usually ask a bunch of scenario-type questions. Your test participant then tries to answer the question by finding an answer on your website. Easy enough, right?

The hard part is writing those questions! When doing a general test for the whole website, your questions have to cover lots of territory – you want at least one question for each “important thing” on your website, while at the same time realizing that no one’s going to sit through a grueling 200 question test (well, not unless you pay them actual money…)

And you want those questions to make sense to the participant. Librarian lingo should be removed (think monograph, reference, ILL, ILS, etc.), hints should be removed (no “go to this page, look in the upper left hand corner, and see if you can find such-and-such”), and
the question should be easy to read.

And DonnaM goes one more step – her post discusses giving the question a real-life scenario. That way, you make the question more vivid and emotional to the test participant. This helps the participant visualize the scenario, thus making it easier for the participant to remember. And ultimately helps the test participant add some realism to his/her answer (thus providing more useful information during the usability test).

Wow – lots to think about for those embarking on usability testing!

Jakob Neilsen isn’t a web designer

From someone’s comments on a previous post –

From Anonymous:
I have found Nielsen to be the most overrated web site design “guru” out there. I read his book “Designing Web Usability” and found it to be pretty far from what I would consider good advice for a web designer, at least in the library world. Maybe if you’re designing a site for the movie “Troy” or some other site for entertainment, Nielsen is the one to turn to. I’m not pointing to specifics, admittedly, but as everybody seems to fawn over Nielsen, I needed to stand up and say that the emperor is not wearing any clothes.

Actually, I’d call it a case of trying to stuff the emperor into farmer’s clothing. Annonymous doesn’t like Neilsen’s ideas – that’s fine. No problem there. But from the comment, I’m not sure this person understands what Neilsen does. Neilsen doesn’t do Web Design – he does Usability. I find the two concepts to be very different:

  • Web design – making a nice-looking website, involving graphics, colors, content, css and other standards, etc.
  • Web usability – making sure that people can use the website.

Neilsen really focuses on usability. Even in his articles about web design mistakes, he mainly discusses usability issues. Now obviously, a usable website will probably be a well-designed website. But from the above comments about Neilsen’s web design book, it seemed to me that the concept of web design vs. the concept of usability could get sorta mangled – because Neilsen usually doesn’t talk about CSS positioning, Flash-enabled layouts, or drop-down menus. Instead, he focuses on making what you have placed on your website into a very usable website – so website visitors can find information quickly and painlessly, and get on with their lives.

The biggest web design mistakes in 2004

This is a great post: Web Pages That Suck presents the biggest web design mistakes in 2004. It’s funny, but it also mentions some good stuff in the process. here’s the list:

1. Believing people care about you and your web site: A website is about customer’s needs… not staff’s needs.

2. A man from Mars can’t figure out what your web site is about in less than 4 seconds: This follows the logic in Steve Krug’s book “Don’t Make Me Think” – he compares a website to a billboard on the highway. that’s how much time you have to connect with your website visitors.

3. Mystical belief in the power of Web Standards, Usability, and tableless CSS: Here’s a great quote from the article: “Remember, nobody gets excited about the tools used to build a house (“Please tell me what brand of hammers you used!”). People get excited about how the house looks and performs.”

4. Using design elements that get in the way of your visitors: they’re talking about splash pages, animations, bad Flash navigation, etc. But I could add Library Catalog navigation to this list! Why do some ILS systems wig-out when I hit the back button – and why am I forced to use their “special” back button? You get the idea.

5. Navigational failure: No links back to the home page, poorly worded links, etc.

6. Using Mystery Meat Navigation: This is a great way to describe links that you have to hover over in order to find out what they link to…

7. Thinking your web site is your marketing strategy: Library websites don’t do this so much… the website is PART of your marketing strategy – not ALL of it.

8. Site lacks Heroin Content: By “heroin,” they mean content that keeps website visitors coming back for more. That’s the goal of my library’s Subject Guides. Another related area is frequently updating information on your library websites – update that tax forms page before you offer it again!

9. Forgetting the purpose of text: When you want to use text – do so. Don’t use graphics or flash.

10. Too much material on one page: pretty clear.

They actually list reasons 11-14, too… go read the article, and take the advice to heart!

Usability of Websites for Teenagers from Jakob Nielsen’s Alertbox

Update: here’s an article from CNN about the new study…

This is a must read: Usability of Websites for Teenagers (Jakob Nielsen’s Alertbox).

From the article: “Many people think teens are technowizards who surf the Web with abandon. It’s also commonly assumed that the best way to appeal to teens is to load up on heavy, glitzy, blinking graphics. Our study refuted these stereotypes.

Teens succeeded in the usability tests only 55% of the time, which in usability is BAD. The study showed that teens’ poor performance centered around three things: “insufficient reading skills, less sophisticated research strategies, and a dramatically lower patience level.” – in other words, they’re… well… teens.

Teens DO like cool-looking websites, and pay more attention to graphics… but found modest clean web design to be more usable.

Here’s another good quote from the article: “Teenagers like to do stuff on the Web, and dislike sites that are slow or that look fancy but behave clumsily.” – think about that one – can teens DO stuff on your library’s website? Or is your teen’s site made up primarily of lists of links and books?

Here are some suggestions from the article about interactive stuff to include on a teen’s website:

online quizzes: How about a Harry Potter quiz, with a drawing for free movie passes for the winners?

feedback/comment/question forms For starters, you could ask teens what they want the website to do (of course, then you just might have to DO what they asked for).

online voting: Have them vote on local issues, surround the voting page with explanations of the issues, and see what happens – could be fun.

games: Gaming is HUGE right now for teens. Buy books on gaming, point to gaming websites, or even go one further and set up gaming days at the library.

sharing pictures or stories: Hold a photography contest, and put the winner’s pictures online.

message boards: teen book/music/dvd clubs, local and world issues, etc – just a place for teens to connect with each other to get and share information.

offering and receiving advice: This can be where you use that virtual reference service to connect with teens.

a way to add their own content: We’ve thought about online poetry slams and articles written by teens/for teens…

These are just a few ideas. Go read the Nielsen article and start thinking!

Usability testing with 5 users challenged

Laura Faulkner wrote an interesting article “Beyond the five-user assumption: Benefits of increased sample sizes in usability testing.” I just heard about it at the UIDesigner blog, but the article has been around for awhile (published in Behavior Research Methods, Instruments, & Computers in August 2003).

In the article, Faulkner argues that “the risk of relying on any one set of 5 users was that nearly half of the identified problems could have been missed; however, each addition of users markedly increased the odds of finding the problems.” Here are some other interesting quotes from the article:

“the average percentage of problem areas found in 100 trials of 5 users was 85%” – this agrees with Nielsen’s claims of 5 users catching 85% of the problems. However…

“The percentage of problem areas found by any one set of 5 users ranged from 55% to nearly 100%. Thus, there was large variation between trials of small samples.” – wow! That’s one WIDE variation!

“groups of 5 found as few as 55% of the problems, whereas no group of 20 found fewer than 95%.” – the hint here is that more testers will find more problems.

Faulkner then goes on to discuss how her study still supports Nielsen’s claims, but that usability practitioners are incorrect about religiously using the 5-user test to always catch 85% of the problems.

The thing I found interesting was this: Faulkner’s article is based on using 5 users in single usability tests. But Nielsen doesn’t actually say to do that! In his article “Why You Only Need to Test With 5 Users” (I know, it sure SOUNDS like he recommends only using 5 users) he actually recommends doing “three tests with 5 users each” and correcting any errors found between each test. That’s VERY different than just testing 5 users, don’t you think?

I think Faulkner is REALLY talking about the perception that usability testers only do single tests, hoping to catch most problems, and then move on to other things. And it’s certainly possible that some testing is done this way. It’s just that, well… Nielsen didn’t suggest doing that (the basis of Faulkner’s article).

Here’s what Nielsen actually says to do:

  1. First test with 5 users: you’ll catch an average of 85% of usability problems
  2. THEN FIX THOSE PROBLEMS!!!
  3. Second test with 5 users: tests the corrections made from results of the first test, catches stuff your first 5 users might not have caught, and even better, “the second test will be able to probe deeper into the usability of the fundamental structure of the site, assessing issues like information architecture, task flow, and match with user needs.” So the second test fixes, tests, and probes much deeper than the first test.
  4. THEN FIX ANY PROBLEMS FOUND DURING THE SECOND TEST!!!
  5. Third test with 5 users: you just fixed more problems, so you need to test those fixes out… hence the third test.
  6. THEN FIX ANY MORE PROBLEMS!!!

One other difference I noticed. Faulkner’s focus in usability testing is the goal of catching all usability problems with testing, but Nielsen’s usability testing goals are different. His goal is “to improve the design and not just to document its weaknesses.” It’s possible that in a life-or-death, mission-critical (egad, did I just write “mission-critical?”) product (like healthcare or flight equipment) with high regulatory standards to meet, documenting weaknesses and catching EVEYTHING would be important. But remember – I’m a library web manager! This isn’t a life-or-death situation, and I’m not a brain surgeon.

Is there a big difference between testing 20 users or multiple groups of 5 users? I’ll let the Research Methods people figure that one out. But it sure seems like three groups of 5 users for usability testing still works just fine, and catches most, if not all, web usability problems.