Confab | Christine Perfetti: Essential Techniques for Measuring Your Content’s Success

Christine Perfetti is talking about how to test and measure your content.

So important: Web analytics tell you WHAT is happening. They do not tell you WHY.

Perfetti is showing really horrifying and totally believable tests….

  • People who got all the way through reserving a room at Disneyland…who wanted a room at Disneyworld.
  • People who were thrilled with the hotel reservation form for another hotel…until they got to the end and discovered it didn’t work the way it appeared, and they didn’t have a reservation set up after all.

Both these examples just make my stomach hurt.

Perfetti’s recommendations on usability testing

Oh, love this: Perfetti just has observes in the room. She says, People know there is someone behind your one-way mirror. Better just to have them in the room and don’t be sneaky. Intros people by first name only, not title. Prefers that observers use paper and pen so as not to have distraction of typing.

Perfetti’s recommended books on usability testing

What you should look for in a test

  • Users going to the FAQ: People aren’t getting what they want
  • Users clicking on Help
  • Users going to a sitemap
  • Users clicking the back button: They’re not seeing what they want
  • Pogosticking: Going back and forth on a list of links, looking for something they never find
  • Going straight to search: People don’t want to search. They haven’t seen their trigger words, so they type them in.

So now we’re going into advanced techniques specifically for content.

5-Second Test: Bring up a page for literally 5 seconds, see if people can figure out how to solve a problem you give them. We’re doing one here about uploading photos…does it seem it will be quick and easy to do? Perfetti shows a page full of text from Photobucket…the room laughs. Is anyone confident they can upload photos? No one.

Now we see 5 seconds of Picasa. This was also text heavy, but also had a big blue button that said, Get Started. Now we’ve seen a Flickr page. It showed a visual representation of the process. Most people agree this is the one that makes it look quick and easy to upload photos. will let you test pages on their site. Perfetti warns: 5-second tests work “very, very poorly” for home pages — these pages have many priority. Use these on content pages or other single-purpose pages.

Great point: People want to spend all their time testing their home page. This is not usually a good use of time. They want to find the page that is useful to them. Test whether your home page directs people to the content they want, but test the effectiveness of your other pages.

First-click test: Give users a specific task, and see where they click first. Are they headed down the wrong path?

Love this: Users don’t like to choose their role. Perfetti shows the WebEx site where users were befuddled when forced to categorize themselves as individual, small, medium, large enterprise. They redesigned the site with a single “Products” link, and users found that much easier to navigate.

Comprehension test: Use to determine if people understand your complex content. Sometimes Perfetti will use a questionnaire, or sometimes will just ask questions. She shows the content, asks them to read through it, and asks the questions while they can still see the content. These are generally information, but will clearly show if users understand the complex content.

Inherent value test: Helps you figure out if you’re conveying the product value to prospects. In phase 1, bring in your most loyal customers. You ask them to give you a tour of the site and talk through the value of the product or service. Have them share what they find most valuable. You can also ask them to complete tasks, but that’s not the main goal. In phase 2, bring in people unfamiliar with the product. Then, ask the prospects to complete the tasks that the loyal customers do all the time. You will find out if they see the same value.

Catalog-based task testing: Find out what’s important to users. Take any printed catalogs or brochures [or print out same info from your site] and ask users to highlight important content. Then ask them to find the same content on the website.

Some other software and resources that were mentioned:

Confab | Testing Content

This session from Angela Colter is slam-packed full. Her popular article last year on testing content on A List Apart obviously precedes her, and everyone wants the details.

She’s going to talk about

  • Readability Formulas: Counting the aspects of text that can be counted, and applying a metric. The Flesch-Kincaid formula measures the number of words per sentence and the number of syllables per word to give you a projected grade level of the text.
    Upside: People can understand and act on this metric.

    Downside: You’re only counting what can be counted. Doesn’t explain anything about text clarity. And my favorite downside, it can push you toward really crappy writing.

    Great quote from George Klare: 

    Merely shortening words and sentences to improve readability is like holding a lighted match under a thermometer when you want to make your house warmer.Don’t use a readability formula alone — it’s good as a red flag.

  • Usability Tests: Find out what people actually understand, not what they say they understand. Get the right people in the room. What are the critical issues for the user and for the business? You have to get users to think aloud, but you need to know what the correct answer looks like.Well written tests are critical to evaluating your content properly.Usability testing ends opinion-based arguments.
  • Cloze Tests: These are the content usability tests where you remove every 5th word and see if people can fill in the blanks. Use 125 words or so and 25 or so blanks. If they’re getting 60% or more, your content works. If they’re getting 40% of less, your content is too difficult. In between, you may need revision or more instruction.If you test on the appropriate audience, this tells you clearly if the content is appropriate.