Electronic Journal of Academic and Special Librarianship
v.4 no.1 (Winter 2003)
Usability Testing at Florida International University Libraries: What We Learned
Electronic Journal of Academic and Special Librarianshipv.4 no.1 (Winter 2003)Usability Testing at Florida International University Libraries: What We Learned |
|
Sarah J. Hammill
Florida International University
hammills@fiu.edu
The Florida International University Libraries’ Web site’s new look was launched in Fall 2001. As a result of the new look, a group formed to undertake a usability study on the top page of the site. The group tested three target groups to determine the usability of the top page. The study pointed out some revisions for the top page; however, more importantly, it suggested areas for future research.
Florida International University (FIU), in Miami, is a public research university serving a student population of 32,000 students and 1,100 full-time faculty. The university has two major campuses, University Park (UP) and Biscayne Bay (BB). The student population is as varied as the city of Miami. Many students are nontraditional, returning to school after many years of employment; 10% are international, from 130 different countries; 51% are Hispanic; 14% are African American; and 3.5% are Asian. In addition, the majority of students are commuters (FIU, 2001).
The Libraries’ Web site was revamped in summer 2001. The new face of the FIU Libraries was launched in August 2001. That fall, an intercampus library group undertook a usability study. The design of the study was based on comparable studies at The University at Buffalo Libraries (Battleson, Booth, and Weintrop, 2001) and the Ferriss Hodgett Library at Sir Wilfred Grenfell College (McGillis & Toms, 2001).
This paper will discuss usability and the actual study done on the FIU Libraries’ Web site.
Usability can be defined in different ways. According to Nielsen (1994), usability is one component of usefulness of a system. The other component is utility, simply defined as whether the system can do what is needed. Usability deals with users’ easy ability to deal with functionality of the system. Usability is the human component of human-computer interaction. Nielsen further defines usability as having five attributes; learning, efficiency of use, memorability, fewer errors equal more usable, and subjective satisfaction. Learning is measured by the initial ease of use. Efficiency of use refers to the point when the learning curve flattens out. Memorability measures how well an interface can be remembered, and subjective satisfaction determines whether the system is pleasant to use (Nielsen, 1994).
Frokjaer, Hertzum, and Hornbaek (2000) define usability as consisting of three components: effectiveness, efficiency, and satisfaction. Effectiveness determines how accurately and completely users can complete certain tasks.
Efficiency is the link between accuracy and completion of goals, including the time to complete a task. Satisfaction measures whether the system is pleasant to use (Frokjaer et al., 2000).
Simply stated, usability involves the carrying out of tests to see if users can find specific information (Horn, 1996).
Norlin and Winters (2002), define Web site usability as a method that involves end users who in return provide feedback on a Web site’s design or Web site from the user’s perspective in a controlled environment.
There are a number of ways to conduct usability testing. This study focuses on formal usability testing of the FIU Libraries Web site http://library.fiu.edu/. Battelson et al. (2001) define formal usability testing as the observation of users performing a number of predefined tasks when using a particular site.
Various researchers detail the steps in formal usability testing, including Norlin and Winters, (2002); Nielsen (1994); Gordon (2000); Horn (1996); and the National Cancer Institute (2001). With some slight variations, each describes the importance of establishing goals and objectives, development of tasks, a written script (including the pre-assessment test), recruitment, actual testing, and analysis of results.
Goals and objectives of usability testing include assessing subjects’ perceptions of a site and/or feedback on what works and what doesn’t (Gordon, 2000). Objectives should be testable. For example, the question “Can our users find a book from the library homepage?” is easier to measure than “Is the site usable?” (Horn, 1996). Goals to consider include whether a user completes a task successfully, how fast tasks are completed, whether the user is satisfied with the time it takes to complete the task, what paths the user takes in completing the task (is the path efficient?), and where the problems are and why (National Cancer Institute, 2001).
The goal of the FIU Library usability study was to determine whether the design and organization of the top page of the site allow users to locate information easily. We focused our study in three main areas: catalog searching, article searching and library services. We asked specific questions related to each of the areas. Questions are graphed and explained below.
The subjects recruited should be representative of “real” users (National Cancer Institute, 2001). The original goal of the FIU study was to recruit 30 respondents from three main target groups: undergraduates, graduate students, and faculty. Librarians chose subjects informally, by approaching patrons within the library. As a result of our recruiting method, we fell short of our goal. We tested 52 subjects: 26 undergraduates, 13 from each of the two campuses; 14 graduate students, 8 from UP and 6 from BB; 5 faculty members, 3 from UP and 2 from BB; and 7 who fell into the “other” category.
The actual testing contains many components, including a pre-test questionnaire, the use of a script, and in our case the Think Aloud Protocol (Horn, 1996).
According to Horn (1996), a pre-test questionnaire can help identify more information about the user, including library Web site use, Internet use, and demographics.
Norlin and Winters (2002) suggest using a written script when conducting actual testing because a script will eliminate variations in the testing procedures. The script should be concise and should be read and followed verbatim. Scripts should include the purpose, and the structure of the test, and a disclaimer about the test. A disclaimer should emphasize that the site is being tested, not the subjects’ ability or knowledge.
The Think Aloud Protocol is a common technique used in usability testing. Subjects are asked to vocalize thoughts, opinions, and decisions to click on one link over another. The Think Aloud Protocol is an inexpensive way of obtaining qualitative feedback on a Web site (Horn, 1996). It is also helps to identify users’ misconceptions of the site. When using the Think Aloud method, it is important to note what the users were doing and where they actually looked when doing the tasks (Nielsen, 1994).
For our research we used a pre-test questionnaire with five questions to determine who are subjects were, and how familiar they were with using the library and searching the Web. We decided on using two scripts (see appendix A), one for recruitment and one for facilitating. Because each researcher did the testing alone, this helped eliminate variations and discrepancies in the testing procedure.
We kept the study simple by using the Think Aloud Protocol (Horn, 1996) whereby the participants verbally expressed their thoughts as they navigated through the tasks. We decided the Think Aloud Protocol would sufficiently define problem areas on the Web site. Subjects were asked to express their thoughts and opinions out loud. The Web site researchers prompted subjects when needed.
The tasks were designed to test the usability of the Web page, including navigation, clarity of vocabulary, and visibility of the different sections. Tasks were given in order, one at a time. The researcher took notes to record what the subject said, where the subject clicked, and what Web pages he/she reached. At the conclusion of all tasks, the subjects answered a questionnaire with rating scales to reflect affective states. In addition, the subjects answered open-ended interview questions with the opportunity to add further comments or questions. The researchers took notes on the users’ answers to the interview questions.
Analyzing the results involves compiling the data from all subjects and then focusing on recurring problems, trends, and comments (National Cancer Institute, 2001). The performance data (time to complete the task and number of clicks to find the answer) can be quantified. Qualitative data, including the thought process, post-test questionnaire, and interview, should be used to back up the performance data (Horn, 1996).
The FIU Libraries’ study gathered both quantitative (number of clicks to complete each task) and qualitative (comments and questions) data intended to reveal the extent to which users are able to make efficient and satisfying use of the FIU Libraries’ Web site.
Each of the questions was graphed to show the number of clicks to completion. We coupled this data with the post interview and questionnaire to define points of revision for the site.
Task #1 Do the FIU Libraries own the book Castles in the Sand?
Task number one asked the subject to identify an item that is part of the libraries’ collection. This task was designed to determine whether the Express Link ‘FIU Library Catalogs’ is visible from the library homepage. Is the title descriptive? Is the purpose and usage of the link clear to users?
Based on the sampling, the link to the FIU Libraries Catalog is clearly identified. Those who were unable to answer this question tended to be non-native speakers of English and/or unfamiliar with using a library in the United States. One student who happens to be employed by the library had no idea where to go or what to look for. He explained that he was a freshman and had never used the library homepage.
Task #2 Is the journal American Anthropologist available in the FIU Libraries?
Task number two asked the subject to identify an item that is part of the libraries’ collection and is a follow-up to task number one. In addition to determining whether students understand the term “catalog,” researchers also wanted to measure whether patrons intuitively go to the FIU Libraries Catalog to find a journal title.
Based on the results of this task it is clear that the majority of subjects do not equate the library catalog with a journal title. The majority attempted to find the journal in Articles by Subject. Most users found their way to the Sociology and Anthropology link under Articles by Subject. Comments included: “It’s somewhere in here.” “You have to search in Anthropological Literature.”
Those who were successful in 2-3 clicks also started in Articles by Subject or tried to link on the heading Database and Article Searching (which at the time was not active).
The results of this task have great implications for the FIU Libraries Information Literacy Program. Of the nineteen subjects who had attended a library instruction session at FIU, 32% were unsuccessful. This compares with 29% of all subjects in answering the question correctly.
Task # 3 Find the database Academic Index.
Task number three asks the subject about online resources. This task was designed to measure whether the links under subject guides is intuitive. Did the user know to click on the “Alphabetical list of all Electronic Resources” for a specific database?
Subjects located Academic Index in a number of ways, including Articles by Subject, Alphabetical List, and Where to Find Articles. The majority of the unsuccessful tried to click on the link to Databases and Article Searching (which was not active at the time). In addition, many attempted to find the database from the Electronic Journals list. There appears to be some confusion over the difference between a database and a journal. One graduate student commented, “Articles by subject has lots of choices. However, the articles aren’t too recent.” He didn’t realize he was looking at the coverage of the databases.
Task # 4 Can you find a journal article on diabetes?
Task number four asked the subject about online resources. This task was designed to measure whether the links under subject guides was intuitive. Did the user know to click on “Articles by Subject” to find specific databases for a particular topic?
Based on the sample, it seems that most of our users are able to find articles on a particular subject.
Task #5 How do you order a book to be delivered from the other FIU campus?
Task number five asked the subject about services available to them online. Were the subjects able to find the links to “Intercampus Loan” and “Other Forms”? Did they understand the difference between Intercampus Loan and Interlibrary Loan? Were the headings clear? (The page has since changed.)
It is interesting to note that many of the unsuccessful were keenly aware of going to the circulation desk and filling out a form. Comments included: “I ask at the reception desk”, “Come to the library and ask”, and “Go to the FIU Catalog, do a title search, get the call number, and take this information downstairs.” Because this study was based on the Web page and online services, we counted this correct answer as unsuccessful. In addition, there was quite a bit of confusion between interlibrary loan and intercampus loan.
Task # 6 How would you find a journal article on discrimination?
Task number six asked the subject about online resources and was a follow-up to task number four, which was designed to measure whether the links under subject guides were intuitive. Did the user know to click on “Articles by Subject” to find specific databases for a particular topic?
Based on the sample, it seems that most of our users are able to find articles on a particular subject. The majority of the subjects went to the Articles by Subject link.
Task # 7 Does the Libraries’ Web site have a guide to doing research in Nutrition and Culture?
Task number seven determined whether the subjects knew where to find guides and background information for doing research. Did they have a clear understanding of the difference between the links “Subject Guides” and “Internet Resources for Subject Collections”? Did they realize that there were guides available to help them do research?
The intent of asking this question was to determine whether patrons knew that the Subject Guide link could be used as a starting point for finding information on a topic. This link can also be found through the Articles by Subject link to Dietetics and Nutrition. Interestingly enough, some librarians thought we were leading the subject to Articles by Subject and not to an actual guide. Most users were unsuccessful in this task. Comments included: “I don’t see the word research anywhere on the page.” This is an area for further discussion; should the word research be on a library’s homepage?
Norlin and Winters (2002) emphasize the fact that usability testing is not a research study. Usability identifies problem areas. A big part of usability testing is subjective: Why does a user click on a certain link and how does he/she understand the terminology? By asking our subjects to think aloud while they conducted the tasks, we gained a better understanding of the rationale and interpretation of the Web site as well as how well they were able to identify key components of the site. The comments and feedback indicate that the users have varying needs. Some prefer all of the information on top; others consider the top page cluttered and disorganized. The one comment that was repeated over and over again was about the Express Links and how clear and easy it was to read the left-hand side of the page.
There are questions about validity with usability testing. Some typical problems with validity and usability testing involve studying the wrong users and having unrealistic tasks. However, testers’ common sense and understanding can increase the level of validity of a test (Nielsen, 1994). In our study, we eliminated two tasks. Originally we had nine tasks, but tasks number eight and nine were eliminated. Task number eight was “Assume you were taking a class in a subject you know very little about, for example, geography, anthropology, or religion. If you had to write a research paper, how would you find information resources on that topic?” The researchers were looking for the subjects to start with the link to Subject Guides. However, when we sample tested librarians, they invariably clicked on the link to Articles by Subject. We suspect the reason for this is that the link to Subject Guides is in the middle of the page and not often used by librarians (even in instruction sessions) except for the librarians who created them. As a result of our pilot testing on librarians, we decided that this was an unreasonable task for our users.
Task number nine was, “There is a book that is checked out. You want this book. How do you request it online?” The information for recalling an item was available under the link Other Forms. However, to actually recall the item the user has to be in the catalog record. It was decided that the question was unfair and not a good measurement tool.
Another point of significance is the small variation in completion and success rates between subjects who had attended an Information Literacy session at FIU and those who had not. For example, in task number three 32% of subjects who had attended a session succeeded in one click compared to 53% of those who had never attended an Information Literacy session at FIU. Currently, we are designing a study to compare differences between first year students who have attended an information literacy session and those who have not. We have decided to focus our research on a combination of usability factors of the library Web site and Information Literacy concepts. It is interesting to note that when this study was proposed, there was some debate among FIU librarians on whether the tasks actually measured usability or information literacy. After much discussion, it was decided that information literacy is more than being able to find a book and an article. However, the results of the 19 subjects who had attended a FIU information literacy session did not vary much from those who hadn’t. It poses a question, “Should students whom we are teaching to be information literate know how to find a book and a journal even if they measure the site as unusable?”
As a result of this study we made the main topic headings such as Databases and Article Searching active. Another subtle change we made was to shorten the link “Alphabetical List of all Electronic Resources” to “A-Z Databases”. Most of our subjects had some idea of what a database is however, they did not connect databases to Electronic Resources. We also changed the link “Other Forms” to “Forms”. One subject asked in the think aloud process, “What other forms?’ We still grapple with using the term forms. How would a student know to click on “forms” to find information on intercampus and interlibrary loan? Finally, we know the importance of being able to search a site as far as usability is concerned. This is something we are working on implementing in the near future.
Future usability studies of the FIU Libraries’ Web site will be conducted. As a result of this study, we have decided that solely testing the top page limited our abilities to clearly measure navigation, and visibility of the different sections. In addition, we realized usability testing takes time. Future studies will focus on a smaller set of subjects pay close attention to questions asked and feedback received. We will spend more time observing and testing each user instead of having many users for a short period of time.
Battleson, B., Booth, A., & Weintrop, J. (2001, May). Usability testing of an academic library web site: a case study. The Journal of Academic Librarianship, 27, 188-198.
Florida International University. (2001, December). Facts and information. Retrieved February 25, 2002, from http://www.fiu.edu/docs/facts_info_stats.htm
Frokjaer, E., Herzum, M., & Hornbaek, K. (2000, April 1-6).Measuring usability: are effectiveness, efficiency, and satisfaction really correlated? CHI, 345-352.
Gordon, S. (2000, February 15). User testing: how to plan, execute, and report on a usability evaluation. Retrieved September 12, 2001, from http://builder.cnet.com/webbuilding/pages/Graphics/Evaluation/
Horn, J. (1996). General concepts of usability testing. Retrieved March 1, 2002, from http://jthom.best.vwh.net/usability/
McGillis, L., & Toms, E. G. (2001, July). Usability of the academic library web site: implications for design. College & Research Libraries, 62, 355-367.
National Cancer Institute. United States. (2001, August 14). Usability.gov: improving the communication of cancer research. Retrieved September 12, 2001, from http://usability.gov/index.html
Nielsen, J. (1994). Usability Engineering. Cambridge, MA: Academic Press.
Norlin, E., & Winters, C. M. (2002). Usability testing for library web sites. Chicago: American Library Association.
Librarians at FIU are conducting a usability test of the FIU Library Web site. The purpose of the study is to evaluate the design and organization of the Web site. We would like you to participate in this study. We will not be measuring or evaluating your ability or performance. Don’t be afraid to answer truthfully. Our objective is to improve the library Web site.
The session will take place in ___________________________________ and will take approximately 20 minutes to complete.
If you choose to participate, we will need to know before your session if you are familiar with the FIU library Web site. If you are, please tell us how often and extensively you have used this site. If you have not used this site, please do not examine it until the time of your session.
If you wish to participate in the usability evaluation, the librarian who has contacted you will arrange a time to conduct the session that accommodates your schedule.
If you have any questions, please contact Sarah J. Hammill at hammills@fiu.edu or 305-919-5604.
Facilitator will open Netscape and go to the library homepage http://www.fiu.edu/~library and minimize the site before the subject arrives.
Facilitator will sit next to the subject.
-Thank you for participating in this usability test. I will be evaluating the usability of the Web site, not your performance.
-While you navigate the Web site, I’m going to ask you to “think out loud” and describe what you’re doing and why you are doing it. I won’t be able to give you feedback or answer your questions while you’re navigating the site. At the end of our session I will ask you a few questions about your experience and at that time I will answer any questions you might have.
-I have nine tasks for you to do and I will give them to you one at a time. While you are working through these tasks, I will be taking notes about what you are saying and doing. I will also be keeping track of approximately the amount of time it takes to complete each task. I estimate it should take 15-25 minutes to complete these tasks but there is not a time limit. Again I’m not measuring your performance; I’m measuring the usability of the site.
-Do you have any questions before you begin?