Electronic Journal of Academic and Special Librarianship

v.4 no.2-3 (Fall 2003)

Digital Reference Evaluation: Assessing the Past to Plan for the Future

Back to Contents

J B Hill, Head of Reference
Southeastern Louisiana University

Cherie Madarash-Hill, Cataloger
Southeastern Louisiana University

Ngoc Pham Thi Bich, Program Officer
State Bank of Vietnam - German Technical Assistance Project


Historically, libraries have routinely evaluated their reference services both quantitatively and qualitatively. With the increasing volume of digital reference transactions, there is a need to modify and apply to digital reference, the same evaluation techniques that have been used successfully in the evaluation of traditional reference service. At Southeastern Louisiana University, a recent evaluation of the library's Ask a Librarian digital reference service included an unobtrusive observation of current practice through a detailed examination of archived reference transactions, as well as an assessment of future needs through a survey of users. Through this evaluation, it was possible to assess the effectiveness of the library's e-mail reference service, as well as to plan for the implementation of a new real-time digital reference service.


Asynchronous digital reference systems (using e-mail, Web forms and bulletin boards) have been in existence in some libraries since the 1980's, however most of the growth in digital reference has been since the late 1990's. Most academic libraries now offer asynchronous, e-mail based reference services, and some also have real-time systems featuring chat and Web page co-browsing. As Lankes and Shostack indicate "real-time systems and asynchronous systems will need to coexist" for the foreseeable future as "different questions and different users will require different forms of interactions" (2002).

As libraries make the transition from asynchronous e-mail reference to synchronous systems, evaluating the past can aid in planning for the future. At Southeastern Louisiana University, two evaluations of the library's Ask A Librarian digital reference service were conducted in 2001 and 2002 to aid in the transition to a real-time service. The results of the 2001 assessment have been previously reported (Hill and Madarash-Hill 2001). This report includes some of the findings of the 2002 assessment.


Founded in 1925, Southeastern Louisiana University is located in Hammond, Louisiana. As a state-supported, regional public university, Southeastern provides instruction to approximately 15,000 undergraduate and graduate students. A large number of these students are non-traditional commuters. To accommodate a new generation of non-traditional learners, Southeastern has become the second largest provider of distance learning in the state of Louisiana. As a result, there is a large body of students using library resources from off-campus.

To provide assistance to this large group of off-campus learners, the Reference Department of Southeastern's Sims Memorial Library launched its Ask a Librarian e-mail reference service in October 1997. The Ask A Librarian service is similar to other asynchronous digital reference services in academic libraries, as reported in the literature (Goetsch 1999; Lederer 2001; White 2001; Pomerantz forthcoming). An e-mail address, Web form, and link (http://www.selu.edu/Library/forms/askref.html) are available on the Library's Web pages.

Responsibility for answering questions was initially assigned to one librarian who answered the questions during times when she was not scheduled on the reference desk. Due to the increasing number of questions and the need for quicker response time, the e-mail service was moved to the reference desk so that the e-mail questions could be answered upon arrival as a routine part of the reference desk activities.

Use of the Ask A Librarian service was initially low but the service experienced rapid growth in the first 3 years of the service before leveling off at approximately 700 questions per year.

In the Fall of 2002, the library began offering Ask a Librarian Live, a real-time digital reference. Prior to the initiation of this real-time service, two annual evaluations were performed in 2001 and 2002. The findings of these evaluations were instructive in setting the initial parameters of the new synchronous service.

Evaluation of Digital Reference

There has been a tremendous growth in the literature related to digital reference. Bernie Sloan's ever-growing bibliography currently contains more than 600 entries (2003). However, as Gross, McClure and Lankes suggest, digital reference evaluation is an underdeveloped area of study (2001). Perhaps due to the small volume of activity and minimal impact on other reference services, most libraries have done little in the way of formal assessment of their services. One survey reports that only 9% of libraries have evaluated their digital reference services (Janes 2002). Another suggests that a large number of libraries do not even maintain statistics of their digital reference transactions (Corrigan, Diamond and Hill 2001).

Nevertheless, a number of authors have written about digital reference evaluation. Most have provided anecdotal reports of specific evaluations. In an early study, Bushallow, DeVinney and Whitcomb give an evaluation of e-mail reference at the University of New York, Buffalo, with an analysis of questions by month, type of question, day of the week, time of day, and department affiliation (1996). Hahn examines e-mail reference at the University of Maryland by type of assistance requested and breadth of assistance provided (1997). Johnson and Grustin (1995) and Henson and Tomajko (2000) examine Georgia Tech's digital reference service. Shelby provides an analysis of e-mail reference at the University of Virginia Law School by type of question (1999) and Schilling-Eccles and Harzbecker document the results of a survey done at the Boston University Medical Center Library (1998). Carter and Janes report on digital reference logs from 3000 questions received by the Internet Public Library (2000). Hoskisson and Wentz profile e-mail reference use and librarian participation at Utah State University (2001). Powell and Bradigan chronicle e-mail reference at the Ohio State University Health Sciences Library (2001). Lederer provides a survey of e-mail reference services in the state of Colorado as well as a detailed analysis of two years of service at Colorado State University (2001). Ruppel and Fagan report on a survey of users of Southern Illinois University's instant messaging service (2003).

Recently there have been some attempts to provide a broader approach to digital reference evaluation. Wasik provides a good overview of the topic in her recent paper (2003). McClure, Lankes, Gross, and Choltco-Devlin provide a detailed guide to developing standards and measuring the performance of digital reference services (2002).

Evaluation Methodology at Southeastern

The methods of evaluation employed at Southeastern included an examination of archived digital reference transactions for the past year (i.e., a variation of unobtrusive observation) and the distribution of a user survey to the patrons of the most recent semester.

As Carter and Janes suggest, "a wealth of information can be culled from the data surrounding a digital reference question" through the use of simple Web forms (2000). However, the collection and use of this data raises some ethical issues (Hill and Johnson forthcoming). In recognition of the rights of patron privacy, we have only gathered the most basic information that is needed to answer the question, including an e-mail address to send the answer, and questions concerning university affiliation, status (i.e., Faculty, Graduate Student, Undergraduate, Staff or Other), and distance learning status, as these give information concerning which resources can be used in answering the question. Nevertheless, a significant amount of data was available through a transaction analysis of 531 questions and responses.

In addition to studying archived transactions, a short questionnaire was developed after examining examples of user surveys published in an Association of Research Libraries' Spec Kit (Engelbrecht and Westerman 1988). User surveys are an established method of evaluated library services in general and reference service in particular. There are a number of examples in library literature of the application of user surveys for digital reference evaluation (Bushallow-Wilbur, DeVinney and Whitcomb 1996; Carter and Janes 2000; Ruppel and Fagan 2003).

At Southeastern, the user survey was intended to determine the effectiveness of the current e-mail reference service as well to help plan for a future migration to a real-time virtual reference service. Consequently, the questionnaire was designed to determine the following:

The questionnaire was kept short to minimize the amount of time required to complete the survey, thus increasing the likelihood of receiving a response.

The questionnaire was sent via e-mail to the e-mail addresses of 261 patrons who had asked questions during the most recent semester. The questionnaire was sent individually (i.e. in separate messages rather than a bulk mailing) in order to maintain the confidentiality of those patrons using the service. Since it was an e-mail questionnaire (rather than a Web questionnaire), respondents did not enjoy complete anonymity (unless they responded via U. S. mail). This may have been a disincentive to respond to the survey.

Respondents were asked to e-mail, fax or mail their responses. All respondents chose to e-mail their responses. The survey response rate was low, as only 37 responses were received, resulting in a 14.2% response rate. Of the respondents, 70% were students, 14% were faculty, 5% were staff members and 11% were not affiliated with the university. Despite the relatively small number of survey responses, the survey results are likely to be indicative of users' opinions, as the results were almost identical to the results of the previous year's survey.

Evaluation Findings

Patrons Served

The Ask A Librarian e-mail reference service is available to all users, regardless of university affiliation. As a result, a significant number of patrons (13%) who identified themselves as not being currently affiliated with Southeastern Louisiana University asked questions during the year. Many of these were residents of southeastern Louisiana, had ties to the university, or were soliciting information specific to the university or the library's collections.

Users of Service Percent
Undergraduate Students 39 %
Graduate Students 15 %
Faculty/Staff 12 %
Other 13 %
Unknown 21 %

Most of the requests came from students. One of the questions patrons were asked on the e-mail reference request form was "Is this for a Distance Learning/Off Campus class?" Of the students asking questions, 22% identified their requests to be for distance learning classes.

The service does have a number of repeat users. However, 88% of the Ask a Librarian patrons used the service only once during the semester.

Peak Times

In the survey of e-mail reference users, when asked what times they would most likely ask a question, 20% of the respondents indicated more than one time period. However, the greatest number of users (51%) indicated that the evening hours between 6 p.m. and midnight were the hours that they would most likely ask a digital reference question. Twenty-two percent indicated a preference for the service between noon and 6 p.m., 11% indicated a preference for the morning hours between 6 a.m. and noon, and 19% were not sure or gave no response.

An examination of the digital reference transactions by the time that the questions were sent indicates that there was some demand for digital reference in the evening hours (between 6 p.m. and midnight); however, the majority of questions were asked during the daytime hours. A breakdown by time of day follows.

Time of Day Percent
Between Midnight and 6 AM 3 %
Between 6 AM and Noon 34 %
Between Noon and 6 PM 42 %
Between 6 PM and Midnight 21 %
During Library Hours 94 %

Surprisingly, the digital reference requests were similar to the physical reference desk requests, peaking around midday. However, late afternoon transactions were proportionally heavier than typical reference room activity. It is interesting that 94% of the questions were submitted during regular library hours (7:30 a.m. to 11:00 p.m.) with 76% being submitted during daytime hours. Since the university is a commuter campus, this is a bit surprising. It might be expected that students would be more active from home in the evenings, resulting in more questions during this time.

Looking at one semester's data broken down by day of the request, the trend of peak activity between noon and 2 p.m. held true for all weekdays, Monday through Friday.

On the weekends, the number of questions tended to be less and the questions tended to come later in the day with the peak activity occurring in the early evening.

Response Time

The Ask a Librarian e-mail reference service only promises a 24-hour turnaround time. Specifically, the Web pages states: "Questions will be answered within 24 hours. Questions submitted during library hours will usually be answered within 2 to 3 hours. If your question involves in-depth research which cannot be answered within a 24 hour period, you will be notified by e-mail." The 2 to 3 hour projection was added as a result of data collected in the 2001 evaluation.

In general, the service provides reasonably fast response times. Eleven percent of the questions were answered within 15 minutes. Forty percent were answered within an hour. Fifty-eight percent were answered within the suggested response time of 2-3 hours. Only 2% of the questions were not answered within 24 hours. In all cases of response times of greater than 24 hours were due to the library being closed for holidays.

Generally, the response time was faster in the evening hours from 6 to 9 p.m. presumably due to decreased reference room activity. Not surprisingly, the slowest response times occurred during late evening, early morning, and weekend hours when the library was closed.

Patrons responding to the user survey indicated a generally high level of satisfaction with the speed of response, 76% were "Very Satisfied" and 16% were "Satisfied." Only 8% were "Not Satisfied." However, one of the most frequent suggestions for an improvement of the service by respondents of the survey was to have faster response to questions.

Questions Received and Answers Provided

The greatest number of questions dealt with using library databases. The other questions were general reference requests, seeking answers to specific questions. Follow-up questions for clarification (where no answer was given) were attempted for only 4% of the questions.

Types of Answers
Assistance in remote use of library resources 28 %
Short answer provided 19 %
Referral to library databases 13 %
Referral to other library departments 10 %
Referral to Web sources 7 %
Referral to print sources in library 7 %
Detailed research assistance 7 %
Follow-up question asked 4 %
Referral to campus resources or departments 3 %
Document delivery 1 %
Patron asked to come to library for assistance 1 %

Based on the questions asked and the answers given, an assessment was made as to the level of staff needed to answer the digital reference questions. Staffing at the physical reference desk at Sims includes both reference librarians and graduate assistants. An analysis of the questions received by complexity and scope indicated that 65% of the questions needed the expertise of a librarian to be answered adequately. Less-experienced, less-trained staff members could reasonably be expected to handle the other 35%.

Quality of Answers

Assessing the quality of digital reference answers is an admittedly subjective process. In cases where factually incorrect answers are provided, the lack of quality is clear. In other instances, an answer may be factually correct but needs improvement due to a failure of the librarian to provide an appropriate level of assistance.

A review by the Head of Reference of the responses sent in 2002 found that 93% of the answers given were judged to be good responses, appropriate to the patron's needs and the department's service parameters. While in an ideal world there would never be any poor reference responses, the service's performance was good, particularly given Kaske and Arnold's findings that e-mail reference services provide correct answers only 60% of the time (2002).

Of the 7% of answers that appeared weak, the greatest number (4%) were judged to be weak due to an incompleteness of the answers. The nature of e-mail correspondence with lags in follow-up questions requires that answers be as complete and specific as possible, suggesting all reasonable possibilities and anticipating follow-up questions. In the other cases, the answers were poor due to the provision of misleading or incorrect answers (1%); the failure to address the question (1%); or negative tone, poor grammar, or typographical errors (1%).

The results of the user survey indicate that users were generally satisfied with the Ask A Librarian service. In regards to their satisfaction with the answer they received, 84% were "Satisfied " or "Very Satisfied" while 11% were "Not Satisfied." An indicator of satisfaction is the willingness to use the service again. Ninety-five percent of the survey respondents indicated that they would use the service again.

Other Results of the User Survey

The great majority (81%) of the users had learned about the service through the Library's Web page. Sixteen percent had been learned about the service through a library instruction session (either a bibliographic instruction session or the library's information literacy course). Five percent had discovered the service by other means (e.g. a Southeastern instructor).

The literature suggests that prominent placement of links on Web pages is important, as that is how most users find a digital reference service. Our survey results indicate that this is the case for our users. The results also may indicate a weakness in our promotion of the service through other means (e.g., library instruction sessions). This is an effort we have begun to make since the completion of this survey.

Two questions on the survey were designed to solicit input on the need for real-time digital reference service. One question asked, "If a chat room were available for immediate response to questions, would it affect your frequency of using the service?"

Survey respondents indicated a preference for the use of chat, as 59% indicated that they would be more likely to use the service if immediate responses were forthcoming, only 3% indicated that they would be less likely to use a chat-based reference service, and 38% indicated that there would be no difference in their likeliness to use the service.

In the comments portion of the survey, the most common suggestion for improvement of the service was faster response. A closer examination of the survey responses found that that some of the survey respondents who suggested a chat room or the need for faster response selected "No Difference" in response to the question, suggesting that they were willing to use either service as the need arose.

The other question, related to the implementation of a real-time digital reference service, was the time that that digital reference users would likely ask questions. As previously reported, the greatest number of users (51%) indicated that the evening hours between 6 p.m. and midnight were the hours that they would most likely ask a digital reference question.

Implications for a Real-Time Digital Reference Service


The results of our user survey indicated that the e-mail reference users were generally satisfied with the speed of response. However, many users indicated a desire for a real-time service and identified evening hours as the period of greatest need.


While the survey respondents indicated a need for night service, the analysis of e-mail reference user behavior indicated that most of the demand was during the daytime hours of the days early in the week. There has been recent discussion in the literature about the need for 24/7 service; however, there is little evidence of significant demand beyond typical library hours (Jackson 2002; Sloan 2001). The digital reference pattern at Southeastern did not indicate a significant demand for a 24/7 service, as over 70% of the demand could be met by day hours and over 90% could be met by typical library hours.

Based on the past trend of digital reference demand, the Ask A Librarian Live service was initially established with daytime hours during the week. Evening coverage was offered for the two most active evenings (Monday and Tuesday). No weekend coverage was initially offered.


Due to the labor-intensive nature of real-time digital reference transactions, it is common for libraries to staff real-time digital reference services away from the reference desk (Ronan 2003). Our e-mail reference transactions indicated a relatively small number of answers based on the print collection. Consequently, there was little perceived need to have the service at the reference desk.

As locating the service away from the desk required the creation of a new service point, staffing was a key issue in the creation of a real-time service. At Southeastern, graduate assistants (from a variety of majors) are employed to assist librarians at the reference desk. Since only a third of the digital reference questions were within the scope of their training and expertise, they could be used in a real-time setting only if a system of triage were established so that questions could be forwarded to reference librarians as needed. To date this has not occurred, as only reference librarians have staffed Ask A Librarian Live away from the reference desk.


Since over one half of past e-mail reference transactions consisted of questions concerning the use of Web databases/resources and/or answers that contained referrals to electronic resources, there was an obvious need for real-time digital reference software that would permit Web page pushing and co-browsing.

In Fall 2002, Southeastern began its real-time digital reference service using the LSSI Virtual Reference Toolkit (VRT). Introduced by Library Systems & Services, LLC (LSSI) in 2000, the Virtual Reference Toolkit permits

Although the LSSI VRT is one of the more expensive digital reference packages, its high degree of functionality effectively addresses the needs of students in the use of Web-based resources. This functionality has enabled Southeastern to not only meet the need of remote students for reference assistance but also their need for instruction (Hill and Stahr 2003).


Planning for the creation of a real-time digital reference service is a challenging undertaking. However through an evaluation of an existing e-mail reference service, needs and trends of digital reference users can be identified, resulting in an improvement in the existing asynchronous service, as well as more effective deployment of a new synchronous service.

Whitlatch (2001) suggests that at the outset of any reference service evaluation that the purpose of the evaluation is clear. The primary purpose of this 2002 digital reference evaluation was to prepare for the creation of a real-time digital reference service. Based on this evaluation, Southeastern was able to better implement a real-time digital reference service. As both digital reference services (asynchronous and synchronous) continue to grow, evaluation will need to be an ongoing process.

Literature Cited

Bushallow-Wilber, Lara, Gemma DeVinney and Fritz Whitcomb. "Electronic Mail Reference Service: A Study." RQ 35, no. 3 (Spring 1996): 359-71.

Carter, David S. and Joseph Janes. "Unobtrusive Data Analysis of Digital Reference Questions and Service at the Internet Public Library: An Exploratory Study." Library Trends 49, no.2 (Fall 2000): 251-265.

Corrigan, Andy, Tom Diamond and J.B. Hill. "Electronic Reference Service Trends Among Louisiana Academic Libraries." Louisiana Libraries 65, no. 1 (Summer 2002): 3-12.

Engelbrecht, Pamela N. and Melvin E. Westerman. User Surveys. SPEC Kit 148. Washington, D.C.: Office of Management Services, Association of Research Libraries, 1988.

Goetsch, Lori. Electronic Reference Service. SPEC Kit 251. Washington, DC: Office of Leadership and Management Services, Association of Research Libraries, 1999.

Gross, Melissa, Charles R. McClure and R. David Lankes. "Assessing Quality in Digital Reference Services: Overview of Key Literature on Digital Reference." Information Use Management and Policy Institute, Florida State University, 2001. http://quartz.syr.edu/quality/VRDphaseIILitReviw.pdf (accessed September 1, 2003).

Hahn, Karla. "College of Library and Information Services University of Maryland at College Park: An Investigation of an E-Mail Based Help Service." CLIS Technical Report no. 97-03, 1997. http://www.clis.umd.edu/research/reports/tr97/03/9703.html (accessed September 1, 2003).

Henson, Bruce and Kathy Gillespie Tomajko. "Electronic Reference Services: Opportunities and Challenges." Journal of Educational Media & Library Services 38, no. 2 (2000): 113-21.

Hill, J. B. and Eric Johnson. "Ethical Issues in Digital Reference." In Information Ethics in the Electronic Age: Current Issues in Africa and the World, edited by Thomas Mendina and Johannes J. Britz. Jefferson, North Carolina: McFarland, (forthcoming).

Hill, J. B. and Cherie Madarash-Hill. 2001. "Evaluation of Electronic Reference: Assessing the Present to Improve the Future." Poster presented at the 20th Annual Poster Sessions of the American Library Association Annual Conference, San Francisco.

Hill, J. B. and Beth Stahr. "Instructional Opportunities in Virtual Reference." THE Forum E-Proceedings. Baton Rouge: Louisiana State University, 2003. http://www.disd.lsu.edu/CFD/E-Proceedings/Instructional%20Opportunities%20in%20Virtual%20Reference.htm (accessed September 1, 2003).

Hoskisson, Tam and Deleyne Wentz. "Simplifying Electronic Reference: A Hybrid Approach to One-On-One Consultation." College & Undergraduate Libraries 8, no. 2 (2001): 89-102.

Jackson, Michael Gordon. "A Rush to Serve: Digital Reference Services and the Commitment to 24/7." Advances in Librarianship 26 (2002): 299-317.

Janes, Joseph. "Digital Reference: Reference Librarians' Experiences and Attitudes." Journal of the American Society for Information Science and Technology 53, no. 7 (2002): 549-566.

Johnston, Pat and Ann Grusin. "Personal Service In an Impersonal World: Throwing Life Preservers To Those Drowning In an Ocean of Information." The Georgia Librarian 32 (1995): 45-9.

Kaske, Neal, and Julie Arnold. "An Unobtrusive Evaluation of Online Real Time Library Reference Services." Paper presented at the Library Research Round Table, American Library Association Annual Conference, Atlanta, Georgia, June 15, 2002. Also available online at http://www.lib.umd.edu/groups/digref/kaskearnoldunobtrusive.html (accessed September 1, 2003).

Lankes, R. David and Pauline Shostack. "The Necessity of Real-Time: Fact and Fiction in Digital Reference Systems." Reference and User Services Quarterly 41, no. 4 (2002): 350-355.

Lederer, Naomi. "E-Mail Reference: Who, When, Where, and What Is Asked." Reference Librarian 74 (2001): 55-73.

McClure, Charles R., R. David Lankes, Melissa Gross, and Beverly Choltco-Devlin. Statistics, Measures, and Quality Standards for Assessing Digital Reference Library Services: Guidelines and Procedures. Syracuse, NY: Information Institute of Syracuse, 2002.

Pomerantz, Jeffrey, Scott Nicholson, Yvonne Belanger and R. David Lankes. "The Current State of Digital Reference: Validation of a General Digital Reference Model through a Survey of Digital Reference Services." Information Processing & Management (forthcoming).

Powell, Carol A. and Pamela S. Bradigan. "E-Mail Reference Services: Characteristics and Effects on Overall Reference Services at an Academic Health Sciences Library." Reference & User Services Quarterly 41, no. 2 (2001): 170-8.

Ronan, Jana Smith. "Staffing a Real-Time Reference Service: The University of Florida Experience." Internet Reference Services Quarterly 8, no.1/2 (2003): 33-47.

Ruppel, Margie, and Jody Condit Fagan. "Instant Messaging Reference: Users' Evaluation of Library Chat." Reference Services Review 30, no. 3 (2002): 183-197. Also available online at: http://www.lib.siu.edu/~jfagan/papers/imref.html (accessed September 1, 2003).

Schilling-Eccles, Katherine and Joseph J. Harzbecker, Jr. "The Use of Electronic Mail at the Reference Desk: Impact of a Computer-Mediated Communication Technology on Librarian-Client Interactions." Medical Reference Services Quarterly 17, no. 4 (Winter 1998): 17-27.

Shelby, Barbara. "Refdesk: UVA Law Library's Approach to E-Mail Reference." Virginia Libraries, 65, no. 2 (April/June 1999): 12-3.

Sloan, Bernie. "Digital Reference Services Bibliography." 2003. http://www.lis.uiuc.edu/~b-sloan/digiref.html (accessed September 1, 2003).

Sloan, Bernie. "Ready for Reference: Academic Libraries Offer Live Web-Based Reference. Evaluating System Use. Final Report." Graduate School of Library and Information Science, University of Illinois at Urbana-Champaign. July 11, 2001. http://www.lis.uiuc.edu/~b-sloan/r4r.final.htm (accessed September 1, 2003).

Wasik, Joann M. "Digital Reference Evaluation." Virtual Reference Desk. http://www.vrd.org/AskA/digref_assess.shtml (accessed September 1, 2003).

White, Marilyn Domas. "Diffusion of an Innovation: Digital Reference Service in Carnegie Foundation Master's Academic Institution Libraries." Journal of Academic Librarianship 27, no. 3 (2001): 173-187.

Whitlatch, J. B. "Evaluating Reference Services in the Electronic Age." Library Trends 50, no. 2 (2001): 207-217.

Back to Contents