Electronic Journal of Academic and Special Librarianship

v.10 no.2 (Summer 2009)

Back to Contents

If You Build It They May Not Come: The Case of QuestionPoint

Lynette Ralph, Assistant Director
Southeastern Louisiana University, USA
lralph@selu.edu

Abstract

The Knowledge Base (KB) of QuestionPoint (QP) is a Knowledge Management (KM) tool capable of capturing the collective knowledge of reference librarians for future use. The goal of the study was to determine if this KB is an effective KM tool. Descriptive research was the methodology used and included an unobtrusive study, a survey instrument, and interviews. This study revealed that despite the technological capabilities of this KB, librarians who had access to the system failed to utilize it.

Introduction

This study investigated whether the Knowledge Base (KB) of QuestionPoint (QP) was an effective Knowledge Management (KM) tool that improved reference services. Over the years many researchers have demonstrated the ineffectiveness of reference services (Hernon and McClure, 1986; Dewdney and Ross, 1996; Kaske & Arnold, 2002; Profeta, 2006). It is evident that there is need for improvement. The literature revealed that many organizations are effective because of the use of KM. Knowledge Management could be defined as a concerted effort to capture critical knowledge, share information and capitalize on the collective organizational memory to enhance efficiency. Indeed, throughout the years, businesses have preserved their competitive edge because of the practice of knowledge sharing (Davenport and Prusak, 2000). In order to share knowledge, these businesses utilized KM and KM tools.

The researcher identified a tool that could be considered a KM tool for reference services. This tool was QuestionPoint (QP), a virtual reference service with a Knowledge Base (KB) component that could serve as a KM tool. The KB of QP is built on a technology that facilitates knowledge retrieval because it is searchable by keyword and can be browsed by subject. This KB also permits the capture both of explicit and tacit knowledge, and therefore facilitates knowledge sharing, knowledge exchange, and knowledge creation. Because the system facilitates knowledge creation, knowledge retrieval, and knowledge sharing, it can be considered a KM tool. This tool is capable of improving reference services because it can serve as a memory bank for reference librarians and help them to reduce duplication of effort.

To determine whether the KB of QP is as an effective KM tool, the researcher asked the following questions:

  1. To what extent is the KB of QP an effective KM tool?
    1. Is the KB of QP used by reference librarians?
    2. Does the use or lack of use affect duplication?
  2. What do librarians perceive as the benefits of using the KB of QP?
  3. What do librarians perceive as the problems of using the KB of QP?

This study consisted of a survey and an unobtrusive study. Twenty-two libraries participated in the survey, while 28 libraries were sent six questions unobtrusively through the use of six proxies. The unobtrusive portion of the study was conducted first. Two questions were repeated and this repetition allowed for the assessment of whether the KB served as a memory bank for librarians and avoided duplication of effort. It was found that reference librarians did not use the KB as a memory bank and thus duplicated their efforts.

Twenty questions were asked through the questionnaire, and these questions answered whether librarians used the KB, and if it served as a memory bank for librarians. The questionnaire also helped reveal the perceptions of librarians regarding the benefits and problems of using the KB of QP. Both the questionnaire and the unobtrusive study were instrumental in answering the research questions posed in the study. Through the questionnaire, it was revealed that while the reference librarians used some other features of QP, they did not generally use the KB of QP. Of the 22 libraries that participated in the questionnaire, 21 (96%) did not use the KB, and only 1 (4%) used the KB less than 5% of the time.

According to the research, while reference librarians acknowledge there could be many benefits to using the KB, they did not use the KB. The problems they identified with the KB included that it was time-consuming, it required an extra step, and that the content was not relevant, accurate, or current. As a result, the reference librarians did not use the KB as a memory bank, but repeatedly performed original research even when the questions were the same. Thus the lack of use of the KB of QP, and its inability to improve efficiency, rendered it ineffective as a KM tool.

Research Methods

The descriptive research design method was chosen to conduct this study. The study was accomplished using unobtrusive testing techniques, an analysis of the unobtrusive testing results, a survey instrument, and selected interviews.

Selection of the Population

The population was composed of the academic libraries in the United States that use QP. According to the Library of Congress (2006), there are 97 academic libraries that use QP. There was a pool of 45 libraries that were neither in the same geographical area, nor of the same university system. Of these 45 libraries, eight no longer used QP, and three declined to participate. Thirty-four libraries agreed to participate in the survey process. However, only 22 of the libraries responded to the questionnaire, resulting in a 64% response rate. Because of a change in summer hours, and the closure of the Virtual Reference Desk (VRD) service at some academic libraries during the summer months, only 28 (82%) of the 34 libraries responded to questions during the unobtrusive process.

Instruments Used

Survey and Interview Script

A survey of 20 questions was used for the survey and the same 20 questions were used for the interview. A total of 65% of QP libraries participated; 53% responded to the survey delivered by SurveyMonkey and 12% responded to the telephone interviews.

Unobtrusive Testing

Six questions were posted through QP to each of the libraries. For each question selected, a background story was written to provide authenticity for the request, in accordance with Elzy et al. and Ward who stressed the need for a cover story to protect the secrecy of the study.

Data Collection

The results of the unobtrusive testing and the responses to the questionnaire and the interviews constituted both qualitative and quantitative data collected for this study.

Unobtrusive Testing

For the unobtrusive testing, six questions were posted through QP to each of the libraries during various times of the days: morning, afternoon, evening, late night, and weekends. The researcher made every effort to ensure a fair distribution of questions by making sure that every library received questions at different times during the day and on weekends (Crowley & Childers, 1971).

Survey

Eighteen libraries (53%) from the pool responded to the survey, delivered by SurveyMonkey, a web-based survey instrument. The qualitative data provided an insight into the group and described the perceptions of the group (Charles, 1998). The quantitative data collected from each respondent was examined, placed in categories, and tabulated.

Interviews

The researcher randomly called 10 reference librarians who did not respond to the survey and tried to set up an appointment for an interview. The same survey questions were given to the non-responders to ensure consistency in the questions asked and to ensure that the non-responders did not represent “a biased group who [would have] answered the questionnaire in a markedly different manner than the responding group” (Borg & Gall 1999, p. 434). Four of the ten were available to be interviewed, resulting in a total of 22 libraries surveyed or interviewed, making an overall 65% response rate. Descriptive statistics including percentages, means, and standard deviations, were used to describe the sample.

Results

Research Question 1(a) - Is the KB of QP Used by Reference Librarians?

Only 1 (5%) library acknowledged using the knowledge base (KB), while twenty-one libraries did not. Twenty (91%) libraries used Chat Reference, 14 (63.6%) libraries used the feature to track questions, 12 (54.6%) libraries used the reporting features, and 5 (23%) used the e-mail feature.

Figure 1. Shows the features of QP that are being used by libraries.
Figure 1

Question 5 – Is there a mandate from your administration to use any specific feature of QP?
Thirteen (59%) libraries answered yes, admitting to having a mandate to use specific features, while nine (41%) libraries answered no.

Question 6 – If there is a mandate, which feature/s is/are mandatory?
Of the 13 libraries that had a mandate, all 13 (100%) were required to use the Chat feature, 5 (38%) were required to track questions, 5 (38%) were required to use the Reporting feature, 5 (38%) used the email, and no library (0%) had a mandate to use the KB of QP.

Question 7 – If there is no mandate, which features do you use?
Of the nine libraries that had no mandate to use any feature, seven (78%) used the chat feature while one used the KB.

Question 8 –When responding to a reference question, do your librarians always first review the KB of QP, or do they perform original research?
None of the librarians acknowledged always first using the KB of QP when responding to a reference question. However, all 22 (100%) libraries acknowledged they performed original research.

Question 9 – If the librarians do not always first review the KB, how often do they use this feature?
Twenty-one (96%) of the librarians said they never use the KB. Only one library said they used KB less than 5% of the time.

Question 10- If the librarians performed original research, what percentage of the time do they perform it?
Twenty-one (96%) of the librarians said they performed original research 100% of the time, while one library said they performed original research between 91 and 99% of the time.

Question 18 – Culture
Question 18 reflects the libraries’ view of their culture (see Table 1).  According to the responses to question 18, 20 (91%) of the libraries either strongly disagreed or disagreed that their reference librarians were required to use the KB. The remaining two libraries responded to this question by being neutral. Again, 20 (91%) of the libraries strongly disagreed or disagreed that their librarians were rewarded for using the KB and all 22 (100%) strongly disagreed or disagreed that their librarians were penalized or recognized for using the KB. Nineteen (87%) of the libraries believed that knowledge sharing was a normal part of everyone’s responsibilities and all 22 (100%) believed that knowledge sharing and collaboration improved performance. Thus while the reference librarians understood the value of sharing knowledge and collaborating with colleagues, as evidenced by their responses to parts “e” and “f”, there was no requirement, reward, penalty or recognition for knowledge sharing as evidenced by their responses to parts “a”, “b”, “c” and “d”.

Table 1. Assessing the Culture through Knowledge Sharing

 

 

Strongly
Disagree

Disagree

Neutral

Agree

Strongly
Agree

(a)

Reference Librarians are required to use the KB

18(82%)

2 (9%)

2 (9%)

 

 

(b)

Reference Librarians are rewarded for using the KB

17(77%)

3(14%)

 

 

 

(c)

Reference Librarians are penalized for using the KB

17(77%)

5(23%)

 

 

 

(d)

Reference Librarians are recognized for using the KB

17(77%)

5(23%)

 

 

 

(e)

Knowledge sharing responsibilities are a normal part of everyone's roles, responsibilities, and duties within the reference department

1(4.5%)

 

2(9%)

9(41%)

10(46%)

(f)

Sharing knowledge and collaborating with colleagues improve performance

 

 

 

7(32%)

15(68%)

Question 14 -   Did your librarians participate in training to use QP?
All 22 (100%) libraries indicated that their librarians participated in some form of training.

Question 15 – Who Provided Training
The method of training varied. In nine (41%) libraries, QuestionPoint provided the training. In seven (32%) libraries, the training was provided by internal trainers who had previously been trained by QP. A variety of other training methods also were employed. For example, one library said that the reference librarians looked at a webcast, then trained each other. Five libraries said that training was provided by QP, and then they trained each other internally. Seven libraries said that both QP and internal trainers provided the training, and these libraries also stated that all the new librarians were trained internally.

Question 16 – Was training adequate for effective use of QP?
Nineteen (86%) libraries believed that the training was adequate, while 3 (14%) believed that it was not adequate.

Question 17- Was the training adequate for the effective use of the KB of QP?
Six libraries (27%) believed that the training was adequate for the effective use of the KB while sixteen libraries (73%) believed that the training was inadequate for the effective use of the KB.

Question 1(b) - Does the Use or Lack of Use Affect Duplication?

Both a survey and an unobtrusive study were conducted. Questions 19 and 20 of the questionnaire answered this question.

Question 19 – Assessing Duplication of Workload
Question 19 assessed the level of duplication at the libraries, and revealed that there was considerable duplication of effort through lack of use of the KB. Most of the libraries, 21 (95%) either strongly disagreed or disagreed that through the use of the KB, duplication was reduced in their organization, as evidenced by part “a” (see Table 2). Part “b” revealed that all 22 of the libraries (100%) either strongly disagreed or disagreed that when their peers with specialized knowledge contributed to the KB, it reduced duplication of effort. Parts “c” and “d” revealed that all the librarians either strongly disagreed or disagreed that they used the KB to supplement training or time spent with unfamiliar resources. Table 2 reveals the duplication through lack of use of the KB.

Table 2. Duplication of Workload

 

 

Strongly
Disagree

Disagree

Neutral

Agree

Strongly
Agree

(a)

Duplication is reduced because the KB of QP is used instead of repeatedly performing original research

13(59%)

8(36%)

1(5%)

 

 

(b)

Peers with specialized expertise contribute to the KB of QP and this reduces the need for every librarian to research the same question

14(64%)

8(36%)

 

 

 

(c)

Using the KB supplements the time that new librarians spend with mentors.

18(82%)

4(18%)

 

 

 

(d)

Using the KB supplements training for new librarians

21(95%)

1(5%)

 

 

 

Question 20 – Assessing the Use of KB as a Memory Bank
Question 20 assessed whether the KB of QP served as a memory bank for the library. If the libraries used the KB as a memory bank, then they would avoid duplication. If they failed to use the KB as a memory bank then they duplicated their efforts. Based on part “a”, (see Table 3), 21 (95%) of the libraries either strongly disagreed or disagreed that the KB of QP serves as a memory bank. One library remained neutral. All the librarians either disagreed or strongly disagreed that the KB, through its use, aided in consistency. All the librarians 22 (100%) either disagreed or strongly disagreed that the KB served as a memory bank, and all the librarians either disagreed or strongly disagreed that the KB, through its use served as an additional resource. Table 3 showed that the KB did not serve as a memory bank, and therefore did not reduce duplication.

Table 3. Memory Bank

 

 

Strongly
Disagree

Disagree

Neutral

Agree

Strongly
Agree

(a)

Librarians at my institution enter offline transactions into the KB of QP and this serves as the institutional memory.

15(68%)

6(27%)

1(5%)

 

 

(b)

Librarians maintain consistency by always reviewing the KB before responding to a question received through QP.

17(77%)

5(23%)

 

 

 

(c)

The shared knowledge on the KB of QP serves as a memory bank for librarians.

19(86%)

3(14%)

 

 

 

(d)

When peers with specialized expertise contribute to QP, it aids in providing access to additional resources.

19(86%)

3(14%)

 

 

 

Unobtrusive Study

Two unobtrusive questions also revealed that the lack of use of the KB of QP affected duplication. Questions four and five, two of the six unobtrusive questions were resubmitted exactly as they were previously asked. In the case of question four, there was no relationship between the responses provided the second time and the responses provided the first time, as none of the libraries provided the identical response both times. In the case of question five, only one (4.5%) library provided the exact response when repeated. Additionally, there was a possible response to question five already existing in the KB. None of the libraries provided this response. It is clear that there was a high level of duplication with only 1 (4.5%) library providing the exact response part of the time.

Research Question 2 - What do librarians perceive as the benefits of using the KB of QP?

Figure 2. Benefits of the Knowledge Base of QuestionPoint
Figure 2

Question 12 of the Questionnaire answered this research question.

Question 12 – What are the benefits of using the KB of QP?
Three main reasons were identified as benefits of using the KB of QP. Seventeen libraries (77%) believed that the KB of QP could facilitate knowledge sharing. Eight (36%) believed that it could avoid duplication; four (18%) believed that it could save time if someone else asked the same question. During the interview, all four of the libraries interviewed made the point that while they did not use the KB, they believed that the benefits cited, represented what they perceive could be beneficial had they used the KB. Other specific responses included:

Research Question 3 – What do Librarians perceive as the problems of using the KB of QP?

Questions 11 and 13 of the Questionnaire addressed this research question.

Questionnaire Question 11 – The librarians do not use the KB of QP because:
There were several reasons given for performing original research in lieu of utilizing the KB. Four (19%) libraries said that the KB of QP is not user-friendly; 8 (38%) said finding content in the KB is difficult; 15 (68%) said it requires an extra step; 10 (48%) said using the KB is time-consuming; and 5 (24%) said using the KB is not required by superiors.

Figure 3. Librarians’ Reasons for Not Using the KB of QP
Figure 3

Other responses included:

Questionnaire Question 13 – What are the problems of using the KB of QP?
Eight problems were identified with using the KB of QP. Five (23%) libraries believed that using the KB of QP was time-consuming; 15 (68%) believed it required an extra step; 1 (4%) said “I never think about it, and I have to think about it to do it;” 4 (18%) libraries believed not using the KB of QP was due to inadequate training; 2 (9%) believed its use was not emphasized in training; and 3 (13.5%) believed that its use and its value were not well known. Eight (38%) of the librarians revealed that finding content is difficult, and 6 (27%) believed that the content was not relevant or adequate for their needs.

Figure 4. Problems Using the KB of QP
Figure 4

Other responses included:

Demographic Responses from the Questionnaire

The demographic responses were obtained from questions 1, 2, and 3.

Question 1 – Title of Person Completing the Questionnaire
Although the titles of the persons completing the questionnaire varied, in every case, the questionnaire was completed by persons who were responsible for the overall reference services of the library. Fourteen responses were from Head of Reference; two responses were from Coordinator of Reference services; two were from Coordinator of Access Services; one was the Assistant Director, Faculty Services; one was Assistant Director, Public Services; one was the Information Commons (IC) Librarian; and one was the Learning Commons Librarian. Overall the responses came from persons of authority with responsibility for the Reference Department.

Question 2 – Size of the Library Collection
The libraries participating in QP varied in size, with the majority of the libraries having collections of over 250,000 volumes. Two libraries (9%) had collections of 50,000-100,000; eight libraries (36.4%) had collections of 100,001–250,000; and 12 libraries (55%) had collections of over 250,000. Overall, 20 (91%) of the libraries who participated had a collection of over 100,000.

 Question 3 – How long has the library used QP?
Two libraries (9%) had used the QP system for less than one year; 17 libraries (77%) used the system between 1-3 years; while three libraries (14%) used the system more than four years; and 20 (91%) of the libraries had used the system for more than one year. The one library that used the KB less than 5% of the time had been using QP for 1-3 years.

Conclusion

The purpose of this study was to determine to what extent the KB of QP was an effective KM tool, and to ascertain the perceptions of librarians on the benefits and problems of using the KB of QP. The KB of QP met technological standards and was capable of serving as a KM tool.  It could facilitate consistency and eradicate duplication. It was also readily accessible to all. However, there was insignificant use made of the system. According to the literature, one of the most critical reasons for KM failure is lack of use. Barth (2004) aptly sums it up when he observed, “Don’t believe if you build it, they will come.” Thus although the Knowledge Base of QuestionPoint was well-built, no one came.

Appendices

Bibliography

Barth, S. (2004). KM horror stories. KM, 3(10), 36-40.

Crowley, T., & Chiders, T. (1971). Information service in public libraries: Two studies. Metuchen, N.J.: Scarecrow.

Davenport, T. H. & Prusak, L. (2000). Working knowledge: How organizations manage what they know. Boston: Harvard Business School Press.

Dewdney, P., & Ross, C.S. (1996). Flying a light aircraft: Reference service evaluation from a user’s viewpoint.Retrieved April 14, 2005, from http://www.ala.org/ ala/rusa/rusapubs/rusq/specialfeatures/rspawardw

Gall, J., Gall, M., & Borg, W. (1999). Applying educational research: A practical guide. New York: Longman.

Herndon, P., & McClure, C. (1986). Unobtrusive reference testing: The 55 percent rule. Library Journal, 11l, 37-41.

Hernon, P., & McClure, C. (1987). Unobtrusive testing and library reference services. Norwood, N.J.: Ablex.

Kaske, N. & Arnold, J. (2002). Unobtrusive evaluation of online real time library reference service. Retrieved March 15, 2007, from http://www.lib.umd.eu/groupsdigref/kaskearnold.unobtrusive.html

Profeta, P. C. (2006). Effectiveness of asynchronous reference services for distance learning students within Florida’s community college system. Digital Dissertations. Proquest Database. (UMI AAT 3212018).

Back to Contents