Electronic Journal of Academic and Special Librarianship

v.8 no.1 (Spring 2007)

Back to Contents

Garbage In, Garbage Out: The Effect of Library Instruction on the Quality of Students’ Term Papers

Susan Hurst
Miami University, Ohio, USA
hurstsj@lib.muohio.edu

Joseph Leonard
Miami University, Ohio, USA

Abstract

The authors report the results of a study which used citation analysis of students’ term papers to determine the effectiveness of a library instruction session.  The research was conducted during the 2004-2005 school year.  In each semester, two sections of the same class received a library instruction session, while the third section of the class did not.  Bibliographies of the students’ term papers were then examined to determine if the numbers and types of sources cited differed between the two groups.  Library instruction was determined to be effective, in that students receiving library instruction were significantly more likely to cite journal articles and other scholarly resources than those students not receiving the library instruction.

Introduction

Tired of reading and grading mediocre papers, all of which cite flimsy sources, if they cite any at all?   It is said that we are living in an “Information Age”.  So, why then, are papers so often lacking in solid, factual information from scholarly sources?  Part of the answer may lie with beliefs and assumptions.  Because today’s undergraduates are skilled at surfing the web, it is often assumed that they will be equally proficient in locating the data needed for their papers and assignments.  Despite their web-surfing skills and technological acumen, however, students may still not know how to effectively search and locate scholarly research articles on a topic.  As the proverb says, when all you have is a hammer, everything looks like a nail.  In this case, the hammer is too often Google or some other Internet search engine. 

Another problem is that while it is assumed that students are taught how to use the library during an introductory English course or through a library orientation, this is frequently not the case.  College and university libraries are also often much larger and have many more resources than the high school or public libraries that students may have used previously.  In addition, academic libraries also increasingly subscribe to commercial databases that include full-text online access to scholarly journal articles, market research reports, and company financial data, all of which are only accessible through the library’s portal.

Thus, a path to improved student papers is making students aware of the wealth of resources available to them.  However, faculty members may not be well-suited to this task either.  Faced with the ever-increasing demands of their own profession as well as new course content to constantly assimilate into the syllabus, keeping up with the latest in library resources often falls by the wayside, to their own detriment as well as the students’.  In fact, a recent study of business faculty found that over one-third (36%) did not themselves use library resources for their own research (Dewald, 2005). Library instruction offers an alternative means of “getting the word out” about these resources, but how effective is it?  Do the students get anything out of it?  Does it make a difference in what resources they eventually use to complete their class assignments?  If they do use library resources, does that result in a better paper, with a subsequently better grade?

The present study was designed to address these issues by examining the effect of library instruction on undergraduates’ term papers.  Through analysis and comparison of the bibliographies, it could be learned what resources students used and what effect if any that had on the grades they received.

Literature Review

Citation analysis of student term papers has become a popular means of determining what sources students are utilizing to support their research.  There are many different approaches to citation analysis, ranging from a simple count of citations to in-depth analysis of source sophistication.  A comparison of these various methodologies was conducted by Gratch (1985).  Gratch concluded that key components of a valid citation analysis study include a carefully chosen sample, working closely with faculty, and having clearly defined and objective criteria for evaluating the citations.  Two different questions are also often raised in the literature.  One, does library instruction have an effect on students’ use of various resources?   Two, is there a correlation between the resources students use and the grades their papers receive? 

Looking at the first question, Davis (2003) conducted a longitudinal study from 1996-2001 to examine the effect of library instruction as compared to the increasing influence of the Internet on students’ research in an introductory economics course.  He also studied the effect of more stringent bibliography guidelines by faculty.  He found that despite library instruction, citations of scholarly sources declined over the time period.  However, not surprisingly, when faculty began to mandate specific types of resources required for the papers, scholarly citations rose again.  However, as only one class was examined each year, there was no control group which did not receive the library instruction.

Robinson and Schlegl (2004) expanded on Davis’ work, testing whether his results were generalizable outside the US and also outside the field of economics.  They included a control group but their sample sizes were small (N=84).  The results were similar to Davis’, in that they found the most powerful effect on students’ bibliographies came from professors mandating specific requirements for sources to be used, rather than the effect of library instruction alone.  They also addressed the second question by correlating the citation analysis results with the grades students’ received on their term papers and found a positive relationship between number of citations and the grade received (Robinson and Schlegl 2004).  Hovde (2000), focusing specifically on the library instruction sessions themselves, found them to be somewhat effective.   She examined the bibliographies from papers written by freshman English students, all of whom had been exposed to a library instruction session, and found that they cited sources located through library databases.  However, once again there was no control group, making it difficult to attribute the results solely to the instruction session. 

Hinchcliffe (2000) examined the effect of expanding the research process on term paper grades.  He gave students in his class the option of completing a more exhaustive research program, including submitting a research plan, a research log, and an annotated bibliography, all prior to writing their papers.  Not surprisingly, he found that those who went through this process received a slightly higher grade on average.  However, once again there was no real control group, with students self-selecting which group to be in.  Also, he too suffered from a very small sample size, with only 26 students participating overall.

The present study builds on the findings of the prior research while addressing some of the methodological limitations. The combination of a control group that was not self-selected and a larger sample size (N = 184) better isolates and identifies the effect of the library instruction.  As both Davis (2003) and Robinson and Schlegl’s (2004) work included faculty mandated bibliography guidelines, we deliberately did not have specific guidelines or requirements, in order to better test the effectiveness of the library instruction session alone. We will address both research questions raised by the following two hypotheses.

Hypotheses

The purpose of this study was to determine whether a one-time library instruction session would have an effect on students’ use of library resources.  Based on the literature that indicates that library instruction has an effect on the sources cited in students’ papers, this leads us to Hypothesis 1.

Hypothesis 1 – Students in the sections that receive library instruction will be more likely to cite a larger number of sources overall, use a greater variety of resources, and cite resources located using the library tools, (e.g. the catalog or databases), than the students in the sections that did not receive instruction.

In turn, it was felt that students who utilized the library’s resources and cited scholarly materials would receive higher grades on their term papers.  This comes from the research indicating that library instruction and an improved research process lead to improved term paper quality.  Thus, Hypothesis 2.

Hypothesis 2 - Students in the sections that received library instruction would earn higher grades on their term papers than the students not receiving the instruction.

Methodology

Data was collected in the Fall and Spring semesters of the 2004-5 school year.  In each semester, a single 45 minute library instruction session was provided to two sections of a junior level International Business class at a Midwestern university.  A third section of the same class did not receive any library instruction.  Out of a total of 184 students over both semesters, 101 (55%) students received library instruction, while 83 (45%) students did not.  Each semester all three sections of the class were taught by Dr. Joseph Leonard, one of the Principal Investigators of the study.  Each of the library instruction sessions was conducted by Susan Hurst, Business Librarian, the other Principle Investigator.  The instruction sessions were tailored to one of the main assignments for the class.  This assignment was a term paper based on research and analysis about a company.  The students usually chose large public companies but in some cases selected smaller family-owned businesses based on personal connections or interests.  The company they chose in turn greatly affected the amount and type of information available to them. 

The library instruction session itself was deliberately designed to be a typical type of session.  Due to constraints on the availability of computer labs, the session was presented in the regular classroom, with no opportunity for hands-on interaction.  The Business Librarian was introduced by the professor who remained in the classroom throughout the presentation.  A computerized instructor station equipped with a projector allowed information to be seen on screens around the room. 

The session began with an overview of the library’s website including information on how to access library resources from off-campus.  With the majority of upper-class students living off campus, this is essential information.  The instruction session then led to demonstrations of specific databases that have relevant types of information based on the assignment requirements (e.g., Business Source Premiere, Marketline {formerly Datamonitor}, and Mergent Online).  Business Source Premiere indexes journals in the popular and scholarly business press, with the majority of articles available in full-text.  Marketline has proprietary full-text market research reports for US and international companies and industries, which tend to be very popular with the students.  The final database demonstrated was Mergent Online, which is an excellent resource for company financial data.  Print reference materials and using the catalog to find books were also mentioned  but the emphasis was on using the databases.  Handouts outlining the sources were provided to the students as well.

The classes that did not receive the session were encouraged by the instructor to use library resources for the assignment but were not given handouts or specific instructions on accessing or using the databases.  No specific requirements were given for the bibliographies for any of the sections.  The students were encouraged to use a variety of resources, both print and electronic, but there were no required numbers or types of sources mandated.

At the end of each semester, the students’ term papers were graded and copies were made of the bibliographies.  Personal identification was removed, and each bibliography was marked with the grade for both the assignment and the course, and a code based on whether the student had been exposed to the instruction session or not.  The bibliographies were then examined to ascertain the nature of each citation and how it had been located (e.g. was it an Internet site, a journal article located through a database, a chapter from the course textbook, etc.).  This was done primarily by examining each citation and determining the source and how it was originally located.  Many of the citations were simply links to online articles, so the URLs provided information as to their provenance.  Links to articles or online sites were followed to determine the nature of the resource and how they had been arrived at, if possible.

The data were analyzed with Minitab (Version 14).  Tests of differences, means, and proportions were conducted to determine if the differences in the types and numbers of resources used by the two groups of students or the grades they received were statistically significant.  The data entered for each bibliography included the total number of unique citations and the number of citations for journal articles, books, library databases, Internet sites, and other.   Those that were coded as “other” were primarily personal interviews that the students had done to gather research.  Also included were the course grade, the paper grade, and the identifier for whether they had been exposed to the library instruction session or not. 

Results

The first piece of data analysis examines the difference in mean numbers of unique citations among the two groups, those exposed to library instruction and those not exposed.  The emphasis is on unique citations as many of the students confuse the purpose of a bibliography with endnotes, so that the same source such as a website or a textbook may be cited multiple times.  Figure 1 and Table 1 demonstrate the differences in mean numbers of citations between the two groups.  While the group receiving library instruction did have a slightly higher mean number of citations (7.16 vs. 6.12), the difference only tends towards statistical significance with a P value of .070.  Thus we cannot attribute the increased number of citations to the affect of the instruction session.

Figure 1

Table 1 –T-Test for Mean Number of Unique Citations
Instruction N Mean StDev SE Mean
No 83 6.12 3.67 0.40
Yes 101 7.16 4.05 0.40
T-Test of difference = 0 (vs. not =): T-Value = -1.82,   P-Value = 0.070

 

Looking beyond the mean number of overall citations, we next examined the types of sources each group cited.  Here we start to see some more significant differences (Figure 2 & Table 2).  The students that had received the instruction session cited on average 2.66 different types of resources (out of 5 possible types; journal articles, Internet sites, library databases, books or other), compared to the average of 1.95 types of resources cited by those not receiving the instruction.  This difference was statistically significant.  This indicates that the library instruction session demonstrated to the students that other sources of information were available and easily accessible.  Once they were made aware of this, the students were more likely to take advantage of the various resources at their disposal, thus bearing out Hypothesis 1.

Figure 2

Table 2 -T-Test for the Number of Source Types Used
Instruction N Mean StDev SE Mean
No 83 1.952 0.936 0.10
Yes 101 2.663 0.863 0.09
T-Test of difference = 0 (vs not =): T-Value = -5.35, P-Value = <0.001

 

Figure 3 depicts the differences in numbers of students citing resources located through the library’s website.  This is a particularly important focus of the research as this is the behavior the library instruction session was meant to influence.  Of students who were exposed to the library instruction session, 86 out of 101 (85%) cited at least one source that they located using library resources.  This might include journal articles located through a library database, a book other than their textbook, or information from one of the other library databases such as Marketline or Mergent Online.   This compares to 36 out of the 83 students (43%) who did not receive library instruction.  This difference is clearly statistically significant.  This further exemplifies the fact that students can only use resources that they know about.  While the library website is as user-friendly and accessible as possible, it still increases usability to have a demonstration, particularly one tailored to the specific resources that are most likely to be useful for a particular assignment.

Figure 3

 Table 3  - Test for Differences in Proportions of Students Using Library Resources
Instruction # Used Library Resources # of Students Sample p Z P-Value
No 36 83 0.434 -6.44 <.001
Yes  (n= 101) 86 101 0.851

 

Now we turn to the opposite side of the equation. Figure 4 and Table 4 present data on the percentage of students who cited only Internet sites in their papers. Here again the difference was particularly marked between the two groups, with fully 35% of those not receiving instruction citing only Internet sites in their bibliographies, compared to 10% of those who received instruction.  The bibliographies of the papers that cited only Internet sites often consisted of a single source, usually a company website.  The fact that the Internet was the sole source of citations for over one-third of the papers in the group not receiving instruction compared to just 10% in the group that received instruction once again strongly points to the power of even a single librarian-led session.

Figure 4

Table 4 - Test for Differences in Proportions of Students Using Internet Sites Exclusively
Instruction Cited Only Internet # of Students Sample p Z P-Value
No 29 83 0.349 4.16 <.001
Yes  (n= 101) 10 101 0.099

 

Figure 5 and Table 5 combine the results of these analyses to examine the differences in types of resources used by students in the two groups.  Those that had received library instruction utilized proportionally more journal articles and databases, and fewer Internet sites, with the differences being statistically significant.  The group receiving library instruction also used proportionally fewer books, although this difference was not statistically significant.  Despite the lack of statistical significance, it may be useful to note that the book most often cited was their textbook.  Thus, any decrease in the proportion of books may actually indicate a greater reliance on other sources.  The increase in library database usage and journal citations by the group that received library instruction continues to bear out Hypothesis 1.  By demonstrating these resources to the students through library instruction, they became aware of them and subsequently used them to locate information for their research papers.

Figure 5

Table 5 – Differences in Types of Materials Used by the Two Groups *

Test for Differences in Proportions for Articles Cited
Received Instruction # of Articles Cited Total # of Unique Citations Sample p Z P-Value
No  (n=83) 85 508 0.167 7.20 <.001
Yes  (n= 101) 247 723 0.341
Test for Differences in Proportions for Internet Sites Cited
Received Instruction # of Internet Sites Total # of Unique Citations Sample p Z P-Value
No  (n=83) 344 508 0.677 -9.32 <.001
Yes  (n= 101) 303 723 0.419
Test for Differences in Proportions for Databases Cited
Received Instruction # of Databases Cited Total # of Unique Citations Sample p Z P-Value
No  (n=83) 19 508 0.037 7.47 <.001
Yes  (n= 101) 113 723 0.156
Test for Differences in Proportions for Books Cited
Received Instruction # of Books Cited Total # of Unique Citations Sample p Z P-Value
No  (n=83) 51 508 0.100 -1.82 0.068
Yes  (n= 101) 51 723 0.071
* N = 184

 

Each of these five figures and their accompanying tables provides empirical evidence of the effectiveness of a library instruction session on increasing the use of library resources, which satisfies Hypothesis 1.  Next we turn to Hypothesis 2, which stated that students who received library instruction would also receive a higher grade on the term paper.

Looking at Figure 6 and Table 6, we see that the mean grades for the term-papers between the two groups were very similar, with the group receiving library instruction having a mean paper grade of 85 versus a mean grade of 84 for those not receiving instruction.  The grades for the course as a whole were even more similar, with each group having a mean course grade of 84.[1] Thus, although those in the group that received library instruction did indeed cite more scholarly sources and more types of sources, this did not result in an improved grade for either the paper itself or for the course overall.   For this reason, Hypothesis 2 was not borne out.

Figure 6

Table 6 –T-Test for Mean Paper Grade
Instruction N Mean Grade StDev SE Mean
No 83 84.452 5.494 0.60
Yes 101 85.366 5.776 0.57
T-Test of difference = 0 (vs. not =): T-Value = -1.09,   P-Value = 0.276

In part, this result was due to a lack of variation among the grades.  Additionally, the papers were graded on multiple criteria and not explicitly according to the number or types of resources cited.  The assignment itself also did not necessitate the use of scholarly sources as much as it required students to analyze a company and discuss its actions.  The papers were graded based primarily on this analysis and how well the students explained and defended their recommendations.  Thus, although good research should lead to improved analysis, the research itself was not specifically graded as a component of the paper.  Therefore there was not as much benefit to those citing or using library resources as might have occurred with a different type of assignment.

Implications

This study provides evidence that library instruction is effective.  Students are much more likely to use and cite scholarly resources when they have been shown how to use and access the databases.  However, it is up to instructors to make quality research a requirement for the class.  While specific mandates for types or numbers of sources are not always necessary, the idea that resources beyond the web are not only available, but desirable, must be conveyed.  This in turn means that assignments ideally would include components that encourage library research.  Once the papers or assignments are turned in, faculty must then evaluate the quality of the research sources used.  Bibliographies can be read, evaluated, and graded just as papers are.  Based on this, students who have utilized good research skills and cited relevant works should then be rewarded with better grades for the project than those who did not.

This research demonstrates that a library instruction session in a class can be very effective in teaching students about new resources and reminding them of ones they have heard about or used in the past.  Including library instruction in a class is also an indication of the importance the instructor places on utilizing research resources.  The benefits of library instruction will carry on throughout students’ college careers and into their futures, where good research skills are becoming an increasingly important part of many job descriptions.  Considering that many students go onto graduate school or else into professions requiring data gathering and analysis, the knowledge of how to use databases and other resources effectively can be a tremendous asset for career satisfaction and advancement.

Conclusions

Overall, the results of this study provide empirical evidence of the effectiveness of librarian-led instruction sessions, particularly when they are tied to specific assignments.  Statistically significant differences were found both in types and numbers of resources used by students in the two groups.  Students not exposed to the instruction session also relied much more heavily on Internet sources, with over one-third citing nothing but Internet sites in their term paper bibliographies.  However, the idea that using more library resources would automatically improve a student’s grade on either the term paper or in the course as whole was not borne out.  Term paper and course grades were virtually identical for each group. 

The fact that grades were not affected indicates that librarians and faculty could work together more in the future to determine the value of students’ work and what sources constitute “good research”.  This is a chance to make information literacy a reality, rather than just a concept often discussed but rarely implemented.  Assignments that are crafted to better take advantage of library resources, combined with library instruction sessions that demonstrate the availability and ease of use of these resources, would be beneficial to both the students and the faculty.  Students would be less frustrated by assignments and faculty would hopefully receive papers that were based on improved research and resources; no more “garbage in, garbage out”.   Ideally, students would be rewarded for this improvement through better grades, thus providing a tangible benefit as well as the intangible gain of their increased knowledge and skills.  This would also spread the news of the importance of better research, as students compare grades and determine that hammering away with Google is no longer sufficient in today’s information-rich environment.

[1] This also suggests that differences in paper quality were not due to chance variations in students’ abilities.

References

Davis, P. M. (2003).  Effect of the web on undergraduate citation behavior: Guiding student scholarship in a networked age. portal: Libraries and the Academy, 3(1), 41-51.

Dewald, N. H. (2005). What do they tell their students? Business faculty acceptance of the web and library databases for student research.   Journal of Academic Librarianship, 31(3), 209-215.

Gratch, B. (1985).  Toward a methodology for evaluating research paper bibliographies.  Research Strategies, 3(4), 170-177.

Hinchcliffe, J. (2000).  Faculty-directed library use instruction: A single class, retrospective study. Research Strategies, 17(4), 281-289.

Hovde, K. (2000).  Check the citation: Library instruction and student paper bibliographies. Research Strategies, 17(1), 3-9.

Robinson, A. M. and Schlegl, K. (2004).  Student bibliographies improve when professors provide enforceable guidelines for citations.  portal: Libraries and the Academy, 4(2), 275-290.

Back to Contents