Introduction

While it may be important for academics to place emphasis on theory and methods in research, practitioners and policymakers mainly want to know the results. However, there may be instances when knowledge about research methods might be of interest to practitioners and policymakers as well. Therefore, the purpose of this paper is to explain the place of research methods, specifically in the area of understanding improvements in literacy research. Researchers use a 'theoretical approach' as a way to look at a certain research problem; it helps to contain the focus of the research questions and the breadth of a study. The methodological approach that researchers choose depends on the questions that they want to ask.

It might be fairly said that research in the social sciences (e.g., linguistics, history, political science, psychology, sociology, etc.) has yet to match the achievements of the natural sciences (e.g., chemistry, biology, medical sciences, mathematics, etc.) in the impact of applied social research on everyday life. One reason for this may be that the rules by which scientific progress is actually undertaken have not been specified. When researchers conduct a systematic review, a meta-analysis, or evidence-based research, they look for commonalities between approaches so that they can compare the findings in each study for use in practice or in policy decisions. This leaves research groups focused on synthesizing and reporting differing findings in limbo, since a common approach that constitutes appropriate research methodology and even what constitutes 'knowledge' does not exist. The aim of this paper is to provide a better understanding of what it means to conduct evidence-based research, systematic reviews, and statistical meta-analyses.

Key Questions

1. What is evidence-based research?
2. What are RCTs and quasi-experiments? Why are they important?
3. What is systematic review?
4. What is statistical meta-analysis?
5. What kind of research gets reviewed in systematic reviews and meta-analyses?

Research Findings

The theoretical and methodological aspects of research
One approach to combine both theoretical and methodological aspects of research is to systematically synthesize evidence bases in an effort to come to an understanding of what is most consistent across a range of studies. Until quite recently, attempts at synthesis simply involved narrative review of what was deemed relevant literature by any given writer (e.g., in reading research specifically, Adams, 1990; Bond & Dykstra, 1967; Chall, 1967). In narrative reviews, the selection of research evidence cited is subjective and can often reflect an individual writer's professional biases. As such, it also does not guarantee the advancement of understanding within a discipline. Indeed, these narrative reviews, which are published in peer-reviewed scientific journals, are frequently followed by a theoretical or empirical rebuttal by another researcher, which in turn, is followed by a response from the author of the original research. Thus, using research findings for uptake into policy and practice from this type of exchange is often extremely difficult (see Merttens, 2003, for a recent example). Engaging in 'evidence-based decision-making' or 'evidence-based practice' is further complicated even if a review is not biased in any way. Even if the studies themselves are methodologically sound, conclusions drawn from isolated studies or from small clusters of studies in narrative review may still be lacking in reliability (i.e., a way that researchers ensure the methodology used in a study was done properly, which gives confidence about the conclusions drawn from the study). In these situations, the confidence with which results can be generalized (or applied) to other populations can only be guessed. Clearly, this is less than ideal if one wishes to base decisions which affect populations, such as the nature of a national or provincial educational curriculum, on reliable research evidence.
 
If all available data, that was basically methodologically sound, was chosen without bias from all available sources and then analyzed using a commonly agreed upon methods, this in principle at least, would represent the holy grail of achieving an evidence base upon which decisions could be made. Such an approach might be particularly important in the field of reading research, which has been marked by conflicts between proponents of different views, especially concerning the role of phonics instruction (e.g., Adams, 1990; Smith, 2004). The use of 'evidence-based' methodologies is now widespread in medicine and its use is catching on in the social sciences in an attempt to more effectively underpin all evidence-based policy and practice, including literacy (e.g., National Reading Panel, 2000; Sackett, Rosenberg, Gray, Haynes, & Richardson, 1996). Five questions regarding the approach are raised and answered below.

1. What is evidence-based research?
It is important to note that every research methodology in existence has flaws or what researchers call 'limitations'. Some methodologies are more reliable than others. Reynolds (2001) describes a hierarchical model of evidence reliability (see Table 1 below). All forms of study are 'evidence' with (in principle) equal claim to be guides to truth. Independent of this, the confidence one might have that any given empirical result from research can be repeated and generalized to the wider population increases the closer the type of evidence is to the top of the list. At least in principle, individual bias also decreases as one moves from Level 7 through to Level 1 because the type of study becomes more and more objective. The scheme is somewhat rigid, and is certainly not comprehensive in terms of possible evidence sources below Level 3. It nevertheless provides a sense of the reasoning in the discipline of meta-analysis.
______________________________________________________________________

1. Systematic Review of a Series of High Quality RCTs
2. Single High Quality RCT
3. Single Low Quality RCT
4. Outcome Evaluation(s) and quasi-experiments
5. Controlled Single Case Studies using formal methodology
6. Case Series
7. Individual / Group Expert Opinion 

_____________________________________________________________
Note: RCT = Randomized Control Trial study

2. What are RCTs and quasi-experiments? Why are they important?
Although evidence-based research has been well-established in the medical sciences, it has yet to reach the same level of development in the social sciences. However, the same principles used for assessing the quality of research findings can be applied across disciplines when ranked according to methodological rigour. In addition, the methodological approaches used in medical research can be adapted for use in social science research concerning literacy and learning. Researchers studying any form of intervention (e.g., intervention of reading programs) are interested in exploring the effect of the presence or absence of their treatment on an outcome (e.g., literacy levels with and without the intervention). To establish the connection between cause (treatment) and effect (outcome) the researcher needs to be able to rule out other factors that might affect the outcomes for the treatment group (those who receive the reading intervention) and the control or comparison group (those who do not) in the real world. RCTs are the only research design that adequately control for all of the unmeasured variables, or factors, that may affect the outcome. Through randomly selecting samples of participants (randomization), all potentially confounding factors (extraneous factors) are distributed in a non-systematic manner across the groups. Because the random selection of people to one group or another (called a sample) reduces measurement error in research findings, it remains a fundamental assumption behind most of the statistical methods used to interpret results in RCTs and other types of statistical analysis.

The next best alternative to RCT studies are well-designed quasi-experiments. In these experiments, groups of children are not randomly allocated but are assigned to specific experimental conditions. For example, classroom x is exposed to the 'intervention' condition (e.g., a reading program) and classroom y is used as the 'control' or 'comparison' condition with no exposure to the 'intervention' condition. To ensure that there are no obvious biases, the two groups are checked to make sure they are similar enough to be compared; for example, whether they match each other on important factors (or variables) such as gender, ethnicity, and socioeconomic status. These quasi-experiment studies do not guarantee that all of the relevant biasing factors outside of the intervention have been controlled for, and so they are clearly inferior to RCT studies. Systematic review of a series of high-quality RCTs represents, to many in this field, a 'gold standard' for building evidence-based practice and government policy (e.g., Barnes, Stein, & Rosenberg, 1999).
 
3. What is systematic review?
A systematic review is a form of review with explicit criteria for the inclusion and exclusion of studies found through exhaustive searches of the databases. These exclusion criteria can be practical (e.g., only studies of the English language) or methodological (e.g., only studies with pre-intervention and post-intervention reading test data). Very often, RCTs are the exclusive or particular focus of reviews since they are the methodologically strongest studies. Whichever studies are included, the inclusion and exclusion criteria should be stated in advance, be consistent across reviews, and be uniformly applied to all studies extracted from basic literature searches.

4. What is statistical meta-analysis?
This method extends the approach of systematic review. In quantitative meta-analysis, relatively simple statistical techniques are applied to explore whether effects (changes) reported in well-designed individual studies are consistent across all studies. The simple statistics used include reporting the 'average' of the literature review which produces a series of similar studies, and reports it as an 'effect size'. This measure gives an indication of the reliability of findings across studies.

5. What kind of research gets reviewed in systematic reviews and meta-analyses?
The degree that RCTs can help us answer a research question is, of course, determined entirely by their frequency in the literature. Sebba (1999) wrote a report to the Cochrane collaboration, an organization set up to collate systematic reviews and meta-analyses of healthcare interventions on a worldwide basis, and noted that: 'In education, there are relatively fewer randomly controlled trials, large quantitative studies and evaluations of experimental interventions. While all these should be encouraged, the existing evidence base would be strengthened through replications and extensions of some existing studies and systematic reviews' (p. 4).Sebba also noted that, in the absence of a well-developed RCT-based literature, other evidence must necessarily be drawn upon as a valuable source in identifying priorities for future RCTs: 'Reviews of 'effective practice' through visits, case studies and reports - often 'grey' literature - will also assist in identifying priorities for systematic reviews' (p. 4).

Conclusions and wider implications
Most websites disseminating meta-analyses are still in their infancy. As a non-expert reader may find, scrolling through systematic review websites (for an example, see The Campbell Collaboration, http://www.campbellcollaboration.org/), the language of meta-analysis is very specific and not familiar to most educators, practitioners, and policymakers. Thus, there is a significant amount of interpretation necessary if evidence-based policy is to become evidence-based practice.

Additionally, there is a need for a more specifically Canadian contribution to field-based evaluation and practical model building. Where program evaluation has been undertaken, it has generally been undertaken outside of Canada (e.g., Rose, 2006, in the UK).
The role of evidence, as currently construed by those advocating evidence-based policy, remains a vital one in guiding policy. Sadly, there is poorer quality evidence from literacy intervention studies than commonly imagined. Of the well-designed studies reported by Torgerson, Brooks, and Hall (2006), three were carried out in Canada. All of these studies were carried out in Ontario; as a result, generalization across Canada must be undertaken very cautiously. In this sense, the context in which the evidence was generated is an important consideration when applying this evidence to practice or policy decisions.

One implication for research is the need for a clear commitment to funding and supporting the publication and dissemination of the 'gold-standard' RCT studies. Such studies need to be large to be reliable; therefore, they are relatively expensive to run and often exceed the budgets currently allocated to single-research studies in Canada.