Positioning Evaluation amongst 4 Theoretical Worldviews

While I’ve engaged in and lead studies focusing on qualitative and quantitative methods and data collection, I have never formally studied the theoretical underpinnings of them. Reading “Research Design” by John Creswell (2014), I find myself nodding along and seeing my work in evaluation reflected back at me as he unpacks 4 specific worldviews from which to view the research: (Post)Positivism, Constructivism, Transformative, Pragmatic.

In summarizing each of the following worldviews, I reflect on their alignment (or misalignment) with evaluation practice.

POSTPOSITIVISM challenges our conception of truth and recognizes that we cannot be positive about our claims of knowledge when studying human behaviour. As such, post-positivists will look to identify and assess the causes of human behaviour and its influence on outcomes by distilling ideas in to small components to test such variables against the hypothesis and research questions.

This, to me, sounded very much like a randomized control trial (RCTs) or (quasi-) experimental evaluation that looks to identify areas of attribution. While the RCTs are typically seen as a ‘gold standard’ in research communities, in evaluation it is less so. This doesn’t mean that evaluators don’t seek to learn attribution, but there are limited circumstances in which it can be applied while still being true to the methodology. There are simply too many variables to control for or otherwise compromise our ethics in attempting to adhere to the gold standard. What there are, are attempts to control for constant and a preference for identifying contribution – be it expressed in either quantified or qualified data.

CONSTRUCTIVISM leans toward qualitative methods and looks to understand the world and the meaning that people put on their experiences. In this worldview, the complexity of viewpoints and opinions are aggregated in to categories and ideas. As such, constructivist research is often broad and general to allow participants to express their subjective experiences, often recounting discussions or interactions with others. It is for this reason that context is incredibly significant and researchers must be cognizant that their own backgrounds and experiences will shape their interpretation of the data being collected. To contrast with post-positivism, constructivism is less concerned on understanding attribution and a theory of results as it is understanding the nuances and meanings of experiences and viewpoints of the research participants.

Oddly enough, some of this description was almost word for word in how I instruct some of my analysts to engage in qualitative data collection – allowing the participant to express in their own words their viewpoints, not letting your bias cloud how you receive the information, emphasizing the importance of context and finding meaning in the ‘grey’. We even triangulate findings from multiple sources to make greater sense of what is happening and allow the data to inform next steps on answering the research questions.

TRANSFORMATIVE worldviews reject the positivist approach as it doesn’t reflect the realities of individualis in our society and does not reflect the political and social realities that permeate our lives, nor does it acknowledge social justice, discrimination or oppression. As such, research may deal with issues such as empowerment, inequality, domination, alienation to name a few. The transformative worldview sees the need for research to be intertwined with politics, a confrontation of social oppression and an acknowledgement that the research can influence not only the lives of the participants of the research, but also for the researcher. As a result, it is not uncommon for the participants to help shape the design of the research, participate in data collection and analysis.

Engaging in evaluation often means confronting challenging circumstances and issues of social justice. By merely engaging in an evaluative process, a client shows openness to learning and understanding the challenge and to what extent progress is being made. I also don’t see how anything in our lives are not touched by the hand of government or the politics that guide it, so this worldview certainly resonated with me. To that end, participatory evaluation reflects that notion of the transformative worldview, by bringing the participants on board as part of the evaluation team – from start to finish – and is often promoted as a good capacity building technique, albeit one that takes a tremendous amount of time. In so doing, participants gain valuable evaluation skills and start to think as an evaluator, which are important as they continue to implement their initiative.

Lastly PRAGMATIC Worldviews are less concerned about application and more about what works and the solutions to problems. As such, there is less emphasis on methods and more examining the nature of the research problem and questioning it as truth. Pragmatic researchers consider social, historical and political contexts and are more concerned about the ‘what’ and the ‘how to’ instead of the ‘why’ of post-positivists or the experiences of constructivists.

I found this worldview interesting and the notion of questioning whether the question is the right one to be a task I’m often engaged in. The exercise itself is often considered by evaluators and likely an aspect of evaluative thinking (more on this later!) Further, formative evaluation looks at process and occurs at the specific stage when an initiaitive is not yet expected to generate results, but sufficient time has passed to assess whether it is being implemented as designed and make recommendations for adjustment, if need be.

I find it difficult to characterize myself under any one of these worldviews, but perhaps – it typical evaluator fashion – would offer that each worldview may contribute to my own understanding of evaluation and how I conduct my evaluation practice. Understanding these worldviews and which an audience my relate to more significantly may influence how I communicate findings.

Creswell continues that these worldviews are more or less linked with specific qualitative or quantitative methods. Some, I think, are more obvious than others. I prefer to allow the research / evaluation questions guide the selection of which method is most appropriate. This, however, is more straightforward that what I think happens in reality. In my experience, questions are not determined in isolation, nor are they made independently; there are multitude of people that consult and provide input in to the creation of questions and each bring specific considerations for how the research or evaluation should look. At the same time, the researcher or evaluator is considerating (perhaps unconsciously?) multiple aspects that concern whether or not the question can be answered; whether they are feasible, and will use this information in their advice and consultation with others. For instance, the evaluator could be thinking any of the following…

• How easy will it be to collect data from x group of people at x time?
• What needs to be done in advance to reach hard to reach populations?
• What software is available?
• How many analysts will work on this research in order to reach x deadline?
• How much money is available to implement this research?
• What tools are available and at what cost to do this research?
• Whether the results generated will be sufficient?

All of these things influence how questions are constructed; without doing so, a researcher and evaluator may run in to problems of implementation.

I’m a proponent of mixed methods approaches and absolutely no less than 3 methods per evaluation – that is, different approaches, questions, timing etc. (To me, data collected from multiple stakeholders but via the same method, like a survey, is still considered one method). Quantitative can only provide the black and white and does not explain the #3 on a likert scale or a 56% completion rate; only qualitative can do that and one without the other is only half the story. And what a compelling story it will be!