Theory-driven evaluation

Apr 23, 2026 - 10:07
0 0
Theory-driven evaluation

Link suggestions feature: 3 links added.

← Previous revision Revision as of 04:36, 23 April 2026
Line 17: Line 17:
Consequently, the terms theory-''driven'' and theory-''based'' evaluation are often used interchangeably in the literature.Birckmayer, J. D., & Weiss, C. H. (2000). [https://doi.org/10.1177/0193841X0002400404 Theory-based evaluation in practice: what do we learn?] Evaluation review, 24(4), 407-431.{{Cite journal| doi = 10.1177/10982140221122764| volume = 45| issue = 1| pages = 110–132| last1 = Matta| first1 = Corrado| last2 = Lindvall| first2 = Jannika| last3 = Ryve| first3 = Andreas| title = The Mechanistic Rewards of Data and Theory Integration for Theory-Based Evaluation| journal = American Journal of Evaluation| date = 2024| url = http://journals.sagepub.com/doi/10.1177/10982140221122764| url-access = subscription}}Dahler-Larsen, P. (2018, p. 9). Theory-Based Evaluation Meets Ambiguity: The Role of Janus Variables. American Journal of Evaluation, 39(1), 6–23. However, ''theory-based evaluation'' is sometimes interpreted more narrowly to mean qualitative or small-''n'' [[case study]]-based evaluations conducted without a comparison group, for example those using [[process tracing]] or [[qualitative comparative analysis]].Stern, E., Stame, N., Mayne, J., Forss, K., Davies, R., & Befani, B. (2012). [https://doi.org/10.22163/fteval.2012.100 Broadening the range of designs and methods for impact evaluations]. Institute for Development Studies. An example of this narrower meaning is present in the UK government handbook on evaluation, the ''Magenta Book''.HM Treasury (2020). [https://www.gov.uk/government/publications/the-magenta-book The Magenta Book].
Consequently, the terms theory-''driven'' and theory-''based'' evaluation are often used interchangeably in the literature.Birckmayer, J. D., & Weiss, C. H. (2000). [https://doi.org/10.1177/0193841X0002400404 Theory-based evaluation in practice: what do we learn?] Evaluation review, 24(4), 407-431.{{Cite journal| doi = 10.1177/10982140221122764| volume = 45| issue = 1| pages = 110–132| last1 = Matta| first1 = Corrado| last2 = Lindvall| first2 = Jannika| last3 = Ryve| first3 = Andreas| title = The Mechanistic Rewards of Data and Theory Integration for Theory-Based Evaluation| journal = American Journal of Evaluation| date = 2024| url = http://journals.sagepub.com/doi/10.1177/10982140221122764| url-access = subscription}}Dahler-Larsen, P. (2018, p. 9). Theory-Based Evaluation Meets Ambiguity: The Role of Janus Variables. American Journal of Evaluation, 39(1), 6–23. However, ''theory-based evaluation'' is sometimes interpreted more narrowly to mean qualitative or small-''n'' [[case study]]-based evaluations conducted without a comparison group, for example those using [[process tracing]] or [[qualitative comparative analysis]].Stern, E., Stame, N., Mayne, J., Forss, K., Davies, R., & Befani, B. (2012). [https://doi.org/10.22163/fteval.2012.100 Broadening the range of designs and methods for impact evaluations]. Institute for Development Studies. An example of this narrower meaning is present in the UK government handbook on evaluation, the ''Magenta Book''.HM Treasury (2020). [https://www.gov.uk/government/publications/the-magenta-book The Magenta Book].


Theory‑driven evaluation is also closely related to ''evidential pluralism'', an approach developed within the philosophy of science from around 2007 which argues that for scientists to make a causal claim, they must provide both evidence of an association between a putative cause and its effect and evidence of the mechanisms that connect the two.{{Cite book |last=Shan |first=Yafeng |url=https://www.taylorfrancis.com/books/9781003143000 |title=Evidential Pluralism in the Social Sciences |last2=Williamson |first2=Jon |date=2023 |publisher=Routledge |isbn=978-1-003-14300-0 |edition= |location=London |language=en |doi=10.4324/9781003143000}} This dual‑requirement aligns naturally with theory‑driven evaluation, which similarly emphasises not only identifying whether an intervention is associated with outcomes but also understanding how those outcomes arise through underlying causal processes.
Theory‑driven evaluation is also closely related to ''evidential pluralism'', an approach developed within the [[philosophy of science]] from around 2007 which argues that for scientists to make a causal claim, they must provide both evidence of an association between a putative cause and its effect and evidence of the mechanisms that connect the two.{{Cite book |last=Shan |first=Yafeng |url=https://www.taylorfrancis.com/books/9781003143000 |title=Evidential Pluralism in the Social Sciences |last2=Williamson |first2=Jon |date=2023 |publisher=Routledge |isbn=978-1-003-14300-0 |edition= |location=London |language=en |doi=10.4324/9781003143000}} This dual‑requirement aligns naturally with theory‑driven evaluation, which similarly emphasises not only identifying whether an intervention is associated with outcomes but also understanding how those outcomes arise through underlying causal processes.


== What is meant by "theory"? ==
== What is meant by "theory"? ==
Line 53: Line 53:


== Examples ==
== Examples ==
Examples discussed in a 2011 systematic review of 45 theory-driven evaluations include:Coryn, C. L. S., Noakes, L. A., Westine, C. D., & Schröter, D. C. (2011). A systematic review of theory-driven evaluation practice from 1990 to 2009. American Journal of Evaluation, 32(2), 199–226. https://doi.org/10.1177/1098214010389321
Examples discussed in a 2011 [[systematic review]] of 45 theory-driven evaluations include:Coryn, C. L. S., Noakes, L. A., Westine, C. D., & Schröter, D. C. (2011). A systematic review of theory-driven evaluation practice from 1990 to 2009. American Journal of Evaluation, 32(2), 199–226. https://doi.org/10.1177/1098214010389321


* An evaluation of the Fort Bragg Child and Adolescent Mental Health Demonstration, a managed mental health care system with a single point of entry, which used individual interviews, focus groups, and document review to assist the development of a theory of change.Bickman, L. (1996). [https://doi.org/10.1016/0149-7189(96)00002-X The application of program theory to the evaluation of a managed mental health care system]. Evaluation and Program Planning, 19, 111-119. The theory explained why it was thought that an integrated care system would be more cost-effective than a fragmented system.
* An evaluation of the Fort Bragg Child and Adolescent Mental Health Demonstration, a managed mental health care system with a single point of entry, which used individual interviews, focus groups, and document review to assist the development of a theory of change.Bickman, L. (1996). [https://doi.org/10.1016/0149-7189(96)00002-X The application of program theory to the evaluation of a managed mental health care system]. Evaluation and Program Planning, 19, 111-119. The theory explained why it was thought that an integrated care system would be more cost-effective than a fragmented system.
Line 61: Line 61:
A 2014 review of theory-driven evaluation in [[school psychology]]Mercer, S. H., Idler, A. M., & Bartfai, J. M. (2014). [https://doi.org/10.1080/02796015.2014.12087439 Theory-Driven Evaluation in School Psychology Intervention Research: 2007–2012]. School Psychology Review, 43(2), 119–131. highlighted two illustrative examples:
A 2014 review of theory-driven evaluation in [[school psychology]]Mercer, S. H., Idler, A. M., & Bartfai, J. M. (2014). [https://doi.org/10.1080/02796015.2014.12087439 Theory-Driven Evaluation in School Psychology Intervention Research: 2007–2012]. School Psychology Review, 43(2), 119–131. highlighted two illustrative examples:


* An evaluation of conjoint behavioral consultation, a "strength-based intervention focused on building behavioral and social competence in children".Sheridan, S. M., Bovaird, J. A., Glover, T. A., Andrew Garbacz, S., Witte, A., & Kwon, K. (2012). [https://doi.org/10.1080/02796015.2012.12087374 A Randomized Trial Examining the Effects of Conjoint Behavioral Consultation and the Mediating Role of the Parent–Teacher Relationship]. School Psychology Review, 41(1), 23–46. The evaluation tested a theory of change using a [[cluster-randomised controlled trial]] and [[mediation analysis]].
* An evaluation of conjoint behavioral consultation, a "strength-based intervention focused on building behavioral and [[social competence]] in children".Sheridan, S. M., Bovaird, J. A., Glover, T. A., Andrew Garbacz, S., Witte, A., & Kwon, K. (2012). [https://doi.org/10.1080/02796015.2012.12087374 A Randomized Trial Examining the Effects of Conjoint Behavioral Consultation and the Mediating Role of the Parent–Teacher Relationship]. School Psychology Review, 41(1), 23–46. The evaluation tested a theory of change using a [[cluster-randomised controlled trial]] and [[mediation analysis]].
* An evaluation of repeated reading and vocabulary previewing which tested causal theory using [[case study]] methodology, an [[Single-subject design#Alternating_treatments_design|adapted alternating treatments design]] with six students.Hawkins, R. O., Hale, A., Sheeley, W., & Ling, S. (2011). [https://doi.org/10.1002/pits.20545 Repeated reading and vocabulary‐previewing interventions to improve fluency and comprehension for struggling high‐school readers]. Psychology in the Schools, 48(1), 59–77.
* An evaluation of repeated reading and vocabulary previewing which tested causal theory using [[case study]] methodology, an [[Single-subject design#Alternating_treatments_design|adapted alternating treatments design]] with six students.Hawkins, R. O., Hale, A., Sheeley, W., & Ling, S. (2011). [https://doi.org/10.1002/pits.20545 Repeated reading and vocabulary‐previewing interventions to improve fluency and comprehension for struggling high‐school readers]. Psychology in the Schools, 48(1), 59–77.


Read More