World Bridge Research, LLC
About this Evaluation Tool Kit
This tool kit is a compilation of materials developed for The Domestic Violence Prevention Enhancement and
Leadership Through Alliances Project (DELTA)--a cooperative agreement made possible by the Centers for
Disease Control and Prevention (CDC) with 14 state domestic violence coalitions that receive DELTA Program
funds. World Bridge Research works with the North Dakota Council on Abused Women’s Services/Coalition
Against Sexual Assault in North Dakota (NDCAWS/CASAND) Empowerment Evaluation Project and the Wisconsin
Coalition Against Domestic Violence (WCADV) Empowerment Evaluation Project.
Empowerment Evaluation is about helping programs work better by using evaluation results to guide strategic
planning and decision-making. Specific tools (e.g., Getting to Outcomes) are used to guide planning,
implementation, and evaluation.
Introduction to Evaluation
During the past 30 years program evaluation for social programs has evolved into a field known as “evaluation
research”. “Evaluation research is the systematic application of social research procedures for assessing the
conceptualization, design, implementation, and utility of social intervention programs,” (Rossi & Freeman, 1996).
Let’s pick this apart – To conduct an evaluation we must create a system or logical way of planning, designing and
putting into place activities and strategies that help to change people’s lives. Then we must document what we do,
what participants experience and the changes that take place in both the program and participants. Evaluation has
steps before, during and after program implementation.
So, what are these evaluation procedures that we use before, during and after the program implementation? We
call them research or evaluation methods and they fall into 2 categories: quantitative and qualitative.
Quantitative methods are a formal, objective, systematic process in which numerical data are utilized to obtain
information about the world. Quantitative methods:
- Use statistics to find results that we can apply for use with other people or situations--what we call
- Deductive --start with a theory or hypothesis and move to corroborate it
- Quantitative methods are most likely used when we are trying to identify a cause or what factors may help
predict a particular outcome.
- May be high or low-tech (high tech might be a telephone survey of 1,000 residents—low tech might be a
simple pre-and post-test measuring knowledge)
Quantitative methods are usually understood to include:
- Pre and post tests
- Surveys – paper and pencil, telephone, web-based
- Environmental surveys
- If the data can not be structured in the form of numbers, they are considered qualitative.
- Goal is to obtain a great deal of depth of understanding.
- Inductive--start with observations and develop theory from it.
- Methods of data collection: In-depth interviewing, observation, document review
- Qualitative methods are usually understood to include:
- Interviews which can range from semi-structured questionnaires to open-ended ad hoc conversations
- Direct observation including participant and non-participant observation, case or field notes, and more
recently photography and video
- Case studies combining different methods to compile a holistic understanding of individuals, households,
communities, markets or institutions
Qualitative methods are useful compliments to quantitative and participatory methods in order to:
Increase understanding of WHAT is happening.
- Qualitative methods are often necessary to investigate more complex and sensitive topics which are not so
easy to quantify or where counting would be extremely time-consuming and costly.
- Qualitative methods are used to investigate more sensitive issues which cannot be easily aired in the public
Contribute to understanding of WHO is affected in which ways.
- Qualitative methods highlight the voices of those who are most disadvantaged in ways which might be
difficult. These qualitative methods capture voices that can be missed in the process of collecting the results
of quantitative methods.
- Qualitative methods can also be used for probing key informants to further investigate issues of diversity
Analyze WHY particular impacts are occurring.
- Qualitative methods enable more probing investigation of contexts and development processes and the
complex interactions between contexts, grassroots aspirations and strategies, institutional structures and
Sometimes qualitative methods lead to quantitative approaches to understanding and vice versa.
Example 1: Qualitative lead to Quantitative
There is an early intervention/prevention program that recruits at-risk mothers for child abuse to participate. It is a
2-year program that begins when the child is born. The program uses psycho-educational approaches and
parenting skills training. There is also a coaching element. The program designers later began a “mentorship”
component to assist parents transition out of the program giving program “graduates” a role in the program.
At first, program designers weren’t exactly sure what kind of results the mentorship component was producing. The
social workers had some intuitive ideas, but wanted more definitive information.
After a series of in-depth interviews (qualitative method), the most striking benefit was found in enhancing the
graduate mentors’ leadership skills.
Now when they evaluate their program they have a group of leadership questions that they ask on their pre- and
post-test survey (quantitative method) to collect more information they learned about with the qualitative
investigation done earlier.
Sometimes qualitative methods fall short and quantitative approaches lead to better understanding.
Example 2: Quantitative was better option
There is an organization that designed re-entry programs for incarcerated men. Before release from prison the
organization taught parenting classes to help inmates prepare for their fathering roles and responsibilities. At the
end of each class the inmates were asked, “Did you learn anything from taking this class? If so, what did you
learn?” 100% of the students said they learned something and the students would write about the topics they
learned something about.
When the inmates were asked to take a “test” at the end of the course based on the curriculum content—there
often was no change in their knowledge of parenting scores from pre-test to the post-test. In other words, students
would answer on average 13 out of 20 parenting knowledge questions correctly on the pre-test and about 13 out
of 20 questions were answered correctly at the post-test.
Most Important Message:
When you select a method for any part of your evaluation it should be a reflection of which approach is most
suitable enhancing your understanding about what is going on with your program. A mistake that we sometimes
make in evaluating our program is picking a method because it is happens to be a method that they have
experience with, it is the latest “fad” or it just sounds good to us.
Focus groups are a great example of the latest fad. It seem like everyone wants to do focus groups no matter what
they are trying to evaluate. Focus groups are great when we are trying to assess need or when we want to know
more about participants’ life experience. Focus groups are not so great if we are trying to assess outcomes. This is
because it is difficult to prove that our program is working (or isn’t working) without some quantifiable numbers. We
will learn more about this later in this course—but the idea here is that certain methods are best paired with certain
questions we are trying to answer. Knowing how best to answer these questions and picking the right methods is
the first step in doing great evaluation work.
Rossi, P.H., Freeman, H.E. Evaluation: A Systematic Approach. Newbury Park, CA: Sage Publications; 1996.