How to Write an Evaluation Plan Webinar
Handout Packet
Fictional AmeriCorps First-time Homelessness Prevention Program
Brief Program Description
Increasing poverty and a growing shortage of affordable housing have contributed to an increase in
first-time homelessness among families in a small rural county over the past decade. To combat the
growing problem of family homelessness in the county, a program was created to meet the needs of
low-income households facing a threat of homelessness due to a possible eviction or foreclosure. By
providing targeted housing relocation and stabilization services (e.g., legal counsel, referrals to
financial aid sources) and other assistance, the program is designed to address the housing crisis facing
low-income families in order to prevent first-time homelessness.
Program Logic Model
Project Resources
Core Project
Components
Evidence of Project
Implementation
and Participation
Evidence of Change
INPUTS ACTIVITIES OUTPUTS
Outcomes
Short-Term
Medium-Term
Long-Term
What we invest (#
and type of
AmeriCorps
members)
What we do
Direct products
from program
activities
Changes in
knowledge, skills,
attitudes,
opinions
Changes in
behavior or action
that result from
participants’ new
knowledge
Meaningful
changes, often in
their condition or
status in life
Funding
Staff
AmeriCorps
members
Non-AmeriCorps
volunteers
Research
Provide housing
relocation and
stabilization
services
Provide
educational
workshops
# families received
relocation and
stabilization
services
# families attended
workshops
Total # of program
participants
Increase in # of
families averting
displacement
Increased
knowledge of
responsible home
owner or tenant
practices/skills
Increased
awareness of
resources/services
in community
Decrease in
foreclosures and
evictions
Increased
adoption of
responsible
practices/skills
Increase in # of
families in stable
housing
Reduction in first-
time
homelessness in
the community
1
Group Exercise 1 – Good or weak research question?
Example research questions for the first-time homelessness prevention program
1. How did the low-income families in the county benefit from participation in the homelessness
prevention program?
□ Clearly stated and specific □ Aligns with program model □ Asks for measurable or observable results
2. What impact did the homelessness prevention program have on beneficiaries’ knowledge of
healthy food practices?
□ Clearly stated and specific □ Aligns with program model □ Asks for measurable or observable results
3. Did program beneficiaries increase their awareness of housing, financial, and other types of
services and benefits in the county as a result of participation in the program?
□ Clearly stated and specific □ Aligns with program model □ Asks for measurable or observable results
4. Is the homelessness prevention program reaching its intended target population?
□ Clearly stated and specific □ Aligns with program model □ Asks for measurable or observable results
5. Which family shelters and temporary housing have the best reputation?
□ Clearly stated and specific □ Aligns with program model □ Asks for measurable or observable results
6. What do people in the county think about homelessness?
□ Clearly stated and specific □ Aligns with program model □ Asks for measurable or observable results
2
3
Research
question
What is collected and how?
From whom / data sources?
When collected and by
whom?
How will you analyze the data?
Is the program’s
activity –
educational
workshops -
being
implemented as
designed?
a) Duration of
workshops
b) Participant workshop
attendance rates
c) Topics covered by
member
c) Members delivery of
program curriculum
during workshops
a, b, and c) Members
report details about
workshops in logs with
pre-defined categories of
reporting
a and b) observations of
workshops
a, b, and c) Member logs
a and b) Evaluator
observes members
delivery of curriculum
a, b, and c) Evaluator
collects the workshop logs
quarterly
a) Quarterly observations
by the evaluator(s) using
structured observation
guides
4
Research
question
What is collected and how?
From whom / data sources?
When collected and by
whom?
How will you analyze the data?
What impact
does the
homelessness
prevention
program have
on beneficiaries’
ability to hold a
stable tenancy
relative to a
comparison
group?
Tenancy status of low-
income families at risk of
homelessness
Low income families’
housing stability is
measured with a survey.
Low-income families
participating in the
program serve as the
intervention group.
Low-income families
facing an immediate
housing crisis that do not
participate in a
homelessness prevention
program serve as the
comparison group.
The evaluator administers
the survey at two time
points:
- before the homelessness
prevention program
begins
- 1 year after the
homelessness prevention
program is implemented
5
Group Exercise 3 - Sample Evaluation Plan for AmeriCorps Food for All Program
Introduction: The AmeriCorps Food for All program has a full-time internal evaluation team that will
lead the development and execution of a comprehensive AmeriCorps Program Evaluation. Our
Evaluation Associate will complete both aspects of the evaluation, under the direction of the Associate
Director of Program Development & Evaluation. Based on the Corporation for National and
Community Service regulations, because Food for All is applying for less than $500,000 annually in
CNCS funds, it may elect to complete a process or impact evaluation using an internal evaluator.
Background: To guide ongoing evaluation efforts, AmeriCorps Food for All uses a logic model (see
Appendix A) which outlines the resources, activities, outputs, and outcomes necessary for success of
the program. Logic models are recommended by the Centers for Disease Control and Prevention to
ensure a common understanding of program strategy and provide a framework for evaluating
programs. The program’s outputs are primarily monitored through an online database, customized for
Food for All’s information needs. The database allows local program staff and tour facilitators to input
information including courses and tours held, number of participants and graduates reached, and
volunteer and staff activities. This database allows local program staff to indicate the involvement of
AmeriCorps members in course and tour coordination and execution. Therefore, Food for All is able to
run accurate and detailed reports on the courses, tours, and other education activities coordinated by
AmeriCorps members, including the number of participants reached and program graduation rates.
Food for All also measures short and intermediate-term outcomes, which correspond to changes in
participant’s skills, attitudes, self-efficacy, and behavior with regard to nutrition, food budgeting, and
cooking. Data on short- and intermediate-term outcomes are collected through participant surveys
administered at the end of the 6-week cooking course or one-hour grocery store tour (tour surveys
focus on participants’ intention to change behavior post-tour). Survey questions are designed and
tested by third party research partners, including State University and the Betty Crocker Center for
Nutrition, to assess the general outcomes defined in the Food for All logic model, the outcomes
outlined for each specific curriculum, and participant satisfaction. Participant surveys are administered
locally on paper surveys, which are mailed to Food for All for data processing and analysis.
Purpose: The evaluation will focus on the impact of the AmeriCorps program on reducing food
insecurity among its low-income participants, and will serve as a tool to drive continued program
improvement. The purpose of this evaluation plan is to ensure that the Food for All program meets the
evaluation requirements of the Corporation for National and Community Service, ensuring that our
AmeriCorps program is accomplishing proposed output and outcome measurements.
Audiences and Decisions: Interim results of this evaluation will be shared with the Corporation for
National and Community Service at the end of each grant year. Final results of the evaluation will be
shared at the end of the three-year grant period. Results will also be shared with program
management staff and funders to assist with continuous improvement efforts, and with the Food for
All Advisory Board for review and future planning.
6
Questions The evaluation will address the following key questions:
Program Success
• Does the program result in an improvement in household food security (National Performance
Measure H12) for program graduates?
• Are improvements in food security maintained three to six months post-course?
• Does the program result in an improvement in key food resource management behaviors shown
to improve food security, including planning meals ahead of time, comparing prices, and
shopping with a list?
• Are improvements in food resource management behaviors maintained three to six months post-
course?
• Do grocery store tours result in an improvement in key food resource management behaviors
shown to improve food security, including planning meals ahead of time, comparing prices, and
shopping with a list?
AmeriCorps Member Contribution
• How many participants are reached through courses or tours coordinated by AmeriCorps
members (National Performance Measure H11)?
• How many new volunteers are leveraged by AmeriCorps members, improving the program’s
sustainability?
• Do graduates of AmeriCorps member-coordinated courses show the same improvements in
household food security and food resource management behaviors as other graduates?
Methods:
Participant Pre- and Post-Course Survey:
All adult participants in Food for All courses complete a participant survey that evaluates their
behavior, attitudes, self-efficacy pre- and post-course. In October 2013, Food for All will release a new
Food for All for Adults participant survey that has been tested for validity and reliability through an
extensive research process in partnership with the Betty Crocker Center for Nutrition. This revised
survey will ask participants “How often do you worry that your food will run out before you get money
to buy more?” both pre- and post-course, which is one of the indicators of food insecurity according to
the USDA Economic Research Service. Additionally, our Food for All for Families survey asks adult
participants “How often do run out of food before you get money to buy more?” which is another
indicator of food insecurity used by the Economic Research Service. Food for All will analyze data from
all participants in Food for All for Adults and Food for All for Families courses to determine whether
participants improve their food security status as a result of the course. Analysis will focus on courses
administered by programs that host AmeriCorps members.
Adult participant surveys also ask about key food resource management behaviors shown to improve
food security, including planning meals ahead of time, comparing prices, and shopping with a list. Food
for All will analyze data from all participants in Food for All for Adults and Food for All for Families
courses to determine whether participants improve their food resource management behaviors as a
result of the course. Analysis will focus on courses administered by programs that host AmeriCorps
members.
7
Food for All Long-Term Follow-up Study:
Food for All is planning an evaluation that will begin in Fall 2013 to measure and document the long-
term impact of Food for All on its participants. This evaluation will be led by an external evaluator, and
will include a comprehensive measure of food security based on definitions provided by the USDA
Economic Research, with the goal of determining whether gains in food security status are achieved
and/or maintained three to six months post-course. The study will also assess whether the key food
resource management practices taught in Food for All courses are maintained three to six months
post-course, and will confirm their correlation with measures of food security in this population. This
follow-up study will use a sample of participants from AmeriCorps member-coordinated courses.
Because of the large number of participants reached by Food for All annually (over 23,000 in 2012), a
sampling structure will allow Food for All to collect detailed information about the long-term impact of
the program without expending more staff time and resources than necessary. For example, for a total
population of 23,000, less than 400 participants would need to be surveyed to report results with a
95% confidence level and a confidence interval of 5. Additionally, since many participants have limited
access to technology, unpredictable work schedules, and frequent changes of address, it would be
impossible to survey every graduate 3- to 6-months after the course and an extremely arduous
process to impose on local partners.
The follow-up study will assess the link between outcomes measured immediately post-course on the
participant survey and the maintenance of those outcomes three to six months post-course. If these
intermediate- and long-term outcomes are shown to be strongly correlated, Food for All will not
continue to measure outcomes three or six months post-course. If they are not shown to be
correlated, Food for All will pilot a three month post-course email survey to determine if the response
rate and respondent demographics make a web-based survey feasible.
Data Analysis:
Food for All will use its existing online database to determine how many participants were reached in
courses coordinated by AmeriCorps members and how many new volunteers were leveraged by
AmeriCorps members during the grant period. Output and outcome data will be analyzed using IBM
SPSS Statistics, which Food for All recently implemented as part of its commitment to maintaining high
quality data and analytics.
Limitations: Food for All chooses to measure food security using one survey item to reduce participant
burden and allow for collection of robust data on other key outcomes, including food budgeting,
nutrition, and food preparation behaviors, attitudes, and self-efficacy as well as participant
satisfaction. The follow-up study will validate the use of this single measure as an accurate assessment
of food security status immediately and three to six months post-course. Additionally, due to the time-
and resource-intensive nature of conducting a follow-up study of food security status, Food for All will
use the results from the already-planned follow-up study to assess long-term food security impact and
will not conduct ongoing follow-up surveys, unless the results of the evaluation suggest ongoing
follow-up is needed.
8
Timeline & Logistics:
Evaluation Element
Timeline
Conducted By:
Course and Tour Data
Collection
Ongoing throughout grant
period
Local program staff; input into
Food for All database
Participant Survey Data
Collection
Ongoing throughout grant
period
Local program staff
Participant Survey Data
Analysis & Reporting
End of each grant year
Food for All Evaluation
Associate
Long-Term Follow-up Study
Fall 2013 to Summer 2015
External Evaluator with support
from Food for All Evaluation
staff
9