FY2019 Low-Cost Short-Duration Evaluation of Education

Hello and welcome to this Institute of
Education Sciences webinar about the Low-Cost, Short-Duration
Evaluation of Education and Special Education Programs Interventions.
My name is Sarah Brasiel, and I’m a program officer at the National
Center for Special Education Research. I am also the contact for 84.324L grant
program. Phil Gagne is a program officer at the National Center for Education
Research, and he is the program contact for 84.305L grants. I will be going
through this webinar explaining a bit about the background of IES in this
competition. Today I’m going to provide a brief overview of IES and its mission,
then we’ll talk about the grant program requirements and specifics,
going through each section of the Project Narrative. Then we will take you
through preparing and submitting an application. The legislative mission of
IES is comprised of three components. IES was established by law in Congress in
2002 to do the following three things: describe the condition and progress of
education in the United States, identify education practices that lead to
improved academic achievement and access to opportunities, and evaluate the
effectiveness of federal and other education programs. This graphic
represents the organizational structure of IES. We are led by a Director,
who receives advice and consultation from the National Board for Education
Sciences. The Board consists of 15 voting members, who are appointed by the
President and confirmed by the Senate. Our Standards and Review Office oversees a
scientific peer review process for IES grant applications and IES reports.
We also have four centers within IES. The National Center for Education
Statistics is the primary federal entity for collecting and analyzing data related
to education. Within NCES, you may be familiar with the NAEP
assessment, the National Assessment of Educational Progress. Under NCES,
you’ll also find many large national longitudinal data sets, including,
for example, the Early Childhood Longitudinal Study. The National Center
for Education Evaluation and Regional Assistance conducts unbiased,
large-scale evaluations of education programs supported by federal funds,
provides technical assistance, and supports the development and use of
research and evaluation through the United States. In NCEE,
you will find the What Works Clearinghouse, (or WWC),
and the Regional Educational Labs (RELs). The two centers that award grants are
highlighted here in blue, the National Center for Education
Research, referred to as NCER, and the National Center for Special
Education Research, or NCSER. The grant opportunities that we will be
talking about today are managed through these two research centers.
You’ll notice here that the research centers are separate from the Standards
and Review Office, meaning that we program officers are not involved in the peer
review process. So this allows us to work closely with you, providing technical
assistance to you on your applications. We will discuss more about that later in
this webinar. IES grant programs. The objective of our research grants is to
answer four questions. What works to improve student educational
outcomes, so we can disseminate what works? What does not work,
so we can stop using it? What works for whom and where,
so we can use it with the appropriate people in the appropriate places?
We also want to answer the question, why does it work, so we understand how to
improve education and can build on this understanding. The director sets the Institute’s
priorities, which are also approved by the board. Partnerships are of growing importance in
our priorities. And that’s in order to help focus research on the issues that
most concern policymakers and practitioners and to help researchers
better communicate their findings, and do it in useful ways for policymakers
and practitioners to use in their practice. These policymakers and
practitioners are to have a strong role throughout these partnerships in setting
the research agenda. And there’s to be constant communication
between the researchers and the practitioners on what is being found and
what research should be done. The partnership is to evaluate a specific
program or policy of high importance to the state or local education agency.
For NCSER, in our center, the National Center for Special Education
Research, this could be another kind of agency that provides overarching
support for education needs for students with or at risk for disabilities.
I’ll say more about this in a couple of slides. A change for FY 2019 is that
grant funds may be used to collect primary data for analyses if the partner agency
deems these data and the associated analyses key to its decision making
regarding the intervention being evaluated. However,
Low-Cost Evaluation projects generally rely primarily on secondary data to obtain
the education outcomes. Opportunities to more directly support
state and local agencies, and their needs, is a priority for these partnership
grants. We want to take advantage of the opportunities to use administrative data
that already exists, provide information to education agencies
that they need in a timely manner, create additional opportunities,
and identify strengths and weaknesses in the applicability of this type of
evaluation. The first requirement
for the application is covered here and in the
next slide. The difference between the two grant programs, 305L and 324L,
is that NCER’s 305L includes students from pre-kindergarten through
post-secondary and adult education as populations that can be studied.
For NCSER and 324L, our range includes students from infants and toddlers through
grade 12. You can study any grade or grade range within any of these boundaries.
For example, you could focus on fifth grade, examines students in
middle school, or study transition between high school and college. In NCSER,
we focus on improving a range of outcomes, ultimately leading to improved education
outcomes overall. But in our group, these can be functional outcomes as well.
And I’ll say more about that in the next few slides. It may also include an
agency other than a state or local agency; for example, one that manages providing
early childhood education for toddlers and infants. A change for fiscal year 2019
is that Grant funds may be used to collect primary data for analyses if the partner
agency deems these data and the associated analyses key to its decision-making
regarding the intervention being evaluated. However,
Low-Cost Evaluation projects generally rely primarily on secondary data to obtain
the education outcomes. Applicants proposing to study children at
risk for developing disabilities must present research-based evidence of an
association between risk factors, specifically defined risk factors in their
proposed sample. In addition, applicants must describe the process of
identification of specific disabilities. The determination of a risk factor for
disability status must be made on an individual child basis and may
include, for example, factors used for moving children to higher
tiers in a response to intervention model. The method to be used for determining if a
child is at risk for developing a specific disability must be made explicit in these
applications and must be completed as part of the sample selection process.
So evidence for at-risk status cannot be they’re from low-income families nor English
learners. That’s not sufficient for this purpose or definition. In addition,
applicants must identify the disability or disability categories sample children are
at risk of developing. IES focuses on research that improves the
quality of education for all students. Therefore, research must address the
education outcomes of students. These student outcomes can be grouped
under academic outcomes, as well social and behavioral outcomes
that support student success in school. These academic outcomes can include test
scores, standardized tests, researcher-developed tests, and of course,
tests or graduation-required tests or grades. Also, measures of progress can
include course and grade completion, high school graduation and dropout,
post-secondary access, progress, and completion. Again,
anything related to post-secondary is funded under 305L and not 324L (but can
include those with risk for disabilities). Social and behavioral outcomes can include
things as broad as social skills, responsibility, and cooperation,
learning strategies, goal-setting, or self-regulated learning, attitudes
(including motivation and academic self-concept) and behaviors
(including attendance and discipline). For students with at risk for
disabilities, student outcomes also include developmental outcomes for younger
students in these domains, cognitive communication, linguistic,
social, emotional, adaptive, functional, or physical, and they can also include
functional outcomes for older students that improve education results and
transitions to employment, independent living, and post-secondary
education. The second general requirement involves the setting and outcomes.
This is a chart of the acceptable outcomes you can include in your application by
age. They are separated for a 324L where the sample can include infant and
toddlers, and then pre-kindergarten and above for both 324L and 305L.
For pre-kindergarten, 324L can also include developmental
outcomes. For participating children in grades K – 12, these are the outcomes
for 305L and 324L, and some additional outcomes specific only to 324L,
where we also include those functional and additional transition outcomes for our
populations. A change for FY 2019 is that student outcomes can also include
employment and earning outcomes. These are defined as long-term post-school
student outcomes that include indicators such as hours of employment,
job stability, wages, and benefits. For 305L only, there are additional
outcomes. Post-secondary education is defined as baccalaureate and
sub-baccalaureate. Outcomes include access to and persistence or progression
through and completion of the education program. These are often degree
or certification programs. Additional outcomes for students in
developmental or remedial education programs in this setting can be included.
You may also examine the outcomes of achievement in reading, writing,
English-language proficiency, and mathematics. Adult education includes
students who are 16 years or above and outside the K through 12 systems,
who are expected to be in such programs as adult basic education,
adult secondary education, adult ESL, English as a second language,
and high school equivalency test preparation; that is preparation for
the General Education Diploma or the GED. The change for FY 2019 is that student
outcomes can also include employment and earning outcomes, also applies to these
programs. Employment and earning outcomes are defined as long-term post-school
student outcomes that include indicators such as hours of employment,
job stability, wages, and benefits. The third general requirement for these
applications is that they evaluate education interventions using secondary
data. Interventions here is defined as the wide range of education curricula and
structural approaches, professional development, technology,
and practices, programs, and policies that are implemented with the
intention to improve student outcomes. The implementation must be of high
importance to the School Education Agency, or SEA, or the Local Education Agency,
LEA. The implementation of the intervention is managed or overseen by
these agencies, not the research partners. An implementation of the intervention will
occur in Year 1 of the project. It must be a newly implemented program so
the evaluation can occur within that first year. The intervention is expected
to produce meaningful improvement that the district is interested in targeting.
And at a minimum, administrative data must contain student education outcomes.
The fourth general requirement is that the applications must come from a partnership.
That is a minimum of one research institution and one education agency.
The Institute does not endorse a specific model of research partnerships. However,
the Institute views research partnerships is going beyond two common forms of
collaboration. That it is going beyond the standard form of the researcher being
hired by the agency performing a specific service, and then reporting the
results, as well as beyond the researcher having an initial interest and obtaining
permission from an agency to carry out that research within the agency
schools. Although these can be covered in our other grant competitions,
these kinds of partnerships are not appropriate for this application.
A joint letter of agreement from the two agencies or more, or agencies or
organizations setting up the partnership, needs to be submitted.
It’s more like a Memorandum of Understanding, or MOU,
showing both sides agree to the work proposed. That is a letter from the
research organization and the education agency and the other partners if they’re
included. Other members of the partnership can provide separate letters of agreement.
Each partner must contribute at least one PI, or Principal Investigator,
and may contribute other Co-PIs or Co-Investigators.
There will be one Principal Investigator Project Director because that is the way
our grants administration considers one individual with responsibility for the
oversight of the grant. Only one institution.
Others will be considered Co-PIs. The agency PI must have decision-making
authority in the agency but does not need to be the superintendent.
This person must be someone who oversees the education issues to be studied or
oversees the policy and program and its implementation across the district or
state. Previous collaborations are appropriate to include,
but they’re stronger if they include personnel from different institutions.
The definition of a research institution is broad. To be eligible to be the
research institution partner, you have to show that the institution has
the ability to do this type of research. This is very rigorous research.
Those agencies can include nonprofits, for-profits, public and private
institutions and agencies, colleges and universities,
as well as other research firms. Note there are no longer any requirements
for the Principal Investigator and the Co-Principal Investigator to have
experience with this research, but the project team should have
experience with and/or training in the evaluation design to be used.
Now let’s turn to the organizations that are eligible to serve as the state
education agency or the local education agency. The examples are all
eligible in this slide as long as they are overseeing some aspect of education.
That’s early learning, elementary, secondary, post-secondary,
higher education, or adult education. Sometimes different types of public
organizations are established to oversee a specific area of education.
And those would be acceptable partners. If there is any link to the main state
education agency, it’s helpful to include that agency as well.
For this grant program, IES uses a broader definition than the
definition which focuses on the primary agency responsible for supervision of
public and elementary and secondary schools so that you have a broader pool of
potential partners at the state education or local education agency level.
In addition, the NCSER age ranges will include other agencies
that serve infants and toddlers. In addition to what
is listed on the slide, there are several
types of organizations that can be useful partners but cannot
serve alone as the agency partner. That is intermediate districts,
sometimes called service districts, that provide services to multiple
districts but do not have decision-making authority over implementing programs and
policies cannot serve as the primary agency. However,
they can serve as the agency partner if they do oversee the program or policy
that’s being studied. Organizations made up of LEAs,
and sometimes other organizations such as universities or practitioner,
like research and implementation networks but do not have decision-making authority
over the implementation of policies and programs within the LEAs,
cannot serve as the primary partner. They need to have some of the LEA members
join as the education agency partner. Non-public organizations that oversee or
administer schools, such as an education management organization or charter
management organization, are also not eligible, although,
they can be partners. You must include the public, state,
or district agency that oversees the schools that are involved as well.
There are also some very small local education agencies. In some cases,
they may have only one school. These can apply as a partner,
but the panel reviewers may consider the work less significant than projects
involving multiple schools. State and local post-secondary systems can
serve as the education agency partner. However, an individual post-secondary
institution that is one campus cannot serve as the agency.
The whole system must sign on to the project. But the evaluation does
not have to be across the whole system. It can take place at only some of the
campuses. If there’s an agency that oversees the post-secondary system,
its inclusion will strengthen the application. And that post-secondary
system cannot study themselves. One department or Center cannot study
another or even the whole system. An outside institution must be involved.
Finally, for adult education, Title II of the Workforce Innovation
Opportunity Act defines organizations eligible for federal funding as adult
education providers, including community-based organizations,
institutes of higher education, public or nonprofit agencies,
and libraries. So, for example, one of our partnership grants research and
practitioner partnerships, Penn State partnered with Miami-Dade
Public Schools, Chicago Citywide Literacy Coalition, and Houston Center for
Literacy, that is the city government, while the Citywide Literacy Coalition is a
non-governmental organization. And this was a partnership to examine
adult education for our low-skilled population as well as immigrants.
One research institution in one state or local agency is the minimum requirement
for a partnership. For example, several education agencies may face a
similar problem, or agencies located in a region, especially small districts,
may share common interests. So it may make more sense to have a group
of them come together. Having multiple education agencies may
also increase the significance of the proposed work. Partnerships may include
additional partners if they’ll increase the quality of the research.
You’ll want to show how the additional partners have similar interests.
Non-research organizations are often part of the research partnership for many
reasons, including their interest in the issue, access to data as providers
of services with links to community. For example, a project seeking to link
early childhood education with kindergarten has the United Way as a
partner because of its role in supporting early childhood education in the district
and its link to stakeholders. As you have more partners,
coordination becomes more difficult. So it’s important to address how the
partnership will work together, keep in contact, and make decisions.
The one type of partnership that we would not recommend is the inclusion of multiple
education agencies, whose only similarity is that they have worked with the same
research institution rather than that they share a common education issue or problem
they want to have more information about. Non-education and state local agencies may
be useful partners as long as the Education agency is also a partner.
If you’re looking at a specific population,
let’s say you wanted to study foster children, you would have your
education agency, but you also might want to include your social service agency that
focuses on work with foster children. For example, another project combines the
Connecticut Department of Education and the Connecticut Department of Justice,
as well as the University Center on Children. And they’re examining the
education of court-involved youth. Again, you can also include more than one
research institution, but you have to argue that there is a
shared interests and they will make a unique contribution to the work beyond the
one agency that’s already the primary partner. The fifth general
requirement relates to dissemination. The Institute considers all types of
findings from low-cost evaluation projects to be potentially useful to researchers,
policymakers, and practitioners. Therefore, the Institute expects
dissemination to include findings of a beneficial impact on student outcomes to
support the wider use of the intervention and more in-depth further evaluation,
as well as findings of no impacts or negative impacts on student outcomes.
These are important for decisions regarding the ongoing youths and wider
dissemination of the intervention and further revisions of the intervention and
its implementation. Dissemination is to be included in Appendix A of your
application. And failure to include this plan in Appendix A can lead to rejection
of your application. So it’s important when writing the
application that the education agency understands that all findings are to be
released. Partners can have the right to the first review of findings so there
are no surprises, but findings cannot be withheld. The education agency has to
want to learn from the project, and the user results have to be shared.
The briefing and written brief are required. The other examples of
dissemination that can be proposed are included in the application.
The Institute expects projects to carry out a broad dissemination to multiple
audiences. On this slide, we provide a quick checklist for you to
determine the fit of your research question and intervention for this
application. We have several other grant competitions that you could explore
submitting an application to, proposing to answer your research
question, and the needs and the goals of the agencies you work with.
It’s important of the Education Agency, again, is the one directing the work to be
done and the evaluation. The focus of the evaluation is based on
the research questions about how these programs or practices
are helping their students. Do not propose that the
partnership will determine what programs
or policies it will evaluate after receiving the grant.
The purpose of this grant is to provide an opportunity for local and state
education agencies to have fairly quick information that they would like to have
about the impact of a program practice or intervention. These partners must drive
the research questions and the plans. You’ll want to identify specific education
interventions in the process of applying for this application.
In consultation with your partner agency, they must drive the question and the
research to be conducted. It must be of high priority to them and
intended to improve student outcomes within a year of implementation.
You must carry out that evaluation while it’s being implemented by the agency using
a very rigorous research design. These can include an RCT
(or randomized controlled trial), or an RDD design. That’s a regression
discontinuity design. Those in the field of special education
can also include single-case designs in these applications.
Secondary data must be used. The estimated overall impacts must be
calculated at the end of the study, and these data must be shared.
If data are available, you must estimate subgroup impacts for
important subgroups and examine other moderators and mediators of interest,
fidelity of implementation where applicable and possible,
and comparison group practice where applicable and possible.
A change for fiscal year 2019 is that grant funds may be used to collect primary
data for analyses if the partner agency deems these data and the associated
analyses key to its decision making regarding the intervention being
evaluated. However, Low-Cost Evaluation projects generally rely primarily on
secondary data to obtain the education outcomes. Expected products of
the grant include the causal evidence of the impact of or lack of a clearly
specified intervention that the agency implements. That is, again,
overall impacts and subgroup impacts where possible. Another product is advice
for the agency. The advice is based on the research in terms of continuing or
expanding the use of the intervention or practice and advice about future research
needs. That is evaluation needs in terms of variation on impacts, moderation,
mediation, generalized ability, and replication, as well as continued
modifications to the intervention or development. The highest standard of
evidence is required, again, for these grants, and the research design
must include randomized control trials, otherwise known as RCTs,
regression discontinuity designs, otherwise known as RDDs. And, also,
for the National Center for Special Education Research, or NCSER,
acceptable designs include single-case designs. Next,
I’m going to go through the sections of the Project Narrative that you’ll be
completing as part of your application. The Project Narrative is the substantive
part of your application. It contains the five sections listed on
the slide. Inside each of these sections, you will see in the Request for
Applications that we’ve set out both requirements and recommendations.
The requirements are the minimum you must address in order to have the application
accepted for review. The recommendations are what the peer
reviewers are asked to look for as they score the quality of your application.
So let’s start with the Significance section.
The purpose of the Significance section is to answer the question why this evaluation
is important to do. You want to lay out the education problem issue that the
intervention you’re evaluating is to address for the state or local
education agency. It’s of secondary importance,
but still valuable to discuss, if it’s relevant to other state or local
education agencies. Then you want to explain the actual education
intervention you’ll be evaluating and describe all of its components,
and then provide a rationale for why this intervention should be able to improve
student outcomes within the short period of time such as a quarter to one year.
You may also include a theory of change. You should discuss why this is different
in the status quo, either in this education agency or in other education
agencies. And you can provide related findings from other studies and how this
study will then improve upon any past work. Then, describe how the
intervention will be implemented because the program or policy that you’ll be
evaluating must be implemented under the education agency’s control.
That means they directly implement it or they’ve contracted out but still oversee
the implementation of the intervention. Show that there’s adequate funding
available for intervention implementation during the first year of the project.
And we do require that implementation must take place during Year 1 of the project.
And you should document why you expect that level of implementation to be high
enough to impact student outcomes. Then, you want to describe the sources of
secondary data to be used in the evaluation. Who actually collects
them? How are they going to be transferred to the research institution by the first
quarter of Year 2 of the project? Also, if this is going to be the first
year that the intervention is actually implemented in Year 1 of the project,
you probably want to provide some additional evidence that it will be
implemented. For example, has some sort of legal authority been set
up to make sure it’s implemented? Is there a special office in charge of
oversight of the intervention? And who will, of course,
implement it in the next year? In the Significance section,
you will also describe sources of primary data to be used in the evaluation.
Describe how these data are collected and how these data will be obtained by the
researchers by the first quarter of Year 2 of the project. Please note,
gathering primary data is not required. A change for fiscal year 2019 is that
grant funds may be used to collect primary data for analyses if the partner agency
deems these data and the associated analyses key to its decision-making
regarding the intervention being evaluated. However,
Low-Cost Evaluation projects generally rely primarily on secondary data to obtain
the education outcomes. So we’ll move on now to the Partnership
section. The purpose of the section is to answer the question,
is this partnership a real one. This is one of the major concerns of the
peer reviewers that this is a partnership, that you are addressing an intervention of
high interest to the education agency and the education agency has helped set out
the research questions to be addressed, as well as this evaluation with such
priority that the agency is quite likely to use the results. We’re not looking for
researchers testing their own interventions and the agency allowing it
but not being signed on to implementing such an intervention if the results are
good. We’re actually looking for interventions that the agency is
testing and that are priority to the agency already and not necessarily in the
future. So, to this end, you want to describe both partners,
the research institution, and the education agency.
And if there are multiple offices within the agency that need to be involved
in this work, please describe them as well. Note any other members of the
partnership. Describe the common interest of all the partners and how they will
benefit from the evaluation, the process by which the partners
determine the intervention to evaluate, again, showing it’s a high priority of the
education agency. Then describe the data sharing agreement, what is the strategy
the data collected by the education agency will be then turned over to the research
institution for analysis by the first quarter of the second
year of the project. Now let’s move on
to the Research Plan. And we’ll walk through a set of issues to
be addressed in your Research Plan. It’s good to restate your research
questions and hypotheses. You’ll probably note these originally in
this Significance section but it’s good to note them again so that it’s clear how
your research design will answer the questions you’re setting out to answer.
So, first, you want to note the sample and setting you’ll be working in and how the
sample that you’ll be working with will allow inferences the full population of
the education agency for which this intervention has been designed.
Note any exclusion inclusion rules and justify them. If you’re including any
retrospective data in your study, you should address these points for it as
well, and also describe whether you will combine the retrospective data with the
prospective data in the analysis or analyze it separately.
And describe how you will address missing retrospective data. In addition,
you should note the strategies that will be used by the agency to increase
participation and reduce attrition, and describe the setting of the study this
work is being done in and how generalizable it is to the full agency.
So, for example, if you’re working in a state, you should discuss whether
schools in your study are from all parts of the state or perhaps only similar to
urban or rural parts of the state. Then, you’ll want to describe the design.
You want to propose a strong research design that can be used to make causal
inferences. Draw conclusions on whether or not the intervention is working and
benefiting students. If there are any weaknesses in the design,
you should openly discuss them. Note their internal validity because the
reviewers will know these weaknesses and will want to see that you can account for
them, or at least recognize that they are there when interpreting your findings.
Discuss how you’ll determine equivalence of baseline. Include any checks for bias
that may come from attrition, either overall or differential. For 305L,
you must propose either a Randomized Controlled Trial or a Regression
Discontinuity Design. And for 324L you must propose an RCT, RDD,
or a single-case experimental design. It is helpful to look at the What Works
Clearinghouse evidence standards for all three of these designs.
Because if you implement them correctly, you should be able to meet the WWC
evidence standards. IES-funded evaluations are automatically submitted to the What
Works Clearinghouse for a review based on the evidence standards.
Let’s start with the Randomized Controlled Trial. You should detail the
unit (for example, students, classrooms, or teachers, schools,
districts) of randomization, the process you will use to randomize
units to the treatment and control groups, and the probability for each unit to be
assigned to the treatment or control groups. Other projects have found
it very helpful in order to maintain the integrity of the research design to
provide technical assistance of the agency and to the on-the-ground implementers on
how important it is to hold to a design. For example, the research institution
might provide a list of students categorized into treatment and control
groups to the education agency based on data provided by the agency.
But the people in the district, who are actually responsible for
scheduling those students, or contacting parents to schedule them,
may have other pressures on them. For example, well, this parent wasn’t home
to give permission, or the student’s schedule is difficult to make it work,
so I’m just going to swap them to another group. It’s important to work
with the education agency, especially those people on the ground tour
making sure those students get into the right group so the design is not
compromised. In your application, you should also describe the informed
consent process for students, parents, teachers, faculty, and other district
staff. RCTs can be done in a number of different ways, and each of them has
its own potential challenges. For example, there may be mandatory assignment where
all schools or all students are included in the assignment. This approach provides
generalizability, but there may be some resistance among some of them that didn’t
want to be in the treatment, so there may be lower fidelity of
implementation. On the other hand, the mandatory assignment may be the only
approach available. So in such cases, it’s important to convince all the groups
involved in the treatment that this is an important intervention to implement in
order to carry out a rigorous evaluation of it. A second approach is to only
include volunteers in the randomization. This approach should lead to high
treatment implementation because only people interested in the treatment will
take part. But as a result, those assigned to the comparison group may
be less interested in remaining in the study because they want to be in the
treatment but they won’t receive it. In a response, they may drop out of the
study or they may try to seek some similar treatment on their own. In some cases,
it may be possible to offer some kind of alternative treatment that isn’t
expected to affect the measured student outcomes or to assure the control group
that they will receive the treatment in the near future. Lotteries are a type
of volunteer randomization. The students are put in the control group
may leave the study in order to obtain a similar intervention. For example,
a study of a magnet school lottery in a city found that parents of students who
did not get in differentially sent their children to non-urban schools.
This differential attrition may bias the comparison group to include more units
that could be less motivated or less interested in the intervention.
In some cases, it may be possible to follow the leavers. For example,
a study in Portland obtained state-level data in order to include students who had
left the Portland schools and could not be found in the Portland administrative data
so they could be included in the intent to treat analysis. Another approach is to
use a staggered rollout in which, for example, some schools are randomly
assigned to receive the intervention in Year 1 of the study,
some are assigned to receive it in Year 2, and others in Year 3.
The later year schools are used as the comparison group for the Year 1 schools.
Year 1 and Year 2 treatment schools may also be compared to Year 3 treatment
schools. This approach may reduce leavers as there is a promise that all will
receive the intervention. For interventions that are expected to
need one or more years of implementation to be fully effective,
the staggered rollout may not provide enough time to provide a fair evaluation.
In addition, it’s important to determine what is occurring in the control group as
they may try to start implementing sooner. And examine if the intervention itself is
changing between years of implementation in the Year 1 schools may lead to
improvements in the intervention and its implementation. Another approach is to
compare variations of an intervention. For example, the treatment group receives
the full intervention and the control group gets some component of the
intervention so they both receive something. It may be that the
district already has evidence that one component of the intervention is widely
implemented and improves student outcomes and wants to determine if an additional
component leads to further improvements. Here are two examples of how low-cost
projects have designed RCTs. The Oakland Unified School District,
in conjunction with Mathematica, evaluated an intervention for students who
are several grades below their expected reading level. For secondary students who
volunteered for the study, they randomized these students within
schools to the level of literacy intervention,
a control group or a third group. The third group allowed Oakland to
maximize the number of students who would receive the treatment.
If a treatment student left their school, a member of the third group was assigned
to take their place. If they only had randomize students to the
treatment and control group, they would not have been able to have a
control group student take the place of a treatment group student who left the
school as they would compromise the design. By having the third group,
which is not in the study, students from that group can be taken to
fill any opening. Their third group students are not included in the study,
and their education outcomes are not analyzed, but they do fulfill
Oakland’s desire to treat the maximum number of students possible.
The University of Wisconsin at Milwaukee and the Milwaukee School District
evaluated an additional component for an ongoing check-in and check-out
intervention to improve behavioral outcomes for elementary students.
Fifty schools took part with half the school’s assigned to receiving the typical
check and connect check-in check-out intervention,
and the other half implemented a modified version that contained additional
reinforcements. Before randomization was done, schools were matched on having
adequate fidelity of the basic program, number of students in the program,
and the number of behavioral incidents. Then each match is randomly assigned to a
treatment and control group. You can also propose a Regression
Discontinuity Design. And here, you want to describe how you will
determine that there is a true discontinuity at the cutoff point and not
at other points where discontinuity would not be expected. You want to describe how
you’ll determine that 1) there’s no manipulation of the assignment
variable, 2) the treatment in comparison groups have similar baseline
characteristics except for the assignment variable so you do not differ in ways that
would indicate selection bias, 3) that there will be high levels of
compliance to assignment, that most or all treatment group members
will receive the intervention and there will be a few crossovers of comparison
group members who receive the intervention (and how you will document the bias).
It is important to talk with people at the school who actually assign the students
about why it’s so important to stick to the cut point. One approach is to allow
a small percentage of “wildcard positions” in which a teacher or principal or
counselor can assign, or not assign, a student to treatment regardless of the
value of their assignment, variable based on other factors.
These students are not included in the study because of these wildcard positions,
but the school and these personnel were more comfortable with the RD design
because of how these students had been placed. You should also discuss the
sensitivity analyses and robustness checks that will be used to assess the influence
of key procedural or analytic decisions. For example, functional forms and
bandwidths on the results. If you’re coming into the study with some
retrospective data from a previous year of the treatment, you could have a
stronger application by doing some preliminary work on the older data to show
that a true discontinuity exists and there are no signs of manipulation of
assignment. Although, again, you would be doing your main analysis on
implementation data from Year 1 of the project. You might also propose to
include the retrospective data in your analysis of the prospective data you will
collect in Year 1 of the study. For 324L applications only,
single-case experimental designs can be used. You will need to justify the use
of a single-case experimental design as opposed to an RCT or RDD.
Describe the repeated systematic measurement of a dependent variable
before, during, and after the active of manipulation of an independent
variable. It’s important to include outcome measures that are not strictly
aligned with the intervention. Also, describe any quantitative analysis
techniques in addition to visual analysis for analyzing the resulting data
(for example, between-case effect size calculations). See the 324L Request
for Applications for additional recommendations. Power is important
to describe for all of these designs I just mentioned.
Adequate power avoids the issue that if you do not find a statistically
significant result, it’s not because your sample size is too small,
so you’ll want to describe your power analysis in detail and justify the method
you’ll use to calculate the power, including all your assumptions.
The reviewers often redo your power calculation and report this out into the
full panel. So if a reviewer says, “This is a good power analysis,” that’s a
very positive comment to be made in panel. While an adverse comment is,
“I couldn’t replicate the power analysis.” For that reason, it’s important to provide
the formula you used to calculate power, the known value of the parameters
used (for example, number of clusters and participants within cluster), the
parameters whose value were estimated (for example,
inter-class correlations, row of covariates), and other aspects of
design that affect power (for example, stratified sampling or blocking repeated
observations), and predicted attrition and how you address it in the power analysis.
You should provide a power analysis for the main impact analysis,
for any confirmatory subgroup analyses, and for tests of mediation and moderation.
You should discuss the practical meaning of the minimal detectable effect size you
identify. Does the change in outcome really matter for the student,
for the teacher, for the school? If you’re proposing a very high minimal
detectable effect size, there’s often a reaction that,
can such a large impact be found? And that may be tempered by what practical
change it represents. Then, for a very low minimal detectable
effect size, that’s often seen as good. But, again, there’s the question of,
is it an important or non-important change?
If it’s seen as small or trivial, you may want to say,
“we can detect this level, but we consider it an important change on
the outcome to be the larger minimum detectable effect size to be practically
important.” You should describe the education outcome measures that you’ll be
using to see if students improve. And these should be measures of high
relevance of the state districts and schools. For example, test scores,
grades, promotion rates, retention rates, high school graduation rates,
discipline referrals, special education placements,
college enrollment, or completion rates. You should note the reliability, validity,
and appropriateness. Depending on the purpose of the
intervention, you may want to also include social and behavioral competencies and
employment and earnings outcomes. Student education outcomes are to be
collected during Year 1 of the project when the intervention was being
implemented. You can also include data from a previous year if the intervention
was implemented in a similar manner. And you should link why these measures
should be affected by the intervention. There are optional measures that you may
obtain from administrative data or from primary data collection. For example,
you may want to look at intermediate outcomes.
If you’re evaluating an intervention that works by changing teaching that is
supposed to affect student outcomes, there may be classroom observations
recorded by principals or coaches or surveys of teacher instruction that you
could obtain as part of your secondary data, or through primary data
collection. That can be used to determine if the intervention was adopted by
teachers and did change teacher instruction.
There may be moderators you want to look at. Often, subgroups are
identified in the administrative data. There may be mediators,
intermediary outcomes, for example. There may be measures of fidelity of
implementation collected. Also, it’s important to know what’s
occurring in the comparison group to provide evidence that there’s a
treatment contrast so that any change in student outcomes can be attributed to
the intervention, or is more likely to be attributed to the intervention.
The use of these measures is optional and their inclusion can be justified by
the education agency’s interest in them. If you do include them,
you should describe how they will be obtained (for example,
from a secondary source or from primary data collection). If primary data are to be
used, you should describe the data to be collected, their collection,
including an encoding, and their transformation into the
variables to be used in the analyses. Keep in mind that low-cost evaluation
projects are to primarily rely on secondary data. The collection analysis
of primary data should not be the main focus of the project.
You will want to detail how you’ll analyze the data. Again, linking the analysis
directly to show how it will answer the research questions, showing that the
analysis fits with a design you proposed. Are you addressing clustering of students
in classrooms and schools? How are you addressing any missing data?
If you’re pulling multiple datasets together, how will they be
linked? Are there IDs for the students in all the data sets?
And then describe the other analysis to be done as well as the main impact if you’re
looking at subgroups, moderators, mediators, and fidelity of implementation.
It’s very helpful to include your model in the equation form,
explain the variables that will be included in it and the coefficients of
interest, and describe the software you’ll use to run the model.
For your impact analyses: 1) include and explain the model(s) you will
estimate to determine the impact of the intervention, 2)
describe your Intent-to-Treat analysis and any additional analyses, for example,
Treatment-on-the-Treated (such as complier average causal effect), variation
in impacts for subgroups or sites. For regression discontinuity designs,
show how your analysis reflects whether you expect to have a sharp or fuzzy design
and describe how you will analyze impacts at the cutoff point.
If you expect to have a sharp design, indicate how you will handle any no-shows
and crossovers. If you expect to have a fuzzy design, describe how you will
determine whether the assignment variable is a strong predictor of participation in
the intervention. Describe how the analysis will address any use of multiple
assignment variables. Describe the sensitivity analysis you will
do to check the robustness of any choices that you have made. If you intend to
impute missing data, describe the approach you will use to
provide unbiased impact estimates. Explain how you’ll measure and report on
effect sizes in ways that policymakers and practitioners can readily understand.
For example, an evaluation of a reading or math program might report on the number of
months gained in reading or math skills as a result of the intervention. Let’s move now on
to the Personnel and Resources section. You want to identify
the key personnel on the project team. The PI may be from either the research
institution or the education agency. That’s up to the choice of the partners.
But there must be a PI from one and a PI from the other (who will serve as Co-PI in
the project). The research team should have some experience and training with the
evaluation design being used. The PI or Co-PI from the education agency
should have some decision-making authority for the intervention being evaluated.
When you write this section, think about all the work you are promising
to do. Identify each person, their expertise or experience in doing the
work, and their time on the project to do that work. For example, you might say,
“So-and-so is an experienced expert in serving methodology as shown by their
work on these three grants and these six papers they’ve published.
They will be in charge of analyzing the district survey that is being used for
teachers. And they are on the grant for 10% of their time a year,
which is enough to carry out that analysis.” In sum,
the Personnel section should identify what work is to be done, who will do it,
how they will have the expertise and experience to do it,
and that they have the time to do it. And you can orient the biosketches as well
in the same way. In a partnership grant like these, it’s helpful to describe any
experience anybody on the team has with partnership work as well as research
or implementation work depending on which partner team they are from.
The PI should have experienced managing a grant of somewhat the size.
They may have had a smaller grant, but they should have some management
experience. And there should always be someone on the project,
it doesn’t have to be the PI or Co-PI, who’s on for a long enough time to make
sure they always keep the project moving. You don’t want a case where there are
always people on for 5% or 3% of the time. So it’s very easy to drop the ball and
forget, oh, what we were supposed to do, I forgot about this.
It’s nice to have somebody on for more time who can always be responsible for
making sure there’s back-and-forth contact going on, and the data is being
transferred, and that intervention is being implemented as expected. Also,
you should be sure to be objective for the evaluation. If anyone was involved
in the development or distribution of the intervention,
they should be separated from some of the evaluation work being done just to show
that this is an objective evaluation. For resources, you want to describe the
institution resources all the partners are bringing and how they will contribute to
the evaluation being done and building the partnership institutional capacity to
manage a grant. Resources available at the partner institutions to support their part
of the project in the joint letter of agreement, which is signed by the major
partners, sets out their expected roles and responsibilities and is put in
Appendix E. Other partners should provide letters in Appendix E as well.
And then you should note what resources you have to carry out the dissemination
plan, which is placed in Appendix A, and we’ll talk about it in a minute.
Change for fiscal year 2019, Appendix A: Dissemination Plan, will now be considered
by scientific peer reviewers as part of their review of the Significance
section of your Research Narrative. Reviewers will consider the resources you
have available for dissemination as part of their review of the Resources section.
I do want to note some other sections of the application because they are also
looked at by the reviewers. This includes the five appendices in the
Budget and Budget Narrative. Appendix A: Dissemination Plan is
required. Again, when we say required, that means, to be accepted for
peer review, it must be included. Required within this plan is dissemination
throughout the partner education agency. So that includes an agency-wide oral
briefing, as well as a free non-technical written brief available both to the agency
and to the public. Then, you want to consider dissemination
to other education agencies and dissemination to the research community.
This can include presentations at practitioner, policymaker,
and researcher meetings, and conferences, as well as publications and practitioner
policymaker journals, and peer-reviewed journals for academics.
The Institute considers all types of findings from these low-cost evaluation
projects to be useful and expects dissemination to include findings of
beneficial impacts on student outcomes as these will support the wider use of the
intervention, and perhaps, a more in-depth further evaluation,
as well as findings of no impacts or negative impacts on student outcomes.
These are obviously important for decisions regarding the ongoing use
and dissemination of the intervention for the revision, and its implementation,
and the revision of the rationale for its use. It’s important when you write
your application to note that the education agency understands that all
findings are to be released. There can certainly be a right of first
review of no surprises to the agency, but findings should not be considered
withholdable. We want the education agency to learn from the project,
and we want the education agency to be willing to want to learn and to
disseminate what it has learned from the project. A change for fiscal year
2019 is that the Appendix A: Dissemination Plan will now be considered
by the scientific peer reviewers as part of their review of the Significance
section of your Research Narrative. In addition, reviewers will consider the
resources you have available for dissemination as part of the review of
the Resources section of the Project Narrative. Appendix B is required
for resubmissions. Its only use is to address reviews of the previous
application. It’s very important to respond to all previous comments.
On the other hand, if you do feel your application has changed so much from a
previous submission that it’s really not a resubmission but a new application,
you can make that argument and why it should be considered that way here.
Appendix C is optional. In here, you can include materials that relate to
the evaluation and the partnership. So, for example, diagrams of the
partnership management structure, a timeline for the project,
this is very useful, examples of instruments that are being
used for collecting administrative or other secondary data. So, for example,
if the district is going to do a survey of teachers regarding their use of an
intervention, you could provide that survey instrument here.
Please do not include narrative text. Appendix D. These are materials that
relate to the education intervention being evaluated. What does the
intervention look like? Does it have its own assessment materials?
If it’s a curriculum, what are some of the curriculum materials?
If it’s an online program, you can provide some computer screenshots.
If teachers are receiving specific types of professional development,
you can show the documents for that professional development.
So the difference between Appendix C and Appendix D is that Appendix C may contain
information regarding the evaluation you are doing. And Appendix D addresses the
materials for the intervention you are evaluating. Appendix E is required.
Here’s where you put in your letters of agreement. There are two required
letters of agreement. One is the joint letter from the research
institution and the education agency. That is to document their participation
and cooperation in the partnership and set out their roles and responsibilities under
the project. The second letter is from the office in charge of the agency’s data.
And this explains that the project will have access to the data required and in
time to do the analysis. It needs to explain that you will have
access to the data required by the first quarter of the second year in time to do
the analysis. Then, you can also have other letters, separate letters from any
other partners taking part, letters from any consultants on the
project and you can always have a small number of schools taking part.
So it’s helpful to have letters from them as well. Obviously, there’s a trade-off in
getting these letters, the cost of getting these letters ahead of
time versus how convincing it makes them look, your partnership looks,
and the probability that the intervention will occur. You will write up a budget in
a Budget Narrative. The maximum project length is 2 years, and the maximum award
is $250,000. A reminder that the grant funds can only be used for the evaluation;
they cannot be used for implementing the intervention. You don’t need to ask
for $250,000 if the project can be done for less. It’s fine to ask for a smaller
amount of money. You include a detailed budget form in a Budget Narrative.
And the Budget Narrative should link to all the areas of the budget form and
describe how each of the areas are being expended, personnel, evaluation,
etc. So if there’s something going on in the Project Narrative,
it should be described in the Budget Narrative as here is funding
for that work to be done. For fiscal year 2019,
Requests for Applications, (RFA), but for
Low-Cost Evaluation, it includes the following changes from the
fiscal year 2018 competition. The Institute has expanded its definition
of student education outcomes to include employment and earning outcomes when
appropriate. Grant funds may be used to collect primary data for analyses if
the partner agency deems these data and the associated analyses key to its
decision-making regarding the intervention being evaluated. However,
Low-Cost Evaluation projects generally rely primarily on secondary data to obtain
the education outcomes. An additional change for fiscal year 2019
RFA is that the Appendix A: Dissemination Plan will now be considered
by the scientific peer reviewers as part of their review of the Significance
section of your Research Narrative. In addition, reviewers will consider the
resources you have available for dissemination as part of their review
of the Resources section of the Project Narrative. There are also
revisions in the research design section that reflect updates in the What Works
Clearinghouse, or WWC, Standards Handbook. For the full updates,
please see the WWC Standards Handbook version 4.0. As you prepare your
application, it’ll be important to keep aware of the dates, to take advantage of
the information sources, and to understand the review process.
So the important dates are listed in the Request for Applications.
The first important date is for the Letter of Intent. The Letter of Intent is a short
description of who the partners are, what is the intervention,
and how are they proposing to evaluate it. It’s not required. Again,
it’s only requested. And if you miss the deadline,
you can still send the email to the program officer saying,
“we are intending to apply, here’s our idea.” The purpose for the
Letter of Intent is that it allows the program officer to respond to you and to
give you feedback, and to note whether this is the right application for you,
and how you could strengthen your application. In addition,
the Standards and Review Office uses them (these letters of intent),
to get an estimate of how many applications will be coming in and what
topic areas so they can recruit peer reviewers who understand the intervention
they’ll be evaluating. The Letter of Intent is not used in the
review process so the reviewers do not see it. All information is superseded by
the information in your application. So if you change your idea a little or
even a lot, you may even change a partner, that’s not a problem.
It is your application that will be evaluated. There will be an application
package posted that you’ll download and fill in, and the RFA will have your application
deadline. That is a very strict deadline. It’s normally set on a
date at 4:30 p.m. Again, if an application
comes in later, it will be considered late and not
accepted for peer review. For that reason, we ask you to upload your applications
several days in advance for two reasons. One, the server gets very slow on the
application deadline date and you may run out of time if you try to upload it later
in the day. And second, you can use a command check for errors,
and if an error is found, you will then have time,
if you apply ahead of time, to correct the error and resubmit.
There are also possible start dates. You don’t need to always start at the
earliest date. There may be reasons to start later. The information sources
requests for applications will be on our funding website. You may also take a look
at some abstracts of projects we’ve already funded. That’s under our grant
search. And you would then go to the grant search and look under program and search
under the Low-Cost Short-Duration Evaluation program. The application
packages that you download are available on grants.gov. These are the two program
officers for the two grant competitions, and you should feel welcome to email us
with your questions regarding the grant program. Just a little bit on the
peer review process, which is run by the Standards and Review
Office. If your application is successfully submitted,
it goes through two screenings. First, a compliance screening, and then,
as well, it will include a responsiveness screening.
I mentioned before that the five sections of the Research Narrative have
requirements. So they’ll be checking for these requirements to be met.
And once it passes those two screenings, it will be assigned to a review panel.
Within that panel, two to three reviewers will do an initial review of your
application. One will be a methodologist, and one is to have substantive knowledge
of the intervention you want to evaluate. Then the most competitive proposals will
go to full panel. So, many panelists on a full panel may be
generalists to your topic to the intervention being used.
So you want to write for both experts in your intervention as well as
generalists to it. The panel also contains experts in the relevant methodologies and
the evaluation designs being proposed. They then score each section of the
Research Narrative, and they provide an overall score as well.
You will create an account on the IES application notification system,
and then you’ll receive an email that the information has been updated so you can
check the status of your award in the final review statement.
If you are not granted an award, we ask that you consider resubmitting and
talking to your program officer. The majority of applications are not
funded. The majority of applications we fund often are through resubmissions.
So it’s worth looking at the comments, talking with your program officer,
“Is this a good application for resubmission,” and then making such a
resubmission. And then, finally, again, I just want to stress that it’s the
program officer’s job, our job, to discuss your ideas with you and to give
you feedback on them as to whether this is the correct program to apply to,
is there another program to apply to that might be better, and then connect you up
with another program officer. So please do feel free to contact us.

You may also like...

Leave a Reply

Your email address will not be published. Required fields are marked *