Common Program Evaluation Needs in STEM
A skier's guide to the most common types of educational outreach program evaluation needs in STEM.
Here’s a short take on the most common types of educational outreach along with general program evaluation advice for each. Thinking about it in this way reminds me of the difficulty signs for downhill skiing going from green circle easiest on the bunny hill to the double black diamond extremely difficult near vertical bowl. That works well to think about program evaluation across these common categories of education and outreach in STEM.
The four most common types of educational activities in STEM are related to:
1. Teaching -- things you do in your courses;
2. Research -- research experiences, e.g., REU or fieldwork;
3. Relationships – mentoring, recruitment/retention of underrepresented groups; and,
4. Public & K-12 Outreach – public lectures, museum partnerships, working with a local teacher or school, etc.
TEACHING: Marked with that tell-tale green circle easiest trails sign, on the bunny hill of program evaluation and assessment, we have all things that happen in the courses you teach. (Now, that is limited to undergraduate and graduate education, we will talk about K-12 in a minute.) This is the easiest and most straightforward of the most common program evaluation and assessment needs. These range from a single assignment to a module to a full course. Courses always have some kind of evaluation of teaching built in and that quantitative evaluation is golden for your program evaluation needs. What makes the ‘things that related to your teaching’ the easiest is that everything is under your control. You have built in access to the audience, who is already invested in and inclined to provide feedback. You can use a variety of things to evidence your teaching, not only the evaluations, but the assignments and your observations (as long as they are structured and well-documented).
RESEARCH EXPERIENCES: On the intermediate slopes with the more difficult blue square sign, we have slightly more difficult program evaluation and assessment for research experiences, like REU. (These can be much easier if your plan comes through a funded REU or RET program in your department or college. In those cases, someone is already doing the program evaluation for you). For most research experience programs, a well-designed survey at the end of the program to capture feedback can work nicely.
RELATIONSHIPS: Off the big chair lift, farther up the mountain with the single black diamond most difficult trails, we get into more difficult terrain in terms of program evaluation and assessment. Relationships are tricky. We need to ask people what they think and how they feel and … it is tricky, or can be. Mentoring programs and all things relationship-based not only involve asking people those questions, but we have to ask more than one person and then match the data. For a mentoring program, it is hard to imagine not needing data from both the mentees and the mentors and then matching that data up for analysis. A professionally designed program evaluation will likely be needed for relationship-based activities.
PUBLIC OUTREACH: This brings us to the bowl: that super steep run through unpacked fresh newly fallen snow, and the double black diamond extremely difficult trails sign. Public outreach is more difficult by definition because the population is not within your control, it may not even be on your campus. But it is a mixed bag. Imagine a program to impact public attitudes towards science through a public lecture series. A very easy (green circle easy) and doable assessment is to count the number of people who participated. But, the extreme difficulty would come from trying finding out how those people’s attitudes changed. Build your public outreach activities to be assessed by simple counts and participation and stay clear of the extremely difficult ...runs.
K-12 EDUCATIONAL OUTREACH: A note on K-12 educational activities -- If you are doing anything for K-12 students, particularly for use in K-12 classrooms or labs, the best approach is to base the evaluation on the observations of your K-12 teacher partner. Without a full IRB and a professional evaluator, it may not be appropriate, feasible or workable to do any evaluation of children -- they are a protected class in human-subjects research. The most appropriate sign for this might be: "Avalanche Danger: Trail Closed."
The bottom line for what your proposal will need is that the program evaluation and assessment is appropriate for your specific plans. Depending on what you want to do with whom, the specific tools for program evaluation and assessment vary.
The mountain is yours. Yours to decide what path -- what run - is best for your program. Some of you may well be on the well-packed moguls, others may be off laying the first tracks where no others have been. A professional evaluator can help you do this efficiently and strategically. I would love to help you with that. As soon as you have the basic program ideas, that’s the best time to reach out and talk about your program evaluation and assessment needs.
This public pavillion was built and provided by researchers from Notre Dame in one of the local county parks, set up as an observation area for bald eagles nested nearby.
public outreach in science & STEM
Bald Eagle at St. Patrick's Park
ScienceAlive! (2020)
