For a variety of reasons, annual experiences surveys are frequently employed for program evaluation in graduate training programs to the great benefit of the programs who use that rich data to translate the feedback received into program improvements. While response rates for traditional surveys continue to be a challenge, particularly since the pandemic, a longer survey of experiences remains one of the best options for collecting feedback from graduate students. Here is a description of my Annual Experiences Survey successfully employed as part of a funded Graduate Education in Areas of National Need (GAANN) doctoral program in chemical engineering.
A GRADUATE STUDENT SURVEY:
RESPONSE SCALES, QUESTIONS, DEFINITIONS, & TOOL MODIFICATION
This survey was developed over the spring of 2022, piloting the tool with graduate students from another department. Distributed to students May 6 to June 1, 2022; June 1-30, 2023; and June 1-30, 2024.
Survey instrument focusses on mentoring (needs vs. received, quality, positive and negative experiences), inclusion and belonging (from the lab to campus), changes to self-confidence and self-efficacy (both concepts defined in the tool itself), and career and professional interests and intentions.
Background/Context: Respondents are asked where they are in the program and what they are doing (e.g., year in program, fellowships, stage of research, pre/post candidacy, working title of thesis, and primary research area). They are also asked about their institutional locations including academic department, labs, sub-discipline, and primary research methods, e.g., theory, experimental and/or data science.
Mentoring Experiences: Respondents are asked about their needs, mentoring received, the quality of mentoring they experience, and overall evaluation of the mentoring received. Students are also asked specifically about negative mentoring experiences and offered follow-up interviews.
Belonging & Inclusion: Respondents are asked about belonging and inclusion using the same question framing as the institution which is an agree/disagree statement “I feel a sense of belonging …”. However, we ask the question assuming that it might vary with closer attachments to the research group and lab than on campus generally, something that is quite common in graduate programs. Respondents are asked to rate their agreement/disagreement on a 5 pt Likert-type in the research group, in the lab, in the department, in the college and on campus. Students are also asked about negative experiences related to inclusion and belonging and offered follow-up interviews.
Self-Confidence & Self-Efficacy: Questions ask for self-reported changes in self-confidence and self-efficacy (5 pt increase/decrease scale from “increased a lot” to “decreased a lot”) where each concept was defined and provided to the respondent. This provided an additional benefit of offering respondent graduate students and their advisors, mentors, and faculty operational definitions self-confidence and self-efficacy.
Methodological Details:
Response Scales: The response scales selected allow for analysis including using a mean or weighted average to compare over time. This is the benefit of a Likert-type scale. The following scales were employed in the tool:
Rating (5 point): 5=Very Good; 4=Good; 3=Fair; 2=Poor; 1=Very Poor
Quantity (5 point): 5=a great deal; 4=moderate amount; 3=occasionally; 2=rarely; 1=never
Frequency (5 point): 5=a lot; 4=frequent; 3=occasional; 2=infrequent; 1=rarely/never
Agreement (5 point): 2=strongly agree; 1=somewhat agree; 0=neutral; -1=somewhat disagree; -2=strongly disagree
Increase/Decrease (5 point): 2=increased a lot; 1=increased a little; 0=stayed the same; -1 decreased a little; -2=decreased a lot
Likelihood (6 pt): 3=extremely likely; 2=very likely; 1=likely; -1=unlikely; -2=very unlikely; -3=extremely unlikely
Types of Mentoring & Mentorship:
Questions cover experiences on mentoring needs, mentoring received, the quality of mentoring received, and negative mentoring experiences across four types of mentoring:
Mentorship related to academic support (courses, coursework and matriculating through program)
Mentorship related to research support (thesis/dissertation, lab and technical skills, data collection & analysis)
Mentorship related to psychosocial or social support (identity, stress, time-management, coping, family)
Mentorship related to professional/career development (career coaching/planning, network building)
Self-Confidence & Self-Efficacy:
Questions ask self-reported changes in self-confidence and self-efficacy (5 pt increase/decrease scale) where the following definitions were provided in the survey tool:
Self-confidence is defined as one’s perceptions of their skills and abilities. High self-confidence means you know your strengths and weaknesses well, that you have a positive view of yourself, you set realistic expectations and goals, communicate assertively, and can handle criticism. Low self-confidence means that you may feel full of self-doubt, be passive or submissive, or be especially sensitive to criticism. Feeling confidence in yourself might depend on the situation and can vary through the course of your graduate research training program.
Self-efficacy here refers to your opinion about what you can and cannot accomplish. Students with high self-efficacy tend to choose complex and challenging tasks while students with lower self-efficacy tend to avoid them. Four areas of your graduate research training program are likely to impact your sense of self-efficacy: (1) mastery of experiences (the more you successfully perform a task the more your sense of self-efficacy strengthens; but, failing to deal with a task or challenge may undermine or even weaken self-efficacy); (2) social modeling (seeing others like you performing tasks can strengthen your self-efficacy); (3) social persuasion (compliments and encouraging statements from others can strengthen self-efficacy); and, (4) psychological responses (one’s moods, emotions, stress can affect how we feel about our ability to succeed, e.g., if nervous before an important speaking event, you may not speak as well, which could affect your self-efficacy in the future).