Working together with Michigan State University, CoreCollaborative International carried out a program evaluation of the Algorithms & Software for SUpercomputers with emerging aRchitEctures (ASSURE) program. The evaluation focused on the extent to which student content knowledge and intercultural competency had increased as a result of the ASSURE framework and its discipline-based education abroad experience.
Michigan State University’s (MSU) ASSURE site is an International Research for Students (IRES) program, supported by funding from the National Science Foundation. The program is open to all US graduate students, and only a handful are accepted annually. Students engage in research projects and professional development activities that prepare them to tackle the kind of complex problems for which supercomputers may be able to find solutions. In the process, students are exposed to international collaborative projects, and grow to understand how they can be initiated, as well as the primary skills needed to sustain them.
Students come to ASSURE from any one of four computational research areas: Biophysics/Biomedicine, Numerical Methods, Big Data, and Machine Learning. Depending on their research interests, during the nine months of their ASSURE experience students collaborate with researchers in Germany, Japan, or China. Students receive stipends as well as round-trip travel and lodging at their research locations. They also participate in cultural activities while abroad.
As well as engaging students globally, ASSURE also benefits participating faculty. In addition to providing funding for faculty mentor travel, the program also promotes faculty research, expands research applications, and demonstrates to faculty the utility of available research facilities.
As part of its agreement to host ASSURE, MSU built in a program evaluation, and approached CoreCollaborative International (CCI) to manage it. The evaluation sought to understand the extent to which students’ knowledge – both in the realms of supercomputing content and of intercultural competency – had increased as a result of their experiences. Research questions focused on these three areas:
- To what extent does ASSURE increase students’ ability to use advanced computing architectures in their chosen field of research?
- To what extent does it broaden students’ understanding of key questions and approaches for addressing disciplinary questions with supercomputers?
- To what extent does ASSURE improve students’ intercultural competencies?
Evaluation instruments were developed, for both students and faculty mentors, that allowed MSU and CCI to understand both the learning outcomes as well as potential improvements that would better prepare students for successful and impactful international computational and data science collaborations and intercultural collaborations. An internally developed survey on students’ self-reported confidence in their abilities to use advanced computing resources was administered both pre- and post-experience; while a post-experience mentor survey assessed students’ skill development. In addition, the Beliefs, Events, and Values Inventory (BEVI) was administered to students. The BEVI established a baseline on who students were, and how they saw the world, both as they entered the program and then again as they exited it.
Because the BEVI is ideally used as a developmental intervention where results are debriefed with participants, CCI’s evaluation approach was to provide a pre-experience briefing and then combine the post-experience BEVI debrief with a focus group to better understand student experiences and perceptions of any change they saw in themselves. Changes in BEVI scores were combined with analysis of focus group transcripts and internal survey responses to provide a more holistic picture of each cohort’s experience. CCI’s approach, based upon prior research, is that desired student outcomes are influenced not only by instructors and assignments, but also by program structures, variations in location, and the backgrounds students bring to the experience. The very small number of student participants in ASSURE provided an excellent opportunity for stakeholders to understand deeply the lived experience of each of the three cohorts of participants analyzed.
The ASSURE program evaluation process provided a real learning experience for its hosts at MSU. While students gained confidence in terms of their ability to do research, in fact the initial outcomes of the program were not what faculty had hoped for regarding intercultural competence. CCI’s evaluation found that intercultural competence was insufficiently addressed within the program materials, and suggested ways in which it could have been more effectively embedded, leading to a likely positive effect in this area.
Having said that, CCI also found a way to help the ASSURE team address, going forward, the gap it had identified. Calling on its robust professional contacts within the larger Michigan State system, CCI introduced its client to MSU’s Education Abroad team, which had not been involved in the initiative earlier. By bringing this cross-institutional partner into the conversation, CCI was able to raise awareness both of additional education abroad opportunities open to graduate students in computational research, and of the existence of an internal unit that could help faculty behind ASSURE achieve their intercultural competency goals. As CCI wrapped up its evaluation, these two teams were planning to collaboratively develop a curriculum to support future scholars. With CCI’s help, then, the ASSURE team was led to network effectively within its own institution so as to build capacity and sustainable support for future initiatives.
The outcomes evident in Michigan State University’s ASSURE program are typical of CoreCollaborative International’s flexible and wide-ranging approach to its work. Program evaluations seldom yield no surprises. By putting the needs of the client first, and by relying on CCI’s extensive knowledge and professional networks, CCI is able to effect positive change for clients even when initial program outcomes fall below hopes.See how CoreCollaborative International can help your organization achieve its goals by contacting the team today.