Visualising digital footprints to enhance learner engagement in work-integrated learning

Project Team

	Associate Professor Helen Wozniak Academic Lead, Assesment Academy for Medical Education		Dr Christine Devine Senior Lecturer in Medical Education Academy for Medical Education
	Associate Professor Justine Gibson Associate Professor School of Veterinary Science		Dr Asela Olupeliyawa Academic Lead and Senior Lecturer (Assessment) Academy for Medical Education
	Dr Christy Noble Clinical Learning & Assessment Lead Academy for Medical Education		Ms Kym Ward e-Learning Manager Academy for Medical Education
	Dr Anna Kull Principal Research Assistant/ Research Project Officer Academy for Medical Education		Ms Shari Bowker Learning Designer (ePortfolio) Institute for Teaching and Learning Innovation
	Dr Aneesha Bakharia Learning Analytics Manager Institute for Teaching and Learning Innovation		Dr Robert Garrard Data Analyst Institute for Teaching and Learning Innovation
	Mrs Chantal Bailey Senior Data Analyst Faculty of Medicine		Mr Samuel Monk Research Assitant Academy for Medical Education

Student Partners

	Miss Kathryn Bird Medical Student Greater Brisbane Clinical School		Daniel Ochayi Medical Student Rural Clinical School
	Zack Kodiyattu Medical Student Ochsner (United States) Clinical School		Zachary Low Veterinary Student School of Veterinary Science

Reference Group

Prof Anna Ryan Head of Department Medical Education; University of Melbourne, VIC Australia

Prof Sandra Kemp Deputy Dean; University of Wollongong, Graduate School of Medicine, NSW Australia

Dr Mike Tweed Convenor Assessment; Medical Program Otago University, New Zealand

Prof Rikki Goddard-Fuller World leader in workplace-based assessment and visualisation of learner progress in WIL; Director of Christie Education, UK

Nancy Davies Leeds University medical program UK, international awards for their WBA work with technologies in medical education

Prof Jason Coe Ontario Veterinary College, University of Guelph, Ontario, Canada

Luke Gaiter Manager Technology Training, UQ Library

This collaborative project between Medicine and Veterinary Science staff and students will adopt a design-based research approach to guide the phases of the project: exploration, design, implementation, analysis, and reflection.

During Phase 1, the exploration and analysis phase, the project team has explored theories about student learning during Work Integrated Learning (WIL) placements to analyse the contextual factors specific to UQ Medicine and Veterinary Science. This phase included focus groups with staff, international reference group members and students gathering perceptions about WIL and identifying digital datasets for inclusion in the project. During Phase 2, a participatory co-design process with the student partners as described by Treasure-Jones (2018) is being adopted. These initial phases will lead to Phase 3 where the WIL interventions will be implemented. This extended phase enables the project team to trial the dashboard designs and other strategies to support their use, gain feedback, and iteratively improve the designs as students move through each WIL placement period. Phase 4 reflection and design adjustment occurs periodically and is informed by the analysis of the student’s digital footprints and further evaluation data. The design-based research approach enables refinements to the interventions and improved theoretical understanding about WIL and learner digital dashboards. 

Main Activities:

Phase 1: Exploration and Analysis

Reference Group Sessions

The purpose of these sessions was to draw on the international reference group members' experiences with student learning in the clinical setting. Specifically, we focused on their experience with WBAs to guide the exploration and analysis stage of our project, prior to moving to the design stage of the project. Through discussion about how WBAs are adopted across different disciplines and contexts principles for both students and supervisors emerged. Colleagues agreed that longitudinal tracking of student progress and the ability to support student learning as they move from supervisor to supervisor or location to location is an important consideration for this project. In addition the reference group highlighted the importance of taking a whole of program approach to the student journey that aims to support students to utilise information, embrace feedback and plan actions to guide their next steps in achieving enhanced practice (not just aiming for getting across the line).

Student Partners Exploration 

Our four student partners representing the School of Veterinary Sciences (1) and the Medical School (3 - Urban, Rural and Ochsner) commenced working with the project team in September reaching out to their student peers through interviews and focus groups to investigate the lived experiences of students in the clinical setting. They specifically asked students to comment on how they complete their WBAs; specifically what they consider before they start a clinical placement, what they do when in the clinical setting as they complete their various WBAs and what they do as they review their progress. Our student partners shared their findings at the co-design session described below. Themes that emerged related to four main areas: aspects related to overarching assessment system design, what would enable students to maximise the value of completing WBAs, the type of features that a progress dashboard could include, suggestions regarding the way an electronic platform could better support capture of WBAs.

Academic and Clinical Colleagues 

In parallel, members of the project team also approached academic staff (course coordinators or support staff) and clinical leaders at a range of clinical sites to investigate their perceptions about WBAs. A similar focus was undertaken to investigate their perceptions about what they required in relation to WBAs prior to, during and following student clinical placements. This included how best to harness information and what type of information would be relevant to the wide range of clinical supervisors who support students during clinical placements. There were common elements from both Veterinary and Medical colleagues focusing on information about a student’s prior experiences, early identification of at risk students, options for students to self-notify when assistance is needed, real time case logs, strategies to capture supervisor feedback, to name a few. In the case of medicine there was a suggestion to consider how to align any features with practices already in place in the post-graduate medical training sphere, and therefore consider of the wider continuum beyond the primary medical degree in the project design phase.

Phase 2: Design and Construction

Co-Design Session 1

The aim of this first co-design session in October 2022 was to unpack information that we have collected from our stakeholders (students, course coordinators, clinical site leads, etc.) and round off the exploration and analysis phase of this project. 

During the co-design workshop with student partners and project team we:  

• Drew on analysis of focus group/interview data to gather information that will inform the development of reports and dashboard prototypes
• Explored the reporting and dashboard visualisation options for WBA data that have been drafted by the Learning analytics team members
• Investigated other ways that WBA data could support student learning in the workplace
• Planned our next steps for project as we moved into the design and construction phase of the project

Co-Design Session 2

The project team met in March 2023 over lunch with the student partners to continue our design and construction phase of the project and begin to consider opportunities for implementing project prototypes. During the co-design session we:

• Discussed our student partners’ feedback received from their peers detailing suggestions about the design of the whole of cohort dashboard prototypes that visualise WBA completions from the prior student cohort (during 2022). This early prototype was made available to medical students beginning their first full time clinical placements in 2023.
• Updated our design parameters for development of an individual student dashboard to assist students to monitor their own progress while completing WBAs in real time.
• Reviewed the team members work analysing qualitative feedback received from clinical supervisors using three different types of mini-CEX’s during 2022 (history, examination and management). The aim of this work is to automate extraction of higher quality feedback using a set of parameters and provide access to students and staff for planning purposes.  
• Considered a series of mock student scenarios depicting different patterns of student engagement with WBAs. This aimed to inform our understanding about  “at risk” patterns of behaviour or where opportunities may arise to support students to increase their engagement in the clinical setting.

Phase 2: Design and Construction/ Phase 3: Implementaion and Analysis

Co-Design Session 3

In December 2023, the project team and student partners convened for the third co-design session to discuss the project progress and work through the upcoming phases. During this session, the project team and student partners accomplished the following:

• Reviewed project stages and timelines. While real-time student facing dashboards were developed to the prototype stage, further development work was unable to be progressed.
• Evaluated the first two (out of three) online modules to support students transitioning to learning in the workplace- Plan, prepare, proceed. The first two modules (screenshots below) draw on the investigations completed by the project team and student partners to highlight strategies that can be used to make the most of clinical learning opportunities.
• Collaboratively planned the design of the 3rd module which will focus on using workplace-based assessment opportunities to gain feedback and plan future actions and clinical skills development
• Formulated strategies for project evaluation and outlined final project outcomes

OTTAWA conference 2024

Title: Exploring student engagement in the workplace: insights from analysis of large WBA datasets

Abstract:

Background

Workplace-based assessments (WBAs) provide evidence of medical students’ clinical and non- clinical skills development in clinical settings. Learning analytics, extensively used to explore student engagement in classroom-based settings, are now emerging for WBA-related insights. Applying learning analytics in WBAs empowers educators and institutions with valuable insights to improve the learning experience through data-informed decisions.

Summary of work

In 2022, we introduced a digital platform to the Workplace Learning Portfolio (WLP), a longitudinal course where students develop their skills through regular completion of WBAs in the final two years of the 4-Year medical program. While this learning is student-led, submission targets and deadlines encourage ongoing engagement. Mid-year and end-of-year reviews assess these targets, guiding progression decisions and supporting students review their learning opportunities. We conducted an analysis of the Year 3 data set (23920 submissions, 320 students) to identify patterns of engagement and associated factors.

Results

Students with higher cumulative WBA completions show early and consistent engagement. Tracking average WBA submissions over the year revealed that students with lower prior semester GPA (<5) were slower to accumulate submissions and meet course requirements, while students with a higher GPA submitted more tasks, more frequently, and met course requirements earlier. This gap in completed tasks is visible by Week 7 of Semester 1. Placement type also matters, with students completing more WBAs during generalist placements.

Discussion and conclusion

Prior GPA and lower WBA completion may inform identification of at-risk students who require more support to transition to workplace-based learning, where deliberate practice should be promoted over just in time completion. Resources to support students as they move between diverse clinical placements may assist to manage expectations around WBA opportunities.

Take-home messages

Data-driven approaches to understand how and when students meet WBA requirements inform a more personalised approach to guiding learning in the workplace.

OTTAWA conference 2024

Title: A co-design approach to enhance workplace-based assessment practices

Abstract:

Background

Learning in the workplace is complex (1). Learners must adapt swiftly to the challenging environment, maximize learning opportunities, and seek actionable feedback from various health professionals as they complete their work-based assessments (WBAs).

Summary of work

A collaborative co-design project at The University of Queensland involving diverse fields (Medicine, Veterinary Science, Learning Analytics, eLearning Design) is reimagining WBAs for workplace learning, harnessing expertise from key stakeholders (assessment leaders, clinical staff, and student partners representing urban, rural, and international cohorts). Co- design, from the human-centred design technology space, involves stakeholders having a ‘seat at the table’ so their collective creativity actively guides design decisions (2).

Results

Thematic analysis of co-design session recordings, transcripts and co-created artefacts revealed two outcomes: firstly design parameters for creating whole-of-cohort and individual digital dashboards for learners to monitor feedback and longitudinal WBA progress; and secondly a framework for optimising workplace learning considering socio-cultural and contextual factors (3). Strategies for learners and supervisors were identified to enhance WBA experiences prior to and during workplace learning, including options for initiating WBAs, gaining actionable feedback, and guiding learner-centred progress reviews.

Discussion

The co-design process integrated varied stakeholder perspectives to identify WBA engagement factors. This collaborative approach foregrounds the role of the learner and supervisor whist being cognisant of the university and workplace culture. It also illuminates how to embed technological options to support learning and ongoing monitoring of progress.

Conclusion

Involving stakeholders in the co-design process provides an essential lens to ensure that the rhythm and flow of the workplace culture informs strategies for optimizing the learning value of WBAs and utilisation of information from digital dashboards.

Take-home messages

Understanding workplace learning and assessment requires ongoing inclusion of all stakeholder viewpoints from design to implementation in the assessment lifecycle.

T&L Week 2023

Title: Digital dashboards that connect learners and challenge our thinking about assessment

The project team delivered an hour-long session in a seminar format during the UQ Teaching and Learning week at UQ. The presentation included four disciplinary case studies and open discussion focused on key questions for debate and consideration. The case studies illustrated novel ways that digitally collected assessment information has led to the development and use of dashboards to facilitate student synthesis of feedback information from multiple sources and challenge learning communities to think differently about assessment and feedback. A recording of the session is available here.

ANZAHPE conference 2023

Title: Co-designing digital dashboards to enhance student engagement in work-integrated learning (WIL)

Abstract: 

Introduction/Background: Work-integrated learning (WIL) provides the essential authentic context for students to apply theoretical knowledge gained from their classroom activities to real-world activities and forms the foundations of all professional programs. However, students often struggle with transition into WIL settings. A teaching innovation grant between the Schools of Medicine and Veterinary Science and the central teaching and learning centre is adopting a participatory co-design approach (Sarmiento, 2022; Treasure-Jones et al, 2018) with student partners to develop digital dashboards depicting engagement in workplace-based assessments (WBAs). This aims to enable students, staff and clinical supervisors to monitor longitudinal progress during WIL placements.

Aim/Objectives: The aim of this presentation is to showcase key outcomes from this co-design process. 

Discussion: The co-design process started with an exploration and analysis phase to evaluate the available digital data sets, review the literature and seek advice about dashboard requirements from both local stakeholders (students and staff) and our international reference group members. This background created a solid base for the first co-design workshop between the project team and student partners. Collaboratively we have developed an initial dashboard prototype that includes real-time visualisations of learner engagement in WBAs. We will present insights from our work completed to date, focusing on strategies that can enable students and supervisors to monitor learning longitudinally, as well as supporting students to set realistic goals and actions in response to feedback they receive from their clinical supervisors. This cross-discipline inclusive approach acknowledges the wider context of student learning in the WIL setting and provides insights for the health professions education community to navigate opportunities for utilising WIL digital datasets.

Issues/Questions for exploration OR Ideas for further discussion:

• Using data visualisations to turn the tide and support proactive engagement in WIL.
• Co-design as an approach to navigate the turbulent WIL waters.

ANZAHPE conference 2023

Title: Navigating ebbs & flows of digital dashboards used in medical student assessments during clinical placements 

Abstract: 

Introduction/Background: Digital dashboards provide information about student engagement in learning activities, assessment performance and comparative cohort metrics1. They are often focused in university examination contexts, with limited development work being undertaken in health professions education and in clinical settings.

Over recent years, medical schools at University of Queensland and Western Sydney University have embarked on workplace-based assessments of their medical students using the same IT provider (MKM MyprogressTM). The general philosophy of programmatic assessment as part of an assessment system is used by both, but bespoke WBA assessments are used at each.  How each institution is adopting learning analytics is different.

Aim/Objectives: We aim to present the learnings from each medical school. We will demonstrate some analytic dashboards and show how dynamic review of this information can enable understand about student learning in the clinical setting.

Discussion: We will propose why digital dashboards need to link to a whole of assessment system and consider how they can be used for coaching students, how they link individual tasks to longer term reports and progress, and how they can be used for remediation. Both medical schools focus on developing feedback literacy when implementing digital dashboards to support the integration of information e.g. learner understanding about when and how to take action. We propose that this whole of system approach could eventually replace high stakes single event clinical examinations.

Issues/Questions for exploration OR Ideas for further discussion

• What do you believe about WBAs for medical students?
• How much can you find about student engagement in clinical placements by using digital dashboards?
• Do you have views about future directions for assessment in medical schools? For example, the use of standardised OSCEs (or similar) compared to the in-situ rating of performance by many for WBAs?

HEA@UQ Forum | T&L Week 2022

Title: Co-designing digital dashboards to enhance student learning in the workplace

Abstract: Background: Work-integrated learning (WIL) provides the essential authentic context for students to apply theoretical knowledge gained from their classroom activities to real-world activities and forms the foundations of all professional programs (K2). However, students often struggle with transition into WIL settings. Thus, universities need to generate strategies to enhance the development of students’ professional capabilities in WIL settings by enabling students, university staff and clinical supervisors to monitor their progress longitudinally (K3), provide effective feedback (A3) and identify students who are falling behind with workplace-based assessments (WBAs) (A4).

The aim of this presentation is to illustrate how our Teaching Innovation Grant is engaging in a co- design process between staff and students from the Schools of Medicine and Veterinary Science and ITaLI to develop digital dashboards and other learner supports for students WBAs in the clinical setting (A4, K4, V1).

Adopting an educational design research framework, the project is initially completing an exploration and analysis phase to evaluate the available digital data sets, review the literature and seek advice from our international reference group members from across the globe (A5, V3). Drawing from an analysis of focus group and interview data with our stakeholders, we are aiming to develop dashboard and reporting prototypes that include real-time visualisation of learner engagement in WIL activities (including WBAs) (K4). This will enable students and supervisors to monitor learning longitudinally, as well as supporting students to set realistic goals and actions in response to feedback they receive from their clinical supervisors (V2). We will present insights from our work completed to date, including outcomes from our initial participatory co-design workshop with student partners and the project team. This cross discipline inclusive approach acknowledges the wider context of student learning in the professional setting and will guide the development of insights for the wider university community (V4).

ANZAHPE conference 2022

Title: A cross-disciplinary partnership between staff and students to support learner growth in diverse clinical settings

Abstract: Background: Achieving optimal outcomes from experiences in clinical settings is challenging due the opportunistic and unplanned nature of this learning environment. Access to real-time information enabling students to track their progress over time is also limited by the dispersed nature of the clinical setting, which in turn also makes it difficult for university staff and clinical supervisors to monitor and identify students who are falling behind with workplace-based assessments (WBAs).

A partnership project between the schools of medicine and veterinary science, and university learning analytics team is aiming to better understand the nature of student engagement in the clinical setting by utilising the available digital learner footprints captured from WBAs. The project team along with student partners and an international reference group is undertaking a co-design process to develop digital dashboards to reveal learner profiles of engagement with WBAs.

The purpose of session is to share initial dashboard designs depicting WBA activity and discuss the patterns of engagement demonstrated by medicine and veterinary science students in the clinical setting.

Issues/Questions for exploration and discussion

• What information needs will best support the monitoring of WBA achievements? Are there different requirements for students, academic staff and clinical supervisors?

• Do the patterns of engagement revealed by the project team align with the experiences of the participants?

• What strategies have participants adopted to foster clinical supervisor engagement in utilising a digital platform? How can a digital platform be maximised to facilitate quality feedback and the feedback interaction/ partnership between supervisor and student?

• What educational strategies could be adopted to foster student feedback literacy and development of their clinical capabilities?

• Could a digital dashboard create efficiencies in the identification and support for at-risk student in the clinical setting?