Assessing Performance and Engagement on a Computer-Based Education Platform for Pharmacy Practice

A computer-based education platform was developed using a theory-based approach to help Canadian pharmacy professionals adopt their full scope of practice. Data from the platform were used to identify factors that impacted user performance and engagement. A de-identified dataset included response data for 21 unique modules, including quiz responses and self-reflection questions. Outcome measures included user performance (mean quiz score) and engagement (completion rate for attempted modules). Analysis of variance (ANOVA), multivariate regression modelling, and machine learning cluster analysis were used to analyze the data. Of the 5290 users, 68% were pharmacists, 11% were technicians, 13% were pharmacy students, and 8% were pharmacy technician students. Four clusters were identified separately for pharmacists and technicians. Clusters with the higher performance and engagement tended to have more users practicing in community pharmacies while the lower performing clusters tended have more internationally trained users. In the regression modelling, pharmacists performed better than technicians and students while students were more engaged (p < 0.0001). Further, internationally trained pharmacists had slightly lower scores but similar engagement compared to domestically trained pharmacists (p < 0.0001). Users demonstrated higher performance on modules related to scope of practice than on clinical topics, and were most engaged with topics directly impacting daily practice such as influenza vaccinations and new and emerging subjects such as cannabis. The cluster analysis suggests that performance and engagement with a computer-based educational platform in pharmacy may be more related to place of practice than to personal demographic factors such as age or gender.


Introduction
The scope of pharmacy practice continues to expand worldwide, especially in areas such as the administration of injections, smoking cessation consulting, medication reviews, and pharmacist prescribing [1]. Research has shown that pharmacists often consider an expanding scope of practice to be a "legitimization" of prior practices and are more likely to offer a new service if they were already doing it in a less formal manner, although there can be a tension between a more professional role and the day-to-day need for technical efficiency [2][3][4][5]. The challenge, however, is that new services associated with an expanding scope can have low uptake, may be delivered in ways that do not align with policy, or may be preferentially offered to less complicated patients [6][7][8][9]. While current methods of continuing professional development commonly used in Canada, such as conference attendance and home study units, are still deemed valuable, they may be insufficient to facilitate widespread practice change [10]. Very little research has been done to date on the use of computer-based education for helping pharmacists close these gaps. Generally, mobile-and computer-based digital education can improve knowledge and skills in ways that are similar to traditional education [11][12][13]. Advantages of computer-based learning are that it can be more interactive [14], that it allows users to ability to apply new knowledge to diverse cases [15], and that it can be accessible to a wider range of healthcare providers such as those in rural practice [16]. That said, there are also significant concerns that computer-based education requires more self-discipline or motivation, which can lead to lower engagement, overall [17].
Pharmacy5in5 was launched in January 2018. It is a computer-based learning platform that aims to help Canadian pharmacy professionals build their knowledge and skills related to the expanding scopes of practice. The development of the platform has been described elsewhere [18]. Briefly, it is an online education platform that can be used on a computer or mobile device. It has a theory-based design that uses case-based quizzes and multimedia resources to apply knowledge to real-life scenarios, while also allowing users to reflect on their past behavior and to compare themselves to their peers. In developing the platform, one of the challenges was to ensure that a variety of users could interact with the content and a diverse audience would find it engaging. For example, pharmacists, students, and technicians may all experience the platform differently, as could those working in community pharmacy or primary care, those who have more or less practice experience, or those who trained domestically or internationally. Thus, the objective of this paper was to identify factors that impact user performance and engagement on a computer-based education platform for pharmacy professionals in Canada.

Materials and Methods
All users of the platform provide consent to the secondary use of their de-identified data for research purposes. The study was conducted in accordance with the Declaration of Helsinki, and the protocol was approved by a University of Waterloo ethics committee (ORE#22642). This paper reports on data from the platform launch in January 2018 to November 2019.
Population: The computer-based education platform is freely available to all pharmacy professionals in Canada. On registering, users provide information about their general demographics and work environment and consent to the secondary use of de-identified data for research and development. For this analysis, data were included for all users who self-identified as pharmacists, pharmacy technicians, pharmacy students, or pharmacy technician students in Canada. Pharmacists and technicians who indicated that they were not registered with a pharmacy regulatory body in Canada (e.g., unlicensed pharmacist) were excluded. Users involved in the development of the module were also excluded.
Dataset: A de-identified dataset was downloaded from the platform and cleaned to remove duplicate data and test accounts. For each user, the dataset included age, gender, years in practice, user type (pharmacist, technician, pharmacy student, technician student), geographic region, main practice location (primary care, hospital, independent community, large chain community, small chain/banner community, and university/academic), and place of training (domestic, international, both). The dataset also included user response data for all quizzes attempted and the date a module was started and completed. Finally, the dataset included reflection question responses, where users answered questions about their past behaviors (e.g., "Over the past 3 months, have you contacted a prescriber with a concern about the safety of an opioid prescription?" or "Have you ever adapted the dose of a medication (increased or decreased the dose?").
Outcomes: The first outcome measure was performance, which was measured by the overall quiz score mean, with each quiz having a raw score out of five. The second outcome measure was engagement, which was measured by the overall number of quizzes completed. The third outcome measure was persistence, which was measured by the overall proportion of quizzes completed in a module attempted (out of a possible seven quizzes per module) and the proportion of modules attempted (out of a possible 21 modules). Persistence was calculated as: Persistence = (overall proportion of quizzes completed + proportion of modules attempted) * 50, where the multiplication factor of 50 is meant to provide an easy to understand scale from 0 to 100.
Analysis: The R software package was used for all data analysis (R Foundation for Statistical Computing, Version 3.6.1). In this paper, the focus was on the data related to the first attempt of any quiz. The analysis was divided into two main areas: (1) Finding natural clusters of performance and engagement; and (2) understanding the relationship between performance, engagement, and user demographics.
Cluster Analysis: To identify clusters, a "cluster analysis" was performed, which is a machine learning technique that creates "unsupervised clusters". The goal of the cluster analysis was to understand the underlying structure in the engagement and performance metrics. Pharmacists and pharmacy technicians were analyzed separately. Four metrics were used to cluster the data: Overall quizzes completed; overall mean quiz score; overall proportion of quizzes completed; and number of modules attempted (modules where at least one quiz was completed). The Hopkins statistic [19] was initially used to identify if the data were clusterable, and the partitioning around medoids (PAM) was used to create clusters [20,21]. The Calinski-Harabasz criterion was used to identify the optimal number of clusters [22]. Bootstrapping by re-sampling was used to evaluate the stability of the clusters, which relied on the Jaccard co-efficient (a similarity measure between sets) to determine how well the clusters behaved [23,24]. Jaccard coefficient values greater than 0.5 indicate a stable solution.
Relationships between demographics and performance and engagement: Multivariate regression modelling was used to determine the relationship between the outcome measures (performance, engagement, and persistence), demographics, and self-reported past behavior. To identify specific differences within variables, analysis of variance (ANOVA) models were used followed by Tukey's honest significance differences test to identify specific differences. It was hypothesized that both performance and engagement would be higher for pharmacists compared to other user types (students and technicians), for domestically trained pharmacists (compared to internationally trained pharmacists), and for users with >10 years in practice (compared to users with <10 years in practice). It was also hypothesized that users with poorer performance would have less engagement with the platform. Finally, it was hypothesized that users who self-reported they had performed a behavior in the past three months would be more likely to complete the associated quiz and would have a higher performance.

Results
The dataset contained data for 5290 users, which included 3579 (68%) pharmacists, 595 (11%) technicians, 711 (13%) pharmacy students, and 405 (8%) pharmacy technician students (Table 1). Of 3739 users (71% of total) were female, the average age was 41 years, and users had practiced for a median of eight years. Of the pharmacists included, 2434 (68%) received their entry-to-practice training in Canada. Of the 861 (24%) pharmacists who received their initial training outside Canada, 284 (32%) had also received training in Canada. The dataset included users from across Canada, but most were from Ontario (4288, 81%), followed by Alberta (451, 9%), and British Columbia (125, 2%). The Hopkins statistic (0.06) indicated that the pharmacist metrics data were clusterable. Using the Calinski-Harabasz criterion, four clusters were identified ( Figure 1). The Jaccard coefficient values corresponding to each cluster (0.98, 0.93, 0.87, 0.93) were greater than 0.5, indicating that the four-cluster solution was stable. The pharmacist clusters were defined as follows using the three outcome measures metrics of performance, engagement, and persistence (more information in Appendix A, Figures A1-A3 In terms of demographics, the clusters had similar age, gender, and education profiles ( Table 2). Cluster two had the highest proportion of internationally trained pharmacists while cluster four had the highest proportion of community pharmacists.

Technician Cluster Analysis
For technician data, the Hopkins statistic (0.12) indicated that the technician metrics data were clusterable. Four clusters were identified using the Calinski-Harabasz criterion ( Figure 2). The Jaccard coefficient values were also greater than 0.5 (0.86, 0.72, 0.91, 0.87), indicating that the four cluster solution was stable ( Figure 2). The technician clusters were defined as follows (more information in Appendix A, Figures A4-A6): • Cluster 1: Low performance, low engagement, low persistence; • Cluster 2: High performance, high engagement, high persistence; • Cluster 3: High performance, low engagement, low persistence; • Cluster 4: High performance, low engagement, high persistence.
Technician clusters had similar age, gender, and education profiles (Table 3). Cluster one had the lowest proportion of Canadian trained technicians while cluster two had the highest proportion of community-based technicians. Technician clusters had similar age, gender, and education profiles (Table 3). Cluster one had the lowest proportion of Canadian trained technicians while cluster two had the highest proportion of community-based technicians.

Relationships between Demographics, Performance, and Engagement
Based on the regression models, pharmacists performed better than all other user categories, while pharmacy students had the highest level of engagement (Table 4). According to the ANOVA, there were statistically significant differences between user categories for performance (F (3, 98,227) = 117.8, p < 0.001), and engagement (F (3, 98,227) = 623.4, p < 0.001). The differences in performance were small, while the differences in engagement were more pronounced (Appendix A, Figure A7). Users practicing in primary care performed better than other users, while engagement was highest for users working in independent practice. According to the ANOVA, there were statistically significant differences between practice locations for performance (F (6, 98,224) = 17.58, p < 0.001) and engagement (F (6, 98,224) = 372.2, p < 0.001). As above, the differences in performance were small, while the differences in engagement were larger (Appendix A, Figure A7).
Users who obtained their entry to practice training in Canada performed better than other users, while engagement was similar whether users trained inside or outside Canada. According to the ANOVA, there were statistically significant differences between training locations for performance (F (2, 98,228) = 98.21, p < 0.001) and engagement (F (2, 98,228) = 231.7, p < 0.001). As with the other comparisons, the differences in performance were small, while the differences in engagement were larger (Appendix A, Figure A8).

Relationship between Topic Type, Performance, and Engagement
The top three modules for user performance were all related to scope of practice ( Figure 3). The bottom three modules for performance were all clinical topics. For engagement, the top three modules with the highest completion rate were related to scope of practice, while the modules with the lowest completion rate related to specialty topics (Table 5, Appendix A, Table A1). After adjusting for demographic factors, both user performance and engagement were higher for users who completed more quizzes in 18 of the 21 modules, with the exception of the modules related to Ramadan, cannabis, and non-sterile compounding.

Relationship between Self-Reported Past Behavior and Performance
Overall, users self-reported they had previously completed a behavior 36% of the time, though the number varied according to topic. For example, in the cannabis module, 21% of users responded that they had performed the target behaviors before (e.g., asking a patient about cannabis use, assessing drug therapy for interactions with cannabis medications). By comparison, 44% of users in the renewals module responded they had performed the target behaviors before (e.g., renewing a short-or long-term medication, renewing a medication for longer than a month, renewing a specialist's prescription). The regression models did not identify any significant differences in performance for users who self-reported that they had previously performed a behavior compared to those who had not, which is also evident from the differences in mean scores (Table 6).

Discussion
Cluster analysis is a powerful tool for evaluating interventions based on performance and engagement, as it pushes the analysis beyond the limits of traditional statistical methods. In this analysis of a national, computer-based education program, the cluster analysis was useful for identifying clear clusters of users based on their performance and engagement, but it also identified that demographic factors were not overly predictive of who was in each cluster. In this way, the cluster analysis segmented users based on how they interacted with the platform rather than who they were. In contrast, traditional statistics identified several differences in the performance and engagement of different demographic groups, however these differences were generally small. For example, pharmacists who had higher overall quiz scores and who completed more modules were slightly more likely to work in community pharmacies compared to other settings. This likely reflects that the content was developed primarily for community pharmacists. Similarly, pharmacists and technicians who had lower scores or who completed fewer modules were more likely to have trained outside Canada, though these differences were so small that they are unlikely to have a significant impact on program evaluation. This latter finding is important because thirty percent of Canadian pharmacists are trained outside Canada [25]. Thus, there is great interest in identifying if place of training impacts performance and engagement. The results showed that while those who were trained internationally had slightly lower overall scores, they had a similar engagement to those trained in Canada. However, the differences were small and there is a risk that the findings could be misinterpreted to devalue the knowledge and experience of internationally trained pharmacists. Thus, while all of the traditional comparisons are interesting and not unexpected, their small size differences limit their usefulness. In contrast, the cluster analysis clearly identified the clusters of users who struggled with either performance or engagement.
The contrast between performance and engagement could potentially be explained by looking closer at motivation. For example, social motivation theory suggests behavior change depends on both intrinsic and extrinsic motivation [26,27]. For computer-based education, users can be engaged to participate through extrinsic motivators such as rewards and points systems, but eventually users must develop intrinsic motivations to continue to use the system or to change the target behavior. In this study, the most engaged users were pharmacy students, suggesting that they may already have more extrinsic motivations through assessments in the classroom or experiential education sites. Thus, pharmacists in practice may benefit from additional features that extrinsically motivate them to engage with the platform to bring their engagement more in line with pharmacy students. Further, users working in independent pharmacies were more engaged than users in other practice settings such as chain stores, suggesting that pharmacists outside chains may have less access to continuing education (e.g., annual conferences). Thus, independent pharmacists may have more intrinsic motivation to complete the modules whereas chain store employees may need more extrinsic motivators to complete a full module.
One notable finding was that a minority of users had typically offered a service or performed an activity prior to completing the module, and that past experience providing a service did not necessarily predict a higher quiz score. Numerous studies have shown that pharmacists can be trained to effectively adopt new scopes of practice [28]. Yet, as Rosenthal, Austin, and Tsuyuki highlighted in 2010, pharmacists are held back by a fear of responsibility, paralysis when facing ambiguity, risk aversion, and the need for approval [29]. Thus, it is unsurprising that the majority of users who engaged with a module had little to no prior experience with the topic in daily practice. Further, demographic factors were not necessarily predictive of whether someone had engaged in an activity in the past.
While these results are based on a large, national database, there are some limitations. The dataset is limited to quiz and self-report data from single time points. Additional information would be needed to assess impact on sustained knowledge or behavior change. Further, there is likely a self-selection bias where users who are more interested in or comfortable with online education signed up to use the platform. That said, all users who had completed at least one quiz were included to ensure a broad sample of pharmacy professionals for the analysis. Other possible factors that are not captured in the database but that could potentially impact performance and engagement include past experience with online games, educational platforms, and other online experience. More research needs to be done to determine the impacts of the platform on knowledge and behavior change and to explore the impact of technology experience on engagement in computer-based education.

Conclusions
Cluster analysis was a useful strategy for analyzing engagement and performance independent of demographics. While it is important to consider demographics like gender and country of training, these variables also lend themselves to bias and cannot be changed. In this study, pharmacists had higher quiz scores than technicians and students, but students were more engaged with the platform. Community pharmacists performed slightly better than hospital pharmacists, but were much more engaged, while internationally trained pharmacists and technicians performed slightly less well, but were as engaged with the platform as others. That said, the cluster analysis using performance and engagement allowed movement beyond demographics to look at user performance regardless of who the users were. It is a promising strategy for evaluating computer-based education interventions on learning.
Funding: This research received no external funding. The development of Pharmacy5in5 was funded, in part, by the Ontario College of Pharmacists.

Conflicts of Interest:
The authors declare no conflict of interest.

Appendix A
The following histograms show the distribution of users based on the number of quizzes completed, quizzes attempted, and the overall quiz mean score.                    Relationships between performance, engagement and user demographics To understand the relationship between different demographic factors and users' performance, the following binomial regression model was built: Correct responses per quiz = intercept + a1 * user type + a2 * gender + a3 * location of training + a4 * practice type + a5* year started practicing + a6 * year of birth.
To understand the relationship between different demographic factors and users' engagement, the following negative binomial regression model was built: Quizzes completed = intercept + a1 * user type + a2 * gender + a3 * location of training + a4 * practice type + a5* year started practicing + a6 * year of birth.
The range of the number of quizzes completed (1 to 147) is large and these data are over-dispersed (variance is much larger than the mean). Therefore, a traditional Poisson model for counts of quizzes would not be appropriate here.
The results of these models are given in Table 4. Based on the regression models, pharmacists performed better than all other user categories and pharmacy students had the highest level of engagement.
Investigation of the differences in performance and engagement for certain demographic factors individually was wanted. Univariate ANOVA models were used, followed by Tukey's honest significance differences test to look for statistically significant differences between groups.
Based on the regression models, pharmacists performed better than all other user categories and pharmacy students had the highest level of engagement (Table 4). According to the ANOVA, there were statistically significant differences between user categories for performance (F (3, 98,227) = 117.8, p < 0.001) and engagement (F(3, 98,227) = 623.4, p < 0.001). The differences in performance were small, while the differences in engagement were more pronounced (Appendix A, Figure A7). Relationships between performance, engagement and user demographics To understand the relationship between different demographic factors and users' performance, the following binomial regression model was built: Correct responses per quiz = intercept + a1 * user type + a2 * gender + a3 * location of training + a4 * practice type + a5* year started practicing + a6 * year of birth.
To understand the relationship between different demographic factors and users' engagement, the following negative binomial regression model was built: Quizzes completed = intercept + a1 * user type + a2 * gender + a3 * location of training + a4 * practice type + a5* year started practicing + a6 * year of birth.
The range of the number of quizzes completed (1 to 147) is large and these data are overdispersed (variance is much larger than the mean). Therefore, a traditional Poisson model for counts of quizzes would not be appropriate here.
The results of these models are given in Table 4. Based on the regression models, pharmacists performed better than all other user categories and pharmacy students had the highest level of engagement.
Investigation of the differences in performance and engagement for certain demographic factors individually was wanted. Univariate ANOVA models were used, followed by Tukey's honest significance differences test to look for statistically significant differences between groups.
Based on the regression models, pharmacists performed better than all other user categories and pharmacy students had the highest level of engagement (Table 4). According to the ANOVA, there were statistically significant differences between user categories for performance (F (3, 98,227) = 117.8, p < 0.001) and engagement (F(3, 98,227) = 623.4, p < 0.001). The differences in performance were small, while the differences in engagement were more pronounced (Appendix A, Figure A7).  Users who obtained their entry to practice training in Canada performed better than other users, while engagement was similar whether users trained inside or outside Canada (Table 4). According to the ANOVA, there were statistically significant differences between training locations for performance (F (2, 98228) = 98.21, p < 0.001) and engagement (F (2, 98,228) = 231.7, p < 0.001). However, as above, the differences in performance were small, while the differences in engagement were more pronounced (Appendix A, Figure A8). Users who obtained their entry to practice training in Canada performed better than other users, while engagement was similar whether users trained inside or outside Canada (Table 4). According to the ANOVA, there were statistically significant differences between training locations for performance (F (2, 98228) = 98.21, p < 0.001) and engagement (F (2, 98,228) = 231.7, p < 0.001). However, as above, the differences in performance were small, while the differences in engagement were more pronounced (Appendix A, Figure A8). Users working in primary care performed better than other users, while engagement was highest for users working in independent practice (Table 4). According to the ANOVA, there were statistically significant differences between practice locations for performance (F (6, 98,224) = 17.58, p < 0.001) and engagement (F (6, 98,224) = 372.2, p < 0.001). The differences in performance were small, while the differences in engagement were more pronounced (Appendix A, Figure A9). Users working in primary care performed better than other users, while engagement was highest for users working in independent practice (Table 4). According to the ANOVA, there were statistically significant differences between practice locations for performance (F (6, 98,224) = 17.58, p < 0.001) and engagement (F (6, 98,224) = 372.2, p < 0.001). The differences in performance were small, while the differences in engagement were more pronounced (Appendix A, Figure A9).
One way to determine if there was a relationship between user performance and engagement was to model quiz scores as a function of quizzes completed. Twenty-one binomial regression models (one for each module) were built with the following structure: Correct responses per quiz = intercept + a* quizzes completed + b1 * user type + b2 * gender + b3 * location of training + b4 * practice type + b5* year started practicing + b6 * year of birth.
Another way to determine the relationship between user performance and engagement, was to model quizzes completed as a function of quiz scores. Twenty-one Poisson regression models (one for each module) were built with the following structure: Quizzes completed = intercept + a* correct responses per quiz + b1 * user type + b2 * gender + b3 * location of training + b4 * practice type + b5* year started practicing + b6 * year of birth.
The results of these models are given in Table 5.