A repeated cross-sectional study on the implementation of electronic medication management system

Information technology has changed our way of life more conveniently and comfortably. It is no exaggeration to say that IT is increasingly being applied to every single activity and providing solutions to the enormous problems we face. One of the major application areas of IT is health, which has received increasing and ongoing investments in it. One thing that differentiates health from other application fields of IT is that it requires a more rigorous and robust evaluation of the technologies, given that it is related to human life. Thus, some health technology evaluation frameworks have been developed and used widely in health-related organizations and governments in the world [1].

Studies on HIT (Health Information Technology), including EMR (Electronic Medical Record), EHR (Electronic Health Record), CPOE (Computerized Physician Order Entry), and eMMS (Electronic Medication Management System), have increased enormously over the last decade [2,3,4]. In particular, the effect of CPOE, EMR, and EHR on clinical work efficiency has been studied for a long time. However, many of such studies mostly focused on the initial perceptions and effects of HIT at the very early stage of HIT implementation [5]. As increased and consistent IT use is believed to improve patient care and clinical outcomes, it is becoming more necessary to continuously monitor HIT use and user satisfaction [6, 7].

While there is increasing adoption of HIT by hospitals, that does not necessarily mean that individual clinicians are satisfied and will continue to use HIT. As most HIT systems are mandatory, clinicians have no choice but to use the system regardless of their intention or system quality [8]. Similarly, it is impossible to discontinue using the system for individual clinicians. Though there has already been a lot of literature about technology acceptance and continuance, many of the extant models are based on the voluntary and individual use of the system, which can be different from the organizational perspective. As the majority of systems introduced in organizational settings, including in the health sector, are expected to be used and therefore are mandatory, more organizational perspectives with the extant models are important in interpreting organizational and mandatory context.

More importantly, many IS continuance studies mostly compare the perceived performances with the expectations. These studies focus on initial perceptions, but these perceptions may change over time [5]. As users do not have enough hands-on experience with the system at the initial stage, they might face considerable difficulties in adapting themselves to the new system functions and the new way of working. According to the theory of belief updating [9], an initial evaluation made by prior knowledge is revised by subsequent interactions or experiences of new information. This theory has been applied to many studies of marketing, which examine intertemporal changes in customer attitudes, customer satisfaction, and intention to purchase. In line with this theory, Venkatesh et al. [10] suggested that initial beliefs such as performance expectancy, effort expectancy, social influence, and facilitating conditions have positive effects on modified beliefs, respectively. In addition, some scholars suggest that the importance of these factors and their effects on outcomes tend to be different at different stages of implementation [10,11,12].

Among many HITs, the electronic medication management system (eMMS) was chosen as a case for this study. The eMMS is changing the way of working in hospitals more efficiently and effectively. With raised awareness of the importance of eMMS, there have been growing investments in eMMS in many countries across the world. Australia is one of the countries which has invested a huge amount of budget into eMMS to provide better healthcare. In particular, the New South Wales (NSW) Government of Australia has been making impressive progress in implementing health IT at the state level. With the eHealth Strategy for NSW Health: 2016–2026, $236 million over seven years, including $18.7 million in 2017–2018, are being invested in the roll-out of the eMMS (Electronic Medication Management System) across NSW [13].

Among the existing IT implementation models, this study employs the three-stage model from Wright et al.[14]: (1) initiation, during which a new IS is evaluated for its usefulness before adoption, (2) adoption, in which the decision to adopt and initially use the IS is made, and (3) assimilation, during which the IS becomes widely used as an integral part of an organization’s business processes. Figure 1 shows different stages of eMMS project in this study to distinguish clearly “perception in initial adoption” from “perception in initial assimilation”. While there have been many studies that examine the factors influencing acceptance of health IS, such as EMR, CPOE, and so on, there is a lack of study that looks at how the importance of these factors changes over time. Adoption or acceptance of a new system indicates success in the early stage, but it is also important to ensure that the system has moved beyond the initial trial and becomes ongoing success in the long term [3].

Stages of eMMS project

Full size image

The primary research questions of this study are as follows: (1) What are the factors driving clinician satisfaction and organizational benefits of eMMS in the initial adoption stage and initial assimilation stage? (2) How do user evaluations change over time? The rest of this paper is organized as follows: First, the theoretical backgrounds are discussed together with the research gaps. Second, a series of research hypotheses are derived to be tested. Third, research methods are discussed, including study context, data collection procedure, etc. Fourth, the results of hypothesis testing are discussed. Lastly, the theoretical and practical implications of the results are discussed, including the limitations.

There are many seminal theoretical frameworks related to technology acceptance, such as TAM [15, 16], IS Success Model [17], and Expectation Confirmation Model (ECM) of Information System Continuance [18]. These theoretical frameworks have been used for the evaluation of HIT as well. TAM [15] is the most widely used model of technology acceptance, which conceptualizes the perceptions and intentions people have in relation to accepting a particular technology, whereas the IS Success model [17] presents a process model which provides a multidimensional view of IS success. Additionally, the ECM of IS Continuance [18] compares expectations and initial perceptions of a system and focuses on continuance intention as an outcome.

2.1 IS success model

Delone and Mclean’s IS Success Model [17] suggested two factors for IS success, namely system quality and information quality, which have effects on use, user satisfaction, individual impacts, and ultimately organizational impacts. Extension of this model has been presented by the authors [19] and some other scholars [20, 21] with an additional success factor, service quality, and applied in evaluating many types of information systems. Another difference from the original model is that Delone and McLean [19] presented net benefits as an ultimate outcome, which can integrate all the “impact” measures such as individual impacts, organizational impacts, and so on.

Petter et al. [22] conducted a qualitative literature review of 180 studies found in the academic literature from 1992 to 2007 dealing with some aspects of IS success. Six dimensions of the DeLone and McLean model [19], such as system quality, information quality, service quality, use, user satisfaction, and net benefits, were used to analyze 90 empirical studies, and the results were synthesized. A total of 15 pairwise associations between the success constructs were examined. Many different kinds of IS under a variety of conditions were considered, and the majority of relationships were reasonably supported, suggesting the value of the DeLone and McLean model [19] when evaluating utilitarian IS.

Moreover, the IS success model has been applied in many HIT studies and provides empirical support in HIT as well [23, 24]. Garcia-Smith and Effken [25] developed the clinical information systems success model (CISSM) based on IS success model and UTAUT (Unified Theory of Acceptance and Use of Technology; [26]. This model was tested with the data from 234 registered nurses in four hospitals and significantly supported IS success model.

A big difference between TAM [15, 16] and IS success model is that TAM focuses on the acceptance of technology in terms of intention to use and actual use, whereas the IS success model focuses on distal outcomes after using information systems such as individual and organizational benefits. It is important and necessary to integrate and link these frameworks under a common logical structure with a common linking pin. In order to do this, it is necessary to have a further look at TAM.

2.2 Technology acceptance and IS continuance model

Two of the most applied models in technology acceptance are TAM and UTAUT. The original TAM [15, 16] proposed two major constructs influencing IT adoption, which are perceived usefulness and perceived ease of use. TAM has been further developed as UTAUT. Many studies showed that TAM and UTAUT could be successfully applied in predicting the acceptance behavior in the health care context [3, 8].

Specifically, perceived usefulness and perceived ease of use are found to have significant effects on technology acceptance and use in healthcare settings [27,28,29]. There are also some studies that successfully applied all four drivers of UTAUT in healthcare contexts [30,31,32]. However, there are some differences in the results of other studies depending on the type of information systems and characteristics of the system users [32]. Holden and Karsh [33] showed that effort expectancy was not a significant predictor of intention to use in six out of 13 HIT studies. Maillet et al. [34] studied 616 nurses’ satisfaction and actual use of EPR (Electronic Patient Record) from four hospitals and found that only performance expectancy influences both satisfaction and actual use and that facilitating conditions and effort expectancy has no significant effect on actual use.

Summarizing these findings, while TAM and UTAUT have been widely used as a general information systems evaluation framework, the findings in health care are quite dependent on the context of the study. That is why many authors added variables to the original TAM and UTAUT models or revised the survey instruments to better adapt to the context of health care.

Another extensively used model with regard to technology acceptance in the post-adoption stage is the Expectation Confirmation Model (ECM) of IS Continuance [18]. Adapted from the traditional Expectation Confirmation Theory (ECT) [35], the ECM is different from the ECT in several ways. First, while the ECT has a construct, “expectation”, which means pre-purchase expectation measured at t1, the ECM does not have a construct like pre-acceptance expectation but focuses only on post-acceptance expectation, represented by perceived usefulness. Originally in the ECT, consumers first have an initial expectation prior to purchase (t1). After the purchase and experience, consumers will compare the perceived performance (t2) with their initial expectations (t1). However, it is hard for users to have a realistic expectation of a new IS without first-hand experience of it, as IS belongs to experience goods whose value can only be assessed after they are experienced [36]. Even in mandatory IS, it may be harder for users to have any specific and defined expectations before use [37]. Thus, in many IS studies, users were given at least a short introduction to the system before the questionnaire survey to measure their initial expectations of the system. This is not the problem of the ECT but a methodological and contextual difference between traditional ECT and IS studies. Thus, whether it is purely a pre-use expectation or an initial post-use expectation is not a critical issue in IS study. The more important point is that both pre-use expectation and initial post-use expectation are initial perceptions that may change over time and provide the baseline or reference level for later perception.

The ECM of IS Continuance [18] suggested that the continuance intention of the information system is determined by user satisfaction and perceived usefulness. The model also showed that user satisfaction is determined by the confirmation of expectation and perceived usefulness. Many studies [10, 38,39,40,41] applied and updated this model. In addition, the ECM of IS Continuance has been applied in the health context as well. Palm et al. [42] suggested Information Technology Post Adoption Model (ITPAM) and conducted a clinical information system post-adoption evaluation by customizing the ECM of IS Continuance. Furthermore, Hadji and Degoulet [43] extended ITPAM (ITPAM2) with a construct (system quality) from IS success model and provided empirical support. ITPAM2 was further developed as UMISC (Unified Model of Information System Continuance [44]) and has been validated in some health settings [45].

2.3 Guiding theoretical framework

Notwithstanding the extensive use of TAM in IS research, it is still valuable to reposition TAM and its extended models in terms of conceptual, methodological, and replication treatments [46, 47]. One of them is the integration with other theories and the contextualization for specific settings [1, 48]. Lim et al. [49] addressed the problem of theoretical simplicity of TAM by integrating UTAUT and IS Success Model [17], and provided an exemplar of contextualizing technology acceptance theories for the specific setting (i.e., social media adoption by non-urban communities for non-profit activities). Moreover, Samhan [50] emphasized the importance of understanding the context as outcomes with the same technology may be different in different settings. Furthermore, the contextualization of technology acceptance needs to consider the type of technology enforcement, such as voluntary versus mandatory, utilitarian versus hedonic systems, etc. [50]. This paper tries to integrate three theories (UTAUT, IS success model, and ECM of IS Continuance) that share some common constructs, and take them as the starting point for a conceptual framework that describes the perceptions and intentions of clinicians in Australian hospitals regarding the use of eMMS. In this way, this study not only addresses the problem of theoretical simplicity of TAM by linking UTAUT, IS Success Model, and IS Continuance model, but also provides more practical value by contextualizing technology acceptance theories for a specific setting (i.e., mandatory eMMS in Australia).

TAM is an individual-level acceptance model and is mostly used for the voluntary use of IT systems. However, most systems in organizations, including HIT, are mandatory systems that users do not have a choice not to use. As eMMS is also a mandatory system, this study applies the constructs from UTAUT for the conceptual framework of this study due to its comprehensiveness and applicability to the organizational context. UTAUT is applicable in explaining technology acceptance in a mandatory setting as UTAUT focuses on large organizations in the business environment, which has a mandatory context [26, 51, 52]. Furthermore, the outcome variable of UTAUT, behavioral intention, needs to be modified to clinician satisfaction to fit the mandatory context, which can be used further as a linkage to connect existing theories. Moreover, TAM [15, 16] or UTAUT [26] emphasizes on user’s perspective, such as individual user’s acceptance. However, these models need to be extended to distal outcomes, which can mean “success” of IS because user satisfaction has a lack of robust theoretical underpinnings as a measure of IS success [53]. Hence, organizational benefits of using eMMS such as effectiveness, efficiency, and safety, need to be added as a measure of eMMS success.

Another common construct in the three theories is perceived usefulness, the degree to which users feel that the system is useful, which means the degree to which users think that the system helps their work and performance [26]. Perceived usefulness matches with the concept of system quality (e.g., usefulness of specific functions) and individual impact (e.g., performing tasks more quickly) in IS success model [17]. Thus, with this conceptual similarity, the perceived usefulness of eMMS in this study represents more integrative meaning which includes the clinician’s individual benefits as well as the quality of eMMS functions. Table 1 shows the mapping of the contextualized constructs of this study and constructs from three theories, which are used to link three theories to this study accordingly.

Last but not least, as this study focuses on how initial perceptions of eMMS change over time, a core guiding theoretical framework of this study is ECM of IS Continuance [18]. The major construct of ECM is confirmation, which compares expectation and actual performance and, in turn, confirms to what extent actual performance is better than expected. The methods of operationalizing confirmation in the ECT literature are divided into three ways: objective, inferred, and perceived [54]. Objective confirmation is not an appropriate way for the experience goods or services like eMMS as they are normally not assessed based on the objective criteria. The ECM employed the method of perceived confirmation, which compares perceptions of t2 and t1 retrospectively at the time of t2 without separately measuring at t1. However, it is hard for users to recall the initial perceptions at a later stage (e.g., one year later). In contrast, inferred confirmation measures both expectation (t1) and actual performance (t2) separately, and it calculates the difference between expectation and actual performance. In this study, both perceptions at each stage are measured respectively. Thus, initial perception can be regarded as expectation, and perception one year later can be regarded as actual performance.

Drawing on the works by Venkatesh et al. [26], DeLone and Mclean [17], and Bhattacherjee [18], this paper proposes a research model on key factors that influence clinician satisfaction and organizational benefits of eMMS. The three theories were connected with clinician satisfaction as the linking pin, and further adapted to a mandatory IS context. In addition, this study looks at not only the relationships at a particular stage, but also how they change from the initial adoption stage to the initial assimilation stage. Figure 2 shows the research model proposed in this paper, which is trying to test a set of hypotheses developed accordingly. The clinician’s perceptions between the initial adoption stage and the initial assimilation stage will be compared with this research model to see if there is any difference. The control variables such as user groups (doctors, nurses, pharmacists) and clinical work experience are included in this research model but not hypothesized as they are used only for controlling extraneous effects.

Research model

Full size image

Perceived ease of use, or effort expectancy, of eMMS refers to the extent of ease associated with the use of eMMS [26]. If clinicians feel that eMMS is very complicated and hard to use, they will not be satisfied with eMMS. Conversely, if they feel eMMS is user-friendly and easy to use, their satisfaction will increase. Particularly, perceived ease of use of eMMS in this specific context stands for the concept of relative ease than the old system (paper-based medication), which gives a novel perspective to UTAUT constructs. It has been found that healthcare professionals’ perceived ease of use has a positive effect on behavioral intention to use health IT [55,56,57]. This is because the relationship between perceived ease of use and behavioral intention has a hidden mediator, user attitude, which means emotion toward health IT itself. The original model of TAM shows such a mechanism that users’ beliefs lead to users’ intention to use a system via user’s attitude. User satisfaction is conceptually similar to user attitude and one of the most comprehensive, appropriate, and important evaluations of IS success [17].

Some literature showed that perceived ease of use of health IT is also a factor positively affecting clinicians’ satisfaction in using health IT [31, 58, 59]. Perceived ease of use in the initial adoption stage is the post-acceptance expectation with very little and superficial use experience of eMMS. This initial post-acceptance expectation will be confirmed positively in the initial assimilation stage as perceived ease of use can be increased by enough use of eMMS for one year. In other words, clinicians may feel that eMMS is getting easier to use as time goes on due to the repetitive use of eMMS for one year. However, the effect of perceived ease of use on clinician satisfaction will be decreased in the initial assimilation stage, and perceived ease of use may not necessarily serve as an influencing factor of clinician satisfaction as clinicians might be already accustomed to the functions of eMMS and ease of use itself does not affect clinician satisfaction anymore. In fact, Karahanna et al. [11] found that effort expectancy has non-significant effects on attitude toward continuance to use in the later stage. Maillet et al. [34] also showed that effort expectancy has no significant effect on the actual use of health IT in the post-implementation stage. Based on such theoretical backgrounds, the following hypotheses are suggested:

H1a

Clinicians’ perceived ease of use on eMMS has a positive effect on clinician satisfaction with eMMS in the initial adoption stage.

H1b

Clinicians’ perceived ease of use on eMMS has no effect on clinician satisfaction with eMMS in the initial assimilation stage.

H1c

The effect of perceived ease of use on clinician satisfaction in the initial adoption stage is greater than the effect in the initial assimilation stage.

H1d

The level of clinicians’ perceived ease of use on eMMS in the initial assimilation stage is higher than in the initial adoption stage.

Perceived usefulness of eMMS in this study is defined as the extent to which a clinician perceives that using eMMS will help him or her to attain gains in job performance [26]. Enhanced job performance is instrumental in achieving various rewards in an organization, such as promotions or monetary gains [18, 60]. As human tends to subconsciously pursue instrumental behaviors or strive for rewards, perceived usefulness is an individual benefit and extrinsic motivation to activate the positive emotional state [18]. In the context of this study, perceived usefulness of eMMS also includes the concept of relative usefulness than the old system (paper-based medication), which contextually means perceived individual improvements in medication management such as increased time saving and convenience. If clinicians think that eMMS is more beneficial for their task performance than the old system, they will be happy to use eMMS. Many studies showed that perceived usefulness, or similarly, performance expectancy, has a positive effect on behavioral intention to use Health IT [61,62,63] or clinician satisfaction in using Health IT [64, 65]. As discussed in hypothesis 1, the relationship between perceived usefulness and behavioral intention also has a hidden mediator, user attitude, so the underlying mechanism of this relationship is the same as the one in hypothesis 1. Perceived usefulness is a cognitive belief noticeable to IS use, which also forms a post-acceptance expectation in the initial acceptance stage, and plays a role of confirmed performance in the initial assimilation stage. In the initial adoption stage, clinicians’ perceived usefulness is relatively lower than in the initial assimilation stage as clinicians are not able to feel the full usefulness of eMMS due to the very short time of use experience of eMMS. In addition, the effect of perceived usefulness of eMMS on clinician satisfaction in the initial assimilation stage will be greater than in the initial adoption stage as clinicians will be satisfied more if they experience actual usefulness fully for one year. Alsyouf and Ishak [66] supported this argument, which showed that nurses’ performance expectancy of EHR had a positive effect on continuous intention to use EHR in the later stage. Thus, the perceived usefulness of health IT can influence a clinician’s intention to use health IT regardless of whether it is for the first time use or continued use. Based on these, the following hypotheses are proposed:

H2a

Clinician’s perceived usefulness on eMMS has a positive effect on clinician satisfaction with eMMS in the initial adoption stage.

H2b

Clinician’s perceived usefulness on eMMS has a positive effect on clinician satisfaction with eMMS in the initial assimilation stage.

H2c

The effect of perceived usefulness on clinician satisfaction in the initial assimilation stage is greater than the effect in the initial adoption stage.

H2d

The level of clinician’s perceived usefulness on eMMS in the initial assimilation stage is higher than in the initial adoption stage.

In this study, facilitating conditions reflect the organizational supports and technical infrastructure of a hospital for using eMMS. Generally, facilitating conditions means an ICT help desk or a call centre which provides users with remote assistance services in most cases. In the context of eMMS, however, the ICT support team provided 24 × 7 (24 h a day, 7 days a week) services, having people on site in order to go to the point of care where issues are being raised. This is crucial point for the users as it gives completely different feeling that they are well catered and well supported. Other facilitating conditions specific to the context of this study are adequate user training on eMMS, a 2:1 device-to-patient ratio, and so on. If clinicians feel that they are well supported by the hospital in using eMMS, they may have satisfaction of the hospital, which in turn leads to satisfaction in using eMMS. Some HIT studies supported this argument, which showed that facilitating conditions have a positive effect on behavioral intention to use health IT [67, 68]. In addition, Garcia-Smith and Effken [25] showed that facilitating conditions in hospitals have a positive effect on clinician satisfaction. Along with other beliefs such as perceived ease of use and perceived usefulness, facilitating conditions in the initial adoption stage is also a post-acceptance expectation with little experience related to eMMS use. This initial evaluation on facilitating conditions will decrease in the initial assimilation stage as clinicians may feel that they do not need much support from IT help desk if they become proficient in using eMMS. In addition, facilitating conditions may not have a significant effect on clinician satisfaction, and the effect of facilitating conditions on clinician satisfaction will also be decreased in the initial assimilation stage as facilitating conditions will become less important in making clinicians satisfied. Some post-implementation studies of health IS supported this argument. Mellikeche et al. [44] showed that facilitating conditions of hospitals have no significant effect on user satisfaction in the later stage. Hadji et al. [69] also suggested that facilitating conditions such as technical support and user training are not determinant in shaping clinicians’ satisfaction in clinical IS use during the later stage. Thus, the following hypotheses can be derived:

H3a

Facilitating conditions of hospitals have a positive effect on clinician satisfaction with eMMS in the initial adoption stage.

H3b

Facilitating conditions of hospitals have no effect on clinician satisfaction with eMMS in the initial assimilation stage.

H3c

The effect of facilitating conditions on clinician satisfaction in the initial assimilation stage is less than the effect in the initial adoption stage.

H3d

The level of facilitating conditions of hospitals in the initial assimilation stage is lower than in the initial adoption stage.

Next, social influence among clinicians is defined as the extent to which other clinicians’ belief on eMMS use has an influence on an individual clinician’s perception [26]. In this study context, social influence among clinicians is mostly about colleagues’ favorable opinions on eMMS use, which can be their perceived benefits. Moreover, social influence in the context of this study is closer to informational influence rather than normative influence [70] as clinicians have relatively high autonomy in the practice of their profession [71]. Another thing special in this context is that there have been proactive and well-informed superusers (1:6 ratio) who act as support to their colleagues in eMMS implementation, which may be a factor that influences other clinicians’ perception on eMMS. If most of the peers agree that the way of working with eMMS is a good idea, favorable peer opinions can increase clinicians’ satisfaction with eMMS. Some health IT studies supported this argument, which showed that social influence among clinicians has a positive effect on behavioral intention to use health IT [61, 72, 73]. Social influence among clinicians is also a post-acceptance expectation, which will be confirmed positively in the initial assimilation stage as more colleagues will become favorable to eMMS with proficiency and adaptation of eMMS for one year. Increased social influence among clinicians will have an effect on clinician satisfaction in the initial assimilation stage, and the effect will be greater than in the initial adoption stage. Kumar and Natarajan [74] supported this argument, which has shown that social influence had a positive effect on continuance intention to use an e-health service in the later stage. Based on these, the following hypotheses can be proposed:

H4a

Social influence among clinicians in a hospital has a positive effect on clinician satisfaction with eMMS in the initial adoption stage.

H4b

Social influence among clinicians in a hospital has a positive effect on clinician satisfaction with eMMS in the initial assimilation stage.

H4c

The effect of social influence among clinicians on clinician satisfaction in the initial assimilation stage is greater than the effect in the initial adoption stage.

H4d

The level of social influence among clinicians in the initial assimilation stage is higher than in the initial adoption stage.

Organizational benefits are defined as the increase in organizational outcomes and the decrease in organizational inputs [75]. In the context of eMMS use, organizational benefits can be improvements in quality, safety, and effectiveness of medication management and reduction of medication errors and adverse drug events, and so on. If clinicians are happy to use eMMS use, they will positively think that eMMS will be beneficial for the hospital as well. Gelderman [76] showed significant correlations between user satisfaction and organizational impact. In the updated IS Success Model [19], Delone and Mclean also showed that user satisfaction has a positive influence on net benefits. Moreover, Garcia-Smith and Effken [25] have found that clinician satisfaction has a positive effect on net benefits. As discussed in Table 1, organizational impact and net benefits are conceptually the same as organizational benefits of eMMS in this study. Thus, these studies empirically supported our arguments. Organizational benefits of eMMS can be increased in the initial assimilation stage, and the effect of clinician satisfaction on organizational benefits will also be increased in the initial assimilation stage as the benefits from IT may not appear immediately but can be felt more after some time [77]. Clinician satisfaction can also be increased in the initial assimilation stage as some concerns regarding eMMS use in the initial adoption stage may be resolved already during one year. According to the mere ownership effect [78], people would evaluate an object more favorably merely because they owned it. This theory can also be applied to the context of clinician satisfaction and organizational benefits in this study. Based on this theoretical background, the following hypotheses can be suggested:

H5a

Clinician satisfaction with eMMS has a positive effect on organizational benefits of eMMS in the initial adoption stage.

H5b

Clinician satisfaction with eMMS has a positive effect on organizational benefits of eMMS in the initial assimilation stage.

H5c

The effect of clinician satisfaction on organizational benefits of eMMS in the initial assimilation stage is greater than the effect in the initial adoption stage.

H5d

The level of clinician satisfaction in the initial assimilation stage is higher than in the initial adoption stage.

H5e

The level of organizational benefits of eMMS in the initial assimilation stage is higher than in the initial adoption stage.

This study is to conduct predictive- and prescriptive-oriented research (i.e., establish cause and effect) rather than descriptive- and association- (e.g., exploratory insights) oriented research. Moreover, in order to see the differences in impact over time, this study takes a conditional approach in which time period (initial adoption stage and initial assimilation stage) can be the boundary condition [79]. With the data collected from two points in time, a quantitative approach is adopted to empirically validate the hypotheses presented in the previous section. In this section, the study setting, data collection procedure, and survey design are described.

4.1 Study setting

The research model is examined using survey data collected from a public tertiary teaching hospital in Sydney, with about 1200 employees. The case hospital was in the position of a leader in electronic medication record (EMR) implementation in New South Wales and had been proactively adopting digital transformation initiatives. Unlike most Australian hospitals that have implemented eMMS prior to a hospital-wide EMR, which consequently led to prolonged adoption [80], the case hospital is the first site in Australia to have full EMR utilisation for 2 years prior to eMMS implementation. The experience of implementing EMR prior to eMMS had raised awareness and readiness of electronic health care.

The eMMS implemented in the case hospital is composed of different modules to support a variety of medication management processes such as prescription, verification, reconciliation, and administration of medication. The different modules are used by different user groups, including doctors, pharmacists, and nurses. For example, modules related to medication prescription processes are used by doctors, whereas modules related to the medication administration processes are used by nurses, and those related to medication verification and dispensing processes are used by pharmacists. There are also modules used by multiple user groups, such as those related to medication reconciliation processes.

4.2 Data collection

A survey study was conducted with clinicians regarding their perceptions of the implementation of eMMS, and it was divided into two stages. The initial adoption survey was conducted for about six weeks in 2017. The questionnaire was distributed to the users of eMMS, including doctors, nurses, and pharmacists. In the initial adoption survey, a total of 202 responses were collected, which included responses from 65 junior doctors (32%), 24 senior doctors (12%), 101 nurses (50%), and 12 pharmacists (6%). The survey in the initial assimilation stage was conducted one year after the initial adoption survey. An online questionnaire was first distributed to the users of eMMS. The online questionnaire was followed by a paper questionnaire distributed to the users in different wards who had not responded to the online questionnaire. In the initial assimilation stage survey, 196 complete responses were collected, including responses from 74 junior doctors (38%), 21 senior doctors (11%), 93 nurses (47%), and 8 pharmacists (4%).

Since this research involved the participation of humans, formal approval from an institutional review board was compulsory. This research was reviewed and approved by a Human Research Ethics Committee and has been undertaken in accordance with the National Statement on Ethical Conduct in Human Research [81]. As per the requirement for consent [81], all participations were voluntary and only the subjects who had provided consent to the survey were given the questionnaire. Moreover, the participants were informed that their answers would be kept anonymous and confidential. In addition, the research team checked whether participants were eligible users of eMMS.

The data collection is conducted using mainly convenience sampling where participants are selected based on their accessibility to the research.. The key advantages of convenience sampling are that it is cheaper, more efficient, less time-intensive, easier, and simpler to implement than other sampling methods [82]. However, the key disadvantage of convenience sampling is that as not every unit in the population has a chance of being selected in the sample, it can have a self-selection bias, so the scientific generalization derived from convenience sampling is limited [83]. To complement this disadvantage of convenience sampling, the research team contacted as many clinicians as possible to collect data in person by visiting ‘ward by ward’. As such, the data collection for each survey took more than five weeks (much longer than originally planned) and almost all eligible eMMS users in the hospital were at least given a chance to be included in the sample. As a result, the risk of self-selection bias was minimized and the shortcomings of convenience sampling are deemed minimal in this research.

4.3 Measures

The survey items were adapted from extant literature and theories such as UTAUT [26] and the same items are used in both surveys to compare the results in the initial adoption stage and initial assimilation stage. Items for perceived ease of use, perceived usefulness, social influence among clinicians, and facilitating conditions of hospitals were adapted from UTAUT [26]. Items for clinician satisfaction were adapted from Sirdeshmukh et al. [84] and Kim and Kankanhalli [85]. Items for organizational benefits were adapted from Moore and Benbasat [86] and Kim and Kankanhalli [85]. Table 2 summarizes the items used in the survey. All the items were measured using 7-point Likert scale (from strongly disagree to strongly agree). To establish the face and content validity of the items, a pre-test was undertaken by three IS researchers and four clinical researchers including a doctor, a nurse, and two pharmacists.

Since there are multiple groups of users in eMMS such as doctors, nurses, and pharmacists, and these different group of users use different modules of eMMS, there could be a possibility of differences between user groups. Hence, to minimize such potential biases, the control variables such as user group and work experience of respondents are included in the analysis. These data of the control variables were collected together with other measures.

5.1 Measurement model

The measurement model was assessed for convergent and discriminant validity by performing a confirmatory factor analysis (CFA) using Smart PLS 2.0 [87]. Firstly, convergent validity was investigated by the standardized factor loadings, CR (Composite Reliability), and AVE (Average variance extracted). As can be seen in Tables 3 and 4, the standardized factor loadings were all significant (p value < 0.001) and greater than 0.7. The values of CR for all latent variables are greater than 0.7, and the values of AVE for all latent variables are also greater than 0.5. Hence, the convergent validity of the survey instrument was supported. Secondly, discriminant validity was examined with the square root of AVE for each latent variable and the correlations between the latent variables. As shown in Tables 3 and 4, the square root of AVE for each latent variable (diagonal element) is greater than the correlations between latent variables (off-diagonal elements). Thus, the discriminant validity of the survey instrument was supported.

As this study is based on the survey data completed by respondents themselves, there can be vulnerability to common method bias, which can have a significant impact on research findings [88]. For this reason, two approaches are adopted to identify common method bias. One is a full collinearity test [89, 90], and the other is a comprehensive CFA marker technique [91]. Firstly, a full collinearity test was conducted for both 2017 and 2018 datasets and generated VIF (Variance Inflation Factor). If any VIF is equal to or greater than 3.3, then it can be concluded that common method bias is present. The results showed that all VIFs of latent variables in both 2017 and 2018 range from 1.082 to 2.862, all below the recommended threshold of 3.3 [89, 90]. Secondly, following the procedures of a comprehensive CFA marker technique described in [91], the Method-C model was compared with the baseline model to test the presence of method variance associated with the marker variable. The null hypothesis that the method factor loadings associated with the marker variable are not related to each of 18 substantive indicators was tested by the chi-square difference test comparing the two models. The result in the initial adoption stage showed that the method-C model (Chi-square = 304.7, df = 158, CFI = 0.965, RMSEA = 0.068) does not fit significantly better than the baseline model (Chi-square = 305.1, df = 159, CFI = 0.965, RMSEA = 0.068). The result in the initial assimilation stage also showed that the method-C model (Chi-square = 299.8, df = 158, CFI = 0.971, RMSEA = 0.068) does not fit significantly better than the baseline model (Chi-square = 305.1, df = 159, CFI = 0.971, RMSEA = 0.067). Thus, there is no evidence of shared common method variance between the indicators of the substantive variables and the latent marker variables. In conclusion, both approaches indicate that common method bias is deemed not a concern in this study.

5.2 Comparison of construct means

Table 5 shows the descriptive statistics of five latent variables in the initial adoption stage and initial assimilation stage. The means of perceived ease of use, perceived usefulness, social influence, clinician satisfaction, and organizational benefits in the initial assimilation stage are greater than those in the initial adoption stage, providing statistical support for hypotheses H1d, H2d, H4d, H5d, and H5e respectively. Only facilitating conditions have no statistical difference in the mean between the initial adoption stage and the initial assimilation stage. Thus, hypothesis H3d is not supported statistically.

5.3 Structural model

Since CFA results showed the validity of the underlying structure of the items measured, we moved to the testing of the structural model. The path analysis of the structural equation model was performed using Smart PLS 2.0 [87] for testing the hypotheses 1–5 proposed in the previous section. The results of the structural model are shown in Figs. 3 and 4.

Results of the structural model (initial adoption stage)

Full size image

Results of the structural model (initial assimilation stage)

Full size image

In the initial adoption stage, all four factors showed significant effects on clinician satisfaction. Among these four factors, perceived usefulness has the highest positive significant effect on clinician satisfaction (path-coefficient = 0.453, p < 0.001), supporting hypothesis H2a. The other three antecedents also have significant positive effects on clinician satisfaction. Perceived ease of use has a lower path-coefficient of 0.227 (p < 0.001), providing empirical support for hypothesis H1a. Similarly, social influence among clinicians shows the path-coefficient of 0.193 (p < 0.001), supporting hypothesis H4a. Facilitating conditions indicate the lowest path-coefficient of 0.134 (p < 0.05), providing statistical support for hypothesis H3a. A total of 75.9 percent of the variance of clinician satisfaction is explained by the above four relationships. Moreover, clinician satisfaction has a significant positive effect on organizational benefits, with a standardized effect size of 0.668 (p < 0.001), statistically supporting hypothesis H5a, which accounts for 42.9 percent of the variance of organizational benefits.

In the initial assimilation stage, only two factors indicated significant effects on clinician satisfaction, which accounts for 74.4 percent of the variance of clinician satisfaction. Perceived usefulness has a higher path-coefficient of 0.543 (p < −0.001), providing empirical support for hypothesis H2b. Social influence among clinicians also has a significant influence on clinician satisfaction with the path-coefficient of 0.257 (p < 0.001), supporting hypothesis H4b. Perceived ease of use and facilitating conditions have no significant effects on clinician satisfaction. Thus, hypotheses H1b and H3b are also supported. Additionally, clinician satisfaction has a significant positive effect on organizational benefits with the path-coefficient of 0.815 (p < 0.001), empirically supporting hypothesis H5b, which accounts for 64.1 percent of the variance of organizational benefits.

The overall results suggest that two factors, perceived usefulness, and social influence, continue to play an important role in driving clinician satisfaction over one year. Whereas the other two factors, perceived ease of use and facilitating conditions, only influence in the first year. Moreover, the result in the initial assimilation stage showed that clinician satisfaction explains a greater amount of the variance in organizational benefits (64.1%) than in the initial adoption stage (42.9%).

5.4 Direct effects over time

The effects of perceived ease of use and facilitating conditions on clinician satisfaction were found to be significant in the initial adoption stage but become insignificant in the initial assimilation stage. To determine whether the changes were significant over time, a multi group analysis using Smart PLS was conducted to test the magnitude and statistical significance of the difference for the coefficients in initial adoption and initial assimilation stages.

As shown in Table 6 above, only the coefficient of clinician satisfaction on organizational benefits has a statistically significant difference between the initial adoption stage and the initial assimilation stage, supporting hypothesis H5c. Hypotheses H1c, H2c, H3c, and H4c are not statistically supported.

5.5 Indirect effects

As four drivers seem to have indirect effects on organizational benefits through clinician satisfaction, a post-hoc analysis to examine the mediation effects of clinician satisfaction has been performed. The results of the test are shown in Table 7. The indirect effects of perceived ease of use, perceived usefulness, social influence among clinicians, and facilitating conditions of hospitals on organizational benefits through the mediator (clinician satisfaction) are all significant in the initial adoption stage. However, the indirect effects of perceived ease of use and facilitating conditions of hospitals on organizational benefits through clinician satisfaction have become insignificant in the initial assimilation stage. In addition, the indirect effects between the initial adoption stage and the initial assimilation stage are not significantly different, which means that there is no statistically differential impact over time.

Our results suggest that the factors have slightly different effects on clinician satisfaction and organizational benefits between the initial adoption stage and the initial assimilation stage. In the initial adoption stage, all four factors significantly affect clinician satisfaction and further organizational benefits, whereas in the initial assimilation stage, only perceived usefulness and social influence among clinicians have a significant effect on clinician satisfaction and further organizational benefits. The importance of perceived ease of use and facilitating conditions have been moving toward a decrease after one year but the degree of their decrease is not enough to be statistically significant yet.

6.1 Confirmation of expectation (initial perception)

As shown in Table 5, the mean differences in perceived usefulness, perceived ease of use, social influence among clinicians, clinician satisfaction, and organizational benefits are statistically significant between the initial adoption stage and the initial assimilation stage. As perceptions in the initial stage can be regarded as expectations, the results can be interpreted as confirming the difference between the expectation and initial assimilation stage perception (actual performance).

The results also suggest that eMMS has moved beyond the initial trial and has become embedded into the routines of a hospital. Clinicians’ perceptions of the same eMMS have changed more positively in a year. The status quo bias theory [92] can provide a plausible interpretation of such results. As people prefer the current state, there can be resistance to the change. However, once they get used to the change, such a change becomes the current state. The eMMS was a change for the clinicians in the initial adoption stage, so the resistance (or reversely satisfaction) is higher (or lower). After one year, however, eMMS became the current state, which reduces (or increases) the resistance (or satisfaction).

6.2 Change of importance

Individual users are not used to a new IT system at the beginning so they may require some experience prior to becoming proficient [77]. According to the learning effect or learning curve [93], workers tend to need less time to perform tasks if task repetition occurs. As clinicians have been using eMMS repetitively for one year, they may become proficient due to familiarity with the operation and tools of eMMS. Once they become proficient, their perception of ease of use will be greater, but that is not the factor that leads to clinician satisfaction. Unfortunately, the decrease in importance over the one-year period is not statistically significant. However, the coefficient values have decreased over the period, suggesting that perceived ease of use has become a less important determinant for the subsequent success of eMMS over time.

Similarly, the importance of facilitating conditions has decreased over the one-year period, but the difference is not statistically significant. Clinicians may feel that they need less help from the IT help desk if they become proficient in using eMMS. Moreover, facilitating conditions such as user training may have an influence on the workers’ learning process. The skill learning process of the workers can be autonomous learning with the passage of time or induced learning by training [94]. Facilitating conditions are related to induced learning which is normally provided during the roll-out period. Venkatesh et al. [26] showed that perceived behavioral control, a root construct of facilitating conditions, has a significant effect on behavioral intention immediately following training (T1), but the effect disappears by the later stage (T2). Thus, users may not think that facilitating conditions in the initial assimilation stage are better than in the initial adoption stage. Consequently, facilitating conditions are not a statistically significant factor influencing clinician satisfaction in the initial assimilation stage in spite of its statistically insignificant decrease in importance.

Unlike perceived ease of use and facilitating conditions, the impact of clinician satisfaction on organizational benefits is significantly different between the two stages. This change of importance statistically means that the effect of clinician satisfaction on organizational benefits has increased after one year. Thus, it can be concluded that the impact of clinician satisfaction on organizational benefits is even stronger in the initial assimilation stage as clinicians may have felt actually realized benefits for one year.

In terms of the indirect effect, the impact of perceived ease of use and facilitating conditions is also moving toward a decrease, but there is no statistically significant difference between the two stages. Again, once clinicians become proficient in eMMS, their perception of ease of use and facilitating conditions does not significantly lead to clinician satisfaction, even though the decrease of the impact is not enough to be statistically significant.

6.3 Theoretical implications

With suggested explanations for changes in user evaluations on eMMS in an Australian hospital, this study contributes to the understanding of temporal dynamics of important beliefs on IS use. The difference between this study and other IS continuance studies is that this study compares the early experience and the later experience but IS continuance studies generally compare expectations and experiences. Expectations can be either confirmed or disconfirmed after actual use. However, there are no special reasons to measure the expectations of users before adoption in a mandatory context. Moreover, little research has been carried out in comparing user evaluations at two different time points, especially at the one-year interval. Thus, this study can contribute especially to the longitudinal studies of mandatory information systems.

This study provides some theoretical implications and contributes to the body of knowledge, especially in the health IS field. As many scholars suggested that technology should be studied with its social context [50, 95,96,97], this study explains changes in clinician’s perception between the initial adoption stage and initial assimilation stage in a specific health setting, which may be different when implementing same technology in another context. This is the first novelty of this study, and the contextualization of technology acceptance in this study is in line with the recommendations by Lim [46] as this study considers the type of technology enforcement, such as mandatory enforcement of utilitarian system (eMMS).

In addition, a major contribution is in linking technology acceptance theory, IS success model, and IS Continuance Model to provide a holistic view of how users evaluate a new information system in a health setting. This linkage to adapt UTAUT and other two theories to the context of eMMS also answers the call by Lim [46, 47] as they illustrate novel and meaningful findings which show the emerging realities of user perception in less researched settings (e.g., health). In addition, as this study focuses not just on technology acceptance but on the changes in clinicians’ perception in the initial assimilation stage, it is a different replication treatment to test whether variables from three major theories can suit the context of changes in clinicians’ perceptions. Furthermore, the adaptation mechanism in this study will help future research to employ the same mechanism for adaptation to generate a novel theory based on reconfiguration of existing theory.

6.4 Practical implications

This study also provides practical implications. This study suggests what factors the hospital’s management should focus on for the success of eMMS during both the initial adoption and initial assimilation stages. In the initial adoption stage, all four factors are important. However, as can be seen in the results, the relative importance of the factors slightly changes over time. Though expectations or initial perceptions have been positively confirmed in the initial assimilation stage, confirming something less important is not that important practically. Thus, confirming perceived ease of use and facilitating conditions does not matter as the importance of such factors is naturally moving toward a decrease in the initial assimilation stage. Confirming something still important after one year has more practical implications.

This study shows that perceived usefulness and social influence among clinicians are critical factors for subsequent success as well as initial success. Thus, work compatibility, usefulness, and utility for the users should be considered a high priority when introducing IT. Additionally, the hospital’s management should promote positive word-of-mouth through influential individuals who have a strong social influence on other users.

6.5 Limitations and future research

The findings of this study should be interpreted together with its limitations. First of all, this study was conducted in a single context, i.e., one Australian hospital with a specific information system, eMMS. Thus, this study has a limitation in applying to other contexts or being a generalized theory at this stage. Currently, this study should be interpreted in the context of eMMS, and future studies could validate this study’s results in other contexts. Secondly, this study is based on self-report survey data on users’ perceptions. Thus, the ultimate outcome construct, organizational benefits, may not actually be realized benefits. Thirdly, the effects of perceived ease of use and facilitating conditions have no statistically significant difference between the two stages. This result may not be conclusive until further examination with more than one-year interval is conducted. Future studies need to see when and whether the impact of perceived ease of use and facilitating conditions becomes significantly different. Additionally, qualitative data such as interviews with different user groups need to be added to the survey data to provide more in-depth and specific implications for the healthcare context.

This paper tried to mainly understand how user evaluations change over one year and examine whether there are any differences in factors influencing clinician satisfaction and organizational benefits between the initial adoption stage and the initial assimilation stage. The results showed that perceived ease of use and facilitating conditions have no significant effect on clinician satisfaction in the initial assimilation stage, whereas all factors have an effect in the initial adoption stage. The means of perceived ease of use, perceived usefulness, social influence, clinician satisfaction, and organizational benefits in the initial adoption stage have significantly increased in the initial assimilation stage, meaning that user evaluations have changed positively after one year. It is believed that the results and experiences of this study can benefit other researchers in the field of study.