Article Preview
TopIntroduction
e-Health (tele-health + health informatics applications) continues to be rapidly implemented despite continued demonstration that studies of adequate rigour to prove value are lacking. We do not know the value proposition of either ehealth or the status quo (Doarn, 2009). A host of limitations in published studies have been noted, as has the reality that many evaluations are performed outside of academia, often by clinical personnel without experience in evaluation, suggesting specific guidance regarding research approaches and methodologies would be beneficial (Poon et al., 2009).
Many examples of proposed approaches to evaluation of health informatics initiatives have been published. For example, DeLone and McLean (1992) presented their Model of Information Systems Success, which they recently upgraded (DeLone & McLean, 2003). The Balanced Scorecard approach, a recognised tool in the corporate sector, measures organisational performance across four quadrants (financial, customer, internal process, and learning and growth) with the goal of balancing corporate performance, was introduced to evaluation in the health sector by Inander and Kaplan (2002). Lau et al. (2006) recently presented a Benefits Evaluation framework for health information systems, which includes measurement of three dimensions of quality (system, information and service), two dimensions of system usage (use and user satisfaction) and three dimensions of net benefits (quality, access and productivity).
For tele-health, perhaps the first significant attempt to structure tele-health (tele-medicine) evaluation was the book Telemedicine: A Guide to Assessing Telecommunications in Health Care, which to this day has great value (Field, 1996). Attempts to gain international support for a common framework began in 1999, when the Tele-health Integrated Research Model (TIRM) was proposed (Scott et al., 1999). Soon thereafter, another model was provided by the Australia New Zealand Tele-health Committee (2000), although it received little praise (Hughes et al., 2002). Hebert (2001) presented a tele-health evaluation framework that built on the DeLone and McLean model and Donabedian’s structure-process-outcomes model for assessing quality of care. More recently, Hicks and Boles (2004) presented a three dimensional model for evaluation of telemedicine that provides a systematic framework to assess the effectiveness, efficiency, and feasibility of telemedicine.
Each of these has sound basis, but lacks general applicability across ehealth applications. The development of a generic evaluation tool has been debated, but to this point conventional wisdom has been that there is no single model that can be recommended, rather it is more important to simply identify one and use it! However, there is value for any single entity (healthcare organisation, sub-national region, country) to identify and adopt a single model or framework. Doing so permits the evidence gathered to be more meaningfully compared and individual evaluations then collectively and rapidly accumulate the broad evidence-base needed to demonstrate the value of ehealth interventions.
In order to provide a truly ‘generic’ framework, a conceptual approach to evaluation of proposed e-health solutions has been developed. Termed ‘Pragmatic Evaluation’ the approach respects the need for rigour, and the need to guide and advise users on evaluation approaches at different stages of development, but limits the need for sophisticated academic research (e.g., avoiding the application of randomized controlled trials).
Pragmatic Evaluation is designed as a process for selecting the most appropriate approach to evaluation of an ehealth intervention. Thus, a major aspect is the ‘staged’ perspective of ehealth implementation, and application of different study designs at these stages. The concept of Pragmatic Evaluation is described below.