- Study protocol
- Open Access
Effectiveness of computerized point-of-care reminders on adherence with multiple clinical recommendations by primary health care providers: protocol for a cluster-randomized controlled trial
SpringerPlus volume 5, Article number: 1505 (2016)
To determine the effectiveness of reminders compared to no reminders in improving adherence to multiple clinical recommendations measured as the resolution of the clinical condition that motivated the reminder, in a primary care setting with a well-established feedback system.
A 12-month, cluster-randomized, controlled clinical trial was designed (randomized by primary care team) to evaluate the impact of computerized reminders. All study participants will continue to receive the usual feedback from the electronic health records system. The control group (well-established feedback) will be compared with reminders and a well-established feedback system. The study will include all general practitioners (3425) and nurses (3262) providing primary care for a population aged 14 years or older in the 282 primary care teams reporting to the Catalan Institute of Health. Up to 10 clinical reminders are offered for each patient, recommending action related to at least one of nine clinical conditions: arterial hypertension, elevated cardiovascular risk, type 2 diabetes mellitus, cerebrovascular accident, ischemic heart disease, heart failure, atrial fibrillation, smoking habit, and hepatitis C. The outcomes are the resolution of the clinical condition that motivated the reminder and the time elapsed between the first reminder message and implementation of the recommended action (months). Due to the obvious correlation between reminders about the same patient, the profile of patients assigned to a particular professional, and the professionals assigned to a particular centre, hierarchical modelling will be used to simultaneously estimate the effect of the study variables at these different levels of analysis. To estimate the impact of the intervention arm, an analysis of adherence to each type of reminder will be carried out, using multi-level logistical regression models at level of the primary care centre. Time to adherence will be estimated by the Kaplan–Meier method and comparisons will be done using the log-rank test.
The results of this study could provide new evidence on the impact of computerized reminders at the point of care on adherence to clinical guidelines in primary care with an established feedback system.
Trial registration ISRCTN42391639. October 8, 2012
Computerization of medical records and the availability of computer access in all medical offices in the primary care setting have made it possible to study the impact of this computerization in daily clinical practice (Fina et al. 2006). The Catalan Institute of Health (Institut Català de la Salut, ICS) is the main public provider of primary care services in the National Health System of Catalonia, with 282 primary care teams serving nearly 6 million people (80 % of the Catalan population). The 8000 health professionals comprising these teams provide family, paediatric, and nursing care. Since 2005, all ICS primary care providers have used the same electronic health records system, called ECAP, and since 2006 have been able to access a feedback system, updated monthly, that reports on quality indicators evaluating the care provided. All professionals have access to their rates of adherence to the evaluation standards and can view detailed information at the patient level. The feedback system also compiles a list of patients who do not meet all established criteria of the 62 evidence-based recommendations that constitute a unique index known as the Quality of Care Standard (Estàndard de Qualitat Assistencial, EQA). At present, health professionals have point-of-care access to all of these results, with no additional data-recording effort on their part (Coma et al. 2013).
Primary health care allows longitudinal follow-up of a population. It also requires attention to multiple health conditions and disease prevention activities in a relatively brief office visit. Bodenheimer estimated that a primary care physician would require 18 h per workday to implement all of the prevention activities recommended by experts in chronic diseases (Bodenheimer 2008). Given the multitude of recommendations, it would be of great interest to health care professionals to have tools available that would help them carry out evidence-based activities. One of the strategies used to improve clinical practice is the incorporation of recommended activities as reminders in the clinical record (Jamtvedt et al. 2003).
The availability of the shared electronic health record allows electronic reminders at the point of care, which has been associated with an improvement in clinical practice, estimated at 4.2 % (interquartile range 0.8–18.8 %). This wide variability could be explained by differences in the settings where the studies were carried out, the basal levels of adherence, or the characteristics of the reminders studied (Shojania et al. 2010).
Most of the randomized clinical trials that have evaluated the effectiveness of reminders in the primary care setting have included reminders for just one disease or clinical condition (Eccles et al. 2002; Filippi et al. 2003; Flottorp et al. 2002; Hicks et al. 2008; Krall et al. 2004; Safran et al. 1995; Sequist et al. 2005; Tierney et al. 2003; van Wyk et al. 2008). The studies that have included interventions with multiple reminders have focussed on preventive measures such as vaccinations and screenings or potential adverse effects of medications (Frank et al. 2004; Tamblyn et al. 2003).
In a retrospective analysis, an association was observed between the level of adherence to the recommendations of interest and the frequency of accessing the feedback provided and the patient list. In other words, the professionals who consulted their own quality results and the status of their patient lists showed greater improvement in their clinical indicators than their colleagues who did not consult the feedback results; in addition, a temporal relationship was detected between the availability of feedback and the clinical improvement (Fina et al. 2006). Even though different clinical trials have studied the impact of paper reminders and other interventions such as economic incentive or educational support materials, there is a lack of scientific evidence on the impact of computerized reminders (Shojania et al. 2010).
In 2012, a maximum of 10 computerized reminders per patient was introduced into the ECAP system, visible to the health professional during the patient visit and related to 25 of the 62 recommendations considered as quality indicators in the EQA. We designed a clinical trial to evaluate whether adding computerized reminders to a well-established feedback system would improve adherence to multiple clinical recommendations in the primary care setting.
The hypothesis was that computerized point-of-care reminders (i.e., presented during the contact between a primary care professional and a patient) improve the level of adherence to the recommended action and reduce the time to implementation of the recommendation.
To determine the effectiveness of reminders compared to no reminders in improving adherence to multiple clinical recommendations measured as the resolution of the clinical condition that motivated the reminder, in a primary care setting with a well-established feedback system.
To describe the effectiveness of the three levels of reminders intervention: (1) pop-up reminder, (2) pop-up reminder and a calendar icon, and (3) pop-up reminder, calendar icon, and configurability (users can select how the reminders are shown).
To describe resolution and time to resolution of the reminders by type of professional, primary care team and by each of the reminders in improving adherence to the clinical recommendations contained in the reminders.
To compare the effectiveness of computerized reminders in improving adherence to multiple clinical recommendations by category (primary, secondary and tertiary prevention).
To compare the effectiveness of computerized reminders in improving adherence to multiple clinical recommendations by category (professional action only and the professional action plus the modification of patient behaviour).
The study is a controlled clinical trial, cluster-randomized by primary care team, designed to evaluate the impact of computerized reminders on adherence to clinical recommendations in a primary care setting with an established feedback system. A pilot study will conducted with primary care centre (PCC) health professionals to improve the development of the reminders. During the clinical trial, all of the study participants, including the control group, continued to receive the usual feedback. The control group is compared with each of the three study arms: (1) pop-up reminder, (2) pop-up reminder and a calendar icon, and (3) pop-up reminder, calendar icon, and configurability.
The inclusion criteria are: being a family physician or nurse who care for adult patients aged 14 years of older with clinical conditions that could generate at least one of the reminders in one of the 282 primary care teams managed by ICS. The exclusion criteria are: professionals who did not adopt the reformed model of primary care defined in Spanish legislation will be excluded due to low utilization of the electronic health record (Real Decreto 137/1984). Furthermore professionals of a primary care team participating in other clinical trials of electronic health record reminders, and those that participate in the pilot stage of this clinical trial will be excluded. It is important to emphasize that this is a global intervention involving all ICS health professionals. The analysis will take into account all included professionals in each PCC (or cluster).
During the clinical trial, all study participants, including those in the control group, will continue to receive the usual feedback established in 2006. This feedback consists of a monthly update for each family physician and primary care nurse of their percentage of adherence to each of the evidence-based recommendations in the EQA, along with a comparison of the month’s results with their own basal data, with the rest of their team, and with the mean of all ICS teams.
For ease of interpretation, the results will be color-coded by target values. For 20 of the 25 recommendations, in addition to the numerical score, professionals will be able to consult the list of patients who do not meet the target; the goal is that the professional will review and take action with those individuals. Access to the feedback screen is provided as a direct link within the electronic health record (Figs. 1, 2, in Catalan; an online demonstration of this feedback system is available at: http://www.amf-semfyc.com/sisap/) (Coma et al. 2013). The monthly results are also used as part of a performance incentive variable in the compensation model; during the intervention year, the incentive will not exceed a total of 750 euros per physician and 380 euros per nurse.
In contrast to the intervention with point-of-care reminders that will be evaluated using the present protocol, the usual monthly feedback screen (control group) does not allow direct interaction with the clinical health record system; this feedback is associated with the professional, not the individual patients, and requires that the health professional actively seek out the numerical results or the patient lists from which the established feedback screen draws the relevant data.
A maximum of 10 clinical reminders per patient will be offered, recommending actions related to at least one of the following nine clinical conditions: arterial hypertension, elevated cardiovascular risk, type 2 diabetes mellitus, cerebrovascular accident, ischemic heart disease, heart failure, atrial fibrillation, smoking habit, and hepatitis C. The reminders are integrated into the electronic health record for each patient so that they are available to any health professional who cares for the patient; the information is updated weekly to reflect any changes that occur.
Each reminder consists of the following information: an orange or red icon that codes the importance of the recommendation as moderate or high, respectively; the relevant clinical condition; the recommended action; and the date and value entered for the recommendation or related laboratory results. A recommendation is classified as highly important if treatment is required (red icon: e.g., atrial fibrillation, anticoagulant/antiplatelet treatment, 12/06/2011) and as moderately important if it refers to a variable, laboratory result, or vaccination (orange icon: e.g., Type 2 diabetes mellitus, monitor HbA1c, most recent value, 8.3, 12/05/2011) (Table 1).
Health professionals can also choose to view the reminders in their daily schedule of patients. The reminder provides a link to the patient’s electronic health record, from which the pertinent action can be taken (generate a prescription, record a vaccination, request a laboratory test, or record a clinical variable). There is also an option to ignore or exclude a recommendation for a particular patient and enter a comment about the reason for the exclusion.
The PCC will be the unit of analysis, so that all professionals from the same centre will be assigned to the same arm of the trial. Randomization will be done with the Stata/SE (v. 11.2, StataCorp) module for the design of randomized clinical trials (Ryan 1990). With the goal of achieving a balanced distribution of the quality of care by PCC in each arm, centres will be stratified by the quintiles of EQA results in the month prior to study initiation. The procedure will consist of obtaining the ICS list of PCCs, importing the data in STAT format, applying the inclusion and exclusion criteria, and adding the EQA value for each PCC. The PCCs will be allocated in randomized blocks of 6, 12, and 18 to achieve the same number of PCCs in each of the study groups (Fig. 3).
The reminders can appear in the electronic health record screens in 3 levels. Each intervention group will receive a distinct, incremental presentation of the reminders: pop-up only, pop-up plus calendar icon, and pop-up plus calendar icon, with a customizable option.
Health professionals allocated to this group will access a patient’s electronic health record; if a reminder has been generated for that patient, a pop-up window will present all of the patient reminders. No other display option will be available. The pop-up appears automatically to the professional on accessing the patient record from their calendar list of patients with an appointment. Before entering any other information they have to close the pop-up or click on the reminder to address the issue.
Pop-up + calendar icon
Each health professional has an online calendar with the list of patients scheduled for the day. The professionals allocated to this arm will be alerted to any reminders pertaining to each patient on the daily schedule. A square orange or red icon will appear next to the patient’s name, indicating the most important reminder (moderate or high) generated, with a small number showing the total number of reminders for that patient. This icon will be linked to the patient’s electronic health record with the pop-up window described for the first study arm.
This level of intervention (in addition to just the pop-up window) gets a calendar icon. As for the pop-up, health professionals sees this window every time they access the patient’s record from the appointments calendar, the content of the pop-up being the unaddressed reminders, As for the calendar icon, the professionals see them every time they navigate to the calendar list of patients with an appointment.
Configurability (pop-up + calendar icon + configuration options)
The default configuration will show all reminders generated for a specific patient. Health professionals allocated to this group will be able to modify the default configuration, adding filters to select only the reminders they want to see integrated into the electronic health record. This option will be added to the pop-up window (which will still allow access to the expanded list of reminders) and the calendar alerts (which will continue to indicate the total number of reminders generated). Screenshots and detailed description of the intervention (Fig. 4).
Length of the intervention
The trial is designed to last one year. After 6 months, the main outcome variable will be analysed and the study will be halted if there is more than 10 % difference between the control group and any of the intervention groups.
Intervention groups will receive the usual ICS training about the functionality of the electronic health record system, with some changes. The usual training is provided for regional ECAP coordinators, who then train the health care teams in the region. For the purposes of the clinical trial, training will be provided only for teams allocated to intervention groups. The directors of each of these PCCs will be contacted before the training process begins.
One 2-h session will be provided for each of the three intervention groups during the first 3 weeks of the trial, to be attended by the regional ECAP coordinators and PCC directors, representing their teams. The research team will explain to each group how the reminder system works in that specific arm of the study. The PCC directors will then be responsible for training the rest of the professional team, independently and without supervision or follow-up by the research team, who will nonetheless be available to answer questions that may arise.
Before the intervention begins, a 2-month pilot study will be conducted in two primary care teams under actual working conditions (patient situations and work setting). All professionals on these two teams will have access to all three reminder formats. In this phase, any problems will be addressed. At the end of the pilot study, an ad hoc survey will be administered to assess the participant’s’ response to the intervention, their level of satisfaction, and any additional suggestions for improving the reminder system. Throughout the intervention, the teams participating in the pilot test will continue to receive the reminders in the format of their choice but will be excluded from the analysis of the clinical trial outcomes.
The main outcome variable is the health professional’s adherence to the reminder, measured as the proportion of patients receiving the recommended attention (yes/no) and the time elapsed between the first reminder message and implementation of the recommended action (months). Reminders will be considered “resolved” when the professional has completed the recommended actions (Table 2).
Data collection and management
The ICS primary care information system (SISAP) will be used to identify patients with a health status that could be improved by the health reminders, and execute a weekly update of patient status. The variables required to execute these algorithms will be obtained from the clinical registry of electronic health records. They include patient diagnoses (by CIE-10 code), primary care visits, clinical variables, drug prescriptions, and vaccinations. In addition to the variables specific to the intervention (resolution of the reminder), other explanatory and descriptive variables will be collected (Table 3). Data for the EQR (quality of data recording) will be collected from the SIDIAP database (Sistema d’Informació per al Desenvolupament de la Investigació en Atenció Primària) (Bolíbar et al. 2012). Moreover, the Medea socioeconomic index (Domínguez-Berjón et al. 2008). The EQA, EQPF (prescription quality standard), and other variables at the level of the professional or PCC will be obtained from the ICS information system.
A 10 % difference between the proportion of resolved reminders and the assumed proportion in the intervention group of 59 % will be considered clinically relevant (Demakis et al. 2000). Assuming an alpha error of 0.01, 95 % power, and 10 % loss to follow-up, and correcting for an intraclass correlation in the PCCs of 0.05, each of the 4 study groups will require 15,950 reminders, a total of 63,800. The estimated number of reminders to be generated is shown in Table 1. Nonetheless, due to the design of the computerized tool, the need for cluster randomization, and the availability of the universal sample, it is not considered necessary to limit the study to the calculated sample size.
Due to the correlation between the reminders for the same patient, the patients assigned to the same professional, and the professionals at the same centre, hierarchical modelling will be used, allowing simultaneous estimation of a factor effect at various levels.
Resolution of the reminder
To estimate the effect of the intervention on improving the resolution of each reminder, a two-level analytical strategy will be used:
The effect of the quality variables—EQA, EQR, and EQPF—on the resolution of the reminder will be evaluated at the aggregate level: percentage of reminders resolved. The Pearson correlation coefficient and scatter plots will be used for this purpose.
Hierarchical or multilevel methods will be used to take into account the structure of the information and be able to introduce variables from different levels as adjustment variables in estimating the intervention effect. Multilevel logistic regression models will be fitted. The analytical strategy will be to estimate the intervention effect, taking into consideration the randomized professionals; next, the effect size will be estimated by introducing the quality variables from the professional level (EQPF, EQA, and EQR). Finally, we will determine whether introducing variables from the reminder level changes the intervention effect. A subgroup analysis will also be done, by type of reminder.
Time to resolution of the improvable clinical situation
Time to resolution will be estimated by the Kaplan–Meier method and comparisons will be done using the log-rank test. Multilevel survival analysis will be used to homogenize the data structure (Yau 2001). The analytical strategy is the same as described above: an initial model will be adjusted to estimate the intervention effect, introduce the professional-level variables (EQA, EQR, EQPF), and then determine whether the variables at the reminder level change the estimated intervention effect.
The level of significance will be established at 5 %. All analysis will be done using the Stata/SE software, version 11.2 (StataCorp 2012).
The study design followed ethical guidelines contained in national legislation and the international standards defined in the Declaration of Helsinki and Tokyo. The study protocol was approved by the Committee on Clinical Research Ethics of the Jordi Gol Institute on Research in Primary Care (Institut Universitàri d’Investigació en Atenció Primària, IDIAP Jordi Gol).
All data extracted from the electronic health records system to create the reminders, as well as all data collected during the study for later analysis, will be anonymized so that it will be impossible for the research team to identify individual patients of the participating health professionals. A computer algorithm will be used that shows patient identification only to the primary care professionals (physicians and nurses) responsible for the patient’s care.
Patients are only indirectly affected by the trial because the researchers are not involved in any intervention or interaction with patients, all patient data are anonymized in accordance with national data privacy laws and National Health System policy, and the intervention is based entirely on best practice recommendations drawn from existing scientific evidence, published in clinical practice guidelines, and included in the objectives of the Catalan government’s Health Plan (Pla de Salut). Indeed, if the intervention results in better adherence by health professionals in the ICS primary care network to these evidenced-based recommendations, patients will benefit from the outcomes achieved by this clinical trial.
There isn’t an independent data monitoring or trial steering committee.
The results of the present clinical trial will contribute new evidence on the impact of computerized point-of-care reminders on the adherence to clinical recommendations by primary care professionals, using an established feedback system. The study will involve almost all health professionals in the ICS primary care network and the population they serve, and therefore will be highly representative of clinical practice in the primary care setting. If the intervention is shown to be effective, the improvements should be almost immediately apparent, benefiting almost 5 million patients in the National Health System in Catalonia. Since the interventions will involve a high number of patients, small improvements in effectiveness could have a great impact on population health, according to the literature, both in terms of direct benefit and side effects. For example, increased use of antiplatelet therapy in patients with cerebrovascular disease reduces the number of coronary or cerebrovascular episodes and the associated mortality, but the number of haemorrhages also increases as a consequence of the treatment. To avoid undesirable effects to the extent possible, we selected indicators for which risk–benefit analysis has provided evidence that clearly favours the intervention.
Previously published studies have described better effectiveness when the feedback is a response to low adherence to clinical practice guidelines (Jamtvedt et al. 2003). In our study population, there is no low level of adherence to recommendations concerning the conditions we are studying, because health professionals have already been receiving performance feedback and lists of patients for 5 years during which their indicators have raised significantly. On the other hand, in contrast to many of the published studies, our design adds the reminders to an established feedback system that will be used in all arms of the study; this may lead to a smaller effect size. Holt et al. compared the use of reminders to a control group with no supporting tools (Holt et al. 2012). Therefore, we selected a 10 % intervention effect as clinically relevant.
Our study has some limitations. Some authors have suggested that “pay for performance” incentives may lead to better recording rather than better clinical practice (Bell and Levinson 2007; Petersen et al. 2006). A Cochrane review concluded that despite the increasing growth in this type of payment system, there is insufficient evidence for or against any impact on improving primary health care services (Scott et al. 2011). The compensation for objectives related to the indicators included in the intervention is low, not exceeding 750 euros per physician and 380 euros per nurse. In any case, this effect would occur in both the control group and the intervention groups; therefore, it would be unlikely to have an impact on the results that may be obtained. On the other hand, most of the selected indicators would not be susceptible to manipulation because they are based on laboratory results such as LDL cholesterol levels or HbA1c levels in patients with diabetes, or prescription data for specific drugs (e.g., antiplatelets, anticoagulants, beta-blockers, angiotensin-converting enzyme inhibitors). Therefore, it is likely that any recording changes would have real impact on health status.
Studies based on data from electronic health records can also be limited by the quality of the record, and any clinical records software such as ECAP could have a certain level of under-recording. Nonetheless, this limitation should affect both the intervention and control groups, and most of the reminders are related to health problems that are long-term indicators with a prevalence similar to what has been reported in the literature. This leads us to conclude that potential under-recording is not a major problem of the study design.
Finally, the health professionals participating in our study cannot be blinded to the intervention, which could possibly generate an information bias. Nonetheless, we selected cluster randomization because most studies of this type use this design; we decided against any design in which the same health professional would have some patient records with reminders and others without. In a study by Chambers et al. participants in one study arm received sporadic reminders about flu vaccine, others received them consistently, and a third group received no reminders (Chambers et al. 1991). The outcomes were worse for the sporadic reminders than for the control group, suggesting the hypothesis that health professionals become dependent on the reminder: if it does not appear, they may assume that the patient does not need the vaccine, which could result in a harm rather than a benefit to the patient.
Clinical Research Ethics Committee (Comité Étic d’Investigació Clínica)
electronic health record system of the Institut Català de la Salut in primary care (Estació Clínica d’Atenció Primària)
electronic health record
quality care indicator
prescription quality indicator
quality of data recording
International Classification of Diseases
Catalan Institute of Health (Institut Català de la Salut)
University Research Institute on Primary Care (Institut Universitari d’Investigació en Atenció Primària)
low density lipids
primary care centre
information system for research on primary care (Sistema de Informació pel Desenvolupament d’Investigació en Atenció Primària)
primary care database of the ICS (Sistema d’Informació dels Serveis d’Atenció Primària)
high blood pressure
angiotensin-converting enzyme inhibitors
angiotensin receptor blockers
clinical related groups
Bell C, Levinson W (2007) Pay for performance: learning about quality. CMAJ 176:1717–1719
Bodenheimer T (2008) Transforming practice. N Engl J Med 359:2086–2089
Bolíbar B, Fina F, Morros R, Garcia-Gil M, Hermosilla E, Ramos R, Rosell M, Rodríguez J, Medina M, Calero S, Prieto-Alhambra D (2012) SIDIAP database: electronic clinical records in primary care as a source of information for epidemiologic research. Med Clin 138:617–621
Chambers CV, Balaban DJ, Carlson BL, Grasberger DM (1991) The effect of microcomputer-generated reminders on influenza vaccination rates in a university-based family practice center. J Am Board Fam Pract 4:19–26
Coma E, Ferran M, Méndez L, Iglesias B, Fina F, Medina M (2013) Creation of a synthetic indicator of quality of care as a clinical management standard in primary care. SpringerPlus 2:1–11
Demakis JG, Beauchamp C, Cull WL, Denwood R, Eisen SA, Lofgren R, Nichol K, Woolliscroft J, Henderson WG (2000) Improving residents’ compliance with standards of ambulatory care: results from the VA Cooperative Study on Computerized Reminders. JAMA 284:1411–1416
Domínguez-Berjón MF, Borrell C, Cano-Serral G, Esnaola S, Nolasco A, Pasarín MI, Ramis R, Saurina C, Escolar-Pujolar A (2008) Construcción de un índice de privación a partir de datos censales en grandes ciudades españolas (Proyecto MEDEA). Gac Sanit 22:179–187
Eccles M, McColl E, Steen N, Rousseau N, Grimshaw J, Parkin D, Purves I (2002) Effect of computerised evidence based guidelines on management of asthma and angina in adults in primary care: cluster randomised controlled trial. BMJ 325:941
Filippi A, Sabatini A, Badioli L, Samani F, Mazzaglia G, Catapano A, Cricelli C (2003) Effects of an automated electronic reminder in changing the antiplatelet drug-prescribing behavior among Italian general practitioners in diabetic patients: an intervention trial. Diabetes Care 26:1497–1500
Fina F, Méndez-Boo, Leonardo, Coma E, Medina M (2010) Sistema de Información de los Servicios de Atención Primaria (SISAP). La experiencia 2006–2009 de l’Institut Català de la Salut. Revista de Innovación Sanitaria y Atención Integrada 2(1):2. http://pub.bsalut.net/risai/vol2/iss1/2/
Flottorp S, Oxman AD, Håvelsrud K, Treweek S, Herrin J (2002) Cluster randomised controlled trial of tailored interventions to improve the management of urinary tract infections in women and sore throat. BMJ 325:367
Frank O, Litt J, Beilby J (2004) Opportunistic electronic reminders: improving performance of preventive care in general practice. Aust Fam Physician 33:87–90
Guies de pràctica clínica. http://www.gencat.cat/ics/professionals/guies/mpoc/mpoc.htm
Hicks LS, Sequist TD, Ayanian JZ, Shaykevich S, Fairchild DG, Orav EJ, Bates DW (2008) Impact of computerized decision support on blood pressure management and control: a randomized controlled trial. J Gen Intern Med 23:429–441
Holt TA, Thorogood M, Griffiths F (2012) Changing clinical practice through patient specific reminders available at the time of the clinical encounter: systematic review and meta-analysis. J Gen Intern Med 27:974–984
Jamtvedt G, Young JM, Kristoffersen DT, Thomson O'Brien MA, Oxman AD (2003) Audit and feedback: effects on professional practice and health care outcomes. Cochrane Database Syst Rev (3):CD000259. doi:10.1002/14651858.CD000259
Krall MA, Traunweiser K, Towery W (2004) Effectiveness of an electronic medical record clinical quality alert prepared by off-line data analysis. Stud Health Technol Inform 107:135–139
Petersen LA, Woodard LD, Urech T, Daw C, Sookanan S (2006) Does pay-for-performance improve the quality of health care? Ann Intern Med 145:265–271
Real Decreto 137/1984, de 11 de enero, sobre estructuras básicas de salud. BOE 1 de febrero de 1984. http://www.boe.es/buscar/doc.php?id=BOE-A-1984-2574
Ryan P. RALLOC: Stata module to design randomized controlled trials. http://ideas.repec.org/c/boc/bocode/s319901.html
Safran C, Rind DM, Davis RB, Sands DZ, Caraballo E, Rippel K, Wang Q, Rury C, Makadon HJ, Cotton DJ et al (1995) A clinical trial of a knowledge-based medical record. Medinfo 8:1076–1080
Scott A, Sivey P, Ait Ouakrim D, Willenberg L, Naccarella L, Furler J, Young D (2011) The effect of financial incentives on the quality of health care provided by primary care physicians. Cochrane Database Syst Rev (9):CD008451. doi:10.1002/14651858.CD008451.pub2
Sequist TD, Gandhi TK, Karson AS, Fiskio JM, Bugbee D, Sperling M, Cook EF, Orav EJ, Fairchild DG, Bates DW (2005) A randomized trial of electronic clinical reminders to improve quality of care for diabetes and coronary artery disease. J Am Med Inform Assoc 12:431–437
Shojania K, Jennings A, Mayhew A, Ramsay C, Eccles M, Grimshaw J (2010) Effect of point-of-care computer reminders on physician behaviour: as systematic review. CMAJ 182(Suppl 5):216–225
StataCorp (2012) Stata Statistical Software: release 11.2. StataCorp LP, College Station
Tamblyn R, Huang A, Perreault R, Jacques A, Roy D, Hanley J, McLeod P, Laprise R (2003) The medical office of the 21st century (MOXXI): effectiveness of computerized decision-making support in reducing inappropriate prescribing in primary care. CMAJ 169:549–556
Tierney WM, Overhage JM, Murray MD, Harris LE, Zhou XH, Eckert GJ, Smith FE, Nienaber N, McDonald CJ, Wolinsky FD (2003) Effects of computerized guidelines for managing heart disease in primary care. J Gen Intern Med 18:967–976
van Wyk JT, van Wijk MA, Sturkenboom MC, Mosseveld M, Moorman PW, van der Lei J (2008) Electronic alerts versus on-demand decision support to improve dyslipidemia treatment: a cluster randomized controlled trial. Circulation 117:371–378
Yau KK (2001) Multilevel models for survival analysis with random effects. Biometrics 57:96–102
MM proposed the research and participated in designing the study and the feedback screens. MI participated in designing the screens and in drafting the protocol. EC and LM participated in the study design and developed the computer algorithm to generate the reminders. EH participated in the study design and manuscript preparation. CO participated in designing the reminder screens. SC participated in designing the study and the reminder screens, and assisted with manuscript preparation. JC-M collaborated in the review of literature and drafted the manuscript. All authors read and approved the final manuscript.
To the SISAP team, and to Carlos Dominguez and Celina Ribera. We also wish to acknowledge the contributions of the primary care professionals, the Catalan Health Institute, and the Management information systems. To Elaine Lilly, Ph.D., for English language revision. Translation of the article was funded by the Institut Universitari d’Investigació en Atenció Primària Jordi Gol (IDIAP Jordi Gol).
The authors declare that they have no competing interests.
The authors neither received, nor will receive, individual or collective financing of their work.