This article describes the methods used to create the EQA, a synthetic indicator of the quality of healthcare, and the results after four years of regularly calculating and reporting the results. The characteristics of the EQA are aligned with those described as essential for good indicators (Houghton and Rouse 2004; Bell and Levinson 2007): it is simple to calculate (the subindicators are proportions); easy to interpret, both for healthcare professionals who periodically look at their results and for various levels of management; relevant in the sense that the subindicators are based on health problems that stand out in primary care; and can be validated by feedback from healthcare professionals.
Among the strengths of the EQA, which it shares with other synthetic indicators such as SQUID (Nietert et al. 2007; Roland 2004), is that it is calculated automatically by a process that takes advantage of data contained in electronic medical records. This has advantages for the speed and reliability of the calculation, and decreases any potential observer bias. In addition, calculating the EQA at the patient, professional and PCT levels permits the identification of patients who are poorly controlled as well as comparisons with other groups, both of patients and of healthcare professionals and centres.
Other efforts to create a synthetic indicator of healthcare quality have been described in the literature before, such as the process used to construct the QOF or SQUID. Although it was possible to use the structure of existing indicators, a new synthetic indicator was performed because of the local acceptability and difficulties in the calculation of some subindicators with existing data. However, these three indicators share certain characteristics, being based on clinical indicators focussed on health problems selected for their prevalence or relevance in terms of impact (Roland 2004; Doran et al. 2006), working with a 12-month timeframe, or using a common algorithm (choose at-risk patients, determine whether they meet indicator, and calculate the result). In addition, the selection of criteria for the choice of the EQA subindicators (scientific evidence, availability, feasibility, etc.) is similar to that of SQUID, and many of the same pathologies were selected in both cases (arterial hypertension, HF, IHD, diabetes, COPD, vaccination, etc.). This should come as no surprise, precisely because both indicators are based on the most prevalent pathologies in primary care.
Among these common characteristics we would highlight the feedback from healthcare professionals, which has been very positive and useful. The EQA has been and remains under continuous development. Although there have been only small modifications in the definition of the indicators, there have been numerous changes in the manner of obtaining data, especially in the first and second years, as a result of user suggestions sent by e-mail. Thanks to the publication of EQA data and to being able to verify them using the patient lists provided, healthcare professionals have served as external validators, demanding changes in the information processing that collected data from the different types of records.
Despite similarities with other synthetic indicators, there are differences as well, such as the EQA weighting of each subindicator by its importance and, especially, the concept of exclusion. In the QOF the physician can exclude those patients considered irrelevant to an indicator. For example, the professional could exclude a patient with terminal cancer from an indicator of controlled cholesterol levels (Roland 2004; Doran et al. 2006). Such exclusions are intended to avoid unnecessary treatment by doctors seeking to improve their performance results. Nonetheless, this type of exclusion also could lead to manipulation of the results: the professional could improve the results on an indicator by improper exclusion of patients who did not meet that indicator.
There are several possible ways of addressing the problem of patients for whom some indicator is contraindicated. Doran et al. (2008) points out three possible solutions: to design indicators that incorporate all possible exceptions in their calculations, to permit healthcare professionals to exclude patients, or to set performance targets below 100%. The first two options have the problem of converting the indicators into something very complicated and susceptible to fraudulent exclusions. Nonetheless, an argument in favour of patient exclusions is that if the limits for the indicators are very high, some patients may be inappropriately treated because they cannot be excluded (Doran et al. 2008). On the other hand, if the limits are too low they allow doctors to achieve the maximum score without treating all eligible patients. In our case, we did not consider the possibility of allowing healthcare professionals to exclude patients in this instance and decided to set targets below 100%, combined with a calculation of expected prevalence, to ensure a minimum denominator (Additional file 1).
Of course, the risk of underreporting some pathologies in poorly controlled patients always exists. To avoid underreporting, the QOF indicators included an audit of randomly selected doctors and of some who were suspected of fraudulent results. This is an important point because the data were entered by the doctors themselves (Doran et al. 2006). In the EQA, the information was collected directly from the medical record, so there should not be many fraudulent cases. In this sense, the development of the EQA opted for a system that could control against potential manipulation and did not demand 100% compliance with the indicator. Therefore, the concept of expected prevalence was used, which allowed us to ensure a minimum prevalence for each indicator. In addition, in order to not penalize situations such as voluntary exclusion from treatment or physician decisions based on specific cases, we determined that 80% compliance (both for expected prevalence and for resolution) would receive the total score for each subindicator. As described in the results section, the number of diagnoses (and therefore of records) has increased each year, which leads us to believe that these possible exclusions are not relevant to the EQA. Nonetheless, it could be of interest to include specific, identifiable exclusions by exploiting the electronic medical record data in these cases of evident contraindications. In this sense, we have considered incorporating these exclusions in a new version of the EQA, as long as they could be assessed in a centralized fashion.
Some authors have found that improvements in indicators linked to pay for performance were due more to increased entries in the registry than to improvements in clinical practice (Bell and Levinson 2007; Petersen et al. 2006). Nonetheless, other studies indicate that economic incentives are a potent stimulus to modify professional conduct (Gené Badia 2007), although this can only be achieved if the objectives are based on records that cannot be manipulated by the healthcare professionals. Therefore, the EQA indicators combine identification of health problems, conduct of laboratory tests, control of chronic diseases, treatments prescribed, administration of vaccines, etc. These fields are difficult to manipulate because of their relevance and variety.
Monitoring the indicators can also provoke unexpected consequences such as the deselection of patients, over-treating patients without deriving any benefit and the neglect of areas not covered because of lack of information (Kerr and Fleming 2007). In this sense, a common criticism of quality indicators linked to pay for performance is that healthcare professionals will focus exclusively on these and not pay attention to other important aspects of clinical practice (Lester and Majeed 2008; Bell and Levinson 2007). Although this outcome is difficult to assess and inherent to all indicators linked to pay for performance, more studies are needed to allow us to quantify the importance of this limitation.
Finally, it is important to point out that the EQA reflects only a small part of the work of primary care professionals, that part that can be measured. Other important but non-quantitative dimensions or pathologies such as mental illness or acute pathologies are difficult to measure and are not included in the EQA. This limitation is also found in other synthetic indicators such as the QOF (Roland 2004), where the indicators represent only part of clinical practice. The underreporting of certain conditions is therefore common to all databases but need not impede our continuing efforts to improve the registry and eventually incorporate other aspects of primary care that cannot be readily evaluated with the resources available to us today.