Creation of a synthetic indicator of quality of care as a clinical management standard in primary care

One of the priorities of health systems is ensuring the quality of primary care (Nietert et al. 2007). To achieve this objective, multiple indicators have been developed to assess the quality of clinical practice (McColl et al. 1998). Nonetheless, utilization of numerous partial indicators makes it difficult to achieve a global interpretation and to compare individuals as well as groups or centres. In many situations, the use of fewer but more comprehensive indicators is preferable. Along this line, some composite, or synthetic, indicators have been defined that condense the measurement of several subindicators into one value, thereby facilitating interpretation and better representing the process of healthcare delivery (Saturno 2004). The possibility of summarizing quality of care in one measurement has made these synthetic indicators very attractive for comparisons between different centres and professionals. Among other characteristics, good health indicators should be valid, evidence-based, comparable, modifiable, directed toward health problems of importance due to their severity or frequency in primary care patients, feasible to obtain without overburdening the healthcare professional and finally, they must deliver health benefits to the population with minimal secondary effects (Brown and Lilford 2006; NHS Institute for Innovation and Improvement). One of the circumstances that has especially favoured this process of constructing synthetic indicators has been the development of electronic medical records and the availability of tools to manage large databases.

In recent years there has also been increasing international interest in “pay for performance” programmes for individual health professionals (Gérvas and Pérez 2008) and in the development of evidence-based synthetic indicators to objectively measure clinical performance. In 1994 this new payment method began in Australia (Medicare Australia) and has been extended to Canada (Pink et al. 2006), the United States (Rosenthal et al. 2005), New Zealand (Perera et al. 2007) and the United Kingdom (Roland 2004; Doran et al. 2006), among others (Gérvas and Pérez 2008; Doran et al. 2008; Engels et al. 2005). Some of these systems have incorporated several different models of clinical indicators. The most well-established effort is that of the National Health Services (NHS) of the United Kingdom, which in 2004 developed the Quality Outcomes Framework (QOF), a payment system for primary care physicians. This system facilitates a single quality measurement that distributes 1000 points across four domains (Roland 2004; Doran et al. 2006; Lester and Majeed 2008). In the United States, there are also systems of clinical indicators that measure outpatient care (Rosenthal et al. 2005; The Ambulatory Care Quality Alliance Recommended Starter Set). One of these systems is the Summary Quality Index (SQUID), a composite measure of the quality of primary care that consists of 36 quality indicators related to a variety of fields and supported by a solid scientific base (Nietert et al. 2007). As has occurred in other environments, utilization of electronic medical records in primary care is widespread in Spain, but there is no global system to describe the clinical performance of healthcare professionals and centres. However, at a more local level, very useful and well-structured efforts have been observed, such as that in Tarragona province (Catalonia) which selected 10 health problems in primary care that became a standardized 17-item scale of healthcare quality (Vila et al. 2006).

The Catalan Health Institute (Institut Català de la Salut, ICS) is a public enterprise of the Generalitat de Catalunya, the Catalan government. It is the primary provider of primary care services and covers approximately 80% of the population (~5 800 000 patients). Beginning in 2005, a series of circumstances combined to encourage administrators to pay greater attention to measuring clinical quality. First, the process of computerizing medical records was practically completed. All primary care practices now routinely enter clinical data in a software application called e-CAP (Estació Clínica d’Atenció Primària), the electronic medical records system for primary care in Catalonia. Secondly, there was a growing effort to incentivize healthcare professionals within a framework of variable, evidence-based compensation known as “pay for performance” (Cortés Pérez et al. 2009; Iglesias Pérez et al. 2008). This led to a need for a standard, homogeneous quality measure for health professionals who provide patient care (Fina Avilés et al. 2010). Initially, then, the established priority was to design a synthetic indicator to base healthcare on the best available scientific knowledge and focus attention on the patient; eventually the indicator also became a tool in “pay for performance” determinations for doctors and nurses (Peiró 2004).

This article describes the construction and shows the results of the healthcare quality standard known as the EQA (Estàndard de Qualitat Assistencial, the Catalan acronym for Healthcare Quality Standard), a synthetic indicator designed to measure the quality of care provided by primary health care professionals and serve as a useful tool to improve clinical practice.

The EQA is a synthetic indicator that combines distinct clinical subindicators. It was designed between May and December 2006 and the objective was to obtain an indicator that would measure quality of care provided to a population assigned to a primary care team (PCT). Later, in 2007, a series of changes in methodology were implemented and these have remained in effect. The subindicators comprising the EQA are calculated by the Information System for Primary Care Services (SISAP), which draws information directly from the e-CAP software.

The methodology for the construction of the EQA included: 1) Establishment of the reference population and the theoretical basis for the standards; 2) Selection of health problems to be included in the EQA; 3) Definition of subindicators; 4) Selection of subindicators; 5) Deliberation about the subindicators and final construction of the EQA; and 6) Publication of EQA results.

This article describes the methods used to create the EQA, a synthetic indicator of the quality of healthcare, and the results after four years of regularly calculating and reporting the results. The characteristics of the EQA are aligned with those described as essential for good indicators (Houghton and Rouse 2004; Bell and Levinson 2007): it is simple to calculate (the subindicators are proportions); easy to interpret, both for healthcare professionals who periodically look at their results and for various levels of management; relevant in the sense that the subindicators are based on health problems that stand out in primary care; and can be validated by feedback from healthcare professionals.

Among the strengths of the EQA, which it shares with other synthetic indicators such as SQUID (Nietert et al. 2007; Roland 2004), is that it is calculated automatically by a process that takes advantage of data contained in electronic medical records. This has advantages for the speed and reliability of the calculation, and decreases any potential observer bias. In addition, calculating the EQA at the patient, professional and PCT levels permits the identification of patients who are poorly controlled as well as comparisons with other groups, both of patients and of healthcare professionals and centres.

Other efforts to create a synthetic indicator of healthcare quality have been described in the literature before, such as the process used to construct the QOF or SQUID. Although it was possible to use the structure of existing indicators, a new synthetic indicator was performed because of the local acceptability and difficulties in the calculation of some subindicators with existing data. However, these three indicators share certain characteristics, being based on clinical indicators focussed on health problems selected for their prevalence or relevance in terms of impact (Roland 2004; Doran et al. 2006), working with a 12-month timeframe, or using a common algorithm (choose at-risk patients, determine whether they meet indicator, and calculate the result). In addition, the selection of criteria for the choice of the EQA subindicators (scientific evidence, availability, feasibility, etc.) is similar to that of SQUID, and many of the same pathologies were selected in both cases (arterial hypertension, HF, IHD, diabetes, COPD, vaccination, etc.). This should come as no surprise, precisely because both indicators are based on the most prevalent pathologies in primary care.

Among these common characteristics we would highlight the feedback from healthcare professionals, which has been very positive and useful. The EQA has been and remains under continuous development. Although there have been only small modifications in the definition of the indicators, there have been numerous changes in the manner of obtaining data, especially in the first and second years, as a result of user suggestions sent by e-mail. Thanks to the publication of EQA data and to being able to verify them using the patient lists provided, healthcare professionals have served as external validators, demanding changes in the information processing that collected data from the different types of records.

Despite similarities with other synthetic indicators, there are differences as well, such as the EQA weighting of each subindicator by its importance and, especially, the concept of exclusion. In the QOF the physician can exclude those patients considered irrelevant to an indicator. For example, the professional could exclude a patient with terminal cancer from an indicator of controlled cholesterol levels (Roland 2004; Doran et al. 2006). Such exclusions are intended to avoid unnecessary treatment by doctors seeking to improve their performance results. Nonetheless, this type of exclusion also could lead to manipulation of the results: the professional could improve the results on an indicator by improper exclusion of patients who did not meet that indicator.

There are several possible ways of addressing the problem of patients for whom some indicator is contraindicated. Doran et al. (2008) points out three possible solutions: to design indicators that incorporate all possible exceptions in their calculations, to permit healthcare professionals to exclude patients, or to set performance targets below 100%. The first two options have the problem of converting the indicators into something very complicated and susceptible to fraudulent exclusions. Nonetheless, an argument in favour of patient exclusions is that if the limits for the indicators are very high, some patients may be inappropriately treated because they cannot be excluded (Doran et al. 2008). On the other hand, if the limits are too low they allow doctors to achieve the maximum score without treating all eligible patients. In our case, we did not consider the possibility of allowing healthcare professionals to exclude patients in this instance and decided to set targets below 100%, combined with a calculation of expected prevalence, to ensure a minimum denominator (Additional file 1).

Of course, the risk of underreporting some pathologies in poorly controlled patients always exists. To avoid underreporting, the QOF indicators included an audit of randomly selected doctors and of some who were suspected of fraudulent results. This is an important point because the data were entered by the doctors themselves (Doran et al. 2006). In the EQA, the information was collected directly from the medical record, so there should not be many fraudulent cases. In this sense, the development of the EQA opted for a system that could control against potential manipulation and did not demand 100% compliance with the indicator. Therefore, the concept of expected prevalence was used, which allowed us to ensure a minimum prevalence for each indicator. In addition, in order to not penalize situations such as voluntary exclusion from treatment or physician decisions based on specific cases, we determined that 80% compliance (both for expected prevalence and for resolution) would receive the total score for each subindicator. As described in the results section, the number of diagnoses (and therefore of records) has increased each year, which leads us to believe that these possible exclusions are not relevant to the EQA. Nonetheless, it could be of interest to include specific, identifiable exclusions by exploiting the electronic medical record data in these cases of evident contraindications. In this sense, we have considered incorporating these exclusions in a new version of the EQA, as long as they could be assessed in a centralized fashion.

Some authors have found that improvements in indicators linked to pay for performance were due more to increased entries in the registry than to improvements in clinical practice (Bell and Levinson 2007; Petersen et al. 2006). Nonetheless, other studies indicate that economic incentives are a potent stimulus to modify professional conduct (Gené Badia 2007), although this can only be achieved if the objectives are based on records that cannot be manipulated by the healthcare professionals. Therefore, the EQA indicators combine identification of health problems, conduct of laboratory tests, control of chronic diseases, treatments prescribed, administration of vaccines, etc. These fields are difficult to manipulate because of their relevance and variety.

Monitoring the indicators can also provoke unexpected consequences such as the deselection of patients, over-treating patients without deriving any benefit and the neglect of areas not covered because of lack of information (Kerr and Fleming 2007). In this sense, a common criticism of quality indicators linked to pay for performance is that healthcare professionals will focus exclusively on these and not pay attention to other important aspects of clinical practice (Lester and Majeed 2008; Bell and Levinson 2007). Although this outcome is difficult to assess and inherent to all indicators linked to pay for performance, more studies are needed to allow us to quantify the importance of this limitation.

Finally, it is important to point out that the EQA reflects only a small part of the work of primary care professionals, that part that can be measured. Other important but non-quantitative dimensions or pathologies such as mental illness or acute pathologies are difficult to measure and are not included in the EQA. This limitation is also found in other synthetic indicators such as the QOF (Roland 2004), where the indicators represent only part of clinical practice. The underreporting of certain conditions is therefore common to all databases but need not impede our continuing efforts to improve the registry and eventually incorporate other aspects of primary care that cannot be readily evaluated with the resources available to us today.