A concise drug alerting rule set for Chinese hospitals and its application in computerized physician order entry (CPOE)

Abstract

Background

A minimized and concise drug alerting rule set can be effective in reducing alert fatigue.

Objectives

This study aims to develop and evaluate a concise drug alerting rule set for Chinese hospitals. The rule set covers not only western medicine, but also Chinese patent medicine that is widely used in Chinese hospitals.

Setting

A 2600-bed general hospital in China.

Methods

In order to implement the drug rule set in clinical information settings, an information model for drug rules was designed and a rule authoring tool was developed accordingly. With this authoring tool, clinical pharmacists built a computerized rule set that contains 150 most widely used and error-prone drugs. Based on this rule set, a medication-related clinical decision support application was built in CPOE. Drug alert data between 2013/12/25 and 2015/07/01 were used to evaluate the effect of the rule set.

Main outcome measure

Number of alerts, number of corrected/overridden alerts, accept/override rate.

Results

Totally 18,666 alerts were fired and 2803 alerts were overridden. Overall override rate is 15.0% (2803/18666) and accept rate is 85.0%.

Conclusions

The rule set has been well received by physicians and can be used as a preliminary medical order screening tool to reduce pharmacists’ workload. For Chinese hospitals, this rule set can serve as a starter kit for building their own pharmaceutical systems or as a reference to tier commercial rule set.

Background

Computerized physician order entry (CPOE) with medication-related clinical decision support (CDS) is an effective solution to reduce drug-related problems and pharmacist workload (Hammar et al. 2015; Claus et al. 2015). Most medication-related decision support functions, such as dosage checking and drug–drug interaction (DDI) checking, are typically implemented by a set of computerized drug alerting rules. One major problem faced by drug alerting rules is the alert fatigue (Nanji et al. 2014), which is usually caused by highly exhaustive and sensitive rules. Recent related work shows override rates can be as high as 53.6% (Nanji et al. 2014), 87.6% (Topaz et al. 2015), and 93% (Bryant et al. 2013) respectively. To address this issue, lots of work has been focused on constructing minimized and concise drug rule sets. For example, Shah et al. (2006) built a tiered medication knowledge subset from a commercial knowledge base. The subset contains clinical significant drug contraindications, and only interrupts physicians for severe alerts. Phansalkar et al. (2012) developed a minimum set of 15 high-severity, clinically significant DDIs from several commercial knowledge bases. Classen et al. (2011) identified 7 most common DDIs by reviewing multiple sources. The public DDI knowledge base SFINX (Swedish, Finnish, INteraction X-referencing) tiers DDIs according to clinical significance (A-D), which enables threshold settings for automated warnings (Andersson et al. 2015).

Aim of the study

The aim of this study to build and evaluate a concise rule set suitable for Chinese hospitals. Compared to existing related work, this rule set not only covers the western medicine, but also includes various Chinese patent medicine (CPM) that is extensively used by Chinese hospitals. For example, a typical Chinese hospital (DaYi Hospital, ShanXi Province, China) uses 1981 drugs, and 462 (23.3%) are Chinese patent medicine.

Ethical approval

This study was approved by the medical ethics committee of DaYi Hospital. All collected data have been de-identified by the information department of the hospital.

Methods

Settings and materials

DaYi Hospital was established in 2011 and is the largest general hospital (2600-bed) in ShanXi Province, China. Until 2013, all the drug checking work in DaYi was performed manually by clinical pharmacists. At the drug dispensing time, the pharmacists would inspect medication orders submitted by the physicians. Unqualified orders would be returned to physicians and recorded by the pharmacists. The recorded medication errors between 2011 and 2013 were used to analyze the most frequent and error-prone drug rules. These records are the initial resource for building the concise rule set.

In 2013, we initiated the KTP (Knowledge Translation Platform) project (Zhang et al. 2015). One of KTP’s goals is to build a medication-related CDS for CPOE, in order to help pharmacists reduce work load and assist the drug checking process. At the beginning of KTP, a preliminary question is: whether to develop own medication-related CDS or use a commercial one. Although there are already mature commercial products on the Chinese market, e.g. Wolters Kluwer/Medicom PASS (Prescription Automatic Screening System), we have our own considerations for not choosing such off-the-shelf systems. (1) Although the rule base of commercial products may be much more comprehensive and detailed, it is still necessary to tier and routinely tailor the complete rule set to suit local hospital situations. For pharmacists, there is not much workload advantage over maintaining a local-developed rule set. (2) From the perspective of the KTP project, the pharmaceutical knowledge is an inseparable part of the entire knowledge base. Inside the KTP knowledge base, there are semantic relations between drug and other medical entities. For example, many clinical rules (e.g. if [Use of Aspirin] == true || [Use of Clopidogrel] == true, recommend [INR monitor]) and clinical treatment protocols (predefined order sets or clinical pathways) involves drug entities. If using third-party products, even if the vendors open their knowledge base or provide external access interfaces, the integration and interaction between different systems (e.g. mapping of drug entities across systems) can be complex and effort-taking. Therefore, we decided to develop an own system.

Information model

To implement a computerized rule set, an information model of drug alerting rules is designed (Fig. 1). It defines 11 rule types (Table 1), including dosage (single intake), daily dosage (accumulated intake), administration route, frequency, skin test, dissolvent, dissolvent dosage, DDI, contra-indication, and prescription restriction.

Fig. 1

The Information model for drug alerting rules

Table 1 Drug alerting rule types

These rule types are designed according to pharmacists’ drug checking requirements. However, there are also other rule types, such as personalized dosing algorithms (e.g. children or elder patients with different body weights and body surface areas, or patients with renal insufficiency based on creatinine clearance). In the current development phase, we haven’t supported such rules because they require lots of patient context data, such as body weight, body surface area, Crcl rate, etc. These data mostly reside in heterogeneous formats in external systems, such as HIS (Hospital Information System), LIS (Laboratory Information System), EMR (Electronic Medical Record), etc. How to extract high-quality and well-structured data in expected formats from various sources is a non-trivial task. In the next development phase, we will try to solve this data acquisition problem and support more rule types.

Authoring tool

Based on the above information model, the database schema for drug alerting rules can be decided, and a corresponding rule authoring tool has been developed (Fig. 2). The tool was developed as a web-based application.

Fig. 2

Drug alerting rule authoring tool. a Main page for editing drug rules. The left panel is the drug list, where user can click one to edit. On the right side is the edit area, which contains three tab pages: basic info, interactions and contraindications. Basic info tab page defines basic rules such as skin test, dosage, etc. b Tab page for editing drug–drug interactions. Users can select drugs that have interactions with the current one. c Tab page for editing contraindication rules. Left panel is the context item (e.g. lab test, symptoms, vital signs, etc.) list used to define contraindicated conditions. The right side is a table of user-selected context items, and a graphical rule composer, as well as a textual rule expression editor

Results

Drug alerting rule set

Based on the recorded medication errors between 2011 and 2013, the pharmacists used the rule authoring tool to define a rule set that was able to cover the most widely used and error-prone drugs. The first version of the rule set was created in June 2013, and contained 150 drugs. The detailed rule set is provided in “Appendix”.

Medication-related CDS based on the rule set

With the rule set, a medication-related clinical decision support was developed and integrated into CPOE (Fig. 3). Reasoning of the rules is executed by a home-grown rule engine (refer to http://ktp.brahma.top/Display/TestRuleEngine, http://ktp.brahma.top/Pages/Evaluation/RuleEngine/Index.html). The CPOE was also developed by our research team, under the product name “MIAS (Medical Information Automation System)”. The interaction between CPOE and CDS was implemented by web services. Whenever the physician submits orders, CPOE will call the drug checking web service of CDS to trigger the rule engine. CDS-detected alerts are then returned to CPOE, and CPOE displays them to the physician as warnings (Fig. 3b). The physician can either cancel order submission or override the alert. All detected alerts are also sent to the notification area (Fig. 3a) for review. In exceptional cases due to patient status, physicians may state their reasons for overriding the alert. While reviewing the drug alerts, physicians can use infobutton (Fig. 3c) to retrieve related drug labels (Fig. 3d). For pharmacists, we provide a backend web portal for viewing the status (accepted or overridden) and override reason for each alert. The information flow of drug alert status is automatically directed and tracked by the system, which has greatly reduced the necessity of face-to-face communication and telephone calls between physicians and pharmacists.

Fig. 3

Medication-related clinical decision support in CPOE. a Notification area for drug alerts. User can review and process all triggered drug alerts in this area. b Drug alert message. c Infobutton for drug labels. d Retrieved drug label by Infobutton

In this system, only physicians have the right to change the status of an alert (accept or override). Pharmacist only have read-only rights for alert statuses, but they can edit (increase threshold or change rule content) or deactivate corresponding rules if they found many occurrences of an unreasonable alert.

Evaluation of the rule set in CPOE

The computerized rule set was first implemented in the inpatient CPOE on 2013/12/25 (The outpatient CPOE was provided by another vendor, and had not been integrated with our system). Until now, the system has been used in 49 inpatient departments for more than 2 years. In order to evaluate the actual effect of the rule set, system log data between 2013/12/25 and 2015/07/01 were collected. During this period, totally 68,182 inpatient visits were enrolled into the system and 2,747,140 medication orders were submitted.

For the submitted medication errors, totally 18,666 alerts were detected by the CDS, and 2803 alerts were overridden by physicians. Therefore, the overall override rate is 15.0% (2803/18,666), and accept rate is 85%. Among the 18,666 alerts, Chinese patent medicine (CPM) takes up 38.4% (7168 in 18,666).

According to Tables 2 and 3, several results caught our attention and we further analyzed these results.

Table 2 Drug alert analysis

Table 3 drug alert analysis grouped by rule types

• 1. Among the detected alerts, “daily dosage” rule type has the highest alert occurrence rate (12,212 alerts in total 18,666). We dived into the “daily dosage” alerts, and found four of the top five drugs are CPM, i.e. “Salvia TMP injection (4039 alerts)”, “Thin Chi glycopeptide injection (1050 alerts)”, “Shuxuening injection (876 alerts)” and “Fufangkushen Injection (761 alerts)”, which are responsible for the majority of “daily dosage” alerts. CPM is mostly extracted or manufactured from Chinese traditional herbs. Compared to western synthesized chemical medicine, though herbs take much longer time to take effect, they also have fewer side effects and adverse reactions. In fact, CPM usually plays an auxiliary or supportive role in treatment regimens. For this reason, some physicians relaxed their vigilance and didn’t pay enough attention when using CPM. This also explains why CPM has a noticeable percentage in all the detected alerts (38.4%).

• 2. The “dissolvent dosage” rule type has the highest override rate (67.9%). The 67.9% override rate is remarkably high compared to other rule types, which means about 2/3 “dissolvent dosage” alerts have been overridden. We consulted with the clinical pharmacists, and found many alerts were related to patients with certain conditions, e.g. renal deficiency or heart failure. For such patients, it is reasonable to use smaller dosage than required by the drug fact sheet. Such false-positive cases have added up to the overridden alerts. To address this issue, we are currently considering using more patient context data to exclude such false-positive alerts.

• 3. The “skin test” rule type has the second highest override rate (41.5%). Investigation reveals that this high override rate is caused by the discrepancy in physicians’ understanding of the “skin test” rule. In this system, the skin test rule is not designed as a mandatory requirement for the current specific patient, but a general risk reminder for nurses. That means, if there is potential allergic risk (either from medical literature or drug fact sheet) for a certain drug, physicians should set the skin test flag for corresponding medication orders. If not, the skin test rule will give an alert. When it comes to the drug administrating phase, the nurses will investigate this flag as well as patient’s specific conditions (e.g. known allergy history towards certain drugs) to judge whether skin test is needed. However, many physicians treated the “skin test” rule as patient-specific flags, i.e. if a certain drug has potential allergic risk, but the physician already knows the current patient is not allergic to this drug, he/she will not set the flag and override the skin test alert.

Besides the above analysis for certain rule types, there are also high alert occurrence and override rates for several individual drugs, which are caused by different reasons and need case-by-case investigation. Base on these periodical retrospective analyses, pharmacists can continually improve the rule set (e.g. change threshold, revise rule content, deactivate rules) to better suit clinical use.

Discussion

The primary contribution of this study is a concise drug alerting rule set oriented to Chinese hospitals. As the rule set was built based on the historical data from a large-scale (2600-bed) general hospital with high patient throughput (e.g. 68,182 inpatient visits from 2013/12/25 to 2015/07/01), the rule set should be able to reflect the medication use profile of large populations and may serve as a reference for other Chinese hospitals.

In this study, the computerized rule set can be used as a “preliminary screening tool” against physicians’ medication orders. In DaYi Hospital, pharmacists need to check 4968 medication orders per day on average, and unqualified orders have to be returned to physicians. This is a time-consuming and laborious work. With the drug alerting CDS, many potential mistakes can be ruled out before they reach the final checkpoint of pharmacists. According to the evaluation result, physicians have revised 85% of detected medication orders. In the long run, the system will not only alleviate the workload of pharmacists (many drug use errors can be revised by the physicians without pharmacists’ intervention) but also enhance the workflow efficiency (avoid the “reject-revise-resubmit” process).

This study has several limitations or arguments:

• 1. The proposed rule set is not suitable for procedural drug rules. For example, the preparation of azithromycin solution is a multi-step procedure. First, azithromycin is dissolved with sterilized water to formulate into 0.1 g/ml. Then, add it to 250–500 ml 0.9% NaCl or 5% glucose solution to get a 1.0–2.0 mg/ml concentration. This procedural logic cannot be easily represented as a single succinct dissolvent rule.

• 2. The current rule set doesn’t support complex personalized dosing algorithms. In certain contexts, such as children or elder patients with different body weights and body surface areas, or patients with renal insufficiency based on creatinine clearance, more complicated personalized dosing algorithms are needed. To support them, the information model needs further extension to represent such individualized knowledge.

• 3. DDI rule subtyping. In current system implementation, all DDI rules are treated as one rule type. However, it’s better to design more DDI sub-types in order to achieve more fine-grained alerts. For example, the SFINX project (Andersson et al. 2015) tiers DDIs according to clinical significance (A–D), which enables fine-grained threshold settings for automated warnings.

• 4. Lack of complete evaluation. In this study, the accept and override rates can be easily calculated from the log data. However, it is not so easy to calculate accuracy and specificity, which requires reviewing every overridden alert in order to identify true positives and false positives. In the future, we will build a “closed-looped” alert tracking workflow, in which the state changes (either by physicians or pharmacists) and change reasons (e.g. why physician override an alert, and why pharmacists reject overriding an alert) of each alert are tracked and logged by the system.

• 5. Use of clinically identified ADEs. ADEs (adverse drug events) are valuable data for analyzing drug use and medication-related CDS. In China, we have a multi-level ADE reporting mechanism. Level I: Physicians submit detected ADE and related clinical data (patient demographics, symptoms, drug use info, etc.) to the hospital’s pharmacy department. Level 2: Pharmacists submit confirmed ADEs to drug regulatory authorities, i.e. China SFDA (a counterpart of US FDA). Level 3: China SFDA evaluates drug risks based on nation-wide collected ADEs. Although this ADE-reporting mechanism is well designed, it’s a sad reality that it hasn’t lived up to its maximum benefit, largely due to the wide-spread under-reporting problems. Most ADE events were concealed or neglected in daily practices, and the few reported ADEs cannot be used as a solid and complete data source for analyzing physicians’ drug use and evaluating our rule set. To address this issue, we are currently cooperating with clinical pharmacists to detect unreported ADEs from clinical documents (e.g. patient daily progress notes) by natural language processing (NLP) technologies.

• 6. Coverage of the rule set. One basic assumption of this study is that drug alerts conform to Pareto-alike distribution, where small portion of drug rules accounts for the majority of alerts. As a supporting case, one US study in 2005 (Reichley et al. 2005) used a commercial drug alerting rule set. It contains 48,262 rules for 1537 drugs, but 90% of alerts are focused on 58 drugs. From their daily work experience, the pharmacists in DaYi hospital also hold the same opinion that small set of drugs generate majority of errors. However, to further verify this assumption, a further evaluation is needed to get the coverage rate of the rule set. This requires a full set for all drugs on the Chinese market, and a parallel comparison of the full set and concise set on a large-scale and long-term patient drug use data set. A coverage rate greater than 80% should be ideal. Otherwise, more rules may have to be added to the rule set.

• 7. Another problem of the rule set is how to keep up with the latest clinical evidence. Occasionally published guidelines or case reports will necessitate adding or revising rules. For example, the China SFDA (State Food and Drug Administration) periodically publish ADE (adverse drug events) reports collected all round the country. A well-maintained rule set should keep up with these public sources. Currently, our research team is developing a semi-automatic program based on NLP, which will help pharmacists extract structured contents from the public ADE reports.

Generally speaking, the overall 85.0% accept rate indicates the rule set has been well received by physicians [compared to the override rates reported in other recent studies, e.g. 53.6% (Nanji et al. 2014), 87.6% (Topaz et al. 2015)] and is effective in reducing pharmacists’ workload. Moreover, the pharmacists are continually analyzing (i.e. analyze those drug alerts with high override rates), improving (e.g. raise alert threshold to reduce false positive alerts) and expanding (i.e. add more drugs and rules) the drug rule set, which will further improve its accuracy and coverage. However, due to the various complex and individualized patient statuses, such a computerized rule set is never meant to substitute the routine work of pharmacists, but can be used an effective supportive tool.

Conclusions

In this study, a concise drug alerting rule set for Chinese hospitals was constructed by pharmacists. The case study in a Chinese hospital indicates the medication-related CDS based on the rule set has been well received by physicians. For other hospitals, they may use this rule set as a starter kit for building their own medication-related CDS systems or use it to tier commercial rule bases.

Appendix

See Tables 4 and 5.

Table 4 Drug rule set (Part 1)

Table 5 Drug rule set (Part 2: DDI pairs)