Skip to main content
SpringerPlus
Download PDF

Predicting the occurrence of major adverse cardiac events within 30 days of a vascular surgery: an empirical comparison of the minimum p value method and ROC curve approach using individual patient data meta-analysis

SpringerPlus volume 5, Article number: 304 (2016) Cite this article

Abstract

We aimed to compare the minimum p value method and the area under the receiver operating characteristics (ROC) curve approach to categorize continuous biomarkers for the prediction of postoperative 30-day major adverse cardiac events in noncardiac vascular surgery patients. Individual-patient data from six cohorts reporting B-type natriuretic peptide (BNP) or N-terminal pro-B-type natriuretic peptide (NTproBNP) were obtained. These biomarkers were dichotomized using the minimum p value method and compared with previously reported ROC curve-derived thresholds using logistic regression analysis. A final prediction model was developed, internally validated, and assessed for its sensitivity to clustering effects. Finally, a preoperative risk score system was proposed. Thresholds identified by the minimum p value method and ROC curve approach were 115.57 pg/ml (p < 0.001) and 116 pg/ml for BNP, and 241.7 pg/ml (p = 0.001) and 277.5 pg/ml for NTproBNP, respectively. The minimum p value thresholds were slightly stronger predictors based on our logistic regression analysis. The final model included a composite predictor of the minimum p value method’s BNP and NTproBNP thresholds [odds ratio (OR) = 8.5, p < 0.001], surgery type (OR = 2.5, p = 0.002), and diabetes (OR = 2.1, p = 0.015). Preoperative risks using the scoring system ranged from 2 to 49 %. The minimum p value method and ROC curve approach identify similar optimal thresholds. We propose to replace the revised cardiac risk index with our risk score system for individual-specific preoperative risk stratification after noncardiac nonvascular surgery.

Background

There are over 200 million individuals receiving major noncardiac surgery worldwide annually (Ford et al. 2010), many carrying a great risk of major adverse cardiovascular events (MACE), including myocardial infarction (MI) and mortality (Rodseth et al. 2011). Preoperative risk-stratification to identify high-risk patients is used to improve perioperative management. The revised cardiac risk index (RCRI) (Lee et al. 1999; Boersma et al. 2005; Fleisher et al. 2007; Poldermans et al. 2009) is the leading instrument used, however it was derived from a heterogeneous noncardiac population, and is unable to produce individual-specific risk scores (Biccard and Rodseth 2011). A systematic review of the RCRI found that prediction of cardiac events (cardiac death, MI, and nonfatal cardiac arrest) are notably less accurate for noncardiac vascular surgery patients (Ford et al. 2010).

Two hormones, B-type natriuretic peptide (BNP) and N-terminal pro-B-type natriuretic peptide (NTproBNP) have recently been investigated as prognostic biomarkers (Rajagopalan et al. 2008, 2011; Rodseth et al. 2008, 2011). They are released into the blood by ventricular cardiomyocytes when there is mechanical or ischemic strain on the atrial or ventricular wall (Rodseth et al. 2011; Harrison et al. 2002). Elevation in either of these hormone concentrations increase the risk of 30-day MACE (Rodseth et al. 2011; Karthikeyan et al. 2009). Optimal thresholds are under investigation.

Rodseth et al. conducted an individual patient data meta-analysis and proposed thresholds of 116 pg/ml for BNP and 277.5 pg/ml for NTproBNP for predicting 30-day MACE (Rodseth et al. 2011). These values were identified using the receiver operating characteristic (ROC) curve approach for categorizing continuous variables (Cook 2008; Zhu et al. 2010). There are other less commonly used discrimination methods, such as the minimum p value method, which may identify a more predictive threshold. Comparisons between these methods have yet to be investigated.

We aimed to (1) explore the BNP and NTproBNP thresholds from Rodseth et al., (2) incorporate the best performing discrimination method into a final predictive model for 30-day MACE after a noncardiac vascular surgery, and (3) propose a decision tool for clinicians to determine individual-patient risk. The secondary objectives were to explore individual prediction rules for all-cause mortality, cardiac death, and nonfatal MI within 30 days after a vascular surgery.

Results

A total of 850 participants who received noncardiac nonvascular surgery were included in our dataset. Only 75 of these individuals experienced a MACE within 30 days (9 %). Participants had an average age of 65.4 years (sd = 12.1), and approximately half were males (391/850, 46 %). Seventy-five patients experienced MACE within 30 days while the modified RCRI used by Rodseth et al. predicted 54 (72 %) of these events. Of those who experienced a MACE, 56 % had a history of coronary artery disease (42/75), 25 % had a history of diabetes mellitus (25/75), 14 % had a history of congestive heart failure (14/75), and 8 % had a history of cerebrovascular disease (8/75). The type of surgery for four participants was not specified and these individuals were not included in our univariate and multivariable analyses. None experienced a MACE within 30 days. Patient characteristics are presented in Table 1 and additional participant information can be found in Rodseth et al.’s study (Rodseth et al. 2011).

Table 1 Participant characteristics
Full size table

Minimal p value method for obtaining thresholds

We excluded 0 pg/ml as a threshold and removed 5 % of outliers from BNP and NTproBNP. The remaining 303 BNP thresholds (range 2–2322 pg/ml) and 204 NTproBNP thresholds (range 22–1572 pg/ml) were investigated. The thresholds corresponding to the minimal p values for BNP and NTproBNP were 115.57 pg/ml (p < 0.001, χ 2 = 88.79, df = 1, RR = 2.09) and 241.70 pg/ml (p = 0.001, χ 2 = 10.98, df = 1, RR = 3.33), respectively (Figs. 1 and 2). These were very close to Rodseth et al.’s ROC thresholds (BNP: 116 pg/ml, NTproBNP: 277.5 pg/ml). Our p value adjustment optimal thresholds were statistically significant with MACE (p BNP  < 0.001, p NTproBNP  < 0.001). Rodseth’s ROC curve results for BNP and NTproBNP were 116 pg/ml (66 % sensitivity, 82 % specificity) and 277.5 pg/ml, respectively (Rodseth et al. 2011). The final BNP and NTproBNP thresholds were combined to form our minimum p value method (MPM) threshold variable.

Fig. 1
figure 1

A minimum p value analysis demonstrating potential BNP threshold values and corresponding p values. This graph presents the corresponding p value of each Chi square test performed on a series of potential B-type natriuretic peptide (BNP) thresholds to predict 30-day MACE after vascular surgery. The threshold with the smallest p value is set as the optimal threshold for dichotomizing NTproBNP

Full size image
Fig. 2
figure 2

A minimum p value analysis demonstrating potential BNP threshold values and corresponding p values. This graph presents the corresponding p value of each Chi square test performed on a series of potential NTproBNP thresholds to predict 30-day MACE after vascular surgery. The threshold with the smallest p value is set as the optimal threshold for dichotomizing NTproBNP

Full size image

Comparing logistic regression methods

The comparison between the previously reported model (ROC threshold, surgery type, and diabetes mellitus) with our model MPM threshold, surgery type, diabetes mellitus) are presented in Table 2. The MPM threshold had a slightly higher odds ratio and a smaller p value than the ROC threshold, indicating a stronger association with 30-day MACE. The new model had a larger AUC and a smaller AIC, indicating a better-fit. We used MPM threshold due to its slightly better overall performance.

Table 2 Comparison of ROC and MPM thresholds using logistic regression analysis
Full size table

Development of a final prediction model

We first evaluated the impact of each predictor in Rodseth et al.’s model by individually and then collectively adding diabetes and surgery type to a model with MPM threshold. A model with only MPM threshold yielded a pMPM threshold < 0.001 and an AUC of 0.72. Diabetes was not significant when adjusting for MPM threshold (pdiabetes = 0.100, AUC = 0.74) while surgery type was (psurgerytype = 0.013, AUC = 0.76). In the full model, however, surgery type and diabetes were significant (Table 2).

We further investigated the MPM threshold with other significant predictors (p < 0.05) from our univariate analyses (Table 1): age greater than 65 years, history of coronary artery disease history, congestive cardiac failure history, diabetes mellitus, and renal insufficiency. We had a low number of observations for study 5 (n5 = 3), therefore we excluded these levels from our logistic regression analysis. The following predictors were significantly associated with 30-day MACE and were included in our final model: MPM threshold (OR = 8.5, 95 % CI 5.03–14.41, p < 0.001), surgery type (OR 2.5, 95 % CI 1.40–4.60, p = 0.002), and diabetes (OR 2.1, 95 % CI 1.11–3.84, p = 0.023). The variance inflation factors were all less than 1.2 and the tolerance levels for each explanatory variable were greater than or equal to 0.9, suggesting low multicollinearity among predictors. The final model is presented as the development model in Table 3, which also presents the results from our sensitivity and validation analysis.  

Table 3 Determining a prediction model for 30-day MACE
Full size table

Validation and sensitivity analysis

Our internal validation model identified similar estimates as our final model (MPM threshold: OR 8.6, 95 % CI 4.95–14.65, p < 0.001; surgery type: OR 2.6, 95 % CI 1.37–4.68, p = 0.004; diabetes: OR 2.1, 95 % CI 1.11–3.84, p = 0.023; AUC = 0.79). Diabetes was not a significant predictor in our mixed effects logistic regression model and our GEE (p > 0.05, α = 0.05). This indicates that some heterogeneity existed among the data. Figure 3 presents the results of each model, separated by predictor to clearly depict the variability between each model.

Fig. 3
figure 3

Forest plot of final model, internal validation model, and sensitivity models. Four models were created: (1) a final prediction model, (2) internal validation (IV) model, generalized estimating equations (GEE), and (3) mixed effects logistic regression (MELR) model with predictors MPM threshold, surgery type, and history of diabetes mellitus. This plot presents the odds ratio (OR), 95 % confidence interval (CI) and p value for each predictor of each model. MPM threshold is a composite predictor of BNP and NTproBNP thresholds determined by the minimum p value method [MPM threshold = 0 (reference) if BNP < 115.57 pg/ml or NTproBNP < 241.7 pg/ml; MPM threshold = 1 if BNP > 115.57 pg/ml or NTproBNP > 241.7 pg/ml). Surgery type is the type of noncardiac vascular surgery [infrainguinal (reference) vs. aortoiliac]. Diabetes is an indicator of whether or not an individual was diagnosed with diabetes [no (reference) vs. yes]

Full size image

Producing a points system

We created a scoring system by first assigning a 0 point to each predictor’s reference level (the least unhealthy level). B, the number of regression units needed for one point in the scoring system and also the smallest regression coefficient in the model, was 0.7262 (β minimum  = β diabetes  = 0.7262). This was used to divide and round all remaining regression coefficients to obtain their corresponding points. History of diabetes was assigned a point of 1, BNP/NTproBNP concentration values greater than the MPM threshold a point of 3, and aortoiliac surgery a point of 1. Thus, individuals scheduled for noncardiac vascular surgery would receive a total score ranging from 0 to 5, which corresponds to an associated risk ranging from 2.49 to 49.10 % (Table 4).

Table 4 Scoring system for 30-day preoperative risk
Full size table

Secondary analyses

We produced mixed effect logistic regression models for our secondary outcomes (all-cause mortality, cardiac death, and non-fatal MI) to account for the within or between study heterogeneity observed in our sensitivity analysis (Table 5). Diabetes was not significant and was removed from all three models. The MPM threshold remained a statistically significant prognostic factor for each outcome. Surgery type was not significantly associated with cardiac death. The AUCs of each model are greater than 0.7, demonstrating a moderate fit for each outcome with only MPM threshold and surgery type as predictors.

Table 5 Prediction models for all-cause mortality, cardiac death, and non-fatal myocardial infarction
Full size table

Discussion

We used data from six cohort studies (n = 832 participants) to propose an individual-specific risk prediction scoring system for 30-day MACE after noncardiac vascular surgery, based on a combined threshold factor for BNP and NTproBNP concentration levels, surgery type, and history of diabetes mellitus. The points system derived from this prediction model can allow clinicians to determine patients’ risk, ranging from 2.49 to 49.10 %, in real time. Thresholds for BNP and NTproBNP were derived using the minimum p value method, which had a slightly better performance than the ROC curve thresholds in our prediction model.

There are various methods for categorizing continuous variables (Williams et al. 2006). Some use clinically relevant thresholds which are arbitrary, and may not be representative of a predictor’s true prognostic capability. Others use data-dependent approaches such as splitting by the mean or by percentiles (25th, median, 75th). Outcome-based methods, such as the ROC curve approach and the minimum p value method, systematically identify an “optimal” threshold from a set of values, using statistical methods and clinical guidance. Previous studies have derived ROC thresholds and validated them using different approaches such as bootstrapping or leave one-out cross-validation (Nougaret et al. 2015). To our knowledge, no study has compared the ROC curve method to the minimum p value method to determine which will identify more informative thresholds. The BNP and NTproBNP thresholds themselves differ by 126.13 pg/ml (115.57 and 241.70 pg/ml, respectively) due to their composition. The half-life of NTproBNP (120 min) is six times longer than that of BNP (20 mins), making NTproBNP concentration levels higher than BNP values (Weber and Hamm 2006).

In our analysis of other predictors, surgery type was clinically important and was included in the final prediction model of 30-day MACE. It was not significant in the univariate analysis (p = 0.229), however it was significant in our multivariable model. In contrast, congestive heart failure was significant (p < 0.001) in our univariate analysis, however was excluded in the final model. Despite a small VIF (0.98), literature describes a strong association between this factor and BNP/NTproBNP concentration levels, thus to avoid potential collinearity it was excluded (Harrison et al. 2002; Maisel 2002). Our secondary analyses identified MPM threshold and surgery type as significant predictors for our secondary outcomes of all-cause mortality and non-fatal MI. For our cardiac death outcome, only MPM threshold was statistically significant. Further exploration of prognostic factors is needed to see the impact of MPM threshold on each outcome and to determine optimal prediction rules.

This study has three key strengths. First, we compared the performance of methods that are widely used for categorizing continuous predictors. Second, we present an internally-valid prediction model for 30-day MACE based on easily attainable measures and robust thresholds. Third, we describe an easy method to convert regression coefficients into risk-prediction scores, which simplifies preoperative risk classification for clinicians.

This study had several limitations. First, we used a composite predictor for BNP and NTproBNP, since no study reported both measures. NTproBNP has a more stable composition with possibly less sensitivity to sudden haemodynamic shifts, making it a potentially better biomarker than BNP for adverse outcomes (Rodseth et al. 2008; Rodseth 2009). However, limited data on NTproBNP was available (n = 218 participants), and the differing optimal thresholds derived using the minimum p value method and the ROC curve approach suggested that they were less robust than the BNP thresholds. Second, the minimum p value method is subject to inflation of Type I error rates (Faraggi and Simon 1996) however we adjusted for this using three formulae to reassess our p values. Third, in our regression analysis, we excluded the ‘not specified’ level of surgery type while Rodseth et al.’s included it (Rodseth et al. 2011). This may explain the slight discrepancy between our ROC threshold variable and their’s. We used different statistical software to run our analyses, and were unable to check the impact of excluding this level.

Future research should further investigate NTproBNP to finalize its optimal threshold, and to determine whether it outperforms BNP in risk prediction. Furthermore, a twofold cross-validation technique could be used to identify optimal thresholds with relatively accurate type I error rates and unbiased estimates of effect size (Maisel 2002; Faraggi and Simon 1996), however larger datasets are required (Faraggi and Simon 1996; Mazumdar et al. 2003). Also, comparing reclassification of patients using the RCRI and using our final prediction model would confirm which approach was more appropriate for pre-operative risk prediction among vascular surgery patients. Lastly, our proposed prediction score should be investigated in the clinical setting to evaluate its impact on patient outcomes and overall perioperative management.

Conclusions

The minimum p value method presents similar optimal thresholds as the ROC curve approach in prediction of 30-day MACE after noncardiac vascular surgery. We propose to replace the RCRI with our risk score system. Reclassification of patients using the RCRI and our final prediction model, along with further investigation of NTproBNP, should be performed to validate our thresholds and determine the most optimal model for pre-operative risk prediction among vascular surgery patients.

Methods

Study population

We used data from a previously conducted individual patient data meta-analysis (Rodseth et al. 2011). Briefly, six cohort studies on patients receiving elective noncardiac vascular surgery (n = 850) investigated associations between preoperative BNP/NTproBNP concentration levels and postoperative cardiovascular outcomes within 30-days, including all-cause mortality, cardiac death, non-fatal MI, and MACE (composite of all-cause mortality and non-fatal MI). Five studies reported BNP (n = 632 participants) (Bolliger et al. 2009; Biccard and Naidoo 2011; Cuthbertson et al. 2007; Gibson et al. 2007; Leibowitz et al. 2008), and one reported NTproBNP (n = 218 participants) (Mahla et al. 2007) concentration levels. No study reported both measurements. Participant characteristics were reported as mean and standard deviation (sd) for continuous variables, and counts and percentages for categorical variables. BNP and NTproBNP are skewed continuous variables and were reported using medians, minimum and maximum values.

Comparison of methods

ROC curve approach

The ROC curve approach is used to measure the accuracy of a test in determining a dichotomous outcome (Maisel 2002). This approach determines an optimal threshold that has the highest accuracy for the prediction of a dichotomous outcome. A graphical presentation of an ROC curve reveals 1-specificity values (false positive rate, x-axis) and the sensitivity values (true positive rate, y-axis) for each potential threshold. Previously, optimal thresholds of 116 pg/ml for BNP (66 % sensitivity, 82 % specificity) and 277.5 pg/ml for NTproBNP (unspecified sensitivity/specificity) were identified. Since no individual reported both BNP and NTproBNP measures, these thresholds were combined into one indicator variable, hereafter referred to as the ROC threshold. Investigation of other prognostic factors revealed a final logistic regression model with ROC threshold, type of surgery, and diabetes mellitus as significant prognostic factors of 30-day MACE. This model was reproduced to make appropriate comparisons between the ROC curve approach and the minimum p value method.

Minimum p value method

The minimum p value method is another method to determine critical thresholds of dichotomous outcomes. A portion of extreme values are removed and each remaining observation is assessed as a potential threshold (Harrison et al. 2002). Chi squared tests were used to test the significant difference between the two groups (low vs. high risk) for 30-day MACE at each threshold. The threshold with the smallest p value is identified as the optimal cut-point where individuals with elevated levels beyond the threshold are considered to have a high risk of 30-day MACE. The Chi square statistics (χ2), corresponding p values (p), degrees of freedom (df), and relative risk (RR) measures are reported for the final BNP and NTproBNP thresholds.

Inflation of Type 1 error caused by multiple-testing can arise and inflate the p values. We evaluated three formulae designed to reduce type I error inflation by adjusting the p values for each of the final BNP and NTproBNP thresholds (Mazumdar and Glassman 2000). The smallest p value between two of these formulae (Miller and Siegmund’s 1982; Lausen and Schumaker’s 1996) was used to determine if each threshold was statistically significant (p < 0.05). These two final cut points were combined into one indicator variable, hereafter referred to as the minimum p value method threshold (MPM threshold).

Comparing methods

We compared the odds ratios (ORs) and corresponding p values of BNP and NTproBNP thresholds using two logistic regression models, the previously reported model (ROC threshold, surgery type, and diabetes mellitus) and a new model (MPM threshold, surgery type, diabetes mellitus). A stronger association of the MPM or ROC threshold with 30-day MACE was implied by a higher OR and a smaller p value. Goodness-of-fit statistics for both models were compared, where a better-fit model had a higher area under the ROC curve (AUC) and lower Akaike information criteria (AIC). We used RStudioVersion 0.96.316 software (RStudio 2012) for the minimum p value method. All other analyses were performed in Version 9.3 of SAS (Cary, NC).

Determining a prediction model for 30-day MACE

Development of a final prediction model

The primary predictor was the optimal threshold variable (MPM or ROC). Other prognostic factors of 30-day MACE were type of surgery (infra-inguinal or aorto-iliac); patient age; and history of diabetes mellitus, congestive cardiac failure, coronary artery disease, cerebrovascular disease, and renal failure (creatinine >2 mg/dl). These were evaluated using t-tests or Chi square tests for continuous or categorical variables, respectively. Significant predictors (p < 0.05) were added to the model using a hierarchical approach. Final prognostic factors were assessed for multicollinearity with the variance inflation factor (VIF), where a VIF greater than 10 indicated high collinearity (Allison 2012).

Internal validation via bootstrapping

We produced logistic regression models for 1000 bootstrapped samples, sampled with replacement, from our dataset (Fox 2002). We averaged these models to produce a final validation model to compare the ORs, 95 % CIs, p values, and the AIC with our prediction model.

Sensitivity analysis

To test whether our final model was sensitive to potential clustering effects created from our multi-leveled data structure, we compared our estimates with a mixed effects logistic regression model and generalized estimating equations (GEE). Our mixed effects logistic regression model was comprised of fixed effects (the covariates) and one random effect (study). The error and random effect term are assumed to follow a normal distribution with a mean of zero (Heo and Leon 2005). Our GEE assumed an exchangeable correlation matrix to consider a constant correlation between responses within a study and the same correlation structure was assumed across studies (Neuhaus et al. 1991).

Producing a points system for preoperative risk of 30-day MACE

We produced a points system based on our final model. The reference level of each variable from our regression analysis was the least unhealthy state, and assigned a value of 0 (W iREF ). Each additional level outside the reference level was given a positive value of (e.g. 1, 2). These levels were referred to as W ij , for the jth level of the ith predictor variable. We then calculated points for each level using:

$$Points_{ij} = \frac{{\hat{\beta }_{i} \left( {W_{ij} - W_{iREF} } \right)}}{B},$$

where, \(\hat{\beta }_{i} ,\) is the parameter estimate for the ith predictor variable and B is the smallest parameter estimate (β min ) among all of the regression coefficients in the model. B represents the number of regression units needed for one point in the scoring system. Using an approximation of the logistic regression formula, we created point totals (rounded to the nearest whole number) and corresponding risk estimates for all possible patient profiles (Sullivan et al. 2004).

Secondary analyses

Exploratory analysis was performed to see if the same model could predict the individual components of 30-day MACE including all-cause mortality, cardiac death, and non-fatal MI.

Abbreviations

AIC:

Akaike information criteria

AUC:

area under the ROC curve (also referred to as the c-statistic)

BNP:

B-type natriuretic peptide concentration level (in pg/ml)

CI:

confidence interval

df:

degrees of freedom

GEE:

generalized estimating equations

MACE:

major adverse cardiac event

MELR:

mixed effects logistic regression

MI:

myocardial infarction

MPM threshold:

indicator variable of BNP and NTproBNP thresholds determined by the minimum p value method

NTproBNP:

N-terminal pro-B-type natriuretic peptide

OR:

odds ratio

p:

p value

RCRI:

revised cardiac risk index

ROC:

receiver operator characteristic curve

ROC threshold:

indicator variable of BNP and NTproBNP thresholds determined by the ROC curve approach

RR:

relative risk

SD:

standard deviation

VIF:

variance inflation factor

References

  • Allison PD (2012) Logistic regression using SAS: theory and application. SAS, Cary

    Google Scholar 

  • Biccard BM, Naidoo P (2011) The role of brain natriuretic peptide in prognostication and reclassification of risk in patients undergoing vascular surgery. Anaesthesia 66(5):379–385. doi:10.1111/j.1365-2044.2011.06694.x

    Article  Google Scholar 

  • Biccard BM, Rodseth RN (2011) Utility of clinical risk predictors for preoperative cardiovascular risk prediction. Br J Anaesth 107(2):133–143. doi:10.1093/bja/aer194

    Article  Google Scholar 

  • Boersma E, Kertai MD, Schouten O, Bax JJ, Noordzij P, Steyerberg EW, Schinkel AF, van Santen M, Simoons ML, Thomson IR, Klein J, van Urk H, Poldermans D (2005) Perioperative cardiovascular mortality in noncardiac surgery: validation of the Lee cardiac risk index. Am J Med 118(10):1134–1141. doi:10.1016/j.amjmed.2005.01.064

    Article  Google Scholar 

  • Bolliger D, Seeberger MD, Lurati Buse GA, Christen P, Rupinski B, Gurke L, Filipovic M (2009) A preliminary report on the prognostic significance of preoperative brain natriuretic peptide and postoperative cardiac troponin in patients undergoing major vascular surgery. Anesth Analg 108(4):1069–1075. doi:10.1213/ane.0b013e318194f3e6

    Article  Google Scholar 

  • Canadian Institute of Health Research, Natrual Sciences and Engineering Research Council of Canada, Social Sciences and Humanities Research Council of Canada (December 2014) Tri-council policy statement: ethical conduct for research involving humans. http://www.pre.ethics.gc.ca/pdf/eng/tcps2-2014/TCPS_2_FINAL_Web.pdf

  • Cook NR (2008) Statistical evaluation of prognostic versus diagnostic models: beyond the ROC curve. Clin Chem 54(1):17–23

    Article  Google Scholar 

  • Cuthbertson BH, Amiri AR, Croal BL, Rajagopalan S, Alozairi O, Brittenden J, Hillis GS (2007) Utility of B-type natriuretic peptide in predicting perioperative cardiac events in patients undergoing major non-cardiac surgery. Br J Anaesth 99(2):170–176. doi:10.1093/bja/aem158

    Article  Google Scholar 

  • Faraggi D, Simon R (1996) A simulation study of cross-validation for selecting an optimal cutpoint in univariate survival analysis. Stat Med 15(20):2203–2213. doi:10.1002/(sici)1097-0258(19961030)15:20<2203:aid-sim357>3.0.co;2-g

    Article  Google Scholar 

  • Fleisher LA, Beckman JA, Brown KA, Calkins H, Chaikof EL, Fleischmann KE, Freeman WK, Froehlich JB, Kasper EK, Kersten JR, Riegel B, Robb JF, Smith SC Jr, Jacobs AK, Adams CD, Anderson JL, Antman EM, Buller CE, Creager MA, Ettinger SM, Faxon DP, Fuster V, Halperin JL, Hiratzka LF, Hunt SA, Lytle BW, Nishimura R, Ornato JP, Page RL, Tarkington LG, Yancy CW (2007) ACC/AHA 2007 guidelines on perioperative cardiovascular evaluation and care for noncardiac surgery: executive summary: a report of the American College of Cardiology/American Heart Association Task Force on Practice Guidelines (Writing Committee to Revise the 2002 Guidelines on Perioperative Cardiovascular Evaluation for Noncardiac Surgery) Developed in Collaboration With the American Society of Echocardiography, American Society of Nuclear Cardiology, Heart Rhythm Society, Society of Cardiovascular Anesthesiologists, Society for Cardiovascular Angiography and Interventions, Society for Vascular Medicine and Biology, and Society for Vascular Surgery. J Am Coll Cardiol 50(17):1707–1732. doi:10.1016/j.jacc.2007.09.001

    Article  Google Scholar 

  • Ford MK, Beattie WS, Wijeysundera DN (2010) Systematic review: prediction of perioperative cardiac complications and mortality by the revised cardiac risk index. Ann Intern Med 152(1):26–35. doi:10.7326/0003-4819-152-1-201001050-00007

    Article  Google Scholar 

  • Fox J (2002) Bootstrapping regression models. Appendix to an R and S-plus companion to applied regression. http://cran.r-project.org/doc/contrib/Fox-Companion/appendix-bootstrapping.pdf (2012)

  • Gibson SC, Payne CJ, Byrne DS, Berry C, Dargie HJ, Kingsmore DB (2007) B-type natriuretic peptide predicts cardiac morbidity and mortality after major surgery. Br J Surg 94(7):903–909. doi:10.1002/bjs.5690

    Article  Google Scholar 

  • Harrison A, Morrison LK, Krishnaswamy P, Kazanegra R, Clopton P, Dao Q, Hlavin P, Maisel AS (2002) B-type natriuretic peptide predicts future cardiac events in patients presenting to the emergency department with dyspnea. Ann Emerg Med 39(2):131–138

    Article  Google Scholar 

  • Heo M, Leon AC (2005) Performance of a mixed effects logistic regression model for binary outcomes with unequal cluster size. J Biopharm Stat 15(3):513–526. doi:10.1081/bip-200056554

    Article  Google Scholar 

  • Karthikeyan G, Moncur RA, Levine O, Heels-Ansdell D, Chan MT, Alonso-Coello P, Yusuf S, Sessler D, Villar JC, Berwanger O, McQueen M, Mathew A, Hill S, Gibson S, Berry C, Yeh HM, Devereaux PJ (2009) Is a pre-operative brain natriuretic peptide or N-terminal pro-B-type natriuretic peptide measurement an independent predictor of adverse cardiovascular outcomes within 30 days of noncardiac surgery? A systematic review and meta-analysis of observational studies. J Am Coll Cardiol 54(17):1599–1606. doi:10.1016/j.jacc.2009.06.028

    Article  Google Scholar 

  • Lausen B, Schumacher M (1996) Evaluating the effect of optimized cutoff values in the assessment of prognostic factors. Comput Stat Data Anal 21(3):307–326. doi:10.1016/0167-9473(95)00016-X

    Article  Google Scholar 

  • Lee TH, Marcantonio ER, Mangione CM, Thomas EJ, Polanczyk CA, Cook EF, Sugarbaker DJ, Donaldson MC, Poss R, Ho KK, Ludwig LE, Pedan A, Goldman L (1999) Derivation and prospective validation of a simple index for prediction of cardiac risk of major noncardiac surgery. Circulation 100(10):1043–1049

    Article  Google Scholar 

  • Leibowitz D, Planer D, Rott D, Elitzur Y, Chajek-Shaul T, Weiss AT (2008) Brain natriuretic peptide levels predict perioperative events in cardiac patients undergoing noncardiac surgery: a prospective study. Cardiology 110(4):266–270. doi:10.1159/000112411

    Article  Google Scholar 

  • Mahla E, Baumann A, Rehak P, Watzinger N, Vicenzi MN, Maier R, Tiesenhausen K, Metzler H, Toller W (2007) N-Terminal pro-brain natriuretic peptide identifies patients at high risk for adverse cardiac outcome after vascular surgery. Anesthesiology 106(6):1088–1095. doi:10.1097/01.anes.0000267591.34626.b0

    Article  Google Scholar 

  • Maisel A (2002) B-type natriuretic peptide levels: diagnostic and prognostic in congestive heart failure: what’s next? Circulation 105(20):2328–2331

    Article  Google Scholar 

  • Mazumdar M, Glassman JR (2000) Categorizing a prognostic variable: review of methods, code for easy implementation and applications to decision-making about cancer treatments. Stat Med 19(1):113–132

    Article  Google Scholar 

  • Mazumdar M, Smith A, Bacik J (2003) Methods for categorizing a prognostic variable in a multivariable setting. Stat Med 22(4):559–571. doi:10.1002/sim.1333

    Article  Google Scholar 

  • Miller R, Siegmund D (1982) Maximally selected Chi square statistics. Biometrics 38(4):1011–1016. doi:10.2307/2529881

    Article  Google Scholar 

  • Neuhaus JM, Kalbfleisch JD, Hauck WW (1991) A comparison of cluster-specific and population-averaged approaches for analyzing correlated binary data. Int Stat Rev/Revue Internationale de Statistique 59(1):25–35. doi:10.2307/1403572

    Google Scholar 

  • Nougaret S, Reinhold C, Alsharif SS, Addley H, Arceneau J, Molinari N, Guiu B, Sala E (2015) Endometrial cancer: combined MR volumetry and diffusion-weighted imaging for assessment of myometrial and lymphovascular invasion and tumor grade. Radiology 30:141212

    Google Scholar 

  • Poldermans D, Bax JJ, Boersma E, De Hert S, Eeckhout E, Fowkes G, Gorenek B, Hennerici MG, Iung B, Kelm M, Kjeldsen KP, Kristensen SD, Lopez-Sendon J, Pelosi P, Philippe F, Pierard L, Ponikowski P, Schmid JP, Sellevold OF, Sicari R, Van den Berghe G, Vermassen F (2009) Guidelines for pre-operative cardiac risk assessment and perioperative cardiac management in non-cardiac surgery. Eur Heart J 30(22):2769–2812. doi:10.1093/eurheartj/ehp337

    Article  Google Scholar 

  • Rajagopalan S, Croal BL, Bachoo P, Hillis GS, Cuthbertson BH, Brittenden J (2008) N-Terminal pro B-type natriuretic peptide is an independent predictor of postoperative myocardial injury in patients undergoing major vascular surgery. J Vasc Surg 48(4):912–917

    Article  Google Scholar 

  • Rajagopalan S, Croal BL, Reeve J, Bachoo P, Brittenden J (2011) N-terminal pro-B-type natriuretic peptide is an independent predictor of all-cause mortality and MACE after major vascular surgery in medium-term follow-up. Eur J Vasc Endovasc Surg 41(5):657–662

    Article  Google Scholar 

  • Rodseth RN (2009) B type natriuretic peptide—a diagnostic breakthrough in peri-operative cardiac risk assessment? Anaesthesia 64(2):165–178. doi:10.1111/j.1365-2044.2008.05689.x

    Article  Google Scholar 

  • Rodseth RN, Padayachee L, Biccard BM (2008) A meta-analysis of the utility of pre-operative brain natriuretic peptide in predicting early and intermediate-term mortality and major adverse cardiac events in vascular surgical patients. Anaesthesia 63(11):1226–1233. doi:10.1111/j.1365-2044.2008.05574.x

    Article  Google Scholar 

  • Rodseth RN, Lurati Buse GA, Bolliger D, Burkhart CS, Cuthbertson BH, Gibson SC, Mahla E, Leibowitz DW, Biccard BM (2011) The predictive ability of pre-operative B-type natriuretic peptide in vascular patients for major adverse cardiac events: an individual patient data meta-analysis. J Am Coll Cardiol 58(5):522–529. doi:10.1016/j.jacc.2011.04.018

    Article  Google Scholar 

  • RStudio (2012) RStudio: integrated development environment for R 0.96, 122nd edn. RStudio, Boston

    Google Scholar 

  • Sullivan LM, Massaro JM, D’Agostino RB Sr (2004) Presentation of multivariate data for clinical use: the Framingham Study risk score functions. Stat Med 23(10):1631–1660. doi:10.1002/sim.1742

    Article  Google Scholar 

  • Weber M, Hamm C (2006) Role of B-type natriuretic peptide (BNP) and NT-proBNP in clinical routine. Heart 92(6):843–849

    Article  Google Scholar 

  • Williams BA, Mandrekar JN, Mandrekar SJ, Cha SS, Furth AF (2006) Finding optimal cutpoints for continuous covariates with binary and time-to-event outcomes. In: Technical Report Series. Division of Biostatistics, Mayo Clinic, Rochester, Minnesota

  • Zhu W, Zeng N, Wang N (2010) Sensitivity, specificity, accuracy, associated confidence interval and ROC analysis with practical SAS® implementations. In: NESUG proceedings: health care and life sciences, Baltimore, Maryland, pp 1–9

Download references

Authors’ contributions

RR, GB, DB, CB, BC, SG, EM, DL, BB acquired the data and LT, RR, TV, and GB were responsible for the conception and design of the study. TV performed the statistical analyses and drafted the manuscript with the statistical and clinical support of LT and RR. All authors read and approved the final manuscript.

Competing interests

The authors declare that they have no competing interests.

Compliance with ethical guidelines

Individual consent for secondary analysis of deidentified and anonymized patient data does not require REB approval as per Tri-Council Policy Statement (Canadian Institute of Health Research et al. December 2014).

Author information

Authors and Affiliations

  1. Department of Clinical Epidemiology and Biostatistics, McMaster University, Health Sciences Centre, Room 2C7, 1280 Main Street West, Hamilton, ON, L8S 4K1, Canada

    Thuva Vanniyasingam & Lehana Thabane

  2. Perioperative Research Unit, Department of Anaesthetics, Nelson R. Mandela School of Medicine, University of KwaZulu-Natal, Pietermaritzburg, South Africa

    Reitze N. Rodseth

  3. Department of Anaesthetics, Grey’s Hospital, Pietermaritzburg, South Africa

    Reitze N. Rodseth

  4. Department of Anaesthesia, Surgical Intensive Care, Prehospital Emergency Medicine and Pain Therapy, University Hospital Basel, Basel, Switzerland

    Giovanna A. Lurati Buse & Daniel Bolliger

  5. Department of Anaesthesiology, Kantonsspital Graubunden, Chur, Switzerland

    Christoph S. Burkhart

  6. Department of Critical Care Medicine, Sunnybrook Health Sciences Centre and University of Toronto, Toronto, ON, Canada

    Brian H. Cuthbertson

  7. Queen Elizabeth University Hospital, Glasgow, UK

    Simon C. Gibson

  8. Department of Anesthesiology and Intensive Care Medicine, Medical University of Graz, Graz, Austria

    Elisabeth Mahla

  9. Division of Cardiology, Hadassah-Hebrew University Medical Center, Jerusalem, Israel

    David W. Leibowitz

  10. Department of Anaesthesia and Perioperative Medicine, University of Cape Town, Cape Town, South Africa

    Bruce M. Biccard

  11. Population Health Research Institute, Hamilton Health Sciences, Hamilton, ON, Canada

    Lehana Thabane

  12. Biostatistics Unit, St Joseph’s Healthcare, Hamilton, ON, Canada

    Lehana Thabane

  13. Departments of Paediatrics and Anaesthesia, McMaster University, Hamilton, ON, Canada

    Lehana Thabane

  14. Centre for Evaluation of Medicine, St Joseph’s Healthcare, Hamilton, ON, Canada

    Lehana Thabane

Authors
  1. Thuva Vanniyasingam
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Reitze N. Rodseth
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Giovanna A. Lurati Buse
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Daniel Bolliger
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Christoph S. Burkhart
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Brian H. Cuthbertson
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Simon C. Gibson
    View author publications

    You can also search for this author in PubMed Google Scholar

  8. Elisabeth Mahla
    View author publications

    You can also search for this author in PubMed Google Scholar

  9. David W. Leibowitz
    View author publications

    You can also search for this author in PubMed Google Scholar

  10. Bruce M. Biccard
    View author publications

    You can also search for this author in PubMed Google Scholar

  11. Lehana Thabane
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Thuva Vanniyasingam.

Rights and permissions

Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Vanniyasingam, T., Rodseth, R.N., Lurati Buse, G.A. et al. Predicting the occurrence of major adverse cardiac events within 30 days of a vascular surgery: an empirical comparison of the minimum p value method and ROC curve approach using individual patient data meta-analysis. SpringerPlus 5, 304 (2016). https://doi.org/10.1186/s40064-016-1936-8

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1186/s40064-016-1936-8

Keywords