# Weighted *K*-means support vector machine for cancer prediction

- SungHwan Kim
^{1}Email author

**Received: **21 December 2015

**Accepted: **25 June 2016

**Published: **25 July 2016

## Abstract

To date, the support vector machine (SVM) has been widely applied to diverse bio-medical fields to address disease subtype identification and pathogenicity of genetic variants. In this paper, I propose the weighted *K*-means support vector machine (wKM-SVM) and weighted support vector machine (wSVM), for which I allow the SVM to impose weights to the loss term. Besides, I demonstrate the numerical relations between the objective function of the SVM and weights. Motivated by general ensemble techniques, which are known to improve accuracy, I directly adopt the boosting algorithm to the newly proposed weighted KM-SVM (and wSVM). For predictive performance, a range of simulation studies demonstrate that the weighted KM-SVM (and wSVM) with boosting outperforms the standard KM-SVM (and SVM) including but not limited to many popular classification rules. I applied the proposed methods to simulated data and two large-scale real applications in the TCGA pan-cancer methylation data of breast and kidney cancer. In conclusion, the weighted KM-SVM (and wSVM) increases accuracy of the classification model, and will facilitate disease diagnosis and clinical treatment decisions to benefit patients. A software package (wSVM) is publicly available at the R-project webpage (https://www.r-project.org).

### Keywords

Support vector machine*K*-means clustering Weighted SVM TCGA

## Introduction

Cutting-edge microarray and sequencing techniques for transcriptome and DNA methylome have received increasing attentions to decipher biological processes and to predict the multi-causes of complex diseases [e.g., cancer diagnosis (Ramaswamy et al. 2001), prognosis (Vijver et al. 2002), and therapeutic outcomes (Ma et al. 2004)]. To this end, the supervised machine learning has considerably contributed to developing tools towards the translational and clinical application. For example, diverse biomarker panels on the basis of transcriptional expressions have been released [e.g. MammaPrint (van ’t Veer 2002), Oncotype DX (Paik et al. 2004), Breast Cancer Index BCI (Zhang et al. 2013) and PAM50 (Parker et al. 2009)] for survival, recurrence, drug response and disease subtypes. It is evident that effective prediction tasks advance clinical diagnosis tools that build on translating models from transcriptomic studies. In this standpoint, rapid and precise classification rules are imperative to support exploring disease-related biomarkers, diagnosis and sub-types identification, and to deliver meaningful information for tailored treatment and precision medicine.

The support vector machine (SVM) was originally introduced by Cortes and Vapnik (1995). Over the decades, the SVM has been applied to a range of study fields, including pattern recognition (Kikuchia and Abeb 2005), disease subtype identification (Gould et al. 2014), pathogenicity of genetic variants (Kircher et al. 2014) and so on. In theory, the forte of the SVM is attributed to its flexibility and outstanding classification accuracy. However, the SVM relies on the quadratic programming (QP), whose computational complexity is commonly costly and subject to size of data. Some methods to circumvent this drawback (Wang and Wu 2005; Lee et al. 2007) were proposed to speed up its computation with minimizing loss of accuracy. Interestingly, Wang and Wu (2005) applied the SVM to centers of *K*-means clustering alone (KM-SVM). Due to small cluster size *K*, this method dramatically diminishes the number of observations, and hence can reduces the high-computational cost. The KM-SVM assumes that cluster centers adequately account for original data. This KM-SVM is also called the Global KM-SVM (Lee et al. 2007) in short. Similarly, Lee et al. (2007) also proposed so-called the By-class KM-SVM, where class labels separate samples into two groups at the outset, to which I apply *K*-means clustering respectively, while the Global KM-SVM, in contrast, employs a majority voting to determine class labels of respective centers. Not surprisingly, it is commonplace that the KM-SVM performs worsen than the standard SVM in most cases. In other words, the KM-SVM pursues computational efficiency at the expense of prediction accuracy.

Yang et al. (2007) and Bang and Jhun (2014) proposed the weighted support vector machine and the weighted KM-SVM to improve accuracy in the context of the outlier sensitivity problem (i.e., WSVM-outlier). The primary idea is to assign weights to each data sample, which manipulates relative importance. It is proved that WSVM-outlier reduces the effect of outliers, and yields higher classification rates. Yet I notice that the WSVM-outlier solely adopts outlier-sensitive algorithms (e.g., a robust fuzzy clustering, kernel-based possibilistic c-means), that are only well-suited to adjusting outlier effects, but not always guarantees to perform best in general cases. It is, therefore, interesting to add other weight schemes applicable to general scenarios.

Boosting is a machine learning ensemble algorithm, making it possible to reduce bias and variance, and to boost predictive power. More specifically, most boosting algorithms (Schapire 1990; Breiman 1998; Freund and Schapire 1997) iteratively glean weak classifiers, and incorporate them to a strong classifier. At each iteration, weak classifiers gain weights in some reasonable ways, and thereby subsequent weak learners focus more on samples that preceding weak learners mis-classified. Over the decades, many have introduced diverse boosting algorithms: Schapire (1990) originally proposed (a recursive majority gate formulation), and Mason et al. (2000) developed boost by majority. Interestingly, Freund and Schapire (1997) then developed AdaBoost.M1, an adaptive algorithm known to be superior to the previous ones.

Taking all things into consideration, I proposed a new algorithm, the weighted KM-SVM (wKM-SVM) and weighted support vector machine (wSVM) to improve the KM-SVM (and SVM) via weights, together with the boosting algorithm. In this paper, I utilize AdaBoost.M1 (Freund and Schapire 1997) in place of the outlier-sensitive algorithms used in WSVM-outlier (Yang et al. 2007). The wKM-SVM (wSVM) adds weights to the hinge loss term, making it straightforward to derive the quadratic programming (QP) objective function, while the WSVM-outlier, to the contrary, directly maneuvers the penalization constant corresponding to each sample. Yang et al. (2007) hardly enables to grasp how each weight is implemented in optimization, whereas my proposed wKM-SVM (wSVM) can demonstrate the numerical relationship between the objective function and weights. The weighted KM-SVM (wKM-SVM) is universally applicable to many different data analysis scenarios, for which comprehensive experiments assess accuracy and provide comparisons with other methods.

In this paper, I applied the proposed method to pan-cancer methylation data (https://tcga-data.nci.nih.gov/tcga/) including breast cancer (breast invasive carcinoma) and kidney cancer (kidney renal clear cell carcinoma). From simulations and real applications, the proposed wKM-SVM (wSVM) is shown to be more efficient in predictive power, as compared to the standard SVM and KM-SVM, including but not limited to many popular classification rules (e.g., decision trees and k-NN and so on). In conclusion, the wKM-SVM (and wSVM) increases accuracy of the classification model that will ultimately improve disease understanding and clinical treatment decisions to benefit patients.

This paper is outlined as follows. In “Backgrounds” section, I review background studies in terms of the SVM and ensemble methods. In “Proposed methods” section, the weighted SVM algorithm is proposed. In “Numerical studies” section, I compare performance of my proposed methods with other methods, and claim biological implications from analysis of the TCGA pan-cancer data. In “Conclusion and discussion” section, conclusions and further studies are discussed.

## Backgrounds

### Support vector machine

*n*th sample. \(\langle \cdot ,\cdot \rangle\) and \(\Vert \cdot \Vert\) denote the inner product and norm in \({\mathbb {R}}^m\). Define hyperplane by \(f(x_n)=\langle w,x_n\rangle +b\). A classification rule that builds on

*f*(

*x*) is

*w*and

*b*are called the weight vector and bias. The optimal vector and bias can be obtained by solving the following quadratic optimization problem,

*C*is the regularization parameter. Note that (1) can be reformulated with the Wolfe dual form by introducing the Lagrange multipliers.

*f*(

*x*) as below:

###
*K*-means SVM

The support vector machine using the *K*-means clustering (KM-SVM) is the SVM algorithm sequentially combined with the *K*-means clustering. Importantly, it is believed that the *K*-means clustering is one of the most popular clustering methods. The following describes how to implement KM-SVM. I first divide samples of train data into several clusters by applying the *K*-means clustering. Given pre-defined *K*, the *K*-means clustering produces clusters \(C_1,\ldots , C_K\). Class labels (i.e., \(-1\) or 1) of \(C_n\) are assigned via majority voting (\(1 \le n \le K\)). Second, I build up a SVM classifier over derived cluster centers. It is interesting to note that the KM-SVM greatly cut down the number of data and support vectors used to estimate solutions, and so has the forte of computational efficiency. Wang and Wu (2005) originally introduced the prototype KM-SVM (Global KM-SVM). Due to its practical utilities, diverse KM-SVM-type classification rules have been proposed afterward (Gu and Han 2013; Lee et al. 2007). In this paper, I mainly focus on the KM-SVM methods proposed by Lee et al. (2007). Wang and Wu (2005) applies the *K*-means clustering to whole input data, while Lee et al. (2007) uses the *K*-means clustering to two sample groups independently separated by each class label (By-class KM-SVM). It is known that the By-class KM-SVM improves error rates, and efficiently circumvents the problem of imbalanced class labels.

## Proposed methods

### Weighted support vector machine

*n*th sample. This penalization form with \(\lambda =\frac{1}{2C}\) is the same as

###
**Lemma 1**

*Let*\(\xi _n^* = c_n [1- y_n h(x_n;\beta ) ]_+\)

*for*\(n=1,\dots ,N\)

*and*\(c_n \ge 0\).

*Then, I get*

*subject to*\(\xi _n^* \ge 0\) and \(\xi _n^* \ge c_n [1- y_n h(x_n;\beta ) ]_+\)

*for*\(n=1,\ldots ,N\).

*The details of the proof are presented in Additional file*1.

The weighted KM-SVM (or SVM) with the boosting algorithm

1. Initialize the weight \(c_n\) with \(\frac{1}{N}\). | |

2. For \(m=1\) to M: | |

(1) Fit a KM-SVM (or SVM) \(G_m(x)\) with weights \(c_n\) to clustering centers of train data. | |

(2) Compute | |

\(err_m = \frac{\sum _{n=1}^N c_n I \big (y_n \ne G_m(x_n) \big ) }{\sum _{n=1}^N w_n }\) | |

(3) Compute \(\alpha _m = \text {log} \big ( \frac{1-err_m}{ err_m } \big )\) | |

(4) Set \(c_n \leftarrow c_n \cdot \text {exp} \big [ \alpha _m \cdot I (y_n \ne G_m(x_n) ) \big ]\) | |

3. Output \(G(x) = \text {Sign}\Big [ \sum _{m=1}^M \alpha _m G_m(x) \Big ]\). |

### Solutions for weighted SVM

*w*and

*b*, and it is enough to solve the dual problem as below:

### Weighted KM-SVM with an ensemble technique

*G*(

*x*) by integrating all weak classifiers via majority voting.

## Numerical studies

### Simulated data

*C*was chosen by 5-fold cross-validation over \(2^{-5},\ldots , 2^{5}\) from train data, and the radial based kernel was applied with \(\sigma =1\) (a.k.a. free parameter). The number of clusters (

*K*) is defined by half size of train data. Making use of the weighted KM-SVM (and SVM) fitted by the optimal parameter, I calculated error rates of test data. The experiment to generate test error rates (= error rates of test data) was repeated 1000 times and average values are presented in Fig. 1a, b. The test error rates were benchmarked to compare with other classification rules. In Fig. 1a, I first observe that the SVM (= 0.265) performs better than the KM-SVM (= 0.291) in accuracy. This is consistent with previous experimental knowledge (Lee et al. 2007). In addition, I notice that the weighted KM-SVM (= 0.278) (and SVM = 0.265) considerably improves the non-weighted KM-SVM (= 0.291) (and SVM = 0.26). Generally, the SVM is believed to be superior to many popular prediction rules. In this simulation, I consider CART (Breiman et al. 1984), kNN (Altman et al. 1992) and Random forest (Ho 1998) for comparison with the family of SVM classifiers. In Fig. 1a, the weighted SVM performs best among all of classification rules. Moreover, it is remarkable to see that the weighted KM-SVM performs better than CART and Random forest despite its data reduction. Figure 2a, b illustrate how the proposed methods reduce error rates as iterated. The test error rates dramatically drop after the first few iterations, and hence boosting evidently helps increasing accuracy. In Fig. 1b, the declining pattern of test error rates are presented as

*r*(i.e., a parameter for \(\mu\) for \(\frac{N}{2} + 1 \le n \le N\)) increases in size ranging from 0.6 to 1.4. It is clear to say that the weighted KM-SVM (and wSVM) is consistently better than the KM-SVM (and SVM).

Shown are the brief descriptions of the nineteen microarray datasets of disease-related binary phenotypes (e.g., case and control). All datasets are publicly available

Name | Study | Type | # of samples | Control | Case | # of matched genes | Reference |
---|---|---|---|---|---|---|---|

BRCA | Breast cancer | Methylation | 343 | 27 | 316 | 10,121 | The Cancer Genome Atlas (TCGA) |

KIRC | Kidney cancer | Methylation | 418 | 199 | 219 | 10,121 | The Cancer Genome Atlas (TCGA) |

### Application to genomic data

*K*) is defined by half size of train data. I examined by the test set weighted KM-SVM’s (and wSVM’s) performance using each SVM constructed by the train set. Similar to simulation studies, I observe that the weighted KM-SVM (and wSVM) outperforms the standard KM-SVM (and SVM) in prediction accuracy. It is also notable that the weighted KM-SVM (and wSVM) better performs than CART, kNN and Random Forest, as shown in Fig. 3a, b. Therefore, I conclude that the proposed weighted SVM can facilitate cancer prediction with enhanced accuracy.

## Conclusion and discussion

In this paper, I propose the new algorithm for the weighted KM-SVM to improve prediction accuracy. Typically, the KM-SVM has higher error rate than that it appears in the SVM, due to data reduction. To circumvent this issue, I suggest the weighted KM-SVM (and SVM) and evaluated performance of each of classifiers through various experimental scenarios. Putting together, I conclude that the proposed weighted KM-SVM (and SVM) is effective to diminish its error rates. In particular, I applied the weighted KM-SVM (and SVM) to TCGA cancer methylation data, and found its improved performance for disease prediction. Due to high accuracy, the weighted KM-SVM (and wSVM) can be widely used to facilitate predicting the complex diseases and therapeutic outcomes. Looking beyond this scope, this precise classification rule advances the upcoming horizon in pursuit of precision medicine, as it is urgently required in the bio-medical field to identify relations between bio-molecular units and clinical phenotype patterns (e.g., candidate biomarker detection, disease subtypes identification and associated biological pathways). The KM-SVM, however, does not involve size of clusters (i.e., the number of samples that belong to a cluster), and so clustering centers may not suitably represent original data structures. This weakness point may potentially results in poor prediction. For future work, I may suggest a new weighting scheme in proportion to size of clusters to improve more in accuracy. I leave this idea to next study.

## Declarations

### Competing interests

The author declares that he has no competing interests.

**Open Access**This article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made.

## Authors’ Affiliations

## References

- Ramaswamy S, Tamayo P, Mukherjee R, Yeang C et al (2001) Multiclass cancer diagnosis using tumor gene expression signatures. Proc Natl Acad Sci 26:15149–54View ArticleGoogle Scholar
- van de Vijver M, He Y, Dai H, Hart A et al (2002) A gene-expression signature as a predictor of survival in breast cancer. New Engl J Med 347:1999–2009View ArticleGoogle Scholar
- Ma X, Wang Z, Ryan P, Isakoff S et al (2004) A two-gene expression ratio predicts clinical outcome in breast cancer patients treated with tamoxifen. Cancer Cell 5:607–616View ArticleGoogle Scholar
- van’t Veer L, Dai H, van de Vijver M, He Y (2002) Gene expression profiling predicts clinical outcome of breast cancer. Nature 415:530–536View ArticleGoogle Scholar
- Paik S, Shak S, Tang G, Kim C et al (2004) A multigene assay to predict recurrence of tamoxifentreated, node-negative breast cancer. N Engl J Med 351:2817–2826View ArticleGoogle Scholar
- Zhang Y, Schnabel C, Schroeder B, Jerevall P et al (2013) Breast cancer index identifies early-stage estrogen receptor-positive breast cancer patients at risk for early- and late-distant recurrence. Clin Cancer Res 19:4196–4205View ArticleGoogle Scholar
- Parker J, Mullins M, Cheang M, Leung S et al (2009) Supervised risk predictor of breast cancer based on intrinsic subtypes. J Clin Oncol 27:1160–1167View ArticleGoogle Scholar
- Cortes C, Vapnik V (1995) Support-vector networks. Mach Learn 20:273–297Google Scholar
- Kikuchia T, Abeb S (2005) Comparison between error correcting output codes and fuzzy support vector machines. Pattern Recognit Lett 26:1937–1945View ArticleGoogle Scholar
- Gould C, Shepherd A, Laurens K, Cairns M et al (2014) Multivariate neuroanatomical classification of cognitive subtypes in schizophrenia: a support vector machine learning approach. Neuroimage Clin 18:229–236View ArticleGoogle Scholar
- Kircher M, Witten D, Jain P, O’Roak B et al (2014) A general framework for estimating the relative pathogenicity of human genetic variants. Nat Genet 46:310–315View ArticleGoogle Scholar
- Wang J, Wu X (2005) Support vector machines based on K-means clustering for real-time business intelligence systems. Int J Bus Intell Data Min 1, 1Google Scholar
- Lee S, Park C, Jhun M, Koo J (2007) Support vector machine using K-means clustering. J Korean Stat Soc 36:175–182Google Scholar
- Yang X, Song Q, Wang Y (2007) Support vector machine using K-means clustering. J Korean Stat Soc 21:961–976Google Scholar
- Schapire R (1990) The strength of weak learnability. Mach Learn 21:197–227Google Scholar
- Breiman R (1998) Arcing classifier (with discussion and a rejoinder by the author). Ann Stat 26:801–849View ArticleGoogle Scholar
- Freund Y, Schapire R (1997) A decision-theoretic generalization of on-line learning and an application to boosting. J Comput Syst Sci 55:119–139View ArticleGoogle Scholar
- Mason L, Baxter J, Bartlett P, Frean M (2000) Boosting algorithms as gradient descent. Adv Neural Inf Process Syst 12:512–518Google Scholar
- Bang S, Jhun M (2014) Weighted support vector machine using k-means clustering. Commun Stat Simul Comput 12:2307–2324View ArticleGoogle Scholar
- Gu Q, Han J (2013) Clustered support vector machines. In: Proceedings of the 16th international conference on artificial intelligence and statistics (AISTATS) 31Google Scholar
- Breiman L, Friedman J, Olshen R, Stone C (1984) Classification and regression trees. Wadsworth & Brooks/Cole Advanced Books & Software, MontereyGoogle Scholar
- Altman N, Friedman J, Olshen R, Stone C (1992) An introduction to kernel and nearest-neighbor nonparametric regression. Am Stat 46:175–185Google Scholar
- Ho N (1998) The random subspace method for constructing decision forests. IEEE Trans Pattern Anal Mach Intell 20:832–844View ArticleGoogle Scholar
- Wang X, Lin Y, Song C, Sibille E et al (2012) Detecting disease-associated genes with confounding variable adjustment and the impact on genomic meta-analysis: with application to major depressive disorder. BMC Bioinform 13:13–52View ArticleGoogle Scholar