On studentized residuals in the quantile regression framework
 Edmore Ranganai^{1}Email author
Received: 14 March 2016
Accepted: 22 July 2016
Published: 2 August 2016
Abstract
Although regression quantiles (RQs) are increasingly becoming popular, they are still playing a second fiddle role to the ordinary least squares estimator like their robust counterparts due to the perceived complexity of the robust statistical methodology. In order to make them attractive to statistical practitioners, an endeavor to studentize robust estimators has been undertaken by some researchers. This paper suggests two versions of RQs studentized residual statistics, namely, internally and externally studentized versions based on the elemental set method. The more preferred externally studentized version is compared to the one based on standardized median absolute deviation (MAD) of residuals using a wellknown data set in the literature. While the MAD based outlier diagnostic seemed to be uniform and more aggressive to flagging outliers the RQ externally studentized one exhibited a dynamic pattern consistent with RQ results.
Keywords
Background
Tukey (1979) recommends that it is perfectly proper to routinely use both the ordinary least squares (OLS) and robust estimators and only examine the data more closely in case of “large” discrepancieswhatever this means (but it is widely accepted that this means that otherwise it suffices to use the OLS). However, this is rarely done as robust estimators are still playing a second fiddle role to the OLS estimator, despite their proliferation. The main reason why this status quo remains is that at the interface of statistics and its applications there are nonspecialists who find it insurmountable to deal with this vague idea of “large” discrepancies and the necessary choices of types of estimators and tuning constants involved in the robust statistical methodology. On the other hand the OLS has a clear and easy to implement methodology to conduct inference and goodness of fit analysis (including residual diagnostics). To make the robust estimators more appealing to statistical practitioners, an endeavor to studentize robust estimators has been undertaken by some researchers (see e.g. Mckean and Sheather 1991; Yohai et al. 1991). This studentization enables users to undertake pertinent statistical tests and obtain confidence intervals and critical values as well as outlier diagnosis which parallel the OLS ones.
Outliers (unusual observations in the Yspace) can adversely influence the regression model fit thereby invalidating the pertinent statistical inferences (see e.g. Rousseeuw and Leroy 2003; Barnett and Lewis 1998). The Koenker and Basset (1978) regression quantiles (RQs) are fairly robust to outliers as their influence functions are bounded in the Yspace. As a result, not only have RQs been employed as alternatives and complementary tools to the OLS estimator but also in robust outlier detection techniques (Portnoy 1991). These detection methods are based on a twofold approach, namely, the “peeling” of observations fit exactly by extreme RQs and those based on RQ computation, i.e., observations lying below the RQs hyperplanes \({{\widehat{q}}_{Y{\mathbf {x}}}}(\tau )\) and/or lying above \({{\widehat{q}}_{Y{\mathbf {x}}}}(1\tau )\) corresponding to \(\widehat{\varvec{\beta }}(\tau )\) and \(\widehat{\varvec{\beta }}(1\tau )\), \(\tau \in (0,1)\), respectively (see expression (7)) may be identified as outliers. Complemented by the ordinary least squares (OLS) one consequence of the latter approach is the Ruppert and Carroll (1980) regression trimmed mean estimator. Outliers in the Xspace are referred to as high leverage points. A worse outcome can result if outliers are further coupled with high leverage points in a data set than when either data aberration manifests alone, especially in the case of RQs. This stems from the fact that RQs are very susceptible to high leverage points since their influence functions are unbounded in the Xspace. This curtails their effectiveness to detect outliers that are also high leverage (outlierleverage) points due to the not yet so wellperceived tradeoff between the RQs high affinity for high leverage points and their exclusion of (resistance to) outliers. Studentization may be a solution as it involves incoorperating some Xinformation.
Most of the existing outlier diagnostics in the RQ framework are in relation to the global orientation (centre) of the data and not relative to each quantile level \(\tau \in (0,1)\), i.e., a conditional quantile model, \({{Q}_{YX}}(\tau )\), especially extreme ones. Very few quantile level specific diagnostics exist. One such single case outlier diagnostic in existence is based on the standardized median absolute deviation (MAD) of residuals (Huber and Ronchetti 2009). Given that it is wellknown that regression outlier diagnostics do not always agree in flagging outliers the conventionally agreed practice of employing a wide spectrum of diagnostics before the analyst arrives at a verdict cannot be exercised in the RQ framework. The focus of this paper is to contribute by adding some new outlier diagnostics to the few existing ones in the RQ framework and further bring in the OLS’s attractiveness to this framework via studentization of residual statistics. This is a convenient approach as RQs have a common link with the OLS estimator that can be fruitfully exploited. This link exists via the elemental set (ES) method (Hawkins et al. 1984). So a studentized residual statistics are suggested for RQs here based on the ES method.
An ES consists of exactly the minimum number (p) of observations to fit the regression model parameters. Such a proposal is motivated by the fact that the basic optimal solution of a linear programming (LP) problem giving a RQ coincides with the p points of an ES (see Koenker and Basset 1978, Theorem 3.1; Ranganai 2016). Applying the OLS procedure to the p ES observations yields a specific elemental regression (ER). Thus RQ leverage and residual statistics and ER ones are identical. A deterrent to employing the ES method is the possibly huge load involved in computing all the \(K=\left( {\begin{array}{c}n\\ p\end{array}}\right)\). However, the number of LP optimization solutions giving RQs is approximately equal to \(n<K\). Thus the ES approach benefits from the existence of efficient LP optimization algorithms giving RQs as solutions. Also, it is shown that the suggested RQ studentized residual statistics follow a t distribution from which a wide spectrum of cutoff values can be obtained like their OLS based counterparts. These are desirable attributes for the practitioner.

Very few RQ \(\tau\) level specific outlier diagnostics with the efficacy to deal with all outlier configurations currently exist in the literature. Therefore the conventionally accepted practice of employing a wide spectrum of diagnostics cannot be carried out in the RQ framework unless more get developed.

Use of of efficient LP algorithms lessens the possibly huge load involved in computing all the K ESs as approximately \(n<K\) RQs from the LP solutions are of interest to this study.

Ease of implementation via OLS and the existence of a wide spectrum of cutoff values from the t distribution brings in the attractive of OLS to practitioners.

There is need to develop more single case outlier diagnostics in light of the not so well perceived opposing phenomena between outlier and high leverage behaviours in outlierleverage points.

Outlierleverage points may be identified better using outlier diagnostics as the suggested studentized diagnostics have some leverage (X information) inherent in them unlike the entirely residual (Y information) based ones.
Motivated by this background, this paper suggests outlier diagnostics based on studentization and ER. The rest of the paper is organized as follows; Some OLS leverage statistics and residuals are elaborated on in the next section; RQ leverage statistics and residuals are discussed in “Regression quantiles leverage statistics and residuals” section; “Studentized residuals in the quantile regression scenario” section dwells on the construction of the suggested RQ studentized residual statistics; Applications are given in “Applications” section while conclusions are given in the last section.
Some OLS leverage statistics and residuals
Regression quantiles leverage statistics and residuals
Studentized residuals in the quantile regression scenario
Theorem 1
Under model (1) the RQ externally studentized residuals \({{\upsilon }_{(i)J_\tau }}\sim {\ }t(n2p1)\).
Proof
Let \({{\theta }_{i}}={{t}_{iJ_\tau }}\sqrt{1+hiJ_\tau },\quad i\notin J_\tau ,\) with \({{t}_{iJ_\tau }}=\frac{{{e}_{iJ_\tau }}}{{{{\widehat{\sigma }}}_{(J_\tau )}}\sqrt{1+{{h}_{iJ_\tau }}}}=\frac{{{{\widehat{\varepsilon }}}_{iJ_\tau }}}{{{{\widehat{\sigma }}}_{(J_\tau )}}}.\)
Theorem 2
Under model (1) the RQ studentized internally residuals \({{\upsilon }_{iJ_\tau }}\sim {\ }t(n2p)\).
Proof
In the next sections the flagging rate of outliers based on this cutoff value in expression (16) and the ones from (14) based on critical values of the t distribution are compared using the Hocking and Pendleton (1983) data set.
Applications
In this Section we consider the Hocking and Pendleton (1983) data set. This data set is a plausible candidate to study the efficacy of the SEPR in flagging outliers as it has various various outlier and high leverage scenarios that are both easy and challenging to deal with in the RQ framework. These include a very high leverage observation 24, an outlier in 17 and two outlierleverage points 11 and 18 with varying degrees of high leverage. Observation 24 will almost always be included in the ES corresponding to RQs due to RQs affinity for high leverage points. Thus it will often have a zero residual while observation 17 will almost always be excluded in this ES and will often have a very large residual. The challenge is on outlierleverage points 11 and 18 which will depend on the tradeoff of the two antagonistic phenomena, namely, the RQs’ affinity for leverage points versus their exclusion (resistance) to outliers.
Hocking data set diagnostics
ESs corresponding to RQs  \(\tau\)  MAD (16)  \({{\upsilon }_{(i)J_\tau }}\) (14)  

8  \(11^{\triangle }\)  16  \(18^{\triangle }\)  0.0853  None  \(17^+({1.789}^{*})\) 
8  \(11^{\triangle }\)  16  19  0.0930  None  \(17^+({2.043}^{*})\) 
8  11  19  \(24^{\times }\)  0.1232  None  \(17^+({2.406}^{*})\) 
8  12  13  \(24^{\times }\)  0.1861  \(17^+({3.645}^{**})\), \(18^{\triangle }({4.582}^{**})\)  \(17^+({1.822}^{*})\), \(18^{\triangle }({6.507}^{**})\) 
8  13  14  \(24^{\times }\)  0.2046  \(11^{\triangle }({ 4.060}^{**})\), \(17^+({5.022}^{**})\), \(18^{\triangle }({7.486}^{**})\)  \(17^+({2.460}^{*})\), \(18^{\triangle }({4.869}^{**})\) 
1  14  \(24^{\times }\)  26  0.2528  \(11^{\triangle }({ 4.060}^{**})\), \(17^+({5.022}^{**})\), \(18^{\triangle }({7.486}^{**})\)  \(17^+({2.315}^{*})\), \(18^{\triangle }({5.439}^{**})\) 
1  5  14  \(24^{\times }\)  0.2593  \(11^{\triangle }({ 4.066}^{**})\), \(17^+({5.022}^{**})\), \(18^{\triangle }({7.494}^{**})\)  \(17^+({2.315}^{*})\), \(18^{\triangle }({5.439}^{**})\) 
1  14  16  \(24^{\times }\)  0.3053  \(11^{\triangle }({ 5.495}^{**})\), \(17^+({6.099}^{**})\), \(18^{\triangle }({9.149}^{**})\)  \(17^+({1.977}^{*})\), \(18^{\triangle }({6.853}^{**})\) 
1  4  16  \(24^{\times }\)  0.3659  \(11^{\triangle }({ 6.205}^{**})\), \(17^+({6.462}^{**})\), \(18^{\triangle }({10.647}^{**})\)  \(17^+({2.246}^{*})\), \(18^{\triangle }({6.394}^{**})\) 
1  14  23  \(24^{\times }\)  0.4018  \(11^{\triangle }({ 6.205}^{**})\), \(17^+({6.462}^{**})\), \(18^{\triangle }({10.647}^{**})\)  \(17^+({2.241}^{*})\), \(18^{\triangle }({6.394}^{**})\) 
14  16  23  \(24^{\times }\)  0.4412  \(11^{\triangle }({ 6.822}^{**})\), \(17^+({6.920}^{**})\), \(18^{\triangle }({11.740}^{**})\)  \(17^+({1.871}^{*})\),\(18^{\triangle }({7.223}^{**})\) 
10  14  16  \(24^{\times }\)  0.4686  \(11^{\triangle }({ 6.602}^{**})\), \(17^+({6.437}^{**})\), \(18^{\triangle }({11.162}^{**})\)  \(17^+({2.143}^{*})\), \(18^{\triangle }({5.689}^{**})\) 
7  10  14  \(24^{\times }\)  0.5370  \(11^{\triangle }({ 6.502}^{**})\), \(17^+({6.277}^{**})\), \(18^{\triangle }({10.923}^{**})\)  \(11^{\triangle }({ 1.741}^{*})\), \(17^+({2.073}^{*})\), \(18^{\triangle }({5.689}^{**})\) 
3  9  10  \(24^{\times }\)  0.5448  \(11^{\triangle }({ 6.728}^{**})\), \(17^+({6.290}^{**})\),\(18^{\triangle }({11.073}^{**})\)  \(11^{\triangle }({ 1.741}^{*})\), \(17^+({2.073}^{*})\), \(18^{\triangle }({5.689}^{**})\) 
3  8  10  \(24^{\times }\)  0.5512  \(11^{\triangle }({ 6.728}^{**})\), \(17^+({6.290}^{**})\), \(18^{\triangle }({11.073}^{**})\)  \(17^+({1.893}^{*})\), \(18^{\triangle }({6.350}^{**})\) 
8  9  10  \(24^{\times }\)  0.6215  \(11^{\triangle }({ 7.205}^{**})\), \(17^+({6.045}^{**})\), \(18^{\triangle }({11.492}^{**})\)  \(17^+({2.013}^{*})\), \(18^{\triangle }({4.843}^{**})\) 
8  9  \(24^{\times }\)  25  0.6315  \(11^{\triangle }({ 7.205}^{**})\), \(17^+({6.045}^{**})\), \(18^{\triangle }({11.492}^{**})\)  \(11^{\triangle }({ 2.301}^{**})\), \(17^+({2.704}^{**})\), \(18^{\triangle }({2.543}^{**})\) 
9  15  \(24^{\times }\)  25  0.6839  \(11^{\triangle }({ 7.224}^{**})\), \(17^+({5.986}^{**})\), \(18^{\triangle }({11.488}^{**})\)  \(11^{\triangle }({ 2.132}^{**})\), \(17^+({2.102}^{**})\), \(18^{\triangle }({4.229}^{**})\) 
8  9  15  \(24^{\times }\)  0.7227  \(11^{\triangle }({ 7.240}^{**})\), \(17^+({5.971}^{**})\), \(18^{\triangle }({11.476}^{**})\)  \(18^{\triangle }({6.832}^{**})\) 
8  10  15  \(24^{\times }\)  0.7304  \(11^{\triangle }({ 7.240}^{**})\), \(17^+({5.971}^{**})\), \(18^{\triangle }({11.476}^{**})\)  \(18^{\triangle }({6.832}^{**})\) 
6  8  21  \(24^{\times }\)  0.7385  \(11^{\triangle }({ 7.240}^{**})\), \(17^+({5.971}^{**})\), \(18^{\triangle }({11.476}^{**})\)  \(11^{\triangle }({ 1.911}^{*})\), \(18^{\triangle }({4.510}^{**})\) 
6  21  22  \(24^{\times }\)  0.7660  \(11^{\triangle }({ 6.866}^{**})\), \(17^+({4.990}^{**})\), \(18^{\triangle }({10.409}^{**})\)  \(11^{\triangle }({ 2.236}^{*})\), \(17^+({2.020}^{*})\), \(18^{\triangle }({2.687}^{*})\) 
6  8  22  \(24^{\times }\)  0.8276  \(11^{\triangle }({ 5.260}^{**})\), \(17^+({3.564}^{**})\), \(18^{\triangle }({7.887}^{**})\)  \(11^{\triangle }({ 2.807}^{*})\), \(17^+({1.908}^{*})\), \(18^{\triangle }({2.526}^{*})\) 
2  6  8  \(24^{\times }\)  0.9549  None  \(11^{\triangle }({ 2.078}^{*})\), \(18^{\triangle }({3.067}^{**})\) 
6  8  16  \(24^{\times }\)  0.9570  \(11^{\triangle }({ 2.184}^{**})\), \(18^{\triangle }({2.897}^{**})\)  \(11^{\triangle }({ 2.078}^{*})\), \(18^{\triangle }({3.067}^{**})\) 
Remark
ESs Corresponding to RQs are the \(p=4\) observations (with zero residuals) in the basic optimal solution of LP problem (7) obtained using effeicient linear programing algorithms.
The two outlier diagnostics do not always agree as is the norm in any regression diagnosis outcome using different diagnostics. Observation 24 with the highest leverage and non outlying is never flagged at all. The major difference to note here is the uniform flagging exhibited by (16) from \(\tau ={0.2046}\) to \(\tau ={0.8276}\) and only otherwise in very extreme \(\tau\) levels. It is hard to conceive that results for below and above \(\tau ={0.50}\) are similar to this extent. This is inconsistent with the wellknown outcome of RQ results due their ability to capture the changing conditional distribution of the response variable, Y given the predictor factors, X at different quantile levels (Chamberlain 1994; Cade and Noon 2003). On the other hand criterion (14) has a dynamic pattern consistent with RQs results as expected.
Conclusion
The version of the studentized RQ predicted residuals (SEPRs) suggested here are useful and of benefit to statistical practitioners as they add to the few existing single case outlier diagnostics in the RQ scenario. Further, the methodology is easy to implement as they have cutoff values that parallel the OLS based versions. Thus they offer alternatives to nonspecialists who may fight it too hard to comprehend the robust outlier detection methodology. However, if possible these diagnostics must be used together as recommended by Tukey (1979).
Declarations
Acknowlegements
The author appreciates the Editor, the Associate Editor and the reviewers inputs which greatly improved the paper as well as the University of South Africa for funding this research.
Competing interests
The author declares that he has no competing interests.
Funding
The research was supported by the University of South Africa’s Research Department.
Open AccessThis article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made.
Authors’ Affiliations
References
 Barnett V, Lewis T (1998) Outliers in statistical data. Wiley, New YorkGoogle Scholar
 Cade BS, Noon BR (2003) A gentle introduction to quantile regression for ecologists. Front Ecol Environ 1(8):412–420View ArticleGoogle Scholar
 Chamberlain G (1994) Quantile regression, censoring, and the structure of wages. In: Sims CA (ed) Advances in econometrics: sixth world congress, vol 1. Cambridge University Press, Cambridge, UK, pp 171–209View ArticleGoogle Scholar
 Chatterjee S, Hadi AS (1988) Sensitivity analysis in linear regression. Wiley, New YorkView ArticleGoogle Scholar
 Hawkins DM, Bradu D, Kass GV (1984) Location of several outliers in multipleregression data using elemental sets. Technometrics 26(3):197–208View ArticleGoogle Scholar
 Hocking RR, Pendleton OJ (1983) The regression dilemma. Commun Stat Theory Methods 12(3):497–527View ArticleGoogle Scholar
 Huber S, Ronchetti EM (2009) Robust statistics, 2nd edn. Wiley, New JerseyView ArticleGoogle Scholar
 Koenker R (2005) Quantile regression: econometric society monographs. Cambridge University Press, New YorkView ArticleGoogle Scholar
 Koenker R, Basset G (1978) Regression quantiles. Econometrica 46(1):33–50View ArticleGoogle Scholar
 Mckean JW, Sheather SJ (1991) Small sample properties of robust analyses of linear models based on Restimates: a survey. In: Stahel W, Weisberg S (eds) Directions in robust statistics and diagnostics Part II. The IMA volumes in mathematics and its applications, vol 34. Springer, New York, pp 1–19View ArticleGoogle Scholar
 Myers RH, Montgomery DC, Vining GG, Robinson TJ (2010) Generalized linear models: with applications in engineering and the sciences. Wiley, New JerseyView ArticleGoogle Scholar
 Portnoy S (1991) Regression quantile diagnostics for multiple outliers. In: Stahel W, Weisberg S (eds) Directions in robust statistics and diagnostics Part II. The IMA volumes in mathematics and its applications, vol 34. Springer, New York, pp 145–157View ArticleGoogle Scholar
 Ranganai E (2016) Quality of fit measurement in regression quantiles: an elemental set method approach. Stat Probab Lett 111(2016):18–25View ArticleGoogle Scholar
 Rousseeuw PJ, Leroy AM (2003) Robust regression and outlier detection, 3rd edn. Wiley, New JerseyGoogle Scholar
 Rousseeuw PJ, Van Driessen K (1999) A fast algorithm for the minimum covariance determinant estimator. Technometrics 41(3):212–223View ArticleGoogle Scholar
 Ruppert D, Carroll RJ (1980) Trimmed least squares estimation in the linear model. J Am Stat Assoc 75(372):828–838View ArticleGoogle Scholar
 Yohai VJ, Stahel W, Zamar RH (1991) A procedure for robust estimation and inference in linear regression. In: Stahel W, Weisberg S (eds) Directions in robust statistics and diagnostics Part II. The IMA volumes in mathematics and its applications, vol 34. Springer, New York, pp 365–374View ArticleGoogle Scholar