- Research
- Open Access

# Unsupervised background-constrained tank segmentation of infrared images in complex background based on the Otsu method

- Yulong Zhou
^{1, 2}Email author, - Min Gao
^{1}, - Dan Fang
^{1}and - Baoquan Zhang
^{3}

**Received:**10 May 2016**Accepted:**17 August 2016**Published:**24 August 2016

## Abstract

In an effort to implement fast and effective tank segmentation from infrared images in complex background, the threshold of the maximum between-class variance method (i.e., the Otsu method) is analyzed and the working mechanism of the Otsu method is discussed. Subsequently, a fast and effective method for tank segmentation from infrared images in complex background is proposed based on the Otsu method via constraining the complex background of the image. Considering the complexity of background, the original image is firstly divided into three classes of target region, middle background and lower background via maximizing the sum of their between-class variances. Then, the unsupervised background constraint is implemented based on the within-class variance of target region and hence the original image can be simplified. Finally, the Otsu method is applied to simplified image for threshold selection. Experimental results on a variety of tank infrared images (880 × 480 pixels) in complex background demonstrate that the proposed method enjoys better segmentation performance and even could be comparative with the manual segmentation in segmented results. In addition, its average running time is only 9.22 ms, implying the new method with good performance in real time processing.

## Keywords

- Infrared images in complex background
- Tank segmentation
- The Otsu method
- Threshold analysis
- Background constraint

## Background

In military fields, forward-looking infrared (FLIR) systems of long wave infrared (LWIR) are widely used to improve the night fighting capability, such as missile guidance and visual supervision systems. Accordingly, infrared image processing techniques, such as infrared image segmentation, target recognition, target tracking and so on, are paid much more and more attentions by lots of researchers and have become one of hot spots in present research (Fan et al. 2011).

During infrared image processing techniques, infrared image segmentation is a basic preprocessing step in infrared image analysis and computer vision (Sen and Pal 2010; Huang and Wang 2009). It intends to separate an object from a background based on some pertinent characteristics such as gray level, gray gradient, texture and location (Tao et al. 2008). Up to now, there have been many effective segmentation algorithms proposed by researchers. However, when the background of an infrared image is complex, the conventional algorithms will tend to be poor and even fail in segmentation. To solve this problem, lots of novel algorithms are proposed to improve segmentation accuracy. But the performance improvement of the approach is always at the cost of huge increase of computing time, such as the 2-D Otsu method, the 2-D maximum entropy method and so on, resulting that the improved method hardly meet the requirement of the real time processing (Guo et al. 2006).

Among the existing segmentation techniques, thresholding is one of the most popular approaches in terms of simplicity, robustness and accuracy (Li et al. 2011). Implicit assumption in image thresholding is that target and background have distinctive gray levels. Thresholding serves a variety of applications, such as biomedical image analysis, character identification and industrial inspection. Many thresholding approaches and their improvements have been developed over the last few years (Long et al. 2012; Liu and Jin 2013; Wu et al. 2010; Li and Tian 2009; Qiao et al. 2013). The most popular thresholding algorithms include the maximum between-class variance method (namely the Otsu algorithm), the entropy based thresholding algorithm, the minimum error method, the paragenetic matrix method, the moments method, the probability relaxation method and so on. Of these methods, the maximum between-class variance method proposed by Otsu was widely used for its simple calculation and good self-adaptive ability (Hu et al. 2010; Zhang et al. 2006). It is based on the single order statistical characteristics of gray histogram, and hence possesses rapid calculation and real-time processing advantages. But to different kinds of infrared images, the Otsu algorithm can’t always produce the results we wanted. Especially when the image background is very complex and the ratio of target to background is very small, the Otsu method will be poor and even fail to segment the area of our interest from the background. So in this paper, with fully considering the real-time processing requirement of infrared imaging guidance system, the threshold obtained by Otsu method will be examined for tank infrared images in complex background and the essential reason of the algorithm failing in tank image segmentation will be found out. Then the improved method will be proposed to solve the exposed problem.

## The Otsu method and its threshold analysis

### The Otsu method

The maximum between-class variance method, namely the Otsu method, was proposed by Otsu, which is based on single dimension gray histogram of the image and makes maximizing the variance between classes of background and target regions as threshold selection criterion. Suppose that the image pixels are divided into two classes of \(C_{0}\) and \(C_{1}\) by gray value *T*. Let \(C_{0} = [1, \ldots ,T]\) and \(C_{1} = [T + 1, \ldots ,L - 1]\). Let \(P_{0} (T)\) and \(P_{1} (T)\) respectively denote the probabilities of \(C_{0}\) and \(C_{1}\). Correspondingly, \(\mu_{0} (T)\) and \(\mu_{1} (T)\) are the gray means, and \(\sigma_{0}^{2} (T)\) and \(\sigma_{1}^{2} (T)\) are the variances of two classes, respectively. Then some important calculations are as follows (Otsu 1979).

Otsu pointed out that the two selection criteria of Eqs. (4) and (5) are equivalent, and hence they will possess the same threshold value (Otsu 1979).

### Threshold analysis of the Otsu method

For the sake of description convenience, the manual segmentation result is regarded as the optimal result for infrared image, and the corresponding threshold will be treated as the optimal threshold. Figure 1c is the segmented result by the Otsu method, which shows that there are lots of background pixels in segmented image and the targets completely overlap with the background. As a result, the visual effect of the segmented result is poor, implying the failure of Otsu method in segmenting the area of interest from infrared images in complex background.

Furthermore, the working mechanism of the Otsu method can also be presented. The small difference between WCVs of the background and the target regions is necessary for the Otsu method to produce satisfying segmented results. Therefore, to improve the Otsu method segmentation ability for the small target infrared images with complex background, the WCV of the background region must be lowered to get close to that of the target region. Subsequently, a new scheme to eliminate the negative influence of the complex background on the Otsu method has been designed and hence an unsupervised background constrained thresholding method is proposed based on the Otsu method in the following content.

## Unsupervised background constrained thresholding method based on the Otsu method

A new scheme will be tested to eliminate the negative influence of the complex background on the Otsu method in this section so as to improve the performance of the Otsu method. The scheme first divides the infrared image into three classes of target region, middle background and lower background via the way of maximizing the between-class variance (BCV)s of three classes, then finds the lowest bound of the gray levels for the infrared image based on the within-class variance (WCV) of the target region and hence simplifies the original infrared image by background constraint with the gray level range, finally applies the Otsu method to the simplified image, thus forming an unsupervised background constrained thresholding algorithm based on the Otsu method with better segmentation performance.

### The classes division of an infrared image with complex background

Based on the analysis result in “Threshold analysis of the Otsu method”, one can conclude that the Otsu method will not get satisfactory segmentation until the WCV of the background gets close to that of the target region. For the infrared images in complex background, the range of gray levels is very big. That is to say, the background is rich in varieties of gray levels including not only the lower gray levels, but also the higher gray levels. So it will not be reasonable to simply divide the image into two classes comprised of background and target regions. Otherwise, there will be more background pixels to be misclassified into the target region, resulting that the Otsu method fails in segmentation. So if the infrared image with complex background is divided into three classes comprised of the target region, the middle background and the lower background by the Otsu method via the image statistical characteristics, it will be more reasonable to reflect the reality of the image and hence there will be more background pixels classified into the class of middle background and less ones misclassified into the class of target region.

- 1.
Compute the mean of the image \(\mu\) with Eq. (1).

- 2.Define two gray levelswhere$$\left\{ {\begin{array}{*{20}c} {T_{1} = \mu - i} \\ {T_{2} = \mu + i} \\ \end{array} } \right.\quad i \ge 1\;{\text{and}}\;i \le \hbox{min} (\mu ,255 - \mu ),$$(6)
*i*is a parameter and its value can be automatically determined by maximizing the proposed criterion in Eq. (11).*T*_{1}and*T*_{2}are the upper and lower bounds for lower background and target region, respectively. - 3.
Find a reasonable value for parameter

*i*via the statistical characteristics of the image. Determining the upper and lower bounds needs choosing a reasonable value for parameter*i*, which in turn involves a statistical criterion to be defined.

*I*is divided into three classes by the two gray levels

*t*

_{1}and

*t*

_{2}, where

*t*

_{1}<

*t*

_{2}, with three classes represented by \(C_{l}\), \(C_{m}\) and \(C_{o}\), where \(C_{l}\) is the lower background class with gray levels \([0, \ldots t_{1} ]\), \(C_{m}\) the middle background class with levels \([t_{1} + 1, \ldots ,t_{2} ]\) and \(C_{o}\) the objet region class with levels \([t_{2} + 1, \ldots ,L - 1]\). The mean of two classes of \(C_{l}\) and \(C_{m}\) can be obtained via

*i*from 1 to \(\hbox{min} (\mu ,255 - \mu )\) to compute \(\sigma_{S}^{2}\) with Eq. (11) until it reaches its maximum, the value for parameter

*i*can be determined and hence the upper and lower bounds,

*T*

_{1}and

*T*

_{2}, can be calculated by Eq. (6). Take the infrared image with three tanks in Fig. 1a for an example. The figure shows that the original infrared image consists of targets, ground background and sky background, with the higher gray levels, middle levels and lower levels, correspondingly. After computation by statistical criterion of Eq. (11), the value of the parameter

*i*is automatically set to 37 and the upper and lower bounds are 65 and 139 respectively. Its target region, middle background and lower background are displayed in Fig. 3a–c, where the bright pixels are our focuses. In the figure, it can be observed that the target region, the ground and sky backgrounds are better separated from the original infrared image respectively. However, concerning the target region, there are still many background pixels left adhering to the targets in target region as shown in Fig. 3a. So it still needs extra steps to make the targets completely segmented. For example, background constraint and image normalization need to be implemented.

### Background constraint based on the within-class variance of the target region

- 1.Calculate the mean of target region bywhere$$\mu_{o} = \frac{1}{{N_{o} }}\sum\limits_{{i = T_{2} + 1}}^{L - 1} {in_{i} } ,$$(14)
*T*_{2}is the lower bound of gray levels for target region. - 2.Calculate the WCV of target region via$$\sigma_{o}^{2} (T) = \frac{1}{{N_{o} }}\sum\limits_{{i = T_{2} }}^{L - 1} {(i - \mu_{o} )^{2} } n_{i} .$$(15)
- 3.
Find the lower bound of gray levels for background. Assume that the lower bound of gray levels of background is denoted by

*T*_{ c }, where*T*_{ c }<*T*_{2}. Let*T*_{ c }decrease from*T*_{2}and thus the gray levels \([T_{c} , \ldots ,T_{2} ]\) are treated as the levels of background. Decrease*T*_{ c }until the WCV of background equals to that of target region, and the lower bound will be determined. - 4.Implement the background constraint via the following way:where \(f_{{}} (i,j)\) and \(f_{c} (i,j)\) are the gray levels at pixel ($$f_{c} (i,j) = \left\{ {\begin{array}{*{20}l} 0 \hfill & {{\text{if}}\;f(i,j) < T_{c} } \hfill \\ {f(i,j)} \hfill & {{\text{if}}\;f(i,j) \ge T_{c} } \hfill \\ \end{array} } \right.,$$(16)
*i*,*j*) of the original infrared image and the constrained form, respectively.

### Detailed steps of the proposed method

- 1.
Divide infrared image into three classes of target region, middle background and lower background via maximazing the statistical criterion of Eq. (11).

- 2.
Find the lower bound of background

*T*_{ c }. Treat the target region as the optimal one approximately and select the value from*T*_{2}to 1 to calculate the within-class variance of background until it equal to the within-class variance of target region. Then the value of*T*_{ c }is determined. - 3.
Implement the background constraint via Eq. (16) and thus simplify the infrared image.

- 4.
Apply the Otsu method to segment the simplified infrared image.

## Experiments and analysis

### Segmentation results comparison of different methods

For description convenience, the manual segmentation results are treated as the optimal ones and displayed in the figures from Figs. 5b, 6b, 7b, 8b, 9b, 10b, 11b and 12b. The segmented results by different methods are also given in Figs. 5, 6, 7, 8, 9, 10, 11 and 12, respectively. From them, it can be observed that, due to the complex background, the segmented results by the standard Otsu method are the worst in visual effect and those by both the 2-D Otsu method and the 2-D maximum entropy method are slightly better. That is to say, there are still more images not well segmented by the two methods. For example, for the 2-D Otsu method, there is only one image to be well segmented as shown in Fig. 10d, and for the 2-D maximum entropy method, there are two images to be well segmented as shown in Figs. 8e and 9e, respectively. In comparison with these three methods, the visual thresholding results by the proposed method indicate that, in addition to a completely segmenting for the targets from complex background, the new method exhibits less background noises. This could be attributed to the utilization of the background constraint for the original image. The background constraint dramatically simplifies the original infrared image by weakening the gray level changes of background. In addition, one can observe that the segmented results of the new method are comparative with those by manual segmentation in visual effect. As a whole, from the visual segmented results it can be concluded that the proposed method enjoys better visual effect, indicating the new method with better segmentation performance.

### Segmentation accuracies comparison of different methods

Thresholds and accuracies of different thresholding methods

Images | Optimal threshold | Otsu method | 2-D Otsu method | 2-D maximum entropy | This paper method | ||||
---|---|---|---|---|---|---|---|---|---|

Threshold |
| Threshold |
| Threshold |
| Threshold |
| ||

Figure 5a | 138 | 84 | 60.9 | (84,85) | 61.2 | (101,108) | 75.7 | 138 | 100.0 |

Figure 6a | 140 | 115 | 82.1 | (115,131) | 87.9 | (118,118) | 84.3 | 145 | 96.4 |

Figure 7a | 100 | 79 | 79 | (80,80) | 80.0 | (122,105) | 86.5 | 99 | 99.0 |

Figure 8a | 135 | 104 | 77.0 | (104,105) | 77.4 | (138,138) | 97.8 | 137 | 98.5 |

Figure 9a | 125 | 105 | 84.0 | (105,106) | 84.4 | (123,123) | 98.4 | 131 | 95.2 |

Figure 10a | 121 | 115 | 95.0 | (115,116) | 95.5 | (153,153) | 73.6 | 126 | 95.9 |

Figure 11a | 155 | 85 | 54.8 | (85,84) | 54.5 | (112,112) | 72.3 | 146 | 94.2 |

Figure 12a | 143 | 92 | 64.3 | (92,94) | 65.0 | (97,103) | 69.9 | 139 | 97.2 |

Average | – | – | 74.7 | – | 75.7 | – | 82.3 | – | 97.1 |

### Running time comparison of different methods

*T*

_{ c },

*L*− 1] and hence saving running time. Nevertheless, the new method needs extra time to estimate the ranges of target and background, with implementing background constraint. In addition, one can observe that the two methods based on 2-dimensional histogram cost too much more times with three orders of magnitude higher than both the standard Otsu method and the proposed method. This is because that the utilization of two-dimension histogram makes that the complexity of the algorithm increases exponentially as gray levels of image. In a word, one can conclude that the proposed method consumes less running time in segmentation of an image, implying its better performance in real time processing.

Running times of different thresholding methods

Methods | Figure 5a | Figure 6a | Figure 7a | Figure 8a | Figure 9a | Figure 10a | Figure 11a | Figure 12a | Average |
---|---|---|---|---|---|---|---|---|---|

Standard Otsu method (ms) | 1.45 | 1.87 | 1.49 | 1.63 | 1.6 | 3.45 | 1.88 | 1.76 | 1.89 |

2-D Otsu method (ms) | 1459.18 | 1393.91 | 1534.09 | 1358.98 | 1391.35 | 2573.63 | 1520.35 | 1478.4 | 1588.74 |

2-D maximum entropy (ms) | 4348.39 | 4361.48 | 4504.75 | 4030.64 | 4287.01 | 8039.61 | 4430.92 | 4411.26 | 4801.76 |

This paper method (ms) | 9.372 | 9.914 | 6.79 | 8.888 | 8.868 | 11.346 | 8.834 | 9.743 | 9.22 |

## Conclusions

In this paper, the threshold of the Otsu method is analyzed and the underlying reason of its failure in segmentation of infrared images in complex background is presented. It is concluded that the extremely large difference of within-class variances between background and target regions leads to deviation of threshold given by Otsu method from the optimal. From this conclusion, an unsupervised background constrained method is proposed for segmentation of infrared images in complex background based on the Otsu method. Due to the complexity of background, the original infrared image is divided into three classes of target region, middle background and lower background based on the statistical characteristics of original image, firstly. Then, the background constraint is implemented based on the within-class variance of target region and hence the original image can be simplified. Finally, the Otsu method is applied to simplified image for threshold selection. Experimental results on a variety of tank infrared images in complex background demonstrate that the proposed method has better segmentation performance and even can be comparative with the manual segmentation. In addition, its average running time is only 9.22 ms, implying the new method with good performance in real time processing.

## Declarations

### Authors’ contributions

The authors’ contributions to this work respectively are: Yulong Zhou, AB; Min Gao, FG; Dan Fang, ES; Baoquan Zhang, ES. All authors read and approved the final manuscript.

### Acknowledgements

This work is supported by Postdoctoral Science Foundation of China (Grant No. 2014M562657).

### Competing interests

The authors declare that we have no competing interests.

**Open Access**This article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made.

## Authors’ Affiliations

## References

- Fan SH, Yang SH, He P et al (2011) Infrared electric image thresholding using two-dimensional fuzzy Renyi entropy. Energy Procedia 12:411–419View ArticleGoogle Scholar
- Guo HT, Tian T, Wang LY et al (2006) Image segmentation using the maximum entropy of the two-dimensional bound histogram. Acta Opt Sin 26(4):506–509Google Scholar
- Hu M, Li M, Wang RG (2010) Application of an improved Otsu algorithm in image segmentation. J Electron Meas Instrum 24(5):443–449View ArticleGoogle Scholar
- Huang DY, Wang CH (2009) Optimal multi-level thresholding using a two-stage Otsu optimization approach. Pattern Recognit Lett 30(3):275–284View ArticleGoogle Scholar
- Kurz L, Benteftifa MH (1997) Analysis of variance in statistical image processing. Cambridge University Press, CambridgeView ArticleGoogle Scholar
- Li YC, Tian Z (2009) Multiscale image segmentation based on graph weighted kernel K-means. Acta Opt Sin 29(10):2762–2767View ArticleGoogle Scholar
- Li ZY, Yang J, Liu GH et al (2011) Unsupervised range-constrained thresholding. Pattern Recognit Lett 32:392–402View ArticleGoogle Scholar
- Liu J, Jin WD (2013) Three-dimensional adaptive minimum error thresholding segmentation algorithm. J Image Graphics 18(11):1416–1424Google Scholar
- Long JW, Shen XJ, Chen HP (2012) Adaptive minimum error threshold algorithm. Acta Automatica Ainica 38(7):1134–1144View ArticleGoogle Scholar
- Otsu N (1979) A threshold selection method from gray-level histograms. IEEE Trans Syst Man Cybern 9(1):62–66View ArticleGoogle Scholar
- Qiao LY, Xu LX, Gao M (2013) Fast maximum entropy thresholding based on two-dime nsional histogram oblique segmentation in infrared imaging guidance. Infrared laser Eng 42(7):1691–1699Google Scholar
- Sen D, Pal SK (2010) Gradient histogram: thresholding in a region of interest for edge detection. Image Vis Comput 28(4):677–695View ArticleGoogle Scholar
- Tao W, Jin H, Zhang Y, Liu L, Wang D (2008) Image thresholding using graph cuts. IEEE Trans Syst Man Cybern A 38(5):1181–1195View ArticleGoogle Scholar
- Wu YQ, Zhang BC, Wu JM (2010) An infrared image segmentation method based on within-class absolute difference and chaotic particle swarm optimization. Acta Opt Sin 30(1):79–85View ArticleGoogle Scholar
- Xu XY, Song EM, Jin LH (2009) Characteristic analysis of threshold based on Otsu criterion. Acta Electron Sin 37(12):2716–2719Google Scholar
- Zhang TX, Zhao GZ, Wang F et al (2006) Fast recursive algorithm for infrared ship image segmentation. J Infrared Millim Waves 25(4):230–295Google Scholar