Color reproduction and processing algorithm based on real-time mapping for endoscopic images
© Khan et al. 2016
Received: 27 August 2015
Accepted: 13 December 2015
Published: 6 January 2016
In this paper, we present a real-time preprocessing algorithm for image enhancement for endoscopic images. A novel dictionary based color mapping algorithm is used for reproducing the color information from a theme image. The theme image is selected from a nearby anatomical location. A database of color endoscopy image for different location is prepared for this purpose. The color map is dynamic as its contents change with the change of the theme image. This method is used on low contrast grayscale white light images and raw narrow band images to highlight the vascular and mucosa structures and to colorize the images. It can also be applied to enhance the tone of color images. The statistic visual representation and universal image quality measures show that the proposed method can highlight the mucosa structure compared to other methods. The color similarity has been verified using Delta E color difference, structure similarity index, mean structure similarity index and structure and hue similarity. The color enhancement was measured using color enhancement factor that shows considerable improvements. The proposed algorithm has low and linear time complexity, which results in higher execution speed than other related works.
Cancer is currently the second-leading cause of death in the United States (Siegel et al. 2015). Furthermore, in 2015 cancer in the digestive system may cause the second highest number of fatalities among all sites (Siegel et al. 2015). Endoscopy plays an important role in diagnostic of colon rectum cancer at an early development stage (Hosokawa et al. 2008). As a result, mortality role for diseases like the stomach cancer, colon cancer, and ulcerative colitis has been drastically decreased in the recent years (Stock et al. 2011). In addition, the ability to capture digital pictures has paved the way for the new field of computer-aided decision support system (CADSS) in medical endoscopy (Liedlgruber and Andreas 2011). These systems focus on aiding in different decision making from endoscopic images such as assessment of different diseases (Kumar et al. 2012; Cong et al. 2014), bleeding detection (Sainju et al. 2014) and frame of interest extraction (Li et al. 2014). In both cases for diagnosis by physician or CADSS, image quality plays a critical role. Wireless Capsule Endoscopy (WCE), as an alternative to the wired endoscopy, offers physicians the capability of examining the interior of the small intestine with a noninvasive procedure (Brownsey and Michalek 2010). In WCE, a battery powered camera is placed in a capsule. When the patient swallows this capsule, it send picture continuously from the gastrointestinal tract (GI). Due to its limited power consumption, the WCE suffers from low image quality (Nakayoshi et al. 2004). Even a high-definition white light endoscopy cannot always detect all mucosal or vascular abnormalities of different positions of GI tract. In both cases for wireless and wired endoscopy, improved image quality can greatly increase early detection and reduce the miss rates of the detection of the mucosal or vascular abnormalities (Liedlgruber and Andreas 2011).
There are both pre-processing and post-processing methods that can significantly enhance mucosal or vascular characteristics in endoscopic images. Pre-processing systems like narrow band imaging (NBI) and auto-fluorescence imaging (AFI) use rotating filters in front of the light source sequentially generating red, blue and green light for tissue illumination (Schmitz-valckenberg et al. 2008; Pohl et al. 2007). Special light sources and filters are utilized to enhance the mucosal structure in the resultant images in NBI and AFI at a cost of higher hardware complexity and power consumption. As an alternative, post-processing system such as virtual chromoendoscopy (CE) decomposes the image into various wavelengths and produces pseudo color added image with enhanced mucosal surface contrast (Chiu et al. 2007). Several researchers concluded that NBI appeared to be a less time-consuming and efficient alternative to CE for the detection of neoplasia. However, NBI has a higher miss rate than CE (Khan and Wahid 2011; Nass and Connolly 2010). Additionally, neither NBI nor CE can improve the adenoma detection or reduce miss rates during screening colonoscopy. As found in works in (Nass and Connolly 2010) and (Khan and Wahid 2011), NBI and CE showed no difference in terms of diagnostic efficacy. Based on the success of CE, several researchers proposed post processing enhancement method for endoscopic image. For example, Okuhata et al. has proposed a real-time enhancement procedure based on retinex theory (Okuhata et al. 2013) and Vogt et al. has proposed real time endoscopic image enhancement scheme based on color normalization (Vogt et al. 2003).
In image processing in general, a well-known procedure for image enhancement is to enhance the luminance channel only while keeping the chrominance channel unchanged (Gonzalez and Woods 2002). Due to psycho-visual redundancy, human eyes are more sensitive to the enhancement of brightness than color. There are a several well-known methods available for enhancing the grayscale image, which can be broadly divided into two categories. Techniques such as contrast stretching (CS) (Wang and Bovik 2002), high boost filtering (HBF) (Srivastava et al. 2009) and unsharp masking (UM) (Polesel et al. 2000) work on the local gradient of the image. On the other hand, techniques such as histogram equalization (HE), contrast limited adaptive histogram equalization (CLAHE) (Zuiderveld 1994) and brightness preserving dynamic fuzzy histogram (BPDFHE) (Sheet et al. 2010) work on the global gradient of the image. The global gradient methods are effective for the low contrast images that contain a single object or no apparent contrast change between object and background (Cheng and Shi 2004).
On the other hand, the psycho-visual redundancy can be utilized to reduce the power consumption in endoscopy. This phenomenon is often utilized in image compression by sending only the grayscale image or the brightness channel. The grayscale image is later colorized using a similar tone theme images, which results in a significant save of power, memory, and bandwidth (Khan et al. 2015). There are several color reproduction algorithms available in the literature. In (Welsh et al. 2002), color information is retrieved from a target swatch. Each pixel in the grayscale image is matched with the pixel in the target swatch based on Euclidean distance metrics. The color is then copied from the matched pixel in the target swatch. This algorithm suffers from high computational complexities and processing time. In another work (Horiuchi and Hirano 2003), the authors have used a set of seed points and their respective color vectors in the RGB format with a YUV-based classification. In (Levin et al. 2004), a quadratic objective function based optimization method is used to interpolate the U and V components of the YUV color space over the entire image using a set of color scribble lines. In (Korostyshevskiy 2006), pseudo colors are employed to colorize the grayscale image using different 64 × 3 color matrices. It does not reproduce a visually appealing color on the entire image and introduces blurriness on the high contrast edges. None of those mentioned above methods has ever been applied to endoscopic imaging. In a recent work by our group, a color enhancement scheme is presented that is dedicated to enhance endoscopic images (Imtiaz et al. 2013 and Imtiaz and Khan 2014). Although this color enhancement scheme is very promising in terms of the color enhancement factor (CEF), it suffers from high algorithm complexities.
In this paper, we propose a dictionary based color reproduction method with low complexities, high color similarity and high CEF. This method enhances the visual quality of GI images. We have also shown three possible scenarios in endoscopy where the proposed method is applicable. The performance of the proposed method is assessed on a relatively diverse dataset based on the reduction of image degradation, structural similarity, and CEF.
Endoscopic images have the intrinsic characteristics of low contrast and inhomogeneous brightness, which stems from the random steering motion of the camera and bending and waving nature of the gastric organs (Vogt et al. 2003). The goal in enhancement stage is to increase the contrast and reduce the inhomogeneous brightness.
Here R, G, and B represent the intensity of red, green and blue channel of the RGB color image respectively using Eq. (1), which uses division by a power of two, so it can be very easily implemented using a shift register. This type of implementation results in a simpler hardware implementation than other conversion methods.
This shifting produces a darkening effect that improves the visibility of mucosa layer. All processed images have a dominance of low-intensity pixel compare to the white lighting image (WLI).
From the experiment it has been found the values of sharpening factor in the range 2–8 provide the best result.
Proposed color reproduction algorithm
At this stage, color is added to the generated enhanced grayscale image to produce a colorized image. The color information is retrieved using the available color image of the nearby anatomical location. Then using the color similarity between the theme image and the grayscale image, the algorithm reproduces the color of the enhanced grayscale image. We have prepared a database of color WCE pictures taken from (Gastrolab—the gastrointestinal site 1996; Atlas of gastrointestinal endoscopy 1996) for different locations of the GI tract from where a theme image is chosen manually. This part of the algorithm consists of two steps. color map generation and color reproduction.
Color map generation
First a color map is generated from a theme image T. Each color pixel is converted to luminance, Y using Eq. (1). Then the corresponding R, G and B values are listed in a color map lookup table. Since there is no one to one correspondence between Y and R, G, B, there may be multiple combinations of R, G, B with a same value of Y. In these cases, the mean values of R, G, B are taken. For an example, RGB triplet (32, 16, 8) and (0, 28, 0) both will produce the same Y value 14. In this case, in the lookup table the average of these combinations, that is (16, 22, 4), will be saved for Y = 14. These may produce color artifacts in the colorized image. However, since, for an anatomically nearby location image, the correlation between the color and luminance would be very high, the trend between the luminance and color of the theme image would provide sufficient information about the color for the enhanced grayscale image.
After completing the table, there might be some empty slots in the color map table if all Y values from 0 to 255 are not generated from the image pixels. The empty slots are filled by linear interpolation between two nearest neighboring entries.
PSNR measures (in dB) with different smoothing function applied to the color table
Linear local regression
Quadratic local regression
Robust linear local regression
Robust quadratic local regression
Applying the color map
Category 1: low contrast WLI images
Category 2: raw NBI images
Category 3: tone enhancement
In the second case, a tone enhanced endoscopic image can be used as a theme image to transfer its characteristics to the WLI images. For example, in the second image in Fig. 9, a tone enhanced theme image from a nearby anatomical location is used to reproduce the color of the original WLI image. The output image showed more enhanced mucosal structure than the original image. In both cases, we have applied the TE-g tone enhancement proposed by i-scan (Kodashima and Fujishiro 2010).
In this section, experiments are conducted to prove the validity of the proposed method in the enhancement of the mucosal structure and color reproduction. Altogether 178 images with different physiological characteristics, taken from GastroLab (Gastrolab—the gastrointestinal site 1996) and Atlas (Atlas of gastrointestinal endoscopy 1996) databases, were used for the purpose of comparison. For theme images, we have used the database of 100 WLI images taken from 20 different location in GI tract. There was no overlapping between these two database meaning that the images used for experimentation were different than the images used for color theme image. The sharpening factor used for different pictures are specified in the figure. Both objective and the subjective evaluation were considered as performance metrics. The objective method evaluated the enhancement, quality of the images and similarity between the original and colorized images. On the other hand, the quality of the proposed method in mucosal structure enhancement has been verified visually by conducting a survey among gastroenterologists.
Reducing the effect of over lighting and low contrast
Image in capsule endoscopy suffers from inhomogeneous lighting and low contrast. In a recent study, Sdiri et al. has shown that the contrast enhancement improved the stereo matching performance and classification results (Sdiri et al. 2015). To measure the performance of the proposed method in contrast enhancement, Statistics of visual representation (SVR) (Balas et al. 2009; Jawahar and Ray 1996) was used. SVR compares the contrast and intensity of the original image and enhanced image. High contrast measurement means that the resultant image has higher contrast than the original image. Similarly, high intensity measurement indicates that the processed image has a higher average intensity than the original image. In addition, to evaluate the image quality, Universal Image Quality (UIQ) (Wang and Bovik 2002) was used. UIQ is a mathematically defined parameter that evaluates the image quality on three factors: loss of correlation, luminance distortion and contrast distortion. UIQ value closer to +1 indicates good quality while value closer to −1 indicates bad quality.
SVR measures with other related works
Universal image quality index
Proposed method (178 images)
Adaptive histogram equalization (AHE)
Contrast stretching (CS)
High boost filtering (HBF)
Unsharp masking (UM)
Color similarity test and color enhancement factor (CEF)
Color similarity and enhancement assessment
Proposed (178 images)
Welsh et al.
Imtiaz et al.
Okuhata et al.
Vogt et al.
Algorithm complexity assessment
Color reprod. time
Proposed (178 images)
256 × 256
512 × 512
Imtiaz et al.
256 × 256
512 × 512
Welsh et al.
256 × 256
512 × 512
256 × 256
512 × 512
Okuhata et al.
256 × 256
512 × 512
Vogt et al.
256 × 256
512 × 512
Subjective evaluation by gastroenterologists
Average mean opinion score (MOS)
This paper presents a novel dictionary based algorithm for colorizing and enhancing a grayscale endoscopic image. The color information and the enhancing improve the visual quality of the endoscopic image. The proposed algorithm generates a dynamic color map from a color theme image. Then the color map is applied to the enhanced grayscale image to produce a color image having a similar color tone of the theme image. The quality of the generated enhanced color image is evaluated using several standard performance metrics, which showed better performance compared to many existing methods. This method can be used in enhancing and colorizing low contrast grayscale WLI images and as an alternative to the color transformation system for NBI system.
THK, SKM, MSI and KAW: The work was done by THK, SKM and MSI and supervised by KAW. All authors conceived the idea. The experiments were designed and conducted by THK, SKM and MSI. All authors read and approved the final manuscript.
The authors would like to acknowledge Grand Challenges Canada Star in Global Health, Natural Science and Engineering Research Council of Canada (NSERC), Canada Foundation for Innovation (CFI) and Western Economic Diversification Canada for its support to this research work. The authors also acknowledge the gastroenterologists who took part in the survey. They are (in no particular order): Dr. Marco Puglia and Dr. Smita Halder from Division of Gastroenterology, McMaster University and Dr. Pierre Ellul and Dr. Mario Vassallo from Gastroenterology Department, Mater Dei Hospital.
The authors declare that they have no competing interests.
Open AccessThis article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made.
- Atlas of gastrointestinal endoscopy 1996 (online). Available: http://www.endoatlas.com/index.html
- Balas Benjamin, Nakano Lisa, Rosenholtz Ruth (2009) A summary-statistic representation in peripheral vision explains visual crowding. J Vis 9(12):1–18View ArticleGoogle Scholar
- Brownsey A, Michalek J (2010) High definition scopes, narrow band imaging, chromoendoscopy. In: American society for gastrointestinal endoscopyGoogle Scholar
- Cheng H, Shi X (2004) A simple and effective histogram equalization approach to image enhancement. Digit Signal Process 14:158–170View ArticleGoogle Scholar
- Chiu H, Chang C, Chen C, Lee Y, Wu M, Lin J, Wang H (2007) A prospective comparative study of narrow-band imaging, chromoendoscopy, and conventional colonoscopy in the diagnosis of colorectal neoplasia. Gut 56(3):373–379View ArticleGoogle Scholar
- Cong Y, Wang S, Liu J, Cao J, Luo J (2014) Deep sparse feature selection for computer aided endoscopy diagnosis. Pattern Recognit 48(3):907–917View ArticleGoogle Scholar
- Dung L, Wu Y (2010) A wireless narrowband imaging chip for capsule endoscope. IEEE Trans Biomed Circuits Syst 4(6):462–468View ArticleGoogle Scholar
- Evis Lucera Spectrum family brochure (online) 2014, Available: http://www.olympus.co.uk/medical/en/medical_systems/mediacentre/media_detail_7450.jsp
- Gastrolab—the gastrointestinal site 1996 (online). Available: http://www.gastrolab.net/index.htm
- Gonzalez RC, Woods RE (2002) Digital image processing. Prentice Hall Upper Saddle River, NJGoogle Scholar
- Horiuchi T, Hirano S (2003) Colorization algorithm for grayscale image by propagating seed pixels, in proceedings of ICIP pp 457–460Google Scholar
- Hosokawa O, Miyanaga T, Kaizaki Y, Hattori M, Dohden K, Ohta K, Aoyagi H (2008) Decreased death from gastric cancer by endoscopic screening: association with a population-based cancer registry. Scand J Gastroenterol 43(9):1112–1115View ArticleGoogle Scholar
- Imtiaz MS, Khan AW (2014) A color reproduction method with image enhancement for endoscopic images, in Proc. IEEE 2nd Middle East Conf Bio-Med EngGoogle Scholar
- Imtiaz MS, Khan TH, Khan AW (2013) New color image enhancement method for endoscopic images, in Proc. IEEE Int Conf Adv Elect Eng: 263–266Google Scholar
- Jawahar CV, Ray AK (1996) Incorporation of gray-level imprecision in representation and processing of digital images. Pattern Recogn Lett 17(5):541–546View ArticleGoogle Scholar
- Khan T, Shrestha R, Imtiaz MS, Wahid KA (2015) Colour-reproduction algorithm for transmitting variable video frames and its application to capsule endoscopy. Healthc Technol Lett 2:52–57View ArticleGoogle Scholar
- Khan TH, Wahid K (2011) Low complexity color-space for capsule endoscopy image compression. IET Electron Lett 47(22):1217–1218View ArticleGoogle Scholar
- Kodashima S, Fujishiro M (2010) Novel image-enhanced endoscopy with i-scan technology. World J Gastroenterol 16(9):1043View ArticleGoogle Scholar
- Korostyshevskiy V (2006) Grayscale to RGB converter, (Online), 2006, Available: http://www.mathworks.com/matlabcentral/fileexchange/13312-grayscale-to-rgb-converter
- Kumar R, Zhao Q, Seshamani S, Mullin G, Hager G, Dassopoulos T (2012) Assessment of Crohn’s disease lesions in wireless capsule endoscopy images. Biomed Eng IEEE Trans 59(2):355–362View ArticleGoogle Scholar
- Levin A et al (2004) Colorization using optimization. ACM Trans Gr (TOG) 23:689–694View ArticleGoogle Scholar
- Li C, Hamza AB, Bouguila N, Wang X, Ming F, Xiao G (2014) Online redundant image elimination and its application to wireless capsule endoscopy. Signal Image Video Process 8(8):1497–1506View ArticleGoogle Scholar
- Liedlgruber M, Andreas U (2011) Computer-aided decision support systems for endoscopy in the gastrointestinal tract: a review. IEEE Rev Biomed Eng 4(2011):73–88View ArticleGoogle Scholar
- Nakayoshi T, Tajiri H et al (2004) Magnifying endoscopy combined with narrow band imaging system for early gastric cancer: correlation of vascular pattern with histopathology. Endoscopy 36(12):1080–1084View ArticleGoogle Scholar
- Nass JP, Connolly SE (2010) Current status of chromoendoscopy and narrow band imaging in colonoscopy. Clin Colon Rectal Surg 23(1):21–30View ArticleGoogle Scholar
- Okuhata H, Nakamura H, Hara S, Tsutsui H, Onoye T (2013) Application of the real-time retinex image enhancement for endoscopic images, Proc Eng Med Biol Soc (EMBC), 35th Annu Int Conf IEEE: 3407–3410Google Scholar
- Pohl J, May A et al (2007) Computed virtual chromoendoscopy: a new tool for enhancing tissue surface structures. Endoscopy 39(1):80–83View ArticleGoogle Scholar
- Polesel A, Ramponi G, Mathews VJ (2000) Image enhancement via adaptive unsharp masking. IEEE Trans Image Process 9:505–510View ArticleGoogle Scholar
- Robertson AR (1990) Incorporation of gray-level imprecision in representation and processing of digital images. Color Res Appl 15(3):167–170View ArticleGoogle Scholar
- Sainju S, Bui FM, Wahid KA (2014) Automated bleeding detection in capsule endoscopy videos using statistical features and region growing. J Med Syst 38(4):1–11View ArticleGoogle Scholar
- Schmitz-valckenberg S, Holz FG, Bird AC, Spaide RF (2008) Fundus autofluorescence imaging: review and perspectives. Retina 28(3):385–409View ArticleGoogle Scholar
- Sdiri B, Cheikh FA, Dragusha K, Beghdadi A (2015) Comparative study of endoscopic image enhancement techniques. colour and visual computing symposium (CVCS). IEEE, New york, pp 1–5Google Scholar
- Sheet Deboot, Garud Hrushikesh et al (2010) Brightness preserving dynamic fuzzy histogram equalization. IEEE Trans Consum Electron 56(4):2475–2480View ArticleGoogle Scholar
- Siegel RL, Miller KD, Jemal A (2015) Cancer Statistics, 2015. CA Cancer J Clin 65(1):5–29View ArticleGoogle Scholar
- Srivastava R, Gupta JRP, Parthasarthy H, Srivastava S (2009) PDF based unsharp masking, crispening and high boost filtering of digital images, Commun Comput Inf Sci, vol. 40, pp 8–13Google Scholar
- Stock C, Knudsen AB, Lansdorp-Vogelaar I, Haug U, Brenner H (2011) Colorectal cancer mortality prevented by use and attributable to nonuse of colonoscopy. Gastrointest Endosc 73(3):435–443View ArticleGoogle Scholar
- Susstrunk SE, Winkler S (2003) Color image quality on the internet. In: Proceedings of the SPIE. Electronic Imaging 2004, International Society for Optics and Photonics, vol 5304, San Jose, CA, p 118–131Google Scholar
- Viera AJ, Garrett JM (2005) Understanding interobserver agreement: the kappa statistic. Fam Med 37(5):360–363Google Scholar
- Vogt F, Krüger S, Niemann H, Schick C (2003) A system for real-time endoscopic image enhancement. In: Ellis RE, Peters TM (eds) Proceedings of the International Conference on Medical Image Computing and Computer Assisted Intervention (MICCAI), ser Lecture Notes Comput Sci, Springer-Verlag, New York, pp 356–363Google Scholar
- Wang Z, Bovik A, Sheikh H, Simoncelli E (2004) Image quality assessment: from error visibility to structural similarity. IEEE Trans Image Process 13:600–612View ArticleGoogle Scholar
- Wang Z, Bovik AC (2002) A universal image quality index. Signal Process Lett IEEE 9:81–84View ArticleGoogle Scholar
- Welsh T, Ashikhmin M, Mueller K (2002) Transferring Color to Greyscale Images. ACM Trans Gr 21(3):277–280View ArticleGoogle Scholar
- Shi Y, Ding Y, Zhang R, Li J (2009) Structure and hue similarity for color image quality assessment, in ICECT, pp 329–333Google Scholar
- Zuiderveld K (1994) Contrast limited adaptive histogram equalization. Graphics Gems IV. Academic Press Professional Inc, San Diego, pp 474–485View ArticleGoogle Scholar