 Research
 Open access
 Published:
Edgeaware spatialfrequency extrapolation for consecutive block loss
SpringerPlus volume 5, Article number: 593 (2016)
Abstract
To improve the spatial error concealment (SEC) for consecutive block loss, an edgeaware spatialfrequency extrapolation (ESFE) algorithm and its edgeguided parametric model are proposed by selectively incorporating the Houghbased edge synthesis into the frequencybased extrapolation architecture. The dominant edges that cross the missing blocks are firstly identified by the Canny detector, and then the robust Hough transformation is utilized to systematically connect these discontinuous edges. During the generation of edgeguided parametric model, the synthesized edges are utilized to divide the missing blocks into the structurepreserving regions, and thus the residual error is reliably reduced. By successively minimizing the weighted residual error and updating the parametric model, the known samples are approximated by a set of basis functions which are distributed in a region containing both known and unknown samples. Compared with other stateoftheart SEC algorithms, experimental results show that the proposed ESFE algorithm can achieve better reconstruction quality for consecutive block loss while keeping relatively moderate computational complexity.
Background
To reduce computational complexity and memory burdens, the blockbased image/video coding has been widely used in visual communication. The transmission of coding data may lead to block losses in packetswitching networks. As a result, the decoded image may be greatly degraded. As a postprocessing technique, error concealment (EC) is used to recover the missing blocks of an image by exploiting the spatial or temporal correlations (Usman et al. 2015). Although the temporal correlation tends to be higher than the spatial one, there are some situations where it is difficult to access temporal information, such as images and intracoded frames (intraframe). Under such circumstances, spatial error concealment (SEC) tries to reconstruct the damaged blocks by utilizing spatially neighboring pixels. Flexible macroblock ordering (FMO) can provide a common benchmark for comparing different SEC algorithms, where a frame is divided into several independentlydecodable block groups, and each block group consists of a sequence of blocks (Panyavaraporn and Aramvith 2011). With FMO, different block groups are encapsulated into different packets, so that the missing blocks in the decoded image can be concealed by neighboring pixels. In some applications, the loss of one packet implies the loss of a block group within a frame.
The typical SEC algorithms mainly depend on the efficient use of spatial correlation. The Markov random field (MRF) is used as a prior model of natural images, and its model parameters are locally adjusted according to the image characteristics around the damaged region (Shirani et al. 1999). The MRF algorithm can produce a visually comfortable but sometimes oversmoothed concealment without a substantial increase in computational complexity. Bilinear interpolation (BI) is a wellknown SEC technique as a nonnormative part in block coding standards (Varsa et al. 2001), which uses the weighted averaging interpolation of neighboring pixels at vertical or horizontal boundaries of a damaged block. The BI algorithm can recover the smooth area but fail to restore the important edge information. To overcome this problem, it becomes necessary to rely on regularization techniques which go from simple lowpass filtering to more sophisticated edgeenhancement solutions (Cafforio et al. 2001). Current research on the problem mainly concentrates its efforts on the tradeoffs between efficiency and accuracy.
Since the edge structures are visually more important than uniform textures, some advanced interpolation methods can exploit the structural information in the neighborhood of missing blocks. Li and Orchard (2002) proposed an orientation adaptive interpolation (OAI) algorithm based on a pixelwise sequential prediction model, which estimates the missing block from eight directions in raster scan order and merges them with the weighted combination. The OAI algorithm alleviates error propagation at the expense of blurred details. In addition, the contentadaptive error concealment (CAEC) (Zhang et al. 2004) classifies each missing block into one of three categories: edge block, texture block and uniform block, and then conceals the missing blocks by different interpolation methods. By a minimum mean square error (MMSE) estimator, the probability function may be used to recovery the missing blocks. Koloda et al (2014a) suggested the MMSEbased error concealment with kernel density estimation (KMMSE), which need more computational complexity to improve reconstruction quality.
Adaptive predictors have been widely researched in lossless coding. If these predictors are directly utilized for SEC, they may cause the severe error propagation. Liu et al (2014) proposed an orderadaptive linear predictor (OALP) to sequentially estimate the missing pixels, where Bayesian information criterion is adopted to explicitly determine the order of the predictor, and error propagation can be well alleviated by a carefully designed scan order. As a typical technique for object removal applications, inpainting is also applicable to the SEC problem. With the loss of Shannon entropy, the inpaintingbased SEC has an implicit advantage in terms of subjective evaluation. Since the image inpainting would require larger amount of computations, it is often difficult to be applied for the SEC applications with runtime constraints. Chung and Yim (2014). proposed a hybrid exemplarbased inpainting and spatial interpolation (HEISI) method, whose unique feature is the thresholdselective reconstruction by inpainting or interpolation. When there is a similar patch, HEISI performs the exemplarbased inpainting; otherwise, it performs the spatial interpolation. The edge synthesis is also used in spatial error concealment, where multidirectional interpolations are combined according to the visual clearness (VC) of the edges (Koloda et al. 2013). However, the VC algorithm is effective only when four neighboring blocks of a missing block are available, and it does not work for consecutive block loss.
The aforementioned SEC algorithms can effectively utilize the multidirectional correlation to combat the scattered block loss. When the consecutive block loss occurs, these SEC algorithms will face lots of difficulties to obtain any horizontal correlation, so the highquality concealment has to rely on much prior knowledge (Usman et al. 2015). In this paper, our work concentrates on the error concealment of consecutive block loss, which is a more challenging scenario where multiple interleaved rows of blocks are missing. An alternative SEC approach is the signal extrapolation (Kaup et al. 2005; Koloda et al. 2014b), which can estimate the unknown signal parts from known samples by assuming that image signals can be sparsely represented in the frequency domain. Based on the successive approximation of parametric model, Koloda et al (2014b) proposed a modified frequency selective extrapolation (XFSE) algorithm that exploits the prior knowledge regarding the lowpass behavior of natural images, and yields a certain smoothing gains for consecutive block loss. However, due to highfrequency decaying of lowpass filter, XFSE cannot progressively improve its reconstruction quality even with more basis functions, whose performance saturates as the number of iterations is further increased. If highfrequency edge information is available, the lowpass filtering module in XFSE should be removed during the generation of parametric model, and thus the concealment performance is likely to be further improved.
The existing extrapolation algorithms don’t fully take into account the edge information of natural images. To further improve the reconstruction performance in case of consecutive block loss, we propose an edgeaware spatialfrequency extrapolation (ESFE) algorithm with its edgeguided parametric model, which incorporates the edge synthesis into the frequencybased extrapolation, and then exploits high frequency terms in image description. The ESFE algorithm firstly performs the segmentation to identify a plausible area of dominant edges, and then conceals the edge pixels across the missing blocks. The ESFE algorithm develops the edgeguided parametric model from the set of Fourier basis functions which can be used to replace the unknown samples with a low computational burden. For consecutive block loss, the ESFE algorithm utilizes the edgeguided parametric model to select optimal basis functions and expansion coefficients while preserving the edge information, and offers a much better solution in terms of reconstruction quality and complexity. To the best of our knowledge, this is the first study aiming to takes the edge information of consecutive block losses into account during the blockbased concealment extrapolation.
The rest of this paper is organized as follows. “Problem formulation” section discusses the SEC problem with consecutive block loss, and provides a short review of signal extrapolation. Our proposed algorithm is described in “Proposed algorithm and its model” section. Extensive experimental results and performance comparisons are presented in “Experimental results” section. Finally, we conclude the paper in “Conclusions” section.
Problem formulation
Consecutive block loss
Different SEC algorithms are designed to estimate the missing blocks from correctly received blocks. The locations of missing blocks can be obtained at the decoder. Figure 1 illustrates two typical loss patterns, i.e., the scattered loss pattern and the consecutive loss pattern, where one image is encoded into two block groups and each square represents a block of pixels. When a block group is missing, the test images are subjected to approximately 50 % block loss. Figure 1a shows a common situation of scattered loss pattern, where fourconnected surrounding blocks of a missing block are correctly received. For the scattered block loss, many SEC algorithms can perform very well as the missing blocks can be reconstructed by their surrounding blocks. Another belongs to the consecutive loss pattern, which is a more challenging pattern since adjacent blocks in one row are lost. Figure 1b shows an example of consecutive loss pattern. Due to the lack of adjacent blocks in each row, many SEC algorithms cannot effectively combat the consecutive block loss. The consecutive block loss is still an open problem for spatial error concealment.
Signal extrapolation
During signal extrapolation, the fitting of unknown samples is subject to a limited number of known samples, which may be used to estimate the missing blocks. In an extrapolation area, (x, y) and (k, l) indicates the (row, column) index in spatial domain and frequency domain, respectively. The samples of known blocks are successively approximated through a parametric model g(x, y), and the missing blocks are extrapolated according to a minimum error criterion of weighted energy function. During extrapolation process, the parametric model is a weighted superposition of twodimensional basis functions φ _{ k,l }(x, y) with expansion coefficients c _{ k,l }.
where \(\varvec{F}_{{\mathbf{b}}}\) denotes the index set in frequency domain, and the number of available basis functions equals the number of samples in the extrapolation area. The extrapolation mechanism iteratively updates a parametric model based on a set of basis functions, in order to approximate the available parts of received image. As the same time, the missing parts of received image can be estimated by the parametric model. Since there is only one basis function added to the model in every iteration step, certain iterations are needed for generating the model. Human visual system is very sensitive to the image structure (e.g., the edge or corner), so the proposed algorithm emphasizes on the concealment of missing region with dominant edges.
Proposed algorithm and its model
Edge synthesis via Hough transformation
Since the consecutive block loss will result in lower reconstruction quality, it is a practical strategy to firstly recover the part information of dominant edges, and the strategy provides a relatively robust basis for the following model generation. In this paper, the edge synthesis via Hough transformation is operated before signal extrapolation, where the edge detection need be introduced firstly to provide a binary edge map. For this purpose, the Canny’s edge detector is chosen due to a good compromise between efficiency and complexity (Canny 1986). Compared with other detectors such as Sobel or Prewitt, the Canny detector is less sensible to noise, and the detected edges are clear. Around the missing blocks, dominant edge points are obtained by the Canny detector. To connect the broken edges, Hough transformation has been widely used as the edgeconnection tool (Robie and Mersereau 2000; Gharavi and Gao 2008). The merging approach in Ref. 16 is rather tedious, especially for a large number of consecutive block losses. In this paper we have utilized a more straightforward approach to connect the dominant edges. Based on the binary edge map from the Canny detector, the Hough transformation can connect the separated segments by a collinear set of points, since each line can be expressed as
where \({\theta }\) (slope) and ρ (offset) denote the Hough coordinate, and the collinear points (x _{ i }, y _{ i }) with i = 1,…,K, are transformed into K sinusoidal curves which intersect at the same Hough coordinate (\(\rho, \theta\)). Each point (x _{ i }, y _{ i }) is transformed into a discretized curve and the accumulator cells along this curve are incremented. Since the collinear edge points in the spatial domain would accumulate into the same cell in the Hough domain, a high peak in the accumulator array would indicate the existence of a straight line in a missing block row.
For each Canny region which includes a missing block row, its upper block row, and its lower block row, a series of Hough procedures are implemented to acquire the dominant linear edges. The Hough transform is continuously applied to the binary regions provided by the Canny edge detector. Above and below a missing block row, two known segments with similar Hough coordinate are selected if their prolongation crosses the missing block row. There may be some near horizontal lines in the vicinity of the missing block row, which cannot be effectively used for edge synthesis. Some known segments with too large slope \({\theta }\) need be eliminated from the candidates. As shown in Fig. 2, if two known segments of a broken edge have similar (ρ, \({\theta }\)) parameters, point 1 in the upper segment and point 2 in the lower segment respectively are the pixels which are closest to the missing block row. In the synthesized edge line, the gray level of a pixel is the bilinear interpolation between point 1 and point 2. After generating a binary edge, the broken edge is reconstructed by bilinear interpolation, whose missing pixel is replaced by a weighted mean of point 1 and point 2.
In order to further clarify the mechanism of edge synthesis via Hough transformation, Fig. 3 shows the experimental results for four typical images and intraframes: 1st intraframe of Foreman (352 × 288, QP = 22), Lena (512 × 512), 1st intraframe of RaceHorses (832 × 480, QP = 37), Airport (1024 × 1024), where the yellow line represents the synthesized edge among missing blocks, and the experimental settings are given in “Experimental results” section. These images and intraframes have different resolutions, ranging from 352 × 288 to 1024 × 1024. It can be seen that the proposed mechanism can obtain basic structural information, and many dominant edges have been detected and connected successfully. The synthesized edges are then used to segment the blocks into different regions for the spatialfrequency extrapolation.
Spatialfrequency extrapolation
Based on the synthesized edge in “Edge synthesis via Hough transformation” section, the proposed algorithm will further improve the XFSE implementation. This approach is based on the XFSE method proposed by Koloda et al (2014b). After the dominant edges are found via the Hough transform, fine concealment can be achieved by spatialfrequency extrapolation. As illustrated in Fig. 4, the size of an extrapolation area \(\varvec{A}\) is X × Y samples which are indexed by spatial variables x and y. All samples in the area \(\varvec{A}\) belong to one of four areas: the correctly received samples build up the received area \(\varvec{R}\); the samples from the missing blocks which have been extrapolated build up the concealed area \(\varvec{C}\); in the current missing blocks, the synthesized edge samples build up the edgesynthesis area \(\varvec{E}\), and other unknown samples such as texture belong to the nonedge missing area \(\varvec{T}\). The following edgeguided parametric model is based on the extrapolation area in Fig. 4.
In order to reconstruct the unknown samples, we need to minimize a weighted residual error between the original signal and its parametric model. To estimate the samples in the nonedge missing area \(\varvec{T}\), the parametric model successively approximates the available samples within the support area (the received area \(\varvec{R}\), concealed area \(\varvec{C}\), edgesynthesis area \(\varvec{E}\)). At the \(n{\text{th}}\) iteration, the edgeguided parametric model g ^{(n)}(x, y) is:
where \(\varvec{F}_{{\mathbf{b}}}\) denotes the set of basis functions \(\varphi_{p,q} \left( {x,y} \right)\) weighted by the expansion coefficients c ^{(n)}_{ p,q } at the \(n{\text{th}}\) iteration; (p, q) denotes the (row, column) index in frequency domain. The samples in the support area are approximated successively by computing the basis function and expansion coefficient per iteration, where the basis function φ _{ p,q }(x, y) is selected which maximizes the decrease of residual error, and the expansion coefficient c ^{(n)}_{ p,q } is computed by minimizing the residual error. The edgeguided parametric model is generated with the initialization g ^{(0)}(x, y) being 0, whose coefficients c ^{(0)}_{ u,v } are also set to 0. At the \(n{\text{th}}\) iteration, we can express the residual error as follows:
where s(x, y) denotes a sample in the extrapolation area; the masking function m(x, y) is zero for (x, y)∈ \(\varvec{T}\) and one otherwise, so as to ensure that the nonedge missing samples are not used. The residual error r ^{(n)}(x, y) between the available sample s(x, y) and the current model g ^{(n)}(x, y) is minimized. The edge synthesis can obtain some structural information in missing block rows, and thus reduce the residual error of edgeguided parametric model. The expansion coefficient is estimated by minimizing the weighted energy from the last residual error:
where ω(x, y) is a weighting function. It has been demonstrated that the influence of the weighting function decays symmetrically with distance from the center of the missing block (Koloda et al. 2013). As a prior knowledge, the known sample in the vicinity of missing block has higher importance than the sample that is far from it, and the influence of the weighting function decreases with distance. Based on the synthesized edge in “Edge synthesis via Hough transformation” section, the proposed ESFE algorithm further refines the weighting function of XFSE by adding the edgeaware spatial correlation constraints. The new weighting function ω(x, y) can be defined as
During the spatialfrequency extrapolation, ω(x, y) is used for quantizing the influence of the distance to the extrapolated sample. As the unknown samples cannot contribute to the model generation, they have to be excluded from the weighting function, and the weight in the area \(\varvec{T}\) is set to 0. The predetermined constant \(\beta \in \left[ {0, 1} \right)\) controls the speed of the decaying. In the edgesynthesis area \(\varvec{E}\), the influence for the sample is further weighted by a factor \(\varepsilon \in \left( {0, 1} \right]\). In the concealed area \(\varvec{C}\), the influence for the sample is further weighted by another factor \(\delta \in \left( {0, 1} \right]\). At each iteration n, the projection variable of expansi coefficient can be expressed as
which is interpreted as a weighted projection variable of r ^{(n)}(x, y) on φ _{ p,q }(x, y). The best basis function and its expansion coefficient need to be searched as the one which maximizes the reduction of error energy, that is,
The process of spatialfrequency extrapolation can be further described as follows:

1.
Initializing the weighted residual error
After the edgeguided parametric model is generated, all the unknown samples are taken from the model, and inserted at the corresponding positions of missing samples. Let us consider the spatiallyweighted version of the residual error. The parametric model is initialized by g ^{(0)}(x, y) = 0. The initialization {n = 0} of the weighted residual error is done by the following:

2.
Determining the best fitting basis function
The Fourier basis can be selected arbitrarily so as to reflect the stochastic properties of an image. By using the twodimensional discrete Fourier transform (\(\varvec{DFT}\)), the reduction of the weighted error energy can be expressed in the frequency domain:
where R ^{(n)}_{ ω } (p, q) and W(p, q) are the \(\varvec{DFT}\) of r ^{(n)}_{ ω } (x, y) and ω(x, y), respectively. As the denominator in the equation above is constant, the division can be calculated in advance and be replaced by a multiplication with 1/W(0, 0) within the iteration loop.

3.
Determining the expansion coefficients
Due to the inclusion of highfrequency edge information, the proposed ESFE algorithm further removes the lowpass filtering module of XFSE. Based on Eq. (7), the projection variable of expansion coefficient can be expressed in the frequency domain:
where R ^{(n)}_{ ω } (u, v) and W(u, v) are another \(\varvec{DFT}\) of r ^{(n)}_{ ω } (x, y) and ω(x, y), respectively. The best basis function will be the one which can better approximate this residual error by the derivation of nearest neighbors. At the \(\left( {n + 1} \right){\text{th}}\) iteration, the expansion coefficient c ^{(n+1)}_{ u,v } is updated by
where the compensation factor \({\gamma }\) is introduced to compensate the orthogonality deficiency of signal extrapolation. Smaller compensation factor yields a better convergence and slower quality decrease after a certain number of iterations.

4.
Updating the edgeguided parametric model
The parametric model is updated at each iteration, which obtains optimal basis functions and expansion coefficients to successively approximate the available samples. The evolution of iterative procedure relies on the computation of weighted residual error R ^{(n)}_{ ω } (p, q). At the next iteration, the weighted residual error is updated as:
where R ^{(n+1)}_{ ω } (p, q) are the \(\varvec{DFT}\) of r ^{(n+1)}_{ ω } (x, y), which provides the weighted residual error for the next iteration directly in the frequency domain.

5.
Final parametric model
After all iterations are done, the final parametric model is obtained by twodimensional inverse discrete Fourier transform (\(\varvec{IDFT}\)). The unknown pixels are properly concealed from the edgeguided parametric model. This parametric model is the closest approximation to the known samples in the available support area:
Experimental results
The proposed ESFE algorithm will be compared with other stateoftheart SEC methods, such as MRF (Shirani et al. 1999), BI (Varsa et al. 2001), OAI (Li and Orchard 2002), CAEC (Zhang et al. 2004), KMMSE (Koloda et al. 2014a), OALP (Liu et al. 2014), HEISI (Chung and Yim 2014), and XFSE (Koloda et al. 2014b). Standard test images and video frames are examined on the consecutive loss pattern in Fig. 1b, which are subjected to about 50 % block loss. In order to facilitate the file operation, the last block row is retained. Peak signaltonoise ratio (PSNR) is chosen as one of the objective quality metrics in the experiments, and the multiscale structural similarity (MSSSIM) metric is also reported (Wang et al. 2003). Most of previous SEC algorithms focus on the block losses with block size of 16 × 16 pixels. So each block has dimensions of 16 × 16 pixels and the size of the area \(\varvec{A}\) is 48 × 48. Based on MATLAB 2012a, the Canny edge detector with adaptive threshold is utilized, and a series of Hough transformbased functions are applied to systematically connect the dominant edges. Typically, the weighting function declines with β = 0.8, and the orthogonality deficiency compensation is set to \({\gamma }\) = 0.2, and the parameters ɛ and δ are set to 1.0 and 0.9, respectively.
Test on still images
To evaluate the performance of the proposed ESFE algorithm during image transmission, extensive experiments are conducted on several still images. The test images are selected from the USCSIPI database (http://sipi.usc.edu/database), and they are uniformed into grayscale images of 8bit depths. These test images include Aerial (256 × 256), Peppers (512 × 512), Lena (512 × 512), Baboon (512 × 512), Boat (512 × 512), House (512 × 512), Airport (1024 × 1024), and Man (1024 × 1024). When the consecutive block loss is applied in test images, the objective quality of different SEC algorithms is given in Table 1. As can be observed from the table, the proposed ESFE algorithm achieves the best average quality for all test images. The average gains over the second best algorithm are 0.48 dB in terms of PSNR and 0.0077 in terms of MSSSIM. Moreover, our algorithm achieves up to 1.89 dB higher PSNR and 0.0701 higher MSSSIM than the edgedirected OAI method. Compared with the recent HEISI, OALP, and XFSE methods, our algorithm averagely obtains gains of 0.48 dB, 0.89 dB, and 0.85 dB in terms of PSNR, and gains of 0.008, 0.019, and 0.0077 in terms of MSSSIM. We attribute this remarkable improvement to the robust edge synthesis and welldesigned signal extrapolation.
For the corrupted images Lena and Airport, Figs. 5 and 6 show the subjective quality comparisons of reconstructed images by different SEC algorithms. From these figures in Fig. 5c, d, we can observe that the CAEC and BI algorithms face the difficulties to recover the edge information in the missing blocks, which result in severe blocking artifacts. In Fig. 5e–j, there are still many edges unrecovered, and they produce lumpy transition. In Fig. 5k, the proposed ESFE algorithm has recovered some dominant edges successfully, and noticeable improvements can be found around some regions (e.g., around the rim of hat and chin of Lena).
Figure 6 gives the similar conclusion. It can be observed that CAEC and BI methods completely blur the inner pixels of missing blocks, which heavily degrade visual quality. The OAI algorithm also produces some annoying artifacts. OALP and KMMSE produce some ghosting parts (e.g., around the head and wings of some airplanes). HEISI overemphasizes the object contour and produces a few pseudoedges. It is noticed that the proposed ESFE algorithm can more accurately recover global object contours with severe losses, such as the edges along the airport runway in the upper part of Airport.
Test on intraframes
The proposed ESFE algorithm is also utilized to recover the intraframes to demonstrate its effectiveness when blocking artifacts and blurring caused by compression are present. Four standard sequences with different resolutions and different levels of activities are chosen: Akiyo (176 × 144), Foreman (352 × 288), BlowingBubbles (416 × 240), and RaceHorses (832 × 480). By H.264 reference software JM19.0 (Extended profile), 1st intraframe of each sequence is encoded in grayscale. The entropy coding method is UVLC with only 4 × 4 transform. Without the rate control, the quantization parameter (QP) is set to 22, 27, 32, and 37, respectively. In the simulation, each frame is encoded into two block groups by using the consecutive loss pattern in Fig. 1b. Most parameters are default settings in the JM19.0 config file (Grayscale = 1, IDRPeriod = 1). Table 2 gives the objective quality comparisons of different SEC algorithms for intraframe concealment with different QP values. It can be seen that the proposed ESFE algorithm can consistently improve the reconstruction quality over other SEC algorithms, and averagely outperforms them in terms of both PSNR and MSSSIM. Compared to the second best algorithm, the average gains of ESFE are 0.77 dB in terms of PSNR and 0.0009 in terms of MSSSIM.
Figures 7 and 8 further show the subjective quality comparisons of reconstructed images by different SEC algorithms, where 1st intraframe of Foreman is encoded with QP = 22, and 1st intraframe of RaceHorses is encoded with QP = 37. As illustrated in these figures, our ESFE algorithm produces the most visually pleasant results among all comparative methods. Although the OALP sometimes gives relatively sharp boundary, it also produces some misleading artificial transition when incorrectly estimating the direction of contours. From these figures in Fig. 7, we can observe that the continuity of hat boundary of Foreman is broken when using HEISI, OALP, and KMMSE, while some dominant edges have been reconstructed gracefully by the proposed ESFE algorithm. It is evident that in all SEC algorithms, the proposed ESFE algorithm is the only one that completely recovers the nose of the Foreman. From these figures in Fig. 8, it is easy to find that the edges across the regions of consecutive block loss cannot be well recovered with the other SEC methods, and the reconstructed image of the proposed ESFE algorithm is visually more plausible and coherent, such as the edge along saddle and the hat of the rider.
Runtime comparison
The proposed ESFE algorithm achieves the best average PSNR and MSSSIM for all test images. To investigate the relative complexity of different SEC algorithms, six images with different sizes (1st intraframe of Akiyo (176 × 144), Aerial (256 × 256), 1st intraframe of BlowingBubbles (416 × 240), 1st intraframe of Foreman (352 × 288), Lena (512 × 512), 1st intraframe of RaceHorses (832 × 480), Man (1024 × 1024)) are tested. By repeating each method 10 times, the average runtime is presented in Table 3, where the runtime is obtained by MATLAB 2012a implementations on Intel Q8200 @ 2.33 GHz CPU and 4 GB memory.
Since these SEC algorithms don’t use any special function, their MATLAB implementations may provide a certain reference for the complexity comparison. From Table 3, it can be seen that the proposed ESFE algorithm is much faster than the recent HEISI and KMMSE algorithms, and it also outperforms OALP when dealing with consecutive block loss. Due to the lack of horizontal correlation, KMMSE has to run a large amount of invalid iterative processes to achieve a convergence result. Although our algorithm requires longer runtime than some methods (e.g., MRF and BI), its advantage is obvious in terms of objective and subjective quality evaluations. Compared with XFSE, simulation results reveal that the proposed ESFE algorithm significantly improves the recovery quality, and increases only a small complexity overhead. Our ESFE algorithm strikes a good balance between the computational complexity and recovery quality.
Conclusions
To combat the consecutive block loss in transmitted images, this paper presented an effective SEC algorithm by selectively incorporating the edge synthesis into the signal extrapolation architecture. During the iterative approximation, the concealment performance can be improved by using structural information. The edge synthesis via Hough transformation can systematically connect the discontinuous edges before signal extrapolation. With the use of Canny detection and Hough transformation, a contour reconstruction is carried out on degraded bocks and utilized to split the missing blocks into separate regions for signal extrapolation. For consecutive block loss, the proposed ESFE algorithm can reconstruct the dominant structures without highlevel semantic knowledge, and thus obtain better subjective and objective quality with a marginal additional computational cost. If objectbased edge synthesis or parallel acceleration are available in the future, the performance of the ESFE algorithm will be further improved.
References
Cafforio C, Di Sciascio E, Guaragnella C (2001) Edge enhancement for subband coded images. Opt Eng 40(5):729–739
Canny J (1986) A computational approach to edge detection. IEEE Trans Pattern Anal Mach Intell 8(6):679–698
Chung Byungjin, Yim Changhoon (2014) Hybrid error concealment method combining exemplarbased image inpainting and spatial interpolation. Sig Process Image Commun 29(10):1121–1137
Gharavi H, Gao S (2008) Spatial interpolation algorithm for error concealment. In: International conference on acoustics, speech, and signal processing. Las Vegas, pp 1153–1156
Kaup A, Meisinger K, Aach T (2005) Frequency selective signal extrapolation with applications to error concealment in image communication. AEU Int J Electron Commun 59(3):147–156
Koloda J, Sánchez V, Peinado AM (2013) Spatial error concealment based on edge visual clearness for image/video communication. Circuits Syst Signal Process 32(2):815–824
Koloda J, Peinado AM, Sanchez V (2014a) Kernelbased MMSE multimedia signal reconstruction and its application to spatial error concealment. IEEE Trans Multimed 16(6):1729–1738
Koloda J, Seiler J, Kaup A, Sanchezy V, Peinado AM (2014b) Frequency selective extrapolation with residual filtering for image error concealment. In: International conference on acoustics, speech, and signal processing, pp 1976–1980, Florence
Li X, Orchard MT (2002) Novel sequential errorconcealment techniques using orientation adaptive interpolation. IEEE Trans Circuits Syst Video Technol 12(10):857–864
Liu J, Zhai G, Yang X, Yang B (2014) Spatial error concealment with adaptive linear predictor. In: IEEE international symposium on circuits and systems. Melbourne, pp 2728–2731
Panyavaraporn J, Aramvith S (2011) Error resilient framework using onepass explicit flexible macroblock ordering map generation and error concealment for H.264/AVC wireless video communication. J Electron Imaging 20(2):1102–1114
Robie DL, Mersereau RM (2000) The use of hough transforms in spatial error concealment. In: International conference on acoustics, speech, and signal processing. Istanbul, pp 2131–2134
Shirani S, Kossentini F, Ward R (1999) An adaptive Markov random field based error concealment method for video communication in an error prone environment. International conference on acoustics, speech, and signal processing (ICASSP). Phoenix, pp 3117–3120
Usman M, He X, Xu M, Lam K (2015) Survey of error concealment techniques research directions and open issues. In: 31st picture coding symposium. Cairns, pp 233–238
Varsa V, Hannuksela MM, Wang YK (2001) Nonnormative error concealment algorithms. Document JVTN62, Geneva
Wang Z, Simoncelli EP, Bovik AC (2003) Multiscale structural similarity for image quality assessment. In: IEEE Asilomar conference on signals, systems and computers. Pacific grove, pp 1398–1402
Zhang R, Zhou Y, Huang X (2004) Contentadaptive spatial error concealment for video communication. IEEE Trans Consum Electron 50(1):335–341
Authors’ contributions
HL: conception and design, drafting of the manuscript. DW: analysis and interpretation of data, statistical analysis. BW, KL: acquisition of data, technical or material support. HT: critical revision of the manuscript for important intellectual content. All authors read and approved the final manuscript.
Acknowledgements
This work is supported by the Natural Science Foundation of Shanghai (14ZR1400500), the China Scholarship Council (201506635022), the National Natural Science Foundation of China (41401486).
Competing interests
The authors declare that they have no competing interests.
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made.
About this article
Cite this article
Liu, H., Wang, D., Wang, B. et al. Edgeaware spatialfrequency extrapolation for consecutive block loss. SpringerPlus 5, 593 (2016). https://doi.org/10.1186/s4006401622136
Received:
Accepted:
Published:
DOI: https://doi.org/10.1186/s4006401622136