Fast digital zooming system using directionally adaptive image interpolation and restoration

Abstract

This paper presents a fast digital zooming system for mobile consumer cameras using directionally adaptive image interpolation and restoration methods. The proposed interpolation algorithm performs edge refinement along the initially estimated edge orientation using directionally steerable filters. Either the directionally weighted linear or adaptive cubic-spline interpolation filter is then selectively used according to the refined edge orientation for removing jagged artifacts in the slanted edge region. A novel image restoration algorithm is also presented for removing blurring artifacts caused by the linear or cubic-spline interpolation using the directionally adaptive truncated constrained least squares (TCLS) filter. Both proposed steerable filter-based interpolation and the TCLS-based restoration filters have a finite impulse response (FIR) structure for real time processing in an image signal processing (ISP) chain. Experimental results show that the proposed digital zooming system provides high-quality magnified images with FIR filter-based fast computational structure.

Introduction

A digital zooming system can increase spatial resolution without a high density image sensor or a high cost optical zoom lens. Recently, most digital cameras adopt a digital zooming system, which is commonly implemented by convolving an up-sampled version of the low-resolution (LR) image with a small kernel using proper weighting coefficients. Popular convolution-based up-sampling methods include linear and cubic-spline interpolation (Wick et al. 2004). Linear interpolation simply averages four neighboring pixels with weights that are inversely proportional to the distance from the target pixel. On the other hand, cubic-spline interpolation determines the pixel intensity value using the weighted average of sixteen neighboring pixels with weights determined by the two dimensional (2D) cubic function. However, neither linear nor cubic-spline interpolation can avoid jagged artifacts in the slanted edge region and blurring artifacts due to the nature of the rectangular-shaped interpolation kernel (Papker et al. 1983; Unser et al. 1991).

To solve this problem, advanced interpolation algorithms have been proposed. Li et al. estimated local covariance coefficients from an LR image and used the estimated coefficients to adapt the interpolation effect based on the geometric duality between the pair of LR and High-resolution (HR) image covariance (Li et al. 2001). Zhang et al. proposed the edge-guided nonlinear interpolation using directional filtering and data fusion (Zhang et al. 2006). Giachetti et al. proposed an up-scaling algorithm based on two-step grid filling and iterative correction of the interpolated pixels by minimizing an objective function depending on the second-order directional derivatives of the image intensity (Giachetti et al. 2011). Zhou et al. proposed the improved cubic-spline interpolation algorithm based on the estimation of the strong edge for a missing pixel location (Zhou et al. 2012). These advanced interpolation algorithms are, however, unsuitable for a fast digital zooming system which has limiter computational power and memory space.

The proposed digital zooming system consists of directionally adaptive image interpolation and restoration. After estimating the edge orientation using steerable filters with edge refinement (Kang et al. 2013a, [b]), the input LR image is adaptively interpolated along the estimated edge orientation using the directionally weighted linear or adaptive cubic-spline interpolation function. The blurring artifacts caused by the interpolation process are then restored using the proposed directionally adaptive truncated constrained least-squares (TCLS) filter (Kim et al. 2009). Both proposed interpolation and restoration filters have a finite impulse response (FIR) structure that is suitable for real-time digital zooming in an image signal processing (ISP) chain.

The subjective observations show that the proposed method can provide high-quality interpolated images without jagging and blurring artifacts with a high magnification ratio. For objective comparison, the proposed method provides higher peak-to-peak signal-to-noise ratio (PSNR) and structural similarity (SSIM) (Wang et al. 2004) values with lower computation time than existing advanced interpolation algorithms. The block diagram of the proposed digital zooming system is shown in Figure 1.

Figure 1

The block diagram of the proposed digital zooming system.

The directionally steerable and truncated constrained least-squares (TCLS) filters

The main advantage of the proposed digital zooming system is implemented simply based on the FIR structure for real-time processing in many digital imaging systems. In this section, our proposed four-direction steerable filters and directionally adaptive TCLS restoration filters are described.

Four-direction steerable filters

In order to determine the edge orientation of an input image, directional steerable filters are used (Freeman et al. 1991). A steerable filter in an arbitrary direction is synthesized using a linear combination of a pair of orthogonal basis filters, such as the 2D circularly symmetric Gaussian function defined in the Cartesian coordinate as

$$G\left(x,y\right)=e^{-\left(x^2+y^2\right)}\text{,}$$

(1)

where scaling and normalization constants have been set to unity for notational simplicity.

Let G^θ be the 1D derivative of G(x, y) in the direction with angle θ. For example, the first-order derivative with θ = 0 is expressed as

$$G^{0^{°}}\left(x,y\right)=\frac{\partial }{\partial x}{e}^{-\left(x^2+y^2\right)}=-2x{e}^{-\left(x^2+y^2\right)}\text{,}$$

(2)

and the same function rotated by 90° is expressed as

$$G^{{90}^{°}}\left(x,y\right)=\frac{\partial }{\partial y}{e}^{-\left(x^2+y^2\right)}=-2y{e}^{-\left(x^2+y^2\right)}\text{.}$$

(3)

It is evident that a 1D Gaussian function in an arbitrary orientation θ can be synthesized by taking linear combination of $G^{0^{°}}$ and $G^{{90}^{°}}$ as

$$G^{\theta }=G^{0^{°}}cos\theta +G^{{90}^{°}}sin\theta \text{.}$$

(4)

The cos θ and sin θ terms are used to express an arbitrary direction. For reducing the computational load of edge orientation, four 5 × 5 steerable filters are used with standard deviation σ = 1.0. The proposed steerable filter coefficients are generated as shown in Table 1.

Table 1 Directionally steerable filter coefficients

Directionally adaptive TCLS restoration filter

In order to remove the blurring artifacts caused by the interpolation process, an image restoration filter is needed. Kim et al. proposed the original version of the TCLS restoration filter for removing spatially adaptive image degradation followed by a spatially adaptive noise smoothing filter (Kim et al. 2009). This sub-section presents a directionally adaptive version of the TCLS restoration filter that removes the blurring artifacts caused by the interpolation process.

The input LR image can be considered as a low-pass filtered and sub-sampled version of the original HR image as

$$g\left(x,y\right)=T\left[f\left(p,q\right)\ast h\left(p,q\right)\right]+\eta \left(x,y\right)\text{.}$$

(5)

where g(x, y) and η(x, y) respectively represent 2D arrays of LR image and additive white Gaussian noise. f(p, q) represents the HR image, and h(p, q) the space-variant point spread function (PSF), which plays a role in anti-aliasing filter for the subsequent subsampling operation denoted as T[·].

Let H(u, v) be the frequency response of the PSF h(p, q), then the frequency response of the constrained least-squares (CLS) restoration filter is given as (Katsaggelos, 1989)

$$R_{CLS}\left(u,v\right)=\frac{H^{\ast }\left(u,v\right)}{{\left|H\left(u,v\right)\right|}^2+\lambda {\left|C\left(u,v\right)\right|}^2}\text{.}$$

(6)

where C(u, v) represents a frequency response of the high-pass filter, and λ the regularization parameter the controls the relative amount of data fidelity and the smoothing constraint.

For the CLS filter to become spatially adaptive, five different smoothness according to the edge orientation, C^θ(u, v), for θ ∈ {0°, 45°, 90°, 135°, Flat}, are generated using the four directional high-pass filter in the spatial domain as (Kang et al. 2013a, [b])

$$c^{0^{°}}\left(x,y\right)=\left(\begin{array}{ccc}\hfill 0\hfill & \hfill 0\hfill & \hfill 0\hfill \\ \hfill -0.1677\hfill & \hfill 0.8333\hfill & \hfill -0.1677\hfill \\ \hfill -0.1677\hfill & \hfill 0.8333\hfill & \hfill -0.1677\hfill \end{array}\right)\text{,}$$

(7)

$$c^{{45}^{°}}\left(x,y\right)=\left(\begin{array}{ccc}\hfill -0.1677\hfill & \hfill 0\hfill & \hfill 0\hfill \\ \hfill -0.1677\hfill & \hfill 0.8333\hfill & \hfill 0\hfill \\ \hfill -0.1677\hfill & \hfill -0.1677\hfill & \hfill -0.1677\hfill \end{array}\right)\text{,}$$

(8)

$$c^{{90}^{°}}\left(x,y\right)=\left(\begin{array}{ccc}\hfill 0\hfill & \hfill -0.1677\hfill & \hfill -0.1677\hfill \\ \hfill 0\hfill & \hfill 0.8333\hfill & \hfill -0.1677\hfill \\ \hfill 0\hfill & \hfill -0.1677\hfill & \hfill -0.1677\hfill \end{array}\right)\text{,}$$

(9)

and

$$c^{{135}^{\circ }}\left(x,y\right)=\left(\begin{array}{ccc}\hfill -0.1677\hfill & \hfill -0.1677\hfill & \hfill -0.1677\hfill \\ \hfill 0\hfill & \hfill 0.8333\hfill & \hfill -0.1677\hfill \\ \hfill 0\hfill & \hfill 0\hfill & \hfill -0.1677\hfill \end{array}\right)\text{.}$$

(10)

For minimizing noise amplification in the flat region, the following constraint is used.

$$c^{\mathrm{Flat}}\left(x,y\right)=\left(\begin{array}{ccc}\hfill -0.1111\hfill & \hfill -0.1111\hfill & \hfill -0.1111\hfill \\ \hfill -0.1111\hfill & \hfill 0.8333\hfill & \hfill -0.1111\hfill \\ \hfill -0.1111\hfill & \hfill -0.1111\hfill & \hfill -0.1111\hfill \end{array}\right)\text{.}$$

(11)

The frequency response of the modified CLS filter is given as

$$R_{CLS}^{\theta }=\frac{H^{\ast }\left(u,v\right)}{{\left|H\left(u,v\right)\right|}^2+\lambda {\left|C^{\theta }\left(u,v\right)\right|}^2}\text{.}$$

(12)

where C^θ(u, v), for θ ∈ {0°, 45°, 90°, 135°, Flat}, plays a role of directionally adaptive smoothness constraints, and λ = 0.2 is experimentally used. The spatial-domain counterpart of $R_{CLS}^{\theta }\left(u,v\right)$ is its inverse DFT expressed as

$$r_{\mathit{TCLS}}^{\theta }\left(x,y\right)=F^{-1}\left[R_{CLS}^{\theta }\left(u,v\right)\right]\text{.}$$

(13)

where F^- 1[·] represents the inverse DFT operation. For reducing the computational load of restoration processing, $r_{\mathit{TCLS}}^{\theta }\left(x,y\right)$ is truncated an m × m FIR filter. Table 2 shows five truncated constrained least-squares (TCLS) filter when m = 5.

Table 2 Directionally adaptive TCLS filter coefficients

Combined directionally adaptive image interpolation and restoration

A typical digital imaging system consists of four functional modules: (i) a set of optical lenses, (ii) the analog front-end (AFE) module including a color filter array (CFA), a complementary metal-oxide-semiconductor image sensor (CIS), and an analog-to-digital converter (ADC), (iii) the digital back-end (DBE) module including various image signal processing subsystems, and (iv) the display devices as shown in Figure 2. The proposed digital zooming system belongs to the DBE module in the ISP chain.

The proposed digital zooming subsystem consists of: (i) estimation of the edge orientation followed by edge refinement, (ii) selective interpolation using either cubic-spline or directionally weighted one-dimensional (1D) linear interpolation along the estimated edge orientation, and (iii) restoration filtering as shown in Figure 3.

Figure 2

The image signal processing chain of a digital camera with the proposed digital zooming system in the digital back-end module.

Figure 3

The block diagram of the proposed digital zooming system combining edge refinement, interpolation, and restoration methods.

Edge orientation estimation and refinement

In order to determine the edge orientation, the input image is convolved with four 5 × 5 FIR steerable filters given in Table 1 as

$$d^{\theta }\left(x,y\right)={\displaystyle \sum _{m=-2}^2{\displaystyle \sum _{n=-2}^2{f}_L\left(m,n\right)G^{\theta }\left(x-m,y-n\right)}}\text{,}$$

(14)

where f _L (m, n) represents a 5 × 5 local block of the input image centered at (x, y) and G^θ(x, y) the 5 × 5 FIR steerable filters rotated by angle θ ∈ {0°, 45°, 90°, 135°}.

The initial edge orientation is determined by minimizing the mean of d^θ(x, y) as (Kang et al. 2013a, [b])

$${\theta }_I\left(x,y\right)=argmin\left\{D^{\theta }\left(x,y\right)\right\}\text{,}$$

(15)

and

$$D^{\theta }\left(x,y\right)=\frac{1}{25}{\displaystyle \sum _{m=-2}^2{\displaystyle \sum _{n=-2.}^2{d}^{\theta }\left(x-m,y-n\right)}}\text{.}$$

(16)

If the mean value D^θ(x, y) is less than a pre-specified threshold, the corresponding pixel is considered to be in the non-slanted edge region. In this work, the threshold value of 0.075 was used for the empirically optimum sensitivity of steerable filters. Given an initial edge orientation θ _I (x, y), the refined edge orientation is selected among eighteen directions. The proposed edge refinement algorithm is summarized as.

The refined edge orientation θ^∗ is finally quantized with the interval of 10°. Figure 4 shows the results of edge orientation estimation using four directionally steerable filters followed by edge refinement. As shown in Figure 4c, the proposed method provides more accurate and continuous edge orientation, which make the result of the proposed directionally adaptive interpolation looks more natural.

Figure 4

Results of edge orientation estimation. a the input image. b color coded edge orientation using four directionally steerable filters. c color coded edge orientation refined by the proposed method.

Directionally adaptive image interpolation

In order to interpolate the slanted edge region without jagging artifacts, the proposed method computes the line on the point P3 with refined edge orientation θ^∗ as shown in Figure 5.

Figure 5

The proposed directionally adaptive interpolation algorithm.

The intensity value of P1 is first determined by the proposed adaptive cubic-spline interpolation using the activity-map (Efstratiadis et al. 1990) at v₂ and four pixels on the same horizontal line as

$$P1=v_{1}f\left(1+\mathrm{S}\right)+v_{2}f\left(\mathrm{S}\right)+v_{3}f\left(1-\mathrm{S}\right)+v_{4}f\left(2-\mathrm{S}\right)\text{,}$$

(17)

where S represents the distance from interpolation point P1, and f(·) represents a cubic-spline weight function defined as

$$f\left(\mathrm{S}\right)=\left\{\begin{array}{c}\hfill \left(a+2\right){\left|\mathrm{S}\right|}^3-\left(a+3\right){\left|\mathrm{S}\right|}^2+1,0\le \left|\mathrm{S}\right|\le 1\hfill \\ \hfill a{\left|\mathrm{S}\right|}^3-5a{\left|\mathrm{S}\right|}^2+8a\left|\mathrm{S}\right|-4a,1\le \left|\mathrm{S}\right|\le 2\hfill \\ \hfill 0,2\le \left|\mathrm{S}\right|\text{.}\hfill \end{array}\right.$$

(18)

where a represents a cubic-spline weight function parameter.

In this work, the initial parameter value of a = - 1 was used. For the spatially adaptive interpolation without blurring artifacts, the cubic-spline weight function parameter is changed according to the strength of the edge using the activity-map (Efstratiadis et al. 1990) as

$${\mathit{\alpha }}_{\mathit{MAP}}\left(x,y\right)=\frac{1}{1+\sigma \mathit{Var}\left(x,y\right)}\text{,}$$

(19)

where the tuning parameter σ is chosen so that α _MAP (x, y) distributes as uniformly as possible in [0,1], and Var(x, y) is the local variance of a pixel located at (x, y). In this work, the tuning parameter of σ = 250 was used.

Thus, the adaptive cubic-spline weight function parameter is determined as

$$\widehat{a}=a\cdot {\mathit{\alpha }}_{\mathit{MAP}}\left(x,y\right),\mathrm{f}\mathrm{o}\mathrm{r}-0.5\le \widehat{a}\le -1\text{,}$$

(20)

where α _MAP (x, y) represents the activity value at v₂.

The intensity value of P2 is also determined in the same manner in the vertical direction. Given P1 and P2, the intensity value of P3 is determined by the weighted linear interpolation along the edge line $\overline{P1P2}$ as (Kang et al. 2013a, [b])

$$P3=\frac{w2}{w1+w2}P1+\frac{w1}{w1+w2}P2\text{.}$$

(21)

where w 1 represents the distance between P1 and P3, and w 2 the distance between P2 and P3.

For reducing the computational load of the interpolation process, a simple cubic-spline interpolation is used with a = - 0.5, if a pixel is not on the salted edge region. By using the directionally optimized interpolation, the proposed method can significantly reduce jagging artifacts in the slanted edge region.

Directionally adaptive image restoration

The proposed TCLS restoration filters are generated using directionally adaptive smoothness constraints C^θ(u, v) according to the estimated edge orientation. To remove blurring artifacts caused by the interpolation process, the proposed restoration method performs 2D convolution using five 5 × 5 directionally adaptive TCLS filters according to the edge orientation θ as

$$\widehat{f}\left(x,y\right)=\widehat{g}\left(x,y\right)\ast r_{\mathit{TCLS}}^{\theta }\left(x,y\right)\text{.}$$

(22)

where ∗ represents the 2D convolution operator, ĝ(x, y) is the interpolated image, $r_{\mathit{TCLS}}^{\theta }\left(x,y\right)$ is the impulse response of the directionally TCLS filter of orientation θ, and $\widehat{f}\left(x,y\right)$ is the restored HR image.

Experimental results

For evaluating the performance of the proposed digital zooming method, we used a set of standard images of size 512 × 512, and outdoor test images of size 1280 × 720 acquired by a mobile phone camera. The performance of the proposed method is evaluated with PSNR, SSIM and processing time in seconds on a personal computer with 3.4 GHz CPU and 8GB memory.

To evaluate the performance of the proposed digital zooming system by comparing with cubic-spline interpolation, the standard images and its eight times magnified version are used. The cubic-spline interpolation uses the weight function with a = - 0.5. The magnification results show that the proposed method can better remove jagging and blurring artifacts in the slanted edge region than the cubic-spline interpolation method as shown in Figure 6.

Figure 6

Experimental results for the eight times magnification. a and c cubic-spline interpolation with a = - 0.5. b and d the proposed method.

For additional experiments, to evaluate the performance and speed of the six interpolation algorithms such as cubic-spline interpolation, Li’s method (Li et al. 2001), Zhang’s method (Zhang et al. 2006), Giachetti’s method (Giachetti et al. 2011), Zhou’s method (Zhou et al. 2012), and the proposed method, standard images are generated by down-sampling 512 × 512 images by a factor of two in both horizontal and vertical directions.

Table 3 shows PSNR, SSIM and processing time for the two times magnification. These evaluation results show that the proposed method provides faster processing time than advanced interpolation method. Since the proposed method is based on the cubic-spline interpolation, it shows almost the same computational complexity as the original cubic-spline interpolation as shown in Table 3. However, the advanced interpolation algorithms use the iterative method to interpolate a new pixel for the magnification. Therefore, these methods needs higher computational complexity than the original cubic-spline interpolation and the proposed methods.

Table 3 PSNR, SSIM, and CPU times of two interpolation methods using standard test images

For additional experiments, 1280 × 720 high-definition (HD) images are generated from 320 × 180 LR mobile camera images by four times magnification using the six advanced interpolation algorithms.

As shown in Figure 7b, strong interpolation artifacts are observed in Li’s method. Zhang’s method results in both jagging and blurring artifacts near edge regions as shown in Figure 7c. Giachetti’s method results in missing pixels near edge regions as shown in Figure 7d. Zhou’s method results in blurring artifacts near edge regions as shown in Figure 7e. The proposed method can successfully remove both jagging and blurring artifacts as shown in Figure 7f. Additional experimental results show that the proposed method provides higher quality magnification results without jagging and blurring artifacts than existing advanced interpolation algorithms.

Figure 7

Experimental results for the four times magnification from 320 × 180 LR mobile camera images. a cubic-spline interpolation. b Li’s method. c Zhang’s method. d Giachetti’s method. e Zhou’s method. f the proposed method.

Conclusion

This paper presents novel directionally adaptive image interpolation and restoration algorithms for a fast digital zooming system in digital cameras. The proposed interpolation algorithm analyzes the edge orientation using computationally efficient steerable filters followed by the edge refinement process. The selective use of directionally weighted 1D interpolation and 2D adaptive cubic-spline interpolation can enhance the image quality without jagging artifacts in the slanted edge region. Blurring artifacts are also removed using directionally adaptive TCLS filters. Both proposed steerable filter-based interpolation and the TCLS-based restoration filters have an FIR structure for real-time processing in an ISP chain. Experimental results show that the proposed method can provide high-quality magnified images without jagging and blurring artifacts. Furthermore, the proposed method gives higher PSNR and SSIM values than existing state-of-the-art interpolation methods with reduced computation load.