Fast digital zooming system using directionally adaptive image interpolation and restoration

A digital zooming system can increase spatial resolution without a high density image sensor or a high cost optical zoom lens. Recently, most digital cameras adopt a digital zooming system, which is commonly implemented by convolving an up-sampled version of the low-resolution (LR) image with a small kernel using proper weighting coefficients. Popular convolution-based up-sampling methods include linear and cubic-spline interpolation (Wick et al. 2004). Linear interpolation simply averages four neighboring pixels with weights that are inversely proportional to the distance from the target pixel. On the other hand, cubic-spline interpolation determines the pixel intensity value using the weighted average of sixteen neighboring pixels with weights determined by the two dimensional (2D) cubic function. However, neither linear nor cubic-spline interpolation can avoid jagged artifacts in the slanted edge region and blurring artifacts due to the nature of the rectangular-shaped interpolation kernel (Papker et al. 1983; Unser et al. 1991).

To solve this problem, advanced interpolation algorithms have been proposed. Li et al. estimated local covariance coefficients from an LR image and used the estimated coefficients to adapt the interpolation effect based on the geometric duality between the pair of LR and High-resolution (HR) image covariance (Li et al. 2001). Zhang et al. proposed the edge-guided nonlinear interpolation using directional filtering and data fusion (Zhang et al. 2006). Giachetti et al. proposed an up-scaling algorithm based on two-step grid filling and iterative correction of the interpolated pixels by minimizing an objective function depending on the second-order directional derivatives of the image intensity (Giachetti et al. 2011). Zhou et al. proposed the improved cubic-spline interpolation algorithm based on the estimation of the strong edge for a missing pixel location (Zhou et al. 2012). These advanced interpolation algorithms are, however, unsuitable for a fast digital zooming system which has limiter computational power and memory space.

The proposed digital zooming system consists of directionally adaptive image interpolation and restoration. After estimating the edge orientation using steerable filters with edge refinement (Kang et al. 2013a, [b]), the input LR image is adaptively interpolated along the estimated edge orientation using the directionally weighted linear or adaptive cubic-spline interpolation function. The blurring artifacts caused by the interpolation process are then restored using the proposed directionally adaptive truncated constrained least-squares (TCLS) filter (Kim et al. 2009). Both proposed interpolation and restoration filters have a finite impulse response (FIR) structure that is suitable for real-time digital zooming in an image signal processing (ISP) chain.

The subjective observations show that the proposed method can provide high-quality interpolated images without jagging and blurring artifacts with a high magnification ratio. For objective comparison, the proposed method provides higher peak-to-peak signal-to-noise ratio (PSNR) and structural similarity (SSIM) (Wang et al. 2004) values with lower computation time than existing advanced interpolation algorithms. The block diagram of the proposed digital zooming system is shown in Figure 1.

The main advantage of the proposed digital zooming system is implemented simply based on the FIR structure for real-time processing in many digital imaging systems. In this section, our proposed four-direction steerable filters and directionally adaptive TCLS restoration filters are described.

A typical digital imaging system consists of four functional modules: (i) a set of optical lenses, (ii) the analog front-end (AFE) module including a color filter array (CFA), a complementary metal-oxide-semiconductor image sensor (CIS), and an analog-to-digital converter (ADC), (iii) the digital back-end (DBE) module including various image signal processing subsystems, and (iv) the display devices as shown in Figure 2. The proposed digital zooming system belongs to the DBE module in the ISP chain.

The proposed digital zooming subsystem consists of: (i) estimation of the edge orientation followed by edge refinement, (ii) selective interpolation using either cubic-spline or directionally weighted one-dimensional (1D) linear interpolation along the estimated edge orientation, and (iii) restoration filtering as shown in Figure 3.

Edge orientation estimation and refinement

In order to determine the edge orientation, the input image is convolved with four 5 × 5 FIR steerable filters given in Table 1 as

$d^{\theta }\left(x,y\right)={\displaystyle \sum _{m=-2}^2{\displaystyle \sum _{n=-2}^2{f}_L\left(m,n\right)G^{\theta }\left(x-m,y-n\right)}}\text{,}$

(14)

where f _L (m, n) represents a 5 × 5 local block of the input image centered at (x, y) and G ^θ (x, y) the 5 × 5 FIR steerable filters rotated by angle θ ∈ {0°, 45°, 90°, 135°}.

The initial edge orientation is determined by minimizing the mean of d ^θ (x, y) as (Kang et al. 2013a, [b])

${\theta }_I\left(x,y\right)=argmin\left\{D^{\theta }\left(x,y\right)\right\}\text{,}$

(15)

and

$D^{\theta }\left(x,y\right)=\frac{1}{25}{\displaystyle \sum _{m=-2}^2{\displaystyle \sum _{n=-2.}^2{d}^{\theta }\left(x-m,y-n\right)}}\text{.}$

(16)

If the mean value D ^θ (x, y) is less than a pre-specified threshold, the corresponding pixel is considered to be in the non-slanted edge region. In this work, the threshold value of 0.075 was used for the empirically optimum sensitivity of steerable filters. Given an initial edge orientation θ _I (x, y), the refined edge orientation is selected among eighteen directions. The proposed edge refinement algorithm is summarized as.

The refined edge orientation θ^∗ is finally quantized with the interval of 10°. Figure 4 shows the results of edge orientation estimation using four directionally steerable filters followed by edge refinement. As shown in Figure 4c, the proposed method provides more accurate and continuous edge orientation, which make the result of the proposed directionally adaptive interpolation looks more natural.

Directionally adaptive image interpolation

In order to interpolate the slanted edge region without jagging artifacts, the proposed method computes the line on the point P3 with refined edge orientation θ^∗ as shown in Figure 5.

The intensity value of P1 is first determined by the proposed adaptive cubic-spline interpolation using the activity-map (Efstratiadis et al. 1990) at v₂ and four pixels on the same horizontal line as

$P1=v_{1}f\left(1+\mathrm{S}\right)+v_{2}f\left(\mathrm{S}\right)+v_{3}f\left(1-\mathrm{S}\right)+v_{4}f\left(2-\mathrm{S}\right)\text{,}$

(17)

where S represents the distance from interpolation point P1, and f(·) represents a cubic-spline weight function defined as

$f\left(\mathrm{S}\right)=\left\{\begin{array}{c}\hfill \left(a+2\right){\left|\mathrm{S}\right|}^3-\left(a+3\right){\left|\mathrm{S}\right|}^2+1,0\le \left|\mathrm{S}\right|\le 1\hfill \\ \hfill a{\left|\mathrm{S}\right|}^3-5a{\left|\mathrm{S}\right|}^2+8a\left|\mathrm{S}\right|-4a,1\le \left|\mathrm{S}\right|\le 2\hfill \\ \hfill 0,2\le \left|\mathrm{S}\right|\text{.}\hfill \end{array}\right.$

(18)

where a represents a cubic-spline weight function parameter.

In this work, the initial parameter value of a = - 1 was used. For the spatially adaptive interpolation without blurring artifacts, the cubic-spline weight function parameter is changed according to the strength of the edge using the activity-map (Efstratiadis et al. 1990) as

${\mathit{\alpha }}_{\mathit{MAP}}\left(x,y\right)=\frac{1}{1+\sigma \mathit{Var}\left(x,y\right)}\text{,}$

(19)

where the tuning parameter σ is chosen so that α _MAP (x, y) distributes as uniformly as possible in [0,1], and Var(x, y) is the local variance of a pixel located at (x, y). In this work, the tuning parameter of σ = 250 was used.

Thus, the adaptive cubic-spline weight function parameter is determined as

$\widehat{a}=a\cdot {\mathit{\alpha }}_{\mathit{MAP}}\left(x,y\right),\mathrm{f}\mathrm{o}\mathrm{r}-0.5\le \widehat{a}\le -1\text{,}$

(20)

where α _MAP (x, y) represents the activity value at v₂.

The intensity value of P2 is also determined in the same manner in the vertical direction. Given P1 and P2, the intensity value of P3 is determined by the weighted linear interpolation along the edge line $\overline{P1P2}$ as (Kang et al. 2013a, [b])

$P3=\frac{w2}{w1+w2}P1+\frac{w1}{w1+w2}P2\text{.}$

(21)

where w 1 represents the distance between P1 and P3, and w 2 the distance between P2 and P3.

For reducing the computational load of the interpolation process, a simple cubic-spline interpolation is used with a = - 0.5, if a pixel is not on the salted edge region. By using the directionally optimized interpolation, the proposed method can significantly reduce jagging artifacts in the slanted edge region.

For evaluating the performance of the proposed digital zooming method, we used a set of standard images of size 512 × 512, and outdoor test images of size 1280 × 720 acquired by a mobile phone camera. The performance of the proposed method is evaluated with PSNR, SSIM and processing time in seconds on a personal computer with 3.4 GHz CPU and 8GB memory.

To evaluate the performance of the proposed digital zooming system by comparing with cubic-spline interpolation, the standard images and its eight times magnified version are used. The cubic-spline interpolation uses the weight function with a = - 0.5. The magnification results show that the proposed method can better remove jagging and blurring artifacts in the slanted edge region than the cubic-spline interpolation method as shown in Figure 6.

For additional experiments, to evaluate the performance and speed of the six interpolation algorithms such as cubic-spline interpolation, Li’s method (Li et al. 2001), Zhang’s method (Zhang et al. 2006), Giachetti’s method (Giachetti et al. 2011), Zhou’s method (Zhou et al. 2012), and the proposed method, standard images are generated by down-sampling 512 × 512 images by a factor of two in both horizontal and vertical directions.

For additional experiments, 1280 × 720 high-definition (HD) images are generated from 320 × 180 LR mobile camera images by four times magnification using the six advanced interpolation algorithms.

As shown in Figure 7b, strong interpolation artifacts are observed in Li’s method. Zhang’s method results in both jagging and blurring artifacts near edge regions as shown in Figure 7c. Giachetti’s method results in missing pixels near edge regions as shown in Figure 7d. Zhou’s method results in blurring artifacts near edge regions as shown in Figure 7e. The proposed method can successfully remove both jagging and blurring artifacts as shown in Figure 7f. Additional experimental results show that the proposed method provides higher quality magnification results without jagging and blurring artifacts than existing advanced interpolation algorithms.

This paper presents novel directionally adaptive image interpolation and restoration algorithms for a fast digital zooming system in digital cameras. The proposed interpolation algorithm analyzes the edge orientation using computationally efficient steerable filters followed by the edge refinement process. The selective use of directionally weighted 1D interpolation and 2D adaptive cubic-spline interpolation can enhance the image quality without jagging artifacts in the slanted edge region. Blurring artifacts are also removed using directionally adaptive TCLS filters. Both proposed steerable filter-based interpolation and the TCLS-based restoration filters have an FIR structure for real-time processing in an ISP chain. Experimental results show that the proposed method can provide high-quality magnified images without jagging and blurring artifacts. Furthermore, the proposed method gives higher PSNR and SSIM values than existing state-of-the-art interpolation methods with reduced computation load.