Seamless images stitching for 3D human models

Realistic 3D model reconstruction (Li et al. 2013) is an important component in computer graphics and computer vision, and comprehensively applied in our daily life, especially in films, animations, video games, virtual reality and so on. It is an underway revolution from 2D to 3D, which gradually makes the reconstruction of the real world come true. As an innovative technological application, 3D human reconstruction makes the humans get rid of the visual fatigue of 2D frames, and it opens a novel and fascinated domain to acquaint ourselves with the real world much more fascinating.

A high-quality texture on the surface of model is a key component of the reconstructed model, which impacts the visual effect significantly. In this paper, our goal is to generate a seamless texture on the surface of given 3D human model, from a set of images obtained by multiple RGB-Depth cameras.

In a large amount of previous works on this problem, they reconstruct an accurate 3D model and generate a preferable texture on the surface of model by using complex equipment, such as laser scanning and structured light. However, these methods require such an expensive and complex facility that can’t be used extensively.

The emergence of RGB-Depth cameras in recent years, such as Kinect produced by Microsoft and Xtion by Asus, paves the way to settle this matter. Evidently, it becomes widely applied due to its low price, simplified operation and hand-held shape, although the accuracy is not as high as the former.

Due to the limitation of accuracy of RGB-Depth camera, some dramatic misaligned seams are produced on the texture. But even using the facility of laser scanning and structured light, the inaccuracy of depth data and multi-view misregistration also result in these seams. Besides, some irrelevant contents of environment are mixed with the objective texture. In addition, the inaccuracy of depth data and multi-view misregistration result in some blank regions without any color contents, which destroy the integrity of texture. Furthermore, due to the different ray gains of cameras from respective directions, as well as surface reflection, the local variations in illumination is noticeable between color images, which damages the visual effect significantly and degrade the quality of texture.

In this paper, to eliminate these artifacts, we treat the texture mapping on model’s surface as an optimizaiton of images stitching, and propose a method of global optimization to compensate for the inaccuracy of depth data and multi-view misregistration. In our method, an extra adaptive iterative factor is introduced into the parameters of images stitching, and the texture images are allowed to move gradually to match the color content at the boundary of adjacent meshes. Finally, the misaligned seams are eliminated by iterative optimization.

Regarding irrelevant environmental contents, we build Gaussian Mixture Models (GMM) (Rother et al. 2004; Ruzon and Tomasi 2000; Chuang et al. 2001) in the color space, and segment the image into two parts, objective and environment. Therefore, only objective content remains in the images while the environmental contents are erased.

To solve the issue of illumination variations between adjacent color images, we construct a composite vector field in the gradient domain of color space, with the points located in the boundary as constraints. Due to the pixels distributing discretely, we solve the resulting discrete Poisson equation (Perez et al. 2003), and blend the adjacent color images smoothly. Thus the color of different images becomes coherent.

Last but not least, we parameterize the surface of model and project it onto a 2D plane. Then the K-Nearest Neighbor algorithm is applied to search for K points with color content, and the pixels in blank regions are filled with the average content of the K points. Then we back project the plane onto the surface, and all the blank regions become textured with color contents. Consequently, an integrated and consistent texture is generated on the surface of 3D human model, as shows in Fig. 6.

It is a classic topic that creating a texture on the surface of objective model, and the key is to minimize the visible seams caused by multi-view misregistration and inaccuracy of data. Catmull (1974, 1980), begin to use the texture mapping skill to improve the visual quality of images generated by raster scanning facility. Rocchini et al. (1999) confirm the best texture for each triangular mesh by projecting the color images onto the surface of 3D model, and apply the region expansion algorithm to complete the ambient texture projection. It guarantees the validity of inner texture for local areas, but results in discontinuities in the boundary of images. Liu et al. (2006) resolve the discontinuities in the boundary effectively by global optimization of the multiple texture images using bundle adjustment. Zhou et al. (2005) implement coarse-to-fine strategy of several levels, considering both geometric and images in overlapping textures, and complete the textures montage.

Different from the previous work, it has become popular recently that projecting the model surface onto the color images and forming the texture as images stitching problem (Wang et al. 2001; Thormahlen and Seidel 2008; Dessein et al. 2014). It combines the images with triangular meshes as a Markov Random Field and constructs an objective energy function of cost. Despite the elegance and power of the energy minimization approach, the algorithms which were previously used, such as iterated conditional modes (ICM) (Besag 1993) and simulated annealing (SA) (Barnard 1989), proved to be ineffective. Boykov et al. (2001) present two algorithms based on graph cuts that effectively find a global minimum of the energy function. As is well known, it has been widely applied in a great amount of cases, such as stereo matching (Barnard 1989; Boykov et al. 2001), image segmentation (Rother et al. 2004; Boykov and Jolly 2001) and image denoising (Felzenszwalb and Huttenlocher 2006). Belief propagation (BP) (Felzenszwalb and Huttenlocher 2006; Yedidia et al. 2000; Tappen and Freeman 2003) is a powerful optimization algorithm to resolve the energy minimization and widely used in a large variety of tasks. Especially in stereo matching, it usually gets a dramatic result. However, BP is not guaranteed to converge and may go into an infinite loop switching between two labelings. If BP converges and there are no ties in the min-marginal for nodes, it often achieves the local minimum.

To eliminate the noticeable seams on the mosaic image, Lempitsky and Ivanov (2007) cast the problem of seams visibility minimization as an optimization of a mesh based Markov Random Field energy, and propose a seam leveling procedure to remove out the mosaic seams without affecting fine details of the texture. Without any move of the texture content, some distortions occur in the boundary of adjacent meshes, accompanied with some slight seams. Ran et al. (2010) present an automatic method to recover high-resolution texture over an object by mapping detailed photographs onto its surface, which minimizes the visible seams by applying the Tree-Reweighted Message Passing algorithm (TRW-s) (Vladimir 2006) to achieve a global minimization, and then assigns smooth texture to adjacent triangular meshes with a set of local image transformations. In practice, there is no guarantee that the energy might not actually increase with time, and the energy sometimes starts to oscillate.

Soler et al. (2002) present an approach to form a seamless texture over the surface meshes by optimizing a set of per-face image transformations in a greedy manner. Eisemann et al. (2008) pose the method of optical flow to handle with these noticeable seams, which calculates the pairwise warping between the images and then blended at the rendering stage. Dessein et al. (2014) present a method to blend the sampled images by least-angle selection of gradients in overlapping patches, but ignore nonlinearity due to specularity and shadows. Dessein et al. (2015) segment the faces into patches, and blend the sampled textures to synthesise a seamless texture over the surface.

Among these methods, it is a common feature that each mesh of the surface can be project onto at least one image. However, in practice, it is general that blank regions without any texture content exist on the surface of model, caused by inaccuracy of depth data and multi-view misregistration. In this paper, we present a method of parameterization of the surface, combined with K-Nearest Neighbor algorithm, to deal with these blank regions, and then fill up them with color contents to create an integrated and coherent texture on the surface of model.

Color fusion in gradient domain (Fattal et al. 2002; Chuang et al. 2009) has been widely applied in a large variety of tasks to produce a new smooth image from the original images. Levin et al. (2004) apply gradient domain fusion in stitching the panoramic mosaics together. Raskar et al. (2004) blend multiple images of a common scene with different illumination intensity in the gradient domain, and consequently generate surrealist images with much more information. Perez et al. (2003) introduce a method of Poisson image editing to blend images into a single image by copying the objective region of a single image into a destination image in the manner of gradient domain. In our work, we add the points in the boundary of adjacent images into the Poisson equation as constraint to make it over-constrained.

In this paper, we propose a method of creating a seamless texture mapping on the model’s surface with a rapid optimization. We introduce an adaptive iterative factor to expand the texture, and the texture is allowed to move towards to match with adjacent meshes at the shared edge. Then the \(\alpha\)-expansion algorithm is applied to handle the energy optimization in an iterative strategy. The results of experiments demonstrate the effectiveness of our method both in visual observation and qualitative analysis.

To evaluate the effectiveness of our method, we perform our method and other three state-of-the-art methods on two human models and a doll model. The results of texture on model’s surface are shown in Figs. 8 and 10 respectively, the corresponding results of energy minimization are shown in Figs. 9 and 11, where the x axis shows the runtimes in seconds on a log scale and the y axis shows the energy of the different methods over time, and the energy and runtime of minimization are shown in Table 1. The other three state-of-the-art methods we compared are Belief Propagation(BP), Mosaicing(using graph cuts), and Montage(using Tree-Reweighted Message Passing, namely TRW-s).

In Fig. 8, we can see that, the texture generated by BP algorithm is much worse than other methods. There are obvious misaligned seams at the boundary between adjacent color images, and some images ghost as well as incorrect details exist in the part of face, hand, jacket and trouser pocket. These artifacts degrade the quality of texture and visual effect significantly. Although the method of Mosaicing achieves a better result than BP, it can not handle with the seams of misalignment and images ghost, which exists in the part of lip, jacket details, hands and so on. Both the methods of Montage and ours generate the best texture on the surface of the model. They remove out these artifacts, such as seams of misalignment, images ghost and incorrect details, and create a seamless, integrated and smooth texture on the surface of model.

In the corresponding result of energy minimization (Fig. 9), the plot shows the energy of different methods over time and some interesting features. The BP algorithm just converges to a local minimum, far away from the global minimum, and the runtime is at the level of \(10^2\) s. The Mosaicing method finds a near-optimal solution most quickly at the level of \(10^1\) s. Although the energy of Montage gradually approaches to the global minimum, it takes a large amount of time at the level of \(10^3\) s. Furthermore, during the process of calculation, it does not decrease progressively but oscillates widely. That means, if it is used in limit time and iterative frequency, it would not achieve the global optimal. Our method achieves the global minimum by aligning the texture contents at the boundary between adjacent color images, and converges quickly and progressively at the level of \(10^1\) s, just slightly slower than the Mosaicing method.

Due to the alignment of texture at the boundary, the optimization of our algorithm is speeded up. With the adaptive factor, the texture of mesh could be moved along the direction, in which the energy of optimization decreases progressively and the adjacent meshes match the color content gradually at the shared edge. Besides, according to the picture of energy minimization, the alpha-expansion algorithm converges straightforward to the global minimum without any oscillation, which reduces the time expenses. As the Fig. 9 shown, the energy of different methods demonstrates that our method obtains the minimum of energy, and achieves effectively enhancement in the aspect of runtime.

Given the second example of a man model (Figs. 10 and 11), similar to the result of the woman model, the BP algorithm gets the worst performance, which converges a local optimal, and can not deal with the previous artifacts. The Mosaicing method obtains a solution close to the global optimal most quickly, but can not remove out the seams of misalignment and images ghost. Our method obtains the global minimum of energy optimization quickly, just a little slower than the Mosaicing method, and creates the best texture on the surface of the man model. In contrast, The result of Montage seems to present something interesting and different. There are some seams of misalignment and images ghost in the region of lip and shirt, presumably caused by its drastic oscillation in the process of energy minimization. According to the result of energy minimization (Fig. 11), it demonstrates that the Montage method oscillates widely and doesn’t achieve the global minimum at the end of optimization.

Furthermore, to evaluate the effectiveness of our method, we implement these methods on the doll model, which is previously published in Mosaicing and Montage. As shown in Fig. 12, some noticeable seams and ghost artifacts are produced on the result of BP, as BP algorithm converges to a local minimum and could not align the adjacent meshes at their shared edge. Obviously, these undesirable artifacts seriously degrade the performance of BP. Although superior to the BP, the Mosaicing can’t eliminate the seams. Due to the error of multi-view registration and inaccuracy of depth data, the adjacent meshes can not match their color content at the shared edge without moving the texture image. Thus the seams and broken texture are still noticeable. In contrast, the Montage brings out a desirable texture in visual observation, without any seams and ghost artifacts. But it needs sufficient iterations to converge to the global minimum, which takes a lot of time. Otherwise, if the number of iteration is insufficient, the performance of Montage will severely decline, especially when it is in the oscillation, just as the result of the man model. Meanwhile, our method also provides a seamless texture. In the process of optimization, the misaligned textures are moved with the adaptive factor to match their color content at the shared edges, and the energy decreases progressively and converges straightforward to the global minimum without any oscillation. Thus, our method achieves the optimal result much faster than Montage, as listed in Table 1. Based on the experiments of our human models and the previously published doll model, the comparison indicates that our method is effective for reduction of seams, with a rapid optimization.

According to the experiments, we know that the BP algorithm may go into a region of local minimum and could not step out to converge to the global minimum, and some misaligned seams and distortions still remain at the shared edge of adjacent meshes through the Mosaicing algorithm. Although the Montage algorithm generates a favorable texture in visual observation without any seams, it needs abundant iterations and larger time expense to converge to the global minimum, especially accompanied with wide oscillations. In order to create a seamless texture on the model’s surface with a rapid optimization, we propose a method of introducing an iterative factor into the \(\alpha\)-expansion algorithm. As the \(\alpha\)-expansion algorithm is able to change the label of several nodes simultaneously, with such a large move, it is effective to get rid of the local minimum and converge rapidly to the global minimum during the optimization. Furthermore, with the iterative factor, the texture of mesh could be moved gradually to match the color contents at the shared edge of adjacent meshes, which compensates for the inaccuracy of depth data and multi-view misregistration. As the texture is aligned at the boundary, the energy of optimization decreases dramatically and converges straight towards the global minimum without any oscillation, leading to a large acceleration of optimization. However, the major limitation of our proposed method is model’s movement. That means, the person should keep motionless as a statue until the color image and depth data have been obtained by the cameras. If the person moves his or her body during the process, the movement will result in terrible damage to the corresponding relationship between color images and depth data, and the error of mismatch is so huge that our method could not handle effectively. In future work, we will take the slight movement of human body into consideration, which involves the nonrigid registration of human body.

In this paper, we present an effective method to generate a seamless, integrated and smooth texture on the surface of 3D human models. Given a set of color images obtained by multiple RGB-Depth cameras around the human model, and a surface mesh reconstructed previously by point cloud, we establish a MRF model. To eliminate the seams at the boundary between adjacent color images, caused by multi-view misregistration and inaccuracy of depth data, we introduce an adaptive iterative factor into the MRF model. To smooth out the noticeable illumination variation between different color images, we apply the state-of-the-art method of Poisson blending to a composite vector field in gradient domain. Furthermore, we perform parameterization of the model’s surface and apply KNN algorithm to repair the blank regions without texture content. To evaluate the effectiveness and robustness of our method, we compare it with another three advanced methods. The results demonstrate that our method achieves the global minimum of optimization almost fastest and creates a best texture on the surface of models. Therefore, our method proves evidently effective and preferable both in visual effect and quantitative analysis.