Computer-assisted coloring and illuminating based on a region-tree structure
© Nascimento et al; licensee Springer. 2012
Received: 7 February 2012
Accepted: 6 March 2012
Published: 6 March 2012
Colorization and illumination are key processes for creating animated cartoons. Computer assisted methods have been incorporated in animation/illustration systems to reduce the artists' workload. This paper presents a new method for illumination and colorization of 2D drawings based on a region- tree representation. Starting from a hand-drawn cartoon, the proposed method extracts geometric and topological information and builds a tree structure, ensuring independence among parts of the drawing, such as curves and regions. Based on this structure and its attributes, a colorization method that propagates through consecutive frames of animation is proposed, combined with an interpolation method that accurately computes a normal mapping for the illumination process. Different operators for curve and region attributes can be applied independently, obtaining different rendering effects.
Keywords2D drawing Region-tree Illumination Colorization Animation
Since the 1960s, computers became a great ally in the production of animated films, contributing to several areas that goes from the creation to the final touches .
In the conventional production of an animation sequence, first of all is necessary a representation of frame by frame . Even today, this technique is used in the production of 2D animated cartoons in which each frame is represented by a free-hand sketch.
Although assisted animation accelerate the procedure, it still has many limitations which drives one of the largest areas of research in computer graphics: Computer Animation.
In the conventional process of computer assisted cartoon animation, the entire process for the production is very costly and may even take months for a short animation to be finished. Among the biggest challenges of an automated solution, the production of inbetweening and frame colorization as mentioned by  is of great importance.
Two important, although tedious, steps in the production of animated cartoons are colorization and illumination. The colorization process transfers the colors from a single frame to the subsequent frames. The most common colorization approaches [4, 5] use structures containing topological information of the drawing, like regions, curves, and graphs, called topological structures.
Although these topological structures perform well in the colorization process, they are not commonly exploited in the illumination process of the animation. The illumination process calculates the interaction of the cartoon with light present in a 3D environment, and is an important task in cartoon rendering, since it helps to produce different effects and styles in the animation sequence.
This paper presents three main contributions. The first one is a new region-tree structure that explores local spacial information in a single frame. The second contribution is a method based on the region-tree to illuminate the objects, by approximating lighting on 2D drawings using a direct and sphere-preserving interpolation. Third, we propose a recursive tracking method for color transfer based on the region-tree. This new approach improves more effective associations of regions of two consecutive frames that can be retrieved through a recursive analysis of previous frames.
An efficient image region representation has been sought in different ways. A common approach is the use of an hierarchical structure which clusters regions at different scales to obtain the representation of images in various resolutions [6–8]. These techniques are ideal in processes such as image compression. However, in images like cartoons this structure could be simplified, since simpler inherent relationship of regions, such as inclusion and adjacency relations, will be necessary during the cartoon illumination and colorization.
In some works [9, 10], automatic normal estimation methods is computed directly from the image. But these techniques calculate the normals from the shading information of the images. They are not suitable for cartoon, because in our input cartoon we have neither normal nor shading. Some techniques to illuminate 2D data provide iterative tools to reconstruct 3D models from available bi-dimensional data [11, 12], followed by traditional 3D illumination . Johnston , observes that, even though these methods provide useful results, they may not be suitable to cartoon rendering. Therefore, the author  proposes a semi-automatic image-based technique to approximate surface normals directly in 2D drawings while avoiding 3D reconstruction. How-ever, this technique does not enable different kinds of interpolation since it does not have a relationship between curves and regions. Besides it obtains an approximated normal propagation to calculate the normal map. Recently, depth-propagation proposed by Sy´cora et al.  obtain correct initial normal map of cartoons, but the interpolation itself is computed later, by using the method proposed by Johnston . In our work we explicitly calculate the normal vector accurately and guarantees smoothness for a coherent illumination. The region- tree flexibility allows different visual effects since it is possible to choose which curve contributes to the regions interpolation process.
Segmentation techniques are generally used to extract topological structures, and employ tracking operations to check the time coherence between consecutive frames, enabling a computer-assisted colorization [4, 16–18]. Although these techniques provide the colorization to animating cartoons, these approaches do not exploit essential information such as curves and interior regions for the illumination process. Garcia et al.  propose a strategy for region matching which deals with the curve closeness issue. Since the matching is rather geometric, the method robustness could be affected Sy´cora et al.  proposes a depth-propagation to handle cartoon animations resulting in automatic colorization. However, this approach does not handle occlusion during tracking. In this work, we propose a topological approach as a way to increase the robustness of region tracking enabling it to handle occlusions.
The remainder of the paper is structured as follows. Section 2 formulates our proposed structure, the region-tree, that represents cartoon geometrical and topological elements. Section 3 describes the proposed illumination method. Section 4 presents a recursive colorization method based on region-tree. Section 5 presents some experimental results and Section 6 concludes.
This section describes a topological structure, which contains the topological and geometric information of cartoons objects. Similar to several other methods, we first apply a segmentation stage to extract a set of curves and regions . In a later stage, we explore the relationship between regions and curves to construct the data structure region-tree.
Curves and regions
Essentially, a cartoon could be represented by their curves and regions. The traces or outlines of the cartoon define the curves, while the interior of closed curves define the regions. We use the operator sets described by Bezerra et al.  to construct the set of regions and curves that will compose our region-tree structure. Those operations includes: skeletonization and region detection.
Region detection consists on performing the segmentation of the thinning image identifying each of its curves and regions. Each curve is defined by a sequence of pixels from the skeleton. In our structure, each region is defined by three informations: its boundary (external curve), its interior points (the region itself), and its internal curves. This structure will help us to calculate more efficiently the relations of inclusion and adjacency between regions in our region-tree. To extract each region, we first compute the internal area by assigning a label to each connected internal pixel. So, each connected component of the image will have the same label and will define the interior of a region. A flood-filling algorithm  is performed to make such labeling (see Figure 1 right).
Now we describe our proposed region-tree structure. Usual topological structures are used in colorization methods to track regions of consecutive frames in the animation process [4, 16–18]. Those structures explore the relationships of adjacency between regions to preserve the temporal coherence. However, they do not explore some local information details in a single frame, such as inclusion relation between regions inside the drawing. In order not only to provide time relational coherence between frames, the structure proposed in this paper aims to also allow exploring each images component individually. The possibility of navigation between the regions structures and their included regions allows treating the components one by one enabling a wide variety of effects in the drawings.
In Section 3, we present how the region-tree structure assists in the illumination process of a 2D drawing. Its advantage stems from the possibility of dealing individually, and in a consistent way, each images region allowing also process them in parallel.
The region-tree structure provides total control in the access of each regions component as boundary, internal curves, internal pixels, adjacent regions and contained regions. This enables separate attributes for each cartoon region: material properties, color, transparency, texture, and others. Analogously, we can define an attribute to each curve, or group of curves: color, line style, width, and others. Regions attributes can be stored as frame buffers allowing the data to be processed directly on the GPU.
In the context of this work, the normal vectors to the curves and regions perform a role of great significance. The inference of a normal field to the image, make it possible for any illuminations model providing three-dimensional effects on the cartoon. The next section describes this attribute in detail.
Illumination and normal mapping
In this work, we propose a novel formulation for the normal field inside each region. This normal field enables the application of illumination effects, like Phong model, providing 3D impression to the cartoon. Our normal definition is accurate and sphere-preserving, which leads to smooth and coherent illumination.
Normal field of the curves
For each region, we first compute the normal field of the boundary : given two consecutive boundary points p1 = (x1, y1) and p2 = (x2, y2), we compute the tangent vector v = (x2− x1, y2− y1). So, the normal vector at p is given by n = (y2− y1, x1− x2, 0).
The coordinate z = 0 indicates that the image plane is the projection plane relative to the viewer and the curve belongs to the object's silhouette.
where v i− 2, v i− 1,v i +1 and v i +2 are the normal vectors of the p i neighbors. The normal vectors of the internal curves of the region are computed in a similar way.
Normal field of the interior
for each point p = (x, y, z) in ℝ.
If the region has internal curves, the normal vectors of theses curves may be used to compose the final normal field of the interior of the region during the interpolation process presented above. If these internal curves are not considered during the interpolation process, only the boundary curves will affect the normal field of the region.
Normal field analysis
We now do an analysis of the surface "generated" by our normal interpolation method. Is it smooth? How it behaves over its domain (the image plane)? First of all, we demonstrate that if C is a circle our interpolation technique provides exactly the normal field of a sphere.
As ny (p) = 0 by the symmetry, . Therefore, the normal field is exactly the same of a sphere of radius 1 centered at the origin.
When we consider internal curves to interpolate the normals, points on these curves are presented as singularities on the surface. On the other hand, if we consider only the boundary, the surface behaves smoothly over the domain.
The proposed region-tree structure is also suitable for applying attribute operators to the regions and to the curves of the cartoon. As the flexibility of our region-tree allows quick access to topological structures present in the cartoon, we can modify, in an efficient and independent way, the attributes of curves and regions. Some attributes are color, thickness, normals, depth, etc. To demonstrate this flexibility, we implemented two normal operators: the scale region operator, and the depth curve operator.
Automatic colorization across a frame sequence
The process of coloring manually each frame in a cartoon animation sequence is one of the most laborious steps. This process requires extensive work by the artist, both due to the large number of frames as the amount of detail in each one of them.
We present an automatic colorization technique based on our region-tree representation. The region-tree of consecutive frames allow the color transfer along the cartoon sequence.
If during the animation the region-tree is changed (regions appear/ disappear) a backtrack is performed to recover the information.
The first step to enable an automatic colorization is to find out the correspondence from consecutive frames. One way to obtain the best associations is to perform a regions tracking between both region-tree. In general, those frames present small variations throughout the scene, therefore, the association can be performed by analyzing parameters like position, area, shape and topology .
Adjacency graph and neighborhood function
The adjacency graph is utilized in this work as way to make the region's tracking in a consistent way.
arg max c R ,dist ( p Qj ).
1≤ j ≤m
So, if an image's region R has n neighbors, it will have n neighborhood functions (see Figure 8 (right)).
Associations between consecutive frames
The simplest way to create associations between regions from two consecutive frames could be comparing all the vertices from both frames graph. However, this option is also computationally expensive. To solve this problem, we propose the use of a region-tree in order to reduce the cost.
We execute a breadth-first search in the source-graph and destiny-graph initialized from the image background, since this is the only region of occurrence guaranteed in all images. In each search iteration, we analyze parameters of the area, position, contours, and neighborhood between regions represented by the compared vertices. The neighborhood parameter is defined by the equivalence degree (ED) from two regions.
Recursive regions tracking - recovering occluded regions
In the animation process, there are cases where a region can disappears in a frame and shows up in another one in the sequence. This happens every time that we have occlusion between regions during the animation. For those cases, the regions tracking (Section 4.1) is unsatisfying for the colorization process. Our work proposes tracking in a recursive way to improve the process by recovering the information of these occluded region.
In the recursive regions tracking, whenever a color information is passed from a region A to a region B in the consecutive frame, the vertex relative to the region A is marked. Once the frame is colorized, the algorithm verifies if there is any region not colorized in the frame. If all regions are colorized, the process will continue making a better association between frames as described in Section 4.3.
For the case where there is a not colorized region, the process is temporarily interrupted. Through the images adjacency graph, we find out which are the n neighboring regions of the non-colored one. In this case a reverse search is done through previous frames. In each analyzed frame, we look for regions corresponding to the n regions we identified. Once the corresponding regions are found, we check if they are unmarked neighboring regions. If this occurs, the unmarked regions are stored in a list of possible candidates to be equivalent to the non-colorized region. These candidates are collected in each frame.
Results and discussion
These regions are new in the frame so the algorithm does not recognize them. In the recursive tracking, when the character's face comes back to being one region representation the algorithm allows recover its color by matching its adjacency graph with the first one.
In these cases, we can observe that meaningful changes in the adjacency graph resulting from occlusion and/or creation of regions may cause less impact on the sequence if the recursive tracking is performed. Once the color information is lost, methods like Bezerra et al.  are not able to recover the informations causing many frames without proper colorization.
Figure 15 shows the occlusion problem in the bear's left leg. In the first image sequence, the color information cannot be recovered since this information is lost in the fifth image because there was a significant topological change such as a region splitting into two. Secondly regions having its neighborhood drastically altered such as the left eye, the hats brim, and the tie are lost in Bezerra et al.  tracking. In the case of the eye, it occurs when the eye links to the bear's face region. In the hats brim case, it occurs when the brim is linked to a new region the bear's ear. Finally, in the tie case, it occurs because one region disappears joining the bear's arm.
In Figure 16 we do not have a region occlusion. On the other hand, when the dog's hands touches his ears two regions are created changing the adjacency graph. Since these regions are new the algorithm do not colorize them. In the first sequence, once the color is lost, the method from Bezerra et al.  is not able to colorize the following frames. The recursive method however (second sequence), allows that the color information is propagated from previous frames making possible the gradual and complete cartoon's colorization (far right image).
Our method present some limitations both in the method of normal vector calculation as in the automatic colorization.
In the normal map calculation, if a point belongs to a curve, its z-component is 0. If this curve is interior to a region, our interpolation scheme will create singularities in the surface. It would be necessary to define normal operators to avoid these singularities.
Another limitation is that illumination is not transferred frame by frame. It is computed separately in each frame.
Conclusion and future works
This work presented a new strategy based on a region-tree structure to represent 2D drawings. Unlike usual structures, our representation scheme explores both the time coherence of the topological structures, as well as the local spatial information of each frame. Therefore our method is both suitable for colorization and to illumination process.
Due to the topological and geometrical information of this region-tree representation, we developed a method for approximating lighting to 2D drawings: we compute the normal mapping of the drawing based on our new topological structure. Our illumination method is suitable for independently shading parts of the drawing, like curves and regions.
The topological structure inspired a colorization method, where color is transferred recursively during the animation. With the recursive tracking method, we can recover most of the lost regions in the animation.
Future works include: verify time coherence together with spacial local structure to transfer frame by frame illumination; define new attribute operators; optimize the normal interpolation, by avoiding that invisible curve points affect the resulting normal inside the region.
The authors would like to thank L. Velho for encouraging this work's production. The authors also would like to thank J. Paixa˜o for the text's revision and D. Cedrim.
- Hunter GM: Computer animation survey. Comput Graphics 1997, 2: 225-229.View Article
- Lasseter J: Principles of traditional animation applied to 3D computer animation. Comput Graphics 1997,21(4):35-44.View Article
- Catmul E: The problems of computer-assisted animation. Sig- graph 1987,12(3):348-353.View Article
- Bezerra H, Velho L, Feijo´ B: A computer-assited colorization algorithm based on topological differences. In Sibgrapi 2006, 121-127.
- Chang CW, Lee SY: Automatic cel painting in computer-assisted cartoon production using similarity recognition. In The Journal of Visualization and Computer Animation 1998.
- Carvalho MA, Couprie M, Lotufo RA: Image segmentation by analysis of scalespace. In Sibgrap 2002, 403-411.
- Carvalho MA, Lotufo RA: Hierarchical regions matching in image sequences through association graph. In Sibgrap 2001, 396-404.
- Garrido S, Garrido L, Salembier P, Casas JR: Representing and retrieving regions using binary partition trees. In International Conference on Image Processing 1999.
- Horn B: Shape from shading. MIT Press, Cambridge, MA; 1989.
- Wu TP, Sun J, Tang CK, Shum HY: Interactive normal reconstruction from a single image. Siggraph Asia 2008,27(119):1-119. 9View Article
- Igarashi T, Matsuoka S, Tanaka H, Teddy : A sketching interface for 3D freeform design. In Siggraph courses 2007.
- Zeleznik RC, Herndon KP, Hughes JF: SKETCH: an interface for sketching 3D scenes. In Siggraph courses 2007 2007.
- Phong BT: Illumination for computer generated pictures. Communications of the ACM 1975, 18: 311-317. 10.1145/360825.360839View Article
- Johnston SF: Lumo:illumination for cel animation. In Non- Photorealistic Animation and Rendering 2002, 45-ff.
- Sýkora D, et al.: Adding depth to cartoons using sparse depth (in)equalities. Computer Graphics Forum 2010,29(2):615-623. 10.1111/j.1467-8659.2009.01631.xView Article
- Hock SS, Tian F: Computer-assisted coloring by matching line drawings. Visual Computer 2000,16(5):289-304. 10.1007/s003719900068View Article
- Sy´kora D, Burianek J, Zara J: Segmentation of black and white cartoons. Spring Conference on Computer Graphics 2003, 245-254.
- Sýkora D, Burianek J, Zara J: Unsupervized colorization of black-and-white cartoons. Non-Photorealistic Animation and Rendering 2004, 121-127.
- Trigo PG, Johan H, Imagire T, Nishita T: Interactive region matching for 2D animation coloring based on feature's variation. Ieice Transactions 2009, 92: 1289-1295.
- Zhang TY, Suen CY: A fast parallel algorithm for thinning digital patterns. Communications of the ACM 1984, 27: 236-239. 10.1145/357994.358023View Article
- Vandevenne L: Lode's computer graphics tutorial flood fill.2004. [http://lodev.org/cgtutor/floodfill.html]
- Parker J: Pratical Computer Vision Using C. John Wiley and Sons, New York; 1994.
- Courant R, Robbins H: What is Mathematics. Oxford University Press, England; 1941.
This article is published under license to BioMed Central Ltd. This is an Open Access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.