Three-Color Balancing for Color Constancy Correction

This paper presents a three-color balance adjustment for color constancy correction. White balancing is a typical adjustment for color constancy in an image, but there are still lighting effects on colors other than white. Cheng et al. proposed multi-color balancing to improve the performance of white balancing by mapping multiple target colors into corresponding ground truth colors. However, there are still three problems that have not been discussed: choosing the number of target colors, selecting target colors, and minimizing error which causes computational complexity to increase. In this paper, we first discuss the number of target colors for multi-color balancing. From our observation, when the number of target colors is greater than or equal to three, the best performance of multi-color balancing in each number of target colors is almost the same regardless of the number of target colors, and it is superior to that of white balancing. Moreover, if the number of target colors is three, multi-color balancing can be performed without any error minimization. Accordingly, we propose three-color balancing. In addition, the combination of three target colors is discussed to achieve color constancy correction. In an experiment, the proposed method not only outperforms white balancing but also has almost the same performance as Cheng’s method with 24 target colors.


Introduction
A change in illumination affects the pixel values of an image taken with an RGB digital camera because the values are determined by spectral information such as the spectra of illumination [1][2][3]. In the human visual system, it is well known that illumination changes (i.e., lighting effects) are reduced, and this ability keeps the entire color perception of a scene constant [4]. In contrast, since cameras do not intrinsically have this ability, white balancing is applied to images [5]. Otherwise, in the field of image segmentation or object recognition, we may suffer from color distortion caused by lighting effects [2,3,[6][7][8][9][10].
In contrast, Cheng et al. [5] proposed a multi-color balancing for reducing lighting effects on both chromatic and achromatic colors. In Cheng's method, multiple colors with remaining lighting effects, called "target colors" in this paper, are used for designing a matrix that maps the target colors into ground truth ones. Although Cheng's method contributes more to improving color constancy correction than white balancing, it still has three problems: choosing the number of target colors, selecting the combination of target colors, and minimizing error which causes the computational complexity to increase.
In this paper, we propose a three-color balance adjustment. This paper has two contributions. The first one is to show that if the combination of three target colors that provides the lowest mean error under three-color balancing is chosen, the three-color balancing will have almost the same performance as Cheng's method with 24 target colors. The second one is to show that there are multiple appropriate combinations of the three target colors, which can offer a low mean error even under various lighting conditions. From the second, some examples of the combination of three target colors are recommended, which can maintain the performance of three-color balancing under general lighting conditions. As a result, the proposed three-color balancing can more improve color constancy correction than WB. Furthermore, the proposed method gives us a new insight for selecting three target colors in multi-color balancing.
In experiments, three-color balancing is demonstrated to have almost the same performance as Cheng's method with 24 target colors, and it outperforms conventional white balance adjustments. Additionally, the computational complexity was tested under various numbers of target colors, and the proposed method achieved faster processing than Cheng's method with error minimization algorithms.

Related Work
Here, we summarize how illumination change affects colors in a digital image, and white balancing and Cheng's multi-color balancing are summarized.

Lighting Effects on Digital Images
On the basis of the Lambertian model [1], pixel values of an image taken with an RGB digital camera are determined by using three elements: spectra of illumination E(λ), spectral reflectance of objects S(λ), and camera spectral sensitivity R C for color C ∈ {R, G, B}, where λ spans the visible spectrum in the range of [400, 720] (see Figure 1) [2,3]. A pixel value P RGB = (r P , g P , b P ) in the camera RGB color space is given by Equation (1) means that a change in illumination E(λ) affects the pixel values in an image. In the human visual system, the changes (i.e., lighting effects) are reduced, and the overall color perception is constant regardless of illumination differences, known as chromatic adaptation [4]. To mimic this human ability as a computer vision task, white balancing is typically performed as a color adjustment method. Figure 1. Imaging pipeline of RGB digital camera.

White Balance Adjustments
By using white balancing, lighting effects on a white region in an image are accurately corrected if the white region under illumination is correctly estimated. Many studies on color constancy have focused on correctly estimating white regions, and a variety of automatic algorithms are available [11][12][13][14][15]17,18].
White balancing is performed by M WB in Equation (2) is given as where P XYZ = (X P , Y P , Z P ) is a pixel value of an input image I XYZ in the XYZ color space [32], and P WB = (X WB , Y WB , Z WB ) is that of a white-balanced image I WB [33]. M A with a size of 3 × 3 is decided in accordance with an assumed chromatic adaptation transform [33]. (ρ S , γ S , β S ) and (ρ D , γ D , β D ) are calculated from a source white point (X S , Y S , Z S ) in an input image and a ground truth white point (X D , We call using the 3 × 3 identity matrix as M A "white balancing with XYZ scaling" in this paper. Otherwise, von Kries's [20] and Bradford's [21] chromatic adaptation transforms were also proposed for reducing lighting effects on all colors under the framework of white balancing. For example, under the use of von Kries's model, M A is given as

Cheng's Multi-Color Balancing
White balancing is a method that maps a source white point in an image into a ground truth one as in Equation (3). In other words, colors other than white are not considered in this mapping, although the goal of color constancy is to remove lighting effects on all colors. To address this problem, various chromatic adaptation transforms such as those of von Kries [20], Bradford [21], CAT02 [22], and the latest CAM16 [23] have been proposed to reduce lighting effects on all colors under the framework of white balancing.
In addition, Cheng et al. proposed a method [5] for considering both achromatic and chromatic colors to further relax the limitation that white balancing has. In their method, multiple colors are used instead of white. Let T 1 = (X T1 , Y T1 , Z T1 ) , T 2 = (X T2 , Y T2 , Z T2 ) , T 3 = (X T3 , Y T3 , Z T3 ) , · · · , T n = (X Tn , Y Tn , Z Tn ) be n colors with remaining lighting effects in the XYZ color space. In this paper, these colors are called "target colors." Additionally, let · · · , G n = (X Gn , Y Gn , Z Gn ) be n ground truth colors corresponding to each target color. To calculate a linear transform matrix, let two matrices T and G be respectively. In Cheng's method, n was set to 24 (i.e., all patches in a color rendition chart) [5]. If n is greater than three, T and G will be a singular matrix, and the inverse matrix of T is not uniquely determined. Hence, the Moore-Penrose pseudoinverse [5,34] is used, and the optimal linear transform matrix M + is given as However, as noted by Funt et al. [35], because the illumination across the target colors is generally not uniform, calculating M + only with Equation (7) is insufficient to perform calibration for such circumstances. Hence, with M + calculated in Equation (7) as the input, the method in [35] is also used to minimize the sum of error: As well as white balancing, a pixel value corrected by Cheng's method P CNG is given as where M † is calculated as in Equations (6)- (8). However, this method still has three problems that have not been discussed: (i) How the number of target colors is decided.
(ii) How the combination of n target colors is selected.
(iii) How the computational complexity of Cheng's method is reduced.
Additionally, as for (iii), if the number of target colors is increased, the computational complexity of Cheng's method will be increased due to the use of Equation (8).

Proposed Method
In this section, we investigate the relationship between the number of target colors and the performance of Cheng's multi-color balancing. From the investigation, we point out that if the combination of three target colors that offers the lowest mean error in three-color balancing is chosen, the three-color balancing will have almost the same performance as Cheng's method with 24 target colors. Accordingly, we propose a three-color balance adjustment that maps three target colors into corresponding ground truth colors without minimizing error. Additionally, the selection of three target colors is discussed, and we recommend some example combinations of three target colors, which can be used under general illumination. Finally, the procedure of the proposed method is summarized.

Number of Target Colors
In this section, we argue the relationship between the number of target colors and the performance of color constancy correction.
White balancing with XYZ scaling and Cheng's multi-color balancing were applied to the images in Figure 2b,c, respectively, where n was set to 1 for white balancing, and it was set to 3, 4, 5, and 24 for Cheng's method. Additionally, corresponding ground truth colors were selected from Figure 2a.
The red band inside boxes indicates the median.
For every combination of target colors, the performance of each adjustment was evaluated by using the mean value of reproduction angular errors for the 24 patches. The reproduction angular error is given by where P is a mean-pixel value of an adjusted patch, and Q is that of the corresponding ground truth one [36]. In this experiment, P corresponds to an adjusted patch in Figure 2b,c, and Q corresponds to a patch in Figure 2a. From the figure, two properties are summarized as follows.
(i) Cheng's method had a lower minimum mean error than white balancing.
(ii) When n ≥ 3, Cheng's method had almost the same minimum mean error as that of n = 24.
Moreover, when n = 3 is chosen, M + is reduced to M † if T and G have full rank, so Equation (8) is not required. Accordingly, in this paper, we propose selecting n = 3. Figure 4 shows an overview of the proposed three-color balancing. In a manner like white balancing, three-color balanced pixel P 3CB is given as

Proposed Three-Color Balancing
Let ) be three target colors in the XYZ color space. Note that the location of each target color is known; or target colors can be estimated by using a color estimation algorithm although color estimation algorithms do not have enough performance data yet for estimating various colors [37][38][39][40]. Additionally, let where When both T and G have full rank, M 3CB is designed by By applying M 3CB in Equation (14) to every pixel value in input image I XYZ , balanced image I 3CB is obtained, where target colors in I XYZ are mapped into ground truth ones.
The proposed method is considered as a special case of Cheng's multi-color balancing. If the number of target colors is three and both T and G have full rank, T −1 is uniquely determined, and M 3CB can be designed without any error minimization algorithms.

Selection of Three Target Colors
Under one light source, the optimal combination of the three target colors has to be selected by testing all of the conceivable combinations, as discussed in Section 3.1. However, because illumination continuously changes in real situations, it is difficult to repeat performing the selection for every light source. Accordingly, in this section, appropriate combinations of three target colors, which offer a low mean reproduction error are recommended by experimentally testing all combinations of three target colors for 500 images taken under various light sources.
We used the ColorChecker dataset prepared by Hemrit et al. [41], in which pixel values of the 24 patches in a color rendition chart were recorded under 551 illumination conditions. Additionally, the ground truth colors selected in Figure 2a were used for this discussion. By using 500 images in the dataset, all combinations of three colors were tested, and the performance of the proposed method with each combination was evaluated in terms of the mean reproduction error for the 24 patches. From the experiment, a combination of three colors (index 6, 9, and 14 in Figure 5) was selected at which the proposed method had the minimum value among 500 mean Err values.  Additionally, we used the other 51 images from the dataset to evaluate the effectiveness of the combination of three target colors. In Table 1, three-color balancing was compared with Cheng's method and white balancing with XYZ scaling, referred to as 3CB (6,9,14), Cheng , and WB (XYZ), respectively. The selected combination shown in Figure 5 was used for three-color balancing, and all of the 24 target colors were chosen for Cheng's method. For white balancing, the white patch (index 6) was selected.  (6,9,14) had almost the same performance as Cheng . In addition, 3CB (6,9,14) and Cheng (1-24) outperformed WB (XYZ) for almost all chromatic colors (index 7-24). In other words, WB (XYZ) did not reduce lighting effects on colors other than achromatic colors (index 1-6). Figure 6 shows box plots of three-color balancing with four more combinations of three target colors in addition to 3CB (6,9,14), where 3CB (6,9,14), 3CB (11,14,24), 3CB (4, 11, 13), 3CB (6,14,16), and 3CB (11,13,24) had the first, second, third, fourth, and fifth smallest mean reproduction error, respectively.
From the figure, the five combinations of three target colors had almost the same result. This means that the effectiveness of three-color balancing can be maintained even when (6,9,14) is replaced with other combinations. Therefore, for example, we recommend (6,9,14) and (11,14,24) from Figure 6 as the top two combinations of target colors, which can maintain high performance under general illumination.

Procedure of Three-Color Balancing
When we assume that three colors selected from 24 colors in Figure 2a are used as ground truth colors, the procedure of the three-color balancing is given as below.
(i) Select three ground truth colors from Figure 2a. Let them G 1 , G 2 , and G 3 , respectively. (ii) Prepare three objects in a camera frame, in which each object color corresponds to one of the ground truth colors. Compute three target colors T 1 , T 2 , and T 3 from the region of each object, respectively. (iii) Apply three-color balancing by using T 1 , T 2 , and T 3 and G 1 , G 2 , and G 3 , following Section 3.2.
Using (6,9,14) or (11,14,24) as a combination is recommended in step (i), as discussed in Section 3.3. Additionally, if the ground truth colors other than those from Figure 2a are used, the selection of target colors discussed in Section 3.3 should be carried out by using the newly determined ground truth colors.

Experiment
We conducted experiments to confirm the effectiveness of the proposed method.

Evaluation of Reducing Lighting Effects
In this experiment, the effectiveness of three-color balancing with the recommended combinations was demonstrated by using various colors and lighting conditions, where these experimental conditions were different from those in Section 3.3. Figure 7 shows images taken under three different light sources, where 10 more color regions were added to the color rendition chart in Figure 2, and the images included 34 color regions numbered from 1 to 34. Additionally, the lighting condition in Figure 7a is the same as that of Figure 2a, which means that the color regions numbered from 1 to 24 in Figure 7a correspond to those of Figure 2a.
The proposed method was applied to the input images, and it was compared with white balancing and Cheng's method ), where XYZ scaling (WB (XYZ)) and von Kries's model (WB (von Kries)) were applied as white balance adjustments. Target colors were selected from the color rendition chart indexed from 1 to 24 in Figure 7b,c. For white balancing, the white region (index 6) was chosen. Additionally, for Cheng's method, all of the 24 color regions were used as target colors. For the proposed method, two combinations of the three target colors determined in Section 3.3 were selected: (6,9,14) and (11,14,24), referred as to 3CB (6,9,14) and 3CB (11,14,24), respectively.
In Tables 2 and 3, the reproduction angular errors for Equation (10) are shown to objectively compare the corrected images. From the tables, the proposed method had almost the same performance as Cheng's method with 24 target colors in terms of the mean value and standard variation. Additionally, the proposed method and Cheng's method outperformed conventional white balance adjustments, although the mean value of 3CB (6,9,14) had a higher error than that of WB (von Kries) in Table 2. Therefore, the effectiveness of the proposed method for color constancy correction was confirmed. Table 2. Reproduction angular error (Err) for every object (deg) in Figure 7b. Color indices correspond to those in Figure 7. Indices inside () indicate target colors used in each adjustment.

Color
Pre  Table 3. Reproduction angular error (Err) for every object (deg) in Figure 7c. Color indices correspond to those in Figure 7. Indices inside () indicate target colors used in each adjustment.

Evaluation of Computational Complexity
In Cheng's method, all of the 24 patches in a color rendition chart are used as target colors. Additionally, when n ≥ 4, the computational complexity of Cheng's method will increase due to the use of Equation (8). In contrast, that of three-color balancing is low because Equation (8) is not required.
To evaluate the complexity, we implemented white balancing with XYZ scaling, three-color balancing, and Cheng's method on a computer with a 3.6 GHz processor and a main memory of 16 Gbytes (see Table 4). Note that the runtime of the equations shown in Sections 2.2, 2.3, and 3.2 was only measured for white balancing, Cheng's method, and three-color balancing, respectively. In other words, for example, image loading runtimes were excluded from the measurement. Figure 10 shows the runtime result in the case that each method was applied to the 51 test images in Section 3.3. As shown in Figure 10, while the proposed three-color balancing and white balancing had almost the same average runtime, Cheng's method required longer runtimes to minimize error. Moreover, in Cheng's method, if the number of target colors increases, it will result in a long runtime.

Discussion on the Rank of T and G
In the proposed three-color balancing, when T and G have full rank, M 3CB can be designed as in Equation (14). If T and G violate the rank constraints, their eigenvalues will include an almost zero value due to rank deficiency [42]. Figure 11 shows the relationship between three eigenvalues of T or G and the performance of three-color balancing under various combinations of three colors. In the figure, for visualization, the product of the three eigenvalues was calculated for the horizontal axis: where λ T1 , λ T2 , and λ T3 denote eigenvalues calculated from T, and λ G1 , λ G2 , and λ G3 denote those from G. λ T was calculated by using various combinations of three target colors selected from Figure 2a, and λ G was calculated by using corresponding ground truth colors from Figure 2b. In a manner like Figure 3, the performance of three-color balancing for every combination was evaluated in terms of the mean reproduction values for the 24 patches. From the figure, we confirmed that if one eigenvalue is nearly zero, i.e., the product of the eigenvalues is nearly zero, three-color balancing results in a high mean Err value. This means that the rank deficiency caused by the combination of linearly dependent color vectors significantly affects the overall color correction performance. Thus, the following results were verified: (i) If the product of eigenvalues is small, the mean value of reproduction errors will be high. (ii) There are many combinations of three target colors at which three-color balancing results in a high error. (iii) Analysis based on eigenvalues enables us to select three target colors without a large dataset as in Section 3.3.

Conclusions
In this paper, we proposed a three-color balance adjustment for color constancy. While conventional white balancing cannot perfectly adjust colors other than white, multi-color balancing including three-color balancing improves color constancy correction by using multiple colors. Additionally, this paper presented the choice of the number of n target colors and the selection of three target colors for the proposed three-color balancing. When the number of target colors is over three, the best performance of color constancy correction is almost the same regardless of n. Additionally, for three-color balancing, a combination of three target colors that enables us to achieve a lower reproduction error was determined with a large dataset. Moreover, no algorithms for minimizing error are required to use the proposed method, and this contributes to reducing computational complexity. Experimental results indicated that the proposed three-color balancing did not only outperform conventional white balancing but also had almost the same performance as multi-color balancing with 24 target colors.

Data Availability Statement:
The data presented in this study are available on request from the corresponding author.