Revisiting Color Efficient Coding through Material Perception

An essential objective of early visual processing is to handle the redundant inputs from natural environments efficiently. For instance, cone signals from natural environments are highly correlated across cone types, indicating channel redundancy. Early visual processing transforms the signals into color and luminance information, such as principle component analysis, and thus achieves efficient and decorrelated representations of natural scenes [1-3]. Building on these findings, previous research has investigated the effect of color on visual tasks such as object recognition using grayscale conversion, which separates luminance from color [4, 5]. However, recent work suggests that when focusing on object materials, color and luminance remain highly redundant due to complex optical properties [6, 7]. Although this finding indicates that there may be a more efficient decomposition of signals, the specific algorithms remain unknown. This study derives that a classic computer graphics algorithm, the median cut [8], offers a novel approach to enhance visual processing efficiency while capturing diagnostic features to separate material from object geometry information (Fig. 1a and 1b). Human behavioral experiments show that color reduction based on the algorithm disturbs material classification while preserving object recognition (Fig. 1c). These findings suggest that object geometric structures are available only from low-bit information. Finally, considering material information can be estimated from summary statistics of image sub-spaces, this study suggests an efficient decomposition of input color images (Fig. 1d). O_FIG O_LINKSMALLFIG WIDTH=200 HEIGHT=146 SRC="FIGDIR/small/644715v1_fig1.gif" ALT="Figure 1"> View larger version (46K): org.highwire.dtl.DTLVardef@1ae46baorg.highwire.dtl.DTLVardef@a56667org.highwire.dtl.DTLVardef@e61973org.highwire.dtl.DTLVardef@90ffbe_HPS_FORMAT_FIGEXP M_FIG O_FLOATNOFig. 1.C_FLOATNO (a) Demonstration of canceling the material appearance difference (wet and dry) of the same stone image while maintaining its geometric structure using a color reduction algorithm called the median cut. The reduced color image consists of four color points in a RGB space, corresponding to 2-bits index color. (b) Schematic visualization of the median cut. The input color space is divided with a plane according to the median of the cumulative histogram of each axis. Each point in the left figure represents a pixel of the RGB color image. The divisions by the plane define the subspaces, and all points within each subspace are replaced with a single mean value. The right figure shows an example with 2-bits. According to the histogram of each subspace, the intensity order of the mean values is maintained consistent with the original distribution (Supplementary Fig. S1). (c) Results of human behavioral experiments. One hundred participants engaged in ensemble 2AFC tasks involving three material classifications and one object classification: Glossy vs. Matte, Wet vs. Dry, Translucent vs. Opaque for material classification, and Animal vs. House for object classification. Participants were divided into four groups based on the color condition to ensure that no participant saw the same original image. Violin plots in each panel illustrate the accuracy distribution for these image conditions, with the mean accuracy indicated by red dots. The horizontal dashed line marks the chance level accuracy. The results demonstrate that color reduction decreases material classification accuracy but does not affect object classification accuracy. (d) Color image decomposition by the median cut with Gaussian ellipsoid projection. While the median cut preserves the geometric structure of an image, the residual material information can be efficiently represented by the summary statistics of the Gaussian ellipsoid within each subspace, i.e., the mean and standard deviation. C_FIG