A Study on Visual Saliency Detection in Infrared Images Using Boolean Map Approach

1. Introduction

Visual saliency is a subjective perceptual quality that makes some objects of an image appear distinct from their surroundings and attract viewers’ attention. Visual saliency detection is a classification task on the pixel level, in which each pixel is assigned a saliency score to determine whether it belongs to the salient objects or to the background. As visual saliency models seek to represent the human attention mechanism, visual saliency research plays an important role in computer vision. Moreover, visual saliency detection is an essential part in various vision-based applications such as content-aware image manipulation [1], video compression [2], image segmentation [3], and object recognition [4].

Saliency detection techniques can be classified into two major categories: bottom-up and top-down. Bottom-up approaches utilize low-level features such as color, texture, orientation; while top-down approaches depend on high-level factors such as prior knowledge regarding tasks or events. In the beginning, saliency detection algorithms were mainly based on bottom-up approaches. Recently, machine learning techniques have been extensively used in computer vision [5,6], furthermore, they become an important part in saliency detection systems that use top-down approaches. Borji et al. [7] performed evaluations on 41 state-of-the-art models using seven data sets. The report provided an in-depth analysis for the development of saliency research over the last few years. Another widely used benchmark for the evaluation of saliency detection methods is the Massachusetts Institute of Technology (MIT) Saliency Benchmark [8], which contains the results from over 70 models.

In addition to color or grayscale images, which are acquired from conventional imaging devices, there are infrared images representing specialized visual data. These images have various applications in critical fields such as security, surveillance, military [9], or medicine [10]. Despite the fact the number of saliency detection algorithms is high and continues to grow, techniques applicable specifically for saliency detection in infrared images are not as common as those applicable to color images. Saliency detection algorithms that work well with color images may result in poor performance when applied to infrared images. This is because infrared images are usually highly noisy and are of low contrast. Moreover, they have low resolution and insufficient features. As shown in Fig. 1, the detail of an infrared image (acquired from [11]) is insufficient even for humans to recognize salient objects. In Fig. 2, a high-performance saliency detection algorithm [12] was applied to an infrared image. This algorithm was originally designed for three-channel color images. As we can see from the result, this algorithm was not able to achieve acceptable result when being applied to the infrared image.

Several attempts have been made to overcome the difficulties discussed above. Li et al. [13] used the Gaussian scale-space representation to compute two types of saliency maps, namely, a dark saliency map, and a bright one. Then signal theory was used to compute phase information for the purpose of detecting both bright and dark regions of an image in question. These regions were finally grouped, and a shape matching algorithm was applied for salient object detection. In [14], the authors first improved the quality and contrast of an image in a frequency domain; then, they combined luminance distribution and the gradient feature to expose the objects that have large gradient and compact distribution. The resulting saliency map for the considered infrared image was constructed by integrating these two features. Qin et al. [15] took advantage of the human attention mechanism and the information theory. They segmented the input image into regions, then calculated the contrast of each region and the spectral residual of the whole image. The final saliency image was obtained by combining the spatial correlation and spectral residual factors retrieved using spatial distance information.

In this research, we utilize Boolean maps [16] to overcome the problems of high noise and low contrast in infrared images. A Boolean map is a spatial representation that divides an image into two comple¬mentary components: the component that is selected and the component that is not selected. Zhang and Sclaroff [17] proposed a saliency detection algorithm based on this concept (namely, Boolean Map Saliency [BMS]). However, it produced imprecise results when applied to infrared images (Fig. 3). More details on the experiment and analysis will be discussed in the following sections.

Saliency detection in infrared images is a challenging problem due to the nature of infrared images. In our proposed method, we utilized several techniques to overcome the existing difficulties. By constructing multiple Boolean maps, we aimed to expose image features in different threshold levels to solve the low-contrast problem. Combining this method with an adaptive thresholding algorithm, we suppressed the noise in infrared images while preserving their visual details. The complete algorithm is described in detail in Section 2. Section 3 demonstrates the performance of the proposed method and provides the results of comparison against the BMS approach. This paper is concluded with Section 4.

2. Methodology

A Boolean map is a spatial representation that divides a visual scene, usually an image, into two complementary components: the component that is selected and the component that is not selected. This theory is an interpretation of the human attention mechanism proposed by Huang and Pashler [16]. The key principle of this theory is that the visual attention of human is limited and it can only capture one object at a time. As illustrated in Fig. 4, an image with two objects is segmented into two Boolean maps showing the mechanisms of human attention.

Boolean maps were first applied for the purpose of saliency detection by Zhang and Sclaroff [17]. In this algorithm, a set of binary images is generated from a given image by thresholding different color channels of the image randomly. Then a saliency map is computed by discovering surrounded regions via the topological analysis of Boolean maps (Fig. 5). The advantage of this algorithm is the ability to achieve superior performance while retaining simplicity. However, as shown previously in Fig. 3, this algorithm does not produce acceptable results when being applied to infrared images. In the following sections we describe our approach to utilize Boolean maps theory for saliency detection in infrared images. The overall flowchart of the proposed system is illustrated in Fig. 6.

Given an infrared image I in which pixel values lie in the interval [0, 255], in the first step, N-1 Boolean maps [TeX:] $$B_{i}$$ are calculated as follows:

where the function [TeX:] $$T h(\cdot)$$ assigns 1 to pixels in I for those values that are greater than [TeX:] $$\theta_{i}$$ and 0 otherwise. The value of [TeX:] $$\theta_{i}$$ is defined as:

We do not perform thresholding at i = 0 (minimum gray level) and i = N (maximum gray level) because the thresholding results in these cases are meaningless, the image will be black or white entirely. Fig. 7 illustrates the Boolean maps created at this step with N=10. The topmost image is the original infrared image, the below nine images are Boolean maps generated from this image. The threshold levels increase from left to right and from top to bottom. As can be seen in Fig. 7, by using multiple threshold values, we successfully exposed visual features from the infrared image.

After obtaining the Boolean maps, the initial saliency map is calculated as a weighted sum of Boolean maps as follows:

where

which means that Boolean maps created using higher thresholds have higher weights. This weighting scheme is based on the observation that infrared imaging captures the temperature of objects. Hence, salient objects in infrared images are usually small and bright indicating heat sources such as humans or vehicles. Fig. 8 shows that the weighted sum enables combining features from multiple Boolean maps exposing potential salient objects. However, the initial saliency map remains unclear at this step.

The initial saliency map S may contain noise due to the nature of infrared imaging. To eliminate the noise, we threshold the initial map by using a modified version of Otsu’s method proposed in [18]. This technique uses entropy information as an additional weighting scheme for Otsu’s method. It is worth noting that values of pixels in [TeX:] $$B_{i}$$ are 0 and 1. Before going to the next step, the values of pixels in S are rescaled to the range [0, 255] (rounded down). For the rescaled initial saliency map S, let the number of pixels with value i be [TeX:] $$m_{i}$$ and M be the total number of pixels in S, a posteriori entropy of the map is then defined as:

Where

If we maximize [TeX:] $$H_{n}^{\prime},$$ we get a trivial result:

This problem is solved by converting (5) into an objective function:

where

Originally, the objective function of Otsu's method is written as follows:

where [TeX:] $$\varphi \text { and } \mu$$ are the probability of class occurrence and the average pixel values of each class (foreground and background in this case) respectively, and the value [TeX:] $$k^{*}$$ is the optimal value for thresholding the image. The entropy calculation [TeX:] $$\psi$$ is combined with Otsu's method to create a new objective function:

The modified version is better than the original method in term of highlighting small salient regions and is not affected by noise, as shown in Fig. 9.

After thresholding, the initial saliency map becomes clearer, exposing salient objects. However, it is still cluttered by small leftover noise particles. In the final step, the saliency map is further refined with the use of opening, a morphological operation to eliminate noise particles. Thereafter, we apply a Gaussian filter to create the final saliency map (Fig. 10).

3. Experimental Results

We evaluated the performance of the proposed method using Ohio State University (OSU) Thermal Pedestrian Database from [19]. This data set contains infrared image sequences from a camera mounted on a rooftop of a building. The main objects of interest in this data set are pedestrians. The images are in gray scale and have a resolution of 360×240 pixels. Because this data set is used for pedestrian detection in the first instance, only bounding boxes around pedestrians are provided as the ground truth. We considered pedestrians as salient objects; hence, we used the bounding boxes as the indicators for salient regions. Fig. 11 shows several sample images from the data set: the first row provides original infrared images and in the second row, the locations of pedestrians in the corresponding images from the first row are indicated by yellow bounding boxes.

The main parameter in the proposed method is N, the number of thresholding levels. If N is small, the number of exposed features is insufficient. If N is large, some Boolean maps will become redundant because the threshold values are very close to each other, which leads to unnecessary computation. In Fig. 12, we demonstrated the calculations of initial saliency maps with different values of N. The visual details of initial map created with N=5 are insufficient, while the maps created with N=20 and N=40 are nearly identical. In the experiments, we selected N = 20 to achieve the best results while retaining low computational cost.

We also compared our method with the Boolean Map Saliency approach by Zhang and Sclaroff [17], which is also a saliency detection technique utilizing Boolean maps. The BMS can be considered as the first attempt to apply the Boolean map theory in saliency detection. Originally, BMS was designed to detect salient objects in three-channel color images. In this experiment we examine the performance of BMS when being applied to infrared images. The implementation of BMS was obtained from the author’s website. It is also worth noting that for BMS we used the maps of eye fixation predictions instead of salient object detection results, because the saliency maps were very imprecise. The saliency detection results are illustrated in Fig. 13.

In Fig. 14, the original infrared images are shown in the left column (Fig. 14(a), (d), (g), (j), and (m)), in which the pedestrians are highlighted by yellow bounding boxes. Each row shows the detection results obtained through BMS and through the proposed method for the respective infrared image. In the first row, there are seven pedestrians in the image (Fig. 14(a)). The result from BMS indicated that there are two large salient regions (Fig. 14(b)), while our algorithm produced six distinct salient regions indicating locations of the pedestrians (Fig. 14(c)); the remaining pedestrian on the right side of the image is small and hard to detect. Detection results are similar for other tests. BMS could only produce unclear large salient regions, while the proposed method was able to produce precise, distinct salient regions that indicated pedestrians’ locations clearly. As can be seen, the proposed method produced better and clearer saliency maps for this data set. In particular, it captured the general shapes of the pedestrians. The detected salient regions locate accurately inside the bounding boxes. In its turn, BMS produced unclear salient regions, especially for regions that are close to image boundaries (in Fig. 14(e), (k), and (m)), the pedestrians at top left corners are unclear). To a certain extent, the precisely detected salient regions from the proposed method can be used as a feature in pedestrian tracking systems employing infrared video recorders.

4. Conclusion

In computer vision, visual saliency detection is a pixel-level classification task, in which each pixel is assigned a saliency score to determine whether it belongs to the salient objects or to the background. Research on visual saliency plays an important role in computer vision as it helps other vision-based algorithms achieve remarkable results because visual saliency models are representations of the human attention mechanism. Among various types of visual data, infrared images are related to a specialized visual data, which has various applications in critical fields. Even though the number of saliency detection algorithms is high and continues growing, techniques for saliency detection in infrared images are not as well developed as the ones in color images. Saliency detection algorithms that work well with color images may show unsatisfactory performance when being applied to infrared images. It is because infrared images are usually highly noisy and low-contrast, besides they have low resolution and insufficient visual features. Saliency detection in infrared images remains a challenging problem due to the nature of input data.

In this paper, we introduced a straightforward approach for saliency detection in infrared images. In this approach, first the input image is thresholded into several Boolean maps, and the number of maps is selected based on observation for the best results. Thereafter, an initial saliency map is calculated as a weighted sum of created Boolean maps. Finally, a high-quality saliency map is constructed by further refining the initial map through thresholding, morphology operation, and Gaussian filter. By constructing multiple Boolean maps, we exposed image features in different threshold levels, solving the low-contrast problem. Combining it with the adaptive thresholding algorithm, we suppressed the noise in infrared images while preserving details of the images. The experiment showed that the proposed method produced high performance results when applied to real-life infrared image sequences. For future work, we plan to utilize Boolean maps further, for such purposes as extracting more features or performing image enhancement to improve and to increase the performance of the proposed method of saliency detection in infrared images system.

Acknowledgement

This research was supported by Basic Science Research Program through the National Research Foundation of Korea (NRF) funded by the Ministry of Education (No. 2020R1F1A1067496).