1. Introduction
The re-identification of individuals across different cameras has fascinated many researchers in the area of computer vision. This technique is mainly used in criminal investigation and image retrieval, however, certain problems still remain to be addressed to effectively re-identify an individual using this approach. These problems include differences in camera position and performance, varying illumination conditions, variability of the individuals themselves, and varying appearances of different backgrounds.
Usually, researchers use standard distances, such as the Mahalanobis distance [1], the Euclidean distance [2], and the Bhattacharyya distance [3], to measure the similarity between two identities. However, when the person with the same ID crosses multiple non-overlapping cameras, different appearance features are affected by different factors such as the angle of view and the illumination.
Four images of the Market-1501 dataset [4] are shown in Fig. 1. The images have the same label and are shoot by disjoint cameras. From the chart, the differences which include the difference of color, contour, texture difference and background are shown. These different points bring great complexity of the person re-identification task.
Pictures of individual cross disparate camera with the same label on Market-1501 dataset [ 4].
There are two ways to address these problems. First, features are designed and selected according to the appearance individual of illumination changes, and the human body deformation robust. Kviatkovsky et al. [5] presented a colour structure of an internal distribution that is invariant to illumination and a covariance descriptor that is used for pedestrian re-identification. This unsupervised learning method is used to extract significant features of human images, which, in turn, can be used for pedestrian reidentification [6]. Second, in this approach, distance metric learning is used to learn a feature transformation or distance measure by which the subject is at the minimum distance from different cameras, while the other people are at the maximum distance from the same camera [7]. The distance learning metric mechanism is optimised so that the distance between the correct matches is less than the error matching and the probability of the distance between the two matches is maximised. A locally adaptive decision function consists of a joint-distance measure model and the locally adaptive threshold rule is proposed to pedestrian re-identification and to ensure that a good recognition rate is achieved [8]. Liao et al. [9] proposed the LOMO feature, which analyses the horizontal occurrence of local features and maximises the occurrence to make a stable representation to viewpoint changes. Similarly, Zheng et al. [10] used metric learning as well, but formulated it in a probabilistic manner. These researchers sought a distance that maximises the probability of the distance between a matching pair being smaller than that between a non-matching pair. Some new researches used the combined joint Bayesian methods to improve the recognition accuracy, but without considering the changes between different cameras [11-13].
Most of the above-mentioned methods are manual methods and do not use depth neural networks to extract features; therefore, these methods have some shortcomings. To overcome these disadvantages, a more efficient approach that can exploit the natural constraints given by the person re-identification task is proposed and constitutes the contribution to this research. First, the features are extracted from a network with potential camera distributions. Second, the Joint Bayesian formulation is redefined that can better serve person re-identification across different camera distributions. Then, the improved Joint Bayesian method is built by combining the characteristics of multi-camera shooting and person reidentification with the Joint-Bayesian method. The proposed method is better adapted to the scene re-ID by learning the same pedestrian lighting and background changes under different cameras.
This paper is organised as follows. In Section 2, the Joint-Bayesian method, which has been implemented for Bayesian face recognition, is introduced. In Section 3, the improved JBR method of related re-identification process analysis is proposed. In Section 4, the implementation and comparison of the existing and proposed methods are presented on two large-scale person re-ID datasets, that is, Market-1501 [4] and DukeMTMC-reID [14]. In Section 5, the concluding remarks are given.
2. Represented Joint Bayesian Method
A brief description of the research results of the former researchers is represented in this section. Based on the previous Bayesian face recognition method [15], the verification task can be formulated as a Bayesian decision problem. The appearance of a face is mainly influenced by two factors: intra-personal variation and extra-personal variation. Here two pictures, which can be faces or other object figures, are represented as [TeX:] $$x_{1} \text { and } x_{2}.$$ represents the difference between the two pictures. Let [TeX:] $$H_{I} \text { and } H_{E}$$ represent the intra-personal hypothesis and the extra-personal hypothesis. Based on the maximum a posterior (MAP) rule, the decision on the verification task is made by testing a log likelihood ratio [TeX:] $$r(\Delta)$$
This ratio r can be used to measure the similarity between two pictures. Chen et al. [15] proposed a joint formulation to directly model the joint distribution of [TeX:] $$\left\{x_{1}, x_{2}\right\}.$$ A verified object can be represented by the sum of two independent Gaussian variables as follows:
where x is the observed object with the mean of all objects subtracted, represents its identity, and is the object variation (e.g., lightings, poses) within the same identity. and follow two Gaussian distributions, [TeX:] $$N\left(0, S_{\mu}\right) \text { and } N\left(0, S_{\xi}\right).$$ Where [TeX:] $$S_{\mu} \text { and } s_{\xi}$$ are two unknown covariance matrices.
Based on the above prior knowledge, we can obtain a Gauss joint distribution with a mean value of 0. Both and are independent, and then the covariance of two face features is obtained as follows:
For the same person, [TeX:] $$\mu_{1} \text { and } \mu_{2}$$ are the same, [TeX:] $$\xi_{1} \text { and } \xi_{2}$$ are independent. Thus, the covariance matrix of the [TeX:] $$\mathrm{P}\left(\mathrm{x}_{1}, \mathrm{x}_{2} | \mathrm{H}_{I}\right)$$ distribution can be calculated as follows:
Based on the premise that people are different from each other, both and are independent. Thus, the covariance matrix of the [TeX:] $$\mathrm{P}\left(\mathrm{x}_{1}, \mathrm{X}_{2} | H_{E}\right)$$ distribution can be calculated as follows:
With the two conditional joint probabilities, the log likelihood ratio [TeX:] $$r\left(x_{1}, x_{2}\right)$$ can be obtained in a closed form with simple algebra operations:
where
Both A and G are negative semi-definite matrices. If A = G, the negative log likelihood ratio will degrade to the Mahalanobis distance. The log likelihood ratio metric is invariant to any full rank linear transform from the feature. More details of the above formulations can be found in the study of Chen et al. [15].
3. Improved Joint Bayesian Method
In this section, to clearly demonstrate the proposed method, the structure of improved Joint Bayesian method is shown in Fig. 2. Fig. 2 shows the images (left) of some pedestrian and the features (right, red filled circles) extracted from the convolutional neural network regarding the distribution of the cameras.Ci indicates the label of the camera, where i=1, …, 6. Different colours with double-headed arrows represent distance measurements with different Joint Bayesian matrices that are trained by features of each pair of cameras. For example, the green double-headed arrow shown in Fig. 2 indicates that if we calculate the feature of C6 to the feature of C3 of the distance, it is necessary to select the Joint-Bayesian matrices that are trained by features from the C6 and C3 cameras. The procedure is to extract features from a photo that is distributed across different cameras and then calculate the distance between these features using the Joint-Bayesian matrix.
Images (left) of some pedestrian and features (right, red filled circles) of the cameras’ distribution that has been extracted from the convolutional neural network. C i indicates the label of each camera. Different colours with double-headed arrows represent distance measurements.
Distribution of the entire region of the representative person, in which the yellow dotted line represents the distribution of the same person, and the blue dotted line represents the same camera under the same label, modelled by Gaussians.
Similar to face recognition, the picture of a pedestrian consists of two parts: differences between different individuals and differences between images of the same individual under varying conditions. The distribution of the entire region of the representative person, in which the yellow dotted line represents the distribution of the same person and the blue dotted line represents the same camera under the same label, is shown in Fig. 3. In the person re-identification task, the person distribution includes not only the distribution of different people in formulation (6) and the distribution of people under varying conditions, but also the feature distribution of the same person under disjoint cameras. When the person characteristic distance is calculated, in most cases, we need to calculate the similarity of two features under different cameras. Thus, the log likelihood ratio of distance can be redefined by camera pairs as follows:
In the above formula, [TeX:] $$S_{\text {pairs}}(\Delta)$$ denotes the similarity between two features that were taken from a pair of cameras, [TeX:] $$P\left(\Delta | H_{\text {Ipurs }}\right)$$ denotes the likelihood ratio of the distance of the two features that captured the pair of cameras in the same individual condition, and [TeX:] $$P\left(\Delta | H_{\text {Epairs }}\right)$$ denotes the likelihood ratio under varying individual conditions. To learn the distribution of different cameras, formula (6) is reorganized as the following formulation:
[TeX:] $$A_{i j} \text { and } G_{i j},$$ where i and j represent the camera transfer matrix from the i camera to the j camera. [TeX:] $$r_{p a i r}\left(x_{1}\right.,\left.x_{2}\right)$$ denotes the similarity of two individuals from a pair of cameras. However, it is clear that some of the information is lost and is not used to train the matrix. Because some of the pictures that are included in the whole dataset are captured by a pair of cameras, a global distance formulation is rebuilt as below:
where [TeX:] $$r_{g l o b a l}\left(x_{1}, x_{2}\right)$$ is the same as in formula (6), and denotes the global distance between two pictures. A and G represent the global Joint Bayesian matrix, and represents the coefficient of the global and the local Joint Bayesian matrix. With the formula (10), we can calculate the similarity of two images that contain global information and local information.
Training: The training strategy is similar to that in [15]. First of all, the dimension of features of 256 is reduced to simplify the computational complexity. Then, the global Joint Bayesian matrix is trained by identity labels, without adding camera labels, and the local Joint Bayesian matrices [TeX:] $$A_{i j}\text { and } G_{i j}$$ are applied to the camera label for training, until convergence occurs.
Testing: During the test period, the application of formulation (6) also needs two features: the reduction of the dimension of 256 and the matrix learned in the training phase. The matrix includes the global matrices (A and G) and the local matrices [TeX:] $$\left(A_{i j} \text { and } G_{i j}\right)$$ and is decided based on which camera the two images are taken with. To calculate the distance between two features using formulation (6), the matrix and features must be obtained.
4. Evaluating Results
4.1 Datasets and Settings
The proposed method is tested on Market-1501 and DukeMTMC-reID [14], which are two benchmark datasets for person recognition.
4.1.1 Market-1501
This is a person re-identification labelled dataset. This dataset contains a shot from six cameras of 32,668 labelled images of 1,501 identities. It uses the DPM algorithm to capture the pictures. In these images, training data includes 12,936 pictures and 751 identities, and test data include 23,100 pictures and 751 identities. In the test phase, 3,368 pictures with 750 identities are contained in the query, and 19,732 pictures with 751 labels are contained in the gallery.
4.1.2 DukeMTMC-reID
This dataset contains 1,812 identities captured by 8 cameras. In the dataset, 1,404 identities appear in 8 cameras, and the remaining 408 identities are the distracted images. The training and testing sets have 702 identities. In total, there are 2,228 query images, 16,522 training images, and 17,661 gallery images.
4.2 Experiments on Market-1501
4.2.1 Comparison with the state-of-the-art
One experiment is implemented on Market-1501, and the Siamese network on ResNet-50 is used [4]. The results of this experiment are shown in Table 1. When the Euclidean is used to evaluate the distance, the recognition rate for Rank-1 is 82.21%, and MAP is 62.35% on the Siamese network. When the Joint Bayesian is used to measure the similarity of features with the global Joint Bayesian [15], the recognition rate for Rank-1 is 84.80%, and MAP is 75.77%. For Rank-1, the recognition rate increases by 2.5%, and for MAP, the recognition rate increases by 13.5%. Thus, all these factors show improvement. From these results, it is clearly shown that the JBR method not only works well for face recognition but also shows a better recognition rate for pedestrian recognition tasks. Once the re-ranking algorithm is integrated, by combining with the Joint Bayesian method for obtaining the distance between person features, the recognition rates are also improved. For Rank-1, the recognition rate is 85.90%, which is an increase of 1.1%. For MAP, the recognition rate is 77.36%, which is an increase of 1.59%. Based on these test results, the proposed method is evaluated. Let = 0; for Rank-1, the recognition rate is 85.51%; for MAP, the recognition rate is 66.06%. When the re-ranking algorithm is combined, for Rank-1, the recognition rate is 88.09%, which is the best one. Simultaneously, for MAP, the best recognition rate is obtained, that is, 78.24%. From these evaluation results, it can be seen that the proposed method shows the best recognition performance for pedestrian recognition tasks. The recent research by Wang et al. [11] used the combined Joint Bayesian method and the experiment results were shown in Table 1. The accuracy of pedestrian recognition was improved. Clearly, the proposed method obtains better results. The main reason is that the proposed method takes into account the factors from the perspective of distance measurement and the changes between different cameras.
Recognition rates (%) on both the Market-1501 and DukeMTMC-reID datasets
4.2.2 Results between camera pairs
As shown in Fig. 4, the results between all camera pairs are compared between the Siamese network [4] and the proposed method. Although camera_6 is a 720×576 camera and captures distinct background with other HD cameras, the re-identification accuracy result by the proposed method is more competitive. Thus, the proposed method improves the performance of multi-camera pedestrian recognition.
4.2.3 Effect of α on experiments
To clearly test the effect of in formulation (10), additional experiments with different values are given on Market-1501. The results are shown in Fig. 5. It can be seen that changing directly affects the recognition performance of Rank-1 and MAP. It is shown that the smaller , the greater Rank-1 and MAP; moreover, when = 0, Rank-1 and MAP approach the maximum.
Confusion matrix between camera pairs for Market-1501: (a) the Siamese network [ 4] and (b) our proposed method.
Recognition results with different α values for Rank-1 (a) and MAP (b).
4.2.4 Ranklist
The performance is compared in a multi-camera scenario under the Joint Bayesian and our algorithm in Fig. 6. The query consists of a pedestrian’s picture that needs to be tested. Ranklist is the 8-image set with the highest score in the current query (on the left, the score is the highest and gradually reduces to the right); the pink box represents the wrong result. Below each picture is the pedestrian digital camera number. From Fig. 6, it can be seen that the proposed algorithm has better pedestrian recognition under different cameras and can better deal with this multi-scene situation.
Comparison of performance in a multi-camera situation under the JB and proposed method. The query consists of a pedestrian picture that needs to be tested. Ranklist is the 8-image set of the pedestrian with the highest score in the current query, the digital camera number is below each picture, and the pink box represents the wrong result.
Finally, in order to demonstrate the performance of the proposed method, the comparison results with other algorithms on Rank-1 and MAP are given in Table 2. The methods named APR and TriNet are selected from [16,17] , these are all well-applied methods in the state-of-the-art algorithms and obtain higher recognition rates on Market-1501. Based on these results, it can be seen that the proposed method shows the highest recognition rate on Rank-1.
Recognition rates of selected methods on Market-1501
4.3 Experiments on DukeMTMC-reID
The proposed method is evaluated in DukeMTMC-reID, which has 8 cameras. The test results are shown in Table 3. With the proposed method in Section 3, the Joint Bayesian matrix is trained based on the features from different cameras. The Siamese network is selected as the network architecture that has proven to be the better option to extract the features of individuals [4].
Selected best recognition rates on DukeMTMC-reID
To further test the performance of the proposed method, similar to Section 4.2, the other four strategies are built to evaluate the recognition rate in the same dataset. Also, the tests are carried out in MAP and Rank-1, Rank-5, Rank-10, and Rank-20. All the recognition rates are shown in Table 1. Based on the results, the highest recognition rate is obtained with the proposed method, either for MAP or for Rank- 1. For Rank-5, Rank-10, and Rank-20, the method without re-ranking obtains better recognition performance.
Also, more comparison results are obtained by using other state-of-the-art algorithms on Rank-1 and MAP, as shown in Table 3. For the other six methods, the results show that SVDNet obtains the best recognition results. That is, the recognition rate is 56.8% for MAP and 76.7% for Rank-1. However, these recognition rates are lower than those of the proposed method, which is 70.91% for MAP and 80.07% for Rank-1. The results clearly show that the proposed method significantly improves the recognition rates and obtains the best performance for pedestrian recognition tasks.
5. Conclusion
A novel method based on the Joint Bayesian method is proposed and implemented to solve the person re-identification under the situation of the same person viewed through multiple cameras. The evaluation results showed that the proposed method achieved better results. Furthermore, the proposed method can be extended to the multi-scene image retrieval field. In this regard, the relevant experiments will be conducted as part of future studies. The limitation of this study is that the computing cost is higher, especially when retrieving a large number of pedestrian features. In the future, it will be continue to study the existence of end-to-end with the Bayesian method, which can be used to identify different persons in the camera scene and to solve the problem of training the Joint Bayesian matrix separately. Meanwhile, it aims to reduce the complexity of the experiment and to enhance the relevance between global and local issue.
Acknowledgement
This work is supported by the Natural Science Foundation of Liaoning Province (No. 201602557), the Program for Liaoning Excellent Talents in University (No. LR2015034), and Liaoning Province Science and Technology Public Welfare Research Fund Project (No. 2016002006).