Modern VPR approaches extract compact image embeddings using a feature extractor backbone followed by a head that implements aggregation or pooling (Kim et al., 2017; Arandjelović et al., 2018; Ge et al., 2020; Ali-bey et al., 2023; Berton et al., 2023; Zhu et al., 2023). These usually employ contrastive learning, using the geotags of the training set as a type of weak supervision to mine negative examples. However, this mining operation is expensive and impractical for scaling to large datasets (Berton et al., 2022b). To mitigate this problem, Ali-bey et al. (2022) proposed the use of a curated training-only dataset in which the images are already split into predefined classes that are far apart from each other, thereby enabling the composition of training batches with images from the same place (positive examples) and from other places (negative examples) very efficiently. The method proposed by Leyva-Vallina et al. (2023) involves annotating the images with a graded similarity, thus enabling training with contrastive losses and full supervision while achieving improvements in terms of both data efficiency and final model quality. Instead of mitigating the cost of mining, Berton et al. (2022a) proposed an approach to remove it entirely through their CosPlace method. The idea of CosPlace is to first partition the training images into disjoint groups with one-hot labels and to then train sequentially on these groups with the CosFace loss (Wang et al., 2018) that was originally designed for large-scale face recognition. Although CosPlace achieves SOTA results on large-scale datasets and even in generalized scenarios, we show here that its sequential training procedure is suboptimal and hampers the convergence speed. In view of these findings, we introduce a parallel-training version of CosPlace that improves the convergence speed and produces new SOTA results on several benchmarks.

The growth of deep-learning methods and training datasets is driving research on distributed training solutions. Among these, data parallelism constitutes a popular family of methods (Lin et al., 2020) wherein different chunks of data are processed in parallel before combining the model updates either synchronously or asynchronously. In particular, to reduce the communication overhead of data movement between the accelerators, local optimization methods are commonly used to allow multiple optimization steps on disjoint sets of data before merging the updates (Stich, 2019; Yu et al., 2019; Wang et al., 2020). In this work, we redefine CosPlace’s training procedure by introducing the parallel training of groups and leveraging local methods to speed up convergence.

The first step in CosPlace’s training protocol involves creating a set of discrete labels from the continuous space of the Universal Transverse Mercator (UTM) coordinates of the area of interest (Berton et al., 2022a). Formally, we define the training distribution D ≔ X × C , where X is the space of possible images and C is the space of UTM coordinates (east, north, heading). We also define a new distribution D ̂ ≔ X × Y , where Y is the label space induced by partitioning C . Formally, a UTM point c ∈ C is discretized to a label y = ⌊ e a s t M ⌋ , ⌊ n o r t h M ⌋ , ⌊ h e a d i n g α ⌋ , where M and α describe the extent of a region covered by any class in meters and degrees, respectively. The set of such classes is then split into groups called CosGroups by fixing the minimum spatial separation between two classes of the same group in terms of both translation and orientation. Formally, a CosPlace group is defined as the set of classes such that G u , v , w ≔ y ∈ Y : e a s t M mod N = u , n o r t h M mod N = v , h e a d i n g α mod L = w , (1) where N and L are hyperparameters for the fixed minimum spatial and angular separations between classes belonging to the same CosGroup. We denote the set of such groups as G , i.e., G = { G u , v , w } ∀ u , v , w ∈ N . Given multiple CosGroups (defined by Eq. 1), it is possible to derive multiple training distributions D i ̂ ≔ X × G i ⊂ D ̂ , where each distribution maps the sample image to a one-hot label within the ith CosGroup. The CosGroups partition is reflected in the model and is composed of two components: a feature extractor B ( ⋅ ) : X → R D parameterized by weights θ ^b and multiple classifiers F i ( ⋅ ) : R D → [ 0,1 ] | G i | that are each associated with a different CosGroup parameterized by the weights θ i f .

The goal of CosPlace is to learn a feature extractor B(⋅) that maps the original distribution X in an embedding space such that the distances between the locations depicted in the images are reflected well. Therefore, CosPlace aims to optimize the following problem: θ b * = arg min θ b ∑ i = 1 | G | E x , y ∼ D i ̂ L lmcl F i ◦ B , x , y . (2)

In practice, the training procedure should minimize the large margin cosine loss (LMCL) (Wang et al., 2018) of the entire model θ ≔ { θ b , ∪ i θ i f } with respect to the label distribution(s) induced by discretization of the GPS coordinates into classes and by the grouping of these classes. The parameters θ i f of the classifiers are used only to train the feature extractor θ ^b and discarded after training. The final performances of B(⋅) are assessed using the kNN algorithm as the proxy with respect to the original distribution D .

Although CosPlace aims to optimize Eq. 2, it is observed that the sequential optimization of θ ^b with respect to each CosGroup is just an approximation of this objective function. Formally, it implements θ G i b * = arg min θ b E x , y ∼ D i ̂ L lmcl F i ◦ B , x , y , θ G i − 1 b * ∀ i ∈ 1 , | G ̄ | θ G 0 b * = θ 0 b i n i t i a l m o d e l , (3) where G ̄ ⊆ G is a subset of all possible CosGroups selected a priori for training. Eq. 3 practically means that at each iteration e, the training procedure selects the ith CosGroup G i , with i ≔ ( e mod | G ̄ | ) , and jointly optimizes the parameters θ ^b and θ i f for s optimization steps starting from the optimal model obtained from the previous CosGroup G i − 1 .

By expressing the CosPlace learning problem in this form, we can revisit it from a continual learning perspective. Accordingly, each distribution associated with a CosGroup can be considered as a task with a disjoint set of labels and dedicated parameters θ i f . Therefore, when CosPlace training iterates to a new CosGroup, it is akin to switching to a new task (Figure 2, left). This is different from solving the original problem in Eq. 2 because there is no guarantee that switching to the new task will not harm the model performances for the older tasks. In practice, the new model updates could be detrimental to the previous tasks, a phenomenon known as catastrophic forgetting (Goodfellow et al., 2014; Pfülb and Gepperth, 2019; Ramasesh et al., 2021). To verify if this phenomenon actually manifests during CosPlace training, we performed an experiment using its original implementation on the SF-XL dataset provided by Berton et al. (2022a). We plot the training loss for this experiment in Figure 3, from which it can be clearly seen that at each iteration, when switching to a new CosGroup, the loss function exhibits a steep increase and requires many steps to recover a loss value similar to the one before group change. This behavior is especially notable in the first few iterations, after which it disappears gradually as it is expected for the model to achieve convergence.

The reason why optimizing Eq. 3 still works remarkably well is that the CosPlace training protocol relies on the fact that each task will be revisited after some iterations. Therefore, the algorithm eventually converges to a solution that is also good for the joint objective function of Eq. 2. However, this is achieved at the cost of increased training time and is hardly scalable with respect to the number of trained groups | G ̄ | , as observed in the original work (Berton et al., 2022a). Together, these problems drastically limit the training time scalability of CosPlace, which is its main purpose.

Given that the most severe jumps in the training loss in Figure 3 occur in the first few iterations, i.e., when the classifiers θ i f associated with each task have not yet been trained, one can consider some engineering solutions to solve this problem. A first modification would be to freeze the backbone model θ ^b for a number of steps s _freeze ≪ s whenever the task is changed. This prevents the weights θ i f from being uninitialized or too stale with respect to the backbone. Additionally, considering the amount of training that the model θ ^b has undergone since the last time task i was selected, it would also be beneficial to reset the optimizer state for model θ i f as it may be excessively biased. However, repeating the same experiment as before with these modifications shows that the effectiveness is limited (Figure 3, orange line). In particular, we observe that resetting the optimizer step is only beneficial during the first few iterations, which slightly speeds up convergence. However, we find that this strategy worsens the final model quality in the long run because maintaining the optimizer states is beneficial as the model finally approaches convergence. A similar observation also holds for freezing θ ^b ; it is initially useful, although a very large number of s _freeze steps are needed for a noticeable reduction in the training loss. In the long run, this becomes detrimental because these steps are wasted.

In conclusion, despite their simplicity, such simple mitigation strategies require careful engineering to determine s _freeze as well as decide when to use them, making them practically ineffective. Moreover, since these issues arise after performing a significant amount of training between two samplings of the same task i, these simple strategies cannot be scaled when the number of training CosGroups | G ̄ | increases.

In this section, we compare the results obtained by D-CosPlace with those from the original CosPlace algorithm in terms of both convergence speed (cf. Table 2) and final model quality given the time budget (cf. Table 1).

Local steps ensure that the distributed training is more efficient from a communication perspective by lowering the synchronization frequency. However, even when a large number of local steps is desirable, too many steps could slow the convergence when the training distributions are different, like in our case. For this reason, J is treated as a hyperparameter. Table 2 shows the impact of the local steps on the convergence speed and final model quality, where the former is expressed in terms of wall-clock time to reach the accuracy of the vanilla CosPlace and the latter is expressed as R@1/R@5. It can be seen that J = 10 produces the optimal balance between training time, convergence speed, and final model quality.

Our analysis in Section 3.3 reveals that CosPlace’s training procedure experiences severe jumps in the loss function due to the optimization procedure not implementing the objective function in Eq. 2 correctly. Indeed, the sharp jumps in loss occur only in the vanilla CosPlace because of the training process that optimizes different CosGroups (and their related classification heads) one at a time. This does not occur in D-CosPlace since all classifiers associated with the CosGroup are jointly optimized (Figure 3). A second challenge that we experienced with CosPlace is the noisy optimization of a single CosGroup, as shown by the loss in Figure 4. It is noted that the training loss is particularly unstable and remains high for many steps before dropping abruptly, with a seemingly periodic cycle every ≈ 1 k steps. We initially associated this behavior with the batch size, especially if it is a low value when compared to the output dimensionality of the final layer. Each CosGroup is in fact associated with ≈ 35 k classes on average, which makes the problem hard to learn. Additionally, the LMCL loss seeks a hard margin boundary, which can be difficult to achieve in high-dimensional problems. To validate this hypothesis, we increased the batch size to fill the memory of an NVIDIA-V100-32 GB GPU. The results in Figure 4 show that the problem persists even after increasing to 1,024 samples. Considering the validation results, the initial value of 32 still gives the best validation performance, substantiating the conclusion that increasing the batch size is not a practical solution. This difficulty of learning a single CosGroup is still present in D-CosPlace since the optimization with respect to a CosGroup is the same as that for CosPlace. We believe this to be an intrinsic limitation of the classification approach of CosPlace that will be an interesting direction for future works.