Multi-Camera Multi-Object Tracking Based on Epipolar Distance and Appearance Similarity

Abstract

Multi-camera multi-object tracking (MC-MOT) is key to accurately tracking people and objects in open and wide areas for video surveillance and traffic monitoring. Accurate MC-MOT requires robust association among the detected bounding boxes of different frames. We propose a multi-camera tracking framework that links each tracklet obtained from single-camera tracking across overlapping fields of view by jointly exploiting geometry and appearance cues. For every cross-view pair we evaluate two complementary similarity scores — an epipolar distance derived from the epipolar geometry between camera pairs, and an appearance-based similarity obtained from image features of each target patch. These similarities drive an efficient one-to-one association scheme that preserves identities even under severe occlusion in a single view, without requiring specialized hardware or heavy-model retraining. Extensive experiments on the MMPTrack and CAMPUS benchmarks show that the proposed framework reduces ID switches and improves overall tracking accuracy compared with both single-camera baselines and recent multi-camera trackers. Code for reproduction is publicly available.

1. Motivation

Single-camera tracking (SC-MOT, e.g., ByteTrack) is simple to deploy but is fragile to occlusion: when a target is briefly hidden, the tracker often re-assigns a new identity once the target reappears. Multi-camera tracking (MC-MOT) with overlapping fields of view can exploit complementary viewpoints to stay robust to occlusion, but recent learning-based MC-MOT methods are data-hungry and typically assume that all cameras are available at both training and inference time — a costly requirement in practice.

Single-camera tracking suffers an ID switch under occlusion — **Fig. 1(a).** Single-camera tracking: occlusion in frame 2 causes an ID switch in frame 3 (ID:2 → ID:3).

Multi-camera tracking preserves identity across cameras — **Fig. 1(b).** Multi-camera tracking: a complementary viewpoint (Camera B) preserves the identity of Camera A despite the occlusion.

Goal. Design a lightweight, training-free cross-camera association method that combines geometric and appearance cues on top of any existing single-camera tracker, and that remains effective with as few as two cameras.

2. Proposed Method

Single-camera tracking is applied independently to each camera stream, producing a pool of per-camera tracklets. An iterative, offline tracklet-matching module then associates tracklets across cameras by fused similarity, assigning the same global identity to matched tracklets. The framework requires no additional training and extends naturally from two to any number of cameras.

Overview of the proposed multi-camera tracklet matching framework — **Fig. 2.** Overview of the proposed framework. Per-camera tracklet pools are iteratively matched: at each step the longest unmatched tracklet is selected as reference, compared against temporally overlapping candidates from other cameras using epipolar distance and appearance similarity, fused, and merged with the best match.

2.1 Iterative Tracklet Matching

Step 1 — Longest tracklet selection. All tracklets are sorted by temporal length; the longest unmatched tracklet (regardless of source camera) is chosen as the reference.
Step 2 — Similarity evaluation. For every temporally-overlapping candidate tracklet from other cameras, the epipolar distance and appearance similarity are computed per co-visible frame and fused.
Step 3 — Matching. The candidate with the best averaged fused similarity is merged into the reference tracklet's identity; any other time-overlapping candidates are discarded to avoid conflicting assignments.
Step 4 — Iteration. Steps 1–3 repeat until no valid candidates remain. Each tracklet is examined a bounded number of times, so overall complexity scales linearly with the number of tracklets.

2.2 Epipolar Distance (Geometric Cue)

For an uncalibrated camera pair with fundamental matrix F, corresponding points must satisfy x̃_B^T F x̃_A = 0. The epipolar distance measures how far a candidate detection in camera B lies from the epipolar line induced by a detection in camera A:

d(A_i, B_j) = |x̃_{B_j}^T F x̃_{A_i}|√(a² + b²), [a, b, c]^T = F x̃_{A_i}

A smaller d(A_i, B_j) indicates a higher geometric likelihood that A_i and B_j are the same person.

Epipolar distance between detections in two camera views — **Fig. 3.** The epipolar line induced by detection A1 constrains where its correspondence can lie in camera B; the perpendicular distance from each candidate detection to this line is the epipolar distance.

2.3 Appearance-Based Similarity (Visual Cue)

A Re-ID feature vector ξ = g(p) is extracted from each detected image patch p using a ResNet-101 encoder pre-trained on Market-1501. The appearance similarity is the (clipped) cosine distance between feature vectors:

θ(A_i, B_j) = min ( cos⁻¹ ξ_{A_i}^T ξ_{B_j}‖ξ_{A_i}‖₂ ‖ξ_{B_j}‖₂, τ_θ )

Clipping at τ_θ limits the influence of outlier appearance pairs.

Appearance-based cosine similarity computation — **Fig. 4.** Appearance-based similarity: features are extracted per detection and compared with cosine distance.

2.4 Fusion of Geometry and Appearance

A small epipolar distance is necessary but not sufficient for a correct match — multiple people can lie near the same epipolar line. We therefore use the epipolar distance to gate the appearance similarity: only pairs that are geometrically consistent are allowed to compete on appearance.

θ_d(A_i, B_j) = θ(A_i, B_j), d(A_i, B_j) < τ_d
τ_θ, d(A_i, B_j) ≥ τ_d

Fusion of epipolar distance and appearance similarity — **Fig. 5.** Per-pair similarity computation: epipolar distance and appearance similarity are computed for each candidate pair and combined by fusion.

3. Experimental Setup

Datasets: MMPTrack (4–5 synchronized cameras, 5 indoor environments: Café, Industry, Lobby, Office, Retail) and CAMPUS (4 calibrated cameras, 3 outdoor environments: Garden1, Garden2, Parking Lot).
Base tracker: YOLO-X detector + ByteTrack association per camera; any single-camera tracker can be substituted.
Appearance encoder: ResNet-101 pre-trained on Market-1501.
Metrics: IDF1 (identity preservation, primary metric), MOTA, and HOTA.
Thresholds (tuned on held-out data): τ_d = 20, τ_θ = π/6.
Baselines: ByteTrack (single-camera), MCTR and DMCT (MC-MOT, MMPTrack), HCT and ReST (MC-MOT, CAMPUS).

Sample frames from the MMPTrack dataset — **Fig. 6.** MMPTrack — densely-annotated indoor multi-camera dataset.

Sample frames from the CAMPUS dataset — **Fig. 7.** CAMPUS — outdoor multi-camera dataset with largely overlapping views.

4. Results

4.1 Ablation: Similarity Cues (MMPTrack, IDF1)

Fusing epipolar distance and appearance outperforms either cue alone, averaged over the five MMPTrack environments.

Cue	Epipolar only	Appearance only	Fusion (ours)
AVG (5 environments)	65.00	64.93	65.48

4.2 Comparison on MMPTrack (IDF1 ↑)

Environment	ByteTrack (1 cam)	MCTR (all cams)	DMCT (all cams)	Ours (2 cams)	Ours (all cams)
Café	57.93	49.71	64.20	63.53	64.78
Industry	72.65	80.21	61.70	76.92	79.08
Lobby	73.51	68.07	69.40	78.41	80.35
Office	54.31	55.01	68.00	60.34	65.60
Retail	35.06	58.48	45.70	36.40	37.60
AVG	58.69	62.30	61.80	63.12	65.48

With all cameras, our method improves IDF1 by +3.18 over MCTR and +3.68 over DMCT. Even with only two cameras, it still surpasses both baselines that use every available view. On HOTA (AVG), our method reaches 56.92 versus 54.00 (ByteTrack) and 55.77 (MCTR).

4.3 Comparison on CAMPUS

IDF1 ↑

Environment	ByteTrack	HCT	ReST	Ours
Garden1	49.86	–	12.70	51.96
Garden2	48.79	–	15.90	49.31
Parking Lot	35.51	–	9.00	40.83
AVG	44.72	–	12.53	47.37

MOTA ↑

Environment	ByteTrack	HCT	ReST	Ours
Garden1	75.30	49.30	58.70	75.30
Garden2	62.60	25.80	49.20	62.60
Parking Lot	47.12	24.10	35.00	47.16
AVG	61.67	33.10	47.60	61.69

Our method achieves the best IDF1 and MOTA on every CAMPUS sequence, consistently outperforming both the single-camera baseline and prior multi-camera methods.

4.4 Qualitative Comparison (MMPTrack)

Single-camera tracking ID switch example — **Fig. 8(a).** Single-camera tracking (ByteTrack): a detection loss at frame t2 causes an ID switch — the target reappears at frame t_n with a new ID.

Proposed method maintains identity example — **Fig. 8(b).** Proposed method: the same identity is correctly maintained across the occlusion, using the complementary camera view.

5. Conclusion

We proposed a simple-yet-effective MC-MOT framework that fuses epipolar geometry and appearance similarity for cross-camera tracklet association — requiring no association training.
The framework substantially reduces ID switches and outperforms both single-camera and recent multi-camera baselines on the MMPTrack and CAMPUS benchmarks, remaining effective with as few as two cameras.
It is plug-and-play with any single-camera tracker (e.g., ByteTrack), so advances in single-camera MOT can be leveraged directly.
We additionally performed an extensive sanitization of the public MMPTrack annotations, correcting ID inconsistencies and removing spurious labels, to ensure a reliable evaluation protocol.
Future work: improving robustness under severe single-view occlusion (e.g., the Retail environment), where missed detections currently limit performance before association even begins.

Publication

Multi-Camera Multi-Object Tracking Based on Epipolar Distance and Appearance Similarity

Masamune Oka, Masayuki Tanaka, Takashi Shibata, Masatoshi Okutomi

Submitted to the 28th International Conference on Pattern Recognition (ICPR 2026), Lyon, France.

[Reproduction Code]

BibTeX

@inproceedings{oka2026mcmot,
  title     = {Multi-Camera Multi-Object Tracking Based on Epipolar Distance and Appearance Similarity},
  author    = {Oka, Masamune and Tanaka, Masayuki and Shibata, Takashi and Okutomi, Masatoshi},
  booktitle = {Proceedings of the International Conference on Pattern Recognition (ICPR)},
  year      = {2026},
  note      = {submitted}
}