Multi-Object Tracking

Cameras, radars, lidars, sonars, and bearing-only sensors all produce a stream of detections frame after frame. Multi-object tracking (MOT) aims to transform these raw detection streams into joint estimates of object counts and their spatial trajectories over time.

Objects enter and leave scenes at unknown times, detectors produce false alarms or miss objects entirely, and detections lack object identity labels. My research addresses multi-object tracking within the labeled random finite set (LRFS) framework (Vo et al., IEEE TSP 2024), treating object collections as set-valued random variables under rigorous Bayesian recursion. We focus on multi-sensor fusion across heterogeneous modalities without relying on tracking-specific training data.

Framework Architecture

01 Estimation Problem

Multi-Object Estimation Under Uncertainty

Cardinality · Kinematic States · Identities

At every time step, sensors report un-ordered detection sets containing real targets, false clutter, and missed detections. The tracker must simultaneously resolve four core sources of uncertainty: time-varying object counts, clutter alarms, missed detections, and data association ambiguity.

Data association is combinatorially complex: as target and measurement counts grow, assigning detections to targets leads to hypothesis expansion. Classical trackers use greedy nearest-neighbour or probabilistic data association, whereas random finite set filters propagate full multi-object densities.

Detections, clutter, missed detections, and the trajectories to be recovered.

02 RFS Formulation

Objects as Labeled Random Finite Sets

Random Finite Sets · Unique Labels · Trajectories

The LRFS formulation models the multi-object state at each time step as a finite set of labeled kinematic states. Each element pairs kinematic variables (position, velocity) with a unique trajectory label that persists over time.

By treating object count as a random variable, target births and deaths are incorporated directly into the Bayesian filtering model, eliminating ad-hoc heuristic track management.

A set-valued state with labels, propagated through births and deaths.

03 Bayesian Filtering

The GLMB Family of Filters

Generalized Labeled Multi-Bernoulli · Bayes Recursion

The Generalized Labeled Multi-Bernoulli (GLMB) filter represents the multi-object density as a weighted mixture of label hypotheses and object spatial distributions. Crucially, the GLMB form is mathematically conjugate under prediction and measurement updates.

To maintain computational tractability against exponential hypothesis growth, truncation methods prune insignificant hypotheses while preserving significant posterior components.

A closed-form recursion, kept tractable by retaining only significant hypotheses.

04 Multi-Sensor Fusion

Measurement-Level Multi-Sensor Fusion

MS-GLMB · Heterogeneous Sensor Networks

Under conditional independence, multi-sensor measurement likelihoods factorize into per-sensor terms. The Multi-Sensor GLMB (MS-GLMB) filter updates target densities using raw measurement sets from all sensors simultaneously.

Fusion occurs at the measurement level inside the Bayesian update rather than combining single-sensor tracks, seamlessly accommodating heterogeneous sensors (cameras, radars, lidars) and non-overlapping fields of view.

Heterogeneous detections enter one filter and leave as labeled trajectories.

05 Optimization

Multi-Dimensional Data Association

Multi-Dimensional Assignment Problem (MDAP) · Gibbs Sampling

Truncating multi-sensor GLMB mixtures requires discovering association maps with maximal weights, framed as a Multi-Dimensional Assignment Problem (MDAP). Each target is assigned measurement indices across all sensors under feasibility constraints.

To bypass NP-hard combinatorial complexity, stochastic Gibbs sampling efficiently draws high-weight association maps directly proportional to posterior component weights.

One cost layer per sensor, one feasible assignment per object across all of them.

References

1. Ba-Ngu Vo, Ba-Tuong Vo, Tran Thien Dat Nguyen, and Changbeom Shim. An Overview of Multi-Object Estimation via Labeled Random Finite Set. IEEE Transactions on Signal Processing, 72:4888-4917, 2024.
2. B.-T. Vo and B.-N. Vo. Labeled random finite sets and multi-object conjugate priors. IEEE Transactions on Signal Processing, 61(13):3460-3475, 2013.
3. B.-N. Vo, B.-T. Vo, and D. Phung. Labeled random finite sets and the Bayes multi-target tracking filter. IEEE Transactions on Signal Processing, 62(24):6554-6567, 2014.
4. B.-N. Vo, B.-T. Vo, and M. Beard. Multi-sensor multi-object tracking with the generalized labeled multi-Bernoulli filter. IEEE Transactions on Signal Processing, 67(23):5952-5967, 2019.