From Pixels to Poses

The problem

Professional football analysis is moving toward objective, high-throughput tactical and biomechanical insight, but the manual tracking and annotation workflows underneath much of the industry are slow, expensive and subjective. Broadcast footage is the most abundant and historically rich source of football data in existence — and extracting reliable 3D skeletal information from it is still an unsolved engineering problem.

The difficulty is that broadcast football violates nearly every assumption the standard 3D pose benchmarks are built on. Human3.6M, which has anchored the field for over a decade, consists of slow, upright, single-subject motion filmed close-up in a controlled laboratory. Broadcast football gives you the opposite on every axis:

• Scale. Players span roughly 50–200 pixels in height. At 50 pixels, fine-grained joint localisation is unreliable.
• Occlusion. Defenders, attackers and referees overlap constantly, and whole body segments disappear for consecutive frames.
• Perspective distortion. Wide-angle cameras foreshorten players near frame edges in ways barely represented in training data.
• Motion distribution. Sprints, dives, tackles and explosive direction changes appear in no laboratory-captured dataset.

So the thesis asks one concrete question: can a supervised spatio-temporal refinement stage, trained on game-engine ground truth, meaningfully correct the systematic errors produced by monocular 3D pose lifting?

The pipeline

Rather than build an end-to-end model, the system decomposes the standard two-stage lifting architecture into five explicitly separated modules. Decoupling matters practically: each stage can be pre-trained, swapped and debugged independently, which is a necessity in a domain where no single model is trained for all sub-tasks.

Stage 1 — Segmentation, tracking and re-identification

The first implementation used a YOLO detector, and evaluation exposed a disqualifying failure mode: the goalkeeper was dropped entirely by the tracker despite remaining clearly visible, because an atypical player posture induced a confidence collapse. Track loss at exactly the moment it is hardest to recover propagates corrupted sequences into everything downstream.

The fix was architectural — split detection into two explicit responsibilities. SAM 3 produces pixel-level masks from a concept prompt ("soccer player"), which is far less sensitive to the appearance variation that collapses YOLO's confidence. Masks become expanded axis-aligned crops fed to the keypoint estimator.

Identity is then maintained by ByteTrack with OSNet-x0.25 appearance embeddings — deliberately a small re-ID model, because at 50–100 pixels of player height the fine-grained appearance detail a heavier model relies on simply isn't present. Association weights appearance over geometry, since trajectories cross constantly in football. A post-hoc fragment-linking pass merges track fragments left over after occlusion.

Stage 2 — Pre-lifting refinement

Errors introduced at the 2D/3D interface are unrecoverable downstream, so the 2D sequences are cleaned before they reach the lifting model: 1-Euro filtering for causal temporal smoothing, cubic spline interpolation for low-confidence joints across gaps of up to 10 frames, and bone-length validation that flags any bone deviating by more than a factor of 2.5 from its track median.

Stage 3 — 3D lifting with MotionBERT

MotionBERT was chosen for an often-overlooked reason: its input interface. Many state-of-the-art lifters are tightly coupled to their own detection pipeline, or need camera intrinsics, pixel features or depth estimates. MotionBERT accepts raw 2D coordinate sequences and nothing else — which makes it entirely agnostic to the visual quality of low-resolution broadcast crops. It attends over 243-frame windows, resolving depth ambiguity from temporal context rather than single-frame appearance.

Long tracks — up to 148,280 frames — are processed by sliding 243-frame windows with 50% overlap, merged with a Gaussian-weighted blend so that predictions near a window's centre dominate and no visible discontinuities appear at window boundaries.

Stage 4 — Post-lifting refinement

Four deterministic passes clean the raw 3D output as far as fixed rules can: zero-phase Butterworth smoothing, velocity and acceleration outlier rejection with Hermite spline replacement, bone-length enforcement with a bilateral symmetry pass, and joint angle limit enforcement using the Rodrigues rotation formula to bring violating joints back into anatomical range by the smallest rotation that restores plausibility.

Stage 5 — The Spatio-Temporal Pose Refinement Transformer

This is the contribution. The deterministic passes operate on individual frames or joints in isolation and cannot reason about the relationship between joint positions across a player's full movement. A heuristic filter cannot distinguish a genuine rapid direction change from a spurious single-frame outlier; a bone-length pass cannot correct a systematic depth bias accumulating over dozens of frames. Those failures need learned temporal reasoning.

The model is a residual corrector: it predicts a delta added to the noisy input rather than regressing absolute positions from scratch. The output projection is initialised near zero, so training starts from the identity function and progressively moves toward meaningful corrections — a model that has learned nothing reproduces its input, which is a genuinely useful safety property.

Each forward pass takes three input streams over a 243-frame window:

• The noisy root-relative 3D pose sequence.
• A binary validity mask, where occluded or missing joints are replaced with a learnable [MASK] token — without it, a missing joint at the root-centred origin is indistinguishable from a silent zero.
• A root context vector carrying field position in metres and frame-to-frame velocity, so the model can correct errors that depend on where a player is on the pitch.

The encoder is 8 stacked blocks, each alternating spatial attention across the 17 joints within a frame and temporal attention across all 243 frames for each joint. Separating the two is deliberate — enforcing anatomical consistency and smoothing a trajectory over time are complementary but distinct tasks, and splitting them is both cheaper than full 3D attention and more interpretable.

Spatial attention carries a learnable skeleton adjacency bias: anatomically adjacent joints get a positive bias in the attention logits at initialisation, injecting skeletal structure as a soft prior. Unlike a hard-coded connectivity mask, the model can override it when the data supports a different pattern.

Training is supervised by a composite loss — masked pose MSE as the primary term, plus velocity smoothness, bone-length integrity and bilateral symmetry as weighted anatomical constraints.

Results

Evaluated on 800 held-out validation windows, the Transformer reduces MPJPE from the pipeline's 184.0 mm to 80.4 mm. The share of frames falling below 100 mm rises from 6.7% to 85.4%.

Body orientation

Beyond joint positions, mean facing-direction error drops from 52.3° to 21.5°, a 58.9% reduction; frames within 45° of ground truth rise from 51% to 90%. This matters for football analytics specifically — whether a player is pressing, retreating or tracking a run is an orientation question, and it is not captured by MPJPE at all. The model recovers it from temporal context alone, with no camera geometry.

Per-joint behaviour

A clear proximal-to-distal error gradient survives in both outputs, consistent with the occlusion patterns of football footage: hips improve from 66 mm to 30 mm, knees from 168 mm to 79 mm, ankles from 293 mm to 125 mm. Upper body shows similarly large gains — shoulders 231 mm to 81 mm, wrists 252 mm to 109 mm.

Data generation

Supervised refinement needs paired noisy and clean 3D sequences, which live broadcast video cannot provide — there is no skeletal ground truth. Ground truth therefore comes from a heavily modified Unity-based football simulation engine that renders matches with full per-player skeletal tracking and a calibrated camera.

• A full 90-minute simulated match, 148,280 frames, roughly 171 GB of raw PNG frames compressed to 10–20 GB for cluster processing.
• The simulator was modified to diversify jersey colour, skin tone and kit type, since OSNet re-identification needs visual discriminability between players.
• The upstream pipeline ran on the Snellius HPC cluster — 64 GB host RAM, 32 wall-hours.
• Ground-truth alignment in three steps: least-squares similarity transform plus Hungarian assignment to match tracks to simulated players, joint remapping to Human3.6M-17, and a global Procrustes alignment via SVD.
• Track merging via graph colouring — the tracker produced 249 IDs for six real players, and the constraint that two tracks alive in the same frame cannot be the same player is encoded as a conflict graph and coloured greedily.

The final dataset is roughly 3,200 training and 800 validation windows, with five augmentation operations. Training ran on a single NVIDIA A100 40 GB on Snellius, AdamW, bfloat16 mixed precision, early stopping at epoch 66 with the best checkpoint at epoch 41.

Limitations

The thesis is explicit about what these numbers do and do not establish, and it is worth repeating here rather than burying:

• Evaluation is in simulation. Training and validation windows are drawn from a single simulated match. Because football motion is strongly temporally correlated and windows overlap, validation windows adjacent to a block boundary share players and phases of play with training data. The reported error should be read as an upper bound on accuracy, not as demonstrated generalisation to unseen footage.
• Single-match training data. One match does not contain enough explosive athletic movement for the model to learn to amplify rather than smooth it — refined output shows mean acceleration of 1952 mm/s² against ground truth's 3823 mm/s², i.e. real over-smoothing of fast dynamics.
• Ground-truth fidelity gap. Engine skeletal data approximates human biomechanics; it is not motion capture of real players.
• 80 mm is not enough for everything. Fine-grained biomechanical analysis needs roughly 40 mm. The system is well suited to coarse tactical analysis — positioning, formation shape, pressing triggers, body orientation — and falls short of joint-angle-level injury assessment.

Why the result matters

The most informative comparison is against sports-domain work. Yeung et al. show MotionAGFormer trained on Human3.6M alone reaching 257 mm on athletic motion, which sport-specific fine-tuning on their AthletePose3D dataset reduces to roughly 98 mm. This pipeline reaches 80.4 mm without any sports-specific retraining of the lifting model.

That a post-hoc refinement stage reaches a similar level of accuracy to domain-specific fine-tuning suggests supervised correction can compensate for the laboratory-to-deployment mismatch without retraining the lifter itself. Practically, MotionBERT can be replaced by any future lifting model and the refinement stage retrained independently — which makes the approach well suited to a field that moves as fast as this one.

Publication