Modern basketball analytics owes much of its depth to computer vision. Optical tracking systems installed in arenas capture the (x, y) coordinates of every player and the ball many times per second throughout a game, quietly generating one of the richest spatial datasets in professional sports. For students who grew up watching that data get turned into shot charts and heat maps without ever thinking about how it was produced, building even a simplified version of that pipeline from scratch was a genuinely eye-opening exercise โ and the subject of this term's computer vision module.
The module was structured around a simple question: what does it actually take to go from raw video frames to a clean, usable trajectory for every player on the court? The answer, as students quickly discovered, involves several distinct subsystems working together โ detection, identity assignment, tracking over time, and noise filtering โ each with its own failure modes and design trade-offs.
Setting the Scene
Because working with real broadcast footage raises licensing and privacy considerations well beyond the scope of a course project, the team built a simulated court environment instead. Synthetic 'players' were represented as moving points with realistic motion constraints โ acceleration limits, minimum turning radii, and boundary rules matching a regulation court โ allowing the class to generate an unlimited supply of controllable, labeled training data without ever needing access to proprietary video.
This turned out to be a pedagogically useful decision in its own right. Because the ground-truth positions were known exactly in the simulation, students could measure their tracking pipeline's error precisely, isolating whether mistakes came from detection, identity assignment, or motion prediction โ a diagnostic clarity that real-world noisy video rarely offers to beginners.
Detection: Finding the Players in Each Frame
The first stage of the pipeline was object detection โ identifying candidate player locations in each individual frame. Students implemented and compared two approaches: a classical background-subtraction method, which flags pixels that differ meaningfully from a static background model, and a lightweight learned detector trained on the synthetic dataset. The background-subtraction method was fast and easy to reason about but broke down whenever the simulated camera introduced motion or lighting changes, while the learned detector generalized better but required considerably more compute and a labeled training set.
Comparing the two approaches side by side gave students a concrete feel for a trade-off that runs throughout computer vision: classical, interpretable methods are easier to debug and cheaper to run, but learned methods tend to win once enough data and compute are available โ and real production systems often end up combining both.
Tracking: Keeping Identities Straight
Detecting a player in a single frame is only half the problem. The harder challenge is linking detections across consecutive frames into a consistent identity โ knowing that the blob at position (34, 12) in frame 100 is the same player who was at (33, 11) in frame 99. Students implemented a Kalman filter to predict each player's next likely position based on their recent velocity, then matched new detections to existing tracks using a distance-based assignment algorithm.
"The moment it clicked for me was watching two simulated players cross paths near the basket and seeing my tracker swap their identities. It's such a simple-sounding bug, but fixing it taught me more about the actual difficulty of real-time tracking than any amount of reading."
โ [Bharat Chaudhary][MCA Student]
Identity switches like the one described above are one of the best-known failure modes in multi-object tracking, and they become especially frequent in basketball, where players cluster tightly near the rim, screen for one another deliberately, and move at high speed during transitions. Students addressed this by incorporating appearance cues โ in the simulation, a simple color signature standing in for jersey and body appearance features used in real systems โ alongside the motion-based predictions, which meaningfully reduced identity-switch rates during crowded sequences.
From Pixels to Insight
Once positional data is extracted, it becomes the foundation for higher-level metrics: distance covered per player, average and peak speed, spacing between teammates, and closeout speed on defense. Students paired their tracking output with analytics scripts to compute these derived statistics, connecting the perception layer of computer vision directly to the decision-making layer of sports analytics that was covered in a companion module.
This connection mattered pedagogically. It is one thing to be told that tracking data 'powers advanced analytics'; it is another to compute a spacing metric yourself from trajectories you extracted, and to watch that number shift sensibly when you deliberately alter simulated player behavior. Several students noted that this was the first project where they felt the full path from raw sensor-like data to an interpretable statistic end to end.
Performance and Real-Time Constraints
A recurring engineering theme was the tension between accuracy and speed. Arena tracking systems operate under real-time constraints, processing dozens of frames per second across an entire court. Students profiled their pipelines and discovered that the identity-matching step, not detection, was the primary bottleneck once player counts grew โ a reminder that in systems work, intuition about where the slowdown 'must be' is often wrong until you actually measure it.
"Every year, students assume the deep learning model will be the slow part. Almost every year, the real bottleneck turns out to be some unglamorous piece of bookkeeping code. That lesson alone is worth the whole exercise."
โ [Dr. Manish Shrivastava, Professor], Department of Computer Science & Applications
Handling Noise and Smoothing Trajectories
Raw tracking output, even from a well-tuned pipeline, is noisy โ small frame-to-frame jitter that would make derived metrics like speed and acceleration unusable if left untreated. Students implemented a smoothing stage using a moving average filter, then compared it against a more sophisticated spline-based approach for reconstructing smooth trajectories from noisy point estimates. The spline approach produced visibly cleaner paths but introduced a subtle risk: over-smoothing could erase genuinely sharp, real changes in direction, such as a hard cut to lose a defender, which are often exactly the moments an analyst cares about most.
Balancing noise reduction against the preservation of real signal turned into a recurring design conversation, and the team ultimately settled on a lightly tuned filter that favored preserving sharp direction changes over producing the smoothest-looking output โ a decision justified by testing how well each approach preserved known ground-truth cuts and stops from the simulation.
Validating Against Ground Truth
Because the simulated environment provided exact ground-truth positions, students could quantify tracking accuracy directly, using average positional error in simulated feet and an identity-switch rate measuring how often the tracker incorrectly swapped two players' identities during a sequence. This gave the project something many student computer vision exercises lack: a clear, objective, and improvable success metric, rather than a purely qualitative sense of whether the output 'looked right.'
Over the course of the semester, the team's identity-switch rate during high-traffic sequences near the basket dropped substantially as appearance cues and improved motion prediction were layered in โ a concrete, measurable improvement that made the abstract engineering work feel tangible and motivating.
Presenting these metrics also gave students practice in an underrated skill: choosing the right way to report an improvement. An error rate that dropped from a middling starting point to a much lower one sounds impressive as a percentage reduction, but the team learned to also report absolute error alongside the relative gain, since a large percentage improvement on an already-small error can be a misleading headline number โ a small but genuine lesson in statistical honesty that applies well beyond computer vision.
What Students Take Away
By the end of the module, most students could speak fluently about a distinction that is easy to state but hard to internalize without building something yourself: detection answers 'where is something right now,' while tracking answers the much harder question of 'is this the same something I saw a moment ago.' That second question, deceptively simple in phrasing, is where the majority of real engineering effort in any tracking system ends up living โ a takeaway several students specifically mentioned carrying into internship interviews later in the year.
Looking Ahead
Next term's extension will introduce pose estimation, moving beyond a single point per player toward tracking joint positions โ enabling analysis of shooting form, defensive stance, and contact detection, topics that connect directly to the department's parallel work on AI-assisted officiating. The team is also exploring camera calibration techniques that would allow the pipeline to eventually process short, fair-use video clips with appropriate rights clearance, bringing the simulated exercise a step closer to a real deployment.
For a department built around teaching transferable systems skills, few exercises make the abstract concrete quite like watching your own code correctly follow five moving points around a simulated court โ and understanding, from first principles, everything that had to go right for it to work.
About the authors: [Dr. Manish Shrivastava, Professor] leads the department's computer vision and robotics coursework. [Bharat Chaudhary] is a student who implemented the tracking and identity-matching components described in this project.
