AI-Assisted Officiating: Instant Replay and Anomaly Detection

Officiating in fast-paced sports is one of the hardest real-time perception problems there is. A referee has a fraction of a second, an obstructed sightline, and a stadium full of noise to make a call that a slow-motion, multi-angle replay might later reveal to be wrong — not because the referee wasn't skilled, but because the human visual system simply was not built to resolve certain events at game speed. This tension is exactly what motivated our student research exercise into whether computer vision and anomaly-detection techniques could, in principle, support rather than replace human referees during instant replay review.

It is worth stating the framing up front, because it shaped every design decision in the project: the goal was augmentation, not automation. The system was never intended to make a final call. Its only job was to flag moments worth a closer look — surfacing candidate plays faster than a human reviewing raw footage manually, while leaving every actual judgment to a trained official. This distinction turns out to matter enormously, both technically and ethically, and it became the throughline of the entire semester.

Why This Problem Is Hard

Before building anything, the team spent time cataloguing exactly why officiating is such a difficult perception problem. Plays happen at high speed, often with multiple players occluding one another. The line between incidental and deliberate contact is frequently a matter of degrees, not a binary. And camera angles that look decisive in one broadcast feed can be genuinely ambiguous from another — which is precisely why professional replay review already uses multiple synchronized camera angles rather than a single view.

This framing pushed the project's technical design toward a multi-signal approach from the start. Rather than trying to build a single model that outputs a verdict, the team built a pipeline that extracts several independent signals — player pose, ball trajectory, and contact proximity — and combines them into a confidence score for whether a moment deserves human review, closer in spirit to an anomaly detector than a classifier making a factual ruling.

Building the Pipeline

Using simulated multi-angle video frames (again avoiding the licensing complications of real broadcast footage, following the same approach as the department's player-tracking project), students implemented a three-stage pipeline. The first stage ran frame-by-frame pose estimation, extracting approximate joint positions for each simulated player. The second stage tracked the ball's trajectory, watching for sudden, physically implausible changes in velocity that might indicate contact or a deflection. The third stage combined these signals with proximity information to compute a single anomaly score for each moment in the sequence.

Figure 1: Conceptual flow of an AI-assisted replay review system, from multi-angle frames to a flagged review moment.

Moments scoring above a tunable threshold were flagged for review, along with the specific signal that triggered the flag — a design choice that mattered enormously for the transparency discussion described below. A flag that simply says 'anomaly detected' is far less useful, and far less trustworthy, than one that says 'unexpected change in ball trajectory coincided with two players in close proximity near the boundary.'

"Early on, our flagged-moments list was almost unusable — dozens of false alarms for perfectly normal, physical basketball contact. Tuning the threshold taught me that in a system like this, precision matters more than recall. A referee who stops trusting the flags because of too many false positives gets no value from the system at all."
  — [Sunita Maurya (MCA Student)]

The Precision Problem

That false-positive problem became the central engineering challenge of the semester. Ordinary basketball involves constant physical contact — screens, boxing out, contested rebounds — that is completely legal and not remotely noteworthy. An anomaly detector tuned too sensitively flags all of it, which is worse than useless in a review context where a human's attention is the scarcest resource in the system. The team iterated extensively on the scoring function, eventually weighting trajectory anomalies more heavily than raw proximity, since unexpected changes in ball motion turned out to be a far more selective signal than mere physical closeness between players.

This iteration process mirrors a broader lesson in applied anomaly detection: the interesting engineering work rarely lives in the initial model architecture. It lives in the evaluation loop, threshold tuning, and the uncomfortable process of confronting how many of your 'detections' are actually just noise dressed up as a signal.

Designing for Trust and Transparency

A significant part of the project was devoted to a question that has no purely technical answer: how should a system explain why it flagged a play? The team prototyped a simple explanation interface that, alongside each flagged moment, displayed which signal triggered the flag and a confidence score, rather than presenting the flag as an unexplained black-box output. Classroom discussion repeatedly returned to the idea that a system referees might actually adopt needs to earn trust incrementally, and unexplained flags are far less likely to earn that trust than ones that show their reasoning.

"The most important slide in the final presentation wasn't a model architecture diagram — it was a mock interface showing exactly what a referee would see, and why. Systems like this succeed or fail on human factors just as much as on accuracy."
  — [Dr. Anita Bhati, Associate Professor], Department of Computer Science & Applications

Open Questions

The exercise raised thoughtful classroom debate that went well beyond the code itself. How should a system explain why it flagged a play in terms a referee can act on quickly, under time pressure, mid-broadcast? What accuracy threshold is high enough to be genuinely useful without overwhelming officials with false positives? Who is accountable when a flagging system misses something significant, or flags something that turns out to be a distraction? And how should such a system be evaluated and audited over time, given that officiating standards themselves evolve from season to season?

These questions mirror open problems across many high-stakes computer vision applications, from sports officiating to medical imaging triage to safety-critical monitoring systems — domains where a model's job is not to replace expert judgment but to direct scarce expert attention efficiently and transparently. Framing the basketball project this way connected it explicitly to broader themes the department addresses in its human-AI interaction and applied ethics coursework.

Comparing to Existing Replay Workflows

To ground the project in reality, the team studied publicly available descriptions of how professional replay-review workflows are currently structured — typically a human-driven process where officials manually cue through multiple camera angles after a play is stopped for review. This research clarified an important framing: the value of an assistive system is not necessarily improving accuracy over a careful human review, since officials already view multiple angles at low speed and are quite accurate given enough time. The real potential value is in speed and completeness of coverage — flagging borderline moments that were never stopped for review in the first place, because in real time nothing about them looked unusual enough to a human observer to warrant a whistle.

This reframing shifted the team's own success criteria partway through the semester, away from 'can the system be as accurate as a human reviewer' and toward 'can the system reliably surface the small number of moments per game that deserved a second look but would otherwise have been missed entirely' — a meaningfully different and, the team concluded, more realistic and more valuable goal for this kind of assistive technology.

A Brief Ethics Discussion

The project closed with a structured classroom discussion on the ethics of introducing AI-assisted tools into officiating. Students raised concerns about over-reliance — the risk that officials might defer to a flag they do not fully understand rather than trusting their own judgment — and about fairness, since any systematic bias in the underlying pose or tracking models could translate into systematically uneven scrutiny across different players or play styles. The team did not resolve these concerns, and did not expect to; the point of the discussion was to make sure a technically capable system was evaluated against more than just its accuracy numbers before any serious discussion of real-world use.

"The most valuable outcome of this project wasn't the pipeline itself. It was watching students learn to ask 'should this be deployed' as a genuinely separate question from 'does this work,' and to treat the second question as necessary but not remotely sufficient."
  — [Dr. Anita Bhati, Associate Professor], Department of Computer Science & Applications

Looking Ahead

Next steps for the project include extending the pose-estimation component to detect specific contact types more precisely — distinguishing incidental brushing from a clear shove, for instance — and running a small user study with volunteer referees or officiating students to evaluate whether the explanation interface actually speeds up review in practice, rather than just looking reasonable in a demo. The team is also interested in formally measuring the system's false-positive and false-negative rates against a larger set of simulated scenarios, to put real numbers behind the precision-focused tuning philosophy the project adopted.

Beyond basketball, the project has already sparked interest from students working on unrelated capstone projects involving anomaly detection in manufacturing and network security — a reminder that the core lesson here, building systems that augment rather than replace human judgment while explaining themselves honestly, generalizes far beyond the court.

About the authors: [Dr. Anita Bhati, Associate Professor] researches human-AI interaction and computer vision in the Department of Computer Science & Applications. [Sunita Maurya (MCA Student)] is a student who designed the anomaly-scoring pipeline and explanation interface described in this project.