Every day, billions of hours of video are captured by cameras embedded in smartphones, vehicles, robots, and surveillance systems. Making sense of that video in real time, identifying who and what is in each frame and keeping track of them as they move, is one of the core challenges of modern AI. Within the EdgeAI European project, researchers at Eindhoven University of Technology (TU/e) are tackling exactly this problem with a new model called VidEoMT, short for Video Encoder-only Mask Transformer.
To appreciate what makes VidEoMT different, it helps to first understand what the problem actually involves. Video segmentation requires a model to do three things at once in every frame of a video: pinpoint each object by drawing a precise boundary around it at the pixel level, label it with the right category (person, car, cyclist) and then follow each object over time, so the system knows that the cyclist visible in frame 1 and the one visible in frame 50 are the same individual. Doing all three accurately and quickly is notoriously difficult.
Comparison between the architecture of a state-of-the-art video segmentation method (CAVIS) and VidEoMT.
A year ago, the TU/e Machine Perception Systems team published EoMT (Encoder-only Mask Transformer), which tackled this problem for still images. EoMT showed that a single large Vision Transformer (ViT), pre-trained on enormous amounts of data with models like DINOv2, can handle image segmentation on its own, without any specialized add-on modules, matching the accuracy of far more complex systems while being up to 4 times faster. The insight is that these vision foundation models develop rich internal representations that already encode object identity, shape, and appearance, making hand-crafted components redundant.
Until now, however, video segmentation had resisted this kind of simplification. The standard approach still stacks many specialized components on top of the image segmenter: re-identification modules, context-aware feature extractors, and a dedicated object tracker. Each component adds computational cost. The result is a system that can be accurate but is extremely slow, often processing fewer than 20 frames per second even on high-end hardware, far below what real-time applications demand and practically out of reach for the constrained hardware that edge devices operate on.
VidEoMT asks whether the same principle that worked for EoMT can be extended to video. The additional challenge is tracking: a plain image model processes each frame independently and has no memory of what came before, so it cannot associate the same object across frames. VidEoMT solves this with a lightweight memory system built directly into the ViT, called Query Fusion. At each new frame, a compact set of learned vectors representing the objects seen so far is carried forward and blended with a fresh set of learnable vectors. This gives the model a memory of what it was tracking while also allowing it to spot objects that appear for the first time, without a dedicated tracking module and at almost no additional computational cost.
Diagram of the VidEoMT architecture, showing how learned queries and propagated track queries are combined by the query fusion module before being processed jointly with image patches in the final layers of the ViT encoder.
The results are compelling. On standard video segmentation benchmarks, VidEoMT with a large ViT backbone processes video at up to 160 frames per second, more than ten times faster than the previous state of the art, while maintaining virtually the same accuracy. On other tasks such as video panoptic and semantic segmentation, VidEoMT is five to nineteen times faster than competing methods, with only marginal differences in quality. Because the model consists almost entirely of plain Transformer operations, it also benefits directly from the hardware and software optimizations increasingly available on modern chips, including those designed for edge deployment.
Speed versus accuracy plot showing VidEoMT (orange) running more than 10× faster than CAVIS (blue) at comparable accuracy across different model sizes, evaluated on YouTube-VIS 2019.
This kind of efficiency gain matters far beyond benchmark leaderboards. A ten-fold speedup means that a video understanding task previously requiring a high-end server GPU can now run on a lightweight edge device, with lower energy consumption and without sending raw video over a network. For the EdgeAI project, which aims to bring powerful AI to sensors, cameras, and embedded systems operating at the edge of industrial and urban infrastructures, this directly enables new classes of applications. Autonomous vehicles can track pedestrians and cyclists in real time from onboard cameras. Smart city cameras can monitor traffic flows locally without uploading footage to the cloud. Industrial robots can perceive and respond to their surroundings faster and more safely. Surgical or endoscopic cameras in medical settings can provide real-time visual assistance to clinicians.
VidEoMT is a reminder that progress in AI does not always mean adding more components. Sometimes it means asking whether all the complexity was necessary in the first place, and discovering that a well-trained model, given enough capacity, can quietly absorb tasks that once required entire sub-systems to solve.
More information and code are available at: https://www.tue-mps.org/videomt/
Blog signed by: Machine Perception Systems (MPS) group, Eindhoven University of Technology (TU/e)