A University of Pittsburgh project is building a robotic mobility and manipulation platform whose eyesight comes from Meta. RAMMP — the Robotic Assistive Mobility and Manipulation Platform for enabling independence of people with disabilities — runs Meta's open vision models, DINO and Segment Anything, directly on battery-powered hardware instead of shipping camera frames to a server. ARPA-H is funding it up to $41.5 million. The population it targets is large and badly served: about 5.5 million people in the United States use wheelchairs, and more than 100,000 wheelchair-related injuries, often from trips and falls, are treated in US emergency rooms every year.
Running perception on board is the decision the rest of the project hangs on. A child can step onto the sidewalk, a curb can appear, a set of keys can drop onto the floor in front of the chair. None of that waits for a network round trip, and on-device inference also removes the failure mode where the platform stops understanding the world because the connection did.
RAMMP combines current robotics hardware, a new operating system and digital twin technology. The digital twin gives the team a virtual environment in which the platform can be tested and reworked safely and at scale — cheaper and less dangerous than putting early builds under people who depend on them to leave the house.
The two Meta models do different jobs. DINO is a self-supervised vision transformer: it extracts visual features from unlabeled data, which makes it suited to situations where curated image sets barely exist. SAM handles segmentation — given a short prompt it can find and outline almost any object in an image or video. Both have already been used in work ranging from medical image analysis to wildlife monitoring.
Those capabilities are already in the first RAMMP prototype. DINO-based tools query the camera feed and locate automatic door buttons, cups, curbs and the ground surface for navigation. With the system ready for real-world testing, the engineers have turned to voice and touch input, so a user can select a specific object nearby and act on it.
The perception system itself is built on RF-DETR, a lightweight detection model fine-tuned using DINOv2 embeddings. SAM is what makes the training data affordable: it labels images automatically, which lets the team produce good annotations quickly across different angles, heights, backgrounds and lighting conditions. Additional data transformations and multi-view strategies keep results consistent when the perspective changes. The pipeline runs in four steps:
1. Capture the environment 2. Automatic labeling with SAM 3. Train RF-DETR 4. 360-degree object detection
Getting DINOv3 and SAM onto compact battery hardware is an exercise in subtraction. The team has to account for battery life, heat dissipation, unstable network connectivity and strict limits on the size and weight of the device. Engineers cut the memory footprint, use reduced-precision compute where it helps, and pick formats suited to field conditions. Practical image resolution and efficient batching keep both models fast and stable on small powered platforms and robotic arms. Sometimes, for the sake of speed and predictability, they give up a little accuracy at object boundaries or some feature detail.
The interaction model is where the project's ambition sits. New assistive robotics has to help with ordinary domestic actions, and the argument is that the user should interact with the robot through the surroundings rather than through a control interface — language plus image, so a person can ask questions about the scene in front of them and give direct commands. Asking the robot to pick up a cup from the table should not require cycling through control modes and holding extra context in your head.
Sivashankar Sivakanthan, RAMMP's chief of staff, says the effectiveness of assistive robotics is determined by how reliably a system performs in unpredictable everyday environments, not by a benchmark result. Running DINOv3 and SAM directly on the device, he says, gives real-time perception a user can trust, without depending on connectivity and while preserving safety requirements.
HERL and ATDev are the consortium's key partners. HERL leads the initiative, drawing on its work in biomedical engineering and user-involved research and setting the direction for assistive mobility; ATDev handles the engineering and turns HERL's work into devices that survive outside a lab. Wheelchair users, clinicians and advocacy organizations take part in the project, which is meant to keep the design anchored to actual needs rather than imagined use cases. The national consortium also includes Kinova Robotics, LUCI Mobility, ATDev, and researchers from Carnegie Mellon, Cornell, Northeastern and Purdue.
Mansoor Khan, an ARPA-H program manager, says the project should help Americans with limited mobility live more independently, work and spend their free time as they choose; Meta's vision models, in his framing, give the platform the ability to see — to understand a scene, orient itself in the space around it and pick out the right objects — which reduces the cognitive load on the user. RAMMP is also supposed to generate jobs and manufacturing capacity in Pittsburgh and across Pennsylvania, in assistive mobility products made domestically. ARPA-H director Alicia Jackson, PhD, says people with limited mobility need technology that adapts to their circumstances, and that working with Meta and using advanced AI models in robotics for people with disabilities will speed up building it.
Sivakanthan's line about benchmarks is the most interesting sentence in the whole announcement, and not only because it is true. It is also unfalsifiable as presented: there is not a single accuracy figure, latency number or benchmark score anywhere in this material, so the claim that benchmarks are not what matters arrives with nothing to compare it against. Meanwhile the perception stack is being rebuilt while the vehicle is moving — the prototype that is ready for real-world testing uses DINOv2 embeddings, and the team's stated next step is to integrate SAM 3.1 and DINOv3. That is a reasonable engineering posture for a research project. It is a stranger one for a device that is about to be tested under people for whom a missed curb is an emergency-room visit.
There is a dependency here that nobody names. A federally funded assistive mobility program has made its ability to see contingent on one company continuing to publish open vision model weights. Meta has done that consistently, and the models are genuinely good, which is why they are in a wheelchair platform and in wildlife surveys at the same time. But nothing obliges Meta to keep doing it, and nothing in this announcement says what RAMMP's perception layer becomes if DINOv4 or SAM 4 arrives behind an API, or does not arrive at all. The digital twin, the operating system and the consortium are all institutions the project controls. The eyes are not.
The other silence is commercial. The announcement talks about manufacturing in Pennsylvania and new jobs, which implies a product, but there is no unit cost, no timeline, no word on how a user would actually pay for one of these — and for a device aimed at 5.5 million people, reimbursement is the difference between a research platform and a thing anyone gets to own. Up to $41.5 million buys a lot of engineering. It does not buy a payment code.
The next stages are aimed squarely at the failure modes that matter: temporal consistency in the model's output, robustness across real-world conditions, and a tighter link between perception and the decision-making and control systems. Further out, RAMMP is meant to account for each user's individual needs, behavior and context, and adapt to them over time.
The digital twin will tell the team how the platform behaves in a simulated kitchen. The 100,000 injuries a year happen in real ones, and the distance between those two kitchens is exactly what the project has to cross — a distance that no volume of auto-labeled training data closes on its own.