Skip to content

FARM: Find Anything using Relational Spatial Memory

Session: Brought up on Day 4 in the Sim2Real2Field workshop, as the example for "robots can benefit from relational spatial memory." See the Day 4 notes. Paper: arXiv 2606.15476 (PDF) by Siming He, Leo Huang, et al. (UC Berkeley, Stanford), 2026 Links: project page, code

Paper summary (from search-result snippets of the abstract and project page; the paper itself is not yet read)

  • Problem: Finding objects from natural-language queries that describe them by their relations to other objects, such as "the mug next to the laptop."
  • Method:
  • Memory: It builds a compact, open-vocabulary 3D memory of objects in real time (about 5–10 Hz) from RGB-D streams. Each object stores its geometry, visual-language descriptors, and the viewpoints it was seen from.
  • Querying: A VLM parses the query into a graph of a target object, anchor objects, and spatial relations. FARM binds those to objects in memory and scores explicit spatial and semantic predicates. A final VLM stage reranks the results.
  • Results (unverified):
  • 44k language queries across 67 indoor and outdoor scenes.
  • +164% Recall@5 and +224% Recall@10 over prior methods. VLM reranking adds +35% Accuracy@1.
  • It runs in real time on a Boston Dynamics Spot with onboard RGB-D and an NVIDIA Jetson Thor.
  • It introduces a seven-scene benchmark, FARM-Scenes.
  • Caveat: An AI-generated third-party review on pith.science flagged gaps in how the results are reported. Treat the headline numbers as provisional until they're checked in the paper.

Talk notes

  • The workshop's point: how a robot builds its memory, and how it queries it, both matter. FARM addresses both, with structured object memory for building and relational query graphs for querying.

The memory structure, as presented (Day 4 slides, 2:17–2:18 pm):

  • Each object node stores attributes: 3D Gaussians, features, views, captions, and possibly more (the list ended with "...").
  • Edges between objects:
  • Covisibility edges: objects seen together in the same view.
  • Adjacency edges: objects next to each other in space.
  • No higher-order relations. Only pairwise edges. (My reading: a relation like "between A and B" has to be assembled at query time from pairwise edges.)

Building the memory online, at scale (slide photo, Day 4)

  • Formally: The memory up to time t is a set of entities, Mt = {Eti}. Each entity is a pair ⟨attributes, relations⟩.
  • Real-time construction (5–10 Hz):
  • Inputs: RGB, depth, and camera poses.
  • Detection: YOLOE does open-vocabulary detection, segmentation, and embedding.
  • 3D lifting: It back-projects the masked depth and fits a 3D Gaussian (μ, Σ) to each detection.
  • Association: Detections are matched to existing objects in parallel. It first filters on geometry, by Hellinger distance between Gaussians, then on semantics, by cosine similarity of embeddings. A match updates the object, and no match creates a new one.
  • Visual-language encoding (2 Hz, 5 objects at a time):
  • Models: Qwen3.5-9B, Qwen3-VL-Embedding-8B, SigLIP2, and Qwen3-Embedding-0.6B.
  • Output: captions as JSON, for example {"category": "bowl", "supercategory": "container", "attributes": ["white", "round", "shiny"], "description": "white round shiny bowl", "decision": "keep"}.
  • Scale shown:
  • A multi-floor house of ~1,600 m², and a construction site of ~15,000 m².
  • A live demo in a 4,000 m² warehouse on a Spot, processing at 10.63 Hz. It was labelled "relational retrieval, accurate, fault tolerant," with "one configuration across scene scales."

FARM: online memory construction, a relational object graph, and query parsing and scoring

Open-vocabulary retrieval through structured reasoning (slide photos)

  • The pipeline:
  • Take a language query, for example "Find my bowl, the one on the left of the other bowl."
  • A Qwen3.5-9B parser turns it into a symbolic JSON query: target "bowl," with predicate LeftOf($target, bowl).
  • That becomes a query graph of target and anchor variables.
  • Retrieve candidate objects from memory.
  • Score every target and anchor assignment against the predicates, then select the top score and rerank. In the example, target x2 with anchor x3 scored 0.83, the highest in the matrix.
  • Output: the target, its 3D position, and the best view, for example "Frame 572."
  • Harder queries:
  • "Find me the cardboard box with labels on a white table, between a sofa that a humanoid robot sits on and a black office chair." This has five variables and the predicates HasAttribute, On, On, and Between. Between is a three-way relation, built from the pairwise memory at query time.
  • "Find the cardboard box on the right of a black storage bin, under a grey plastic crate, and near a metal industrial ladder." This uses RightOf, Below, and Near.