SeededGrasp: Language-Guided Grasping in Complex Scenes with Multiple Embodiments

SeededGrasp: Language-Guided Grasping in Complex Scenes with Multiple Embodiments

arXiv:2607.20207 | Domain: Dexterous Grasping / Natural Language / Multi-Embodiment | Dataset: 2.56M grasps, 610 scenes, 334 objects, 3 grippers | Project: uoft-isl.github.io/seeded-grasp

One-Sentence Summary

SeededGrasp decouples language-guided grasping into two stages: an off-the-shelf VLM zero-shot predicts a 3D seed point encoding semantic intent, then a lightweight flow-matching model generates a multi-embodiment grasp pose conditioned on that seed—requiring no language-annotated dataset or expensive end-to-end training, achieving 72% success in simulation and 78% real-world, with the first multi-embodiment tabletop cluttered-scene grasping dataset (2.56M grasps).

Background and Motivation

Dexterous grasping is fundamental to general manipulation but remains challenging. A robot must contact objects precisely, stably, and task-awarely; in multi-object cluttered scenes it must also respect surroundings. Since downstream tasks impose different functional requirements, following language instructions is highly advantageous. The challenge is compounded by end-effector variety: simple two-finger grippers are limited to pinch grasps on complex geometries; high-dexterity grippers like the Allegro Hand can approach objects in multiple ways but are hard to control. Supporting multiple embodiments also improves grasping performance—a single multi-embodiment model is more useful and data-efficient than specialized ones.

Existing methods address only parts of the problem. Some focus on isolated objects, simplifying perception but missing clutter complexity. Language-grounded grasping works often require large annotated datasets and expensive end-to-end VLM training, or use pretrained VLMs for high-level representations with complex pose optimization—these do not scale to dense-occlusion multi-object multi-embodiment settings.

SeededGrasp's core insight: use a seed point as an efficient interface between the VLM and the grasp-generation model. VLMs excel at identifying semantically relevant grasp regions from BEV images and language but are poor at predicting precise 3D grasp poses; flow-matching models excel at generating geometrically precise grasps from point clouds but lack semantic understanding. The seed point compresses semantic intent into a 3D coordinate that bridges both—avoiding both the VLM's spatial-awareness deficit in direct grasp prediction and the data/compute cost of end-to-end joint training.

Figure 1: Example grasps and performance summary of SeededGrasp.

Dataset Generation

Training a seed-point-conditioned grasp model requires valid grasps in cluttered scenes for each gripper. SeededGrasp builds on the MultiGripperGrasp (MGG) dataset, which provides millions of synthetic mid-air grasp poses across diverse grippers and objects. Adapting it involves generating cluttered scenes, placing objects on tabletops, projecting mid-air grasps into scenes, and filtering collisions. The final dataset contains 2.56M grasps, 610 scenes, 334 objects, 3 grippers (Franka Panda, Robotiq 3-Finger, Allegro Hand)—the first multi-embodiment cluttered-scene grasping dataset.

Method

Two-Stage Architecture

Figure 2: VLM selects a seed point; point-cloud encoding and grasp prediction are trained end-to-end, predicting a grasp pose conditioned on the seed point. Training uses dataset-derived seed points.

The pipeline has two stages: seed-point prediction, then grasp-pose prediction conditioned on it. Inspired by DGN2.0 but not requiring a trained seed-point model—instead an off-the-shelf VLM zero-shot predicts a suitable grasping location from a BEV scene image and user instruction. The seed point is passed to a trained flow-matching model synthesizing a grasp for the selected gripper. This modular design makes the VLM swappable, naturally benefiting from future VLM improvements.

Point Cloud Encoding

Gripper and scene geometries are colorless point clouds $p_r \in \mathbb{R}^{1024 \times 3}$ and $p_c \in \mathbb{R}^{2048 \times 3}$. Point coordinates are Fourier-encoded and concatenated with local normal $n \in \mathbb{R}^3$ and curvature $\sigma(p) \in \mathbb{R}$ (estimated via local-neighborhood PCA) to enrich surface representation. Each scene point's representation also includes its distance to the seed point. Graph Convolutional Networks encode point clouds using 16 nearest neighbors, yielding per-point features $f_p \in \mathbb{R}^{512}$; global representations use max and mean pooling. To capture local geometry around the seed, features from its 16 nearest neighbors are concatenated with global scene features. A learnable query $q_r \in \mathbb{R}^{512}$ encodes gripper selection. Final features: $f_r \in \mathbb{R}^{3 \times 512}$, $f_c \in \mathbb{R}^{18 \times 512}$.

Grasp Pose Parameterization and Flow-Matching Generation

Each pose is parameterized as $g = (T, R, \theta)$, with $T \in \mathbb{R}^3$ translation, $R \in \text{SO}(3)$ rotation (Euler angles), and $\theta$ the gripper joint configuration. The pose representation is padded to the maximum DoF (Allegro's 16 joints). The stable-grasp distribution conditioned on robot and scene features $p(g \mid f_r, f_c)$ is complex and multimodal, so flow matching is used. The model $u_\theta(t, g_t, f_r, f_c)$ is trained to predict the ground-truth vector field $u_t(t, g_t, f_r, f_c)$, transforming a noisy sample $g_t$ into a target-distribution sample given $t \in [0,1]$. The network uses a standard DiT architecture with cross-attention for conditioning.

Training Objective

The network is optimized with a conditional flow-matching loss $\mathcal{L}_{\text{cfm}}$ across translation/rotation/joint components. The total loss is a linear combination:

$$\mathcal{L} = \lambda_{\text{trans}} \mathcal{L}_{\text{trans}} + \lambda_{\text{rot}} \mathcal{L}_{\text{rot}} + \lambda_{\text{joints}} \mathcal{L}_{\text{joints}}$$

with $\lambda = (0.4, 0.04, 0.8)$. Unused joint dimensions are masked. An L1 loss replaces standard L2:

$$\mathcal{L}_{\text{cfm}} = \mathbb{E}_{t \sim \beta(1.5, 1.0),\; (g_t, f_r, f_c) \sim \mathcal{U}} \left[ \left| u_\theta(t, g_t, f_r, f_c) - u_t(t, g_t, f_r, f_c) \right| \right]$$

L1 improves precision of generated grasps adhering to scene geometry. The time step $t$ is sampled from $\beta(1.5, 1.0)$ to bias toward later steps ($t \to 1$) where accurate vector fields are critical. Classifier-free guidance randomly drops conditioning features during training to learn both conditional and unconditional fields, extrapolating at inference ($w=1.1$) toward stronger conditioning.

Inference: VLM Seed Point and ODE Solving

At inference the VLM is prompted zero-shot to identify a stable grasping location aligned with the user instruction; the predicted image point is projected onto the 3D scene point cloud and passed to the generation module. The final grasp pose is generated by initializing $g_0$ as a fixed canonical grasp pose adjacent to the seed point (not pure noise), then integrating through the predicted vector field from $t=0$ to $t=1$ via forward Euler (step size 0.2)—simplifying the trajectory and minimizing numerical error accumulation.

Experimental Results

Seed Point Selection and Conditional Grasp Prediction

Method (Allegro only)	Graspness Seeds	VLM Seeds
DGN2.0	53.15	66.67
Ours	66.22	72.97

Table 2 (left): Trained on Allegro data only, SeededGrasp beats DGN2.0 by 13% on the same seeds; VLM seeds outperform DGN2.0's dedicated seed module.

SeededGrasp outperforms DGN2.0 by 13% on single-embodiment grasping with the same seed points. Gemini 3.1 Flash seed points yield higher success than DGN2.0's specially trained seed module. Unlike DGN2.0, the model needs no explicit segmentation masks or extra feature vectors and supports multiple end-effectors.

Multi-Embodiment Performance

Method	Franka	Robotiq	Allegro
Geomatch	25.35	35.71	37.97
Ours	71.38	72.16	72.07

Table 2 (right): On multi-embodiment data, SeededGrasp outperforms Geomatch by ~35% on complex objects in cluttered scenes.

SeededGrasp outperforms Geomatch by ~35% on complex objects and cluttered scenes. The gap stems from Geomatch's architecture of predicting contact points then relying on IK optimization—fragile on complex geometries, exacerbated by multi-camera point-cloud fusion yielding incomplete hollow objects without bottoms.

Language-Conditioned Grasp Prompting

Method	Obj. & Part Recog. Succ.	Overall Grasp Succ.	Easy Prompt	Difficult Prompt
ShapeGrasp w/ Obj.Seg.	57.5	51.6	31.1
GraspMAS	42.5	29.0	24.5
Ours	89.6	80.6	58.2

Table 3: Language-conditioned grasp quality. SeededGrasp far exceeds baselines in correctly recognizing the target object/part and producing valid language-following grasps.

On 226 scene-prompt query pairs (164 easy + 62 difficult), SeededGrasp far exceeds ShapeGrasp and GraspMAS in correctly recognizing target objects/parts and producing valid grasps. Both baselines rely exclusively on BEV images and cannot use point clouds; the seed-point interface eliminates the need for complex prompting architectures.

Real-World Evaluation

Figure 3: Real-world grasping experiments.

Evaluated with a Delto DG-3F-B three-finger gripper (not in training), mapping Robotiq 3-Finger predictions to Delto. Two ZED stereo cameras generate the tabletop point cloud. 15 unseen objects, 10 trials each with 4 random distractors: overall success rate 78%, validating sim-to-real and embodiment-transfer capability.

Figure 4: Additional experimental results and grasp examples.

Ablation: Gripper Encoding

Method	Franka	Robotiq	Allegro
w/o Query Vector	72.41	69.07	59.01
w/o Robot PC	72.76	67.53	62.16
SeededGrasp	71.38	72.16	72.07

Table 4: Removing either gripper representation decreases success, especially for the more kinematically complex Robotiq and Allegro.

The model encodes gripper info using both robot point cloud and a learnable gripper-specific query vector. Both are important, especially for the more complex Robotiq 3-Finger and Allegro—they let the network capture each embodiment's geometric intricacies and generalize across grippers.

Seed Point as Information Bottleneck

The seed-point interface can be understood from an information-bottleneck perspective. Let the semantic information of language instruction $l$ be $H(l)$, the seed point $s \in \mathbb{R}^3$'s capacity be $H(s)$, and the grasp-pose distribution entropy be $H(g \mid f_r, f_c)$. The design goal is to maximize semantic information retention while minimizing over-constraint on geometry:

$$\min_{s} \; H(l \mid s) - eta \cdot H(g \mid f_r, f_c, s)$$

i.e., the seed point should preserve as much language semantics as possible ($H(l \mid s)$ small) while restricting grasp-pose diversity as little as possible ($H(g \mid \cdot, s)$ large). A 3D coordinate strikes this balance: it carries the "where to grasp" semantics without the "how to grasp" geometry.

Conditional Flow-Matching Generation

Flow matching constructs a probability path from noise $g_0 \sim \mathcal{N}(0, I)$ to the target distribution $p(g \mid f_r, f_c)$. The training objective learns a vector field $u_ heta$ matching the true conditional vector field:

$$\min_ heta \; \mathbb{E}_{t, g_t, f_r, f_c} \left\| u_ heta(t, g_t, f_r, f_c) - u_t(t, g_t \mid g_1, f_r, f_c) ight\|$$

At inference the ODE $rac{dg_t}{dt} = u_ heta(t, g_t, f_r, f_c)$ is integrated from $g_0$ to $g_1$. The paper uses forward Euler (step 0.2, 5 steps), initializing $g_0$ as a canonical pose rather than pure noise to reduce numerical error.

Classifier-Free Guidance Conditional Extrapolation

CFG enhances conditioning adherence at inference by extrapolating between conditional and unconditional vector fields:

$$ ilde{u}_ heta = u_ heta(\cdot \mid \emptyset) + w \cdot \left( u_ heta(\cdot \mid f_r, f_c) - u_ heta(\cdot \mid \emptyset) ight)$$

with guidance strength $w=1.1$. Training randomly drops conditioning so the network learns both $u_ heta(\cdot \mid \emptyset)$ and $u_ heta(\cdot \mid f_r, f_c)$. $w>1$ pushes samples toward stronger conditioning alignment, improving fit to scene geometry.

Motivation for Beta-Distribution Time-Step Sampling

Sampling $t$ from $eta(1.5, 1.0)$ biases toward later time steps. In flow matching, $t o 1$ corresponds to the refinement stage near the target distribution, where vector-field accuracy directly determines final grasp-pose precision. The $eta(1.5, 1.0)$ density is higher near $t=1$, focusing training on refinement. This works with L1 loss to improve generated-grasp fit to scene geometry.

From an architectural-design perspective, SeededGrasp's two-stage decoupling contrasts sharply with end-to-end joint training. End-to-end methods (training VLM with the grasp model jointly) can co-optimize semantics and geometry but require large-scale language-annotated datasets and enormous compute—especially costly in multi-embodiment cluttered scenes. SeededGrasp's modular strategy delegates semantic reasoning entirely to a frozen off-the-shelf VLM (zero-shot, no fine-tuning) and geometric generation to a lightweight flow-matching model (trained on geometric data only), coupled via the minimal seed-point interface. This "frozen semantic module + trained geometric module + minimal interface" design reduces training cost from "language+geometry end-to-end" to "geometry only" while preserving semantic flexibility (VLM is swappable).

The gripper-encoding ablation reveals an interesting finding: for the kinematically simple Franka Panda, removing either representation has little impact (even slight improvement), but for the complex Robotiq and Allegro both are indispensable. This suggests that simple grippers' geometric information is sufficiently captured by the point cloud, while complex grippers need additional learnable queries to encode their unique kinematic structure. This finding guides future multi-embodiment learning: embodiment-representation complexity should match the embodiment's own kinematic complexity.

The 78% real-world success on the untrained Delto gripper via Robotiq mapping is more significant than the number alone. It shows the flow-matching model learns grasp distributions tied to gripper geometry rather than memorizing specific joint configurations—enabling transfer to geometrically similar but untrained grippers via simple joint-offset mapping. This "geometric transfer" capability is preliminary evidence for open-embodiment grasping, though currently still requiring manual mapping.

System Architecture Diagram

flowchart LR
  subgraph Stage1["Stage 1: VLM Seed Point Prediction"]
    BEV["BEV scene image
+ language instruction"]
    VLM["Off-the-shelf VLM
Gemini 3.1 Flash
zero-shot"]
    SEED["3D seed point
projected to point cloud"]
  end
  subgraph Stage2["Stage 2: Flow-Matching Grasp Generation"]
    PC["Scene PC p_c
+ gripper PC p_r"]
    ENC["GCN encoding
Fourier+normal+curvature
+seed distance"]
    FEAT["Conditioning f_r, f_c
+learnable gripper query"]
    DIT["DiT flow matching
u_theta(t,g_t,f_r,f_c)"]
    GRASP["Grasp pose g=(T,R,theta)
Euler integration t:0->1"]
  end
  BEV --> VLM --> SEED
  PC --> ENC --> FEAT
  SEED --> ENC
  FEAT --> DIT --> GRASP

Limitations

No arm-kinematics awareness (author-acknowledged). In real deployment grippers are mounted on arms that cannot reach every predicted pose, especially when avoiding obstacles. The model currently predicts grasp poses without arm-kinematics awareness and may propose infeasible actions. Future work should jointly address grasp selection and arm motion planning.

Single-image VLM conditioning (author-acknowledged). Currently only a single BEV image conditions the VLM; multi-view perspectives could improve spatial reasoning and robustness when optimal grasp regions are occluded in some views.

Embodiment-specific queries limit generalization (author-acknowledged). Current learnable queries are embodiment-specific; moving to a unified joint-space representation could facilitate better transfer to unseen hardware.

Training-data bias toward power grasps (author-acknowledged). The over-representation of power grasps in MGG limits effectiveness on flat objects sitting flush against a surface.

Summary and Outlook

SeededGrasp uses a VLM-predicted seed point to encode semantic intent combined with a flow-matching model for grasp generation, achieving strong performance in both simulation (72%) and real-world (78%). Seed-point conditioning is an effective bridge between language grounding and grasp generation—it decouples semantic reasoning (VLM's strength) from geometric execution (flow-matching's strength) via a minimal interface (one 3D point), avoiding both the VLM's spatial deficit in direct grasp prediction and the data/compute cost of end-to-end training. The modular design makes the VLM swappable, naturally benefiting from future improvements.

From a broader perspective, SeededGrasp embodies a "decouple semantics from geometry via a minimal interface" design philosophy. The seed point's elegance lies in its information content being just right: enough to constrain grasp location (the geometric projection of semantic intent), yet not so much as to limit grasp-pose diversity (flow matching can still explore multimodal grasps near the seed). This "compress semantics to a point, release geometry as a distribution" idea may inspire other semantic-geometric bridging tasks. With arm-kinematics awareness and multi-view conditioning, the seed-point interface could become a standard for general language-guided manipulation.

Golden Lines

"The seed point compresses semantic intent into a 3D coordinate—enough to constrain grasp location, not enough to limit pose diversity—the minimal effective interface between language and geometry."

"Compress semantics to a point, release geometry as a distribution—let the VLM do the semantic reasoning it excels at, and let flow matching do the geometric generation it excels at."