Coordinated Diffusion: Generating Multi-Agent Behavior Without Multi-Agent Demonstrations

Lasse Peters¹ Laura Ferranti^2,† Andrea Bajcsy^3,† Javier Alonso-Mora^2,†

¹UC Berkeley ²TU Delft ³Carnegie Mellon University ^†Equal advising

Paper (arXiv) Code (coming soon) Related Publications

Abstract

Imitation learning powered by generative models has proven effective for modeling complex single-agent behaviors. However, teaching multi-agent systems, like multiple arms or vehicles, to coordinate through imitation learning is hindered by a fundamental data bottleneck: as the joint state-action space grows exponentially with the number of agents, collecting a sufficient amount of coordinated multi-agent demonstrations becomes extremely costly. In this work, we ask: how can we leverage single-agent demonstration data to learn multi-agent policies? We present CoDi, a framework that couples independently trained single-agent diffusion policies through a user-defined multi-agent cost function, without requiring any coordinated demonstrations. We derive a new diffusion-based sampling scheme wherein the diffusion score function decomposes into independent, single-agent pre-trained base policies plus a cost-driven guidance term that coordinates these base policies into cohesive multi-agent behavior. We show that this guidance term can be estimated in a gradient-free manner, making CoDi applicable to black-box, non-differentiable cost functions without additional training. Theoretically and empirically, we analyze the conditions under which this composition can faithfully approximate a target multi-agent behavior. We find a complementary role for demonstration data versus the cost function: single-agent demonstrations must cover the support of the desired multi-agent behavior, while the cost function must promote desired behavior from this product of single-agent policies. Our results in simulation and hardware experiments of a two-arm manipulation task show that CoDi discovers robust coordinated behavior from single-agent data, is more data-efficient than multi-agent baselines, and highlights the importance of joint guidance, base-policy support, and cost design.

Method Overview

CoDi inverts the conventional centralized training, decentralized execution paradigm: instead, it enables decentralized (single-agent) pre-training at the cost of requiring centralized execution. The method proceeds in three steps:

Train independent single-agent diffusion policies from single-agent demonstrations of basic skills (e.g., picking and placing objects).
Specify a user-defined multi-agent cost function that encodes the desired coordination behavior (e.g., move the object to a goal, avoid collisions).
At deployment, solve a KL-regularized cooperative game that combines the single-agent base policies with a cost-driven guidance term into a cohesive multi-agent policy.

The resulting multi-agent policy takes the composite form $$\pi(\mathbf{a} \mid \mathbf{s}) \propto \exp\!\left(-J(\mathbf{s}, \mathbf{a}) / \lambda\right) \cdot \prod_i p^{(i)}(a^{(i)} \mid s^{(i)}),$$ where each $p^{(i)}$ is a single-agent diffusion policy and $J$ is the shared cost function. Crucially, the guidance term is estimated in a gradient-free, sampling-based manner, making the approach applicable to black-box simulators and non-differentiable costs.

In the paper, we formally analyze under which conditions this structure can recover a desired target behavior $\pi^*$. The key result is that exact recovery is possible when the product of single-agent policies is an over-approximation of $\pi^*$ — i.e., wherever $\pi^*$ assigns positive probability to a joint action, the product must as well (a support-coverage condition). Under this condition, the cost $J$ can compensate for the KL gap between the product of single-agent policies and $\pi^*$: the more informative the single-agent demonstrations are for the targeted multi-agent behavior, the less design effort needs to go into $J$.

Results

Key Takeaways

(C1) CoDi discovers object-handover strategies from single-agent demonstrations alone — no multi-agent data required.
(C2) CoDi is robust enough for real-hardware deployment and generalizes to unseen objects.
(C3) CoDi outperforms methods trained on multi-agent demonstrations at the same data budget, using cheaper single-agent data that also transfers better when data is scarce.
(C4) Joint guidance is critical: independently guiding single-agent policies leads to arm collisions and task failures.
(C5) The sampling-based guidance estimator is necessary — classifier guidance fails when applied to independently pre-trained single-agent policies.
(C6) The cost function must supply the coordination signal absent from single-agent demonstrations; a simplified cost degrades CoDi but not a jointly trained baseline.

Hardware Experiments

We deploy CoDi on two 7-DoF Franka Research 3 arms placed at opposite ends of a table (1.8 m wide × 1.2 m deep) — wide enough that no single arm can reach across. CoDi guides the left arm to pick up the cube and place it at an intermediate location reachable by the right arm, which then completes the task by placing the cube at the goal (green star). (C1) Object handovers are not present in the single-agent training data — CoDi discovers this collaboration strategy by composing the individual manipulation skills through the multi-agent cost function. The supplementary video also includes examples where the cube is replaced with other objects, demonstrating (C2) robustness of the learned behavior beyond the training conditions.

Simulation Results

CoDi is more data-efficient than methods requiring multi-agent demonstrations (C3)

We compare CoDi (trained on single-agent data only) against four baselines that use multi-agent demonstration data: classifier-guided diffusion (CG-Joint), diffusion policy mirror descent (DPMD-Joint), stochastic diffusion actor-critic (SDAC-Joint), and expert policy optimization (EXPO-Joint). Despite the less informative training data, CoDi achieves better task efficiency and manipulation accuracy than all baselines. As an additional ablation, applying our guidance scheme to a joint base policy (CoDi-Joint) also improves performance but falls short of CoDi with single-agent pre-training — confirming that the single-agent state-action space yields better manipulation skills per demonstration.

Task efficiency

Manipulation accuracy

We further study the sensitivity to demonstration data density. Training CoDi on only 20% of the single-agent data still achieves a task success rate close to CG-Joint trained on the full multi-agent dataset (88% vs. 92%), while substantially outperforming CG-Joint trained on the same reduced multi-agent dataset (54% success rate). This highlights that single-agent demonstrations — which are far cheaper to collect — transfer more efficiently to the multi-agent setting.

Task efficiency

Manipulation accuracy

Joint guidance is critical for safe coordination (C4)

A key design choice is that CoDi generates joint actions for all robots simultaneously, rather than guiding each robot independently. Without joint guidance (CoDi-Indep.), robots can fail to coordinate: both arms attempt to reach the cube at the same time, leading to arm collisions and task failures. Joint guidance eliminates this failure mode by explicitly accounting for inter-agent interactions through the collision-avoidance and engagement cost components.

Task efficiency

Manipulation accuracy

Safety (inter-arm distance)

Independent guidance failure: both arms reach for the cube simultaneously, causing a collision.

The guidance estimator matters when composing single-agent policies (C5)

We compare our sampling-based guidance estimator against applying vanilla classifier guidance to the product-of-marginals base policy (CG-Product). While classifier guidance works reasonably well when the base policy already encodes coordination (as in the joint baseline setting), it fails when applied to independently pre-trained single-agent policies. The product-of-marginals policy exhibits a larger tilt relative to the coordinated target distribution, making the learned cost model a poor guide. Our sampling-based estimator bridges this gap without requiring an additional trained cost model.

Task efficiency

Manipulation accuracy

The cost function must compensate for the coordination gap in single-agent data (C6)

We isolate the role of the guidance cost by evaluating a simplified variant that retains only the goal term, removing the collision-avoidance and engagement components. CoDi with single-agent base policies degrades substantially under this simplified cost, whereas a joint base policy trained on multi-agent demonstrations remains largely unaffected. This confirms the complementary roles of demonstrations and the cost function: when coordination is not present in the training data, the cost must explicitly supply it.

Task efficiency

Manipulation accuracy

Limitations & Future Work

Using available multi-agent data: This work studies the extreme case of no multi-agent demonstrations. In practice, some such data may exist. How to combine it with single-agent demonstrations — e.g., for fine-tuning the base policy or learning the guidance term directly — is a natural next step.
Centralized execution: CoDi coordinates agents centrally at test time. Distilling the resulting joint policy into a decentralized one is an important direction for deployment in settings where communication is limited.
Cooperative assumption: CoDi assumes all agents optimize a shared cost. Extending the framework to non-cooperative settings with distinct per-agent objectives is an open avenue for future work.

Back to Publications