Choreographing CAV Formations on the Fly with Reflection

GRF-TAFC: a Grounded and Reflective Framework for Task-Adaptive Formation Choreography of CAVs.
GRF-TAFC integrates context-grounded formation generation, optimization-based execution, and execution-driven reflection into a closed loop.

aTsinghua University, bNational University of Singapore, cUniversity of Michigan

On-the-Fly Formation Choreography in Simulation

Seven demonstrations cover lane unavailability, accident-induced lane closures, longitudinal spacing adjustment, speech-driven structure switching, and image-guided shape transformation. Quantitative results for these simulation tasks are reported below.

1. Independent Lane-Availability Tasks

1.1 Lane 1 Becomes Unavailable

Three-lane road; the formation adapts to the remaining two lanes.

1.2 Lane 3 Becomes Unavailable

The symmetric lane-closure condition, executed independently.

1.3 Accident in Lane 1

Four-lane road; the formation translates into available lanes.

1.4 Accident in Lane 2

Four-lane road; target positions avoid the closed lane.

2. Successive Task Inputs During Operation

2.1 Longitudinal Spacing Adjustment

The desired spacing increases to 45 m and then decreases to 30 m.

2.2 Speech-Driven Structure Switching

The interlaced formation changes to a parallel formation; a subsequent instruction moves the single vehicle to the rear.

Audio is included: the spoken task inputs can be heard in this video.

2.3 Image-Guided Shape Transformation

An image specifies a circular shape; a subsequent instruction changes the radius reference to the three middle vehicles.

Real-World Experiments

Seven demonstrations cover spacing adjustment, ambulance yielding, obstacle avoidance, speech-driven structure switching, and image-guided shape transformation.

1. Longitudinal Spacing Adjustment

1.1 Decreasing Longitudinal Spacing

The vehicles reduce their longitudinal spacing in response to the operator's instruction.

Because the experimental track is limited in length, the formation does not start moving at the beginning of the demonstration.

1.2 Increasing Longitudinal Spacing

The vehicles increase their longitudinal spacing in response to the operator's instruction.

Because the experimental track is limited in length, the formation does not start moving at the beginning of the demonstration.

2. Yielding to High-Priority Vehicles

2.1 Yielding via Formation Transition

The vehicles switch formation to clear a lane for the approaching ambulance.

Because the experimental track is limited in length, the formation does not start moving at the beginning of the demonstration.

2.2 Yielding via Formation Translation

The vehicles translate the formation into available lanes to make way for the ambulance.

Because the experimental track is limited in length, the formation does not start moving at the beginning of the demonstration.

3. Obstacle Avoidance

The vehicles adapt their formation to avoid the obstacle.

Because the experimental track is limited in length, the formation does not start moving at the beginning of the demonstration.

4. Speech-Driven Structure Switching

Spoken instructions change the interlaced formation to a parallel formation and then move the single vehicle from the front to the rear.

Audio is included: the spoken task inputs can be heard in this video.

Because the experimental track is limited in length, the formation does not start moving at the beginning of the demonstration. The next instruction is issued while the vehicles are operating.

5. Image-Guided Shape Transformation

An image specifies a circular formation, and a subsequent instruction changes the radius reference from the vehicles on both sides to the three middle vehicles.

Because the experimental track is limited in length, the formation does not start moving at the beginning of the demonstration. The next instruction is issued while the vehicles are operating.

Simulation Details and Quantitative Results

The following experimental details and quantitative results correspond to the simulation videos above. Seven CAVs adapt to five task categories. Each of the ten conditions is repeated ten times independently with the same initial experience memory.

100/100Successful generations
0/100Invalid outputs
90/100Optimal outputs
100/100Completed executions

1. Tasks and inputs

The initial interlaced formation has a 2–3–2 distribution across Lanes 1–3, a 30 m within-lane following distance, and 4 m-wide lanes. Accident cases provide a fourth lane; all other cases use three lanes. These repeated tests use the Gemini 3.1 Pro API with Low reasoning effort, temperature 1.0, and top-p 0.95.

Two conditions per category, in the same order as the plots
Task / roadCondition 1Condition 2
Lane unavailability3 lanes · text“Lane 1 will become unavailable.”“Lane 3 will become unavailable.”
Accident closure4 lanes · text“There is an accident in Lane 1!”“There is an accident in Lane 2!”
Spacing adjustment3 lanes · text“Increase the longitudinal spacing to 45 m.”“Decrease the longitudinal spacing to 30 m.”
Structure switching3 lanes · speech“Change the interlaced formation to a parallel formation.”“For this parallel shape, move the single vehicle to the rear instead of the front.”
Shape transformation3 lanes · text + image, then text“Change the formation shape to match desired_shape.jpg in the current directory while keeping the vehicles on both sides fixed.”“For the new circle shape, use the three middle vehicles as the radius reference instead of the vehicles on both sides.”

Speech inputs are prerecorded WAV files; the image input is an actual picture of a black circle. “Single vehicle to rear” moves the lone vehicle at the front of the parallel formation to its rear. The second circular-shape condition changes the radius reference to the three middle vehicles.

Lane-unavailability and accident conditions start independently from the initial formation. The other task pairs run sequentially: condition 2 is dispatched after condition 1 converges, while the vehicles continue driving. This gives 70 simulation runs and 100 task instances. Each condition is repeated ten times independently with the same initial experience memory; experience is not accumulated across repetitions.

Measurement definitions and execution protocol

Generation success means the initial response passes explicit format and formation-feasibility checks; invalid outputs fail these checks. Optimal outputs prioritize feasible whole-formation translation over reconfiguration and otherwise minimize the number of lane changes. When sufficient usable lanes exist, the prompt favors translating the formation while satisfying the task and environmental constraints. Thus, moving all seven CAVs from Lanes 1–3 to Lanes 2–4 is optimal under this priority when an accident closes Lane 1. Lane changes count actual lane-boundary crossings across all seven vehicles, with hysteresis to avoid counting boundary jitter.

Generation time spans the API request to receipt of the complete response. Completion time runs from formation dispatch until the root-mean-square slot-position error first falls below 0.5 m, with a 90 s observation limit.

The root-mean-square slot-position error measures distance to assigned target positions after removing common longitudinal translation, while retaining lateral errors:

\[\Delta x(t)=\frac{1}{N}\sum_{i=1}^{N}(x_i(t)-x_i^*),\qquad e_{\mathrm{slot}}(t)=\sqrt{\frac{1}{N}\sum_{i=1}^{N}\left[(x_i(t)-x_i^*-\Delta x(t))^2+(y_i(t)-y_i^*)^2\right]}.\]

Here, N = 7; (xᵢ, yᵢ) and (xᵢ*, yᵢ*) are actual and assigned target positions in meters. Unlike a pairwise-distance formation-shape metric, this measure detects whether the formation has reached its target lanes.

Generation and execution measurements are reported separately. Generation results characterize the MLLM outputs; execution results characterize realization of the dispatched target formations.

2. Quantitative results

Ten trials per condition; times are mean ± sample standard deviation
ConditionGeneration
success
Invalid
outputs
Optimal
outputs
CompletedGeneration
time (s)
Completion
time (s)
Lane
changes
Lane 1 unavailable10/100/109/1010/1014.31 ± 2.195.52 ± 1.593–4
Lane 3 unavailable10/100/108/1010/1015.18 ± 2.085.79 ± 1.723–4
Accident: Lane 110/100/1010/1010/1011.29 ± 1.821.76 ± 0.087
Accident: Lane 210/100/103/1010/1019.96 ± 5.151.90 ± 0.215–6
Increase spacing10/100/1010/1010/1013.39 ± 3.245.12 ± 0.370
Decrease spacing10/100/1010/1010/1012.79 ± 1.704.18 ± 0.230
Interlaced to parallel10/100/1010/1010/1011.07 ± 1.494.06 ± 0.360
Single vehicle to rear10/100/1010/1010/1015.18 ± 1.518.71 ± 0.530
Circle: sides fixed10/100/1010/1010/1024.54 ± 5.322.38 ± 0.130
Circle: middle reference10/100/1010/1010/1017.12 ± 2.384.39 ± 0.210

All recorded generations pass validation, and all evaluated executions reach the completion threshold. Mean generation times span 11.07–24.54 s; mean completion times span 1.76–8.71 s.

Generation success

All ten trials pass the stated checks in each condition.Click to enlarge
All ten trials pass the stated checks in each condition.

Optimal outputs

90 of 100 outputs are optimal under the translation-first priority and lane-change criterion; validity and optimality are evaluated separately.Click to enlarge
90 of 100 outputs are optimal under the translation-first priority and lane-change criterion; validity and optimality are evaluated separately.

Generation time

Individual measurements, means, and standard deviations for the ten generation trials per condition.Click to enlarge
Individual measurements, means, and standard deviations for the ten generation trials per condition.

Completion time

Time from formation dispatch to convergence in the execution experiments.Click to enlarge
Time from formation dispatch to convergence in the execution experiments.

3. Completion time and lane-change demand

The slower lane-reduction trials correspond exactly to non-optimal outputs: one trial in condition 1 and two in condition 2 require four lane changes instead of three. They take 8.89–9.94 s, compared with 4.50–5.58 s for optimal outputs. For the Lane 2 accident, seven non-optimal outputs require six lane changes and take 1.84–2.40 s; three optimal outputs require five and take about 1.69 s.

Together, these results demonstrate GRF-TAFC's on-the-fly adaptation to different instructions and road configurations.

Reflection and Experience Transfer

We compare repeated formation generation with and without reflection, then evaluate whether experience from one ambulance-yielding scenario can guide generation in a different but similar scenario.

Original scenario

Seven CAVs across three lanes. The ambulance approaches in Lane 3; Pattern B minimizes lane changes. Click to enlarge
7 CAVs · 3 lanes · Ambulance in the lowermost lane

Transfer scenario

Ten CAVs across four lanes. The ambulance approaches in Lane 1; Pattern D allocates the vehicles as 4–3–3 over Lanes 2–4. Click to enlarge
10 CAVs · 4 lanes · Ambulance in the uppermost lane

Repeated generation: with and without reflection

In the original scenario, Kimi K3 and Claude Sonnet 5 initially generate Pattern A, requiring four lane changes. Both retain this non-optimal formation in a second attempt without reflection. With reflection enabled after the first attempt, both generate the optimal Pattern B, requiring three lane changes. This comparison supports reflection's contribution to formation quality.

Transferring the lane-change-minimization principle

The original formation has a 2–3–2 distribution across three lanes. The transfer scenario increases the fleet to ten CAVs, initially distributed as 2–3–2–3 across four lanes, and moves the approaching ambulance from the lowermost to the uppermost lane. Candidate distributions over the remaining Lanes 2–4 include 4–3–3 (Pattern D), 3–4–3 (Pattern E), and 3–3–4. They share the same task of clearing the ambulance's lane but entail different numbers of lane changes.

Using experience from the original scenario, both Kimi K3 and Claude Sonnet 5 generate Pattern D (4–3–3), requiring the minimum of three lane changes. Their generation times are 41.36 s and 35.59 s, respectively. An excerpt from the generated reasoning explicitly identifies the retrieved experience:

Experience Memory (Lane-Change Minimization): Similar to the retrieved experience of reducing three lanes to two for a 2-3-2 distribution, the strategy here is to allocate the CAVs in a manner that closely mirrors the existing lane-wise distribution to strictly minimize the total number of lane changes.
Retrieved experience

Key: Reduction of available lanes from three to two for a seven-CAV formation with a 2–3–2 lane-wise distribution.

Value: Place four target positions in Lane 2 and three in Lane 1, rather than the reverse, because this reduces the lane-change count from four to three.

These outputs provide evidence that the stored experience conveys a reusable decision principle and can transfer to this related scenario despite changes in fleet size, road configuration, and ambulance approach lane.

Formation generation results

Model comparison, reflection ablations, and experience transfer; Medium reasoning effort
MLLM and configurationGenerated formationLane-change countGeneration time (s)
Original scenario: 7 CAVs, ambulance in Lane 3
Grok 4.6Pattern Boptimal344.75
Kimi K3Pattern Anon-optimal424.14
Kimi K3(second turn, without reflection)Pattern Anon-optimal412.05
Kimi K3(second turn, with reflection)Pattern Boptimal315.45
Claude Sonnet 5Pattern Anon-optimal414.09
Claude Sonnet 5(second turn, without reflection)Pattern Anon-optimal46.02
Claude Sonnet 5(second turn, with reflection)Pattern Boptimal37.97
DeepSeek V4 ProPattern Boptimal3128.85
Qwen3.8-MaxPattern Boptimal341.62
GPT-5.6 SolPattern Boptimal316.48
Gemini 3.1 Pro(without tools)Pattern Cinfeasible/50.72
Gemini 3.1 Pro(without knowledge)Pattern Boptimal332.04
Gemini 3.1 ProPattern Boptimal336.77
Transfer scenario: 10 CAVs, ambulance in Lane 1
Kimi K3(transfer)Pattern Doptimal341.36
Claude Sonnet 5(transfer)Pattern Doptimal335.59

Pattern C places vehicles in Lanes 2–3 and is infeasible because it fails to clear Lane 3 for the ambulance in the original scenario. Pattern D is the optimal 4–3–3 formation in the transfer scenario shown above. A non-optimal formation is not necessarily infeasible. “/” indicates that no lane-change count is reported for the infeasible output.

Prompt Example

1. Context-Grounded Formation Generation

Complete prompt example for context-grounded formation generation

2. Execution-Driven Reflection

Complete prompt example for execution-driven reflection

Implementation and Response Time

Multimodal input. The prompt composer sends the task prompt together with recorded WAV audio or the specified image to the MLLM. In the structure-switching demonstration, the operator presses R to start recording and types T followed by Enter to stop recording. The shape demonstration supplies an image of a black circle alongside the text request.

Validation and fallback. Prompt-guided self-checking helps the MLLM refine candidates. After generation, a separate rule-based parser checks output tags, the vehicle count, finite two-dimensional coordinates, usable road regions and lanes, target separation, and obstacle clearance. Relative longitudinal coordinates are scaled by half the desired within-lane following distance. Rejected outputs trigger regeneration; after three consecutive invalid outputs, a rule-based interlaced formation is generated and checked. If it remains infeasible, the system initiates a safe stop. Each CAV also has independently operating automatic emergency braking. Formation-level validation is distinct from the downstream trajectory optimizer's treatment of dynamic feasibility.

Experience retrieval and memory updates

Keys describe applicable task contexts; values record formation adjustments and their rationale. The retrieval configuration uses multilingual-e5-base with query: and passage: prefixes for context queries and stored keys, respectively. Their normalized embeddings are ranked by cosine similarity, and the top three associated values are retrieved (or all available entries if fewer than three exist), without a fixed similarity cutoff. Domain knowledge takes precedence over conflicting experience. Before writing a reflection insight, similar keys are retrieved and their values checked for substantive duplication; redundant insights are not stored again.

Lead time and limitations. Normal operation requires timely task issuance, prompt MLLM responses, and reliable cloud–CAV communication. Combining the reported 36.77 s generation time with the 5.52 s formation-switching time gives an estimated 42.3 s from request to completion. At an ambulance speed of 20 m/s, this corresponds to approximately 846 m before the intended yielding location. Operators can use connected ambulance-location data to issue the task in advance. This estimate is not a worst-case latency bound: delayed generation or communication increases the required lead time and may prevent timely completion of tasks with stringent real-time requirements.

Abstract

Formation control of connected and automated vehicles (CAVs) has shown great potential for improving traffic efficiency and safety. However, most existing studies focus on tracking predefined formations for fixed tasks, offering limited adaptability to changing task requirements. To address this limitation, we propose GRF-TAFC, a Grounded and Reflective Framework for Task-Adaptive Formation Choreography of CAVs. GRF-TAFC integrates context-grounded formation generation, optimization-based execution, and execution-driven reflection into a closed loop. A formation generator elicits the formation generation capabilities of pretrained MLLMs and grounds their reasoning over unstructured task inputs in real-time tool outputs, domain knowledge, and experience memory, yielding formations tailored to current task requirements while respecting environmental constraints. An execution module realizes the generated formations through polynomial-time CAV-to-target assignment and joint spatio-temporal trajectory optimization, facilitating collision avoidance and safe execution. A reflection module reasons over task context and actual execution outcomes, distilling reusable experience for iterative refinement. Real-world experiments across dynamic tasks and environments demonstrate GRF-TAFC’s effectiveness and practicality, with experimental videos available on the project website: https://dongjh20.github.io/GRF-TAFC.

GRF-TAFC Framework

Overview of the GRF-TAFC framework

Scenario