How to submit
.zip. The lab unpacks it into
submissions/<team_name>/, builds your environment, and runs
tools/run_custom_eval.py across the three benchmark buckets. Scoring is per-bucket
and overall Driving Score (DS) and Success Rate (SR).
ZIP structure
team_name.zip
├── agents.py # AutonomousAgent subclass
├── Dockerfile # one of Dockerfile (preferred) or model_env.yaml
├── model_env.yaml # conda env spec, used when Dockerfile is absent
├── config/
│ └── submission_config.yaml # whatever your setup() loads
├── src/ # additional Python modules your agent imports
├── weights/ # bundled checkpoints, or a download link in README.md
└── README.md # external services, port assumptions, anything env can't express
Paths inside submission_config.yaml should be relative to the submission
directory or absolute under it. The lab invokes your planner with
--agent submissions/<team_name>/agents.py --agent-config submissions/<team_name>/config/submission_config.yaml.
Agent contract
agents.py must subclass
leaderboard.autoagents.autonomous_agent.AutonomousAgent and define a
module-level get_entry_point() returning the class name.
setup(path_to_conf_file, ego_vehicles_num=1)— load your config and weights.ego_vehicles_numis the number of egos in the current scenario.sensors()— return the CARLA leaderboard sensor list. IDs must be unique; the framework appends_<ego_index>per ego, so read each ego's data viainput_data[f"{id}_{i}"].run_step(input_data, timestamp)— returnlist[carla.VehicleControl]of lengthego_vehicles_num.destroy()— release any acquired resources.
Multi-ego scenarios
Scenarios can have more than one ego vehicle. run_step() must always return a
list[carla.VehicleControl], one per ego, even when ego_vehicles_num == 1.
If your model handles only a single ego, pad the remainder with neutral controls:
def run_step(self, input_data, timestamp):
primary = self._compute_control(input_data)
controls = [primary]
for _ in range(self.ego_vehicles_num - 1):
controls.append(carla.VehicleControl()) # neutral
return controls
Returning a single VehicleControl (or a list of the wrong length) errors the
scenario out and counts as a route failure.
Environment
Submit either a Dockerfile or a model_env.yaml (or both). When
both are present, the lab uses the Dockerfile.
Option A: Docker (preferred)
Provide a Dockerfile that builds a self-contained image containing your
agent, weights, configs, and all dependencies. The lab builds the image with
docker build -t mdrive_<team_name> . and invokes
tools/run_custom_eval.py inside the running container with the host's CARLA
installation mounted at /workspace/carla_root. Your image needs Python 3.7
(the CARLA 0.9.12 PythonAPI egg is compiled for 3.7) and the CARLA PythonAPI on
PYTHONPATH
(/workspace/carla_root/PythonAPI/carla/dist/carla-0.9.12-py3.7-linux-x86_64.egg).
Option B: Conda
Provide model_env.yaml. The lab runs
conda env create -f model_env.yaml -n mdrive_<team_name> and activates
it before launching run_custom_eval.py.
- Pin to Python 3.7 unless you have a reason not to. The CARLA 0.9.12 PythonAPI egg is Python 3.7.
- Do not include CARLA itself in the env; the lab injects
CARLA_ROOTand the PythonAPI egg intoPYTHONPATH.
Anything env-time setup can't express (extra pip install from git, custom CUDA
extensions, external service startup like a vLLM endpoint) goes in README.md
as numbered shell commands.
Weights
Either bundle under weights/, or include a download URL in
README.md (Google Drive, HuggingFace, S3). Prefer a link for checkpoints
over a few GB. If weights live behind auth, document the access path.
Example: a complete TCP-style submission
A full submission directory mirroring the built-in tcp planner. Every file
below is the minimum needed for a valid submission; copy this scaffold and swap in your
own model, config, weights, and environment.
my_tcp/
├── agents.py
├── Dockerfile
├── model_env.yaml
├── config/
│ └── submission_config.yaml
├── weights/
│ └── my_tcp.ckpt
└── README.md
agents.py
import carla, torch, yaml
from leaderboard.autoagents.autonomous_agent import AutonomousAgent, Track
def get_entry_point():
return "MyTcpAgent"
class MyTcpAgent(AutonomousAgent):
def setup(self, path_to_conf_file, ego_vehicles_num=1):
self.track = Track.SENSORS
self.ego_vehicles_num = int(ego_vehicles_num)
with open(path_to_conf_file) as f:
cfg = yaml.safe_load(f)
self.model = torch.load(cfg["checkpoint"]).cuda().eval()
def sensors(self):
return [
{"type": "sensor.camera.rgb", "id": "rgb_front",
"x": 1.3, "y": 0.0, "z": 2.3, "roll": 0.0, "pitch": 0.0, "yaw": 0.0,
"width": 800, "height": 600, "fov": 100},
{"type": "sensor.speedometer", "id": "speed", "reading_frequency": 20},
]
def run_step(self, input_data, timestamp):
# The framework suffixes each sensor id with `_<ego_index>`. A single-ego
# model reads ego 0 and pads neutral controls for any additional egos.
img = input_data["rgb_front_0"][1]
speed = input_data["speed_0"][1]["speed"]
with torch.no_grad():
throttle, steer, brake = self.model.predict(img, speed)
primary = carla.VehicleControl()
primary.throttle = float(throttle)
primary.steer = float(steer)
primary.brake = float(brake)
controls = [primary]
for _ in range(self.ego_vehicles_num - 1):
controls.append(carla.VehicleControl())
return controls
def destroy(self):
pass
config/submission_config.yaml
checkpoint: weights/my_tcp.ckpt
model_env.yaml (used when Dockerfile is absent)
name: my_tcp
channels:
- pytorch
- conda-forge
- defaults
dependencies:
- python=3.7
- cudatoolkit=11.3
- pytorch=1.10.1
- torchvision=0.11.2
- pip
- pip:
- pyyaml
- numpy
Dockerfile
Build on continuumio/miniconda3 and reuse the same model_env.yaml
so versions stay in sync between the Docker and Conda paths.
FROM continuumio/miniconda3
WORKDIR /workspace
COPY . /workspace/
RUN conda env create -f model_env.yaml -n my_tcp \
&& conda clean -afy
ENV PATH=/opt/conda/envs/my_tcp/bin:$PATH
ENV PYTHONPATH=/workspace/carla_root/PythonAPI/carla/dist/carla-0.9.12-py3.7-linux-x86_64.egg:/workspace
weights/my_tcp.ckpt
The trained model checkpoint. For TCP, this is the new.ckpt file from the
public TCP release. For checkpoints over a few GB, omit weights/ from the
ZIP and document the download URL in README.md instead.
README.md
# my_tcp submission
Single-camera imitation-learning planner based on TCP.
## Weights
Bundled at weights/my_tcp.ckpt. If you need to re-download, the original TCP
checkpoint is at:
https://drive.google.com/file/d/1D-10aMUAOPk1yiOr-PvSOJMS_xi_eR7U/view
## External services
None.
How your planner is run
For each of the three benchmark buckets:
python tools/run_custom_eval.py \
--agent submissions/<team_name>/agents.py \
--agent-config submissions/<team_name>/config/submission_config.yaml \
--routes-dir scenarioset/<bucket> \
--results-tag <team_name>_<bucket> \
--scenario-pool --start-carla --gpus 0
| Bucket | Scenarios | Description |
|---|---|---|
scenarioset/interaction | 112 | Agentic-generated interaction scenarios + NHTSA pre-crash |
scenarioset/interdrive | 46 | InterDrive multi-agent negotiation |
scenarioset/v2xpnp | 67 | Real2Sim from V2X-PnP logs |
We recommend running one scenario per bucket locally before zipping, to catch sensor or env issues early.
Scoring
Per-bucket DS and SR, plus HM = 2 * DS * SR / (DS + SR). Overall HM is the arithmetic mean of the three per-bucket HM values and is the leaderboard ranking metric. Per-scenario results are published alongside.