How to submit

Submit your planner as a single .zip. The lab unpacks it into submissions/<team_name>/, builds your environment, and runs tools/run_custom_eval.py across the three benchmark buckets. Scoring is per-bucket and overall Driving Score (DS) and Success Rate (SR).

ZIP structure

team_name.zip
├── agents.py                       # AutonomousAgent subclass
├── Dockerfile                      # one of Dockerfile (preferred) or model_env.yaml
├── model_env.yaml                  # conda env spec, used when Dockerfile is absent
├── config/
│   └── submission_config.yaml      # whatever your setup() loads
├── src/                            # additional Python modules your agent imports
├── weights/                        # bundled checkpoints, or a download link in README.md
└── README.md                       # external services, port assumptions, anything env can't express

Paths inside submission_config.yaml should be relative to the submission directory or absolute under it. The lab invokes your planner with --agent submissions/<team_name>/agents.py --agent-config submissions/<team_name>/config/submission_config.yaml.

Agent contract

agents.py must subclass leaderboard.autoagents.autonomous_agent.AutonomousAgent and define a module-level get_entry_point() returning the class name.

  • setup(path_to_conf_file, ego_vehicles_num=1) — load your config and weights. ego_vehicles_num is the number of egos in the current scenario.
  • sensors() — return the CARLA leaderboard sensor list. IDs must be unique; the framework appends _<ego_index> per ego, so read each ego's data via input_data[f"{id}_{i}"].
  • run_step(input_data, timestamp) — return list[carla.VehicleControl] of length ego_vehicles_num.
  • destroy() — release any acquired resources.

Multi-ego scenarios

Scenarios can have more than one ego vehicle. run_step() must always return a list[carla.VehicleControl], one per ego, even when ego_vehicles_num == 1. If your model handles only a single ego, pad the remainder with neutral controls:

def run_step(self, input_data, timestamp):
    primary = self._compute_control(input_data)
    controls = [primary]
    for _ in range(self.ego_vehicles_num - 1):
        controls.append(carla.VehicleControl())  # neutral
    return controls

Returning a single VehicleControl (or a list of the wrong length) errors the scenario out and counts as a route failure.

Environment

Submit either a Dockerfile or a model_env.yaml (or both). When both are present, the lab uses the Dockerfile.

Option A: Docker (preferred)

Provide a Dockerfile that builds a self-contained image containing your agent, weights, configs, and all dependencies. The lab builds the image with docker build -t mdrive_<team_name> . and invokes tools/run_custom_eval.py inside the running container with the host's CARLA installation mounted at /workspace/carla_root. Your image needs Python 3.7 (the CARLA 0.9.12 PythonAPI egg is compiled for 3.7) and the CARLA PythonAPI on PYTHONPATH (/workspace/carla_root/PythonAPI/carla/dist/carla-0.9.12-py3.7-linux-x86_64.egg).

Option B: Conda

Provide model_env.yaml. The lab runs conda env create -f model_env.yaml -n mdrive_<team_name> and activates it before launching run_custom_eval.py.

  • Pin to Python 3.7 unless you have a reason not to. The CARLA 0.9.12 PythonAPI egg is Python 3.7.
  • Do not include CARLA itself in the env; the lab injects CARLA_ROOT and the PythonAPI egg into PYTHONPATH.

Anything env-time setup can't express (extra pip install from git, custom CUDA extensions, external service startup like a vLLM endpoint) goes in README.md as numbered shell commands.

Weights

Either bundle under weights/, or include a download URL in README.md (Google Drive, HuggingFace, S3). Prefer a link for checkpoints over a few GB. If weights live behind auth, document the access path.

Example: a complete TCP-style submission

A full submission directory mirroring the built-in tcp planner. Every file below is the minimum needed for a valid submission; copy this scaffold and swap in your own model, config, weights, and environment.

my_tcp/
├── agents.py
├── Dockerfile
├── model_env.yaml
├── config/
│   └── submission_config.yaml
├── weights/
│   └── my_tcp.ckpt
└── README.md

agents.py

import carla, torch, yaml
from leaderboard.autoagents.autonomous_agent import AutonomousAgent, Track


def get_entry_point():
    return "MyTcpAgent"


class MyTcpAgent(AutonomousAgent):
    def setup(self, path_to_conf_file, ego_vehicles_num=1):
        self.track = Track.SENSORS
        self.ego_vehicles_num = int(ego_vehicles_num)
        with open(path_to_conf_file) as f:
            cfg = yaml.safe_load(f)
        self.model = torch.load(cfg["checkpoint"]).cuda().eval()

    def sensors(self):
        return [
            {"type": "sensor.camera.rgb", "id": "rgb_front",
             "x": 1.3, "y": 0.0, "z": 2.3, "roll": 0.0, "pitch": 0.0, "yaw": 0.0,
             "width": 800, "height": 600, "fov": 100},
            {"type": "sensor.speedometer", "id": "speed", "reading_frequency": 20},
        ]

    def run_step(self, input_data, timestamp):
        # The framework suffixes each sensor id with `_<ego_index>`. A single-ego
        # model reads ego 0 and pads neutral controls for any additional egos.
        img = input_data["rgb_front_0"][1]
        speed = input_data["speed_0"][1]["speed"]
        with torch.no_grad():
            throttle, steer, brake = self.model.predict(img, speed)

        primary = carla.VehicleControl()
        primary.throttle = float(throttle)
        primary.steer = float(steer)
        primary.brake = float(brake)

        controls = [primary]
        for _ in range(self.ego_vehicles_num - 1):
            controls.append(carla.VehicleControl())
        return controls

    def destroy(self):
        pass

config/submission_config.yaml

checkpoint: weights/my_tcp.ckpt

model_env.yaml (used when Dockerfile is absent)

name: my_tcp
channels:
  - pytorch
  - conda-forge
  - defaults
dependencies:
  - python=3.7
  - cudatoolkit=11.3
  - pytorch=1.10.1
  - torchvision=0.11.2
  - pip
  - pip:
      - pyyaml
      - numpy

Dockerfile

Build on continuumio/miniconda3 and reuse the same model_env.yaml so versions stay in sync between the Docker and Conda paths.

FROM continuumio/miniconda3

WORKDIR /workspace
COPY . /workspace/

RUN conda env create -f model_env.yaml -n my_tcp \
    && conda clean -afy

ENV PATH=/opt/conda/envs/my_tcp/bin:$PATH
ENV PYTHONPATH=/workspace/carla_root/PythonAPI/carla/dist/carla-0.9.12-py3.7-linux-x86_64.egg:/workspace

weights/my_tcp.ckpt

The trained model checkpoint. For TCP, this is the new.ckpt file from the public TCP release. For checkpoints over a few GB, omit weights/ from the ZIP and document the download URL in README.md instead.

README.md

# my_tcp submission

Single-camera imitation-learning planner based on TCP.

## Weights
Bundled at weights/my_tcp.ckpt. If you need to re-download, the original TCP
checkpoint is at:
https://drive.google.com/file/d/1D-10aMUAOPk1yiOr-PvSOJMS_xi_eR7U/view

## External services
None.

How your planner is run

For each of the three benchmark buckets:

python tools/run_custom_eval.py \
  --agent submissions/<team_name>/agents.py \
  --agent-config submissions/<team_name>/config/submission_config.yaml \
  --routes-dir scenarioset/<bucket> \
  --results-tag <team_name>_<bucket> \
  --scenario-pool --start-carla --gpus 0
BucketScenariosDescription
scenarioset/interaction112Agentic-generated interaction scenarios + NHTSA pre-crash
scenarioset/interdrive46InterDrive multi-agent negotiation
scenarioset/v2xpnp67Real2Sim from V2X-PnP logs

We recommend running one scenario per bucket locally before zipping, to catch sensor or env issues early.

Scoring

Per-bucket DS and SR, plus HM = 2 * DS * SR / (DS + SR). Overall HM is the arithmetic mean of the three per-bucket HM values and is the leaderboard ranking metric. Per-scenario results are published alongside.