{
 "repo": "huggingface/trl-recipies",
 "branch": "main",
 "generated_at": "2026-11-19 11:10 UTC",
 "projects": [
  {
   "id": "AsyncGRPOTrainer/harness-adapters",
   "trainer": "AsyncGRPOTrainer",
   "name": "harness-adapters",
   "readme": "# Harness adapters\n\nSmall LoRA adapters trained with RL inside unmodified agent harnesses (pi, hermes, opencode): one adapter per model\nand harness, all served from one base model. A tool call is checked by\n[safeagents](https://github.com/huggingface/safeagents) before it runs. Blog post:\n[harness-hub/harness-adapters](https://huggingface.co/spaces/harness-hub/harness-adapters). Live training dashboard:\n[harness-adapters-rl](https://harness-hub-harness-adapters-rl.hf.space/).\n\n| Model | Steps | Training tasks | HF Jobs | Recipe |\n|---|--:|---|---|---|\n| `LiquidAI/LFM2.5-24B-A2B` | 200 | 100 medium + 200 hard SmolDataEnvs, 230 traj2envs | h200x4 | [LFM2.5-24B-A2B/](LFM2.5-24B-A2B/) |\n| `LiquidAI/LFM3-1.7B` | 400 | 300 medium + 250 hard SmolDataEnvs, 230 traj2envs | h200x2 | [LFM3-1.7B/](LFM3-1.7B/) |\n| `webAI-Official/TwIL-8B` (experiment) | 100 | 150 medium + 250 hard SmolDataEnvs, 230 traj2envs | h200x4 | [TwIL-8B/](TwIL-8B/) |\n| `Qwen/Qwen3.8-27B` (training now) | 1000 | 750 scale800, 300 TermGrade | cluster: hopper-b200, 2 GPUs per harness | [Qwen3.8-27B/](Qwen3.8-27B/) |\n\nQwen3.8-27B runs on the cluster with its own training script, TRL branch and settings. Its README lists the\ndifferences; the rest of this page describes the LFM / TwIL runs.\n\n## How a run works\n\n`scripts/launcher_train.py` runs everything inside one job:\n- one `vllm serve --enable-lora` that serves the base model and every adapter version\n- one OpenEnv Harbor server per harness, with its sandboxes on HF sandboxes\n- one AsyncGRPO trainer per harness (`--run NAME:HARNESS:DIFFICULTY`) running `scripts/async_grpo_harbor.py`\n\nAfter each optimizer step, a trainer loads only its new adapter into vLLM, so the three harnesses share one server.\nFlags the launcher does not know are passed on to the training script.\n\nEvery model uses the same settings:\n- LoRA r=32 on all linear layers, learning rate 1e-5, 8 rollouts per task\n- 4 gradient-accumulation steps, rollouts at most 4 weight updates stale, temperature 0.8\n- up to 32,768 output tokens per turn\n- a checkpoint every 10 steps\n\nThe reward is the task's own verifier plus `0.1 \u00d7 (1 \u2212 calls / 15)`, given only to correct rollouts\n(`--tool-efficiency-weight 0.1`). safeagents runs with two atif-scan plugins: `scripts/atif_smoldataenvs.py` and\n`scripts/atif_traj2envs.py`. They block the agent from reading a task's answers at their source.\n\nEvaluation (`scripts/launcher_eval.py --eval-lora`) serves the base and every checkpoint on one vLLM and measures\npass@1 per harness on two held-out splits:\n- `final-heldout`: 150 easy and medium SmolDataEnvs tasks from index 1000\n- `traj2envs-heldout`: the 40 traj2envs tasks from index 230\n\n## Layout\n\n```\nscripts/                 shared by every model\n  launcher_train.py      HF Jobs / cluster entry point for training (PEP 723: `hf jobs uv run` installs its deps)\n  launcher_eval.py       same for evaluation\n  async_grpo_harbor.py   the training script (AsyncGRPOTrainer + LoRA + Harbor rollouts)\n  evaluate_harbor.py     the eval client the eval launcher runs per checkpoint\n  atif_*.py              safeagents plugins (atif_harbor.py: generic, used by Qwen3.8-27B)\n  eval_sweep.py          scheduled job that evaluated every checkpoint of the released runs as it landed\n  release_adapters.py    packages the 9 released adapters (+ GGUF) and writes their model cards\n  slurm_setup.sh         node setup sourced by every *_slurm.sh\n<model>/\n  recipe.yaml            metadata: hyperparameters, datasets, hardware, released checkpoints, TRL fork\n  trl.patch              the TRL fork's changes to trl/, relative to the upstream commit it branched from\n  train_local.sh  train_hf_jobs.sh  train_slurm.sh  eval_local.sh  eval_hf_jobs.sh  eval_slurm.sh\n```\n\n## HF Jobs\n\nYou need the `hf` CLI logged in, and a bucket the job mounts on `/outputs`. The default is `BUCKET=ybelkada/harbor-evals`.\n\n```sh\nBUCKET=<user>/<bucket> bash LFM3-1.7B/train_hf_jobs.sh\n```\n\nEach script uploads the files the job reads from the bucket (the training or eval script, the safeagents plugins, and\nTwIL's chat template), then submits the job detached. Runs are written to `/outputs/runs/<model>-<harness>-<job id>-<hex>/`\nand evals to `/outputs/eval/`.\n\nTo evaluate every new checkpoint of a run on a schedule, copy `scripts/launcher_eval.py` to the bucket and edit\n`MODELS` in `scripts/eval_sweep.py` to point at your job id:\n\n```sh\nhf jobs scheduled uv run \"0 */6 * * *\" --no-concurrency --flavor h200 --image huggingface/trl --secrets HF_TOKEN \\\n    --volume hf://buckets/<user>/<bucket>:/outputs scripts/eval_sweep.py -- LFM2.5-24B-A2B\n```\n\n**TRL version.** The released runs installed a TRL wheel from the bucket\n(`--trl-wheel /outputs/wheels/lfm3/trl-1.17.0.dev0-py3-none-any.whl`). The branch it was built from was not recorded.\nWithout `TRL_WHEEL`, the scripts use the launcher's pinned TRL (`younesbelkada/trl@add-lfm-tree-attn`, tree-attention\npacking for LFM). To use a wheel, set `TRL_WHEEL=/outputs/<path>.whl`.\n\n**TRL fork changes.** Each model folder has a `trl.patch`. It holds the diff of `trl/` between the TRL fork commit\nthat recipe installs and the upstream `huggingface/trl` commit the fork branched from. Both commits are recorded in\n`recipe.yaml` under `trl:` and in the patch header. To rebuild the same TRL from upstream, check out `upstream_base`\nand run `git apply trl.patch`. To regenerate the patch after the fork moves:\n\n```sh\ngit diff $(git merge-base origin/main <fork commit>) <fork commit> -- trl/\n```\n\n## Local machine\n\nThe `*_local.sh` scripts run the same launcher with the same flags on the machine you are on: `uv run` installs the\nlauncher's inline dependencies, and the training script, plugins and chat template are read from this folder instead of\na bucket. You need `uv`, `HF_TOKEN` (the HF sandboxes also run under it), and as many GPUs as the run uses: 4 for\nLFM2.5-24B-A2B and TwIL-8B, 2 for LFM3-1.7B and for each Qwen3.8-27B harness, all H200 class. Runs and evals go to\n`./outputs` (`DATA_ROOT`). These are **untested**: none of the runs was made this way.\n\n```sh\ncd LFM3-1.7B && MAX_STEPS=2 bash train_local.sh\n```\n\n## Cluster (SLURM)\n\nThe `*_slurm.sh` scripts run the same launcher with the same flags on one node. They are **untested**: the released\nadapters were all trained on HF Jobs. The node setup (`scripts/slurm_setup.sh`) comes from the Qwen3.8 TermGrade runs\non `hopper-b200`. Change the partition with `sbatch -p`.\n\nOne-time setup on a login node:\n\n```sh\nROOT=/fsx/$USER/harness-adapters\nmkdir -p $ROOT && rsync -a ./ $ROOT/\nuv venv $ROOT/venv --python 3.12 && source $ROOT/venv/bin/activate\nuv pip install \"trl @ git+https://github.com/younesbelkada/trl.git@add-lfm-tree-attn\" \\\n    \"safeagents@git+https://github.com/huggingface/safeagents@5e683fa\" atif-scan==0.8.15 rich markdown-it-py mdurl pygments\n```\n\nThen install the rest of `scripts/launcher_train.py`'s inline dependency list (the `# /// script` block at the top)\ninto the same venv. Ship the venv with `venv-ship $ROOT/venv`. Download the model into `$ROOT/hf-hub`\n(`HF_HUB_CACHE=$ROOT/hf-hub hf download <model>`) and ship it with `dir-ship $ROOT/hf-hub`.\n\nThen, from the model's folder:\n\n```sh\nsbatch train_slurm.sh\nCHECKPOINTS=\"LiquidAI/LFM3-1.7B@pi,hermes,opencode $ROOT/data/runs/<run>/checkpoint-400@pi\" sbatch eval_slurm.sh\n```\n\nRuns go to `$ROOT/data/runs/`, evals to `$ROOT/data/eval/`, and logs to `/fsx/$USER/logs/`.\n",
   "shared_files": [
    "async_grpo_harbor.py",
    "atif_harbor.py",
    "atif_smoldataenvs.py",
    "atif_traj2envs.py",
    "eval_sweep.py",
    "evaluate_harbor.py",
    "launcher_eval.py",
    "launcher_train.py",
    "release_adapters.py",
    "slurm_setup.sh",
    "sync_live.sh"
   ]
  },
  {
   "id": "SFTTrainer/sft-qlora",
   "trainer": "SFTTrainer",
   "name": "sft-qlora",
   "readme": "# SFT with QLoRA\n\nSupervised fine-tuning with QLoRA on one small GPU: the base model is loaded in 4 bits and only a LoRA adapter is\ntrained. The dataset is [HuggingFaceH4/Multilingual-Thinking](https://huggingface.co/datasets/HuggingFaceH4/Multilingual-Thinking),\nwhose reasoning traces are translated into several languages, so a reasoning model learns to think in the language the\nsystem prompt asks for. Ported from TRL's\n[`examples/sft_qlora/sft_qlora.ipynb`](https://github.com/huggingface/trl/blob/ebf09f6b7e126af6e34ff172bb74e1629ff24156/examples/sft_qlora/sft_qlora.ipynb),\nwhich runs on a free Colab T4.\n\n| Model | VRAM (notebook, T4) | Recipe |\n|---|---|---|\n| `Qwen/Qwen3-8B` | ~12.8 GB | [Qwen3-8B/](Qwen3-8B/) |\n| `meta-llama/Llama-3.2-3B-Instruct` (gated) | ~4.7 GB | [Llama-3.2-3B-Instruct/](Llama-3.2-3B-Instruct/) |\n| `LiquidAI/LFM2-2.6B` | ~5.9 GB | [LFM2-2.6B/](LFM2-2.6B/) |\n\n## Settings\n\nEvery model uses the same settings, which are the notebook's:\n- **Data:** the `messages` column. Each message's `thinking` field is folded into its content as `<think>\u2026</think>`,\n  since these models' chat templates have no separate thinking field.\n- **Model:** loaded in 4-bit NF4 with double quantization and float16 compute. Training uses float16 and SDPA, which is\n  what a T4 supports. On Ampere or newer GPUs, bfloat16 and Flash Attention are faster.\n- **LoRA:** r=32, alpha 32, on `q,k,v,o,gate,up,down_proj`.\n- **Training:**\n  - 30 steps: batch size 1 with 4 gradient-accumulation steps, learning rate 2e-4 with 5 warmup steps\n  - `paged_adamw_8bit`, sequences up to 1,024 tokens\n  - activation offloading to CPU, which is what fits 8B models in 16 GB\n- **TRL:** the release `trl[peft]==1.17.1` from PyPI, with no fork and no patch.\n\nThe adapter is pushed to `<your user>/<model>-SFT`, and the run is logged to the Trackio Space of the same name.\n\n## Scripts\n\n```\nscripts/sft_qlora.py             training (PEP 723: `uv run` installs its dependencies)\nscripts/compare_generations.py   the notebook's check: base model vs adapter on a prompt asking for German reasoning\n<model>/train_local.sh  train_hf_jobs.sh  train_slurm.sh  eval_local.sh  eval_hf_jobs.sh  eval_slurm.sh\n```\n\nOn a **local machine**, you need `uv` and one GPU with 16 GB or more. The scripts write the run to `./outputs`.\n\nOn **HF Jobs**, you need the `hf` CLI logged in. Each job runs on a `t4-small` with your `HF_TOKEN` as a secret.\n\nOn the **cluster**, submit from the model's folder: the job finds `../scripts/` from the directory it was submitted from.\nIt needs `uv` on the path. Weights and uv's package cache go to `/fsx/$USER`, and the run to `/fsx/$USER/sft-qlora/`.\n\nNone of these scripts has been run as written yet. The notebook they come from ran on Colab.\n",
   "shared_files": [
    "compare_generations.py",
    "sft_qlora.py"
   ]
  },
  {
   "id": "AsyncGRPOTrainer/openenv-coding-tasks",
   "trainer": "AsyncGRPOTrainer",
   "name": "openenv-coding-tasks",
   "title": "RL Environment Pipeline",
   "readme": "Build verifiable coding tasks and select examples that give the base model a mix of successful and unsuccessful rollouts.",
   "guide_url": "guides/openenv-coding-tasks/index.html"
  }
 ],
 "recipes": [
  {
   "id": "AsyncGRPOTrainer/harness-adapters/LFM2.5-24B-A2B",
   "trainer": "AsyncGRPOTrainer",
   "project": "harness-adapters",
   "name": "LFM2.5-24B-A2B",
   "org": "LiquidAI",
   "avatar": "https://cdn-avatars.huggingface.co/v1/production/uploads/61b8e2ba285851687028d395/EsTgVtnM2IqVRKgPdfqcB.png",
   "meta": {
    "title": "LFM2.5-24B-A2B harness adapters",
    "project": "harness-adapters",
    "trainer": "AsyncGRPOTrainer",
    "model_id": "LiquidAI/LFM2.5-24B-A2B",
    "description": "One LoRA adapter per agent harness (pi, hermes, opencode), trained with async GRPO inside the unmodified harness on verifiable Harbor tasks, with safeagents checking every tool call.",
    "tags": [
     "lora",
     "agentic-rl",
     "harbor",
     "safeagents",
     "multi-harness"
    ],
    "harnesses": [
     "pi",
     "hermes",
     "opencode"
    ],
    "date_added": "2026-11-13",
    "blog": "https://huggingface.co/spaces/harness-hub/harness-adapters",
    "dashboard": "https://harness-hub-harness-adapters-rl.hf.space/"
   },
   "hardware": {
    "hf_jobs": "h200x4",
    "slurm": "hopper-b200, 4 GPUs (untested)",
    "local": "4 GPUs, H200 class (untested)"
   },
   "datasets": {
    "train": [
     "FineEnvs/SmolDataEnvs-harbor-train",
     "harness-hub/traj2envs-rl@191a751bda511eada76f6a9a76513d143321a64a"
    ],
    "eval": {
     "final-heldout": "FineEnvs/SmolDataEnvs-harbor-train, 150 easy/medium tasks from index 1000",
     "traj2envs-heldout": "harness-hub/traj2envs-rl, the 40 tasks from index 230"
    }
   },
   "hyperparameters": {
    "lora_r": 32,
    "lora_target_modules": "all-linear",
    "learning_rate": 1e-05,
    "num_generations": 8,
    "gradient_accumulation_steps": 4,
    "max_staleness": 4,
    "temperature": 0.8,
    "max_completion_length": 16384,
    "max_output_tokens": 32768,
    "max_model_len": 131072,
    "agent_step_limit": 17,
    "agent_timeout_s": 600,
    "max_inflight": 48,
    "tool_efficiency_weight": 0.1,
    "max_steps": 200,
    "save_steps": 10
   },
   "vllm": {
    "tool_call_parser": "lfm2",
    "reasoning_parser": "qwen3"
   },
   "trl": {
    "fork": "younesbelkada/trl@ee6e9ad9512b83c80bb0051942770c8798c0dbf2 (add-lfm-tree-attn)",
    "upstream_base": "7fdad81d04303e6f43db168e903565769c0b8642",
    "patch": "trl.patch"
   },
   "provenance": {
    "hf_job": "6aedb47a0e0338d1bb8107e8",
    "released": {
     "pi": {
      "step": 170,
      "repo": "harness-hub/LFM2.5-24B-A2B-pi-v2"
     },
     "hermes": {
      "step": 50,
      "repo": "harness-hub/LFM2.5-24B-A2B-hermes-v2"
     },
     "opencode": {
      "step": 160,
      "repo": "harness-hub/LFM2.5-24B-A2B-opencode"
     }
    }
   },
   "scripts": {
    "train": {
     "hf_jobs": "train_hf_jobs.sh",
     "slurm": "train_slurm.sh",
     "local": "train_local.sh"
    },
    "eval": {
     "hf_jobs": "eval_hf_jobs.sh",
     "slurm": "eval_slurm.sh",
     "local": "eval_local.sh"
    }
   },
   "builder": {
    "verified": {
     "hf_jobs": true,
     "slurm": false,
     "local": false
    },
    "eval": [
     {
      "env": "SPLIT",
      "label": "Held-out split",
      "choices": [
       "final-heldout",
       "traj2envs-heldout"
      ]
     },
     {
      "env": "CHECKPOINTS",
      "label": "Checkpoints",
      "input": true,
      "placeholder": "<path|glob|hub id>@<harnesses> ... (empty: the script's default)"
     }
    ]
   },
   "readme": "# LFM2.5-24B-A2B harness adapters\n\nLoRA adapters for pi, hermes and opencode on [`LiquidAI/LFM2.5-24B-A2B`](https://huggingface.co/LiquidAI/LFM2.5-24B-A2B),\ntrained for 200 steps on 300 SmolDataEnvs tasks (100 medium, 200 hard) and 230 traj2envs tasks. Setup and shared\nsettings: [../README.md](../README.md). Full hyperparameters: [recipe.yaml](recipe.yaml).\n\n| | HF Jobs | Cluster (untested) | Local (untested) |\n|---|---|---|---|\n| Train | `bash train_hf_jobs.sh` (h200x4, ~36h) | `sbatch train_slurm.sh` (4 GPUs) | `bash train_local.sh` (4 GPUs) |\n| Eval | `bash eval_hf_jobs.sh` (h200) | `CHECKPOINTS=... sbatch eval_slurm.sh` (1 GPU) | `bash eval_local.sh` (1 GPU) |\n\nModel-specific flags:\n- `--tool-call-parser lfm2 --reasoning-parser qwen3`\n- vLLM on GPU 0 and one trainer per harness on GPUs 1, 2 and 3 (`--train-device 1,2,3`)\n\nReleased checkpoints (job `6aedb47a0e0338d1bb8107e8`):\n\n| Harness | Step | Repo |\n|---|--:|---|\n| pi | 170 | `harness-hub/LFM2.5-24B-A2B-pi-v2` |\n| hermes | 50 | `harness-hub/LFM2.5-24B-A2B-hermes-v2` |\n| opencode | 160 | `harness-hub/LFM2.5-24B-A2B-opencode` |\n",
   "files": {
    "train_local.sh": "#!/bin/bash\n# LFM2.5-24B-A2B harness adapters on one local machine with 4 GPUs:\n# vLLM on GPU 0, one trainer per harness on GPUs 1, 2 and 3. The released run used H200s (141 GB).\n#\n# Same launcher and training flags as train_hf_jobs.sh, run here with `uv run` instead of on HF Jobs. Not run locally\n# yet. Needs uv, 4 GPUs and HF_TOKEN (also what the HF sandboxes run under). Runs are written to $DATA_ROOT/runs/.\n#\n#   bash train_local.sh\n#   MAX_STEPS=2 bash train_local.sh                   short smoke run\nset -euo pipefail\n\nMAX_STEPS=\"${MAX_STEPS:-200}\"\nDATA_ROOT=\"${DATA_ROOT:-$PWD/outputs}\"\nTRL_WHEEL=\"${TRL_WHEEL:-}\"\nTRACKIO_SPACE_ID=\"${TRACKIO_SPACE_ID:-ybelkada/async-grpo-harbor}\"\nREV=191a751bda511eada76f6a9a76513d143321a64a  # harness-hub/traj2envs-rl revision the adapters were trained on\nHERE=\"$(cd \"$(dirname \"$0\")\" && pwd)\"\nSCRIPTS=\"$HERE/../scripts\"\nexport HF_TOKEN=\"${HF_TOKEN:-$(cat ~/.cache/huggingface/token)}\"\n# Names this run's folders (logs/<id>, runs/<model>-<harness>-<id>-...), as HF_JOB_ID does on HF Jobs\nexport JOB_ID=\"${JOB_ID:-local-$(date +%Y%m%d-%H%M%S)}\"\nexport PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True\n\nTRL_ARGS=()\nif [ -n \"$TRL_WHEEL\" ]; then TRL_ARGS=(--trl-wheel \"$TRL_WHEEL\"); fi\n\nuv run \"$SCRIPTS/launcher_train.py\" \\\n    --data-root \"$DATA_ROOT\" ${TRL_ARGS[@]+\"${TRL_ARGS[@]}\"} \\\n    --train-script-url \"file://$SCRIPTS/async_grpo_harbor.py\" --expose gradio \\\n    --pip-install \"safeagents@git+https://github.com/huggingface/safeagents@5e683fa atif-scan==0.8.15 rich markdown-it-py mdurl pygments\" \\\n    --model LiquidAI/LFM2.5-24B-A2B \\\n    --tool-call-parser lfm2 --reasoning-parser qwen3 --max-model-len 131072 \\\n    --run pi:pi:medium:100,hard:200 \\\n    --run hermes:hermes:medium:100,hard:200 \\\n    --run opencode:opencode:medium:100,hard:200 \\\n    --split FineEnvs/SmolDataEnvs-harbor-train --n-tasks 300 \\\n    --mix-split \"harness-hub/traj2envs-rl@$REV:230\" --max-output-tokens 32768 \\\n    --sandbox hf-sandbox --docker-image docker.io/ybelkada/env-data-agent-train:opencode-1.19.5 \\\n    --vllm-device 0 --train-device 1,2,3 \\\n    --max-inflight 48 --max-lora-rank 32 --max-loras 18 \\\n    --reward-key correctness,reward --num-generations 8 --gradient-accumulation-steps 4 --max-staleness 4 \\\n    --max-completion-length 16384 --agent-step-limit 17 --agent-timeout 600 --temperature 0.8 \\\n    --lora-r 32 --lora-target-modules all-linear --learning-rate 1e-5 --tool-efficiency-weight 0.1 \\\n    --safeagents --safeagents-plugin \"$SCRIPTS/atif_smoldataenvs.py:checks\" --safeagents-plugin \"$SCRIPTS/atif_traj2envs.py:checks\" \\\n    --max-steps \"$MAX_STEPS\" --save-steps 10 --save-total-limit 0 \\\n    --trackio-space-id \"$TRACKIO_SPACE_ID\"\n",
    "train_hf_jobs.sh": "#!/bin/bash\n# LFM2.5-24B-A2B harness adapters on HF Jobs: one vLLM (GPU 0) serving the base and every adapter version, and one LoRA\n# AsyncGRPO trainer per harness (pi, hermes, opencode on GPUs 1, 2, 3), on 4x H200.\n#\n# This is the command of the released run (HF job 6aedb47a0e0338d1bb8107e8, 2026-11-06), with the bucket as a variable.\n#\n#   bash train_hf_jobs.sh\n#   BUCKET=<user>/<bucket> MAX_STEPS=2 bash train_hf_jobs.sh          short smoke run\n#   TRL_WHEEL=/outputs/wheels/trl-....whl bash train_hf_jobs.sh       a TRL wheel already on the bucket, over the\n#                                                                     launcher's pinned TRL\nset -euo pipefail\n\nBUCKET=\"${BUCKET:-ybelkada/harbor-evals}\"\nMAX_STEPS=\"${MAX_STEPS:-200}\"\nTRL_WHEEL=\"${TRL_WHEEL:-}\"\nTRACKIO_SPACE_ID=\"${TRACKIO_SPACE_ID:-ybelkada/async-grpo-harbor}\"\nREV=191a751bda511eada76f6a9a76513d143321a64a  # harness-hub/traj2envs-rl revision the adapters were trained on\nSCRIPTS=\"$(cd \"$(dirname \"$0\")/../scripts\" && pwd)\"\n\n# The job reads the training script and the safeagents plugins from the bucket, mounted on /outputs\nfor f in async_grpo_harbor.py atif_smoldataenvs.py atif_traj2envs.py; do\n    hf buckets cp \"$SCRIPTS/$f\" \"hf://buckets/$BUCKET/$f\"\ndone\n\nTRL_ARGS=()\nif [ -n \"$TRL_WHEEL\" ]; then TRL_ARGS=(--trl-wheel \"$TRL_WHEEL\"); fi\n\nhf jobs uv run --detach \\\n    --flavor h200x4 --image huggingface/trl --secrets HF_TOKEN --timeout 36h \\\n    --env PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True \\\n    --volume \"hf://buckets/$BUCKET:/outputs\" \\\n    \"$SCRIPTS/launcher_train.py\" -- \\\n    --data-root /outputs ${TRL_ARGS[@]+\"${TRL_ARGS[@]}\"} \\\n    --pip-install \"safeagents@git+https://github.com/huggingface/safeagents@5e683fa atif-scan==0.8.15 rich markdown-it-py mdurl pygments\" \\\n    --model LiquidAI/LFM2.5-24B-A2B \\\n    --tool-call-parser lfm2 --reasoning-parser qwen3 --max-model-len 131072 \\\n    --run pi:pi:medium:100,hard:200 \\\n    --run hermes:hermes:medium:100,hard:200 \\\n    --run opencode:opencode:medium:100,hard:200 \\\n    --split FineEnvs/SmolDataEnvs-harbor-train --n-tasks 300 \\\n    --mix-split \"harness-hub/traj2envs-rl@$REV:230\" --max-output-tokens 32768 \\\n    --sandbox hf-sandbox --docker-image docker.io/ybelkada/env-data-agent-train:opencode-1.19.5 \\\n    --vllm-device 0 --train-device 1,2,3 \\\n    --max-inflight 48 --max-lora-rank 32 --max-loras 18 \\\n    --reward-key correctness,reward --num-generations 8 --gradient-accumulation-steps 4 --max-staleness 4 \\\n    --max-completion-length 16384 --agent-step-limit 17 --agent-timeout 600 --temperature 0.8 \\\n    --lora-r 32 --lora-target-modules all-linear --learning-rate 1e-5 --tool-efficiency-weight 0.1 \\\n    --safeagents --safeagents-plugin /outputs/atif_smoldataenvs.py:checks --safeagents-plugin /outputs/atif_traj2envs.py:checks \\\n    --max-steps \"$MAX_STEPS\" --save-steps 10 --save-total-limit 0 \\\n    --trackio-space-id \"$TRACKIO_SPACE_ID\"\n",
    "train_slurm.sh": "#!/bin/bash -l\n# LFM2.5-24B-A2B harness adapters on one cluster node: one vLLM (GPU 0) serving the base and every adapter version, and\n# one LoRA AsyncGRPO trainer per harness (pi, hermes, opencode on GPUs 1, 2, 3).\n#\n# Same launcher and training flags as train_hf_jobs.sh (the released run). Not run on the cluster yet: the released\n# adapters were trained on HF Jobs. One-time setup on a login node: see ../README.md.\n#\n#   sbatch train_slurm.sh\n#   MAX_STEPS=2 sbatch -p hopper-prod train_slurm.sh\n#SBATCH --job-name=ha-lfm2.5\n#SBATCH --partition=hopper-b200\n#SBATCH --nodes=1\n#SBATCH --ntasks-per-node=1\n#SBATCH --gpus-per-node=4\n#SBATCH --cpus-per-task=64\n#SBATCH --time=2-00:00:00\n#SBATCH --output=/fsx/%u/logs/%x-%j.out\nset -euo pipefail\n\nROOT=\"${ROOT:-/fsx/$USER/harness-adapters}\"\nMAX_STEPS=\"${MAX_STEPS:-200}\"\nREV=191a751bda511eada76f6a9a76513d143321a64a  # harness-hub/traj2envs-rl revision the adapters were trained on\nsource \"$ROOT/scripts/slurm_setup.sh\"\n\npython \"$ROOT/scripts/launcher_train.py\" \\\n    --data-root \"$ROOT/data\" \\\n    --train-script-url \"file://$ROOT/scripts/async_grpo_harbor.py\" \\\n    --vllm-port \"$VLLM_PORT\" --server-port \"$SERVER_PORT\" --capture-port \"$CAPTURE_PORT\" --expose gradio \\\n    --model LiquidAI/LFM2.5-24B-A2B \\\n    --tool-call-parser lfm2 --reasoning-parser qwen3 --max-model-len 131072 \\\n    --run pi:pi:medium:100,hard:200 \\\n    --run hermes:hermes:medium:100,hard:200 \\\n    --run opencode:opencode:medium:100,hard:200 \\\n    --split FineEnvs/SmolDataEnvs-harbor-train --n-tasks 300 \\\n    --mix-split \"harness-hub/traj2envs-rl@$REV:230\" --max-output-tokens 32768 \\\n    --sandbox hf-sandbox --docker-image docker.io/ybelkada/env-data-agent-train:opencode-1.19.5 \\\n    --vllm-device 0 --train-device 1,2,3 \\\n    --max-inflight 48 --max-lora-rank 32 --max-loras 18 \\\n    --reward-key correctness,reward --num-generations 8 --gradient-accumulation-steps 4 --max-staleness 4 \\\n    --max-completion-length 16384 --agent-step-limit 17 --agent-timeout 600 --temperature 0.8 \\\n    --lora-r 32 --lora-target-modules all-linear --learning-rate 1e-5 --tool-efficiency-weight 0.1 \\\n    --safeagents --safeagents-plugin \"$ROOT/scripts/atif_smoldataenvs.py:checks\" \\\n    --safeagents-plugin \"$ROOT/scripts/atif_traj2envs.py:checks\" \\\n    --max-steps \"$MAX_STEPS\" --save-steps 10 --save-total-limit 0\n",
    "eval_local.sh": "#!/bin/bash\n# Held-out pass@1 of LFM2.5-24B-A2B and its harness adapters on one local GPU. One vLLM serves the base and every\n# checkpoint as an adapter, with one sandbox server per checkpoint. Results land in $DATA_ROOT/eval/<label>_<split>/.\n#\n# Same flags as eval_hf_jobs.sh, run here with `uv run`. Not run locally yet.\n#\n#   bash eval_local.sh                                                         the base model\n#   CHECKPOINTS=\"LiquidAI/LFM2.5-24B-A2B@pi,hermes,opencode $PWD/outputs/runs/<run>/checkpoint-50@pi\" bash eval_local.sh\n#   SPLIT=traj2envs-heldout bash eval_local.sh\nset -euo pipefail\n\nDATA_ROOT=\"${DATA_ROOT:-$PWD/outputs}\"\nSPLIT=\"${SPLIT:-final-heldout}\"  # final-heldout | traj2envs-heldout\n# <path|glob|hub id>@<harnesses>, space-separated\nCHECKPOINTS=\"${CHECKPOINTS:-LiquidAI/LFM2.5-24B-A2B@pi,hermes,opencode}\"\nHERE=\"$(cd \"$(dirname \"$0\")\" && pwd)\"\nSCRIPTS=\"$HERE/../scripts\"\nexport HF_TOKEN=\"${HF_TOKEN:-$(cat ~/.cache/huggingface/token)}\"\nexport JOB_ID=\"${JOB_ID:-local-$(date +%Y%m%d-%H%M%S)}\"\n\nif [ \"$SPLIT\" = final-heldout ]; then\n    SPLIT_ARGS=(--split FineEnvs/SmolDataEnvs-harbor-train --task-offset 1000 --difficulty easy,medium --n-tasks 150\n        --docker-image docker.io/ybelkada/env-data-agent-train:opencode-1.19.5)\nelse\n    SPLIT_ARGS=(--split harness-hub/traj2envs-rl@191a751bda511eada76f6a9a76513d143321a64a --task-offset 230 --n-tasks 40)\nfi\nread -r -a CHECKPOINT_LIST <<< \"$CHECKPOINTS\"\n\nuv run \"$SCRIPTS/launcher_eval.py\" \\\n    --data-root \"$DATA_ROOT\" --eval-script-url \"file://$SCRIPTS/evaluate_harbor.py\" \\\n    --eval-lora --thinking --sandbox hf-sandbox --expose gradio \\\n    --model LiquidAI/LFM2.5-24B-A2B --max-model-len 131072 --eval-tag \"$SPLIT\" \\\n    --eval-harnesses pi,hermes,opencode \\\n    --eval-checkpoints \"${CHECKPOINT_LIST[@]}\" \\\n    --tool-call-parser lfm2 --reasoning-parser qwen3 --max-inflight 32 \\\n    \"${SPLIT_ARGS[@]}\"\n",
    "eval_hf_jobs.sh": "#!/bin/bash\n# Held-out pass@1 of LFM2.5-24B-A2B and its harness adapters on HF Jobs (1x H200). One vLLM serves the base and every\n# checkpoint as an adapter, with one sandbox server per checkpoint, all evaluated at once. Results land in\n# /outputs/eval/<label>_<split>/summary.json on the bucket.\n#\n# Same flags as the scheduled sweep that evaluated the released run (../scripts/eval_sweep.py).\n#\n#   bash eval_hf_jobs.sh                                      base + the released checkpoints, on SmolDataEnvs held-out\n#   SPLIT=traj2envs-heldout bash eval_hf_jobs.sh              the traj2envs held-out tasks instead\n#   CHECKPOINTS=\"/outputs/runs/<run>/checkpoint-50@pi\" bash eval_hf_jobs.sh\nset -euo pipefail\n\nBUCKET=\"${BUCKET:-ybelkada/harbor-evals}\"\nSPLIT=\"${SPLIT:-final-heldout}\"  # final-heldout | traj2envs-heldout\nJOB=6aedb47a0e0338d1bb8107e8     # the released training job, whose runs are under /outputs/runs\n# <path|glob|hub id>@<harnesses>, space-separated\nCHECKPOINTS=\"${CHECKPOINTS:-LiquidAI/LFM2.5-24B-A2B@pi,hermes,opencode \\\n/outputs/runs/LFM2.5-24B-A2B-pi-$JOB-*/checkpoint-170@pi \\\n/outputs/runs/LFM2.5-24B-A2B-hermes-$JOB-*/checkpoint-50@hermes \\\n/outputs/runs/LFM2.5-24B-A2B-opencode-$JOB-*/checkpoint-160@opencode}\"\nSCRIPTS=\"$(cd \"$(dirname \"$0\")/../scripts\" && pwd)\"\n\nif [ \"$SPLIT\" = final-heldout ]; then\n    SPLIT_ARGS=(--split FineEnvs/SmolDataEnvs-harbor-train --task-offset 1000 --difficulty easy,medium --n-tasks 150\n        --docker-image docker.io/ybelkada/env-data-agent-train:opencode-1.19.5)\nelse\n    SPLIT_ARGS=(--split harness-hub/traj2envs-rl@191a751bda511eada76f6a9a76513d143321a64a --task-offset 230 --n-tasks 40)\nfi\nread -r -a CHECKPOINT_LIST <<< \"$CHECKPOINTS\"\n\n# The job reads the eval client from the bucket, mounted on /outputs\nhf buckets cp \"$SCRIPTS/evaluate_harbor.py\" \"hf://buckets/$BUCKET/evaluate_harbor.py\"\n\nhf jobs uv run --detach \\\n    --flavor h200 --image huggingface/trl --secrets HF_TOKEN --timeout 6h \\\n    --volume \"hf://buckets/$BUCKET:/outputs\" \\\n    \"$SCRIPTS/launcher_eval.py\" -- \\\n    --data-root /outputs --eval-lora --thinking --sandbox hf-sandbox --expose gradio \\\n    --model LiquidAI/LFM2.5-24B-A2B --max-model-len 131072 --eval-tag \"$SPLIT\" \\\n    --eval-harnesses pi,hermes,opencode \\\n    --eval-checkpoints \"${CHECKPOINT_LIST[@]}\" \\\n    --tool-call-parser lfm2 --reasoning-parser qwen3 --max-inflight 32 \\\n    \"${SPLIT_ARGS[@]}\"\n",
    "eval_slurm.sh": "#!/bin/bash -l\n# Held-out pass@1 of LFM2.5-24B-A2B and its harness adapters on one cluster GPU. One vLLM serves the base and every\n# checkpoint as an adapter, with one sandbox server per checkpoint. Results land in $ROOT/data/eval/<label>_<split>/.\n#\n# Same flags as eval_hf_jobs.sh. Not run on the cluster yet. One-time setup on a login node: see ../README.md.\n#\n#   CHECKPOINTS=\"LiquidAI/LFM2.5-24B-A2B@pi,hermes,opencode /fsx/$USER/harness-adapters/data/runs/<run>/checkpoint-50@pi\" \\\n#       sbatch eval_slurm.sh\n#   SPLIT=traj2envs-heldout CHECKPOINTS=... sbatch eval_slurm.sh\n#SBATCH --job-name=ha-lfm2.5-eval\n#SBATCH --partition=hopper-b200\n#SBATCH --nodes=1\n#SBATCH --ntasks-per-node=1\n#SBATCH --gpus-per-node=1\n#SBATCH --cpus-per-task=16\n#SBATCH --time=6:00:00\n#SBATCH --output=/fsx/%u/logs/%x-%j.out\nset -euo pipefail\n\nROOT=\"${ROOT:-/fsx/$USER/harness-adapters}\"\nSPLIT=\"${SPLIT:-final-heldout}\"  # final-heldout | traj2envs-heldout\n# <path|glob|hub id>@<harnesses>, space-separated\nCHECKPOINTS=\"${CHECKPOINTS:-LiquidAI/LFM2.5-24B-A2B@pi,hermes,opencode}\"\nsource \"$ROOT/scripts/slurm_setup.sh\"\n\nif [ \"$SPLIT\" = final-heldout ]; then\n    SPLIT_ARGS=(--split FineEnvs/SmolDataEnvs-harbor-train --task-offset 1000 --difficulty easy,medium --n-tasks 150\n        --docker-image docker.io/ybelkada/env-data-agent-train:opencode-1.19.5)\nelse\n    SPLIT_ARGS=(--split harness-hub/traj2envs-rl@191a751bda511eada76f6a9a76513d143321a64a --task-offset 230 --n-tasks 40)\nfi\nread -r -a CHECKPOINT_LIST <<< \"$CHECKPOINTS\"\n\npython \"$ROOT/scripts/launcher_eval.py\" \\\n    --data-root \"$ROOT/data\" --eval-script-url \"file://$ROOT/scripts/evaluate_harbor.py\" \\\n    --vllm-port \"$VLLM_PORT\" --server-port \"$SERVER_PORT\" --capture-port \"$CAPTURE_PORT\" \\\n    --eval-lora --thinking --sandbox hf-sandbox --expose gradio \\\n    --model LiquidAI/LFM2.5-24B-A2B --max-model-len 131072 --eval-tag \"$SPLIT\" \\\n    --eval-harnesses pi,hermes,opencode \\\n    --eval-checkpoints \"${CHECKPOINT_LIST[@]}\" \\\n    --tool-call-parser lfm2 --reasoning-parser qwen3 --max-inflight 32 \\\n    \"${SPLIT_ARGS[@]}\"\n",
    "recipe.yaml": "meta:\n  title: \"LFM2.5-24B-A2B harness adapters\"\n  project: harness-adapters\n  trainer: AsyncGRPOTrainer\n  model_id: LiquidAI/LFM2.5-24B-A2B\n  description: \"One LoRA adapter per agent harness (pi, hermes, opencode), trained with async GRPO inside the unmodified harness on verifiable Harbor tasks, with safeagents checking every tool call.\"\n  tags: [lora, agentic-rl, harbor, safeagents, multi-harness]\n  harnesses: [pi, hermes, opencode]\n  date_added: 2026-11-13\n  blog: https://huggingface.co/spaces/harness-hub/harness-adapters\n  dashboard: https://harness-hub-harness-adapters-rl.hf.space/\n\nhardware:\n  hf_jobs: h200x4            # vLLM on GPU 0, one trainer per harness on GPUs 1-3\n  slurm: \"hopper-b200, 4 GPUs (untested)\"\n  local: \"4 GPUs, H200 class (untested)\"\n\ndatasets:\n  train:\n    - FineEnvs/SmolDataEnvs-harbor-train       # 100 medium + 200 hard\n    - harness-hub/traj2envs-rl@191a751bda511eada76f6a9a76513d143321a64a  # first 230 tasks\n  eval:\n    final-heldout: \"FineEnvs/SmolDataEnvs-harbor-train, 150 easy/medium tasks from index 1000\"\n    traj2envs-heldout: \"harness-hub/traj2envs-rl, the 40 tasks from index 230\"\n\nhyperparameters:\n  lora_r: 32\n  lora_target_modules: all-linear\n  learning_rate: 1.0e-5\n  num_generations: 8\n  gradient_accumulation_steps: 4\n  max_staleness: 4\n  temperature: 0.8\n  max_completion_length: 16384\n  max_output_tokens: 32768\n  max_model_len: 131072\n  agent_step_limit: 17\n  agent_timeout_s: 600\n  max_inflight: 48\n  tool_efficiency_weight: 0.1\n  max_steps: 200\n  save_steps: 10\n\nvllm:\n  tool_call_parser: lfm2\n  reasoning_parser: qwen3\n\ntrl:\n  fork: \"younesbelkada/trl@ee6e9ad9512b83c80bb0051942770c8798c0dbf2 (add-lfm-tree-attn)\"\n  upstream_base: 7fdad81d04303e6f43db168e903565769c0b8642\n  patch: trl.patch            # the fork's changes to trl/, relative to upstream_base\n\nprovenance:\n  hf_job: 6aedb47a0e0338d1bb8107e8\n  released:\n    pi: {step: 170, repo: harness-hub/LFM2.5-24B-A2B-pi-v2}\n    hermes: {step: 50, repo: harness-hub/LFM2.5-24B-A2B-hermes-v2}\n    opencode: {step: 160, repo: harness-hub/LFM2.5-24B-A2B-opencode}\n\nscripts:\n  train: {hf_jobs: train_hf_jobs.sh, slurm: train_slurm.sh, local: train_local.sh}\n  eval: {hf_jobs: eval_hf_jobs.sh, slurm: eval_slurm.sh, local: eval_local.sh}\n\n# What the command builder on the recipes site offers. Each option is an environment variable of the script it runs.\nbuilder:\n  verified: {hf_jobs: true, slurm: false, local: false}   # whether the scripts of each platform ran as written\n  eval:\n    - env: SPLIT\n      label: Held-out split\n      choices: [final-heldout, traj2envs-heldout]\n    - env: CHECKPOINTS\n      label: Checkpoints\n      input: true\n      placeholder: \"<path|glob|hub id>@<harnesses> ... (empty: the script's default)\"\n",
    "trl.patch": "TRL changes these recipes run on, relative to upstream huggingface/trl.\n  upstream base: 7fdad81d04303e6f43db168e903565769c0b8642 (2026-10-28, fix: add peft on asyncgrpo openenv harbor + fixes (#7948))\n  fork:          younesbelkada/trl add-lfm-tree-attn = ee6e9ad9512b83c80bb0051942770c8798c0dbf2\nFork commits:\n  ee6e9ad9 Merge branch 'main' into add-lfm-tree-attn\n  c474c2cc Apply suggestion from @younesbelkada\n  eb5d5b9b Merge branch 'main' into add-lfm-tree-attn\n  2fa050c4 revert\n  b467898a feat: add LFM support for tree attn\nApply on a checkout of the upstream base: git apply trl.patch\nRegenerate: git diff $(git merge-base origin/main <fork commit>) <fork commit> -- trl/\n\ndiff --git a/trl/experimental/async_grpo/async_grpo_trainer.py b/trl/experimental/async_grpo/async_grpo_trainer.py\nindex 66a5d595..2e609d6f 100644\n--- a/trl/experimental/async_grpo/async_grpo_trainer.py\n+++ b/trl/experimental/async_grpo/async_grpo_trainer.py\n@@ -58,7 +58,7 @@ from ...trainer.utils import (\n )\n from .async_grpo_config import AsyncGRPOConfig\n from .async_rollout_worker import AsyncRolloutWorker, RolloutSample\n-from .packing import TREE_ATTENTION, TreeRow, patch_tree_gdn, register_tree_attention, unpad_tree\n+from .packing import TREE_ATTENTION, TreeRow, patch_tree_recurrent, register_tree_attention, unpad_tree\n from .vllm_client import VLLMClient\n from .weight_transfer import WeightTransferClient\n \n@@ -1122,11 +1122,17 @@ class AsyncGRPOTrainer(_BaseTrainer):\n                     \"before it, these layers call the functions they stored at init rather than the module-level \"\n                     \"ones tree packing patches, so they would scan across samples.\"\n                 )\n-            if layer_types != {\"linear_attention\"} or not patch_tree_gdn(model):\n+            if layer_types == {\"conv\"} and Version(transformers.__version__) <= Version(\"5.21.0.dev0\"):\n                 raise ValueError(\n-                    f\"{type(model).__name__} has {sorted(layer_types)} layers. Tree packing routes attention and \"\n-                    \"transformers' gated DeltaNet layers along each sample's own path; any other recurrent or \"\n-                    \"convolutional layer would scan across samples.\"\n+                    f\"Tree packing of {type(model).__name__}'s short convolution layers requires transformers>=5.22.0: \"\n+                    \"before it, these layers do not pass the forward's keyword arguments down to the module-level \"\n+                    \"function tree packing patches, so they would scan across samples.\"\n+                )\n+            if layer_types not in ({\"linear_attention\"}, {\"conv\"}) or not patch_tree_recurrent(model, *layer_types):\n+                raise ValueError(\n+                    f\"{type(model).__name__} has {sorted(layer_types)} layers. Tree packing routes attention, \"\n+                    \"transformers' gated DeltaNet layers and LFM2's short convolutions along each sample's own path; \"\n+                    \"any other recurrent or convolutional layer would scan across samples.\"\n                 )\n         if any(name.endswith(\".sinks\") for name, _ in model.named_parameters()):\n             raise ValueError(\ndiff --git a/trl/experimental/async_grpo/packing.py b/trl/experimental/async_grpo/packing.py\nindex c22132f3..24ed508d 100644\n--- a/trl/experimental/async_grpo/packing.py\n+++ b/trl/experimental/async_grpo/packing.py\n@@ -119,7 +119,7 @@ class TreeRow:\n                 - `tree_cu_seqlens_q` (`S + 1`), `tree_cu_seqlens_k` (`S + 1`): segment boundaries in the row, path\n                   boundaries in `tree_kv_index`.\n                 - `tree_segment_parents` (`S`), `tree_segment_depths` (`S`): the segment tree, for the recurrent layers\n-                  patched by [`patch_tree_gdn`].\n+                  patched by [`patch_tree_recurrent`].\n                 - `tree_sizes` (`4 + depth`): longest segment, longest path, `K`, 64-token chunks, then the segments\n                   per depth. Every length the kernels launch with, so a rank reads them in one go (see [`unpad_tree`]).\n                 - `token_index` (`M`): position whose prediction is each trained token. A shared position serves every\n@@ -278,20 +278,27 @@ def register_tree_attention() -> None:\n _PATCHED = set()\n \n \n-def patch_tree_gdn(model) -> bool:\n+def patch_tree_recurrent(model, layer_type: str) -> bool:\n     \"\"\"\n-    Make the model's gated DeltaNet layers follow the tree of a tree-packed forward.\n+    Make the model's gated DeltaNet or short convolution layers follow the tree of a tree-packed forward.\n \n-    Transformers' hybrids (Qwen3.5, Qwen3-Next, OLMo-Hybrid, ...) run these layers through two module-level functions,\n-    `causal_conv1d_fn` and `torch_chunk_gated_delta_rule`, and pass the forward's keyword arguments down to both. Both\n-    are wrapped: given the tree of [`~packing.TreeRow.pack`] they follow it, otherwise they run unchanged.\n+    Transformers' gated DeltaNet hybrids (Qwen3.5, Qwen3-Next, OLMo-Hybrid, ...) run their `\"linear_attention\"` layers\n+    through two module-level functions, `causal_conv1d_fn` and `torch_chunk_gated_delta_rule`, and LFM2's `\"conv\"`\n+    layers through the first alone. These layers pass the forward's keyword arguments down to them. Each is wrapped:\n+    given the tree of [`~packing.TreeRow.pack`] they follow it, otherwise they run unchanged.\n+\n+    Args:\n+        model ([`~transformers.PreTrainedModel`]):\n+            Model to patch.\n+        layer_type (`str`):\n+            The model's only non-attention layer type, `\"linear_attention\"` or `\"conv\"`.\n \n     Returns:\n-        `bool`: whether the model's modeling module has those functions.\n+        `bool`: whether the model's modeling module has the functions its layers run through.\n     \"\"\"\n     modeling = sys.modules[type(model).__module__]\n     conv, chunk = vars(modeling).get(\"causal_conv1d_fn\"), vars(modeling).get(\"torch_chunk_gated_delta_rule\")\n-    if conv is None or chunk is None:\n+    if conv is None or (layer_type == \"linear_attention\" and chunk is None):\n         return False\n     if modeling in _PATCHED:\n         return True\n@@ -307,6 +314,8 @@ def patch_tree_gdn(model) -> bool:\n             return chunk(query, key, value, g=g, beta=beta, use_qk_l2norm_in_kernel=use_qk_l2norm_in_kernel, **kwargs)\n         return tree_chunk_gated_delta_rule(query, key, value, g, beta, kwargs, use_qk_l2norm_in_kernel), None\n \n-    modeling.causal_conv1d_fn, modeling.torch_chunk_gated_delta_rule = causal_conv1d_fn, torch_chunk_gated_delta_rule\n+    modeling.causal_conv1d_fn = causal_conv1d_fn\n+    if chunk is not None:\n+        modeling.torch_chunk_gated_delta_rule = torch_chunk_gated_delta_rule\n     _PATCHED.add(modeling)\n     return True\ndiff --git a/trl/trainer/utils.py b/trl/trainer/utils.py\nindex 5e53bc49..e72efc9b 100644\n--- a/trl/trainer/utils.py\n+++ b/trl/trainer/utils.py\n@@ -1802,18 +1802,21 @@ def compute_flops_per_token(config: PretrainedConfig, seq_len: int) -> int:\n         mlp_flops = 2 * 3 * h * config.intermediate_size\n         total_layer_flops = L * (attn_flops + mlp_flops)\n     else:\n-        # Routed experts (gate + up + down, 3 matmuls each) + router.\n-        if Version(transformers.__version__) >= Version(\"5.1.0\"):\n-            num_experts = config.num_local_experts\n-        else:\n+        # Routed experts (gate + up + down, 3 matmuls each) + router. LFM2-MoE keeps `num_experts` on every version, and\n+        # opens with `num_dense_layers` dense layers where Qwen-style MoE spaces them by `decoder_sparse_step`.\n+        if config.model_type == \"lfm2_moe\":\n             num_experts = config.num_experts\n+            is_sparse = [layer_idx >= config.num_dense_layers for layer_idx in range(L)]\n+        else:\n+            if Version(transformers.__version__) >= Version(\"5.1.0\"):\n+                num_experts = config.num_local_experts\n+            else:\n+                num_experts = config.num_experts\n+            is_sparse = [layer_idx % config.decoder_sparse_step == 0 for layer_idx in range(L)]\n         moe_mlp_flops = num_experts_per_tok * 2 * 3 * h * config.moe_intermediate_size\n         moe_mlp_flops += 2 * h * num_experts\n         dense_mlp_flops = 2 * 3 * h * config.intermediate_size  # interspersed dense layers\n-        sparse_step = config.decoder_sparse_step\n-        total_layer_flops = sum(\n-            attn_flops + (moe_mlp_flops if layer_idx % sparse_step == 0 else dense_mlp_flops) for layer_idx in range(L)\n-        )\n+        total_layer_flops = sum(attn_flops + (moe_mlp_flops if sparse else dense_mlp_flops) for sparse in is_sparse)\n \n     # Embedding is a lookup (no FLOPs); lm_head is a 2*V*h matmul, tied or not.\n     lm_head_flops = 2 * V * h\n"
   },
   "other_files": []
  },
  {
   "id": "AsyncGRPOTrainer/harness-adapters/LFM3-1.7B",
   "trainer": "AsyncGRPOTrainer",
   "project": "harness-adapters",
   "name": "LFM3-1.7B",
   "org": "LiquidAI",
   "avatar": "https://cdn-avatars.huggingface.co/v1/production/uploads/61b8e2ba285851687028d395/EsTgVtnM2IqVRKgPdfqcB.png",
   "meta": {
    "title": "LFM3-1.7B harness adapters",
    "project": "harness-adapters",
    "trainer": "AsyncGRPOTrainer",
    "model_id": "LiquidAI/LFM3-1.7B",
    "description": "One LoRA adapter per agent harness (pi, hermes, opencode), trained with async GRPO inside the unmodified harness on verifiable Harbor tasks, with safeagents checking every tool call.",
    "tags": [
     "lora",
     "agentic-rl",
     "harbor",
     "safeagents",
     "multi-harness"
    ],
    "harnesses": [
     "pi",
     "hermes",
     "opencode"
    ],
    "date_added": "2026-11-13",
    "blog": "https://huggingface.co/spaces/harness-hub/harness-adapters",
    "dashboard": "https://harness-hub-harness-adapters-rl.hf.space/"
   },
   "hardware": {
    "hf_jobs": "h200x2",
    "slurm": "hopper-b200, 2 GPUs (untested)",
    "local": "2 GPUs, H200 class (untested)"
   },
   "datasets": {
    "train": [
     "FineEnvs/SmolDataEnvs-harbor-train",
     "harness-hub/traj2envs-rl@191a751bda511eada76f6a9a76513d143321a64a"
    ],
    "eval": {
     "final-heldout": "FineEnvs/SmolDataEnvs-harbor-train, 150 easy/medium tasks from index 1000",
     "traj2envs-heldout": "harness-hub/traj2envs-rl, the 40 tasks from index 230"
    }
   },
   "hyperparameters": {
    "lora_r": 32,
    "lora_target_modules": "all-linear",
    "learning_rate": 1e-05,
    "num_generations": 8,
    "gradient_accumulation_steps": 4,
    "max_staleness": 4,
    "temperature": 0.8,
    "max_completion_length": 16384,
    "max_output_tokens": 32768,
    "max_model_len": 131072,
    "token_budget": 65536,
    "agent_step_limit": 17,
    "agent_timeout_s": 600,
    "max_inflight": 48,
    "tool_efficiency_weight": 0.1,
    "max_steps": 400,
    "save_steps": 10
   },
   "vllm": {
    "tool_call_parser": "qwen3_coder",
    "reasoning_parser": "qwen3",
    "trust_remote_code": true,
    "commit": "fac2ce2eb3633c44a104a66dbb30d0e6c7f4493e",
    "overlays_from": "https://github.com/atmikshetty/vllm/tree/5f67fbcbb2336b0e551c0df2b4c0eb51b030c553"
   },
   "trl": {
    "fork": "younesbelkada/trl@ee6e9ad9512b83c80bb0051942770c8798c0dbf2 (add-lfm-tree-attn)",
    "upstream_base": "7fdad81d04303e6f43db168e903565769c0b8642",
    "patch": "trl.patch"
   },
   "provenance": {
    "hf_job": "6aedb4820e0338d1bb8107eb",
    "released": {
     "pi": {
      "step": 400,
      "repo": "harness-hub/LFM3-1.7B-pi"
     },
     "hermes": {
      "step": 400,
      "repo": "harness-hub/LFM3-1.7B-hermes"
     },
     "opencode": {
      "step": 400,
      "repo": "harness-hub/LFM3-1.7B-opencode"
     }
    }
   },
   "scripts": {
    "train": {
     "hf_jobs": "train_hf_jobs.sh",
     "slurm": "train_slurm.sh",
     "local": "train_local.sh"
    },
    "eval": {
     "hf_jobs": "eval_hf_jobs.sh",
     "slurm": "eval_slurm.sh",
     "local": "eval_local.sh"
    }
   },
   "builder": {
    "verified": {
     "hf_jobs": true,
     "slurm": false,
     "local": false
    },
    "eval": [
     {
      "env": "SPLIT",
      "label": "Held-out split",
      "choices": [
       "final-heldout",
       "traj2envs-heldout"
      ]
     },
     {
      "env": "CHECKPOINTS",
      "label": "Checkpoints",
      "input": true,
      "placeholder": "<path|glob|hub id>@<harnesses> ... (empty: the script's default)"
     }
    ]
   },
   "readme": "# LFM3-1.7B harness adapters\n\nLoRA adapters for pi, hermes and opencode on [`LiquidAI/LFM3-1.7B`](https://huggingface.co/LiquidAI/LFM3-1.7B),\ntrained for 400 steps on 550 SmolDataEnvs tasks (300 medium, 250 hard) and 230 traj2envs tasks. Setup and shared\nsettings: [../README.md](../README.md). Full hyperparameters: [recipe.yaml](recipe.yaml).\n\n| | HF Jobs | Cluster (untested) | Local (untested) |\n|---|---|---|---|\n| Train | `bash train_hf_jobs.sh` (h200x2, ~48h) | `sbatch train_slurm.sh` (2 GPUs) | `bash train_local.sh` (2 GPUs) |\n| Eval | `bash eval_hf_jobs.sh` (h200) | `CHECKPOINTS=... sbatch eval_slurm.sh` (1 GPU) | `bash eval_local.sh` (1 GPU) |\n\nModel-specific flags:\n- LFM3 is not in released vLLM yet. The launcher builds vLLM at commit `fac2ce2e`\n  (`--vllm-commit`) and overlays three model files from\n  [atmikshetty/vllm@5f67fbcb](https://github.com/atmikshetty/vllm/tree/5f67fbcbb2336b0e551c0df2b4c0eb51b030c553)\n  (`--vllm-overlay`).\n- `--trust-remote-code` for both the trainer and vLLM.\n- `--tool-call-parser qwen3_coder --reasoning-parser qwen3`, `--token-budget 65536`\n- The model is small, so the three trainers share GPU 1 (`--train-device 1,1,1`).\n\nReleased checkpoints (job `6aedb4820e0338d1bb8107eb`):\n\n| Harness | Step | Repo |\n|---|--:|---|\n| pi | 400 | `harness-hub/LFM3-1.7B-pi` |\n| hermes | 400 | `harness-hub/LFM3-1.7B-hermes` |\n| opencode | 400 | `harness-hub/LFM3-1.7B-opencode` |\n",
   "files": {
    "train_local.sh": "#!/bin/bash\n# LFM3-1.7B harness adapters on one local machine with 2 GPUs:\n# vLLM on GPU 0, the three trainers sharing GPU 1. The released run used H200s (141 GB).\n#\n# Same launcher and training flags as train_hf_jobs.sh, run here with `uv run` instead of on HF Jobs. Not run locally\n# yet. Needs uv, 2 GPUs and HF_TOKEN (also what the HF sandboxes run under). Runs are written to $DATA_ROOT/runs/.\n#\n#   bash train_local.sh\n#   MAX_STEPS=2 bash train_local.sh                   short smoke run\nset -euo pipefail\n\nMAX_STEPS=\"${MAX_STEPS:-400}\"\nDATA_ROOT=\"${DATA_ROOT:-$PWD/outputs}\"\nTRL_WHEEL=\"${TRL_WHEEL:-}\"\nTRACKIO_SPACE_ID=\"${TRACKIO_SPACE_ID:-ybelkada/async-grpo-harbor}\"\nREV=191a751bda511eada76f6a9a76513d143321a64a  # harness-hub/traj2envs-rl revision the adapters were trained on\nP=https://raw.githubusercontent.com/atmikshetty/vllm/5f67fbcbb2336b0e551c0df2b4c0eb51b030c553  # LFM3 vLLM model files\nHERE=\"$(cd \"$(dirname \"$0\")\" && pwd)\"\nSCRIPTS=\"$HERE/../scripts\"\nexport HF_TOKEN=\"${HF_TOKEN:-$(cat ~/.cache/huggingface/token)}\"\n# Names this run's folders (logs/<id>, runs/<model>-<harness>-<id>-...), as HF_JOB_ID does on HF Jobs\nexport JOB_ID=\"${JOB_ID:-local-$(date +%Y%m%d-%H%M%S)}\"\nexport PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True\n\nTRL_ARGS=()\nif [ -n \"$TRL_WHEEL\" ]; then TRL_ARGS=(--trl-wheel \"$TRL_WHEEL\"); fi\n\nuv run \"$SCRIPTS/launcher_train.py\" \\\n    --data-root \"$DATA_ROOT\" ${TRL_ARGS[@]+\"${TRL_ARGS[@]}\"} \\\n    --train-script-url \"file://$SCRIPTS/async_grpo_harbor.py\" --expose gradio \\\n    --pip-install \"safeagents@git+https://github.com/huggingface/safeagents@5e683fa atif-scan==0.8.15 rich markdown-it-py mdurl pygments\" \\\n    --model LiquidAI/LFM3-1.7B --trust-remote-code \\\n    --vllm-commit fac2ce2eb3633c44a104a66dbb30d0e6c7f4493e \\\n    --vllm-overlay \"$P/vllm/model_executor/models/lfm3.py\" \\\n    --vllm-overlay \"$P/vllm/model_executor/models/registry.py\" \\\n    --vllm-overlay \"$P/vllm/model_executor/layers/mamba/ops/causal_conv1d.py\" \\\n    --vllm-extra-args=--trust-remote-code \\\n    --tool-call-parser qwen3_coder --reasoning-parser qwen3 --max-model-len 131072 \\\n    --run pi:pi:medium:300,hard:250 \\\n    --run hermes:hermes:medium:300,hard:250 \\\n    --run opencode:opencode:medium:300,hard:250 \\\n    --split FineEnvs/SmolDataEnvs-harbor-train --n-tasks 550 \\\n    --mix-split \"harness-hub/traj2envs-rl@$REV:230\" --max-output-tokens 32768 \\\n    --sandbox hf-sandbox --docker-image docker.io/ybelkada/env-data-agent-train:opencode-1.19.5 \\\n    --vllm-device 0 --train-device 1,1,1 \\\n    --max-inflight 48 --max-lora-rank 32 --max-loras 18 \\\n    --reward-key correctness,reward --num-generations 8 --gradient-accumulation-steps 4 --max-staleness 4 \\\n    --max-completion-length 16384 --agent-step-limit 17 --agent-timeout 600 --temperature 0.8 \\\n    --lora-r 32 --lora-target-modules all-linear --learning-rate 1e-5 --tool-efficiency-weight 0.1 \\\n    --token-budget 65536 \\\n    --safeagents --safeagents-plugin \"$SCRIPTS/atif_smoldataenvs.py:checks\" --safeagents-plugin \"$SCRIPTS/atif_traj2envs.py:checks\" \\\n    --max-steps \"$MAX_STEPS\" --save-steps 10 --save-total-limit 0 \\\n    --trackio-space-id \"$TRACKIO_SPACE_ID\"\n",
    "train_hf_jobs.sh": "#!/bin/bash\n# LFM3-1.7B harness adapters on HF Jobs: one vLLM (GPU 0) serving the base and every adapter version, and three LoRA\n# AsyncGRPO trainers (pi, hermes, opencode) sharing GPU 1, on 2x H200.\n#\n# This is the command of the released run (HF job 6aedb4820e0338d1bb8107eb, 2026-11-06), with the bucket as a variable.\n# LFM3 needs a vLLM build from a pinned commit plus three model files from atmikshetty/vllm (--vllm-overlay).\n#\n#   bash train_hf_jobs.sh\n#   BUCKET=<user>/<bucket> MAX_STEPS=2 bash train_hf_jobs.sh          short smoke run\n#   TRL_WHEEL=/outputs/wheels/trl-....whl bash train_hf_jobs.sh       a TRL wheel already on the bucket, over the\n#                                                                     launcher's pinned TRL\nset -euo pipefail\n\nBUCKET=\"${BUCKET:-ybelkada/harbor-evals}\"\nMAX_STEPS=\"${MAX_STEPS:-400}\"\nTRL_WHEEL=\"${TRL_WHEEL:-}\"\nTRACKIO_SPACE_ID=\"${TRACKIO_SPACE_ID:-ybelkada/async-grpo-harbor}\"\nREV=191a751bda511eada76f6a9a76513d143321a64a  # harness-hub/traj2envs-rl revision the adapters were trained on\nP=https://raw.githubusercontent.com/atmikshetty/vllm/5f67fbcbb2336b0e551c0df2b4c0eb51b030c553  # LFM3 vLLM model files\nSCRIPTS=\"$(cd \"$(dirname \"$0\")/../scripts\" && pwd)\"\n\n# The job reads the training script and the safeagents plugins from the bucket, mounted on /outputs\nfor f in async_grpo_harbor.py atif_smoldataenvs.py atif_traj2envs.py; do\n    hf buckets cp \"$SCRIPTS/$f\" \"hf://buckets/$BUCKET/$f\"\ndone\n\nTRL_ARGS=()\nif [ -n \"$TRL_WHEEL\" ]; then TRL_ARGS=(--trl-wheel \"$TRL_WHEEL\"); fi\n\nhf jobs uv run --detach \\\n    --flavor h200x2 --image huggingface/trl --secrets HF_TOKEN --timeout 48h \\\n    --env PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True \\\n    --volume \"hf://buckets/$BUCKET:/outputs\" \\\n    \"$SCRIPTS/launcher_train.py\" -- \\\n    --data-root /outputs ${TRL_ARGS[@]+\"${TRL_ARGS[@]}\"} \\\n    --pip-install \"safeagents@git+https://github.com/huggingface/safeagents@5e683fa atif-scan==0.8.15 rich markdown-it-py mdurl pygments\" \\\n    --model LiquidAI/LFM3-1.7B --trust-remote-code \\\n    --vllm-commit fac2ce2eb3633c44a104a66dbb30d0e6c7f4493e \\\n    --vllm-overlay \"$P/vllm/model_executor/models/lfm3.py\" \\\n    --vllm-overlay \"$P/vllm/model_executor/models/registry.py\" \\\n    --vllm-overlay \"$P/vllm/model_executor/layers/mamba/ops/causal_conv1d.py\" \\\n    --vllm-extra-args=--trust-remote-code \\\n    --tool-call-parser qwen3_coder --reasoning-parser qwen3 --max-model-len 131072 \\\n    --run pi:pi:medium:300,hard:250 \\\n    --run hermes:hermes:medium:300,hard:250 \\\n    --run opencode:opencode:medium:300,hard:250 \\\n    --split FineEnvs/SmolDataEnvs-harbor-train --n-tasks 550 \\\n    --mix-split \"harness-hub/traj2envs-rl@$REV:230\" --max-output-tokens 32768 \\\n    --sandbox hf-sandbox --docker-image docker.io/ybelkada/env-data-agent-train:opencode-1.19.5 \\\n    --vllm-device 0 --train-device 1,1,1 \\\n    --max-inflight 48 --max-lora-rank 32 --max-loras 18 \\\n    --reward-key correctness,reward --num-generations 8 --gradient-accumulation-steps 4 --max-staleness 4 \\\n    --max-completion-length 16384 --agent-step-limit 17 --agent-timeout 600 --temperature 0.8 \\\n    --lora-r 32 --lora-target-modules all-linear --learning-rate 1e-5 --tool-efficiency-weight 0.1 \\\n    --token-budget 65536 \\\n    --safeagents --safeagents-plugin /outputs/atif_smoldataenvs.py:checks --safeagents-plugin /outputs/atif_traj2envs.py:checks \\\n    --max-steps \"$MAX_STEPS\" --save-steps 10 --save-total-limit 0 \\\n    --trackio-space-id \"$TRACKIO_SPACE_ID\"\n",
    "train_slurm.sh": "#!/bin/bash -l\n# LFM3-1.7B harness adapters on one cluster node: one vLLM (GPU 0) serving the base and every adapter version, and three\n# LoRA AsyncGRPO trainers (pi, hermes, opencode) sharing GPU 1.\n#\n# Same launcher and training flags as train_hf_jobs.sh (the released run). Not run on the cluster yet: the released\n# adapters were trained on HF Jobs. One-time setup on a login node: see ../README.md.\n#\n#   sbatch train_slurm.sh\n#   MAX_STEPS=2 sbatch -p hopper-prod train_slurm.sh\n#SBATCH --job-name=ha-lfm3\n#SBATCH --partition=hopper-b200\n#SBATCH --nodes=1\n#SBATCH --ntasks-per-node=1\n#SBATCH --gpus-per-node=2\n#SBATCH --cpus-per-task=32\n#SBATCH --time=2-00:00:00\n#SBATCH --output=/fsx/%u/logs/%x-%j.out\nset -euo pipefail\n\nROOT=\"${ROOT:-/fsx/$USER/harness-adapters}\"\nMAX_STEPS=\"${MAX_STEPS:-400}\"\nREV=191a751bda511eada76f6a9a76513d143321a64a  # harness-hub/traj2envs-rl revision the adapters were trained on\nP=https://raw.githubusercontent.com/atmikshetty/vllm/5f67fbcbb2336b0e551c0df2b4c0eb51b030c553  # LFM3 vLLM model files\nsource \"$ROOT/scripts/slurm_setup.sh\"\n\npython \"$ROOT/scripts/launcher_train.py\" \\\n    --data-root \"$ROOT/data\" \\\n    --train-script-url \"file://$ROOT/scripts/async_grpo_harbor.py\" \\\n    --vllm-port \"$VLLM_PORT\" --server-port \"$SERVER_PORT\" --capture-port \"$CAPTURE_PORT\" --expose gradio \\\n    --model LiquidAI/LFM3-1.7B --trust-remote-code \\\n    --vllm-commit fac2ce2eb3633c44a104a66dbb30d0e6c7f4493e \\\n    --vllm-overlay \"$P/vllm/model_executor/models/lfm3.py\" \\\n    --vllm-overlay \"$P/vllm/model_executor/models/registry.py\" \\\n    --vllm-overlay \"$P/vllm/model_executor/layers/mamba/ops/causal_conv1d.py\" \\\n    --vllm-extra-args=--trust-remote-code \\\n    --tool-call-parser qwen3_coder --reasoning-parser qwen3 --max-model-len 131072 \\\n    --run pi:pi:medium:300,hard:250 \\\n    --run hermes:hermes:medium:300,hard:250 \\\n    --run opencode:opencode:medium:300,hard:250 \\\n    --split FineEnvs/SmolDataEnvs-harbor-train --n-tasks 550 \\\n    --mix-split \"harness-hub/traj2envs-rl@$REV:230\" --max-output-tokens 32768 \\\n    --sandbox hf-sandbox --docker-image docker.io/ybelkada/env-data-agent-train:opencode-1.19.5 \\\n    --vllm-device 0 --train-device 1,1,1 \\\n    --max-inflight 48 --max-lora-rank 32 --max-loras 18 \\\n    --reward-key correctness,reward --num-generations 8 --gradient-accumulation-steps 4 --max-staleness 4 \\\n    --max-completion-length 16384 --agent-step-limit 17 --agent-timeout 600 --temperature 0.8 \\\n    --lora-r 32 --lora-target-modules all-linear --learning-rate 1e-5 --tool-efficiency-weight 0.1 \\\n    --token-budget 65536 \\\n    --safeagents --safeagents-plugin \"$ROOT/scripts/atif_smoldataenvs.py:checks\" \\\n    --safeagents-plugin \"$ROOT/scripts/atif_traj2envs.py:checks\" \\\n    --max-steps \"$MAX_STEPS\" --save-steps 10 --save-total-limit 0\n",
    "eval_local.sh": "#!/bin/bash\n# Held-out pass@1 of LFM3-1.7B and its harness adapters on one local GPU. One vLLM serves the base and every\n# checkpoint as an adapter, with one sandbox server per checkpoint. Results land in $DATA_ROOT/eval/<label>_<split>/.\n#\n# Same flags as eval_hf_jobs.sh, run here with `uv run`. Not run locally yet.\n#\n#   bash eval_local.sh                                                         the base model\n#   CHECKPOINTS=\"LiquidAI/LFM3-1.7B@pi,hermes,opencode $PWD/outputs/runs/<run>/checkpoint-50@pi\" bash eval_local.sh\n#   SPLIT=traj2envs-heldout bash eval_local.sh\nset -euo pipefail\n\nDATA_ROOT=\"${DATA_ROOT:-$PWD/outputs}\"\nSPLIT=\"${SPLIT:-final-heldout}\"  # final-heldout | traj2envs-heldout\n# <path|glob|hub id>@<harnesses>, space-separated\nCHECKPOINTS=\"${CHECKPOINTS:-LiquidAI/LFM3-1.7B@pi,hermes,opencode}\"\nP=https://raw.githubusercontent.com/atmikshetty/vllm/5f67fbcbb2336b0e551c0df2b4c0eb51b030c553  # LFM3 vLLM model files\nHERE=\"$(cd \"$(dirname \"$0\")\" && pwd)\"\nSCRIPTS=\"$HERE/../scripts\"\nexport HF_TOKEN=\"${HF_TOKEN:-$(cat ~/.cache/huggingface/token)}\"\nexport JOB_ID=\"${JOB_ID:-local-$(date +%Y%m%d-%H%M%S)}\"\n\nif [ \"$SPLIT\" = final-heldout ]; then\n    SPLIT_ARGS=(--split FineEnvs/SmolDataEnvs-harbor-train --task-offset 1000 --difficulty easy,medium --n-tasks 150\n        --docker-image docker.io/ybelkada/env-data-agent-train:opencode-1.19.5)\nelse\n    SPLIT_ARGS=(--split harness-hub/traj2envs-rl@191a751bda511eada76f6a9a76513d143321a64a --task-offset 230 --n-tasks 40)\nfi\nread -r -a CHECKPOINT_LIST <<< \"$CHECKPOINTS\"\n\nuv run \"$SCRIPTS/launcher_eval.py\" \\\n    --data-root \"$DATA_ROOT\" --eval-script-url \"file://$SCRIPTS/evaluate_harbor.py\" \\\n    --eval-lora --thinking --sandbox hf-sandbox --expose gradio \\\n    --model LiquidAI/LFM3-1.7B --max-model-len 131072 --eval-tag \"$SPLIT\" \\\n    --eval-harnesses pi,hermes,opencode \\\n    --eval-checkpoints \"${CHECKPOINT_LIST[@]}\" \\\n    --vllm-commit fac2ce2eb3633c44a104a66dbb30d0e6c7f4493e \\\n    --vllm-overlay \"$P/vllm/model_executor/models/lfm3.py\" \\\n    --vllm-overlay \"$P/vllm/model_executor/models/registry.py\" \\\n    --vllm-overlay \"$P/vllm/model_executor/layers/mamba/ops/causal_conv1d.py\" \\\n    --vllm-extra-args=--trust-remote-code \\\n    --tool-call-parser qwen3_coder --reasoning-parser qwen3 --max-inflight 32 \\\n    \"${SPLIT_ARGS[@]}\"\n",
    "eval_hf_jobs.sh": "#!/bin/bash\n# Held-out pass@1 of LFM3-1.7B and its harness adapters on HF Jobs (1x H200). One vLLM serves the base and every\n# checkpoint as an adapter, with one sandbox server per checkpoint, all evaluated at once. Results land in\n# /outputs/eval/<label>_<split>/summary.json on the bucket.\n#\n# Same flags as the scheduled sweep that evaluated the released run (../scripts/eval_sweep.py).\n#\n#   bash eval_hf_jobs.sh                                      base + the released checkpoints, on SmolDataEnvs held-out\n#   SPLIT=traj2envs-heldout bash eval_hf_jobs.sh              the traj2envs held-out tasks instead\n#   CHECKPOINTS=\"/outputs/runs/<run>/checkpoint-50@pi\" bash eval_hf_jobs.sh\nset -euo pipefail\n\nBUCKET=\"${BUCKET:-ybelkada/harbor-evals}\"\nSPLIT=\"${SPLIT:-final-heldout}\"  # final-heldout | traj2envs-heldout\nJOB=6aedb4820e0338d1bb8107eb     # the released training job, whose runs are under /outputs/runs\n# <path|glob|hub id>@<harnesses>, space-separated\nCHECKPOINTS=\"${CHECKPOINTS:-LiquidAI/LFM3-1.7B@pi,hermes,opencode \\\n/outputs/runs/LFM3-1.7B-pi-$JOB-*/checkpoint-400@pi \\\n/outputs/runs/LFM3-1.7B-hermes-$JOB-*/checkpoint-400@hermes \\\n/outputs/runs/LFM3-1.7B-opencode-$JOB-*/checkpoint-400@opencode}\"\nP=https://raw.githubusercontent.com/atmikshetty/vllm/5f67fbcbb2336b0e551c0df2b4c0eb51b030c553  # LFM3 vLLM model files\nSCRIPTS=\"$(cd \"$(dirname \"$0\")/../scripts\" && pwd)\"\n\nif [ \"$SPLIT\" = final-heldout ]; then\n    SPLIT_ARGS=(--split FineEnvs/SmolDataEnvs-harbor-train --task-offset 1000 --difficulty easy,medium --n-tasks 150\n        --docker-image docker.io/ybelkada/env-data-agent-train:opencode-1.19.5)\nelse\n    SPLIT_ARGS=(--split harness-hub/traj2envs-rl@191a751bda511eada76f6a9a76513d143321a64a --task-offset 230 --n-tasks 40)\nfi\nread -r -a CHECKPOINT_LIST <<< \"$CHECKPOINTS\"\n\n# The job reads the eval client from the bucket, mounted on /outputs\nhf buckets cp \"$SCRIPTS/evaluate_harbor.py\" \"hf://buckets/$BUCKET/evaluate_harbor.py\"\n\nhf jobs uv run --detach \\\n    --flavor h200 --image huggingface/trl --secrets HF_TOKEN --timeout 6h \\\n    --volume \"hf://buckets/$BUCKET:/outputs\" \\\n    \"$SCRIPTS/launcher_eval.py\" -- \\\n    --data-root /outputs --eval-lora --thinking --sandbox hf-sandbox --expose gradio \\\n    --model LiquidAI/LFM3-1.7B --max-model-len 131072 --eval-tag \"$SPLIT\" \\\n    --eval-harnesses pi,hermes,opencode \\\n    --eval-checkpoints \"${CHECKPOINT_LIST[@]}\" \\\n    --vllm-commit fac2ce2eb3633c44a104a66dbb30d0e6c7f4493e \\\n    --vllm-overlay \"$P/vllm/model_executor/models/lfm3.py\" \\\n    --vllm-overlay \"$P/vllm/model_executor/models/registry.py\" \\\n    --vllm-overlay \"$P/vllm/model_executor/layers/mamba/ops/causal_conv1d.py\" \\\n    --vllm-extra-args=--trust-remote-code \\\n    --tool-call-parser qwen3_coder --reasoning-parser qwen3 --max-inflight 32 \\\n    \"${SPLIT_ARGS[@]}\"\n",
    "eval_slurm.sh": "#!/bin/bash -l\n# Held-out pass@1 of LFM3-1.7B and its harness adapters on one cluster GPU. One vLLM serves the base and every\n# checkpoint as an adapter, with one sandbox server per checkpoint. Results land in $ROOT/data/eval/<label>_<split>/.\n#\n# Same flags as eval_hf_jobs.sh. Not run on the cluster yet. One-time setup on a login node: see ../README.md.\n#\n#   CHECKPOINTS=\"LiquidAI/LFM3-1.7B@pi,hermes,opencode /fsx/$USER/harness-adapters/data/runs/<run>/checkpoint-50@pi\" \\\n#       sbatch eval_slurm.sh\n#   SPLIT=traj2envs-heldout CHECKPOINTS=... sbatch eval_slurm.sh\n#SBATCH --job-name=ha-lfm3-eval\n#SBATCH --partition=hopper-b200\n#SBATCH --nodes=1\n#SBATCH --ntasks-per-node=1\n#SBATCH --gpus-per-node=1\n#SBATCH --cpus-per-task=16\n#SBATCH --time=6:00:00\n#SBATCH --output=/fsx/%u/logs/%x-%j.out\nset -euo pipefail\n\nROOT=\"${ROOT:-/fsx/$USER/harness-adapters}\"\nSPLIT=\"${SPLIT:-final-heldout}\"  # final-heldout | traj2envs-heldout\n# <path|glob|hub id>@<harnesses>, space-separated\nCHECKPOINTS=\"${CHECKPOINTS:-LiquidAI/LFM3-1.7B@pi,hermes,opencode}\"\nP=https://raw.githubusercontent.com/atmikshetty/vllm/5f67fbcbb2336b0e551c0df2b4c0eb51b030c553  # LFM3 vLLM model files\nsource \"$ROOT/scripts/slurm_setup.sh\"\n\nif [ \"$SPLIT\" = final-heldout ]; then\n    SPLIT_ARGS=(--split FineEnvs/SmolDataEnvs-harbor-train --task-offset 1000 --difficulty easy,medium --n-tasks 150\n        --docker-image docker.io/ybelkada/env-data-agent-train:opencode-1.19.5)\nelse\n    SPLIT_ARGS=(--split harness-hub/traj2envs-rl@191a751bda511eada76f6a9a76513d143321a64a --task-offset 230 --n-tasks 40)\nfi\nread -r -a CHECKPOINT_LIST <<< \"$CHECKPOINTS\"\n\npython \"$ROOT/scripts/launcher_eval.py\" \\\n    --data-root \"$ROOT/data\" --eval-script-url \"file://$ROOT/scripts/evaluate_harbor.py\" \\\n    --vllm-port \"$VLLM_PORT\" --server-port \"$SERVER_PORT\" --capture-port \"$CAPTURE_PORT\" \\\n    --eval-lora --thinking --sandbox hf-sandbox --expose gradio \\\n    --model LiquidAI/LFM3-1.7B --max-model-len 131072 --eval-tag \"$SPLIT\" \\\n    --eval-harnesses pi,hermes,opencode \\\n    --eval-checkpoints \"${CHECKPOINT_LIST[@]}\" \\\n    --vllm-commit fac2ce2eb3633c44a104a66dbb30d0e6c7f4493e \\\n    --vllm-overlay \"$P/vllm/model_executor/models/lfm3.py\" \\\n    --vllm-overlay \"$P/vllm/model_executor/models/registry.py\" \\\n    --vllm-overlay \"$P/vllm/model_executor/layers/mamba/ops/causal_conv1d.py\" \\\n    --vllm-extra-args=--trust-remote-code \\\n    --tool-call-parser qwen3_coder --reasoning-parser qwen3 --max-inflight 32 \\\n    \"${SPLIT_ARGS[@]}\"\n",
    "recipe.yaml": "meta:\n  title: \"LFM3-1.7B harness adapters\"\n  project: harness-adapters\n  trainer: AsyncGRPOTrainer\n  model_id: LiquidAI/LFM3-1.7B\n  description: \"One LoRA adapter per agent harness (pi, hermes, opencode), trained with async GRPO inside the unmodified harness on verifiable Harbor tasks, with safeagents checking every tool call.\"\n  tags: [lora, agentic-rl, harbor, safeagents, multi-harness]\n  harnesses: [pi, hermes, opencode]\n  date_added: 2026-11-13\n  blog: https://huggingface.co/spaces/harness-hub/harness-adapters\n  dashboard: https://harness-hub-harness-adapters-rl.hf.space/\n\nhardware:\n  hf_jobs: h200x2            # vLLM on GPU 0, the three trainers sharing GPU 1\n  slurm: \"hopper-b200, 2 GPUs (untested)\"\n  local: \"2 GPUs, H200 class (untested)\"\n\ndatasets:\n  train:\n    - FineEnvs/SmolDataEnvs-harbor-train       # 300 medium + 250 hard\n    - harness-hub/traj2envs-rl@191a751bda511eada76f6a9a76513d143321a64a  # first 230 tasks\n  eval:\n    final-heldout: \"FineEnvs/SmolDataEnvs-harbor-train, 150 easy/medium tasks from index 1000\"\n    traj2envs-heldout: \"harness-hub/traj2envs-rl, the 40 tasks from index 230\"\n\nhyperparameters:\n  lora_r: 32\n  lora_target_modules: all-linear\n  learning_rate: 1.0e-5\n  num_generations: 8\n  gradient_accumulation_steps: 4\n  max_staleness: 4\n  temperature: 0.8\n  max_completion_length: 16384\n  max_output_tokens: 32768\n  max_model_len: 131072\n  token_budget: 65536\n  agent_step_limit: 17\n  agent_timeout_s: 600\n  max_inflight: 48\n  tool_efficiency_weight: 0.1\n  max_steps: 400\n  save_steps: 10\n\nvllm:\n  tool_call_parser: qwen3_coder\n  reasoning_parser: qwen3\n  trust_remote_code: true\n  commit: fac2ce2eb3633c44a104a66dbb30d0e6c7f4493e\n  overlays_from: https://github.com/atmikshetty/vllm/tree/5f67fbcbb2336b0e551c0df2b4c0eb51b030c553\n\ntrl:\n  fork: \"younesbelkada/trl@ee6e9ad9512b83c80bb0051942770c8798c0dbf2 (add-lfm-tree-attn)\"\n  upstream_base: 7fdad81d04303e6f43db168e903565769c0b8642\n  patch: trl.patch            # the fork's changes to trl/, relative to upstream_base\n\nprovenance:\n  hf_job: 6aedb4820e0338d1bb8107eb\n  released:\n    pi: {step: 400, repo: harness-hub/LFM3-1.7B-pi}\n    hermes: {step: 400, repo: harness-hub/LFM3-1.7B-hermes}\n    opencode: {step: 400, repo: harness-hub/LFM3-1.7B-opencode}\n\nscripts:\n  train: {hf_jobs: train_hf_jobs.sh, slurm: train_slurm.sh, local: train_local.sh}\n  eval: {hf_jobs: eval_hf_jobs.sh, slurm: eval_slurm.sh, local: eval_local.sh}\n\n# What the command builder on the recipes site offers. Each option is an environment variable of the script it runs.\nbuilder:\n  verified: {hf_jobs: true, slurm: false, local: false}   # whether the scripts of each platform ran as written\n  eval:\n    - env: SPLIT\n      label: Held-out split\n      choices: [final-heldout, traj2envs-heldout]\n    - env: CHECKPOINTS\n      label: Checkpoints\n      input: true\n      placeholder: \"<path|glob|hub id>@<harnesses> ... (empty: the script's default)\"\n",
    "trl.patch": "TRL changes these recipes run on, relative to upstream huggingface/trl.\n  upstream base: 7fdad81d04303e6f43db168e903565769c0b8642 (2026-10-28, fix: add peft on asyncgrpo openenv harbor + fixes (#7948))\n  fork:          younesbelkada/trl add-lfm-tree-attn = ee6e9ad9512b83c80bb0051942770c8798c0dbf2\nFork commits:\n  ee6e9ad9 Merge branch 'main' into add-lfm-tree-attn\n  c474c2cc Apply suggestion from @younesbelkada\n  eb5d5b9b Merge branch 'main' into add-lfm-tree-attn\n  2fa050c4 revert\n  b467898a feat: add LFM support for tree attn\nApply on a checkout of the upstream base: git apply trl.patch\nRegenerate: git diff $(git merge-base origin/main <fork commit>) <fork commit> -- trl/\n\ndiff --git a/trl/experimental/async_grpo/async_grpo_trainer.py b/trl/experimental/async_grpo/async_grpo_trainer.py\nindex 66a5d595..2e609d6f 100644\n--- a/trl/experimental/async_grpo/async_grpo_trainer.py\n+++ b/trl/experimental/async_grpo/async_grpo_trainer.py\n@@ -58,7 +58,7 @@ from ...trainer.utils import (\n )\n from .async_grpo_config import AsyncGRPOConfig\n from .async_rollout_worker import AsyncRolloutWorker, RolloutSample\n-from .packing import TREE_ATTENTION, TreeRow, patch_tree_gdn, register_tree_attention, unpad_tree\n+from .packing import TREE_ATTENTION, TreeRow, patch_tree_recurrent, register_tree_attention, unpad_tree\n from .vllm_client import VLLMClient\n from .weight_transfer import WeightTransferClient\n \n@@ -1122,11 +1122,17 @@ class AsyncGRPOTrainer(_BaseTrainer):\n                     \"before it, these layers call the functions they stored at init rather than the module-level \"\n                     \"ones tree packing patches, so they would scan across samples.\"\n                 )\n-            if layer_types != {\"linear_attention\"} or not patch_tree_gdn(model):\n+            if layer_types == {\"conv\"} and Version(transformers.__version__) <= Version(\"5.21.0.dev0\"):\n                 raise ValueError(\n-                    f\"{type(model).__name__} has {sorted(layer_types)} layers. Tree packing routes attention and \"\n-                    \"transformers' gated DeltaNet layers along each sample's own path; any other recurrent or \"\n-                    \"convolutional layer would scan across samples.\"\n+                    f\"Tree packing of {type(model).__name__}'s short convolution layers requires transformers>=5.22.0: \"\n+                    \"before it, these layers do not pass the forward's keyword arguments down to the module-level \"\n+                    \"function tree packing patches, so they would scan across samples.\"\n+                )\n+            if layer_types not in ({\"linear_attention\"}, {\"conv\"}) or not patch_tree_recurrent(model, *layer_types):\n+                raise ValueError(\n+                    f\"{type(model).__name__} has {sorted(layer_types)} layers. Tree packing routes attention, \"\n+                    \"transformers' gated DeltaNet layers and LFM2's short convolutions along each sample's own path; \"\n+                    \"any other recurrent or convolutional layer would scan across samples.\"\n                 )\n         if any(name.endswith(\".sinks\") for name, _ in model.named_parameters()):\n             raise ValueError(\ndiff --git a/trl/experimental/async_grpo/packing.py b/trl/experimental/async_grpo/packing.py\nindex c22132f3..24ed508d 100644\n--- a/trl/experimental/async_grpo/packing.py\n+++ b/trl/experimental/async_grpo/packing.py\n@@ -119,7 +119,7 @@ class TreeRow:\n                 - `tree_cu_seqlens_q` (`S + 1`), `tree_cu_seqlens_k` (`S + 1`): segment boundaries in the row, path\n                   boundaries in `tree_kv_index`.\n                 - `tree_segment_parents` (`S`), `tree_segment_depths` (`S`): the segment tree, for the recurrent layers\n-                  patched by [`patch_tree_gdn`].\n+                  patched by [`patch_tree_recurrent`].\n                 - `tree_sizes` (`4 + depth`): longest segment, longest path, `K`, 64-token chunks, then the segments\n                   per depth. Every length the kernels launch with, so a rank reads them in one go (see [`unpad_tree`]).\n                 - `token_index` (`M`): position whose prediction is each trained token. A shared position serves every\n@@ -278,20 +278,27 @@ def register_tree_attention() -> None:\n _PATCHED = set()\n \n \n-def patch_tree_gdn(model) -> bool:\n+def patch_tree_recurrent(model, layer_type: str) -> bool:\n     \"\"\"\n-    Make the model's gated DeltaNet layers follow the tree of a tree-packed forward.\n+    Make the model's gated DeltaNet or short convolution layers follow the tree of a tree-packed forward.\n \n-    Transformers' hybrids (Qwen3.5, Qwen3-Next, OLMo-Hybrid, ...) run these layers through two module-level functions,\n-    `causal_conv1d_fn` and `torch_chunk_gated_delta_rule`, and pass the forward's keyword arguments down to both. Both\n-    are wrapped: given the tree of [`~packing.TreeRow.pack`] they follow it, otherwise they run unchanged.\n+    Transformers' gated DeltaNet hybrids (Qwen3.5, Qwen3-Next, OLMo-Hybrid, ...) run their `\"linear_attention\"` layers\n+    through two module-level functions, `causal_conv1d_fn` and `torch_chunk_gated_delta_rule`, and LFM2's `\"conv\"`\n+    layers through the first alone. These layers pass the forward's keyword arguments down to them. Each is wrapped:\n+    given the tree of [`~packing.TreeRow.pack`] they follow it, otherwise they run unchanged.\n+\n+    Args:\n+        model ([`~transformers.PreTrainedModel`]):\n+            Model to patch.\n+        layer_type (`str`):\n+            The model's only non-attention layer type, `\"linear_attention\"` or `\"conv\"`.\n \n     Returns:\n-        `bool`: whether the model's modeling module has those functions.\n+        `bool`: whether the model's modeling module has the functions its layers run through.\n     \"\"\"\n     modeling = sys.modules[type(model).__module__]\n     conv, chunk = vars(modeling).get(\"causal_conv1d_fn\"), vars(modeling).get(\"torch_chunk_gated_delta_rule\")\n-    if conv is None or chunk is None:\n+    if conv is None or (layer_type == \"linear_attention\" and chunk is None):\n         return False\n     if modeling in _PATCHED:\n         return True\n@@ -307,6 +314,8 @@ def patch_tree_gdn(model) -> bool:\n             return chunk(query, key, value, g=g, beta=beta, use_qk_l2norm_in_kernel=use_qk_l2norm_in_kernel, **kwargs)\n         return tree_chunk_gated_delta_rule(query, key, value, g, beta, kwargs, use_qk_l2norm_in_kernel), None\n \n-    modeling.causal_conv1d_fn, modeling.torch_chunk_gated_delta_rule = causal_conv1d_fn, torch_chunk_gated_delta_rule\n+    modeling.causal_conv1d_fn = causal_conv1d_fn\n+    if chunk is not None:\n+        modeling.torch_chunk_gated_delta_rule = torch_chunk_gated_delta_rule\n     _PATCHED.add(modeling)\n     return True\ndiff --git a/trl/trainer/utils.py b/trl/trainer/utils.py\nindex 5e53bc49..e72efc9b 100644\n--- a/trl/trainer/utils.py\n+++ b/trl/trainer/utils.py\n@@ -1802,18 +1802,21 @@ def compute_flops_per_token(config: PretrainedConfig, seq_len: int) -> int:\n         mlp_flops = 2 * 3 * h * config.intermediate_size\n         total_layer_flops = L * (attn_flops + mlp_flops)\n     else:\n-        # Routed experts (gate + up + down, 3 matmuls each) + router.\n-        if Version(transformers.__version__) >= Version(\"5.1.0\"):\n-            num_experts = config.num_local_experts\n-        else:\n+        # Routed experts (gate + up + down, 3 matmuls each) + router. LFM2-MoE keeps `num_experts` on every version, and\n+        # opens with `num_dense_layers` dense layers where Qwen-style MoE spaces them by `decoder_sparse_step`.\n+        if config.model_type == \"lfm2_moe\":\n             num_experts = config.num_experts\n+            is_sparse = [layer_idx >= config.num_dense_layers for layer_idx in range(L)]\n+        else:\n+            if Version(transformers.__version__) >= Version(\"5.1.0\"):\n+                num_experts = config.num_local_experts\n+            else:\n+                num_experts = config.num_experts\n+            is_sparse = [layer_idx % config.decoder_sparse_step == 0 for layer_idx in range(L)]\n         moe_mlp_flops = num_experts_per_tok * 2 * 3 * h * config.moe_intermediate_size\n         moe_mlp_flops += 2 * h * num_experts\n         dense_mlp_flops = 2 * 3 * h * config.intermediate_size  # interspersed dense layers\n-        sparse_step = config.decoder_sparse_step\n-        total_layer_flops = sum(\n-            attn_flops + (moe_mlp_flops if layer_idx % sparse_step == 0 else dense_mlp_flops) for layer_idx in range(L)\n-        )\n+        total_layer_flops = sum(attn_flops + (moe_mlp_flops if sparse else dense_mlp_flops) for sparse in is_sparse)\n \n     # Embedding is a lookup (no FLOPs); lm_head is a 2*V*h matmul, tied or not.\n     lm_head_flops = 2 * V * h\n"
   },
   "other_files": []
  },
  {
   "id": "AsyncGRPOTrainer/harness-adapters/Qwen3.8-27B",
   "trainer": "AsyncGRPOTrainer",
   "project": "harness-adapters",
   "name": "Qwen3.8-27B",
   "org": "Qwen",
   "avatar": "https://cdn-avatars.huggingface.co/v1/production/uploads/6215ca5692c0ecfba9186921/hrRM50-6XcdWgg2AKpENG.jpeg",
   "meta": {
    "title": "Qwen3.8-27B harness adapters",
    "project": "harness-adapters",
    "trainer": "AsyncGRPOTrainer",
    "model_id": "Qwen/Qwen3.8-27B",
    "description": "One LoRA adapter per agent harness (pi, hermes, opencode) on Qwen3.8-27B, trained with async GRPO on agentic coding and terminal tasks (scale800 + TermGrade), one job per harness, thinking off.",
    "tags": [
     "lora",
     "agentic-rl",
     "harbor",
     "safeagents",
     "multi-harness",
     "live"
    ],
    "harnesses": [
     "pi",
     "hermes",
     "opencode"
    ],
    "date_added": "2026-11-13",
    "dashboard": "https://huggingface.co/spaces/harness-hub/qwen-livestream"
   },
   "hardware": {
    "slurm": "hopper-b200, 2 GPUs per harness",
    "local": "2 GPUs per harness, H200 or B200 class (untested)",
    "hf_jobs": "h200x2 per harness (untested)"
   },
   "datasets": {
    "train": [
     "harness-hub/scale800-harbor@56c7f75d01f70859ec1354e97866e32a175c1a40",
     "harness-hub/termgrade-train-harbor"
    ],
    "eval": {
     "sbp-hard51": "harness-hub/swebench-pro-v2-hard51-harbor",
     "tb21-20pct": "harborframework/terminal-bench-2.1@3e235dff6880252a587fa479c09bcb1e16edf2eb, 18 tasks",
     "s800-heldout": "harness-hub/scale800-heldout-harbor (the repositories left out of training), 1800 s and 50 steps as in training",
     "automationbench": "Zapier AutomationBench, 10 tasks per domain (60), as the base on the dashboard"
    }
   },
   "hyperparameters": {
    "lora_r": 32,
    "lora_target_modules": [
     "q_proj",
     "k_proj",
     "v_proj",
     "o_proj",
     "gate_proj",
     "up_proj",
     "down_proj"
    ],
    "learning_rate": 2e-05,
    "num_generations": 8,
    "gradient_accumulation_steps": 16,
    "max_staleness": 16,
    "temperature": 0.8,
    "max_completion_length": 16384,
    "max_model_len": 65536,
    "token_budget": 65536,
    "agent_step_limit": 50,
    "agent_timeout_s": 1800,
    "max_inflight": 64,
    "tool_efficiency_weight": 0.1,
    "max_steps": 1000,
    "save_steps": 10,
    "thinking": false
   },
   "vllm": {
    "tool_call_parser": "qwen3_xml",
    "reasoning_parser": "qwen3",
    "extra_args": [
     "--language-model-only",
     "--enable-prefix-caching",
     "--mamba-cache-mode align"
    ]
   },
   "trl": {
    "fork": "younesbelkada/trl@e498acc637084cffff14deb09ccdc423a766adb2 (async-grpo-shared-vllm)",
    "upstream_base": "d115286bc5aef8d346e8510bfe312803c7881c4a",
    "patch": "trl.patch"
   },
   "provenance": {
    "slurm_jobs": {
     "pi": 374053,
     "hermes": 374054,
     "opencode": 374055
    },
    "status": "training"
   },
   "scripts": {
    "train": {
     "hf_jobs": "train_hf_jobs.sh",
     "slurm": "train_slurm.sh",
     "local": "train_local.sh"
    },
    "eval": {
     "hf_jobs": "eval_hf_jobs.sh",
     "slurm": "eval_slurm.sh",
     "local": "eval_local.sh"
    },
    "dashboard": "../scripts/sync_live.sh"
   },
   "builder": {
    "verified": {
     "hf_jobs": false,
     "slurm": true,
     "local": false
    },
    "train": [
     {
      "env": "HARNESS",
      "label": "Harnesses",
      "choices": [
       "pi",
       "hermes",
       "opencode"
      ],
      "multi": true,
      "hf_jobs_env": "HARNESSES",
      "platforms": [
       "hf_jobs",
       "slurm"
      ]
     },
     {
      "env": "HARNESS",
      "label": "Harness",
      "choices": [
       "pi",
       "hermes",
       "opencode"
      ],
      "platforms": [
       "local"
      ]
     }
    ],
    "eval": [
     {
      "env": "BENCH",
      "label": "Benchmark",
      "choices": [
       "sbp",
       "tb",
       "s800",
       {
        "value": "automationbench",
        "platforms": [
         "hf_jobs",
         "local"
        ]
       }
      ]
     },
     {
      "env": "POLICIES",
      "label": "Policies",
      "input": true,
      "default": "Qwen/Qwen3.8-27B@pi,hermes,opencode",
      "unless": {
       "BENCH": "automationbench"
      }
     },
     {
      "env": "CHECKPOINTS",
      "label": "Checkpoints",
      "input": true,
      "default": "base",
      "when": {
       "BENCH": "automationbench"
      }
     }
    ]
   },
   "readme": "# Qwen3.8-27B harness adapters\n\nLoRA adapters for pi, hermes and opencode on [`Qwen/Qwen3.8-27B`](https://huggingface.co/Qwen/Qwen3.8-27B), trained on\nthe cluster for up to 1,000 steps. Training data is the 750 scale800 tasks interleaved with 300 TermGrade tasks. Live\ndashboard: [harness-hub/qwen-livestream](https://huggingface.co/spaces/harness-hub/qwen-livestream). Setup and shared\nsettings: [../README.md](../README.md). Full hyperparameters: [recipe.yaml](recipe.yaml).\n\n| | Cluster | HF Jobs (untested) | Local (untested) |\n|---|---|---|---|\n| Train | `HARNESS=pi sbatch train_slurm.sh`, one job per harness (2 GPUs) | `bash train_hf_jobs.sh`, one h200x2 job per harness | `HARNESS=pi bash train_local.sh`, one harness per machine (2 GPUs) |\n| Eval | `BENCH=sbp\\|tb\\|s800 POLICIES=... sbatch eval_slurm.sh` (1 GPU) | `BENCH=sbp\\|tb\\|s800\\|automationbench ... bash eval_hf_jobs.sh` (h200) | `BENCH=sbp\\|tb\\|s800\\|automationbench ... bash eval_local.sh` (1 GPU) |\n\nHow this differs from the LFM / TwIL recipes:\n- **Training script and TRL.** This folder has its own [async_grpo_harbor.py](async_grpo_harbor.py), built on\n  `HarnessAsyncGRPOTrainer`. That trainer comes from TRL's `async-grpo-shared-vllm` branch\n  (`younesbelkada/trl@e498acc6`), not the launcher's pinned TRL. Its changes relative to upstream are in\n  [trl.patch](trl.patch).\n  - On the cluster, the venv has that branch installed editable.\n  - On HF Jobs, `--trl-wheel` installs it from git.\n  - The cluster checkout also has an uncommitted `rollout/solved_tool_calls_mean` metric. The HF Jobs install leaves\n    it out.\n- **One job per harness**, not one job for all three.\n- **Thinking off.** Each run has 50 agent steps and 1800 s, and `--no-responses-thinking`.\n- **vLLM settings:**\n  - `--language-model-only`, plus prefix caching with `--mamba-cache-mode align`, which the gated-DeltaNet layers\n    need.\n  - On B200, vLLM runs from its own venv (`$ROOT/venv-vllm`, Triton 3.6.0), because the trainer's Triton 3.7.1 kills\n    vLLM there.\n- **Safeagents plugin.** It is the generic Harbor plugin, `scripts/atif_harbor.py`.\n\n## Cluster\n\n`ROOT` defaults to the existing TermGrade setup, `/fsx/$USER/termgrade`, which holds `venv`, `venv-vllm`, `hf-hub` and\n`data`. Copy this project next to them once from your laptop:\n\n```sh\nrsync -a AsyncGRPOTrainer/harness-adapters/ <login node>:/fsx/<user>/termgrade/\n```\n\nThe scripts then read `$ROOT/scripts/` and `$ROOT/Qwen3.8-27B/`. The final runs were launched as follows (the\n3-minute stagger keeps the jobs from starting sandboxes all at once):\n\n```sh\nI=0; for H in pi hermes opencode; do HARNESS=$H sbatch --begin=now+$((I * 180)) train_slurm.sh; I=$((I + 1)); done\n```\n\nTo resume a run, pass the checkpoint through `TRAIN_EXTRA`:\n`HARNESS=pi TRAIN_EXTRA=\"--resume-from-checkpoint $ROOT/data/runs/<run>/checkpoint-200\" sbatch train_slurm.sh`.\n\nEvals compare each checkpoint with a base eval run the same way, under the same tag: run the base once per benchmark\n(`POLICIES=\"Qwen/Qwen3.8-27B@pi,hermes,opencode\"`). `s800` is the scale800 held-out set: the repositories left out of\ntraining, under training's own limits (1800 s, 50 steps).\n\nThe dashboard reads the bucket, which [../scripts/sync_live.sh](../scripts/sync_live.sh) keeps current from a login\nnode (`tmux new -d -s live \"bash $ROOT/scripts/sync_live.sh\"`).\n\n## Earlier runs (same scripts, other settings)\n\n| Slurm jobs | Harnesses | Learning rate | Data |\n|---|---|---|---|\n| 372966, 372967 | hermes, opencode | 1e-5 | all 1,000 TermGrade tasks (`SPLIT=harness-hub/termgrade-train-harbor`, no mix) |\n| 373528\u2013373530 | pi, hermes, opencode | 5e-5 | TermGrade + 50 agentic-coding-envs tasks |\n| 374053\u2013374055 | pi, hermes, opencode | 2e-5 | scale800 + 300 TermGrade (this recipe) |\n",
   "files": {
    "train_local.sh": "#!/bin/bash\n# Qwen3.8-27B harness adapter for one harness on one local machine with 2 GPUs: one vLLM (GPU 0) serving the base and\n# the adapter versions, and one LoRA AsyncGRPO trainer (GPU 1). The 27B trainer needs ~140 GB: H200 or B200 class.\n#\n# The settings of the final cluster runs (train_slurm.sh), with the HF Jobs install (train_hf_jobs.sh): TRL from the\n# async-grpo-shared-vllm commit and Triton 3.7.1, run here with `uv run`. Not run locally yet. Runs are written to\n# $DATA_ROOT/runs/. On B200, vLLM needs its own Triton (see train_slurm.sh): use the cluster setup there.\n#\n#   HARNESS=pi bash train_local.sh\n#   HARNESS=pi MAX_STEPS=5 MAX_INFLIGHT=16 bash train_local.sh            short smoke run\nset -euo pipefail\n\nHARNESS=\"${HARNESS:?HARNESS=pi, hermes or opencode}\"\nDATA_ROOT=\"${DATA_ROOT:-$PWD/outputs}\"\nSPLIT=\"${SPLIT:-harness-hub/scale800-harbor@56c7f75d01f70859ec1354e97866e32a175c1a40}\"\nLEARNING_RATE=\"${LEARNING_RATE:-2e-5}\"\nMAX_INFLIGHT=\"${MAX_INFLIGHT:-64}\"\nMAX_STALENESS=\"${MAX_STALENESS:-16}\"\nGRAD_ACCUM=\"${GRAD_ACCUM:-16}\"\nTOKEN_BUDGET=\"${TOKEN_BUDGET:-65536}\"  # the 27B trainer peaked at 138 GiB of 141 on an H200 with this\nMAX_STEPS=\"${MAX_STEPS:-1000}\"\nMAX_NUM_SEQS=$((2 * MAX_INFLIGHT))\nMAX_LORAS=$((MAX_STALENESS + 2))\n# HarnessAsyncGRPOTrainer, which this training script uses, is on the async-grpo-shared-vllm branch, not in the\n# launcher's pinned TRL\nTRL=\"${TRL:-trl @ git+https://github.com/younesbelkada/trl.git@e498acc637084cffff14deb09ccdc423a766adb2}\"\nTRACKIO_SPACE_ID=\"${TRACKIO_SPACE_ID:-ybelkada/async-grpo-harbor}\"\nHERE=\"$(cd \"$(dirname \"$0\")\" && pwd)\"\nSCRIPTS=\"$HERE/../scripts\"\nexport HF_TOKEN=\"${HF_TOKEN:-$(cat ~/.cache/huggingface/token)}\"  # also what the HF sandboxes run under\n# Names this run's folders (logs/<id>, runs/<model>-<harness>-<id>-...), as HF_JOB_ID does on HF Jobs\nexport JOB_ID=\"${JOB_ID:-local-$(date +%Y%m%d-%H%M%S)}\"\nexport PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True\n\n# triton 3.7.1: 3.6.0's gated-DeltaNet backward is wrong on Hopper (fla #640)\nuv run \"$SCRIPTS/launcher_train.py\" \\\n    --data-root \"$DATA_ROOT\" --trl-wheel \"$TRL\" \\\n    --pip-install \"triton==3.7.1 safeagents@git+https://github.com/huggingface/safeagents@5e683fa atif-scan==0.8.15 rich markdown-it-py mdurl pygments\" \\\n    --train-script-url \"file://$HERE/async_grpo_harbor.py\" \\\n    --model Qwen/Qwen3.8-27B --harness \"$HARNESS\" --split \"$SPLIT\" --n-tasks 1000 \\\n    --mix-split harness-hub/termgrade-train-harbor:300 --interleave \\\n    --sandbox hf-sandbox --expose gradio \\\n    --vllm-device 0 --train-device 1 \\\n    --tool-call-parser qwen3_xml --reasoning-parser qwen3 --max-model-len \"$TOKEN_BUDGET\" \\\n    --vllm-extra-args=\"--language-model-only --enable-prefix-caching --mamba-cache-mode align --max-num-seqs $MAX_NUM_SEQS --default-chat-template-kwargs '{\\\"enable_thinking\\\": false}'\" \\\n    --max-inflight \"$MAX_INFLIGHT\" --max-lora-rank 32 --max-loras \"$MAX_LORAS\" \\\n    --no-responses-thinking \\\n    --reward-key correctness,reward --num-generations 8 \\\n    --gradient-accumulation-steps \"$GRAD_ACCUM\" --max-staleness \"$MAX_STALENESS\" --token-budget \"$TOKEN_BUDGET\" \\\n    --max-completion-length 16384 --agent-step-limit 50 --agent-timeout 1800 --temperature 0.8 \\\n    --lora-r 32 --lora-target-modules q_proj,k_proj,v_proj,o_proj,gate_proj,up_proj,down_proj \\\n    --learning-rate \"$LEARNING_RATE\" --tool-efficiency-weight 0.1 \\\n    --safeagents --safeagents-plugin \"$SCRIPTS/atif_harbor.py:checks\" \\\n    --max-steps \"$MAX_STEPS\" --save-steps 10 --save-total-limit 0 \\\n    --trackio-space-id \"$TRACKIO_SPACE_ID\"\n",
    "train_hf_jobs.sh": "#!/bin/bash\n# Qwen3.8-27B harness adapters on HF Jobs: one h200x2 job per harness, each with one vLLM (GPU 0) serving the base and\n# the adapter versions, and one LoRA AsyncGRPO trainer (GPU 1).\n#\n# The settings of the final cluster runs (train_slurm.sh, Slurm jobs 374053/54/55), moved to HF Jobs. Not run in this\n# exact form: the final runs were on the cluster, and the HF Jobs Qwen3.8 runs before them used SmolDataEnvs.\n#\n#   bash train_hf_jobs.sh                                   pi, hermes and opencode, one job each\n#   HARNESSES=pi MAX_STEPS=5 MAX_INFLIGHT=16 bash train_hf_jobs.sh\nset -euo pipefail\n\nBUCKET=\"${BUCKET:-ybelkada/harbor-evals}\"\nHARNESSES=\"${HARNESSES:-pi hermes opencode}\"\nSPLIT=\"${SPLIT:-harness-hub/scale800-harbor@56c7f75d01f70859ec1354e97866e32a175c1a40}\"\nLEARNING_RATE=\"${LEARNING_RATE:-2e-5}\"\nMAX_INFLIGHT=\"${MAX_INFLIGHT:-64}\"\nMAX_STALENESS=\"${MAX_STALENESS:-16}\"\nGRAD_ACCUM=\"${GRAD_ACCUM:-16}\"\nTOKEN_BUDGET=\"${TOKEN_BUDGET:-65536}\"  # the 27B trainer peaked at 138 GiB of 141 on an H200 with this\nMAX_STEPS=\"${MAX_STEPS:-1000}\"\nMAX_NUM_SEQS=$((2 * MAX_INFLIGHT))\nMAX_LORAS=$((MAX_STALENESS + 2))\n# HarnessAsyncGRPOTrainer, which this training script uses, is on the async-grpo-shared-vllm branch, not in the\n# launcher's pinned TRL\nTRL=\"${TRL:-trl @ git+https://github.com/younesbelkada/trl.git@e498acc637084cffff14deb09ccdc423a766adb2}\"\nTRACKIO_SPACE_ID=\"${TRACKIO_SPACE_ID:-ybelkada/async-grpo-harbor}\"\nHERE=\"$(cd \"$(dirname \"$0\")\" && pwd)\"\nSCRIPTS=\"$HERE/../scripts\"\n\n# The job reads the training script and the safeagents plugin from the bucket, mounted on /outputs. The Qwen training\n# script differs from the LFM / TwIL one, so it gets its own name there.\nhf buckets cp \"$HERE/async_grpo_harbor.py\" \"hf://buckets/$BUCKET/qwen3.8_async_grpo_harbor.py\"\nhf buckets cp \"$SCRIPTS/atif_harbor.py\" \"hf://buckets/$BUCKET/atif_harbor.py\"\n\nfor HARNESS in $HARNESSES; do\n    # triton 3.7.1: 3.6.0's gated-DeltaNet backward is wrong on Hopper (fla #640)\n    hf jobs uv run --detach \\\n        --flavor h200x2 --image huggingface/trl --secrets HF_TOKEN --timeout 72h \\\n        --env PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True \\\n        --volume \"hf://buckets/$BUCKET:/outputs\" \\\n        \"$SCRIPTS/launcher_train.py\" -- \\\n        --data-root /outputs --trl-wheel \"$TRL\" \\\n        --pip-install \"triton==3.7.1 safeagents@git+https://github.com/huggingface/safeagents@5e683fa atif-scan==0.8.15 rich markdown-it-py mdurl pygments\" \\\n        --train-script-url file:///outputs/qwen3.8_async_grpo_harbor.py \\\n        --model Qwen/Qwen3.8-27B --harness \"$HARNESS\" --split \"$SPLIT\" --n-tasks 1000 \\\n        --mix-split harness-hub/termgrade-train-harbor:300 --interleave \\\n        --sandbox hf-sandbox --expose gradio \\\n        --vllm-device 0 --train-device 1 \\\n        --tool-call-parser qwen3_xml --reasoning-parser qwen3 --max-model-len \"$TOKEN_BUDGET\" \\\n        --vllm-extra-args=\"--language-model-only --enable-prefix-caching --mamba-cache-mode align --max-num-seqs $MAX_NUM_SEQS --default-chat-template-kwargs '{\\\"enable_thinking\\\": false}'\" \\\n        --max-inflight \"$MAX_INFLIGHT\" --max-lora-rank 32 --max-loras \"$MAX_LORAS\" \\\n        --no-responses-thinking \\\n        --reward-key correctness,reward --num-generations 8 \\\n        --gradient-accumulation-steps \"$GRAD_ACCUM\" --max-staleness \"$MAX_STALENESS\" --token-budget \"$TOKEN_BUDGET\" \\\n        --max-completion-length 16384 --agent-step-limit 50 --agent-timeout 1800 --temperature 0.8 \\\n        --lora-r 32 --lora-target-modules q_proj,k_proj,v_proj,o_proj,gate_proj,up_proj,down_proj \\\n        --learning-rate \"$LEARNING_RATE\" --tool-efficiency-weight 0.1 \\\n        --safeagents --safeagents-plugin /outputs/atif_harbor.py:checks \\\n        --max-steps \"$MAX_STEPS\" --save-steps 10 --save-total-limit 0 \\\n        --trackio-space-id \"$TRACKIO_SPACE_ID\"\ndone\n",
    "train_slurm.sh": "#!/bin/bash -l\n# Qwen3.8-27B harness adapter for one harness on one hopper-b200 node: one vLLM (GPU 0) serving the base and the\n# adapter versions, and one LoRA AsyncGRPO trainer (GPU 1). One job per harness.\n#\n# The settings of the final livestream runs (Slurm jobs 374053/54/55, 2026-11-13): scale800 interleaved with 300\n# TermGrade tasks, learning rate 2e-5, 16 token-budgeted micro-batches per step.\n#\n#   I=0; for H in pi hermes opencode; do\n#       HARNESS=$H sbatch --begin=now+$((I * 180)) train_slurm.sh; I=$((I + 1))\n#   done\n#   HARNESS=pi MAX_STEPS=5 MAX_INFLIGHT=16 sbatch train_slurm.sh                   short smoke run\n#\n# The 3-minute stagger keeps the three jobs from preparing tasks and starting sandboxes all at once.\n# hopper-b200's Capacity Block ends 2026-11-19 11:30 UTC and running jobs die with it: keep --time inside it.\n#SBATCH --job-name=ha-qwen3.8\n#SBATCH --partition=hopper-b200\n#SBATCH --nodes=1\n#SBATCH --ntasks-per-node=1\n#SBATCH --gpus-per-node=2\n#SBATCH --cpus-per-task=32\n#SBATCH --time=3-00:00:00\n#SBATCH --output=/fsx/%u/logs/%x-%j.out\nset -euo pipefail\n\n# The TermGrade setup on /fsx: venv (TRL from the async-grpo-shared-vllm branch, editable), venv-vllm, hf-hub, data/\nROOT=\"${ROOT:-/fsx/$USER/termgrade}\"\nHARNESS=\"${HARNESS:?HARNESS=pi, hermes or opencode}\"\nSPLIT=\"${SPLIT:-harness-hub/scale800-harbor@56c7f75d01f70859ec1354e97866e32a175c1a40}\"\nLEARNING_RATE=\"${LEARNING_RATE:-2e-5}\"\nMAX_INFLIGHT=\"${MAX_INFLIGHT:-64}\"  # sandboxes at once\n# A rollout can last up to the 1800 s agent limit, many steps at this rate: past MAX_STALENESS versions it is dropped\nMAX_STALENESS=\"${MAX_STALENESS:-16}\"\nGRAD_ACCUM=\"${GRAD_ACCUM:-16}\"  # token-budgeted micro-batches per step: more groups per step\nTOKEN_BUDGET=\"${TOKEN_BUDGET:-65536}\"  # the 27B trainer peaked at 138 GiB of 141 on an H200 with this; a B200 has 180\nMAX_STEPS=\"${MAX_STEPS:-1000}\"\n# Extra training-script arguments, e.g. TRAIN_EXTRA=\"--resume-from-checkpoint $ROOT/data/runs/<run>/checkpoint-20\"\nTRAIN_EXTRA=\"${TRAIN_EXTRA:-}\"\nMAX_NUM_SEQS=$((2 * MAX_INFLIGHT))  # each rollout has one model call in flight at a time\nMAX_LORAS=$((MAX_STALENESS + 2))  # the versions requests may still name, plus the one being loaded\nsource \"$ROOT/scripts/slurm_setup.sh\"\n\n# vLLM from its own venv, with the Triton torch 2.11 pins (3.6.0). The main venv has 3.7.1 for the trainer (3.6.0's\n# gated-DeltaNet backward is wrong on Hopper, fla #640), which kills vLLM on B200 (a worker dies warming up Qwen's\n# Triton kernels). The launcher runs the first `vllm` on PATH, so only that command is put first.\nmkdir -p vllm-bin\nln -sf \"$(dirname \"$(venv-load \"$ROOT/venv-vllm\")\")/vllm\" vllm-bin/vllm\nexport PATH=\"$PWD/vllm-bin:$PATH\"\n\n# Prefix caching needs both flags on this hybrid model: without a mamba cache mode the gated-DeltaNet layers keep no\n# reusable state, and every agent turn re-reads its whole history (95% hit rate, 3x the generation throughput).\n# --no-responses-thinking: hermes speaks the Responses API, whose OpenEnv dialect turns thinking on, unlike the\n# chat-completions harnesses; with it on, every hermes rollout hit the 1800 s agent limit. Off, all harnesses run\n# non-thinking alike.\npython \"$ROOT/scripts/launcher_train.py\" \\\n    --data-root \"$ROOT/data\" \\\n    --train-script-url \"file://$ROOT/Qwen3.8-27B/async_grpo_harbor.py\" \\\n    --model Qwen/Qwen3.8-27B --harness \"$HARNESS\" --split \"$SPLIT\" --n-tasks 1000 \\\n    --mix-split harness-hub/termgrade-train-harbor:300 --interleave \\\n    --sandbox hf-sandbox --expose gradio \\\n    --vllm-port \"$VLLM_PORT\" --server-port \"$SERVER_PORT\" --capture-port \"$CAPTURE_PORT\" \\\n    --vllm-device 0 --train-device 1 \\\n    --tool-call-parser qwen3_xml --reasoning-parser qwen3 --max-model-len \"$TOKEN_BUDGET\" \\\n    --vllm-extra-args=\"--language-model-only --enable-prefix-caching --mamba-cache-mode align --max-num-seqs $MAX_NUM_SEQS --default-chat-template-kwargs '{\\\"enable_thinking\\\": false}'\" \\\n    --max-inflight \"$MAX_INFLIGHT\" --max-lora-rank 32 --max-loras \"$MAX_LORAS\" \\\n    --no-responses-thinking \\\n    --reward-key correctness,reward --num-generations 8 \\\n    --gradient-accumulation-steps \"$GRAD_ACCUM\" --max-staleness \"$MAX_STALENESS\" --token-budget \"$TOKEN_BUDGET\" \\\n    --max-completion-length 16384 --agent-step-limit 50 --agent-timeout 1800 --temperature 0.8 \\\n    --lora-r 32 --lora-target-modules q_proj,k_proj,v_proj,o_proj,gate_proj,up_proj,down_proj \\\n    --learning-rate \"$LEARNING_RATE\" --tool-efficiency-weight 0.1 \\\n    --safeagents --safeagents-plugin \"$ROOT/scripts/atif_harbor.py:checks\" \\\n    --max-steps \"$MAX_STEPS\" --save-steps 10 --save-total-limit 0 $TRAIN_EXTRA\n",
    "eval_local.sh": "#!/bin/bash\n# Evaluate Qwen3.8-27B policies on one local GPU (H200 or B200 class), with `uv run`. Results land in\n# $DATA_ROOT/eval/. Same settings as eval_hf_jobs.sh; not run locally yet.\n#\n#   BENCH=sbp|tb|s800 each LoRA checkpoint merged into the base and served alone, one at a time.\n#                     POLICIES: `<checkpoint path or Hub id>@<harnesses>`, space-separated\n#   BENCH=automationbench\n#                     Zapier AutomationBench with its own agent loop (no sandboxes): one vLLM serves the base and every\n#                     adapter (eval_automationbench.py). CHECKPOINTS: `base` and/or checkpoint paths, space-separated\n#\n#   BENCH=sbp POLICIES=\"Qwen/Qwen3.8-27B@pi,hermes,opencode\" bash eval_local.sh\n#   BENCH=automationbench CHECKPOINTS=\"base $PWD/outputs/runs/<run>/checkpoint-50\" bash eval_local.sh\nset -euo pipefail\n\nDATA_ROOT=\"${DATA_ROOT:-$PWD/outputs}\"\nBENCH=\"${BENCH:?BENCH=sbp, tb, s800 or automationbench}\"\n# scale800 held-out: the tasks of the repositories left out of training (all of them), with training's own agent limits,\n# since it measures what training produced. Set S800_HELDOUT to the held-out dataset, pinned like the training one.\nS800_HELDOUT=\"${S800_HELDOUT:-harness-hub/scale800-heldout-harbor}\"\nHERE=\"$(cd \"$(dirname \"$0\")\" && pwd)\"\nSCRIPTS=\"$HERE/../scripts\"\nexport HF_TOKEN=\"${HF_TOKEN:-$(cat ~/.cache/huggingface/token)}\"\nexport JOB_ID=\"${JOB_ID:-local-$(date +%Y%m%d-%H%M%S)}\"\n\nif [ \"$BENCH\" = automationbench ]; then\n    CHECKPOINTS=\"${CHECKPOINTS:-base}\"\n    read -r -a CHECKPOINT_LIST <<< \"$CHECKPOINTS\"\n    # Resumable: a run stopped early keeps every finished chunk, and the same command runs only the rest.\n    # 10 tasks per domain (60), the size of the base's eval on the dashboard: a checkpoint is only comparable with a\n    # base run on the same tasks\n    uv run \"$HERE/eval_automationbench.py\" \\\n        --data-root \"$DATA_ROOT\" \\\n        --model Qwen/Qwen3.8-27B --checkpoints \"${CHECKPOINT_LIST[@]}\" --per-domain \"${PER_DOMAIN:-10}\" \\\n        --vllm-extra-args=\"--language-model-only --max-num-seqs 64\"\n    exit 0\nfi\n\nPOLICIES=\"${POLICIES:?POLICIES=\\\"<checkpoint>@<harness> ...\\\"}\"\ncase \"$BENCH\" in\n    sbp)\n        TAG=\"${TAG:-sbp-hard51-solo}\"\n        BENCH_ARGS=(--split harness-hub/swebench-pro-v2-hard51-harbor --n-tasks 51 --sandbox-memory-mb 4096\n            --agent-step-limit 100 --agent-timeout 7200)\n        ;;\n    tb)\n        TAG=\"${TAG:-tb21-20pct-solo}\"\n        BENCH_ARGS=(--split harborframework/terminal-bench-2.1@3e235dff6880252a587fa479c09bcb1e16edf2eb\n            --difficulty easy:1,medium:11,hard:6 --n-tasks 18 --sandbox-memory-mb 8192\n            --agent-step-limit 100 --agent-timeout 3600)\n        ;;\n    s800)\n        TAG=\"${TAG:-s800-heldout-solo}\"\n        BENCH_ARGS=(--split \"$S800_HELDOUT\" --n-tasks 1000 --sandbox-memory-mb 8192\n            --agent-step-limit 50 --agent-timeout 1800)\n        ;;\n    *)\n        echo \"unknown BENCH=$BENCH: sbp, tb, s800 or automationbench\" >&2\n        exit 1\n        ;;\nesac\nread -r -a POLICY_LIST <<< \"$POLICIES\"\n\nuv run \"$SCRIPTS/launcher_eval.py\" \\\n    --data-root \"$DATA_ROOT\" --eval-script-url \"file://$SCRIPTS/evaluate_harbor.py\" \\\n    --model Qwen/Qwen3.8-27B --eval-checkpoints \"${POLICY_LIST[@]}\" --eval-tag \"$TAG\" \"${BENCH_ARGS[@]}\" \\\n    --sandbox hf-sandbox --expose gradio --max-inflight 32 --max-model-len 131072 \\\n    --vllm-device 0 \\\n    --tool-call-parser qwen3_xml --reasoning-parser qwen3 --no-responses-thinking \\\n    --vllm-extra-args=\"--max-num-seqs 64 --enable-prefix-caching --mamba-cache-mode align\"\n",
    "eval_hf_jobs.sh": "#!/bin/bash\n# Evaluate Qwen3.8-27B policies on HF Jobs (1x H200).\n#\n#   BENCH=sbp|tb|s800 as eval_slurm.sh: each LoRA checkpoint merged into the base and served alone, one at a time.\n#                     POLICIES: `<checkpoint path or Hub id>@<harnesses>`, space-separated\n#   BENCH=automationbench\n#                     Zapier AutomationBench with its own agent loop (no sandboxes): one vLLM serves the base and every\n#                     adapter (eval_automationbench.py). CHECKPOINTS: `base` and/or checkpoint paths, space-separated\n#\n#   BENCH=sbp POLICIES=\"Qwen/Qwen3.8-27B@pi,hermes,opencode\" bash eval_hf_jobs.sh\n#   BENCH=automationbench CHECKPOINTS=\"base /outputs/runs/<run>/checkpoint-50\" bash eval_hf_jobs.sh\n#\n# The sbp / tb settings are those of the cluster evals, not run on HF Jobs in this form. AutomationBench ran on HF Jobs.\nset -euo pipefail\n\nBUCKET=\"${BUCKET:-ybelkada/harbor-evals}\"\nBENCH=\"${BENCH:?BENCH=sbp, tb, s800 or automationbench}\"\n# scale800 held-out: the tasks of the repositories left out of training (all of them), with training's own agent limits,\n# since it measures what training produced. Set S800_HELDOUT to the held-out dataset, pinned like the training one.\nS800_HELDOUT=\"${S800_HELDOUT:-harness-hub/scale800-heldout-harbor}\"\nHERE=\"$(cd \"$(dirname \"$0\")\" && pwd)\"\nSCRIPTS=\"$HERE/../scripts\"\n\nif [ \"$BENCH\" = automationbench ]; then\n    CHECKPOINTS=\"${CHECKPOINTS:?CHECKPOINTS=\\\"base /outputs/runs/<run>/checkpoint-N ...\\\"}\"\n    read -r -a CHECKPOINT_LIST <<< \"$CHECKPOINTS\"\n    # Resumable: a job that hits its timeout keeps every finished chunk, and the same command runs only the rest.\n    # 10 tasks per domain (60), the size of the base's eval on the dashboard (Qwen_Qwen3.8-27B_automationbench): a\n    # checkpoint is only comparable with a base run on the same tasks\n    hf jobs uv run --detach \\\n        --flavor h200 --image huggingface/trl --secrets HF_TOKEN --timeout 4h \\\n        --volume \"hf://buckets/$BUCKET:/outputs\" \\\n        \"$HERE/eval_automationbench.py\" -- \\\n        --model Qwen/Qwen3.8-27B --checkpoints \"${CHECKPOINT_LIST[@]}\" --per-domain \"${PER_DOMAIN:-10}\" \\\n        --vllm-extra-args=\"--language-model-only --max-num-seqs 64\"\n    exit 0\nfi\n\nPOLICIES=\"${POLICIES:?POLICIES=\\\"<checkpoint>@<harness> ...\\\"}\"\ncase \"$BENCH\" in\n    sbp)\n        TAG=\"${TAG:-sbp-hard51-solo}\"\n        BENCH_ARGS=(--split harness-hub/swebench-pro-v2-hard51-harbor --n-tasks 51 --sandbox-memory-mb 4096\n            --agent-step-limit 100 --agent-timeout 7200)\n        ;;\n    tb)\n        TAG=\"${TAG:-tb21-20pct-solo}\"\n        BENCH_ARGS=(--split harborframework/terminal-bench-2.1@3e235dff6880252a587fa479c09bcb1e16edf2eb\n            --difficulty easy:1,medium:11,hard:6 --n-tasks 18 --sandbox-memory-mb 8192\n            --agent-step-limit 100 --agent-timeout 3600)\n        ;;\n    s800)\n        TAG=\"${TAG:-s800-heldout-solo}\"\n        BENCH_ARGS=(--split \"$S800_HELDOUT\" --n-tasks 1000 --sandbox-memory-mb 8192\n            --agent-step-limit 50 --agent-timeout 1800)\n        ;;\n    *)\n        echo \"unknown BENCH=$BENCH: sbp, tb, s800 or automationbench\" >&2\n        exit 1\n        ;;\nesac\nread -r -a POLICY_LIST <<< \"$POLICIES\"\n\n# The job reads the eval client from the bucket, mounted on /outputs\nhf buckets cp \"$SCRIPTS/evaluate_harbor.py\" \"hf://buckets/$BUCKET/evaluate_harbor.py\"\n\nhf jobs uv run --detach \\\n    --flavor h200 --image huggingface/trl --secrets HF_TOKEN --timeout 14h \\\n    --volume \"hf://buckets/$BUCKET:/outputs\" \\\n    \"$SCRIPTS/launcher_eval.py\" -- \\\n    --data-root /outputs \\\n    --model Qwen/Qwen3.8-27B --eval-checkpoints \"${POLICY_LIST[@]}\" --eval-tag \"$TAG\" \"${BENCH_ARGS[@]}\" \\\n    --sandbox hf-sandbox --expose gradio --max-inflight 32 --max-model-len 131072 \\\n    --vllm-device 0 \\\n    --tool-call-parser qwen3_xml --reasoning-parser qwen3 --no-responses-thinking \\\n    --vllm-extra-args=\"--max-num-seqs 64 --enable-prefix-caching --mamba-cache-mode align\"\n",
    "eval_slurm.sh": "#!/bin/bash -l\n# Evaluate Qwen3.8-27B policies one at a time on one hopper-b200 GPU: each LoRA checkpoint is merged into its base and\n# served alone by vLLM, with prefix caching, as the base model is. No policy shares the server, so a score does not\n# depend on how many others were evaluated next to it.\n#\n#   BENCH     sbp (SWE-bench Pro v2 HARD-51), tb (Terminal-Bench 2.1, 20% subset) or s800 (scale800 held-out)\n#   POLICIES  space-separated launcher_eval.py entries, `<checkpoint path or Hub id>@<harnesses>`, evaluated in order\n#   TAG       suffix of the eval/ folders, default <bench tag>-solo\n#\n#   BENCH=sbp POLICIES=\"Qwen/Qwen3.8-27B@pi,hermes,opencode\" sbatch eval_slurm.sh\n#   BENCH=tb POLICIES=\"$ROOT/data/runs/<run>/checkpoint-100@pi\" sbatch eval_slurm.sh\n#\n# Results: $ROOT/data/eval/<label>_<tag>/, copied to the bucket's eval/ every 10 minutes and at the end (where the\n# livestream Space reads them). Rerunning the same POLICIES redoes only pairs with no grade yet.\n#SBATCH --job-name=ha-qwen3.8-eval\n#SBATCH --partition=hopper-b200\n#SBATCH --nodes=1\n#SBATCH --ntasks-per-node=1\n#SBATCH --gpus-per-node=1\n#SBATCH --cpus-per-task=16\n#SBATCH --time=14:00:00\n#SBATCH --output=/fsx/%u/logs/%x-%j.out\nset -euo pipefail\nROOT=\"${ROOT:-/fsx/$USER/termgrade}\"\nMODEL=\"${MODEL:-Qwen/Qwen3.8-27B}\"\nBENCH=\"${BENCH:?BENCH=sbp, tb or s800}\"\n# scale800 held-out: the tasks of the repositories left out of training (all of them), with training's own agent limits,\n# since it measures what training produced. Set S800_HELDOUT to the held-out dataset, pinned like the training one.\nS800_HELDOUT=\"${S800_HELDOUT:-harness-hub/scale800-heldout-harbor}\"\nPOLICIES=\"${POLICIES:?POLICIES=\\\"<checkpoint>@<harness> ...\\\"}\"\nBUCKET=\"${BUCKET:-ybelkada/harbor-evals}\"\n# Per benchmark: the split and its task selection, and the agent limits. SWE-bench Pro caps model calls rather than\n# wall time: with the tasks' own limits (about an hour), 70% of the base's rollouts timed out at 72-129 s per call, so\n# its score measured server speed. 7200 s only stops a rollout that is stuck.\ncase \"$BENCH\" in\n    sbp)\n        TAG=\"${TAG:-sbp-hard51-solo}\"\n        BENCH_ARGS=(--split harness-hub/swebench-pro-v2-hard51-harbor --n-tasks 51 --sandbox-memory-mb 4096\n            --agent-step-limit 100 --agent-timeout 7200)\n        ;;\n    tb)\n        TAG=\"${TAG:-tb21-20pct-solo}\"\n        BENCH_ARGS=(--split harborframework/terminal-bench-2.1@3e235dff6880252a587fa479c09bcb1e16edf2eb\n            --difficulty easy:1,medium:11,hard:6 --n-tasks 18 --sandbox-memory-mb 8192\n            --agent-step-limit 100 --agent-timeout 3600)\n        ;;\n    s800)\n        TAG=\"${TAG:-s800-heldout-solo}\"\n        BENCH_ARGS=(--split \"$S800_HELDOUT\" --n-tasks 1000 --sandbox-memory-mb 8192\n            --agent-step-limit 50 --agent-timeout 1800)\n        ;;\n    *)\n        echo \"unknown BENCH=$BENCH: sbp, tb or s800\" >&2\n        exit 1\n        ;;\nesac\nexport JOB_ID=\"$SLURM_JOB_ID\"\nexport HF_TOKEN=\"${HF_TOKEN:-$(cat ~/.cache/huggingface/token)}\"  # also what the HF sandboxes run under\nexport HF_HUB_CACHE=\"$(dir-load \"$ROOT/hf-hub\")\"\nexport PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True\nexport PYTHONFAULTHANDLER=1\nmkdir -p \"/fsx/$USER/logs\"\n# The job's private /tmp, on the node's NVMe and deleted by Slurm when the job ends. Each merged checkpoint (~54 GB) is\n# written here and deleted once evaluated; the CPU merge fits the default 248 GB of memory per GPU.\nmkdir -p \"/tmp/eval-solo-$SLURM_JOB_ID\"\ncd \"/tmp/eval-solo-$SLURM_JOB_ID\"\nexport TRITON_CACHE_DIR=\"$PWD/cache/triton\"\nexport TORCHINDUCTOR_CACHE_DIR=\"$PWD/cache/inductor\"\nexport VLLM_CACHE_ROOT=\"$PWD/cache/vllm\"\nexport XDG_CACHE_HOME=\"$PWD/cache/xdg\"\nsource \"$(venv-load \"$ROOT/venv\")\"\n# vLLM from its own venv, as in train_slurm.sh: only that command is put first on PATH\nmkdir -p vllm-bin\nln -sf \"$(dirname \"$(venv-load \"$ROOT/venv-vllm\")\")/vllm\" vllm-bin/vllm\nexport PATH=\"$PWD/vllm-bin:$PATH\"\n# Ports free right now, from the OS: ports derived from the GPU index (as in train_slurm.sh) collided with a process an\n# earlier job on the node left on them (\"capture port :20000 is already in use\")\nread -r VLLM_PORT SERVER_PORT CAPTURE_PORT < <(python -c \"\nimport socket\nsocks = [socket.socket() for _ in range(3)]\nfor s in socks:\n    s.bind(('', 0))\nprint(*(s.getsockname()[1] for s in socks))\n\")\necho \"ports: vllm $VLLM_PORT server $SERVER_PORT capture $CAPTURE_PORT\"\n\n# This job's eval folders, copied to the bucket: every 10 minutes, and once more when the job ends\nsync_evals() {\n    for dir in \"$ROOT\"/data/eval/*_\"$TAG\"; do\n        [ -d \"$dir\" ] || continue\n        hf buckets sync \"$dir\" \"hf://buckets/$BUCKET/eval/$(basename \"$dir\")\" > /dev/null 2>&1 \\\n            || echo \"$(date +%T) failed: sync $(basename \"$dir\")\"\n    done\n}\n(while true; do sleep 600; sync_evals; done) &\nSYNC_PID=$!\ntrap 'kill $SYNC_PID 2> /dev/null; sync_evals' EXIT\n\nread -r -a POLICY_LIST <<< \"$POLICIES\"\npython \"$ROOT/scripts/launcher_eval.py\" \\\n    --data-root \"$ROOT/data\" --eval-script-url \"file://$ROOT/scripts/evaluate_harbor.py\" \\\n    --model \"$MODEL\" --eval-checkpoints \"${POLICY_LIST[@]}\" --eval-tag \"$TAG\" \"${BENCH_ARGS[@]}\" \\\n    --sandbox hf-sandbox --expose gradio --max-inflight 32 --max-model-len 131072 \\\n    --vllm-port \"$VLLM_PORT\" --server-port \"$SERVER_PORT\" --capture-port \"$CAPTURE_PORT\" \\\n    --vllm-device 0 \\\n    --tool-call-parser qwen3_xml --reasoning-parser qwen3 --no-responses-thinking \\\n    --vllm-extra-args=\"--max-num-seqs 64 --enable-prefix-caching --mamba-cache-mode align\"\n",
    "recipe.yaml": "meta:\n  title: \"Qwen3.8-27B harness adapters\"\n  project: harness-adapters\n  trainer: AsyncGRPOTrainer\n  model_id: Qwen/Qwen3.8-27B\n  description: \"One LoRA adapter per agent harness (pi, hermes, opencode) on Qwen3.8-27B, trained with async GRPO on agentic coding and terminal tasks (scale800 + TermGrade), one job per harness, thinking off.\"\n  tags: [lora, agentic-rl, harbor, safeagents, multi-harness, live]\n  harnesses: [pi, hermes, opencode]\n  date_added: 2026-11-13\n  dashboard: https://huggingface.co/spaces/harness-hub/qwen-livestream\n\nhardware:\n  slurm: \"hopper-b200, 2 GPUs per harness\"   # vLLM on GPU 0, the trainer on GPU 1\n  local: \"2 GPUs per harness, H200 or B200 class (untested)\"\n  hf_jobs: \"h200x2 per harness (untested)\"\n\ndatasets:\n  train:\n    - harness-hub/scale800-harbor@56c7f75d01f70859ec1354e97866e32a175c1a40   # 750 tasks\n    - harness-hub/termgrade-train-harbor                                      # 300 tasks, interleaved\n  eval:\n    sbp-hard51: harness-hub/swebench-pro-v2-hard51-harbor\n    tb21-20pct: \"harborframework/terminal-bench-2.1@3e235dff6880252a587fa479c09bcb1e16edf2eb, 18 tasks\"\n    s800-heldout: \"harness-hub/scale800-heldout-harbor (the repositories left out of training), 1800 s and 50 steps as in training\"\n    automationbench: \"Zapier AutomationBench, 10 tasks per domain (60), as the base on the dashboard\"\n\nhyperparameters:\n  lora_r: 32\n  lora_target_modules: [q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj]\n  learning_rate: 2.0e-5\n  num_generations: 8\n  gradient_accumulation_steps: 16\n  max_staleness: 16\n  temperature: 0.8\n  max_completion_length: 16384\n  max_model_len: 65536\n  token_budget: 65536\n  agent_step_limit: 50\n  agent_timeout_s: 1800\n  max_inflight: 64\n  tool_efficiency_weight: 0.1\n  max_steps: 1000\n  save_steps: 10\n  thinking: false\n\nvllm:\n  tool_call_parser: qwen3_xml\n  reasoning_parser: qwen3\n  extra_args: [--language-model-only, --enable-prefix-caching, --mamba-cache-mode align]\n\ntrl:\n  fork: \"younesbelkada/trl@e498acc637084cffff14deb09ccdc423a766adb2 (async-grpo-shared-vllm)\"\n  upstream_base: d115286bc5aef8d346e8510bfe312803c7881c4a\n  patch: trl.patch            # the fork's changes to trl/, relative to upstream_base\n\nprovenance:\n  slurm_jobs: {pi: 374053, hermes: 374054, opencode: 374055}\n  status: training\n\nscripts:\n  train: {hf_jobs: train_hf_jobs.sh, slurm: train_slurm.sh, local: train_local.sh}\n  eval: {hf_jobs: eval_hf_jobs.sh, slurm: eval_slurm.sh, local: eval_local.sh}\n  dashboard: ../scripts/sync_live.sh  # live.json and checkpoints -> the bucket the Space reads\n\n# What the command builder on the recipes site offers. Each option is an environment variable of the script it runs.\nbuilder:\n  verified: {hf_jobs: false, slurm: true, local: false}   # whether the scripts of each platform ran as written\n  train:\n    - env: HARNESS\n      label: Harnesses\n      choices: [pi, hermes, opencode]\n      multi: true                # one job per harness: a loop on SLURM, HARNESSES=\"...\" on HF Jobs\n      hf_jobs_env: HARNESSES\n      platforms: [hf_jobs, slurm]\n    - env: HARNESS\n      label: Harness\n      choices: [pi, hermes, opencode]\n      platforms: [local]         # one machine trains one harness\n  eval:\n    - env: BENCH\n      label: Benchmark\n      choices:\n        - sbp\n        - tb\n        - s800\n        - {value: automationbench, platforms: [hf_jobs, local]}\n    - env: POLICIES\n      label: Policies\n      input: true\n      default: \"Qwen/Qwen3.8-27B@pi,hermes,opencode\"\n      unless: {BENCH: automationbench}\n    - env: CHECKPOINTS\n      label: Checkpoints\n      input: true\n      default: \"base\"\n      when: {BENCH: automationbench}\n",
    "trl.patch": "TRL changes these recipes run on, relative to upstream huggingface/trl.\n  upstream base: d115286bc5aef8d346e8510bfe312803c7881c4a (2026-11-11, Add UDPO loss for noisy preferences to DPOTrainer (#8211))\n  fork:          younesbelkada/trl async-grpo-shared-vllm = e498acc637084cffff14deb09ccdc423a766adb2\nFork commits:\n  e498acc6 AsyncGRPO: start the rollout concurrency cap at max_concurrent_rollouts\n  965cb093 Harbor example: LoRA, safeagents, live log, token budget and resume, on HarnessAsyncGRPOTrainer\n  83bdbc7c Let several AsyncGRPO/AsyncDistillation trainers share one vLLM server\nApply on a checkout of the upstream base: git apply trl.patch\nRegenerate: git diff $(git merge-base origin/main <fork commit>) <fork commit> -- trl/\n\ndiff --git a/trl/experimental/async_distillation/async_distillation_config.py b/trl/experimental/async_distillation/async_distillation_config.py\nindex bc1d422e..38a1e7a2 100644\n--- a/trl/experimental/async_distillation/async_distillation_config.py\n+++ b/trl/experimental/async_distillation/async_distillation_config.py\n@@ -84,6 +84,9 @@ class AsyncDistillationConfig(_BaseConfig):\n             Timeout in seconds for calls to the vLLM servers: waiting for the student's server to be ready at startup,\n             each generation or scoring request, and each weight-sync call. Waiting for the server and weight-sync calls\n             raise when it runs out; a generation or scoring request that times out is retried.\n+        vllm_lora_name (`str`, *optional*, defaults to `\"trl-policy\"`):\n+            Prefix of the adapter names published to the student's vLLM server under adapter-only sync\n+            (`{vllm_lora_name}-v{N}`). Give each trainer its own prefix when several share one server.\n         teacher_server_urls (`dict[str, str]`, *optional*, defaults to `{\"default\": \"http://localhost:8001\"}`):\n             Teacher vLLM server(s), each started with `vllm serve <teacher-model> --logprobs-mode processed_logprobs\n             --max-logprobs -1`. Every teacher is static: this trainer never streams weight updates to any of them, so a\n@@ -295,6 +298,13 @@ class AsyncDistillationConfig(_BaseConfig):\n             \"weight-sync calls raise when it runs out; a generation or scoring request that times out is retried.\"\n         },\n     )\n+    vllm_lora_name: str = field(\n+        default=\"trl-policy\",\n+        metadata={\n+            \"help\": \"Prefix of the adapter names published to the student's vLLM server under adapter-only sync \"\n+            \"(`{vllm_lora_name}-v{N}`). Give each trainer its own prefix when several share one server.\"\n+        },\n+    )\n     teacher_server_urls: dict[str, str] | str | None = field(\n         default=None,\n         metadata={\ndiff --git a/trl/experimental/async_distillation/async_distillation_trainer.py b/trl/experimental/async_distillation/async_distillation_trainer.py\nindex 69a5da64..b42c97c9 100644\n--- a/trl/experimental/async_distillation/async_distillation_trainer.py\n+++ b/trl/experimental/async_distillation/async_distillation_trainer.py\n@@ -1348,7 +1348,7 @@ class AsyncDistillationTrainer(_BaseTrainer):\n         # Adapter-only vLLM sync is derived from a PEFT model plus a server started with `--enable-lora`, rather\n         # than configured. Rank 0 probes and broadcasts the answer below.\n         self._lora_sync = False\n-        self._lora_name = \"trl-policy\"\n+        self._lora_name = self.args.vllm_lora_name\n         # Absolute: the path is resolved by the *server's* process, which has its own working directory and may not\n         # be on this machine. vLLM reads a path it cannot resolve as a Hub repo id, failing deep inside the engine.\n         self._lora_dir = os.path.abspath(os.path.join(self.args.output_dir, \".vllm_lora\"))\n@@ -1967,11 +1967,10 @@ class AsyncDistillationTrainer(_BaseTrainer):\n         # Under DDP `self.model` is the wrapper, which does not expose `save_pretrained`.\n         model = self.accelerator.unwrap_model(self.model)\n \n-        logger.info(\"Weight sync: pausing vLLM...\")\n-        if self.accelerator.is_main_process:\n-            self.vllm_client.pause()\n+        # No `/pause`: in-flight requests keep the adapter they named, and pausing would stall every other trainer\n+        # sharing this server.\n         t_pause = time.time()\n-        logger.info(f\"Weight sync: pause took {t_pause - t0:.1f}s, waiting for all ranks...\")\n+        logger.info(\"Weight sync: waiting for all ranks...\")\n \n         self.accelerator.wait_for_everyone()\n         t_barrier = time.time()\n@@ -1984,12 +1983,11 @@ class AsyncDistillationTrainer(_BaseTrainer):\n \n         self.accelerator.wait_for_everyone()\n \n-        logger.info(f\"Weight sync: resuming vLLM... (transfer took {t_transfer - t_barrier:.1f}s)\")\n+        logger.info(f\"Weight sync: loading adapter... (transfer took {t_transfer - t_barrier:.1f}s)\")\n         if self.accelerator.is_main_process:\n             # The adapter has to exist on the server BEFORE the version moves below: the rollout worker derives the\n             # adapter it requests from `model_version`, so bumping first would name one that is not loaded yet.\n             self.vllm_client.load_lora_adapter(f\"{self._lora_name}-v{version}\", adapter_dir)\n-            self.vllm_client.resume()\n             stale_version = version - (self.args.max_staleness + 1)\n             if stale_version > 0:\n                 self.vllm_client.unload_lora_adapter(f\"{self._lora_name}-v{stale_version}\")\ndiff --git a/trl/experimental/async_grpo/async_grpo_config.py b/trl/experimental/async_grpo/async_grpo_config.py\nindex 6335eb5f..26a20220 100644\n--- a/trl/experimental/async_grpo/async_grpo_config.py\n+++ b/trl/experimental/async_grpo/async_grpo_config.py\n@@ -88,6 +88,9 @@ class AsyncGRPOConfig(_BaseConfig):\n             Timeout in seconds for calls to the vLLM server: waiting for it to be ready at startup, each generation\n             request, and each weight-sync call. Waiting for the server and weight-sync calls raise when it runs out; a\n             generation request that times out is retried.\n+        vllm_lora_name (`str`, *optional*, defaults to `\"trl-policy\"`):\n+            Prefix of the adapter names published to the vLLM server under adapter-only sync (`{vllm_lora_name}-v{N}`).\n+            Give each trainer its own prefix when several share one server.\n \n         > Parameters that control the training\n \n@@ -296,6 +299,13 @@ class AsyncGRPOConfig(_BaseConfig):\n             \"it runs out; a generation request that times out is retried.\"\n         },\n     )\n+    vllm_lora_name: str = field(\n+        default=\"trl-policy\",\n+        metadata={\n+            \"help\": \"Prefix of the adapter names published to the vLLM server under adapter-only sync \"\n+            \"(`{vllm_lora_name}-v{N}`). Give each trainer its own prefix when several share one server.\"\n+        },\n+    )\n \n     # Parameters that control the training\n     epsilon: float = field(\ndiff --git a/trl/experimental/async_grpo/async_grpo_trainer.py b/trl/experimental/async_grpo/async_grpo_trainer.py\nindex d716d5c7..28251f89 100644\n--- a/trl/experimental/async_grpo/async_grpo_trainer.py\n+++ b/trl/experimental/async_grpo/async_grpo_trainer.py\n@@ -1247,7 +1247,7 @@ class AsyncGRPOTrainer(_BaseTrainer):\n         # Adapter-only vLLM sync is derived from a PEFT model plus a server started with `--enable-lora`, rather\n         # than configured. Rank 0 probes and broadcasts the answer below.\n         self._lora_sync = False\n-        self._lora_name = \"trl-policy\"\n+        self._lora_name = self.args.vllm_lora_name\n         # Absolute: the path is resolved by the *server's* process, which has its own working directory and may not\n         # be on this machine. vLLM reads a path it cannot resolve as a Hub repo id, failing deep inside the engine.\n         self._lora_dir = os.path.abspath(os.path.join(self.args.output_dir, \".vllm_lora\"))\n@@ -1799,11 +1799,10 @@ class AsyncGRPOTrainer(_BaseTrainer):\n         # Under DDP `self.model` is the wrapper, which does not expose `save_pretrained`.\n         model = self.accelerator.unwrap_model(self.model)\n \n-        logger.info(\"Weight sync: pausing vLLM...\")\n-        if self.accelerator.is_main_process:\n-            self.vllm_client.pause()\n+        # No `/pause`: in-flight requests keep the adapter they named, and pausing would stall every other trainer\n+        # sharing this server.\n         t_pause = time.time()\n-        logger.info(f\"Weight sync: pause took {t_pause - t0:.1f}s, waiting for all ranks...\")\n+        logger.info(\"Weight sync: waiting for all ranks...\")\n \n         self.accelerator.wait_for_everyone()\n         t_barrier = time.time()\n@@ -1816,12 +1815,11 @@ class AsyncGRPOTrainer(_BaseTrainer):\n \n         self.accelerator.wait_for_everyone()\n \n-        logger.info(f\"Weight sync: resuming vLLM... (transfer took {t_transfer - t_barrier:.1f}s)\")\n+        logger.info(f\"Weight sync: loading adapter... (transfer took {t_transfer - t_barrier:.1f}s)\")\n         if self.accelerator.is_main_process:\n             # The adapter has to exist on the server BEFORE the version moves below: the rollout worker derives the\n             # adapter it requests from `model_version`, so bumping first would name one that is not loaded yet.\n             self.vllm_client.load_lora_adapter(f\"{self._lora_name}-v{version}\", adapter_dir)\n-            self.vllm_client.resume()\n             stale_version = version - (self.args.max_staleness + 1)\n             if stale_version > 0:\n                 self.vllm_client.unload_lora_adapter(f\"{self._lora_name}-v{stale_version}\")\ndiff --git a/trl/experimental/async_grpo/async_rollout_worker.py b/trl/experimental/async_grpo/async_rollout_worker.py\nindex 754a02d4..d55e043d 100644\n--- a/trl/experimental/async_grpo/async_rollout_worker.py\n+++ b/trl/experimental/async_grpo/async_rollout_worker.py\n@@ -564,6 +564,9 @@ class AsyncRolloutLoop:\n             max_workers=max(1, config.max_concurrent_rollouts), thread_name_prefix=\"grpo-tool\"\n         )\n \n+        # Starts at the requested concurrency, not at `KV cache tokens / max_model_len`: agent rollouts use a small\n+        # fraction of max_model_len, and growing 25% per turnover of the pool took hours when rollouts last minutes.\n+        # The control loop still cuts it when the server's KV cache fills up.\n         self._kv_cap = config.max_concurrent_rollouts\n         self._total_rollouts = 0\n         self._metrics_poll_s = 5.0\n@@ -667,7 +670,6 @@ class AsyncRolloutLoop:\n         async with aiohttp.ClientSession(connector=connector, timeout=timeout) as session:\n             self.session = session\n             await self._check_sampling_replay()\n-            await self._init_kv_cap()\n             logger.info(\n                 f\"vllm worker started: num_generations={self.config.num_generations}, \"\n                 f\"max_concurrent_rollouts={self.config.max_concurrent_rollouts}, kv_cap={self._kv_cap}, \"\n@@ -694,26 +696,6 @@ class AsyncRolloutLoop:\n         expected_tokens = self._produced_tokens + self._rollouts_unqueued * self._tokens_per_rollout\n         return expected_tokens < self._token_allowance.value\n \n-    async def _init_kv_cap(self) -> None:\n-        \"\"\"Start the cap at `KV cache tokens / max_model_len`: as many requests as fit at full length.\"\"\"\n-        try:\n-            async with self.session.get(f\"{self.vllm_server_url}/metrics\") as response:\n-                response.raise_for_status()\n-                text = await response.text()\n-            async with self.session.get(f\"{self.vllm_server_url}/v1/models\") as response:\n-                response.raise_for_status()\n-                models = await response.json()\n-        except Exception as e:\n-            logger.warning(f\"Could not read the vLLM server's KV capacity, starting at max_concurrent_rollouts: {e!r}\")\n-            return\n-        kv_matches = re.findall(r'^vllm:cache_config_info\\{.*kv_cache_size_tokens=\"(\\d+)\"', text, re.M)\n-        if not kv_matches:\n-            logger.warning(\"`/metrics` has no `kv_cache_size_tokens`, starting at max_concurrent_rollouts\")\n-            return\n-        kv_tokens = sum(map(int, kv_matches))\n-        max_model_len = max(model[\"max_model_len\"] for model in models[\"data\"] if model[\"max_model_len\"])\n-        self._kv_cap = max(1, min(self.config.max_concurrent_rollouts, kv_tokens // max_model_len))\n-\n     async def _control_loop(self, stop_event: asyncio.Event) -> None:\n         \"\"\"Size the KV cap from the vLLM server's load, polled from `/metrics`.\n \n"
   },
   "other_files": [
    "async_grpo_harbor.py",
    "eval_automationbench.py"
   ]
  },
  {
   "id": "AsyncGRPOTrainer/harness-adapters/TwIL-8B",
   "trainer": "AsyncGRPOTrainer",
   "project": "harness-adapters",
   "name": "TwIL-8B",
   "org": "webAI-Official",
   "avatar": "https://cdn-avatars.huggingface.co/v1/production/uploads/6483b6ffa7e293d5722577cd/Dh0hsjEncJ0NOAIZzEahN.png",
   "meta": {
    "title": "TwIL-8B harness adapters",
    "project": "harness-adapters",
    "trainer": "AsyncGRPOTrainer",
    "model_id": "webAI-Official/TwIL-8B",
    "description": "Experiment: one LoRA adapter per agent harness (pi, hermes, opencode) on a model not trained on agentic data, with more agent steps and a longer rollout timeout.",
    "tags": [
     "lora",
     "agentic-rl",
     "harbor",
     "safeagents",
     "multi-harness",
     "experiment"
    ],
    "harnesses": [
     "pi",
     "hermes",
     "opencode"
    ],
    "date_added": "2026-11-13",
    "blog": "https://huggingface.co/spaces/harness-hub/harness-adapters",
    "dashboard": "https://harness-hub-harness-adapters-rl.hf.space/"
   },
   "hardware": {
    "hf_jobs": "h200x4",
    "slurm": "hopper-b200, 4 GPUs (untested)",
    "local": "4 GPUs, H200 class (untested)"
   },
   "datasets": {
    "train": [
     "FineEnvs/SmolDataEnvs-harbor-train",
     "harness-hub/traj2envs-rl@191a751bda511eada76f6a9a76513d143321a64a"
    ],
    "eval": {
     "final-heldout": "FineEnvs/SmolDataEnvs-harbor-train, 150 easy/medium tasks from index 1000",
     "traj2envs-heldout": "harness-hub/traj2envs-rl, the 40 tasks from index 230"
    }
   },
   "hyperparameters": {
    "lora_r": 32,
    "lora_target_modules": "all-linear",
    "learning_rate": 1e-05,
    "num_generations": 8,
    "gradient_accumulation_steps": 4,
    "max_staleness": 4,
    "temperature": 0.8,
    "max_completion_length": 16384,
    "max_output_tokens": 32768,
    "max_model_len": 131072,
    "token_budget": 65536,
    "agent_step_limit": 30,
    "agent_timeout_s": 1200,
    "max_inflight": 12,
    "tool_efficiency_weight": 0.1,
    "max_steps": 100,
    "save_steps": 10
   },
   "vllm": {
    "tool_call_parser": "qwen3_coder",
    "reasoning_parser": "qwen3",
    "chat_template": "chat_template.jinja",
    "trainer_flags": [
     "--qwen3-5-response-template"
    ]
   },
   "trl": {
    "fork": "younesbelkada/trl@ee6e9ad9512b83c80bb0051942770c8798c0dbf2 (add-lfm-tree-attn)",
    "upstream_base": "7fdad81d04303e6f43db168e903565769c0b8642",
    "patch": "trl.patch"
   },
   "provenance": {
    "hf_job": "6aedb49dc78863f9c63408be",
    "released": {
     "pi": {
      "step": 90,
      "repo": "harness-hub/TwIL-8B-pi"
     },
     "hermes": {
      "step": 60,
      "repo": "harness-hub/TwIL-8B-hermes"
     },
     "opencode": {
      "step": 100,
      "repo": "harness-hub/TwIL-8B-opencode"
     }
    }
   },
   "scripts": {
    "train": {
     "hf_jobs": "train_hf_jobs.sh",
     "slurm": "train_slurm.sh",
     "local": "train_local.sh"
    },
    "eval": {
     "hf_jobs": "eval_hf_jobs.sh",
     "slurm": "eval_slurm.sh",
     "local": "eval_local.sh"
    }
   },
   "builder": {
    "verified": {
     "hf_jobs": true,
     "slurm": false,
     "local": false
    },
    "eval": [
     {
      "env": "SPLIT",
      "label": "Held-out split",
      "choices": [
       "final-heldout",
       "traj2envs-heldout"
      ]
     },
     {
      "env": "CHECKPOINTS",
      "label": "Checkpoints",
      "input": true,
      "placeholder": "<path|glob|hub id>@<harnesses> ... (empty: the script's default)"
     }
    ]
   },
   "readme": "# TwIL-8B harness adapters (experiment)\n\nLoRA adapters for pi, hermes and opencode on [`webAI-Official/TwIL-8B`](https://huggingface.co/webAI-Official/TwIL-8B),\na model not trained on agentic data. Trained on 400 SmolDataEnvs tasks (150 medium, 250 hard) and 230 traj2envs tasks.\nSetup and shared settings: [../README.md](../README.md). Full hyperparameters: [recipe.yaml](recipe.yaml).\n\n| | HF Jobs | Cluster (untested) | Local (untested) |\n|---|---|---|---|\n| Train | `bash train_hf_jobs.sh` (h200x4, ~48h) | `sbatch train_slurm.sh` (4 GPUs) | `bash train_local.sh` (4 GPUs) |\n| Eval | `bash eval_hf_jobs.sh` (h200) | `CHECKPOINTS=... sbatch eval_slurm.sh` (1 GPU) | `bash eval_local.sh` (1 GPU) |\n\nModel-specific flags:\n- vLLM serves the model with [chat_template.jinja](chat_template.jinja), which the HF Jobs scripts upload to the bucket\n  as `twil_chat_template.jinja`. The trainer uses `--qwen3-5-response-template`.\n- `--tool-call-parser qwen3_coder --reasoning-parser qwen3`, `--token-budget 65536`\n- `--agent-step-limit 30 --agent-timeout 1200`: the model needs more steps per task.\n- `--max-inflight 12` (48 for the LFM models): its long reasoning fills vLLM's memory sooner.\n\n**Steps:** the launch command and the released checkpoints use 100 steps (`--max-steps 100`), but the blog post's table\nsays 300.\n\nReleased checkpoints (job `6aedb49dc78863f9c63408be`):\n\n| Harness | Step | Repo |\n|---|--:|---|\n| pi | 90 | `harness-hub/TwIL-8B-pi` |\n| hermes | 60 | `harness-hub/TwIL-8B-hermes` |\n| opencode | 100 | `harness-hub/TwIL-8B-opencode` |\n",
   "files": {
    "train_local.sh": "#!/bin/bash\n# TwIL-8B harness adapters on one local machine with 4 GPUs:\n# vLLM on GPU 0, one trainer per harness on GPUs 1, 2 and 3. The released run used H200s (141 GB).\n#\n# Same launcher and training flags as train_hf_jobs.sh, run here with `uv run` instead of on HF Jobs. Not run locally\n# yet. Needs uv, 4 GPUs and HF_TOKEN (also what the HF sandboxes run under). Runs are written to $DATA_ROOT/runs/.\n#\n#   bash train_local.sh\n#   MAX_STEPS=2 bash train_local.sh                   short smoke run\nset -euo pipefail\n\nMAX_STEPS=\"${MAX_STEPS:-100}\"\nDATA_ROOT=\"${DATA_ROOT:-$PWD/outputs}\"\nTRL_WHEEL=\"${TRL_WHEEL:-}\"\nTRACKIO_SPACE_ID=\"${TRACKIO_SPACE_ID:-ybelkada/async-grpo-harbor}\"\nREV=191a751bda511eada76f6a9a76513d143321a64a  # harness-hub/traj2envs-rl revision the adapters were trained on\nHERE=\"$(cd \"$(dirname \"$0\")\" && pwd)\"\nSCRIPTS=\"$HERE/../scripts\"\nexport HF_TOKEN=\"${HF_TOKEN:-$(cat ~/.cache/huggingface/token)}\"\n# Names this run's folders (logs/<id>, runs/<model>-<harness>-<id>-...), as HF_JOB_ID does on HF Jobs\nexport JOB_ID=\"${JOB_ID:-local-$(date +%Y%m%d-%H%M%S)}\"\nexport PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True\n\nTRL_ARGS=()\nif [ -n \"$TRL_WHEEL\" ]; then TRL_ARGS=(--trl-wheel \"$TRL_WHEEL\"); fi\n\nuv run \"$SCRIPTS/launcher_train.py\" \\\n    --data-root \"$DATA_ROOT\" ${TRL_ARGS[@]+\"${TRL_ARGS[@]}\"} \\\n    --train-script-url \"file://$SCRIPTS/async_grpo_harbor.py\" --expose gradio \\\n    --pip-install \"safeagents@git+https://github.com/huggingface/safeagents@5e683fa atif-scan==0.8.15 rich markdown-it-py mdurl pygments\" \\\n    --model webAI-Official/TwIL-8B \\\n    --vllm-extra-args=\"--chat-template $HERE/chat_template.jinja\" \\\n    --tool-call-parser qwen3_coder --reasoning-parser qwen3 --max-model-len 131072 \\\n    --run pi:pi:medium:150,hard:250 \\\n    --run hermes:hermes:medium:150,hard:250 \\\n    --run opencode:opencode:medium:150,hard:250 \\\n    --split FineEnvs/SmolDataEnvs-harbor-train --n-tasks 400 \\\n    --mix-split \"harness-hub/traj2envs-rl@$REV:230\" --max-output-tokens 32768 \\\n    --sandbox hf-sandbox --docker-image docker.io/ybelkada/env-data-agent-train:opencode-1.19.5 \\\n    --vllm-device 0 --train-device 1,2,3 \\\n    --max-inflight 12 --max-lora-rank 32 --max-loras 18 \\\n    --reward-key correctness,reward --num-generations 8 --gradient-accumulation-steps 4 --max-staleness 4 \\\n    --max-completion-length 16384 --agent-step-limit 30 --agent-timeout 1200 --temperature 0.8 \\\n    --lora-r 32 --lora-target-modules all-linear --learning-rate 1e-5 --tool-efficiency-weight 0.1 \\\n    --token-budget 65536 \\\n    --qwen3-5-response-template \\\n    --safeagents --safeagents-plugin \"$SCRIPTS/atif_smoldataenvs.py:checks\" --safeagents-plugin \"$SCRIPTS/atif_traj2envs.py:checks\" \\\n    --max-steps \"$MAX_STEPS\" --save-steps 10 --save-total-limit 0 \\\n    --trackio-space-id \"$TRACKIO_SPACE_ID\"\n",
    "train_hf_jobs.sh": "#!/bin/bash\n# TwIL-8B harness adapters on HF Jobs: one vLLM (GPU 0) serving the base and every adapter version, and one LoRA\n# AsyncGRPO trainer per harness (pi, hermes, opencode on GPUs 1, 2, 3), on 4x H200.\n#\n# This is the command of the released run (HF job 6aedb49dc78863f9c63408be, 2026-11-06), with the bucket as a variable.\n# TwIL is not trained on agentic data: it gets 30 agent steps and 20 minutes per rollout, and fewer rollouts at once\n# (its long reasoning fills vLLM's memory sooner). It is served with its own chat template (chat_template.jinja).\n#\n#   bash train_hf_jobs.sh\n#   BUCKET=<user>/<bucket> MAX_STEPS=2 bash train_hf_jobs.sh          short smoke run\n#   TRL_WHEEL=/outputs/wheels/trl-....whl bash train_hf_jobs.sh       a TRL wheel already on the bucket, over the\n#                                                                     launcher's pinned TRL\nset -euo pipefail\n\nBUCKET=\"${BUCKET:-ybelkada/harbor-evals}\"\nMAX_STEPS=\"${MAX_STEPS:-100}\"\nTRL_WHEEL=\"${TRL_WHEEL:-}\"\nTRACKIO_SPACE_ID=\"${TRACKIO_SPACE_ID:-ybelkada/async-grpo-harbor}\"\nREV=191a751bda511eada76f6a9a76513d143321a64a  # harness-hub/traj2envs-rl revision the adapters were trained on\nHERE=\"$(cd \"$(dirname \"$0\")\" && pwd)\"\nSCRIPTS=\"$HERE/../scripts\"\n\n# The job reads the training script, the safeagents plugins and the chat template from the bucket, mounted on /outputs\nfor f in async_grpo_harbor.py atif_smoldataenvs.py atif_traj2envs.py; do\n    hf buckets cp \"$SCRIPTS/$f\" \"hf://buckets/$BUCKET/$f\"\ndone\nhf buckets cp \"$HERE/chat_template.jinja\" \"hf://buckets/$BUCKET/twil_chat_template.jinja\"\n\nTRL_ARGS=()\nif [ -n \"$TRL_WHEEL\" ]; then TRL_ARGS=(--trl-wheel \"$TRL_WHEEL\"); fi\n\nhf jobs uv run --detach \\\n    --flavor h200x4 --image huggingface/trl --secrets HF_TOKEN --timeout 48h \\\n    --env PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True \\\n    --volume \"hf://buckets/$BUCKET:/outputs\" \\\n    \"$SCRIPTS/launcher_train.py\" -- \\\n    --data-root /outputs ${TRL_ARGS[@]+\"${TRL_ARGS[@]}\"} \\\n    --pip-install \"safeagents@git+https://github.com/huggingface/safeagents@5e683fa atif-scan==0.8.15 rich markdown-it-py mdurl pygments\" \\\n    --model webAI-Official/TwIL-8B \\\n    --vllm-extra-args=\"--chat-template /outputs/twil_chat_template.jinja\" \\\n    --tool-call-parser qwen3_coder --reasoning-parser qwen3 --max-model-len 131072 \\\n    --run pi:pi:medium:150,hard:250 \\\n    --run hermes:hermes:medium:150,hard:250 \\\n    --run opencode:opencode:medium:150,hard:250 \\\n    --split FineEnvs/SmolDataEnvs-harbor-train --n-tasks 400 \\\n    --mix-split \"harness-hub/traj2envs-rl@$REV:230\" --max-output-tokens 32768 \\\n    --sandbox hf-sandbox --docker-image docker.io/ybelkada/env-data-agent-train:opencode-1.19.5 \\\n    --vllm-device 0 --train-device 1,2,3 \\\n    --max-inflight 12 --max-lora-rank 32 --max-loras 18 \\\n    --reward-key correctness,reward --num-generations 8 --gradient-accumulation-steps 4 --max-staleness 4 \\\n    --max-completion-length 16384 --agent-step-limit 30 --agent-timeout 1200 --temperature 0.8 \\\n    --lora-r 32 --lora-target-modules all-linear --learning-rate 1e-5 --tool-efficiency-weight 0.1 \\\n    --token-budget 65536 \\\n    --qwen3-5-response-template \\\n    --safeagents --safeagents-plugin /outputs/atif_smoldataenvs.py:checks --safeagents-plugin /outputs/atif_traj2envs.py:checks \\\n    --max-steps \"$MAX_STEPS\" --save-steps 10 --save-total-limit 0 \\\n    --trackio-space-id \"$TRACKIO_SPACE_ID\"\n",
    "train_slurm.sh": "#!/bin/bash -l\n# TwIL-8B harness adapters on one cluster node: one vLLM (GPU 0) serving the base and every adapter version, and one\n# LoRA AsyncGRPO trainer per harness (pi, hermes, opencode on GPUs 1, 2, 3).\n#\n# Same launcher and training flags as train_hf_jobs.sh (the released run). Not run on the cluster yet: the released\n# adapters were trained on HF Jobs. One-time setup on a login node: see ../README.md.\n#\n#   sbatch train_slurm.sh\n#   MAX_STEPS=2 sbatch -p hopper-prod train_slurm.sh\n#SBATCH --job-name=ha-twil\n#SBATCH --partition=hopper-b200\n#SBATCH --nodes=1\n#SBATCH --ntasks-per-node=1\n#SBATCH --gpus-per-node=4\n#SBATCH --cpus-per-task=64\n#SBATCH --time=2-00:00:00\n#SBATCH --output=/fsx/%u/logs/%x-%j.out\nset -euo pipefail\n\nROOT=\"${ROOT:-/fsx/$USER/harness-adapters}\"\nMAX_STEPS=\"${MAX_STEPS:-100}\"\nREV=191a751bda511eada76f6a9a76513d143321a64a  # harness-hub/traj2envs-rl revision the adapters were trained on\nsource \"$ROOT/scripts/slurm_setup.sh\"\n\npython \"$ROOT/scripts/launcher_train.py\" \\\n    --data-root \"$ROOT/data\" \\\n    --train-script-url \"file://$ROOT/scripts/async_grpo_harbor.py\" \\\n    --vllm-port \"$VLLM_PORT\" --server-port \"$SERVER_PORT\" --capture-port \"$CAPTURE_PORT\" --expose gradio \\\n    --model webAI-Official/TwIL-8B \\\n    --vllm-extra-args=\"--chat-template $ROOT/TwIL-8B/chat_template.jinja\" \\\n    --tool-call-parser qwen3_coder --reasoning-parser qwen3 --max-model-len 131072 \\\n    --run pi:pi:medium:150,hard:250 \\\n    --run hermes:hermes:medium:150,hard:250 \\\n    --run opencode:opencode:medium:150,hard:250 \\\n    --split FineEnvs/SmolDataEnvs-harbor-train --n-tasks 400 \\\n    --mix-split \"harness-hub/traj2envs-rl@$REV:230\" --max-output-tokens 32768 \\\n    --sandbox hf-sandbox --docker-image docker.io/ybelkada/env-data-agent-train:opencode-1.19.5 \\\n    --vllm-device 0 --train-device 1,2,3 \\\n    --max-inflight 12 --max-lora-rank 32 --max-loras 18 \\\n    --reward-key correctness,reward --num-generations 8 --gradient-accumulation-steps 4 --max-staleness 4 \\\n    --max-completion-length 16384 --agent-step-limit 30 --agent-timeout 1200 --temperature 0.8 \\\n    --lora-r 32 --lora-target-modules all-linear --learning-rate 1e-5 --tool-efficiency-weight 0.1 \\\n    --token-budget 65536 \\\n    --qwen3-5-response-template \\\n    --safeagents --safeagents-plugin \"$ROOT/scripts/atif_smoldataenvs.py:checks\" \\\n    --safeagents-plugin \"$ROOT/scripts/atif_traj2envs.py:checks\" \\\n    --max-steps \"$MAX_STEPS\" --save-steps 10 --save-total-limit 0\n",
    "eval_local.sh": "#!/bin/bash\n# Held-out pass@1 of TwIL-8B and its harness adapters on one local GPU. One vLLM serves the base and every\n# checkpoint as an adapter, with one sandbox server per checkpoint. Results land in $DATA_ROOT/eval/<label>_<split>/.\n#\n# Same flags as eval_hf_jobs.sh, run here with `uv run`. Not run locally yet.\n#\n#   bash eval_local.sh                                                         the base model\n#   CHECKPOINTS=\"webAI-Official/TwIL-8B@pi,hermes,opencode $PWD/outputs/runs/<run>/checkpoint-50@pi\" bash eval_local.sh\n#   SPLIT=traj2envs-heldout bash eval_local.sh\nset -euo pipefail\n\nDATA_ROOT=\"${DATA_ROOT:-$PWD/outputs}\"\nSPLIT=\"${SPLIT:-final-heldout}\"  # final-heldout | traj2envs-heldout\n# <path|glob|hub id>@<harnesses>, space-separated\nCHECKPOINTS=\"${CHECKPOINTS:-webAI-Official/TwIL-8B@pi,hermes,opencode}\"\nHERE=\"$(cd \"$(dirname \"$0\")\" && pwd)\"\nSCRIPTS=\"$HERE/../scripts\"\nexport HF_TOKEN=\"${HF_TOKEN:-$(cat ~/.cache/huggingface/token)}\"\nexport JOB_ID=\"${JOB_ID:-local-$(date +%Y%m%d-%H%M%S)}\"\n\nif [ \"$SPLIT\" = final-heldout ]; then\n    SPLIT_ARGS=(--split FineEnvs/SmolDataEnvs-harbor-train --task-offset 1000 --difficulty easy,medium --n-tasks 150\n        --docker-image docker.io/ybelkada/env-data-agent-train:opencode-1.19.5)\nelse\n    SPLIT_ARGS=(--split harness-hub/traj2envs-rl@191a751bda511eada76f6a9a76513d143321a64a --task-offset 230 --n-tasks 40)\nfi\nread -r -a CHECKPOINT_LIST <<< \"$CHECKPOINTS\"\n\nuv run \"$SCRIPTS/launcher_eval.py\" \\\n    --data-root \"$DATA_ROOT\" --eval-script-url \"file://$SCRIPTS/evaluate_harbor.py\" \\\n    --eval-lora --thinking --sandbox hf-sandbox --expose gradio \\\n    --model webAI-Official/TwIL-8B --max-model-len 131072 --eval-tag \"$SPLIT\" \\\n    --eval-harnesses pi,hermes,opencode \\\n    --eval-checkpoints \"${CHECKPOINT_LIST[@]}\" \\\n    --vllm-extra-args=\"--chat-template $HERE/chat_template.jinja\" \\\n    --tool-call-parser qwen3_coder --reasoning-parser qwen3 \\\n    --max-output-tokens 32768 --max-inflight 24 --agent-step-limit 30 --agent-timeout 1200 \\\n    \"${SPLIT_ARGS[@]}\"\n",
    "eval_hf_jobs.sh": "#!/bin/bash\n# Held-out pass@1 of TwIL-8B and its harness adapters on HF Jobs (1x H200). One vLLM serves the base and every\n# checkpoint as an adapter, with one sandbox server per checkpoint, all evaluated at once. Results land in\n# /outputs/eval/<label>_<split>/summary.json on the bucket.\n#\n# Same flags as the scheduled sweep that evaluated the released run (../scripts/eval_sweep.py).\n#\n#   bash eval_hf_jobs.sh                                      base + the released checkpoints, on SmolDataEnvs held-out\n#   SPLIT=traj2envs-heldout bash eval_hf_jobs.sh              the traj2envs held-out tasks instead\n#   CHECKPOINTS=\"/outputs/runs/<run>/checkpoint-50@pi\" bash eval_hf_jobs.sh\nset -euo pipefail\n\nBUCKET=\"${BUCKET:-ybelkada/harbor-evals}\"\nSPLIT=\"${SPLIT:-final-heldout}\"  # final-heldout | traj2envs-heldout\nJOB=6aedb49dc78863f9c63408be     # the released training job, whose runs are under /outputs/runs\n# <path|glob|hub id>@<harnesses>, space-separated\nCHECKPOINTS=\"${CHECKPOINTS:-webAI-Official/TwIL-8B@pi,hermes,opencode \\\n/outputs/runs/TwIL-8B-pi-$JOB-*/checkpoint-90@pi \\\n/outputs/runs/TwIL-8B-hermes-$JOB-*/checkpoint-60@hermes \\\n/outputs/runs/TwIL-8B-opencode-$JOB-*/checkpoint-100@opencode}\"\nHERE=\"$(cd \"$(dirname \"$0\")\" && pwd)\"\nSCRIPTS=\"$HERE/../scripts\"\n\nif [ \"$SPLIT\" = final-heldout ]; then\n    SPLIT_ARGS=(--split FineEnvs/SmolDataEnvs-harbor-train --task-offset 1000 --difficulty easy,medium --n-tasks 150\n        --docker-image docker.io/ybelkada/env-data-agent-train:opencode-1.19.5)\nelse\n    SPLIT_ARGS=(--split harness-hub/traj2envs-rl@191a751bda511eada76f6a9a76513d143321a64a --task-offset 230 --n-tasks 40)\nfi\nread -r -a CHECKPOINT_LIST <<< \"$CHECKPOINTS\"\n\n# The job reads the eval client and the chat template from the bucket, mounted on /outputs\nhf buckets cp \"$SCRIPTS/evaluate_harbor.py\" \"hf://buckets/$BUCKET/evaluate_harbor.py\"\nhf buckets cp \"$HERE/chat_template.jinja\" \"hf://buckets/$BUCKET/twil_chat_template.jinja\"\n\nhf jobs uv run --detach \\\n    --flavor h200 --image huggingface/trl --secrets HF_TOKEN --timeout 6h \\\n    --volume \"hf://buckets/$BUCKET:/outputs\" \\\n    \"$SCRIPTS/launcher_eval.py\" -- \\\n    --data-root /outputs --eval-lora --thinking --sandbox hf-sandbox --expose gradio \\\n    --model webAI-Official/TwIL-8B --max-model-len 131072 --eval-tag \"$SPLIT\" \\\n    --eval-harnesses pi,hermes,opencode \\\n    --eval-checkpoints \"${CHECKPOINT_LIST[@]}\" \\\n    --vllm-extra-args=\"--chat-template /outputs/twil_chat_template.jinja\" \\\n    --tool-call-parser qwen3_coder --reasoning-parser qwen3 \\\n    --max-output-tokens 32768 --max-inflight 24 --agent-step-limit 30 --agent-timeout 1200 \\\n    \"${SPLIT_ARGS[@]}\"\n",
    "eval_slurm.sh": "#!/bin/bash -l\n# Held-out pass@1 of TwIL-8B and its harness adapters on one cluster GPU. One vLLM serves the base and every checkpoint\n# as an adapter, with one sandbox server per checkpoint. Results land in $ROOT/data/eval/<label>_<split>/.\n#\n# Same flags as eval_hf_jobs.sh. Not run on the cluster yet. One-time setup on a login node: see ../README.md.\n#\n#   CHECKPOINTS=\"webAI-Official/TwIL-8B@pi,hermes,opencode /fsx/$USER/harness-adapters/data/runs/<run>/checkpoint-50@pi\" \\\n#       sbatch eval_slurm.sh\n#   SPLIT=traj2envs-heldout CHECKPOINTS=... sbatch eval_slurm.sh\n#SBATCH --job-name=ha-twil-eval\n#SBATCH --partition=hopper-b200\n#SBATCH --nodes=1\n#SBATCH --ntasks-per-node=1\n#SBATCH --gpus-per-node=1\n#SBATCH --cpus-per-task=16\n#SBATCH --time=6:00:00\n#SBATCH --output=/fsx/%u/logs/%x-%j.out\nset -euo pipefail\n\nROOT=\"${ROOT:-/fsx/$USER/harness-adapters}\"\nSPLIT=\"${SPLIT:-final-heldout}\"  # final-heldout | traj2envs-heldout\n# <path|glob|hub id>@<harnesses>, space-separated\nCHECKPOINTS=\"${CHECKPOINTS:-webAI-Official/TwIL-8B@pi,hermes,opencode}\"\nsource \"$ROOT/scripts/slurm_setup.sh\"\n\nif [ \"$SPLIT\" = final-heldout ]; then\n    SPLIT_ARGS=(--split FineEnvs/SmolDataEnvs-harbor-train --task-offset 1000 --difficulty easy,medium --n-tasks 150\n        --docker-image docker.io/ybelkada/env-data-agent-train:opencode-1.19.5)\nelse\n    SPLIT_ARGS=(--split harness-hub/traj2envs-rl@191a751bda511eada76f6a9a76513d143321a64a --task-offset 230 --n-tasks 40)\nfi\nread -r -a CHECKPOINT_LIST <<< \"$CHECKPOINTS\"\n\npython \"$ROOT/scripts/launcher_eval.py\" \\\n    --data-root \"$ROOT/data\" --eval-script-url \"file://$ROOT/scripts/evaluate_harbor.py\" \\\n    --vllm-port \"$VLLM_PORT\" --server-port \"$SERVER_PORT\" --capture-port \"$CAPTURE_PORT\" \\\n    --eval-lora --thinking --sandbox hf-sandbox --expose gradio \\\n    --model webAI-Official/TwIL-8B --max-model-len 131072 --eval-tag \"$SPLIT\" \\\n    --eval-harnesses pi,hermes,opencode \\\n    --eval-checkpoints \"${CHECKPOINT_LIST[@]}\" \\\n    --vllm-extra-args=\"--chat-template $ROOT/TwIL-8B/chat_template.jinja\" \\\n    --tool-call-parser qwen3_coder --reasoning-parser qwen3 \\\n    --max-output-tokens 32768 --max-inflight 24 --agent-step-limit 30 --agent-timeout 1200 \\\n    \"${SPLIT_ARGS[@]}\"\n",
    "recipe.yaml": "meta:\n  title: \"TwIL-8B harness adapters\"\n  project: harness-adapters\n  trainer: AsyncGRPOTrainer\n  model_id: webAI-Official/TwIL-8B\n  description: \"Experiment: one LoRA adapter per agent harness (pi, hermes, opencode) on a model not trained on agentic data, with more agent steps and a longer rollout timeout.\"\n  tags: [lora, agentic-rl, harbor, safeagents, multi-harness, experiment]\n  harnesses: [pi, hermes, opencode]\n  date_added: 2026-11-13\n  blog: https://huggingface.co/spaces/harness-hub/harness-adapters\n  dashboard: https://harness-hub-harness-adapters-rl.hf.space/\n\nhardware:\n  hf_jobs: h200x4            # vLLM on GPU 0, one trainer per harness on GPUs 1-3\n  slurm: \"hopper-b200, 4 GPUs (untested)\"\n  local: \"4 GPUs, H200 class (untested)\"\n\ndatasets:\n  train:\n    - FineEnvs/SmolDataEnvs-harbor-train       # 150 medium + 250 hard\n    - harness-hub/traj2envs-rl@191a751bda511eada76f6a9a76513d143321a64a  # first 230 tasks\n  eval:\n    final-heldout: \"FineEnvs/SmolDataEnvs-harbor-train, 150 easy/medium tasks from index 1000\"\n    traj2envs-heldout: \"harness-hub/traj2envs-rl, the 40 tasks from index 230\"\n\nhyperparameters:\n  lora_r: 32\n  lora_target_modules: all-linear\n  learning_rate: 1.0e-5\n  num_generations: 8\n  gradient_accumulation_steps: 4\n  max_staleness: 4\n  temperature: 0.8\n  max_completion_length: 16384\n  max_output_tokens: 32768\n  max_model_len: 131072\n  token_budget: 65536\n  agent_step_limit: 30\n  agent_timeout_s: 1200\n  max_inflight: 12\n  tool_efficiency_weight: 0.1\n  max_steps: 100              # the blog post says 300; the launch command and the released checkpoints stop at 100\n  save_steps: 10\n\nvllm:\n  tool_call_parser: qwen3_coder\n  reasoning_parser: qwen3\n  chat_template: chat_template.jinja\n  trainer_flags: [--qwen3-5-response-template]\n\ntrl:\n  fork: \"younesbelkada/trl@ee6e9ad9512b83c80bb0051942770c8798c0dbf2 (add-lfm-tree-attn)\"\n  upstream_base: 7fdad81d04303e6f43db168e903565769c0b8642\n  patch: trl.patch            # the fork's changes to trl/, relative to upstream_base\n\nprovenance:\n  hf_job: 6aedb49dc78863f9c63408be\n  released:\n    pi: {step: 90, repo: harness-hub/TwIL-8B-pi}\n    hermes: {step: 60, repo: harness-hub/TwIL-8B-hermes}\n    opencode: {step: 100, repo: harness-hub/TwIL-8B-opencode}\n\nscripts:\n  train: {hf_jobs: train_hf_jobs.sh, slurm: train_slurm.sh, local: train_local.sh}\n  eval: {hf_jobs: eval_hf_jobs.sh, slurm: eval_slurm.sh, local: eval_local.sh}\n\n# What the command builder on the recipes site offers. Each option is an environment variable of the script it runs.\nbuilder:\n  verified: {hf_jobs: true, slurm: false, local: false}   # whether the scripts of each platform ran as written\n  eval:\n    - env: SPLIT\n      label: Held-out split\n      choices: [final-heldout, traj2envs-heldout]\n    - env: CHECKPOINTS\n      label: Checkpoints\n      input: true\n      placeholder: \"<path|glob|hub id>@<harnesses> ... (empty: the script's default)\"\n",
    "trl.patch": "TRL changes these recipes run on, relative to upstream huggingface/trl.\n  upstream base: 7fdad81d04303e6f43db168e903565769c0b8642 (2026-10-28, fix: add peft on asyncgrpo openenv harbor + fixes (#7948))\n  fork:          younesbelkada/trl add-lfm-tree-attn = ee6e9ad9512b83c80bb0051942770c8798c0dbf2\nFork commits:\n  ee6e9ad9 Merge branch 'main' into add-lfm-tree-attn\n  c474c2cc Apply suggestion from @younesbelkada\n  eb5d5b9b Merge branch 'main' into add-lfm-tree-attn\n  2fa050c4 revert\n  b467898a feat: add LFM support for tree attn\nApply on a checkout of the upstream base: git apply trl.patch\nRegenerate: git diff $(git merge-base origin/main <fork commit>) <fork commit> -- trl/\n\ndiff --git a/trl/experimental/async_grpo/async_grpo_trainer.py b/trl/experimental/async_grpo/async_grpo_trainer.py\nindex 66a5d595..2e609d6f 100644\n--- a/trl/experimental/async_grpo/async_grpo_trainer.py\n+++ b/trl/experimental/async_grpo/async_grpo_trainer.py\n@@ -58,7 +58,7 @@ from ...trainer.utils import (\n )\n from .async_grpo_config import AsyncGRPOConfig\n from .async_rollout_worker import AsyncRolloutWorker, RolloutSample\n-from .packing import TREE_ATTENTION, TreeRow, patch_tree_gdn, register_tree_attention, unpad_tree\n+from .packing import TREE_ATTENTION, TreeRow, patch_tree_recurrent, register_tree_attention, unpad_tree\n from .vllm_client import VLLMClient\n from .weight_transfer import WeightTransferClient\n \n@@ -1122,11 +1122,17 @@ class AsyncGRPOTrainer(_BaseTrainer):\n                     \"before it, these layers call the functions they stored at init rather than the module-level \"\n                     \"ones tree packing patches, so they would scan across samples.\"\n                 )\n-            if layer_types != {\"linear_attention\"} or not patch_tree_gdn(model):\n+            if layer_types == {\"conv\"} and Version(transformers.__version__) <= Version(\"5.21.0.dev0\"):\n                 raise ValueError(\n-                    f\"{type(model).__name__} has {sorted(layer_types)} layers. Tree packing routes attention and \"\n-                    \"transformers' gated DeltaNet layers along each sample's own path; any other recurrent or \"\n-                    \"convolutional layer would scan across samples.\"\n+                    f\"Tree packing of {type(model).__name__}'s short convolution layers requires transformers>=5.22.0: \"\n+                    \"before it, these layers do not pass the forward's keyword arguments down to the module-level \"\n+                    \"function tree packing patches, so they would scan across samples.\"\n+                )\n+            if layer_types not in ({\"linear_attention\"}, {\"conv\"}) or not patch_tree_recurrent(model, *layer_types):\n+                raise ValueError(\n+                    f\"{type(model).__name__} has {sorted(layer_types)} layers. Tree packing routes attention, \"\n+                    \"transformers' gated DeltaNet layers and LFM2's short convolutions along each sample's own path; \"\n+                    \"any other recurrent or convolutional layer would scan across samples.\"\n                 )\n         if any(name.endswith(\".sinks\") for name, _ in model.named_parameters()):\n             raise ValueError(\ndiff --git a/trl/experimental/async_grpo/packing.py b/trl/experimental/async_grpo/packing.py\nindex c22132f3..24ed508d 100644\n--- a/trl/experimental/async_grpo/packing.py\n+++ b/trl/experimental/async_grpo/packing.py\n@@ -119,7 +119,7 @@ class TreeRow:\n                 - `tree_cu_seqlens_q` (`S + 1`), `tree_cu_seqlens_k` (`S + 1`): segment boundaries in the row, path\n                   boundaries in `tree_kv_index`.\n                 - `tree_segment_parents` (`S`), `tree_segment_depths` (`S`): the segment tree, for the recurrent layers\n-                  patched by [`patch_tree_gdn`].\n+                  patched by [`patch_tree_recurrent`].\n                 - `tree_sizes` (`4 + depth`): longest segment, longest path, `K`, 64-token chunks, then the segments\n                   per depth. Every length the kernels launch with, so a rank reads them in one go (see [`unpad_tree`]).\n                 - `token_index` (`M`): position whose prediction is each trained token. A shared position serves every\n@@ -278,20 +278,27 @@ def register_tree_attention() -> None:\n _PATCHED = set()\n \n \n-def patch_tree_gdn(model) -> bool:\n+def patch_tree_recurrent(model, layer_type: str) -> bool:\n     \"\"\"\n-    Make the model's gated DeltaNet layers follow the tree of a tree-packed forward.\n+    Make the model's gated DeltaNet or short convolution layers follow the tree of a tree-packed forward.\n \n-    Transformers' hybrids (Qwen3.5, Qwen3-Next, OLMo-Hybrid, ...) run these layers through two module-level functions,\n-    `causal_conv1d_fn` and `torch_chunk_gated_delta_rule`, and pass the forward's keyword arguments down to both. Both\n-    are wrapped: given the tree of [`~packing.TreeRow.pack`] they follow it, otherwise they run unchanged.\n+    Transformers' gated DeltaNet hybrids (Qwen3.5, Qwen3-Next, OLMo-Hybrid, ...) run their `\"linear_attention\"` layers\n+    through two module-level functions, `causal_conv1d_fn` and `torch_chunk_gated_delta_rule`, and LFM2's `\"conv\"`\n+    layers through the first alone. These layers pass the forward's keyword arguments down to them. Each is wrapped:\n+    given the tree of [`~packing.TreeRow.pack`] they follow it, otherwise they run unchanged.\n+\n+    Args:\n+        model ([`~transformers.PreTrainedModel`]):\n+            Model to patch.\n+        layer_type (`str`):\n+            The model's only non-attention layer type, `\"linear_attention\"` or `\"conv\"`.\n \n     Returns:\n-        `bool`: whether the model's modeling module has those functions.\n+        `bool`: whether the model's modeling module has the functions its layers run through.\n     \"\"\"\n     modeling = sys.modules[type(model).__module__]\n     conv, chunk = vars(modeling).get(\"causal_conv1d_fn\"), vars(modeling).get(\"torch_chunk_gated_delta_rule\")\n-    if conv is None or chunk is None:\n+    if conv is None or (layer_type == \"linear_attention\" and chunk is None):\n         return False\n     if modeling in _PATCHED:\n         return True\n@@ -307,6 +314,8 @@ def patch_tree_gdn(model) -> bool:\n             return chunk(query, key, value, g=g, beta=beta, use_qk_l2norm_in_kernel=use_qk_l2norm_in_kernel, **kwargs)\n         return tree_chunk_gated_delta_rule(query, key, value, g, beta, kwargs, use_qk_l2norm_in_kernel), None\n \n-    modeling.causal_conv1d_fn, modeling.torch_chunk_gated_delta_rule = causal_conv1d_fn, torch_chunk_gated_delta_rule\n+    modeling.causal_conv1d_fn = causal_conv1d_fn\n+    if chunk is not None:\n+        modeling.torch_chunk_gated_delta_rule = torch_chunk_gated_delta_rule\n     _PATCHED.add(modeling)\n     return True\ndiff --git a/trl/trainer/utils.py b/trl/trainer/utils.py\nindex 5e53bc49..e72efc9b 100644\n--- a/trl/trainer/utils.py\n+++ b/trl/trainer/utils.py\n@@ -1802,18 +1802,21 @@ def compute_flops_per_token(config: PretrainedConfig, seq_len: int) -> int:\n         mlp_flops = 2 * 3 * h * config.intermediate_size\n         total_layer_flops = L * (attn_flops + mlp_flops)\n     else:\n-        # Routed experts (gate + up + down, 3 matmuls each) + router.\n-        if Version(transformers.__version__) >= Version(\"5.1.0\"):\n-            num_experts = config.num_local_experts\n-        else:\n+        # Routed experts (gate + up + down, 3 matmuls each) + router. LFM2-MoE keeps `num_experts` on every version, and\n+        # opens with `num_dense_layers` dense layers where Qwen-style MoE spaces them by `decoder_sparse_step`.\n+        if config.model_type == \"lfm2_moe\":\n             num_experts = config.num_experts\n+            is_sparse = [layer_idx >= config.num_dense_layers for layer_idx in range(L)]\n+        else:\n+            if Version(transformers.__version__) >= Version(\"5.1.0\"):\n+                num_experts = config.num_local_experts\n+            else:\n+                num_experts = config.num_experts\n+            is_sparse = [layer_idx % config.decoder_sparse_step == 0 for layer_idx in range(L)]\n         moe_mlp_flops = num_experts_per_tok * 2 * 3 * h * config.moe_intermediate_size\n         moe_mlp_flops += 2 * h * num_experts\n         dense_mlp_flops = 2 * 3 * h * config.intermediate_size  # interspersed dense layers\n-        sparse_step = config.decoder_sparse_step\n-        total_layer_flops = sum(\n-            attn_flops + (moe_mlp_flops if layer_idx % sparse_step == 0 else dense_mlp_flops) for layer_idx in range(L)\n-        )\n+        total_layer_flops = sum(attn_flops + (moe_mlp_flops if sparse else dense_mlp_flops) for sparse in is_sparse)\n \n     # Embedding is a lookup (no FLOPs); lm_head is a 2*V*h matmul, tied or not.\n     lm_head_flops = 2 * V * h\n"
   },
   "other_files": [
    "chat_template.jinja"
   ]
  },
  {
   "id": "SFTTrainer/sft-qlora/LFM2-2.6B",
   "trainer": "SFTTrainer",
   "project": "sft-qlora",
   "name": "LFM2-2.6B",
   "org": "LiquidAI",
   "avatar": "https://cdn-avatars.huggingface.co/v1/production/uploads/61b8e2ba285851687028d395/EsTgVtnM2IqVRKgPdfqcB.png",
   "meta": {
    "title": "LFM2-2.6B SFT with QLoRA",
    "project": "sft-qlora",
    "trainer": "SFTTrainer",
    "model_id": "LiquidAI/LFM2-2.6B",
    "description": "QLoRA SFT on Multilingual-Thinking, so the model reasons in the language the system prompt asks for. 4-bit base, LoRA r32, 30 steps: fits one 16 GB T4 (~5.9 GB VRAM).",
    "tags": [
     "sft",
     "qlora",
     "4-bit",
     "reasoning",
     "multilingual"
    ],
    "date_added": "2026-11-13",
    "source": "https://github.com/huggingface/trl/blob/ebf09f6b7e126af6e34ff172bb74e1629ff24156/examples/sft_qlora/sft_qlora.ipynb"
   },
   "hardware": {
    "hf_jobs": "t4-small",
    "slurm": "hopper-prod, 1 GPU",
    "local": "1 GPU with 16 GB or more"
   },
   "datasets": {
    "train": [
     "HuggingFaceH4/Multilingual-Thinking"
    ]
   },
   "hyperparameters": {
    "quantization": "4-bit nf4, double quant, float16 compute",
    "lora_r": 32,
    "lora_alpha": 32,
    "lora_target_modules": [
     "q_proj",
     "k_proj",
     "v_proj",
     "o_proj",
     "gate_proj",
     "up_proj",
     "down_proj"
    ],
    "learning_rate": 0.0002,
    "optim": "paged_adamw_8bit",
    "per_device_train_batch_size": 1,
    "gradient_accumulation_steps": 4,
    "warmup_steps": 5,
    "max_steps": 30,
    "max_length": 1024,
    "dtype": "float16",
    "activation_offloading": true
   },
   "trl": {
    "version": "1.17.1"
   },
   "provenance": {
    "notebook": "huggingface/trl examples/sft_qlora/sft_qlora.ipynb @ ebf09f6b",
    "notebook_vram": "~5.9 GB on a free Colab T4"
   },
   "scripts": {
    "train": {
     "hf_jobs": "train_hf_jobs.sh",
     "slurm": "train_slurm.sh",
     "local": "train_local.sh"
    },
    "eval": {
     "hf_jobs": "eval_hf_jobs.sh",
     "slurm": "eval_slurm.sh",
     "local": "eval_local.sh"
    }
   },
   "builder": {
    "verified": {
     "hf_jobs": false,
     "slurm": false,
     "local": false
    },
    "eval": [
     {
      "env": "ADAPTER",
      "label": "Adapter",
      "input": true,
      "default": "<your user>/LFM2-2.6B-SFT"
     }
    ]
   },
   "readme": "# LFM2-2.6B: SFT with QLoRA\n\n[`LiquidAI/LFM2-2.6B`](https://huggingface.co/LiquidAI/LFM2-2.6B) fine-tuned with QLoRA on\n[HuggingFaceH4/Multilingual-Thinking](https://huggingface.co/datasets/HuggingFaceH4/Multilingual-Thinking), so it\nwrites its reasoning in the language the system prompt asks for. The notebook measured ~5.9 GB of VRAM on a free\nColab T4. Shared settings: [../README.md](../README.md). Full hyperparameters: [recipe.yaml](recipe.yaml).\n\nLFM2 names its layers differently: of the notebook's LoRA targets only `q_proj`, `k_proj` and `v_proj` exist in it, so the adapter trains the attention projections only (as in the notebook). Pass `--lora-target-modules q_proj,k_proj,v_proj,out_proj,in_proj,w1,w2,w3` to `sft_qlora.py` to adapt every linear layer.\n\n| | HF Jobs | Cluster | Local |\n|---|---|---|---|\n| Train | `bash train_hf_jobs.sh` (t4-small) | `sbatch train_slurm.sh` (1 GPU) | `bash train_local.sh` (1 GPU, 16 GB) |\n| Check | `ADAPTER=<your user>/LFM2-2.6B-SFT bash eval_hf_jobs.sh` | `ADAPTER=... sbatch eval_slurm.sh` | `ADAPTER=... bash eval_local.sh` |\n\nTraining pushes the adapter to `<your user>/LFM2-2.6B-SFT` and logs to the Trackio Space of the same name. The check prints\nthe base model's answer, then the adapter's: the adapter's reasoning should be in German.\n",
   "files": {
    "train_local.sh": "#!/bin/bash\n# SFT with QLoRA of LiquidAI/LFM2-2.6B on this machine's GPU (16 GB is enough: the notebook ran on a T4), with `uv run`: the\n# settings of TRL's sft_qlora notebook. The adapter is saved to $OUTPUT_ROOT/LFM2-2.6B-SFT, pushed to <your user>/LFM2-2.6B-SFT,\n# and the run is logged to the Trackio Space of the same name.\n#\n#   bash train_local.sh\n#   MAX_STEPS=2 bash train_local.sh                   short smoke run\nset -euo pipefail\n\nMAX_STEPS=\"${MAX_STEPS:-30}\"\nOUTPUT_ROOT=\"${OUTPUT_ROOT:-$PWD/outputs}\"\nSCRIPTS=\"$(cd \"$(dirname \"$0\")/../scripts\" && pwd)\"\n\nuv run \"$SCRIPTS/sft_qlora.py\" \\\n    --model LiquidAI/LFM2-2.6B --output-root \"$OUTPUT_ROOT\" --max-steps \"$MAX_STEPS\"\n",
    "train_hf_jobs.sh": "#!/bin/bash\n# SFT with QLoRA of LiquidAI/LFM2-2.6B on HF Jobs, one 16 GB T4: the settings of TRL's sft_qlora notebook. The adapter is pushed to\n# <your user>/LFM2-2.6B-SFT and the run is logged to the Trackio Space of the same name.\n#\n#   bash train_hf_jobs.sh\n#   MAX_STEPS=2 bash train_hf_jobs.sh                 short smoke run\nset -euo pipefail\n\nMAX_STEPS=\"${MAX_STEPS:-30}\"\nSCRIPTS=\"$(cd \"$(dirname \"$0\")/../scripts\" && pwd)\"\n\nhf jobs uv run --detach \\\n    --flavor t4-small --secrets HF_TOKEN --timeout 2h \\\n    \"$SCRIPTS/sft_qlora.py\" -- \\\n    --model LiquidAI/LFM2-2.6B --max-steps \"$MAX_STEPS\"\n",
    "train_slurm.sh": "#!/bin/bash -l\n# SFT with QLoRA of LiquidAI/LFM2-2.6B on one cluster GPU: the settings of TRL's sft_qlora notebook (made for a 16 GB T4, so an\n# H100 has room to spare). The adapter is saved to $OUTPUT_ROOT/LFM2-2.6B-SFT and pushed to <your user>/LFM2-2.6B-SFT.\n#\n#   sbatch train_slurm.sh                              from this folder: the shared scripts are found from it\n#   MAX_STEPS=2 sbatch train_slurm.sh                  short smoke run\n#SBATCH --job-name=sft-qlora-lfm2-2.6b\n#SBATCH --partition=hopper-prod\n#SBATCH --nodes=1\n#SBATCH --ntasks-per-node=1\n#SBATCH --gpus-per-node=1\n#SBATCH --cpus-per-task=12\n#SBATCH --time=2:00:00\n#SBATCH --output=/fsx/%u/logs/%x-%j.out\nset -euo pipefail\n\nMAX_STEPS=\"${MAX_STEPS:-30}\"\nOUTPUT_ROOT=\"${OUTPUT_ROOT:-/fsx/$USER/sft-qlora}\"\nexport HF_TOKEN=\"${HF_TOKEN:-$(cat ~/.cache/huggingface/token)}\"\n# Model weights and uv's package cache on /fsx, not the small NFS home\nexport HF_HUB_CACHE=\"${HF_HUB_CACHE:-/fsx/$USER/hf-hub}\"\nexport UV_CACHE_DIR=\"${UV_CACHE_DIR:-/fsx/$USER/.cache/uv}\"\nmkdir -p \"/fsx/$USER/logs\" \"$OUTPUT_ROOT\"\n\n# sbatch runs a copy of this script, so the shared scripts are found from the folder it was submitted from\nuv run \"$SLURM_SUBMIT_DIR/../scripts/sft_qlora.py\" \\\n    --model LiquidAI/LFM2-2.6B --output-root \"$OUTPUT_ROOT\" --max-steps \"$MAX_STEPS\"\n",
    "eval_local.sh": "#!/bin/bash\n# The notebook's check of the trained adapter on this machine's GPU: generate for a prompt asking for German reasoning,\n# with the base model and then with the adapter.\n#\n#   ADAPTER=<your user>/LFM2-2.6B-SFT bash eval_local.sh\n#   ADAPTER=$PWD/outputs/LFM2-2.6B-SFT bash eval_local.sh\nset -euo pipefail\n\nADAPTER=\"${ADAPTER:?ADAPTER=<your user>/LFM2-2.6B-SFT or the run folder train_local.sh saved}\"\nSCRIPTS=\"$(cd \"$(dirname \"$0\")/../scripts\" && pwd)\"\n\nuv run \"$SCRIPTS/compare_generations.py\" --model LiquidAI/LFM2-2.6B --adapter \"$ADAPTER\"\n",
    "eval_hf_jobs.sh": "#!/bin/bash\n# The notebook's check of the trained adapter on HF Jobs (one T4): generate for a prompt asking for German reasoning,\n# with the base model and then with the adapter. Read the two outputs with `hf jobs logs <job id>`.\n#\n#   ADAPTER=<your user>/LFM2-2.6B-SFT bash eval_hf_jobs.sh\nset -euo pipefail\n\nADAPTER=\"${ADAPTER:?ADAPTER=<your user>/LFM2-2.6B-SFT, the adapter train_hf_jobs.sh pushed}\"\nSCRIPTS=\"$(cd \"$(dirname \"$0\")/../scripts\" && pwd)\"\n\nhf jobs uv run --detach \\\n    --flavor t4-small --secrets HF_TOKEN --timeout 1h \\\n    \"$SCRIPTS/compare_generations.py\" -- \\\n    --model LiquidAI/LFM2-2.6B --adapter \"$ADAPTER\"\n",
    "eval_slurm.sh": "#!/bin/bash -l\n# The notebook's check of the trained adapter on one cluster GPU: generate for a prompt asking for German reasoning,\n# with the base model and then with the adapter. The two outputs are in the job's log.\n#\n#   ADAPTER=<your user>/LFM2-2.6B-SFT sbatch eval_slurm.sh\n#   ADAPTER=/fsx/$USER/sft-qlora/LFM2-2.6B-SFT sbatch eval_slurm.sh\n#SBATCH --job-name=sft-qlora-lfm2-2.6b-eval\n#SBATCH --partition=hopper-prod\n#SBATCH --nodes=1\n#SBATCH --ntasks-per-node=1\n#SBATCH --gpus-per-node=1\n#SBATCH --cpus-per-task=12\n#SBATCH --time=0:30:00\n#SBATCH --output=/fsx/%u/logs/%x-%j.out\nset -euo pipefail\n\nADAPTER=\"${ADAPTER:?ADAPTER=<your user>/LFM2-2.6B-SFT or the run folder train_slurm.sh saved}\"\nexport HF_TOKEN=\"${HF_TOKEN:-$(cat ~/.cache/huggingface/token)}\"\nexport HF_HUB_CACHE=\"${HF_HUB_CACHE:-/fsx/$USER/hf-hub}\"\nexport UV_CACHE_DIR=\"${UV_CACHE_DIR:-/fsx/$USER/.cache/uv}\"\nmkdir -p \"/fsx/$USER/logs\"\n\nuv run \"$SLURM_SUBMIT_DIR/../scripts/compare_generations.py\" --model LiquidAI/LFM2-2.6B --adapter \"$ADAPTER\"\n",
    "recipe.yaml": "meta:\n  title: \"LFM2-2.6B SFT with QLoRA\"\n  project: sft-qlora\n  trainer: SFTTrainer\n  model_id: LiquidAI/LFM2-2.6B\n  description: \"QLoRA SFT on Multilingual-Thinking, so the model reasons in the language the system prompt asks for. 4-bit base, LoRA r32, 30 steps: fits one 16 GB T4 (~5.9 GB VRAM).\"\n  tags: [sft, qlora, 4-bit, reasoning, multilingual]\n  date_added: 2026-11-13\n  source: https://github.com/huggingface/trl/blob/ebf09f6b7e126af6e34ff172bb74e1629ff24156/examples/sft_qlora/sft_qlora.ipynb\n\nhardware:\n  hf_jobs: t4-small\n  slurm: \"hopper-prod, 1 GPU\"\n  local: \"1 GPU with 16 GB or more\"\n\ndatasets:\n  train:\n    - HuggingFaceH4/Multilingual-Thinking\n\nhyperparameters:\n  quantization: \"4-bit nf4, double quant, float16 compute\"\n  lora_r: 32\n  lora_alpha: 32\n  lora_target_modules: [q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj]\n  learning_rate: 2.0e-4\n  optim: paged_adamw_8bit\n  per_device_train_batch_size: 1\n  gradient_accumulation_steps: 4\n  warmup_steps: 5\n  max_steps: 30\n  max_length: 1024\n  dtype: float16\n  activation_offloading: true\n\ntrl:\n  version: \"1.17.1\"            # from PyPI, no fork\n\nprovenance:\n  notebook: \"huggingface/trl examples/sft_qlora/sft_qlora.ipynb @ ebf09f6b\"\n  notebook_vram: \"~5.9 GB on a free Colab T4\"\n\nscripts:\n  train: {hf_jobs: train_hf_jobs.sh, slurm: train_slurm.sh, local: train_local.sh}\n  eval: {hf_jobs: eval_hf_jobs.sh, slurm: eval_slurm.sh, local: eval_local.sh}\n\n# What the command builder on the recipes site offers. Each option is an environment variable of the script it runs.\nbuilder:\n  verified: {hf_jobs: false, slurm: false, local: false}   # ported from the notebook, which ran on Colab; not run as these scripts yet\n  eval:\n    - env: ADAPTER\n      label: Adapter\n      input: true\n      default: \"<your user>/LFM2-2.6B-SFT\"\n"
   },
   "other_files": []
  },
  {
   "id": "SFTTrainer/sft-qlora/Llama-3.2-3B-Instruct",
   "trainer": "SFTTrainer",
   "project": "sft-qlora",
   "name": "Llama-3.2-3B-Instruct",
   "org": "meta-llama",
   "avatar": "https://cdn-avatars.huggingface.co/v1/production/uploads/646cf8084eefb026fb8fd8bc/oCTqufkdTkjyGodsx1vo1.png",
   "meta": {
    "title": "Llama-3.2-3B-Instruct SFT with QLoRA",
    "project": "sft-qlora",
    "trainer": "SFTTrainer",
    "model_id": "meta-llama/Llama-3.2-3B-Instruct",
    "description": "QLoRA SFT on Multilingual-Thinking, so the model reasons in the language the system prompt asks for. 4-bit base, LoRA r32, 30 steps: fits one 16 GB T4 (~4.7 GB VRAM).",
    "tags": [
     "sft",
     "qlora",
     "4-bit",
     "reasoning",
     "multilingual"
    ],
    "date_added": "2026-11-13",
    "source": "https://github.com/huggingface/trl/blob/ebf09f6b7e126af6e34ff172bb74e1629ff24156/examples/sft_qlora/sft_qlora.ipynb"
   },
   "hardware": {
    "hf_jobs": "t4-small",
    "slurm": "hopper-prod, 1 GPU",
    "local": "1 GPU with 16 GB or more"
   },
   "datasets": {
    "train": [
     "HuggingFaceH4/Multilingual-Thinking"
    ]
   },
   "hyperparameters": {
    "quantization": "4-bit nf4, double quant, float16 compute",
    "lora_r": 32,
    "lora_alpha": 32,
    "lora_target_modules": [
     "q_proj",
     "k_proj",
     "v_proj",
     "o_proj",
     "gate_proj",
     "up_proj",
     "down_proj"
    ],
    "learning_rate": 0.0002,
    "optim": "paged_adamw_8bit",
    "per_device_train_batch_size": 1,
    "gradient_accumulation_steps": 4,
    "warmup_steps": 5,
    "max_steps": 30,
    "max_length": 1024,
    "dtype": "float16",
    "activation_offloading": true
   },
   "trl": {
    "version": "1.17.1"
   },
   "provenance": {
    "notebook": "huggingface/trl examples/sft_qlora/sft_qlora.ipynb @ ebf09f6b",
    "notebook_vram": "~4.7 GB on a free Colab T4"
   },
   "scripts": {
    "train": {
     "hf_jobs": "train_hf_jobs.sh",
     "slurm": "train_slurm.sh",
     "local": "train_local.sh"
    },
    "eval": {
     "hf_jobs": "eval_hf_jobs.sh",
     "slurm": "eval_slurm.sh",
     "local": "eval_local.sh"
    }
   },
   "builder": {
    "verified": {
     "hf_jobs": false,
     "slurm": false,
     "local": false
    },
    "eval": [
     {
      "env": "ADAPTER",
      "label": "Adapter",
      "input": true,
      "default": "<your user>/Llama-3.2-3B-Instruct-SFT"
     }
    ]
   },
   "readme": "# Llama-3.2-3B-Instruct: SFT with QLoRA\n\n[`meta-llama/Llama-3.2-3B-Instruct`](https://huggingface.co/meta-llama/Llama-3.2-3B-Instruct) fine-tuned with QLoRA on\n[HuggingFaceH4/Multilingual-Thinking](https://huggingface.co/datasets/HuggingFaceH4/Multilingual-Thinking), so it\nwrites its reasoning in the language the system prompt asks for. The notebook measured ~4.7 GB of VRAM on a free\nColab T4. Shared settings: [../README.md](../README.md). Full hyperparameters: [recipe.yaml](recipe.yaml).\n\nThe model is gated: accept its license on the Hub with the account whose `HF_TOKEN` the job uses.\n\n| | HF Jobs | Cluster | Local |\n|---|---|---|---|\n| Train | `bash train_hf_jobs.sh` (t4-small) | `sbatch train_slurm.sh` (1 GPU) | `bash train_local.sh` (1 GPU, 16 GB) |\n| Check | `ADAPTER=<your user>/Llama-3.2-3B-Instruct-SFT bash eval_hf_jobs.sh` | `ADAPTER=... sbatch eval_slurm.sh` | `ADAPTER=... bash eval_local.sh` |\n\nTraining pushes the adapter to `<your user>/Llama-3.2-3B-Instruct-SFT` and logs to the Trackio Space of the same name. The check prints\nthe base model's answer, then the adapter's: the adapter's reasoning should be in German.\n",
   "files": {
    "train_local.sh": "#!/bin/bash\n# SFT with QLoRA of meta-llama/Llama-3.2-3B-Instruct on this machine's GPU (16 GB is enough: the notebook ran on a T4), with `uv run`: the\n# settings of TRL's sft_qlora notebook. The adapter is saved to $OUTPUT_ROOT/Llama-3.2-3B-Instruct-SFT, pushed to <your user>/Llama-3.2-3B-Instruct-SFT,\n# and the run is logged to the Trackio Space of the same name.\n#\n#   bash train_local.sh\n#   MAX_STEPS=2 bash train_local.sh                   short smoke run\nset -euo pipefail\n\nMAX_STEPS=\"${MAX_STEPS:-30}\"\nOUTPUT_ROOT=\"${OUTPUT_ROOT:-$PWD/outputs}\"\nSCRIPTS=\"$(cd \"$(dirname \"$0\")/../scripts\" && pwd)\"\n\nuv run \"$SCRIPTS/sft_qlora.py\" \\\n    --model meta-llama/Llama-3.2-3B-Instruct --output-root \"$OUTPUT_ROOT\" --max-steps \"$MAX_STEPS\"\n",
    "train_hf_jobs.sh": "#!/bin/bash\n# SFT with QLoRA of meta-llama/Llama-3.2-3B-Instruct on HF Jobs, one 16 GB T4: the settings of TRL's sft_qlora notebook. The adapter is pushed to\n# <your user>/Llama-3.2-3B-Instruct-SFT and the run is logged to the Trackio Space of the same name.\n#\n#   bash train_hf_jobs.sh\n#   MAX_STEPS=2 bash train_hf_jobs.sh                 short smoke run\nset -euo pipefail\n\nMAX_STEPS=\"${MAX_STEPS:-30}\"\nSCRIPTS=\"$(cd \"$(dirname \"$0\")/../scripts\" && pwd)\"\n\nhf jobs uv run --detach \\\n    --flavor t4-small --secrets HF_TOKEN --timeout 2h \\\n    \"$SCRIPTS/sft_qlora.py\" -- \\\n    --model meta-llama/Llama-3.2-3B-Instruct --max-steps \"$MAX_STEPS\"\n",
    "train_slurm.sh": "#!/bin/bash -l\n# SFT with QLoRA of meta-llama/Llama-3.2-3B-Instruct on one cluster GPU: the settings of TRL's sft_qlora notebook (made for a 16 GB T4, so an\n# H100 has room to spare). The adapter is saved to $OUTPUT_ROOT/Llama-3.2-3B-Instruct-SFT and pushed to <your user>/Llama-3.2-3B-Instruct-SFT.\n#\n#   sbatch train_slurm.sh                              from this folder: the shared scripts are found from it\n#   MAX_STEPS=2 sbatch train_slurm.sh                  short smoke run\n#SBATCH --job-name=sft-qlora-llama3.2-3b\n#SBATCH --partition=hopper-prod\n#SBATCH --nodes=1\n#SBATCH --ntasks-per-node=1\n#SBATCH --gpus-per-node=1\n#SBATCH --cpus-per-task=12\n#SBATCH --time=2:00:00\n#SBATCH --output=/fsx/%u/logs/%x-%j.out\nset -euo pipefail\n\nMAX_STEPS=\"${MAX_STEPS:-30}\"\nOUTPUT_ROOT=\"${OUTPUT_ROOT:-/fsx/$USER/sft-qlora}\"\nexport HF_TOKEN=\"${HF_TOKEN:-$(cat ~/.cache/huggingface/token)}\"\n# Model weights and uv's package cache on /fsx, not the small NFS home\nexport HF_HUB_CACHE=\"${HF_HUB_CACHE:-/fsx/$USER/hf-hub}\"\nexport UV_CACHE_DIR=\"${UV_CACHE_DIR:-/fsx/$USER/.cache/uv}\"\nmkdir -p \"/fsx/$USER/logs\" \"$OUTPUT_ROOT\"\n\n# sbatch runs a copy of this script, so the shared scripts are found from the folder it was submitted from\nuv run \"$SLURM_SUBMIT_DIR/../scripts/sft_qlora.py\" \\\n    --model meta-llama/Llama-3.2-3B-Instruct --output-root \"$OUTPUT_ROOT\" --max-steps \"$MAX_STEPS\"\n",
    "eval_local.sh": "#!/bin/bash\n# The notebook's check of the trained adapter on this machine's GPU: generate for a prompt asking for German reasoning,\n# with the base model and then with the adapter.\n#\n#   ADAPTER=<your user>/Llama-3.2-3B-Instruct-SFT bash eval_local.sh\n#   ADAPTER=$PWD/outputs/Llama-3.2-3B-Instruct-SFT bash eval_local.sh\nset -euo pipefail\n\nADAPTER=\"${ADAPTER:?ADAPTER=<your user>/Llama-3.2-3B-Instruct-SFT or the run folder train_local.sh saved}\"\nSCRIPTS=\"$(cd \"$(dirname \"$0\")/../scripts\" && pwd)\"\n\nuv run \"$SCRIPTS/compare_generations.py\" --model meta-llama/Llama-3.2-3B-Instruct --adapter \"$ADAPTER\"\n",
    "eval_hf_jobs.sh": "#!/bin/bash\n# The notebook's check of the trained adapter on HF Jobs (one T4): generate for a prompt asking for German reasoning,\n# with the base model and then with the adapter. Read the two outputs with `hf jobs logs <job id>`.\n#\n#   ADAPTER=<your user>/Llama-3.2-3B-Instruct-SFT bash eval_hf_jobs.sh\nset -euo pipefail\n\nADAPTER=\"${ADAPTER:?ADAPTER=<your user>/Llama-3.2-3B-Instruct-SFT, the adapter train_hf_jobs.sh pushed}\"\nSCRIPTS=\"$(cd \"$(dirname \"$0\")/../scripts\" && pwd)\"\n\nhf jobs uv run --detach \\\n    --flavor t4-small --secrets HF_TOKEN --timeout 1h \\\n    \"$SCRIPTS/compare_generations.py\" -- \\\n    --model meta-llama/Llama-3.2-3B-Instruct --adapter \"$ADAPTER\"\n",
    "eval_slurm.sh": "#!/bin/bash -l\n# The notebook's check of the trained adapter on one cluster GPU: generate for a prompt asking for German reasoning,\n# with the base model and then with the adapter. The two outputs are in the job's log.\n#\n#   ADAPTER=<your user>/Llama-3.2-3B-Instruct-SFT sbatch eval_slurm.sh\n#   ADAPTER=/fsx/$USER/sft-qlora/Llama-3.2-3B-Instruct-SFT sbatch eval_slurm.sh\n#SBATCH --job-name=sft-qlora-llama3.2-3b-eval\n#SBATCH --partition=hopper-prod\n#SBATCH --nodes=1\n#SBATCH --ntasks-per-node=1\n#SBATCH --gpus-per-node=1\n#SBATCH --cpus-per-task=12\n#SBATCH --time=0:30:00\n#SBATCH --output=/fsx/%u/logs/%x-%j.out\nset -euo pipefail\n\nADAPTER=\"${ADAPTER:?ADAPTER=<your user>/Llama-3.2-3B-Instruct-SFT or the run folder train_slurm.sh saved}\"\nexport HF_TOKEN=\"${HF_TOKEN:-$(cat ~/.cache/huggingface/token)}\"\nexport HF_HUB_CACHE=\"${HF_HUB_CACHE:-/fsx/$USER/hf-hub}\"\nexport UV_CACHE_DIR=\"${UV_CACHE_DIR:-/fsx/$USER/.cache/uv}\"\nmkdir -p \"/fsx/$USER/logs\"\n\nuv run \"$SLURM_SUBMIT_DIR/../scripts/compare_generations.py\" --model meta-llama/Llama-3.2-3B-Instruct --adapter \"$ADAPTER\"\n",
    "recipe.yaml": "meta:\n  title: \"Llama-3.2-3B-Instruct SFT with QLoRA\"\n  project: sft-qlora\n  trainer: SFTTrainer\n  model_id: meta-llama/Llama-3.2-3B-Instruct\n  description: \"QLoRA SFT on Multilingual-Thinking, so the model reasons in the language the system prompt asks for. 4-bit base, LoRA r32, 30 steps: fits one 16 GB T4 (~4.7 GB VRAM).\"\n  tags: [sft, qlora, 4-bit, reasoning, multilingual]\n  date_added: 2026-11-13\n  source: https://github.com/huggingface/trl/blob/ebf09f6b7e126af6e34ff172bb74e1629ff24156/examples/sft_qlora/sft_qlora.ipynb\n\nhardware:\n  hf_jobs: t4-small\n  slurm: \"hopper-prod, 1 GPU\"\n  local: \"1 GPU with 16 GB or more\"\n\ndatasets:\n  train:\n    - HuggingFaceH4/Multilingual-Thinking\n\nhyperparameters:\n  quantization: \"4-bit nf4, double quant, float16 compute\"\n  lora_r: 32\n  lora_alpha: 32\n  lora_target_modules: [q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj]\n  learning_rate: 2.0e-4\n  optim: paged_adamw_8bit\n  per_device_train_batch_size: 1\n  gradient_accumulation_steps: 4\n  warmup_steps: 5\n  max_steps: 30\n  max_length: 1024\n  dtype: float16\n  activation_offloading: true\n\ntrl:\n  version: \"1.17.1\"            # from PyPI, no fork\n\nprovenance:\n  notebook: \"huggingface/trl examples/sft_qlora/sft_qlora.ipynb @ ebf09f6b\"\n  notebook_vram: \"~4.7 GB on a free Colab T4\"\n\nscripts:\n  train: {hf_jobs: train_hf_jobs.sh, slurm: train_slurm.sh, local: train_local.sh}\n  eval: {hf_jobs: eval_hf_jobs.sh, slurm: eval_slurm.sh, local: eval_local.sh}\n\n# What the command builder on the recipes site offers. Each option is an environment variable of the script it runs.\nbuilder:\n  verified: {hf_jobs: false, slurm: false, local: false}   # ported from the notebook, which ran on Colab; not run as these scripts yet\n  eval:\n    - env: ADAPTER\n      label: Adapter\n      input: true\n      default: \"<your user>/Llama-3.2-3B-Instruct-SFT\"\n"
   },
   "other_files": []
  },
  {
   "id": "SFTTrainer/sft-qlora/Qwen3-8B",
   "trainer": "SFTTrainer",
   "project": "sft-qlora",
   "name": "Qwen3-8B",
   "org": "Qwen",
   "avatar": "https://cdn-avatars.huggingface.co/v1/production/uploads/6215ca5692c0ecfba9186921/hrRM50-6XcdWgg2AKpENG.jpeg",
   "meta": {
    "title": "Qwen3-8B SFT with QLoRA",
    "project": "sft-qlora",
    "trainer": "SFTTrainer",
    "model_id": "Qwen/Qwen3-8B",
    "description": "QLoRA SFT on Multilingual-Thinking, so the model reasons in the language the system prompt asks for. 4-bit base, LoRA r32, 30 steps: fits one 16 GB T4 (~12.8 GB VRAM).",
    "tags": [
     "sft",
     "qlora",
     "4-bit",
     "reasoning",
     "multilingual"
    ],
    "date_added": "2026-11-13",
    "source": "https://github.com/huggingface/trl/blob/ebf09f6b7e126af6e34ff172bb74e1629ff24156/examples/sft_qlora/sft_qlora.ipynb"
   },
   "hardware": {
    "hf_jobs": "t4-small",
    "slurm": "hopper-prod, 1 GPU",
    "local": "1 GPU with 16 GB or more"
   },
   "datasets": {
    "train": [
     "HuggingFaceH4/Multilingual-Thinking"
    ]
   },
   "hyperparameters": {
    "quantization": "4-bit nf4, double quant, float16 compute",
    "lora_r": 32,
    "lora_alpha": 32,
    "lora_target_modules": [
     "q_proj",
     "k_proj",
     "v_proj",
     "o_proj",
     "gate_proj",
     "up_proj",
     "down_proj"
    ],
    "learning_rate": 0.0002,
    "optim": "paged_adamw_8bit",
    "per_device_train_batch_size": 1,
    "gradient_accumulation_steps": 4,
    "warmup_steps": 5,
    "max_steps": 30,
    "max_length": 1024,
    "dtype": "float16",
    "activation_offloading": true
   },
   "trl": {
    "version": "1.17.1"
   },
   "provenance": {
    "notebook": "huggingface/trl examples/sft_qlora/sft_qlora.ipynb @ ebf09f6b",
    "notebook_vram": "~12.8 GB on a free Colab T4"
   },
   "scripts": {
    "train": {
     "hf_jobs": "train_hf_jobs.sh",
     "slurm": "train_slurm.sh",
     "local": "train_local.sh"
    },
    "eval": {
     "hf_jobs": "eval_hf_jobs.sh",
     "slurm": "eval_slurm.sh",
     "local": "eval_local.sh"
    }
   },
   "builder": {
    "verified": {
     "hf_jobs": false,
     "slurm": false,
     "local": false
    },
    "eval": [
     {
      "env": "ADAPTER",
      "label": "Adapter",
      "input": true,
      "default": "<your user>/Qwen3-8B-SFT"
     }
    ]
   },
   "readme": "# Qwen3-8B: SFT with QLoRA\n\n[`Qwen/Qwen3-8B`](https://huggingface.co/Qwen/Qwen3-8B) fine-tuned with QLoRA on\n[HuggingFaceH4/Multilingual-Thinking](https://huggingface.co/datasets/HuggingFaceH4/Multilingual-Thinking), so it\nwrites its reasoning in the language the system prompt asks for. The notebook measured ~12.8 GB of VRAM on a free\nColab T4. Shared settings: [../README.md](../README.md). Full hyperparameters: [recipe.yaml](recipe.yaml).\n\n| | HF Jobs | Cluster | Local |\n|---|---|---|---|\n| Train | `bash train_hf_jobs.sh` (t4-small) | `sbatch train_slurm.sh` (1 GPU) | `bash train_local.sh` (1 GPU, 16 GB) |\n| Check | `ADAPTER=<your user>/Qwen3-8B-SFT bash eval_hf_jobs.sh` | `ADAPTER=... sbatch eval_slurm.sh` | `ADAPTER=... bash eval_local.sh` |\n\nTraining pushes the adapter to `<your user>/Qwen3-8B-SFT` and logs to the Trackio Space of the same name. The check prints\nthe base model's answer, then the adapter's: the adapter's reasoning should be in German.\n",
   "files": {
    "train_local.sh": "#!/bin/bash\n# SFT with QLoRA of Qwen/Qwen3-8B on this machine's GPU (16 GB is enough: the notebook ran on a T4), with `uv run`: the\n# settings of TRL's sft_qlora notebook. The adapter is saved to $OUTPUT_ROOT/Qwen3-8B-SFT, pushed to <your user>/Qwen3-8B-SFT,\n# and the run is logged to the Trackio Space of the same name.\n#\n#   bash train_local.sh\n#   MAX_STEPS=2 bash train_local.sh                   short smoke run\nset -euo pipefail\n\nMAX_STEPS=\"${MAX_STEPS:-30}\"\nOUTPUT_ROOT=\"${OUTPUT_ROOT:-$PWD/outputs}\"\nSCRIPTS=\"$(cd \"$(dirname \"$0\")/../scripts\" && pwd)\"\n\nuv run \"$SCRIPTS/sft_qlora.py\" \\\n    --model Qwen/Qwen3-8B --output-root \"$OUTPUT_ROOT\" --max-steps \"$MAX_STEPS\"\n",
    "train_hf_jobs.sh": "#!/bin/bash\n# SFT with QLoRA of Qwen/Qwen3-8B on HF Jobs, one 16 GB T4: the settings of TRL's sft_qlora notebook. The adapter is pushed to\n# <your user>/Qwen3-8B-SFT and the run is logged to the Trackio Space of the same name.\n#\n#   bash train_hf_jobs.sh\n#   MAX_STEPS=2 bash train_hf_jobs.sh                 short smoke run\nset -euo pipefail\n\nMAX_STEPS=\"${MAX_STEPS:-30}\"\nSCRIPTS=\"$(cd \"$(dirname \"$0\")/../scripts\" && pwd)\"\n\nhf jobs uv run --detach \\\n    --flavor t4-small --secrets HF_TOKEN --timeout 2h \\\n    \"$SCRIPTS/sft_qlora.py\" -- \\\n    --model Qwen/Qwen3-8B --max-steps \"$MAX_STEPS\"\n",
    "train_slurm.sh": "#!/bin/bash -l\n# SFT with QLoRA of Qwen/Qwen3-8B on one cluster GPU: the settings of TRL's sft_qlora notebook (made for a 16 GB T4, so an\n# H100 has room to spare). The adapter is saved to $OUTPUT_ROOT/Qwen3-8B-SFT and pushed to <your user>/Qwen3-8B-SFT.\n#\n#   sbatch train_slurm.sh                              from this folder: the shared scripts are found from it\n#   MAX_STEPS=2 sbatch train_slurm.sh                  short smoke run\n#SBATCH --job-name=sft-qlora-qwen3-8b\n#SBATCH --partition=hopper-prod\n#SBATCH --nodes=1\n#SBATCH --ntasks-per-node=1\n#SBATCH --gpus-per-node=1\n#SBATCH --cpus-per-task=12\n#SBATCH --time=2:00:00\n#SBATCH --output=/fsx/%u/logs/%x-%j.out\nset -euo pipefail\n\nMAX_STEPS=\"${MAX_STEPS:-30}\"\nOUTPUT_ROOT=\"${OUTPUT_ROOT:-/fsx/$USER/sft-qlora}\"\nexport HF_TOKEN=\"${HF_TOKEN:-$(cat ~/.cache/huggingface/token)}\"\n# Model weights and uv's package cache on /fsx, not the small NFS home\nexport HF_HUB_CACHE=\"${HF_HUB_CACHE:-/fsx/$USER/hf-hub}\"\nexport UV_CACHE_DIR=\"${UV_CACHE_DIR:-/fsx/$USER/.cache/uv}\"\nmkdir -p \"/fsx/$USER/logs\" \"$OUTPUT_ROOT\"\n\n# sbatch runs a copy of this script, so the shared scripts are found from the folder it was submitted from\nuv run \"$SLURM_SUBMIT_DIR/../scripts/sft_qlora.py\" \\\n    --model Qwen/Qwen3-8B --output-root \"$OUTPUT_ROOT\" --max-steps \"$MAX_STEPS\"\n",
    "eval_local.sh": "#!/bin/bash\n# The notebook's check of the trained adapter on this machine's GPU: generate for a prompt asking for German reasoning,\n# with the base model and then with the adapter.\n#\n#   ADAPTER=<your user>/Qwen3-8B-SFT bash eval_local.sh\n#   ADAPTER=$PWD/outputs/Qwen3-8B-SFT bash eval_local.sh\nset -euo pipefail\n\nADAPTER=\"${ADAPTER:?ADAPTER=<your user>/Qwen3-8B-SFT or the run folder train_local.sh saved}\"\nSCRIPTS=\"$(cd \"$(dirname \"$0\")/../scripts\" && pwd)\"\n\nuv run \"$SCRIPTS/compare_generations.py\" --model Qwen/Qwen3-8B --adapter \"$ADAPTER\"\n",
    "eval_hf_jobs.sh": "#!/bin/bash\n# The notebook's check of the trained adapter on HF Jobs (one T4): generate for a prompt asking for German reasoning,\n# with the base model and then with the adapter. Read the two outputs with `hf jobs logs <job id>`.\n#\n#   ADAPTER=<your user>/Qwen3-8B-SFT bash eval_hf_jobs.sh\nset -euo pipefail\n\nADAPTER=\"${ADAPTER:?ADAPTER=<your user>/Qwen3-8B-SFT, the adapter train_hf_jobs.sh pushed}\"\nSCRIPTS=\"$(cd \"$(dirname \"$0\")/../scripts\" && pwd)\"\n\nhf jobs uv run --detach \\\n    --flavor t4-small --secrets HF_TOKEN --timeout 1h \\\n    \"$SCRIPTS/compare_generations.py\" -- \\\n    --model Qwen/Qwen3-8B --adapter \"$ADAPTER\"\n",
    "eval_slurm.sh": "#!/bin/bash -l\n# The notebook's check of the trained adapter on one cluster GPU: generate for a prompt asking for German reasoning,\n# with the base model and then with the adapter. The two outputs are in the job's log.\n#\n#   ADAPTER=<your user>/Qwen3-8B-SFT sbatch eval_slurm.sh\n#   ADAPTER=/fsx/$USER/sft-qlora/Qwen3-8B-SFT sbatch eval_slurm.sh\n#SBATCH --job-name=sft-qlora-qwen3-8b-eval\n#SBATCH --partition=hopper-prod\n#SBATCH --nodes=1\n#SBATCH --ntasks-per-node=1\n#SBATCH --gpus-per-node=1\n#SBATCH --cpus-per-task=12\n#SBATCH --time=0:30:00\n#SBATCH --output=/fsx/%u/logs/%x-%j.out\nset -euo pipefail\n\nADAPTER=\"${ADAPTER:?ADAPTER=<your user>/Qwen3-8B-SFT or the run folder train_slurm.sh saved}\"\nexport HF_TOKEN=\"${HF_TOKEN:-$(cat ~/.cache/huggingface/token)}\"\nexport HF_HUB_CACHE=\"${HF_HUB_CACHE:-/fsx/$USER/hf-hub}\"\nexport UV_CACHE_DIR=\"${UV_CACHE_DIR:-/fsx/$USER/.cache/uv}\"\nmkdir -p \"/fsx/$USER/logs\"\n\nuv run \"$SLURM_SUBMIT_DIR/../scripts/compare_generations.py\" --model Qwen/Qwen3-8B --adapter \"$ADAPTER\"\n",
    "recipe.yaml": "meta:\n  title: \"Qwen3-8B SFT with QLoRA\"\n  project: sft-qlora\n  trainer: SFTTrainer\n  model_id: Qwen/Qwen3-8B\n  description: \"QLoRA SFT on Multilingual-Thinking, so the model reasons in the language the system prompt asks for. 4-bit base, LoRA r32, 30 steps: fits one 16 GB T4 (~12.8 GB VRAM).\"\n  tags: [sft, qlora, 4-bit, reasoning, multilingual]\n  date_added: 2026-11-13\n  source: https://github.com/huggingface/trl/blob/ebf09f6b7e126af6e34ff172bb74e1629ff24156/examples/sft_qlora/sft_qlora.ipynb\n\nhardware:\n  hf_jobs: t4-small\n  slurm: \"hopper-prod, 1 GPU\"\n  local: \"1 GPU with 16 GB or more\"\n\ndatasets:\n  train:\n    - HuggingFaceH4/Multilingual-Thinking\n\nhyperparameters:\n  quantization: \"4-bit nf4, double quant, float16 compute\"\n  lora_r: 32\n  lora_alpha: 32\n  lora_target_modules: [q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj]\n  learning_rate: 2.0e-4\n  optim: paged_adamw_8bit\n  per_device_train_batch_size: 1\n  gradient_accumulation_steps: 4\n  warmup_steps: 5\n  max_steps: 30\n  max_length: 1024\n  dtype: float16\n  activation_offloading: true\n\ntrl:\n  version: \"1.17.1\"            # from PyPI, no fork\n\nprovenance:\n  notebook: \"huggingface/trl examples/sft_qlora/sft_qlora.ipynb @ ebf09f6b\"\n  notebook_vram: \"~12.8 GB on a free Colab T4\"\n\nscripts:\n  train: {hf_jobs: train_hf_jobs.sh, slurm: train_slurm.sh, local: train_local.sh}\n  eval: {hf_jobs: eval_hf_jobs.sh, slurm: eval_slurm.sh, local: eval_local.sh}\n\n# What the command builder on the recipes site offers. Each option is an environment variable of the script it runs.\nbuilder:\n  verified: {hf_jobs: false, slurm: false, local: false}   # ported from the notebook, which ran on Colab; not run as these scripts yet\n  eval:\n    - env: ADAPTER\n      label: Adapter\n      input: true\n      default: \"<your user>/Qwen3-8B-SFT\"\n"
   },
   "other_files": []
  },
  {
   "id": "AsyncGRPOTrainer/openenv-coding-tasks/Qwen3.8-27B",
   "trainer": "AsyncGRPOTrainer",
   "project": "openenv-coding-tasks",
   "name": "Qwen3.8-27B",
   "org": "Qwen",
   "meta": {
    "model_id": "Qwen/Qwen3.8-27B",
    "description": "Build verifiable coding tasks and select examples that give the base model a mix of successful and unsuccessful rollouts.",
    "tags": [
     "RL Environment Pipeline",
     "OpenEnv",
     "task preparation"
    ]
   },
   "guide_url": "guides/openenv-coding-tasks/index.html",
   "title": "RL Environment Pipeline"
  }
 ]
}