PLABench

A benchmark for protein-ligand binding affinity prediction. It runs nine models, sequence-based and structure-based, over the same targets under the same metrics, and varies the input structure so that pose quality can be told apart from model quality.

This is the code and data behind "Leakage-controlled benchmarking reveals generalization limits of deep learning for protein-ligand binding affinity prediction", by Lyuwei Wang and Jianlin Cheng. See Citation.

git clone --recurse-submodules https://github.com/BioinfoMachineLearning/PLABench.git
python run_benchmark.py model=boltz2 dataset=boltz2_chembl35_full

Each model needs its own conda environment and its own weights, so read Install before the second line.

Every number and figure below is reproduced from results/. Full tables, including the leakage-filtered arms and the per-target breakdowns, are in results/README.md.

The three sequence models retrained on the splits MixingDTA defined (results/benchmark_summary.csv).

Kendall's tau on the two ligand series, with the black rule giving the size-weighted average over all 140 compounds (results/casp16/casp16_stage1_kendall.csv).

4,862 ligands over 71 targets, after the leakage filter: every test ligand within ECFP4 Tanimoto 0.85 of any ligand in the fourteen audited training corpora is dropped, then targets left with fewer than ten ligands go too (results/chembl35/weighted_summary_chembl35_filtered.csv). The unfiltered arm, 7,650 ligands over all 84 targets, is in the same directory.

FLOWR.ROOT leads even after the filter, but the filter is what makes that readable. SAIR, the corpus it trains on, is built from ChEMBL 35 and reaches Tanimoto 0.85 against 32.7% of the unfiltered benchmark, which is 2,413 of the 2,735 rows the similarity cut removes. results/chembl35_leakage/leak_by_corpus.csv gives the per-corpus breakdown.

Pearson correlation for each model on each protein family, over the filtered ChEMBL35 arm plus the two CASP16 series (results/chembl35/per_class_pearson_wide_chembl35_casp16_filtered.csv).

Clone with the model sources, which are git submodules under forks/:

git clone --recurse-submodules https://github.com/BioinfoMachineLearning/PLABench.git

If you already cloned without them, git submodule update --init fills forks/ in. Each fork sits on a plabench branch holding the exact upstream commit that was benchmarked plus the patches needed to run it; forks/README.md lists those patches.

Every model runs in its own conda environment, since their dependencies conflict. See Environments below for how to point the benchmark at yours.

The weights PLABench trained itself (DeepDTA, LLF and MixingDTA on PDBbind, MixingDTA's Davis and KIBA cold-start folds, and MFE) ship in archive 3 of the Zenodo deposit. bash scripts/download_checkpoints.sh fetches it from there and checks the SHA256; both are baked into the script, so it takes no arguments. The benchmark also attempts this by itself when a configured checkpoint is missing. To install from a local copy or a mirror instead, set PLABENCH_CHECKPOINT_URL, and PLABENCH_CHECKPOINT_SHA256="" to skip the checksum test.

Third-party weights are never re-uploaded. That covers Boltz-2, FLOWR.ROOT, FlowDock, BA-Pred, Graph_RG/Haiping, ESM3, and the MixingDTA authors' Davis and KIBA warm-start models. checkpoints/THIRD_PARTY.tsv records the official URL, SHA256 and license of each file, and bash scripts/download_third_party.sh fetches and verifies them. Boltz-2 is the exception: it downloads itself into ~/.boltz/ on first use.

checkpoints/README.md has the directory layout and the full inventory.

Benchmark inputs, ground truth and the AlphaFold3 structures are too large for git and live on Zenodo instead, at doi:10.5281/zenodo.22716174. Four archives, 1.4 GB to download and 5.8 GB unpacked, over data/, checkpoints/, outputs/ and results/:

Unpack archives 1, 2 and 4, and 46 of the 68 dataset configs run as they are. The other 22 wait on a corpus nobody may redistribute: the CASF core sets, the CSAR-HiQ structures, or the CASP16 stage-2 complexes. Each is a registration or a download away, and the bullets below say where.

What ships is decided by who made it: PLABench artifacts go in, corpora other people built are linked instead. checkpoints/ splits the same way, between MANIFEST.tsv and THIRD_PARTY.tsv. data/SOURCES.tsv gives the call, the origin and the license for every path the benchmark reads, 25 of them. Four of those calls change what you get:

• PDBbind v2020 forbids redistribution without written permission, so the deposit carries the 4,465 / 497 refined partition as compound_id,split and nothing else. Rejoin it against photonmz/pdbbindpp-2020.
• CSAR-HiQ never came with terms that allow redistribution and its portal is gone, so the 36 and 51 structure sets are not shipped. The standardized PDB-code / SMILES / pKd tables are, which is enough to rescore once you have the complexes. Binding MOAD's download page still serves the CSAR-NRC HiQ set and its update, but Binding MOAD is sunset and static now, so fetching the 87 entries from the RCSB by PDB code is the safer route.
• CASF-2013 and CASF-2016 come from the CASF authors under their own terms, so the two core sets are not shipped either. Request them at pdbbind.org.cn/casf.php. The Boltz-2 poses for both are in archive 2 and the standardized affinity tables in archive 1, so only the experimental arms wait on the request.
• The experimental CASP16 stage-2 complexes are the organizers' release. No number in the paper needs them: CASP16 is scored against the labels in archive 1, and configs/manifests/ records which targets each stage-2 run covered, so scripts/collect_results.py still reports 93 of 93 with data/ half empty. Re-running the six stage-2 configs does need them, from the Prediction Center.

MFE reads its structures as one directory per complex holding protein.pdb and ligand.mol2. Those eight data/*_prepared directories are symlink farms with no bytes of their own, so they are not deposited. Rebuild them after unpacking:

python scripts/data_prep/link_mfe_inputs.py

The four that point at Boltz-2 poses work straight from archive 2; the other four wait on CASF and CSAR-HiQ.

Davis and KIBA are complete: train, validation and test, for the warm-start arm and for both cold-start arms. Everything except the test folds is in the pickle format MixingDTA released, because the target sequence repeats on every row and pickle stores it once, which is 28 MB against 621 MB for the same folds as CSV. For CSVs with the same four columns as the test files:

python scripts/cv/export_davis_kiba_folds.py

It rebuilds every CSV that already exists before writing anything, so a change in the pickled row layout fails there rather than in a training run.

The AlphaFold3 structures are distributable under the AlphaFold3 Output Terms of Use, non-commercially, and the archive carries the modification notice those terms require. No AlphaFold3 model parameters are redistributed anywhere here.

bash scripts/package_zenodo.sh --dry-run lists what each archive holds; that script is what built the deposit.

Everything except the Davis and KIBA cross-validation runs through Hydra:

python run_benchmark.py model=<model> dataset=<dataset>

configs/model/ holds one file per model (bapred, boltz2, deepdta, flowdock, flowr_root, haiping, llf, mfe, mixingdta), each naming the checkpoint, the model source under forks/, and the interpreter to run it with. configs/dataset/ holds the 68 evaluation settings the paper reports, one per combination of input source, benchmark and evaluation mode. Their names encode which is which, so read configs/README.md before picking one; the ordering of tokens in a name is easy to misread. Runs that did not make the paper are in archive/configs/dataset/.

Predictions land in outputs/<model>/<dataset>/latest/. Score them from the analysis environment (environments/analysis.yaml, see below) with python scripts/collect_results.py, which refreshes results/benchmark_summary.csv and the per-target tables, then python scripts/weighted_summary.py for the weighted aggregates. scripts/README.md says what each of the remaining scripts produces and in what order to run them.

Override any config field on the command line:

python run_benchmark.py model=mfe dataset=af3_chembl35_full ++model.device=cpu

The three sequence models are cross-validated over five folds in three regimes (warm start, cold drug, cold target), which the generic runner does not cover. Use the dedicated runners, one per model, then score them together:

python scripts/cv/run_deepdta_cv.py          # or a subset: davis_warm kiba_cold_drug
python scripts/cv/run_llf_cv.py
python scripts/cv/run_mixingdta_cv.py
python scripts/cv/collect_kiba_davis_cv.py   # add --dry-run to score without rewriting the summary

Each runner writes outputs/<model>/<config>/fold_<n>/predictions.csv, one row per ground-truth row in the same order. The collector reads that one layout for all three models, averages the folds, and refreshes the Davis and KIBA rows of results/benchmark_summary.csv. Per-fold metrics land in analysis/davis_kiba_cv_metrics.csv.

MixingDTA cold-start inference needs forks/MixingDTA initialized even if you skip the other submodules. It reads the authors' pickled test sets from there, because it looks up per-compound and per-target embeddings by identifiers the CSV copies do not carry. The pickles are row-identical to the CSV ground truth.

Every model has its own conda environment, exported to environments/. Create one with conda env create -f environments/<model>.yaml; environments/README.md covers the two post-create steps a conda export cannot carry, and how AlphaFold3 was built.

The analysis scripts get a tenth environment of their own, analysis.yaml. It runs no model, so it installs in a couple of minutes, and it is what everything under scripts/ was run in:

conda env create -f environments/analysis.yaml
conda activate plabench-analysis
python scripts/collect_results.py

Five models are spawned as a subprocess, so they take a path. Override it with an environment variable rather than editing configs/model/*.yaml:

LCDD-team, Graph_RG, MFE and the non-CV DeepDTA runs use the interpreter that started run_benchmark.py, so activate their environment and launch from inside it.

MIXINGDTA_ROOT points the MixingDTA runner at a checkout other than forks/MixingDTA, and PLABENCH_DEVICE sets the torch device for the cross-validation runners.

Every number in the paper was produced by the code pinned in the three tables below: the nine model forks, the external tools, and the hardware each model ran on.

Each submodule is pinned to a commit on the plabench branch of its fork, which is the upstream commit listed here plus the benchmark patches. forks/README.md gives the patch list per fork; git submodule status prints the pinned SHAs of your own checkout.

Weight files are inventoried separately: checkpoints/THIRD_PARTY.tsv records the official download URL, SHA256 and license of every third-party checkpoint, and checkpoints/MANIFEST.tsv covers the ones PLABench trained.

The environment each model was benchmarked in. They conflict, which is why there is one per model. environments/ holds a conda export of all nine, and environments/README.md is the install guide.

@unpublished{wang2026plabench,
  title  = {Leakage-controlled benchmarking reveals generalization limits of
            deep learning for protein-ligand binding affinity prediction},
  author = {Wang, Lyuwei and Cheng, Jianlin},
  year   = {2026},
  note   = {Manuscript in submission}
}

Two DOIs go with the paper. The benchmark inputs, structures, checkpoints and predictions are at 10.5281/zenodo.22716174, and the archived v1.0.0 source snapshot of this repository is at 10.5281/zenodo.22782934. Cite whichever you used on its own.

If you report a number from one of the nine models, cite that model's own paper as well. forks/README.md lists them.

The code is MIT, in LICENSE. The data and the weights are not, and each carries the terms of whoever produced it:

data/SOURCES.tsv is the per-path record and is the file to check before reusing anything here.