Benchmarks

Introduction

Baseline results obtained using the tfcaidm pipeline. These results can act as a leaderboard for individuals interested in benchmarking their different models and methodologies. A user can submit their results by writing a few lines of code, with examples shown below. For those new to the lab, this is a great place to start!

Note: Benchmarks are currently only supported on the CAIDM clusters, additionally, tfcaidm must be installed.

Getting started

To get started, users should checkout all related benchmark datasets and their descriptions using the help method. They can also checkout the current leaderboard standings with the leaderboard method. This site will be updated with the best submissions periodically, but for real-time updates, use the methods below.

from tfcaidm import Benchmark

# --- Get benchmark related information
Benchmark.help()

# --- View the leaderboards
Benchmark.leaderboard()

To run a benchmark, a user needs to specify the benchmark dataset path and a trained model. From there they will have to align the dataset inputs and targets with their model inputs and outputs. After doing so, they will be able to run model evaluation and view their results. A submission can then optionally be made to the leaderboards.

Running a benchmark

from tfcaidm import Model
from tfcaidm import Benchmark
from jarvis.utils.general import gpus

DATA_PATH = "benchmark/kits"
MODEL_PATH = "checkpoints/unet"

# --- Autoselect GPU (use only on caidm cluster)
gpus.autoselect()

# --- Load a trained model
model = Model.load_model(MODEL_PATH)

# --- Ensure model inputs and outputs matches the dataset inputs and targets
model = Model.inference_mode(model, inputs=["dat"], outputs=["lbl"])

# --- Instantiate a benchmark object
benchmark = Benchmark(DATA_PATH)

# --- Run model evaluation
benchmark.run(model)

# --- If you are happy with your score, submit it
benchmark.submit()

Datasets

Dataset	Client	Description	Annotation	Inputs	Outputs	Additional Information	# Evaluation Samples	Fold
ct/kits	/data/ymls/client-2d.yml	2D kidney tumor segmentation	tumor segmentation	["dat"]	["lbl"]	ct-kidney-kits		0
ct/pna	/data/ymls/client.yml	2D pneumonia segmentation	pneumonia segmentation	["dat", "lng"]	["pna"]	ct-lung-pna		0
mr/brats	/data/ymls/client-5-class.yml	2D brain tumor segmentation	tumor segmentation	["t2", "flair", "pre", "post"]	["lbl"]	mr-brain-tumor		0
mr/breast-fgt	/data/ymls/client-3-class.yml	2D breast fibroglandular tissue segmentation	fibroglandular tissue segmentation	["dat"]	["lbl"]	mr-breast-fgt		0

---

Results

Rank	User	Dataset	Model	# Params	Description	Metrics	Score ↑	Date
1	tfcaidm	ct/pna	unet		pixel classifier w/ class weights	F1
2	tfcaidm	ct/pna	unet++		pixel classifier w/ class weights	F1
3	tfcaidm	ct/pna	unet3+		pixel classifier w/ class weights	F1