Learn Template Tricks
Override any config parameter from command line
python train.py trainer.max_epochs=20 model.optimizer.lr=1e-4
python train.py +model.new_param="owo"
Train on CPU, GPU, multi-GPU and TPU
# train on CPU
python train.py trainer=cpu
# train on 1 GPU
python train.py trainer=gpu
# train on TPU
python train.py +trainer.tpu_cores=8
# train with DDP (Distributed Data Parallel) (4 GPUs)
python train.py trainer=ddp trainer.devices=4
# train with DDP (Distributed Data Parallel) (8 GPUs, 2 nodes)
python train.py trainer=ddp trainer.devices=4 trainer.num_nodes=2
# simulate DDP on CPU processes
python train.py trainer=ddp_sim trainer.devices=2
# accelerate training on mac
python train.py trainer=mps
Train with mixed precision
# train with pytorch native automatic mixed precision (AMP)
python train.py trainer=gpu +trainer.precision=16
Train model with any logger available in PyTorch Lightning, like W&B or Tensorboard
# set project and entity names in `configs/logger/wandb`
wandb:
project: "your_project_name"
entity: "your_wandb_team_name"
# train model with Weights&Biases (link to wandb dashboard should appear in the terminal)
python train.py logger=wandb
Train model with chosen experiment config
python train.py experiment=example
Attach some callbacks to run
python train.py callbacks=default
Use different tricks available in Pytorch Lightning
# gradient clipping may be enabled to avoid exploding gradients
python train.py +trainer.gradient_clip_val=0.5
# run validation loop 4 times during a training epoch
python train.py +trainer.val_check_interval=0.25
# accumulate gradients
python train.py +trainer.accumulate_grad_batches=10
# terminate training after 12 hours
python train.py +trainer.max_time="00:12:00:00"
Easily debug
# runs 1 epoch in default debugging mode
# changes logging directory to `logs/debugs/...`
# sets level of all command line loggers to 'DEBUG'
# enforces debug-friendly configuration
python train.py debug=default
# run 1 train, val and test loop, using only 1 batch
python train.py debug=fdr
# print execution time profiling
python train.py debug=profiler
# try overfitting to 1 batch
python train.py debug=overfit
# raise exception if there are any numerical anomalies in tensors, like NaN or +/-inf
python train.py +trainer.detect_anomaly=true
# use only 20% of the data
python train.py +trainer.limit_train_batches=0.2 \
+trainer.limit_val_batches=0.2 +trainer.limit_test_batches=0.2
Resume training from checkpoint
python train.py ckpt_path="/path/to/ckpt/name.ckpt"
Evaluate checkpoint on test dataset
python eval.py ckpt_path="/path/to/ckpt/name.ckpt"
Create a sweep over hyperparameters
# this will run 6 experiments one after the other,
# each with different combination of batch_size and learning rate
python train.py -m data.batch_size=32,64,128 model.lr=0.001,0.0005
Create a sweep over hyperparameters with Optuna
# this will run hyperparameter search defined in `configs/hparams_search/mnist_optuna.yaml`
# over chosen experiment config
python train.py -m hparams_search=mnist_optuna experiment=example
Execute all experiments from folder
python train.py -m 'experiment=glob(*)'
Execute run for multiple different seeds
python train.py -m seed=1,2,3,4,5 trainer.deterministic=True logger=csv tags=["benchmark"]
Use Hydra tab completion
**Note**: Hydra allows you to autocomplete config argument overrides in shell as you write them, by pressing `tab` key. Read the [docs](https://hydra.cc/docs/tutorials/basic/running_your_app/tab_completion).Run tests
# run all tests
pytest
# run tests from specific file
pytest tests/test_train.py
# run all tests except the ones marked as slow
pytest -k "not slow"
Use tags
Each experiment should be tagged in order to easily filter them across files or in logger UI:python train.py tags=["mnist","experiment_X"]
>>python train.py tags=[]
[2022-07-11 15:40:09,358][src.utils.utils][INFO] - Enforcing tags! <cfg.extras.enforce_tags=True>
[2022-07-11 15:40:09,359][src.utils.rich_utils][WARNING] - No tags provided in config. Prompting user to input tags...
Enter a list of comma separated tags (dev):
>>python train.py -m +x=1,2,3 tags=[]
ValueError: Specify tags before launching a multirun!