Optimization & Training

Optimizers, loss functions, activation functions, schedulers, regularization, and inference

1.

Activation Functions

Sigmoid, Tanh, ReLU, GeLU, Swish and other activation functions with derivatives

2.

Inference & Model Compression WIP

Latency, throughput, quantization, pruning, and distillation

3.

Loss Functions

MSE, Cross-Entropy, Focal, Triplet, Contrastive, KL Divergence and more

4.

Optimizers

SGD, Momentum, Nesterov, AdaGrad, RMSProp, Adam with update rules

5.

PyTorch Lightning WIP

High-level framework for organizing PyTorch training code

6.

Regularization

L1, L2, Elastic Net, Dropout, Early Stopping and other regularization techniques

7.

Learning Rate Schedulers

StepLR, MultiStepLR, ExponentialLR scheduling strategies