optimizer

The largest collection of PyTorch image encoders / backbones. Including train, eval, inference, export scripts, and pretrained weights -- ResNet, ResNeXT, EfficientNet, NFNet, Vision Transformer (ViT), MobileNetV4, MobileNet-V3 & V2, RegNet, DPN, CSPNet, Swin Transformer, MaxViT, CoAtNet, ConvNeXt, and more

Python SQL Parser and Transpiler

AdalFlow: The library to build & auto-optimize LLM applications.

torch-optimizer -- collection of optimizers for Pytorch

On the Variance of the Adaptive Learning Rate and Beyond

SAM: Sharpness-Aware Minimization (PyTorch)

Portfolio optimization and back-testing.

The official implementation of “Sophia: A Scalable Stochastic Second-order Optimizer for Language Model Pre-training”

Scour - An SVG Optimizer / Cleaner

Adan: Adaptive Nesterov Momentum Algorithm for Faster Optimizing Deep Models

End-to-end Generative Optimization for AI Agents

The official implementation of MARS: Unleashing the Power of Variance Reduction for Training Large Models

TorchOpt is an efficient library for differentiable optimization built upon PyTorch.

Code for Adam-mini: Use Fewer Learning Rates To Gain More https://arxiv.org/abs/2406.16793

Model Compression Toolkit (MCT) is an open source project for neural network model optimization under efficient, constrained hardware. This project provides researchers, developers, and engineers advanced quantization and compression tools for deploying state-of-the-art neural networks.

optimizer & lr scheduler & loss function collections in PyTorch

AdamP: Slowing Down the Slowdown for Momentum Optimizers on Scale-invariant Weights (ICLR 2021)

Effortless plugin and play Optimizer to cut model training costs by 50%. New optimizer that is 2x faster than Adam on LLMs.

APOLLO: SGD-like Memory, AdamW-level Performance; MLSys'25 Oustanding Paper Honorable Mention

lookahead optimizer (Lookahead Optimizer: k steps forward, 1 step back) for pytorch