PyTorch

PyTorch Lightning & Lightning Hydra Template are neat. Tensor Puzzles is great intro.

TorchScale is nice library for making Transformers efficiently. BackPACK is interesting too.

WebDataset is interesting.

Notes

• torch.fx allows you to write composable transformations in Python that take in PyTorch code and output PyTorch code

Links

• PyTorch Tutorials
• RL Adventure
• PyTorch template project
• PyTorch GAN - PyTorch implementations of Generative Adversarial Networks.
• PyTorch examples
• Grokking PyTorch
• Awesome PyTorch list
• OCaml PyTorch - OCaml bindings for PyTorch.
• Apex - PyTorch Extension: Tools for easy mixed precision and distributed training in PyTorch.
• Translate - PyTorch Language Library.
• What is torch.nn really?
• Introduction to PyTorch Workshop at the AMLD 2019
• Neural Pipeline - Neural networks training pipeline based on PyTorch. Designed to standardize training process and accelerate experiments.
• Faster R-CNN and Mask R-CNN in PyTorch
• tch-rs - Rust bindings for PyTorch.
• PyTorch Geometric - Geometric deep learning extension library for PyTorch.
• PyTorch Geometric Temporal - Temporal (dynamic) extension library for PyTorch Geometric.
• BigGAN-PyTorch - Contains code for 4-8 GPU training of BigGANs from Large Scale GAN Training for High Fidelity Natural Image Synthesis.
• Incredible PyTorch - Curated list of tutorials, papers, projects, communities and more relating to PyTorch.
• NimTorch - Nim frontend for pytorch, aiming to be mostly auto-generated and internally using ATen.
• PyTorch adds new dev tools as it hits production scale (2019)
• PyTorch tutorials demonstrating modern techniques with readable code
• PyTorch internals
• Towards Reproducible Research with PyTorch Hub (2019)
• XLM - PyTorch original implementation of Cross-lingual Language Model Pretraining.
• BoTorch - Bayesian optimization in PyTorch.
• PyTorch Lightning - Lightweight PyTorch wrapper for high-performance AI research. Scale your models, not the boilerplate. (Tutorial) (Finding why Pytorch Lightning made my training 4x slower) (PyTorch Lightning for Dummies) (Tutorial) (Docs) (Web) (Intro) (Intro Code)
• PointNet.pytorch - PyTorch implementation for "PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation".
• former - Simple transformer implementation from scratch in PyTorch.
• PyTorch Implementation of Differentiable ODE Solvers
• PyTorch BigGraph - Software used for generating embeddings from large-scale graph-structured data.
• Deep Learning (PyTorch) - Projects and exercises for the latest Deep Learning ND program.
• PyWarm - Cleaner way to build neural networks for PyTorch.
• PyTorch implementations of algorithms for density estimation
• Higher - PyTorch library allowing users to obtain higher order gradients over losses spanning training loops rather than individual training steps.
• Detectron2 - FAIR's next-generation research platform for object detection and segmentation.
• Deep learning with PyTorch: a 60 minute blitz (HN)
• Neural Network Programming - Deep Learning with PyTorch (2019) - Series is all about neural network programming and PyTorch.
• A quick tour of Torch internals (2015)
• How to Run PyTorch with TPUs
• EfficientNet PyTorch
• Captum - Model interpretability and understanding for PyTorch.
• Pytorch-Struct - Fast, general, and tested differentiable structured prediction in PyTorch.
• Dive Into Deep Learning 2 PyTorch
• PyTorch at Tesla - Andrej Karpathy, Tesla (2019) (HN)
• 3D Ken Burns Effect - Implementation of 3D Ken Burns Effect from a Single Image using PyTorch.
• Fast LSTMs in PyTorch (2019)
• F8 2019: Developing and Scaling AI Experiences at Facebook with PyTorch
• Kaolin - PyTorch Library for Accelerating 3D Deep Learning Research.
• SpConv - PyTorch Spatially Sparse Convolution Library.
• Segmentation models - Python library with Neural Networks for Image Segmentation based on PyTorch.
• LSTM and QRNN Language Model Toolkit
• cvxpylayers - Python library for constructing differentiable convex optimization layers in PyTorch and TensorFlow using CVXPY.
• Frustum ConvNet - Sliding Frustums to Aggregate Local Point-Wise Features for Amodal 3D Object Detection.
• PyTorch Elastic - Framework that enables distributed training jobs to be executed in a fault tolerant and elastic manner.
• torchgpipe - GPipe implementation in PyTorch.
• C++ Implementation of PyTorch Tutorial for Deep Learning Researchers
• Neural Painters: A learned differentiable constraint for generating brushstroke paintings
• CycleGAN and pix2pix in PyTorch - Image-to-Image Translation in PyTorch.
• Easily Extendable Basic Deep Metric Learning Pipeline
• TTAch - Image Test Time Augmentation with PyTorch.
• Hyperspherical Variational Auto-Encoders in PyTorch
• Hyperbolic Graph Convolutional Networks in PyTorch
• Deep Learning with PyTorch Book (2020) - Provides a detailed, hands-on introduction to building and training neural networks with PyTorch. (HN) (Code)
• OpenProtein - PyTorch framework for tertiary protein structure prediction.
• GPyTorch - Highly efficient and modular implementation of Gaussian Processes in PyTorch. (Docs)
• Memory-efficient implementation of DenseNets
• PyTorch Toolbelt - PyTorch extensions for fast R&D prototyping and Kaggle farming.
• Reformer - Efficient Transformer, in PyTorch.
• Which PyTorch-compatible training abstraction libraries do you use and why? (2020)
• Deep Learning with PyTorch Course (Code)
• PyTorch: An Imperative Style, High-Performance Deep Learning Library (2019)
• Open Graph Benchmark (OGB) - Collection of benchmark datasets, data-loaders and evaluators for graph machine learning in PyTorch.
• Ignite - High-level library to help with training neural networks in PyTorch.
• PyTorch Forum
• Collection of PyTorch notebooks for learning and practicing deep learning
• face.evoLVe - High-Performance Face Recognition Library based on PyTorch.
• PyTorch implementation for 3D Bounding Box Estimation Using Deep Learning and Geometry
• Fast AutoAugment - Official Implementation of 'Fast AutoAugment' in PyTorch.
• Rethinking the Value of Network Pruning
• PyTorch Metric Learning
• Deep Tutorials for PyTorch
• Matchbox - Enables deep learning researchers to write PyTorch code at the level of individual examples, then run it efficiently on minibatches.
• torchtext - Data loaders and abstractions for text and NLP.
• Relational Networks - PyTorch implementation of "A simple neural network module for relational reasoning".
• nflows - Normalizing flows in PyTorch.
• torchlayers - Shape inference for PyTorch (like in Keras) + new SoTA layers.
• PyTorch re-implementation of Neural Radiance Fields
• Torchmeta - Collection of extensions and data-loaders for few-shot learning & meta-learning in PyTorch.
• PyTorch SimCLR: A Simple Framework for Contrastive Learning of Visual Representations
• Neural Spline Flow, RealNVP, Autoregressive Flow, 1x1Conv in PyTorch
• PyTorch implementation of NeRF (Neural Radiance Fields)
• PyTorch learning rate finder
• PyTorch Scatter - PyTorch Extension Library of Optimized Scatter Operations.
• TorchServe - Flexible and easy to use tool for serving PyTorch models.
• torch-optimizer - Collection of optimizers for PyTorch.
• PyTorch 1.5 (2020) (HN)
• Tacotron 2 (without wavenet) - PyTorch implementation of Natural TTS Synthesis By Conditioning Wavenet On Mel Spectrogram Predictions.
• Tacotron 2 latest papers
• pytorch-crf - Conditional random field in PyTorch.
• torchprof - PyTorch layer-by-layer model profiler.
• A Taste of PyTorch C++ frontend API (2020)
• Tutorial: Mixture Density Networks with JAX (2020)
• Deep learning CNN feature visualization using PyTorch
• rectorch - PyTorch-based framework for state-of-the-art top-N recommendation.
• neural-dream - PyTorch implementation of DeepDream algorithm.
• PyProf2 - PyTorch Profiling tool.
• Blitz - Simple and extensible library to create Bayesian Neural Network layers on PyTorch.
• PyContrast - PyTorch implementation of Contrastive Learning methods.
• PyVacy - Differentially Private Optimization Algorithms for PyTorch.
• Gaussian mixture models in PyTorch
• Effective PyTorch - PyTorch tutorials and best practices.
• PyTorch for Numpy Users (Code)
• PyTorch implementation of Fully Convolutional Networks
• Lite Transformer with Long-Short Range Attention
• PyTorch implementation of the KWNG estimator
• Classification with PyTorch
• PyTorch Toolbox - Aiming to make you write Pytorch code more easier, readable and concise.
• Sinusoidal Representation Networks (SIREN)
• PyTorch Hub - Discover and publish models to a pre-trained model repository designed for research exploration. (Code)
• Pretrained models for PyTorch
• DrQA - Reading Wikipedia to Answer Open-Domain Questions. In PyTorch.
• minGPT - Minimal PyTorch re-implementation of the OpenAI GPT (Generative Pretrained Transformer) training. (Code) (HN)
• MiniTorch - Teaching library for machine learning engineers who wish to learn about the internal concepts underlying deep learning systems.
• PyTorch Performance Tuning Guide - Szymon Migacz, NVIDIA (2020) (Summary)
• Opacus - Training PyTorch models with differential privacy. (Web) (Opacus Lab Research)
• Towards Fast, Accurate and Stable 3D Dense Face Alignment
• Hivemind - Library to train large neural networks across the internet. Imagine training one huge transformer on thousands of computers from universities, companies, and volunteers.
• Prediction and Policy-learning Under Uncertainty (PPUU)
• PyTorch Lightning Bolts - Toolbox of models, callbacks, and datasets for AI/ML researchers.
• LabML Neural Networks - Collection of simple PyTorch implementation of various neural network architectures and layers. (Code) (HN)
• PyTorch Interpret - PyTorch library for interpreting your deep learning results, using both visualisations and attributions.
• Lambda Networks - PyTorch - Implementation of Lambda Networks, an attention-based solution for image recognition that reaches SOTA.
• PyTorch image data augmentation
• Bootstrap Your Own Latent (BYOL), in PyTorch
• Vision Transformer - PyTorch - Implementation of Vision Transformer, a simple way to achieve SOTA in vision classification with only a single transformer encoder, in PyTorch.
• Benchmkarking Optimizers in PyTorch (2020)
• ShrinkBench - PyTorch library to facilitate development and standardized evaluation of neural network pruning methods.
• Asteroid - PyTorch-based audio source separation toolkit for researchers.
• The Hitchhikers's Guide to PyTorch for Data Scientists (2020)
• Pointnet2/Pointnet++ PyTorch
• Dreamer PyTorch - Dream to Control: Learning Behaviors by Latent Imagination, implemented in PyTorch.
• project-NII-pytorch-scripts
• fairscale - PyTorch extension library for high performance and large scale training for optimizing training on one or across multiple machines/nodes.
• FastSpeech PyTorch
• Lightwood - Pytorch based framework that breaks down machine learning problems into smaller blocks that can be glued together.
• PyTorch Loss Functions: Guide (2020)
• David's Tips on How to Read PyTorch
• MeshCNN in PyTorch - General-purpose deep neural network for 3D triangular meshes, which can be used for tasks such as 3D shape classification or segmentation.
• Tinygrad - Simple and powerful neural network framework. (tinygrad: tiny efficientnet is recognize cat?) (HN) (Web) (HN)
• Lucent - Lucid library adapted for PyTorch.
• pytorch-pfn-extras - Supplementary components to accelerate research and development in PyTorch.
• PyHessian - PyTorch library for second-order based analysis and training of Neural Networks.
• AdaHessian - Second order based optimizer for the neural network training based on PyTorch.
• TorchMTL - Lightweight module to do Multi-Task Learning in PyTorch.
• TRTorch - PyTorch/TorchScript compiler for NVIDIA GPUs using TensorRT. (Docs)
• Mastering TorchScript: Tracing vs Scripting, Device Pinning, Direct Graph Modification (2020)
• Flow Forecast - Deep learning PyTorch library for time series forecasting, classification, and anomaly detection.
• SpykeTorch - High-speed simulator of convolutional spiking neural networks with at most one spike per neuron.
• PyTorch Loss - Label-smooth, amsoftmax, focal-loss, triplet-loss.
• Cleora - Faster alternative to PyTorch-BigGraph.
• The Original Transformer (PyTorch)
• PyTorch implementation of the hamburger module from the ICLR 2020 paper "Is Attention Better Than Matrix Decomposition"
• PyTorch Developer Day 2020
• MLflow and PyTorch integration – autologging and TorchServe plugin (2020)
• HiddenLayer - Lightweight library for neural network graphs and training metrics for PyTorch, Tensorflow, and Keras.
• LabML - Monitor PyTorch & TensorFlow model training on mobile phones. (Docs)
• Old Photo Restoration (Official PyTorch Implementation)
• Torch-Pruning - PyTorch pruning toolkit for structured neural network pruning and layer dependency maintaining.
• torchac - Fast Arithmetic Coding for PyTorch.
• AdaHessian - Easy-to-use AdaHessian optimizer (PyTorch).
• PyTorch Dynamic Quantization (2020)
• Performer in PyTorch - Implementation of Performer, a linear attention-based transformer, in PyTorch.
• GoTch - Go binding for Pytorch C++ API (libtorch).
• auraloss - Collection of audio-focused loss functions in PyTorch.
• Normalizing flows with PyTorch - Implementation and tutorials of normalizing flows with the novel distributions module.
• PyTorch Lightning Snippets
• Stylized Neural Painting (2021) (PyTorch Code)
• PyTorch Tools, Best Practices & Style Guide
• NeuralProphet - Neural Network based Time-Series model, inspired by Facebook Prophet and AR-Net, built on PyTorch.
• Pytorch-Lightning Implementation of Self-Supervised algorithms
• Reimplementation of LiteFlowNet using PyTorch
• Julius - Fast PyTorch based DSP for audio and 1D signals.
• PyTorch Points 3D - PyTorch framework for doing deep learning on point clouds.
• PyTorch Lightning project template
• TorchData - PyTorch dataset extended with map, cache etc. (tensorflow.data like).
• pytorch_memlab - Profiling and inspecting memory in PyTorch.
• NeRF (Neural Radiance Fields) using PyTorch (PyTorch Lightning)
• Faster PyTorch Implementation of Faster R-CNN
• Axial Attention in PyTorch - Implementation of Axial Attention - attending to multi-dimensional data efficiently.
• torchaudio - Audio library for PyTorch.
• Benchmark Suite for Deep Learning - Provides a wrapper to run PyTorch benchmarks using NVidia's Deep Learning Examples repo.
• Fast Transformers - PyTorch library for fast transformer implementations.
• PyTorch Quantization Aware Training (2020)
• Unsupervised Learning of Image Segmentation Based on Differentiable Feature Clustering
• WebDataset - High-performance Python-based I/O system for large (and small) deep learning problems, with strong support for PyTorch.
• Semantic Segmentation in PyTorch
• torchlambda - Lightweight tool to deploy PyTorch models to AWS Lambda.
• torchfunc - PyTorch functions and utilities to make your life easier.
• PyTorch-Reparam-Module - Reparameterize your PyTorch modules.
• Pytorch Forecasting - Time series forecasting with PyTorch.
• DALL-E in PyTorch - Implementation / replication of DALL-E, OpenAI's Text to Image Transformer, in PyTorch.
• PyTorch Architectures - PyTorch implementations of research paper model architectures.
• Ways of making PyTorch training faster (2021)
• RepVGG: Making VGG-style ConvNets Great Again (PyTorch)
• PyTorch-OpCounter - Count the MACs / FLOPs of your PyTorch model.
• PyTorch Memory optimizations via gradient checkpointing
• Efficient PyTorch - Best practices of training large datasets using PyTorch.
• SAM: Sharpness-Aware Minimization (PyTorch)
• Factorization Machine models in PyTorch
• graftr - Interactive shell to view and edit PyTorch checkpoints.
• EagerPy - Writing Code That Works Natively with PyTorch, TensorFlow, JAX, and NumPy. (Web)
• Evidential Deep Learning to Quantify Classification Uncertainty (2018) (Code)
• PyProf - GPU performance profiling tool for PyTorch models.
• ctcdecode - Implementation of CTC (Connectionist Temporal Classification) beam search decoding for PyTorch.
• OpenMM PyTorch Plugin - OpenMM plugin to define forces with neural networks.
• Totally Versatile Miscellania for PyTorch
• Lightning Flash - Collection of tasks for fast prototyping, baselining, finetuning and solving problems with deep learning.
• FlashTorch - Python visualization toolkit, built with PyTorch, for neural networks in PyTorch.
• torchinfo - View model summaries in PyTorch.
• Compressive transformer PyTorch implementation with notes
• DALL-E - PyTorch package for the discrete VAE used for DALL·E. (HN)
• PyTorch: From Centralized To Federated (2021)
• PyTorch 1.8, with AMD ROCm support (2021) (HN)
• Linear Attention Transformer in PyTorch - Transformer based on a variant of attention that is linear complexity in respect to sequence length.
• GeoTorch - Constrained optimization toolkit for PyTorch.
• LieTorch: Tangent Space Backpropagation
• FastMoE - Fast MoE implementation for PyTorch. (Code)
• NN Template - Generic template to bootstrap your PyTorch project with PyTorch Lightning, Hydra, W&B, DVC, and Streamlit.
• Lightning Hydra Template - PyTorch Lightning + Hydra. A very general, feature-rich template for rapid, scalable ML experimentation process and best practices.
• Memformer - PyTorch - Implementation of Memformer, a Memory-augmented Transformer, in PyTorch.
• torchtyping - Runtime type annotations for the shape, dtype etc. of PyTorch Tensors.
• torchbench - Easily benchmark machine learning models in PyTorch.
• pytorch-world - Nix scripts for pytorch-related libraries.
• Butterfly matrix multiplication in PyTorch
• GLOM - PyTorch - Geoffrey Hinton's new idea that integrates concepts from neural fields, top-down-bottom-up processing, and attention (consensus between columns) for learning emergent part-whole heirarchies from data.
• nnAudio - Audio processing by using PyTorch 1D convolution network.
• Lightning-Transformers - Flexible interface for high performance research using SOTA Transformers leveraging Pytorch Lightning, Transformers, and Hydra.
• PyTorch Tutorial for Deep Learning Researchers
• torchsynth - Based upon traditional modular synthesis written in PyTorch. It is GPU-optional and differentiable. (Docs)
• PyTorch Tabular - Standard framework for modelling Deep Learning Models for tabular data.
• PyTorch implementation of EfficientDet
• Transfer Learning for Semantic Segmentation using PyTorch DeepLab v3
• Class Activation Map methods implemented in PyTorch
• Neural Network Pruning - PyTorch-based pruning and research.
• Deep Learning with PyTorch Step-by-Step: Beginner's Guide (2021) (Code)
• learn2learn - PyTorch Library for Meta-learning Research.
• PyTorch builds the future of AI and machine learning at Facebook (2021)
• allRank - Framework for training learning-to-rank neural models based on PyTorch.
• KD_Lib - PyTorch library to easily facilitate knowledge distillation for custom deep learning models.
• PyTorch Sparse - PyTorch Extension Library of Optimized Autograd Sparse Matrix Operations.
• train-CLIP - PyTorch Lightning solution to training CLIP from both scratch and fine-tuning.
• Norse - Deep learning with spiking neural networks (SNNs) in PyTorch.
• Dense Matching - Dense matching library based on PyTorch.
• TorchShard - Slicing a PyTorch Tensor Into Parallel Shards.
• Growing open-source: from Torch to PyTorch (2021)
• PyTorch implementation of "Supervised Contrastive Learning"
• Equinox - Callable PyTrees and filtered JIT/grad transformations => neural networks in JAX.
• PonderNet: Learning to Ponder (HN)
• PyNIF3D - Open-source PyTorch-based library for research on neural implicit functions (NIF)-based 3D geometry representation.
• PyGCL - Graph Contrastive Learning Library for PyTorch.
• Fast Transformer - PyTorch
• Fast Transformer - TensorFlow
• Bonito - PyTorch Basecaller for Oxford Nanopore Reads.
• Working through "What is torch.nn really?" (2021)
• PyTorch WideDeep - Flexible package to combine tabular data with text and images using Wide and Deep models in PyTorch.
• Vector Quantization - PyTorch
• PytorchLightning Tutorials - Collection of PyTorch lightning tutorial form as rich scripts automatically transformed to ipython notebooks.
• torchkit - Lightweight library containing PyTorch utilities useful for day-to-day research.
• PyTorch implementation of various Attention Mechanisms, MLP, Re-parameter, Convolution
• stoke - Lightweight wrapper for PyTorch that provides a simple declarative API for context switching between devices, distributed modes, mixed-precision, and PyTorch extensions.
• pystiche - Framework for Neural Style Transfer (NST) built upon PyTorch.
• TorchData - PyTorch repo for data loading and utilities to be shared by the PyTorch domain libraries.
• Efficient Video Dataset Loading and Augmentation in PyTorch
• Accelerate PyTorch models with ONNX Runtime
• Bagua - Deep learning training acceleration framework for PyTorch.
• TorchMetrics - Machine learning metrics for distributed, scalable PyTorch applications. (Docs) (Article)
• AutoDL-Projects - Automated deep learning algorithms implemented in PyTorch.
• PyTorch 1.10 (2021) (HN)
• TorchX - Library containing standard DSLs for authoring and running PyTorch related components for an E2E production ML pipeline.
• classy - PyTorch-based library for fast prototyping and sharing of deep neural network models.
• Probabilistic Torch - Library for deep generative models that extends PyTorch.
• jax2torch - Use JAX functions in PyTorch with DLPack.
• Differentiable Neural Computers, Sparse Access Memory and Sparse Differentiable Neural Computers, for Pytorch
• Toroidal - Lightweight transformer library for PyTorch.
• Ultimate PyTorch Source-Build Template - Template repository to build PyTorch projects from source on any version of PyTorch/CUDA/cuDNN.
• hf-hub-lightning - PyTorch Lightning Callback for pushing models to the Hugging Face Hub.
• Audio Augmentations - Library for PyTorch for audio in the time-domain, with support for stochastic data augmentations.
• e2cnn - PyTorch extension for equivariant deep learning.
• Deep Learning With PyTorch - Full Course (2021)
• All You Need to Know About PyTorch (2021)
• TorchSnooper - Debug PyTorch code using PySnooper.
• PyTorch-LIT - Lite Inference Toolkit (LIT) for PyTorch which focuses on easy and fast inference of large models on end-devices.
• Interactive Deep Colorization in PyTorch
• PyTorch Batched SVD - 100x faster SVD for PyTorch.
• PyTorch Adapt - Modular domain adaptation library written in PyTorch.
• Torch Memory-adaptive Algorithms (TOMA) - Helps you write algorithms in PyTorch that adapt to the available (CUDA) memory.
• Bayesian-Torch - Bayesian neural network layers for uncertainty estimation.
• PyTorch: Where we are headed and why it looks a lot like Julia (but not exactly) (HN)
• torchfilter - Bayesian filters in PyTorch.
• PopTorch - PyTorch integration for the Graphcore IPU.
• FlowTorch - PyTorch library for learning and sampling from complex probability distributions using a class of methods called Normalizing Flows.
• Attention is all you need implemented from scratch in PyTorch
• Winners of the 2021 PyTorch Annual Hackathon
• PyTorch vs TensorFlow in 2022 (HN)
• Perceiver IO - PyTorch implementation of DeepMind's Perceiver IO with PyTorch Lightning scripts for distributed training.
• Neural network inferencing for PyTorch and TensorFlow with ONNX, WebAssembly System Interface, and WASI NN (2021)
• hamiltorch - PyTorch-based library for Riemannian Manifold Hamiltonian Monte Carlo (RMHMC) and inference in Bayesian neural networks.
• Machine Learning with PyTorch and Scikit-Learn Book (2021) (Tweet) (Code)
• Web Application to generate your training scripts with PyTorch Ignite
• torchinterp1d - CUDA 1-D interpolation for PyTorch.
• torchquad - High-performance numerical integration on the GPU with PyTorch.
• Efficient PyTorch: Tensor Memory Format Matters (2021)
• TorchSSL - PyTorch-based Toolbox for Semi-Supervised Learning.
• PyTea - PyTorch Tensor Shape Error Analyzer.
• PyTorch Builder - Scripts to build PyTorch binaries and do end-to-end integration tests.
• PyTorch VAE - Collection of Variational Autoencoders (VAE) in PyTorch.
• Pytorch MS-SSIM - Fast and differentiable MS-SSIM and SSIM for PyTorch.
• PyTorch Live - Easy to use library of tools for creating on-device ML demos on Android and iOS.
• Modeling uncertainty with PyTorch (2022)
• PyTorch distributions examples
• moolib - Library for distributed ML training with PyTorch.
• v objective diffusion inference code for PyTorch
• Texar-PyTorch - Integrating the Best of TF into PyTorch, for Machine Learning, Natural Language Processing, and Text Generation.
• solo-learn - Library of self-supervised methods for unsupervised visual representation learning powered by PyTorch Lightning.
• torch-max-mem - Decorators for maximizing memory utilization with PyTorch & CUDA.
• PyTorch implementation of various Knowledge Distillation (KD) methods
• Padertorch - Designed to simplify the training of deep learning models written with PyTorch. Focusing on speech and audio processing.
• VFormer - Modular PyTorch library for Vision Transformers.
• BindsNET - Simulation of spiking neural networks (SNNs) using PyTorch.
• Hammer - Efficient PyTorch-based library for training deep models.
• (Generic) EfficientNets for PyTorch - Pretrained EfficientNet, EfficientNet-Lite, MixNet, MobileNetV3 / V2, MNASNet A1 and B1, FBNet, Single-Path NAS.
• sbi - PyTorch package for simulation-based inference.
• torchmd - PyTorch differentiable molecular dynamics.
• TorchCAM - Simple way to leverage the class-specific activation of convolutional layers in PyTorch.
• torch2trt - PyTorch to TensorRT converter which utilizes the TensorRT Python API.
• PyTorch Toolbox for Image Quality Assessment
• Human Segmentation PyTorch - Human segmentation models, training/inference code, and trained weights, implemented in PyTorch.
• Breaching - Framework for Attacks against Privacy in Federated Learning in PyTorch.
• PyTorchViz - Small package to create visualizations of PyTorch execution graphs.
• PyTorch on Apple M1 MAX GPUs with SHARK – faster than TensorFlow-Metal (2022) (HN)
• Deep Robust - PyTorch adversarial library for attack and defense methods on images and graphs.
• TorchRec - Library for modern production recommendation systems. (Article) (HN)
• TorchStudio - IDE for PyTorch. (Web) (Beginner's Guide)
• sk2torch - Convert scikit-learn models to PyTorch modules.
• Multi-Task Learning in PyTorch
• PyTorch C++ Samples - Deep Learning sample programs of PyTorch written in C++.
• SparseLinear - PyTorch package that allows a user to create extremely wide and sparse linear layers efficiently.
• Complete Guide of Swin Transformer with Full PyTorch Implementation (2022)
• Kernel Point Convolution implemented in PyTorch
• executorch - Simple and portable executor of PyTorch programs.
• Image 2 Vec with PyTorch
• Tez - Super-simple and lightweight Trainer for PyTorch.
• TorchMultimodal - PyTorch library for training state-of-the-art multimodal multi-task models at scale.
• MiniTorch Module 0
• baukit - Kit of David's secret tools to help with productive research prototyping with PyTorch.
• Cyclemoid PyTorch - Implementation of the cyclemoid activation function for PyTorch.
• Graph Embedding - Reimplementation of Graph Embedding methods by PyTorch.
• Distributed PyTorch Lightning Training on Ray
• Lightning Pose - Accelerated pose estimation and tracking using semi-supervised convolutional networks.
• Disent - Modular VAE disentanglement framework for python built with PyTorch Lightning.
• MQBench - Open-source model quantization toolkit based on PyTorch fx.
• Self-supervised learning tutorial: Implementing SimCLR with PyTorch lightning (2022) (Code)
• PyTracking - Visual tracking library based on PyTorch.
• TorchDynamo - Python-level JIT compiler designed to make unmodified PyTorch programs faster.
• Einops Extensions - Implementation of some personal helper functions for Einops.
• TorchOpt - High-performance optimizer library built upon PyTorch for easy implementation of functional optimization and gradient-based meta-learning.
• PyTorch Connectomics - Segmentation toolbox for EM connectomics.
• TedNet - PyTorch Toolkit for Tensor Decomposition Networks.
• TorchData: PyTorch Composable Data loading utility library (2022)
• Equivariant Steerable CNNs Library for PyTorch
• PyTorch C++ inference with LibTorch
• pl_cross - Cross validation in PyTorch lightning made easy.
• diffsptk - Differentiable version of SPTK based on the PyTorch framework.
• minDiffusion - Self-contained, minimalistic implementation of diffusion models with PyTorch.
• torchdistX - Torch Distributed Experimental.
• grafog - Graph Data Augmentation Library for PyTorch Geometric.
• High-fidelity performance metrics for generative models in PyTorch
• Denoising Diffusion Probabilistic Models in PyTorch
• PyTorch Geometric Install Helper
• PyTorch Benchmarks - Collection of open source benchmarks used to evaluate PyTorch performance.
• Torchprofile - General and accurate MACs / FLOPs profiler for PyTorch models.
• K Means using PyTorch
• surgeon-pytorch - Library to inspect itermediate layers of PyTorch models. (Reddit)
• PyMDE - Minimum-distortion embedding with PyTorch.
• torchdistill - Modular, Configuration-Driven Framework for Knowledge Distillation.
• PyTorch on Angel - Arming PyTorch with a powerful Parameter Server, which enable PyTorch to train very big models.
• NITorch - Library written in PyTorch aimed at medical image processing and analysis, with a focus on neuroimaging.
• torch-ppr - Page-Rank computation using PyTorch.
• PyTorch-Energizer - Active learning library for PyTorch based on PyTorch-Lightning.
• Torchshow - Visualize PyTorch tensor in one-line of code.
• Accelerated PyTorch Training on M1 Mac (2022) (HN) (Reddit)
• Metatorch - Library and framework for prototyping PyTorch models with Haskell.
• Running PyTorch on the M1 GPU (2022) (HN)
• PyTorch Topological - Topological machine learning framework based on PyTorch.
• Optimizers - PyTorch optimization algorithms.
• MinT - Minimal Transformer Library and Tutorials.
• Apple Neural Engine (ANE) Transformers - Reference implementation of the Transformer architecture optimized for Apple Neural Engine (ANE.
• Taking Datasets, DataLoaders, and PyTorch’s New DataPipes for a Spin (2022) (Code)
• Lightning EcoSystem CI - Automated Testing for Lightning EcoSystem Projects.
• Run PyTorch models in the browser using ONNX.js
• torchsnapshot - Light-weight library for adding fault tolerance to large-scale PyTorch distributed training workloads.
• PyTorch Project Template
• PyTorch Model Compare - Tiny package to compare two neural networks in PyTorch.
• PyEPO (PyTorch-based End-to-End Predict-and-Optimize Tool) - Python-based, open-source software that supports modeling and solving predict-and-optimize problems with the linear objective function.
• Parti - PyTorch - Implementation of Parti, Google's pure attention-based text-to-image neural network, in PyTorch.
• OpenMMLab - Open source optical flow toolbox based on PyTorch.
• simpleT5 - Quickly train T5/mT5/byT5 models in just 3 lines of code.
• ngp_pl - Instant-ngp (only NeRF) in pytorch+cuda trained with pytorch-lightning (high quality with high speed).
• PyTorch time series - PyTorch implementations of neural networks for time series classification.
• DiffMat - PyTorch-based differentiable material graph library for procedural material capture.
• Torchsort - Fast, differentiable sorting and ranking in PyTorch.
• FL_PyTorch - Optimization Research Simulator for Federated Learning.
• PyTorch TestInfra - Infrastructure components that are supporting the PyTorch CI/CD system.
• TorchScene - Scene recognition tool based on PyTorch. Provide training, test and deployment functions, as well as many pretrained models.
• torchimize - Numerical optimization algorithms in PyTorch.
• 1D, 2D, and 3D Sinusoidal Postional Encoding (PyTorch and TensorFlowi)
• Kaolin Wisp - PyTorch Library and Engine for Neural Fields Research.
• Notebook walks through a single forward pass of the Transformer architecture in PyTorch
• EvoTorch - Advanced evolutionary computation library built directly on top of PyTorch.
• Writing Snake in 12 Lines of PyTorch (2022) (HN)
• PyTorch iOS Example Apps
• Bgflow - Boltzmann Generators and Normalizing Flows in PyTorch.
• Frechet Audio Distance in PyTorch
• Satellite Cloud Generator - PyTorch-based tool to generate clouds for satellite images.
• bitsandbytes - 8-bit CUDA functions for PyTorch.
• Horizontally Fused Training Array (HFTA) - Boost hardware utilization for ML training workloads via Inter-model Horizontal Fusion.
• Stannum - Fusing Taichi into PyTorch.
• Learn PyTorch for Deep Learning (Code)
• audio-diffusion-pytorch - Audio generation using diffusion models, in PyTorch.
• TorchEval - Library that contains a rich collection of performant PyTorch model metrics.
• C++ Torch Server - Serve PyTorch / torch models using Drogon.
• Deep Learning with PyTorch Step-by-Step: Beginner's Guide (2022)
• NeRF-Factory: An awesome PyTorch NeRF collection
• NestedTensor - Tools for the concurrent manipulation of variably sized Tensors.
• PyTorch Boiler - Minimalistic boiler plate code for training PyTorch models.
• PyEPO - PyTorch-based End-to-End Predict-then-Optimize Library for Linear and Integer Programming.
• torchmocks - Test PyTorch code with minimal computational overhead.
• Continual Learning - PyTorch implementation of various methods for continual learning (XdG, EWC, SI, LwF, FROMP, DGR, BI-R, ER, A-GEM, iCaRL, Generative Classifier) in three different scenarios.
• Hedwig - PyTorch deep learning models for document classification.
• GenForce - Efficient PyTorch library for deep generative modeling.
• NeuroMANCER - PyTorch-based framework for solving parametric constrained optimization problems.
• light-the-torch - Install PyTorch distributions with computation backend auto-detection.
• OpenMetricLearning - Library for metric learning pipelines.
• NerfAcc - PyTorch Nerf acceleration toolbox for both training and inference.
• PyTorch Estimate Flops - Estimate/count FLOPS for a given neural network using PyTorch.
• Bert-squeeze - Tools for Transformers compression using PyTorch Lightning.
• PyTorch dual-attention LSTM-autoencoder for multivariate Time Series
• torch-scan - Seamless analysis of your PyTorch models (RAM usage, FLOPs, MACs, receptive field, etc.).
• Kernl - Lets you run PyTorch transformer models several times faster on GPU with a single line of code, and is designed to be easily hackable.
• Open MatSci ML Toolkit - Single framework for prototyping and scaling out deep learning models for materials discovery, built on top of OpenCatalyst, PyTorch Lightning, and the Deep Graph Library.
• Torch-DCT - DCT (discrete cosine transform) functions for PyTorch.
• MultiPy - C++ library that enables you to run eager mode PyTorch models in production without any modifications to your model to support tracing.
• Video Swin Transformer in PyTorch
• PyTorch Fundamentals Learning Path
• Auto-PyTorch - Automatic architecture search and hyperparameter optimization for PyTorch.
• Auto-Tuning Hyperparameters with Optuna and PyTorch (2020)
• PyTorch Symbolic - Provides symbolic API for model creation in PyTorch.
• PyTorch 3D U-Net Convolution Neural Network (CNN) designed for medical image segmentation
• torchnnprofiler - Context Manager to profile the forward and backward times of PyTorch's nn.Module.
• Minimal IK PyTorch
• Open-Vocabulary Semantic Segmentation with Mask-adapted CLIP (2022) (Code)
• Torchview - Visualize PyTorch models.
• LAMPE - Likelihood-free AMortized Posterior Estimation with PyTorch.
• Plan2Vec: Unsupervised Representation Learning by Latent Plans in PyTorch
• BackPACK - Backpropagation package built on top of PyTorch which efficiently computes quantities other than the gradient.
• TrueGrad - PyTorch interface for TrueGrad-AdamW.
• torchplate - Minimal Experiment Workflows in PyTorch.
• TorchScale - PyTorch library that allows researchers and developers to scale up Transformers efficiently and effectively.
• PyTorch 2.0 (HN)
• Implementation of Hinton's forward-forward (FF) algorithm - an alternative to back-propagation
• Transformer encoder in PyTorch
• PiPPy - Pipeline Parallelism for PyTorch.
• Forward Forward algorithm (by Geoffrey Hinton) implemented with PyTorch
• TencentPretrain: Tencent Pre-training Framework
• Zuko - Normalizing flows in PyTorch.
• PIQA - PyTorch Image Quality Assessment.
• Neural Network Force Field based on PyTorch
• MVTorch - Modular PyTorch library for multi-view research on 3D understanding and 3D generation.
• TorchMD - End-To-End Molecular Dynamics (MD) Engine using PyTorch.
• PyTorch Vulkan Back End User Workflow
• tiny-diffusion - Minimal PyTorch implementation of probabilistic diffusion models for 2D datasets.
• Transformer implementation in PyTorch
• PyTorch Snake - Memory and compute-efficient fused PyTorch implementation of the Snake activation function from Neural Networks Fail to Learn Periodic Functions and How to Fix It.
• OctoML PyTorch Profiler
• Lion PyTorch - Optimizer discovered by Google Brain using genetic algorithms that is purportedly better than Adam.
• PyTorch tutorial for beginners
• Holistic Trace Analysis - Library to analyze PyTorch traces.
• Lightning Pod - Deep Learning Research Template for the Lightning AI Ecosystem.
• Rising - High-performance data loading and augmentation library for 2D and 3D data completely written in PyTorch.
• minLoRA - Minimal PyTorch library that allows you to apply LoRA to any PyTorch model.
• Implementation of the Hyena Operator in PyTorch
• PyTorchTS - PyTorch based Probabilistic Time Series forecasting framework based on GluonTS backend.
• Zipslicer - Library for incremental loading of large PyTorch checkpoints. (HN)
• Torch.fx.Transformer - Symbolically modify PyTorch modules. (HN)
• PyTorch 2.0 (2023) (HN)
• PyTorch 2.0 Native Flash Attention 32k Context Window (2023)