LLM-Distributed-Quantization

Accelerating multi-node Large Language Model training with per-layer selective quantization (FP32 -> FP16) of the transformer architecture.

Stars

2

Forks

0

Language

Python

Last Updated

Oct 07, 2022

Similar Repos
Repo Language Stars Description Updated At
Jupyter Notebook 31 Post-training static quantization using ResNet18 architecture Apr 20, 2023
Python 77 Reorder-based post-training quantization for large language model Apr 12, 2023
Python 4 Code for paper "Accelerating Federated Learning for IoT in Big Data Analytics with Pruning, Quantization … Mar 25, 2022
Python 106 [IJCAI 2022] FQ-ViT: Post-Training Quantization for Fully Quantized Vision Transformer Aug 12, 2022
Python 84 SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models Nov 23, 2022
Python 3 Baseline natural/adversarial training framework with mixed precision (amp fp16) + multi-gpu support. May 17, 2021
Python 190 Feel free to fine tune large BERT models with Multi-GPU and FP16 support. Aug 09, 2022
Python 2 A model weights quantization library which converts weights to fp32, fp16 or calibrated int8 for … Feb 22, 2024
Python 44 A pre-trained model with multi-exit transformer architecture. Aug 22, 2022
JavaScript 65 Generic Repository Pattern, Singleton, Multi-layer Architecture Pattern - Examples Oct 04, 2022
Python 168 Code repo for the paper "LLM-QAT Data-Free Quantization Aware Training for Large Language Models" Jan 12, 2024
Python 74 Simple large-scale training of stable diffusion with multi-node support. Apr 20, 2023
Python 14 Official pytorch implementation of paper "Remote Sensing Image Captioning Based on Multi-Layer Aggregated Transformer" May 11, 2023
Python 2 Quantized NN (Q7D-NN) is our framework for testing quantization in single- and multi-node DNN training … Jun 30, 2022
Python 28 HUMUS-Net: a Transformer-convolutional Hybrid Unrolled Multi-Scale Network architecture for accelerated MRI reconstruction May 20, 2023
Python 208 Swarm training framework using Haiku + JAX + Ray for layer parallel transformer language models … May 09, 2023
Python 40 TensorFlow implementation of multi-layer recurrent neural networks for training and sampling from texts Mar 02, 2021
Python 73 Code for NeurIPS'19 "Layer-Dependent Importance Sampling for Training Deep and Large Graph Convolutional Networks" Apr 03, 2023
Jupyter Notebook 7 The goal of this project is to create multi-modal implementation of Transformer architecture in Swift. Jan 24, 2022
Jupyter Notebook 2 Abstractive summarisation on financial news articles with Chinese-English translation via multi-headed attention transformer architecture Feb 13, 2022
Python 65 A simple recipe for training and inferencing Transformer architecture for Multi-Task Learning on custom datasets. … Apr 29, 2023
Python 2 This work is an adaptation of CNN+Transformer architecture to training text recognition models for Yorùbá … Jan 24, 2022
Python 20 Implementation of DocFormer: End-to-End Transformer for Document Understanding, a multi-modal transformer based architecture for the … Feb 12, 2023
Jupyter Notebook 4 Transformers Workshop on behalf of ML India. Contains resource notebook for training/inferring large scale transformer … Apr 14, 2023
Python 133 The first comprehensive Robustness investigation benchmark on large-scale dataset ImageNet regarding ARchitecture design and Training … Apr 10, 2023
Python 16 "Understanding and Accelerating Neural Architecture Search with Training-Free and Theory-Grounded Metrics" by Wuyang Chen, Xinyu … Jan 06, 2022
Python 40 Unofficial Pytorch implementation of SNAC: Speaker-normalized affine coupling layer in flow-based architecture for zero-shot multi-speaker … May 30, 2023
Cuda 15 A library for accelerating Transformer models on NVIDIA GPUs, including using 8-bit floating point (FP8) … Oct 04, 2022
Cuda 2 A library for accelerating Transformer models on NVIDIA GPUs, including using 8-bit floating point (FP8) … May 05, 2023
None 29 PyTorch-Direct code on top of PyTorch-1.8.0nightly (e152ca5) for Large Graph Convolutional Network Training with GPU-Oriented … Apr 13, 2023
Kotlin 4 AlinChinese is based on Multi-Module Clean Architecture using Dagger Hilt, Coroutines, Flow, Android Jetpack and … Nov 24, 2022
Jupyter Notebook 56 PyTorch Implementation of Implicit Quantile Networks (IQN) for Distributional Reinforcement Learning with additional extensions like … May 04, 2023
Python 3 [ICONIP 2021] "Training-Free Multi-Objective Evolutionary Neural Architecture Search via Neural Tangent Kernel and Number of … Apr 19, 2022
Python 15 Implementation for paper MLP-Mixer: An all-MLP Architecture for Vision. MLP-Mixer, an architecture based exclusively on … Jul 26, 2022
Jupyter Notebook 2 This project compares deep learning techniques for multilingual text classification, focusing on language detection and … Dec 28, 2023
None 2 Vicuna-13B is a new open-source chatbot developed by researchers from UC Berkeley, CMU, Stanford, and … May 04, 2023
C++ 7 Code for Large Graph Convolutional Network Training with GPU-Oriented Data Communication Architecture (accepted by PVLDB).The … Apr 13, 2023
C# 69 This project exemplifies the implementation and dockerization of a simple Razor Web MVC Core consuming … Mar 10, 2023
Java 2 To create an CRUD-based web application, with utilisation of supporting tools, methodologies, and technologies, that … May 13, 2022
Jupyter Notebook 2 Convolution Neural Network to predict Covid-19. This is a CNN model which predicts whether you … Apr 13, 2022
Jupyter Notebook 48 In this project, urban traffic videos are collected from the middle section of Xi 'an … May 08, 2023
Python 3 By utilizing the developer's clean and simple API, users can use this project to perform … Apr 12, 2022
Python 82 # AD-Prediction Convolutional Neural Networks for Alzheimer's Disease Prediction Using Brain MRI Image ## Abstract … Apr 15, 2023
Python 3 Eulerian Video Magnification Jan 22, 2022
Python 3 Repology bulk exporter Aug 12, 2022
Python 3 Repology vulnerability data updating daemon Jun 10, 2022
Python 3 Bot to update Wikidata software project entries with information from Repology Oct 06, 2020
Python 3 Python solution to conference scheduling problem Apr 28, 2020
    todo-list-django
Python 3 None Sep 21, 2022
Python 3 3D slicer extension for interactive medical image segmentation Sep 05, 2022