Megatron-DeepSpeed

Ongoing research training transformer language models at scale, including: BERT & GPT-2

Stars

1255

Forks

209

Language

Python

Last Updated

May 24, 2024

Similar Repos
Repo Language Stars Description Updated At
None 2 Ongoing research training transformer language models at scale, including: BERT & GPT-2 Jun 12, 2021
Python 133 Ongoing research training transformer language models at scale, including: BERT & GPT-2 Aug 07, 2022
None 2 Ongoing research training transformer language models at scale, including: BERT & GPT-2 May 22, 2023
Python 4 Ongoing research training transformer language models at scale, including: BERT & GPT-2 May 30, 2023
Python 4 Ongoing research training transformer language models at scale, including: BERT & GPT-2 Jun 02, 2023
Python 14 Ongoing research training transformer language models at scale, including: BERT Apr 16, 2021
Python 3149 Ongoing research training transformer models at scale Aug 10, 2022
None 2 Ongoing research training transformer models at scale Jun 17, 2022
Python 4 Ongoing research training transformer models at scale Apr 18, 2023
Python 5 Ongoing research training transformer models at scale Dec 02, 2022
Python 14 Ongoing research training transformer models at scale Dec 08, 2023
Python 2 Ongoing research training transformer models at scale May 09, 2024
C++ 4 Transformer related optimization, including BERT, GPT Nov 12, 2021
C++ 1244 Transformer related optimization, including BERT, GPT Aug 31, 2022
C++ 2 Transformer related optimization, including BERT, GPT Apr 22, 2023
C++ 16 Transformer related optimization, including BERT, GPT Apr 24, 2023
C++ 2 Transformer related optimization, including BERT, GPT May 21, 2023
C++ 3 Transformer related optimization, including BERT, GPT May 16, 2023
Jupyter Notebook 10 Staged Training for Transformer Language Models Jul 21, 2022
Python 4 Training transformer models (e.g. RoBERTa, GPT2 and GPT-J) from scratch. Oct 09, 2023
Python 17 Pre-training BERT masked language models with custom vocabulary Feb 19, 2023
Python 96 Transformer Grammars: Augmenting Transformer Language Models with Syntactic Inductive Biases at Scale, TACL (2022) Mar 19, 2023
Python 4 Understanding "A Lite BERT". An Transformer approach for learning self-supervised Language Models. Aug 09, 2022
Python 4 PyTorch DistributedDataParallel training for Transformer models. Apr 04, 2023
Python 230 Transformer with Untied Positional Encoding (TUPE). Code of paper "Rethinking Positional Encoding in Language Pre-training". … Aug 30, 2022
Jupyter Notebook 8 📖The Big-&-Extending-Repository-of-Transformers: Pretrained PyTorch models for Google's BERT, OpenAI GPT & GPT-2, Google/CMU Transformer-XL. Mar 23, 2022
Jupyter Notebook 2 📖The Big-&-Extending-Repository-of-Transformers: Pretrained PyTorch models for Google's BERT, OpenAI GPT & GPT-2, Google/CMU Transformer-XL. May 05, 2023
Python 2 📖The Big-&-Extending-Repository-of-Transformers: Pretrained PyTorch models for Google's BERT, OpenAI GPT & GPT-2, Google/CMU Transformer-XL. Aug 16, 2022
Python 11 📖The Big-&-Extending-Repository-of-Transformers: Pretrained PyTorch models for Google's BERT, OpenAI GPT & GPT-2, Google/CMU Transformer-XL. Mar 03, 2023
Python 2 Colab notebooks and code used for training and inferencing BERT and other transformer models from … Nov 29, 2019
None 129 DBMDZ BERT, DistilBERT, ELECTRA, GPT-2 and ConvBERT models Aug 12, 2022
Python 36 Implement Question Generator with SOTA pre-trained Language Models (RoBERTa, BERT, GPT, BART, T5, etc.) May 05, 2023
Python 163 Transformer language model (GPT-2) with sentencepiece tokenizer Mar 12, 2023
Python 11 Transformer language model (GPT-2) with sentencepiece tokenizer Dec 23, 2021
Python 222 BERT as language model, fork from https://github.com/google-research/bert Aug 17, 2022
Python 2 Transformer Language Models using Pytorch Lightning. Apr 08, 2022
Python 2 Efficiently training Transformer-based models in single line. May 06, 2023
Python 27 Code for pre-training CharacterBERT models (as well as BERT models). May 09, 2022
None 117 Arabic edition of BERT pretrained language models Aug 30, 2022
None 180 PERT: Pre-training BERT with Permuted Language Model Aug 13, 2022
HTML 4 Data Management in large-scale education research training series Feb 04, 2023
Python 3167 Train transformer language models with reinforcement learning. Apr 24, 2023
Python 430 Can large language models provide useful feedback on research papers? A large-scale empirical analysis. Jan 18, 2024
None 8 A collection of German GPT language models Feb 12, 2023
Python 523 Keras library for building (Universal) Transformers, facilitating BERT and GPT models Sep 08, 2022
Python 85 Training GPT-2 on a Russian language corpus Feb 26, 2023
Jupyter Notebook 18 Neural Language Models for Historical Research Mar 20, 2023
Python 68 Evaluation suite for large-scale language models. Jul 07, 2022
Python 2 Components for training large language models Nov 22, 2022
Python 88 Efficient Training (including pre-training and fine-tuning) for Big Models Aug 12, 2022