attention_with_linear_biases

Code for the ALiBi method for transformer language models (ICLR 2022)

Stars

479

Forks

37

Language

Python

Last Updated

May 17, 2024

Similar Repos
Repo Language Stars Description Updated At
Python 2 Code for ICLR 2022 submission "Differentially Private Fine-tuning of Language Models". Jul 20, 2022
Python 59 [ICLR 2022] Towards Continual Knowledge Learning of Language Models Sep 01, 2022
Python 17 Official code for "Dynamic Token Normalization Improves Vision Transformer", ICLR 2022. Jun 30, 2022
Python 161 [ICLR 2022 spotlight]GreaseLM: Graph REASoning Enhanced Language Models for Question Answering Apr 24, 2023
Python 179 [ICLR 2023] Code for the paper "Binding Language Models in Symbolic Languages" Apr 26, 2023
Python 27 Code for SegTree Transformer (ICLR-RLGM 2019). Jan 28, 2023
Python 96 Transformer Grammars: Augmenting Transformer Language Models with Syntactic Inductive Biases at Scale, TACL (2022) Mar 19, 2023
Python 4 Code for the ACL 2021 paper "Structural Guidance for Transformer Language Models" Aug 01, 2022
Python 37 [ICLR 2022] RelViT: Concept-guided Vision Transformer for Visual Relational Reasoning Aug 12, 2022
Jupyter Notebook 10 Staged Training for Transformer Language Models Jul 21, 2022
Jupyter Notebook 96 PyTorch code for “TVLT: Textless Vision-Language Transformer” (NeurIPS 2022 Oral) May 29, 2023
Python 41 Code for text augmentation method leveraging large-scale language models Aug 17, 2022
Python 12 ICLR 2022 May 05, 2023
Python 143 Label-Efficient Semantic Segmentation with Diffusion Models (ICLR'2022) Aug 15, 2022
Python 2 Transformer Language Models using Pytorch Lightning. Apr 08, 2022
Python 2 This is the code for the ICLR 2023 paper "Leveraging Large Language Models for Multiple … Mar 27, 2023
Python 6 [ICLR 2022] Huber Additive Models for Non-stationary Time Series Analysis Aug 29, 2022
Python 59 Official code for Long Expressive Memory (ICLR 2022, Spotlight) Apr 05, 2023
Python 2 Pytorch implementation for the ICLR 2022 paper Mar 21, 2023
Python 3167 Train transformer language models with reinforcement learning. Apr 24, 2023
Python 60 [ICLR 2023] Code for our paper "Selective Annotation Makes Language Models Better Few-Shot Learners" Apr 17, 2023
Python 14 Pytorch code for some vision transformer models Apr 10, 2022
Python 207 Code for the paper "How Attentive are Graph Attention Networks?" (ICLR'2022) Apr 25, 2023
OCaml 16 Transformer-based models for Natural Language Processing in OCaml Dec 14, 2022
Python 2 Transformer as Scorer (Language Model) for STT accoustic models. Mar 14, 2023
Python 21 Code for the paper "Post-hoc Concept Bottleneck Models". Spotlight @ ICLR 2023 Apr 21, 2023
Python 3 The official code base of Expressivity of Emergent Languages (ICLR-2022). Oct 20, 2022
Python 252 [ICLR 2022] code for "How Much Can CLIP Benefit Vision-and-Language Tasks?" https://arxiv.org/abs/2107.06383 Jul 01, 2022
Jupyter Notebook 222 [ICLR 2023] ReAct: Synergizing Reasoning and Acting in Language Models Mar 30, 2023
Python 107 About Code release for "Anomaly Transformer: Time Series Anomaly Detection with Association Discrepancy" (ICLR 2022 … Aug 14, 2022
Python 2 An intuitive API for analyzing the inner workings of Transformer-based language models Dec 07, 2023
Python 14 Code for the ICLR'22 paper "Improving Non-Autoregressive Translation Models Without Distillation" Apr 13, 2023
Python 120 The official code of LM-Debugger, an interactive tool for inspection and intervention in transformer-based language … May 11, 2023
Python 34 Code for ICLR 2022 Paper, "Controlling Directions Orthogonal to a Classifier" Mar 07, 2023
Jupyter Notebook 4 Code for our ICLR 2022 paper "Missingness Bias in Model Debugging" Feb 13, 2023
Python 13 Source code for ICLR 2023 spotlight paper "Hidden Markov Transformer for Simultaneous Machine Translation" May 01, 2023
Python 255 Code for the AAAI 2022 paper "SSAST: Self-Supervised Audio Spectrogram Transformer". May 05, 2023
Python 67 Code for the paper Analytic-DPM: an Analytic Estimate of the Optimal Reverse Variance in Diffusion … Aug 06, 2022
Python 88 The official implementation for Pseudo Numerical Methods for Diffusion Models on Manifolds (ICLR 2022) and … Jun 08, 2022
Python 2 Code Supplementary for Unsupervised Disentanglement with Tensor Product Representations on the Torus (ICLR 2022) May 16, 2023
Jupyter Notebook 2 Terminology extraction on ACTER using Transformer-based language models Jun 17, 2022
Jupyter Notebook 79 Adapt Transformer-based language models to new text domains Feb 28, 2023
Python 7 Code for the DEQ experiments of the ICLR 2022 spotlight "SHINE: SHaring the INverse Estimate … Nov 16, 2022
Python 33 Open-source benchmark datasets and pretrained transformer models in the Filipino language. Jul 11, 2022
Jupyter Notebook 11 Code that accompanies the PyData New York (2022) talk: Addressing the sensitivity of Large language … Apr 10, 2023
Python 194 Code and Checkpoints for "Generate rather than Retrieve: Large Language Models are Strong Context Generators" … May 13, 2023
Python 618 [ICLR 2023 Spotlight] Vision Transformer Adapter for Dense Predictions May 02, 2023
Python 8 Code for EMNLP 2022 paper titled "Making Pretrained Language Models Good Long-tailed Learners" Apr 27, 2023
Python 216 [ECCV 2022]Code for paper "DaViT: Dual Attention Vision Transformer" Apr 26, 2023
Python 25 Code for "Continuous-Time Meta-Learning with Forward Mode Differentiation" (ICLR 2022) Apr 15, 2023