InferLLM

a lightweight LLM model inference framework

Stars

632

Forks

78

Language

C++

Last Updated

May 13, 2024

Similar Repos
Repo Language Stars Description Updated At
C++ 2 a lightweight LLM model inference framework Jun 03, 2023
Python 25 Large Language Model (LLM) Inference API and Chatbot Apr 27, 2023
Python 138 LLM Inference benchmark Jan 18, 2024
Python 8 Reward Model framework for LLM RLHF May 09, 2023
None 639 LLM papers I'm reading, mostly on inference and model compression Jan 12, 2024
C++ 205 C++ model train&inference framework Sep 11, 2022
Jupyter Notebook 4 Benchmarking LLM Inference Speeds Oct 12, 2023
Python 1376 LightLLM is a Python-based LLM (Large Language Model) inference and serving framework, notable for its … Jan 19, 2024
Python 628 A lightweight framework for building LLM-based agents Jan 19, 2024
Python 5 LLM inference with HuggingFace (experimental) Mar 27, 2023
C++ 80 WebGPU LLM inference tuned by hand May 20, 2023
C++ 399 TinyChatEngine: On-Device LLM Inference Library Jan 17, 2024
C++ 95 RTP-LLM: Alibaba's high-performance LLM inference engine for diverse applications. Jan 19, 2024
Python 62 llm-export can export llm model to onnx. Jan 18, 2024
Cuda 122 Flash-LLM: Enabling Cost-Effective and Highly-Efficient Large Generative Model Inference with Unstructured Sparsity Jan 17, 2024
Python 16 ChatGPT LangChain LLM model Mar 10, 2023
C++ 2 Super lightweight yolov7 inference Jan 04, 2023
Python 2 LLaMa model inference Mar 08, 2023
Jupyter Notebook 123 LLM (Large Language Model) FineTuning Jan 17, 2024
TypeScript 102 ✦ The intuitive LLM framework Dec 24, 2023
Rust 2 Fast inference of LLM on cpu written in rust Jul 13, 2023
Python 8 Lightweight framework for structured and repeatable model validation Aug 18, 2022
Python 7 Inference Model for BertSum Nov 17, 2021
Python 116 Large-scale model inference. Aug 28, 2022
Python 2 Language Model Inference API Apr 24, 2023
Go 3 GPT-2 Model Inference Mar 08, 2023
C++ 60 Model-less Inference Serving Apr 04, 2023
Jupyter Notebook 1304 Simple UI for LLM Model Finetuning Apr 09, 2023
Jupyter Notebook 25 Enhanced BiLSTM Inference Model for Natural Language Inference Jan 31, 2023
C++ 23 RidgeRun Inference Framework Jan 23, 2023
Python 2 Lightweight implementation of a model router for multi LLM chain dispatch. Alternatively can be assumed … Apr 27, 2023
Python 165 ✦ The intuitive python LLM framework Jan 12, 2024
C++ 49 Lightweight Component Model and Messaging Framework based on ØMQ Jul 19, 2022
Python 1651 AutoChain: Build lightweight, extensible, and testable LLM Agents Jan 19, 2024
MATLAB 14 The COntextual INference (COIN) model Mar 10, 2023
Prolog 3 Bayesian inference of model structure Jan 02, 2022
Python 5 Inference YOLO-NAS ONNX model May 25, 2023
Python 46 Inference Model Manager for Kubernetes May 25, 2023
None 3 paper list about large language model (LLM) Mar 23, 2023
Jupyter Notebook 4 Korean Grammar Correction Model based on LLM Jun 14, 2023
Jupyter Notebook 15 [Deprecated] PyTorch Lite is a lightweight machine learning framework for on-device mobile inference. Jul 30, 2022
Python 2 Common Framework for Inference May 12, 2022
Python 3 General / Global Inference Framework Dec 17, 2022
None 680 📖A curated list of Awesome LLM Inference Paper with codes, TensorRT-LLM, vLLM, streaming-llm, AWQ, SmoothQuant, … Jan 19, 2024
None 9 Awesome series for Large Language Model(LLM)s Mar 30, 2023
None 3 Mental Model for better understanding LLM/Agent space Apr 21, 2023
None 3 An overview of Large Language Model (LLM) options Apr 24, 2023
Jupyter Notebook 47 What can I do with a LLM model? Apr 26, 2023
None 2 An overview of Large Language Model (LLM) options May 07, 2023
Java 2 AI features using a LLM (Large Language Model) Sep 02, 2023