tokenizer

Tokenizer (lexer) for golang

Stars

86

Forks

5

Language

Go

Last Updated

May 05, 2024

Similar Repos
Repo Language Stars Description Updated At
JavaScript 2 Tokenizer/Lexer (Scanner) for JSDoc. Mar 04, 2023
Go 5 Parsing utility (lexer/tokenizer) Jan 07, 2023
TypeScript 7 Tokenizer/lexer generator for Deno like moo Dec 04, 2020
PHP 9 A pure PHP lexer / tokenizer for RTF files Jan 28, 2023
HTML 4 Math expressions tokenizer/lexer and evaluator in JS Dec 04, 2020
JavaScript 3 Optimised tokenizer/lexer generator! 🐄 Uses /y for performance. Moo. Sep 05, 2022
JavaScript 4 A fast tokenizer/lexer core that is well tested. Apr 18, 2023
Go 9 A Text Tokenizer library for Golang Jul 14, 2022
JavaScript 13 Tokenizer for ACE editor to do syntax highlighting using an ANTLR4 lexer. Jun 16, 2022
Go 4 High performance SQL tokenizer - Golang Feb 19, 2023
Rust 2 This is Interpreter written by rust. this implements lexer tokenizer parser Mar 27, 2021
Go 9 golang basic lexer + lr parser generator Jul 18, 2021
C# 2 Contains a simplie lexer / tokenizer for splitting strings into tokens with a custom ruleset Oct 05, 2020
Go 5 graphql golang go parser lexer linter proxy Jun 19, 2022
Go 361 A multilingual command line sentence tokenizer in Golang May 04, 2023
Go 2 Lexer for Go Nov 02, 2014
JavaScript 3 Lexer for Node.js Mar 20, 2023
Rust 4 Extensions for Logos lexer Aug 11, 2022
Pascal 5 Lexer adapters for ATSynEdit May 08, 2022
Go 8 Simple lexer for Go Apr 05, 2022
Python 2 LAMMPS Lexer for Pygments Feb 25, 2023
Python 15 Pygments lexer for Julia Jul 01, 2022
Elixir 2 C lexer for Makeup Apr 20, 2022
Elixir 30 Elixir lexer for Makeup Sep 13, 2022
Elixir 2 Erlang lexer for Makeup Apr 20, 2022
D 2 lexer library for d May 19, 2020
Python 2 IOS lexer for pygments Nov 13, 2022
Python 2 JunOS lexer for pygments Nov 13, 2022
Elixir 2 Rust lexer for Makeup Mar 05, 2023
Kotlin 2 Lexer Generator for Kotlin Jan 13, 2022
Python 5 Zephir Lexer for Pygments Jun 17, 2021
Python 3 Pygments lexer for Gherkin Jan 28, 2023
Elixir 4 JSON lexer for Makeup Jul 19, 2023
JavaScript 12 Basic lexer Feb 27, 2021
Go 5 Go lexer Dec 01, 2020
Haskell 11 Applicative lexer Mar 22, 2023
PEG.js 2 TSQL lexer Feb 25, 2022
Kotlin 3 Lexer madness Dec 02, 2018
Python 3 DL‘s Lexer Jun 27, 2022
PEG.js 2 PLSQL lexer Jan 09, 2024
Python 54 Japanese tokenizer for Transformers Oct 15, 2022
Rust 35 Lindera tokenizer for Tantivy. Sep 20, 2022
JavaScript 11 A tokenizer for French Mar 31, 2021
JavaScript 2 RWKV tokenizer for node.js Jun 08, 2023
Python 2 Tokenizer For Indian Languages Mar 29, 2023
Scala 799 Korean tokenizer Aug 09, 2022
JavaScript 101 MooTools tokenizer Aug 04, 2021
TypeScript 6 Esperanto tokenizer Mar 19, 2021
JavaScript 3 efficient tokenizer Jan 28, 2023
JavaScript 2 MooTools tokenizer Jun 10, 2017