saku

A Japanese Sentence Tokenizer written in Rust.

Stars

2

Forks

0

Language

Rust

Last Updated

Jun 28, 2022

Similar Repos
Repo Language Stars Description Updated At
Python 19 Yet another sentence-level tokenizer for the Japanese text Jan 15, 2023
Rust 13 Compact Japanese tokenizer Mar 22, 2023
Python 54 Japanese tokenizer for Transformers Oct 15, 2022
Perl 7 zenrize Japanese sentence. Sep 08, 2014
Kotlin 40 A Japanese tokenizer and morphological analysis engine written in Kotlin May 17, 2023
Jupyter Notebook 2 Demonstrate using mecab Japanese Tokenizer Mar 14, 2022
Python 43 :japanese_goblin: tokenizer specified for Japanese Jul 05, 2022
Java 616 A Japanese Tokenizer for Business Oct 15, 2022
Python 3 Japanese Tokenizer for transformers library Feb 03, 2023
Python 6 NPlusOne Japanese sentence miner Aug 15, 2021
Rust 7 An Erlang source code tokenizer written in Rust. Mar 20, 2022
Rust 3 lightweight japanese ime written in rust Jul 27, 2022
Rust 69 Japanese Morphological Analysis written in Rust Apr 02, 2023
Python 3 Japanese tokenizer with KyTea for nltk Jan 28, 2023
Python 4 Thai tokenizer, POS-tagger and sentence segmenter. Feb 15, 2021
Python 3 Tokenizer and sentence splitter based on opennlp Sep 05, 2015
Python 12 🧨 Japanese Sentence Breaker 🧨 Jun 14, 2022
Rust 178 WIP: CSS tokenizer, parser, transformer, minifier, written in Rust. Jun 03, 2022
Python 8 A tokenizer and lemmatizer for Japanese text Mar 31, 2020
Python 322 Python version of Sudachi, a Japanese tokenizer. Oct 19, 2022
Julia 20 Julia version of TinySegmenter, compact Japanese tokenizer Dec 02, 2022
TypeScript 4 A Japanese tokenizer Sudachi in JavaScript (incomplete) Nov 13, 2022
Objective-C 15 Super compact Japanese tokenizer in Objective-C Nov 17, 2017
Go 361 A multilingual command line sentence tokenizer in Golang May 04, 2023
PHP 2 PHP library of zenrize Japanese sentence. Mar 23, 2014
Python 335 A Japanese tokenizer based on recurrent neural networks May 01, 2023
Rust 2 This is Interpreter written by rust. this implements lexer tokenizer parser Mar 27, 2021
Rust 3 A ruby tokenizer and type inferencer written in Rust (an experiment) Jun 13, 2022
JavaScript 9 Japanese Light Novel liked Sentence in Chinese Sep 05, 2020
Python 3 Japanese sentence compressor using the 1st algorithm in [Clarke & Lapata, 2008] written in Python3 Jan 28, 2023
JavaScript 3 A Japanese tokenizer and stopwords for Lunr JavaScript library Oct 21, 2020
Rust 17 A quiz/training app for Japanese learners. Written in Rust. Mar 22, 2023
JavaScript 11 Automated glossing of Japanese texts based on the Kuromoji tokenizer Sep 29, 2020
Emacs Lisp 2 Super compact Japanese tokenizer in Ruby ported to emacs lisp Apr 30, 2018
Python 13 Sentencepiece based BPE tokenizer for English and Japanese language text. Apr 30, 2023
Rust 2 A JSON tokenizer for Rust :cat: Aug 16, 2020
Rust 3 Rust wrapper for the Alpino tokenizer Aug 09, 2022
Rust 9 A string tokenizer library for Rust Mar 21, 2022
Ruby 6 :link: Japanese random sentence generator based on Markov chain May 05, 2022
Python 55 Plugin for sentence/vocab mining Japanese books in Anki. Aug 19, 2022
Rust 57 Rust programming, in Japanese Apr 09, 2023
Rust 51 Rust port of sentence-transformers (https://github.com/UKPLab/sentence-transformers) May 13, 2023
Java 31 Async Japanese Tokenizer Native Plugin for React Native for iOS and Android May 30, 2022
Rust 4 Japanese sentence bank program. Add and find sentences for language learning. May 03, 2021
Rust 8 ik-analyzer for rust; chinese tokenizer for tantivy May 26, 2023
None 19 Summrize ActivityPub written by Japanese. Feb 21, 2023
Go 2 Self-contained Korean Tokenizer written in pure Go Aug 22, 2020
C# 247 Open source NLP tools (sentence splitter, tokenizer, chunker, coref, NER, parse trees, etc.) in C# Sep 17, 2022
Rust 2 Rust crate with a shell-like tokenizer & token expander Jan 31, 2020
Python 4 Japanese implementation of [Filippova & Altun, 2013] (building training data for sentence compression in Japanese … Jan 28, 2023