word-segmentation · GitHub Topics

google / sentencepiece

#自然语言处理#Unsupervised text tokenizer for Neural Network-based text generation.

翻译 - 用于基于神经网络的文本生成的无监督文本标记器。

neural-machine-translation 自然语言处理 word-segmentation

C++ 10.68 k

11 天前

baidu / lac

百度NLP：分词，词性标注，命名实体识别，词重要性

word-segmentation part-of-speech-tagger named-entity-recognition chinese-word-segmentation chinese-nlp Parsing Python Java

C++ 3.91 k

4 年前

wolfgarbe / SymSpell

SymSpell: 1 million times faster spelling correction & fuzzy search through Symmetric Delete spelling correction algorithm

翻译 - SymSpell：通过“对称删除”拼写校正算法快一百万倍

levenshtein fuzzy-search approximate-string-matching edit-distance spellcheck spell-check levenshtein-distance damerau-levenshtein spelling fuzzy-matching word-segmentation chinese-text-segmentation chinese-word-segmentation spelling-correction

C# 3.21 k

1 个月前

PyThaiNLP / pythainlp

#自然语言处理#Thai natural language processing in Python

Python nlp-library 自然语言处理 word-segmentation thai Hacktoberfest computational-linguistics text-processing

Python 1.02 k

15 天前

VKCOM / YouTokenToMe

#自然语言处理#Unsupervised text tokenizer focused on computational efficiency

翻译 - 无监督文本令牌生成器专注于计算效率

自然语言处理 word-segmentation bpe tokenization

C++ 965

1 年前

mammothb / symspellpy

Python port of SymSpell: 1 million times faster spelling correction & fuzzy search through Symmetric Delete spelling correction algorithm

Python spellcheck spell-check fuzzy-matching fuzzy-search spelling-correction damerau-levenshtein approximate-string-matching levenshtein edit-distance levenshtein-distance spelling word-segmentation chinese-text-segmentation chinese-word-segmentation

Python 821

3 天前

ckiplab / ckip-transformers

CKIP Transformers

transformers language-model word-segmentation part-of-speech-tagging named-entity-recognition

Python 716

2 年前

cbaziotis / ekphrasis

#自然语言处理#Ekphrasis is a text processing tool, geared towards text from social networks, such as Twitter or Facebook. Ekphrasis performs tokenization, word normalization, word segmentation (for splitting hashta...

自然语言处理 text-processing nlp-library spelling-correction Parsing tokenization word-segmentation

Python 666

1 年前

vncorenlp / VnCoreNLP

#自然语言处理#A Vietnamese natural language processing toolkit (NAACL 2018)

dependency-parsing named-entity-recognition pos-tagging word-segmentation vietnamese-nlp 自然语言处理 pos-tagger ner Parsing vietnamese Java Python

Java 608

2 年前

bab2min / Kiwi

#自然语言处理#Kiwi(지능형 한국어 형태소 분석기)

自然语言处理 korean morphological-analysis word-segmentation C++

C++ 562

4 天前

JayYip / m3tl

#自然语言处理#BERT for Multitask Learning

bert named-entity-recognition 自然语言处理 word-segmentation multitask-learning cws pretrained-models ner text-classification multi-task-learning transformer encoder-decoder

Jupyter Notebook 546

2 年前

modelscope / AdaSeq

#自然语言处理#AdaSeq: An All-in-One Library for Developing State-of-the-Art Sequence Understanding Models

named-entity-recognition 自然语言处理 natural-language-understanding PyTorch sequence-labeling word-segmentation ner relation-extraction bert chinese-nlp crf information-extraction

Python 431

1 年前

taishi-i / nagisa

#自然语言处理#A Japanese tokenizer based on recurrent neural networks

dynet word-segmentation pos-tagging japanese nlp-library sequence-labeling 自然语言处理 Parsing

Python 396

9 个月前

ku-nlp / jumanpp

#自然语言处理#Juman++ (a Morphological Analyzer Toolkit)

自然语言处理 japanese morphological-analysis pos-tagging pos-tagger part-of-speech-tagger word-segmentation cjk Parsing

C++ 384

1 年前

jacksonllee / pycantonese

#自然语言处理#Cantonese Linguistics and NLP

cantonese computational-linguistics 自然语言处理 Python word-segmentation part-of-speech-tagging

Python 371

10 个月前

yongzhuo / Pytorch-NLU

Pytorch-NLU，一个中文文本分类、序列标注工具包，支持中文长文本、短文本的多类、多标签分类任务，支持中文命名实体识别、词性标注、分词、抽取式文本摘要等序列标注任务。 Ptorch NLU, a Chinese text classification and sequence annotation toolkit, supports multi class and multi label c...

Python PyTorch text-classification sequence-labeling named-entity-recognition word-segmentation pos-tagging chinese-text-segmentation transformers bert pretrained-models

Python 340

8 个月前

bab2min / kiwipiepy

#自然语言处理#Python API for Kiwi

自然语言处理 korean morphological-analysis word-segmentation python-library

Python 312

4 天前