bpe

Unsupervised Word Segmentation for Neural Machine Translation and Text Generation

翻译 - 用于神经机器翻译和文本生成的无监督分词

#自然语言处理#Unsupervised text tokenizer focused on computational efficiency

翻译 - 无监督文本令牌生成器专注于计算效率

#计算机科学#The fastest JavaScript BPE Tokenizer Encoder Decoder for OpenAI's GPT-2 / GPT-3 / GPT-4 / GPT-4o / GPT-o1. Port of OpenAI's tiktoken with additional features.

#自然语言处理#Fast and customizable text tokenization library with BPE and SentencePiece support

Ready-made tokenizer library for working with GPT and tiktoken

#自然语言处理#Explains nlp building blocks in a simple manner.

#自然语言处理#Byte Pair Encoding for Python!

nfelib - bindings Python para e ler e gerir XML de NF-e, NFS-e nacional, CT-e, MDF-e, BP-e

Fast bare-bones BPE for modern tokenizer training

Go BPE tokenizer (Encoder+Decoder) for GPT2 and GPT3

#计算机科学#Machine Learning for Phishing Website Detection

Subword Encoding in Lattice LSTM for Chinese Word Segmentation

Simple-to-use scoring function for arbitrarily tokenized texts.

#大语言模型#GPT3 encoder & decoder tool written in Swift

Kotlin multiplatform BPE tokenizer library for OpenAI models

BBPE 底层实现

High performance unsupervised text tokenization for Ruby

Learning BPE embeddings by first learning a segmentation model and then training word2vec

Sentiment-based classification for stock article title using PhoBert

✂️ OpenAI's tiktoken tokenizer written in Go