Simple-to-use scoring function for arbitrarily tokenized texts.
-
Updated
Jun 12, 2024 - Python
Simple-to-use scoring function for arbitrarily tokenized texts.
Tools and resources for the computational processing of Nheengatu (Modern Tupi)
Implementation of the LongRoPE: Extending LLM Context Window Beyond 2 Million Tokens Paper
A Python library for interacting with TI-(e)z80 (82/83/84 series) calculator files
Code for Zero-Shot Tokenizer Transfer
Basis Theory Developer Documentation
XFT's tokenized treasuries protocol.
Strukt Common Utilities
VGS Show - Android SDK that enables you to securely display sensitive data https://www.verygoodsecurity.com/docs/vgs-show
A package to download and preprocess a Wikipedia dump, in any language.
Contracts for Spiko's tokenized securities.
Rule engine used by the CMTAT token framework to implement transfer restriction.
Sudachi in Rust 🦀 and new generation of SudachiPy
Laboratory 4 - Retrieval Information
Laboratory 3 - Retrieval Information
Laboratory 2 - Retrieval Information
Laboratory 1 - Retrieval Information
Easy token price estimates for LLMs
Simple, reliable and well tested training code for quick experiments with transformer based models
Add a description, image, and links to the tokenization topic page so that developers can more easily learn about it.
To associate your repository with the tokenization topic, visit your repo's landing page and select "manage topics."