Skip to content
generated from openpeeps/pistachio

Generic tokenizer written in Nim language πŸ‘‘ Powered by std/lexbase and Nim's Macros

License

Notifications You must be signed in to change notification settings

openpeeps/toktok

Folders and files

NameName
Last commit message
Last commit date

Latest commit

Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 

Repository files navigation


Generic tokenizer written in Nim language, powered by Nim's Macros πŸ‘‘

nimble install toktok

API reference

Github Actions Github Actions

😍 Key Features

  • ✨ Powered by Nim's Macros
  • πŸͺ„ Based on std/lexbase / Zero Regular Expression
  • Compile-time generation using macro-based TokenKind enum, lexbase
  • Runtime generation using TokenKind tables, lexbase
  • Open Source | MIT

Note

This is a generic Lexer, based on std/ streams, lexbase and macros. It is meant to be used by higher level parsers for writing any kind of tools or programs.

Note

Compile with -d:toktokdebug to inspect the generated code.

Quick Example

# Register your custom handlers
handlers:
  proc handleImport(lex: var Lexer, kind: TokenKind) =
    # tokenize `import x, y, z`
    lex.kind = kind

# Register your tokens
registerTokens defaultSettings:
  `const` = "const"
  `echo` = "hello"
  asgn = '=':   # `=` is tkAsgn
    eq = '='    # `==` is tkEQ
  excl = '!':
    ne = '='
  at = '@':
    import = tokenizer(handleImport, "import") 


# Tokenizing...
var
  tok = lexer.init(sample)
  prev: TokenTuple
  curr: TokenTuple = tok.getToken
  next: TokenTuple = tok.getToken 

proc walk(tok: var Lexer) =
  prev = curr
  curr = next
  next = tok.getToken

while likely(curr.kind != tkEOF):
  if tok.hasError: break
  echo curr # use `getToken` consumer to get token by token
  walk tok

TODO

  • Runtime Token generation using tables/critbits

❀ Contributions & Support

🎩 License

MIT license. Made by Humans from OpenPeeps.
Copyright Β© 2023 OpenPeeps & Contributors β€” All rights reserved.