Skip to content

Easily scan a string with an object of regex patterns to produce an array of tokens. ~100 sloc.

Notifications You must be signed in to change notification settings

here-be/snapdragon-scanner

Repository files navigation

snapdragon-scanner NPM version NPM monthly downloads NPM total downloads Linux Build Status

Easily scan a string with an object of regex patterns to produce an array of tokens. ~100 sloc.

Please consider following this project's author, Jon Schlinkert, and consider starring the project to show your ❤️ and support.

Install

Install with npm:

$ npm install --save snapdragon-scanner

What is this?

This is a simple Lexical Scanner that takes an object of regex patterns, and uses those patterns to process an input string into an array of tokens.

What is the difference between this and snapdragon-lexer?

snapdragon-lexer uses registered handler functions to capture and handle tokens, snapdragon-scanner simply iterates over an object of regular expression patterns to create tokens. You can think of snapdragon-scanner as the "lite" version of snapdragon-lexer.

Usage

const Scanner = require('snapdragon-scanner');

API

Create a new Scanner with the given str and optional rules.

Params

  • input {String}: Input string to scan.
  • options {Object}: (optional) Pass an object of regex patterns on options.rules, or use .addRules() or .addRule() after instantiating.

Example

const Scanner = require('snapdragon-scanner');
const scanner = new Scanner('var foo = "bar";', {
  rules: {
    space: /^ +/,
    tab: /^\t+/,
    newline: /^\n+/,
    text: /^\w+/,
    equal: /^=/,
    quote: /^["']/,
    semi: /^;/,
    dot: /^\./
  }
});

Add a rule to the scanner.

Params

  • rule {String}
  • match {RegExp}: Match array from RegExp.exec().

Example

console.log(scanner.token('text', ['foo']);
//=> { rule: 'text', value: 'foo', match: [foo] };

Add a rule to the scanner.

Params

  • rule {String}
  • regex {RegExp}: Regular expression to use when scanning.

Example

scanner.addRule(rule, regex);
// example
scanner.addRule('text', /^\w+/);

Add an object of rules to the scanner.

Params

  • rules {Object}

Example

scanner.addRules({
  text: /^\w+/,
  slash: /^\//,
  dot: /^\./
});

Attempts to match scanner.string with the given regex. Also validates the regex to ensure that it starts with ^ since matching should always be against the beginning of the string, and throws if the regex matches an empty string, to avoid catastrophic backtracking.

Params

  • regex {RegExp}: (required)
  • returns {Array|null}: Returns the match array or null from RegExp.exec.

Example

const scanner = new Scanner('foo/bar', { text: /^\w+/ });
const match = scanner.match(scanner.rules.get('text'));
console.log(match);
//=> [ 'foo', index: 0, input: 'foo/bar', groups: undefined ]

Remove the given length of substring from scanner.string.

Params

  • len {Number}
  • value {String}: Optionally pass the value being consumed for minor performance improvement.
  • returns {String}: Returns the consumed value

Example

scanner.consume(1);
scanner.consume(1, '*');

Push a token onto the scanner.queue array.

Params

  • token {object}
  • returns {Object}: Returns the token.

Example

console.log(scanner.queue.length); // 0
scanner.enqueue({ rule: 'foo' });
console.log(scanner.queue.length); // 1

Shift a token from scanner.queue.

  • returns {Object}: Returns the first token in the scanner.queue.

Example

console.log(scanner.queue.length); // 0
scanner.enqueue({ rule: 'foo' });
console.log(scanner.queue.length); // 1
scanner.dequeue();
console.log(scanner.queue.length); // 0

Iterates over the registered regex patterns until a match is found, then returns a token from the match and regex rule.

  • returns {Object}: Returns a token with rule, value and match properties.

Example

const token = scanner.advance();
console.log(token) // { rule: 'text', value: 'foo' }

Lookahead n tokens and return the last token. Pushes any intermediate tokens onto scanner.tokens. To lookahead a single token, use .peek().

Params

  • n {number}
  • returns {Object}

Example

const token = scanner.lookahead(2);

Returns a token representing the next match, but without consuming the matched substring (e.g. the cursor position is not advanced).

  • returns {Object|undefined}: Returns a token, or undefined if no match was found.

Example

const token = scanner.peek();

Returns a token representing the next match, but without consuming the matched substring (e.g. the cursor position is not advanced).

  • returns {Object|undefined}: Returns a token, or undefined if no match was found.

Example

const token = scanner.peek();

Returns the next token and advances the cursor position.

  • returns {Object|undefined}: Returns a token, or undefined if no match was found.

Example

const token = scanner.scan();

Scan until the given fn does not return true.

Params

  • fn {Function}: Must return true to continue scanning.
  • returns {Array}: Returns an array if scanned tokens.

Example

scanner.scanWhile(tok => tok.rule !== 'space');

Returns true if the scanner has not consumed any of the input string.

  • returns {Boolean}

Returns true if scanner.string and scanner.queue are empty.

  • returns {Boolean}

Token objects

Scanner tokens are plain JavaScript objects with the following properties:

{
  type: String;
  value: String
  match: Array
}

Token properties

  • type {String} - The name of the regex that matched the substring.
  • value {String} - The substring that was captured by the regex.
  • match {Array} - The match array from RegExp.exec()

Release history

See the changelog.

About

Contributing

Pull requests and stars are always welcome. For bugs and feature requests, please create an issue.

Running Tests

Running and reviewing unit tests is a great way to get familiarized with a library and its API. You can install dependencies and run tests with the following command:

$ npm install && npm test
Building docs

(This project's readme.md is generated by verb, please don't edit the readme directly. Any changes to the readme must be made in the .verb.md readme template.)

To generate the readme, run the following command:

$ npm install -g verbose/verb#dev verb-generate-readme && verb

Related projects

You might also be interested in these projects:

Author

Jon Schlinkert

License

Copyright © 2018, Jon Schlinkert. Released under the MIT License.


This file was generated by verb-generate-readme, v0.8.0, on November 19, 2018.

About

Easily scan a string with an object of regex patterns to produce an array of tokens. ~100 sloc.

Topics

Resources

Stars

Watchers

Forks

Sponsor this project

  •  
  •  

Packages

No packages published