class Puppet::Parser::Lexer::TokenList

Maintain a list of tokens.

Attributes

regex_tokens[R]

string_tokens[R]

Public Class Methods

new() click to toggle source

# File lib/puppet/parser/lexer.rb, line 89
def initialize
  @tokens = {}
  @regex_tokens = []
  @string_tokens = []
  @tokens_by_string = {}
end

Public Instance Methods

add_token(name, regex, options = {}, &block) click to toggle source

Create a new token.

# File lib/puppet/parser/lexer.rb, line 73
def add_token(name, regex, options = {}, &block)
  raise(ArgumentError, "Token #{name} already exists") if @tokens.include?(name)
  token = Token.new(regex, name, options)
  @tokens[token.name] = token
  if token.string
    @string_tokens << token
    @tokens_by_string[token.string] = token
  else
    @regex_tokens << token
  end

  token.meta_def(:convert, &block) if block_given?

  token
end

add_tokens(hash) click to toggle source

Define more tokens.

# File lib/puppet/parser/lexer.rb, line 102
def add_tokens(hash)
  hash.each do |regex, name|
    add_token(name, regex)
  end
end

each() { |name, value| ... } click to toggle source

Yield each token name and value in turn.

# File lib/puppet/parser/lexer.rb, line 115
def each
  @tokens.each {|name, value| yield name, value }
end

lookup(string) click to toggle source

Look up a token by its value, rather than name.

# File lib/puppet/parser/lexer.rb, line 97
def lookup(string)
  @tokens_by_string[string]
end

sort_tokens() click to toggle source

Sort our tokens by length, so we know once we match, we’re done. This helps us avoid the O(n^2) nature of token matching.

# File lib/puppet/parser/lexer.rb, line 110
def sort_tokens
  @string_tokens.sort! { |a, b| b.string.length <=> a.string.length }
end