tcalc::token::Tokenizer Class Reference

Tokenize the input string into tokens. More...

#include <tokenizer.hpp>

Public Types
using	KeywordComp = std::greater< std::string_view >

Public Member Functions
	Tokenizer (std::string_view input)
	Construct a new Tokenizer object.
error::Result< Token >	next ()
	Get the next token.
TCALC_INLINE auto	pos () const noexcept
	Get the current position of the tokenizer.
TCALC_INLINE auto	spos () const noexcept
	Get the current position of the tokenizer as an index.

Static Public Attributes
static const std::map< std::string_view, TokenType, KeywordComp >	KEYWORDS
static constexpr char	QUOTE = '\''

Detailed Description

Tokenize the input string into tokens.

Member Typedef Documentation

KeywordComp

using tcalc::token::Tokenizer::KeywordComp = std::greater<std::string_view>

Comparator for keywords.

Constructor & Destructor Documentation

Tokenizer()

tcalc::token::Tokenizer::Tokenizer ( std::string_view  input )

inlineexplicit

Construct a new Tokenizer object.

Parameters

input

The input string.

Member Function Documentation

next()

error::Result< Token > tcalc::token::Tokenizer::next

(

)

Get the next token.

Returns

Token The next token.

pos()

TCALC_INLINE auto tcalc::token::Tokenizer::pos ( ) const

inlinenoexcept

Get the current position of the tokenizer.

Returns

std::string_view::const_iterator The current position.

spos()

TCALC_INLINE auto tcalc::token::Tokenizer::spos ( ) const

inlinenoexcept

Get the current position of the tokenizer as an index.

Returns

size_t The current position as an index.

Member Data Documentation

KEYWORDS

const std::map<std::string_view, TokenType, KeywordComp> tcalc::token::Tokenizer::KEYWORDS

inlinestatic

Initial value:

= {
      { "def", TokenType::DEF },         { "let", TokenType::LET },
      { "if", TokenType::IF },           { "then", TokenType::THEN },
      { "else", TokenType::ELSE },       { "import", TokenType::IMPORT },
      { "==", TokenType::EQUAL },        { "!=", TokenType::NOTEQUAL },
      { ">=", TokenType::GREATEREQUAL }, { "<=", TokenType::LESSEQUAL },
      { "&&", TokenType::AND },          { "||", TokenType::OR },
      { "+", TokenType::PLUS },          { "-", TokenType::MINUS },
      { "*", TokenType::MULTIPLY },      { "/", TokenType::DIVIDE },
      { "(", TokenType::LPAREN },        { ")", TokenType::RPAREN },
      { ",", TokenType::COMMA },         { ";", TokenType::SEMICOLON },
      { "=", TokenType::ASSIGN },        { ">", TokenType::GREATER },
      { "<", TokenType::LESS },          { "!", TokenType::NOT },
    }

Tcalc keywords, must be ordered by length.

QUOTE

constexpr char tcalc::token::Tokenizer::QUOTE = '\''

staticconstexpr

The quote character.

---

The documentation for this class was generated from the following files:

• include/tcalc/tokenizer.hpp
• src/tokenizer.cpp