|
Gyoji Compiler
|
Represents a token read from the input stream. More...
#include <token-stream.hpp>
Public Member Functions | |
| Token (TokenID _typestr, std::string _value, const std::string &_filename, size_t _line, size_t _column, size_t _length) | |
| ~Token () | |
| const TokenID & | get_type () const |
| const std::string & | get_value () const |
Represents a token read from the input stream.
A token represents the result of the lexical analysis and contains the line number and column number of where the token was found as well as the type of token and the literal value of the token from the input. Some tokens participate in the grammar such as language keywords while others represent whitespace or comments.
While it might seem that this belongs in the lex/parse code of the front-end, this token is less concrete in the sense that it is actually agnostic of the way the data is broken up and parsed. This token stream can actually be re-used for multiple different lexical and syntactical structures.
It also carries the source location where the token was read from so that its source-reference can be used to derive and report the location of any errors. It is declared in the compiler context because the token stream must have a lifetime at least as long as the error reporting system because the errors hold references to the tokens and their source references.
| Token::Token | ( | TokenID | _typestr, |
| std::string | _value, | ||
| const std::string & | _filename, | ||
| size_t | _line, | ||
| size_t | _column, | ||
| size_t | _length | ||
| ) |
Creates a new token of the given type and value with the line number and column number where it was found in the input stream.
| Token::~Token | ( | ) |
Destructor, nothing fancy.
| const TokenID & Token::get_type | ( | ) | const |
This returns the type of the token. In most cases, this will be the token symbol from the grammar definition but in some cases may refer to a whitespace or comment directly from the input and bypassing the grammar entirely.
| const std::string & Token::get_value | ( | ) | const |
This is the literal value that was found in the input stream and matched one of the lexical rules.
1.9.8