A simple tokenizer for parsing text files. More...

Classes
class	BadMSVCSpecial
	While parsing a number of the form 1. More...

class	Settings
	Tokenizer configuration options. More...

class	TokenException

class	WrongString
	String read from input did not match expected string. More...

class	WrongSymbol

class	WrongTokenType
	Thrown by the read methods. More...

Public Types
enum	FS { FROM_STRING }

Public Member Functions
	TextInput (const String &filename, const Settings &settings=Settings())

	TextInput (FS fs, const String &str, const Settings &settings=Settings())
	Creates input directly from a string. More...

	TextInput (FS fs, const char *str, size_t strLen, const Settings &settings=Settings())
	Creates input directly from a fixed-length, non-nullptr terminated string. More...

const String &	filename () const
	Returns the filename from which this input is drawn, or the first few characters of the string if created from a string. More...

bool	hasMore ()
	Returns true while there are tokens remaining. More...

Token	peek ()
	Return a copy of the next token in the input stream, but don't remove it from the input stream. More...

int	peekCharacterNumber ()
	Returns the character number (relative to the line) for the next token in the input stream. More...

int	peekLineNumber ()
	Returns the line number for the next token. More...

void	popSettings ()

void	push (const Token &t)
	Take a previously read token and push it back at the front of the input stream. More...

void	pushSettings (const Settings &settings)
	Temporarily switch parsing to use settings. More...

Token	read ()
	Read the next token (which will be the END token if ! hasMore()). More...

void	read (Token &t)
	Avoids the copy of read() More...

bool	readBoolean ()

String	readComment ()
	Like readCommentToken, but returns the token's string. More...

void	readComment (const String &s)
	Reads a specific comment token or throws either WrongTokenType or WrongString. More...

Token	readCommentToken ()
	Reads a comment token or throws WrongTokenType, and returns the token. More...

int	readInteger ()
	Reads a number that must be in C integer format: `[ '+' \| '-' ] #+ \| '0x'#+` More...

String	readNewline ()
	Like readNewlineToken, but returns the token's string. More...

void	readNewline (const String &s)
	Reads a specific newline token or throws either WrongTokenType or WrongString. More...

Token	readNewlineToken ()
	Reads a newline token or throws WrongTokenType, and returns the token. More...

double	readNumber ()
	Read one token (or possibly two, for minus sign) as a number or throws WrongTokenType, and returns the number. More...

Token	readSignificant ()
	Calls read() until the result is not a newline or comment. More...

String	readString ()
	Like readStringToken, but returns the token's string. More...

void	readString (const String &s)
	Reads a specific string token or throws either WrongTokenType or WrongString. More...

Token	readStringToken ()
	Reads a string token or throws WrongTokenType, and returns the token. More...

String	readSymbol ()
	Like readSymbolToken, but returns the token's string. More...

void	readSymbol (const String &symbol)
	Reads a specific symbol token or throws either WrongTokenType or WrongSymbol. More...

void	readSymbols (const String &s1, const String &s2)
	Read a series of two specific symbols. More...

void	readSymbols (const String &s1, const String &s2, const String &s3)
	Read a series of three specific symbols. More...

void	readSymbols (const String &s1, const String &s2, const String &s3, const String &s4)
	Read a series of four specific symbols. More...

Token	readSymbolToken ()
	Reads a symbol token or throws WrongTokenType, and returns the token. More...

void	readSymbolToken (Token &t)
	Avoids the copy of readSymbolToken() More...

String	readUntilDelimiterAsString (const char delimiter1, const char delimiter2='\0')
	Read from the beginning of the next token until the following delimiter character and return the result as a string, ignoring all parsing in between. More...

String	readUntilNewlineAsString ()
	Read from the beginning of the next token until the following newline and return the result as a string, ignoring all parsing in between. More...

Static Public Member Functions
static bool	parseBoolean (const String &_string)
	toLower(_string) == "true" More...

static double	parseNumber (const String &_string)
	Includes MSVC specials parsing. More...

Detailed Description

A simple tokenizer for parsing text files.

TextInput handles a superset of C++,Java, Matlab, and Bash code text including single line comments, block comments, quoted strings with escape sequences, and operators. TextInput recognizes several categories of tokens, which are separated by white space, quotation marks, or the end of a recognized operator:

Token::SINGLE_QUOTED_TYPE string of characters surrounded by single quotes, e.g., 'x', '\0', 'foo'.
Token::DOUBLE_QUOTED_TYPE string of characters surrounded by double quotes, e.g., "x", "abc\txyz", "b o b".
Token::SYMBOL_TYPE legal C++ operators, keywords, and identifiers. e.g., >=, Foo, _X, class, {
Token::HEX_INTEGER_TYPE numbers in hexadecimal integer. e.g., 0x17F, -0x1
Token::INTEGER_TYPE numbers without decimal places or exponential notation, in decimal notation. e.g., 10, 32, 0, -155
Token::FLOATING_POINT_TYPE numbers with decimal places or exponential notation. e.g., 1e3, -1.2, .4, 0.5
Token::BOOLEAN_TYPE special symbols like "true" and "false"; the exact details can be configured in TextInput::Settings
Token::LINE_COMMENT_TYPE (disabled by default); generated for line comments as specified by TextInput::Settings
Token::BLOCK_COMMENT_TYPE (disabled by default); generated for c-style block comments as specified by TextInput::Settings
Token::NEWLINE_TYPE (disabled by default); generated for any of "\\r", "\\n" or "\\r\\n"

The special ".." and "..." tokens are always recognized in addition to normal C++ operators. Additional tokens can be made available by changing the Settings.

Negative numbers are handled specially because of the ambiguity between unary minus and negative numbers– see the note on TextInput::read.

TextInput does not have helper functions for types with non-obvious formatting, or helpers that would be redundant. Use the serialize methods instead for parsing specific types like int, Vector3, and Color3.

Inside quoted strings escape sequences are converted. Thus the string token for ["a\\nb"] is 'a', followed by a newline, followed by 'b'. Outside of quoted strings, escape sequences are not converted, so the token sequence for [a\nb] is symbol 'a', symbol '\', symbol 'nb' (this matches what a C++ parser would do). The exception is that a specified TextInput::Settings::otherCommentCharacter preceeded by a backslash is assumed to be an escaped comment character and is returned as a symbol token instead of being parsed as a comment (this is what a LaTex or VRML parser would do).

Examples

TextInput ti(TextInput::FROM_STRING, "name = \"Max\", height = 6");
Token t;
t = ti.read(); 
debugAssert(t.type == Token::SYMBOL);
debugAssert(t.sval == "name");
ti.read();
debugAssert(t.type == Token::SYMBOL);
debugAssert(t.sval == "=");
String name = ti.read().sval;
ti.read();

TextInput ti(TextInput::FROM_STRING, "name = \"Max\", height = 6");
ti.readSymbols("name", "=");
String name = ti.readString();
ti.readSymbols(",", "height", "=");
double height = ti. readNumber();

Assumes that the file is not modified once opened.

Member Enumeration Documentation

◆ FS

enum G3D::TextInput::FS

Enumerator
FROM_STRING

Constructor & Destructor Documentation

◆ TextInput() [1/3]

G3D::TextInput::TextInput	(	const String &	filename,
		const Settings &	settings = `Settings()`
	)

◆ TextInput() [2/3]

G3D::TextInput::TextInput	(	FS	fs,
		const String &	str,
		const Settings &	settings = `Settings()`
	)

Creates input directly from a string.

The first argument must be TextInput::FROM_STRING.

◆ TextInput() [3/3]

G3D::TextInput::TextInput	(	FS	fs,
		const char *	str,
		size_t	strLen,
		const Settings &	settings = `Settings()`
	)

Creates input directly from a fixed-length, non-nullptr terminated string.

The first argument must be TextInput::FROM_STRING.

Member Function Documentation

◆ filename()

const String& G3D::TextInput::filename ( ) const

Returns the filename from which this input is drawn, or the first few characters of the string if created from a string.

If settings::filename is non-empty that will replace the true filename.

◆ hasMore()

bool G3D::TextInput::hasMore ( )

Returns true while there are tokens remaining.

◆ parseBoolean()

static bool G3D::TextInput::parseBoolean ( const String & _string )

static

toLower(_string) == "true"

◆ parseNumber()

static double G3D::TextInput::parseNumber ( const String & _string )

static

Includes MSVC specials parsing.

◆ peek()

Token G3D::TextInput::peek ( )

Return a copy of the next token in the input stream, but don't remove it from the input stream.

◆ peekCharacterNumber()

int G3D::TextInput::peekCharacterNumber ( )

Returns the character number (relative to the line) for the next token in the input stream.

◆ peekLineNumber()

int G3D::TextInput::peekLineNumber ( )

Returns the line number for the next token.

◆ popSettings()

void G3D::TextInput::popSettings ( )

inline

◆ push()

void G3D::TextInput::push ( const Token & t )

Take a previously read token and push it back at the front of the input stream.

Can be used in the case where more than one token of read-ahead is needed (i.e., when peek doesn't suffice).

◆ pushSettings()

void G3D::TextInput::pushSettings ( const Settings & settings )

inline

Temporarily switch parsing to use settings.

Note that this will override the currently recorded sourceFilename unless you explicitly set it back.

See also: popSettings

◆ read() [1/2]

Token G3D::TextInput::read ( )

Read the next token (which will be the END token if ! hasMore()).

Signed numbers can be handled in one of two modes. If the option TextInput::Settings::signedNumbers is true, A '+' or '-' immediately before a number is prepended onto that number and if there is intervening whitespace, it is read as a separate symbol.

If TextInput::Settings::signedNumbers is false, read() does not distinguish between a plus or minus symbol next to a number and a positive/negative number itself. For example, "x - 1" and "x -1" will be parsed the same way by read().
In both cases, readNumber() will contract a leading "-" or "+" onto a number.

◆ read() [2/2]

void G3D::TextInput::read ( Token & t )

Avoids the copy of read()

◆ readBoolean()

bool G3D::TextInput::readBoolean ( )

◆ readComment() [1/2]

String G3D::TextInput::readComment ( )

Like readCommentToken, but returns the token's string.

Use this method (rather than readCommentToken) if you want the token's value but don't really care about its location in the input. Use of readCommentToken is encouraged for better error reporting.

◆ readComment() [2/2]

void G3D::TextInput::readComment ( const String & s )

Reads a specific comment token or throws either WrongTokenType or WrongString.

If the next token in the input is a comment matching s, it will be consumed.

Use this method if you want to match a specific comment from the input. In that case, typically error reporting related to the token is only going to occur because of a mismatch, so no location information is needed by the caller.

WrongTokenType will be thrown if the next token in the input stream is not a comment. WrongString will be thrown if the next token in the input stream is a comment but does not match the s parameter. When an exception is thrown, no tokens are consumed.

◆ readCommentToken()

Token G3D::TextInput::readCommentToken ( )

Reads a comment token or throws WrongTokenType, and returns the token.

Use this method (rather than readComment) if you want the token's location as well as its value.

WrongTokenType will be thrown if the next token in the input stream is not a comment. When an exception is thrown, no tokens are consumed.

◆ readInteger()

int G3D::TextInput::readInteger ( )

Reads a number that must be in C integer format: [ '+' | '-' ] #+ | '0x'#+

◆ readNewline() [1/2]

String G3D::TextInput::readNewline ( )

Like readNewlineToken, but returns the token's string.

Use this method (rather than readNewlineToken) if you want the token's value but don't really care about its location in the input. Use of readNewlineToken is encouraged for better error reporting.

◆ readNewline() [2/2]

void G3D::TextInput::readNewline ( const String & s )

Reads a specific newline token or throws either WrongTokenType or WrongString.

If the next token in the input is a newline matching s, it will be consumed.

Use this method if you want to match a specific newline from the input. In that case, typically error reporting related to the token is only going to occur because of a mismatch, so no location information is needed by the caller.

WrongTokenType will be thrown if the next token in the input stream is not a newline. WrongString will be thrown if the next token in the input stream is a newlin but does not match the s parameter. When an exception is thrown, no tokens are consumed.

◆ readNewlineToken()

Token G3D::TextInput::readNewlineToken ( )

Reads a newline token or throws WrongTokenType, and returns the token.

Use this method (rather than readNewline) if you want the token's location as well as its value.

WrongTokenType will be thrown if the next token in the input stream is not a newline. When an exception is thrown, no tokens are consumed.

◆ readNumber()

double G3D::TextInput::readNumber ( )

Read one token (or possibly two, for minus sign) as a number or throws WrongTokenType, and returns the number.

If the first token in the input is a number, it is returned directly.

If TextInput::Settings::signedNumbers is false and the input stream contains a '+' or '-' symbol token immediately followed by a number token, both tokens will be consumed and a single token will be returned by this method.

WrongTokenType will be thrown if one of the input conditions described above is not satisfied. When an exception is thrown, no tokens are consumed.

◆ readSignificant()

Token G3D::TextInput::readSignificant ( )

Calls read() until the result is not a newline or comment.

◆ readString() [1/2]

String G3D::TextInput::readString ( )

Like readStringToken, but returns the token's string.

Use this method (rather than readStringToken) if you want the token's value but don't really care about its location in the input. Use of readStringToken is encouraged for better error reporting.

◆ readString() [2/2]

void G3D::TextInput::readString ( const String & s )

Reads a specific string token or throws either WrongTokenType or WrongString.

If the next token in the input is a string matching s, it will be consumed.

Use this method if you want to match a specific string from the input. In that case, typically error reporting related to the token is only going to occur because of a mismatch, so no location information is needed by the caller.

WrongTokenType will be thrown if the next token in the input stream is not a string. WrongString will be thrown if the next token in the input stream is a string but does not match the s parameter. When an exception is thrown, no tokens are consumed.

See also: readString(), readStringToken(), readUntilNewlineAsString(), readUntilDelimiterAsString()

◆ readStringToken()

Token G3D::TextInput::readStringToken ( )

Reads a string token or throws WrongTokenType, and returns the token.

Use this method (rather than readString) if you want the token's location as well as its value.

WrongTokenType will be thrown if the next token in the input stream is not a string. When an exception is thrown, no tokens are consumed.

◆ readSymbol() [1/2]

String G3D::TextInput::readSymbol ( )

Like readSymbolToken, but returns the token's string.

Use this method (rather than readSymbolToken) if you want the token's value but don't really care about its location in the input. Use of readSymbolToken is encouraged for better error reporting.

Referenced by readSymbols().

◆ readSymbol() [2/2]

void G3D::TextInput::readSymbol ( const String & symbol )

Reads a specific symbol token or throws either WrongTokenType or WrongSymbol.

If the next token in the input is a symbol matching symbol, it will be consumed.

Use this method if you want to match a specific symbol from the input. In that case, typically error reporting related to the token is only going to occur because of a mismatch, so no location information is needed by the caller.

WrongTokenType will be thrown if the next token in the input stream is not a symbol. WrongSymbol will be thrown if the next token in the input stream is a symbol but does not match the symbol parameter. When an exception is thrown, no tokens are consumed.

◆ readSymbols() [1/3]

void G3D::TextInput::readSymbols	(	const String &	s1,
		const String &	s2
	)

inline

Read a series of two specific symbols.

See readSymbol.

◆ readSymbols() [2/3]

void G3D::TextInput::readSymbols	(	const String &	s1,
		const String &	s2,
		const String &	s3
	)

inline

Read a series of three specific symbols.

See readSymbol.

◆ readSymbols() [3/3]

void G3D::TextInput::readSymbols	(	const String &	s1,
		const String &	s2,
		const String &	s3,
		const String &	s4
	)

inline

Read a series of four specific symbols.

See readSymbol.

◆ readSymbolToken() [1/2]

Token G3D::TextInput::readSymbolToken ( )

Reads a symbol token or throws WrongTokenType, and returns the token.

Use this method (rather than readSymbol) if you want the token's location as well as its value.

WrongTokenType will be thrown if the next token in the input stream is not a symbol. When an exception is thrown, no tokens are consumed.

◆ readSymbolToken() [2/2]

void G3D::TextInput::readSymbolToken ( Token & t )

Avoids the copy of readSymbolToken()

◆ readUntilDelimiterAsString()

String G3D::TextInput::readUntilDelimiterAsString	(	const char	delimiter1,
		const char	delimiter2 = `'\0'`
	)

Read from the beginning of the next token until the following delimiter character and return the result as a string, ignoring all parsing in between.

The delimiter is not returned in the string, and the following token read will begin at the delimiter or end of file token (if they are enabled for parsing).

◆ readUntilNewlineAsString()

String G3D::TextInput::readUntilNewlineAsString ( )

Read from the beginning of the next token until the following newline and return the result as a string, ignoring all parsing in between.

The newline is not returned in the string, and the following token read will be a newline or end of file token (if they are enabled for parsing).

Classes

Public Types

Public Member Functions

Static Public Member Functions

Detailed Description

Member Enumeration Documentation

◆ FS

Constructor & Destructor Documentation

◆ TextInput() [1/3]

◆ TextInput() [2/3]

◆ TextInput() [3/3]

Member Function Documentation

◆ filename()

◆ hasMore()

◆ parseBoolean()

◆ parseNumber()

◆ peek()

◆ peekCharacterNumber()

◆ peekLineNumber()

◆ popSettings()

◆ push()

◆ pushSettings()

◆ read() [1/2]

◆ read() [2/2]

◆ readBoolean()

◆ readComment() [1/2]

◆ readComment() [2/2]

◆ readCommentToken()

◆ readInteger()

◆ readNewline() [1/2]

◆ readNewline() [2/2]

◆ readNewlineToken()

◆ readNumber()

◆ readSignificant()

◆ readString() [1/2]

◆ readString() [2/2]

◆ readStringToken()

◆ readSymbol() [1/2]

◆ readSymbol() [2/2]

◆ readSymbols() [1/3]

◆ readSymbols() [2/3]

◆ readSymbols() [3/3]

◆ readSymbolToken() [1/2]

◆ readSymbolToken() [2/2]

◆ readUntilDelimiterAsString()

◆ readUntilNewlineAsString()