Public Member Functions
boolean	atEnd ()

void	consume (final String token) throws ParseException

boolean	consumeBoolean () throws ParseException

ByteString	consumeByteString () throws ParseException

double	consumeDouble () throws ParseException

float	consumeFloat () throws ParseException

String	consumeIdentifier () throws ParseException

int	consumeInt32 () throws ParseException

long	consumeInt64 () throws ParseException

String	consumeString () throws ParseException

int	consumeUInt32 () throws ParseException

long	consumeUInt64 () throws ParseException

boolean	lookingAt (String text)

boolean	lookingAtInteger ()

void	nextToken ()

ParseException	parseException (final String description)

ParseException	parseExceptionPreviousToken (final String description)

boolean	tryConsume (final String token)

boolean	tryConsumeDouble ()

boolean	tryConsumeFloat ()

boolean	tryConsumeIdentifier ()

boolean	tryConsumeInt64 ()

boolean	tryConsumeString ()

boolean	tryConsumeUInt64 ()

UnknownFieldParseException	unknownFieldParseExceptionPreviousToken (final String unknownField, final String description)

Private Member Functions
void	consumeByteString (List< ByteString > list) throws ParseException

ParseException	floatParseException (final NumberFormatException e)

ParseException	integerParseException (final NumberFormatException e)

void	skipWhitespace ()

	Tokenizer (final CharSequence text)

Private Attributes
int	column = 0

String	currentToken

int	line = 0

final Matcher	matcher

int	pos = 0

int	previousColumn = 0

int	previousLine = 0

final CharSequence	text

Static Private Attributes
static final Pattern	DOUBLE_INFINITY

static final Pattern	FLOAT_INFINITY

static final Pattern	FLOAT_NAN = Pattern.compile("nanf?", Pattern.CASE_INSENSITIVE)

static final Pattern	TOKEN

static final Pattern	WHITESPACE = Pattern.compile("(\\s\|(#.*$))++", Pattern.MULTILINE)

Detailed Description

Represents a stream of tokens parsed from a

String

.

The Java standard library provides many classes that you might think would be useful for implementing this, but aren't. For example:

java.io.StreamTokenizer

: This almost does what we want – or, at least, something that would get us close to what we want – except for one fatal flaw: It automatically un-escapes strings using Java escape sequences, which do not include all the escape sequences we need to support (e.g. '\x').
java.util.Scanner

: This seems like a great way at least to parse regular expressions out of a stream (so we wouldn't have to load the entire input into a single string before parsing). Sadly,
Scanner

requires that tokens be delimited with some delimiter. Thus, although the text "foo:" should parse to two tokens ("foo" and ":"),
Scanner

would recognize it only as a single token. Furthermore,
Scanner

provides no way to inspect the contents of delimiters, making it impossible to keep track of line and column numbers.

Luckily, Java's regular expression support does manage to be useful to us. (Barely: We need

Matcher.usePattern()

, which is new in Java 1.5.) So, we can use that, at least. Unfortunately, this implies that we need to have the entire input in one contiguous string.

Definition at line 741 of file TextFormat.java.

Constructor & Destructor Documentation

◆ Tokenizer()

com.google.protobuf.TextFormat.Tokenizer.Tokenizer ( final CharSequence text )

inlineprivate

Construct a tokenizer that parses tokens from the given text.

Definition at line 776 of file TextFormat.java.

Member Function Documentation

◆ atEnd()

boolean com.google.protobuf.TextFormat.Tokenizer.atEnd ( )

inline

Are we at the end of the input?

Definition at line 800 of file TextFormat.java.

◆ consume()

void com.google.protobuf.TextFormat.Tokenizer.consume ( final String token ) throws ParseException

inline

If the next token exactly matches

token

, consume it. Otherwise, throw a {}.

Definition at line 864 of file TextFormat.java.

◆ consumeBoolean()

boolean com.google.protobuf.TextFormat.Tokenizer.consumeBoolean ( ) throws ParseException

inline

If the next token is a boolean, consume it and return its value. Otherwise, throw a {}.

Definition at line 1083 of file TextFormat.java.

◆ consumeByteString() [1/2]

ByteString com.google.protobuf.TextFormat.Tokenizer.consumeByteString ( ) throws ParseException

inline

If the next token is a string, consume it, unescape it as a ByteString, and return it. Otherwise, throw a ParseException.

Definition at line 1123 of file TextFormat.java.

◆ consumeByteString() [2/2]

void com.google.protobuf.TextFormat.Tokenizer.consumeByteString ( List< ByteString > list ) throws ParseException

inlineprivate

Like consumeByteString() but adds each token of the string to the given list. String literals (whether bytes or text) may come in multiple adjacent tokens which are automatically concatenated, like in C or Python.

Definition at line 1137 of file TextFormat.java.

◆ consumeDouble()

double com.google.protobuf.TextFormat.Tokenizer.consumeDouble ( ) throws ParseException

inline

If the next token is a double, consume it and return its value. Otherwise, throw a {}.

Definition at line 1007 of file TextFormat.java.

◆ consumeFloat()

float com.google.protobuf.TextFormat.Tokenizer.consumeFloat ( ) throws ParseException

inline

If the next token is a float, consume it and return its value. Otherwise, throw a {}.

Definition at line 1045 of file TextFormat.java.

◆ consumeIdentifier()

String com.google.protobuf.TextFormat.Tokenizer.consumeIdentifier ( ) throws ParseException

inline

If the next token is an identifier, consume it and return its value. Otherwise, throw a ParseException.

Definition at line 889 of file TextFormat.java.

◆ consumeInt32()

int com.google.protobuf.TextFormat.Tokenizer.consumeInt32 ( ) throws ParseException

inline

If the next token is a 32-bit signed integer, consume it and return its value. Otherwise, throw a ParseException.

Definition at line 925 of file TextFormat.java.

◆ consumeInt64()

long com.google.protobuf.TextFormat.Tokenizer.consumeInt64 ( ) throws ParseException

inline

If the next token is a 64-bit signed integer, consume it and return its value. Otherwise, throw a ParseException.

Definition at line 953 of file TextFormat.java.

◆ consumeString()

String com.google.protobuf.TextFormat.Tokenizer.consumeString ( ) throws ParseException

inline

If the next token is a string, consume it and return its (unescaped) value. Otherwise, throw a ParseException.

Definition at line 1105 of file TextFormat.java.

◆ consumeUInt32()

int com.google.protobuf.TextFormat.Tokenizer.consumeUInt32 ( ) throws ParseException

inline

If the next token is a 32-bit unsigned integer, consume it and return its value. Otherwise, throw a ParseException.

Definition at line 939 of file TextFormat.java.

◆ consumeUInt64()

long com.google.protobuf.TextFormat.Tokenizer.consumeUInt64 ( ) throws ParseException

inline

If the next token is a 64-bit unsigned integer, consume it and return its value. Otherwise, throw a ParseException.

Definition at line 980 of file TextFormat.java.

◆ floatParseException()

ParseException com.google.protobuf.TextFormat.Tokenizer.floatParseException ( final NumberFormatException e )

inlineprivate

Constructs an appropriate ParseException for the given

NumberFormatException

when trying to parse a float or double.

Definition at line 1187 of file TextFormat.java.

◆ integerParseException()

ParseException com.google.protobuf.TextFormat.Tokenizer.integerParseException ( final NumberFormatException e )

inlineprivate

Constructs an appropriate ParseException for the given

NumberFormatException

when trying to parse an integer.

Definition at line 1179 of file TextFormat.java.

◆ lookingAt()

boolean com.google.protobuf.TextFormat.Tokenizer.lookingAt ( String text )

inline

Returns

true

if the current token's text is equal to that specified.

Definition at line 881 of file TextFormat.java.

◆ lookingAtInteger()

boolean com.google.protobuf.TextFormat.Tokenizer.lookingAtInteger ( )

inline

Returns

true

if the next token is an integer, but does not consume it.

Definition at line 871 of file TextFormat.java.

◆ nextToken()

void com.google.protobuf.TextFormat.Tokenizer.nextToken ( )

inline

Advance to the next token.

Definition at line 805 of file TextFormat.java.

◆ parseException()

ParseException com.google.protobuf.TextFormat.Tokenizer.parseException ( final String description )

inline

Returns a ParseException with the current line and column numbers in the description, suitable for throwing.

Definition at line 1161 of file TextFormat.java.

◆ parseExceptionPreviousToken()

ParseException com.google.protobuf.TextFormat.Tokenizer.parseExceptionPreviousToken ( final String description )

inline

Returns a ParseException with the line and column numbers of the previous token in the description, suitable for throwing.

Definition at line 1170 of file TextFormat.java.

◆ skipWhitespace()

void com.google.protobuf.TextFormat.Tokenizer.skipWhitespace ( )

inlineprivate

Skip over any whitespace so that the matcher region starts at the next token.

Definition at line 840 of file TextFormat.java.

◆ tryConsume()

boolean com.google.protobuf.TextFormat.Tokenizer.tryConsume ( final String token )

inline

If the next token exactly matches

token

, consume it and return

true

. Otherwise, return

false

without doing anything.

Definition at line 851 of file TextFormat.java.

◆ tryConsumeDouble()

boolean com.google.protobuf.TextFormat.Tokenizer.tryConsumeDouble ( )

inline

If the next token is a double, consume it and return

true

. Otherwise, return

false

without doing anything.

Definition at line 1032 of file TextFormat.java.

◆ tryConsumeFloat()

boolean com.google.protobuf.TextFormat.Tokenizer.tryConsumeFloat ( )

inline

If the next token is a float, consume it and return

true

. Otherwise, return

false

without doing anything.

Definition at line 1070 of file TextFormat.java.

◆ tryConsumeIdentifier()

boolean com.google.protobuf.TextFormat.Tokenizer.tryConsumeIdentifier ( )

inline

If the next token is an identifier, consume it and return

true

. Otherwise, return

false

without doing anything.

Definition at line 912 of file TextFormat.java.

◆ tryConsumeInt64()

boolean com.google.protobuf.TextFormat.Tokenizer.tryConsumeInt64 ( )

inline

If the next token is a 64-bit signed integer, consume it and return

true

. Otherwise, return

false

without doing anything.

Definition at line 967 of file TextFormat.java.

◆ tryConsumeString()

boolean com.google.protobuf.TextFormat.Tokenizer.tryConsumeString ( )

inline

If the next token is a string, consume it and return true. Otherwise, return false.

Definition at line 1110 of file TextFormat.java.

◆ tryConsumeUInt64()

boolean com.google.protobuf.TextFormat.Tokenizer.tryConsumeUInt64 ( )

inline

If the next token is a 64-bit unsigned integer, consume it and return

true

. Otherwise, return

false

without doing anything.

Definition at line 994 of file TextFormat.java.

◆ unknownFieldParseExceptionPreviousToken()

UnknownFieldParseException com.google.protobuf.TextFormat.Tokenizer.unknownFieldParseExceptionPreviousToken	(	final String	unknownField,
		final String	description
	)

inline

Returns a UnknownFieldParseException with the line and column numbers of the previous token in the description, and the unknown field name, suitable for throwing.

Definition at line 1195 of file TextFormat.java.

Member Data Documentation

◆ column

int com.google.protobuf.TextFormat.Tokenizer.column = 0

private

Definition at line 751 of file TextFormat.java.

◆ currentToken

String com.google.protobuf.TextFormat.Tokenizer.currentToken

private

Definition at line 744 of file TextFormat.java.

◆ DOUBLE_INFINITY

final Pattern com.google.protobuf.TextFormat.Tokenizer.DOUBLE_INFINITY

staticprivate

Initial value:

=

Pattern.compile("-?inf(inity)?", Pattern.CASE_INSENSITIVE)

Definition at line 769 of file TextFormat.java.

◆ FLOAT_INFINITY

final Pattern com.google.protobuf.TextFormat.Tokenizer.FLOAT_INFINITY

staticprivate

Initial value:

=

Pattern.compile("-?inf(inity)?f?", Pattern.CASE_INSENSITIVE)

Definition at line 771 of file TextFormat.java.

◆ FLOAT_NAN

final Pattern com.google.protobuf.TextFormat.Tokenizer.FLOAT_NAN = Pattern.compile("nanf?", Pattern.CASE_INSENSITIVE)

staticprivate

Definition at line 773 of file TextFormat.java.

◆ line

int com.google.protobuf.TextFormat.Tokenizer.line = 0

private

Definition at line 750 of file TextFormat.java.

◆ matcher

final Matcher com.google.protobuf.TextFormat.Tokenizer.matcher

private

Definition at line 743 of file TextFormat.java.

◆ pos

int com.google.protobuf.TextFormat.Tokenizer.pos = 0

private

Definition at line 747 of file TextFormat.java.

◆ previousColumn

int com.google.protobuf.TextFormat.Tokenizer.previousColumn = 0

private

Definition at line 756 of file TextFormat.java.

◆ previousLine

int com.google.protobuf.TextFormat.Tokenizer.previousLine = 0

private

Definition at line 755 of file TextFormat.java.

◆ text

final CharSequence com.google.protobuf.TextFormat.Tokenizer.text

private

Definition at line 742 of file TextFormat.java.

◆ TOKEN

final Pattern com.google.protobuf.TextFormat.Tokenizer.TOKEN

staticprivate

Initial value:

=
        Pattern.compile(
            "[a-zA-Z_][0-9a-zA-Z_+-]*+|" 
                + "[.]?[0-9+-][0-9a-zA-Z_.+-]*+|" 
                + "\"([^\"\n\\\\]|\\\\.)*+(\"|\\\\?$)|" 
                + "\'([^\'\n\\\\]|\\\\.)*+(\'|\\\\?$)", 
            Pattern.MULTILINE)

Definition at line 761 of file TextFormat.java.

◆ WHITESPACE

final Pattern com.google.protobuf.TextFormat.Tokenizer.WHITESPACE = Pattern.compile("(\\s|(#.*$))++", Pattern.MULTILINE)

staticprivate

Definition at line 760 of file TextFormat.java.

The documentation for this class was generated from the following file:

TextFormat.java

Public Member Functions

Private Member Functions

Private Attributes

Static Private Attributes

Detailed Description

Constructor & Destructor Documentation

◆ Tokenizer()

Member Function Documentation

◆ atEnd()

◆ consume()

◆ consumeBoolean()

◆ consumeByteString() [1/2]

◆ consumeByteString() [2/2]

◆ consumeDouble()

◆ consumeFloat()

◆ consumeIdentifier()

◆ consumeInt32()

◆ consumeInt64()

◆ consumeString()

◆ consumeUInt32()

◆ consumeUInt64()

◆ floatParseException()

◆ integerParseException()

◆ lookingAt()

◆ lookingAtInteger()

◆ nextToken()

◆ parseException()

◆ parseExceptionPreviousToken()

◆ skipWhitespace()

◆ tryConsume()

◆ tryConsumeDouble()

◆ tryConsumeFloat()

◆ tryConsumeIdentifier()

◆ tryConsumeInt64()

◆ tryConsumeString()

◆ tryConsumeUInt64()

◆ unknownFieldParseExceptionPreviousToken()

Member Data Documentation

◆ column

◆ currentToken

◆ DOUBLE_INFINITY

◆ FLOAT_INFINITY

◆ FLOAT_NAN

◆ line

◆ matcher

◆ pos

◆ previousColumn

◆ previousLine

◆ text

◆ TOKEN

◆ WHITESPACE