Package org.stringtemplate.v4.compiler
Class STLexer
java.lang.Object
org.stringtemplate.v4.compiler.STLexer
- All Implemented Interfaces:
org.antlr.runtime.TokenSource
This class represents the tokenizer for templates. It operates in two modes:
inside and outside of expressions. It implements the
TokenSource
interface so it can be used with ANTLR parsers. Outside of expressions, we
can return these token types: TEXT
, INDENT
, LDELIM
(start of expression), RCURLY
(end of subtemplate), and
NEWLINE
. Inside of an expression, this lexer returns all of the
tokens needed by STParser
. From the parser's point of view, it can
treat a template as a simple stream of elements.
This class defines the token types and communicates these values to
STParser.g
via STLexer.tokens
file (which must remain
consistent).
-
Nested Class Summary
Nested ClassesModifier and TypeClassDescriptionstatic class
We buildSTToken
tokens instead of relying onCommonToken
so we can overrideSTLexer.STToken.toString()
. -
Field Summary
FieldsModifier and TypeFieldDescriptionstatic final int
static final int
static final int
(package private) char
current characterstatic final int
static final int
static final int
(package private) char
The char which delimits the start of an expression.(package private) char
The char which delimits the end of an expression.static final int
static final int
static final int
static final int
static final int
static final char
static final int
static final int
(package private) ErrorManager
static final int
static final int
static final int
static final int
(package private) org.antlr.runtime.CharStream
static final int
static final int
static final int
static final int
static final int
static final int
static final int
static final int
static final int
static final int
static final int
static final int
(package private) boolean
This keeps track of the current mode of the lexer.static final int
static final org.antlr.runtime.Token
static final int
(package private) int
When we started token, track initial coordinates so we can properly build token objects.(package private) int
(package private) int
static final int
int
To be able to properly track the inside/outside mode, we need to track how deeply nested we are in some templates.static final int
(package private) org.antlr.runtime.Token
template embedded in a group file? this is the templatestatic final int
(package private) List
<org.antlr.runtime.Token> Our lexer routines might have to emit more than a single token.static final int
-
Constructor Summary
ConstructorsConstructorDescriptionSTLexer
(org.antlr.runtime.CharStream input) STLexer
(ErrorManager errMgr, org.antlr.runtime.CharStream input, org.antlr.runtime.Token templateToken) STLexer
(ErrorManager errMgr, org.antlr.runtime.CharStream input, org.antlr.runtime.Token templateToken, char delimiterStartChar, char delimiterStopChar) -
Method Summary
Modifier and TypeMethodDescriptionorg.antlr.runtime.Token
(package private) org.antlr.runtime.Token
COMMENT()
protected void
consume()
void
emit
(org.antlr.runtime.Token token) (package private) org.antlr.runtime.Token
ESCAPE()
protected org.antlr.runtime.Token
inside()
static boolean
isIDLetter
(char c) static boolean
isIDStartLetter
(char c) static boolean
isUnicodeLetter
(char c) static boolean
isWS
(char c) (package private) void
void
match
(char x) Consume ifx
is next character on the input stream.(package private) org.antlr.runtime.Token
mID()
ID : ('a'..'z'|'A'..'Z'|'_'|'/') ('a'..'z'|'A'..'Z'|'0'..'9'|'_'|'/')* ;(package private) org.antlr.runtime.Token
mSTRING()
STRING : '"' ( '\\' '"' | '\\' ~'"' | ~('\\'|'"') )* '"' ;(package private) org.antlr.runtime.Token
mTEXT()
org.antlr.runtime.Token
newToken
(int ttype) org.antlr.runtime.Token
org.antlr.runtime.Token
org.antlr.runtime.Token
newTokenFromPreviousChar
(int ttype) org.antlr.runtime.Token
protected org.antlr.runtime.Token
outside()
static String
str
(int c) (package private) org.antlr.runtime.Token
(package private) org.antlr.runtime.Token
UNICODE()
(package private) void
WS()
-
Field Details
-
EOF
public static final char EOF- See Also:
-
EOF_TYPE
public static final int EOF_TYPE- See Also:
-
SKIP
public static final org.antlr.runtime.Token SKIP -
RBRACK
public static final int RBRACK- See Also:
-
LBRACK
public static final int LBRACK- See Also:
-
ELSE
public static final int ELSE- See Also:
-
ELLIPSIS
public static final int ELLIPSIS- See Also:
-
LCURLY
public static final int LCURLY- See Also:
-
BANG
public static final int BANG- See Also:
-
EQUALS
public static final int EQUALS- See Also:
-
TEXT
public static final int TEXT- See Also:
-
ID
public static final int ID- See Also:
-
SEMI
public static final int SEMI- See Also:
-
LPAREN
public static final int LPAREN- See Also:
-
IF
public static final int IF- See Also:
-
ELSEIF
public static final int ELSEIF- See Also:
-
COLON
public static final int COLON- See Also:
-
RPAREN
public static final int RPAREN- See Also:
-
COMMA
public static final int COMMA- See Also:
-
RCURLY
public static final int RCURLY- See Also:
-
ENDIF
public static final int ENDIF- See Also:
-
RDELIM
public static final int RDELIM- See Also:
-
SUPER
public static final int SUPER- See Also:
-
DOT
public static final int DOT- See Also:
-
LDELIM
public static final int LDELIM- See Also:
-
STRING
public static final int STRING- See Also:
-
PIPE
public static final int PIPE- See Also:
-
OR
public static final int OR- See Also:
-
AND
public static final int AND- See Also:
-
INDENT
public static final int INDENT- See Also:
-
NEWLINE
public static final int NEWLINE- See Also:
-
AT
public static final int AT- See Also:
-
REGION_END
public static final int REGION_END- See Also:
-
TRUE
public static final int TRUE- See Also:
-
FALSE
public static final int FALSE- See Also:
-
COMMENT
public static final int COMMENT- See Also:
-
SLASH
public static final int SLASH- See Also:
-
delimiterStartChar
char delimiterStartCharThe char which delimits the start of an expression. -
delimiterStopChar
char delimiterStopCharThe char which delimits the end of an expression. -
scanningInsideExpr
boolean scanningInsideExprThis keeps track of the current mode of the lexer. Are we inside or outside an ST expression? -
subtemplateDepth
public int subtemplateDepthTo be able to properly track the inside/outside mode, we need to track how deeply nested we are in some templates. Otherwise, we know whether a'}'
and the outermost subtemplate to send this back to outside mode. -
errMgr
ErrorManager errMgr -
templateToken
org.antlr.runtime.Token templateTokentemplate embedded in a group file? this is the template -
input
org.antlr.runtime.CharStream input -
c
char ccurrent character -
startCharIndex
int startCharIndexWhen we started token, track initial coordinates so we can properly build token objects. -
startLine
int startLine -
startCharPositionInLine
int startCharPositionInLine -
tokens
List<org.antlr.runtime.Token> tokensOur lexer routines might have to emit more than a single token. We buffer everything through this list.
-
-
Constructor Details
-
STLexer
public STLexer(org.antlr.runtime.CharStream input) -
STLexer
public STLexer(ErrorManager errMgr, org.antlr.runtime.CharStream input, org.antlr.runtime.Token templateToken) -
STLexer
public STLexer(ErrorManager errMgr, org.antlr.runtime.CharStream input, org.antlr.runtime.Token templateToken, char delimiterStartChar, char delimiterStopChar)
-
-
Method Details
-
nextToken
public org.antlr.runtime.Token nextToken()- Specified by:
nextToken
in interfaceorg.antlr.runtime.TokenSource
-
match
public void match(char x) Consume ifx
is next character on the input stream. -
consume
protected void consume() -
emit
public void emit(org.antlr.runtime.Token token) -
_nextToken
public org.antlr.runtime.Token _nextToken() -
outside
protected org.antlr.runtime.Token outside() -
inside
protected org.antlr.runtime.Token inside() -
subTemplate
org.antlr.runtime.Token subTemplate() -
ESCAPE
org.antlr.runtime.Token ESCAPE() -
UNICODE
org.antlr.runtime.Token UNICODE() -
mTEXT
org.antlr.runtime.Token mTEXT() -
mID
org.antlr.runtime.Token mID()ID : ('a'..'z'|'A'..'Z'|'_'|'/') ('a'..'z'|'A'..'Z'|'0'..'9'|'_'|'/')* ;
-
mSTRING
org.antlr.runtime.Token mSTRING()STRING : '"' ( '\\' '"' | '\\' ~'"' | ~('\\'|'"') )* '"' ;
-
WS
void WS() -
COMMENT
org.antlr.runtime.Token COMMENT() -
LINEBREAK
void LINEBREAK() -
isIDStartLetter
public static boolean isIDStartLetter(char c) -
isIDLetter
public static boolean isIDLetter(char c) -
isWS
public static boolean isWS(char c) -
isUnicodeLetter
public static boolean isUnicodeLetter(char c) -
newToken
public org.antlr.runtime.Token newToken(int ttype) -
newTokenFromPreviousChar
public org.antlr.runtime.Token newTokenFromPreviousChar(int ttype) -
newToken
-
newToken
-
getSourceName
- Specified by:
getSourceName
in interfaceorg.antlr.runtime.TokenSource
-
str
-