Class STLexer

java.lang.Object
org.stringtemplate.v4.compiler.STLexer
All Implemented Interfaces:
org.antlr.runtime.TokenSource

public class STLexer extends Object implements org.antlr.runtime.TokenSource
This class represents the tokenizer for templates. It operates in two modes: inside and outside of expressions. It implements the TokenSource interface so it can be used with ANTLR parsers. Outside of expressions, we can return these token types: TEXT, INDENT, LDELIM (start of expression), RCURLY (end of subtemplate), and NEWLINE. Inside of an expression, this lexer returns all of the tokens needed by STParser. From the parser's point of view, it can treat a template as a simple stream of elements.

This class defines the token types and communicates these values to STParser.g via STLexer.tokens file (which must remain consistent).

  • Nested Class Summary

    Nested Classes
    Modifier and Type
    Class
    Description
    static class 
    We build STToken tokens instead of relying on CommonToken so we can override STLexer.STToken.toString().
  • Field Summary

    Fields
    Modifier and Type
    Field
    Description
    static final int
     
    static final int
     
    static final int
     
    (package private) char
    current character
    static final int
     
    static final int
     
    static final int
     
    (package private) char
    The char which delimits the start of an expression.
    (package private) char
    The char which delimits the end of an expression.
    static final int
     
    static final int
     
    static final int
     
    static final int
     
    static final int
     
    static final char
     
    static final int
     
    static final int
     
    (package private) ErrorManager
     
    static final int
     
    static final int
     
    static final int
     
    static final int
     
    (package private) org.antlr.runtime.CharStream
     
    static final int
     
    static final int
     
    static final int
     
    static final int
     
    static final int
     
    static final int
     
    static final int
     
    static final int
     
    static final int
     
    static final int
     
    static final int
     
    static final int
     
    (package private) boolean
    This keeps track of the current mode of the lexer.
    static final int
     
    static final org.antlr.runtime.Token
     
    static final int
     
    (package private) int
    When we started token, track initial coordinates so we can properly build token objects.
    (package private) int
     
    (package private) int
     
    static final int
     
    int
    To be able to properly track the inside/outside mode, we need to track how deeply nested we are in some templates.
    static final int
     
    (package private) org.antlr.runtime.Token
    template embedded in a group file? this is the template
    static final int
     
    (package private) List<org.antlr.runtime.Token>
    Our lexer routines might have to emit more than a single token.
    static final int
     
  • Constructor Summary

    Constructors
    Constructor
    Description
    STLexer(org.antlr.runtime.CharStream input)
     
    STLexer(ErrorManager errMgr, org.antlr.runtime.CharStream input, org.antlr.runtime.Token templateToken)
     
    STLexer(ErrorManager errMgr, org.antlr.runtime.CharStream input, org.antlr.runtime.Token templateToken, char delimiterStartChar, char delimiterStopChar)
     
  • Method Summary

    Modifier and Type
    Method
    Description
    org.antlr.runtime.Token
     
    (package private) org.antlr.runtime.Token
     
    protected void
     
    void
    emit(org.antlr.runtime.Token token)
     
    (package private) org.antlr.runtime.Token
     
     
    protected org.antlr.runtime.Token
     
    static boolean
    isIDLetter(char c)
     
    static boolean
     
    static boolean
     
    static boolean
    isWS(char c)
     
    (package private) void
     
    void
    match(char x)
    Consume if x is next character on the input stream.
    (package private) org.antlr.runtime.Token
    mID()
    ID : ('a'..'z'|'A'..'Z'|'_'|'/') ('a'..'z'|'A'..'Z'|'0'..'9'|'_'|'/')* ;
    (package private) org.antlr.runtime.Token
    STRING : '"' ( '\\' '"' | '\\' ~'"' | ~('\\'|'"') )* '"' ;
    (package private) org.antlr.runtime.Token
     
    org.antlr.runtime.Token
    newToken(int ttype)
     
    org.antlr.runtime.Token
    newToken(int ttype, String text)
     
    org.antlr.runtime.Token
    newToken(int ttype, String text, int pos)
     
    org.antlr.runtime.Token
     
    org.antlr.runtime.Token
     
    protected org.antlr.runtime.Token
     
    static String
    str(int c)
     
    (package private) org.antlr.runtime.Token
     
    (package private) org.antlr.runtime.Token
     
    (package private) void
    WS()
     

    Methods inherited from class java.lang.Object

    clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait
  • Field Details

  • Constructor Details

    • STLexer

      public STLexer(org.antlr.runtime.CharStream input)
    • STLexer

      public STLexer(ErrorManager errMgr, org.antlr.runtime.CharStream input, org.antlr.runtime.Token templateToken)
    • STLexer

      public STLexer(ErrorManager errMgr, org.antlr.runtime.CharStream input, org.antlr.runtime.Token templateToken, char delimiterStartChar, char delimiterStopChar)
  • Method Details

    • nextToken

      public org.antlr.runtime.Token nextToken()
      Specified by:
      nextToken in interface org.antlr.runtime.TokenSource
    • match

      public void match(char x)
      Consume if x is next character on the input stream.
    • consume

      protected void consume()
    • emit

      public void emit(org.antlr.runtime.Token token)
    • _nextToken

      public org.antlr.runtime.Token _nextToken()
    • outside

      protected org.antlr.runtime.Token outside()
    • inside

      protected org.antlr.runtime.Token inside()
    • subTemplate

      org.antlr.runtime.Token subTemplate()
    • ESCAPE

      org.antlr.runtime.Token ESCAPE()
    • UNICODE

      org.antlr.runtime.Token UNICODE()
    • mTEXT

      org.antlr.runtime.Token mTEXT()
    • mID

      org.antlr.runtime.Token mID()
        ID  : ('a'..'z'|'A'..'Z'|'_'|'/')
              ('a'..'z'|'A'..'Z'|'0'..'9'|'_'|'/')*
            ;
        
    • mSTRING

      org.antlr.runtime.Token mSTRING()
        STRING : '"'
                 (   '\\' '"'
                 |   '\\' ~'"'
                 |   ~('\\'|'"')
                 )*
                 '"'
               ;
       
    • WS

      void WS()
    • COMMENT

      org.antlr.runtime.Token COMMENT()
    • LINEBREAK

      void LINEBREAK()
    • isIDStartLetter

      public static boolean isIDStartLetter(char c)
    • isIDLetter

      public static boolean isIDLetter(char c)
    • isWS

      public static boolean isWS(char c)
    • isUnicodeLetter

      public static boolean isUnicodeLetter(char c)
    • newToken

      public org.antlr.runtime.Token newToken(int ttype)
    • newTokenFromPreviousChar

      public org.antlr.runtime.Token newTokenFromPreviousChar(int ttype)
    • newToken

      public org.antlr.runtime.Token newToken(int ttype, String text, int pos)
    • newToken

      public org.antlr.runtime.Token newToken(int ttype, String text)
    • getSourceName

      public String getSourceName()
      Specified by:
      getSourceName in interface org.antlr.runtime.TokenSource
    • str

      public static String str(int c)