Package team.unnamed.mocha.lexer

Interface MolangLexer

All Superinterfaces:
AutoCloseable, Closeable
@NonExtendable public interface MolangLexer extends Closeable
Lexical analyzer for the Molang language.

The lexical analyzer converts character streams to token streams

Note that this is a stream-based lexer, this means that it will not consume the entire reader if it doesn't continue having next() calls

See the following example on correctly lexing a string:


     MolangLexer lexer = MolangLexer.lexer(new StringReader("1 + 1"));
     List<Token> tokens = new ArrayList<>();
     Token token;
     while ((token = lexer.next()).kind() != TokenKind.EOF) {
         tokens.add(token);
     }
     // tokens: [ Double, Plus, Double ]

Or using the shorter, convenience method:


      List<Token> tokens = MolangLexer.tokenizeAll("1 + 1");
      // tokens: [ Double, Plus, Double ]
Since:
3.0.0

  • Method Summary

    Modifier and Type
    Method
    Description
    void
    close()
    Closes this lexer and the internal Reader.
    @NotNull Token
    current()
    Returns the last emitted token (the last token value returned when calling next())
    @NotNull Cursor
    cursor()
    Returns the cursor for this lexer, the cursor maintains track of the current line and column, it is used for error reporting.
    static @NotNull MolangLexer
    lexer(@NotNull Reader reader)
    Creates a new lexer that will read the characters from the given reader.
    static @NotNull MolangLexer
    lexer(@NotNull String string)
    Creates a new lexer that will read the characters from the given string.
    @NotNull Token
    next()
    Reads the internal reader until it gets a token and then returns it.
    default @NotNull List<Token>
    tokenizeAll()
    Reads all the tokens until it finds a TokenKind.EOF.
    static @NotNull List<Token>
    tokenizeAll(@NotNull Reader reader)
    Tokenizes all the data from the given reader.
    static @NotNull List<Token>
    tokenizeAll(@NotNull String string)
    Tokenizes the provided string.

  • Method Details

    • lexer

      @NotNull static @NotNull MolangLexer lexer(@NotNull @NotNull Reader reader) throws IOException
      Creates a new lexer that will read the characters from the given reader.
      Parameters:
      reader - The reader to use.
      Returns:
      The created lexer
      Throws:
      IOException - If lexer initialization fails.
      Since:
      3.0.0

    • lexer

      @NotNull static @NotNull MolangLexer lexer(@NotNull @NotNull String string) throws IOException
      Creates a new lexer that will read the characters from the given string.
      Parameters:
      string - The string to tokenize.
      Returns:
      The created lexer
      Throws:
      IOException - If lexer initialization fails.
      Since:
      3.0.0

    • tokenizeAll

      @NotNull static @NotNull List<Token> tokenizeAll(@NotNull @NotNull Reader reader) throws IOException
      Tokenizes all the data from the given reader.
      Parameters:
      reader - The reader.
      Returns:
      The emitted tokens.
      Throws:
      IOException - If reading fails.
      Since:
      3.0.0

    • tokenizeAll

      @NotNull static @NotNull List<Token> tokenizeAll(@NotNull @NotNull String string) throws IOException
      Tokenizes the provided string.
      Parameters:
      string - The string.
      Returns:
      The emitted tokens.
      Throws:
      IOException - If reading fails.
      Since:
      3.0.0

    • cursor

      @NotNull @NotNull Cursor cursor()
      Returns the cursor for this lexer, the cursor maintains track of the current line and column, it is used for error reporting.
      Returns:
      The lexer cursor
      Since:
      3.0.0

    • current

      @NotNull @NotNull Token current()
      Returns the last emitted token (the last token value returned when calling next())

      Requires the user to call next() at least once first.

      Returns:
      The last emitted token
      Throws:
      IllegalStateException - If there is no current token
      Since:
      3.0.0

    • next

      @NotNull @NotNull Token next() throws IOException
      Reads the internal reader until it gets a token and then returns it.

      The returned token will never be null, but it can be of kind TokenKind.EOF or TokenKind.ERROR.

      We can stop lexing when we find a TokenKind.EOF token for the first time, since following tokens will be EOF too.

      Returns:
      The emitted token after reading characters from the internal reader
      Throws:
      IOException - If reading fails
      Since:
      3.0.0

    • tokenizeAll

      @NotNull default @NotNull List<Token> tokenizeAll() throws IOException
      Reads all the tokens until it finds a TokenKind.EOF.

      After this method is called, the lexer should be done and all next tokens should be EOF

      Returns:
      All the read tokens
      Throws:
      IOException - If reading fails
      Since:
      3.0.0

    • close

      void close() throws IOException
      Closes this lexer and the internal Reader.
      Specified by:
      close in interface AutoCloseable
      Specified by:
      close in interface Closeable
      Throws:
      IOException - If closing fails
      Since:
      3.0.0