Module org.seasar.doma.core
Package org.seasar.doma.internal.jdbc.sql

Class SqlTokenizer

java.lang.Object
org.seasar.doma.internal.jdbc.sql.SqlTokenizer
public class SqlTokenizer extends Object
High-performance SQL tokenizer for Doma's two-way SQL processing.

Architecture Overview

This tokenizer processes SQL strings character by character, identifying tokens such as:
  • SQL keywords (SELECT, FROM, WHERE, etc.)
  • Block comments (/* */), including Doma directives (/*%if*/, /*%expand*/, etc.)
  • Line comments (--)
  • Quoted strings ('text')
  • Bind variables (/*param*/value)
  • Parentheses, delimiters, and other SQL syntax elements

Key Design Principles

  • Performance-Critical: Used extensively in SQL parsing pipelines
  • Memory-Efficient: Uses CharBuffer for zero-copy string operations
  • Lookahead Optimization: 10-character lookahead buffer minimizes buffer operations

Internal State Management

The tokenizer maintains several critical state variables:
  • buf: CharBuffer positioned at current parsing location
  • lookahead: Fixed 10-char array for efficient keyword matching
  • tokenStartIndex: Start position of current token for substring extraction
  • currentLineNumber: Line tracking for error reporting

Parsing Strategy

The core parsing logic in peek() uses a two-phase approach:
  1. Character Classification: Determines if first character starts a word
  2. Optimized Parsing: Uses character-based branching for efficient keyword matching

Performance Optimizations

  • Bitwise Case Folding: (ch | 0x20) for fast case-insensitive comparison
  • Character-based Branching: Groups keywords by first character and checks in frequency order for optimal performance
  • Direct Character Classification: Uses SqlTokenUtil for O(1) character classification with pre-computed lookup tables
  • Minimal Buffer Operations: Reduces CharBuffer position changes
  • Specialized Handlers: Separate methods for quotes, comments, and words

Error Handling

Provides precise error reporting with line/column information for:
  • Unterminated quoted strings (DOMA2101)
  • Unterminated block comments (DOMA2102)
  • Invalid directive syntax (DOMA2119)

Maintenance Guidelines

  • Keyword Changes: Update both peekWord() character-based branching and corresponding isXxxWord() methods
  • Performance Testing: Run JMH benchmarks after modifications (see SqlTokenUtilBenchmark)
  • Buffer Position: Always ensure buffer position is correctly managed in parsing methods
  • Lookahead Consistency: Verify lookahead array size matches maximum keyword length

Thread Safety

This class is NOT thread-safe. Each parsing operation requires a separate instance.
See Also:

  • Constructor Details

    • SqlTokenizer

      public SqlTokenizer(String sql)

  • Method Details

    • next

      public SqlTokenType next()
      Advances to the next token and returns its type.

      This method coordinates the tokenization process:

      1. Handles EOF and EOL special cases
      2. Prepares the current token (extracts substring and position info)
      3. Advances parsing position and determines next token type
      Returns:
      the type of the current token (before advancing)

    • getToken

      public String getToken()

    • getLineNumber

      public int getLineNumber()

    • getPosition

      public int getPosition()