Token vs. Production

[manticore-projects]

Greetings! Question please: Why are <S_CHAR_LITERAL> and <S_QUOTED_IDENTIFIER> defined as Token instead of a Production? Would it not be possible to replace those tokens by a production, which allowed LOOKAHEAD and better configuration of Escape- and Quote-Characters?

Of course such a production would need to mimic the token attributes, such as image.

TOKEN:
{
<S_IDENTIFIER: <LETTER> (<PART_LETTER>)*>
|   <#LETTER: ["$","A"-"Z","_","#","a"-"z",...,"\uffe5"-"\uffe6"]>
|   <#PART_LETTER: ["\u0000"-"\b",...,"\ufff9"-"\ufffb"]>
|   < S_CHAR_LITERAL: (["U","E","N","R","B"]|"RB"|"_utf8")? (("'" ( <ESC> | ~["'", "\\", "\n", "\r"] )* "'") | ("'" ("''" | ~["'"])* "'")) >
|   < S_QUOTED_IDENTIFIER: "\"" (~["\n","\r","\""])* "\"" | "$$" (~["\n","\r","\""])* "$$" | ("`" (~["\n","\r","`"])+ "`") | ( "[" (~["\n","\r","]"])* "]" ) >
    { if ( !configuration.getAsBoolean(Feature.allowSquareBracketQuotation) && matchedToken.image.charAt(0) == '[' ) {
         matchedToken.image = "[";
         for (int i=0;i<CCJSqlParserConstants.tokenImage.length;i++) {
            if (CCJSqlParserConstants.tokenImage[i].equals("\"[\"")) {
                matchedToken.kind = i;
            }
         }
         input_stream.backup(image.length() - 1);
       }
    }
|   < #ESC: "\\" ["n","t","b","r","f","\\","'","\""] >
}

[wumpz]

This divides the token manager and the parser itself. Tokens are parsed before and sent to the parser. There are parser generators that try to do that. JavaCC does not. Here tokens are simply regular expressions, even the simple ones, while productions for the parser accept a larger grammar space. Much of the optimizations of parsers are based on this tokenizing pre step.

What benefit do you think one would have to do that?

[manticore-projects]

This divides the token manager and the parser itself. Tokens are parsed before and sent to the parser. There are parser generators that try to do that. JavaCC does not. Here tokens are simply regular expressions, even the simple ones, while productions for the parser accept a larger grammar space. Much of the optimizations of parsers are based on this tokenizing pre step.

What benefit do you think one would have to do that?

Configuration related to different SQL dialects. Examples: Quotes and Escape Characters.
Specific example: LIKE ... ESCAPE ..., in this case you won't be able to define an SQL Escape Character, which is pre-defined as an Escape Character in the Grammar yet.

[wumpz]

No, but you could parse the token with some code like the bracket stuff is done.

[manticore-projects]

I have fixed the backslash escaping, but I am more than ever of the opinion that we need to refactor <S_IDENTIFIER> and <S_CHAR_LITERAL> into productions. This would affect other literal tokens too, since we would access productions and not tokens anymore.
I would like to suggest to release 4.6 first and then incorporate that change during the 4.7 cycle.