Details > Tokenizers

For translation that does not need the full force of a parser, a tokenizer is used instead. For instance when translating from a GAMS equation to a Gekko statement. When doing this, Gekko has a quite advanced tokenizer: StringTokenizer.GetTokensWithLeftBlanksRecursive(). It uses the concept of leftblanks (remembering whitespace), and parses recursively regarding parentheses of different kinds (and can also split up with commas).

The capabilities of tokens and basic tokenizers have been explained in this section, so the current section will explain how the more advance recursive blanks-preserving tokenizer works. The GetTokensWithLeftBlanksRecursive() method creates a tree with alternating TokenHelper and TokenList objects/nodes. TokenHelper contains the token and other info, and may contain a TokenList (for instance the contents of a (...) parenthesis). TokenList is basically just a very simple List<TokenHelper>.

An example regarding the string "a + (b + c)":

TokenHelper this node is artificial and contains nothing but a TokenList

TokenList a + (b + c) --> at this level only a + (...) is seen

/ \

TokenHelper TokenHelper the TokenHelper with children will be empty with a TokenList

containing what is inside the parenthesis. Before and

/ after this token there will be '(' and ')'

TokenList b + c --> at this level only b + c is seen

/ \

TokenHelper TokenHelper

The highest object in the illustration, the TokenHelper, is only a placeholder here. It contains a field .subnodes, which is a TokenList, in this case containing three TokenHelpers, representing "a", "+", and the (...) parenthesis. So the TokenHelper that corresponds to "(b + c)" will contain a TokenList that

The TokenHelper object also contains (among other things) info on the string contents (for instance "a"), the number of blanks to the left of the token (the field leftblanks), line and column in the original string or text file. As mentioned, the TokenList object is very simple: it is basically just a List<TokenHelper>. So when a node is "normal" (not for instance containing parentheses), it acts much like a normal token, whereas if it is not-normal, the token is basically empty except for the field .subnodes, which contains a list of the sub-tokens.

The way GetTokensWithLeftBlanksRecursive() works is that it first constructs a 1-dimensional normal sequence of tokens, using GetTokensWithLeftBlanks(). After that, the 2d structure is constructed with GetTokensWithLeftBlanksRecursiveHelper(). You may call GetTokensWithLeftBlanksRecursive() with the following arguments (examples provided):

•textInputRaw --> the raw text string

•commentsClosed --> for instance "/*" + "*/"

•commentsNonClosed --> for instance "!!" and "#"

•commentsClosedOnlyStartOfLine --> for instance "$ontext" + "$offtext"

•commentsNonClosedOnlyStartOfLine --> for instance "*"

Here, we allow different kinds of comments (this is taken from a GAMS parser). Some of the comments have start/end tags (closed), and some of them only work at the first position of the line. Parentheses recognized in the recursive parser are (...), [...] and {...}, beware that tags like <...> are not recognized as sub-nodes. But putting the contents of (...), [...] and {...} into sub-nodes makes it much easier to navigate mathematical expressions, because it is easy to skip behind a pair of parentheses to see what comes next, and comments etc. will be stored in separate tokens regardless of what they contain. The tokens themselves can be of these types: Word, Number, QuotedString, Comment, Symbol. Whitespace/blanks are stored in the tokens themselves, in the field leftblanks.

There are some useful methods:

•Equal() --> used to test for string equality from a list of tokens

•FindOptionFieldInSeriesAssignment() --> in a list of tokens, locates an option field like <2010 2020> in prt <2010 2020> x;.

•FindS() --> finds the next occurrence of a string in a list of tokens

•TokenHelper.Flatten() --> flatten 2d nested structure

•TokenHelper.SplitCommas() --> for a TokenHelper that contains for instance "(a, b, c)", split the string by commas into a List<TokenHelperComma> with three elements. Useful for splitting up function arguments and similar.

•TokenHelper.SiblingBefore()/After() --> finds the next token that is not a comment, end-of-line or end-of-file token. Useful for finding the previous/next word, symbol etc. without worring about blanks and comments (and nested parentheses for that matter).

Using a recursive tokenizer that preserves blanks has a lot of advantages when translating or extracting information from "foreign" languages like older Gekko versions, AREMOS, GAMS, etc. It does not have the full force of a parser like ANTLR, but it is much, much easier to setup and use.

It is not completely inconceivable that one day Gekko command files will be parsed and translated into C# with the help of such a recursive tokenizer instead of using ANTLR.