Previous: Grammar styles, Up: Wisent Semantic [Contents][Index]

### 4.2 The Wisent Lex lexer

The lexical analysis step of Semantic is performed by the general function `semantic-lex`. For more information, See (semantic-langdev)Writing Lexers.

`semantic-lex` produces lexical tokens of the form:

    (token-class start . end)

token-class

  Is a symbol that identifies a lexical token class, like `symbol`, `string`, `number`, or `PAREN_BLOCK`.

start
end

  Are the start and end positions of mapped data in the input buffer.

The Wisent’s parser doesn’t depend on the nature of analyzed input stream (buffer, string, etc.), and requires that lexical tokens have a different form (see Writing a lexer):

    (token-class value [start . end])

`wisent-lex` is the default Wisent’s lexer used in Semantic.

Function: wisent-lex

  Return the next available lexical token in Wisent’s form.

  The variable `wisent-lex-istream` contains the list of lexical tokens produced by `semantic-lex`. Pop the next token available and convert it to a form suitable for the Wisent’s parser.

Mapping of lexical tokens as produced by `semantic-lex` into equivalent Wisent lexical tokens is straightforward:

    (token-class start . end)
         ⇒ (token-class value start . end)

value is the input `buffer-substring` from start to end.