Flex Tutorial - Using Flex [Lexical Analyzer Generator][Flex]

Flex (Fast lexical analyzer) is a lexical analyzer generator. It is a free software alternative to lex. It is a computer program that generates lexical analyzers ("scanners" or "lexers"). It is frequently used with the free Bison parser generator. Unlike Bison, flex is not part of the GNU project.

Download flex (windows) : http://gnuwin32.sourceforge.net/packages/flex.htm
Download tdm-gcc (windows) : http://tdm-gcc.tdragon.net/download

You need to set the path environment variable after you install these softwares properly.

Steps to write a lexical analyzer using flex:

1. Specify what your lexical analyzer should take care of using the format recognized by flex. Save this as a .l file. (lets say myLex.l)
2. Run "flex myLex.l" using cmd. This generates a .c source code file. ("lex.yy.c");
3. Compile the .c source code file using "gcc lex.yy.c". This generates "a.exe" as output file.
4. Run a.exe throgh cmd.

A flex program consists of three parts.

Part one may include C source code (definitions).
Second part specifies the working (rules) of the lexical analyzer you are making.
Third part may include other C code (subroutines).
The C code is incuded as-it-is in the generated .c file.

Parts are separated by %% sign.

... definitions ...
%%
... rules ...
%%
... subroutines ...

yytext is a global variable that is a pointer to the pattern matched in the input (null terminated string).
yyleng is another global variable that denotes the length of the matched pattern.
yylex() starts the lexical analyzer.

Some flex programs:
1. A lexical analyzer that accepts digits.

Source:

%option noyywrap
%%
[0-9] printf("Digit : %s\n", yytext); 
. ;
%%
int main(){
 yylex();
}

2. A lexical analyzer that counts the number of characters, words and newlines.

Source:

%option noyywrap
%{
int chars, words, newLines;
%}

%%
[a-zA-Z]+ {chars+=yyleng;words++;}
[ \t]    {chars++;}
[\n]    {newLines++;}

%% 
int main(){
 yylex();
 printf("words = %d characters = %d newlines = %d", words, chars, newLines);
}

3. A lexical analyzer that counts identifiers and digits.

%option noyywrap
%{
 int i = 0, d = 0;
%}

%% 
[a-zA-Z]+([a-zA-Z]|[0-9])*  i++; 
[0-9]+     d+=yyleng;
%%

int main(){
 yylex();
 printf("identifiers = %d digits = %d\n", i, d);
}

4. A lexical analyzer that converts vowels to uppercase.

vowel [aeiou]
%%
{vowel} {printf("%c", toupper(*yytext));}
%%
int main(){
 yylex();
 return 0;
}
int yywrap(){
 return 1;
}

5. A lexical analyzer that accepts identifiers, letters, constants, keywords and operators.

%%
"="   printf("\n%s\tOperator is ASSIGNMENT", yytext);
"++"  printf("\n%s\tOperator is INCREMENT", yytext);
"--"  printf("\n%s\tOperator is DECREMENT", yytext);
"+="  printf("\n%s\tOperator is INCREMENT and ASSIGN",      yytext);
"-="  printf("\n%s\tOperator is DECREMENT and ASSIGN",      yytext);
"+"  {printf("\n%s\tOperator is PLUS",yytext);}
"*"  {printf("\n%s\tOperator is MULTIPLICATION",yytext);}
"-"  {printf("\n%s\tOperator is MINUS",yytext);}
"=="  {printf("\n%s\tOperator is  EQUAL TO",yytext);}
"/"  {printf("\n%s\tOperator is DIVISION",yytext);}
"<"  {printf("\n%s\tOperator is LESS THEN",yytext);}

">"  {printf("\n%s\tOperator is GREATER THEN",yytext);}

">="  {printf("\n%s\tOperator is GREATER THEN EQUAL TO",yytext);}

"<="  {printf("\n%s\tOperator is LESS THEN EQUAL TO",yytext);}

"!="  {printf("\n%s\tOperator is NOT EQUAL TO",yytext);}

","  {printf("\n%s\tComma",yytext);}
";"  {printf("\n%s\tSemi Colon",yytext);} 
[(|{|)|}]     {printf("\n%s\tBraces",yytext);} 
"if"  {printf("\n%s\tKEYWORD",yytext);}
"while" {printf("\n%s\tKEYWORD",yytext);}
"for"  {printf("\n%s\tKEYWORD",yytext);}
"float" {printf("\n%s\tKEYWORD",yytext);}
"char"  {printf("\n%s\tKEYWORD",yytext);}
"int"   {printf("\n%s\tKEYWORD",yytext);}
[-+]*[0-9]* {printf("\n%s\tConstant",yytext);}
[a-zA-Z]+([0-9]|[a-zA-Z])* {printf("\n%s\tIdentifier",yytext);}
[ \t\n]+ //ignore whitespace and newlines
%%
int main()
{
yylex();
}

int yywrap()
{
return 1;
}

More here: http://epaperpress.com/lexandyacc/download/LexAndYaccTutorial.pdf