-
Notifications
You must be signed in to change notification settings - Fork 1.4k
Expand file tree
/
Copy pathlemon.html
More file actions
1296 lines (1109 loc) · 49.9 KB
/
lemon.html
File metadata and controls
1296 lines (1109 loc) · 49.9 KB
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
8000
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
997
998
999
1000
<html>
<head>
<title>The Lemon Parser Generator</title>
</head>
<body>
<a id="main"></a>
<h1 align='center'>The Lemon Parser Generator</h1>
<p>Lemon is an LALR(1) parser generator for C.
It does the same job as "bison" and "yacc".
But Lemon is not a bison or yacc clone. Lemon
uses a different grammar syntax which is designed to
reduce the number of coding errors. Lemon also uses a
parsing engine that is faster than yacc and
bison and which is both reentrant and threadsafe.
(Update: Since the previous sentence was written, bison
has also been updated so that it too can generate a
reentrant and threadsafe parser.)
Lemon also implements features that can be used
to eliminate resource leaks, making it suitable for use
in long-running programs such as graphical user interfaces
or embedded controllers.</p>
<p>This document is an introduction to the Lemon
parser generator.</p>
<a id="toc"></a>
<h2>1.0 Table of Contents</h2>
<ul>
<li><a href="#main">Introduction</a>
<li><a href="#toc">1.0 Table of Contents</a>
<li><a href="#secnot">2.0 Security Notes</a><br>
<li><a href="#optheory">3.0 Theory of Operation</a>
<ul>
<li><a href="#options">3.1 Command Line Options</a>
<li><a href="#interface">3.2 The Parser Interface</a>
<ul>
<li><a href="#onstack">3.2.1 Allocating The Parse Object On Stack</a>
<li><a href="#ifsum">3.2.2 Interface Summary</a>
</ul>
<li><a href="#yaccdiff">3.3 Differences With YACC and BISON</a>
<li><a href="#build">3.4 Building The "lemon" Or "lemon.exe" Executable</a>
</ul>
<li><a href="#syntax">4.0 Input File Syntax</a>
<ul>
<li><a href="#tnt">4.1 Terminals and Nonterminals</a>
<li><a href="#rules">4.2 Grammar Rules</a>
<li><a href="#precrules">4.3 Precedence Rules</a>
<li><a href="#special">4.4 Special Directives</a>
</ul>
<li><a href="#errors">5.0 Error Processing</a>
<li><a href="#history">6.0 History of Lemon</a>
<li><a href="#copyright">7.0 Copyright</a>
</ul>
<a id="secnot"></a>
<h2>2.0 Security Note</h2>
<p>The language parser code created by Lemon is very robust and
is well-suited for use in internet-facing applications that need to
safely process maliciously crafted inputs.</p>
<p>The "lemon.exe" command-line tool itself works great when given a valid
input grammar file and almost always gives helpful
error messages for malformed inputs. However, it is possible for
a malicious user to craft a grammar file that will cause
lemon.exe to crash.
We do not see this as a problem, as lemon.exe is not intended to be used
with hostile inputs.
To summarize:</p>
<ul>
<li>Parser code generated by lemon → Robust and secure
<li>The "lemon.exe" command line tool itself → Not so much
</ul>
<a id="optheory"></a>
<h2>3.0 Theory of Operation</h2>
<p>Lemon is computer program that translates a context free grammar (CFG)
for a particular language into C code that implements a parser for
that language.
The Lemon program has two inputs:</p>
<ul>
<li>The grammar specification.
<li>A parser template file.
</ul>
<p>Typically, only the grammar specification is supplied by the programmer.
Lemon comes with a default parser template
("<a href="https://sqlite.org/src/file/tool/lempar.c">lempar.c</a>")
that works fine for most applications. But the user is free to substitute
a different parser template if desired.</p>
<p>Depending on command-line options, Lemon will generate up to
three output files.</p>
<ul>
<li>C code to implement a parser for the input grammar.
<li>A header file defining an integer ID for each terminal symbol
(or "token").
<li>An information file that describes the states of the generated parser
automaton.
</ul>
<p>By default, all three of these output files are generated.
The header file is suppressed if the "-m" command-line option is
used and the report file is omitted when "-q" is selected.</p>
<p>The grammar specification file uses a ".y" suffix, by convention.
In the examples used in this document, we'll assume the name of the
grammar file is "gram.y". A typical use of Lemon would be the
following command:</p>
<pre>
lemon gram.y
</pre>
<p>This command will generate three output files named "gram.c",
"gram.h" and "gram.out".
The first is C code to implement the parser. The second
is the header file that defines numerical values for all
terminal symbols, and the last is the report that explains
the states used by the parser automaton.</p>
<a id="options"></a>
<h3>3.1 Command Line Options</h3>
<p>The behavior of Lemon can be modified using command-line options.
You can obtain a list of the available command-line options together
with a brief explanation of what each does by typing</p>
<pre>
lemon "-?"
</pre>
<p>As of this writing, the following command-line options are supported:</p>
<ul>
<li><b>-b</b>
Show only the basis for each parser state in the report file.
<li><b>-c</b>
Do not compress the generated action tables. The parser will be a
little larger and slower, but it will detect syntax errors sooner.
<li><b>-d</b><i>directory</i>
Write all output files into <i>directory</i>. Normally, output files
are written into the directory that contains the input grammar file.
<li><b>-D<i>name</i></b>
Define C preprocessor macro <i>name</i>. This macro is usable by
"<tt><a href='#pifdef'>%ifdef</a></tt>",
"<tt><a href='#pifdef'>%ifndef</a></tt>", and
"<tt><a href="#pifdef">%if</a></tt> lines
in the grammar file.
<li><b>-E</b>
Run the "%if" preprocessor step only and print the revised grammar
file.
<li><b>-g</b>
Do not generate a parser. Instead write the input grammar to standard
output with all comments, actions, and other extraneous text removed.
<li><b>-l</b>
Omit "#line" directives in the generated parser C code.
<li><b>-m</b>
Cause the output C source code to be compatible with the "makeheaders"
program.
<li><b>-p</b>
Display all conflicts that are resolved by
<a href='#precrules'>precedence rules</a>.
<li><b>-q</b>
Suppress generation of the report file.
<li><b>-r</b>
Do not sort or renumber the parser states as part of optimization.
<li><b>-s</b>
Show parser statistics before exiting.
<li><b>-T<i>file</i></b>
Use <i>file</i> as the template for the generated C-code parser implementation.
<li><b>-x</b>
Print the Lemon version number.
</ul>
<a id="interface"></a>
<h3>3.2 The Parser Interface</h3>
<p>Lemon doesn't generate a complete, working program. It only generates
a few subroutines that implement a parser. This section describes
the interface to those subroutines. It is up to the programmer to
call these subroutines in an appropriate way in order to produce a
complete system.</p>
<p>Before a program begins using a Lemon-generated parser, the program
must first create the parser.
A new parser is created as follows:</p>
<pre>
void *pParser = ParseAlloc( malloc );
</pre>
<p>The ParseAlloc() routine allocates and initializes a new parser and
returns a pointer to it.
The actual data structure used to represent a parser is opaque —
its internal structure is not visible or usable by the calling routine.
For this reason, the ParseAlloc() routine returns a pointer to void
rather than a pointer to some particular structure.
The sole argument to the ParseAlloc() routine is a pointer to the
subroutine used to allocate memory. Typically this means malloc().</p>
<p>After a program is finished using a parser, it can reclaim all
memory allocated by that parser by calling</p>
<pre>
ParseFree(pParser, free);
</pre>
<p>The first argument is the same pointer returned by ParseAlloc(). The
second argument is a pointer to the function used to release bulk
memory back to the system.</p>
<p>After a parser has been allocated using ParseAlloc(), the programmer
must supply the parser with a sequence of tokens (terminal symbols) to
be parsed. This is accomplished by calling the following function
once for each token:<p>
<pre>
Parse(pParser, hTokenID, sTokenData, pArg);
</pre>
<p>The first argument to the Parse() routine is the pointer returned by
ParseAlloc().
The second argument is a small positive integer that tells the parser the
type of the next token in the data stream.
There is one token type for each terminal symbol in the grammar.
The gram.h file generated by Lemon contains #define statements that
map symbolic terminal symbol names into appropriate integer values.
A value of 0 for the second argument is a special flag to the
parser to indicate that the end of input has been reached.
The third argument is the value of the given token. By default,
the type of the third argument is "void*", but the grammar will
usually redefine this type to be some kind of structure.
Typically the second argument will be a broad category of tokens
such as "identifier" or "number" and the third argument will
be the name of the identifier or the value of the number.</p>
<p>The Parse() function may have either three or four arguments,
depending on the grammar. If the grammar specification file requests
it (via the <tt><a href='#extraarg'>%extra_argument</a></tt> directive),
the Parse() function will have a fourth parameter that can be
of any type chosen by the programmer. The parser doesn't do anything
with this argument except to pass it through to action routines.
This is a convenient mechanism for passing state information down
to the action routines without having to use global variables.</p>
<p>A typical use of a Lemon parser might look something like the
following:</p>
<pre>
1 ParseTree *ParseFile(const char *zFilename){
2 Tokenizer *pTokenizer;
3 void *pParser;
4 Token sToken;
5 int hTokenId;
6 ParserState sState;
7
8 pTokenizer = TokenizerCreate(zFilename);
9 pParser = ParseAlloc( malloc );
10 InitParserState(&sState);
11 while( GetNextToken(pTokenizer, &hTokenId, &sToken) ){
12 Parse(pParser, hTokenId, sToken, &sState);
13 }
14 Parse(pParser, 0, sToken, &sState);
15 ParseFree(pParser, free );
16 TokenizerFree(pTokenizer);
17 return sState.treeRoot;
18 }
</pre>
<p>This example shows a user-written routine that parses a file of
text and returns a pointer to the parse tree.
(All error-handling code is omitted from this example to keep it
simple.)
We assume the existence of some kind of tokenizer which is created
using TokenizerCreate() on line 8 and deleted by TokenizerFree()
on line 16. The GetNextToken() function on line 11 retrieves the
next token from the input file and puts its type in the
integer variable hTokenId. The sToken variable is assumed to be
some kind of structure that contains details about each token,
such as its complete text, what line it occurs on, etc.</p>
<p>This example also assumes the existence of a structure of type
ParserState that holds state information about a particular parse.
An instance of such a structure is created on line 6 and initialized
on line 10. A pointer to this structure is passed into the Parse()
routine as the optional 4th argument.
The action routine specified by the grammar for the parser can use
the ParserState structure to hold whatever information is useful and
appropriate. In the example, we note that the treeRoot field of
the ParserState structure is left pointing to the root of the parse
tree.</p>
<p>The core of this example as it relates to Lemon is as follows:</p>
<pre>
Parse
527E
File(){
pParser = ParseAlloc( malloc );
while( GetNextToken(pTokenizer,&hTokenId, &sToken) ){
Parse(pParser, hTokenId, sToken);
}
Parse(pParser, 0, sToken);
ParseFree(pParser, free );
}
</pre>
<p>Basically, what a program has to do to use a Lemon-generated parser
is first create the parser, then send it lots of tokens obtained by
tokenizing an input source. When the end of input is reached, the
Parse() routine should be called one last time with a token type
of 0. This step is necessary to inform the parser that the end of
input has been reached. Finally, we reclaim memory used by the
parser by calling ParseFree().</p>
<p>There is one other interface routine that should be mentioned
before we move on.
The ParseTrace() function can be used to generate debugging output
from the parser. A prototype for this routine is as follows:</p>
<pre>
ParseTrace(FILE *stream, char *zPrefix);
</pre>
<p>After this routine is called, a short (one-line) message is written
to the designated output stream every time the parser changes states
or calls an action routine. Each such message is prefaced using
the text given by zPrefix. This debugging output can be turned off
by calling ParseTrace() again with a first argument of NULL (0).</p>
<a id="onstack"></a>
<h4>3.2.1 Allocating The Parse Object On Stack</h4>
<p>If all calls to the Parse() interface are made from within
<a href="#pcode"><tt>%code</tt> directives</a>, then the parse
object can be allocated from the stack rather than from the heap.
These are the steps:
<ul>
<li> Declare a local variable of type "yyParser"
<li> Initialize the variable using ParseInit()
<li> Pass a pointer to the variable in calls to Parse()
<li> Deallocate substructure in the parse variable using ParseFinalize().
</ul>
<p>The following code illustrates how this is done:
<pre>
ParseFile(){
yyParser x;
ParseInit( &x );
while( GetNextToken(pTokenizer,&hTokenId, &sToken) ){
Parse(&x, hTokenId, sToken);
}
Parse(&x, 0, sToken);
ParseFinalize( &x );
}
</pre>
<a id="ifsum"></a>
<h4>3.2.2 Interface Summary</h4>
<p>Here is a quick overview of the C-language interface to a
Lemon-generated parser:</p>
<blockquote><pre>
void *ParseAlloc( (void*(*malloc)(size_t) );
void ParseFree(void *pParser, (void(*free)(void*) );
void Parse(void *pParser, int tokenCode, ParseTOKENTYPE token, ...);
void ParseTrace(FILE *stream, char *zPrefix);
</pre></blockquote>
<p>Notes:</p>
<ul>
<li> Use the <a href="#pname"><tt>%name</tt> directive</a> to change
the "Parse" prefix names of the procedures in the interface.
<li> Use the <a href="#token_type"><tt>%token_type</tt> directive</a>
to define the "ParseTOKENTYPE" type.
<li> Use the <a href="#extraarg"><tt>%extra_argument</tt> directive</a>
to specify the type and name of the 4th parameter to the
Parse() function.
</ul>
<a id="yaccdiff"></a>
<h3>3.3 Differences With YACC and BISON</h3>
<p>Programmers who have previously used the yacc or bison parser
generator will notice several important differences between yacc and/or
bison and Lemon.</p>
<ul>
<li>In yacc and bison, the parser calls the tokenizer. In Lemon,
the tokenizer calls the parser.
<li>Lemon uses no global variables. Yacc and bison use global variables
to pass information between the tokenizer and parser.
<li>Lemon allows multiple parsers to be running simultaneously. Yacc
and bison do not.
</ul>
<p>These differences may cause some initial confusion for programmers
with prior yacc and bison experience.
But after years of experience using Lemon, I firmly
believe that the Lemon way of doing things is better.</p>
<p><i>Updated as of 2016-02-16:</i>
The text above was written in the 1990s.
We are told that Bison has lately been enhanced to support the
tokenizer-calls-parser paradigm used by Lemon, eliminating the
need for global variables.</p>
<a id="build"><a>
<h3>3.4 Building The "lemon" or "lemon.exe" Executable</h3>
<p>The "lemon" or "lemon.exe" program is built from a single file
of C-code named
"<a href="https://sqlite.org/src/tool/lemon.c">lemon.c</a>".
The Lemon source code is generic C89 code that uses
no unusual or non-standard libraries. Any
reasonable C compiler should suffice to compile the lemon program.
A command-line like the following will usually work:</p>
<blockquote><pre>
cc -o lemon lemon.c
</pre></blockquote
<p>On Windows machines with Visual C++ installed, bring up a
"VS20<i>NN</i> x64 Native Tools Command Prompt" window and enter:
<blockquote><pre>
cl lemon.c
</pre></blockquote>
<p>Compiling Lemon really is that simple.
Additional compiler options such as
"-O2" or "-g" or "-Wall" can be added if desired, but they are not
necessary.</p>
<a id="syntax"></a>
<h2>4.0 Input File Syntax</h2>
<p>The main purpose of the grammar specification file for Lemon is
to define the grammar for the parser. But the input file also
specifies additional information Lemon requires to do its job.
Most of the work in using Lemon is in writing an appropriate
grammar file.</p>
<p>The grammar file for Lemon is, for the most part, a free format.
It does not have sections or divisions like yacc or bison. Any
declaration can occur at any point in the file. Lemon ignores
whitespace (except where it is needed to separate tokens), and it
honors the same commenting conventions as C and C++.</p>
<a id="tnt"></a>
<h3>4.1 Terminals and Nonterminals</h3>
<p>A terminal symbol (token) is any string of alphanumeric
and/or underscore characters
that begins with an uppercase letter.
A terminal can contain lowercase letters after the first character,
but the usual convention is to make terminals all uppercase.
A nonterminal, on the other hand, is any string of alphanumeric
and underscore characters than begins with a lowercase letter.
Again, the usual convention is to make nonterminals use all lowercase
letters.</p>
<p>In Lemon, terminal and nonterminal symbols do not need to
be declared or identified in a separate section of the grammar file.
Lemon is able to generate a list of all terminals and nonterminals
by examining the grammar rules, and it can always distinguish a
terminal from a nonterminal by checking the case of the first
character of the name.</p>
<p>Yacc and bison allow terminal symbols to have either alphanumeric
names or to be individual characters included in single quotes, like
this: ')' or '$'. Lemon does not allow this alternative form for
terminal symbols. With Lemon, all symbols, terminals and nonterminals,
must have alphanumeric names.</p>
<a id="rules"></a>
<h3>4.2 Grammar Rules</h3>
<p>The main component of a Lemon grammar file is a sequence of grammar
rules.
Each grammar rule consists of a nonterminal symbol followed by
the special symbol "::=" and then a list of terminals and/or nonterminals.
The rule is terminated by a period.
The list of terminals and nonterminals on the right-hand side of the
rule can be empty.
Rules can occur in any order, except that the left-hand side of the
first rule is assumed to be the start symbol for the grammar (unless
specified otherwise using the <tt><a href='#start_symbol'>%start_symbol</a></tt>
directive described below.)
A typical sequence of grammar rules might look something like this:</p>
<pre>
expr ::= expr PLUS expr.
expr ::= expr TIMES expr.
expr ::= LPAREN expr RPAREN.
expr ::= VALUE.
</pre>
<p>There is one non-terminal in this example, "expr", and five
terminal symbols or tokens: "PLUS", "TIMES", "LPAREN",
"RPAREN" and "VALUE".</p>
<p>Like yacc and bison, Lemon allows the grammar to specify a block
of C code that will be executed whenever a grammar rule is reduced
by the parser.
In Lemon, this action is specified by putting the C code (contained
within curly braces <tt>{...}</tt>) immediately after the
period that closes the rule.
For example:</p>
<pre>
expr ::= expr PLUS expr. { printf("Doing an addition...\n"); }
</pre>
<p>In order to be useful, grammar actions must normally be linked to
their associated grammar rules.
In yacc and bison, this is accomplished by embedding a "$$" in the
action to stand for the value of the left-hand side of the rule and
symbols "$1", "$2", and so forth to stand for the value of
the terminal or nonterminal at position 1, 2 and so forth on the
right-hand side of the rule.
This idea is very powerful, but it is also very error-prone. The
single most common source of errors in a yacc or bison grammar is
to miscount the number of symbols on the right-hand side of a grammar
rule and say "$7" when you really mean "$8".</p>
<p>Lemon avoids the need to count grammar symbols by assigning symbolic
names to each symbol in a grammar rule and then using those symbolic
names in the action.
In yacc or bison, one would write this:</p>
<pre>
expr -> expr PLUS expr { $$ = $1 + $3; };
</pre>
<p>But in Lemon, the same rule becomes the following:</p>
<pre>
expr(A) ::= expr(B) PLUS expr(C). { A = B+C; }
</pre>
<p>In the Lemon rule, any symbol in parentheses after a grammar rule
symbol becomes a place holder for that symbol in the grammar rule.
This place holder can then be used in the associated C action to
stand for the value of that symbol.</p>
<p>The Lemon notation for linking a grammar rule with its reduce
action is superior to yacc/bison on several counts.
First, as mentioned above, the Lemon method avoids the need to
count grammar symbols.
Secondly, if a terminal or nonterminal in a Lemon grammar rule
includes a linking symbol in parentheses but that linking symbol
is not actually used in the reduce action, then an error message
is generated.
For example, the rule</p>
<pre>
expr(A) ::= expr(B) PLUS expr(C). { A = B; }
</pre>
<p>will generate an error because the linking symbol "C" is used
in the grammar rule but not in the reduce action.</p>
<p>The Lemon notation for linking grammar rules to reduce actions
also facilitates the use of destructors for reclaiming memory
allocated by the values of terminals and nonterminals on the
right-hand side of a rule.</p>
<a id='precrules'></a>
<h3>4.3 Precedence Rules</h3>
<p>Lemon resolves parsing ambiguities in exactly the same way as
yacc and bison. A shift-reduce conflict is resolved in favor
of the shift, and a reduce-reduce conflict is resolved by reducing
whichever rule comes first in the grammar file.</p>
<p>Just like in
yacc and bison, Lemon allows a measure of control
over the resolution of parsing conflicts using precedence rules.
A precedence value can be assigned to any terminal symbol
using the
<tt><a href='#pleft'>%left</a></tt>,
<tt><a href='#pright'>%right</a></tt> or
<tt><a href='#pnonassoc'>%nonassoc</a></tt> directives. Terminal symbols
mentioned in earlier directives have a lower precedence than
terminal symbols mentioned in later directives. For example:</p>
<pre>
%left AND.
%left OR.
%nonassoc EQ NE GT GE LT LE.
%left PLUS MINUS.
%left TIMES DIVIDE MOD.
%right EXP NOT.
</pre>
<p>In the preceding sequence of directives, the AND operator is
defined to have the lowest precedence. The OR operator is one
precedence level higher. And so forth. Hence, the grammar would
attempt to group the ambiguous expression</p>
<pre>
a AND b OR c
</pre>
<p>like this</p>
<pre>
a AND (b OR c).
</pre>
<p>The associativity (left, right or nonassoc) is used to determine
the grouping when the precedence is the same. AND is left-associative
in our example, so</p>
<pre>
a AND b AND c
</pre>
<p>is parsed like this</p>
<pre>
(a AND b) AND c.
</pre>
<p>The EXP operator is right-associative, though, so</p>
<pre>
a EXP b EXP c
</pre>
<p>is parsed like this</p>
<pre>
a EXP (b EXP c).
</pre>
<p>The nonassoc precedence is used for non-associative operators.
So</p>
<pre>
a EQ b EQ c
</pre>
<p>is an error.</p>
<p>The precedence of non-terminals is transferred to rules as follows:
The precedence of a grammar rule is equal to the precedence of the
left-most terminal symbol in the rule for which a precedence is
defined. This is normally what you want, but in those cases where
you want the precedence of a grammar rule to be something different,
you can specify an alternative precedence symbol by putting the
symbol in square braces after the period at the end of the rule and
before any C-code. For example:</p>
<pre>
expr = MINUS expr. [NOT]
</pre>
<p>This rule has a precedence equal to that of the NOT symbol, not the
MINUS symbol as would have been the case by default.</p>
<p>With the knowledge of how precedence is assigned to terminal
symbols and individual
grammar rules, we can now explain precisely how parsing conflicts
are resolved in Lemon. Shift-reduce conflicts are resolved
as follows:</p>
<ul>
<li> If either the token to be shifted or the rule to be reduced
lacks precedence information, then resolve in favor of the
shift, but report a parsing conflict.
<li> If the precedence of the token to be shifted is greater than
the precedence of the rule to reduce, then resolve in favor
of the shift. No parsing conflict is reported.
<li> If the precedence of the token to be shifted is less than the
precedence of the rule to reduce, then resolve in favor of the
reduce action. No parsing conflict is reported.
<li> If the precedences are the same and the shift token is
right-associative, then resolve in favor of the shift.
No parsing conflict is reported.
<li> If the precedences are the same and the shift token is
left-associative, then resolve in favor of the reduce.
No parsing conflict is reported.
<li> Otherwise, resolve the conflict by doing the shift, and
report a parsing conflict.
</ul>
<p>Reduce-reduce conflicts are resolved this way:</p>
<ul>
<li> If either reduce rule
lacks precedence information, then resolve in favor of the
rule that appears first in the grammar, and report a parsing
conflict.
<li> If both rules have precedence and the precedence is different,
then resolve the dispute in favor of the rule with the highest
precedence, and do not report a conflict.
<li> Otherwise, resolve the conflict by reducing by the rule that
appears first in the grammar, and report a parsing conflict.
</ul>
<a id="special"></a>
<h3>4.4 Special Directives</h3>
<p>The input grammar to Lemon consists of grammar rules and special
directives. We've described all the grammar rules, so now we'll
talk about the special directives.</p>
<p>Directives in Lemon can occur in any order. You can put them before
the grammar rules, or after the grammar rules, or in the midst of the
grammar rules. It doesn't matter. The relative order of
directives used to assign precedence to terminals is important, but
other than that, the order of directives in Lemon is arbitrary.</p>
<p>Lemon supports the following special directives:</p>
<ul>
<li><tt><a href='#pcode'>%code</a></tt>
<li><tt><a href='#default_destructor'>%default_destructor</a></tt>
<li><tt><a href='#default_type'>%default_type</a></tt>
<li><tt><a href='#destructor'>%destructor</a></tt>
<li><tt><a href='#pifdef'>%else</a></tt>
<li><tt><a href='#pifdef'>%endif</a></tt>
<li><tt><a href='#extraarg'>%extra_argument</a></tt>
<li><tt><a href='#pfallback'>%fallback</a></tt>
<li><tt><a href='#reallc'>%free</a></tt>
<li><tt><a href='#pifdef'>%if</a></tt>
<li><tt><a href='#pifdef'>%ifdef</a></tt>
<li><tt><a href='#pifdef'>%ifndef</a></tt>
<li><tt><a href='#pinclude'>%include</a></tt>
<li><tt><a href='#pleft'>%left</a></tt>
<li><tt><a href='#pname'>%name</a></tt>
<li><tt><a href='#pnonassoc'>%nonassoc</a></tt>
<li><tt><a href='#parse_accept'>%parse_accept</a></tt>
<li><tt><a href='#parse_failure'>%parse_failure</a></tt>
<li><tt><a href='#pright'>%right</a></tt>
<li><tt><a href='#reallc'>%realloc</a></tt>
<li><tt><a href='#stack_overflow'>%stack_overflow</a></tt>
<li><tt><a href='#reallc'>%stack_size_limit</a></tt>
<li><tt><a href='#stack_size'>%stack_size</a></tt>
<li><tt><a href='#start_symbol'>%start_symbol</a></tt>
<li><tt><a href='#syntax_error'>%syntax_error</a></tt>
<li><tt><a href='#token'>%token</a></tt>
<li><tt><a href='#token_class'>%token_class</a></tt>
<li><tt><a href='#token_destructor'>%token_destructor</a></tt>
<li><tt><a href='#token_prefix'>%token_prefix</a></tt>
<li><tt><a href='#token_type'>%token_type</a></tt>
<li><tt><a href='#ptype'>%type</a></tt>
<li><tt><a href='#pwildcard'>%wildcard</a></tt>
</ul>
<p>Each of these directives will be described separately in the
following sections:</p>
<a id='pcode'></a>
<h4>4.4.1 The <tt>%code</tt> directive</h4>
<p>The <tt>%code</tt> directive is used to specify additional C code that
is added to the end of the main output file. This is similar to
the <tt><a href='#pinclude'>%include</a></tt> directive except that
<tt>%include</tt> is inserted at the beginning of the main output file.</p>
<p><tt>%code</tt> is typically used to include some action routines or perhaps
a tokenizer or even the "main()" function
as part of the output file.</p>
<p>There can be multiple <tt>%code</tt> directives. The arguments of
all <tt>%code</tt> directives are concatenated.</p>
<a id='default_destructor'></a>
<h4>4.4.2 The <tt>%default_destructor</tt> directive</h4>
<p>The <tt>%default_destructor</tt> directive specifies a destructor to
use for non-terminals that do not have their own destructor
specified by a separate <tt>%destructor</tt> directive. See the documentation
on the <tt><a href='#destructor'>%destructor</a></tt> directive below for
additional information.</p>
<p>In some grammars, many different non-terminal symbols have the
same data type and hence the same destructor. This directive is
a convenient way to specify the same destructor for all those
non-terminals using a single statement.</p>
<a id='default_type'></a>
<h4>4.4.3 The <tt>%default_type</tt> directive</h4>
<p>The <tt>%default_type</tt> directive specifies the data type of non-terminal
symbols that do not have their own data type defined using a separate
<tt><a href='#ptype'>%type</a></tt> directive.</p>
<a id='destructor'></a>
<h4>4.4.4 The <tt>%destructor</tt> directive</h4>
<p>The <tt>%destructor</tt> directive is used to specify a destructor for
a non-terminal symbol.
(See also the <tt><a href='#token_destructor'>%token_destructor</a></tt>
directive which is used to specify a destructor for terminal symbols.)</p>
<p>A non-terminal's destructor is called to dispose of the
non-terminal's value whenever the non-terminal is popped from
the stack. This includes all of the following circumstances:</p>
<ul>
<li> When a rule reduces and the value of a non-terminal on
the right-hand side is not linked to C code.
<li> When the stack is popped during error processing.
<li> When the ParseFree() function runs.
</ul>
<p>The destructor can do whatever it wants with the value of
the non-terminal, but its design is to deallocate memory
or other resources held by that non-terminal.</p>
<p>Consider an example:</p>
<pre>
%type nt {void*}
%destructor nt { free($$); }
nt(A) ::= ID NUM. { A = malloc( 100 ); }
</pre>
<p>This example is a bit contrived, but it serves to illustrate how
destructors work. The example shows a non-terminal named
"nt" that holds values of type "void*". When the rule for
an "nt" reduces, it sets the value of the non-terminal to
space obtained from malloc(). Later, when the nt non-terminal
is popped from the stack, the destructor will fire and call
free() on this malloced space, thus avoiding a memory leak.
(Note that the symbol "$$" in the destructor code is replaced
by the value of the non-terminal.)</p>
<p>It is important to note that the value of a non-terminal is passed
to the destructor whenever the non-terminal is removed from the
stack, unless the non-terminal is used in a C-code action. If
the non-terminal is used by C-code, then it is assumed that the
C-code will take care of destroying it.
More commonly, the value is used to build some
larger structure, and we don't want to destroy it, which is why
the destructor is not called in this circumstance.</p>
<p>Destructors help avoid memory leaks by automatically freeing
allocated objects when they go out of scope.
To do the same using yacc or bison is much more difficult.</p>
<a id='extraarg'></a>
<h4>4.4.5 The <tt>%extra_argument</tt> directive</h4>
<p>The <tt>%extra_argument</tt> directive instructs Lemon to add a 4th parameter
to the parameter list of the Parse() function it generates. Lemon
doesn't do anything itself with this extra argument, but it does
make the argument available to C-code action routines, destructors,
and so forth. For example, if the grammar file contains:</p>
<pre>
%extra_argument { MyStruct *pAbc }
</pre>
<p>Then the Parse() function generated will have an 4th parameter
of type "MyStruct*" and all action routines will have access to
a variable named "pAbc" that is the value of the 4th parameter
in the most recent call to Parse().</p>
<p>The <tt>%extra_context</tt> directive works the same except that it
is passed in on the ParseAlloc() or ParseInit() routines instead of
on Parse().</p>
<a id='extractx'></a>
<h4>4.4.6 The <tt>%extra_context</tt> directive</h4>
<p>The <tt>%extra_context</tt> directive instructs Lemon to add a 2nd parameter
to the parameter list of the ParseAlloc() and ParseInit() functions. Lemon
doesn't do anything itself with these extra argument, but it does
store the value make it available to C-code action routines, destructors,
and so forth. For example, if the grammar file contains:</p>
<pre>
%extra_context { MyStruct *pAbc }
</pre>
<p>Then the ParseAlloc() and ParseInit() functions will have an 2nd parameter
of type "MyStruct*" and all action routines will have access to
a variable named "pAbc" that is the value of that 2nd parameter.</p>
<p>The <tt>%extra_argument</tt> directive works the same except that it
is passed in on the Parse() routine instead of on ParseAlloc()/ParseInit().</p>
<a id='pfallback'></a>
<h4>4.4.7 The <tt>%fallback</tt> directive</h4>
<p>The <tt>%fallback</tt> directive specifies an alternative meaning for one
or more tokens. The alternative meaning is tried if the original token
would have generated a syntax error.</p>
<p>The <tt>%fallback</tt> directive was added to support robust parsing of SQL
syntax in <a href='https://sqlite.org/'>SQLite</a>.
The SQL language contains a large assortment of keywords, each of which
appears as a different token to the language parser. SQL contains so
many keywords that it can be difficult for programmers to keep up with
them all. Programmers will, therefore, sometimes mistakenly use an
obscure language keyword for an identifier. The <tt>%fallback</tt> directive
provides a mechanism to tell the parser: "If you are unable to parse
this keyword, try treating it as an identifier instead."</p>
<p>The syntax of <tt>%fallback</tt> is as follows:</p>
<blockquote>
<tt>%fallback</tt> <i>ID</i> <i>TOKEN...</i> <b>.</b>
</blockquote></p>
<p>In words, the <tt>%fallback</tt> directive is followed by a list of token
names terminated by a period.
The first token name is the fallback token — the
token to which all the other tokens fall back to. The second and subsequent
arguments are tokens which fall back to the token identified by the first
argument.</p>
<a id='pifdef'></a>
<h4>4.4.8 The <tt>%if</tt> directive and its friends</h4>
<p>The <tt>%if</tt>, <tt>%ifdef</tt>, <tt>%ifndef</tt>, <tt>%else</tt>,
and <tt>%endif</tt> directives
are similar to #if, #ifdef, #ifndef, #else, and #endif in the C-preprocessor,
just not as general.
Each of these directives must begin at the left margin. No whitespace
is allowed between the "%" and the directive name.</p>
<p>Grammar text in between "<tt>%ifdef MACRO</tt>" and the next nested
"<tt>%endif</tt>" is
ignored unless the "-DMACRO" command-line option is used. Grammar text
between "<tt>%ifndef MACRO</tt>" and the next nested "<tt>%endif</tt>" is
included except when the "-DMACRO" command-line option is used.<p>
<p>The text in between "<tt>%if</tt> <i>CONDITIONAL</i>" and its
corresponding <tt>%endif</tt> is included only if <i>CONDITIONAL</i>
is true. The CONDITION is one or more macro names, optionally connected
using the "||" and "&&" binary operators, the "!" unary operator,
and grouped using balanced parentheses. Each term is true if the
corresponding macro exists, and false if it does not exist.</p>
<p>An optional "<tt>%else</tt>" directive can occur anywhere in between a
<tt>%ifdef</tt>, <tt>%ifndef</tt>, or <tt>%if</tt> directive and
its corresponding <tt>%endif</tt>.</p>
<p>Note that the argument to <tt>%ifdef</tt> and <tt>%ifndef</tt> is
intended to be a single preprocessor symbol name, not a general expression.
Use the "<tt>%if</tt>" directive for general expressions.</p>
<a id='pinclude'></a>
<h4>4.4.9 The <tt>%include</tt> directive</h4>
<p>The <tt>%include</tt> directive specifies C code that is included at the
top of the generated parser. You can include any text you want —
the Lemon parser generator copies it blindly. If you have multiple
<tt>%include</tt> directives in your grammar file, their values are concatenated
so that all <tt>%include</tt> code ultimately appears near the top of the
generated parser, in the same order as it appeared in the grammar.</p>
<p>The <tt>%include</tt> directive is very handy for getting some extra #include
preprocessor statements at the beginning of the generated parser.
For example:</p>
<pre>
%include {#include <unistd.h>}
</pre>
<p>This might be needed, for example, if some of the C actions in the
grammar call functions that are prototyped in unistd.h.</p>
<p>Use the <tt><a href="#pcode">%code</a></tt> directive to add code to
the end of the generated parser.</p>
<a id='pleft'></a>
<h4>4.4.10 The <tt>%left</tt> directive</h4>
The <tt>%left</tt> directive is used (along with the
<tt><a href='#pright'>%right</a></tt> and
<tt><a href='#pnonassoc'>%nonassoc</a></tt> directives) to declare
precedences of terminal symbols.
Every terminal symbol whose name appears after
a <tt>%left</tt> directive but before the next period (".") is
given the same left-associative precedence value. Subsequent
<tt>%left</tt> directives have higher precedence. For example:</p>
<pre>
%left AND.
%left OR.
%nonassoc EQ NE GT GE LT LE.
%left PLUS MINUS.
%left TIMES DIVIDE MOD.
%right EXP NOT.
</pre>
<p>Note the period that terminates each <tt>%left</tt>,
<tt>%right</tt> or <tt>%nonassoc</tt>
directive.</p>
<p>LALR(1) grammars can get into a situation where they require
a large amount of stack space if you make heavy use or right-associative
operators. For this reason, it is recommended that you use <tt>%left</tt>
rather than <tt>%right</tt> whenever possible.</p>
<a id='pname'></a>
<h4>4.4.11 The <tt>%name</tt> directive</h4>
<p>By default, the functions generated by Lemon all begin with the
five-character string "Parse". You can change this string to something
different using the <tt>%name</tt> directive. For instance:</p>
<pre>
%name Abcde
</pre>
<p>Putting this directive in the grammar file will cause Lemon to generate
functions named</p>
<ul>
<li> AbcdeAlloc(),
<li> AbcdeFree(),
<li> AbcdeTrace(), and
<li> Abcde().
</ul>
</p>The <tt>%name</tt> directive allows you to generate two or more different
parsers and link them all into the same executable.</p>
<a id='pnonassoc'></a>
<h4>4.4.12 The <tt>%nonassoc</tt> directive</h4>
<p>This directive is used to assign non-associative precedence to
one or more terminal symbols. See the section on
<a href='#precrules'>precedence rules</a>
or on the <tt><a href='#pleft'>%left</a></tt> directive
for additional information.</p>
<a id='parse_accept'></a>
<h4>4.4.13 The <tt>%parse_accept</tt> directive</h4>
<p>The <tt>%parse_accept</tt> directive specifies a block of C code that is
executed whenever the parser accepts its input string. To "accept"
an input string means that the parser was able to process all tokens
without error.</p>
<p>For example:</p>
<pre>
%parse_accept {
printf("parsing complete!\n");