генерировать программы для лексических задач (РАЗРАБОТКА) (generate programs for lexical tasks (DEVELOPMENT))
Расширенное описание (Extended description)
Each input file shall contain lex source code, which is a table
of regular expressions with corresponding actions in the form of
C program fragments.
When lex.yy.c is compiled and linked with the lex library (using
the -l l operand with c99), the resulting program shall read
character input from the standard input and shall partition it
into strings that match the given expressions.
When an expression is matched, these actions shall occur:
* The input string that was matched shall be left in yytext as
a null-terminated string; yytext shall either be an external
character array or a pointer to a character string. As
explained in Definitions in lex, the type can be explicitly
selected using the %array or %pointer declarations, but the
default is implementation-defined.
* The external int yyleng shall be set to the length of the
matching string.
* The expression's corresponding program fragment, or action,
shall be executed.
During pattern matching, lex shall search the set of patterns for
the single longest possible match. Among rules that match the
same number of characters, the rule given first shall be chosen.
The general format of lex source shall be:
Definitions %% Rules %% UserSubroutines
The first "%%" is required to mark the beginning of the rules
(regular expressions and actions); the second "%%" is required
only if user subroutines follow.
Any line in the Definitions section beginning with a <blank>
shall be assumed to be a C program fragment and shall be copied
to the external definition area of the lex.yy.c file. Similarly,
anything in the Definitions section included between delimiter
lines containing only "%{" and "%}" shall also be copied
unchanged to the external definition area of the lex.yy.c file.
Any such input (beginning with a <blank> or within "%{" and "%}"
delimiter lines) appearing at the beginning of the Rules section
before any rules are specified shall be written to lex.yy.c after
the declarations of variables for the yylex() function and before
the first line of code in yylex(). Thus, user variables local to
yylex() can be declared here, as well as application code to
execute upon entry to yylex().
The action taken by lex when encountering any input beginning
with a <blank> or within "%{" and "%}" delimiter lines appearing
in the Rules section but coming after one or more rules is
undefined. The presence of such input may result in an erroneous
definition of the yylex() function.
C-language code in the input shall not contain C-language
trigraphs. The C-language code within "%{" and "%}" delimiter
lines shall not contain any lines consisting only of "%}", or
only of "%%".
Definitions in lex
Definitions appear before the first "%%" delimiter. Any line in
this section not contained between "%{" and "%}" lines and not
beginning with a <blank> shall be assumed to define a lex
substitution string. The format of these lines shall be:
name substitute
If a name does not meet the requirements for identifiers in the
ISO C standard, the result is undefined. The string substitute
shall replace the string {name} when it is used in a rule. The
name string shall be recognized in this context only when the
braces are provided and when it does not appear within a bracket
expression or within double-quotes.
In the Definitions section, any line beginning with a <percent-
sign> ('%') character and followed by an alphanumeric word
beginning with either 's' or 'S' shall define a set of start
conditions. Any line beginning with a '%' followed by a word
beginning with either 'x' or 'X' shall define a set of exclusive
start conditions. When the generated scanner is in a %s state,
patterns with no state specified shall be also active; in a %x
state, such patterns shall not be active. The rest of the line,
after the first word, shall be considered to be one or more
<blank>-separated names of start conditions. Start condition
names shall be constructed in the same way as definition names.
Start conditions can be used to restrict the matching of regular
expressions to one or more states as described in Regular
Expressions in lex.
Implementations shall accept either of the following two
mutually-exclusive declarations in the Definitions section:
%array Declare the type of yytext to be a null-terminated
character array.
%pointer Declare the type of yytext to be a pointer to a null-
terminated character string.
The default type of yytext is implementation-defined. If an
application refers to yytext outside of the scanner source file
(that is, via an extern), the application shall include the
appropriate %array or %pointer declaration in the scanner source
file.
Implementations shall accept declarations in the Definitions
section for setting certain internal table sizes. The
declarations are shown in the following table.
Table: Table Size Declarations in lex
┌────────────┬────────────────────────────────────┬───────────────┐
│Declaration │ Description │ Minimum Value │
├────────────┼────────────────────────────────────┼───────────────┤
│%p n │ Number of positions │ 2500 │
│%n n │ Number of states │ 500 │
│%a n │ Number of transitions │ 2000 │
│%e n │ Number of parse tree nodes │ 1000 │
│%k n │ Number of packed character classes │ 1000 │
│%o n │ Size of the output array │ 3000 │
└────────────┴────────────────────────────────────┴───────────────┘
In the table, n represents a positive decimal integer, preceded
by one or more <blank> characters. The exact meaning of these
table size numbers is implementation-defined. The implementation
shall document how these numbers affect the lex utility and how
they are related to any output that may be generated by the
implementation should limitations be encountered during the
execution of lex. It shall be possible to determine from this
output which of the table size values needs to be modified to
permit lex to successfully generate tables for the input
language. The values in the column Minimum Value represent the
lowest values conforming implementations shall provide.
Rules in lex
The rules in lex source files are a table in which the left
column contains regular expressions and the right column contains
actions (C program fragments) to be executed when the expressions
are recognized.
ERE action
ERE action
...
The extended regular expression (ERE) portion of a row shall be
separated from action by one or more <blank> characters. A
regular expression containing <blank> characters shall be
recognized under one of the following conditions:
* The entire expression appears within double-quotes.
* The <blank> characters appear within double-quotes or square
brackets.
* Each <blank> is preceded by a <backslash> character.
User Subroutines in lex
Anything in the user subroutines section shall be copied to
lex.yy.c following yylex().
Regular Expressions in lex
The lex utility shall support the set of extended regular
expressions (see the Base Definitions volume of POSIX.1‐2017,
Section 9.4, Extended Regular Expressions), with the following
additions and exceptions to the syntax:
"..." Any string enclosed in double-quotes shall represent
the characters within the double-quotes as themselves,
except that <backslash>-escapes (which appear in the
following table) shall be recognized. Any
<backslash>-escape sequence shall be terminated by the
closing quote. For example, "\01""1" represents a
single string: the octal value 1 followed by the
character '1'.
<state>r, <state1,state2,...>r
The regular expression r shall be matched only when the
program is in one of the start conditions indicated by
state, state1, and so on; see Actions in lex. (As an
exception to the typographical conventions of the rest
of this volume of POSIX.1‐2017, in this case <state>
does not represent a metavariable, but the literal
angle-bracket characters surrounding a symbol.) The
start condition shall be recognized as such only at the
beginning of a regular expression.
r/x The regular expression r shall be matched only if it is
followed by an occurrence of regular expression x (x is
the instance of trailing context, further defined
below). The token returned in yytext shall only match
r. If the trailing portion of r matches the beginning
of x, the result is unspecified. The r expression
cannot include further trailing context or the '$'
(match-end-of-line) operator; x cannot include the '^'
(match-beginning-of-line) operator, nor trailing
context, nor the '$' operator. That is, only one
occurrence of trailing context is allowed in a lex
regular expression, and the '^' operator only can be
used at the beginning of such an expression.
{name} When name is one of the substitution symbols from the
Definitions section, the string, including the
enclosing braces, shall be replaced by the substitute
value. The substitute value shall be treated in the
extended regular expression as if it were enclosed in
parentheses. No substitution shall occur if {name}
occurs within a bracket expression or within double-
quotes.
Within an ERE, a <backslash> character shall be considered to
begin an escape sequence as specified in the table in the Base
Definitions volume of POSIX.1‐2017, Chapter 5, File Format
Notation ('\\', '\a', '\b', '\f', '\n', '\r', '\t', '\v'). In
addition, the escape sequences in the following table shall be
recognized.
A literal <newline> cannot occur within an ERE; the escape
sequence '\n' can be used to represent a <newline>. A <newline>
shall not be matched by a period operator.
Table: Escape Sequences in lex
┌─────────┬──────────────────────────┬──────────────────────────┐
│ Escape │ │ │
│Sequence │ Description │ Meaning │
├─────────┼──────────────────────────┼──────────────────────────┤
│\digits │ A <backslash> character │ The character whose │
│ │ followed by the longest │ encoding is represented │
│ │ sequence of one, two, or │ by the one, two, or │
│ │ three octal-digit │ three-digit octal │
│ │ characters (01234567). │ integer. Multi-byte │
│ │ If all of the digits are │ characters require │
│ │ 0 (that is, │ multiple, concatenated │
│ │ representation of the │ escape sequences of this │
│ │ NUL character), the │ type, including the │
│ │ behavior is undefined. │ leading <backslash> for │
│ │ │ each byte. │
├─────────┼──────────────────────────┼──────────────────────────┤
│\xdigits │ A <backslash> character │ The character whose │
│ │ followed by the longest │ encoding is represented │
│ │ sequence of hexadecimal- │ by the hexadecimal │
│ │ digit characters │ integer. │
│ │ (01234567abcdefABCDEF). │ │
│ │ If all of the digits are │ │
│ │ 0 (that is, │ │
│ │ representation of the │ │
│ │ NUL character), the │ │
│ │ behavior is undefined. │ │
├─────────┼──────────────────────────┼──────────────────────────┤
│\c │ A <backslash> character │ The character 'c', │
│ │ followed by any │ unchanged. │
│ │ character not described │ │
│ │ in this table or in the │ │
│ │ table in the Base │ │
│ │ Definitions volume of │ │
│ │ POSIX.1‐2017, Chapter 5, │ │
│ │ File Format Notation │ │
│ │ ('\\', '\a', '\b', '\f', │ │
│ │ '\n', '\r', '\t', '\v'). │ │
└─────────┴──────────────────────────┴──────────────────────────┘
Note: If a '\x' sequence needs to be immediately followed by a
hexadecimal digit character, a sequence such as "\x1""1"
can be used, which represents a character containing the
value 1, followed by the character '1'.
The order of precedence given to extended regular expressions for
lex differs from that specified in the Base Definitions volume of
POSIX.1‐2017, Section 9.4, Extended Regular Expressions. The
order of precedence for lex shall be as shown in the following
table, from high to low.
Note: The escaped characters entry is not meant to imply that
these are operators, but they are included in the table to
show their relationships to the true operators. The start
condition, trailing context, and anchoring notations have
been omitted from the table because of the placement
restrictions described in this section; they can only
appear at the beginning or ending of an ERE.
Table: ERE Precedence in lex
┌──────────────────────────────────┬──────────────────────┐
│ Extended Regular Expression │ Precedence │
├──────────────────────────────────┼──────────────────────┤
│collation-related bracket symbols │ [= =] [: :] [. .] │
│escaped characters │ \<special character> │
│bracket expression │ [ ] │
│quoting │ "..." │
│grouping │ ( ) │
│definition │ {name} │
│single-character RE duplication │ * + ? │
│concatenation │ │
│interval expression │ {m,n} │
│alternation │ | │
└──────────────────────────────────┴──────────────────────┘
The ERE anchoring operators '^' and '$' do not appear in the
table. With lex regular expressions, these operators are
restricted in their use: the '^' operator can only be used at the
beginning of an entire regular expression, and the '$' operator
only at the end. The operators apply to the entire regular
expression. Thus, for example, the pattern "(^abc)|(def$)" is
undefined; it can instead be written as two separate rules, one
with the regular expression "^abc" and one with "def$", which
share a common action via the special '|' action (see below). If
the pattern were written "^abc|def$", it would match either "abc"
or "def" on a line by itself.
Unlike the general ERE rules, embedded anchoring is not allowed
by most historical lex implementations. An example of embedded
anchoring would be for patterns such as "(^| )foo( |$)" to match
"foo" when it exists as a complete word. This functionality can
be obtained using existing lex features:
^foo/[ \n] |
" foo"/[ \n] /* Found foo as a separate word. */
Note also that '$' is a form of trailing context (it is
equivalent to "/\n") and as such cannot be used with regular
expressions containing another instance of the operator (see the
preceding discussion of trailing context).
The additional regular expressions trailing-context operator '/'
can be used as an ordinary character if presented within double-
quotes, "/"; preceded by a <backslash>, "\/"; or within a bracket
expression, "[/]". The start-condition '<' and '>' operators
shall be special only in a start condition at the beginning of a
regular expression; elsewhere in the regular expression they
shall be treated as ordinary characters.
Actions in lex
The action to be taken when an ERE is matched can be a C program
fragment or the special actions described below; the program
fragment can contain one or more C statements, and can also
include special actions. The empty C statement ';' shall be a
valid action; any string in the lex.yy.c input that matches the
pattern portion of such a rule is effectively ignored or skipped.
However, the absence of an action shall not be valid, and the
action lex takes in such a condition is undefined.
The specification for an action, including C statements and
special actions, can extend across several lines if enclosed in
braces:
ERE <one or more blanks> { program statement
program statement }
The program statements shall not contain unbalanced curly brace
preprocessing tokens.
The default action when a string in the input to a lex.yy.c
program is not matched by any expression shall be to copy the
string to the output. Because the default behavior of a program
generated by lex is to read the input and copy it to the output,
a minimal lex source program that has just "%%" shall generate a
C program that simply copies the input to the output unchanged.
Four special actions shall be available:
| ECHO; REJECT; BEGIN
| The action '|' means that the action for the next rule
is the action for this rule. Unlike the other three
actions, '|' cannot be enclosed in braces or be
<semicolon>-terminated; the application shall ensure
that it is specified alone, with no other actions.
ECHO; Write the contents of the string yytext on the output.
REJECT; Usually only a single expression is matched by a given
string in the input. REJECT means ``continue to the
next expression that matches the current input'', and
shall cause whatever rule was the second choice after
the current rule to be executed for the same input.
Thus, multiple rules can be matched and executed for
one input string or overlapping input strings. For
example, given the regular expressions "xyz" and "xy"
and the input "xyz", usually only the regular
expression "xyz" would match. The next attempted match
would start after z. If the last action in the "xyz"
rule is REJECT, both this rule and the "xy" rule would
be executed. The REJECT action may be implemented in
such a fashion that flow of control does not continue
after it, as if it were equivalent to a goto to another
part of yylex(). The use of REJECT may result in
somewhat larger and slower scanners.
BEGIN The action:
BEGIN newstate;
switches the state (start condition) to newstate. If
the string newstate has not been declared previously as
a start condition in the Definitions section, the
results are unspecified. The initial state is indicated
by the digit '0' or the token INITIAL.
The functions or macros described below are accessible to user
code included in the lex input. It is unspecified whether they
appear in the C code output of lex, or are accessible only
through the -l l operand to c99 (the lex library).
int yylex(void)
Performs lexical analysis on the input; this is the primary
function generated by the lex utility. The function shall
return zero when the end of input is reached; otherwise, it
shall return non-zero values (tokens) determined by the
actions that are selected.
int yymore(void)
When called, indicates that when the next input string is
recognized, it is to be appended to the current value of
yytext rather than replacing it; the value in yyleng shall
be adjusted accordingly.
int yyless(int n)
Retains n initial characters in yytext, NUL-terminated, and
treats the remaining characters as if they had not been
read; the value in yyleng shall be adjusted accordingly.
int input(void)
Returns the next character from the input, or zero on end-
of-file. It shall obtain input from the stream pointer
yyin, although possibly via an intermediate buffer. Thus,
once scanning has begun, the effect of altering the value
of yyin is undefined. The character read shall be removed
from the input stream of the scanner without any processing
by the scanner.
int unput(int c)
Returns the character 'c' to the input; yytext and yyleng
are undefined until the next expression is matched. The
result of using unput() for more characters than have been
input is unspecified.
The following functions shall appear only in the lex library
accessible through the -l l operand; they can therefore be
redefined by a conforming application:
int yywrap(void)
Called by yylex() at end-of-file; the default yywrap()
shall always return 1. If the application requires yylex()
to continue processing with another source of input, then
the application can include a function yywrap(), which
associates another file with the external variable FILE *
yyin and shall return a value of zero.
int main(int argc, char *argv[])
Calls yylex() to perform lexical analysis, then exits. The
user code can contain main() to perform application-
specific operations, calling yylex() as applicable.
Except for input(), unput(), and main(), all external and static
names generated by lex shall begin with the prefix yy or YY.
