Looking for:
Awk manual |
April 1993 - Awk manual
AWK - Wikipedia. Awk manual
The operators! Variables may be scalars, array elements denoted x[i] or fields. Variables are initialized to the null string. Array subscripts may be any string, not necessarily numeric; this allows for a form of associative memory. Multiple subscripts such as [i,j,k] are permitted; the constituents are concatenated, separated by the value of SUBSEP see the section on variables below. The printf statement formats its expression list according to the format see printf 1.
Patterns are arbitrary Boolean combinations with! Isolated regular expressions in a pattern apply to the entire line. A pattern may consist of two patterns separated by a comma; in this case, the action is performed for all lines from an occurrence of the first pattern through an occurrence of the second.
A conditional is an arithmetic expression, a relational expression, or a Boolean combination of these. The include keyword can be used to read external awk source files. This gives you the ability to split large awk source files into smaller, more manageable pieces, and also lets you reuse common awk code from various awk scripts. In other words, you can group together awk functions used to carry out specific tasks into external files.
Note that source files may also be included using the -i option. Here is the test1 script:. So, to include external awk source files, you just use include followed by the name of the file to be included, enclosed in double quotes. NOTE: Keep in mind that this is a language construct and the file name cannot be a string variable, but rather just a literal string constant in double quotes.
This is very helpful in constructing gawk function libraries. If you have a large script with useful, general-purpose awk functions, you can break it down into library files and put those files in a special directory.
Of course, you can keep library files in more than one directory; the more complex the working environment is, the more directories you may need to organize the files to be included. Given the ability to specify multiple -f options, the include mechanism is not strictly necessary. However, the include keyword can help you in constructing self-contained gawk programs, thus reducing the need for writing complex and tedious command lines.
In particular, include is very useful for writing CGI scripts to be run from web pages. The load keyword can be used to read external awk extensions stored as system shared libraries.
Using load is completely equivalent to using the -l command-line option. For command-line usage, the -l option is more convenient, but load is useful for embedding inside an awk source file that requires access to an extension. It also describes the ordchr extension. The arbitrary precision arithmetic feature is deprecated as of gawk version 5.
The feature will be removed in the release of A regular expression , or regexp , is a way of describing a set of strings. Because regular expressions are such a fundamental part of awk programming, their format and use deserve a separate chapter.
The simplest regular expression is a sequence of letters, numbers, or both. Such a regexp matches any string that contains that sequence. Other kinds of regexps let you specify more complicated classes of strings. Initially, the examples in this chapter are simple. As we explain more about how regular expressions work, we present more complicated instances.
A regular expression can be used as a pattern by enclosing it in slashes. Then the regular expression is tested against the entire text of each record.
Normally, it only needs to match some part of the text in order to succeed. Regular expressions can also be used in matching expressions. These expressions allow you to specify the string to match against; it need not be the entire current input record. Expressions using these operators can be used as patterns, or in if , while , for , and do statements. See Control Statements in Actions. For example, the following is true if the expression exp taken as a string matches regexp :.
This next example is true if the expression exp taken as a character string does not match regexp :. One use of an escape sequence is to include a double-quote character in a string constant.
Other escape sequences represent unprintable characters such as TAB or newline. There is nothing to stop you from entering most unprintable characters directly in a string constant or regexp constant, but they may look ugly. The following list presents all the escape sequences used in awk and what they represent. Unless noted otherwise, all these escape sequences apply to both string constants and regexp constants:.
This often makes some sort of audible noise. Any further hexadecimal digits are treated as simple letters or numbers. For many years, gawk would continue incorporating hexadecimal digits into the value until a non-hexadecimal digit or the end of the string was encountered.
However, using more than two hexadecimal digits produced undefined results. A literal slash should be used for regexp constants only. Because the regexp is delimited by slashes, you need to escape any slash that is part of the pattern, in order to tell awk to keep processing the rest of the regexp.
A literal double quote should be used for string constants only. Because the string is delimited by double quotes, you need to escape any quote that is part of the string, in order to tell awk to keep processing the rest of the string. In gawk , a number of additional two-character sequences that begin with a backslash have special meaning in regexps.
See gawk -Specific Regexp Operators. In a regexp, a backslash before any character that is not in the previous list and not listed in gawk -Specific Regexp Operators means that the next character should be taken literally, even if it would normally be a regexp operator.
For complete portability, do not use a backslash before any character not shown in the previous list or that is not an operator. If you place a backslash in a string constant before something that is not one of the characters previously listed, POSIX awk purposely leaves what happens as undefined. There are two choices:. This is what BWK awk and gawk both do. Because this is such an easy bug both to introduce and to miss, gawk warns you about it.
Some other awk implementations do this. Suppose you use an octal or hexadecimal escape to represent a regexp metacharacter. See Regular Expression Operators. Does awk treat the character as a literal character or as a regexp operator? Historically, such characters were taken literally. However, the POSIX standard indicates that they should be treated as real metacharacters, which is what gawk does.
In compatibility mode see Command-Line Options , gawk treats the characters represented by octal and hexadecimal escape sequences literally when used in regexp constants. You can combine regular expressions with special characters, called regular expression operators or metacharacters , to increase the power and versatility of regular expressions. The escape sequences described earlier in Escape Sequences are valid inside a regexp. Here is a list of metacharacters.
All characters that are not escape sequences and that are not listed here stand for themselves:. This suppresses the special meaning of a character when matching. This matches the beginning of a string. The condition is not true in the following example:. The condition in the following example is not true:. This matches any single character, including the newline character. Otherwise, NUL is just another character. Other versions of awk may not be able to match the NUL character.
This is called a bracket expression. A full discussion of what can be inside the square brackets of a bracket expression is given in Using Bracket Expressions.
This is a complemented bracket expression. It matches any characters except those in the square brackets. This is the alternation operator and it is used to specify alternatives. Parentheses are used for grouping in regular expressions, as in arithmetic. These are Texinfo formatting control sequences. The left or opening parenthesis is always a metacharacter; to match one literally, precede it with a backslash. However, the right or closing parenthesis is only special when paired with a left parenthesis; an unpaired right parenthesis is silently treated as a regular character.
This symbol means that the preceding regular expression should be repeated as many times as necessary to find a match. One or two numbers inside braces denote an interval expression.
If there is one number in the braces, the preceding regexp is repeated n times. If there are two numbers separated by a comma, the preceding regexp is repeated n to m times. If there is one number followed by a comma, then the preceding regexp is repeated at least n times:. As in arithmetic, parentheses can change how operators are grouped.
However, many other versions of awk treat such a usage as a syntax error. Is this useful? What does it match? It is useful. It matches the invisible empty string at the start and end of a string of characters, as well as the empty string between characters.
This is best illustrated with the gsub function, which makes global substitutions in a string see String-Manipulation Functions. Normal usage of gsub is like so:. Interval expressions were not traditionally available in awk. However, beginning with version 4. This is because compatibility with POSIX has become more important to most gawk users than compatibility with old programs.
Then the regexp constants are valid and work the way you want them to, using any version of awk. As mentioned, interval expressions were not traditionally available in awk. In March of , BWK awk finally acquired them. Starting with version 5. POSIX says that interval expressions containing repetition counts greater than produce unspecified results.
Interval expressions may be implemented internally via repetition. A large repetition count may exhaust memory or greatly slow matching. Fortunately, regular expressions like these are typically artificial, and cascaded repetitions do not conform to POSIX so cannot be used in portable programs anyway. Next: How Much Text Matches? As mentioned earlier, a bracket expression matches any character among those listed between the opening and closing square brackets.
Within a bracket expression, a range expression consists of two characters separated by a hyphen. This is mainly of historical interest. With the increasing popularity of the Unicode character standard , there is an additional wrinkle to consider. Octal and hexadecimal escape sequences inside bracket expressions are taken to represent only single-byte characters characters whose values fit within the range 0— To match a range of characters where the endpoints of the range are larger than , enter the multibyte encodings of the characters directly.
For example, the notion of what is an alphabetic character differs between the United States and France. A character class is only valid in a regexp inside the brackets of a bracket expression. Table 3. If your character set had other alphabetic characters in it, this would not match them. This matches all values numerically between zero and , which is the defined range of the ASCII character set. NOTE: Some older versions of Unix awk treat [:blank:] like [:space:] , incorrectly matching more characters than they should.
Caveat Emptor. Two additional special sequences can appear in bracket expressions. These apply to non-ASCII character sets, which can have single symbols called collating elements that are represented with more than one character.
They can also have several characters that are equivalent for collating , or sorting, purposes. These sequences are:. Locale-specific names for a list of characters that are equal. This example uses the sub function to make a change to the input record. But when doing text matching and substitutions with the match , sub , gsub , and gensub functions, it is very important.
It may be any expression. The expression is evaluated and converted to a string if necessary; the contents of the string are then used as the regexp. A regexp computed in this way is called a dynamic regexp or a computed regexp :. If you are going to use a string constant, you have to understand that the string is, in essence, scanned twice : the first time when awk reads your program, and the second time when it goes to match the string on the lefthand side of the operator with the pattern on the right.
What difference does it make if the string is scanned twice? The answer has to do with escape sequences, and particularly with backslashes. To get a backslash into a regular expression inside a string, you have to type two backslashes. Only one backslash is needed. Given that you can use both regexp and string constants to describe regular expressions, which should you use? Some older versions of awk do not allow the newline character to be used inside a bracket expression for a dynamic regexp:.
GNU software that deals with regular expressions provides a number of additional regexp operators. These operators are described in this section and are specific to gawk ; they are not available in other awk implementations.
Most of the additional operators deal with word matching. Matches any space character as defined by the current locale. Matches any character that is not a space, as defined by the current locale. Matches any word-constituent character—that is, it matches any letter, digit, or underscore. Matches any character that is not word-constituent.
Matches the empty string at the beginning of a word. Matches the empty string at the end of a word. Matches the empty string at either the beginning or the end of a word i. Matches the empty string that occurs between two word-constituent characters. There are two other operators that work on buffers.
In Emacs, a buffer is, naturally, an Emacs buffer. Other GNU programs, including gawk , consider the entire string to match as the buffer.
The operators are:. They are provided for compatibility with other GNU software. An alternative method would have been to require two backslashes in the GNU operators, but this was deemed too confusing. The various command-line options see Command-Line Options control how gawk interprets characters in regexps:. Interval expressions are allowed. Match traditional Unix awk regexps. The GNU operators are not special. So too, interval expressions are allowed.
Characters described by octal and hexadecimal escape sequences are treated literally, even if they represent regexp metacharacters. Case is normally significant in regular expressions, both when matching ordinary characters i. However, this can be cumbersome if you need to use it often, and it can make the regular expressions harder to read. There are two alternatives that you might prefer. Prior to version 5. However, as of version 5.
Case is always significant in compatibility mode. In the typical awk program, awk reads all input either from the standard input by default, this is the keyboard, but often it is a pipe from another command or from files whose names you specify on the awk command line.
If you specify input files, awk reads them in order, processing all the data from one before going on to the next. The input is read in units called records , and is processed by the rules of your program one record at a time.
By default, each record is one line. Each record is automatically split into chunks called fields. This makes it more convenient for programs to work on the parts of a record.
On rare occasions, you may need to use the getline command. The getline command is valuable both because it can do explicit input from any number of files, and because the files used with it do not have to be named on the awk command line see Explicit Input with getline. It keeps track of the number of records that have been read so far from the current input file. This value is stored in a predefined variable called FNR , which is reset to zero every time a new file is started.
Another predefined variable, NR , records the total number of input records read so far from all data files. It starts at zero, but is never automatically reset to zero. Normally, records are separated by newline characters.
You can control how records are separated by assigning values to the built-in variable RS. If RS is any single character, that character separates records. Otherwise in gawk , RS is treated as a regular expression.
This mechanism is explained in greater detail shortly. Records are separated by a character called the record separator. By default, the record separator is the newline character. This is why records are, by default, single lines. To use a different character for the record separator, simply assign that character to the predefined variable RS.
The new record-separator character should be enclosed in quotation marks, which indicate a string constant. Often, the right time to do this is at the beginning of execution, before any input is processed, so that the very first record is read with the proper separator. Then the input file is read, and the second rule in the awk program the action with no pattern prints each record. Here are the results of running the program on mail-list :.
In the original data file see Data files for the Examples , the line looks like this:. In fact, this record is treated as part of the previous record; the newline separating them in the output is the original newline in the data file, not the one added by awk when it printed the record! Another way to change the record separator is on the command line, using the variable-assignment feature see Other Command-Line Arguments :.
The moral is: Know Your Data. Reaching the end of an input file terminates the current input record, even if the last character in the file is not the character in RS. The empty string "" a string without any characters has a special meaning as the value of RS. It means that records are separated by one or more blank lines and nothing else.
See Multiple-Line Records for more details. If you change the value of RS in the middle of an awk run, the new value is used to delimit subsequent records, but the record currently being processed, as well as records already processed, are not affected. After the end of the record has been determined, gawk sets the variable RT to the text in the input that matched RS. When using gawk , the value of RS is not limited to a one-character string.
If it contains more than one character, it is treated as a regular expression see Regular Expressions. In general, each record ends at the next string that matches the regular expression; the next record starts at the end of the matching string. This general rule is actually at work in the usual case, where RS contains just a newline: a record ends at the beginning of the next matching string the next newline in the input , and the following record starts just after the end of this string at the first character of the following line.
The newline, because it matches RS , is not part of either record. When RS is a single character, RT contains the same single character. However, when RS is a regular expression, RT contains the actual input text that matched the regular expression. If the input file ends without any text matching RS , gawk sets RT to the null string.
The following example illustrates both of these features. The square brackets delineate the contents of RT , letting you see the leading and trailing whitespace. The final value of RT is a newline. This is because gawk views the input file as one long string that happens to contain newline characters. It is thus best to avoid anchor metacharacters in the value of RS. Record splitting with regular expressions works differently than regexp matching with the sub , gsub , and gensub see String-Manipulation Functions.
Those functions allow a regexp to match the empty string; record splitting does not. The use of RS as a regular expression and the RT variable are gawk extensions; they are not available in compatibility mode see Command-Line Options. In compatibility mode, only the first character of the value of RS determines the end of the record.
As of October, , BWK awk also supports it. Neither version supplies RT , however. There are times when you might want to treat an entire data file as a single record. This is hard to do in a general way, such that a program always works for arbitrary input files. You might think that for text files, the NUL character, which consists of a character with all bits equal to zero, is a good value to use for RS in this case:.
However, this usage is not portable to most other awk implementations. Almost all other awk implementations 20 store strings internally as C-style strings. C strings use the NUL character as the string terminator.
It happens that recent versions of mawk can use the NUL character as a record separator. However, this is a special case: mawk does not allow embedded NUL characters in strings. This may change in a future version of mawk. See Reading a Whole File at Once for an interesting way to read whole files. If you are using gawk , see Reading an Entire File for another option. When awk reads an input record, the record is automatically parsed or separated by the awk utility into chunks called fields.
By default, fields are separated by whitespace , like words in a line. Whitespace in awk means any string of one or more spaces, TABs, or newlines; other characters that are considered whitespace by other languages such as formfeed, vertical tab, etc. The purpose of fields is to make it more convenient for you to refer to these pieces of the record.
Unlike in the Unix shells, the field numbers are not limited to single digits. For example, suppose the following is a line of input:. NF is a predefined variable whose value is the number of fields in the current record. If used in a numeric operation, you get zero.
Use it when you are not interested in specific fields. Here are some more examples:. A field number need not be a constant.
The value of the expression specifies the field number. If the value is a string, rather than a number, it is converted to a number. Consider this example:. Recall that NR is the number of records read so far: one in the first record, two in the second, and so on. So this example prints the first field of the first record, the second field of the second record, and so on.
For the twentieth record, field number 20 is printed; most likely, the record has fewer than 20 fields, so this prints a blank line. Here is another example of using expressions as field numbers:. To avoid confusion with the incompatible older version, this version was sometimes called "new awk" or nawk. This implementation was released under a free software license in and is still maintained by Brian Kernighan see external links below.
Jump to content Navigation. Help Learn to edit Community portal Recent changes Upload file. Download as PDF Printable version. In other projects. Wikimedia Commons Wikibooks.
On this Wikipedia the language links are at the top of the page across from the article title. Go to top. Contents move to sidebar hide. Article Talk. Read Edit View history. More Read Edit View history. This article is about the programming language. For other uses, see AWK disambiguation. IEEE Std Aho [13]. Archived PDF from the original on Retrieved Pilavakis UNIX Workshop. Macmillan International Higher Education. O'Reilly Media. Livingston May 2, Digital Review.
The Art of Unix Programming. Case Study: awk. Archived from the original on July 30, Retrieved May 11, The awk action language is Turing-complete, and can read and write files. September 1, Unix Seventh Edition Manual, Volume 2. Bell Telephone Laboratories, Inc. Retrieved February 1, Addison-Wesley Publishing Company.
ISBN Retrieved 16 May September 30, You can create an algorithm to generate the indices to an associative array, and control the order this way.
However, this is difficult to do. Since UNIX provides an excellent sort utility, more programmers separate the information processing from the sorting. I'll show you what I mean. This example will demonstrate these techniques, and illustrate the power and elegance of AWK. The program is simple and common.
The disk is full. Who's gonna be blamed? I just hope you use this power wisely. Remember, you may be the one who filled up the disk. Having resolved my moral dilemma, by placing the burden squarely on your shoulders, I will describe the program in detail.
I will also discuss several tips you will find useful in large AWK programs. First, initialize all arrays used in a for loop.
There will be four arrays for this purpose. Selecting the names of the arrays, and the indices for each array is very important. In a complex program, it can become confusing to remember which array contains what. I suggest you clearly identify the indices and contents of each array. Even when a quick hack comes back to haunt you three years later. I've been there. The third suggestion is to make sure your input is in the correct form.
It's generally a good idea to be pessimistic, but I will add a simple but sufficient test in this example. I placed the test and error clause up front, so the rest of the code won't be cluttered. AWK doesn't have user defined functions. The next piece of advice for complex AWK scripts is to define a name for each field used. In this case, we want the user, group and size in disk blocks.
We could use the file size in bytes, but the block size corresponds to the blocks on the disk, a more accurate measurement of space. Disk blocks can be found by using "ls -s". This adds a column, so the username becomes the fourth column, etc. Of course this is confusing. That's why it's a good idea to assign names to the fields. I've been there too. Next the AWK script will count how many times each combination of users and groups occur. That is, I am going to construct a two-part index that contains the username and groupname.
Consider this: how would you calculate the total for just a user, or for just a group? You could rewrite the script. You could do it, but it's not the AWK way to do it.
If you had to examine a bazillion files, and it takes a long time to run that script, it would be a waste to repeat this task. It's also inefficient to require two scripts when one can do everything. The proper way to solve this problem is to extract as much information as possible in one pass through the files.
I don't really need 4 arrays, as I can use the format of the index to determine which array is which. But this does maake the program easier to understand for now. The next tip is subtle, but you will see how useful it is. I mentioned the indices into the array can be anything. If possible, select a format that allows you to merge information from several arrays. I realize this makes no sense right now, but hang in there. All will become clear soon. There is a space between the two values.
What about the other three arrays? The heart of the script totals up the number and size of each file, putting the information into the right category. Also important is to sort the information in an order that is useful. You can try to force a particular output order in AWK, but why work at this, when it's a one line command for sort? The difficult part is finding the right way to sort the information. This script will sort information using the size of the category as the first sort field.
The largest total will be the one for all files, so this will be one of the first lines output. However, there may be several ties for the largest number, and care must be used. The second field will be the number of files. This will help break a tie.
The third and fourth fields will be generated by the index of the array. This is the tricky part I warned you about. The script will output one string, but the sort utility will not know this. Instead, it will treat it as two fields.
This will unify the results, and information from all 4 arrays will look like one array. The sort of the third and fourth fields will be dictionary order, and not numeric, unlike the first two fields.
That's the essence of the script. The results is sorted, and I converted the space into a tab for cosmetic reasons. All of the files are owned by root. There are 3 files with group daemon, which takes up 88 disk blocks.
As you can see, the first line of information is the total for all users and groups. The second line is the sub-total for the user "root". The third line is the sub-total for the group "staff". Therefore the order of the sort is useful, with the sub-totals before the individual entries. You could write a simple AWK or grep script to obtain information from just one user or one group, and the information will be easy to sort.
There is only one problem. Where's the discrepancy? The script does not understand hard links. This may not be a problem on most disks, because many users do not use hard links.
Still, it does generate inaccurate results. In this case, the program vi is also e , ex , edit , view , and 2 other names. The program only exists once, but has 7 names. You can tell because the link count field 2 reports 7. This causes the file to be counted 7 times, which causes an inaccurate total.
The fix is to only count multiple links once. Examining the link count will determine if a file has multiple links. However, how can you prevent counting a link twice?
There is an easy solution: all of these files have the same inode number. You can find this number with the -i option to ls. To save memory, we only have to remember the inodes of files that have multiple links.
This means we have to add another column to the input, and have to renumber all of the field references. It's a good thing there are only three. Adding a new field will be easy, because I followed my own advice.
The final script should be easy to follow. I have used variations of this hundreds of times and find it demonstrates the power of AWK as well as provide insight to a powerful programming paradigm. AWK solves these types of problems easier than most languages. But you have to use AWK the right way. Note - this version was written for a Solaris box. You have to verify if ls is generating the right number of arguments.
The -g argument may need to be deleted, and the check for the number of files may have to be modified. Updated I added a Linux version below - to be downloaded. This is a fully working version of the program, that accurately counts disk space, appears below:! I can use just one. This is more confusing, but more concise!
Columns might not line up properly, and it is often hard to find patterns or trends without this unity. As you use AWK more, you will be desirous of crisp, clean formatting. To achieve this, you must master the printf function.
C programmers should have no problem using printf function. Printf has one of these syntactical forms: printf format ; printf format, arguments I only use the first format to be consistent with other nearby printf statements. A print statement would do the same thing.
Printf reveals it's real power when formatting commands are used. The first argument to the printf function is the format. This is a string, or variable whose value is a string. This string, like all strings, can contain special escape sequences to print control characters. Hopefully I'll provide enough examples to demonstrate the differences. See the man page ascii 7 for more information.
Printf does nothing unless you specify the action. Table 3 illustrates the differences. When a string with numbers and letters are coverted into an integer, AWK will return a zero, while NAWK will convert as much as possible. The second example, marked with "NAWK? Using format specifiers, there is another way to print a double quote with NAWK.
This demonstrates Octal, Decimal and Hexadecimal conversion. As you can see, it isn't symmetrical. Decimal conversions are done differently. This is the width field. Spaces are added so the number of printed characters equal this number. Note that this is the minimum field size.
If the field becomes to large, it will grow, so information will not be lost. Spaces are added to the left. This format allows you to line up columns perfectly. If the string is too long, then the two fields will run together, making it hard to read. You may want to consider placing a single space between the fields, to make sure you will always have one space between the fields. This is very important if you want to pipe the output to another program.
Adding informational headers makes the output more readable. Be aware that changing the format of the data may make it difficult to get the columns aligned perfectly. Consider the following script:! More complicated formats would require a lot of trial and error. You have to adjust the first printf to agree with the second printf statement. I suggest! This may not seem like it's very useful, but when you have multiple formats and multiple columns, it's very useful to have a set of templates like the above.
If you have to add an extra space to make things line up, it's much easier to find and correct the problem with a set of format strings that are together, and the exact same width. CHainging the first columne from 10 characters to 11 is easy. Left Justification The last example places spaces before each field to make sure the minimum field width is met.
What do you do if you want the spaces on the right? The Field Precision Value The precision field, which is the number between the decimal and the format character, is more complex. With the octal, decimal or hexadecimal format, it specifies the minimum number of characters. Zeros are added to met this requirement. The precision specifies the number of digits displayed, before and after the decimal point. Explicit File output Instead of sending output to standard output, you can send output to a named file.
Appending to the file does not delete the old contents. However, there is a subtle difference between AWK and the shell. Consider the shell program:! Now consider the equivalent AWK program:! If you find this to be a problem, look into PERL. I hope this gives you the skill to make your AWK output picture perfect. Flow Control with next and exit You can exit from an awk script using the exit command. Let's say to expect all lines of a file to be 60 characters, and you want to use an awk program as a filter to exit if the number of characters is not Some sample code could be!
❿
No comments:
Post a Comment