AWK
Mohamed Mukthar Ahmed
CONTENTS
Introduction
Structure of awk Program
First awk Program
Searching For a String
Records and Fields
Awk Relational and Logical Operators
Formatted Output
BEGIN and END Statements
Awk Defined Variables
User Defined Variables
Regular Expressions
Match Operators
Concatenation Operator
Mohamed Mukthar Ahmed
Page 1
CONTENTS
Combining Patterns
Pattern Range
Awk Defined Variables Again
Built In Arithmetic Functions
Built In String Functions
Control Flow Statements
next Statement
Arrays in Awk
for Statement
User Defined Functions
Output to File
Output to Pipes
Awk Input
getline( ) Function
Mohamed Mukthar Ahmed
Introduction
A powerful pattern matching and scanning tool.
SpecialSpecial-purpose language for lineline-oriented pattern
processing
Typically used to scan an input string, grab certain
portions of the string, then output the information in
another format.
Developed by Alfred Aho,
ho, Peter Weinberger, and
Brian Kernighan (hence the name AWK).
AWK).
There are several variants of awk: standard awk
(awk), Gnu awk (gawk
), and new awk (nawk
(gawk),
(nawk)) are the
most common.
.
common
Mohamed Mukthar Ahmed
Page 2
Introduction
AWK is a UNIX/Linux programming language used for
manipulating data and generating reports.
AWK can be used at the command line for simple
operations.
It can also be written into programs or scripts for
larger applications.
Mohamed Mukthar Ahmed
Structure of AWK Program
AWK scans a file (or any input source) line by line,
searching for lines that match a certain pattern
(regular expression) or condition.
For each pattern, an action is specified. The action is
performed when the pattern matches.
In short, an awk program consists of a number of
patterns and associated actions.
Actions are enclosed using curly braces, and separated
using semi-colons.
If there is an action without a pattern, all lines are
executed.
Mohamed Mukthar Ahmed
Page 3
First AWK Program
Consider the following simple awk program.
{ print $0 }
The action prints field 0 ( the entire line )
$ awk f myawk1
/etc/group
The general syntax of awk is as follows:
awk Fchar /pattern/{action} input_file
awk Fchar f awkscript input_file
The default delimiting character is Tab or Space.
Space.
However, if it is other than tab or space, we need to
explicitly state it using the F option.
$ awk F: -f myawk1
/etc/passwd
Mohamed Mukthar Ahmed
Searching For a String
To search for a string in an input line, specify it as a
pattern.
Patterns are enclosed using forward slash symbols.
# Searching for a pattern mukthar
/mukthar/ { print $0 }
$ awk f myawk2
$ who
/etc/passwd
| awk -f myawk2
Mohamed Mukthar Ahmed
Page 4
Records and Fields
awk sees input as a table of rows and columns.
The table consists of rows that represent records.
The entire row is identified by $0
Columns in the table are the fields. The columns are
identified as $1,
$1, $2,
$2, $3
awk expects that the fields are delimited by either a
spaces or a tabs. We can change the delimiter by
using the F option
# Searching for a condition
$3 == 501 { print $0 }
Mohamed Mukthar Ahmed
AWK Relational Operators
awk uses the following relational operators
Relational Operators
Operator
Meaning
Equal to
==
!=
>
>=
<
<=
~
!~
Not Equal to
Greater than
Greater than or equal to
Less than
Less than or equal to
Matches
Does not match
Logical Operators
Operator
Meaning
AND
&&
||
!
OR
NOT
Mohamed Mukthar Ahmed
Page 5
Formatted Output
awk uses the printf(
printf( ) for formatted output.
It is similar to that of C programming.
# Formatted output
/mukthar/ { printf(UID = %d\tGID = %d\n,$3, $4) }
Mohamed Mukthar Ahmed
BEGIN and END Statements
The keywords BEGIN and END are used to perform
specific actions relative to the programs execution.
BEGIN Action before the first input line is read.
END Action after all input lines have been
processed
# myawk4
# Searching for a pattern mukthar
BEGIN { print Locating User mukthar }
/mukthar/ { printf(UID = %d\tGID = %d\n, $3, $4) }
END { print End of Report }
$ awk F: -f myawk4
/etc/passwd
Mohamed Mukthar Ahmed
Page 6
AWK Defined Variable
awk supports a number of prepre-defined variables.
PrePre-defined Variables
Variable
Meaning
The current input line.
NR
NF
Number of fields in the input line.
# myawk5
# Number of valid users
END { print There are , NR , users }
$ awk F: -f myawk5
/etc/passwd
Mohamed Mukthar Ahmed
User Defined Variable
awk supports the use of variables.
There is no need to explicitly initialize the variable to
zero, awk does this by default.
# myawk5b
# Counting users of training group
$4 == 505 { training++ }
END { print The number of users in ;
print training group are , training
}
$ awk F: -f myawk5b
/etc/passwd
Mohamed Mukthar Ahmed
Page 7
Regular Expressions
awk provides pattern matching which is more
comprehensive. These patterns are called regular
expressions.
expressions.
Similar to those supported by UNIX / Linux grep
command.
# myawk6
# Searching for a user1 to user5
BEGIN { print Locating User1 to User5 }
/^user[1-5]/ { print $1, $3, $4 }
END { print End of Report }
$ awk F: -f myawk6
/etc/passwd
Mohamed Mukthar Ahmed
Regular Expressions
Use the ~ or !~ match operators for matching.
# myawk7
# Searching for UIDs 501-505
BEGIN { print Locating UIDs 501-505 }
$3 ~ /50[1-5]/ { print $1, $3, $4 }
END { print End of Report }
Pattern
Meaning
. DOT any one character
\
[ ]
[^
[^]
^
$
despecialize character
any one character in the list. Character Class
any one character not in the list
beginning of line
Also called as anchors
end of line
Mohamed Mukthar Ahmed
Page 8
Regular Expressions
We can match a repeating pattern by adding a
modifier or repetition operator.
operator.
Regular expressions can have any of the three
modifiers.
modifiers.
Modifier
Meaning
?
Match at most once the preceding character
*
+
{n}
Match preceding character exactly n times
{n,}
Match preceding character at least n times
{n,m}
n,m}
zero or more occurrences of preceding character
one or more occurrence of preceding character
Match preceding character at least n times but
not more than m times.
Alternative patterns can be specified by a
alternate separator | (pipe)
Mohamed Mukthar Ahmed
Concatenation Operator
The plus (+
(+) symbol concatenates one or more
strings in pattern matching.
# myawk8
# Searching for a pattern $unix
BEGIN { print Locating $Unix or $unix }
$1 ~ /\$+[Uu]nix/ { print $0 }
END { print End of Report }
awk interprets any string or variable on the right
side of ~ or !~ as a regular expression.
Thus, the regular expression can be assigned to a
variable, and the variable can be used in pattern
matching.
Mohamed Mukthar Ahmed
Page 9
Combining Patterns
Patterns can be combined to provide more
powerful and complex matching.
# myawk9
# Combineing patterns
BEGIN { print Combining Patterns }
$1 == 486 && $5 > 250 { print $0 }
END { print End of Report }
awk pattern range can be specified by having two
patterns separated by a comma.
The action is performed for each input line between
the occurrence of the first and second pattern.
# myawk9b
/user1/,/user8/ { print $0 }
Mohamed Mukthar Ahmed
AWK Defined Variable - Again
awk supports a number of prepre-defined variables.
Variable
NR
NF
FS
FILENAME
FNR
OFS
ORS
ARGC
ARGV
PrePre-defined Variables
Meaning
The current input line.
Number of fields in the input line.
Input field separator.
Name of current input file.
Record number in current input file.
Output field separator.
Output record separator.
Number of command line arguments.
Array of command line arguments.
Mohamed Mukthar Ahmed
Page 10
Examples
Examples on using prepre-defined variables of awk.
awk.
# myawk10
# Print the first five input lines.
FNR == 1, FNR == 5 { print $0 }
# myawk11
# Print each input line with line number.
# Print the heading with file name.
BEGIN { print "File :", FILENAME }
{ print NR, ":\t", $0 }
# myawk12
BEGIN { print "There are ", ARGC, "parameters on the
command line";
print "The first argument is ", ARGV[0];
print "The second argument is ", ARGV[1];
}
Mohamed Mukthar Ahmed
AWK Built In Arithmetic Functions
Following is a summary of awk
awks builtbuilt-in arithmetic
functions.
All operations are done in floatingfloating-point format.
Name
int(x)
int(x)
sqrt(x)
sqrt(x)
rand(x)
rand(x)
srand(x)
srand(x)
exp(x)
exp(x)
log(x)
log(x)
sin(x)
sin(x)
cos(x)
cos(x)
Arithmetic Functions
Description
Integer part of x
Square root of x
Random number between 0 and 1
x is a new seed for rand( )
Exponential function of x
Natural Logarithm of x
Sine of x, with x in radians
Cosine of x, with x in radians
Mohamed Mukthar Ahmed
Page 11
Examples
Examples on builtbuilt-in functions of awk.
awk.
# myawk13
# Print the square root of input value
{ print sqrt( $1 ) }
$ awk f myawk13
2
1.41421
3
1.73205
4
2
If no data file is specified, awk reads from the stdin
input file (i.e. keyboard)
keyboard)
Mohamed Mukthar Ahmed
AWK Built In String Functions
Following is a summary of awk
awks builtbuilt-in string
functions.
Strings are enclosed within quotes ( )
Name
String Functions
Description
Return length of s
length(s)
length(s)
Returns substring of s from position p
substr(s,p)
substr(s,p)
substr(s,p,n)
substr(s,p,n) Returns substring of s from position p of
index(s,t)
index(s,t)
match(s,t)
match(s,t)
split(s,a)
split(s,a)
split(s,a,fs)
split(s,a,fs)
length n
Returns position of t in string s
Returns position of t in string s
Splits s into elements of a defined by FS
Splits s into elements of a defined by fs
Mohamed Mukthar Ahmed
Page 12
AWK Built In String Functions
Following is a summary of awk
awks builtbuilt-in string
functions.
Strings are enclosed within quotes ( )
Name
gsub(r,s)
gsub(r,s)
gsub(r,s,t)
gsub(r,s,t)
String Functions
Description
Substitutes s in place of r in $0 globally.
Returns the number of substitutions made
Substitutes s in place of r in t. Returns the
number of substitutions made.
Substitutes s for first r.
sub(r,s)
sub(r,s)
Substitutes s for first r in t.
sub(r,s,t)
sub(r,s,t)
sprintf(fmt Returns expression list formatted according
to format string specified by fmt.
fmt.
,expr,expr-lst)
lst)
Mohamed Mukthar Ahmed
Control Flow Statements
awk provides constructs to implement selection and
iteration.
iteration.
awk control flow statements are similar to C
language constructs.
# myawk15
# Finding biggest disk
{
if (disksize < $5 )
{
disksize = $5;
computer = $0
}
}
END
{ print
computercomp_data
}
$ awk
-f myawk15
Mohamed Mukthar Ahmed
Page 13
Control Flow Statements - Examples
Examples
# myawk16
# To print out each second field for 286 computers
BEGIN { printf("Type\tLoc\tDisk\n"); }
/286/ { field = 1;
while( field <= NF )
{
printf("%s\t", $field);
field += 2;
}
print "";
}
$ awk
-f myawk16
comp_data
Mohamed Mukthar Ahmed
Control Flow Statements - 2
We are already familiar with break and continue
statements.
The next statement skips to the next input line then
restarts from the first patternpattern-action statement.
# myawk17
# Print out computer type 286 using next
{
while($1 != 286)
next;
print $0
}
$ awk
-f myawk17
comp_data
Mohamed Mukthar Ahmed
Page 14
Arrays in AWK
awk provides single dimensioned arrays.
Arrays need not be declared, they can be created.
Arrays are heterogeneous.
heterogeneous.
Array subscripts are strings.
strings.
# Array Examples
ARRAY[e001] = 100
print ARRAY[e001]
ARRAY[
ARRAY[num
num] and ARRAY[num] are not the same.
ARRAY[1] and ARRAY[
ARRAY[1] is the same.
Mohamed Mukthar Ahmed
Arrays - Examples
Examples
# myawk18
# diskspace[] holds the sum of the disk space for all
# computers
# computers[] holds number of computers of specific type
$1 == "486" { computers["486"]++ }
$5 > 0
{ diskspace[0] += $5 }
END {
print "Number of 486 computers :" ,
computers[486];
print "Total disk space :",
diskspace[0];
}
$ awk
-f myawk18
comp_data
Mohamed Mukthar Ahmed
Page 15
Arrays in AWK
awk array checking.
To check for a subscript in an array, awk provides us
the in operator.
To remove an array element, awk provides us the
delete operator.
# Array Examples
if (e001 in ARRAY)
delete ARRAY[e001]
Mohamed Mukthar Ahmed
for Statement
awk provides a for construct for handling arrays
iterations.
Syntax:
for ( var in array ) statement(s)
The var get one element at a time from the array and
executes the statement until the array elements are
not exhausted.
Mohamed Mukthar Ahmed
Page 16
for - Examples
Examples
# myawk19
# Counting number of computers of all types
{ computers[$1]++ }
END {
for(name in computers)
print Number of , name, computers
is , computers[name]
}
$ awk
-f myawk19
comp_data
Mohamed Mukthar Ahmed
User Defined Functions
awk supports user defined functions.
Syntax:
function name(arg_list)
{
statements
}
There must be NO space between the function name
and the left bracket of the argument list.
The return statement is used to return a value by the
function.
Mohamed Mukthar Ahmed
Page 17
UDF - Examples
Examples
# myawk20
# Finding factorial
function factorial(n) {
if(n<=1) return 1
else return n*factorial(n-1)
} #End of function
{
print Factorial of,$1,is,factorial($1)
}
$ awk
-f myawk20
Mohamed Mukthar Ahmed
Output To File
awk output generated by print or printf can be
redirected to a file by using the redirection concept.
concept.
The name of the file MUST be in quotes.
# myawk21
# Output to file
$1 == "486" {
print "Type = ", $1 , "Location = ", $3
}
$ awk
> "comp486.dat"
-f myawk21 comp_data
The output of awk programs can be piped into a
UNIX / Linux command. Termed as Output To Pipes
Pipes
Mohamed Mukthar Ahmed
Page 18
AWK Input getline Function
awk getline( ) function reads input from the following
Current Pipe
Current File
Specific File
Internal Pipe
# myawk22
# Using getline function
if ($1 == "486) {
FIRSTLINE = $0;
getline;
SECONDLINE = $0
}
$ awk
-f myawk22 comp_data
Mohamed Mukthar Ahmed
AWK Input getline Function
Reads next line, sets $0 and NF.
increments NR and FNR
Moreover,
# myawk23
{
print NR, $0
getline;
print NR, $0
}
$ echo 100 200 300 400 500 600 | awk
-f myawk23
Mohamed Mukthar Ahmed
Page 19
AWK Input getline Function
getline var
FNR
Read next line into var,
var, increment NR
getline < file Read next line from file.
file. Set $0, NF
# myawk24
# Using getline function
while( getline < data )
print $0;
$ awk
-f myawk24
Mohamed Mukthar Ahmed
AWK Input getline Function
getline var < file Read next line from file into var,
var,
increment NR FNR
cmd | getline Read next line from cmd.
cmd. Set $0, NF
cmd | getline var Read next line from cmd into var
# myawk25
# Using getline function
while( who | getline )
print user , $1, tty, $3;
$ awk
-f myawk25
Mohamed Mukthar Ahmed
Page 20
