0% found this document useful (0 votes)

31 views19 pages

Complete Unix and AWK Guide

The document is a comprehensive guide on Unix and AWK programming, providing practical examples and solutions for various data manipulation tasks using Unix commands. It covers core utilities such as grep, sort, uniq, and sed, along with detailed AWK command examples for data processing. Additionally, it includes exercises and case studies to reinforce learning and application of the concepts discussed.

Uploaded by

ragerspirit037

We take content rights seriously. If you suspect this is your content, claim it here.

Available Formats

Download as PDF, TXT or read online on Scribd

0% found this document useful (0 votes)

31 views19 pages

Complete Unix and AWK Guide

Uploaded by

ragerspirit037

We take content rights seriously. If you suspect this is your content, claim it here.

Available Formats

Download as PDF, TXT or read online on Scribd

You are on page 1/ 19

Complete Unix & AWK Programming Guide

1. Unix Basics with Examples

This section uses real datasets and output examples to demonstrate core Unix utilities like cat, grep, sort,

uniq, sed, etc., using practical data.

We'll use the fruits.txt dataset:

fruit_id,fruit_name,fruit_qty,unit_price,total_price

1,Mango,2,10,20

2,Apple,6,15,90

3,Banana,4,8,32

4,Watermelon,7,9,63

5,apple,3,15,45

Example:

cut -d',' -f2 fruits.txt

=> Output: Mango, Apple, Banana... (second column)

2. Examples from Unix 1 (1).txt with Explanations

Example 1: To Print the 2nd, 3rd and 5th Columns only

fruit_id,fruit_name,fruit_qty,unit_price,total_price

1,Mango,2,10,20

2,Apple,6,15,90

3,Banana,4,8,int

4,Watermelon,7,9,63

Solution:

awk 'BEGIN{

FS=",";

OFS="|";

}
{

print $2,$3,$5;

}' fruits.txt

--------------------------------------------------------------------------------------

Example 2: To Find the total price of an apple (PIPELINING grep & awk)

fruit_id,fruit_name,fruit_qty,unit_price,total_price

1,Mango,2,10,20

2,Apple,6,15,90

3,Banana,4,8,int

4,Watermelon,7,9,63

5,apple,3,15,45

Solution:

grep -i "[Aa]pple" fruits.txt | awk 'BEGIN{FS=",";s=0;}

s = s+$5;

END{

print "The total price spent on apple fruit is " s;

--------------------------------------------------------------------------------------

Example 3: Printing the Multiplication of Number 5

Solution:

awk 'BEGIN{

i=1;
while(i<=10)

print "5 * "i" = "i*5;

i++;

--------------------------------------------------------------------------------------

Example 4: Find the number of fields in a record and total no.of records present in the text file

mango

watermelon

pine apple

custard apple

banana

Solution:

awk 'BEGIN{FS=" ";}

print "The Number of fields present in record " NR " is " NF".";

END{

print NR" records are present.";

}' fruits2.txt

---------------------------------------------------------------------------------------------

Input Data:

29,Arun

26,Karthik
28,Kiran

52,Raju

78,Rachel

Example 5:

awk 'BEGIN{FS=",";}

if($1>50)

print "Value is greater than 50.";

else

print "Value is less than 50";

i=1;

while(i<=1)

print "Row " NR ", Column 2 (loop "i "): " $2;

i++;

END{

print match("End of Code", /of/);

print RSTART, RLENGTH;

}' example.txt

Example 6:

grep ',A' input3.txt | awk 'BEGIN{FS=",";total_age=0;}

{
total_age += $1;

count++;

END{

if(count>0)

avg_age = total_age/count;

printf "Average Age for Group A:%.2f\n", avg_age;

else

print "No Data Found";

--------------------------------------------------------------------------------------

ROYAL MAIL HOTEL (CASE STUDY)

employeeDetails.txt (DATA) :

Name,Age,Place,Experience,Salary

Anish,26,Chennai,2,10000

Jai,24,Chennai,2,10000

Kumar,29,Hyderabad,5,32000

John,32,Mumbai,2,11000

Neethu,21,Nagpur,3,13000

Satish,22,Ahmedabad,2,10000

Situation: To Print the complete data

awk 'BEGIN{FS=",";}

print;
}' employeeDetails.txt

Situation: Manager wish to display the employee name and salary working in royal mail hotel

awk 'BEGIN{FS=",";}

print $1,$5;

}' employeeDetails.txt

Situation: Manager wishes to print details of Kumar and Satish

awk '/Kumar|Satish/' employeeDetails.txt

Situation: Manager wishes to find the total expenses of hotel per month in the form of salary

awk -F"," 'BEGIN{

s=0;

s=s+$5;

END{

print "Total Exprenses per month in form of salary is " s;

}' employeeDetails.txt

Situation: Manager wishes to find the total no.of employees earning 10000 per month

awk '/10000/{

++count;

END{

print "No.of employees earning 10000 per month is " count;

}' employeeDetails.txt
Situation: Manager wishes to find the employees as best performers who completed 2 years of exprerience

and earning more than 10000

awk 'BEGIN{

FS=",";

print "Performance Report";

if(NR!=1)

if($4>=2 && $5>10000)

print $1" is a good performer";

else if($4==2 && $5<=10000)

print $1" Needs to improve";

END{

print "Performance Report";

}' employeeDetails.txt

-----------------------------------------------------------------------------------------

SORT COMMAND

input data file: list

1,John Cena,Title 758,Price $7.30

2,Randy Orton,Title 739,Price $6.20

4,Triple H,Title 893,Price $6.42

5,GoldBerg,Title 392,Price $1.98

input data file: list1

7,Shawn Michales,Title 620,Price $1.64

8,Roman Reigns,Title 920,Price $1.03

3,Brock Lesnar,Title 201,Price $6.71

6,Rey mysterio,Title 109,Price $12.4

Situation: Sort on the 2nd field of file named "list". File list is comma seperated value

Solution: sort -k 2 list;

Situation: Sort the input data file

Solution: sort list;

Situation: Sort can be applied on multiple files as well

Solution: sort -n list list1;

Situation: Sort in reverse order of first numeric column from multiple files

Solution: sort -nr list list1;

Situation: Sort the above input file by removing the duplicate records/lines.

Solution: sort -u list;

Situation: Sort two input files and merge it.

Solution: sort -m list list1;

---------------------------------------------------------------------------------------------------------

UNIQ COMMAND

input data file: list.txt

unix operating system

unix dedicated server

linux dedicated server

Situation: surpass the duplicate records

Solution: uniq list.txt

Situation: Pipelining command using (Sort and Uniq)

Solution: sort list.txt | uniq

Situation: Count the repeated lines

Solution: uniq -c list.txt

Situation: Display only the duplicate lines

Solution: uniq -d list.txt

Situation: Display all duplicate lines

Solution: uniq -D list.txt

Situation: Print only unique lines

Solution: uniq -u list.txt

------------------------------------------------------------------------------------------

GREP COMMAND

demo_file

THIS LINE IS THE 1ST UPPER CASE IN THIS FILE.

this line is the 1st lower case line in the file.

This Line Has All Its First Characters as Upper Case.

Two lines above this line is empty.

And this is the last line.

Situation: Search for a given string and also you can check a string in multiple files

Solution: grep "this" demo_file.txt

Situation: Search for a given string with caseinsensitive approach

Solution: grep -i "this" demo_file.txt

Situation: Search for a line starting with 'Two' and ending with 'empty'

Solution: grep "Two.*empty" example.txt

Situation: To Search for a word and to avoid it to match the substrings -w option is used

Solution: grep -w "is" example.txt

Situation: If you want to display the lines which does not matches the given string/pattern then

Solution: grep -v "Two" example.txt

-----------------------------------------------------------------------------------------------------------

SED COMMAND

file_name

1.Linux - System Admin, Scripting etc.

2.Database - Oracle, mySQL etc.

3.Hardware

4.Security (Firewall, Network etc.)

5.Storage

Operation in Sed

-p = Prints specific number of lines based on line number or pattern

-n = suppress automatic printing of patternspace. It will not print any thing until explicit request to print is

found

-d = delete the line when pattren matches

-i = modifies the text in the input file

SYNTAX:

sed -n '/pattern/'p file_name

Replaces the occurrences of 'Security' with 'security' from file_name.txt

sed 's/Security/security/g' file_name.txt

Prints lines containing 'Security' by duplicating from file_name.txt

sed '/Security/p' file_name.txt

Prints the lines containing 'Security' until explicit request to print is found

sed -n '/Security/p' file_name.txt

Prints lines 2 to 4 from file_name.txt

sed -n '2,4p' file_name.txt

Modifies the text 'Hardware' to 'hardwares' from file_name.txt

sed -i 's/Hardware/hardwares/g' file_name.txt

Prints every 2nd line starts from line 3.

sed -n '3~2p' file_name.txt

------------------------------------------------------------------------------AWK Command-------------------------------------

Regular Expressions

DOT

echo -e "cat\ncut\nfun\nfin\nfan" | awk '/c.t/'

START OF THE LINE

echo -e "This\nThat\nThere\nTheir\nthese" | awk '/^The/'

END OF THE LINE

echo -e "knife\nknow\nfun\nfin\nfan\nnine" | awk '/n$/'

MATCH CHARACTER SET

echo -e "Call\nTall\nBall" | awk '/[CT]all/'

EXCLUSIVE SET

echo -e "Call\nTall\nBall" | awk '/[^CT]all/'

ALTERATION

echo -e "Call\nTBall\nSmall\nShall" | awk '/Call|Ball/'

ZERO OR ONE OCCURRENCE

echo -e "Colour\nColor" | awk '/Colou?r/'

ZERO OR MORE OCCURRENCE

echo -e "ca\ncat\ncatt" | awk '/cat*/'

ONE OR MORE OCCURRENCE

echo -e "111\n22\n123\n234\n456\n222" | awk '/2+/'

GROUPING

echo -e "Apple Juice\nApple Pie\nApple Tart\nApple Cake" | awk '/Apple (Juice|Cake)/'

----------------------------------------------------------------------------------

USER DEFINED FUNCTIONS

# Returns minimum number

function find_min(num1, num2){

if (num1 < num2)

return num1

return num2

# Returns maximum number

function find_max(num1, num2){

if (num1 > num2)

return num1

return num2

# Main function

function main(num1, num2){

# Find minimum number

result = find_min(10, 20)

print "Minimum =", result

# Find maximum number

result = find_max(10, 20)

print "Maximum =", result

# Script execution starts here

BEGIN {

main(10, 20)

--------------------------------------------------------------------------------

CONTROL FLOW STATEMENTS

IF STATEMENT

awk 'BEGIN {num = 10; if (num % 2 == 0) printf "%d is even number.\n", num }'

IF-ELSE STATEMENT

awk 'BEGIN {

num = 11;

if (num % 2 == 0)

printf "%d is even number.\n", num;

else

printf "%d is odd number.\n", num;

IF-ELSE-IF LADDER

awk 'BEGIN {

a = 30;

if (a==10)

print "a = 10";

else if (a == 20)

print "a = 20";

else if (a == 30)

print "a = 30";

}'
3. AWK Command Exercises - Solutions and Explanations
Q1: Display names, skip header
Command:

awk -F@ 'NR > 1 { print $2 }'

Explanation:

This command uses '@' as the field separator and applies a condition to extract specific information.