Linux Challenge

Now it's your turn to put into practice what we've gone through. Answer as many of the questions as you can.

Contents
Linux Challenge ..............................................................................................................................................................1
Objective....................................................................................................................................................................1
Requirements .......................................................................................................................................................1
Pre-Task configuration ..............................................................................................................................................1
Task 1.........................................................................................................................................................................3
Objective ..............................................................................................................................................................3
Instructions...........................................................................................................................................................3
Team Questions....................................................................................................................................................4
Tasks 2 - 6 .......................................................................................................................................................................5
Objective....................................................................................................................................................................5
Task 2.........................................................................................................................................................................5
Task 3.........................................................................................................................................................................6
Task 4.........................................................................................................................................................................7
Task 5.........................................................................................................................................................................8
Task 5.........................................................................................................................................................................9

Objective
You have 2 hours to complete the following tasks using the commands that you have learned this morning.

There are 6 tasks in total, some of which may contain multiple steps.

If you have any issues, or need clarification please ask your allocated instructor.

You must make sure that your allocated instructor knows your Meeting room link for when you need help, and ask
the instructor to join your meeting in the main learning room when you need help. They may already be helping
someone, so you can always leave a message for them.

Requirements
You should ensure that you attempt as many of the tasks as possible, all of them if you can.

You will need to record your commands and most of the outputs in a document that you can send to the instructors
to review.

IMPORTANT: you should be prepared to present your answers toward the end of the day. You may be selected to
present a particular task, not all of them, but you should complete as many as you can.

Pre-Task configuration
Make sure you have access to the log file named ExampleFinance.log. Ask an instructor if you are unsure where to
find this. All of the following tasks use this file.
Task 1
Exploring the Linux file system, using variables and running scripts.
You will get 15 minutes to workout the game and play it.
We will then regather for a review of your map, explanation and score - for 15 minutes

Objective
The object of this exercise is to gain some hands-on experience with navigating a Linux filesystem and some basic
bash commands and operations such as ls, cd, cat, environmental variables, running a script.
For this exercise we will be using a very simple command line “adventure game” called bashcrawl. This is simply a
directory structure containing some very simple bash scripts and text files. This directory structure is organised in
such a way so that you can navigate it like a very simple command line “game”.

For this exercise you will be in breakout rooms of small teams. Each person should download the filesystem onto
their Linux machine using the “Git” tool (instructions below). You will then each navigate the filesystem according
to the instructions given. There are some questions below that we would like each team to answer at the end of the
exercise. While each person should download and explore the filesystem individually, you should work as a team to:
1. Introduce yourselves to each other!
2. Ensure that everyone on your team understands what you’re doing, if you’ve got no idea within the team then
ask an instructor to come into your breakout room.
3. Work together to answer the questions at the end of the exercise

Instructions
The instructions below give you the steps to get started with bashcrawl, try to understand what you’re actually doing
rather than blindly following the steps. Talk to your teammates if you’re unsure about any particular step
1. On your linux machine, go to your home directory and then download the bashcrawl filesystem with git:
◦ cd
◦ git clone https://gitlab.com/slackermedia/bashcrawl.git
2. You will now have a directory called bashcrawl, move into it and read the README.md file with less or cat
◦ cd bashcrawl
◦ less README.md
◦ When reading a file with less use the arrows to move around and press ‘q’ to exit
3. To start your ‘adventure’ move into the ‘entrance’ directory and read the ‘scroll’ text file with cat. Each time
you move to a new directory there should be a text file called ‘scroll’ that should give you instructions.
◦ cd entrance
◦ cat scroll
4. Some directories have a bash script e.g. ‘treasure’ or ‘potion’. The instructions in the ‘scroll’ files will tell you
what to do with these.
Team Questions
1. We would like a map of the entire game as a directory tree. You can create this in a text file or draw in white-
board or whatever your team prefers.
2. How do the script files (treasure, potion etc.) produce text and give an interactive prompt
3. When you first `cd entrance` you will notice there is only a ‘cellar’ subdirectory. Later there is more subdirecto-
ries e.g. entrance/chapel How are these new subdirectories created?
4. Try to understand how the game works so that as a team you can explain any questions the instructors may ask.
5. Bonus points for the team with the highest health points and biggest inventory at the end of the game.
Tasks 2 - 6
Objective
The next tasks are all about dealing with log files and working out the format of the file, extracting data and present-
ing it. For each of the tasks we want you to document the commands used and your output.

Task 2
In this task we want you to view the data in the log file in different ways so that you get to understand the format of
the log file. Once you think you have the format of the logfile write out the syntax for it.

1. View the ExampleFinance.log file page by page, making note of the lines and working out what the format of
the line is. You should look for the default field separator and whether the log file has multiple lines for a mes-
sage, or whether each line is a unique message
2. Check that you have identified the correct field separator by using a command to count the number of lines that
contain your field separator
• ANSWER: grep '|' ExampleFinance.log | wc -l Output = 100949
3. Check that there are not any lines that do not contain the field separator
• ANSWER: grep -v '|' ExampleFinance.log | wc -l Output = 0
4. How many lines are there in the file in total?
• ANSWER: wc -l ExampleFinance.log Output = 100949
5. Write down your idea of what each column of the log file is called;
• ANSWER:
• Column 1 = Date and time of the entry
• Column 2 = Command or Action and ID in square brackets
• Column 3 = Message type, error, info, notice, warning
• Column 4 = The message, some of the messages have multiple | in their field and are made up as;
o Column 5 = time of transaction in epoch
o Column 6 = stock
o Column 7 = Market
o Column 8 = text
o Column 9 = message
Task 3
Now that you have worked out the format of the data we would like you to provide a summary of the following. In
your work you should tell us;
1. The command you ran
2. The result of the command

• Tell me how many of each type of message there are in the log file and sort them biggest first;
• ANSWER: awk -F"|" '{print $3}' ExampleFinance.log | sort | uniq -c | sort –k1nr
o 100276 notice
o 395 warning
o 228 error
o 50 info
• Tell me the top 5 times with the highest number of messages in the log file that occurred at the same time
on specific days, with the highest being printed first.
• ANSWER: awk -F'|' '{print $1}' ExampleFinance.log | sort | uniq -c | sort -k1nr | head –5
o 110 2013/01/18 05:13:32.263
o 110 2013/01/18 05:13:32.264
o 109 2013/01/18 05:13:32.265
o 109 2013/01/18 05:13:32.274
o 108 2013/01/18 05:13:32.272
• Using your command in 2, change it so that you get the bottom 5
• ANSWER: awk -F'|' '{print $1}' ExampleFinance.log | sort | uniq -c | sort -k1nr | tail –5
• 1 2013/01/18 11:24:48.457
• 1 2013/01/18 11:24:49.001
• 1 2013/01/18 11:24:50.428
• 1 2013/01/18 11:24:52.011
• 1 2013/01/18 11:24:52.417
• Using your commands in 2 and 3, write the outputs so that they are both in a file called topbottomtimes.txt
• ANSWER:
• awk -F'|' '{print $1}' ExampleFinance.log | sort | uniq -c | sort -k1nr | head -5 > topbottomtimes.txt
• awk -F'|' '{print $1}' ExampleFinance.log | sort | uniq -c | sort -k1nr | tail -5 >> topbottomtimes.txt
Task 4
Data separation. In this task we want you to separate out data into separate files.
• Separate out each type of message into its own file.
◦ e.g. all lines with errors into a file called errors.log, all lines with info into a file called info.log, etc
◦ ANSWER:
▪ grep error ExampleFinance.log >errors.log
▪ grep warning ExampleFinance.log >warning.log
▪ grep notice ExampleFinance.log >notice.log
▪ grep info ExampleFinance.log >info.log
• Let's check to make sure that we only have errors for error messages. Construct a command that will check
this for us, and provide us with a short summary output, e.g. tells us how many of each message.
◦ ANSWER: awk -F'|' '{print $3}' errors.log | sort | uniq –c
▪ 228 error
▪ 2 warning
• Check the other files too
◦ ANSWER: Only the outputs here;
▪ Warning = 395 warnings only
▪ Info = 50 info, 3416 notice
▪ Notice = 100276 notice only
• How many lines are there in each of the files? Do this in one command line
◦ ANSWER:
▪ wc -l error.log notice.log warning.log info.log
▪ 230 errors.log
▪ 100276 notice.log
▪ 395 warning.log
▪ 3466 info.log
▪ 104367 total
Task 5
Earlier we asked you to look for those lines that contained the extra field separators which were financial trading
information.
• Is there a correlation between the lines in the log file that contain the word time= and the type of notice?
◦ ANSWER:
▪ awk -F'|' '$5 ~ /^time/ {print $0}' ExampleFinance.log | wc –l
▪ awk -F'|' '$5 ~ /^time/ {print $0}' ExampleFinance.log | grep notice | wc –l
▪ Both commands return the same number 3728
▪ Yes, all financial transaction information in the log file, that is those lines containing the word
time= are all of type notice
• What are the 3 eventTypes in the log file?
◦ ANSWER: awk -F'|' '$5 ~ /^time/ {print $4}' ExampleFinance.log | sort | uniq
▪ alert
▪ instrumentStatus
▪ marketPhase
Task 6
In this task we're upping the anti on the commands. Where a command has lots of output we only want the com-
mand. If the command is less then 10 lines then show the command and output.

1. Some of the lines in the log file have extra columns, 5 - 8. Compose a command that will only show me
the lines that have the extra columns;
• ANSWER: awk -F'|' '$5 ~ /^time/ {print $1,$2,$3,$4,$5,$6,$7,$8}' ExampleFinance.log
2. How many unique symbols are there in the data?
• ANSWER: awk -F'|' '$5 ~ /^time/ {print $6}' ExampleFinance.log | sort | uniq | wc –l
• 2111
3. How many symbols are there in total?
• ANSWER: awk -F'|' '$5 ~ /^time/ {print $6}' ExampleFinance.log | wc –l
• 3728
4. From Task 3 you wrote some commands to extract the different types of messages from the file. Now we
want you to extract only those message types without grabbing any of the other lines which did not belong
to that category, e.g. errors.log should only have error messages, etc. To help with this question you can
use $0 in AWK to print the entire line.
• ANSWER:
o awk -F'|' '$3 ~ /error/ {print $0}' ExampleFinance.log >errors.log
o awk -F'|' '$3 ~ /notice/ {print $0}' ExampleFinance.log >notice.log
o awk -F'|' '$3 ~ /warning/ {print $0}' ExampleFinance.log >warning.log
o awk -F'|' '$3 ~ /info/ {print $0}' ExampleFinance.log >info.log
5. Reformatting data. Create a separate file for the instrumentStatus eventType, but;
◦ include a single line header for each field to the top of the file, comma seperated
◦ the output for each line should be comma separated
◦ reduce the amount of data per line to;
▪ Date and Time, symbol, currency, orderEntry
▪ You should remove the fieldname= part so that we are only left with the value
◦ the file should be named after the eventType with the extension being .csv
◦ HINT: AWK has a section called BEGIN{ }, and in there you could print your header in double quotes
".
▪ ANSWER: grep eventType=instrumentStatus ExampleFinance.log | awk -F '|' 'BEGIN{print
"Date & Time,Symbol,Currency,Order Entry"}{print $1","$6","$8","$10}' | sed 's/ *,[a-zA-
Z]*=/,/g'