- [Instructor] Hello, and welcome to the next video
of Section Four, Using awk for Advanced Text Processing.
In the previous video, we have learned how to use sed
to perform text replacement.
In this video, we will see how to use awk
for advanced text processing.
Awk is a tool designed to work with data streams.
It's very interesting, as it can operate
on columns and rows.
It supports much built-in functionality, such as arrays
and functions in the C programming language.
Its biggest advantage is its flexibility.
The structure of an awk script is as follows.
The awk command can read from standard in, also.
An awk script usually consists of three parts:
begin, end, and a common statement block with
the pattern match option.
The three of them are optional, and any of them can
be absent in the script.
Let's write a simple awk script enclosed in single quotes,
or double quotes, as follows.
Or, alternatively, use the following command, for example...
Let's see how it works.
The awk command works in the following manner.
Execute the statements, and they begin.
Brackets commands block.
Read one line from the file of standard in,
and execute pattern, brackets commands.
Repeat this step until the end of the file is reached.
When the end of the input stream is reached,
execute the end commands block.
The begin block is executed before awk starts reading
the lines from the input stream.
It's an optional block.
The statements, such as variable initialization now,
and printing the output header for an output table
are common statements that are written in the begin block.
The end block is similar to the begin block.
It gets executed when an awk completes reading
all the lines from the input stream.
The statements, such as printing results after analyzing
all the values calculated for all the lines,
or printing the conclusion, are the commonly used statements
in the end block.
For example, after comparing all the lines,
print the maximum number from a file.
This is an optional block.
The most important block, is of the common commands,
within the pattern block.
This block is also optional.
If this block is not provided, by default,
print gets executed, so as to print each of the lines read.
This block gets executed for each line read by awk.
It's like a wire loop for lines read
with statements provided inside the body of the loop.
When a line is read, it checks whether the provided pattern
matches the line.
The pattern can be a regular expression match, conditions,
range of lines match, and so on.
If the current read line matches with the pattern,
it executes the current statements
enclosed in curly brackets.
The pattern is optional.
If it is not used, all the lines are matched,
and the statements inside curly brackets are executed.
Let's go through the following example.
When print is used without an argument, it will print
the current line.
There are two important things to be kept in mind about it.
When the arguments of the print are separated by commas,
they are printed with a space delimiter.
Double quotes are used as the concatenation upright
in the context of print in awk.
For example...
The preceding statement will print the values
of the variables as follows.
The echo command writes a single line into
the standard output.
Hence, the statements in the curly brackets block
of awk are executed once.
If the standard input to awk contains multiple lines,
the commands in awk would be executed multiple times.
Concatenation can be used as follows.
The output will be as follows.
Curly brackets is like a block in a loop,
iterating through each line of a file.
Usually, we place initial variable assignments,
such as var=0, and like statements, print the file header,
in the begin block.
In the end block, we place statements
such as printing results.
The awk command comes with a lot of rich features.
In order to master the art of awk programming,
you should be familiar with
the important awk options, and functionalities.
Let's go through the essential functionalities of awk.
Special variables.
Some special variables that can be used with awk,
are as follows.
NR: it stands for the current record number,
which corresponds to the current line number
when it uses lines as records.
NF: it stands for the number of fields,
and corresponds to the number of fields
in the current record under execution.
Fields are delimited by space.
$0: it's a variable that contains the text content
of the current line under execution.
$1: it's a variable that holds the text of the first field.
$2: it's the variable that holds the text
of the second field.
For example...
We can print the last field of a line as print$NF last,
but the second as $[NF-1], and so on.
Awk also provides the print F function with
the same syntaxes in C.
We can also use that instead of print.
Let's see some basic awk usage examples.
Print the second and third field of every line as follows.
In order to count the number of lines in a file,
use the following command.
Here, we only used the end block.
NR will be updated on entering each line by awk
with its line number.
When it reaches the end of a line, it will have
the value of the last line number.
Hence, in the end block, NR will have the values
of the last line number.
You can sum up all the numbers
from each line field one as follows.
Passing an external variable to awk,
by using the -V arguments, we can pass external values,
other than standard, into awk, as follows.
There is a flexible alternate method to pass
many variable values from outside awk.
For example...
When an input is given through a file,
rather than standard input, use the following command.
In the preceding method, variables are specified
as key value pairs, separated by a comma
and space, and [V1=$var1, V2=$var2,
and its command arguments to awk, soon after they begin,
currently brackets and end blocks.
Reading a line explicitly using getline.
Usually awk reads all the lines in a file by default.
If you want to read one specific line, you can use
the getline function.
Sometimes you may need to read the first line
from the begin block.
The syntax is getline var.
The variable var will contain the content for the line.
If getline is called without an argument,
we can access the content of a line by using $0, $1, and $2.
For example...
Filtering lines processed by awk with filter patterns.
We can specify some conditions for lines to be processed.
For example...
Setting delimiter for fields.
By default, the delimiter for fields is a space.
We can explicitly specify a delimiter by using
the -F delimiter.
We can set the output fields separated
by setting OFS=2" delimiter, in the begin block.
Reading the command output from awk, in the following code,
echo will produce a single blank line.
The command out variable will contain the output
of the command greproute/etc/password,
and it will print the line containing the route.
The syntax of reading out commands in a variable output
is as follows.
For example...
By using getline, you can read the output
of external shell commands in a variable called command out.
Awk supports associative arrays, which can use
the text as the index, using a loop inside awk.
String minute relation function in awk.
Awk comes with many
built-in string minute relation functions.
Let's have a look at a few of them.
This returns the string length.
This returns the position of which the search_string
is found in the string.
This stores the list of strings generated
by using the delimiter in the array.
This returns the substring created from the string
by using the start and end character offsets.
This replaces the first occurring regular expression match
from the string replacement_str.
This is similar to sub, but it replaces
every regular expression match.
This returns the result of whether
our regular expression, regex, match
is found in the string or not.
It returns a non-zero output if a match is found.
Otherwise, it returns zero.
Two special variables are associated with a match.
They are R start, and R length.
The R start variable contains the position at which
the regular expression match starts.
The R length variable contains the length of the string
matched by the regular expression.
Also, we've successfully learned how to use awk
for advanced text processing.
In the next video, we'll learn about finding the frequency
of words used in a given file.