- [Instructor] Hi and welcome to the fifth section
of this course: "Tangled Web? Not at all!"
The web is becoming the face of technology
and the central axis point for data processing.
Though Shell Scripting cannot do everything
that languages like PHP can do on the web,
there are still many tasks to
which Shell Script are ideally suited.
In this section we will explore some videos
that can be used to pass website content,
download and obtain data, send data to forms,
and automate website usage tasks and similar activities.
We can automate many activities
that we perform interactively through a browser
with a few lines of scripting.
Access to the functionalities provided
by the HTTP protocol with command line utilities
enables us to write scripts that are suitable
for solving most of the web automation utilities.
Have fun while going through the videos in this section.
Now we move onto the first video of Section Five:
"Downloading from a Web Page".
In this video, we'll see how to download a file
or webpage from a given URL.
A few command line, download utilities
are available to perform this task.
Wget is a file download, command line utility.
It's very flexible and can be configured with many options.
Let's see how to do it.
A webpage or a remote file can be
downloaded by using wget as follows.
For example.
It's also possible to specify
multiple download URLs as follows.
Usually files are downloaded
but the same filename is in the URL.
And the download log information, or progress,
is written to standard out.
You can specify the output filename
with the "-o" option.
If the file with the specified filename already exists,
it will be truncated first and the downloaded file
will be returned to the specified file.
You can also specify a different log file path
rather than printing logs to standard out
by using the "-o" option.
By using the preceding command,
nothing will be printed onscreen.
The log, or progress, will be written to the log
and the output file will be "dloaded_file.img".
There is a chance that downloads might break
due to unstable internet connections.
In that case, we can use the number of tries as an argument.
So that once interrupted the utility will retry
the download that many times before giving up.
To specify the number of tries use the "-t" flag as follows.
Or to ask wget to keep trying indefinitely
use "-t" as follows.
The wget utility has several additional options
that can be used under different problem domains.
Let's go through a few of them.
Restricting the download speed.
When we have a limited internet bandwidth
and many applications are sharing the internet connection,
if a large file is given for download,
it will suck all the bandwidth
and may cause other processes to starve for the bandwidth.
The wget command comes with a built-in option
to specify the maximum bandwidth
the download job can possess.
Hence all the applications can simultaneously
access the internet fairly.
We can restrict these speed limits in wget
by using the "--limit-rate" argument as follows.
In this command, K, kilobyte, specifies the speed limit.
You can also use M for megabyte.
We can also specify the maximum quota for the download.
It will stop when the quota is exceeded.
It's useful when downloading multiple files
limited by the total download size.
This is useful to prevent the download
from accidentally using too much disc space.
Use "--quota" or "-Q" as follows.
Resume downloading and continue.
If a download using wget gets interrupted before it is
complete we can resume the download
from where we left off by using the "-C" option as follows.
Copying a complete website.
Mirroring.
Wget has an option to download the complete website
by recursively collecting all the URL links
in the webpages and downloading all of them like a crawler.
Hence we can completely download all the pages of a website.
In order to download the pages,
use the "--mirror" option as follows.
Or use the following command.
"-l" specifies the depth of webpages' levels.
This means it will traverse only that many levels.
It is used with the "-r" recursive.
The "-N" argument is used to enable time stamping
for the files.
"URL" is the base URL for a website
for which the download needs to be initiated.
The "-k" or "--convert-links" option
instructs wget to convert the links
to other pages in a downloaded page
to the local copy of those pages.
Exercise discretion when mirroring other websites.
Unless you have permission
only perform this for your personal use
and don't do it too frequently.
Accessing pages with HTTP or FTP authentication.
Some pages require authentication for HTTP or FTP URLs.
It can be obtained by using
the "--user" and "--password" arguments.
It's also possible to ask for a password
without specifying the password inline.
For this you use "--ask-password"
instead of the "--password" argument.
Great.
We've successfully learned about downloading from a webpage.
In the next video, we'll see
about downloading a webpage as plain text.