- [Instructor] Welcome to the next video of section five,
image crawler and downloader.
In the previous video we've seen
how to parse data from a website.
In this video we'll see how to write a script
to parse the image files and download them automatically.
Image crawlers are very useful when we need to download
all the images that appear in a webpage.
Instead of going through the html sources
and picking all the images,
we can use a script to parse the image files
and download them automatically.
Let's see how to do it.
Let's write a bash script to crawl and download the images
from a website as follows.
The preceding image downloader script parses an html page,
strips out all the tags except image,
then parses source=URL from the image tag
and downloads them to the specified directory.
This script accepts a webpage URL
and a destination directory path as command line arguments.
The statement checks whether the total number
of arguments to the script is three,
otherwise it exits and returns a usage example.
If there are three arguments,
we parse the URL and destination directory.
This is done as follows.
A while loop is used.
It runs as long as there are more arguments to be processed.
The case statement will evaluate
the first argument, dollar one,
and matches -d or any other string arguments it checked.
The advantage of parsing arguments in this way
is that we can place the -d argument
anywhere in the command line.
Shift is used to shift arguments to the left
in such a way that when shift is called,
dollar one will take the next arguments behave,
that is, dollar two and so on.
Hence we can evaluate all arguments
through dollar one itself.
When -d is matched, it is obvious that the next argument
is the value for the destination directory.
Star corresponds to a default match.
It will match anything other than -d.
Hence while iteration dollar one equals quote
or dollar one equals URL in the default match,
we need to take the dollar one URL,
avoiding quotes to overwrite.
Hence we use this expression.
It will return a URL value if already not quotes,
otherwise it will assign dollar one.
This will print only the matching strings,
which are the image tags, including their attributes.
It is used to match all the characters
except those in the closing greater than.
That is, parses source=URL, so that all the image URLs
can be parsed from the image tags already parsed.
There are two types of image source paths,
relative and absolute.
Absolute paths contain full URLs that start with...
Relative URLs start with / or image_name itself.
An example of an absolute URL is example.com/image.jpg.
An example of a relative URL is /image.jpg.
For relative URLs the starting / should be replaced
with the base URL to transform it into example.com/image.jpg
For this transformation we initially find out
the base URL by parsing,
and then in every URL that starts with /
we replace it with base URL.
Then a while loop is used to iterate each line of the list
and download the images by using curl.
The minus minus silent argument is used with curl
to avoid extra progress messages
from being printed on the screen.
Awesome.
We have successfully learned
about image crawler and downloader.
In the next video,
we'll learn about web photo album generator.