- Hi, and welcome to the next video of section five.
A Primer on cURL.
In the previous video,
we've learned how to download a webpage's plain text.
In this video, we'll learn a list of the most important
features of cURL.
cURL is a powerful utility that support protocols
such as HTTP, HTTPS, and FTP.
It supports many features including posts, cookies,
authentication, downloading partial files from a specific
offset, referer, user agent string,
extra headers, limit speed, maximum file size,
progress bar, and so on.
cURL is very useful when we play around with automating
a webpage usage sequence, and to retrieve data.
cURL mostly doesn't come with any Linux distros.
You may have to install it by using the package manager.
By default, distribution ship with Wget.
cURL usually dumps the downloaded files to standard out,
and progress information to standard error.
To avoid the progress information from being shown,
we use the --silent option.
Let's see how to do it.
The cURL command can be used to perform different activities
such as downloading, sending different HTTP requests,
and specifying HTTP headers.
Let's see how we can perform different tasks with cURL.
To dump the downloaded file into the terminal,
the downloaded data is written to standard out,
use the following command.
To prevent the cURL command from displaying
the progress information mentioned, --silent.
To write the downloaded data into a file with the filename
passed from the URL, rather than writing to standard output,
use the -O option.
To show the hash progress bar while downloading,
use the --progress instead of --silent.
Let's see how it works.
cURL write a webpage or file to the filename as in the URL.
Instead of writing to standard out.
If the filenames are not present in the URL,
it will produce an error.
Hence, make sure the URL is a URL to remote file.
This will display an error since the filename
cannot be passed from the URL.
In such cases, we can manually specify the filename
as follows by using the -O option.
In the proceeding sections, we learned how to download files
and dump HTML pages to the terminal.
There are several advanced options
that come along with cURL.
Let's explore more of cURL.
Continuing and resuming downloads.
cURL has advanced resume download features
to continue at given offset.
It helps to download portions of files
by specifying an offset.
The offset is an integer value in bytes.
cURL doesn't require us to know the exact byte offset,
if we want to resume downloading a file.
If you want cURL to figure out the correct resume point,
use the -C option as follows.
cURL will automatically figure out where to restart
the download of the specified file.
Setting the referer string with cURL.
Referer is a string in the HTTP header
used to identify the page from which the user reaches
the current webpage.
When a user clicks on a link on a webpage A
to go to webpage B, the referer header string for webpage B
will contain the URL of page A.
Some dynamic pages check the referer string
before returning the HTML data.
For example, a webpage shows a Google logo attached page
when a user navigates to a website by searching on Google,
and shows a different page when the user navigates
to the webpage by manually typing in the URL.
The offer of a website can write a condition to return
a Google page if the referer is www.google.com,
or else return a different page.
You can use --referer with the cURL command
to specify the referer string as follows.
For example, cookies with cURL.
By using cURL, we can specify as well as store
the cookies that are encountered during HTTP operations.
To specify cookies, use the --cookie
"cookies" options.
Cookies should be provided as name=value.
Multiple cookies should be delimited by a semicolon.
For example, to specify a file to which the cookies
encountered are to be stored,
use the --cookie-jar option.
For example, setting a user agent string with cURL.
Some webpages that check the user agent won't work
if there is no user agent specified.
You must have noticed that certain old websites only work
on Internet Explorer, IE.
If a different browser's used, it will show a message
that it will only work on IE.
This is because the website checks for a user agent.
You can set the user agent with cURL.
By using cURL, it can be set using
- -user-agent,
or -A as follows.
Additional headers can be passed with cURL.
Use -H header to pass multiple additional headers.
For example, there are many different user agent strings
across multiple browsers and crawlers on the web.
You can find the list of some of them at this site.
Specifying a bandwidth limit on cURL.
When the available bandwidth is limited
and multiple users are sharing the internet,
in order to share bandwidth smoothly,
we can limit a download rate to a specified limit in cURL
by using the --limit-rate option as follows.
In this command, k, kilobyte,
specifies the download rate limit.
You can also use m for megabyte.
Specifying the maximum download size.
The maximum download file size for cURL
can be specified using the
- -max-filesize option as follows.
It will return a non zero exit code if the filesize exceeds.
It will return zero if it succeeds.
Authenticating with cURL.
HTTP authentication or FTP authentication can be done
by using cURL with the -U argument.
The username and password can be specified
by using the -U username code on password.
It is not possible to provide a password in such a way
that it will prompt for the password when executing.
For example, if you prefer to be prompted for the password,
you can do that using only the -U username.
Printing response headers excluding the data.
It's useful to print only response headers
to apply mini checks or statistics.
For example, to check whether a page is reachable or not,
we don't need to download the entire page contents.
Just reading the HTTP response header can be used
to identify whether a page is available or not.
An example use case for checking the HTTP header,
is to check the filesize before downloading.
We can check the content hyphen length parameter
in the HTTP header to find out the length
of a file before downloading.
Also, several useful parameters
can be retrieved from the header.
The last hyphen modified parameter enable us to know
the last modification time for the remote file.
Use the -I or -head option with cURL
to dump only the HTTP headers
without downloading the remote file, for example.
Awesome! We've successfully learned about a primer on cURL.
In the next video,
we'll learn about parsing data from a website.