- [Instructor] Welcome to the next video of section five,
Finding Broken Links in a Website.
In the previous video, we've seen
how to create a defined utility by using the web backend.
In this video, we'll find the broken links
by using HTTP manipulation tools.
Some people manually check every page
on a website to search for broken links.
It's feasible for websites having very few pages,
but gets difficult when the number of pages becomes large.
It becomes really easy if we can automate
the process of finding broken links.
To identify the links and find
the broken ones from the links, we can use lynx and Curl.
It has an option, namely minus traversal,
which will recursively visit pages
on the website and build a list
of all hyperlinks in the website.
We can use Curl to verify each of the links
for whether they're broken or not.
Let's write a bash script with the help of Curl command
to find out the broken links on a web page.
Lynx minus traversal url will produce
a number of files in the working directory.
It includes a file reject.dat,
which will contain all the links in the website.
Sort -u is used to build a list by avoiding duplicates.
Then we iterate for each link and check
the header response by using curl -I.
If the header contains the first line
to have HTTP/1.0 200 OK as the response,
it means that the target is not broken.
All other responses corresponding
to the broken links are printed on the screen.
From its name, it might seem that rejects.dat should contain
a list of URLs which were broken or unreachable.
However, this is not the case.
A link just adds the all the URLs there.
Also note that lynx generates a file called traverse.errors,
which contains all the URLs that had problems in browsing.
However, lynx will only add URLs
which return HTTP 404 Not Found.
And so we'll lose other errors,
for instance, HTTP 403 Forbidden.
This is why we manually check for statuses.
Awesome, we successfully learned
how to find broken links in a website.
In the next video, we'll learn
how to track changes to a website.