Introduction

A Web crawler, sometimes called a spider, is an Internet bot that systematically browses the World Wide Web, typically for the purpose of Web indexing and data mining. (Wikipedia)

Web Crawler (Wikipedia)

Project #1

Search the class test web site. Collect and display:

• the Author's name (if any) and page count
• how many web pages do not have author meta data
• the number of links in a given page and
  plot the count information

Note: Before proceeding with this project, the class will need to build a test web site.

Links

Scrapy Home
Develop your first web crawler in Python Scrapy
3 Python web scrapers and crawlers