Python Web Penetration Testing #3. Programming World!

Section 3: Web Crawling with Scrapy

3.1 Web Application Mapping

You’ll learn about web application mapping, what it is, and how to do it.
Revisit the penetration testing process
Check out the different options of performing the mapping

Check out a mapping example from Burp suite.

Link:
Watch Video

3.2 Creating a Crawler with Scrapy

We want to create a web application crawler to help us map an application.
Define the objective

Create a Scrapy project

Run the project and verify the results.

Link:
Watch Video

3.3 Recursive Crawling

We created a basic crawler, but now, we want to make it recursive to cover all the web application content.

Create the code to extract all the links
Check whether the links are valid

Run the crawler and check the results.

Link:
Watch Video

3.4 Extracting Information

We need to extract information from the web application that will be useful for our Security testing.

Learn how to extract e-mails

Extract forms

Extract comments.

Link:
Watch Video

Click Here for Section 4