CoCalc -- crawler.md

CoCalc provides the best real-time collaborative environment for Jupyter Notebooks, LaTeX documents, and SageMath, scalable from individual users to large groups and classes!

GitHub Repository: rapid7/metasploit-framework
Path: blob/master/documentation/modules/auxiliary/scanner/http/crawler.md
Views: ¹⁹⁰⁴

Description

This module is a http crawler, it will browse the links recursively from the web site. If you have loaded a database plugin and connected to a database, this module will report web pages and web forms.

Vulnerable Application

You can use any web application to test the crawler.

Options

URI

Default path is /

DirBust

Bruteforce common url path, default is true but may generate noise in reports.

HttpPassword, HttpUsername, HTTPAdditionalHeaders, HTTPCookie

You can add some login information

UserAgent

Default User Agent is Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)

Verification Steps

Do: use auxiliary/scanner/http/crawler
Do: set RHOST [IP]
Do: set RPORT [PORT]
Do: set URI [PATH]
Do: run

Scenarios

Example against WebGoat

msf> use auxiliary/scanner/http/crawler
msf auxiliary(crawler) > set RHOST 127.0.0.1
msf auxiliary(crawler) > set RPORT 8080
msf auxiliary(crawler) > set URI /webgoat/
msf auxiliary(crawler) > set DirBust false
msf auxiliary(crawler) > run
[*] Crawling http://127.0.0.1:8008/webgoat/...
[*] [00001/00500]    302 - 127.0.0.1 - http://127.0.0.1:8008/webgoat/ -> /webgoat/login.mvc
[*] [00002/00500]    200 - 127.0.0.1 - http://127.0.0.1:8008/webgoat/login.mvc
[*]                         FORM: POST /webgoat/j_spring_security_check;jsessionid=8B1EAF2554B60EFC93A52AFCA4B6C202
[-] [00003/00500]    404 - 127.0.0.1 - http://127.0.0.1:8008/webgoat/images/favicon.ico
[*] [00004/00500]    200 - 127.0.0.1 - http://127.0.0.1:8008/webgoat/plugins/bootstrap/css/bootstrap.min.css
[*] [00005/00500]    200 - 127.0.0.1 - http://127.0.0.1:8008/webgoat/css/font-awesome.min.css
[*] [00006/00500]    200 - 127.0.0.1 - http://127.0.0.1:8008/webgoat/css/animate.css
[*] [00007/00500]    302 - 127.0.0.1 - http://127.0.0.1:8008/webgoat/j_spring_security_check;jsessionid=8B1EAF2554B60EFC93A52AFCA4B6C202 -> /webgoat/login.mvc;jsessionid=8B1EAF2554B60EFC93A52AFCA4B6C202?error
[*] [00008/00500]    200 - 127.0.0.1 - http://127.0.0.1:8008/webgoat/login.mvc;jsessionid=8B1EAF2554B60EFC93A52AFCA4B6C202?error
[*]                         FORM: GET /webgoat/login.mvc
[*]                         FORM: POST /webgoat/j_spring_security_check;jsessionid=8B1EAF2554B60EFC93A52AFCA4B6C202
[*] [00009/00500]    200 - 127.0.0.1 - http://127.0.0.1:8008/webgoat/css/main.css
[*] [00010/00500]    302 - 127.0.0.1 - http://127.0.0.1:8008/webgoat/start.mvc -> http://127.0.0.1:8008/webgoat/login.mvc
[*] [00011/00500]    200 - 127.0.0.1 - http://127.0.0.1:8008/webgoat/login.mvc
[*]                         FORM: POST /webgoat/j_spring_security_check
[*] Crawl of http://127.0.0.1:8008/webgoat/ complete
[*] Auxiliary module execution completed

Follow-on: Wmap

As you see, the result is not very user friendly...

But you can view a tree of your website with the Wmap plugin. Simply run :

msf auxiliary(crawler) > load wmap
msf auxiliary(crawler) > wmap_sites -l
[*] Available sites
===============

     Id  Host           Vhost          Port  Proto  # Pages  # Forms
     --  ----           -----          ----  -----  -------  -------
     0   127.0.0.1      127.0.0.1      8080  http   70       80


msf auxiliary(crawler) > wmap_sites -s 0

    [127.0.0.1] (127.0.0.1)
    └── webgoat (7)
        ├── css (3)
        │   ├── animate.css
        │   ├── font-awesome.min.css
        │   └── main.css
        ├── j_spring_security_check;jsessionid=8B1EAF2554B60EFC93A52AFCA4B6C202
        ├── login.mvc
        ├── login.mvc;jsessionid=8B1EAF2554B60EFC93A52AFCA4B6C202
        ├── plugins (1)
        │   └── bootstrap (1)
        │       └── css (1)
        │           └── bootstrap.min.css
        ├── start.mvc
        └── j_spring_security_check