All projects

Python 3.9+ · source CLI

WebCrawler

Map one host and flag administration paths, interesting query parameters, and input surfaces for manual review. Resume when a crawl gets interrupted.

What it does

  • Follows links on a single host and respects robots.txt by default.
  • Reports heuristic admin, SQLi-candidate, and XSS-related input findings in JSON.
  • Supports checkpointing and an optional Selenium/Chrome fallback for dynamic pages.

Compatibility

Requires Python 3.9+, requests, and Beautiful Soup. Optional packages are documented separately.

Without Selenium or Chrome it can continue with ordinary HTTP requests. No standalone executable or installer is published.

Download & installation

Available as a documented source installation. The download contains the public repository's source files.

Source code · requires the setup described below · not a prebuilt executable

  1. Download and extract the source ZIP, or clone the repository.
  2. Install dependencies with python -m pip install -r requirements.txt. Install requirements-optional.txt only if needed.
  3. Run python webcrawler.py for the interactive prompts, or pass an explicitly authorized URL. Use --help for the available limits and resume options.

Limitations

Findings are reconnaissance leads, not confirmed vulnerabilities. The tool deliberately leaves active exploitation out of the crawl.

Documentation

All projects