Python 3.9+ · source CLI
WebCrawler
Map one host and flag administration paths, interesting query parameters, and input surfaces for manual review. Resume when a crawl gets interrupted.
What it does
- Follows links on a single host and respects robots.txt by default.
- Reports heuristic admin, SQLi-candidate, and XSS-related input findings in JSON.
- Supports checkpointing and an optional Selenium/Chrome fallback for dynamic pages.
Compatibility
Requires Python 3.9+, requests, and Beautiful Soup. Optional packages are documented separately.
Without Selenium or Chrome it can continue with ordinary HTTP requests. No standalone executable or installer is published.
Download & installation
Available as a documented source installation. The download contains the public repository's source files.
Source code · requires the setup described below · not a prebuilt executable
- Download and extract the source ZIP, or clone the repository.
- Install dependencies with python -m pip install -r requirements.txt. Install requirements-optional.txt only if needed.
- Run python webcrawler.py for the interactive prompts, or pass an explicitly authorized URL. Use --help for the available limits and resume options.
Limitations
Findings are reconnaissance leads, not confirmed vulnerabilities. The tool deliberately leaves active exploitation out of the crawl.