Frameworks information gathering tool Vulnerability scanner

dcrawl : Multi-threaded Web Crawler

Written by Mubassir patel


dcrawl is a simple, but smart, multi-threaded web crawler for randomly gathering huge lists of unique domain names.


dcrawl : Multi Threaded Web crawler

How it works?

dcrawl takes one site URL as input and detects all <a href=...> links in the site’s body. Each found link is put into the queue. Successively, each queued link is crawled in the same way, branching out to more URLs found in links on each site’s body.

How  smart crawling works:

  • Branching out only to predefined number of links found per one hostname.
  • Maximum number of allowed different hostnames per one domain (avoids subdomain crawling hell e.g.
  • Can be restarted with same list of domains – last saved domains are added to the URL queue.
  • Crawls only sites that return text/html Content-Type in HEAD response.
  • Retrieves site body of maximum 1MB size.
  • Does not save inaccessible domains.

How to run?

go build dcrawl.go
./dcrawl -url -out ~/domain_lists/domains1.txt -t 8


     ___                          __
  __| _/________________ __  _  _|  |
 / __ |/ ___\_  __ \__  \\ \/ \/ /  |
/ /_/ \  \___|  | \// __ \\     /|  |__
\____ |\___  >__|  (____  /\/\_/ |____/
     \/    \/           \/       v.1.0

usage: dcrawl -url URL -out OUTPUT_FILE -t THREADS

  -ms int
        maximum different subdomains for one domain (def. 10) (default 10)
  -mu int
        maximum number of links to spider per hostname (def. 5) (default 5)
  -out string
        output file to save hostnames to
  -t int
        number of concurrent threads (def. 8) (default 8)
  -url string
        URL to start scraping from
  -v bool
        verbose (default false)


dcrawl was made by Kuba Gretzky from and released under the MIT license.

About the author

Mubassir patel

Mubassir is a founder and developer of this site. He is a computer science engineer. He has a very deep interest in ethical hacking, penetration testing, website development and including all technology topic.

Leave a Comment