Sitemap Generator

Need a sitemap? This tech-stack agnostic generator will crawl a website like a real user generating a sitemap along the way.

This extension lets you generate a sitemap for any public website right in your browser. While generating the sitemap, the extension renders the web pages and waits for javascript to load, making it great for crawling dynamic single-page applications made with angular, react, etc.

This extension works by taking some start URL, crawling that page for more links, and then recursively crawling those pages for more links. Once all discovered links have been checked, the extension outputs a sitemap file.

This implementation is not practical for website containing tens of thousands of pages. It can however, crawl a few thousand entries in a reasonable amount of time.

System diagram

This is a simplified diagram showing how the process works