website-crawler

Here are 4 public repositories matching this topic...

puneetindersingh / open-seo-crawler

Free SEO crawler & website audit tool — a self-hosted, open-source SEO spider that crawls any website for technical SEO issues. Concurrent, CMS-aware (Shopify/WordPress/Webflow), XLSX export. A free alternative to Screaming Frog.

python flask crawler spider seo web-crawler sitemap-generator site-audit website-crawler wordpress-seo shopify-seo seo-tools seo-crawler cms-detection seo-audit technical-seo broken-link-checker screaming-frog-alternative seo-spider

Updated Sep 29, 2026
JavaScript

simeon-zhelev / componentize

Star

Discover the components hiding in every website.

web-components visual-analysis perceptual-hashing component-analysis design-system website-crawler playwright website-audit

Updated Jul 14, 2026
JavaScript

Kenil-Mangukiya / llm-txt-generator-wordpress

Star

WordPress plugin that generates LLM-friendly documentation files (llm.txt, llm-full.txt) from website URLs. Perfect for SEO optimization and AI/LLM consumption.

wordpress wordpress-plugin seo text-generation file-management website-documentation website-crawler ai-tools llm ai-integration

Updated Dec 29, 2025
JavaScript

AmaanHaider / News-crawler

Star

bootstrap node mongodb cheerio crawling express-js news-crawler website-crawler cheerio-js cheerio-node news-crawler-website

Updated Aug 24, 2023
JavaScript

Add this topic to your repo

To associate your repository with the website-crawler topic, visit your repo's landing page and select "manage topics."

Learn more