scrapy/protego
A pure-Python robots.txt parser with support for modern conventions.
Stars: 85Language: DIGITAL Command Language
Give AlbumentationsX a star on GitHub — it powers this leaderboard
Star on GitHubA pure-Python robots.txt parser with support for modern conventions.