tomayac’s Twitter Archive—№ 9,825

Node.js-based robots.txt parser: github.com/samclarke/robots-parser 🤖 x.com/kdzwinel/status/947867088777039872