This list contains network and data processing tools with command line interface written in any programming langauge.
EMPTY CONTENT
- pipet - A swiss-army tool for scraping and extracting data using selectors, JavaScript and unix pipes
- trafilatura - Gather text and metadata on the Web: Crawling, scraping, extraction, output as CSV, JSON, HTML, MD, TXT, XML
- Wayback-Archive - Download a whole website from the Wayback Machine and rewrite its links for offline viewing
- courlan - Clean, filter and sample URLs to optimize data collection: Deduplication, spam, content and language filters