Crawler for CTF Writeups
http://www.postgresql.org/ Please ensure the PostgreSQL is installed and running on machine with the following configured
- Ensure that there is a database named: ctfcrawler
- Ensure that there is a postgres user/password: ctfcrawler/ctfcrawler
The program is developed on Eclipse IDE. To run, either import folder to Eclipse and run using Eclipse or extract the jar file using "jar xf " and run using "java "
The driver class to start the WebCrawler. To run, "java WebCrawlerDriver"
The Database Frontend for queries. To run, "java DatabseFrontend"
Contains domains that is allowed during crawling
Contains the categories to tag the links during crawling
Contains new domains found during crawling that might be suitable to be whitelisted
Contains new categories found during crawling that might be suitable for tagging