Skip to content
This repository has been archived by the owner on Apr 20, 2019. It is now read-only.
/ heroshi Public archive

Heroshi – open source web crawler.

Notifications You must be signed in to change notification settings

temoto/heroshi

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

Heroshi, open source web crawler.

Motivation 1: learn HTTP, libraries, real world quirks.
Motivation 2: collection of libraries and tools for building custom crawlers.
Motivation 3: provide access to representative subset of Web for educational and research purposes.

As of 2012-10-12, last goal is not even started, but these guys did amazing job at it http://commoncrawl.org/

See http://temoto.github.com/heroshi/ for more information.