Project Description

A web crawling frame work built on top of GoFish. CrawlFish is meant to do web crawls in a cluster, storing all data locally.

About CrawlFish

This basic web crawler is built to create localized content for GoFish (http://gofish.codeplex.com). Currently you can generate data buy using the TestApp console application and supplying parameters. In the future this will be modified to accept urls and configs through web services as the final goal is to be able to distribute crawl loads over a cluster of boxes, storing the results locally, and returning meaningful data through GoFish's distributed computing framework.

Last edited Mar 12, 2009 at 1:55 PM by gerunddev, version 3