Snippets tagged crawler
-
AsyncSeq - Introduction and Crawler
This snippet demonstrates programming using asynchronous sequences. It contains (hidden) implementation of AsyncSeq type and combinators for working with it. More importantly, it demonstrates how to use asynchronous sequences to implement a simple sequential on-demand crawler.
10 people like this
Like the snippet!Posted: 13 years ago by Tomas Petricek
-
Web Crawler
This snippet features an F# Web crawler that i'm already using in 2 applications (slightly modified). It's based on a scalable network of communicating agents that follow URLs extracted from HTML pages until reaching the specified limit.
8 people like this
Like the snippet!Posted: 3 years ago by Taha Hachana
-
Parallel recursive crawler using agents
The aim here is to demonstrate a method of distributing work using the built in F# agent across multiple nodes in parallel the result of crawling one page might result in finding multiple new pages to fetch. This is a recursive process which will continue until no new URLs are found. The main focus is how to process a potentially indefinite queue of work across a pool of workers, rather than how to parse web pages.
6 people like this
Like the snippet!Posted: 8 years ago by Daniel Bradley