Snippets tagged web crawler
-
AsyncSeq - Introduction and Crawler
This snippet demonstrates programming using asynchronous sequences. It contains (hidden) implementation of AsyncSeq type and combinators for working with it. More importantly, it demonstrates how to use asynchronous sequences to implement a simple sequential on-demand crawler.
10 people like this
Like the snippet!Posted: 12 years ago by Tomas Petricek
-
Web Crawler extensions
The snippet extends a web crawler from snippet http://fssnip.net/3K. It synchronizes all printing using an additional agent (so printed text does not interleave) and the crawling function returns an asynchronous workflow that returns when crawling completes.
0 people like this
Like the snippet!Posted: 8 months ago by Tomas Petricek
-
Parallel recursive crawler using agents
The aim here is to demonstrate a method of distributing work using the built in F# agent across multiple nodes in parallel the result of crawling one page might result in finding multiple new pages to fetch. This is a recursive process which will continue until no new URLs are found. The main focus is how to process a potentially indefinite queue of work across a pool of workers, rather than how to parse web pages.
5 people like this
Like the snippet!Posted: 7 years ago by Daniel Bradley