Snippets tagged web crawler

  • Web Crawler extensions

    The snippet extends a web crawler from snippet http://fssnip.net/3K. It synchronizes all printing using an additional agent (so printed text does not interleave) and the crawling function returns an asynchronous workflow that returns when crawling completes.

    9 people like this

    Posted: 5 years ago by Tomas Petricek

  • Parallel recursive crawler using agents

    The aim here is to demonstrate a method of distributing work using the built in F# agent across multiple nodes in parallel the result of crawling one page might result in finding multiple new pages to fetch. This is a recursive process which will continue until no new URLs are found. The main focus is how to process a potentially indefinite queue of work across a pool of workers, rather than how to parse web pages.

    3 people like this

    Posted: 9 months ago by Daniel Bradley

  • AsyncSeq - Introduction and Crawler

    This snippet demonstrates programming using asynchronous sequences. It contains (hidden) implementation of AsyncSeq type and combinators for working with it. More importantly, it demonstrates how to use asynchronous sequences to implement a simple sequential on-demand crawler.

    10 people like this

    Posted: 5 years ago by Tomas Petricek