Snippets tagged crawling

  • Web Crawler extensions

    The snippet extends a web crawler from snippet http://fssnip.net/3K. It synchronizes all printing using an additional agent (so printed text does not interleave) and the crawling function returns an asynchronous workflow that returns when crawling completes.

    9 people like this

    Posted: 5 years ago by Tomas Petricek

  • Parallel recursive crawler using agents

    The aim here is to demonstrate a method of distributing work using the built in F# agent across multiple nodes in parallel the result of crawling one page might result in finding multiple new pages to fetch. This is a recursive process which will continue until no new URLs are found. The main focus is how to process a potentially indefinite queue of work across a pool of workers, rather than how to parse web pages.

    3 people like this

    Posted: 8 months ago by Daniel Bradley