Snippets tagged crawler

  • Web Crawler

    This snippet features an F# Web crawler that i'm already using in 2 applications (slightly modified). It's based on a scalable network of communicating agents that follow URLs extracted from HTML pages until reaching the specified limit.

    22 people like this

    Posted: 6 years ago by Taha Hachana

  • Parallel recursive crawler using agents

    The aim here is to demonstrate a method of distributing work using the built in F# agent across multiple nodes in parallel the result of crawling one page might result in finding multiple new pages to fetch. This is a recursive process which will continue until no new URLs are found. The main focus is how to process a potentially indefinite queue of work across a pool of workers, rather than how to parse web pages.

    3 people like this

    Posted: 8 months ago by Daniel Bradley

  • AsyncSeq - Introduction and Crawler

    This snippet demonstrates programming using asynchronous sequences. It contains (hidden) implementation of AsyncSeq type and combinators for working with it. More importantly, it demonstrates how to use asynchronous sequences to implement a simple sequential on-demand crawler.

    10 people like this

    Posted: 5 years ago by Tomas Petricek