Live briefing
Everything about Distributed web crawling
Distributed computing technique
Distributed web crawling is a distributed computing technique whereby Internet search engines employ many computers to index the Internet via web crawling. Such systems may allow for users to voluntarily offer their own computing and bandwidth resources towards crawling web pages. By spreading the load of these tasks across many computers, costs that would otherwise be spent on maintaining large computing clusters are avoided. Wikipedia →
🧠Start here: a 6-step reading path
- Read the overview: Distributed web crawlingA 5-minute orientation
- Skim a recent paper: Improvisation of Crawling on Web Maps Using Distributed Algorithms with Socket Programming ModelInfluential work since 2022
- Pick up a classic book: Introduction to information retrievalThe most-reprinted book on the subject
- Read the landmark paper: A Fast Distributed Focused-web CrawlingCited 24 times
- Go sideways: Web crawlerA closely related topic
- Go sideways: Apache NutchA closely related topic
đź“– Definitions
No dictionary entry for this exact phrase.
🗂️ The subject at a glance
🏆 Landmark papers (most cited)
A Fast Distributed Focused-web Crawling
Distributed Fundamentals based Conducting the Web Crawling Approaches and Types (Focused, Incremental, Distributed, Parallel, Hidden Web, Form Focused and Breadth First) Crawlers
WAN-Based Distributed Web Crawling
Distributed Focused Web Crawling for Context Aware Recommender System using Machine Learning and Text Mining Algorithms
Improvised Architecture for Distributed Web Crawling
Exploring Web Partition in DHT-Based Distributed Web Crawling
Efficient Distributed Web Crawling Utilizing Internet Resources
A Novel Distributed Web Crawling Approach Based on Mongodb
🆕 Influential since 2022
đź“š Books worth reading
Introduction to information retrievalChristopher D. Manning, Prabhakar Raghavan · 2008
Emerging Technologies for Information Systems, Computing, and ManagementW. Eric Wong · 2013
Apache Solr 3 Enterprise Search ServerDavid Smiley · 2011
Web DynamicsMark Levene · 2004
Ranked by number of editions on Open Library, a rough measure of how lasting a book is.
đź”— Related topics
- Web crawler A web crawler, sometimes called a spider or spiderbot and often shortened to crawler, is an Internet bot that systematically browses the World Wide Web…
- Apache Nutch Apache Nutch is a highly extensible and scalable open source web crawler software project. Nutch is coded entirely in the Java programming language, but…
- Googlebot Googlebot is the web crawler software used by Google that collects documents from the web to build a searchable index for the Google Search engine. This…
- Grub (search engine) Grub was an open source distributed search crawler platform. Users of Grub could download the peer-to-peer client software and let it run during their…
- Focused crawler A focused crawler is a web crawler that collects Web pages that satisfy some specific property, by carefully prioritizing the crawl frontier and managing…
- YaCy YaCy (pronounced “ya see”) is a free distributed search engine built on the principles of peer-to-peer (P2P) networks, created by Michael Christen in 2003…
- Distributed search engine A distributed search engine is a search engine where there is no central server. Unlike traditional centralized search engines, work such as crawling,…
- Load balancing (computing) In computing, load balancing is the process of distributing a set of tasks over a set of resources (computing units) with the aim of making their overall…