Live briefing
Everything about Web crawler
Software that systematically browses the World Wide Web
A web crawler, sometimes called a spider or spiderbot and often shortened to crawler, is an Internet bot that systematically browses the World Wide Web and that is typically operated by search engines for the purpose of Web indexing. Wikipedia →
🧭 Start here: a 6-step reading path
- Read the overview: Web crawlerA 5-minute orientation
- Skim a recent paper: Digital finance and green growth in China: Appraising inclusive digital finance using web crawler technology and big dataInfluential work since 2022
- Pick up a classic book: Introduction to Computing Using PythonThe most-reprinted book on the subject
- Read the landmark paper: UbiCrawler: a scalable fully distributed Web crawlerCited 330 times
- Go sideways: Distributed web crawlingA closely related topic
- Go sideways: Focused crawlerA closely related topic
📖 Definitions
Noun Alternative letter-case form of Web crawler.
🏆 Landmark papers (most cited)
UbiCrawler: a scalable fully distributed Web crawler
Mercator: A scalable, extensible Web crawler
A web crawler design for data mining
A focused crawler for Dark Web forums
Web Crawler: A Review
Keyword query based focused Web crawler
Results from a web impact factor crawler
Methodologies for crawler based Web surveys
🆕 Influential since 2022
Digital finance and green growth in China: Appraising inclusive digital finance using web crawler technology and big data
Water-Quality Assessment and Pollution-Risk Early-Warning System Based on Web Crawler Technology and LSTM
Perspective of People With Type 2 Diabetes Toward Self-management: Qualitative Study Based on Web Crawler Data
Reinforcement learning based web crawler detection for diversity and dynamics
A novel combining method of dynamic and static web crawler with parallel computing
📚 Books worth reading
Introduction to Computing Using PythonLjubomir Perkovic · 2011
Semantic Web and Web ScienceJuanzi Li · 2013
Emerging Technologies for Information Systems, Computing, and ManagementW. Eric Wong · 2013- 📕Managing the web of thingsQuan Z. Sheng, Yongrui Qin · 2017
- 📕Marvel Spider-ManDickson Adejo · 2019
Modeling and data mining in blogosphereNitin Agarwal · 2009
Intellectual property law in cyberspaceG. Peter Albert · 1999- 📕Enhancement in Web Crawler using Weighted Page Rank Algorithm based on VOLSachin Gupta · 2014
Ranked by number of editions on Open Library, a rough measure of how lasting a book is.
🔗 Related topics
- Distributed web crawling Distributed web crawling is a distributed computing technique whereby Internet search engines employ many computers to index the Internet via web crawling…
- Focused crawler A focused crawler is a web crawler that collects Web pages that satisfy some specific property, by carefully prioritizing the crawl frontier and managing…
- Googlebot Googlebot is the web crawler software used by Google that collects documents from the web to build a searchable index for the Google Search engine. This…
- Sitemaps Sitemaps is a protocol in XML format meant for a webmaster to inform search engines about URLs on a website that are available for web crawling. It allows…
- Deep web The deep web, also known as the invisible web or the hidden web, is the parts of the World Wide Web whose contents are not indexed by standard web search-engine…
- Search engine optimization Search engine optimization (SEO) is the practice of improving the visibility and overall performance of websites and web pages in search engine results…
- Robots.txt The Robots Exclusion Protocol (often referred to by the filename used to implement it, robots.txt) is a standard used by websites to indicate to visiting…
- DeepPeep DeepPeep was a search engine that aimed to crawl and index every database on the public Web. Unlike traditional search engines, which crawl existing webpages…