Web scraping

Web scraping, web harvesting, or web data extraction is <a href="/facts/Data_scraping/8zLlaECG">data scraping</a> used for <a href="/facts/Data_extraction/lsU5arIq">extracting data</a> from <a href="/facts/Website/SAsjJnca">websites</a>. Web scraping software may directly access the <a href="/facts/World_Wide_Web/wvnTrWFq">World Wide Web</a> using the <a href="/facts/Hypertext_Transfer_Protocol/TWtvf7JD">Hypertext Transfer Protocol</a> or a web browser. While web scraping can be done manually by a software user, the term typically refers to automated processes implemented using a <a href="/facts/Internet_bot/yPTtRLyq">bot</a> or <a href="/facts/Web_crawler/ra7D1xyb">web crawler</a>. It is a form of copying in which specific data is gathered and copied from the web, typically into a central local <a href="/facts/Database/VYcpkevb">database</a> or <a href="/facts/Spreadsheet/0cvc1GeB">spreadsheet</a>, for later <a href="/facts/Data_retrieval/Hqe9K9qd">retrieval</a> or <a href="/facts/Data_analysis/CNtARxLW">analysis</a>.
Scraping a web page involves fetching it and then extracting data from it. Fetching is the downloading of a page (which a browser does when a user views a page). Therefore, web crawling is a main component of web scraping, to fetch pages for later processing. Having fetched, extraction can take place. The content of a page may be <a href="/facts/Parsing/nRKB01S3">parsed</a>, searched and reformatted, and its data copied into a spreadsheet or loaded into a database. Web scrapers typically take something out of a page, to make use of it for another purpose somewhere else. An example would be finding and copying names and telephone numbers, companies and their URLs, or e-mail addresses to a list (contact scraping).
As well as <a href="/facts/Contact_scraping/p38pJgrT">contact scraping</a>, web scraping is used as a component of applications used for <a href="/facts/Web_indexing/ege3X1B6">web indexing</a>, <a href="/facts/Web_mining/7fbuMUpN">web mining</a> and <a href="/facts/Data_mining/AH53q5ac">data mining</a>, online price change monitoring and <a href="/facts/Comparison_shopping_website/0Y6z92CI">price comparison</a>, product review scraping (to watch the competition), gathering real estate listings, weather data monitoring, <a href="/facts/Change_detection_and_notification/NUEvfoYI">website change detection</a>, research, tracking online presence and reputation, <a href="/facts/Web_mashup/gn6mpfH9">web mashup</a>, and <a href="/facts/Web_data_integration/QlQczyaU">web data integration</a>.
<a href="/facts/Web_page/5JK7xQLe">Web pages</a> are built using text-based mark-up languages (<a href="/facts/HTML/sSc3fT78">HTML</a> and <a href="/facts/XHTML/6qHnmJSg">XHTML</a>), and frequently contain a wealth of useful data in text form. However, most web pages are designed for human <a href="/facts/End-user_(computer_science)/7IM9qu7N">end-users</a> and not for ease of automated use. As a result, specialized tools and software have been developed to facilitate the scraping of web pages. Web scraping applications include <a href="/facts/Market_research/grCXQpTf">market research</a>, price comparison, content monitoring, and more. Businesses rely on web scraping services to efficiently gather and utilize this data.
Newer forms of web scraping involve monitoring <a href="/facts/Data_feed/3mKJecHR">data feeds</a> from web servers. For example, <a href="/facts/JSON/bs1rUAhd">JSON</a> is commonly used as a transport mechanism between the client and the web server.
There are methods that some websites use to prevent web scraping, such as detecting and disallowing bots from crawling (viewing) their pages. In response, web scraping systems use techniques involving <a href="/facts/Document_Object_Model/GglE6jGx">DOM</a> parsing, <a href="/facts/Computer_vision/Tl2Yyk66">computer vision</a> and <a href="/facts/Natural_language_processing/1hjMKsSN">natural language processing</a> to simulate human browsing to enable gathering web page content for offline parsing.

Web scraping open-in-new

Web scraping