Automatically extracting data from websites by parsing their HTML content, used when data is not available through an API or structured download.
Web scraping uses software to load web pages and extract specific data elements — tables, lists, text, links — from the HTML. It is a last-resort data acquisition method: if a website offers an API or downloadable files, those are always preferable to scraping.
Web scraping raises important legal and ethical considerations. Many websites prohibit scraping in their terms of use. Scraping personal information may violate privacy laws. Aggressive scraping can overload servers. Always check the terms of use and robots.txt before scraping a website.
Example: A researcher wants to analyze property listing prices from a real estate website that does not offer a data download or API. They write a web scraper that loads each listing page, extracts the price, address, and property attributes from the HTML, and saves the results to a CSV file. Before running the scraper, they check the website's terms of use and find that scraping for non-commercial research is permitted with attribution.