An automated sequence of steps that moves and transforms data from one system to another.
A data pipeline handles the flow of data from its source — a database, an API, a file — through a series of processing steps, and into a destination such as a data warehouse or analytics tool. Each step may filter, clean, enrich, or reformat the data.
Pipelines can run on a schedule (batch) or continuously as new data arrives (streaming). They are the infrastructure that keeps data products up to date without manual intervention.
Example: A city's open data team runs a nightly pipeline that pulls permit records from an internal system, standardizes address formats, removes test entries, and loads the cleaned dataset into the public portal by 6 a.m.
Understanding pipelines helps you appreciate why published datasets have a lag, why formats are consistent, and what ETL processes do behind the scenes.