Data Validation

Data validation runs automated tests on data: checking that columns are not null when required, that values fall within expected ranges, that formats are correct, and that relationships between tables are valid. For example, a schema says the customers table has columns (customer_id: integer, name: string, created_at: timestamp), and validation ensures all rows conform. Validation that fails can either raise alerts or automatically remediate (blocking data load, marking data as invalid, or filling with defaults).

Data validation emerged because manual quality checks don't scale. With thousands of tables and daily refreshes, checking quality manually is impossible. Automated validation runs tests continuously, catching issues quickly. Early detection prevents bad data from propagating: if a validation fails at data ingestion, the bad data never reaches downstream consumers.

Validation rules are typically defined using domain knowledge: analysts and engineers understand what valid data looks like. Rules might be simple (customer_id must be non-null) or complex (if order status is shipped, shipping_date must equal actual ship date). Some validation rules are business-specific (subscription start date cannot be after contract end date); others are structural (foreign key references must exist). Modern data platforms support declarative validation: writing rules in YAML or configurations rather than custom code.