The lake pattern
The projects here share one data lake design: a medallion architecture, with layers named for what they do rather than for the metals. Compute is matched to the workload, not standardized on one service — short ingestion and transformation jobs run on Lambda, while longer-running scans run as Fargate container tasks. Projects share a single S3 bucket, separated by prefix and governed by IAM roles scoped to each project's own data.
Architecture decisions, in full →When a failed wind sensor produced a month of bad readings, the fix was applied surgically in the reporting layer — nulling only the affected fields for the affected month — rather than rebuilding the silver layer.
A purist would push that correction into the lake; shipping it at the reporting layer preserved the raw record, fixed the published data the same day, and left a clean path to promote the fix upstream later. Knowing where to deviate from the textbook, and keeping the deviation documented and reversible, is most of what the pattern is for.
