data542.combellingham, wa

data542

Bruce McMinn, Data Architecture & Engineering

I build data pipelines and the reporting on top of them for a university business intelligence team. This site is a personal data architecture project: the same kind of work, done end to end on AWS with data I collect myself. Every decision, from ingestion to the published report, is one I made and can explain. Two pipelines run in production, a Tempest weather station and a daily equities scanner. A third project, twenty years of Euro exchange-rate volatility, is earlier analysis work in Python from 2022. The notebook is posted here. Future projects planned on the same foundation are listed as well.

Platformaws us-west-2

The lake pattern

Every project lands in the same S3 bucket, each under its own prefix, with IAM roles scoped to its own data. Each keeps a control prefix recording where its last successful run ended, written only after the data lands, so a failed run changes nothing and the next one resumes from the last good point. Compute is chosen per job rather than standardized: the weather station's short jobs run on Lambda, and the daily equities scan runs longer than Lambda allows, so it runs as a Fargate container. EventBridge schedules both, and Power BI reads the results through CloudFront.

Two pipelines run this way today. The Euro study below is next on the platform. It runs now from a frozen dataset that stops in 2020; rebuilt here, it pulls from the source that publishes those rates daily. A one-time terrain calculation and engine telemetry collected offline come later.

data542 platform architecture: Tempest and market data flow through Lambda and Fargate into an S3 medallion lake, then out via CloudFront to Power BI

Architecture decisions, in full

Livehistoric_data/

WeatherFlow Live Dashboard

Raw observations land from the WeatherFlow API exactly as received, giving every downstream step a replayable source of record. A cleaning pass repairs structural defects without touching granularity, at the station's native 1-minute resolution. A final aggregation builds a 5-minute materialized view, the only layer Power BI ever reads: roughly 272,000 pre-aggregated rows instead of millions, cheap to cache and fast to refresh.

View the live report

Liveswing_trader/

Swing Trader

A containerized daily scan of high-volatility NASDAQ equities using ATR, RSI, Supertrend, and volume rules, appending to a growing scan history. The universe of scanned symbols rebuilds weekly; scans run each weekday after market close.

It runs on Fargate rather than Lambda for a specific reason: the scanner processes hundreds of symbols and can exceed Lambda's 15-minute execution ceiling. A single compute standard would be simpler to describe and worse to operate.

View the scan

Notebookeuro-forex

Euro on FOREX

An early exploratory notebook: twenty years of Euro exchange rate volatility across forty currencies, geopolitical events mapped against market movement, and an animated annual volatility world map. It predates the platform; a rebuild on it is on the roadmap below.

View the notebook

Euro on FOREX annual volatility world map

Roadmapwhat's next on the platform

What's coming

A one-time terrain analysis of west-facing viewpoints around Puget Sound, computing sunset azimuth against horizon profiles derived from USGS elevation data, to rank, for any date, which site offers the latest visible sunset and the longest civil twilight. Its heavy computation runs once, offline; the report reads a static result table, a deliberately different shape from the two scheduled pipelines.

Planned

FOREX modernization

The exploratory notebook rebuilt on the platform, with a currency-pair selector, multiple timescales, and a three-dimensional volatility visualization across twenty pairs.

Exploring

Marine Telemetry

An NMEA 2000 gateway capturing live data from an outboard engine and onboard instruments (fuel flow, heading, position) during time on the water. Unlike the platform's scheduled API pipelines, this data originates at the edge, collected in intermittent sessions and batch-uploaded to the lake afterward. Early targets: fuel-efficiency curves by speed, and cross-referencing heading against position to estimate the effect of current.

Deferred

The shared catalog: by design

A common bucket convention and a Glue Data Catalog across all projects, sequenced to begin once the Sunset project's requirements are concrete. Building them before the next project exists would mean designing for workloads that haven't taken shape. Knowing what not to build yet is an architecture decision too.