docs / architecture / decisions

Architecture Decisions

Key decisions made during the project with rationale.

ADR-001Python + asyncio for ingestion

Decision: Python 3.11 with asyncio for recup.py

Reason: I/O-bound task. asyncio allows concurrent sensor requests within the 60s window. Plain Python for all other pipeline stages.

ADR-002PostgreSQL for Silver and Gold

Decision: PostgreSQL on the VM

Reason: Multi-user access, Power BI connector, proper SQL for OLAP queries, free and production-grade.

ADR-003Apache Airflow for orchestration

Decision: Airflow on the VM

Reason: Full control over DAGs, visual monitoring, retry logic, logging. Each pipeline step is a visible task.

ADR-004File system for Bronze

Decision: Timestamped folder structure on disk

Reason: Immutable raw storage, no DB overhead, easy to inspect and replay.