Economic Indicators Dashboard

The problem

Macroeconomic and financial indicators live scattered across public APIs, each with its own format, frequency and rate limits. Comparing the Fed funds rate against oil, gold or the S&P 500 means repeating the same collection and cleaning work every time — and the data goes stale the moment you stop.

The approach

I built a production-style pipeline instead of one-off scripts: extractors per source with local caching, a normalization layer, idempotent upsert loads into PostgreSQL, and a dashboard on top. The whole stack — database, ETL, cron scheduler and dashboard — reproduces with one Docker Compose command and refreshes itself daily without manual intervention.

How it works

1. 01

   Extract

   Python extractors pull 13 indicators from the FRED and Alpha Vantage APIs, caching raw JSON responses with a 24-hour TTL to respect Alpha Vantage's 25-calls/day free tier.

2. 02

   Transform

   Raw payloads are cleaned and normalized into a common schema — consistent dates, units and symbols across both sources.

3. 03

   Load

   Observations are upserted into PostgreSQL via ON CONFLICT, one atomic transaction per indicator — re-runs never duplicate rows, and one failing source never aborts the run.

4. 04

   Schedule

   A cron job inside the container re-runs the pipeline every night, keeping the data fresh with zero manual intervention.

5. 05

   Serve

   A Streamlit + Plotly dashboard queries the database, with Base-100 indexing, percentage change and day-to-year resampling to compare indicators with different units.

Stack

Language

Python (pandas, psycopg2)

Storage

PostgreSQL

Dashboard

Streamlit + Plotly

Infrastructure

Docker Compose + Cron

Sources

FRED · Alpha Vantage APIs

What I learned

• — Idempotency is the single most valuable property of a pipeline — upsert loads and safe re-runs simplified every failure scenario.
• — Isolating each indicator in its own transaction with its own error handling lets partial failures degrade gracefully instead of crashing the whole run.
• — Caching raw responses is an engineering and an economics decision at once: it is what makes a 25-calls/day API viable in production.

Outcome

• ✓ One `docker compose up` builds the database, loads all 13 indicators and starts the scheduler and the dashboard.
• ✓ Data refreshes automatically every day; a failed source is logged and skipped, never fatal.
• ✓ The dashboard compares indicators with different units through Base-100 and percentage-change transformations.