My Data Journey

A portable Data Analytics stack using Docker, Mage, dbt-core, DuckDB and Superset

Constantin LunguUpdated 1 min read

Photo by frank mckenna on Unsplash

Just wanted to share a small learning-by-doing project of mine. It's a containerized Data Analytics suite, covering end-to-end analytics process for a small (imaginary) company.

We're talking about:
- generating example data in parquet files using Python
- ingesting data into DuckDB
- model data using dbt-core
- loading a DuckDB datamart
- orchestrate using MageAI
- displaying it all in a Superset dashboard.

Apache Superset Sales dashboard screenshot with a sales per region table, a sales per country map, a channel sales evolution line chart for 2019 to 2020, an average order size of 8.99 USD, sales by month and a day-of-week trends pivot by city.

Each of the components is in a separate Docker container, tied all together with docker-compose.

I've previously set up similar projects with Airflow and Dagster.

It's pretty bare bones (somewhat as intended) and has some rough edges, but it should be a good starting point for a demo, template or learn how all these components works together.

I would of course appreciate any feedback or suggestions on how to make it better.