Data Homelab for Mini Testing

• 2 min read

Data Homelab is a data platform for data engineers to build and deploy data pipelines, data lake, and data warehouse.

Long’s Notes

  • Using K3s to deploy OSS components
  • Hardware: Any laptop with 8GB RAM and 20GB SSD
  • Software: Ubuntu (preferably 20.04 LTS), Docker, Kubernetes, K3s, Git
  • OSS components: Mage AI, Grafana, Prometheus, PostgreSQL. (All alternatives)

Solution to backup data

  • AWS S3 ****

[[tooling]] help me to set up the homelab

  • Chip Laptop: during time I was in university, the Dell is really powerfull laptop in my thinking because it is for construction major engineers to learn and study, it runs the AutoCad, Solidwork, Semilation. Even I was learing Audomation with PLC tool, MathLab with semulator. Dell is the chosen one in that cases. Keeping that practice, I use Dell for hosting all infrastructure.

  • K3s: basically, I followed the online K3s installation instruction.
  • DataPods - OSS: I constructed the repository for hosting 3 essential components of mini ETL includes: Mage ETI, Postgres Store, Grafana Visualize. You also can deploy the template with Railway datapods-mini-elt - Railway Help Station

Searching template

Railway mini ETL

Check it out.

railway-etl-template.png

I mentioned in a few posts or talks that we are trying to use the published knowledge and contributing back to community. That is how we grow and maintain the world.