Bring dbt to Production with seamless CI/CD

• 1 min read

How to bring dbt to production with seamless CI/CD

##TODO ref: https://jaehyeon.me/blog/2024-09-13-dbt-guide/

Key takeaways

Checklist:

  • Best practice:
    • Split dbt project into modules
    • Separate macros and tests into different packages
    • Use base layer for source data & staging layer for complex logic
    • Implement CI/CD via GitLab/GitHub Actions
    • Build simple web service for backfilling/scheduling/monitoring
  • Testing:
    • Generic tests: unique, not_null, accepted_values
    • Singular tests for complex business logic
    • Create custom schema tests with macros
  • Development:
    • Use dbt completion for easier development
    • Modular project structure with individual packages
    • Follow clean code practices
  • Common challenges:
    • Hard dependencies between models
    • Workflow monitoring
    • Data quality testing
    • Scheduling & orchestration

Call to Action