Slim CI Job

A Slim CI Job on 5X Managed dbt Core™ runs by testing only modified dbt models upon a pull request (PR). A continuous integration (CI) workflow is implemented by setting up automation that tests code changes by running CI Jobs before merging to production. Slim CI runs the modified models exclusively using the data from the previous job run, using differ. Users can control the job run by selecting an environment to compare code changes against.

Slim CI builds only modified models and the modified models will reference upstream, unchanged models from the comparison environment. 5X Managed dbt Core™ tracks the status of the production environments to run a CI Job for the modified data assets and their downstream dependencies.

5X Managed dbt Core™ supporting Slim CI,

  • Optimizes CI/CD pipelines and resource utilization, and Slim CI identifies the modified models and their downstream dependencies, to run tests on them exclusively. This enables 5X Managed dbt Core™ to optimize efficiency since it does not test the entire dbt project repository for code changes, saving unnecessary computational resources, time and development cycle effort.

  • Enhances development workflow and developer productivity Slim CI reduces CI/CD pipeline execution time to help with faster feedback and quick iterations. 5X Managed dbt Core ™ users can streamline their development and deployment processes more efficiently with Slim CI and enhance development workflow to save infrastructure costs.

CI Jobs are only supported on external GitHub repositories for 5X Managed dbt Core™ projects. Refer to Set up and manage dbt Project to set up a dbt project with the existing GitHub repository.

Prerequisites

  • User must use a dbt project connected to their external Github repository.

  • Users must have DBT_TOKEN connected to their repository. Refer to Set up a DBT_TOKEN to add Access Token on GitHub Secrets.

Create a CI Job

To create a CI Job, follow the steps:

  1. Navigate to Jobs on a 5X Managed dbt Core™ project with an external GitHub repository connection. Click Continuous Integration Job from the Create Job dropdown.

  2. Configure the Job settings. You can run a CI Job on deployment environments only.

  • Threads represent the number of concurrent models that can be executed simultaneously. More threads can potentially speed up the execution of the job, we recommend using 2 threads to get started, but the optimal number depends on factors like the available resources and the complexity of the transformations.

  • Enabling Triggerd by pull request to trigger the CI Job when you create a pull request.

  • A CI Job can run on the draft pull-request or the final pull-request. Enable Run on draft pull request when you want the CI Job to run whenever the draft pull request is opened.

  1. Configure the execution settings that are required and Save. Add dbt commands to execute during the job run and select an environment to defer Slim CI, against which all changes will be compared.

The default dbt command for the Slim CI job is dbt build --select state:modified+. dbt will build only the models that have been modified since the last successful run, along with their dependent models. state:modified+ instructs dbt to build models that have been modified since the last successful run. The + indicates that it includes all models that have been modified, including their dependents.

  1. Once saved, you’ve successfully added the Slim CI job.

The CI Job get executed when you commit modified changes from the Console and open a pull request.

Users can check the status of their CI Job on their repository.

Last updated