Using Pulumi

Deployment Order

When deploying for the first time, pulumi projects should be deployed in the following order to ensure that inter-project dependencies are managed appropriately.

• core
• pipelines_base
• metadata
• pipelines
• glue_database

Thereafter, you should mostly be able to update projects independently unless you are making changes that affect dependent resources in another project.

There are the following dependencies between projects:

• pipelines depends on pipelines-base implicitly
• pipelines depends on core explicitly

If the 'dmsRole' is updated or replaced in 'pipelines' the policy on the 'mojap-land...' bucket will need to be recreated so that it contains the correct ARN. This is managed in the data-engineering-infra repository.

Backend

All projects use the same pulumi backend:

s3://analytical-platform-data-engineering-pulumi-backend

which is defined in https://github.com/ministryofjustice/data-engineering-pulumi-backends and deployed to the Data Engineering AWS account.

Stacks

Each project have a prod stack. Most projects will also have a preprod stack and a dev stack.

In order to distinguish between stacks for each environment across different projects while using the same backend, stacks are given names of the form project-env.

New Pulumi Projects

New pulumi projects should be created in a subdirectory. To create a new project, follow the Pulumi AWS guidance. This will automatically create a .gitignore file and a requirements.txt file. These files can be removed from the project directory.

You should also create a README file that outlines the purpose of the project and how to use it, including any requirements.

Manual updates

You should update the stacks through the automated deployment pipelines. However you may sometimes need to update the stacks manually if something goes wrong:

1. Create a shell session with temporary credentials for the restricted-admin@data-engineering role using AWS Vault:

   aws-vault exec restricted-admin@data-engineering
   

2. Activate a virtual environment with the dependencies in requirements.txt installed.

3. Login to the cloud backend:

   pulumi login -c s3://analytical-platform-data-engineering-pulumi-backend
   

4. Select the stack:

   pulumi stack select <stack>
   

5. Preview any changes:

   pulumi preview --refresh --diff --policy-pack=../policy --policy-pack-config=../policy-config.json
   

6. Create or update the resources:

   pulumi up --refresh --policy-pack=../policy --policy-pack-config=../policy-config.json
   

Troubleshooting

Resources sometimes do not deploy correctly after running pulumi up. You can try re-running the pulumi_up github action.

You can also make sure the resource is correctly deployed by adding --replace urn and running locally:

    pulumi up --replace urn

Here, urn is the URN of the resource to be replaced. You can see the URNs of all resources in a stack by running:

    pulumi stack -u