How to Contribute

Architecture Decision Records

Please refer to Managed Pipelines ADRs. You will need an account on the dsdmoj Confluence site.

Git branch strategy

We use GitHub-Flow for branch management with a single long-lived main branch. We use feature flags to release code to production more quickly and often and ensure changes are deployed to one environment at a time.

Tickets

Please refer to dmet-probation github project

Runbooks

Please refer to Managed Pipelines Runbooks. You will need an account on the dsdmoj Confluence site.

Prerequisites

To work with this repository, you must have the following installed:

• Python 3.9 or later
• Pulumi

You must also have a python environment activated with the relevant python libraries installed.

You must also have the necessary AWS IAM permissions to update AWS resources. For further information about IAM permissions on the Analytical Platform, see the analytical-platform-iam repository.

Unit Testing

Please refer to tests.

Integration Testing

• tests_glue tests the glue pyspark and and python shell jobs in glue_database
• tests_lamdba tests the validate lambda function in pipelines
• tests_extraction(exp. 2023) tests the metadata and data extraction in metadata and pipelines

Alerts

Please subscribe to the following slack channels:

• #data-engineering-alerts-dev for notifications about the dev environment
• #data-engineering-alerts-preprod for notifications about the preprod environment
• #data-engineering-alerts-prod for notifications about the preprod environment

You will need an account on the ASD Slack Worspace.

Pulumi

This repository uses pulumi to define and deploy Infrastructure as Code (IAC). Please refer to using pulumi for more details.

Resource Names

Resources should be named according to the Terraform naming conventions.

Object Names

Names for all objects should follow the conventions outlined in the PEP 8 rules, except for instances of Pulumi AWS resource classes, which should be styled in mixedCase. For example:

securityGroup = aws.ec2.SecurityGroup(...)

Tagging

All resources that allow tagging should be tagged in line with the MoJ technical guidance.

You should use the following values for the mandatory tags by default:

Diagrams

This repository uses draw.io (also known as diagrams.net) to create diagrams. To update the diagrams install the draw.io visual code extension. You can preview images in markdown files using Markdown Preview in Visual Studio.

Documentation

This repository uses mkdocs-tech-docs-template and various mkdocs plugins to build and publish the documentation to GitHub pages. The mkdocs-tech-docs-template is experimental and should only be used for prototyping.

To preview the documentation locally

1. Create a new virtual environment with Python 3.10

   python -m venv env
   

2. Activate the virtual environment

   source env/bin/activate
   

3. Install the python docs dependencies

   pip install -r requirements-doc.txt
   

4. Preview the documentation

   mkdocs serve