How to Contribute

Architecture Decision Records

Please refer to Managed Pipelines ADRs. You will need an account on the dsdmoj Confluence site.

Git branch strategy

We use GitHub-Flow for branch management with a single long-lived main branch. We use feature flags to release code to production more quickly and often and ensure changes are deployed to one environment at a time.

Tickets

Please refer to Managed Pipelines and Modernisation Board. You will need an account on the dsdmoj Jira site.

Runbooks

Please refer to Managed Pipelines Runbooks. You will need an account on the dsdmoj Confluence site.

Prerequisites

To work with this repository, you must have the following installed:

You must also have a python environment activated with the relevant python libraries installed.

You must also have the necessary AWS IAM permissions to update AWS resources. For further information about IAM permissions on the Analytical Platform, see the analytical-platform-iam repository.

Unit Testing

Please refer to tests.

Integration Testing

tests_glue tests the glue pyspark and and python shell jobs in glue_database
tests_lamdba tests the validate lambda function in pipelines
tests_extraction(exp. 2023) tests the metadata and data extraction in metadata and pipelines

Alerts

Please subscribe to the following slack channels:

#data-engineering-alerts-dev for notifications about the dev environment
#data-engineering-alerts-preprod for notifications about the preprod environment
#data-engineering-alerts-prod for notifications about the preprod environment

You will need an account on the ASD Slack Worspace.

Pulumi

This repository uses pulumi to define and deploy Infrastructure as Code (IAC). Please refer to using pulumi for more details.

Resource Names

Resources should be named according to the Terraform naming conventions.

Object Names

Names for all objects should follow the conventions outlined in the PEP 8 rules, except for instances of Pulumi AWS resource classes, which should be styled in mixedCase. For example:

securityGroup = aws.ec2.SecurityGroup(...)

Tagging

All resources that allow tagging should be tagged in line with the MoJ technical guidance.

You should use the following values for the mandatory tags by default:

Tag	Value
`business-unit`	`Platforms`
`application`	`Data Engineering`
`is-production`	`true` or `false`
`owner`	`Data Engineering:dataengineering@digital.justice.gov.uk`

Diagrams

This repository uses draw.io (also known as diagrams.net) to create diagrams. To update the diagrams install the draw.io visual code extension. You can preview images in markdown files using Markdown Preview in Visual Studio.

Documentation

This repository uses mkdocs-tech-docs-template and various mkdocs plugins to build and publish the documentation to GitHub pages. The mkdocs-tech-docs-template is experimental and should only be used for prototyping.

To preview the documentation locally

Create a new virtual environment with Python 3.10
```
python -m venv env
```
Activate the virtual environment
```
source env/bin/activate
```
Install the python docs dependencies
```
pip install -r requirements-doc.txt
```
Preview the documentation
```
mkdocs serve
```

Last update: January 9, 2024
Created: January 9, 2024