Extraction Process Components

pulumi is used to define and deploy Infrastructure as Code (IAC).

The repository uses a micro-stack approach to increase agility and reduce conflicts. The repository is split between multiple pulumi projects in the following subdirectories:

• core Provisions core network infrastructure to connect to the source database
• metadata Uploads metadata about the source databases to AWS S3
• pipelines Provisions the AWS DMS instance and the validation lambda function
• glue_database Updates the metadata in the AWS Glue Data Catalogue. It also used to creates the data curation glue job and infrastructure but this is now deprecated in favour of create-a-derived-table

Please refer to the respective subdirectories for more detailed information and diagrams about the infrastructure created by each pulumi project. Please refer to using pulumi for more details on using pulumi.