ALZ Backup and Restore Process
This page lists the current ALZ Backup and Restore process for Hub and Shared services used to protect the critical service elements for ALZ product.
Backup And Restore
We currently backup the following resources spread across Hub and Shared Services.
| Resource | Backup Method | RPO | RTO |
|---|---|---|---|
| General Virtual Machines | Azure Virtual Machine | 24 Hours ( Daily once backup) | 12 to 24 Hours depends on the role of the VM |
| Domain Controller | Azure Virtual Machine | 24 Hours ( Daily once backup) | 4 Hours - per Major incident guidelines |
| Terrafrom State | Blob soft delete and version control | continuous | 4 Hours - per Major incident guidelines |
| Azure Workbooks used for Monitoring | Version control in git repositories | continuous | 24 hours as it is not a major or significant incident |
| Credentials-Secrets in Keyvault | Keyvault soft delete and version control | continuous | 4 Hours or more -depends on the use of the key/secret |
Access Required for Restore
ALZ team members do not have the required RBAC role in Production to carry out the restore steps mentioned in the document below.
Therefore it is proposed to have a new custom role made available with Backup Operator rights.
This role can be activated using Azure AD PIM process as and when a restore has to be carried out.
Until then the steps can be performed by contacting Matt White /Aaron Ruecroft as they have the required access over the Hub and SharedServices subscription.
Restore - Virtual Machines with non ADE encryption
Our Virtual machines use SSE + PMK ( platform managed) encryption. Hence restoring them in place is easy and can be carried out using following steps.
Navigate to Backup center in the Azure portal and click Restore from the Overview tab.
Select Azure Virtual machines as the Datasource type, and then select a Backup instance.
Select a VM and click Continue.
In the next screen that appears, select a restore point to use for the recovery.
Restore - Virtual Machines with ADE encryption
If a VM ( in future) has ADE encryption then restoring in place is not an option and recovery process becomes little complicated.
As this option will not allow to restore the backup on top of original disk, you will need to restore the disk in a staging location ( another storage account in the same sub).
Ensure the disk has same name as that of the original disks otherwise terraform will complain on next run.
Restore - Domain Controller VMs
Important: No restore should be performed without a approved Change and unless requested and approved by ATOS/DXC.
The restore process for the general vms and domain controllers is similiar upto the point the disks are restored.
This is then followed by Active Directory database restoration. This step is out of ALZ support remit and is carried out by ATOS/DXC or relevant TechOps team in future.
Restore - Terraform State
The terraform state files used by various Azure DevOps pipelines are stored as individual files as Block blob. Thsese blobs are located in the following storage accounts.
| Enviornment | Subscription | Storage Account | Container |
|---|---|---|---|
| Production | Hub | samojtfstate003 | tfstate |
| PreProd | Hub | samojtfstate002 | tfstate |
| Development | Hub | samojtfstate001 | tfstate |
All terraform state file end with a .tfstate extension and all of them have softdelete and version history enabled.
Restore Process
Navigate to the respective storage account –container–tfstate.
Enbable the Show Deleted Blob radio button.
You will then see the deleted state file.
Click on the ellipsis of the deleted blon and select Undelete from the contextual menu to recover the deleted blob (and thus the file).
Restore- Azure Workbooks
Our Azure workbooks are used primarily for Azure Monitor and are version controlled at all times in the following path
Restoring a deleted workbook is thus only a matter of running the Pipelines again.
Restore- Keyvault Secrets and Credentials
We have enabled soft delete but purge protection is disabled. Soft delete functionality is sufficient to recover the keys and secrets in an event of accidental deletion.
Furthur each key and secret also has version history enabled by default.
Steps
- Follow the steps given in list-recover-or-purge-soft-deleted-secrets-keys-and-certificates