Deployments
Deployment configuration
The main deployment scripts used by all the repos:
fb-builder
This is a base image for building the other applications and has all the 3rd party software included. It has recently been updated to be store in Dockerhub instead of AWS ECR.
It is pushed to docker hub on merging into main using CircleCI using the formbuilder developer account.
fb-deploy
This is a bash script that is triggered during the pipeline which builds and deploys the applications into cloud platform. It covers almost all the applications in the platform.
Editor
Like other MoJ Forms applications, the Editor is deployed in a Continous Delivery manner using CircleCI with a manual gate to production.
Since it has a significant amount of feature development work that is done concurrently it has a slightly different workflow than the other applications. A developer can deploy out a separate editor in the Test environment that is entirely independent of the one that takes the code from main.
The Editor that uses the code on main can be found at: https://fb-editor-test.apps.live.cloud-platform.service.justice.gov.uk/.
Workflow
It is important that all the unit tests and acceptance tests have been updated and are passing before creating a testable Editor. The purpose of this separate Editor is primarily for manual testing away from the main branch. Developers should have already written or updated any unit tests and acceptance tests that are associated with their feature.
The Developer checks out a branch locally and prepends the name of that branch with testable-. eg testable-some-new-awesome-feature
This branch is what is used to create the URL that a user will visit. The example above becomes https://testable-some-new-awesome-feature.apps.live.cloud-platform.service.justice.gov.uk. Therefore it is important to not use / in branch names as that would then create an invalid URL.
Good: testable-make-it-work
Bad: testable-feature/make-it-work
Once your branch has been deployed out you will see a message in the #form-builder-deployments channel in Slack complete with a URL which can be visited. similar to this one:
If however your branch build fails for some reason Jean Luc will express his disappointment with your failure and hope you will do better next time.
Pipeline overview:
Caveats
Shared Database
All the different Editors share the use of the same database. This means that we should be very careful when doing anything that involves database migrations as this can lead to many issues.
Also the Workers containers for all the Editor apps share the same database too. When a Delayed Job (Publishing or Unpublishing) is first created on the database there is no guarantee that the Worker that picks up the job to process it will be running the same code as the app where the Job was actually created. Any of the available Editor apps could be the one to pick up the job and process it.
Cross Container Environment Variable Contamination
This one is a bit of tech debt but only relates to the Test environment. As a consequence of having n number of Editor containers the fb-deploy scripts do not take into account multiple versions of the configurations. Therefore some of Kubernetes default environment variables for the service ports are being set inside all the other containers that are deployed after a testable editor is deployed.
We will fix this at some point, but it does not impact performance directly so has been deprioritised, but developers should be aware.
Testable Editor teardown
Once all testing has been completed the Developer can raise a PR from the testable branch as normal for review etc. Once the branch has been merged (and deleted in Github, which is automatic for the Editor repo settings) then a job in CircleCI is triggered which runs a rake task:
bundle exec rails remove:testable_editors
The task looks for any branches prepended with testable- that have been removed from Github but still exist as deployments in Kubernetes and then proceeds to remove all the required configuration associated with those specific Editors and should notify in the #form-builder-deployments channel in Slack.
It is dependent upon the environment variable K8S_SAAS_TOKEN being set. This is the token for the CircleCI service account associated with the formbuilder-saas-test namespace. Most other CircleCI jobs use the token associated with the formbuilder-repos namespace CircleCI service account.
If for some reason this is not working or the Developer needs to run it independently of CircleCI they can run the same rake task on their laptop. Before being able to run this the Developer should have set up their Kubernetes config as documented in the Cloud Platform user guide.
Manual Deployments
It is also possible to manually deploy out a specific branch to the test environment. This method will overwrite the Editor at https://fb-editor-test.apps.live.cloud-platform.service.justice.gov.uk/.
On the branch you are working on just make some changes to the circle config workflows to add the name of your branch so it triggers CircleCI to deploy it out. For example, given a branch named my-amazing-branch:
workflows:
version: 2
test_and_build:
jobs:
- build_and_deploy_to_test:
filters:
branches:
only:
- my-amazing-branch
Pushing up your branch will then trigger CircleCi to deploy it out to the test environment. It might be worth considering commenting out all the jobs in the workflows related to Live and perhaps even Slack notifications.
This approach works for pretty much any app should it be required. The recommendation would be that you would add the above changes to a temporary commit that you would remove from your branch before raising the final PR for review.
Make sure to notify the other developers that you are deploying your branch.