Alerts Quick Guide
Most of the information can be found in this Runbook
This is a quick guide about the common alerts you may come across in the #form-builder-alerts channel.
Alerts in this channel are high severity and affect live environment. Whereas alerts in the low severity channel concerns the test environment.
Application Related Alerts
Failed Delayed Jobs
This is something that needs to be investigated. This issue can occur for submitter or the editor.
A walk through of how to investigate for the submitter is found in this documentation.
For the editor, the guide can be found here.
Client Error Response
These usually resolve within 5 minutes of firing. If there are a number (say 5 or more) of these alerts in short succession, it could be a bot trying to get into the service.
Slow Responses
These usually resolve quickly after firing with no intervention needed.
Kubernetes Related Alerts
Kube Namespace Quota Nearing
We have an alert that lets us know when we hit 80%. If this goes off it could be because the pods are creating and draining, which is normal. To check this run:
kubectl get pods -n <namespace>
This will give you all the pods in a namespace. There should be 2 for each form, if there are some that are restarting then you may have more than 2 - which could be why the quota has been reached. If this is the case, then there is no issue. If there are no pods restarting in the list, the best to chat with the team about whether we need to increase the number of pods.
Kube Pod Crash Looping
Each form has 2 pods, so the form should still be running despite a pod being down. Check the form, then check the logs. A way to fix this could be to restart the problem pod.
Documentationon checking the logs and restarting the pods.
Namespace Missing
To check whether the namespace is missing:
kubectl get namespace | grep < namespace >
If the namespace is there, then there isn’t much to worry about. If it doesn’t resolve itself and the namespace is there, it might be worth flagging with cloud platform. If the namespace is in fact not there, then we have a problem and will need to let the other devs know.
Kube Pod Not Ready
There are always 2 pods for every form, as such this alert is generally nothing to worry about. This should resolve itself once the pod is up and running.
Testable Editors
Sometime we will see a ‘Kube Pod Not Ready’ error for a testable editor pod.
When an editor branch has a prefix of testable every push on that branch creates a web and worker pod.
If a testable editor branch is merged into main, the deployment pipeline will trigger a deletion of the testable branch and the associated pods.
However, sometimes testable branches may not need to be merged in and are deleted without being merged, which leaves us with testable editor pods that may trigger alerts.
If a testable editor pod is causing alerts and the testable branch is stale or has been deleted, we will need to manually remove the deployment resource.
To check what deployment resources are running:
kubectl get deployments -n <namespace>
For each testable editor branch, there will be an associated web and worker deployment, for example:
NAME READY UP-TO-DATE AVAILABLE AGE
testable-my-branch-web-test 2/2 2 2 5d
testable-my-branch-workers-test 2/2 2 2 5d
Once we are happy that the testable editor is no longer being used, we can remove both the web and worker deployment resources.
kubectl delete deployments -n <namespace> <deployment>
For example:
kubectl delete deployments -n formbuilder-saas-test testable-my-branch-web-test
Sentry related alerts
All Sentry alerts are found on the Sentry dashboard. Please ensure the alert is resolved once you have completed your investigation, otherwise we will not be alerted for the unresolved error in future.
Faraday::Unprocessable Entity Error
This is the metadata-api returning a 422, this happens when either:
- The user has tried to create a page with a URL that already exists
- The editor is trying to send metadata to the API that it shouldn’t be
If the issue is the former, then that is not anything we need to investigate.
However, if it is due to the Editor sending metadata it should not be sending, we will need to investigate the issue further.
Action View Template Error
This could be due to a form owner interacting with a form that was made prior to breaking changes were introduced. This is not a serious issue, we may need to ask the form owner to re-publish a form to get the latest changes.
Net Read Timeout
This usually occurs on the HMCTS complaints adapter. Generally it is a communication error with Optics, it is best to keep an eye out for any ‘Failed Delay Jobs’ alerts as this will point to the job not being processed. We can also check if there any delayed jobs on the api pods of the hmcts-complaints-formbuilder-adapter-production namespace by running Delayed::Job.count in a rails console.
JSON::Schema::ValidationError
First seen in 2023, and happens when the Editor sends metadata to the API that does not conform to the schema. This is a similar issue to the Faraday::Unprocessable Entity Error but with the submitter, however this is a more specific error that points to the exact issue with the metadata (null value in the schema). We can not replicate this error to provide a fix. However, it does not happen frequently. Okay to Resolve and check for a delayed submission alert.
Aws::SESV2::Errors:: BadRequestException - Unbalanced quoted string
This occurs when sending an email via Amazon SES in app/services/adapters/amazon_ses_adapter.rb. The from_email_address is built from the service name (opts[:from].split('<')[0]) which, when it contains an unbalanced quotation mark, produces an invalid from header such as Some "Service<no-reply@example.com> and SES rejects it with Unbalanced quoted string.
The root cause is a service name containing an unbalanced quote. The fix is to ensure the service name portion is properly quoted/escaped before being used in the from_email_address.
Affects the form filler (email fails to send); caused by the form owner’s service name. Do not resolve until the service name is fixed, or it will keep recurring.
Adapters::PdfApi::ClientRequestError
This is also seen in the pdf-generator as PDFKit::ImproperWkhtmltopdfExitStatus (Command failed exitstatus=1).
This happens when someone submits a form that emails a PDF copy of their answers.
The submitter asks the pdf-generator to turn the answers into a PDF. The pdf-generator uses a tool called wkhtmltopdf to build the PDF. Sometimes that tool fails on a particular submission, so the pdf-generator returns an error and the submitter reports it too.
You may see it in two places:
- In the submitter:
Adapters::PdfApi::ClientRequestError - ... returned response status of: 500 - In the pdf-generator:
PDFKit::ImproperWkhtmltopdfExitStatus - Command failed (exitstatus=1)
Both are the same problem. The cause is something in that one submission’s answers that the PDF tool can’t handle. It only affects that submission.
- Both error reports share the same submission ID. Use it to confirm they come from the same submission.
- Use the service ID from the error to find the form in the Editor Admin feature (see “how do I find out which form is causing the error?”). This tells you which form and user were affected.
- The full details of the tool’s failure are only in the pdf-generator error report, so check there.
Note the affected submission may not have received its PDF.
Affects the form filler (may not get their PDF); one submission only. Okay to resolve once you’ve identified the form and submission.
Faraday::ForbiddenError
This is raised by V2::ProcessSubmissionJob while it downloads a submission’s uploaded attachments in app/services/download_attachments.rb (around the connection.get('', {}, headers) call). The submitter fetches each uploaded file from the user filestore and authenticates that request with the headers built in DownloadAttachments#headers. A 403 Forbidden means the filestore rejected that request.
Due to the age of the error when investigating (>28 days), the exact cause is not clear, but there are a few likely causes:
- Expired / out-of-skew access token - the JWT access token saved with the submission is generated at submission time. If the job runs (or is retried) long after the submission was created, the token’s
iatcan fall outside the allowed skew window (MAX_IAT_SKEW_SECONDS) and the filestore returns a 403. - The uploaded file has already been removed from the user filestore, so the request is no longer authorised.
If it was a normal run, a stale token past the 90 second skew window might be the cause. If it was a replay, the token skew has been overridden to 28 days.
Affects the form filler (attachments may not reach the form owner). Okay to resolve if it’s a one-off — check for a ‘Failed Delayed Jobs’ alert first.
Adapters::JweWebhookDestination:: ClientRequestError -> Prawn::Errors::UnknownFont
This appears as a 422 from Adapters::JweWebhookDestination#send_webhook when V2::ProcessSubmissionJob posts to https://track-a-query.service.justice.gov.uk/api/rpi. The 422 is the symptom. Api::RpiController#create on track-a-query catches an exception and returns head :unprocessable_entity.
The real error in track-a-query is Prawn::Errors::UnknownFont, raised in RequestPersonalInformation::FileBuilder#zip_pdf while generating the submission’s PDF. To handle arbitrary submitted characters, the PDF uses a fallback font, “LastResort”, but only a normal style is registered:
pdf.font_families.update(
"LastResort" => { normal: LAST_RESORT_FONT_FILE },
)
pdf.fallback_fonts(%w[LastResort])
If the submitted content (rpi.to_markdown(target)) needs italics, Prawn looks for an italic “LastResort” that doesn’t exist, and raises instead of generating the PDF.
A potential fix would be to add the italic font by including italic: LAST_RESORT_FONT_FILE.
Affects the form filler (submission never reaches track-a-query). Do not resolve — needs a fix from the track-a-query team.
TypeError: Cannot read properties of undefined (reading ‘position’)
Occurs in fb-editor applyPageFlowConnectorPaths (app/javascript/src/controller_services.js) when drawing connector arrows in the detached flows section of the flow diagram.
The page is trying to draw an arrow to its “next” page, but that page isn’t in the diagram section being drawn, so the lookup returns nothing and the code fails reading .position.
This is already caught and reported via view.sentry.send(err), so it does not break the page, it just skips drawing that one arrow leaving a detached branch with a missing line.
Affects the form owner in the Editor only, and is cosmetic (one missing arrow). Okay to resolve.
TypeError: Cannot read properties of undefined (reading ‘$node’)
Happens in the Editor when a form owner selects “Delete option” twice for the same option on a radio or checkbox question. Selecting it does not close the menu, and the confirmation dialog is fetched from the server before it appears, so a second selection is possible while the first is still loading. Two dialogs then open, one behind the other.
Confirming the first dialog deletes the chosen option correctly. Confirming the second dialog looks for that same option again, cannot find it because it has already gone, and deletes the last option in the list instead. It then fails before removing that last option from the page.
So two options are deleted: the one the form owner chose, and the last option in the list, which they never selected and were never asked about. The last option is still visible on screen, so the form owner has no way of knowing it has been removed. Saving deletes it permanently, with no warning.
More likely when the API is responding slowly, as this widens the window for a second selection.
Reproduced (Aug 2026):
- Open a radio or checkbox question with at least 3 options.
- In developer tools, add a network throttling profile with a latency of 3000ms and select it. This mimics a slow API response.
- Open the menu for an option that is NOT the last one in the list, and select “Delete option”.
- Wait around 1.5 seconds and select “Delete option” again from the same menu.
- Two confirmation dialogs open. Confirm both.
- The chosen option is removed, and so is the last option in the list. The error is thrown. The last option stays visible on the page until it is reloaded, even though the Editor has already discarded it.
Affects the form owner, who silently loses the last option in the question. Form fillers are affected once the form is published, as that option will be missing from the live form.
Do not resolve until fixed. Contact the form owner and ask them to check whether the last option is missing from the affected question, as they will not have been told it was deleted.
Error: cannot call methods on dialog prior to initialization; attempted to call method ‘open’
This is raised in fb-editor at app/javascript/src/component_dialog_api_request.js when open() is called on a dialog that no longer exists.
Dialogs in the Editor are ephemeral, a new DialogApiRequest is created each time an action is triggered, and the dialog is destroyed and removed from the DOM when it closes.
The activator (the link or button that opens the dialog, for example “Delete page…” in the page menu) is not destroyed with it. When the same activator is used again, the click listener added for the previous dialog is still attached and calls open() on that now-destroyed dialog, which throws this error.
The user does not see a problem and the new dialog still opens correctly, but the stale call is reported to Sentry. It only occurs from the second time onwards when the activator is used.
Affects the form owner in the Editor only, with no visible problem. Okay to resolve, or archive if noisy.
UserDataStoreClientError: ENOERROR
Happens when a form filler starts or returns to a form and the user data store refuses to save or load their session.
The error appears in two places, both with the same cause: saving a new session (loadUserData → saveData, a POST) when a form filler starts a form, and reading an existing session (loadData → getData, a GET) when a form filler returns to one. On the read path a 404 (no saved data) is handled quietly, but a 403 surfaces as the ENOERROR. Everything below applies to both.
“ENOERROR” is not a real error. It is what the code says when it cannot work out what went wrong, so it tells you nothing on its own. Look at the Sentry breadcrumbs for the web request just before it — that is the real error. So far the underlying error has always been a “403 Forbidden” from the user data store.
A 403 means the user data store did not trust the request. It gives the same 403 for every reason, so the cause is one of these:
- A missing key — before trusting a request, the user data store asks another service for a key to check it against. If that request fails, or returns something unexpected, it carries on with no key and rejects everything. Usually a brief network problem.
- The key did not match — the key was found but was the wrong one. More serious, raise with the team.
- The request’s timestamp was outside the allowed window — set by
MAX_IAT_SKEW_SECONDSon the user data store. The check trips if the timestamp is too old or too far ahead, and usually means the clocks on two servers disagree. Note this setting is per-service, so the value quoted in the Faraday::ForbiddenError entry does not apply here. IfMAX_IAT_SKEW_SECONDSis ever unset, the allowance becomes zero and almost everything is rejected.
Affects the form filler, who could not start or resume the form. Form owners are not affected.
Okay to resolve if it is a one-off. Investigate if it happens again or affects several form fillers at once.
There is a Sentry alert - how do I find out which form is causing the error?
When a Sentry alert is triggered, general information regarding the alert can be accessed via the Sentry dashboard. There will be a URL that is provided in the Sentry Issue, this will have the service ID in the path.
We can search for this service ID in the Admin feature of the Editor. Once you have identified the service in the Admin dashboard you will be able to pinpoint the form and user that relate to the issue. If required, you can copy the form metadata to debug further.