About

Data Engineering

What do we do?

Data engineers bridge the gap between data producers and data users.

We work with digital teams to help them make data from their services available on the Analytical Platform.

We do this by building and running data pipelines ourselves, and by supporting digital teams to use self-service tools and infrastructure on the Analytical Platform.

We also contribute to the development of the Analytical Platform and other foundational data infrastructure, including the data uploader, Airflow, Create a Derived Table, Register my Data (internal only), the data extractor (internal only), and various packages.

What value do we provide?

We enable access to data in a way that is more reliable, secure, auditable and timely than was ever previously possible. Where possible, we provide historical data to support reproducible and longitudinal analysis.

Analysts and data scientists can access complete data from a range of sources in a single location and in a standardised format, reducing preparation time.

Data can now be received daily, instead of weekly, monthly or quarterly, generating insight that is more relevant and current.

Analytics Engineering

What do we do?

Analytics engineers start with the raw data from source systems provided by data engineers.

We transform, cleanse and link it to produce curated data sets that are more useful and useable for analysts and data scientists, accelerating the production of statistics, MI, analysis, and data science products.

What value do we provide?

Analysts and data scientists need to spend less time transforming and understanding complex and messy data sets, so they can start deriving insights faster and make better use of their technical skills to solve new problems.

Modelled data is trusted and consistent across use cases, meaning we get the same answers to the same questions no matter where we ask them.

Data is domain-aligned, not source-aligned, facilitating a simpler transition from heritage systems to new services.