Data platforms and data management

A data platform is a central place where data from different source systems is collected, cleaned, combined and made available for reports, analyses and other applications. The source system stays authoritative; the platform is a copy set up to answer questions.

The signs that you need one are usually recognisable:

• two departments report a different number for the same concept;
• reports are produced by combining exports from several systems in a spreadsheet;
• heavy reporting queries slow down a source system for the people working in it;
• nobody knows exactly how a figure in the annual report came about.

A data platform is not needed if you have one source system with good built-in reporting, or if the questions come up so rarely that a manual export is enough. If the problem is mainly one spreadsheet that has grown into a business application, custom software often fits better. If you are unsure, start with one question and see whether the answer is worth the effort.

A data platform does not have to be a big project. It consists of four layers that you can build up step by step, starting with one source and one report.

In the set-up we propose, the raw data is kept as it arrived. If a transformation goes wrong, you recalculate from the raw layer instead of querying the source again. That also makes the platform auditable: every result can be traced back to the original records.

This stack is chosen for portability. PostgreSQL is an open source object-relational database that has been developed for decades and that almost every data tool can read. Python is the usual language for collecting and processing data. Almost every data analyst knows SQL. That makes it easy to let your own people or another party work on the platform later.

With dbt, transformations are written as SQL files kept under version control. Every change to a definition can therefore be traced, including who made it and why. Apache Airflow is a platform for defining, scheduling and monitoring workflows programmatically in Python, so it is visible which load succeeded and when.

Open source does not mean there is no maintenance. Every PostgreSQL version has an end-of-life date: PostgreSQL 14, for example, receives no more fixes after 12 November 2026. A maintenance agreement for a data platform therefore includes an upgrade plan, not only fixing outages. If you have location data, PostGIS adds spatial functions to the same database; see GIS and digital map solutions.

A data platform is only valuable if people trust the figures. That trust does not come from an attractive dashboard but from checks that run automatically and from definitions that are written down.

In practice that means:

• Data tests on every load. Checks that a column contains no empty values or duplicate keys run with every load. If a test fails, the report is not silently updated.
• One definition per concept. What counts as an active customer, a resolved report or an occupied address? That definition exists once in the platform, with a description in plain language.
• Provenance per figure. For every model it is visible which sources and intermediate steps it is built from.
• Monitoring of loads. If a source stays empty or suddenly far fewer records arrive, the administrator receives a notification.

The dbt documentation defines such checks as follows:

Data tests are assertions you make about your models and other resources in your dbt project (for example, sources, seeds, and snapshots).

Source: dbt Developer Hub, "Add data tests to your DAG". The platform does not fix errors in the source. It does make them visible, so the owner of the source system can correct them. That is often one of the first benefits.

We propose building a data platform around one concrete question rather than a complete design up front. The steps below are the default approach; the plan for your organisation is agreed after the first step.

1. One question, one report. Choose a report that now costs a lot of manual work and record which figures it contains and who uses them.
2. Source analysis. For each figure, trace the source system, the fields involved and the owner who can explain their meaning.
3. Raw layer. Load the sources unchanged and on a schedule, with the load time stored for every record.
4. Models and tests. Write the definitions as SQL models, each with data tests, and compare the outcome with the current manual report.
5. Exposure and handover. Connect the report or dashboard, document definitions and loads, and only then add the next source.

Comparing the new figures with the old report in step 4 almost always uncovers a definition difference. Settling it is part of the work, not a delay.

For government bodies and public organisations, a data platform is often also the source for open data. It is then wise to include publication in the design instead of building a separate export later.

Two things are needed. First, metadata following DCAT-AP-NL, the Dutch profile for describing datasets and data catalogues. Version 3.0 is a Geonovum standard; the adopted version is dated 26 May 2025. Second, a catalogue or portal where the datasets can be found. CKAN is a widely used open source data management system for data portals.

In the platform this means that every publishable dataset gets its own controlled publication layer, without fields that may not be public. The metadata is generated from the platform itself, so description and content do not drift apart. If you are looking for a full open data portal, see the open data platform. How a municipality organises publishing is covered in the knowledge base under publishing open data as a municipality (in Dutch).

A data platform brings together data that was previously separate. That increases its value, but also the risk of misuse. Anyone with access to the platform can make combinations that were not possible in the source systems.

For that reason we set up the platform on a few principles:

• collect only the fields needed for an agreed purpose, and record that purpose per source;
• replace directly identifying data with a pseudonym as early as possible, so analysts do not work with it;
• arrange access per layer and per dataset, with separate roles for administration, analysis and read-only use;
• enforce retention periods technically, so old data really disappears;
• record who runs which queries on sensitive tables.

Whether your platform needs a data protection impact assessment is something you decide with your data protection officer. We supply the technical description needed for it: which data, which links, who has access and how long everything is kept. That description grows with every new source.

A data platform touches many systems and departments. The technology is rarely the bottleneck; agreements on ownership are.

• For each source system, an owner who arranges access and answers questions about the meaning of fields.
• Preferably read access through a copy or replica, so the platform does not load the production environment.
• Someone who decides on definitions when departments disagree.
• A first concrete question to start with, preferably a report that currently costs a lot of manual work.
• A choice of where the platform runs; the trade-offs are covered under cloud and data infrastructure.

Tekin Uslu, founder of TechinFlow, has by his own account worked on geo-information, data engineering and IT architecture since 2006. More about his background is on the Tekin Uslu page. Preferably start small: one source, one question and a report that currently costs a lot of manual work. If you want to discuss such a first step, get in touch.