Data-driven applications, from business intelligence (BI) tools and data services to AI Agents need data. Some in particular need demographic data to enrich profiles and analyze groups, such as population, age structure, households, education, income and media use. The Federal Statistical Office provides census and population data through the GENESIS-Online database, the Federal Institute for Research on Building, Urban Affairs and Spatial Development (BBSR) publishes regional indicators in the INKAR online atlas, and the ARD/ZDF media analysis reports figures on media usage. Each source comes with its own formats, interfaces, and metadata conventions. Open data promises substantial benefits, yet in practice it fails on access, format, and processing barriers. 

Existing general-purpose ETL/ELT platforms (Extract-Transform-Load/ Extract-Load-Transform) focus primarily on data movement rather than the integration of official statistical data. Although such platforms can access official sources through generic REST/HTTP or custom connectors, they generally lack dedicated connectors for (German) official statistical APIs and do not transform heterogeneous data sources behind a unified, standardized API for its consumption. Anyone who needs such data today researches manually, converts formats and cleans data and, in the best cases, builds its own toolbox to repeat the process and, in the worst case, they repeat the manual process each time. 

This project therefore builds an ETL/ELT pipeline with multiple predefined and pre-configured connectors (with an extensible architecture) that makes German open data sources machine-accessible, maps them onto a unified target schema (to be defined as part of the project) and exposes them through a documented REST API and MCP Server, so that clients systems can query clean, normalized data.

Project Objectives:

The goal of this project is the conception and implementation of an extensive ETL/ELT pipeline with a unified REST Interface and MCP Server for German open data sources. The focus is on

1. Selecting at least three concrete sources (e.g., GENESIS-Online, the 2022 census, INKAR) and identifying their requirements, including documentation of their interfaces, formats, and usage licenses,
2. designing a unified data model (target schema) that standardizes regional keys and values, 
3. identifying the state-of-the-art for ETL/ELT pipelines and selecting an open-source tool to use, or build one as part of the project if none is found to fulfill the requirements, 
4. building an ETL/ELT pipeline for each source based on (3),
5. providing a documented REST API and MCP Server through which client systems can query clean, normalized data, 
6. making these pipelines and its results monitorable and reproducible and 
7. evaluating the pipeline against defined criteria (query time, data quality, resource usage, extensibility to one additional source).

The choice of the frameworks and languages is to be defined in the first project meeting. Python and established, maintained open-source tools are recommended.

Project Requirements: 

• Creation of a project plan and task allocation among group members
• Requirements analysis and documentation of the selected data sources (interfaces, formats, licenses)
• Design of a unified data model (target schema) including a metadata concept
• Identification or ELT/ETL pipeline existing alternatives and selecting one by comparing the relevant features for the project
• Implementation of a working ETL/ELT pipeline for at least three sources
• Development of a documented REST API (including an OpenAPI specification) and MCP Server
• Evaluation of the pipeline against defined quality criteria
• Project documentation and presentation of interim and final results

Prerequisites:

• Students of the Faculty of Computer Science in the fields of Information Systems or Software Engineering at bachelor’s level (with at least 90 ECTS) or at master’s level
• Basic knowledge of databases/SQL and at least one programming language (e.g., Python)
• Interest in data integration, API development, and official open data sources
• Experience with Docker, workflow orchestration, or data warehousing is a plus, but not required

References

• Vassiliadis, P. (2009). A survey of extract–transform–load technology. *International Journal of Data Warehousing and Mining*, *5*(3), 1–27. https://doi.org/10.4018/jdwm.2009070101 
• Vetrò, A., Canova, L., Torchiano, M., Orozco Minotas, C., Iemma, R., & Morando, F. (2016). Open data quality measurement framework: Definition and application to Open Government Data. *Government Information Quarterly*, *33*(3), 325–337. doi.org/10.1016/j.giq.2016.02.001 
• Wang, R. Y., & Strong, D. M. (1996). Beyond accuracy: What data quality means to data consumers. *Journal of Management Information Systems*, *12*(4), 5–33. doi.org/10.1080/07421222.1996.11518099 
• Willekens, F. (2023). Programmatic access to open statistical data for population studies: The SDMX standard. *Demographic Research*, *49*(40), 1117–1162. doi.org/10.4054/DemRes.2023.49.40 
• Zuiderwijk, A., & de Reuver, M. (2021). Why open government data initiatives fail to achieve their objectives: Categorizing and prioritizing barriers through a global survey. *Transforming Government: People, Process and Policy*, *15*(4), 377–395. doi.org/10.1108/TG-09-2020-0271 

If you are interested in this porject, follow the steps below to submit your application.

1. Form a project group
Projects are typically carried out in groups of 3–5 students. We recommend forming a group with fellow students before applying.

2. Prepare your application
Send a short application including:

• your transcript of records
• a short motivation letter (about one page) explaining why your group is interested in the project.

3. Submit your application
Send your application via email to:jannis.nacke (at) icb.uni-due.de