abaQon Project Report: Provision of Personal Information With Kafka Streams

THE PROJECT

A big Swiss Retail Bank required a resilient Kafka Streams application that aggregates customer information and provides a single output Kafka topic with all relevant customer information. The output topic will serve as the source of truth for several consumers, such as the banks credit application or the leasing platform. Data is exported from the bank’s core banking system into various Kafka topics and any relevant change in any of the input topics should be available within minutes to the consumers.

OUR CONTRIBUTION

abaQon was involved in the design and the implementation of the Kafka Streams application. By analyzing the stakeholder requirements we outlined the desired topic, its properties and the consequences on the application design. The implementation was done in Kotlin. We used a test-driven development approach which helped massively in shaping the application to cover all edge-cases. Additionally we used Python to generate large amounts of synthetic data that helped us to perform load tests.

REALISED ADDED VALUE

The application provides a unified and real-time view on customers base data that satisfies the needs of all consuming services. It is fed directly from the core banking system and due to its scalability it is able to manage large initial loads of data as well as the daily stream of updates. Changes in any of the six input topics that are not relevant for our output do not trigger a new message. This helps to reduce the data load on the application and the output topic.