REAL-TIME INVENTORY MANAGEMENT PLATFORMBack Case Studies
Client: Global provider of technology solutions for retail industry
Project Duration: 6 months
Goal: Assist in reviewing and implementing a Kafka-based real-time stock counting subsystem
Tech: Java, Spring, Apache Kafka, RabbitMQ, Apache Cassandra, InfluxDB, Grafana
The client has developed a new addition to their existing solution - a real-time stock counting subsystem (RTS). It is integrated into the existing system. RTS receives a stream of EPCIS events, processes them in real-time and maintains the status of all known EPCs and stock account information per business location. RTS needs to answer high data volume demands as well as to ensure 100% correct stock information (no EPC event can be lost).
In parallel, the whole system is connected to ERP systems of clients and in batch mode, stock is imported from the ERP system as well. When there is both real-time stock count and stock count from the ERP system, the difference list is calculated and presented to the client through web UI.
The engagement was split into phases. The goal for the first phase was to review the existing RTS project, to apply the best practice applicable to the development of a Kafka-based system and, finally, to suggest any necessary modifications and tuning options. The scope of the second phase included the implementation of the suggested modifications. The final, third phase was implementing a monitoring system as a part of RTS to be used in production environment and a load generator used to perform end-to-end tests.
The first phase was executed taking into account the latest Kafka development (Kafka version 0.9) and best practices and recommendations related to it, and matching them against the existing version of the RTS project (Kafka version 0.8.2). The review also revealed problems related to horizontal scaling and no message loss guarantees. The findings from the first phase were addressed in the second phase by switching to the new Kafka consumer API. The necessary preparations for the third phase were made during this phase by integrating metrics libraries/tools (Java Metrics and Kafka Burrows) into RTS. In the third phase a monitoring stack (consisting of Telegraf, InfluxDB, Grafana, Burrow) was developed and integrated with RTS.
At the end of this engagement, RTS was deployed in the production environment together with the monitoring stack. Having RTS in the production environment enables end users (retail organizations) to access the stock information such as stock counts and product lists in real-time easing and improving their daily operations. The monitoring stack allows for faster spotting and preventing situations that could lead to potential problems in production. Using Apache Kafka for implementing the data pipeline builds a scalable platform for further system extensions easing the new development and cutting down the time-to-market.