System integration · Enterprise Service Bus

Data bus (ESB): a custom enterprise service bus for e-commerce

A custom Enterprise Service Bus for a large online store: high-speed exchange between heterogeneous databases with multithreaded sync, a load-control admin panel and transaction logging. Data migration moved from night windows to a continuous, controlled flow, and performance grew 10x.

ClientA large online store with high volumes of commercial and user data

IndustryE-commerce, high-load integrations

PlatformsServer software, data integration bus

TimelineAbout 3 months

ESB data busE-commerce, high-load integrations

About the product

The client is a large online marketplace that works with high volumes of commercial and user data. As the business grew, moving and synchronizing data between internal systems put more and more strain on the infrastructure.

To avoid disrupting daytime operations, data migration could only run at night, when server load is minimal. But volumes kept growing, and at some point even the night window was no longer enough: the processes started affecting the service during the day. The client needed high-speed data exchange without downtime.

The challenge

Because of server load, data migration could only run at night. As data volumes grew, the night window became too short: sync processes began overlapping with daytime operations and interfering with customer service.

The client needed an architecture that would move and synchronize data between heterogeneous databases continuously throughout the day without degrading the core service, while giving administrators control over load and priorities.

Project highlights

Multithreaded transferReal-time load controlLogged transactionsDatabase optimizationAdmin panel for prioritiesContinuous flow instead of night windows

Solution and approach

We designed and built a custom Enterprise Service Bus that handles exchange between the online store’s heterogeneous systems. At its core is multithreaded transfer: several databases sync at the same time rather than one after another, which dramatically speeds up migration.

To keep load under control, we built an admin panel for managing priorities and throttling the flow in real time. Every operation runs as a logged transaction, and the database architecture was further optimized for high-speed exchange. As a result, data transfer moved from a night window to a continuous, controlled flow.

Product features

Multithreaded transfer

Simultaneous synchronization of several heterogeneous databases

Load management

Priority allocation and real-time flow throttling

Transaction processing

Reliable transfer with every operation logged

Admin panel

Exchange monitoring and manual priority control

Database optimization

Database architecture reworked for high-speed exchange

Access control

Role-based permissions and data protection under the client’s regulatory requirements

ESB architecture diagram connecting e-commerce, data warehouse, ERP, CRM, EDI and third-party applications

ESB data bus architecture

The diagram shows how the Enterprise Service Bus becomes a single exchange point between heterogeneous systems. E-commerce, the data warehouse and ERP connect to the bus from above; CRM, custom applications, the business analyst workstation, the EDI notification service and third-party applications connect from below.

Instead of many point-to-point integrations, all systems communicate through the bus: it routes messages, syncs data in multiple threads and lets new participants join the exchange without rewriting existing connections.

Why a custom ESB instead of an off-the-shelf product

For a large online store with heterogeneous databases and non-standard load-management requirements, off-the-shelf ESB products (MuleSoft, Apache Kafka, IBM MQ) require lengthy customization and licensing costs. The team designed a solution for the client’s specific data architecture: high-speed multithreaded transfer between heterogeneous databases with flexible load control through an admin panel.

A dedicated architect on the project kept technical decisions consistent and reduced the risks of integrating with the client’s existing systems. This made it possible to move from night-time operations to a continuous flow without affecting the service during the day.

By the numbers

10xfaster data exchange (the target was 5x)

~3 moproject delivery time

24/7continuous exchange instead of night windows

RBACrole-based access and logging of every operation

Goal: make data migration 5x faster than the original baseline.

ESB plus admin panel: two components in one solution, a data bus and a load-management tool.

Multithreaded transfer: simultaneous synchronization of the large online store’s heterogeneous databases.

A dedicated architect on the project ensured consistent technical decisions and reduced integration risks.

Night-time operations replaced by a continuous flow with load managed through the admin panel.

Typical delivery window for an ESB of this class is 12–24 weeks (industry range, not a guarantee).

Data, architecture and security

The bus handles the commercial and user data of a large online store, so reliability and data protection were critical. Every operation runs as a logged transaction: changes are never lost, and the flow can be traced and controlled through the admin panel.

Security and data integrity are built into the architecture: transactional processing, operation logging and role-based access to the admin panel. Data is processed under the requirements of the client’s jurisdiction, personal data is stored on locally hosted servers, and access is restricted by role.

Tech stack

Node.jsData bus core and stream processing

RabbitMQMessage queues and routing between systems

PostgreSQLState storage, transaction log and metadata

DockerContainerization and scaling of bus services

Build a data bus like this — talk to us

Are data syncs between your systems slowing down the business or eating into night windows? Tell us about your architecture, and we’ll propose an integration approach and a realistic plan.

Book a discovery call

Frequently asked questions

How much does it cost to build an enterprise service bus (ESB)?

The cost depends on the number of source systems, data volumes, transfer speed requirements and whether a load-management admin panel is needed. An ESB of this class falls within the standard range for custom enterprise development; we give an exact estimate after a discovery phase that analyzes your current data architecture.

Why build a custom ESB when off-the-shelf solutions exist?

Off-the-shelf ESB products work well for standard scenarios but need lengthy customization when the data structure is non-standard or load management has specific requirements. A custom solution implements exactly the flow-distribution logic you need and integrates with existing databases without unnecessary abstraction layers.

How does the ESB manage load on server hardware?

The admin panel lets an operator see active transfer threads in real time, set task priorities and cap load during peak periods. This replaced manual management through night maintenance windows: migrations can now run at any time with a controlled impact on daytime operations.

How is user data protected when it passes through the ESB?

Data is transferred over secure channels with transactional processing and logging of every operation. Access to the admin panel is role-based. Personal data localization and protection requirements of the client’s jurisdiction are built into the architecture.

How long does it take to implement an ESB for a large online store?

For an enterprise data bus with multithreaded sync of heterogeneous databases and an admin panel, a typical implementation window is around 12–24 weeks; this is an industry range. The timeline depends on the number of data sources, schema complexity and SLA requirements. We fix exact timelines after analyzing the architecture.

Related cases