The challenge
A marketplace integrates constantly — sellers, catalogue, pricing, stock, orders, fulfilment — and the connections tend to accumulate as point-to-point jobs, each with its own schedule, error handling and owner. Stock and price reached the search index on a batch cadence, so the storefront could show items as available at prices that had already changed. Meanwhile the platform itself needed structural work: a clustered order-management topology, and a production index estate that had accreted over years of releases with no record of which indexes were deliberate. None of it could interrupt trading.
Approach
- Established a single orchestration layer for integration flows, with an event backbone beneath it, replacing scattered point-to-point jobs.
- Introduced schema governance so producers and consumers could evolve without breaking each other.
- Moved stock and price onto near real-time synchronisation into search, rather than a batch cadence.
- Classified every production index by origin before proposing changes, so the discussion concerned a known estate rather than a guess.
- Broke the clustering programme into independently shippable sub-activities with their own environments and acceptance criteria.
- Wrote release notes at a level the systems integrator's build team could execute without the architect present.
Innovation
- Treated integration as a platform with its own operability requirements, rather than as a set of jobs owned by whoever wrote them.
- Made configuration capture the deliverable — the durable outcome was that the estate became reviewable, not merely faster.
- Designed each change to be shippable on its own, so a slip in one workstream did not hold the others.
Outcome
- Integration orchestration layer established as the standard route for new flows.
- Stock and price synchronised into search in near real time, replacing batch propagation.
- Order-management tier moved to a clustered topology without a trading interruption.
- Production index estate classified, captured in configuration and brought under review.
- Subsequent releases taken up by the client's integrator team using the documented procedures.
Recommendations
- Point-to-point integration is cheap to add and expensive to own. The cost shows up as operational load, not as project cost.
- Batch-synchronised stock and price is a customer-experience problem before it is a technical one — customers meet the staleness, not you.
- Classify before you optimise. An index estate nobody can explain is a change-management problem before it is a performance one.
- Write release notes for the team that will run them, not for the person who wrote the change.
- On a platform that cannot go offline, shippable increments beat a well-planned big-bang cutover every time.
Architecture
In their words
Great effort holding fort during crucial stages of the project go-live.
Delivery leadership, platform vendor · 2016-09
Technology
- Apache NiFi
- Apache Kafka
- Schema Registry
- Apache SOLR / SOLR Cloud
- SAP Commerce (hybris)
- Order Management