Market Data Ingestion Engine
Ingests raw data from Markit, ICE, MSCI, and other providers, reads it from S3, processes it through Lambda, stores results in DynamoDB, and loads XML output into Fusion Invest.
Modernizing Trade Booking and Market Data Orchestration
An AWS-native, event-driven platform that replaced a legacy trade booking application and automated market data integration from Markit, ICE, and MSCI, delivering financial data transformation into Fusion Invest and Finastra for brokers and traders.
Project Overview
A financial technology client needed to modernize how its brokers and traders handle complex financial instruments and market data. Its legacy trade booking application slowed down execution, and raw feeds from Markit, ICE, MSCI, and other providers arrived in disparate formats that required heavy transformation before use in Fusion Invest. Ditstek engineered a fully AWS-native, event-driven platform that removes the outdated front-end, automates financial data transformation into XML and FIXML formats, and connects critical financial systems, giving the client scalable market data integration with less manual intervention.
Challenges
Legacy Trade Booking Application
Brokers and traders were forced to book Foreign Exchange (FX), Interest Rate Swaps (IRS), and Cross-Currency Swaps (CCS) trades through an inefficient legacy trade booking application.
Friction in Fast-Paced Trading
The outdated interface created unnecessary friction in fast-paced trading environments, slowing down critical trade execution at the moments when speed and accuracy mattered most to traders.
Disparate Market Data Formats
Raw market data feeds from Markit, ICE, MSCI, and other providers arrived in different formats, making market data integration a significant operational bottleneck for the business.
Complex Data Transformation
Before Fusion Invest could use any incoming feed, the data needed complex financial data transformation, a demanding process that the existing setup could not handle efficiently.
Risk of Delays and Failures
Without the right architecture, the transformation process risked data delays, manual intervention, and downstream reporting failures that could disrupt trading operations and the teams depending on them.
Need for a Scalable, Flexible Architecture
The client needed a highly scalable and flexible architecture that could handle changing data loads and new provider formats without constant code changes or custom maintenance.
Solution
Ditstek replaced the legacy front-end and built a serverless, event-driven AWS platform for financial workflow automation, combining direct trade booking, automated orchestration, configuration-driven transformation, and privileged access security in one scalable architecture.
Trade Booking Moved to Finastra
Brokers and traders now book FX, IRS, and CCS trades directly within Finastra, removing the legacy application as the front-end and consolidating the entire trading ecosystem.
Serverless AWS Lambda Architecture
Stateless AWS Lambda functions replace continuously running servers, so the system scales automatically in response to API calls, storage events, and message queues across the whole data pipeline.
EventBridge Central Event Bus
AWS EventBridge serves as the central event bus, routing events to Lambda, SNS, SQS, and Step Functions, and reshaping data through input transformation before delivery.
Step Functions Orchestration
AWS Step Functions coordinate multiple Lambda functions across each data workflow and react to AWS service events automatically, freeing the platform from fragile custom state management logic.
Configuration-Driven Transformation
Parsing and transformation logic lives outside the core code in YAML, Jinja2, and XSLT configuration files, so new data formats can be supported without rewriting the application.
CyberArk Privileged Access Security
CyberArk Privileged Access Management secures the infrastructure, with strict network and IP restrictions limiting CyberArk administration to approved AWS subnets instead of all VPC traffic.
Features
The platform covers the full trade and market data lifecycle, from file ingestion and email processing to transformation, trade booking, downstream responses, and secure access, all orchestrated through AWS serverless services.
Direct Trade Booking
Replaces the legacy trade booking application, letting brokers and traders import and book FX, IRS, and CCS trades directly into Finastra through automated AWS services.
Automated Markit File Processing
A control job sweeps new Markit files to S3, where a Step Functions state machine identifies each file, converts it to FIXML via Lambda, and hands it to Finastra's SFTP pickup.
Email Watcher & Processor
A kickoff Lambda runs every 10 minutes, passing S3 configuration through EventBridge to Step Functions, which fetch emails via Microsoft Graph API and save attachments and content automatically.
Asynchronous Response Handling
Downstream responses land in S3 and trigger SQS and Step Functions, which run a Response Lambda to update DynamoDB and an Output/Excel Writer Lambda to finalize data.
Secure Web Frontend
Users access the interface through an Application Load Balancer with Cognito/OIDC authentication, upload files for Excel parsing and validation, and track long-running operations through live status tables.
Configuration-Driven Data Processing
Files are processed with Python 3.12, Polars for structured data, openpyxl for Excel, and lxml, XSLT, and Jinja2 for XML handling and template-based financial message generation.
State Machine Orchestration & Monitoring
State machines defined in Amazon States Language can be triggered via the AWS SDK for .NET, the AWS CLI, or S3 uploads, with CloudWatch monitoring executions and Lambda logs.
Privileged Access Management
CyberArk secures AWS-hosted resources with session monitoring, session recording, automated password rotation, and secure credential vaulting, so privileged access to the platform stays tightly controlled and fully traceable.
Tech Implementation
- jQuery
- .NET Core
- SQL Server
- AWS Cloud
- AWS Lambda
- Jackson
- Amazon DynamoDB
- Amazon EventBridge
- Python
- .NET 8
- Jinja
- AWS CloudWatch
- Grafana
- Amazon Chime SDK
Impact Created
Legacy Front-End Retired
Moving trade booking into Finastra retired the legacy application completely, giving brokers and traders one simpler workflow for booking all FX, IRS, and CCS trades.
Automated Market Data Pipeline
Market data integration from Markit, ICE, MSCI, and other providers now runs automatically end to end, reducing the manual intervention that previously slowed data delivery.
Scales Without Server Management
The serverless, event-driven architecture scales automatically with API calls, storage events, and message queues, so the client no longer runs or manages continuously running application servers.
Easier Support for New Formats
Configuration-driven financial data transformation lets the team support new provider formats by updating YAML, Jinja2, or XSLT files instead of changing the core application code.
Reliable, Low-Maintenance Orchestration
Step Functions and EventBridge removed the need for fragile custom state logic, making workflows more reliable, lower-maintenance, and easier to monitor through CloudWatch and Grafana.
Tighter Security Controls
CyberArk PAM with subnet-level restrictions, session recording, and automated password rotation gives the client strong control over privileged access to its financial infrastructure on AWS.