Complex systems such as travel planning platforms usually require integration with partner systems that provide important information to users. People using such a solution expect a quick response and consistent information - which can be compared. During this collaboration, we faced a number of challenges in carrying out integrations with flight providers: Kiwi, Duffel, Amadeus or Tripstack. Our goal was to create a cohesive solution that can query the systems efficiently and provide users with the latest, personalised flight data in real time. How do you manage the complexity of this solution to make it scalable and cost-effective? Our answer.
Each system has its own rules
One of the biggest challenges I faced in building the Getaway platform was integrating multiple flight providers - such as Kiwi, Duffel, Amadeus and Tripstack - in a way that could efficiently deliver the latest and personalised flight data to our users. Each of these providers had its own API, which differed not only in terms of structure and protocols, but also in terms of query limits and compliance rules.
Overview of flight providers
Kiwi, Duffel, Amadeus, Tripstack - each of these platforms operate on different principles, meaning that developing a universal solution that can handle their specific requirements required careful thought and the use of appropriate technologies. The key question I faced was how to create a system that would not only connect these various APIs, but would also be able to operate efficiently, scalably and deliver real-time personalised data to Getaway users without overloading individual service providers.
Prerequisites:
✅ Integrating with 4 different flight providers: Duffel, Kiwi, Amadeus, Tripstack
✅ Each provider has a unique API with different rules, limits, and complexities
Description of the problem
Trying to unify data collection and conversion across all these platforms for seamless flight search results. Major issue: each provider has its own API limitations, rate limits, and requires different levels of personalisation for queries. How to handle this effectively?
We're facing the same issue. Each API behaves differently. Any successful strategies?
Spent hours looking into it - Google, ChatGPT, dev forums. It's a mess. API inconsistencies everywhere.
Yeah, it's a common headache. Here's what we found:
- Duffel - offers a simple and well-documented API. It's great for real-time data, but the rate limits are strict. We had to implement aggressive caching and intelligent throttling to avoid hitting their limits.
- Kiwi - their API is powerful but complex. It requires a lot of fine-tuning in your queries to get relevant results. We found that building a layer of abstraction to standardise the search parameters was crucial.
- Amadeus - super comprehensive but comes with a steep learning curve. High requirements for compliance and authorization. We had to build a dedicated module just to handle their security protocols and maintain consistency in data format.
- Tripstack - great for real-time searching but managing their query limits was a challenge. We used dynamic rate limiting and fallback mechanisms to ensure stability when they hit their limits.
How did you manage these APIs simultaneously? Especially with different rate limits and response formats?
We went with a unified search interface backed by AWS Lambda. Each provider is queried in parallel, and the data is normalised on the fly. AWS Lambda's scalability lets us handle varying loads effectively, and the serverless nature means we're only paying for what we use. Plus, we implemented a common caching layer to reduce the load on each provider's API, which also helps in managing rate limits.
Solution: Building gateway for flight provider queries - the power of AWS Lambda
To address the challenge of integrating with multiple flight providers, we used the power of AWS Lambda to design a common API for the Getaway platform. As a serverless tool, AWS Lambda allowed us to create a solution that queries all suppliers simultaneously according to specific API rules. This allowed us to create one common search interface that works in a unified way, rather than adapting to the specifics of each supplier each time.
The solution was designed as a separate project using the Serverless Framework (https://www.serverless.com/), which greatly streamlined the entire deployment process. Serverless solves the problem of provisioning - it automatically manages resource allocation, configuration and provisioning of the necessary functions in the cloud. In addition, the framework provides an intuitive CLI to easily deploy, configure and monitor our solution. This allowed us to focus on business logic instead of worrying about infrastructure.
The use of AWS Lambda has enabled us to scale the flight search process in response to actual demand. When a user initiates a query on the Getaway platform, AWS Lambda automatically triggers the relevant functions that communicate in parallel with APIs from all providers such as Kiwi, Duffel, Amadeus, and Tripstack. In this way, we can quickly collect and process data from a variety of sources, providing users with the latest and personalised flight information in real time.
Another key element was the creation of a common search interface that masks the complexity of communicating with different supplier systems. This allows users of the Getaway platform to easily search for flights and receive consistent responses, regardless of the differences in the APIs of the different providers.
By using AWS Lambda and the Serverless Framework, we have not only gained scalability, but also cost control. Lambda operates on a 'pay-as-you-go' model, meaning that we only pay for the resources actually used when handling queries. This solution has proved ideal for our platform, providing not only performance but also cost efficiency, especially during periods of fluctuating traffic.
Implementation process: Gateway for flight provider queries
The process of implementing the solution was based on several key steps, with a focus on using AWS Lambda and the Serverless framework. The first step was to write sensitive parts of the code in Python and Node.js. The choice of these languages was based on their ability to handle HTTP requests efficiently, as well as easy integration with AWS Lambda. Python allowed us to quickly process and analyse data from various flight provider APIs, while Node.js was used to manage asynchronous operations, which are essential in the context of simultaneously querying multiple systems.
The prepared code was then ported to AWS Lambda handlers. Handlers are functions that AWS Lambda calls in response to events, such as queries from users. This allowed us to take advantage of Lambda's scalability and flexibility to query APIs from different providers such as Kiwi, Duffel, Amadeus and Tripstack in parallel. Each handler was optimised for the specifics of each provider, taking into account their unique requirements for authorisation, query limits and data format.
The next step was to package the entire solution in Serverless. The framework automated the process of deploying and managing Lambda functions. This allowed us to define and deploy the entire infrastructure with simple configuration files, avoiding manual configuration of AWS resources. This tool also enabled us to monitor the performance of the handlers in real time, which was key to reacting quickly to potential problems and optimising performance.
Finally, using Serverless, we deployed the whole thing on the AWS platform. This gave us full control over the deployment process, including automatic version management of Lambda functions, which allowed us to securely deploy fixes and enhancements. The entire solution was tested in a production environment, using real data, which confirmed its effectiveness and scalability.
Interesting approach. Any gotchas we should be aware of when building that unified interface?
Yeah, a few:
- Error Handling - be ready for different error formats and codes. We created a standard error-handling module to interpret and normalise errors across APIs.
- Data Normalization - You'll need a mapping layer to convert each provider's data into a common format. It adds complexity but pays off in the long run.
- Testing - simulate various scenarios for each provider. They all behave differently under load or when they hit rate limits.
OK, thanks a lot!
Implementation effect: Gateway for flight provider queries
The decision to choose AWS Lambda as the basis for the solution was determined by several key factors. First and foremost, cost and time mattered. AWS Lambda, with its pay-as-you-go model, allows you to pay only for the resources actually used, which is particularly beneficial for functionality that is not constantly active.
This solution fitted perfectly into our budget and schedule, eliminating the need to invest in maintaining a traditional server infrastructure. In addition, our previous experience with AWS cloud technologies gave us confidence that we could quickly implement and optimise this solution.
None of the traditional solutions offered the flexibility and cost efficiency that AWS Lambda did. They would have required constant resource management, which not only increased costs but also complicated the deployment and scaling process. AWS Lambda, combined with the Serverless Framework, enabled us to deploy a scalable and efficient system in a much shorter time.
So we created a scalable service that automatically adapts to the load and is extremely flexible. In addition, using the Serverless Framework in conjunction with AWS CloudFormation gave us full control over the deployment process, allowing us to easily manage and monitor features. The result is a stable, efficient and easy-to-maintain solution that perfectly meets the needs of the Getaway platform and its users.