When I am faced with the task of integrating data from different suppliers in Traveltech, I often encounter a major challenge - the variety of data formats. Managing them can not only be complicated, but also raises many questions and concerns. How do I effectively test these integrations to avoid errors and ensure the system runs smoothly? Is mocking or stubbing a better solution? Or are there tools that can simplify the process? In this article, I will share my experiences and the methods I use at Neoncube to tackle these issues, from analysing the data format to implementing new suppliers into our model.

As many suppliers, as many formats.... ​

Integrating with different suppliers in the Traveltech industry involves managing multiple data formats. Each supplier may use a different standard, which complicates the whole process.

The problem with multiple formats is that the data must be properly processed and converted to a uniform format before it enters the system. Differences in data structure, file types and communication protocols can lead to difficulties in interpreting the data and increase the risk of errors.

All formats must be correctly recognised and processed by the system. Even minor differences in data structure can cause errors that are difficult to identify and fix.

The challenges of integrating... ​

The main challenges are the scale and complexity of the data. Each new format introduces an additional level of complexity, requiring the creation of specific tests to validate the processing and integration of the data. In practice, this means that appropriate mocks and stubs need to be developed to mimic the providers' actual data.

One of the most common problems is the lack of data consistency and standardisation. Testing has to take into account different scenarios, such as changing data structures, incomplete information or differences in data types. This requires sophisticated testing tools and careful planning. Lack of standardisation can lead to inconsistencies in test results and difficulties in identifying the source of errors.

Another challenge is the need to constantly monitor and update tests. As suppliers make changes to their data formats, tests need to be kept up-to-date to ensure they are effective. Without this, there is a risk that new data will not be processed correctly, which can lead to errors in the production system. This requires close collaboration between the development teams and the teams responsible for testing and maintaining the system.

Mocking vs Stubbing ​

Mocking and stubbing are two key techniques used in unit testing, but they differ in purpose and use.

Mocking involves the creation of objects that simulate the behaviour of real components in a controlled way, allowing verification that a component interacts with its dependencies as expected. Mockings allow verification of the number of method calls and their parameters. They are mainly used to test interactions and system behaviour.

Mocks tend to be more complex as they can provide detailed call-out information, which is useful in more complex test scenarios.

Stubbing, on the other hand, focuses on providing predefined responses to method calls without verifying interactions. Stubs are simpler and mainly used to isolate the code under test from external dependencies, providing a controlled test environment.

Stubbing is ideal for testing specific scenarios where the outcome of a method is more important than how it is called. Stubs typically do not record the number of calls or parameters, making them less complex and easier to use in simple tests.

A combination of Mocking and Stubbing ​

In many cases, it is useful to combine both techniques for more comprehensive tests. Stubbing can be used to generate predefined responses for specific methods, allowing you to control the test environment and eliminate external dependencies. At the same time, mocking can be used to verify interactions between components, ensuring that the system works as expected at the interaction level.

The combination of mocking and stubbing allows for more flexible and accurate testing, especially in complex systems where both behaviour and interactions between components are crucial to the correct operation of the application.

Attention, bugs! ​

Inappropriate use of existing solutions - in practice, it is common for developers to use copy-paste methods from other integrations without fully understanding the specific requirements of the new data provider, leading to errors and inconsistencies in the system.

Lack of dedicated domain models - a problem that complicates the integration process. Instead of creating appropriate models that reflect the unique needs of different providers, developers often try to adapt existing models, resulting in poor data quality and processing problems.

Lack of proper code monitoring - without proper logging of query parameters and error reporting, it is more difficult to track and diagnose issues that arise during integration. Adequate monitoring tools are key to maintaining system quality and stability.

Lack of collaboration with the support team - collaboration with suppliers is crucial to quickly resolve issues and obtain the necessary technical support. Its absence often results in delays and difficulties in maintaining data quality.

Insufficient focus on testing - both unit and integration testing are often neglected, leading to bugs only being detected at the production stage.

Overview of available tools and solutions ​

Unittest.mock: A comprehensive object mocking library, allowing detailed inspection and verification of method calls.

Factory Boy: A tool for creating complex test objects, with easy integration with popular ORMs.

Model Mommy: A simple tool to quickly generate instances of Django models, ideal for creating test database data.

Faker: a library for generating fake data, versatile and easy to integrate with other testing tools.

1. Python's Mocking Library (unittest.mock) ​

Unittest.mock is Python's built-in library for creating mocks. It is versatile and allows the creation of objects that can simulate the behaviour of real objects in a controlled way. Mocking allows method calls, their parameters and results to be monitored, which is crucial for testing interactions between components.

The main advantages of this library are its integration with the built-in unittest module and its extensive documentation, which makes it easy to use. The disadvantages can be the complex syntax and learning curve for novice users.

2. Factory Boy ​

Factory Boy is a library that facilitates the creation of complex test objects. It integrates with popular ORMs such as Django ORM or SQLAlchemy, making it a flexible tool for generating test database data. Factory Boy allows you to define object factories that can generate realistic model instances with minimal effort. This is particularly useful for unit and integration testing.

The advantage of Factory Boy is its ability to easily create objects with predefined states and the ability to modify factories according to testing needs. A disadvantage may be the need to understand and configure the factories before using the tool.

3. The Mommy Model ​

Model Mommy is a tool for quickly generating instances of Django models for testing purposes. It is easy to use and allows models to be automatically populated with random data, which significantly speeds up the test development process. The advantage of Model Mommy is the ease of integration with Django projects and the minimal configuration needed to get started.

However, compared to Factory Boy, Model Mommy offers fewer options for customising the generated objects, which can be a limitation in more complex test scenarios.

4. Faker ​

Faker is a library for generating fake data that is extremely useful for testing applications. It can generate a variety of data such as names, addresses, phone numbers, dates and even more complex structures such as user profiles. Faker is flexible and can be easily integrated with other test tools such as Factory Boy or Model Mommy, making it a versatile tool for generating test data.

Its main advantage is its simplicity of use and extensive documentation, while the disadvantage may be the need to define more complex data scenarios manually.

How to integrate multiple data formats? ​

In the case of the Getaway project, we developed a process by which we implemented each new supplier.

1. Analyse the data format that each supplier operates in order to improve correct integration.
2. Next, we create the appropriate mocks and stubs to suit the data format.
3. The next step is to develop unit tests using an iterative method, gradually adding integrations.
4. Once we have written the tests and integration, we attach the partner to a data model that is common to all suppliers in the category.
5. The previously mapped integration is completed with an adapter that creates a layer of abstraction over the integration with the partner and allows us to use the domain model.

A key element of the process is monitoring the implementation. Dedicated mechanisms were implemented to monitor code, log query parameters and report relevant errors. Collaboration with the vendors' support team is also essential to quickly resolve issues as they arise and ensure uninterrupted data quality.