Beyond ‘Just integrate it’: Real challenges of software integration

Integrations are the essence of modern software, fundamentally coming down to the exchange of information. 

Unfortunately, they are hard to generalize and are known as notoriously difficult topics to tackle. 

The complexity comes mainly from the overlap between business and technology. Business means organizational challenges, reaching a common understanding, working with people from different cultures and backgrounds, addressing legal/compliance issues, and, at the center of it all – delivering value through integration. Technology is the way to get us there. All these factors impact the final shape of the solution.

In this article, we’ll dive into what makes integrations both inevitable and challenging, explore the hidden costs behind seemingly simple integration requests, and learn how to make better decisions about them.

You need software integrations.

Source: Google Trends. Note: Zapier was founded in 2011.

Think about the universe of SaaS products you’ve come in contact with, apps within apps, and the need to connect everything.

It’s no wonder that the business of gluing things together gets bigger year by year. Companies like Zapier, Make, or Mulesoft are all about making integrations as seamless as possible.

The complexity of everything grows alongside the push to make things easy to use. All these companies are trying to fill their little market niche. They do it so well that often it doesn’t make sense to reinvent the wheel when you can just use existing solutions.

Now that we understand the need for integrations, we can think about how to do them right.

Technical challenges of software integration (checklist)

In the trenches, we know that integrations generate huge overhead in our workflows. 

Let’s first review the technical challenges we have to consider before diving into implementation (and this is even before we start talking about the actual business):

1. What’s the mode of communication – are we pulling data from someone, or is someone pushing it to us? Could it be unidirectional or bidirectional? Who sends the data, and who receives it?
2. Making sure there’s no data loss – what happens if we lose some data over the wire, what might be the impact on the business? How do we ensure that nothing gets lost?
3. Authentication/Authorization – as two parties integrate, we need a way to recognize each other and acknowledge that we indeed have the right to access specific information or perform certain actions.
4. Security – are we sure there’s no one else in between who can hook into our data exchange? How do we find out? What’s the procedure in case of a data breach?
5. Now that we trust each other on a surface level, we have to be prepared for the provider’s unexpected/unwanted behaviors, like rate limiting, flooding the other party with huge amounts of data, or (hopefully) unintentional infinite loops.
6. There’s also the possibility that something goes wrong on our side– are we prepared to retry operations that would fix the situation? Conversely, if someone retries a request to our system, are we prepared for that?
7. Are we sure there are no bottlenecks on our side in terms of bandwidth and storage?
8. What are our requirements for data to be up-to-date? Is being “real-time” important for us?
9. Just in case, we need good insight into whether everything works as expected, hence we employ monitoring and observability tools. For legal reasons, we may also need to be prepared for auditing.
10. Error handling – circuit breakers, exponential backoffs, dead letter queues – there’s a wide range of tools and techniques meant to fix something that’s been promised to work automatically without supervision.
11. Automation and housekeeping – are we prepared for key rotation and additional tasks that support our integration?
12. API versioning and documentation – what happens to our integration when we need to change our internal logic? It shouldn’t impact the existing integration.
13. Testing, sandbox environment, datasets for testing – before we release our precious integration to the real world, we need a way to test it thoroughly and ideally be able to mimic the behavior expected in production
14. Data compliance and regulatory requirements – let’s give a simple example: modern privacy-preserving regulations often have a concept called “right to be forgotten,” which means users can request their PIIs to be completely deleted/obfuscated from our system. What are we going to do if we already have upstream and downstream 3rd party integrations?
15. HA and SLAs – as we aspire to “enterprise” status, we have to be sure the other party delivers reliable service, therefore we enter the territory of legal agreements
16. Related to the previous point, there’s a wide range of challenges around scaling our solution. What if we need to provide a reliable service to many of our enterprise customers?
17. Finally, we can decide about actual implementation details – are we using REST API, event-based systems, CSV files, or webhooks? Are we using some specific protocol like FIX in FinTech or HL7 in Medtech?

Business challenges of software integration

Now you get the idea of how awesome the services that can handle almost all of this for us in the background are. While they surely work, we also need to go back to the business aspect of the process.

On integration complexity

Sometimes from the business perspective, it’s easy to say, “We’ll just integrate with X.” 

When it comes to implementation, we discover that we don’t know something about the other party’s internal logic. To handle that, we need to set up a whole process for exchanging information and creating documentation, which effectively can mean we spend more time agreeing on the subject matter than on implementation (points listed above). It may happen that we’ll need additional customization on top of the current offering – who would pay for that? Then if we decide to go with custom integration – which has a cost in money, time, and effort – are we sure we’ll be satisfied with the final result?

On organizational structure

To answer that question, we need to take into account Conway’s Law:

Organizations which design systems are constrained to produce designs which are copies of the communication structures of these organizations.

As developers, we always keep this at the back of our minds – the software we produce will mimic the structure of our ways of communication. In practice, going against that law results in software teams that block each other’s work and get stuck generating a huge dose of inefficiency, much of which has to do with communication friction.

In summary, our integration not only has immediate costs in terms of resources involved but also influences our organizational structure. If this isn’t accounted for, we can spend great amounts of money, time, and effort to get a result detrimental to our organization.

Unfortunately, from developer experience, we often conclude that the total cost of integration surpasses the cost of developing something from scratch, with the additional disadvantage of adding a dependency on the other party, making moving forward harder.

On people involved

Another “people” factor to consider is who will support our existing integration. 

Tech teams dedicated to integrations are rarely enthusiastic about it – instead of working on the core business, they have to deal with the same issues over and over again. In some cases, this might influence the quality of service (what’s your experience with support departments?). 

On the other side of the integration, we often have people from different cultures and time zones, which multiplies the effects of potential misunderstandings.

Business partnership

You need to consider your position in terms of being a business priority to the other party. What if every integration needs dedicated resources, and the integration pipeline is already booked for the quarter ahead?

When integrate with an organization, it means you become business partners. Can we trust the other party to deliver their service? What’s the procedure if a bigger player buys a company or goes out of business?

Role of integration in your business

When you place the integration at the center of your business, the issues above might hit you 10x as hard. Think about the most fashionable business idea of 2024 – GPT wrapper. Imagine a situation where OpenAI raises the price of their service, and suddenly all these apps go out of business.

Mitigating the risk of failed software integration

Not doing it

It’s trivial advice, but always consider the scenario of not doing the integration at all. Remember the challenges outlined above – maybe there’s a way we could still deliver value without the integration, or maybe there’s a way to work around it or deliver 80% of the functionality at 20% of the cost.

Coming up with your own implementation has the benefit of being able to cover your needs exactly and gives you the ability to adjust to new requirements easily (especially using agile software principles).

Use software designed for integration.

When integrating with a 3rd party, look for software built with integration in mind. Does the service provide documentation? That’s a good first sign. Someone knowledgeable about the technical side should review it. Such a service should ideally provide client libraries and SDKs for different languages; if the code of client libraries is available open source – that’s even better. 

When you sign up for a service – does it provide a dashboard for you to track the integration? Well-established services should. Ideally, in such cases, we should be able to integrate with a service without reaching out to support.

Choose integrations that fit your organization’s structure.

Make sure the integration matches your workflow according to Conway’s Law. Analyze who is dependent on whom – it’s important to minimize your dependencies on others (most likely it’s a scenario where you pay the other party). The best way to think about it is to minimize the surface level of the integration. 

A good example of a part of a system that can be integrated with an external partner is a service for sending out emails and text messages.

If I have an app from which I need to send an occasional email notification to a user, it’s best to integrate with a service dedicated to email delivery. This is because sending emails involves a significant level of complexity, such as ensuring they pass through spam filters.

A bad example would be building an ERP system with broad and deep integration with a CRM. This setup relies on constantly syncing information back and forth, which can lead to inefficiencies and complications. The situation becomes worse if the CRM is managed by a different team within your organization. And it gets even more problematic if you eventually need to contact the CRM provider to request customizations specific to your use case.

A good question to ask yourself to verify is: how easy would the integration be to replace and use another provider?

Beware of integration dependency.

Be wary of putting the integration at the heart of your business – that means you’re absolutely dependent on your provider, and a change in terms can put you out of business. Think of a ChatGPT wrapper startup – what if OpenAI decides to provide the same service as you?

Act according to the process and create documentation.

Before stepping into implementation, have in mind the business processes that the integration is meant to cover. Is everything understood on your side? If yes, documenting the flows would be very helpful along the way for each party involved. Documentation would also serve as a contract for future reference.

Summary. Software Integration

In the modern technology landscape, delivering any digital product or service means there’s a near 100% chance that integrations will happen. That’s because nowadays everything is interconnected, and integrations help us deliver enhanced functionality and increase operational efficiency.

Need to glue things together? We can help you make the right decisions and implement reliable integrations.

Software integrations. FAQ