Why integrations go wrong; an incomplete list
When I talk with customers about integration, often the focus is on the technology side of things. Can OutSystems integrate with X technology?
To which the answer is nearly always “Yes, and it’s actually quite easy.”
And yet, integration is almost always one of the main reasons projects get delayed. So how can that be?
Well, the hard part about integration is not building the integration, it is the organizational aspects around it. So, allow me to introduce you to an incomplete list of why integrations go wrong, and how to avoid it.
- We are ready, but they aren’t
The Laufenburg bridge crosses borders as well as water: One side is in Switzerland and one side in Germany. It was built by both countries from their own side and connected in the middle. They agreed upon measurements and started measuring from sea level. Only, Germans measurement for sea-level is 27 cm higher than the Swiss side. They compensated for this in the calculations… in the wrong direction. In the end, the German side was lowered to complete the bridge, leading to inevitable delays.
In many ways, your integration is a bridge to another country. First of all, you need to make sure that you both agree on where you meet, and what height sea-level has. Additionally, while integration can be done quickly on OutSystems side, the other side can often have difficulties meeting that same speed. So it is important to account in your project planning that sometimes you need to wait a couple of months for the other side of the integration to be ready. (Even if it can be executed fast in the technology, there might be planning or organizational barriers that cause delays.)
If that is the case, make sure you request changes to the other team with sufficient upfront time, and make sure to agree on a good specification.
Product Increment sessions can be invaluable organizational tools to catch such situations early, and align planning upfront.
2. The keys to the castle
Integrations allow other technologies to access and write information into your databases, and vice versa. And that is a privilege you don’t want to just give to anyone. Solution: add authentication. Next!
Not quite. Authentication comes in many forms and shapes, and they all have their up- and downsides. So let’s look at some of them.
User-based authentication.
When an organization utilizes a centralized authentication method for users, utilizing the logged-in users credentials to get/give access on an integrated service is often desirable.
After all, we can now limit access of some users, while giving more access to others. And we can see who did what, no matter whether they log in directly or use an integration to access/update information.
But, there is no silver bullet, so here are some downsides:
In some cases- for example when running asynchronously on the server in BPT- you won’t have an user to use, so how to authenticate in that situation?
Additionally, some technologies don’t support user-based authentication on their API’s.
I’ve also found quite some organizations who are unwilling to use the user-based authentication method with technologies that charge per (service) user, in an effort to decrease costs.
System users, service tokens and the lot
When we don’t authenticate based on users, we tend to authenticate based on static values. Whether they expire, come in multiple parts or require handshakes, in the end a lot of technologies come down to one or two key pieces of information that unlock all doors. There are so many variations of this that it is hard to capture them all explicitly, so let’s just focus on the storage implication that comes from all of them.
Let’s assume that anyone who is able to access your stored token can access the API it protects freely. A good system is built more robustly than that, of course, but for simplicity sake, lets assume that when one manages to steal this piece of information your whole API is compromised. Lets build our storage security requirements around that assumption.
The first place to store the secret/token/password is in a security vault, a specialized piece of technology for that explicit purpose. This should be a tool outside of your various technology stacks, so that all the different technologies that utilize this key can grab it from the same place. You use a security vault because you would like to store this information in as little places as possible, and the place that you do store it in needs to be buffed up to protect the information. Depending on where you deploy you might have access to Azure Key Vault, AWS KMS, or AWS S3 Glacier for this use case, or a Key management/HSM tool on-premise.
Not everyone has such a technology lying underneath their bed, however. So if you are unable to utilize such a security vault, we can enable the storage of such credentials on OutSystems too. In those cases we recommend encrypting the items and storing them either in database or in site properties. Encryption can best be done by CryptoAPI a tool that allows you to encrypt and decrypt the data based on the private platform encryption key. This key can be found on the server in a .txt file, but is otherwise not accessible by humans.
Integration Builder utilizes this technique in a nifty way, storing the credentials encrypted in a database table that is not accessible from Service Studio. This way Integration Builder also allows credential holders to enter the credentials of an API though the wizard without ever having to give the secret information to the development team. ( Integration Builder is pretty neat :-) )
TLDR: Locking your doors is a good idea. Having keys that register who is using them is even better. If your key can be used by anyone, don’t leave them lying around.
3. Different cadences
Let’s say we’ve established our connections, stored our credentials and our systems live in harmony. But everything changes when the fire nation attacks!
That is to say: our modern IT chains tend to evolve, and so both us and the target system will consistently need to adapt our architecture. Theoretically speaking this should lead to no problems as integrations work with standardized interfaces, written down in contracts that should never change. And if they do change such a change will be subject to heavy coordination.
In practice contracts are not always as set in stone as they should be and changes are not always announced. Not to mention the times that we are swapping out an entire tool in the toolchain, meaning that all contracts are null and void. Change is what makes the world go round. Here is how we can prepare for it:
3.1 Plan that Increment!
Bring out the red string, cause this is what the Product Increment planning session was literally invented for.
A Product Increment planning session is a meeting held at, for example, a quarterly basis. Everyone whose work might intersect is there, and we figure out how our future features and other changes might impact integrated technologies. Traditionally we indicate dependencies between work items across teams with red string, pinned on the features involved. This is how we identify the impacts and the way we need to organize work to mitigate them. The best feature of Product Increments is that they help address situations where teams work in different cadences from each other or the business around them, as long as they all roll-up in the same PI cadence. Sometimes a well-organized co-located meeting with all stakeholders can address things that no planning board tool can.
3.2 Trust, but add an Integration Service
A PI planning session can make you aware of upcoming changes in integrated systems that influence you. It does nothing, however, to make the implementation of that change easy for you. That is what we have architecture for!
To be more specific, the Integration Service pattern helps to reduce the impact of an external change on your landscape.
It works like this: You integrate with an external party who promises that they are not going to change the contract. Double promise, cross-my-hear-hope-to-die. You trust them on this, but to be sure you decouple this outside integration from your internal consumption, by adding a little layer on top. This way if the outside party changes the contract, you simply fix it in the layer you have added on top of it. That way you only have to implement the change once.
Parameters name changed? Map it in the Integration Service.
Data type changed? Map it in the Integration Service. New validations implemented in the Integration? Pre-validate them in the Integration Service. Need to reconnect to a different system entirely? Build the new integration and map things around in the Integration Service.
With an Integration Service no one else needs to know about the changes in the actual integration.
(Btw: this is exactly how Integration Builder does it. As I said, Integration Builder is pretty neat!)
This pattern limits the impact of external landscape changes on your platform. It gives you a measure of control in a chaotic world. In some ways the Integration Service is the IT equivalent of mindfulness; it allows you to interact with the outside world, while simultaneously diminishing the impact of changes outside of your control.
We should all strive towards implementing an Integration Service on our lives.
Namaste!
