Citizen Integration Tools. The peaks and the pitfalls of DIY integration.
You've probably heard the term citizen developer mentioned with increased frequency over the last few years. So, who are these mysterious citizens? What do they do, how do they do it, and why?
Citizen developers are people within your company who have access to low-code/no-code software which provides them with some very handy drag-and-drop application components. They can connect these components together to generate new web, mobile or business applications.
There's also a range of self-service tools that allow citizen developers to connect applications and integrate data between systems without the help from IT integration specialists. Some examples of these tools are Dell Boomi, Flow Power Automate, MuleSoft, Microsoft Logic Apps, Decisions, Zapier, SnapLogic, AWS Redshift, etc.
So, instead of adopting the traditional approach of involving the business and technical teams to design and deliver application integrations, your business team can now bypass IT altogether. As citizen integration tools don't require you (or your nominated citizens), to have any programming background and experience, or go through a steep learning curve to use them, there's no need for a solution architect or developer.
And when compared with the traditional integration process, these tools allow you to deliver business solutions for less, and more quickly. So why wouldn't you?
Citizen integration tools sound great, don't they? So what's the catch?
Now no offence meant, but while these tools are straightforward and easy to use, building a solution that meets most or all your business requirements, and adheres to best practice, is often beyond the ability of a typical citizen developer.
Let's look at some of the problems you may run into.
Problem: It costs us how much?
Because these self-service tools are so accessible and easy to build a solution with, you may not have considered the cost implications if you get it wrong.
For example, if you create more connectors than you need to do the job, your costs may go up. Many integration platforms use a pricing model based on the number of connectors you have in place.
Or you may end up storing data in inappropriate storage tiers (hot, cool, archive) in Azure Blob Storage. If you store data that is still in active use in the archive tier, instead of the hot or cool tier, you will incur a data rehydration cost.
Solution: Do your homework
Invest more time in researching the available tools (e.g. pricing calculator in Azure). Take note, though; this will not guarantee the most cost-efficient solution long term. You'll need to keep track of the cost through a process of continuous reviews.
It's also important to look at more than one product. And to fully understand how each product you evaluate can enable you to achieve the solution, and at what cost. We guarantee it will vary.
To meet your company KPIs, we also suggest regularly reviewing the usage and bills associated with each service so you can identify where you are paying too much.
Problem: Umm, this solution doesn't follow development best practices!
Oh, dear. You've got a brand new citizen integrator solution that performs poorly and is hard to maintain, scale and reuse.
For example, it's making multiple calls to an external system (e.g. a database) and is causing unnecessary loads/traffic. Or it's creating separate and different solutions for each department of your organisation. Or you've ended up with 'spaghetti code' which is hard to maintain and reuse.
Solution: Apply best practice (sorry, we know that's kind of obvious)
Our recommendation is to maximise solution performance by making a single, bulk entry call for insert operations. To cache your frequently used data to reduce traffic between your applications/systems. And to consolidate common workflows/processes by implementing a single solution that can handle workflows/processes of all or most of the departments of your organisation (loosely coupled). For example, if you created a standard template for the client onboarding, then it will become a matter of rinse and repeat.
Problem: Oh-no, there's no way you are implementing that! It doesn't conform to our compliance and regulations!
No-one wants to be the person who implemented 'that' solution. The one that resulted in your organisation failing an internal security audit, or resulted in a data breach, hack, or denial of service.
Some examples of fail points are incorrectly storing GDPR data in Europe, or not meeting PCI (Payment Card Industry Data Security Standard) for the finance industry. Or hosting your solution offshore when regulations demand you host it onshore.
Solution: Leverage tools such as Azure Blueprint
When developing the solution, review its compliance with the appropriate team (e.g. infrastructure). For example, if you provide details on what Azure storage account your solution complies with, then your infrastructure team can provide expert feedback.
Problem: Sorry, but this solution doesn't meet our security requirements
Not many people have an in-depth understanding of security best practices; it’s a specialised field. Security doesn't just mean IT security, but it also involves network security, user access security, etc.
So, you have a problem if your data is not encrypted as it flows through various components within a solution (data transient security, IP restrictions and Port restrictions). It's not encrypted or hashed while in storage (security in rest). Or your users can freely access resources using basic or no authentication.
Solution: We have a list
- Enforce HTTPS and latest TLS version.
- Encrypt data with the company's domain certificate
- Hash data before storing.
- Implement multi-factor authentication (MFA), SHA256, least privilege user access policy.
- Leverage security check tools such as Burp Passive Scan
Where do citizen integration tools shine?
While it may all seem a bit daunting, and even leave the citizen developer asking, 'why bother?', the self-service integration tools are useful, and do have their place.
Here's a list of things you can do that will add value to your business:
- Create a Proof of Concept (POC) for quick business turnaround and decision making, compare the pros and cons of each tool.
- Make use of the built-in features of the tools where applicable. For example, Dell Boomi has Boomi Suggest to help you out with data mapping exercises and a deployment process that is only a few clicks away.
- Look into the reporting capability of the tool for real-time data. For example, Application Insights in Azure visualises the data that's flowing through the solution, helping you to understand the best and worst-performing sales products.
While citizen integration tools have their place, and can add value (see above), they should be approached with caution and common sense.
Not because they don’t work – far from it.
But because having access to a hammer and spirit level doesn’t necessarily qualify you to build an architecturally and structurally sound house, which is compliant with health and safety regulations.
The cost to remediate errors can far outweigh the investment in using integration experts or working with your IT team to design and build a cost-efficient, best practice, scalable and secure solution.
Author: Lisi Chan - Senior Consultant
Co-Author: Harris Kristanto - Consultant