A Discussion of the Question of Integrations

Blog Post created by Rick.Brown Employee on Jun 24, 2015

I wrote this on 17 December 2014, but never uploaded it here. As DevTest development has become even more agile, and discussions have continued regarding version control, perhaps I should publish it?


We’ve all been there. In a meeting with an existing or a prospective customer, the question will be asked

Does your product integrate with our project planning* / ALM* / source control* / logging* / ESB* / banking system* / SAP* / spread sheet* / word processing* / search tool* / development tool* / continuous delivery mechanism* / single sign on*?

*delete or add as appropriate

The answer is sometimes yes, but more often no.

Why is this, what are the implications, should we provide integrations, is there value in doing so?


As a real example, my partner wanted to upload a photo at the weekend, because the Internet doesn’t have enough cat photos. Her go-to photo sharing application has changed over the years, as many sites have that functionality. She had a choice of Facebook, Instagram, Photobucket, Flickr, Picasa, LiveJournal, iCloud, or even our own media server at home from where we can make photos available on the cloud. She decided that Flickr would be where she wanted to put this photo.


Fast forward 30 minutes and much swearing at poor user experiences, and the photo appeared on Imgur.

The photo

A quick aside here – the hyperlink above might not display correctly in all browsers, and isn’t as in-line as others you might have seen. This is because I am writing this article in MS Word rather than in a browser-based editor – I have been stung too many times by integration failures where pressing the “backspace” button does a “back” in my browser and loses my whole entry, where an integration link between cloud-based storage and cloud-based version control fails and loses my whole entry, where things just don’t work cleanly because providers try, and fail, to provide seamless integrations.


It’s hardly a ground-breaking photo - the soppy cat is cute but I don’t think much of the accompanying human, but it should have been simple to share, so I asked her what had been wrong with Flickr that meant she uploaded to Imgur. She mentioned a painful login process, not knowing where her old Flickr login name and password have gone, why the login page was plastered by Tesco advertisements, why Yahoo! Logos were all over the place, and a complete lack of enjoyment in the experience.


I am aware that Flickr was bought by Yahoo! a few years ago, and that it was a great photo sharing site before then, but has something gone wrong with it? I must admit, I don’t take many photos, and when I do, my phone just syncs any photos to whatever service I have it linked to. I’m not certain, but I presume my Android phones automatically upload to some Google service, and my iPhone automatically uploads to Dropbox (unless I have blown my usage limit … again). So, perhaps some investigation is needed into Flickr, and the replacement of that site as the preferred place for sharing photos.


I found various articles talking about a slow demise; an increasing lack of relevance of Flickr for today’s consumers. This is strange, as Flickr had user-contributed tags, comments, multiple photo sizes, in fact all the facilities I would expect in a photo sharing service, and it had these 10 years ago, when Facebook was a University project, when Twitter was what birds did, when Imgur was simply a misspelling. Investigating further, it seems that the Flickr engineering teams were tasked with integrating Flickr into the Yahoo! ecosystem instead of creating new innovative facilities that would have kept it at the forefront of photo sharing. The descriptions of its decreased usage explained about mobile strategy coming from Yahoo!, about how a large company can mismanage a small one whilst making good business decisions, how the reason for takeovers can sometimes not match the perception of either company in the marketplace, how user requirements can sometimes be neglected in the push for business requirements.


I wondered if I can draw parallels from this in my own working life, as I have been employed by a couple of companies who have been subject to takeover, and I have occasionally been frustrated that our product does not integrate cleanly, or sometimes at all, with complementary products, both sold by our company and third party products.


What is integration? It sounds easy, and it should be, but there are actually two distinct areas that can sometimes get blurred:

  1. Front-end integration, where I perform a specific action to link what I’m doing to something elsewhere. An example of this might be pressing a button in Imgur to send my photo to Facebook, or it might be to drag service performance profiles from APM into a virtual service.
  2. Back-end integration, where a server-side linkage makes a difference to what services are provided to me. An example of this might be the Yahoo! single sign-on for Flickr, or the way your mobile and online banking keep your data in sync.


These integrations all sound great! Why wouldn’t we want all these seamless links? They all make our lives better, increasing usability and adding value to every service. In fact, some of the facilities on which we rely just wouldn’t be possible without these integrations. All of this is true, but there are some caveats and problems that don’t become evident until we investigate further.


Let’s look at what happens with a product that is designed from the ground-up to be integrated. In the “consumer world”, this type of application would include modern photo sharing apps (I’ll come back to Flickr later), payments for online shopping, mobile banking and checking-in for a flight. In the “corporate world”, this would include a continuous delivery tool chain and straight through processing in an investment bank. In every one of these types of applications, there are commonalities in developing small pieces of functionality very quickly. If the functionality is not developed quickly and released frequently, problems appear such as the issues that ING bank had with their mobile app being voted by users as the worst of all banking apps. How did they fix it? They moved to frequent updates, so the app had added facilities every day. This meant that users would be able to provide a different voting score every day, comment on enhancements, add to the requirements of the app while knowing that their requirements would be instantly catered-for, and the ING mobile app has recently been voted by its users as the best of all banking apps. Empowering users like this is a great way to build brand loyalty, as users feel part of the improvement process.


What about applications that aren’t constructed with integration at their heart? For consumers, this might be MS Office. For corporates, this might be SAP. There are various after-market additions that can be made to most consumer applications, and the ability to export files in multiple formats, but for corporates, a long-running integration effort needs to be undertaken to make the data available from the place it’s been created to the place that needs to consume it. This is why SAP implementations can take multiple years, why your Excel spread sheet doesn’t quite display correctly in Libre Office, and why version control for documents has never completely taken over from file system storage.


Returning to Flickr … it was an extremely early implementation of social networking. It had built-in facilities that we take for granted nowadays, but there was nothing for it to integrate with, so when it was purchased by Yahoo!, the integration options were to integrate mainly with other Yahoo! services, hence the Yahoo! single sign-on option for logging in to Flickr, or the fact that a Yahoo! search will display indexed Flickr photos. However, this is not what users now want to do – users want to not have to log in, they want to have everything indexed rather than one provider of photos, they want to blog with embedded photos, they want a rich mobile app experience, they want a Web 2.0 experience, and Flickr hasn’t kept up with the trends of user requirements. Visionaries in Flickr and visionaries in Yahoo! had different ideas, and somehow, this meant that users didn’t see continuous improvement.


What is “continuous improvement”? A trite response is “make things better, and doing it constantly”. The Flickr / Yahoo! story is about making things better, but who were they making things better FOR? Flickr developers were concentrating on making things better for the Yahoo! corporation, which was not necessarily making things better for users. While they were doing this, users saw no improvement of services, and saw impediments to performing the actions they wanted (for example; needing a Yahoo! login, and the mobile app simply being a façade to the web page), and so other online services gained traction.


Let’s bring this to corporate services. We work for a huge software company, where there are some overlapping products and some complementary products. We also have competition who has tools in similar areas, and there are third party tools that could provide interesting value propositions. So, what should we do about them? In CA, we have a large number of developers, both for specific product lines and specifically for integration. Corporate pressures mean that cost justification needs to be made for product enhancements, and the implementation of inter-product integrations can divert development funds away from product lines. This justification is good and necessary for the company, but not necessarily good for users. Users have specific requirements for integration, and we need to make sure we continue to do the right thing for users, assuming it’s also the right thing for the company, whilst making sure we don’t ignore either users OR the company. User requirements can be gathered in a number of ways. For us, the CA Communities are a great resource, where users are able to input their requirements and other users are able to vote on them. If enough users add their vote to a requirement, product management are able to view these and decisions can be made as to whether the user requirements match corporate direction. If so, these can be added to the backlog of future enhancements. A backlog intimates that any development would happen in an agile manner, so the specific piece of functionality to match a validated requirement can be delivered while it’s still relevant to user needs.


What about other integration requirements? I know that users have requested version control integration in LISA since I started working with it 5 years ago (5 year anniversary last week!!), that users have always asked about more complete integrations with ALM tools, project management integration, direct linking from development tools, and more recently, continuous delivery and agile portfolio management. Where it has made the most sense, both with internal tools and providing automation to select third party tools, LISA (now “CA DevTest”) already contains integration points, But there is a vast number of other tools where integration might be useful.


What kinds of integration would be useful? Earlier in this article, I mentioned two types of integration. For our platform, and the use cases that provides the most value for users, the initial integration point is the front-end, or automation of user operations. This means that DevTest needs to be able to be orchestrated by external tools, such as CA Release Automation , or even Jenkins or Ant.


The secondary integration point would be the back-end, or facilities providing data to DevTest that come from remote applications. This could be things like version control, PPM or service catalogue. All of these integrations would be useful, but do they both add value to users and to the company, while being more important than other functionality that we need to keep implementing to keep DevTest innovative, thought-leading and ahead of any competition? This is potentially far more difficult than the initial integration points, as we need to ensure that DevTest does not lose focus, but still keeping everyone happy. I am not sure that we are best positioned to actually do this work!


That is a vast and potentially divisive statement! What on earth could I mean by this? Well, keeping DevTest bounded into what it excels at is the best way of not losing focus on it. Much development has been done in the recent past, which is still ongoing, to provide RESTful interfaces to the major functionality provided by DevTest, and DevTest has supported command-line interactions for a long time. By continuing to enhance these open integration server components, we provide the ability for others to actually do the integration work. If others do that work, we don’t have control. What we do have is the focus of doing what we do to the best of our ability. If we don’t have control, who does? Whoever actually writes an integration layer has the control. Integration layers are backbone systems that have been bubbling under the radar of most people for many years, all the way from point-to-point synchronisation servers to move a single piece of data from one database to another, through to enterprise service buses to allow applications to publish information and others to subscribe to it.


But how would it work in our environment?


We have an integration team. Product lines fund them. Product lines only have a finite amount of funds, and therefore any integration work must divert funds from core functionality, hampering how many innovations can be included. So they need a different funding model. Assuming they add value to cross-product sales, it would logically come from cross-product revenue (eg: ELA renewals). If these integrations aren’t adding value, the technology would presumably be made open source instead of a product.


This might, actually, be a good idea. Open-sourcing some internal CA integration bus development work, providing third parties the ability to create their own back-end integrations with our tools, would mean that we have the potential for first-class integrations of our own tools to many applications from many vendors and open source projects. The profile of CA Technologies as a FOSS-friendly company would increase, we would have the opportunity to become thought leaders in the technology side of the application economy, our corporate marketing could make a big push on this and we would even be able to evolve “Business, Rewritten By Software” to something like “Business Reimagined By Software”.