Published on the 03/05/2018 | Written by Jonathan Cotton

The numbers are in: Complexity, lack of skills and lack of cloud visibility are the biggest headaches for those on the front line…

The study, Reducing Complexity in IT Infrastructure Monitoring: A Study of Global Organisations, conducted by big data company Splunk and the Ponemon Institute looks at the challenges companies face in troubleshooting and monitoring the cloud and on-premises environments within their organisations.

“There is a lack of visibility, especially into cloud services,” says the report published last month. “As a result, finding problems is a challenge for organisations. Further, when there is a loss of service or downtime, it takes too much time to identify the location of the problem, determine what needs to be done and fix the problem.”

The Ponemon Institute surveyed 2,497 IT, cloud and application development managers and directors on the complexity and scale of their infrastructure monitoring. The average professional was responsible for monitoring around 59 servers, with many reporting a feeling of overwhelm around meeting expectations, performance and availability.

According to the research, just 24 percent of respondents said the handling of scale and complexity of IT infrastructure has improved in the past year and only 29 percent of respondents said their ability to easily deploy and maintain server monitoring technologies has improved over the same time period.

“Companies are looking to simplify IT infrastructure monitoring,” says the report. “The most influential factors in deciding an organisation’s approach to infrastructure troubleshooting and monitoring are the simplification of IT complexity through consolidation of technologies and systems and the automation of IT maintenance/management processes.”

“When system availability and performance degrades, there is friction between IT and lines of business and loss of revenue. Sixty-one percent of respondents say the consequences of a lack of system availability and poor performance is friction between IT and lines of business.”

According to the businesses surveyed, complexity and lack of visibility are the greatest challenges faced. The top two challenges to troubleshooting, monitoring and cloud migration specifically are “a lack of insights to quickly pinpoint issues and identify the root cause” (50 percent of respondents) and “complexity and diversity of IT systems and technology” (47 percent of respondents).

Furthermore, application complexity and lack of skills and expertise seem to pose the biggest risk. Fifty-five percent of respondents said that the biggest risk to their ability to troubleshoot, monitor and migrate to the cloud is the increase in complexity of applications running on infrastructure.

When migrating to the cloud, ensuring application performance and availability causes the most stress, say those surveyed. Sixty-eight percent of respondents say application performance and availability is one of their biggest concerns when migrating to the cloud. This is followed by cost and the inability to monitor and troubleshoot applications (both 51 percent of respondents).

The use of data to diagnose IT issues and determine root cause is cumbersome and difficult to make real-time decisions.

“The biggest problem experienced by companies is that data ingestion and normalisation is cumbersome and tedious (70 percent of respondents) and data is in different formats and types (63 percent of respondents).”

“The biggest problem experienced by companies is that data ingestion and normalisation is cumbersome and tedious.”

Current monitoring tools also do not provide the capability to ingest real-time data said more than half the respondents, with a similar figure saying uncertainty over what data is relevant for problems that arise is a problem.

Following the loss or service or downtime, the most time is spent on identifying the location of the problem in servers, applications or networks, followed by the need to determine what needs to be done to fix the problem and then fix the problem.

Troublingly, most companies seem unprepared to respond to interruptions to IT systems or loss of service even when the cause is understood. Only 29 percent of respondents say their organisation has documented workflows and automated processes to follow in the event of a system failure (e.g. unplanned downtime).

Managers want the ability to perform infrastructure monitoring and troubleshooting in the cloud or an on-premises environment. The most important way to do this – according to 87 percent of respondents – is automated investigations to find trends and root causes easier and faster based on machine learning. In addition, more than half of those surveyed said that implementing automation and operational analytics would improve their ability to deliver projects within budget.

Click here to read the full (and highly detailed) Reducing Complexity in IT Infrastructure Monitoring: A Study of Global Organisations.