The Big Data skills gap and other issues ⁰

A new report from techUK claims that the widespread use of Big Data could boost the UK economy by £241 billion and create around 157,000 jobs by 2020, but the Government needs to act fast to address the skills gap that is holding back the implementation of more Big Data applications. The survey of techUK members found that almost two thirds (62 percent) say they need more specialists in emerging disciplines including Data Analysts, Data Infrastructure Engineers and Solutions Architects. However there may be wider challenges ahead for the application of Big Data as a survey carried out by Gartner suggests that the number of organisations willing to invest in Big Data is set to fall.

The techUK report gives a few suggestions on how to solve the skills gap including:

• Data Analysts, Data Infrastructure Engineers and Solution Architects should feature prominently on any future Government preferred shortage occupation list, joining Data Scientists
• The UK’s existing tech workforce should be upskilled
• The Department for Education must work to ensure there is more uptake of higher-level apprenticeships from a younger age with the forthcoming Apprenticeship Levy
• Programming and Analytics should be introduced earlier in the school curriculum
• Industry and government must do more to promote the value and importance of Big Data and Data Analytics

However, the survey by Gartner based on a survey of 199 Garter Research Circle members found that while Big Data investments continue to rise, there are signs of a contraction in investment intentions in the immediate future as firms start to assess how best to make use of Big Data. The survey found that 48 percent of respondents has invested in Big Data in 2016, up 3 percent from 2015. However, those who plan to invest within the next two years fell from 31 to 25 percent in 2016.

“Investment in big data is up, but the survey is showing signs of slowing growth with fewer companies having a future intent to invest,” said Nick Heudecker, research director at Gartner. “The big issue is not so much Big Data itself, but rather how it is used. While organisations have understood that this is not just about a specific technology, they need to avoid thinking about Big Data as a separate effort.”

While nearly three quarters of respondents said that their organisation has invested or is planning to invest, many remain stuck at the pilot stage. Only 15 percent of businesses reported deploying their project to production, effectively unchanged from last year (14 percent).

“One explanation for this is that projects appear to be receiving less spending priority than competing IT initiatives,” said Mr. Heudecker. Only 11 percent of respondents from organisations that have already invested in Big Data reported that their investments were as important, or more important, than other IT initiatives, while 46 percent stated that they were less important. This could be due to the fact that many projects don’t have a tangible return on investment (ROI) that can be determined upfront,” added Mr. Heudecker. “Another reason could be that the initiative is a part of a larger funded initiative. This will become more common as the term “Big Data” fades away, and dealing with larger datasets and multiple data types continues to be the norm.”

A further factor to consider is the lack of effective business leadership or involvement in data initiatives, claims the report. Too often, pilots and experiments are built with ad-hoc technologies and infrastructure that are not created with production-level reliability in mind. “When it comes to Big Data, many organisations are still finding themselves at the crafting stage,” said Jim Hare, research director at Gartner. “Industrialisation — and the performance and stability guarantees that come with it — have yet to penetrate Big Data thinking.”