Published on the 13/11/2014 | Written by Beverley Head
There has been a surge in interest in data lakes over the last three months according to technology analyst Gartner, as enterprises scramble to make sense of their data…
Data lakes which are expressly designed to act as repositories of all forms of information – structured and unstructured – are being constructed by large enterprises keen to understand customer behaviour. Large enterprises including insurance companies, retailers and banks are leading the charge to build central pools of data that can then be interrogated.
Speaking at FST Media’s Future of Banking and Financial Services conference in Sydney recently Greg Booker, chief information officer of RACQ, said that the organisation had; “Created a data lake in Amazon. It’s now a free for all for people to go in and play “what if”.”
RACQ, like other companies is looking to offer a user-friendly system that business users can harness to look for patterns and trends that can inform decision making.
In the US, online retailers have been working with Pivotal, an EMC spin-off which has developed tools for data lake analysis, to build a data lake that can be trawled to identify customer spending patterns and preferences. In a recent blog detailing the retail deployment, Pivotal data scientist Ellie Dobson said that the data lake was growing multiple terabytes every day and that analysis of that lake was providing “a much clearer picture about what is ringing the cash register and what isn’t…”.
In Australia the banks are also building huge centralised information repositories. NAB View, for example pools 12 million customer records and details 150 million banking relationships. Westpac is building a customer service hub to provide a central repository of customer information, though it balks at describing the hub as a data lake.
Whatever these huge repositories are called, they still have to be analysed and interpreted, and Gartner analyst Andrew White, has warned this might not be as simple as vendors suggest.
While there are a growing number of tools, such as Apache Hadoop that can be used to interpret data lakes, White this warned that there were no silver bullets. “Data lakes became a hot topic about three months ago,” he said, as enterprises started to recognise them as a physical repository for all their data which did not have the same sorts of barriers to entry as a conventional data warehouse.
While White said having a central data location was a generally good thing, users were still underestimating the difficulties of being able to make sense of that data.
