For all of its advances, the IT sector’s first five decades could be characterized as the electronic storing of systems of record.
The move to the electronic era saw paper ledgers, tax returns, bank statements stored and archived safely and legally on machines instead of paper files.
Rows of data were worked on in spreadsheets and stored in SQL relational databases. Since then data has been everywhere. It has been in data warehouses, it has been in data lakes, up mountains where it has been mined and in pools.
It is now so voluminous that it can even be measured in something called a Brontobyte. (Though it is a generally accepted we are in the Zettabyte era.)
And more recently this digital era of data and big data saw enterprises embark on the quest to extract value from the information in order to make accurate forecasts.
The Holy Grail of data value began within a subset of mathematics with a discipline called probability and statistical analysis.
Specialists interrogated the data for patterns in order to conduct fraud detection, measure marketing campaign effectiveness or grade insurance claim assessments.
Probability and statistical analysis (a tough and not a very popular career choice) became Business Intelligence that in turn evolved into Data Science (highly sought after and well paid).
What a data scientist is and does has been described in many ways. It requires deep understanding of probability and statistics, a domain expertise such as finance or health and a high level of expertise in machine learning and the workings of big data frameworks such whether commercial such as SAP Hana or open source like Hadoop and their associated platforms, languages and methods.
Analytics now spans five categories: descriptive, diagnostic, predictive, prescriptive, and cognitive, with each building on the last.
The current effort of gaining value from business data is called advanced analytics.
Advanced analytics means using methods to pull meaning from data that will allow for accurate forecasts and predictions. It can also mean managing the added complexity of analysing combinations of structured and unstructured data and doing so in real time.
So, if a company is collecting a petabyte of data each day, say for example a telco or mobile network operator, then – it is argued – the use of advanced analytics will help that company better serve its customers by knowing that they will want or will do next.
This could help reduce churn and allow companies to up- and cross-sell features and services.
The big data storage challenges
The implications of advanced analytics for the IT professional with responsibility for the storage, security and accessibility of this vast data pool are huge. Simply managing the volumes of data pouring into the organization is proving to be a challenge.
For example, even powering and cooling enough HDD RAID arrays to store an Exabyte of raw data would break the budget of most companies.
Increasingly it will be software-defined storage and flash that will be deployed for big data as advanced analytics promises more insight for direct business benefit.
This will be thanks to these media’s improved speed, density, performance and reliability relative to disk and it could fundamentally change the storage infrastructure strategies of enterprises and organizations.
Click here to continue reading Ambrose McNevin’s article.