The main driver for implementing Big Data solutions is tangible business benefits. Web pages, transactions, clicks, likes and tweets – business value is squeezed out of literally every bit of information available. Now many Internet companies have switched to a monetization model in which the user pays for services with their personal data. They turn into personalized ads, ads drive sales, and the company makes a profit.
In addition to this model, there are other scenarios for extracting value from big data. For example, banks compete for customers, competing as algorithms to detect fraudulent transactions. Another way to use the accumulated data for the benefit of the business is to build descriptive analytics that help companies understand their customers, categorize them and adjust their development strategy.
The area in which Big Data technologies are just beginning to reveal their potential can be called the Internet of Things. The idea is simple: you want everyday items to adapt to the user, interacting both with ordinary home objects and with other smart devices on the home network.
But in order to learn how to work with an arbitrary home environment from scratch, such a device will need to collect a huge amount of information about the parameters of the house and its inhabitants. The obvious solution is to use the experience of smart devices already living safely in other homes.
A number of barriers make it difficult to use data received from other users without violating their privacy – smart devices are strongly connected to the privacy of their owners. In addition, the issue of security must be carefully considered. Using a smart heater must not cause a fire, and the stove must not allow gas to leak.
This raises another problem – who is legally responsible for accidents related to the use of IoT? Uncertainty regarding these issues hinders the conduct of business in the field of the Internet of things and, accordingly, the implementation of industrial Big Data solutions.
Why Big Data is useful to the end user
First of all, we are talking about personal, when the user is the subject and initiator of the use of Big Data. Often this question is answered with banal phrases like “every time you use Google or Yandex search engines, you work with big data.”
However, the harsh truth is that so far, talking about Big Data applications for mass users (B2C big data, if you like) is more like advertising posters from the era of “atomic romanticism” of the 50s, where housewives are promised atomic vacuum cleaners, and children – atomic toys . It is difficult to imagine a situation in which a mass user will be not just a source of information or a consumer of ready-made applications, but a full-fledged participant in the process.
Where Big Data has best shown its benefits
The first big data technologies began to be used by those industries whose activities are tied to the processing of large flows of information on a daily basis – banks, mobile operators, retail chains. Basically, working with data in these areas is aimed at forming a portrait of the client in order to offer him the most suitable services for him.
Each of them has its own specifics, for example, mobile operators now operate with sufficiently detailed data about subscribers in order to extract serious profit from them.
Moreover, some mobile operators do not just use this data to improve efficiency, but also create separate branches of the business to develop B2B services based on the data they collect.
At the same time, Big Data is very slowly penetrating government structures. It would seem that one should be glad that the state is in no hurry to turn the lives of citizens into a matrix. In fact, often outdated methods of working with data, which are due to both administrative barriers and a lack of expertise in government agencies, prevent the use of BD for the benefit of the population.
Although, in terms of maturity in the field of Big Data, Russian government agencies are extremely heterogeneous: for example, almost all organizations in one way or another related to public finance can outperform many prominent commercial banks in terms of working with data.
Of course, we are still far from fully realizing the potential of technology in improving the efficiency of public administration. In addition to obvious examples such as assistance in solving crimes and a unified registry of documents, there are several tasks that BD could help solve at a qualitatively higher level.
For example, the full potential of big data in medicine has not yet been unlocked. Machine learning algorithms are already being actively used in cancer diagnosis, but this approach is not used in other areas, such as flu treatment and personalized diet advice.
It would be interesting to look at a bunch of big data and augmented reality. City and museum guides, instructions for everything that falls into the lens of your mobile camera, first aid tips – now there is simply not enough imagination to imagine the synergy effect of these two technologies in the future.
What hinders further development
The technical problems associated with the use of Big Data solutions have been completely eliminated in recent years. Tasks for which the current set of technologies is not enough are extremely rare. However, there are several factors slowing down the development of big data.
Often, business processes in a company are not sufficiently debugged to apply new technologies. Different departments of companies create analytical repositories for their needs, the data in these repositories turns out to be inconsistent, and as a result, when it becomes necessary to solve larger-scale analytical tasks, the integration of data from different sources turns out to be very difficult – a transition to a common technology stack and painstaking collaboration of analysts is required. Such situations can be avoided if you initially take a responsible approach to data storage – strive for centralization, the so-called “single version of the truth.”
The psychological barrier to the implementation of Big Data is still the imaginary high cost of such solutions. At the words big data, a picture of a data center with orderly rows of servers of astronomical cost arises in the head. In fact, there are now a large number of platforms that provide virtual computing resources. The largest of them – such as Amazon Web Services and Microsoft Azure – take on almost all the management of client clusters.
In terms of software development for Big Data, the situation has also changed a lot over the past three years. Many open source projects have moved from the testing stage to stable release versions, virtualization and containerization technologies make it possible to deploy applications of any complexity on clusters of any configuration. There are more and more specialists on the market who are ready to work with these technologies.
Artificial Intelligence, Neural Networks and Big Data
Without this bundle, neural networks and deep learning tools lose their meaning. The fact is that for effective work, a significant amount of initial data is required for training the model. In such a situation, one simply cannot do without Big Data tools.
Now artificial intelligence tools, hand in hand with Big Data, are marching around the world at a frightening pace.
We asked the Head of Data Science Teradata Russia to talk about how Big Data technologies are used in the field of analytics Alexandra Smirnova
“Teradata has been using neural network and artificial intelligence technologies long before there was a fuss about them.
Of the latest technological innovations, the Teradata Analytical Platform is a fusion of a high-performance database and advanced analytical tools, including artificial neural network algorithms.
In addition, we have implemented many projects in the field of deep machine learning in various industries, from road object recognition systems for car manufacturers to bank fraud detection using artificial intelligence.
The most striking case is probably from the banking sector – Teradata created a mathematical model that allowed to reduce the number of false positives of the card fraud detection system by at least two times. In this situation, Big Data and artificial intelligence not only and not so much help large corporations make money, but also make the life of an ordinary person better.”
Where you can see big data in everyday life
Most people already use Big Data, but most often indirectly through web applications, ATMs, even subway turnstiles.
In the future, big data may become more “personal” precisely in areas such as the “Internet of Things” – sensors of smart objects will produce enough information, for example, to apply machine learning algorithms to take into account user preferences.
Big data: the new face of humanity
Total control: how supermarkets monitor customers
A photo: manufacturing companies, www.pexels.com