June 30, 2020

The Sudden Shrinking of Big Data

Written by Tyler Moore

The world is changing quickly and so are our business and data models. The right cloud services coupled with the right data can help you to manage through a "black swan" event and have a successful pandemic era and post pandemic era business.

It’s hard to imagine that we went from normal business discussions in January, to discussing toilet paper shortages in March to discussing social unrest in June. Talk about living in interesting times.

Being in the cloud services business we are seeing a data world turned upside down. For those building business models and performing data science, big data may never have seemed so small. While the volume of data continues to grow, the volume of relevant data has seemingly diminished. 

So what do you do when the world suddenly changes and your models and historical data are no longer helping you to accurately predict the future?  Well, quite simply, you collect more data, create more models, and do it more rapidly. Teams are now busy:

  • Creating new dashboards and algorithms to solve problems that weren’t even being considered in March.
  • Revising forecasts based on just a few months of data.
  • Weighing the benefits of changes to, or expansion of, data access with its impact on security and governance.
  • Quickly identifying and locking down security holes that new data and machine learning relationships can cause.

We aren’t going back

The world isn’t going to one day reset itself to the point where the old data and models are perfectly relevant, so you can expect more rapid changes in the short run. This means that every new day is adding a new round of data that is helping to tune the dashboards and models that will drive business, organization, and government decisions going forward. Could anyone have predicted that our predictions would be so far off?

The crisis has highlighted the benefits of the cloud’s flexible pay-for-what-you-use model over on-premises data centers for many customers, and accelerated migrations and digital transformations. As customers mix and match data sets stored in different public clouds, they are more interested in performing workloads on platforms that can access data across data platforms.

CFOs are looking for ways to cut their budgets, so now the focus is more on using AI, machine learning, and automation to save money. In our business we see companies trying to preserve cash and do more with less.

AI used to be more about growing the business, whether by expanding existing customers’ spend or by finding new business. Now, companies are more intent on building models to hold on to what they’ve got. AI and machine learning are being used to determine which customers are likely to churn so account managers can reach out and be more proactive with them. Speed, agility, and clarity at this time are critically important because businesses are having to make decisions extremely quickly.

Taking pandemic measures

The pandemic is impacting just about any business you can imagine, from the volume of an item that a store should stock to the volume of people that may be safely admitted to a store or public venue. Which is why virtually every public and private entity is trying to acquire as much data as it can.

Many cloud-related service providers are offering customers access to collections of publicly available virus-related data from sources like universities, research centers and the World Health Organization. 

In my role in marketing we have had to readjust our market automation models and digital marketing methodologies as behaviors and data models have changed significantly. Professionals in my discipline are sharing experiences, lessons learned, and even data sets to collectively supplement the smaller amounts of individual data available.

Business as unusual

Just when it started to feel as though industries had tamed big data to help make organizations safer and more efficient, the pandemic rendered many of these seemingly rock solid models useless. Piecing together new models based on fleeting interdependencies while executives demand instant insights is now the norm.

We are all in this together

We’ve all seen the signs calling for us all to pull together in the fight against Covid-19 and that call for unity is happening in the business world where there is a clear fight for survival. Working with cloud service providers that can manage a multi-cloud implementation and provide the control and governance of data across cloud providers is now more important than ever. 

In today’s world, businesses are the rapid arbiters of data access, security and governance and cloud platforms must be flexible and robust enough to accommodate their needs. While relevant data sets might be smaller these days, the need to quickly collaborate in the cloud is bigger than ever.

To learn more about selecting the right cloud service platform click here.

You may also like…