Five reasons your data is failing you

With access to data growing exponentially, many businesses are still trying to understand what they should be doing with it once they have it and how their customer data can provide valuable insights.

But there are also other factors which can block you getting to this point. So what are the most common issues around big data — how do you avoid them?

In order to understand the issues, you first need to understand the term.

Snijders et al* define it as ‘data, which is beyond the ability of commonly used software tools to capture, manage and process within a reasonable time’.

This description gives you some indication of all the stages big data can fail. If there’s no reason, there’s no need.

The hype around big data doesn’t mean your business has to do it right away. Do you have a user case or a problem to solve, and if so the problem should be something that traditional tools are failing to solve.

Find something relevant to your business, but if you can’t come up with anything, it probably means you don’t need big data.

Data capture

The most obvious way your data pipeline can fail is when you capture it. It won’t help having the infrastructure to store and process big data, if you’re unable to capture the right type in the first place.

It’s better to have less and the right data, than have more and the wrong data. Failing to capture the right, wrong or invalid data is a big issue and more common than you’d think.

Processing pains

Data processing can be difficult and proving the worth of some attributes might require a platform more powerful than the traditional tools you use. Try using a cloud provider’s infrastructure or a commercial provider for a trial where you can feed your data in temporarily to ensure it works in the way you need it, while repeating processes and doing quality control while you’re moving data into it.

It’s a good testbed because once you’re merging different data streams you may realise you’re missing attributes.

Remember to invest in people

The hardware investment you should reduce in the ‘experimentation’ phase is something you’ll face at some point, but an equal or even bigger part of your investment needs to be in people.

I’ve witnessed instances where expensive hardware was in place but no in-house capability of using it. Take your people with you and the long term gains will be much higher and the results rewarding.

Big data’s for life, not just for Christmas campaign

Don’t treat data as a project. A project has a start, milestones and an end and you shouldn’t expect things to work like this.

Your data strategy needs to allow you to adapt to change quickly and incorporate new data streams as needed. If you don’t take care of the long term management, your data ‘project’ might transform into a data tomb.

You’ll have noticed that technical issues didn’t feature prominently and there’s a good reason for that. There aren’t many reasons why big data does not deliver.

If there’s a technical issue it can likely be solved before the data enters the system or by ensuring you have bright people to work up a solution.

Failure is often caused by either a lack of purpose, lack of prior experimentation, the right people or the lack of adapting to change.

Before assuming your data isn’t working, take a step back and think about why.