Big Data’s Hidden Sources: Where to Find Big Data

Big data is all around you – including places where you may not think to look. Is your organization missing out on hidden datasets that can drive analytics and insight? Read on to review four big data sources you might be overlooking.

Some big data sources are obvious. Software log files, databases that house customer records and the like are designed for the specific purpose of collecting and storing data.

As a result, these are the places where organizations first tend to look when they seek data sources for analytics.

Hidden big data sources

Yet making the very most of big data requires thinking beyond the obvious.

Consider also the following big data sources that can offer valuable insight for business operations, marketing and beyond:

1. Email

The average office employee sends 40 business emails per day and receives 121. That’s a lot of data – especially when you count the attachments that are included with many messages.

Mining data from your organization’s email accounts can deliver insight into everything from the productivity level of employees and the health of your business pipeline to the times of day when your customers are most likely to respond to emails (and, probably, other forms of engagement, too).

2. Social media

Between Tweets, Facebook posts, Instagram images, and all the other social media data streams out there, social media platforms offer a wealth of information that you can analyze to learn more about, for example, how people are talking about your business and which topics relevant to your business are trending.

3. Open data

There are many gigabytes of “open” data that is free for the taking. Much of it is provided by government agencies, such as the city of New York and the United States federal government, which publish open datasets that can be used by anyone. You don’t have control over which sorts of data get collected and reported, of course.

But there’s a good chance that you can find information within these datasets that is relevant to your business. As a bonus, many open datasets are relatively well maintained and ready to be analyzed out-of-the-box.

4. Sensor data

Traditionally, the go-to sources for machine data were log files produced by devices like servers and network switches. Increasingly, however, organizations are adding Internet of Things (IoT) devices to their infrastructures. IoT devices also generate machine data, which may or may not be recorded in conventional logs. If you use sensors or other smart devices, don’t overlook the vast data that they produce.

In short, big data sources are everywhere. You just need to look below the surface to find rich data sources that you might otherwise be missing.

Download our eBook to learn more about building successful big data projects on a solid foundation of data integration.