As data geeks, we get some good-n-geeky data questions. A common one is, “What criteria should I use to decide which data to store and which to discard?”
Which is great — I love questions with straightforward answers. Answer: If you think there’s even a slight possibility it could be valuable at some point, store it. For most people that means STORE ALL OF YOUR DATA.
But isn’t it a waste of money to save all of that data if you’re not sure whether you’ll use it? No — it’s probably not.
Innovation and aggressive competition are driving a rapid decline in storage costs, and new technologies are making data analysis very accessible. As a result, the potential opportunity costs far outweigh what you’d spend on cloud storage.
For example, let’s take a look at a particularly popular dataset: the Twitter Firehose. [For those who aren’t familiar, the Twitter Firehose is a stream of all tweets on Twitter.]