Three things IT can do to get control of the big structured data dilemma

Even thought storage prices are falling at a good clip, the reality is that more and more data is being retained online. Corporate database applications are getting bloated, requiring expensive production class disk to keep them responsive and healthy. Meanwhile new classes of structured data generated from machines such as logs, quality control tests and automated trades, numbering in the billions per day, present a new type of storage dilemma.

Addressing these challenges by continuously buying new higher capacity disk is not only cost prohibitive, but building and maintaining massive storage farms is definitely not environmentally friendly. The reality is that existing disk can be reclaimed and reused to good effect allowing IT to “do more with the same”. This explains the tremendous popularity of products that perform compression or de-duplication of unstructured data. The issue is that these technologies cannot do much for the structured data which is held in databases and being continuously generated in the wake of the big data revolution.

Three things IT can do to get control of the big structured data dilemma:

1.       Find stuff that’s OLDR – Data as it comes into an application typically goes into Online transactional processing (OLTP) databases such as Oracle. When the application is new, 100% of that data has the potential to be modified and updated. Over time, the data becomes historical and will be only occasionally accessed. A new class of Online data retention (OLDR) repositories provide a more efficient way of ingesting and retaining historical data. Relocating historical data to OLDR can free up significant storage and cost less to operate and maintain

2.       Put your big data on a diet – Simple binary compression products that use file and block level de-duplication are limited in how much they can compress structured data. OLDR repositories have unique de-duplication that work at the value and pattern level, achieving compression rates of 40-1 or higher. That’s like getting 97% of your disk back!

3.       Get your head in the Cloud(s) –  Planning for growth by having excess idle disk capacity is a waste. Using Cloud storage capacity as a retention contingency or a ongoing supplemental strategy can free up your local storage for more critical production use cases. As an added bonus if the Cloud provider has an OLDR offering transporting and storing 25GB (after 40-1 compression) vs. the original footprint of 1TB represents compelling economics

Leave a Reply