|
|||||||||
|
Thank you to Mike Vizard of CTOEdge for accepting my guest blog post Surfing the Big Data Retention Wave “Companies in many other industries face Big Data Retention imperatives, so when push comes to shove, you can analyze for show, but you need to comply for dough.” Please read the entire post at: http://www.ctoedge.com/content/surfing-big-data-retention-wave OLTP or Online Transaction Processing databases and OLAP or Online Analytics Processing tools and data warehouses are widely used in data management applications and IT infrastructures. But as George Crump of Storage Switzerland points out in his Information Week blog post “Keeping Data Forever vs. Data Retention” the decision of what data to keep and when to purge it has always been a dilemma which has yet to be resolved. As one person commented, the IT department should not be determining what data should be kept or removed. It is fundamentally a business decision based on compliance, regulatory requirements and usefulness of the data relative to the business. With the Big Data explosion occurring, disk space continues to get gobbled up faster than Pacman going after power pills. With compliance drivers such as the online availability of Electronic Health Care records for everyone here in the USA, you can be sure that IT departments and ISV providers serving this industry sector will be looking at ways to intelligently purge the data that is no longer required to be retained. The question remains how can this be done based on configurable rules and how can you ensure compliance without significant changes to applications or custom code/scripts to execute these tasks. If you could ensure that older, historical data can be retained and kept accessible online in limitless volumes, with purge capabilities only in order to reduce liabilities where needed. And as an added bonus, by removing the static data from your production applications or data warehouses, those systems get “younger” in that their indices get smaller and they run more efficiently without the need to add new hardware. In short, they return to their youthful original self before they got clogged up through the passage of time. If you are a data management professional or a product manager in charge of applications generating lots of data in your OLTP and OLAP systems, you owe it to yourself (and your customers) to “get OLDR”.
Scroll down to vote for who you think will be the next to be acquired? The hot news in the “Big Data” world this afternoon is the acquisition of Greenplum by EMC. Greenplum has been one of the major high performance analytics players focusing on “Big Data” (see my analysis of Greenplum’s Enterprise Data Cloud strategy last year). A few of the others being Aster Data, Cloudera (and Hadoop), ParAccel and Vertica. The space also has established giants in the form of integrated Data Warehousing Appliances, such as Teradata, Netezza and Oracle with (Sun) Exadata. Clearly this was a bit of a surprise to many, including Curt Monash of DBMS2 who tweeted (you can read a more serious Curt review of the acquisition here): Holy shit. EMC is acquiring Greenplum. http://www.emc.com/about/news/press/2010/20100706-01.htm Neither side told me in advance, of course. Apart from the generic press release typically issued for such events, Chuck Hollis VP — Global Marketing CTO at EMC Corporation sheds more light in his blog around what the acquisition means for EMC, and for partners such as ParAccel (ominously mispelled ParAccell in Chuck’s blog) who clearly must not be pleased about this turn of events. More details around the ParAccel “Scalable Analytic Appliance” can be found here on the EMC Solution Gallery. Merv Adrian of IT Strategies noted in a tweet: #ParAccel Scalable Analytic Appliance was partnership w #EMC. New version was expected. #Greenplum Deal a blow: EMC promises an appliance. But the most interesting paragraph in Chuck’s Blog post listed “Many Targets For Value-Added Integration” If you think about it, we’ve got a very long list of candidate technologies that might be interesting to consider integrating in the future:
While many companies might be scrambling around reading into those bullets and sizing up their opportunities to also be acquired by EMC, conspicuous by it’s absence is any mention of Master Data Management (MDM). As many of us who are and have been in the MDM world know (Disclosure: I used to be the VP of Product Marketing for Siperian – acquired by Informatica), MDM is a critical component of an enterprise data management strategy to ensure that the dimensions utilized by high performance analytics and data warehouses are accurate. MDM, which started out as Customer Data Integration (CDI), has seen many of its initial successes and popularity from demonstrating that it is critical to ensure that reference data fed into expensive high-end analytics are true and accurate. Otherwise you could be analyzing garbage. Of course Informatica and IBM realized this via their respective MDM acquistions last year (in the case of IBM, they acquired DWL as far back as 2005). So it’s rather strange that EMC, who own Enterprise Content Management (ECM) giant Documentum, which could be considered MDM for unstructured data, has to date not made any moves in this particular area. Even more surprising given that EMC Consulting (as part of the BusinessEdge acquisition) have been very active as System Integration experts in the area of MDM. All this is food for thought and fun to speculate, especially since we haven’t had an MDM acquisition for a while since Tibco bought Netrics. Meanwhile, I’m also eagerly awaiting Rob Karel of Forrester’s take on his blog. P.S. EMC acquires Greenplum #DW http://bit.ly/cSvP6p #ETL #MDM #BI are sure to follow. HP is once again sitting on its laurels. P.P.S A list of the major bloggers and their thoughts around the acquisition: 1) EMC’s CTO, Chuck Hollis – http://chucksblog.emc.com/chucks_blog/2010/07/emc-to-acquire-greenplum.html 2) Netezza’s response to EMC/Greenplum acquisition – from Phil Francisco (VP Prod Mgmnt, Netezza) http://www.enzeecommunity.com/blogs/nzblog/2010/07/07/emc-swallows-a-green-plum 3) Merv Adrian - http://mervadrian.wordpress.com/2010/07/06/emc-buys-greenplum-big-data-realignment-continues/ 4) Curt Monash – http://www.dbms2.com/2010/07/06/emc-is-buying-greenplum/ 5) Rob Karel – http://blogs.forrester.com/rob_karel/10-07-07-emc_moving_tackle_data_management_market 6) James Kobius – http://blogs.forrester.com/james_kobielus/10-07-06-emc_acquiring_greenplum_paving_way_emergence_virtualized_cloud_data_warehousing Cloud ‘N Clear Poll – You Make The Call! Your Votes: Your Votes: For all the advances the human race has delivered over countless centuries, one could argue that this is still a very complex world indeed. True, things can now be accomplished a lot faster, cheaper and more efficiently than ever before, but exactly how far have we come that it still takes an average person till the age of 21 before they amass the skills needed to seek their fortune in the business world. Tack on additional MBA time or Doctor/Lawyer level focused education, and on the job training and all of a sudden you are pushing late 20s before you feel that you’ve found your niche and “career”. I went through the traditional path of school then University (computer science degree) then pursued a programming career (specializing in Oracle and Informix database administration), followed by product management and overall technology marketing to end up where I am today. Back in the early 90s when I was knee deep in shell scripts, Oracle SQL*Plus command line entries, I was gratified to know that the skills I was learning through courses, on the job training and volumes of articles from experts and books would enrich me with significant compensation had I decided to pursue a career as an Oracle DBA. Fast forward 20 years later, being an Oracle DBA is still a great job, highly compensated and a critical role within an IT organization. Much of this is no doubt due to the ubiquity and success of Oracle. However, you would think that in 20 years, an Oracle database would be smart enough by now to forgo the need for specialized human babysitting. Not so, since in an effort to become more ubiquitous, the powerful Oracle RDBMS has added more and more complex features and capabilities which make it a very sensitive, powerful database that requires dedicated care, feeding and handling in order to make the most of your significant investment. This comes as no surprise as it mirrors real life. If you shell out big bucks for a Ferrari the corresponding maintenance costs are higher due to the specialized mechanics compared with that of an average car. If you buy a big expensive house, your repairs, upkeep and property taxes are also going to be higher. Oracle, together with expensive hardware, fiber connected SANs and expert DBAs adds up to quite a hefty sum of the total cost of ownership (TCO) of managing and retaining the valuable data captured by your applications. If you were going to spend big bucks, you’d expect results and performance to match your use cases and requirements. With Big Data upon us and years of data warehousing experience, we already know that transactional RDBMS’ are not the best repositories for analytics. Furthermore Teradata, Netezza and a cadre of new providers such as Cloudera (with Hadoop), Aster Data, Greenplum, Paraccel and others are further specializing in big data analytics & insights. Similarly, Oracle and other RDBMS’ with all their glorious complexity should be the least likely repository to keep massive amounts of data if long term retention is your focus. Would you use your Ferrari to store your furniture? Features such as two phase commit, referential integrityand even update would be considered over kill for immutable retention. Throw in an Oracle expert or two and you get the picture why ISVs are looking for specialized repositories to tackle their Big Data retention problems. They expect their repositories to not only leverage low cost commodity hardware, be enabled for the cloud, but require close to zero administration. That doesn’t mean there isn’t room for that Ferrari. Oracle is still the king for running production critical applications and the Ferrari mechanics should be 100% focused on managing and maintaining those systems. For Big Data retention however, the complexity and TCO of the target repository should be so low that it is a negligible part of the overall IT budget. All this can be achieved by selecting the right specialized repository for the right job.
In IT, high performance production operational or analytic applications end up in a similar dilemma as they experience “weight increases” through data volume growth over time. To make matters worse, research shows that as much as 90+% of the data is static and historical in nature. For example, some medical applications servicing hospitals can contain as much as 96% static data, made up of prior patients who may never return to the hospital. Just like a person, these applications end up suffering from sluggishness in the form of poor performance. They face a constant battle to manage data within high performance RDBMS’ and data warehouses by spending billions of dollars a year on tuning, extra storage and expensive hardware to tackle this ongoing problem. While Big Data has been generally used to describe Google-scale analytic problems, the reality is that for most companies their own internal Big Data problem is one of the growing expense to manage, retain and keep data accessible, while dealing with the hit to their production applications. Fortunately, the way to speed up an application, without throwing piles of cash for bigger faster larger boxes, is just like it is in real life, you need to put your production databases on a diet. A BIG DATA diet! However, just like people diet fads are expensive and short-lived, many of the solutions today don’t deal with the root cause of the problem, nor the recognition that Big Data just keeps on growing. What is needed is a conclusive way to unclog the arteries of production databases of the over 90+% of static, historical data that is only occasionally accessed. This would leave the RDBMS to expertly handle actively modified data, thereby returning the application to its former glorious slimmed down self. Of course, as we all know the way to avoid having to diet is to eat right in the first place. In the case of Big Data that might mean never putting any data that is machine generated data (e.g. logs, Call Data Records, text messages) and therefore immediately historical, requiring no modification or updates into a RDBMS as it is simply not the right repository for the job. On the subject of discipline, most of us need help keeping the weight off and generally respond better when there are good reasons and incentives to do so. The explosion of Big Data has been accompanied by government regulatory retention requirements in many industries, enforced through stiff fines and penalties as discussed in The Other Side Of The Big Data Problem. For ISVs this can be an opportunity since retention periods for large amounts of data, together with online accessibility requirements are forcing companies to re-examine their architectures, processes and ultimately the databases they are using to store the data generated by their applications. ISVs can provide solutions to meet these needs resulting in new revenue streams. As it happens, my current company RainStor has a new release, version 4 that supports ISVs who are looking to provide diets to their production systems. With a less filling but with the same great taste SQL and BI tool access to the data, it also provides capabilities that manage and automate the discipline needed to stay compliant at a very granular level. Now if you’ll excuse me, all this diet talk is making me hungry and I need to get me some artery clogging cookies. Claudia Chandra, Product Manager of the Data Archive suite at Informatica wrote a nice piece today titled A Secure, Cost Effective Home for Your Retired Application Data. (Disclosure: Informatica is a major partner and investor in my current company RainStor) Here are some exerpts of her post (in italics) and how the specialized repository that Informatica uses gives them a competitive advantage: Informatica Data Archive is different from other archiving solutions because it provides you with exactly that option – archiving to a highly optimized, immutable, secure, and compressed file format. It also yields up to 98% compression, thereby reducing your storage capacity footprint dramatically. At the same time, you can still access and report against your archived data from any reporting or Business Intelligence tool, as though it is still stored in traditional relational databases. The optimized file-based archive format accomplishes all this without the burden of specialized administration or expensive high performance hardware. 3 fundamental tenants differentiate Informatica’s offering from traditional relational databases: Reduce. Retain. Retrieve. Reduce – A unique combination of value, pattern and algorithmic software techniques examine each data value as it is loaded into the repository. Unlike a traditional RDBMS, the values of the records are stored as unique nodes in a tree structure. Duplicate values are not stored again thereby saving at both a value and pattern level. While the data de-duplication process significantly compresses the size of the data being stored, there is no loss in the integrity of the original form of the records as the repository retains the original structure through a system of pointers. When a query is executed, the information is returned in its original form, just like a traditional database. Retain – Apart from being immutable, which means no updates are allowed, the repository also provides configurable retention rules and metadata annotation options not available in a typical database. With fine grained control down to the record level, the data can be secured, annotated, policies set for expiry, while ensuring secure access to only those individuals who are authorized. Retrieve – Traditional techniques to reduce the size of data usually takes the form of brute force compression, e.g. WinRAR, ZLib, Gzip and so forth. As such, the data requires a re-inflation step prior to access which results in delay which means slower performance. the repository that Informatica uses has data reduction techniques which provides exceptional performance with no re-inflation needed. The data structures are stored and queried in exactly the same form as illustrated in the diagram. The intelligence of the optimizer allows any SQL dialect (Oracle, MySQL, SQL Server etc.) to deliver the results from with excellent response times. Furthermore, the repository is not disk-bound and very CPU-friendly. If you need your queries to run faster, adding CPUs or cores can proportionately speed up the return of results. Finally once the data is safely loaded, the physical representation of the data is optimized for low-cost commodity or specialized disk and systems such as EMC Centera. A great option and very timely given Informatica’s newly announced partnership with EMC. The architecture also makes it ideal for cloud deployment as illustrated with Informatica Data Archive Cloud Store Option All this yields great TCO as indicated by Informatica’s case study customer Adelphia: Informatica customer, Adelphia was able to shut down their legacy financial, treasury, and franchise legal application by using Informatica Data Archive to retire the data. They then use standard ODBC/JDBC interfaces to query the Informatica file-based archive and deliver the necessary application data to their business users on demand. They were able to retain all of the data and maintain access through inexpensive server and storage infrastructure, with 5% of the original database size and 5% of the staff time required to maintain the original application. The resulting ROI of 211% speaks for itself. Looks like Informatica has really cracked the code on retiring applications, while providing visitation opportunities. Certainly enough to satisfy business users and compliance requirements while saving their customers a bundle. Today EMC Corporation announced an expansion of its EMC Atmos cloud partner ecosystem to help customers manage and optimize external clouds as part of an overall private cloud strategy. RainStor (my current company) participated in the early evaluation and feedback of the Atmos platform. Our close partnership with EMC has ensured that our ISV partners, who embed RainStor as part of their overall solution, immediately gain the ability to leverage Atmos as their platform of choice with little or no effort on their part. As Chuck Hollis, VP Global Marketing CTO eloquently articulated in today’s blog post Building The Atmos Storage Ecosystem Atmos is “… designed to deliver services, rather than simply storage. And if you’re a hardware type, you’re frustrated that the hardware of Atmos is relatively uninteresting—almost all of the value comes from software.” This philosophy is what makes the EMC Atmos and RainStor a perfect combination to preserve and make online retrievable large amounts of historical data for regulatory or business purposes. With RainStor’s specialized repository integrated, ISV partners can deliver enterprise-ready cloud services utilizing powerful Atmos built-in features such as policy-based management and GeoProtect. All this is good technical stuff, but the real silver-lining that makes Atmos and other cloud platforms so compelling is the promise of significant cost-efficiencies as well as new revenue generating opportunities as “big data” growth continues unabated. As indicated in EMC’s Atmos press release: “According to the new EMC-sponsored IDC study titled The Digital Universe Decade – Are You Ready?, in 2009, amid the “Great Recession,” the amount of digital information grew 62% over 2008 to 800 billion gigabytes (0.8 Zettabytes). One Zettabyte equals one trillion gigabytes. In addition, based on the use of cloud computing services by companies to reduce the portion of their IT budget devoted to legacy system maintenance, IDC estimates that the increase in IT dollars spent on innovation could drive more than $1 trillion in increased business revenues between now and the end of 2014. This projection will increase substantially as private cloud and other cloud computing models move into mainstream adoption.” If you are an ISV, reseller, managed service provider or SaaS application vendor, you may already be encountering such “opportunities” in the form of increased data growth and usage by your customers, coupled with a new tide of regulatory and compliance mandates across industries and sectors which require that historical data be maintained and made available online accessible for longer periods. Does cloud computing and the Atmos platform provide you with the best way to monetize? Or should you be also thinking beyond traditional RDBMS’ such as Oracle and MySQL and focusing on new technologies that are optimized for longer term retention and retrieval of big data. This was to be a 3 part series, but I’ve received such interest and kind emails wanting to know more about KM, I’ve decided to extend this series of posts to provide more details than I intended. So this is not the final post, but rather will continue as time and questions permit.
The first thing you need to do in a KM project is to determine who are the main “fountains” of knowledge or go to “experts” in your company. If you are a relatively small organization (less than 100 people) you should know this without much analysis. However, given that many organizations do not attempt or think it worth doing KM until they are much, much larger, finding out the individuals that everyone gravitates towards can be accomplished in a more formal manner. 1. Organizational Network Analysis (ONA) – Initially someone (a Product Manager would be idea) should create a map of roles in the organization that are responsible for specific processes. It adds a layer of specificity and identifies what incumbents and managers think is needed to be successful. Much like gathering requirements for products, gathering requirements for knowledge capture and dissemination requires interviewing stakeholders, documenting use cases and identifying gaps and requirements. One-on-one interviews can be supplemented by a survey which would include the following questions:
This information can be used to build a deeper interviewee list and to identify the individuals who should ultimately be part of the KM asset creation and review cycle . Of course you’ll find that participation in this survey may initially be limited, given that a) it’s not everyone’s priority to participate b) there may be a lot of skepticism about the gathering of this type of data and the KM project in general. It is the responsibility of the Executive team to do their best to encourage participation, but you can offer an incentive or two. For example, I offered $50 Amazon gift certificates to every 10th survey received back (in the order in which they were received) and ultimately got 80% participation by my company. 2. Knowledge Inventory Map – Next an inventory of all knowledge assets and the form they are in should be defined to ultimately help in the gap identification process. This can be done by trolling the intranet, documentation, training assets, product marketing, best practices docs from solutions delivery, customer FAQs, CRM knowledge base and more. This can be quite frustrating as the discovery process will reveal multiple versions of the same document, no idea of which is more accurate. All of which is contributes greatly to exactly why the company needs this KM process. When all this has been accomplished a framework for the contributing organizations (and identified individuals) can be visually mapped out. An example can be seen in the graphic below:
In my next post I will dive into this diagram and detail the design of a central knowledge repository that helps align and make accessible all of the knowledge within an organization.
This is Part 2 of a TBD # of parts series:
In Part 1 of this series I reviewed some of the common problems and challenges that may be warning signs that a company could do with some good solid Knowledge Management (KM). Today we’ll take a look at who should be charged with driving KM in an organization, who needs to be part of the solution, and how to get it done. As ever, comments welcome and appreciated. Align KM with company’s key objectives KM is not a small side project. It requires significant resources and commitment which starts from executive leadership. In order to transform an organization, the metrics and goals of KM should align with the company’s targets such as sales, revenue or customer/partner satisfaction levels. Given KM might be considered a “new” function, there might be a temptation to go hire “KM experts” to help in the analysis and planning. While this may be helpful, it could be a costly endeavor and they should not be the main leaders of the KM initiative, rather it is critical that the KM team and project be led by an internal employee. So who should own and drive KM for the company? The characteristics of the individual or group should be profiled as follows: 1. Has good and broad relationships across the organization, interacting with all departments Sound like a familiar profile? Indeed, a good senior Product Manager(PM) is ideally suited to drive your KM initiative. Unless they don’t meet the criteria above (in which case you should question if they should be doing PM in the first place) Make no mistake though, a full blown KM initiative is not a part-time endeavor, therefore it requires some additional resources and full blown commitment from the executive team and in some cases the board of directors. Additionally the KM Czar would not be able to simultaneously fulfill critical PM duties in parallel. This should be a dedicated project for the duration. The first step in any KM project should be to create a KM Charter upon which there should be no ambiguity and various department heads who are required to invest time and effort need to sign up (literally get some signatures) and commit. The contents of this charter might look something like this:
Furthermore, the opening summary of the charter should emphasize the seriousness of the KM initiative: The KM Charter recommends and requests the following, without which the KM initiative cannot succeed:
There are other risks and dependencies as listed later in the document, but immediate commitment and approval of the charter with full alignment by the executive team will allow the KM initiative to be launched. Clearly stated goals and measurements should be defined up-front. An example is below:
Let me reiterate for the hundredth time in this article, no KM initiative should proceed unless the executive team are agreed and commitments from each individual department group responsible for contributing and managing content has been received. KM when not executed properly CAN BE A WASTE OF TIME, RESOURCES AND MONEY, as illustrated by the Dilbert cartoon. To avoid ridicule and to be successful, KM must be viewed in all seriousness as a major contributor to the company’s goals and objectives for growth. In the next post, I will delve into how to get started and present a content capture and sharing framework for a KM project illustrating how groups should participate and contribute. This is Part 1 of a TBD # part series:
Please leave me any comments or questions you may have in between posts so that I may incorporate any ideas or suggestions you may have. What is Knowledge Management (KM)? Before I answer that question, see if the following sounds familiar to you:
Internal staff communicates with each other through the following channels/mediums:
External Communication with partners, customers and prospects takes place through:
Due to a combination of lack of resources, inconsistencies, lack of commitment and infrastructure issues, both internal and external use of the channels previously listed have been known to be inaccurate, siloed, out-of-date or inaccessible. This has been documented to affect implementations that may need to be “rescued”, re-architected through the use of limited key personnel. The result is cost inefficiencies as well as lowered customer satisfaction. Why Knowledge Management (KM)? If any of the above resonated with you, you have identified one of the most powerful, but as yet untapped opportunities in organizational process, development and execution. There is an old saying “knowledge is power”. That reference was focused on keeping a tight hold of information individually and leveraging it for personal gain. In today’s dynamic, fast moving business environment, “knowledge is still power” but power which should be leveraged for the good of the company, it’s partners, customers and prospects. As such, I define KM as the Consistent capture and efficient dissemination of accurate information internally and externally. Whose ROI can be measured by process efficiencies, increased productivity, improved customer and partner satisfaction and ultimately greater profits. In Part 2 I’ll describe how to begin thinking about KM, who should be involved and where to start. |
|||||||||
|
Copyright © 2010 Ramon Chen: Cloud 'N Clear - All Rights Reserved 6 guests, 0 members Max visitors today: 14 at 05:46 am GMT+8 This month: 67 at 07-28-2010 11:02 pm GMT+8 This year: 91 at 05-04-2010 07:10 am GMT+8 All time: 91 at 05-04-2010 07:10 am GMT+8 Twitter links powered by Tweet This v1.6.1, a WordPress plugin for Twitter. |
|||||||||
Loading ...
According to the 
Join the Discussion