For most business intelligence and data warehouse architects, your informed CIO and/or CTO have already asked you about the growing trends in the industry, namely Big Data and Apache Hadoop. Hopefully, this was a measured and thoughtful request for information in increasingly detailed levels of granularity, matched against your awareness of the company’s need, maturity, and capability. Hopefully.
Unfortunately for many of us, that conversation started out as a hallway drive-by, when the CIO (on the way to another meeting) caught us by the lapel and asked casually, “Hey, I’m hearing about this Big Data thing. Should we be doing that?”
This is every BI Architect’s nightmare scenario.
It’s a no-win obviously, because Big Data solutions are never as simple as “yes, we should” or “no, we shouldn’t”, at least at the outset. A Big Data initiative needs to flow out of the company’s data architecture direction and strategic planning for how data will be used to drive the company’s success. To get asked by the CIO for a snap answer like that isn’t fair, but hey, this is business.
If you say “Yes, we need to get right on this,” then likely you’re tasked with making it happen, and if it fails it’s your fault. Inevitably our off-the-cuff answers cannot possibly anticipate the kinds of technological, integration, and organisational challenges to overcome in making a Big Data solution work for the enterprise. And by “work”, I mean generate demonstrable value. What’s worse, many companies have reluctantly admitted that initial Big Data projects (like initial BI projects) don’t generate quite the advertised ROI out of the box, but only really provide a competitive and internal advantage after a period of time. Generally, mining big data eventually becomes an essential part of the company’s business processes.
On the other hand if you say “No, we need to sit this out while the industry sorts the wheat from the chaff”, though that’s a likely safe answer (which might also buy you time to go do your homework and prepare a business case for it, since now you know that it’s on your CIO’s radar), it nonetheless can appear resistant-to-change or worse yet, uninformed — hardly the image the effective BI architect wants to project!
So how to answer your CIO’s hallway drive-by, then?
The best answers to this question contain a bit of optimism, a bit of caution, a bit of industry wisdom, and a bit of your own analysis. Here are a few phrases that you can consider (read: steal) when you get asked about Big Data:
- Some companies that embrace bleeding-edge technology advances are seeing promising results.
- Others are seeing that they do not yet have a suitable Big Data challenge.
- The vendor landscape is still maturing.
- Our systems are/are not ready for deep integration.
- We do/don’t have the kind of data that Big Data solutions thrive on and from which we can likely generate ROI.
- I have some preliminary analysis about candidates we can look at for a proof of concept.
Obviously for the last one to work, you need to have done some homework already. But this will at least get you through the volatile initial conversation in the hallway and into the longer, more involved, and more measured dialogue that guides your organisation into the right decision-making framework.
DataHub Writer: Douglas R. Briggs
“Mr. Briggs has been active in the fields of Data Warehousing and Business Intelligence for the entirety of his 17-year career. He was responsible for the early adoption and promulgation of BI at one of the world’s largest consumer product companies and developed their initial BI competency center. He has consulted with numerous other companies and is regard to effective BI practices. He holds a Master of Science degree in Computer Science from the University of Illinois at Urbana-Champaign and a Bachelor of Arts degree from Williams College (Mass).
View Linkedin Profile->
Other Articles by Douglas->