What 127 Hours Tells us About Social Networks

In an interview this week for Mark Kermode’s Film Review show, Danny Boyle made it clear that he used the success of Slumdog to make a film that might otherwise not have been made. To leverage the success of 2010 most acclaimed film is an indication that 127 hours is more than this years ‘would you?’ movie.

For those few that have not yet heard about it, it is the true story of Aron Ralston who gets trapped under a boulder whilst canyoneering alone in Utah. The desperate measure that he takes to free himself is well documented so it is not giving anything away to say that he was trapped by his arm, he has a multi-tool (a really cheap one) and a little under 127 hours to debate if he should … or should not.

Before, I go on you might be wondering what’s the connection between BI and Social Networks let alone the connection between Danny Boyle’s latest movie and Social Networks. Those that follow my posts and tweets will know that Social platforms interest me because I think they are changing the way we share and use information in business and will profoundly change the Business Analytics space over the coming years. A social platform has already made it into IBM Cognos 10 because these guys, again, are ahead of the game. Many don’t see it yet because the original use of social platforms have trivialised their significance but it’s there nonetheless.

 

Back to the connection. Aron Ralston, played by James Franco, is an all-American hero. He’s young, fit, strong, intrepid and independent. He is good at what he does, he has spent a lot of time in his chosen wilderness and is able to navigate it with speed and ease. In fact, at one point in his story, he briefly but convincingly takes the role of park guide. The hopelessness of his literal and figurative fall takes a long time to sink in for our hero. Indeed, when it does dawn on him that he could have shared his hiking plan with his friends or family it wouldn’t be exaggerating to call it an epiphany. It’s clearly a powerful realisation for Aron that he’s not a hero, he’s an arse.

 

There is a moment in the movie where Aron says ‘thank you’. It’s a strange moment. I don’t want to give away why it is strange but once you have seen the movie, you will know why. For me, it was significant because he knew that if he made it home alive (which was still, by no means certain) then he would be changed forever. He would live the rest of his life in the knowledge that however strong, smart and experienced he was that those tiny connections we all make each day matter. Sometimes in small ways because it’s just about about sharing. Sometimes in significant and surprising ways.

For me, I am continually and pleasantly surprised by what I learn on the subjects of analytics, organisational leadership, productivity, start-ups and social media in my twitter stream. It’s full of links to content that cover important ideas from solid thinkers. Admittedly none of them are life-saving but, at a stretch, a rare few might be described as life-changing. Each of them make a tiny but positive difference and sometimes someone in my network helps me (or me them) in a surprising way.

 

More and more choices for BI Solution Architects

We analytics practitioners have always had the luxury of alternatives to the RDBM as part of our data architectural choices. OLAP of one form or another has been providing what one of my colleagues calls ‘query at the speed of thought’ for well over a decade. However, the range of options available to a solutions architect today is bordering on overwhelming.

First off, the good old RDBMS offers hashing, materialised views, bitmap indexes and other physical implementation options that don’t really require us to think too differently about the raw SQL. The columnar database and implementations of it in products like Sybase IQ are another option. The benefits are not necessarily obvious. We data geeks always used to think the performance issues where about joining but then the smart people at InfoBright, Kickfire et al told us that shorter rows are the answer to really fast queries on large data volumes. There is some sense in this given that disk i/o is an absolute bottleneck so less columns means less redundant data reading. The Oracle and Microsoft hats are in the columnar ring (if you will excuse the garbled geometry and mixed metaphor) with Exadata 2 and Gemini/Vertipaq so they are becoming mainstream options.

Data Warehouse appliances are yet another option. The combined hardware, operating systems and software solution usually using massively parallel (MPP) deliver high performance on really large volumes. And by large we probably mean Peta not Tera. Sorry NCR, Tera just doesn’t impress anyone anymore. And whilst we are on the subject of Teradata, it was probably one of the first appliances but then NCR strategically decided to go open shortly before the data warehouse appliance market really opened up. The recent IBM acquisition of Netezza and the presence of Oracle and NCR is reshaping what was once considered niche and special into the mainstream.

We have established that the absolute bottleneck is disk i/o so in memory options should be a serious consideration. There are  in-memory BI products but the action is really where the data is.Databases include TimesTen (now Oracle’s) and IBM’s solidDB. Of course, TM1 fans will point out that they had in-memory OLAP when they were listening to Duran Duran CD’s and they would be right.

The cloud has to get a mention here because it is changing everything. We can’t ignore those databases that have grown out of the need for massive data volumes like Google’s BigTable, Amazon’s RDS and Hadoop. They might not have been built with analytics in mind but they are offering ways of dealing with unstructured and semi-structured data and this is becoming increasingly important as organisations include data from on-line editorial and social media sources in their analytics. All of that being said, large volumes and limited pipes are keeping many on-premises for now.

So, what’s the solution? Well that is the job of the Solutions Architect. I am not sidestepping the question (well actually, I am a little) However, it’s time to examine the options and identify what information management technologies should form part of your data architecture. It it is no longer enough to simply chose an RDBMS.

 

Traditional IT teams are missing the boat on Social Analytics

Being a natural owl and not a lark, it takes something really important or deeply interesting to get me into the City for a 7.30 am breakfast meeting. Ed Thompson of Gartner speaking on how Sales, Marketing and Customer Services are making use of social media last week more than qualified.

 

The focus was not the usual ‘if facebook were a country’ hype but very much on how ordinary businesses are adapting to the world of social media and getting ahead through practical application of new and innovative solutions. Interestingly, the most common applications are brand monitoring and company watching in the form of B2B CRM and Competitive Intelligence. Sectors already adopting include Retail, Hi-Tech, Media and Consumer Goods businesses.

 

Insight came thick and fast but one thing that stood out was that IT are nowhere to be seen. This is, at least, partially because these are new solutions, usually cloud based and IT involvement isn’t mandatory. However, with the internal department involved in less than 2 out of 10 initiatives, they are getting left behind. It could be argued that they only have themselves to blame. When I work with my customers and they tell me that a new server will take 15 weeks to build or that it will be 8 weeks before a new report will run for the first time then I find it difficult to side with the ‘professionals’. Business cycles are getting shorter and shorter whilst IT surrounds themselves with processes and models designed to reduce risk, increase quality and security but that also kick delivery dates so far over the horizon that the business have stopped asking for help.

 

Those that are involved are busy defining standards, mandating architectures and generally slowing things down. My advice to IT departments, BI teams and competency centres involved in such activity is stop. Just stop.Things are moving quickly and by the time you have updated the version control on your feasibility study, it’s out of date. Now is the time for adoption and execution (Ed’s words not mine, btw) The business needs support in getting information on what their customers are saying about their products or the latest marketing campaign. The sales team want to identify reasons to pick up the phone and sell to their prospects and they want it embedded in their CRM systems and processes. Marketing want to understand what competitors are doing, if they are forming new partnerships, announcing new products and how the market is responding. All of this, delivered regularly and routinely, is becoming as critical as daily sales, fulfilment, basket analysis or the senior management team’s dashboards.

As information professionals we should be helping the business corral the world of social media and on-line content. We should be investing time in understanding the new challenges and opportunities that semantic and content analytics represent.  We should also be embracing, experimenting and learning from the emerging technologies that address them. Most of all we should be adopting and implementing.

The growth of SaaS means that the business has a choice now. When it comes to social analytics the early adopters are looking at a range of vendors with innovative solutions that require no more implementation than adding a new bookmark. Then they are looking at their IT teams who are offering them a four page ‘IT request approval’ form. Where would you go?

 

BI Project Managers and Eyebrows

Like eyebrows, you don’t really notice project managers when they are there but if you are rash enough to let them go you will end up looking startled and stupid.

I point this out because over a period of more than 10 years I have had the opportunity to observe many, many BI projects and one of the most surprising patterns is the scaling back of project management largely because the project is going well!

The openly declared reason is usually cost or some other misdirection but it is invariably preceded with pointed questions about what value the project manager has been adding to a project that is going so well. Perversely, the better the project is doing, the higher the risk that there will be murmurings about things like the overhead of project reporting and that project management activity will ultimately be reduced or even removed altogether. It has become as common and predictable as it is deeply and logically flawed.

Perhaps this is one of the phenomena that explains why the trend for project failure is not getting any better. According to the latest Standish Group report which is covered by Peter Taylor, author of ‘The Lazy Project Manager’, in his blog ‘Are your Project Managers working too hard to be successful?‘ instances of challenged (late, over budget or reduced deliverable) projects continues to rise.

As BI practitioners we often value technical skills, competency in the reporting tool and the deep musing of the data architect and yet have a blind spot when it comes to project management. This may be partly because early BI projects were often departmental in scale. It may also be because many of today’s BI Competency Centres originated as ‘skunk works’ initiatives and see project management as all methodology and meetings but we ignore it at our peril.

It is true that project management can be at its most obviously valuable when priorities need resetting, additional resources have to be secured or controlled management escalation is called for. However, we shouldn’t assume that if a Project Manager is not doing these things that they are not doing anything.

Planned projects with predictable timescales along with accurate project reporting are rewarded with confidence from our business sponsors. A considered set of risks based on real-life experience of BI projects will mitigate against them becoming time sucking issues and properly managed issues will prevent them becoming show-stoppers.

A good Project Manager may make it look easy but don’t take the lack of fire fighting and crisis meetings as an indication that nothing is being done. Look deeper for the benefits of order over chaos or be prepared to invest in an eyebrow pencil for a look that is decidedly a poor second best.

 

Small Children, Energy and Efficient Data Warehousing

Last week, I referred to Peter Thomas, and his article Using multiple BI tools in a BI implementation – Part II, In the article, Peter points out that the way to drive consistency across dimensions and measures is to define as much logic as possible in the data warehouse.

I was musing over this again this weekend (I hear the cries of what an interesting life you have) whilst out for lunch with friends and their small children (ages 9 and 7) On the short walk to the restaurant I was amused by how different their approach at getting from A to B was to the ‘grown ups’. We were focused on getting to the destination in a relatively direct and efficient way. However, the children ran to and fro, stopped, doubled-back, looped a few circles and even randomly waved their arms in the air. They generally spent as much time in a state of motion as possible. Clearly the criteria of small children, when en-route to a destination, is to use the maximum amount of energy possible!

This, if you will go with me on this, is rather like trying to implement data warehouse consistency in the BI tools rather than further downstream in the data warehouse. You do eventually get to the destination, but will probably be exhausted, out of breath and hungry. This is fine if you are two children out for dinner with some stuffy grown-ups but not an efficient use of the somewhat limited time of a BI practitioner.

A typical BI architecture comprises tiers that include;

• Source Systems
• ETL
• Data (Data Warehouse, Data Marts, OLAP Cubes)
• BI Metadata
• BI Application (Reports, Scorecards, Dashboards)

A properly architected data warehouse (more on this in later blogs) should have been built against an enterprise schema and is therefore *the* consistent representation of business information. Common definitions of customers, departments, profit and products live here. If there is one good reason for this (although there are many) then it is simply that there these can all be defined once in the data warehouse but would have to be defined many, many times in what can often be hundreds of reports that comprise a BI solution.

One of the reasons that we fail to do this is that inconsistency is often made visible for the first time by the BI tools. At this point the project momentum is around building metadata models and reports. Inconsistencies are fixed where the resources are focused … in reports. Add to this that revisiting the design may need involvement from the ETL developer, the DW designer and the business analyst and, if there is a lack of clarity, the business users. It is no surprise that the report author is inclined to fix it where they stand. After all, the tools make the fix simple and it is only when the report author has built the same calculation for the tenth time that they become suspicious about starting it in the first place. And of course, the initial build of the BI application is only the beginning. Many more reports will be built over the life of the BI application.

So work hard to establish the correct definitions during the design of the data warehouse and it will reap productivity gains. Leave it to the BI application only if you have the carefree attitude, the free-time and the energy levels of an eight year old.