Polarization of BI Solutions

It almost seems like the big BI companies figured out they had to be a part of something even bigger to justify their already sky-high license costs. Just within last 12 months, we've seen Hyperion get bought Oracle, Business Objects get bought by SAP, and now today Cognos got bought by IBM.

And I can't help but feel that Enterprise BI software has become a dying breed.

If you are a fortune 1000 type of organization, chances are you already have some sort of elaborate licensing agreement with at least one of these acquirers (Oracle, SAP, or IBM). So now, they have one more thing to sell you so that you can have a check mark for your BI initiatives without having to drive too far to a different vendor.

I guess I shouldn't be so cynical. As BI software, Hyperion, Business Objects, and Cognos are pretty feature-rich, and have many compelling qualities. My problem is with their complex enterprise deployment model, proprietary nature, and staggering cost of ownership that continues standing in the way of making BI available to the masses. And their recent acquisitions further polarizes the world of BI haves and have-nots.

Perhaps this is how corporate world will come to demand a new breed of BI solutions. Back in 2002, when we started our marketing analytics company, we thought (and still do) open source and software-as-a-service were going to drive this new trend. And it's good to see companies like JasperSoft, Pentaho, Swivel, and LucidEra lead the way in this direction. What's more important than just looking for open source and/or SaaS solutions to BI, is to demand "openness" -- openness in terms of architecture, sharing of insights, and just as importantly, openness about cost of ownership. How many times you have seen a BI initiative where the entire budget got spent just on data integration and/or cleansing? How about the cost of those special consultants who seemed to be the only earthlings that understood the esoteric workings of a popular yet proprietary BI software package?

Above all, one needs to keep in mind that BI is about leveraging all available data to get a clearer picture of what's going on in the business, be able to focus on the most relevant issues, and make better decisions. Instead of taking a centralized approach of one system doing it all, a good BI system today needs to act more like a network that can connect to various data sources, systems, API's, web services, etc. What if your BI system acted more like a mashup that lets you combine compatible information sources and cross-reference them as you please? Swivel has an approach close to this except that it expects all information to be uploaded by its users. It could be interesting if Swivel could connect to some common public data repositories (like geographical locations, weather, stock prices).

The key here is to have BI systems that enable mashup of information resources and analyses, a la web 2.0, as opposed to being traditional "enterprise" solutions that need your business to bend over backwards to fit into their proprietary framework and terminologies. Which to me conjures up images of Oracle, SAP, and IBM.

Maybe I liked them better when they were Hyperion, Business Objects, and Cognos. At least they didn't say -- "BTW, we also do BI".

Tweet

Freeway to Yosemite

Freeways are ugly. After driving for a few hours, once I took the exit off Highway 99 to get on Hwy 140 towards Yosemite National Park -- I felt the 2-lane highways much more a part of the landscape than the concrete mesh work I'd just driven through that meandered through equally artificial and out-of-place looking subdivisions. But I digress..

On the way, a rock slide had closed a section of the highway. There was a detour that alternately stopped traffic for 15 minutes each way so vehicles can take turn on a single-lane highway. The geek in me kept wishing for sensors that optimized wait time on each end, but I told myself the whole point about being in Yosemite was to slow down.


More to reinforce rather than repeat something you may have have heard a thousand times -- Yosemite in fall is simply wonderful. Not crowded, the weather is sunny, nice, warm. Ok -- most of the waterfalls were dried up. But, would you consider the following images any less breathtaking?



Since we had our 2 little kids with us, we opted for a relatively short hike to Vernal Falls Bridge (categorized as "mildly strenuous"). We got around the park in these cool hybrid buses. The kids got real excited when we spotted a small bear strolling around the trees, and I was happy we weren't planning on camping out. The hike to Vernal Falls is a part of a much longer trek, most popular of which is the one that goes all the way to the top of the half dome. Maybe next time when we have some babysitting help (or if the kids are old enough to climb with us).



We learned that a few weeks earlier, a man in his 60s was on his way on the 211-mile trek to Mount Whitney. Apparently he'd made arrangements for food/supplies to be air-dropped via a plane or maybe a helicopter. Sounded like a cool (literally) adventure.



On our way back, it was close to 7:30 pm, both kids were asleep in their car seats, we drove up to Glacier Point (about a 20 mile detour) to watch the stars in the clear night sky. Living in the city, you tend to forget what they sky looks like at night without any artificial light. We even managed to see some shooting stars. Too bad my camera couldn't photograph that -- some things are just meant to be experienced.

Driving back on 120 towards 99, suddenly a deer crossed the road in front of me. Good thing I wasn't driving fast and was able to slow down in time, without waking up anyone inside either. Pretty soon, we got back on the concrete mesh work by way of Highway 99, making me even more certain -- freeways are really ugly.

Tweet

Halloween Pumpkin Carving Contest

In the spirit of the day, we decided to have a little pumpkin carving contest at the office. Here're are the contestants:

Mine is the one on the right below:



And the winner is:

Tweet

Commoditization of Business Intelligence

I received an email from Seth Grimes, a very familiar voice in the BI community, with an interesting question -- Do you now see BI as a commodity market?

He was referring to an earlier post about my open source BI project OpenI, where I'd mentioned:

The state of business intelligence software market has been very much controlled by a few big players. The situation is very similar to how the J2EE application server market was before JBoss, or how database server market used to be before MySQL and Postgres emerged as serious alternatives, or how OS market was before Linux. Pretty soon we will talk about the BI platform market in the same manner, because open source and open standards are driving the commoditization of BI as we speak. It is just a matter of time.

That was a few years ago. So, I asked myself -- well, how do I feel now? Have I learned anything?

The question is a tough one -- something I've always grappled with. Ultimately, it depends on what do we mean by "business intelligence". If we go by the current big commercial players' definition -- then BI is more about a software tool providing capabilities around data warehousing/ETL, OLAP, analytical modeling, and visualization. So by that account, I'd definitely stick to my original thoughts and say it's a commodity market.

However, a more relevant question might be -- does having these capabilities make a business intelligent? In reality, what I've seen is that it comes down to an analyst (or group of analysts) who (a) know how to work a "BI" tool, and (b) have some fundamental expertise in the business domain they are analyzing. So, the "BI" tool is more about facilitating the job of an analyst or a general business user. You could argue that by making performance metrics, etc. more easily accessible to a business user, the BI tool is helping them make more effective decisions, but it is making a big assumption that the user knows how relevant the performance metrics are for the business.

In the end, my take on this is that, the most effective BI tools are domain-centric, i.e. they embody some inherent knowledge about a particular business domain -- so, not only they are extremely efficient and accurate about compiling all the performance metrics and making them available, they also "understand" the applicability of those metrics and can almost act like expert systems in guiding crucial decisions. This, I don't think is a commodity market. It needs to be grounded into specific industry domains to be effective.

Would love to hear your thoughts.

Tweet

Conquerer of Crises

Prayer lamps @ Sankata Mandir - Kathmandu, Nepal

I've been on sort of a hiatus all summer (if you can't tell from the blog posts or lack thereof). Anyway, I worked from our Kathmandu (Nepal) office for almost all of July/August, and also spent quality time with the entire extended family.

It's one thing to read and hear about globalization, but utterly amazing when you are smack dab in the middle of it. Kathmandu is, well, increasingly chaotic. A coalition government is struggling to restore stability, while there are protests of all kinds happening on a daily basis, sometimes going to the extremes of calling a "bandh" (general strike), which mostly means no vehicles allowed on the streets (except for ambulances and tourist buses, after all the local economy badly needs the tourists).

Still, life goes on, people figure out a way to get their business done. For IT-based businesses, these "bandh"'s really don't matter a lot. "High-speed" (usually 128 kbps) connection literally provides the information superhighway to get past the "bandh". Let's just hope no one stops the traffic there.

For logical thinkers, it is hard to explain how the city/country functions at all -- but it does, and has been doing so for quite some time. The rate of technology adoption is just nuts. Entrepreneurship is on the rise. All the while an uncertain political climate looms, but perhaps people have figured out over time -- this too shall pass -- and carry a strange confidence that their families, jobs, businesses will somehow always find a way to survive.

The Sankata Mandir is an old temple right in the heart of old Kathmandu. We got there as I followed my family through lots of temples. I was mostly looking after the kids, not being much for organized religion myself, but hoping somehow this might help our young ones develop an appreciation for the higher powers. Most of these temple, I'm sorry to say, are uninspiring -- but something about Sankata felt different. Perhaps it's the name -- a loose translation will be "conquerer of crises" -- that symbolized the current state of Kathmandu and Nepal.

And so even after all the daily accounts of various "crises" -- I came back hopeful, for just like these prayer lamps in Sankata Mandir, there are enough souls in Nepal that carry a strong yet silent power inside them. While these people claim they don't know what's keeping the place together in midst of these chaos, they don't need to look any further, just need to look inside themselves. I truly hope they conquer all their crises.

Tweet

We'll have more women employees, if we only had a separate restroom

Since my earlier post re men vs women engineers, I have had several discussion with our offshore counterparts on why they don't have as many female employees. Something they mentioned a week or so ago was particularly funny, but also a harsh reminder of some of the unusual (from a western point of view) infrastructural challenges:

Here's the issue: the office has only 1 restroom (for a staff of about 10). And women employees don't like that.

And they aren't unique in this challenge. Most of the office buildings have restrooms built as an afterthought, and even if they have multiple restrooms, even allocated as "Ladies" and "Gentlemen" (part of old colonial legacy) -- the allocations aren't honored all the time. So, a typical complaint across all women staff is the lack of clean, and more importantly, dedicated restroom facilities.

Now, I doubt if this is a key reason for the lack of female engineers -- but in a generally reserved culture, I'm bound to assume that the female employees don't discuss this issue as much with the management, and the management (usually male) doesn't think much of the issue either to actually take some actions. Maybe it's too trivial, not core to the business -- whatever, but it is an issue nevertheless. Maybe offshore locations plagued by high turnover problems can use this as a differentiator for retention.

I should point out that our offshore counterpart has a female president.

Tweet

Email Marketing is about Engagement First, Revenue Second

For the last couple of months, I have been trying to get my head around the analytical nuances of email-centric marketing (as a part of my new gig with Responsys). At a first glance, there seems to be a plethora of metrics – some around deliverability, then open rates, click rates, conversion rates, etc, etc. Which ones of these are the true measures of success? To some extent, they all are – but I can’t help but look for a top-down hierarchy which looks at the bottom line first, and then delves into the various supporting factors.

The success of your email (or any other customer contact for that matter) depends on whether the email was successful in influencing a desired action on the recipient’s part -- Did they actually open the email? Did they actually click on any of the external links in the email? Did they actually perform the action that was the goal of the email – like making a purchase or signing up for a program? Measuring each one of these events and understanding how they are influenced by external factors are crucial to optimizing the success of your email campaigns.

A common misperception about emails is that they are more-or-less “free”. Compared to the cost of a direct mail piece or a telemarketing call, the physical cost of sending an email is definitely lower by several orders of magnitude. As a result, what is also common is that organizations get somewhat sloppy about measuring the cost and impact of their email campaigns, and they tend to measure email campaign effectiveness on a more general level.

Particularly important are the hidden costs of email campaigns. Since it is easier and cheaper to send several emails to a recipient, often “list fatigue” is realized much sooner in the email than in offline channels. This means people in the email lists opt out more quickly and the response rates decrease more sharply, indicating a more negative experience for the email recipients.

This breakdown in relationship manifests itself in the following email engagement metrics:

• Decreasing click-through rates and conversion rates
• Higher opt-out and spam-out rates

However, a common trap most email marketers fall in is to address the above issues by increasing the volume of emails. The thought pattern is something like this:

I am sending out 100,000 emails today, and I only get a 0.1% conversion rate. Since my target is to drive 10,000 new web sales by email this quarter, I must send out 10,000*100 / 0.1 = 10,000,000 emails this quarter. And because it costs me only a penny per email, I am basically spending $100,000. If the gross margin per sale is higher than $10, then I even have positive ROI.

While this calculation may be mathematically correct, the major flaw in this approach is that it evaluates email purely on short-term revenue potential, and does not consider the impact of email volume on the overall quality of customer engagement. In a way, it is very similar to the bad image telemarketing has gotten over the years because telemarketers have been entirely focused on making a large volume of calls without caring much how many people get infuriated in the process. The difference is that in the case of emails, people can easily unsubscribe with a click, tag you as a spammer, or simply hit delete any time they see an email from you. They can do a lot more damage to you in a much shorter amount of time.

So, it is essential to think of email as an engagement tool first, and revenue generator second. The primary goal of your email campaign should be engagement, which in turn drives value. It does not work the other way around.

In the coming days, I will post more of my thoughts around measuring engagement and its influencers. I will also like to hear from you – what are your thoughts around measuring engagement driven by email, and/or do you see the problem space differently? What should be the role of analytics in driving success of email campaigns?

The more I think about this, it seems that analyzing success of email campaigns is not just looking at one particular success outcome, or even looking at individual metrics like open rates or click rates -- the task seems more about developing some sort of "Email Engagement Index" (EEI -- do we really need another acronym?) that incorporates all the different manifestations of engagement that were influenced by the email (open, click, purchase, forum posts, calls to the call center, web site visit, etc.) with their appropriate "weights". Then we need to track the impact of our email campaigns with respect to this index. Else we run the danger of looking at this multi-dimensional issue in a more unary manner (such as looking at only revenue or open rate, one at a time), and thus, not realize the actual impact of the email campaign.

I'll have more on this soon, but thought I should at least get the dialog started.

Tweet

Loyalty Matrix becoming a part of Responsys

Some of you may have seen the press release today -- Loyalty Matrix, the company I founded back in 2001, was acquired by Responsys earlier this month. We moved offices last week (it is only 3 blocks away in downtown San Francisco -- so wasn't that bad), and now a new phase begins for the Loyalty Matrix team (including myself).

Since I did become a parent during my tenure at Loyalty Matrix, interestingly I find a lot of parallels between founding a company and raising children. They both need a lot of nurturing at the beginning, many sacrifices, but it is your love/passion that keeps you going. And at some point, you realize that your child/company will need help from beyond the parental units to ensure proper growth. With children, it is the schools, the soccer coaches, friends, guidance counselor, etc. -- and with a company, it is the investors and/or an acquirer. In a way, I feel like we finally got Loyalty Matrix admitted into college where it will have the necessary infrastructure to flourish to its full potential.

So it was fortunate that (a) we as Loyalty Matrix finally recognized that we were ready for the next step in our evolution; and (b) Responsys got the big picture that combining analytics with execution is the next big thing in marketing automation.

Deals like these are always time consuming and intense, doesn't matter if you are a small company or big. Now that the deal is done, here comes the fun part -- we get to work! plus I get to blog when I want to!!

So friends -- a new journey begins. You will certainly hear more often from me. Let's keep the hope alive.

Tweet

Do You Prefer Men or Women?... Engineers, That Is.

We have been offshoring for about 2 years now with a small firm in Nepal. Recently someone referred a different offshore firm and suggested I should check them out as a backup option.



The contact happened to be someone based out of the San Francisco bay area, but he travelled frequently to India and eastern Europe, where he managed different offshore teams. The meeting started cordially, with the person describing different areas of technical prowess, fat communication lines where we can call a local phone number which rings an IP phone in Mumbai next to our offshore developers. He also had pictures showing the office -- very similar to the cube farms one would find in silicon valley.



Then he started asking me about our particular technology needs, what type of work we were trying to support with offshore help.. and then, out of the blue -- he asked me:



"Now, Sandeep... do you prefer men or women?"



What?? What kind of question is that, I thought. "Excuse me..?" I said.



"Well, do you prefer male engineers in your team, or women? We can assign team members however you like."



I tried to explain that it didn't matter. Not just because we have equal opportunities law here, but more from a fundamental belief that what's important is that we have productive software engineers, and we could care less about their gender or any other demographic traits.



Yet this person asked again -- implying I shouldn't get hung up on equal opportunity law, since apparently they don't in India, and if I had any deep-rooted prejudices against women software developers, this was my chance to make that clear, and he'd be more than happy to make that happen. He presented it almost a big differentiator that set his offshoring practice apart from others.



Needless to say, we didn't do any business with this person. However, it brings back the issues with gender prejudices in software engineering. Granted that the number of women in software field is very low compared to men, and on top of that we have stereotypes about women not being as good in math and sciences -- but, is it as bad where an offshore company can get away with asking for gender preference for your offshore team's makeup?



I hope not.



I'm not sure if my or anyone's chastising this person on gender equality would make any big of a difference. But if enough people say No, and say that gender doesn't matter, maybe he'll get the message.



Still, how many times do you get asked by a consulting firm -- "do you prefer men or women for your projects?" I can't help but scratch my head every time I think about it.

powered by performancing firefox

Tweet

What do Howard Stern and Rush Limbaugh Have in Common?

Apparently their birthday. Both were born on today's date -- according to the local public radio I was listening to this morning on my bus ride to work.

I thought that was funny -- polar opposites of talk radio born on the same day (I'm sure not the same year, although that would really be funny)

Tweet

OpenI Listed in Top 10 Free BI Apps

Ask anyone who is involved in an open source project, getting recognized is one of the greatest kicks you get out of the whole deal.

So, when my Google alert picked up this news on OpenI listed in the Top 10 Free BI Apps list, it absolutely made my day. Of course, a lot of the credit goes to all the folks who have contributed to this project, and the open source community that has supported us all this time. And thanks to Tamina Vahidy for recognizing the project.

When we started OpenI back in July 2005, we just wanted to subsidize our R&D. We needed a BI platform to deploy our analytical models, and instead of opting for commercial BI platforms which would have never fit into our cost model, we decided to develop a BI platform using available open source components, and also as an open source project of its own. We figured if we get even a couple of people outside of our company to pitch -- whether it was design help, or just thinking through requirements, use cases we hadn't encountered -- that alone would pay for the efforts to make it open source.

Well, not only we got design help and advice from a great deal of smart folks in the space, we even have people contributing code. I remember someone (maybe Steve Weber) making a point about open source development model -- not all the smart people in the world work for you, so the only way to get them involved in your projects is via open source (ok, you may argue crowdsourcing ideas such as Netflix's contest, but I don't have a $ 1 million to give away in prize money :-)

So -- here we are -- working on version 1.3 of the product. We are using it internally as the web front-end of our commercial product. Of course, it has ways to go (see roadmap) -- but as contribution and recognition keep coming in, it just seems like a much more rewarding way to develop software.

Tweet

Is Open Source Java More Restrictive under GPL?

Today will most probably go down as an important day in the history of software engineering -- Sun open sourced Java under GPL license! (Sun's CEO Jonathan Schwarz's blogs on it as well)

While this is a very welcome news, the choice of GPL as the license has me a bit concerned. Two years ago, I was researching the various open source licenses like mad for our own open source project OpenI, and decided against GPL because of its "viral" nature (we ended up with Mozilla Public License, MPL). The "viral" aspect of GPL (section 2(b)) says:

You must cause any work that you distribute or publish, that in whole or in part contains or is derived from the Program or any part thereof, to be licensed as a whole at no charge to all third parties under the terms of this License.

and later further clarifies:

..it is not the intent of this section to claim rights or contest your rights to work written entirely by you; rather, the intent is to exercise the right to control the distribution of derivative or collective works based on the Program.

In addition, mere aggregation of another work not based on the Program with the Program (or with a work based on the Program) on a volume of a storage or distribution medium does not bring the other work under the scope of this License.

When someone uses programming language software like Java to build their own software, it is tricky to determine if their work is "derivative", "collective", or a "mere aggregation". In such a case, all the software code is written in Java. The software requires a Java Runtime Environment, which acts like a virtual operating system, to run. So, Java as a software construct becomes very tightly coupled with each new software program built on Java.

This is a contentious issue where there has been a lot of debate. See this blog entry at O'Reilly and its comments back in 2004

For example, with Java being GPL now, what happens when I build a software program in Java and would like to distribute it along with a Java runtime engine (JRE) for user convenience? Since I have built the software using Java and I am combining the JRE in my distribution, will my software be considered a "derivative work" or a "collective work"? If my software is considered "derivative" than by definition of GPL, I *MUST* distribute my software under GPL, which can severely affect the commercial viability of my software.

So, who decides what is derivative vs collective? Unfortunately, most likely it will be lawyers rather than software developers. And that may very well cause at least some amount of concern amongst those who build commercial (i.e. non open source) software using Java for a wider distribution (i.e. for use outside of one's organization, most probably selling commercially licensed copies of software to customers).

Now it is probably unlikely that Sun will come after commercial software developers who use Java claiming their software fits the category of "derivative". However, if any organization has to spend any legal resources to investigate this, it affects the use of Java in commercial space.

I think Sun should make it very clear to the community why they chose GPL, and how they want to assure the community (both open source and commercial) that their software development efforts in Java are not subject to the "viral" aspect of GPL.

UPDATE: CLASSPATH EXCEPTION TO THE RESCUE

After digging through more details, I found 2 things that address my concern:

1. Sun is distributing Java under GPL with the ClassPath exception, and describes in their FAQ the reasons behind this -- with an example almost the same as I posted earlier where the classpath exception allows me to distribute my software written in Java bundled with a JDK or JRE without requiring GPL:
It allows you to link an application available under any license to a library that is part of software licensed under GPL v2, without that application being subject to the GPL's requirement to be itself offered to the public under the GPL.

2. Following one anonymous commenter's suggestion, I read the original GNU Classpath license text , and it does clearly describe the exception:
As a special exception, the copyright holders of this library give you permission to link this library with independent modules to produce an executable, regardless of the license terms of these independent modules, and to copy and distribute the resulting executable under terms of your choice, provided that you also meet, for each linked independent module, the terms and conditions of the license of that module.

Still, my suggestion to Sun still remains the same: you can't assume everyone will go through the trouble (granted it wasn't much) of digging through the details of the classpath exception, and as such, they are likely to remain under false assumptions re GPL restrictions on Java. So, don't assume -- everytime you mention Java and GPL, make sure you talk about the classpath exception and how it enables you to distribute your software written in Java under any license.

Tweet

"All marketers are liars". Seth Godin is a marketer. Hence?

Sorry for the cheeky title, but Seth sort-of ticked me off today with his post "Nobody Knows Anything".

First off, as someone who makes his living providing marketing analytics, I get a bit, let's say annoyed, when someone like Seth starts a blog post with "There are two kinds of marketing analysis, both pretty useless".

I calmed down a little bit when later in the post he conceded "Here’s the really good news: in addition to analysis, marketing today offers something that actually works: a process".

But his post in general has an attitude that says "Marketing is not science", and that "most marketing breakthroughs come down, sooner or later, to luck".

Well, I am not a guru like Seth (or as "lucky" in terms of having a couple of bestsellers under my belt), and he definitely knows a lot of things I don't -- but this is my blog :-) -- so, I will say that marketing is just as much of a science as social science is. It may not have equivalents to Newton's law of gravity or Einstein's theory of relativity, but marketing analytics does have some tried and tested ways to leverage data to make smart predictions about future behavior of customers and prospects.

Now, once equipped with the intelligence that marketing analytics provides, it is completely up to the marketers on how successfully they can change their strategy and tactics to yeild results (so, marketing is only partially scientific) -- but marketing analytics will almost always put a marketer somewhere above pure dumb luck.

Of course, you may agree with Seth, or not -- but I just had to get this off my chest.

Tweet

Marketing Analytics for Salesforce.com

This week and last, we've had 2 good conferences in San Francisco, very relevant for marketing analytics. Last week it was DreamForce'06 from salesforce.com, and earlier this week, we had DMA'06.

Let me talk about DreamForce first because it has become a fad these days (specially for anyone working in BI or SaaS) to integrate with salesforce.com using AppExchange. Since early this year, I have been eagerly trying to find an angle between our business and salesforce.com, so DreamForce'06 was obviously very relevant.

A good friend of mine from college, John Barnes, was at DreamForce'06. John is the VP of Technology at Model Metrics, a Chicago-based firm that specializes in customization and integration of salesforce.com with legacy systems. I pinged John to find out more about what salesforce.com is actually doing about marketing automation and analytics, which was very insightful (thanks John).

Salesforce.com website describes marketing autmation as a key component of their platform, which also includes marketing analytics. Turns out while it is possible to define and manage campaigns using salesforce.com, the marketing analytics bit only provides some very basic reports.

So my idea is fairly simple. If people are actually running high-volume direct marketing campaigns using salesforce.com, then we will write an AppExchange component to suck in all the campaign and customer data into our platform and then provide much more sophisticated predictive analytics and other fun stuff on our platform -- make a lot of money, retire early, speak at next DreamForce (sorry I get carried away with this vision thing).

Anyway, about 10-20% of SF customers are using the marketing functionality today. The SFA is the main use but marketing is growing more and more. But 10% of 22,000+ customers is still a good market to go after. And the marketing analytics (and reporting) on SF is lacking. The main obstacle to making it better is that their API does not have a join capability so the only way to do better reporting today is to have a copy of SFDC locally and use replication software to keep it up to date (by DBAmp or Relational Junction on the AppExchange).

Sounds like a good opportunity.

So, overall I am happy with DreamForce'06. I had a chance to shoot breeze with an old college buddy, and learn about a very feasible way for us to get into the AppExchange game. Now I need someone who will go in on this with me as a "design partner" :-)

Tweet

Better Blogging by Chemistry

Elliott Ng, a friend, ex-colleague and fellow blogger, has put an interesting commentary on Top 10 tips from (to) a novice blogger posted by Avinash Kaushik.

My take on it is that a blog's success depends on how effective it is in starting conversations -- which means you either get comments like this, or someone links your post on their blog expanding on the topic -- or simply email the link around with some comments.

Why would someone do that?

Well, only if they actually care about what you write about. And caring is more of an emotional response rather than intellectual one. Scoble, Doc Searls, et al really stress on "having a voice", which happens when you combine passion and get awawy from corporate-speak, IMHO.

Scoble's point on being easy to find is also important -- but instead of going the SEO route, it is more important to find interesting conversations in the blogosphere and participate. If you are an active participator with a unique and compelling voice, the search engines are bound to pick you up.

Personally, I found it helpful to write a post outlining my reasons for blogging -- and to the rest of the bloggers out there, novice and experts alike, I'd love to lob the question -- what have you found to be most effective at starting conversations? Was it different than what you'd initially expected?

Tweet

How to win $1 Million from Netflix?

Fellow blogger Michael Fassnacht noted rightly that Netflix has caused quite a stir for marketing data geeks with their recent $1 million prize offer for "substantially improving" their existing Cinematch algorithm to make more accurate predictions of "how much someone is going to love a movie based on their movie preferences".

Call it "crowdsourcing", or harnessing "group smart" -- the approach is intriguing, and one of a kind. Being a curious soul myself, I decided to register a team from our company to check this out (who knows what may happen? we can be smart sometimes with enough luck :-)

A few interesting facts:

• The contest is actually slated to go another 5 years until 2011, the bar being raised each year to improve over last year's winner
• A fine but important distinction: the algorithm needs to predict how someone will rent a movie, NOT what movie someone will rent
• At first glance, the data provided by Netflix seems pretty "skimpy" in terms of richness. Basically you get:
• List of movies
• List of ratings assigned for each movie by an extensive list of Netflix members
• My first reaction was that having extra information on the movies themselves might help. There's a bunch of stuff available from IMDB . However, apparently there are license restictions and also Netflix doesn't really consider extra data to be valuable in improving their algorithm (see the discussion thread )

The "enjoy the journey, not the destination" mantra may be apt for this contest. As you can see on the discussion forum on netflix , this process has invited all sorts of interesting conversation on the validity of approaches, whether Netflix has provided enough data, why should one even bother, etc. etc. -- a dream peer review IMHO, albeit a bit too noisy. So, Netflix should be getting a lot more than their money's worth via this process -- not just by getting better algorithms and the PR buzz, but also by leveraging an almost open-source-type process to involve external community for their internal R&D.

At the moment, I agree with Michael's assessment that trying to solve this with ratings data alone might not be the best way to go. There seem to be so many other interesting dimensions that should influence somone's movie rating: movie characteristics like the cast, director, etc., review from critics, local media review, geo/demographic information about the Netflix member, among others. None of these are being considered in the current algorithm. I can understand Netflix's hesitancy to interface with 3rd party resources, but perhaps they should make all the datapoints within Netflix's movie database available for this contest -- and second, encourage contestants to add their own qualitative datapoints. If the goal is to approach this as a pure improvement of a data mining problem -- then increasing the depth of data should help.

I'll keep you all posted how far we get on this. Being a small company, we will do this in the copious amount of spare time left over after working on existing client work that pays the bills. Still, it should be a lot of fun.

Tweet

An Asterisk that Cost 2 Million Dollars

This just in: a colleague of mine pointed out an analysis that showed a sudden spike in the number of new trial subscriber signups for one of our clients. In early 2005, they had just introduced a new product version and through mid 2006, they were averaging around 3,000 new trial subscribers each week -- which was less than half of what they were getting with the previous product version.

What had happened?

Turns out this client had several product feature descriptions listed during the trial sign-up process. One of these product descriptions happened to have an asterisk next to it, which was explained at the bottom of the page saying it required a credit card number upfront to use it. For a casual observer, it wasn't clear to which product feature the asterisk actually belonged. So, during a content review session, someone caught this and said -- wow, people are looking at our new product and they think they need a credit card to subscribe, which they really don't, and they get very hesitant.

So, in July, a quick content change was made. The asterisk was removed.

Since we are tracking all the transactions, within 7 days after this change, we started seeing a sudden spike in new trial subscriptions, which has levelled off above 7,000 new trial signups a week (for now). More than double of what was happening prior to the removal of the infamous asterisk.

My colleague did a quick calculation on revenue impact, and it basically translated roughly $2,000,000 in additional revenue in the coming 12-months because of the asterisk removal.

I hope you don't have a similarly expensive asterisk ANYWHERE on your website.

Tweet

Can Analytics Influence Direct Marketing Creative Process?

Direct Marketing Association (DMA's) Northern California chapter hosted its first independent meeting yesterday (Thursday Sep 21st) at Intuit's campus in Sunnyvale. This summer, the national DMA "abandoned" formal support for all its local chapters asking them all to go on their own. It's good to see that Northern California DMA has managed to make this independent start, hopefully they'll get adequate local support to thrive.

The main speaker was Bill Mirbach, VP of Direct marketing and direct sales at Intuit -- presenting a talk titled " Owner's Manual for the Creative Process". Now, I'll be the first admit that professionally direct marketing creative is the last thing we deal with -- at our work, we leverage direct marketing data for predictive analytics, so while we can measure and predict if creative version A will perform better than version B for a target audience, that is very different from the "creative process" itself. So, I was intrigued with the topic.

Bill has been around the silicon valley high tech industry. The highlight of the talk for me was a story he shared going back to 1984 when he helped the founder of a fledgling software company called Intuit with their direct ads. The talk was mostly about how companies should chose vendors and vice versa for the creative process, not the artistic aspect itself -- so, during Q&A, I asked Bill how does knowledge of your audience impact the direct marketing creative process? Can one leverage their direct marketing data, knowledge of their customer segments, etc. to make the creative process more effective? What's been his experience?

Here's what I heard:

Yes, a better knowledge of one's audience and their likes and dislikes about an organization's products and services certainly helps the creative person to craft their message more effectively? However, while this sounds logical, this is not what normally happens. The creative process is more the product of the discipline and idiosyncracies of the "creator", rather than driven by data-derived intelligence.

Bill mentioned a particular test where they wanted to measure the performance of a scare-tactic message ("if you don't use our product, you'll be sorry") versus a benefits-focused message ("and you can get X, Y, and Z at the click of a button") -- where the creative person just didn't believe in scare-tactic and came up with a very tepid "scary" message (which obviously didn't perform well). Whereas he used a different creative person who specialized in scare-tactic message (scary thought, pardon the pun) -- which turned out to be very effective.

What does this tell us?

I don't know how much of Bill's story is the norm or exception, but he certainly has been around direct marketing creative people a lot more than I have -- so I must respect his POV. Still, it seems like rather than asking creative people to use marketing data driven intelligence to fine tune their message -- it's probably better the other way around -- i.e. leverage the analytics to find out what type of messages you want to be send out to different segments -- THEN find the right creative team to craft those messages.

What do you think?

Tweet

Passion for Data Visualization

I found this on Christopher Ahlberg's blog, and I completely agree that this is a true display of "passion for information visualization".

http://www.ted.com/tedtalks/tedtalksplayer.cfm?key=hans_rosling&flashEnabled=1

The software used for the presentation is at http://www.gapminder.org and is also a google tool (http://tools.google.com/gapminder) It appears to be "bundled" with the global economic data, not sure if there is an open decoupled version that one can point to their data and play around. Although it seems flash-based and pulling data from static data sources (didn't seem like rdbms, but I could be wrong) -- but this would be a great way visualize OLAP data.

What's interesting is the concept of a "play" button for the time dimension, which makes a great use of animation to see how different quantities (measures) change over time. It also manages the screen real-estate well to put different dimensions on X or Y-axis. But most of all, this truly exemplifies what data visualization is all about -- it goes beyond the realm of charts and graphs that take a while to decipher, and rather tells a very clear, compelling, and visual story. Very impressive!

Tweet

We've got charts and graphs to back us up, so f@#$ off!

Recently I had a potential partnership discussion to evaluate whether our predictive analytics technology could provide key insights from this potential partner's (I'll call them company XYZ) marketing database. Here's how my conversation went with Mr. X, an exec at company XYZ, (which is a marketing technology company):

Me: "So, what are they key business pain points for your clients that we can analyze?"
Mr. X: "Well, you know, the usual stuff -- marketing ROI, cut costs, increase sales, etc."
Me: "ah.. yes, but can we delve a bit deeper? Where exactly clients' marketing programs need help?"
Mr. X: "what do you mean?"
Me: "Well, are they more worried about increasing acquisition volume, or is it more about predicting high-LTV customers, or is it more about retention? I'm trying to get a sense of what is their #1 issue?"
Mr. X: "they don't know.. it's probably all of that stuff"
Me: "It's important that we get a sense of priority, because otherwise we are talking about applying analytics without really knowing what we are trying to optimize"
Mr. X: "well, you are the analytics expert -- you need to tell them what to analyze. They don't think like you, worrying about success metrics, etc. They ask us to run marketing programs, and now we'd like to sell them some analytics. I can tell you what data we have on their marketing programs, now you tell me what kind of analytics you can provide me that I can sell."
Me: "But we do need to understand their business objectives before determining what analytics is relevant enough so that they'll pay for it"
Mr. X: "What I need from you is some screenshots --- some charts and graphs that show what kind of analytics you can do -- I'll be more than happy to review that and tell you if we can work together"

Needless to say, I didn't sense a true spirit of partnership here, but I did sense an attitude that I find more often than I'd like that analytics is all about producing charts and graphs that the user will somehow find useful.

Which, IMHO, is total BS!

But I can't blame Mr. X too much because this is a pretty common perception of analytics in the marketplace. Recently I talked to a marketing exec who said --"everytime I meet with the analytics guys from our agency, they basically have this big ream of a powerpoint deck filled with one chart after another -- and I don't want to see all that stuff; all I want them to do is to tell me what relevant insight(s) did they find, and what course of marketing action would they recommend, and it's like pulling teeth to get them to move beyond charts and graphs and talk about action."

Look at the websites of any business intelligence software provider, or analytical software provider -- and I will guarantee you that you will see a bunch of fancy charts and graphs, and dashboards with enough dials and speedometers to make you dizzy. Somewhere along the line, maybe we have forgotten that the purpose of analytics is to equip us with insights that enable better decision making.

So, first off, the type of analysis being done has to be aware of what type of decisions we are exactly expecting to improve; and second, the result of the analysis needs to be presented in a fashion that is "integrated" into the decision making. Maybe you list your recommended actions next to your charts and graphs, or maybe you somehow highlight the figures and trends that demand attention. My point -- don't leave it upto your user figure out the action based on the fancy charts and graphs, find out what decisions users are trying to make, and provide information that fills in that gap between analytics and actionable insight.

Tweet