Here is the session info for those that are interested:

Topic: Pentaho Community Technical WebEx – James Dixon – Pentaho BI 4.0 Thin Client APIs and Theming
Date and Time:

Wednesday, August 31, 2011 10:00 am, Eastern Daylight Time (New York, GMT-04:00)
Wednesday, August 31, 2011 2:00 pm, Greenwich Time (Reykjavik, GMT)
Wednesday, August 31, 2011 4:00 pm, Europe Summer Time (Paris, GMT+02:00)
Thursday, September 1, 2011 12:00 am, Australia Eastern Standard Time (Sydney, GMT+10:00)

Event number: 712 776 191
Event password: opensource

——————————————————-
To join the online event
——————————————————-
1. Click here to join the online event.
Or copy and paste the following link to a browser:

https://pentaho.webex.com/pentaho/onstage/g.php?d=712776191&t=a&EA=dev.pentaho%40pentaho.org&ET=4b3a4deb21d439ad703449b420e9bb36&ETR=de22fea937a9b0976b3a1b466f7ab631&RT=MiMxMQ==&p

2. Click “Join Now”.

——————————————————-
To join the teleconference only
——————————————————-
Call-in toll-free number (US/Canada): 1-866-699-3239
Call-in toll number (US/Canada): 1-408-792-6300
Global call-in numbers: https://pentaho.webex.com/pentaho/globalcallin.php?serviceType=EC&ED=161656302&tollFree=1
Toll-free dialing restrictions: http://www.webex.com/pdf/tollfree_restrictions.pdf
Access code: 712 776 191

The sin in open source comes from contributing neither one’s time nor money

He makes the bold statement that using open source software without contributing to that project is a sin. I’m not misquoting him, he says its a sin.

He goes on to say that if you use JasperSoft community edition you are required to contribute.

Your contribution is required.

This is called quid-pro-quo – they does something for you and you must do something for them. In some parts of the planet, certain forms of quid-pro-quo are classified as harassment and are illegal. This attitude towards community also rates very low on the openness scale – Gentile is only open to community members who are going to contribute  directly, and within a time-frame that Jaspersoft finds agreeable. I’m not sure the OSI will approve that license.

Obviously I disagree with his statements. Completely. Utterly. On multiple fronts.

Self-Interest

Firstly, open source works best when all participants act purely out of self-interest, and by doing so, the other participants benefit as a side effect. Here are some examples:

• Reduce Future Work: I have contributed bug fixes to Apache foundation and JBoss projects. I did this not because I feel an obligation to do so, but because I want them to maintain the patch in future versions so I don’t have to continually merge my change with each new version. I contribute my bug fixes purely out of self-interest – I want to reduce my future workload. I also didn’t go looking for those bugs. I came across those bugs while trying to use their open source software to solve a real problem or requirement I faced. By contributing my fixes out of self-interested all the other participants gain higher quality software.

• Education and Career: In prior decades it was not easy to show potential employers your prior work – because often all of it was the proprietary intellectual property of your previous employers. Today, if you are a software engineer, technical writer, graphic artist etc, you can contribute to open source projects as a way of demonstrating your work. This is particularly true of certain open source projects – if you have contributed code to the Linux kernel your lifetime earning potential is probably higher as a result. If you are a software engineer and you want to work at RedHat, your application for a position will considered significantly more if you are active in their community.

• Getting It Done: Community members don’t post questions on forums for fun. They do it because they have a problem of some kind, and it is stopping them from getting something done. Maybe their question get answered, maybe it doesn’t. Either way the question is a contribution – often a question it identifies a problem with the installer, or the documentation, or a feature, or the download site, or the expectations set by the web site etc. The asker of the question did not ask it in order to gently identify an opportunity for improvement, they did it out of self-interest, but it still counts as a contribution.

• Status and Satisfaction: It is satisfying to contribute in a community. There is also status and reputation to be gained. While this form of reward is not monetary, career-driven, or externally noticeable, is a benefit that is derived by the participant. If the community member did not feel this satisfaction, they would not waste their time contributing. If there is a particular application area you are interested in (music, graphics, gaming, mobile devices etc), but there are no career opportunities open to you, you can participate in an open source project to satisfy your interest/curiosity.

• Business: Some companies sponsor projects by encouraging (sometimes requiring) their employees to contribute to open source projects. In some cases these companies do this because they are services companies that want to demonstrate expertise with certain technologies to help attract new customers. The company gains attention and reputation, and in doing so the open source project and community benefit from the time and resources (selfishly) donated. Other companies sponsor their employees to contribute because the company relies on the software operationally. To reduce their risk of failures and downtime, the company needs to have engineers on staff who have deep technical knowledge of the software.

• Changing Market Dynamics: The act of choosing to use open source software instead of a proprietary alternative changes the market dynamics in favor of commercial open source companies. Just using the software is a contribution. Sometimes those users tell other people about the software they are using, and encourage them to use it. The community grows, and the market changes more. Over time this dynamic and the continual community growth help all the participants.

As you can see there are many varied reasons people contribute to open source. In all these cases the participants are acting out of self-interest, but the community as a whole benefits from these selfish actions. But guilt and obligation are not on this list – they are very weak motivations.

With a commercial open source software (COSS) project you have a company that is the single most important participating organization. The motivations of this company are very important. It does not matter which company you look at, whether it’s RedHat, MySQL (Oracle), Pentaho, or Jaspersoft, that company exists for self-interest – to make money. There is nothing wrong with this, it’s actually very healthy. If the stewarding company is acting out of self-interest, you should not expect the community to act differently.

Monetizing the Community

Secondly Gentile is making an elementary mistake, one that is common among those that are new to the COSS business model. Gentile states ”Open source communities thrive based on the community members donating either their time and/or money”. Within the COSS world getting community members to buy software and services is called ‘Monetizing the Community’. It is a clumsy term that encourages misguided actions. I refer to my Beekeeper model. In there I state:

Customers are corporations, the community are people. They have very different needs.

The Beekeeper model makes it clear that it is pointless for a COSS company to try to sell it’s enterprise software and services to community members. This statement is true for all COSS companies that focus on business software (it is not true for open source consumer software such as games). This distinction is important, and it is missing from Gentile’s desperate plea. I’m guessing that only a very small percentage of the Jaspersoft installations are for personal use. Reporting software just isn’t that much fun. The majority of Jaspersoft’s installations are for business purposes. The individual who installed the software is usually a technical end user or an IT developer. Either way, the individual is not going to buy the software for their own needs – if a purchase of software is made, the customer will be the employer, not the community member. As a COSS company you can provide tools for your community members to persuade their employers to become customers, and you can explain how this benefits both companies involved and the community. For most COSS companies is it impossible to monetize the community directly, and therefore ridiculous to try. Trying to berate community members into purchasing business software shows a remarkable lack of understanding of the commercial open source business model. Asking, begging, or requiring community members to purchase your software is futile, Gentile.

I’m not pretending that the COSS business model is easy. I’m not pretending that Pentaho does not have room for improvement either. I am stating that Pentaho provides more BI functionality in open source than Jaspersoft, and that you are welcome to use that software – sin-free and guilt-free.

How refreshing, then, to have olap4j reach the 1.0 milestone – http://www.olap4j.org. Created by consumers and producers of open source BI software, olap4j shows the advantage of open collaboration by motivated parties. Already olap4j has a Mondrian driver, and an XMLA driver for Microsoft SQL Server Analysis Services, SAP BW, and Jedox Palo. There are also several clients who use olap4j servers, some from Pentaho, and Saiku, Wabit, and ADANS.

olap4j is very cool stuff. You can read more on Julian Hyde’s blog. Congratulations for everyone that has worked on olap4j.

Then I had a lengthy and interesting chat with Dan Woods. Amongst other things Dan runs http://www.citoresearch.com/ and also blogs for Forbes. We talked about Pentaho’s history and our experiences so far with the commercial open source model.   We also talked about Hadoop and big data and about the vision and roadmap of our Agile BI offering.

Next I met with Steve Lohr who is a technology reporter for the New York Times. We talked about many topics including the enterprise software markets and how open source is affecting them. We also talked about Hadoop, of course.

Next was a co-meet-up of the New York Predictive Analytics and No-SQL groups where I presented decks about Weka and Hadoop, separately and together. There were lots of interesting questions and side discussions earlier. By the time we finished all these topics a blizzard was going on out side. Cabs were nowhere to be seen so Matt Gershoff of Conductrics was kind enough to lead me via the subway to the vicinity of my hotel.

Being the first Business Intelligence vendor to bring reporting and ETL to the Hadoop space sets us apart from all the other vendors. We have so much recognition in this space that I’ve spoken to a few people in the last month who thought we were ‘THE’ visualization and data transformation provider for Hadoop and didn’t connect to other data sources.

This afternoon I’m meeting with reporters and columnists from a couple of different publications to chat about Big Data / Hadoop stuff. Tonight I’m presenting at the New York Predictive Analytics Meetup to talk about Hadoop from an analysis perspective.

January 19th and 20th is our Global 2011 Summit in San Francisco. I have three sessions Pentaho for Hadoop, Extending Pentaho’s Capabilities, and an Architecture Overview. So I’m creating and digging up some new sample plug-ins and extensions. I’m also going to take part in an Q&A session with the Penaho architects since Julian Hyde (Mondrian), Matt Casters (PDI/Kettle), Thomas Morgner (Pentaho Reporting) will all be there. Who should attend?

CTOs, architects, product managers, business executives and partner-facing staff from System Integrators and Resellers, as well as Software Providers with a need to embed business intelligence or data integration software into your products.

We usually have customers and prospects attending our summits as well.

We are also having an architect’s summit that same week to work on our 2011 technology road-map. That should be a lot of fun.

The week after that I’ll be in New York presenting at the NYC Hadoop User Group on Tuesday, January 25 and the NYC Predictive Analytics Meetup on Wednesday January 26th.

An analysis by country of these numbers shows interesting results.

The Long Tail

This chart shows the number of new installations year-to-date for each country. Our data shows new Pentaho installations in 176 countries so far this year. That’s out of a total of 229 countries.

This is clearly a classic long tail. In fact after the first 20 or 30 countries it is difficult to read values from the chart. This second chart uses a log scale. The line on this chart is almost perfectly linear, showing that the distribution by country is pretty much logarithmic.

Over the same time period Pentaho has customers in 46 countries. This is a larger geographic spread than most of the proprietary BI companies.

Since we are dealing with country-based data, here is the analysis I did using Google Geo Map, Pentaho Data Integration, and Pentaho BI Server.

New Pentaho Installs Jan-Oct 2010

This shows the geographic spread of the installations.

It is fairly obvious from the map above that the highest number of installations were in the USA, China, and Brazil, followed by India and parts of Europe. But this simplistic graphic does take into account the economics or demographics of the countries. How does the number of installations relate to the size or economic power of each country?

New Installations Per $Billion GDP

If we look at the number of new installations of Pentaho software per billion dollars of GDP we see a different picture. The GDP data is from the CIA World Factbook

I capped ‘Installs Per Bn GDP’ at 10 to prevent outliers from skewing the color gradient.

Compared with the first map the prominence of the USA and Chine is reduced, and the areas of high activity are shown to be South America followed by Europe and parts of Asia. But analysis using GDP alone does not take into account things like exchange rates and the cost-of-living within a country – as a result there is probably a bias towards countries like the South American ones. So I went to find metrics that should remove bias of economic factors.

New Installations Per 100k Labor Force

If we look at the number of new Pentaho installs compared with the labor force of each country we  get a slightly different picture. The labor force data is from the CIA World Factbook

I capped ‘Installs Per 100k Labor Force’ at 50 to prevent outliers from skewing the color gradient.

Compared with the first two maps, this one shows the South America, Europe, and North America countries roughly equal to each other. Australia and New Zealand are also comparable. Asia, Africa, and the Middle East are shown to be generally behind. What is odd about this graphic is that countries like India, generally considered to be significant open source consumers, are not shown to be within the leading countries. This is because, I’m assuming, that a large percentage of the labor force is agricultural, and as such, less likely to be doing much BI.

New Installations Per 100k Internet Users

So instead of labor force, let’s look at new installations of Pentaho for every 100k internet users within a country.

I capped ‘Installs Per 100k Internet Users’ at 50 to prevent outliers from skewing the color gradient.

Here we see that South America is still prominent, along with southern Europe. The rest of Europe and North America come second along with India, other parts of Asia and Australia. South Africa also makes a showing for the first time. China however does not show strongly.

This metric – Installations per 100k Internet Users – seems like a reasonable way to compare the adoption of software between countries. ‘Internet Users’, by definition, have access to a computer (needed to run FOSS) and to the internet (needed to get FOSS). This metric is not skewed by the percentage of the population that are not internet users, and is not skewed by cost-of-living or exchange rates.

Here are the top 40 countries for new Pentaho installations per 100k internet users ( for countries with over 1 million internet users).

There is a bias still. Countries with a lower percentage of internet users in the total population will be rated higher than those with very high percentage. This is because in the first case, the individuals with internet access will tend to be those in business, i.e. those with a higher than normal need for BI tools. Whereas in the second case the internet users include relatively more families and individuals – those with a lower need for BI tools. This bias would not affect the installation figures of software such as Firefox, but would affect the ratings in Pentaho’s case.

If we group the countries into regions we see some other interesting things. This scatter chart shows number internet users on the X axis and new installations of Pentaho software on the Y axis.

Interestingly the USA, South America, and Asia come out with around the same total number of installations (approx 30,000), but the chart shows a large difference (100m up to 550m) in the number of internet users within those regions. Europe, as a region, has the highest number of new installations, with a 50% margin over the second place region.

So which metric do you think is most valuable? And for what purpose?

Also interesting to note is that the 2010 installations numbers represent, for each country, 40-50% of the all-time (2007-2010) installation figures. This means that the number of new installations so far in 2010 is about the same as the number of installations in the previous 3 years combined.

About This Analysis

And yes, I used Pentaho software to do this analysis – I used Pentaho’s Agile BI process.

• Iteration 1: I first loaded the ‘new installations’ data into a table to do the histograms and the first map. After seeing the map, it occurred to me that just looking at the installation figures was not very interesting, and that comparing installations to GDP might be better.
• Iteration 2: So I went to find GDP data and added it to the table using Pentaho Data Integration. After seeing the ‘Pentaho Installs Per $dn GDP’  map it occurred to me that other metrics might show different, and better, results – so I went to find other data sets, not knowing what I might find.
• Iteration 3: At the CIA World Factbook I found Labor Force and Internet Users. I added these to the table and looked at the maps. At this point I decided that comparing installation counts with the total number of internet users in a country was a good metric.

It took three iterations of finding data, merge/calculate/load, and visualization before I settled on an analysis I thought was optimal. The important point here is that until I saw the data visualized, the next question did not occur to me, so a one-pass ‘requirements’ -> ‘design’ -> ‘implement’ -> ‘visualize’ process would not have worked.

For example: the State Department sends Google bad border data – and Costa Rica gets invaded.

http://searchengineland.com/nicaragua-raids-costa-rica-blames-google-maps-54885

http://edition.cnn.com/2010/TECH/web/11/05/nicaragua.raid.google.maps/

That’s my new favorite data quality anecdote

It’s not a good demonstration of their product capabilities if the king of SaaS Customer Relationship Management can’t internally identify who their customer are. #Fail

SalesForce needs a better CRM system. Any suggestions?