Freekshow

December 14, 2010

The scientific method

Filed under: Community,Process,Reading — Freek Leemhuis @ 11:42 pm

The ‘scientific method’ – or is it?

I like the work of David Anderson on Kanban, which I think is deserving of the attention it is now getting in the software development industry. His latest post describes the basic principles well. I noticed he and others in the field describe the use of the ‘scientific model’ as approach to improve one’s process. In this post I would like to examine the use of this term, because I think there’s a lot of misunderstanding about scientific methods and how they can be applied to software projects.
In David’s post it is described as follows:

The use of models allows a team to make a prediction about the affect of a change (or intervention). After the change is implemented the outcome can be observed by measuring the flow and examining the data. The outcome can be compared to the prediction expected from the model and the change can be assessed as an improvement, or not. This process of evaluating an empirical observation with a model, suggesting an intervention and predicting the outcome based on the model, then observing what really happens and comparing with the prediction, is use of the scientific method in its fundamental sense. This scientific approach, is I believe, more likely to lead to learning at both the individual and organizational level. Hence the use of the scientific approach in Kanban will lead directly to the emergence of learning organizations.

It makes a lot of sense, change a variable and then measure the difference it makes. I do believe however that some scientific methods are overlooked in this type of description. I’ll try to explain.

Suppose you’ve run a project and you have gathered all kinds of metrics regarding performance of your team. The data shows that since this one bald developer was introduced in an otherwise hairy developer team the velocity increased. This leads you to the hypothesis that having at least one bald developer on the team leads to better team performance. In order to test this hypothesis, you will introduce a bald programmer to another team, and again you collect empirical evidence to see if your hypothesis is confirmed. Again, the numbers show an increase of productivity. You’ve followed ‘the scientific method’, so you can now state firmly that

bald programmers lead to better team productivity.

And you have numbers to back it up.

The example is deliberately silly, you will immediately think of alternative explanations. Quite often, bald developers are of advanced age and therefore on average more senior and experienced developers. There’s all sorts of alternative explanations that would, when true, show the correlation between performance and amount of hair to be spurious. That is one problem that is not easily solved, there can always be an underlying correlation that is the true predictor for the result.
However, the outcome can also be influenced by other factors, and these can be controlled by using methods that are ‘more scientific’.

Let’s talk about two types of controls that are common in social scientific research: the use of control groups and sample distribution.

control groups

In the social sciences, experiments to gather data to test a hypothesis are executed in controlled experiments. Usually, to test a hypothesis a researcher use two different (sets of) subjects simultaneous: a treatment group, for which the effect of an manipulation is observed, and a control group, which uses all of the same conditions as the first with the exception of the actual manipulation. The effect of the manipulation can then be measured by comparing the results of both groups. If you do not use a control group, the data that you so carefully collect might be influenced by factors outside of your manipulation. Increasing performance might for example be in fact due to the fact that the team was working on easier parts of the system for that time, or the bout of flue that was going around and kept important team members from contributing was over by that time. Using a control group will eliminate all factors related to the time of your experiment from influencing the outcome of your manipulation. Well established misinterpretations such as the Hawthorne effect can thus be avoided.

Sample characteristics

Secondly, you’re looking for findings that are applicable not just for the team in question but somewhat broader. But can you say that findings for the one team are expected to apply to all teams in your organization? To do so you need a sample (the team) that is representative for the population (all teams) that you want to target.
In science the sample size and distribution of these characteristics are used to measure the probability that the effect of manipulation is in fact valid for the population. If you want to measure the popularity of Ajax football club in the whole of Holland for example, you would be wise to include people from different age, sex, from different parts of the country, with different incomes and so on. Obviously, the closer the sample size is to the population, the higher the probability that the findings will apply to the population as a whole. We know teams can vary greatly. If you add the bald developer to a team of java developers, the effect might be totally different then adding him to a team of php developers. Adding a baldy to a new team might increase productivity, adding him to a well-established team however could be counter-productive.
There’s all sorts of team attributes that might be important to the effect of manipulation. One should look carefully at the attributes of a team in order to decide if the sample is representative.

Finally

I don’t think there is such a thing as one ‘scientific method’. There’s different methods, one more scientific than the other, that can lead to a measurement of probability that a hypothesis is true or false. And sure, in practice it will not always be feasible to improve our process in a rigorous scientific manner. I do think for best results we should at least understand and communicate these factors and account for them where possible.

Advertisements

October 15, 2009

On the nature of community driven events

Filed under: Community,Devnology,Events,Learning,Speaking — Freek Leemhuis @ 5:45 pm

 
 

lowscore Last week I was involved in organizing a Scala workshop, and from the feedback we received it was notable that some of the issues that were mentioned were more to do with the expectation of participants than with the actual content.The average rating from the evaluation for this event was 7.3 (on a scale from 1 to 10), and while that might seem decent it is dramatically lower then scores for our previous events.Some of the feedback that was provided was very fair. For example, it was mentioned that the tempo could have been higher, and it could have. The typical participants of Devnology meetings have way more smarts and passion about them than any other group I have participated in, so we need to become better at tailoring our meetings accordingly.

Another point of criticism we received was that the facilitators did not seem to know all the ins and outs of the subject matter. This in my opinion is a result of the position we take in organizing these types of events: there should not be one expert explaining to the rest everything there is to know, but rather a situation where there’s enough preparation done to go through the material, and in the process learn from each other.

Under these conditions, chances are a lot bigger that it will be an interactive event, with active participation.

This way, you get to know the attendees, learn what knowledge and experience they can contribute, possibly even beyond the event itself. The meetings will become more lively and interesting, and more in line with the ideas behind the Unconference.

Having an expert capable of explaining everything on a particular subject may not improve the experience, expecially if you lose a level of interaction. We will therefore continue to encourage members of the community to take on a subject and create a learning events that is geared toward learning together.

Obviously, this will only be a success if the more knowledgeable participants are aware of this and willing to share their skills and ideas. If you are aware of points being missed or if you have information or skills that can contribute, one way to react it to sit back and be bored. It might however be better to ask yourself this question: what have I done to improve the experience of this event for some (or all) of the other participants?

Let me paraphrase what this guy said:

Ask not what your community can do for you, ask what you can do for your community!

I don’t mean to sound obnoxious here and I certainly don’t want to put people off from attending. What I hope to achieve with this post is to make our viewpoints regarding these issues clear and hopefully prevent misunderstanding in the future.

jfk

 

Please feel free to respond!

July 17, 2009

DDD Course… coming to a theatre near you very soon!

Filed under: Community,Domain Driven Design,Events — Freek Leemhuis @ 7:23 am

There’s a great opportunity to immerse yourself for 2 days in Domain Driven Design. On the 14th and 15th of September Gregory Young will speak about the essentials of applying DDD concepts in the design and implementation of your application. This is a very practical, hands-on event. Places are limited, so sign up now at http://www.jworks.nl/

You can see Greg in action in this recent presentation at QCon

April 10, 2009

More DDD

Filed under: Community,Domain Driven Design,Learning,Speaking — Freek Leemhuis @ 4:12 pm

The second part of the DDD workshop contained a modelling session (modelling out loud!), which I think helped a lot explaining some of the concepts.

picture-030

I’ve attached the slides for the second day below.

ddd-day2 slide deck

April 8, 2009

Spreading the word

Filed under: Community,Domain Driven Design,Speaking — Freek Leemhuis @ 7:33 am

Eindhoven, 7th of april 2009, Workshop Domain Driven Design.

ddd_eindhoven

 

Smiles all around yesterday as each participant received a copy of the DDD bible. I think a few heads were spinning toward the end of the evening trying to take it all in. It is hard to cram a lot of the ideas in just a few hours, especially if there’s good discussion.

The slide deck can be found here, and the assignment here.

Second part on thursday, see you all there!

April 2, 2009

Code Fest 01

Filed under: Community,Devnology — Freek Leemhuis @ 4:00 pm

Some night that was! I was well impressed with the people that showed up, discussion and banter flowed between them based on a shared passion for technology and interest in other people’s approaches.

codefest1

On this first Code Fest (all in Dutch, apologies) we had implementations of that old classic, James Conway’s Game of Life. On the night we had demonstrations of implementations in Flex, Erlang, Haskell, C++, Alloy, Java, CosMos and the Google App Engine. It certainly had my head spinning, and with what I thought was a cracking atmosphere we couldn’t have asked for more. We have invested a lot of time and energy in getting this thing of the ground, but yesterday evening alone was worth every bit of it.

Thanks all for a great evening.

On a sidenote, I was playing with this domain model to describe Devnology, and thought I’d share it here:

model

 

Geek as I am, it helps me focus and explain what I think we can achieve : Building a technology agnostic community by organising events that have high interaction between passionate developers. Enable interaction between academia and the software development industrie (software craftsmen, if you like).

With result like yesterday and the software testing event at TU Delft things are shaping up. Come and join us at our next meeting!

February 20, 2009

Devnology!

Filed under: Community,Devnology — Freek Leemhuis @ 9:24 am

The cat is out of the bag! It’s been a long time in the making, but finally…. coming to a theatre near you very soon… It’s…..devnology

Our first event is planned for the 1st of april (no joke, I promise) and is now open for registration. Do come!

Blog at WordPress.com.