Freekshow

December 14, 2010

The scientific method

Filed under: Community,Process,Reading — Freek Leemhuis @ 11:42 pm

The ‘scientific method’ – or is it?

I like the work of David Anderson on Kanban, which I think is deserving of the attention it is now getting in the software development industry. His latest post describes the basic principles well. I noticed he and others in the field describe the use of the ‘scientific model’ as approach to improve one’s process. In this post I would like to examine the use of this term, because I think there’s a lot of misunderstanding about scientific methods and how they can be applied to software projects.
In David’s post it is described as follows:

The use of models allows a team to make a prediction about the affect of a change (or intervention). After the change is implemented the outcome can be observed by measuring the flow and examining the data. The outcome can be compared to the prediction expected from the model and the change can be assessed as an improvement, or not. This process of evaluating an empirical observation with a model, suggesting an intervention and predicting the outcome based on the model, then observing what really happens and comparing with the prediction, is use of the scientific method in its fundamental sense. This scientific approach, is I believe, more likely to lead to learning at both the individual and organizational level. Hence the use of the scientific approach in Kanban will lead directly to the emergence of learning organizations.

It makes a lot of sense, change a variable and then measure the difference it makes. I do believe however that some scientific methods are overlooked in this type of description. I’ll try to explain.

Suppose you’ve run a project and you have gathered all kinds of metrics regarding performance of your team. The data shows that since this one bald developer was introduced in an otherwise hairy developer team the velocity increased. This leads you to the hypothesis that having at least one bald developer on the team leads to better team performance. In order to test this hypothesis, you will introduce a bald programmer to another team, and again you collect empirical evidence to see if your hypothesis is confirmed. Again, the numbers show an increase of productivity. You’ve followed ‘the scientific method’, so you can now state firmly that

bald programmers lead to better team productivity.

And you have numbers to back it up.

The example is deliberately silly, you will immediately think of alternative explanations. Quite often, bald developers are of advanced age and therefore on average more senior and experienced developers. There’s all sorts of alternative explanations that would, when true, show the correlation between performance and amount of hair to be spurious. That is one problem that is not easily solved, there can always be an underlying correlation that is the true predictor for the result.
However, the outcome can also be influenced by other factors, and these can be controlled by using methods that are ‘more scientific’.

Let’s talk about two types of controls that are common in social scientific research: the use of control groups and sample distribution.

control groups

In the social sciences, experiments to gather data to test a hypothesis are executed in controlled experiments. Usually, to test a hypothesis a researcher use two different (sets of) subjects simultaneous: a treatment group, for which the effect of an manipulation is observed, and a control group, which uses all of the same conditions as the first with the exception of the actual manipulation. The effect of the manipulation can then be measured by comparing the results of both groups. If you do not use a control group, the data that you so carefully collect might be influenced by factors outside of your manipulation. Increasing performance might for example be in fact due to the fact that the team was working on easier parts of the system for that time, or the bout of flue that was going around and kept important team members from contributing was over by that time. Using a control group will eliminate all factors related to the time of your experiment from influencing the outcome of your manipulation. Well established misinterpretations such as the Hawthorne effect can thus be avoided.

Sample characteristics

Secondly, you’re looking for findings that are applicable not just for the team in question but somewhat broader. But can you say that findings for the one team are expected to apply to all teams in your organization? To do so you need a sample (the team) that is representative for the population (all teams) that you want to target.
In science the sample size and distribution of these characteristics are used to measure the probability that the effect of manipulation is in fact valid for the population. If you want to measure the popularity of Ajax football club in the whole of Holland for example, you would be wise to include people from different age, sex, from different parts of the country, with different incomes and so on. Obviously, the closer the sample size is to the population, the higher the probability that the findings will apply to the population as a whole. We know teams can vary greatly. If you add the bald developer to a team of java developers, the effect might be totally different then adding him to a team of php developers. Adding a baldy to a new team might increase productivity, adding him to a well-established team however could be counter-productive.
There’s all sorts of team attributes that might be important to the effect of manipulation. One should look carefully at the attributes of a team in order to decide if the sample is representative.

Finally

I don’t think there is such a thing as one ‘scientific method’. There’s different methods, one more scientific than the other, that can lead to a measurement of probability that a hypothesis is true or false. And sure, in practice it will not always be feasible to improve our process in a rigorous scientific manner. I do think for best results we should at least understand and communicate these factors and account for them where possible.

Advertisements

March 17, 2010

Adrenaline Junkies and Template Zombies

Filed under: Learning,Reading — Freek Leemhuis @ 8:34 pm

I’ve followed Tom DeMarco and Tim Lister ever since I read Peopleware, a great book about building great software teams. When a new book by these authors (and a few others) came out last year, I ordered it straight away. It’s called Adrenaline Junkies and Template Zombies, and it’s a great list of patterns of project behaviour.
It’s a fun read, although I’m sure for some of my colleagues and former colleagues some of the stories are too close for comfort.
I’ll describe why a few of the patterns really resonated with me:

At one of my previous assignment there were a lot of procedures to follow. Using project management procedures like Prince2 really revolved around having the specified documents in place at certain to pass certain milestones. Usually, the templates for these documents have little descriptions here and there where information is required.At the start of a project, one had to fill in an document listing predefined architectural concerns and design decisions for the system to be delivered. In itself not a bad practice, but ofcourse only valuable if you put the right amount of detail and consideration into it. As a test, I filled in the template with completely inane comments to see who would pick up on it. Where it asked: what are the logging and tracing requirements for this application I wrote This application has no specific needs for logging and tracing other than standard.
Under the heading: what security concerns need to be adressed for the information contained in the database I wrote : the information needs to be authorized by the guidelines specified in the technical design (note: there was at that time no technical design document – this was to be delivered only in the next project phase). You get the picture. I filed the document and it passed with flying colours: all boxes where checked. In reality, it amounted to nothing.
This kind of behaviour is what DeMarco et al refer to as Template Zombies. If procedures are more about form than content, be very careful. The Template Zombies are out to get you.
The other pattern that lends it’s name to the book is one I deal with quite a bit: Adrenaline Junkies. You will find these people in organizations where priorities are constantly shifting to the project that is particular urgent at that time. A new project was about to start, and after calculating the manpower required to finish on time we concluded that a 4man team should start right away to have any chance of actually meeting the deadline. However, after confronting the project manager with this information, he looked at us stunned, and responded: `are you kidding me? There are 10 more projects that we need to work on before we start that one!’
And so we did not start until it was too late, and ofcourse the project was late like any other project they were doing.
This is the thing when you are dealing with Adraneline Junkies: they react rather than consider, and there is no long-time planning, and generally they confuse being very busy with being very productive. There is generally a lot of burnout and turnover at companies that follow this pattern. Junkie behaviour does not scale, and companies that fail to recognize it will struggle to make a long term impact.
Mind you, the book does not really go into detail on how to handle the situations it describes. It merely gives a name to patterns that are, unfortunately, rife in our industrie. When dealing with Adraneline Junkies I have made my mind up to avoid in particular one other pattern mentioned by DeMarco: I will not be a Film Critic. Go ahead and look that one up, and many other similarly apt named patterns.

August 18, 2008

JP’s contest

Filed under: .Net,Reading — Freek Leemhuis @ 9:47 am

Jean-Paul Boodhoo is an well-known authority on agily methodologies and patterns in the .Net community. He is author of many published articles, and I have tremendously enjoyed the series of webcasts he has produced for DNRtv on Demystifying Design Patterns.

JP also organizes the Nothing but .Net training bonanza’s, and reading the description of these bootcamp style events have always had me thinking of ways to be able to sign up for one of them. So when he opened a competition for submitting stories that would foster the passion in developers I put in a few words. And lo and behold, I made it in the top 5! So now you can vote for me or any of the other contestants to give your favorite author the chance to enjoy a great week of training.

UPDATE:

So I did not win the top price, so I guess I need to persuade my company to stump up the money for the training course! I did come in third, which JP is generous enough to award with a 130 dollar Amazon voucher. I’m planning to spend it on this set of books:

Feather’s book has had some rave reviews, and I’m curious what tips and practices it offers, especially since I will be working on a lot of legacy code in the near future it seams…
This book apparantly has set a new standard in the publishing of programming books. It’s got syntax code coloring! And it’s a very good book on WPF, which I want to dive into a bit more
McConnel is my favorite author, and this is one of his books I have yet to get my hands on
Uncle Bob is another one of my favorites, and this brand new tome will no doubt contain many a pearl of wisdom. Dispite the ugly cover.

Thanks to JP for stacking up my reading list!

April 5, 2008

Come see Jimmy Nilsson

Filed under: .Net,Domain Driven Design,Events,Reading — Freek Leemhuis @ 7:43 pm
Tags:

Great news: on 24th of april, the DotNed usergroup will host another top speaker and Domain Driven Design guru Jimmy Nilsson. Keep an eye on their website for the anouncement. Found out more about Jimmy on his blog. I highly recommend his excellent book Applying Domain Driven Design and Patterns.

March 29, 2008

Reading: The Change Function

Filed under: Reading — Freek Leemhuis @ 4:29 pm

changefunctioncover.jpg

 If you build it, they will come.

This quote from the rather corny movie ‘Field of Dreams’ is used by author Pip Coburn to illustrate the mindset with which a lot of technology is build. Innovation is often driven by suppliers and not based on what users are after. In his book The Change Function:Why Some Technologies Take Off and Others Crash and Burn,  he addresses two key issues in the technology industrie:

1. High-Tech Failure rates stink

2. Suppliers think they are in charge but in reality users are in charge

Coburn describes the old school thinking that is paramount in the field as

Change = f(Moore’s Law * Grove’s Law)

Former Intel CEO Andy Grove’s mantra suggests that the surest way to success is to focus on creating disruptive technologies that produce order of magnitude, or “10x” changes that alter the landscape. Moore’s Law can be ascribed to anyone suggesting that as the price drops, the market will flourish.

Coburn does not consider this thinking wrong, but argues that since it focuses on the supplier, not on the users, it is incomplete. It would be better to include the user’s perspective, as decribed by his Change Function:

f(user crises vs. total perceived pain of adoption)

This is a much more user-focused : It’s essential the user is in a crisis to motivate him to move to new products. Just telling him he should use this new technology usually won’t work. The total perceived pain of adoption is the user’s estimate of what it will cost him to adopt the technology. This can be monetary value, but usually the bigger components here are time and effort it takes to learn how to use the technologies.

For example, the Blackberry took off because the perceived crisis is high (lack of access to email while traveling), while the total perceived pain of adoption is relatively low (it’s not that expensive, and easy to use). On the other hand, things like interactive TV have been less succesfull then anticipated because of the lack of perceived crisis.

What I like about Coburn’s reasoning is that it’s user-centric. It’s all about the perception of the user.

usersmind.jpg

It’s a good read, with loads of great quotes, although the large number of examples does get a bit tiresome. It is ofcourse very speculative, and I would have liked to see some scientific research results to back it up. You’d think that with all the money going on in the industry it should not be hard to come up with some solid research.

You can download the intro to the books from the Coburn Ventures website here.

Create a free website or blog at WordPress.com.