posted on Friday, February 22, 2008 6:46 PM by AnjaliSastry

Will your project have the effect you want it to?

NOTE THIS IS A DRAFT POST TO BE UPDATED

How do you make sure that your work is effective? First and foremost, you need to get stuff done. Designing and using a good workplan is key, of course. Last week’s post offers some ideas to get you started on workplanning. For instance, you need to sort out what it is you want to have finished by the end of the project (the awkwardly-named “deliverables”) as well as the time, effort, data, and other resources needed to produce these things.

But working effectively also requires you to address some big questions about the effect of your efforts.

For this post, I want to introduce some ideas that I think could help you sort out two linked questions that get at effectiveness: What’s our underlying model for how this project will actually work, and how will we know if we’re right?

If your project is a real one—and a fun one!—your answers to these questions will evolve. Our starting point for addressing the questions is a set of structured approaches called theory of change or logic models or, sometimes, evaluation models

Theory of Change evaluations make the team’s theories of change explicit so that the work can best address the causal model that undergirds the project’s plans and goals. A good theory of change includes working versions of:

  • situation and stakeholder analyses
  • actions taken
  • anticipated outcomes

The reason to make it all explicit is to set up the team to seek evidence as to whether the theory is matched by reality.

Example: boys and girls club http://www.evaluationtools.org/plan_theory.asp

introductory instructions: http://www.uwex.edu/ces/pdande/evaluation/pdf/LMinstructions.pdf

source: http://www.uwex.edu/ces/pdande/evaluation/evallogicmodelworksheets.html

for more please go to
http://www.uwex.edu/ces/pdande/evaluation/evallogicmodel.html

tutorial: http://www.uwex.edu/ces/lmcourse/#

Other benefits of a theory-of-change approach, or why you should address these questions (now and again!):

Where outcomes are expected only in the long term – perhaps after the evaluation is complete - they give early indication as to whether predicted changes are happening and therefore whether the intended outcomes are likely to emerge in due course.

They are able to trace complex links between action and outcome, so that the problem of attribution is diminished.

The process of explicating leaders’ theories of change can be helpful in planning the initiative with greater clarity.

They also provide leaders with early feedback as to the effects of their actions, making it possible for those actions to be modified at an early stage and linking the evaluation process closely with the development of the initiative.

Qualitative data (generated from interviews with a wide range of stakeholders) and quantitative data are both useful for the evaluation process.

The expectation is that, by the end of the evaluation process, it should be possible not only to articulate the theory (and any ways in which, by then, it has changed or has been contested by other stakeholders) but also to present convincing evidence of the sorts of changes that are being brought about by the project and to predict the sorts of long-term outcomes which are likely to emerge.

More ideas?

 

Interesting to link this to your first cut at the rationale for your project—recall our discussion of your projects via three simple questions from Week 2:

  • What’s the problem?
  • What’s your solution?
  • How does it work?

 

If you go through the processes outlined in the resources linked to here, you will have a theory of change. What next?

answer #1: talk this over with your team

AND (answer #2):

How are you testing this theory as you go? Do your experiences and the totality of your emerging set of data fit with it? Do you need to revisit your theory of change?

links:

http://www.wkkf.org/Pubs/Tools/Evaluation/Pub3669.pdf

http://ctb.ku.edu/tools/en/section_1877.htm

http://www.innonet.org/client_docs/File/logic_model_workbook.pdf

http://www.theoryofchange.org/index.html

http://www.aecf.org/upload/PublicationFiles/CC2977K440.pdf

http://www.uwex.edu/ces/pdande/evaluation/evallogicbiblio.html for a bibliography, including research

Comments

# re: Will your project have the effect you want it to?

Thursday, February 28, 2008 2:29 PM by Dwane Morgan
So far, we have been testing our theory with research. There is a lot of readily available research online about both Marathoning and the BAA. We were able to determine that the BAA is currently a very healthy organization. They have experienced asset growth at a constant rate of 15% per year over the past five years. We believe that this is contributing to their complacency. We also found statistics showing that the 30 largest Marathons in the US (Boston ranks #3) accounted for 71% of marathon finishers in 2006, but just 57% of the finishers in 2007. That confirms our next hypothesis which is that the BAA will not be able to keep their competitive advantage forever with their current race model. Communicating this delicately will be key to making strides with this organization.

# re: Will your project have the effect you want it to?

Tuesday, March 04, 2008 7:31 PM by Kyle
When thinking about past projects in the context of this class, I’m often struck by the number of “frameworks” a team can choose with which to work. I feel damned, doubly damned, actually, because I have a bachelor’s and master’s degree in engineering and I operate within a different type of “framework” that is commonly referred to as the scientific method. There are many similarities between the scientific method and business school “frameworks”, but the scientific method values failures just as much as successes. A failure in the scientific method is when the results of test do not match the hypothesis – but steps are taken to determine why and knowledge is gained.

When I think of business in the context of frameworks and the scientific method, two stories come to mind. One is of Archimedes and King Heiros’s crown: Archimedes exclaimed, “Eureka!” when he discovered how to determine if the king’s crown was made of real gold. The other story is of well-known scientist and author Isaac Asimov, who is quoted, “The most exciting phrase to hear in science, the one that heralds new discoveries, is not ‘Eureka!’ but ‘That’s funny ….”.

I have not worked with consultants much (and I don’t whether I want to insert fortunately or unfortunately here). What I have witnessed, though, is that consultants like to say “Eureka” – here is your answer, here is your gold(en crown). And that’s all well and good – as long as the problem being solved is well-defined. But what if the problem is an open-ended question, as I imagine most to be in business? In that case, I think the scientific method, forming a hypothesis, testing that hypothesis, examining the results, and constantly questioning the outcomes and assumptions to ensure the greatest robustness, and being able to say, “That’s funny …”, is so much more valuable.

I guess another way of saying the same thing as above is to say that the approach of “answer first” is extraordinarily difficult for me to get my head around. At this point, someone could easily argue that answer first and form a hypothesis are the same thing – but a hypothesis is tested, reformulated, tested again, reformulated, tested again, etc. In my experience, the first answer in the answer first approach is never tested or reformulated. How can anyone argue that is a robust methodology?

Perhaps this is my personal dilemma with which no one else grapples. But I cannot imagine a day when I would rather take an untested theory and stick with it to solve a problem than work and rework a hypothesis until I have a robust solution in which I have complete faith, and whole lot of data to support my recommendation.

# re: Will your project have the effect you want it to?

Tuesday, March 04, 2008 7:32 PM by Kyle
When thinking about past projects in the context of this class, I’m often struck by the number of “frameworks” a team can choose with which to work. I feel damned, doubly damned, actually, because I have a bachelor’s and master’s degree in engineering and I operate within a different type of “framework” that is commonly referred to as the scientific method. There are many similarities between the scientific method and business school “frameworks”, but the scientific method values failures just as much as successes. A failure in the scientific method is when the results of test do not match the hypothesis – but steps are taken to determine why and knowledge is gained.

When I think of business in the context of frameworks and the scientific method, two stories come to mind. One is of Archimedes and King Heiros’s crown: Archimedes exclaimed, “Eureka!” when he discovered how to determine if the king’s crown was made of real gold. The other story is of well-known scientist and author Isaac Asimov, who is quoted, “The most exciting phrase to hear in science, the one that heralds new discoveries, is not ‘Eureka!’ but ‘That’s funny ….”.

I have not worked with consultants much (and I don’t whether I want to insert fortunately or unfortunately here). What I have witnessed, though, is that consultants like to say “Eureka” – here is your answer, here is your gold(en crown). And that’s all well and good – as long as the problem being solved is well-defined. But what if the problem is an open-ended question, as I imagine most to be in business? In that case, I think the scientific method, forming a hypothesis, testing that hypothesis, examining the results, and constantly questioning the outcomes and assumptions to ensure the greatest robustness, and being able to say, “That’s funny …”, is so much more valuable.

I guess another way of saying the same thing as above is to say that the approach of “answer first” is extraordinarily difficult for me to get my head around. At this point, someone could easily argue that answer first and form a hypothesis are the same thing – but a hypothesis is tested, reformulated, tested again, reformulated, tested again, etc. In my experience, the first answer in the answer first approach is never tested or reformulated. How can anyone argue that is a robust methodology?

Perhaps this is my personal dilemma with which no one else grapples. But I cannot imagine a day when I would rather take an untested theory and stick with it to solve a problem than work and rework a hypothesis until I have a robust solution in which I have complete faith, and whole lot of data to support my recommendation.