Firstly, there are a lot of very smart people thinking about these issues – it was thoroughly enjoyable to have to take on so many ideas in such a period of time.

The second thing that stood out was that most of the people at the conference were involved with the development / supplier side of the IT equation. Hardly surprising, but it does mean that some of what was covered was of particular relevance for the “supply side”. However pretty much every topic had things which can be of use from the customer perspective.

Thirdly, a number of people are applying thought to the problems in getting the benefits of Agile in multi-party environments, and how to support the process with appropriate contracts.

It’s in everyone’s interest that there should be consensus on how to reliably deliver software projects, and how to assess the risk of such delivery (and I don’t mean CMMI) – if we can get to that, then there is more opportunity to deploy financial engineering (e.g. Graham Oake’s idea of completion bonds) to facilitate business improvements through software.

There are clear lessons about the importance of getting a common view of risk between project participants.

DSDM/Atern doesn’t seem to have much visibility yet outside the IT world, but looks worthy of further investigation.

Last but not least, Agile is not just for software (nice to see other people have spotted this to!).

Topics for Further Research and/or reflection

Agile Contracts, and the work of Cem Kaner

DSDM/Atern

How could we use a Acceptance-Test-Driven approach from the customer side?

I need to reflect on, document and develop the work we have been doing on using agile approaches for such things as business analysis / programme shaping etc.

Finally, thanks for thought-provoking conversations to Rachel Davies, Graham Oakes and Antony Marcano.

Acceptance Test Driven Development

David Peterson

Summary

• What happens if you put acceptance tests in the
  driving seat?
• Fresh ideas about the agile development process
• Practical techniques to improve your project’s agility
• Emphasis on process and practice (non-technical)

Notes

Whilst working at EasyNet, David modified their normal XP iteration cycle to insert a phase where, for each story, acceptance test criteria were agreed and documented. Alongside this their testing harnesses were adapted to provide autoamted testing of acceptance tests wherever possible.

A key enabler was to separate out test case definition (which depends on the business requirement) from test scripting (which is dependent on, and coupled with, the system under test).

The technique adopted was to build test fixtures which interfaced between the (HTML) test cases and the system under test. This way the test cases can stay unchanged whilst the system changes. If the system changes in a way that breaks the test fixture that shows up as broken acceptance tests.

The tool is released as Concordion. Good write up on that site.

Update: See this post from Keith Braithwaite

DSDM Atern: The next step in agile!

Keith Richards, Keith Richards Consulting, on behalf of DSDM Consortium

Summary

DSDM Atern (login required) “The Agile Project Delivery Framework”

Latest iteration of DSDM – much more aimed at being a generic project management method rather than IT-specific.

Often used as a wrapper around Scrum and XP to scale them.

Philosophy / Principles / Process / People / Products / Practices

Presentation was a run through material that can be found on the website…

When XP Met Outsourcing

Angela Martin, Martin IT Consulting Ltd

Outsourcing is common for software development, and is the context for many projects using agile development processes. This paper presents two case studies concentrating on the customer role in projects using outsourcing and extreme programming (XP).

The studies follow an interpretive approach based on in-depth interviews, and suggest some tensions between some contractual arrangements in outsourcing, and the XP process.

In particular, one suggests XP worked well in the context of their particular outsourcing arrangements, and the other study suggests difficulty in aligning XP with a different set of outsourcing arrangements.

Notes

(to be) Published as a paper on MartinIT website

Method – two interpretative in-depth case studies. Multiple perspectives via semi-structured interviews. Validated data and interpretation with each interviewee.

Two Case studies

Case Study 1 (T&M)

KiwiCorp (customer), DevCorp (outsourcing/software house), BureauCorp(facilities management and infrastructure). 15 months, 11 people. Seen as a success.

Customer saw benefit from the XP process. DevCorp project manager recognised that on fixed price would have had to be harder on the client.

Lots of vendor issues because of differences between DevCorp and BureauCorp.

Case study 2

Project Pinta. Custom-build, fixed price. FalconCorp (US developer), RetailCorp(UK retailer), ManageCorp (big consulting organisation who hired FalconCorp to do the job). FalconCorp to take the cusotm build and sell as a project.

Everyone thought it was doomed… 6 month deadline. In 2 weeks ramped up to 60 people over four XP labs. Weekly iterations. So within a month knew it wouldn’t fly.

FalconCorp felt that as fixed price very little room to move, so moved more to waterfall… Stopped asking questions, to make sure they could just get signoff and the cheque. Kicked the customer rep (ManageCorp) out of the lab – seen as being a “spy”.

Is this because of the process, the contract, or Winner’s Curse? (over-bid)

Got to 6 month demo – and custoemr accepted it! (the demo didn’t show the broken bits…).

FalconCorp went into bug-fix mode and laid off 2/3 the staff. Then new management came in and found that actually the product was full of holes. Treated it as a throwaway prototype, and went into a fixed-price waterfal project to re-engineer as a saleable product.

Questions (and for Duncan too)

Q: How do you sell Agile to your client?

A: Don’t – until you are sure you can deliver in an agile way

Q: What are the client drivers for fixed price?

A: Need someone to blame “not the client’s problem”

Q: Which has pects of Agile do you keep/drop when customer insists on fixed price:

A: All of them (internally) as a delivery engine.

Q: Why does the industry encourage under-bidding?

A: (Keith) We have allowed purchasers of IT to think it is a fungible commodity so competition is on price. Look for shared-risk, shared-reward…

Can IT Projects be Insured?

Graham Oakes, Graham Oakes Ltd

Summary

In the movie industry, the people financing new productions can buy “Completion Bonds” – effectively insurance policies that repay their investment if the film isn’t completed on time and in line with the original proposal. Such bonds cost perhaps 2-6% of the total production budget. Could we do the same for IT projects?

Notes

Graham Oakes – independent consultant, help people to set up projects in early stages, and help organisations/sponsors to keep in touch with the project.

We like to think about a project as a contained world – but what about the “space” around it – the things outside the project which can impact but which we don’t pay enough attention to?

Zoom out even further – most organisations have lots of shapes and sizes of projects – lots of hidden interactions. “Gravitational force” of large projects affecting other… How can we control the “black space”?

Zoom out even further – universe of projects – but even more space…

Chaos. Standish Chaos Reports – most projects unhappy. But how come most project manager CVs look like most projects are good!! Sampling the same universe!!

Perhaps it’s about how you define success…

Creates lots of Confusion and Blame – procurement processes which are self-defeating. Contractual structures which only hide or defer risk, don’t transfer it in the way that is intended.

Hence downward spiral – expectations of failure, lead to procurement and governance processes which have unintended consequence of making failure more likely, which increase fear…. and so on.

We’re all in this, and it’s hard to break out. For people with “skin in the game” it’s hard to risk change…

Look at film industry – completion bonds. Insurance policy – 3–way between producer, money and insurer – typical premium 2%-6% of production budget. If film not completed on time and budget, insurance repays the financier.

Part of the contract is insurer has right to vet people on the film, check script, come on location etc. If they think project is off course, they have right to intervene. In their interest to not intervene – happens about 20% of projects in UK, fewer cases where they need to takeover.

So how would this work for software?

If company could buy cover, get independent advice, would help more SMEs take advantage of opportunities to get value from IT projects.

So what would be needed to make it happen?

some group work happened…

some of the ideas were:

• agreed measure of progress – e.g. automated acceptance test figures
• minimum engineering practices
• risk management
• transparency
• a market in insurers… which implies that someone can take a view on how to judge risk…
• process, methods, maturity…
• where are you in the process?
• evidence of capability
• completion and intervention criteria
• co-location and collaboration

Graham’s input:

• evidence of alignment between all the stakeholders
• clear definition of requirements
• shared plans and practices
• team and track record
• IPR – if project doesn’t deliver, insurer gets IPR in what has been done…

Visibility is critical…

What would this do to the customer-developer dynamic?

Lack of trust is preventing people starting projects, and killing projects which would otherwise be possible.

Not enough to just say “trust me”

Triad relationship might build boldness, expectations, accountability, management of the business priorities bottleneck…

All parties need skin in the game

Could this work?

I (JE) asked the question “why hasn’t this been tried already?” – according to Graham there was apparently one large project which looked at it a couple of years ago, but the insurers quoted a premium of 40% of the prOject value…

Fit for the Future: The future of Agile Acceptance Test Tools

Antony Marcano, testingReflections.com

• The role of acceptance test tools in agile teams
• Where have they come from; what are they; what is their future?
• Report on the vision created during the Agile Alliance Functional Test Tools Visioning Workshop, which included Ward Cunningham, Brian Marick, Jim Shore, Elisabeth Hendrickson and the presenter, Antony Marcano.
• Delegates will be encouraged to discuss these ideas and suggest some of their own.

Notes

Power of acceptance tests to drive out requirements as well as prove success.

ATDD – Acceptance Test Driven Development or Story Driven Development

Example from Jason Gorman – Test-driven Kitchen Design – should have designed based on examples of things that he wanted to to in the kitchen.

Get examples of how you are going to use first before thinking about what you need to enable it – then wrap those examples in the form of tests.

Target audience for tests is the customer/Product Owner, developers and tester: one of common problems with Test Driven Development is that often written in automated test tools which fall short of the customer-communication requirements

A commonly-used tool is FIT – others e.g. Concordion

How do you slice up the work – e.g. horizontally (by layers / components) or vertically (e.g. by feature) ? ATDD leads to a feature-based approach, growing the design with each iteration by incremental addition of capabilities.

Workshop report on workshop sponsored by Agile Alliance in Portland Oregon October 2007. “to discuss cutting-edge advancements in, and envision possibilities for, the future of automated tools”

Highlights:

Levels of abstraction at which tests are applied – Kevin Lawrence – Goals/Activities/Tasks – e.g. task level testing in Selenium. However activity-level testing much easier to maintain as the application changes. Next generation of tools should better support activity-level tests.

Vocabulary – many people using FIT in a standard way. e.g. Given (context), When (something happens), Then (expect something) – especially relevant if writing tests at Activity level.

Visualisation of Flow – e.g. seeing images of workflow for tests in that area. Brian Marick doing some work on before-the-fact workflow visualisation tools. Alternative approach by Ward Cunningham requires textual description of test flow, but when text can execute gives a visual view. Another tool – CubicTest – Eclipse plugin to capture workflow and make it easier to write Watir and Selenium tests.

Multiple Views – how about IDE presenting a “customer view” of test code that maps onto underlying code but which is expressed in customer language?

Augmenting with model-based testing – Ben Simo presented model-based testing tool. Idea (Antony) how about generating the model from the first tests

Patterns of self-testing software – integrate into the software, part of the documentation (technical, business, online help) – use to drive development.

Workshop Yahoo! group: aa-ftt@yahoogroups.com

Q: How useful for testing non-functional requirements?

A: Workshop didn’t look at this. Some personal experiments with e.g. FitDecorator. Not a lot in the literature yet.

Q: Is passing the Acceptance Tests enough?

A: No – need to combine Prospective Testing (as described above) with consideration of fault path conditions and with Inspective Testing – as soon as a story is passing all its tests why not start more Inspective testing to explore the application.

Q: What’s the right time to do acceptance testing in an agile environment?

A: Ideally every time there is releasable code. During the iteration plan at least come up with the names of tests to accept the story, then treat creation of the tests as a story in that iteration.

Examples, Exemplars, Requirements, Tests

Keith Braithwaite, Zuhlke (but speaking on behalf of SPA)

Summary

• Automated Functional Tests ensure quality and drive process change
• Test failures are more often due to misunderstood requirements than sloppy coding
• Treating tests as executable specifications can help with both
• Tests based on examples lead to an exploration of the problem space that discovers requirements and provides a foundation for trapping defects.

Notes

Reporting on practical experience at Zuhlke over last ~18months

What business value should we see from automated testing?

IT response to the uncertainty of software delivery has traditionally been about forcing the customer to speak in IT terms – e.g. requirements engineering, waterfall etc.

Big shock – it’s about people!

And people want IT tools to help them carry out a task in order to achieve a goal in some context – and it’s the goal and the context which are important.

Context is fuzzy and messy.

Writing rules about fuzzy contexts is hard/impossible – but the people in the business can almost always give examples

In example, users built examples (of FX trades) in Excel. Team used FitLibrary to take these example data, drive the system under test and display results

Several hundred scenarios tested. Revealed defects in existing systems (very early). Having tests before features encourage incremental development.

It was hard to write the adapters from FIT to the system-under-test – if it’s hard to instrument the design that in itself suggests the design is faulty.

This delivery of the tool was defect-free: i.e. no defects to fix after the release! After that, this approach became mandatory.

What about “correctness”? Almost certainly not in the exact computer science sense – but who cares – not the business!

Second example – messaging system, again for financial trades.

Again, examples set out in a tabular format. this time captured in Fitnesse.

Some issue – domain specific messages (SWIFT) have many fields, so example tables v. sparse – hard to work with.

Issues with shared test environment – required semi-manual intervention.

Reporting – tests written = scope captured, tests passing = scope delivered.

Project board said first time they believed a status. Users said they want it this way always in future, even though it required a lot of their time.

These tests get written early (preferably first) – which means they are testing something that doesn’t exist. Think of them as a form of a specification – in fact they act as a gauge.

Q: How do you estimate?

A: Probably doesn’t affect…

Q: How should this impact the buying side – should we frame our requirements in an example-driven way?

A: yes…

Q: What if the tests do not cover all areas of the user requirement?

A: dangerous – should aim for as full coverage as possible..

Update: See this post from Keith

Using Agile: the QA perspective

Chris Ambler – QA Director Electronic Arts

Not a developer, never have been, never will be.

Not a techie! But have worked in testing (of various sorts) for 28 years.

Focus is product, and quality.

What does quality really mean? How does it affect the business? How can we measure it?

Product quality, quality of efforts…

Product quality – Stability, Excitement / Experience, Progression

Hard to measure the user experience?

Quality is the responsibility of everybody.

Quality of efforts – P * T^2 = Q

Team – up for the challenge, skilled, motivated?

Processes – and are they followed?

Timescales – does the plan allow for sufficient time to execute?

How does this relate to the business world…

The business – driven by fears – share price, not knowing what’s going on etc. Hate surprises. Want predictability, consistency, quality, profitability, transparency.

Project teams – driven by fears – losing job, not getting bonus, not delivering product. Have to deal with technology, complexity, customer expectation, environment changes, pressure to ship, floating features, avoiding surprises!

The business don’t really know what the team are doing, and the team don’t want them to know!

The team care about the link to the business – the business mostly only care about the product.

So how to pull the three together?

Review the Agile Manifesto

Highest priority is customer satisfaction

Deliver product on a regular basis – only true metric

We can handle late change – it happens!

Need direct face-to-face communication

Commit as a team – a whole team

Appropriate level of planning and documentation

Transparent

Secret weapon is inspect and adapt

Seems like Agile Thinking to Chris – but how do the business think?

Game development to asset lock-down point is “a melee” – whereas the bug fix to final release time can be driven very tightly in agile way.

3 phases;

Initial Creativity – agile-ish

Massive Change – melee

Delivery – agile

But can still work in agile ways… Agile is a state of mind, not a process!

Traverse the bridge of uncertainty

Get rid of surprise with strong reporting

Use light-weight processes when you can

Don’t get into bug debt

Learn to know when you are “done”

Make sure the business do this too!

Scaling Scrum

Rachel Davies, Agile Experience Ltd.

Summary

Scrum is the simplest possible agile method so it’s easy to get started with a small team of software developers. What happens when you try to apply Scrum to large projects? This talk shares experiences of working with large projects with multiple scrum teams and distributed scrum teams. Come to this talk to explore how to scale Scrum without losing the essence.

Notes

Sees lots of teams being successful with Scrum in small projects, but people tend to get lost when they look at scaling up… It’s an emergent field.

The challenges – highly collaborative, need the team involved in the planning, risk of losing easy way of resolving issues, hard work!

Scrum recommended team size is 7 +/– 2. Scaling could be many teams, geographical distributions, organisational level.

Rachel asked if anyone had experience above 5 teams – very few.

Scaling hits roles differently:

Product Owner – workload, being at all the meetings

Scrum Master – more inter-team issues, dependencies

Scrum Team – less empowered, large or distributed.

Challenges for Scrum Practices:

Establishing Sprint goal, distributed team members at Scrum meeting, Increment has dependencies on other teams, emphasis on electronic management of sprint backlog (remoteness), political impact on product backlog (more at stake on big project), sprint review – who’s in it?, combining lessons from teams in retrospective.

Organisational issues – greater potential for culture clash, impact of organisational impedance. Logistics of working environment and technical infrastructure.

Advice from the literature:

Scrum of Scrums

Start with a “Beachhead team” the use the members to seed other teams. (assumes you are starting a fresh project – Rachel sees lots of organisations getting benefit from Scrum on existing products)

For existing products – extract a virtual team to focus on forward plan, and another to focus on integration, keeping it all working.

May need communities of common disciplines e.g. DBAs

Align iteration boundaries across teams.

Think about series or parallel for multiple daily Scrum meetings

Keep clear about “levels” in plans – vision, releases roadmap, themes per sprint, product backlog, sprint backlogs per team.

Think about using a Product Owner team to share the load.

Common language, domain model, glossary. Architecture guidelines

Think about integration of the product – maybe a virtual team across the teams.

Think about testing – may need additional test team looking at whole product as well as the people in the scrums teams

Organise product showcase meetings for all teams to see whole product

Think about scaling the development infrastructure

Strike a balance:

Larger teams v. More teams

Contention for single Product Owner v. Product Owner team

Cross-functional team v. Specialists

Questions from the floor

Q: You mention feature teams v. component teams – difference?

A: Component teams might be a silo working on particular bundle of software e.g. a layer, possibly across multiple products. Feature team tend to focus on aspect of user-facing functionality for one product, possibly across multiple components – requires huge breadth of knowledge in the team.

Q: Thanks, trade-offs?

A: Knowledge-sharing. Team per component – might find one group is busier than other, so delays features. Feature-based teams address that, but the challenge is the amount of knowledge and skill in each team.

Q: How do you stop multiple sprints in between releases merging into one big sprint

A: Have to keep up strict discipline. Product Showcase on each sprint helps. Sprint Review of completion against velocity.

Leading Agile Teams

Dot Tudor, TCC

Summary

One school of thought is that good leaders make a big difference to the successful outcome of any initiative. On the other hand, some agilists want to eliminate leaders and go with situational leadership and self-organising teams. There is also a large contingent in the agile community with the view that the right approach is to change the style of leadership, not to eliminate leaders.

This interactive workshop will explore the workings of an agile team and attempt to identify the issues and approaches to leadership styles that support an agile environment. This will be fun, informative – and there may even be prizes!

Notes

Workshop format…

Will have to think about how to transcribe this!