Agile Testing and BDD eXchange 2010

Yesterday I attended the Agile Testing and BDD eXchange. Interesting day, few good talks. I however did not take any notes. However John Stevenson created a Googlewave and took some awesome notes. I've appropriated them for my blog.

The podcasts are making their way online here


Dan North
why ignorance is the greatest enemy to success, and presents some strategies and techniques for deliberately reducing ignorance, increasing learning and moving towards more deterministic and lower risk software delivery.

wrote a blog on the perils of estimation, negative aspects of planning
as we do decomposition, we accidently discover features, stories, etc.
- some external stake holder

Its not deliberate discovery...

Deliberate discovery (half baked theory - in progress)

Close your eyes and think about the last project you did, imagine you had to redo it over again - the same objective, people, everything - except 20/20 hindsight - all the nastly little gotchas your are forwarned about

how long would it take you?

People usually answer a dramatically shorter time...

Theory of constraints...
In any system there is currently one thing that is the most constraining thing - trying to put out a fire with a line of people passing buckets - one guy is slow

Any work you do behind that constraint is waste, any work upstream you may as not be doing...
Any time you spend not addressing that constraint can be waste
When you resolve that constraint, another replaces it...

Problem: you dont know where the constraint is !! You need a big arrow to show you where it is

Ignorance is a constraint !!
- we dont know things and this is stopping us delivering

Kinds of ignorance
- what do people actually want - requirement - the nature of the problem
- vendors - response time, availability
- unfamiliar technology
- people - the team
- performance - what is acceptable, technology performance
- we dont understand the domain - mary pop. dont deliver what is asked for, find out what the problem is and deliver on it
- organisational constraints
- relationships with stakeholders - knowing who the stake holders are

The right choice of technology can make a project an effective experience and can really make a project flow
The wrong technology can bake in all kinds of problems

Second order ignorance - unknown unknowns
Ignorance is multivariant



Ignorance reduces in steps
The bigest discoveries in physics are not eureka moments

oh Oh! Crap

Even with BDD you still end up with Oh Crap ! moments --

At the begining of a project
- this time we will know better - attribution bias - people make mistakes but we dont - the more you know about this the more you do this
- this time we will come in on time
-- this time will be exactly the same as last time


Deliberate discovery
Assume you are always operating in ignorance
- immediately go into Oh crap mode and work out how you are going to get caught out
Assume some specific axis of ignorance is your current constraint - a fairy comes along and tells you one thing that will majorly affect your project - thinking like thatputs you on the lookout
- improve through put by actively addressing ignorance

Second order ignorance is a given
- if you try write these down is not in your head

Book: A mind of its own

We are wired to resitst this
- suffer with attribution bias / confirmation bias (pro life vs pro choice - see confirmation of their position in the same data)


Remember how the project started?
Get everyone in a room
You decomposed the problem into stories - and more stories.... etc
what that really the best use of your time...? No (from the audience)


So how can we apply this?

Even if you have a great team of developers it could be the company that is holding them back - constraining process, lack of vision

Software has a half-life - like a physics half life
- software is replaced by other software

Shorter halflife means less 2nd Order Ignorance
- a smaller code base means you are less ignorant about it
- if you create a very short half-life then you are less ignorant about it
- less opportunity to not know what I dont know

Once you know the domain better you can Iterate over an entire faster than you can rework


Deliberatediscovery in planning

Plan for at least some unexpedted bad things
- you can predict that there are things you cant predict

Try natural planning (GTD) - an options approach
1 Purpose
2 Mission / vision / goals
3 Brainstorming
4 Organise
5 Next actions - the thing to do after discovery may be more discovery! - with a complex problem its okay to know that you dont know where you are going and leaving the room knowing there are things that will still blind site you - "planning" is iterative

Is planning pointless - Dan: No, but what I am calling planning is not what other people call planning - run a series of discovery sessions (war stories, scenarios, 6 thinking hats, pubs)
- dynamic of having people in the room with different perspective that is a highly productive way of discovering things

In business - we do unnatural planning
As a human we do it differently - lunch for 10 people? - lots of unordered unsqeuential

Beware the perils of fractal estimation

You need a lot of discipline to make this work (6 thinking hats - thinking creatively in the same direction - avoids counter productive criticism)


Deliberate discovery in analysis

Dont fear analysis paralisis
- as long as its reducing relevant ingnorance - if you have a constraint, the best thing you can do is more analysis


Ethnography
- go find out what your users are doing by watching them so you know what they need to do
- what the stakeholders caring about
- what should they be caring about

Play it forward
- who are you tring to reach
- what do you want therei experience to be
-- UI dev is defining wht the user experience is going to be, the life of the people you are going to touch with your design (its not just about a screeen) - you get to choose what that experience is !!


Deliberate discovery about programming

Spike and stabalise - typical agile approach - get creative and throw it away and write something stable
- learn through evolving the spike
- choose to stabilise later - deferred, test-driven testing

- if we say this is spike or production code, we have already taken the option - we should really differ that option until the last responsible moment
- you can code and decide later if it is appropriate for production if you havent excerpted your opttions

Design for the second case
- you might gonna need it

Indirect discovery
- have an end goal to learn something and learn along the way - go solve Y using X to learn about X - learning about X is less effective
- focus on different aspects

Travle in pairs
- huges amounts of discover and learning in paris

Deliberate discovery in testing

Exploratory testing
ramdomising testing - come up with crazy stuf
Not just running the same old automated test
Good testers are already doing this

DD in devops
get into production early - walking skelleton (Cockburn)

Design for monitorability
- push diagniostics rather than pulling an autopsy
- make it easy to listen

Design for discoverability
....


Summary
You dont know what you dont know - this is a fact that is still ture
Ignorance is killing your throughput
Sometimes you cant know what you dont know
For everything else there is deliberate discovery



Experiences using SpecFlow - Patters and practices of building a Gherkin Based Living documents

Implementing your first user story is easy: describe your first Gherkin scenario (Given/When/Then) and implement it outside-in using TDD. But as the system grows, several questions arise that you will need answer...Gaspar and Christian will talk about their experience on several .NET projects using SpecFlow and about questions they ran into when doing BDD with SpecFlow. View the podcast here..

Deriving scope from goals ... its not easy

Product backlog, users stories, epics....

Deriving user stories from...
- vision/busines goals - too far away
- actor goals - not focused enough

Roadmap helped to focus
- first milestone goal - use specLog for SpecLog -- even if it hurts.

Illustrating user stories with examples: acceptance criteria


I recommend viewing the slides and podcast for this one - should be uploaded before the end of the conference.

(I am struggling to makes notes on this one - but its an interesting session - John Stevenson)



TEST-DRIVING AT HIGH SPEED: KLARNA'S AGILE STORY

David Evans and Erik Stenman present an experience report on agile testing at Klarna, transition to Scrum and then Kanban, Erlang test automation environment, the strategies they used to adopt agile acceptance testing and their experience with Fitnesse + SlimErl View the podcast here...

A small start up running a couple of servers
Important to get up an running quickly
Started billing after a few months
Profitable by the end of the year

Introducing trust in online shopping
Klara the consumer doesnt play untilthey get their goods - the merchant is guaranteed to get money for products

Always up and running - onnline credit decision
- 80% of all Swedish onlne stores now using Klarna

Written in Erlang
- build a system that is up and running for ever and ever.
- a good way to get great developers
- process oriented (built in concept)
- erlan virtual machine (BEAM) has support for symmetric multiprocessing
- no shared memory - easier to program
- each year your sequentional programm will go slower and you concurrent program will go faster (multi-core hardware with slower cpu cycles)

Instinctively Agile
Develop quickly
Short iterations
Do only important and neccessary features
- can deploy into a running system.

Tight deadline coming up to Christmas, so worked on taking orders, then taking money and then paying merchants (as we could get away without paying them for an extra month)


Dont worry about performance
- until we have to
- Uneven but predictable load patterns

History of rapid growth

Challenges
Testing
speciications
knowledge sharing
documentation
reviews
- growth of a small company with a lack of communication

Lack of consistency around testing
Business and IT communication challenged
No time for documentation - problem when getting new people joining
growing divide between the language the business people (especially new people) and the rest of the org.
- vision was evolving without understanding the original vision
- everyday terms are not necessarily shared in understanding between all people


2005: we need a product
cowboy coding - cant scale, high risk, organic architecture
A few programmers - all Erlangers, solid but reactive coding, every code change reinvented the wheel.
Huge interdependencies

2009: We need a process
Scrum 2 week iterations, 3 teams, separate backlogs, technical product owners, no testers in teams
- lack of knowledge, focus on business understanding

The business didnt like it
- scrum has slowed us down - they were used to released in an hour, releasing every day not in two weeks
- sprint goals violated
- poor estimations
- stories not succinct


2010: we need flow
Kanban
6 teams, common backlog,WIP Limit of 4 stories, embeded testers
- kanban along would not fix the issues without discipline, but fitted the business model and practices the business were used too
- used physical slot to show wip limits - but cat-like programmers doubled up in slots...

Communication still an issue and hasnt been solved by changing the process


ATDD
Can we solve all these issues?
Testing
speciications
knowledge sharing
documentation
reviews

Wanted to get a regular cadence but timebox so you could specifically make time for a sprint review
- can we merge ATDD with Kanban to resolve all the above issues - especially communication

Specification by Example
building up a body of living documentation
selfchecking business rules
Helping business people and POs get better at writing stories and illustrating with examples
separating the "what" from "why"

Refactoring test suites to utilise common primatives for objec tsetup and teardown
SlimErl - very lightweight slim port for Erlang - slim makes creating a runner an easier process - open source
Simplifying automation by parsing Fitness tables to generate fixture code

Its a journey - perfect is a verb, Kent Beck - you cant be perfect but you can try to perfect

What have we learnt
- success can be a poison challece - huge growth can cripple a company that cant manage it
People are the most important part of any process - they are also the most complex
Culture affects process in surprising ways
Every process ust include time for reflection
Processes, lik products, must evolve iteratively

Where do we want to be?
Many smart people with good ideas making great solutions quickly through good documentaition


John Smart

discusses several case studies of automating web testing using BDD and ATDD tools and techniques. View the podcast here...


Dont let our web tests ends up like a fallen down building (derelick building)

3-6 months half life of your automated tests typically


3 ways of automated testing
- record and replay
- scriptiong
- page objects - a really good approach, especailly with ATDD


Record / replay
very basic quick and ditry specific tests and Nothing else

scripting
still get a lot of duplication

what we like to have
avoid duplication (code)


DRY dont repeat yourself
tests should be treated with even more respect than your production code as they are your insurance policy

Selenium RC is a low level tesing approach, talking about names of fields - not particularly business friendly


Introductiong page objects
Reusable
Low Maintenance
Speak you language - talk in terms of business concepts

pages objects
classes that represent parts of your page
lot more power in your tests

Present reasonable understandable point of view of what is hidden in your page

You need to design the page objects, so there is more work up front, but it is the developers who work on these so its not too much of a problem.

Allows the techy people to talk the business language and think about those business concerns.



Page objects in action

Selenium RC - very low level and quite slow (have to wait for the page to load)

Webdriver/Selenium2 - simple abstraction page.open(), page.searchFor("cats"), open(), close(), clickOnFeelingLucky(),

Use a list to check search box lists and other nasty under the hood ajax stuff - keeping the code pretty simple - only looking at the business functions.


So page objects are easy, but would you want your testers to have to deal with them?


using "google-search"

scenario "searching for 'cats' in the search box" {
....
}


Make a distinction between the work in progress and regression tests

Architecture - fitting it all together
(see slides and podcast)


Summary
Using page objects in this way allows for a level of abstraction that results in maintainable, reusable tests that become communication tools, work with BDD or ATDD frameworks, under the hood you can make them as robust and reliable as you like

You need to invest the time to maintain those page object as a deliverable

You can develop your page objects incrementally and map it to the BDD / ATDD approach

Works as an awesome documentation tool