Agile Testing and BDD eXchange 2010

Yesterday I attended the Agile Testing and BDD eXchange. Interesting day, few good talks. I however did not take any notes. However John Stevenson created a Googlewave and took some awesome notes. I've appropriated them for my blog.

The podcasts are making their way online here


Dan North
why ignorance is the greatest enemy to success, and presents some strategies and techniques for deliberately reducing ignorance, increasing learning and moving towards more deterministic and lower risk software delivery.

wrote a blog on the perils of estimation, negative aspects of planning
as we do decomposition, we accidently discover features, stories, etc.
- some external stake holder

Its not deliberate discovery...

Deliberate discovery (half baked theory - in progress)

Close your eyes and think about the last project you did, imagine you had to redo it over again - the same objective, people, everything - except 20/20 hindsight - all the nastly little gotchas your are forwarned about

how long would it take you?

People usually answer a dramatically shorter time...

Theory of constraints...
In any system there is currently one thing that is the most constraining thing - trying to put out a fire with a line of people passing buckets - one guy is slow

Any work you do behind that constraint is waste, any work upstream you may as not be doing...
Any time you spend not addressing that constraint can be waste
When you resolve that constraint, another replaces it...

Problem: you dont know where the constraint is !! You need a big arrow to show you where it is

Ignorance is a constraint !!
- we dont know things and this is stopping us delivering

Kinds of ignorance
- what do people actually want - requirement - the nature of the problem
- vendors - response time, availability
- unfamiliar technology
- people - the team
- performance - what is acceptable, technology performance
- we dont understand the domain - mary pop. dont deliver what is asked for, find out what the problem is and deliver on it
- organisational constraints
- relationships with stakeholders - knowing who the stake holders are

The right choice of technology can make a project an effective experience and can really make a project flow
The wrong technology can bake in all kinds of problems

Second order ignorance - unknown unknowns
Ignorance is multivariant



Ignorance reduces in steps
The bigest discoveries in physics are not eureka moments

oh Oh! Crap

Even with BDD you still end up with Oh Crap ! moments --

At the begining of a project
- this time we will know better - attribution bias - people make mistakes but we dont - the more you know about this the more you do this
- this time we will come in on time
-- this time will be exactly the same as last time


Deliberate discovery
Assume you are always operating in ignorance
- immediately go into Oh crap mode and work out how you are going to get caught out
Assume some specific axis of ignorance is your current constraint - a fairy comes along and tells you one thing that will majorly affect your project - thinking like thatputs you on the lookout
- improve through put by actively addressing ignorance

Second order ignorance is a given
- if you try write these down is not in your head

Book: A mind of its own

We are wired to resitst this
- suffer with attribution bias / confirmation bias (pro life vs pro choice - see confirmation of their position in the same data)


Remember how the project started?
Get everyone in a room
You decomposed the problem into stories - and more stories.... etc
what that really the best use of your time...? No (from the audience)


So how can we apply this?

Even if you have a great team of developers it could be the company that is holding them back - constraining process, lack of vision

Software has a half-life - like a physics half life
- software is replaced by other software

Shorter halflife means less 2nd Order Ignorance
- a smaller code base means you are less ignorant about it
- if you create a very short half-life then you are less ignorant about it
- less opportunity to not know what I dont know

Once you know the domain better you can Iterate over an entire faster than you can rework


Deliberatediscovery in planning

Plan for at least some unexpedted bad things
- you can predict that there are things you cant predict

Try natural planning (GTD) - an options approach
1 Purpose
2 Mission / vision / goals
3 Brainstorming
4 Organise
5 Next actions - the thing to do after discovery may be more discovery! - with a complex problem its okay to know that you dont know where you are going and leaving the room knowing there are things that will still blind site you - "planning" is iterative

Is planning pointless - Dan: No, but what I am calling planning is not what other people call planning - run a series of discovery sessions (war stories, scenarios, 6 thinking hats, pubs)
- dynamic of having people in the room with different perspective that is a highly productive way of discovering things

In business - we do unnatural planning
As a human we do it differently - lunch for 10 people? - lots of unordered unsqeuential

Beware the perils of fractal estimation

You need a lot of discipline to make this work (6 thinking hats - thinking creatively in the same direction - avoids counter productive criticism)


Deliberate discovery in analysis

Dont fear analysis paralisis
- as long as its reducing relevant ingnorance - if you have a constraint, the best thing you can do is more analysis


Ethnography
- go find out what your users are doing by watching them so you know what they need to do
- what the stakeholders caring about
- what should they be caring about

Play it forward
- who are you tring to reach
- what do you want therei experience to be
-- UI dev is defining wht the user experience is going to be, the life of the people you are going to touch with your design (its not just about a screeen) - you get to choose what that experience is !!


Deliberate discovery about programming

Spike and stabalise - typical agile approach - get creative and throw it away and write something stable
- learn through evolving the spike
- choose to stabilise later - deferred, test-driven testing

- if we say this is spike or production code, we have already taken the option - we should really differ that option until the last responsible moment
- you can code and decide later if it is appropriate for production if you havent excerpted your opttions

Design for the second case
- you might gonna need it

Indirect discovery
- have an end goal to learn something and learn along the way - go solve Y using X to learn about X - learning about X is less effective
- focus on different aspects

Travle in pairs
- huges amounts of discover and learning in paris

Deliberate discovery in testing

Exploratory testing
ramdomising testing - come up with crazy stuf
Not just running the same old automated test
Good testers are already doing this

DD in devops
get into production early - walking skelleton (Cockburn)

Design for monitorability
- push diagniostics rather than pulling an autopsy
- make it easy to listen

Design for discoverability
....


Summary
You dont know what you dont know - this is a fact that is still ture
Ignorance is killing your throughput
Sometimes you cant know what you dont know
For everything else there is deliberate discovery



Experiences using SpecFlow - Patters and practices of building a Gherkin Based Living documents

Implementing your first user story is easy: describe your first Gherkin scenario (Given/When/Then) and implement it outside-in using TDD. But as the system grows, several questions arise that you will need answer...Gaspar and Christian will talk about their experience on several .NET projects using SpecFlow and about questions they ran into when doing BDD with SpecFlow. View the podcast here..

Deriving scope from goals ... its not easy

Product backlog, users stories, epics....

Deriving user stories from...
- vision/busines goals - too far away
- actor goals - not focused enough

Roadmap helped to focus
- first milestone goal - use specLog for SpecLog -- even if it hurts.

Illustrating user stories with examples: acceptance criteria


I recommend viewing the slides and podcast for this one - should be uploaded before the end of the conference.

(I am struggling to makes notes on this one - but its an interesting session - John Stevenson)



TEST-DRIVING AT HIGH SPEED: KLARNA'S AGILE STORY

David Evans and Erik Stenman present an experience report on agile testing at Klarna, transition to Scrum and then Kanban, Erlang test automation environment, the strategies they used to adopt agile acceptance testing and their experience with Fitnesse + SlimErl View the podcast here...

A small start up running a couple of servers
Important to get up an running quickly
Started billing after a few months
Profitable by the end of the year

Introducing trust in online shopping
Klara the consumer doesnt play untilthey get their goods - the merchant is guaranteed to get money for products

Always up and running - onnline credit decision
- 80% of all Swedish onlne stores now using Klarna

Written in Erlang
- build a system that is up and running for ever and ever.
- a good way to get great developers
- process oriented (built in concept)
- erlan virtual machine (BEAM) has support for symmetric multiprocessing
- no shared memory - easier to program
- each year your sequentional programm will go slower and you concurrent program will go faster (multi-core hardware with slower cpu cycles)

Instinctively Agile
Develop quickly
Short iterations
Do only important and neccessary features
- can deploy into a running system.

Tight deadline coming up to Christmas, so worked on taking orders, then taking money and then paying merchants (as we could get away without paying them for an extra month)


Dont worry about performance
- until we have to
- Uneven but predictable load patterns

History of rapid growth

Challenges
Testing
speciications
knowledge sharing
documentation
reviews
- growth of a small company with a lack of communication

Lack of consistency around testing
Business and IT communication challenged
No time for documentation - problem when getting new people joining
growing divide between the language the business people (especially new people) and the rest of the org.
- vision was evolving without understanding the original vision
- everyday terms are not necessarily shared in understanding between all people


2005: we need a product
cowboy coding - cant scale, high risk, organic architecture
A few programmers - all Erlangers, solid but reactive coding, every code change reinvented the wheel.
Huge interdependencies

2009: We need a process
Scrum 2 week iterations, 3 teams, separate backlogs, technical product owners, no testers in teams
- lack of knowledge, focus on business understanding

The business didnt like it
- scrum has slowed us down - they were used to released in an hour, releasing every day not in two weeks
- sprint goals violated
- poor estimations
- stories not succinct


2010: we need flow
Kanban
6 teams, common backlog,WIP Limit of 4 stories, embeded testers
- kanban along would not fix the issues without discipline, but fitted the business model and practices the business were used too
- used physical slot to show wip limits - but cat-like programmers doubled up in slots...

Communication still an issue and hasnt been solved by changing the process


ATDD
Can we solve all these issues?
Testing
speciications
knowledge sharing
documentation
reviews

Wanted to get a regular cadence but timebox so you could specifically make time for a sprint review
- can we merge ATDD with Kanban to resolve all the above issues - especially communication

Specification by Example
building up a body of living documentation
selfchecking business rules
Helping business people and POs get better at writing stories and illustrating with examples
separating the "what" from "why"

Refactoring test suites to utilise common primatives for objec tsetup and teardown
SlimErl - very lightweight slim port for Erlang - slim makes creating a runner an easier process - open source
Simplifying automation by parsing Fitness tables to generate fixture code

Its a journey - perfect is a verb, Kent Beck - you cant be perfect but you can try to perfect

What have we learnt
- success can be a poison challece - huge growth can cripple a company that cant manage it
People are the most important part of any process - they are also the most complex
Culture affects process in surprising ways
Every process ust include time for reflection
Processes, lik products, must evolve iteratively

Where do we want to be?
Many smart people with good ideas making great solutions quickly through good documentaition


John Smart

discusses several case studies of automating web testing using BDD and ATDD tools and techniques. View the podcast here...


Dont let our web tests ends up like a fallen down building (derelick building)

3-6 months half life of your automated tests typically


3 ways of automated testing
- record and replay
- scriptiong
- page objects - a really good approach, especailly with ATDD


Record / replay
very basic quick and ditry specific tests and Nothing else

scripting
still get a lot of duplication

what we like to have
avoid duplication (code)


DRY dont repeat yourself
tests should be treated with even more respect than your production code as they are your insurance policy

Selenium RC is a low level tesing approach, talking about names of fields - not particularly business friendly


Introductiong page objects
Reusable
Low Maintenance
Speak you language - talk in terms of business concepts

pages objects
classes that represent parts of your page
lot more power in your tests

Present reasonable understandable point of view of what is hidden in your page

You need to design the page objects, so there is more work up front, but it is the developers who work on these so its not too much of a problem.

Allows the techy people to talk the business language and think about those business concerns.



Page objects in action

Selenium RC - very low level and quite slow (have to wait for the page to load)

Webdriver/Selenium2 - simple abstraction page.open(), page.searchFor("cats"), open(), close(), clickOnFeelingLucky(),

Use a list to check search box lists and other nasty under the hood ajax stuff - keeping the code pretty simple - only looking at the business functions.


So page objects are easy, but would you want your testers to have to deal with them?


using "google-search"

scenario "searching for 'cats' in the search box" {
....
}


Make a distinction between the work in progress and regression tests

Architecture - fitting it all together
(see slides and podcast)


Summary
Using page objects in this way allows for a level of abstraction that results in maintainable, reusable tests that become communication tools, work with BDD or ATDD frameworks, under the hood you can make them as robust and reliable as you like

You need to invest the time to maintain those page object as a deliverable

You can develop your page objects incrementally and map it to the BDD / ATDD approach

Works as an awesome documentation tool

Introducing Weeknight Testing!

A few people have recognised that while Weekend Testing is a fantastic thing not everybody can make it.

To help with this Weeknight Testing is being introduced.

First session details are:
Date: Wednesday 24th November 2010
Time: 7.30pm – 9.30pm UTC/GMT

Full details here

Let's exchange testers.

I went to school in Sydney, Australia where we had a student exchange. A student (it was probably a couple rather than just one) from another school and country would swap with a student from our school. I imagine this is done throughout Australia and throughout the world but as far as I know it stops after high school. Why?

I didn't take part but I imagine it to be richly rewarding, experiencing a different cultural, a different school, different people, different foods, I could go on.

Can you imagine the possibilities if this continued in the workplace? If you could take your own personal knowledge and experience as well as your organisation's and share it? And at the same time learn from other people and an organisation?

Just a random thought that was going through my head.

Take part in a testing book, share experiences/knowledge and raise money for charity

The STC is spreading a little love this Christmas. They are creating a ebook which will be sold to raise money for Oxfam. The ebook will have contributions from testers all over the world and you can be one of them.

You can help by filling in this form, promoting the book and/or donating.

Full details here.

November London Tester Gathering write up.

On the 2nd November another London Tester Gathering was held. I was quite surprised, the turnout was the normal(ish) 50 odd people. I thought the tube strike would make it a very quiet night.

Reflective Solutions demo'd StressTester which looks intuitive and easy to pick up and seemed to generate some interest, I noticed the Reflective Solutions guys answering a lot of questions. If you are looking for a performance testing tool it's worth checking out.

It was great to have Michael Bolton along to another gathering, he shared with us his 'Burning Issues' talk which was thought provoking in a worrisome kind of way but also entertaining.

There were lots of new faces and lots of regulars, great to see both.

It was also a great pleasure to meet Sharath Byregowda who is one of the co-founders of Weekend Testing which has now become a worldwide 'movement'.
Sharath is in London looking for testing challenges so if you need someone, grab him quick, he won't be free for long.

There were some great discussions all around, way too many to list. One that is of particular interest was the idea of weeknight testing, something to keep an eye out for.

All in all a very interesting night, there were discussions, things for people to take away and think about, visitors from India, Toronto, Netherlands and Scotland and lots of merriment.

See you at the next one.

Testing the future - Test challenge

Darren McMillan and his team have been giving each other testing challenges. He was kind enough to share one with me. As I ate my lunch today I completed it, my questions following.

Testing the Future
Scenario :

Your good friend Dr. MacDonald has been developing a time-travelling car which he aims to use to allow a person to travel backwards or forwards in time. He states that he has tested each of the components separately and has tested the system as a whole by using two synchronised clocks (one of which is sent 1 minute into the future with the car. When the car reappeared, there was exactly 1 minute of difference between the clocks).
He states that he is now happy to use the system and wishes to be the first person to travel in time.
Task :
Dr. MacDonald has hired you at this late stage in development to consult about the level of testing he has carried out so far and if he needs to do any more.
So, the task is simply this. What questions would you ask Dr. MacDonald about the system and its testing?
Time :
Please take no longer than 30 minutes to come up with as many questions as you can.

My questions:

Is there a min/max timeframe for the travel?

Is he concerened about any of the components?

How confident is he that the clocks are correct? In good working order?

What type of clock was it? Digital? Analog? Could anything have affected the clocks?

Is there a date on the clock?

Are the clocks still working?

Could the car have actually travelled 24hrs and 1minute? Or 48hrs, etc.

What does his synchronised clock test tell him? Did the car go backward or forward?

Is that actually what he was aiming for? 1 minute time difference?

What if that's all the car can actually do? Time travel 1 minute into the future. Is it worth continuing?

Does the car take time to warm up before time travel?

There is plenty that could go wrong with a car, is he confident that every aspect of the car is in good working order? Tires? Engine? Breaks? etc.

Does the car actually move when it time travels or disappear and reappear? If it moves does it need to be at a specific speed? Specific fuel?

Specifically what components have been tested and how?

Specifically how was the system tested as a whole? Just the 1 minute test?

Did he check the car for any possible issues after it returned?

Can he confirm the car and the clock went to the same place? Is there a possibility they went separate places? How do you know?

Should he not test travelling backwards? Why did he just test going forward?

How confident is he on the whole? Would he send his mum on a trip?

What affects the travel? Weather? Weight of occupant/s? Has he tried a life size dummy? Something organic? What if he repeats the test with some fruit in the car and it supposedly was gone for a minute but the fruit is rotten on the return?

If he is happy to use the system and ready to travel why has he hired me? What is niggling at him?

Does he have any experience with time travel or cars?

Do I have any experience with time travel or cars? - Why was I hired general and why me specifically?

Why does he want to time travel?

Why a car?

Why do I only have 30mins to come up with questions?

What is he a doctor of? What if he is he a crazy vet?

How does he know the car travelled? What if the car stayed still and everything else travelled?

Has anybody else done any testing?