Jan 1, 2012
Working in a different industry that has different demands, and brings different challenges to the table, is a thrilling experience. During my time in the development process for end-user applications and telecom applications never have I seen the chaos and change rate and pace that I have seen in the aggressive e-commerce industry. Using processes and methods that you are accustomed to from other industries just does not cut it here. Managing development for an e-commerce site is like funneling chaos through a pin hole. Committing to issues/tasks on a (bi)weekly basis is a dream in this ever so fast transforming universe. So you have to devise a new method of making the production line run. So here is exactly what is not working:
- Choose N number of issues, and try to get them developed by time X
- Get them tested.
- Fix reopened stuff.
- Get the reopens tested
- deploy live
The bad thing about the first step is that, by the time you are half way through the N chosen issues, something new with a higher priority than everything else pops up, that must be done now. It is almost impossible to eliminate this interference without loosing competitive edge.
The bad thing about the second item is that the QA team is working constantly in burst loads. One minute they don’t have anything related to the release, the next minute they are expected to test N issues in a short time.
The bad thing about the 3rd item is that QA is idle again waiting for a new release with the fixes.This can be mitigated by opting to continue the tests even if a bug is found, and sending all fixed reopened items to test later. This is trading off robustness for speed, as ideally the test would stop, the bug would be fixed, and the tests would start from the beginning, in case that bug has caused any side effects.
The 4th item can also be avoided by not submitting reopened items at all (except for really crucial issues), and waiting for the next release to deploy them.
So this usual way of developing is not suited for such a fast changing environment. Here is the alternative method we are experimenting with
- every task is assign a priority during creating
- every body picks the next most important task from the pool
- every hour the release manager integrates any finished task to the development branch, from the topic branch of that issue
- the QA team constantly tests the issues that are integrated into the development branch
- if there is a reopen it sent back to the developer every six hours the release manager selects all the issues that have passed testing in the development branch and integrates them into the release branch
- QA runs an integration test on the release branch
- if there is a task breaking integration, it is unmerged
- deployment is done every day
I think the major advantages of this type of development cycle is that the future-reopen issues aren’t loosing time waiting to be included in a release but are tested on the spot, and are never merged into a release. Also another great advantage is that the load distribution on the QA team is evened out. I will publish some empirical data on both processes once I have collected enough for the second process type.
Having a test server that is independent of cocky sys-admins is crucial to the success of this methodology. You absolutely need a smaller version of your live environment at your finger tips, that you can break restore at will.
A further improvement will be added by actually being able to test each issue in isolation in it’s own branch. This brings the problem of having to cope with DB changes (since each branch’s deployed code will be using the same DB) that may lead to conflict.