HOME / Community / Guidelines

Quality Evangelism

Motivation: Increase the quality and reliability of our codebase - the NetBeans modules.

When developing - writing new code, refactoring or just bug fixing - people always make mistakes and produce new bugs. This is an unavoidable fact and there is almost nothing we can do to prevent ourselves from making mistakes when coding (perhaps writing less code). That's why we should carefully learn how to find our own mistakes reliably and quickly. The sooner the bug is found and fixed the better, because once the bug is fixed can no longer negatively influence further work. The best solution would be to find and fix all bugs before the code containing them is committed into the version control system and the bugs can disrupt others who use the code.

Every developer is excited when designing and writing new features and every developer hates fixing and code maintenance. In reality developers spend significantly more time on bug fixing than developing new features, don't you think? Why? They haven't found their own mistakes before putting the code into the user's hands! Finding bugs immediately after writing new code will give you more time for new features and for fun. Besides that, it will make your users happy and your boss satisfied. Yes, so much good can come just from having no bugs.

How to find bugs?

In NetBeans there are several instruments which we use to find bugs. They are focused on different levels of testing, use different methodology and have different complexity, but in the end they all test the code and are capable of detecting bugs. Hopefully the list isn't final and will soon grow.

Milestones Process

The NetBeans QE team releases Milestones to assure a basic level of quality during the release cycle. The Milestone Process is well-established among the developer community. Milestones reports are published on the NetBeans site and announced on the mailing list. There are strict rules followed by all developers about when and how to fix Milestone stopper bugs.

Commit Validation Suite

The Commit Validation Suite should help all developers to verify that the changes they are going to push to the NetBeans Mercurial repository don't break any vital functionality provided by the NetBeans modules. This is the suite of tests gathered among NetBeans modules which developers should run before pushing their changes to the Mercurial repository. Remember, your bugs may disturb other developers around the globe.

The Commit Validation Suite can be extended by any module whose functionality the developers consider vital and should be tested before every commit. You can read more about commit guidelines here.

Unit tests

The NetBeans developers and QA team invested a lot of time in implementing an easy-to-use framework allowing to write and run unit tests. Nowadays I doubt there are any developers who have never heard of unit testing or think that writing unit tests is a bad thing. Unfortunately, the reality is that even those developers still do NOT write any tests. Why? They haven't discovered the benefits.

Why write unit tests?

Tests represent the most practical design spec possible - Unit tests constitute design documentation that evolves naturally with a system. This is the Holy Grail of software development, documentation that evolves naturally with a system. What better way to document a class than to provide a coded set of use cases. That's what these unit tests are: a set of coded use cases that document what a class does, given a controlled set of inputs. As such, this design document is always up-to-date because the unit tests always have to pass.

Tests give a developer confidence in the code - you can always check whether your change breaks something or not by running tests. Well-written tests should catch all potential regressions.

Tests are the lab - when testing your code this is usually done in laboratory environment which eliminates dependencies and side-effects caused by other parts of code. You can test each functionality separately which will allows you to quickly track the problem. You can't expect your code to work in the production environment if it doesn't in the lab.

Tests speed up refactoring - by coding a set of use cases into your tests you can easily verify that you are not compromising any functionality or causing incompatibilities when refactoring your code. Well written tests should catch all potential incompatibilities.

Tests help to fight memory leaks - it's usually hard to find memory leaks in a program, and it's harder when the program is big and complex. Running tools like OptimizeIt doesn't necessarily help in all cases and it's usually slow. It's better to write tests for isolated pieces of code (subsystems) which check whether your objects are really getting garbage collected when they aren't used anymore.

Tests will make your boss more happy - by writing tests your code will be more stable attracting more users, making the product more successful. That's what all bosses like!

How many tests do I need to write?

All of them, and it still will not be enough! Seriously, you should write as many tests as possible, ideally the tests should cover all the code in your module. Practically this is hard to achieve and software projects usually define some lower level of code coverage as sufficient. However, the code coverage by unit tests should be at least 70% for production quality code. This is a high number!

The problem with code coverage by tests is that even 100% coverage doesn't mean that there are no bugs in the tested code. This is because your code behaves according to the state of memory and thus you shouldn't only exercise each line of your code but exercise it for each possible state of memory. Obviously threading issues bring even more complexity to the problem.

When to write Unit tests?

In NetBeans, currently we have a lot of code which isn't covered by tests. Trying to write tests for all this code sounds like a never-ending task and in fact it's not really practical to stop all work and write tests for all the code now! Instead we should establish few rules about how to continuously improve the situation and increase the number of code lines covered by tests. And here it goes:

All newly written code should be covered by unit tests

When fixing a bug, a test should be written to simulate the bug and then verify that the fix really solved the problem.

Whenever you feel tired by your current work try to write some tests for existing code which isn't tested enough yet (there is a lot of untested code overall in the NetBeans sources).