Having push access to the NetBeans Mercurial repository is a privilege
but
also a responsibility: responsibility to the quality of the code
and responsibility to other people working on NetBeans. In order to
ensure that the quality of the code base is high and that you will not
disturb anybody else's work, here are a few guidelines that will help
you to validate your changesets before you push them into the NetBeans
Mercurial repository.
When your modifications are ready to be pushed do a clean build
to make sure that it does not negatively influence other
modules. Execute
ant clean
to clear the sources and
ant
to compile the NetBeans IDE. Currently we are building with JDK 1.5, so
you
can use all its methods, but be careful not to depend on anything
newer.
Execute commit validation test suite which is designed
to test the basic functionality of NetBeans and shall affirm you
that your integration is not completely broken. Execute
ant commit-validation
Of course no test shall fail otherwise there is something wrong -
probably with your
integration. You should evaluate
why it fails.
Execute unit tests that might be influenced by your changes. For
example when committing into the form
module its test shall be run by
ant -f form/build.xml test
and shall not fail.
If you use Linux, you can run a premade script that does the
clean build and test suites for you. Try nbbuild/misc/full-build-linux.sh
(see the shell script comments for usage instructions).
If you really want to verify everything that we verify on continuous build, run:
ant -f nbbuild/build.xml commit-validation
-D"xtest.userdata|com.sun.aas.installRoot"=<glassfish install dir>
-Dwtk.dir=<path to wtk22_win.zip or wtk22_linux.zip>
-Dxtest.config=commit-validation-all
For detailed description of these tests look at this
document.
If the IDE compiles, tests pass you are ready for integration.
Check what you have changed by evaluating the output of hg out -p
and if the output really contains what you have
modified do hg push.
Evaluate commit
validation results
Once you have executed commit validation test suite you have to
evaluate its results. If you run it from command line, at the end it is
printed out summary information about pass ratio of the suite. There is
also provided a link to HTML presentation of the results. If something
fails, you should follow this link and check error messages and log
files to determine what is wrong with your build.
Main page of the results is split into two frames. The left navigation
frame shows you links to results of particular modules and also list of
modules which failed. The right frame shows summary of results in
several levels.
The shortest way to see the reason of failure is to click on module
name in Failures report in the left navigation frame. Then you can see
name of failed test case and error message. If you need more details,
click on its name. You will see test cases for whole test suite and
failed test in red color. To see exception stack trace follow the link
in Status column. There might be also additional proprietary log files
in working directory of the test case. In that case there should be
"Yes" link in Workdir column. For UI functional tests (module ide) are
useful jemmy.log with trace messages of performed operations and
screen.png containing B&W screen shot at the time of the failure.
If you are not able to identify reason of failure, or you think failure
is not caused by your changes, please write an email to
.
How to quickly find useful things from results main page
messages.log and similar
click on module name, then testbag name - there should be link
IDE Logs which points to messages.log
ant logs
follow the link Logs from build scripts - you can find there
compilation errors etc.
test case error message
click on module name, then testbag name, suite name - you will
see list of executed test cases and failed one in red color with error
message
exception stack trace
click on module name, then testbag name, suite name, link in
Status column - you will see stack trace of exception which
caused that test failed
jemmy.log
click on module name, then testbag name, suite name, link in
Workdir column, jemmy.log file - it contains trace messages of
performed operations and it can be useful to know the last operation
before the failure
screen.png
click on module name, then testbag name, suite name, link in
Workdir
column, screen.png file - it contains B&W screen shot at the time
of the failure
Add a new test case into commit validation suite
To add a test case into the commit validation suite you have to modify
module's tests config file and also xtest's master config if module is
not already included in the suite. If you need more information about
XTest, please visit the http://xtest.netbeans.org/
site.
Reasons to add a test
The commitValidation suite shall serve as a global check
suite, that
will try to catch possible regressions spread over wide range of
functionality.
For example a modification in org.openide.text.Line can
affect
the correctness of debugging of JSPs. It would be the best to write
correct unit
test of the Line but if that is not possible, than the JSP
module developer is encouraged to add a test for the correct JSP
debugging functionality into the commitValidation tests
of
the JSP module. That way he will help the core developer
recognize
that there is something wrong before the commit is done.
Another example might be related to correct garbage collection. The
whole
system is designed to free itself from memory when not used, but small
mistake
in any module can prevent this feature to function correctly. As this
can be
broken by any commit in any module, this is a reasonable test candidate
for a
commitValidation suite.
In no way the commitValidation is a replacement for unit
tests. Each
module is responsible for its own set of tests for basic functionality
that
shall be executed before any change of the module code. The
commitValidation suite cannot contain such tests as it would
not scale
and we need it to execute quickly as it shall be run before every
commit.
Think about that before extending it - if it is not reasonable to
provide
validation of an important functionality on a level of unit tests,
then commitValidation is the right place. But try to minimize
such
cases for the sake of scalability.
Modify module's config file
Create a new test bag with attribute commit and add test cases in it
which you want to be executed in commit validation suite. The test bag
should look like this:
Above example is taken from openide/test/cfg-unit.xml
in openide module. If test bag with commit attribute already exists
in config file, you only need to add your test cases by adding an
appropriate include
element.
Modify XTest's master config file
You have to modify master config file only if your module is not
already included in commit validation suite. In that case simply add a
line to config named commit-validation-nb.
Master config is located in xtest/instance/master-config.xml
and commit validation part looks like this:
If you participate in a development, you should be subscribed to
.
Failures in a daily builds as well as continuous ones are notified
there and if your integration breaks something, you can react
quickly. If you are going to fix such problem please reply to the
mail, so others know that it is being solved.
Test and test tools are also just a software this means if
you find bug in them please fill it. If you don't now which
Issuezilla module it is then go to http://testtools.netbeans.org/
where you can find more info,
If you don't know how to evaluate commit validation suite
results, read above.
The automatic tests opens the IDE and popup and close various
windows which prevents me from work. How can I prevent it? Easy on
Unix - start new X server (Xnest is useful) and execute
the
validation test there. Using the full-build-linux.sh
driver script, just pass spawndisplay=yes.
Usually one is commiting change that has a record in Issuezilla
(bug, some task, enhancement). In such case mention the issue number
in the commit message.
hg ci -m '#31043: Fixing the screen resolution'
and include the changeset hash of the commit into the description field
when resolving the issue. Then one can it is possible to find out
more about the commit in the Issuezilla and it is possible to find
the sources that were modified to resolve the issue.
Common mistakes
Do not check in debug code that depends on a modified API
with something made public that isn't Solution: Always
do a sanity diff
Check in a class that also contains a change that's part
of a different bug fix Solution: Always do a sanity
diff
