I think that java guys are working on this.

Can someone point me to info on what is being done?  Thanks,
Tom

Perhaps more important than its absolute lack of speed is that relative to other IDEs, specically Eclipse, NB is much,
much slower. Reports are that it is five to ten times longer to make small changes, and get back into the code.
Performance of the edit/compile/debug cycle is critical for programmer productivity, if it is too long, you lose your
place in the code. Reference F. Brooks "no silver bullet"

To be specific, the speed of run-single and test-single (IMO the most frequently used run cases) has to be improved. 

The lack of speed for test-single practically kicks NetBeans out of any shop that uses Test Driven/Agile approach.

 

Same here. As I already offered for issue #115996, I can provide (under NDA) our project for testing.

Is there any particular reason this was placed in java/source? AFAIK Retouche has nothing to do with this.

Reporter left Version field unspecified and no comments say anything about which version of NB is being discussed.
5.5.1? Some 6.0 milestone, or Beta 1?

This issue was filed as a result of nbusers discussion: http://tinyurl.com/23s62l
In the first post dated by May 2007, the reporter states he's running daily builds of NB6.

This is a problem with all versions of NB that I have access to (5.5x, 6.0) and,
I suspect with every version since NB started relying on ant for all build related
matters (4.0?)  Since there is no choice for "All", I left it blank.

I have no clue what "Retouche" is, so I can't comment on my issue's relevance to it.
All I know is what I indicated in the issue: NB is at an order of magnitude
slower than any of its competitors I'm familiar with (Eclipse, IntelliJ) in the
simple edit->compile->debug cycle on any project that has >2000 Java files in it.

My guess is that solving this issue would require hacking into Ant/javac and removing the overhead of scanning the
source tree for changed files by feeding it the information that NetBeans has in this regard.

It's more generic problem not specific to j2se project, the overhead is caused by the Ant, probably ant/project is the
right component. Java/source cannot help to solve this.

The problem may be with ant overhead, but some possible solutions may lie elsewhere.  It seems that a lot (most?) of the
use/advantage of ant comes from packaging and post processing flexibility, and that this doesn't come into play for most
edit/compile/debug activity. For this, its all about the "*.java-->*.class" translation needed when "DEBUG" button is
pushed.

What dependency information and compilations can be kept/calculated in the background?  Highly important is knowing when
cached info should be thrown out and a full build can be done. I'd be willing to bet that some/many users would be
willing to turn on a "go-fast" option, that didn't completely guarantee all dependencies and occasionally required a
manual full build, or if that's not acceptable from a user experience perspective, seems there are still many cases
where go-fast would be reliable.

Just put into j2seproject for evaluation purposes - perhaps <javac> is being used incorrectly, perhaps javac is slow,
perhaps Ant is invoking javac poorly, etc. ant/project does not contain any code related to compilation so it doesn't
make sense there, and ant just provides the integration but also does not contain any code related to compilation.

There are of course various possibilities for running a compiler differently, if that would indeed help anything (yet to
be confirmed), but a major change like this is very likely out of scope for 6.0.

err-

As a user, I for one would expect the "background calculation of dependency" for test/run/debug-single to be accurate,
WITH the "complete guarantee". In case if that cannot be guaranteed, I'd expect a pop-up saying something like

"IDE can't be sure if this will work out: Recommend do a full build, or proceed at your own risk".

I don't know how exactly this can be done (if at all). But per my understanding of Ant, it's difficult to use Ant for this.

I mean he ant/project as a common for all java based projects because this problem is not specific to j2seproject but to
all java projects (j2se, web, j2ee, api support - even more due to module scan).
1) <javac> is used incorrectly - in case of default platform the javac runs in the IDE vm - probably fastest way.
2) Javac is slow - javac has some performance problems, mostly caused by the nonefficient IO, but the JavaFileManagers
were optimized but I am not sure if these fixes will go into JDK 6.0. But IntelliJ uses the same javac for building, so
javac is not a problem. Yes, we can provide faster JavaFileManager.list() but we will need to modify ant <javac> (when
running from IDE it will use CachingFileManager) but I will not definitely do this for NB 6.0.
3) Ant invokes javac poorly - I hope that the Ant in case of default platform just calls
com.sun.tools.javac.main.JavaCompiler

The next question is the performance of <depend>.

The flip side of my previous mentioning of "guarantee": re-build everything possible is certainly not the right way to
achieve that, which is the whole point of this IZ.

To wqtnetbeans: The IDE internally knows the dependency among files, when you save a file the IDE indexes the file and
also dependent files in the same project, but the problem is how to provide this information to the ant build system.

Maybe my message from the mentioned nbusers thread could shed some light on what's going on or give some directions for
investigation.
--------------------------------------------------------------

Here's a log directly from NB 6.0 console, when building a web project of 3200 source files (without generating war).

init:
deps-module-jar:
deps-ear-jar:
deps-jar:
library-inclusion-in-archive:
library-inclusion-in-manifest:
compile:
compile-jsps:
do-dist:
dist:
BUILD SUCCESSFUL (total time: 1 minute 2 seconds)

As you can see, there were no changes, nothing to compile, no work done. Still, it took 1 minute. Admittedly, when
building with a warm cache, things usually go more smoothly: four consequent builds took 9, 11, 9, 9 seconds. I would
guess changed files scanning is to blame. OTOH, Eclipse with its built-in compiler probably doesn't have to scan through
all your source tree, if it assumes that all the changes are done from inside the IDE.

So it gets down to integration with compiler. In NB, there's no integration: when you press F11, it runs a separate Ant
process that needs to check every .java file and compare its timestamp to the .class file's timestamp. Thus, in order to
improve performance (assuming I'm correct guessing that change scanning is the culprit), it is necessary to make
coupling between NB, Ant and javac more tight. One way to do it would be providing Eclipse-style background compiler
that bypasses Ant completely. OTOH, maybe some kind of clever integration with Ant and javac could keep Ant's
advantages, while reducing overhead, by letting Ant look at IDE's list of dirty files instead of scanning everything.
--------------------------------------------------------------


Later, I enabled printing verbose Ant info and redid the same test. It showed that -do-compile target took the longest.
I think I could trace it even deeper into the targets, but I wasn't sure, in which of two consequently printed targets
the slowdown occurred (is it the target whose name was printed before the delay or after?). Anyway, this should be easy
enough to determine by taking any large project and pressing F11 repeatedly without making any changes.

To tzezula: actually, running the compiler in NB process has two big disadvantages, when working a large source tree:
1. , you need to add significant amount of memory to NetBeans purely for compiler's use, just so that it won't run out
of memory when compiling
2. it considerably slows down the IDE in comparison with running it in a different process (at least on my WinXP laptop)

I have recently fallen on this, when switching from 1.4 compiler to 6.0, and didn't find anything better than
registering a fake JDK6.0 platform in NetBeans (which actually was configured on the same JDK directory) just for
compilation purposes.

You can easily switch to an external invocation of javac by adding

build.compiler=extJavac

to project.properties. I think this is in the FAQs somewhere.

Running Ant with -v etc. is not likely useful; need method-level profiling for "warm" runs, plus additional tracking of
I/O (D scripts??) for "cold" runs.

Modifications to Ant would probably not be necessary simply to restrict the list of files considered for compilation
(under the assumption that this is the root performance problem, and not something totally unrelated); the IDE could
pass an includes list of recently touched files, and I think Ant's DirectoryScanner is smart enough to not examine other
dirs when a list of exact paths (w/o wildcards) is passed to it. The real question is how to make the list be correct
often enough to not produce a dozen new P2 bugs. The IDE can certainly track any file modified in the editor and saved,
but there are other ways source files get touched. (Consider doing a CVS update after a colleague has checked in changes.)

> the IDE could pass an includes list of recently touched files
The list of changed source files can be rather big, the files can be modified en masse by things like source control,
search/replace, refactoring, etc. Is there an efficient way to pass several thousands of file paths to Ant without
modifying it? Will Ant efficiently handle them internally?

> Consider doing a CVS update after a colleague has checked in changes.
AFAIK, NetBeans' javacvs module uses the same filesystem API as the editor, so unless you're using an external CVS
client, intercepting all changes at this level should do the job. Of course, if you do use an external tool, you're in
trouble. But this should be no different from the Eclipse ways, and people live with it happily. Though I guess Eclipse
does additional background scanning of the source tree continuously or from time to time (or is there some native API
for getting notified on changes in a directory tree?).

Perhaps a simple per-project or per-source-root, or per-source-tree (or all) configuration flag could be used to let the
user explicitly specify, how to treat the sources? If it's off (default), the IDE would assume no external tools touch
the files and do fast compilation, otherwise fall back to the old way. Power users that rely heavily on external tools
would switch it on, everyone else would just be happy with the fast compilation.

Additionally, I would suggest resurrecting the "Refresh" option in the directory context menu (I think it was there
around NB 3.0, but maybe I'm wrong) that would tell IDE to recheck dirty files itself. This is for users, who do most of
the work from IDE, but occasionally need to use an external tool. Only show this option on source trees, for which the
flag is off.

Some of the external tool use cases might be handled without the need to fall back to all-tree scanning, when the
external tool is launched by an Ant target (CVS update for example) from the IDE. This can be handled by wrapping a list
of well-known Ant targets in custom logic (or modifying them) that would determine (maybe even approximately), which
files were or could be changed by the command and marking them dirty.

> a list of well-known Ant targets
Err.. this was supposed to be Ant _tasks_

> You can easily switch to an external invocation of javac by adding
> build.compiler=extJavac
> to project.properties. I think this is in the FAQs somewhere.
This is offtopic, but is there any chance to make it a GUI configuration option? If an active user with.. 4+? 5+? years
of NB experience (that would be me) didn't know this could be changed via properties, it's really obscure :)

It appears that NB does scan for external changes to source trees.  At least it refreshes it's view when I add or delete files outside of NB.  So combined with 
passing an explicit includes list to the Ant javac task, I would think some improvement could be made that is relatively low risk, Particularly for this use-case 
of speeding up the edit->debug cycle which is the big boat anchor holding back NB.  This one issue is the most significant reason that I have seen for people 
switching away from NB.  The loss in productivity caused by the relative slowness of the NB edit-compile-test cycle is enormous.
It's great to see this getting some attention.

> To wqtnetbeans: The IDE internally knows the dependency among files, when you save a file the IDE indexes the file and
> also dependent files in the same project, but the problem is how to provide this information to the ant build system.

I see. If that is the case (again this is from a layman's perspective), rather than trying to tackle the difficult route
with Ant, why don't we forget about Ant and use an easier way to do small tasks like test-single? Ant was born to do
bulk build anyway (replacement of make for Java). There must be some easier way to handled this kind of deal: Eclipse
and IntelliJ don't use Ant, and they handle this kind of small tasks much better....

I think the evaluation in j2seproject is over.

I will do the dtrace evaluation (IO) and attach results, unfortunately I was not able to profile ant process.

Created attachment 49422 [details]
d trace script to measure io time

Created attachment 49423 [details]
dtrace output

Comments about the dtrace output:
It measures the real io time (when the file content is inside a cache it's not logged). Both runs were on a "warm"
project the project was already built and cleaned to make the numbers comparable. The tested project was a j2seproject
for java compiler (327 source files). The io time was 4 sec from 25 sec of the build ~ 16%.

Thanks for the I/O evaluation. It doesn't seem to be *the* problem (while 3s of I/O is still nontrivial time in case of
hot caches).

> unfortunately I was not able to profile ant process.
To profile (external) ant process, open ant sources, open org.apache.tools.ant.launch.Launcher, right-click on the main
method and add it as profiler root method. Then you can follow the Attach Profiler wizard (changing the attach method if
already set) with following settings:
* Application
* Local
* Direct
---
* JDK1.6
and copy the agent setup string from the last wizard step. Then start the ant process using the agent setup string
inside ANT_OPTS, in my case:
ANT_OPTS="-agentpath:/opt/java/NB/profiler2/lib/deployed/jdk16/linux/libprofilerinterface.so=/opt/java/NB/profiler2/lib,5140"
ant

Then you can finish the wizard and perform attach.

I have received a private email with response to my question regarding the native APIs to get notifications of changes
in files/directories. Turns out various OSes indeed have APIs for that, where some allow recursive notifications
(Windows, at least), and others don't (Linux inotify). Solaris supports some kind of similar mechanism through /dev/poll
and something called FEM.

The sender of the email argued against making NB dependent on native APIs, which I certainly understand. But I think
it's important to point this out, since what is unacceptable for NB, might be acceptable for Eclipse. I can see how,
through the use of efficient (read: native) notifications APIs, one could blend the two kinds of notifications, relying
primarily on IDE information regarding dirty files, but also responding (probably somewhat slower, but still at
acceptable speed) to changes in files made by external tools.

The difference it all makes is whether the path (having two changes detection modes) that I was suggesting earlier is
proven in real life or not. If Eclipse does in fact use native APIs, then it goes a different path.

Native notification is irrelevant in this case.
NetBeans infrastructure does keep the built status anyway and pretty accurately. It needs to do so for keeping the java
metadata accurate.

Moreover, once the native notification is available (and it should be in JDK7), filesystems API will switch over to it
and everybody would benefit w/o any rewrite.

pnejedly, I'm concerned with the case, when an external tool (e.g. cvs client) modifies .java source files, but NB
doesn't know about it. Is there currently a mechanism that watches over the entire source root(s) for externally-made
changes? If not, having a fast and reliable notifications mechanism would make a difference.

> Is there currently a mechanism that watches over the entire source root(s) for externally-made
> changes? If not, having a fast and reliable notifications mechanism would make a difference.

Unfortunately there is not such mechanism. There is already JSR 203 for it.

Excuse me butting in, but I'm sure it would be acceptable - and help solve the problem faster - to expect/require all
changes to be made within the IDE. If someone does mass changes outside Netbeans that's their lookout.

Well, there is one. Not 100%, but should cover most use cases - the IDE refreshes (checks) everything it cares about
during focus switch from external app into the IDE.
It covers using external versioning client, but doesn't cover e.g. if you let the client perform large update in the
background while you're already working inside the IDE. The IDE will eventually catch up though, as soon as you'll check
your mail client, for example ;-) 

I agree - I'm only guessing here, but I would venture to say that most developers
work entirely within the IDE - at least that is the case for the 15 developers in
our group.  If they do a CVS update, they do so from within Eclipse (or NB, in my
case), so the IDE would be aware of what changed.

I guess you guys are trying to solve the performance problem generically, but from my
simplistic user perspective (and the reason I wrote the issue in the first place),
in a edit->compile->debug cycle, I would be extremely happy if you could simply
compile the 1-2 files I've changed, replace them in the already existing jar,
and run!  Doesn't seem all that complicated and should be doable in 1-2 seconds.
That's all I'm really asking for!

I care about dependency checking and about external tools changing the source when
I'm doing a full "build".  I don't care about this, when I'm changing a single
String in a single Java file and just want to see that new string appear on the
screen.

At this moment, when I hit "Run Main Project" after a clean/build, I see:
init:
deps-jar:
Compiling 8 source files to /home/twolf/nb/neptune/build/classes
compile:
run:
This takes about 15 seconds on my 3400 file project.  If I Run Main Project a
second time, I see the same thing (I have no clue why 8 files are being compiled).

On a totally unrelated note: this issue had 25+ votes before it was moved to
another sub-component.  Now it has 21 - it's really annoying that people have to
recast their vote every time an issue is moved (or marked a duplicate of an
existing one).  Is this a problem with the bug reporting system or with the
way it's been setup.  If it's the latter, I hope it can be changed.

Thnx,
tom

So you're saying each time I bring NB to focus it goes over all 3500 source files in my project and checks their
modification dates? One could argue doing this at a constant interval + on demand would be a better solution. But
actually I don't notice any slowdown on focus, are you sure it really does that?..

Anyway, I did a quick search on how Eclipse handles notifications, and here's what I discovered: it has a configuration
option that enables/disables notifications functionality. It currently uses the win32 native API on windows and a Java
polling mechanism on other OSes.

http://www.eclipse.org/articles/Article-Resource-deltas/resource-deltas.html
http://wiki.eclipse.org/index.php/FAQ_When_should_I_use_refreshLocal%3F

This is the "not 100%", the IDE checks only timestamps of files it has FileObjects for it (someone hods a reference to
this file) not for all 3500 files.

I see. Could this be 1% in the average case? :) I mean, it holds FileObjects for open files, probably also ones
currently visible in the expanded directory trees in Files view and project configuration files, but all this is much
less than the overall amount of source files. So, I guess, there's no general modification polling mechanism to speak of.

Regarding the Java7 changes - this is very good news, and would probably solve the problem with external tools The Right
Way, but 2009 release date is more than a year away.

So, to reduce change scanning time (presuming native APIs aren't an option), the following things can be done:
1. (mandatory) change the way of doing build. One of the following:
1.1. make NB pass its list of dirty files to Ant/javac
1.2. Eclipse-style integrated compilation, bypassing Ant
2. (optional) provide a configuration option to choose between the traditional Ant-based compilation and the improved
one. The option can be configured on one or more of the following levels: global, project, source root, directory tree.
3. (optional) background thread that polls the controlled projects/source roots/directory trees for changes. Again,
configuration options can be created for configuring it on certain level and overriding on lower levels.
4. (optional) "Refresh" item in context menu on a directory to force rescan.
5. (optional) intercepting calls to a set of known ant tasks; estimating, which files/directories could be changed as a
result and automatically refreshing their modification status.

This is to follow the comments of "twolf2919" in terms of internal/external change tracking: 

IMO NetBeans should focus on 

1. Resolve the "simple" and "internal" tracking first, and catch up to the running speed of other IDEs. 

2. At the same time, provide a "refresh" option to the user, in case she made "external" changes. Some slowness while
doing this is perfectly acceptable. And, if it is hard to achieve, this doesn't even need to automatically detect
external changes and ask the user if she wants to refresh, as Eclipse does. As someone pointed out earlier, if you
change things outside the IDE, it's your watch-out. However, it's vital for the IDE to give a "refresh" option.

A side note: External changes happens very often when comes to generated code - JAXB binding, MDA code generation (such
AndroMDA), ORM code generation such as Hibernate Tools.... Currently NetBeans does not support this kind of stuff very
well. ex. External changes is not even supported by the Subversion module, which is very odd. 

http://www.netbeans.org/issues/show_bug.cgi?id=105802

 

Hmm, everybody comes with a solution without having really analyzed the problem.
Please stop posting solutions before we have relevant profiling results and analysis of what's doable with architecture
in place.

For the record, the people that I'm aware of in our shop using netbeans use an external subversion tool.  I think it's
very reasonable to do a scan on task switch, as someone else commented is the current behavior.  It's a fact of life,
people use external tools... please don't break the IDE for those people.

To pnejedly:

>Hmm, everybody comes with a solution without having really analyzed the problem.
>Please stop posting solutions before we have relevant profiling results and analysis of what's doable with architecture
>in place.

If you are referring to my earlier 2 points of comments, they are not solutions, rather they are requests from a
NetBeans user.

 

In response to jn's last comment: my hope would be that the current build system
become as fast as other IDEs when it comes to small changes.  If that is not
feasible, given the dependence on ant and how it does things, then I'd be
perfectly happy with a new button, call it "fast build" or something that will
forego all the usual build checks and simply patch the executable jar file(s)
with whatever small changes were made.  I imagine this could be done outside
of ant - and, hopefully, very fast.

I think pnejedly's comment was rather pointed in my general direction. Well, I fully accept the blame for running ahead
and proposing solutions to a problem that hasn't been formally analyzed yet. Burn karma, burn.

Prior to 6.0 beta I have been using 5.0. Running a file (Shift-F6) in 5.0 compiled only that file. In 6, that is no 
longer true - as I understand, it compiles all classes in the project. After digging around in the 5.0 build-impl.xml, 
I changed the depends of 6.0's run-single target (in the Execution Section) from "compile" to "compile-single". This 
resulted in a compile of only the file being run, duplicating the behavior of 5.0. 

When I first inquired about this I was referred to some earlier material that was said to be the cause of the change 
from 5.0. However, reading these posts, I wonder if it was really necessary to change the behavior for run-single, 
since the real problem appears to be involved with insuring the integrity of projects - and maybe that was 
accomplished at the expense of usability and turnaround time for a single run. 

The behavior of run single was changed by excludes/includes from source roots. It is a workaround preventing the java
compiler to follow the transitive dependencies into the excluded subtree.

Created attachment 49768 [details]
Script to create compilable ~5400 *.java + 20 *.jar from nb_all, as a test case

I found a dumb inefficiency in Ant which when fixed seems to shave off 20-30% of the time from a no-op
(non-)compilation. Fix will be in Ant 1.7.1. (Which might or might not go into NB 6, depending on when it releases.
Looking unlikely.)

Good news! Do you have the link to Ant bug tracker?

This raises the question, whether Ant is pluggable into NB. Ideally, it should be possible to simply replace the
installed Ant module with a different one. So someone would take Ant 1.7.1, its beta or even patched 1.7.0, wrap it in a
module and distribute it through the UC.

It is good news - but I'm currently waiting 15 seconds for a no-op build to finish.  With the fix, I'd still be waiting
10 seconds - which is 9 seconds too long (in other words, I wouldn't even notice the improvement).

I don't think ant improvements or tighter integration with ant are going to cut this delay down to the sub-one-second
range where Eclipse and IntelliJ are.  So why spend time/resources on it?  Didn't NB used to do its own builds in the
3.x days?  Couldn't some of that code be revived to handle the simple case (edit a couple files, compile, run)?

tom

No bug filed for the optimization in Ant. You can replace the bundled version of Ant as a regular user in the Options
dialog. If 1.7.1 is released after the cutoff date for 6.0 we will probably release it on the update center.

Playing with the large sample project previously mentioned, it seems that scanning for modified sources takes some time,
but not such a huge amount. Just for <javac>, on those 5000+ files a no-op build with warm disk caches is maybe 3
seconds for me, which seems manageable. You don't even need to do a full build all the time; F9, S-F6, and C-S-F5 let
you compile, run, and debug single files with <1 second overhead (and if you think this doesn't work like you want and
you are running 6.0 beta 1, get development builds because there were several other changes in build-impl.xml recently).
Also (to a previous comment) you *can* reload files in a running program using NB without doing a build - just run the
app in the debugger and use Run > Apply Code Changes.

What is slow:

1. <depend> adds some overhead. In my experimentation, it will roughly double the time taken for a no-op compile. (This
is probably sensitive to size of classpath etc.) Of course it can also save you time overall if it notifies you early of
incompatible changes you have made, but this matters much more for some projects than for others.

2. Running <jar> can be quite slow if the project is large. Even if the JAR does not need to be touched, it is fairly
expensive to compare the *.class timestamps to the existing JAR entries. If the user is only running the Build action in
order to make sure all classes are built, e.g. in preparation for running something in the debugger, then this time is
wasted. (See also issue #46721, an old RFE.)

Still have the option for the future of passing a restricted fileset to <javac> based on an IDE-maintained dirty list if
that proves to be useful, but a much simpler and safer solution for 6.0 is to permit the user to disable dependency
checking and/or JAR generation on a per-project basis. Proposed patch adds checkboxes correspondingly to the Compiling
and Packaging panels of the project. Default values are on for both; setting stored in private.properties (i.e. treated
as a personal configuration choice if working in a team). Note that JARs are still generated for subprojects - turning
off JAR building just makes Build (or Clean & Build) run the 'compile' rather than 'jar' target.

I will add this to http://wiki.netbeans.org/wiki/view/NB6LateUIChanges and see if I can get approval for the UI change.

Created attachment 50053 [details]
Proposed patch

Created attachment 50054 [details]
Screenshot of Compiling panel

Created attachment 50056 [details]
Screenshot of Packaging panel

About the "build jar after compiling"... I'm not entirely clear on what targets/menu-items are affected by this. Clearly
the Debug menu item doesn't need the jar.

Is there (should there be) a menu item for build-distribution. Should the jar/no-jar decision be based on a menu item
rather than a project parameter?

How about an obvious indicator for "jar out of date", like some button, eg build, having a red background.

I understand that at this late date it's difficult to do anything for nb6. But avoiding those dozen P1 bugs...

> How about an obvious indicator for "jar out of date", like some button, eg build, having a red background.

Of course a different project icon in the Projects window makes more sense.

I think I'm hung up on wanting dynamic control (menus) rather than static (project parameter).

Jess, "Apply Code Changes" seems to work on all the small changes I've tried (adding a printf to single file; trying
multiple printfs across multiple files; trying a Matisse-based change) - fantastic, so far.  Is there a document that
describes the limitations of "Apply Code changes"?

Thanks,
tom

1. I have some past experience with building projects without checking deps and know that it quickly gets frustrating.
So I won't switch the checkbox on and keep it that way. Thus I support err's suggestion of having two build actions -
with deps and without them. This will allow to skip checking after small changes, but still perform it in other cases.
2. I think that adding the checkbox and the special build action can only serve an intermediate purpose. I'd rather wait
for the full-blown improvement to enter in the 6.1 release cycle, than have the bug closed with half-measures.

The "Build JAR..." checkbox simply controls which target to use for the Build (and Clean & Build) actions - jar, or
compile. It has no effect on debug etc., nor on subproject builds.

Introducing new major actions (menu/toolbar/KB) is out of the question for 6.0 and probably not a good idea ever. The
context menu could easily be made to differentiate build with and without JAR, but this would lead to confusion as to
which corresponds to the global actions, since it would vary depending on project settings. Differentiating builds with
and without dependency tracking would likely be confusing as well. If you do like to use dependency tracking, it is
probably best to pair it with F9 for the minor changes.

Adding dynamic status display for the output JAR could introduce new performance problems, and would likely be baffling.

Off-topic, but Apply Code Changes uses JPDA reload and is subject to its restrictions - no class signature changes.
Enabling broader reload capabilities is an active research topic for people interested in the JVM.

> Adding dynamic status display for the output JAR could introduce new performance problems,
Seems like a build that does not generate a jar set's the no-jar flag, and one that generates a jar clears it. When in
doubt, set it. Don't see a performance issue.
> and would likely be baffling.
Perhaps. But it would never be seen unless the "Build JAR" checkbox is unchecked, in which case the user takes
responsibility for being an expert of sorts. I would argue it is less baffling than working with a jar that is out of date.

> the IDE could pass an includes list of recently touched files ... to Ant.

Was it possible to get an estimate on how this would affect performance of build?

to err: The IDE (java/module) tries to track the changed java source files, it also provides API to notify about changed
java types. As discussed above in this issue the only problem is with external changes which the java/module cannot
detect, but it may be solved by some action causing refresh of sources in project.

*** Issue 117579 has been marked as a duplicate of this issue. ***

The attached patch allows users to turn off depend and jar creation. The issue was originally reported to NB 5.5 which
hadn't support for depend (turning off depend only removes NB 6.0 regression). Turning off jar file creation helps to
j2seproject but it will break build when someone depends on this project, for example web application needs to copy the
jar into war file. From the UI perspective I am not sure if these options,especially the "no depend" which is rather
implementation details, should be a part of the project's customizer, the customizer misses options like java.target
because of simplicity. Probably setting them in project.properties and well documenting them will be enough.

Again: the proposed patch *does not* affect subproject builds. If a WAR projects depends on the j2seproject, it will get
its JAR regardless of the Build JAR setting.

Not including the new options in the GUI effectively means no one except those on CC here will ever use them. If that is
the goal, the patch doesn't need to be applied at all - anyone can already override whatever target in their build.xml
to tune the build process however they like, in any IDE release, and it is quite simple to write up instructions for
doing so in the wiki or bundled help. The point is that the _default_ build script, as easily customizable using only
the GUI, should be at least adequate for most users, even if they happen to have large projects. It will never be
perfect for anyone, and will always be poor for a few people with unusual setups.

Regarding "JAR-out-of-date" flag: implemented as you suggest it is possible with no performance impact, though it will
not always be an accurate representation of whether the JAR exists and is up to date - does not account for external
builds or even direct invocation of Ant targets not through the project action UI, and may not be correct after
non-build actions such as run-single which can do compilation but usually would not. I am not sure I see the real
purpose. Most IDE actions do not rely on the JAR anyway. Subproject builds would always create it. Any custom build
targets written to expect the JAR should simply depend on the jar target and will always be correct. For remaining cases
where you simply want to build the JAR and will then use it somehow outside the IDE, it seems simplest to just enable
the checkbox and build (or run the 'jar' target straight from build.xml).

Passing a modified list to <javac> remains an open option to investigate. I am not sure this can work with <depend>
(since sources which were not touched and thus not in the file list may need recompilation) but I will check. My main
concern is whether the IDE will reliably keep a reference to every *.java FileObject: automatic refresh of timestamps
after switching back from an external process is reliable in my experience, but you need to be holding the file
reference in memory or the event is lost. (Adding accurate change notification using e.g. inotify is not an option for
6.0 and probably not for 6.x.) I know that changes to arbitrary files in the project, e.g. some test data, are not
noticed unless you happen to have expanded that folder. From a quick experiment, it seems that java/source does not
necessarily hold onto all sources in a project, either (which is probably good from a performance perspective!): if the
package is not visibly expanded, and GC has occurred since project opening or other usage of the files, external changes
are not detected until the package is expanded. That means that some external changes (e.g. big CVS update) might modify
a lot of Java files that the IDE does not know about, causing the next "full build" to erroneously fail to compile most
of the real changes. Whether this would be a serious problem and whether there is some effective heuristic to address it
in most cases, I do not know yet.

I have introduced some further optimizations for Ant 1.7.1 after doing some profiling of I/O calls for no-op builds and
noticing that it made a lot more system calls than it really needed to. On my test case, running the compile target with
<depend>, Ant 1.7.0 takes 8 seconds while the dev version of 1.7.1 (or 1.8 alpha) takes 5 seconds. Without <depend>, it
is 3 seconds down to 2 seconds. So some improvement.

Out of curiosity: did you save commit logs for Ant fixes?

Ant SVN revs: 581753 in 1.7.x (merge of 581748) for I/O optzns; 581395 (merge of 581394) for CPU.

Option to produce no JAR file would finally close issue #46721.
Regarding dependencies for single compiles, I think the NB 5.x way would be nice, a complete new approach without Ant
would be even better.

Just a little visual nitpicking .. could the two checkboxes be grouped together? (see screenshot suggestion) We
generally try to keep the same elements next to each other wherever their order is flexible (e.g. in option panel). JLF
suggests to lay elements so that the most important ones are in the top. According to both criteria, it would IMO make
sense to move the "Build JAR after compiling" checkbox below the textfields.

Created attachment 50165 [details]
packaging panel reordering proposal

"Regarding dependencies for single compiles, I think the NB 5.x way would be nice" - please reread my comment about NB
6.0 beta 1 vs. dev builds. "a complete new approach without Ant would be even better" - such major changes are not on
the table for NB 6.x.

Implemented options incl. GUI to skip JAR creation and dependency checking. Packaging panel rearranged according to
suggestion.

Also added code to pass -Dincludes=.... when running compile/jar, run, debug, and debug-stepinto targets when dependency
checking is off. (The optimization does not work when <depend> is used. Probably this is OK: if you have a smaller
project for which speed is not a problem to begin with, or you want to be careful, leave <depend> on and the IDE will
behave conservatively. If you want to make compilation faster, turn it off.)

Precisely: any files (of any extension) among non-test source roots of the project which were modified (with the IDE's
knowledge) since the last such target was run are collected. (Files modified in the editor are also included, since they
are about to be saved.) This list is passed as ${includes}, with the exception that if the list was empty and you are
running compile/jar targets, for safety the property is not passed. Effectively the behavior should be intuitive: if you
modify something and build or run, the IDE will not scan the whole source root, but just know to compile that class (or
copy that resource). If you just run (or debug) without any changes, no time is spent looking for changes. If you hit
Build with no apparent changes, a full check will be done (so this is an easy way to recover if the IDE did not refresh
correctly after an external command). Clean & Build always does a full check, of course.

While disabling <depend> works also for test roots and yields better performance, the ${includes} optimization is not
used on test sources, nor for compilations run as part of test builds. I hope this is acceptable for now - it would be
especially evil for a test to fail (or, worse, pass) because some modified classes were not recompiled.

If you find some specific problems or would like to suggest a refinement to some aspect of this solution, please _do not
reopen_ but file separate issues blocking this one, to be prioritized and tracked independently.

Checking in src/org/netbeans/modules/java/j2seproject/ui/customizer/J2SEProjectProperties.java;
/shared/data/ccvs/repository/java/j2seproject/src/org/netbeans/modules/java/j2seproject/ui/customizer/J2SEProjectProperties.java,v
 <--  J2SEProjectProperties.java
new revision: 1.68; previous revision: 1.67
done
Checking in src/org/netbeans/modules/java/j2seproject/ui/customizer/CustomizerJar.form;
/shared/data/ccvs/repository/java/j2seproject/src/org/netbeans/modules/java/j2seproject/ui/customizer/CustomizerJar.form,v
 <--  CustomizerJar.form
new revision: 1.11; previous revision: 1.10
done
Checking in src/org/netbeans/modules/java/j2seproject/ui/customizer/CustomizerCompile.form;
/shared/data/ccvs/repository/java/j2seproject/src/org/netbeans/modules/java/j2seproject/ui/customizer/CustomizerCompile.form,v
 <--  CustomizerCompile.form
new revision: 1.12; previous revision: 1.11
done
Checking in src/org/netbeans/modules/java/j2seproject/ui/customizer/CustomizerCompile.java;
/shared/data/ccvs/repository/java/j2seproject/src/org/netbeans/modules/java/j2seproject/ui/customizer/CustomizerCompile.java,v
 <--  CustomizerCompile.java
new revision: 1.17; previous revision: 1.16
done
Checking in src/org/netbeans/modules/java/j2seproject/ui/customizer/CustomizerJar.java;
/shared/data/ccvs/repository/java/j2seproject/src/org/netbeans/modules/java/j2seproject/ui/customizer/CustomizerJar.java,v
 <--  CustomizerJar.java
new revision: 1.14; previous revision: 1.13
done
Checking in src/org/netbeans/modules/java/j2seproject/ui/customizer/Bundle.properties;
/shared/data/ccvs/repository/java/j2seproject/src/org/netbeans/modules/java/j2seproject/ui/customizer/Bundle.properties,v
 <--  Bundle.properties
new revision: 1.94; previous revision: 1.93
done
Checking in src/org/netbeans/modules/java/j2seproject/J2SEActionProvider.java;
/shared/data/ccvs/repository/java/j2seproject/src/org/netbeans/modules/java/j2seproject/J2SEActionProvider.java,v  <-- 
J2SEActionProvider.java
new revision: 1.63; previous revision: 1.62
done
Checking in nbproject/project.xml;
/shared/data/ccvs/repository/java/j2seproject/nbproject/project.xml,v  <--  project.xml
new revision: 1.57; previous revision: 1.56
done
Checking in src/org/netbeans/modules/java/j2seproject/resources/build-impl.xsl;
/shared/data/ccvs/repository/java/j2seproject/src/org/netbeans/modules/java/j2seproject/resources/build-impl.xsl,v  <--
 build-impl.xsl
new revision: 1.103; previous revision: 1.102
done
Checking in test/unit/src/org/netbeans/modules/java/j2seproject/J2SEActionProviderTest.java;
/shared/data/ccvs/repository/java/j2seproject/test/unit/src/org/netbeans/modules/java/j2seproject/J2SEActionProviderTest.java,v
 <--  J2SEActionProviderTest.java
new revision: 1.21; previous revision: 1.20
done

Stupid question: will I be able to test the fix with the next daily?

I presume the next "daily" will have the fix, though I know little about the status and timing of the official daily
builds, so you are on your own there. You will definitely be able to find a build with the fix on deadlock.netbeans.org
within an hour or two; looks like #3708 will get it.

Questions to Jesse:

1. Does the fix only affect situation, where the user chooses to disable deps checking, or is there also optimization
for the general case?

2. 
> any files <...> which were modified (with the IDE's knowledge) since the last such target was run are collected
  a) I modify A.java file, hit F11, javac says there is an error
  b) I modify B.java file, hit F11
  => will A.java be compiled after step b?

3. 
> if the list was empty and you are running compile/jar targets, for safety the property is not passed
  What is the purpose of the property and what happens, when you don't pass it? Is this the situation, in which there
were no changes, you hit F11, and Ant does scanning/deps checking on the whole tree?

4. Consider the following case. Dependency checking is off, you have A.java and B.java files, both compiled to disk.
   a) A.java depends on B.java by calling its run() method.
   b) A.java is not open in the editor, and IDE doesn't currently hold its FileObject
   c.1) you rename B.run() into nur() via refactoring and hit F11
      => will A.java be recompiled?
   c.2) you rename B.run() into nur() *without refactoring* and hit F11
      => will A.java be recompiled?
   c.3) you rename B.run() into nur() *without refactoring*, save it, restart the IDE and hit F11
      => will A.java be recompiled?

5. Are there any other ways you can confuse the build system or compiler (assuming no external tools touch the sources)?

4.
   c4) you rename B.run() into nur() *with refactoring*, save it, restart the IDE and hit F11
      => will A.java be recompiled?

You can answer most of these questions for yourself by simply trying it, but:

To #1: the optimization is applied only if deps checking is disabled; it does not work with <depend>.

#2: currently no. The solution - assuming for some perverse reason you decide to leave A.java broken - is simple: press
Build again. (Note for testing: if you press F11 while A.java is still modified, the scenario will work by accident,
since in this case the file is saved after the Ant command line has been prepared, so it shows up as a potential
modification for the next build in addition to B.java. If you save explicitly and then build, then modify B.java and
build again, just B.java will be considered.)

#3: if not passed, then all files are checked to see if any are out of date, in the usual way.

#4: (1) yes (the IDE would necessarily load its FileObject as part of the refactoring); (2) no (A.java looks up to
date); (3) no (A.java still looks up to date). Obviously if you do such an incompatible change without refactoring and
break your source base, if you have Java error badges turned on, then you will find out quickly enough by the sudden
appearance of red X's all over your project. If you have turned off Java error badging, and you have also turned off
dependency tracking (which exists precisely to catch this kind of mistake), then you need to be more careful, or do
occasional clean builds.

#5: probably.

#4 (4) - if you hit F11 before editing anything else, sure. This seems a rather artificial case though.

#4 - How will the IDE know that the file is changed? Does it save this information anywhere on disk? What if I close
A.java, so that after restart the IDE won't load its FileObject - will it compile it then?

I don't think this is artificial. Simplified - yes. A real scenario would be to change several files via refactoring,
close the IDE at the end of the day, the next day launch the IDE again, do some more work and then compile everything.
It's possible you never even opened some of the files that were affected by refactoring.

I think you can figure out the implications of various scenarios from the details I already gave, plus it is easy to
experiment with for yourself. If you have modified some files since the last build (with the IDE's knowledge) just those
are compiled. Otherwise, any files newer than their classfiles are compiled.

I won't be able to test it during the several next days. But I asked these questions here mainly because many people
will like to know, how reliable the new mode is, but not everyone will want to think and play with all the corner cases
alone. Unfortunately, it doesn't sound reliable at all. So while I welcome improvements in compilation speed in simple
cases, I was hoping for a reliable performance solution for the general case. I wonder, whether the development will be
continuing in that direction and if yes, in which bug will it be tracked.

2 observations about the depends change:

1. Naming
I think the text "Track Java Dependencies" can be misleading. As far as I understand, switching this OFF does not run
the <depends> ant task. So, NB itselfs "tracks" the changes. See what I mean? Maybe a text like "Ants dependency
checking" or "Full dependency checking" would be more clear.

2. Testing
I modified a file outside of NB (file is not open in NB), so .class is out of date. Then I single-run a different file.
No matter if this setting is ON or OFF, the changed source is not being compiled on run-single of the other file.
In Beta 1, the changed file is always being recompiled. So, <depends> seems now to be always disabled.
We have .class and .java in the same folders (like NB 3.x or javac) and a slightly modified build.xml:
    <target name="-post-clean" depends="init">
        <echo message="Deleting all classes from ${build.classes.dir}"/>
        <delete>
            <fileset dir="${build.classes.dir}" includes="**/*.class"/>
        </delete>
        <echo message="Deleting all classes from ${build.test.classes.dir}"/>
         <delete>
           <fileset dir="${build.test.classes.dir}" includes="**/*.class"/>
        </delete>
     </target>

#1 - I don't think so. With the checkbox off, NB does _not_ track dependencies among classes; it simply uses a shortcut
to make an Ant target run more quickly, whose impl involves remembering modifications. For example, if class B {A a;}
and you modify A.java, with the checkbox on both A.java and B.java will be compiled on the next build; with it off, with
or without the optimization, just A.java.

#2 - run.single compiles only the file you run, as in previous releases. Unrelated to this change. See issue #116230.

Jesse,
You're the man!  Thank you so much - I just tried the daily and here is what I see:
Clean Build: 53 seconds
Run Main project (F6):  17 seconds
Run Main project again: 13 seconds
Make a single-file change, then F6: 17 seconds
Make a single file change (same file) then F6: 15 seconds

Now I turn off the dependency check and the jar creation:
Clean Build: 52 seconds
Run Main project (F6):  ~1.2 seconds
Run Main project again: ~1.2 seconds
Make a single-file change, then F6: 4.5 seconds
Make a single file change (same file) then F6: 3 seconds

The changes are reflected in the running program - good enough for me!  Hurray!

Now, it's still not quite as good as Eclipse or IntelliJ, but it's close enough
for me to stick with Netbeans, since it's got overall the better usability as far
as I'm concerned.

Thanks again.
tom

twolf2919, 

Just wanted to be clear: When you say turn off dep check, make a single change, and rerun, you are running on the same
changed file, right? So if I change Aclass.java, and rerun AclassTest.java, it won't pick up the change in Aclass.java? 

That's definitely not good enough.  

In response to wqtnetbeans' question:

I guess I wasn't clear when I said that all changes were reflected in the running
program (vs. class).  My program starts with class A.java.  If I make a change to
B.java and do F6 (run main program), then the changes to B.java are reflected in
the running program.

Most of my project files don't have a "main()", so typically I don't try
to run a file by itself - I suspect, though, this limited case would work too.

hope this is clearer.
tom

In twolf2919's case, Run will compile B.java; Run File (on A.java) generally will not. (It gets a bit complicated
because javac does its own automatic compilation of direct dependencies under some circumstances without being asked.)
Comments or questions about this belong in issue #116230.

Again about the naming of the dependency checkbox. I still think, it can be misleading. How about "Track external changes"?

So as twolf2919 pointed out earlier, after the fix is put in, the performance is good enough for some, but in general
it's still not as good as Eclipse/IntelliJ. 

And now the issue is marked as resolved and fixed, does that mean that's the closure of the whole issue, and NetBeans is
just going to be genetically slower in these smaller scale operations than other IDEs?

I was wondering the same things as wqt. But a point I want to stress is that actually, from my perspective, nothing
improved yet (except the fixes to Ant 1.7.1). I am not going to disable the deps, since I don't want to get into all
kinds of awkward situations this causes. So the question is - will there be further improvements after 6.0? If yes, then
there should be a bug to track the process (and for people to vote for).

"Track External Changes" is also wrong. Again, the function of this checkbox is to control whether or not Java-specific
dependencies between classes are considered when deciding which files to (re-)compile. That is why "Track Java
Dependencies" is correct, though there might be some clearer phrasing that has not been suggested yet.

Generally speaking: if you turn on Java error badging (and notice when the errors appear, e.g. at least occasionally
look at the Projects tab, or have the Task List window open), Track Java Dependencies will not be of much value since
you would likely notice any breakages as you make them. The exception would be if you make a change to a class which
changes the semantics of another class without breaking it, e.g.

class A {protected int x = 3;}
class B extends A {}
class C extends B {main() {System.out.println(x);}}

and edit B to read

class B extends A {protected String x = "hello world";}

Here after rerunning C without a rebuild with dep tracking off, you will still see "3". (In this case NB will show a
warning in B.java suggesting you rename the field, exactly because field shadowing like this is a frequent source of
confusion and generally a bad idea.)

One caveat with <depend> is that since dependencies are tracked at the bytecode level, not source level, changes to
compile-time constants (constants of primitive or String type with init exprs following certain restrictions) will not
propagate into classes where they are used unless you also touch the referring classes (or do a clean build). E.g.:

class A {public static final int TYPE_ONE = 1;}
class B {main() {System.out.println(A.TYPE_ONE);}}

If you compile, then change TYPE_ONE to 2, then run B, 1 will still be printed. This is annoying - a premature
performance optimization in the original JLS which hardly matters under HotSpot. If you use enums, I think this will not
happen, and there are also tricks for forcing a static final field to not be treated as a compile-time constant.
Generally you need to use care when exposing compile-time constants to other classes; for example, few people doing API
design realize that you can never change such a constant after it has been introduced, since old binary clients will
continue to use the old definition.

Closing an issue is a matter of tracking whether a given change was implemented and when. If there are resources
available to improve build performance for subsequent releases, and a way to do so is found, obviously we will do that.
As a rule, the broader the scope of a filed issue, the less valuable it is - in the extreme case, an issue "make
NetBeans faster" could be closed or not, it would not matter, because everyone would ignore it. An issue tracking system
is valuable insofar as it records and tracks very specific and concrete problems and their fixes which would otherwise
be unknown or forgotten. Filing some new issue identical to this one would accomplish nothing on its own.

This patch makes a real speed different for F6! The Dependencies checkbox should be off by default.

> Closing an issue is a matter of tracking whether a given change was implemented and when.
There are several ideas floating in the air. At least the following have been expressed here:
1. Fast compile-on-save or background compilation, bypassing Ant.
2. Smart integration with Ant. This is all about tracking changes and dependencies in the IDE and findnig a way to feed
them to Ant/javac instead of making Ant do the job.

Are these goals too wide to be achievable? Sure, someone will have to decide, in which of the above directions to push.
But, I think, the choice should not be that hard - obviously, the second suggestion is easier and more in line with the
current ways, provided that it's going to work. This point is hard stop for a user like me - I can't do this checking,
so I have to suggest both solutions.

I didn't notice any changes to the GUI of my web project. Is deps fix only for J2SE?

BTW, I did an install of a nightly Ant build and there is a measurable speed improvement over 1.7.0 - 2 to 3 seconds are
regularly shaved off the build time (hard to talk in percents, since the code is not compilable right now).

Like kirillkh, I am also for leaving the issue open. 

The current fix put in for 6.0 is for sure a relief for some. If you compare with other IDEs in regard to this kind of
small but often happen operations, NetBeans still doesn't have a good answer.

Currently only j2seproject's have dependency tracking; see issue #77666. I don't think changes in web app projects are
planned for 6.0.

Why do EE projects have different way of tracking dependency than SE ones, such that changes to address the same kind of
problem need to be applied separately?   

Different project types share some code but not currently that much. It is an ongoing effort to merge codebases.

So if the same kind of change doesn't get applied in EE projects in 6.0, as a normal user, I'd expect different
behaviors for the same kind of IDE operation depending on which type of project I am in, then? That's sad. :( 

I think the fact that web project doesn't use the deps task implies an important point that deserves to be stated
explicitly. According to your research, slow compilation in J2SE projects was caused by inefficiencies in the deps task.
But web projects don't use it at all. So the peformance problems I experience must be caused by something else.

Use of <depend> adds _some_ overhead. Obviously there are various other factors which affect the speed of a build:
overhead of normal <fileset> scanning in Ant 1.7.0 vs. 1.7.1, subproject builds (if enabled), creation of JAR (WAR,
...), warm vs. cold disk caches, javac's speed, local vs. network drives, initial startup cost of launching an Ant
project and perhaps opening the Output Window and saving modified files, etc.

The speed of normal build of a project without subprojects on a local drive, where no files were changed and no war
created, with Ant 1.7.0 was 12 seconds repeatedly (i.e. warm cache). Is the javac process run in this case? I think it's
safe to say that the overhead of running ant and of console window is negligible, when run repeatedly.

Looks like this rules out everything except the <fileset> scanning (I'm not sure: is it the same scanning that Ant uses
to determine, which files are changed?)

That is probably <fileset> scanning, though would need to run a profiler to be sure.

*** Issue 119089 has been marked as a duplicate of this issue. ***

really glad that there is now an option to turn dependency tracking off!
(However I think it should be default off or it should be removed completely)

I notice one more (related) regression with the new ant build/debug targets.

I edit (vim) files outside netbeans and debug with netbeans. This all worked very good with 5.5. But not so with 6.0.
Let me explain:
If I edit (and save) a file outside netbeans and then start the debugger.
- Sometimes netbeans doesn't compile the just edited file before debugging.
- Most of the times it compiles it after I start the debugger twice (I know it sounds strange but it is the truth!).
- Sometimes it will never compile that file before debugging.

However it always is build when I press the 'build' button (F11).
So I ask: please, execute the same targets that 'build' when I start a debug session.

Kees.

keeskuip - for specific problems that look like bugs, please file a fresh report with complete steps to reproduce from
scratch and it will be evaluated. Random comments here will just be lost.

jglick, sorry about that but I cannot reproduce it consistently. In fact if I turn on verbose (in search of a good
bugreport) on ant it doesn't happen. I thought I would mention it here because it looks related.  

Tom W.

Out of curiosity, have you tried these options? And if so, what's your experience?

Tom, nevermind. I just saw your comment of "Oct 5 18:40:40". (I was out of town then)