This Bugzilla instance is a read-only archive of historic NetBeans bug reports. To report a bug in NetBeans please follow the project's instructions for reporting issues.
E.g. A depends on B and C, both of which depend on D. Currently if you set A as the main project and build it, D's script is run twice (though the second time will do nothing). Problem: standard Ant tasks do not permit subprojects to transmit information back to a parent (though a task in antcontrib does permit this, apparently). Nor does Ant have a stock way of skipping builds that have already occurred within a parent build. Possible solution: main project defines location of a *.properties file in tmpdir; cleared or deleted by the main project at the start of every build. Each subproject, if it sees this property set, adds its name (path?) to the properties file. Each (sub-)project skips a subproject build if it sees the name (or path) already in that file. Pretty hacky, so should probably be optional, with a toggle in the project.xml read by XSLT; perhaps turned on automatically when a change in project dependencies warrants it.
Probably not going to try it for D.
I have this same comment in another issue, but it was off-topic for that issue, so just moving it here and modifying and adding adding a little... I have run into this with projects also. I have projects which depend on the same projects then these projects get aggregated into other projects which other projects depend on, but in the case of a medium project (I call it medium and it has roughly 20 sub-projects and 15-20 jar dependencies) this takes a long time running through the ANT scripts as I keep seeing... compile: jar: dist: over and over many more times than the number of projects I have. The end result is it takes a long time to run or debug a large project even when nothing is compiled. This does not only apply to web applications. It takes about 1 minute to run a single main class even when nothing needs to be compiled in it or the sub-projects. It would be nice if we had dirty hashes or codes (properties maybe something as simple as dirty.dist=Y, dirty.compile=Y...which could be set on edit and save or something)...basically something to check with a quick load and exit the ANT scripts if they are not dirty. Clean would obviously reset these as well. It might add an issue to checking out then building, but I suppose a quick check for the dist and build folders would tell whether the files needed to be built or not...if the user did "not" just get just the updated project files and not the source, but then I would think they have other problems anyways such as files out of sync with their project or possibly files missing. If not then I suppose an option on the context menu to reset dirty flags or not would help with issues such as that. I do not know if ANT should support such a feature directly however. Because it would not be relevant to a standalone build system. With a build system you either clean or build. I suppose ANT could set flags if it were given a command option to do so. Though if the user had no easy way to manage these flags they would have to manually reset these as properties for each project upon edits if they were not using an IDE which supported them. I suppose it would be possible to add the base functionality to ANT instead of NB tasks or project scripts for NB to use. Basically an attribute could be turned on in a given project tag for ANT to use the system or not. So, it loads one build file, checks the attribute, then looks for a companion file or a file supported by the attribute. So, the attribute points to a file. The file can be relative to the build.xml file. It can then have a structure to support this feature. It could be either property file format or an XML document. The file could contain a set of directories to check for existence and could support flag properties. Needs more though obviously on the flags if one is thinking of generic usage vs. NB project system. Then it would take collaboration with the ANT community. I setup this main class to bomb out as soon as it started. See below the ANT output as it ran over the different sub-projects targets. Notice the final output. This to run a single and simple main class which itself bombs out at start to not skew the time too much. I only did this to give a feel for the issue. Sorry if it is too long, and I'll understand if it is edited at some point to shorten it: init: deps-jar: init: deps-jar: compile: jar: init: deps-jar: init: deps-jar: compile: jar: init: deps-jar: compile: jar: init: deps-jar: init: deps-jar: compile: jar: init: deps-jar: compile: jar: init: deps-jar: init: deps-jar: compile: jar: init: deps-jar: compile: jar: compile: jar: compile: jar: init: deps-jar: init: deps-jar: compile: jar: init: deps-jar: compile: jar: compile: jar: init: deps-jar: init: deps-jar: compile: jar: init: deps-jar: compile: jar: init: deps-jar: init: deps-jar: compile: jar: init: deps-jar: compile: jar: init: deps-jar: init: deps-jar: compile: jar: init: deps-jar: compile: jar: compile: jar: compile: jar: init: deps-jar: init: deps-jar: compile: jar: init: deps-jar: compile: jar: compile: jar: init: deps-jar: compile: jar: compile: jar: compile: jar: init: deps-jar: compile: jar: init: deps-jar: init: deps-jar: compile: jar: init: deps-jar: compile: jar: init: deps-jar: init: deps-jar: compile: jar: init: deps-jar: compile: jar: compile: jar: compile: jar: init: deps-jar: init: deps-jar: compile: jar: init: deps-jar: compile: jar: compile: jar: init: deps-jar: init: deps-jar: compile: jar: init: deps-jar: compile: jar: init: deps-jar: init: deps-jar: compile: jar: init: deps-jar: compile: jar: init: deps-jar: init: deps-jar: compile: jar: init: deps-jar: compile: jar: compile: jar: compile: jar: init: deps-jar: init: deps-jar: compile: jar: init: deps-jar: compile: jar: compile: jar: init: deps-jar: compile: jar: compile: jar: init: deps-jar: compile: jar: init: deps-jar: init: deps-jar: init: deps-jar: init: deps-jar: compile: jar: init: deps-jar: compile: jar: init: deps-jar: init: deps-jar: compile: jar: init: deps-jar: compile: jar: init: deps-jar: init: deps-jar: compile: jar: init: deps-jar: compile: jar: compile: jar: compile: jar: init: deps-jar: init: deps-jar: compile: jar: init: deps-jar: compile: jar: compile: jar: init: deps-jar: init: deps-jar: compile: jar: init: deps-jar: compile: jar: init: deps-jar: init: deps-jar: compile: jar: init: deps-jar: compile: jar: init: deps-jar: init: deps-jar: compile: jar: init: deps-jar: compile: jar: compile: jar: compile: jar: init: deps-jar: init: deps-jar: compile: jar: init: deps-jar: compile: jar: compile: jar: init: deps-jar: compile: jar: compile: jar: init: deps-jar: compile: jar: compile: jar: init: deps-jar: compile: jar: init: deps-jar: compile: jar: init: deps-jar: compile: jar: init: deps-jar: compile: jar: init: deps-jar: init: deps-jar: init: deps-jar: compile: jar: init: deps-jar: compile: jar: init: deps-jar: init: deps-jar: compile: jar: init: deps-jar: compile: jar: init: deps-jar: init: deps-jar: compile: jar: init: deps-jar: compile: jar: compile: jar: compile: jar: init: deps-jar: init: deps-jar: compile: jar: init: deps-jar: compile: jar: compile: jar: init: deps-jar: init: deps-jar: compile: jar: init: deps-jar: compile: jar: init: deps-jar: init: deps-jar: compile: jar: init: deps-jar: compile: jar: init: deps-jar: init: deps-jar: compile: jar: init: deps-jar: compile: jar: compile: jar: compile: jar: init: deps-jar: init: deps-jar: compile: jar: init: deps-jar: compile: jar: compile: jar: init: deps-jar: compile: jar: compile: jar: init: deps-jar: compile: jar: compile: jar: init: deps-jar: compile: jar: init: deps-jar: compile: jar: init: deps-jar: compile: jar: init: deps-jar: init: deps-jar: compile: jar: init: deps-jar: init: deps-jar: compile: jar: init: deps-jar: compile: jar: init: deps-jar: init: deps-jar: compile: jar: init: deps-jar: compile: jar: init: deps-jar: init: deps-jar: compile: jar: init: deps-jar: compile: jar: compile: jar: compile: jar: init: deps-jar: init: deps-jar: compile: jar: init: deps-jar: compile: jar: compile: jar: init: deps-jar: compile: jar: compile: jar: compile: jar: init: deps-jar: init: deps-jar: compile: jar: init: deps-jar: compile: jar: init: deps-jar: init: deps-jar: compile: jar: init: deps-jar: compile: jar: compile: jar: compile: jar: init: deps-jar: init: deps-jar: compile: jar: init: deps-jar: compile: jar: compile: jar: init: deps-jar: init: deps-jar: compile: jar: init: deps-jar: compile: jar: init: deps-jar: init: deps-jar: compile: jar: init: deps-jar: compile: jar: init: deps-jar: init: deps-jar: compile: jar: init: deps-jar: compile: jar: compile: jar: compile: jar: init: deps-jar: init: deps-jar: compile: jar: init: deps-jar: compile: jar: compile: jar: init: deps-jar: compile: jar: compile: jar: compile: jar: init: deps-jar: init: deps-jar: compile: jar: init: deps-jar: init: deps-jar: compile: jar: init: deps-jar: compile: jar: init: deps-jar: init: deps-jar: compile: jar: init: deps-jar: compile: jar: init: deps-jar: init: deps-jar: compile: jar: init: deps-jar: compile: jar: compile: jar: compile: jar: init: deps-jar: init: deps-jar: compile: jar: init: deps-jar: compile: jar: compile: jar: init: deps-jar: compile: jar: compile: jar: compile: jar: init: deps-jar: init: deps-jar: compile: jar: init: deps-jar: compile: jar: init: deps-jar: init: deps-jar: compile: jar: init: deps-jar: compile: jar: compile: jar: compile: jar: init: deps-jar: init: deps-jar: compile: jar: init: deps-jar: compile: jar: compile: jar: init: deps-jar: init: deps-jar: compile: jar: init: deps-jar: compile: jar: init: deps-jar: init: deps-jar: compile: jar: init: deps-jar: compile: jar: init: deps-jar: init: deps-jar: compile: jar: init: deps-jar: compile: jar: compile: jar: compile: jar: init: deps-jar: init: deps-jar: compile: jar: init: deps-jar: compile: jar: compile: jar: init: deps-jar: compile: jar: compile: jar: init: deps-jar: compile: jar: init: deps-jar: compile: jar: compile: jar: init: deps-jar: compile: jar: compile: jar: init: deps-jar: compile: jar: compile-single: run-single: BUILD SUCCESSFUL (total time: 1 minute 2 seconds)
More information which I have broken down. In the example I provide the project has 9 direct dependency projects. The first project out of the 9 has 5 dependencies. Those 4 of the 5 are part of the 9 of the top level project. They are common libraries. The second only has jar dependencies and no project dependencies. The third has 2. One with project dependencies. Which probably is dependent on all the other 9 and then some, but no more than a possible 17. I have in total 17 projects which make up this project and all other higher level projects are included, so it wouldn't actually be 17, but could only possibly be 17 in total ever at the current state of the overall project, so I can never have more than 17 actual projects being built for this overall program/project. All the rest are static jar dependencies. On and on this goes for the 9 dependencies. The problem appears definitely in the D situation. It is also in the C situation, but is pushed overboard by the D situation. In conclusion, of a total possible 17 project builds if a project were only built once 190 builds (or attempted builds to be more clear) are taking place. The above was factored from counting all the "jar:" from the above example. 17*9 = 153, so it is easy to see how the number gets high just by taking the total number of known top level dependencies and multiplying by the hard limit of 17 known projects. I know projects are built more than 9 times and that 17 projects will not be included, but it at least shows the possibility. I would have to add an echo to -pre-jar or something to get more exact numbers, but I think this example illustrates the problem well. The only way to even attempt to speed this up from this projects point of view would be to break everything out into interfaces and have lookups. That would just add overhead to the process and not gain anything as far as breaking the APIs apart as most builds are from mid-level dependencies having dependencies on common libraries. Then some of the common libraries have dependencies on even more generic libraries such as logging etc which I have made wrappers to integrate with configuration and dynamic file output placement.
Maybe a half way solution would be to have targets to copy dependent jars and run and debug without the checks. dist-nodeps and run-nodeps and debug-nodeps which could make dist and running and debugging faster when one knows they do not have any changes in anything other than a single sub-project which they build manually. This way jars can be copied and the process can be started, but all the checks do not have to take place. The problem then is different targets and not easily integrated into the UI and process. But, it's just an idea for an in between step while the issue is worked out. I wonder how much speed could be obtained by knowing ahead of time if a project is dirty or not. I mean at each sub-project not just a top-level project. If a project is dirty the project build just exits for that script nothing to run through checks and hashes. It would be possible with new tasks to use in the build scripts which could check for dirty early and set/reset it in the end. It would have to be something which was checked early, and the script targets would have to be wrapped in conditional executions to support it. The UI would have to allow users to override it just in case to account for unknowns. Currently not only are the scripts loaded, but the class files, jars, etc have to be checked before the target completes. Tough issue all around. I'll dwell on it much more before I post any other input.
Jesse, please reevaluate. Thanks
No longer working on j2seproject. Would be trivial to do in any kind of scriptable build system. Unfortunately tricky with Ant because the child build cannot easily pass information back up to the parent build, so probably needs to use a delete-on-exit text file as a back door communication channel. Possibly but quite ugly.
I think this issue deserves much higher priority than P3. The problem is not so evident for an "educational sample" of project A depending on B and C, which both depend on D. In this case the build time is still fairly good. I have defined 37 projects (java projects with existing sources) in NB 6.0, many of them depend on a "core project" defining the common types/classes, and there are also other dependencies. Running build task of a project which is on the top of the hierarchy does not mean the "core project" is compiled "only" 36 (37-1) times, but due to other project dependencies it is more than 10000 times (I did not count this exactly, see below the exact numbers for a smaller number of projects). For a comparison I provide the time achieved by an external ant script (which solves the project dependencies) for a project which depends on other 19 projects and the (partial) "build" output of NB 6.0: 1) build by external ant script: BUILD SUCCESSFUL Total time: 7 seconds 2) build in NB 6.0 (17259 lines of output): compile: <4307 times> jar: <8630 times> deps-init: <4291 times> BUILD SUCCESSFUL (total time: 56 minutes 51 seconds)
The main project script should generate build ID (System.currentTimeMillis) and pass it to the subants. When build the subant will store it into the build folder. If the build enters again into the same subant the build ID == stored buid ID => nothing to do.
Yes, that would work. Still tricky to implement in Ant without scripting, though. There is no way for the init target to just "return" if some condition holds. It cannot <fail> or the whole build will fail. It could set some "skip" property, but then every target in the rest of the project would need to add unless="skip", which is quite ugly. Perhaps the calling project, rather than the called project, could keep track of whether someone has called it already, though then the calling project would need to hardcode the location of the 'build' dir without permitting an override, which is also a problem. I think my original proposal might be somewhat simpler. Main project would create a ${basedir}/build/built.properties with one key per basedir of built subproject (value irrelevant), defining a property for the location of this file and letting subbuilds inherit it. deps-jar target would need to be split up into one target per dep, each checking whether the basedir of the subprj is already listed, and if so, skipping. Still quite ugly.
I will try this for Nb 6.1 but the pure Ant solution looks very ugly.
Does it look any uglier than rebuilding the "root" project thousand times?
It depends: Makes build-impl even less readable. If buggy it may cause that some project will not be compiled at all.
> If buggy it may cause that some project will not be compiled at all. Taking an hour to compile real user project that should compile in 7s is equivalent to not being compiled at all in my eyes :-)
moving opened issues from TM <= 6.1 to TM=Dev
We are also suffering this same issue with 6.1 and we think it deserves a higher priority. We are migrating from JBuilder and this is getting us stuck. We have a big architecture of projects and dependencies. We have managed to override ant targets to compile only the projects really needed but this doesn't solve the problem completely. If you try to compile a single class all the dependencies are opened and checked taking half an hour to complete, and obviously this is not acceptable. Is this issue targeted for 6.5? It's really important for big sized projects.
I agree with all of you. For big sized projects is impossible to work with Netbeans. We are thinking in migrating to Eclipse, because the more dependencies in the project, longer the time it takes to compile. The time is increased in an exponential way. To my eyes, this issue is so important that no large projects will be done in Netbeans, because you can not afford to modify the build scripts every time you modify the definition of the project.
Related thread on nbusers: http://forums.netbeans.org/topic3919.html
We have tested this issue again with the 7.0M1 version and it's still present (200901271556 development build fails to start in my machine). It would very sad if this issue is not resolved in version 7 either. It's present since version 4 and our team needs to know when is it going to be resolved. We are very interested in using Netbeans but this issue doesn't let us migrate from JBuilder, so if this is not working in 7.0 we probably go for Eclipse.
I've written a build-override.xml which can be imported before build-impl.xml in your build.xml files and implements basically what is described here. I was able to achieve a 13x improvement in build speeds on our project. The first project that gets called creates a file to keep track of which sub-projects have been called and the code prevents re-calling them during this ant invocation. There are a couple of potential problems: 1) If your project has no project dependencies (is a leaf node in your system), the code fails. Instead of fixing it I just don't import it for leaf node projects. 2) I ignore completely the deps-module-jar and deps-ear-jar distinction. We don't use that and I don't know anything about it. The code works by overriding deps-jar and deps-clean with special code that scans project.properties for lines that start with "project.*" and dynamically writing out a temp ant build file on the fly. We are using Netbeans 5.5.1 on Windows and Linux. I have not tried it with any other versions. I will attach the file in a moment.
Created attachment 76925 [details] build override file to implement smart dependency checking. Import before build-impl.xml in all your non-leaf projects.
looks like P3 for me (see http://wiki.netbeans.org/BugPriorityGuidelines) + looks like there is some workaround described by parcellsharp
I tested this workaround and helps a bit but still takes too long to compile our EAR, 15 min. With our hand crafted ant xml takes less than a minute. It is not a viable workaround for us, so it looks like P2 for me... P2 doesn't work workaround exists Product feature doesn't work, a workaround may exist but it's difficult to use or impractical
Making it P2 then.
Fixed in core-main #c01e2c997e19, essentially along the lines I originally proposed five years ago. :-) Implementation is complicated by several aspects of Ant's semantics, which conspire to make any kind of coordination across projects very hard: 1. There is no <if> task in the standard distribution, so you need to use conditional targets driven by properties, which is made worse by #2. 2. Properties are single-assignment, so you need to generate new property names for each new situation (no scoping). 3. <ant>/<antcall> cannot return values to the caller, so you need to communicate using files on disk. 4. Passing some, but not all, properties along to a subproject is not trivial. 5. There is no way that I know of to tell if you are running a new top-level target (e.g. argument on Ant CLI) as opposed to running a target as a dependency of another. There are two drawbacks with the impl that I know of: 1. Some additional messages are printed during the build. Could be addressed using an AntLogger if it is considered too annoying. 2. 'ant jar jar' will fail, not that you would have any real reason to do this. 'ant jar test' will also fail since 'test' depends on 'jar'; just use 'ant test'. 'ant clean jar' (or 'ant clean test') works even though both 'clean' and 'jar' recurse, because the cycle check is done separately by target ('clean' vs. 'jar'). If there are a lot of people out there who insist on running 'ant jar test' even though it is redundant, the cycle check could be downgraded to a warning and it might work.
Regardless of not being able to run particular targets a certain way, speeding up this area will be just awesome. Thanks Jesse. I plan to try it out sometime very soon. Will report anything I find. Thanks again.
Note also that the fix applies only to j2seproject's. I do not imagine people would normally build deep trees of web projects etc. - the dependencies would likely be plain libraries. If there is a need, the fix could be ported to other Ant-based project types. This _would_ be useful if you had, say: WebApp -> Java1 > Java2 | v \ v +----> Java3 > Java4 Running 'ant -f Java1/build.xml jar' will run just 4 scripts, as expected. But 'ant -f WebApp/build.xml war' will currently run 7 rather than the expected 5: - Java4, Java2, Java3, Java1 (i.e. all deps of Java1 in reverse topological order) - Java4, Java3 (did not know these had already been built!) - WebApp This is not as bad an issue as the potentially exponential explosion of a deeply nested and heavily crosslinked DAG of j2seproject's, but still undesirable. I don't know if such a setup is really common in practice, however. CC'ing dkonecny who might have a better idea.
Hi, this is good news! But what about EJB projects? We have deep trees of EJB projects inside an EAR...
helius - I filed issue #167371 for a port to Java EE-related project types. I don't expect it to be difficult to make an analogous fix, though there are additional complexities of ejbjarproject's build that I do not personally understand, so I will leave it to the experts.
Integrated into 'main-golden', will be available in build *200906191401* on http://bits.netbeans.org/dev/nightly/ (upload may still be in progress) Changeset: http://hg.netbeans.org/main-golden/rev/c01e2c997e19 User: Jesse Glick <jglick@netbeans.org> Log: #42683: enter shared subprojects at most once per build.
I am reluctant for this to be a 67patch-candidate unless it receives thorough testing for regressions. (Issue #167371 ought to be considered part of the package as well.) For a possible backport I would also recommend changing the "cycle detected" error to a warning. So far I have learned of one case where this causes a problem: someone implementing a target (e.g. 'test') in the project to <antcall> some other target, but neglecting to pass inheritall="false" (which they should have been passing to begin with).
Marian, will you be able to test this enough for the patch? The thing is that this has 7 votes and was a P2 for some time already ...
unfortunately I cannot verify this fix due to issue 168294
OK, we are not going to fix this issue into NB 6.7.1 - neither following NB 6.7 patches
I am going to fix 168294 today. I think we can still consider this one for inclusion if Jirka is able to verify it. What do you think? I don't think that backporting to 6.7 is tied to 168294, it can be done in the trunk inly IMHO.
oops, I meant "tied to 167371".
I still recommend that issue #167371 be included in any backport, as well as a possible removal of the <fail>.
The fix of issue #174799 should also be included in any potential backport.
*** Bug 103490 has been marked as a duplicate of this bug. ***