This Bugzilla instance is a read-only archive of historic NetBeans bug reports. To report a bug in NetBeans please follow the project's instructions for reporting issues.
Need API to add and remove dynamically generated source root to the classpath. This will allow generated sources to be available in code completion. Such an API is helpful for modules like JAXB and Webserives, which generate Java sources from XSD schema and WSDL files. Same or similar API is also needs to be implemented in Web and EJB project types. Description of the requirement from Milan: I'd prefer to ask for the API extension of ProjectClassPathModifier : ability to add the "hidden" source root What I mean with "hidden" source root - "hidden" source root is accessible for Code Completion - "hidden" source root is not accessible in wizards - in project customizer-source root section it may be visible (marked with an extra icon) Suggested method : ProjectClassPathModifier.addRoots(classPathRoots, FileObject projectArtifact, ClassPath.SOURCE, boolean hidden); Milan
We need that API pretty much. In web services/jaxb we generate java artifacts into build directory (e.g. build/generated/wsimport/client).Then we'd like to see those classes available for CC and for refactoring (the java refactoring requires those artifacts to be under source roots). Currently we hack somehow the CC in all particulear project types that supports web services, by hacking the ClassPathImplementation:getResources(), that is not good. We have few (p2) bugs that depend on this. This is the list : issue 75410 issue 81751 issue 96936 issue 112237
The reasons we don't want the "hidden" source root be visible in UI are : - we don't want to allow user to modify those files - we don't want to expose those packages in wizards (location combo box in java wizards)
We will be looking for workaround for 6.0, because implementing the API would require changes in project.xml schema and that's not what we would want to do around Beta1.
Milane, what is the status of this issue? Thanks
I'm sorry, but this issue is not in NetBeans 6.5 plan and I'm afraid that it won't be fixed during bugfixing of 6.5 either.
Does anyone have a more or less clean hack to include the generated files at least into the JavaDoc for the project? Currently I'm using a manual ant task to copy the generated Java source files to the regular source path. This works for the JavaDoc but of course it also pollutes the source directory with the duplicated source files which could easily be out of sync with the up-to-date generated files. I'd appreciate any ideas!
Created attachment 69625 [details] Test case
See the multiple issues that depend on this. I'm just summarizing: - javadoc for generated classes is not available - CC for generated classes should be hacked - Go to Class feature (Source for Binary Query) needs to be extra implemented - problems with Compile on Save feature (Clean and Build doesn't add generated classes to project) Increasing to P1. I like the MAVEN way to implement a specific "parent folder" for all generated sources, e.g. "generated-sources" to enable all supports to put those sources there, e.g. : project_dir - generated-sources - jaxb - jax-ws - jax-rpc ... Ant based projects should handle all common features for those sources automatically (CC, Javadoc, ...). Moreover these source roots shouldn't be available for wizards.
These are the detailed requirements on Generated Sources behaviour (it's very similar as it works in Maven project type) : Consider all sub-folders of "build/generated-sources" folder are source roots (we can call them generated source roots) - don't expose generated source roots in wizards - involve generated source roots into project ClassPath - implement class path for all generated sources - implement SourceForBinaryQuery for all generated source roots - generate javadoc for all generated source roots - project's infrastructure doesn't care how these sources are generated - clean action will remove all generated sources (obvious, since entire build directory is removed) - expose packages from generated source roots in project view (to be able to examine those files, set breakpoints etc.) Suggested Project's view (J2SE Project): JavaApplication - Source Packages - Test Packages - Generated Sources [or Generated Source Packages] /* collected packages from all generated source roots */ - Libraries - Test Libraries Files View : JavaApplication - build - classes - generated-sources - jaxb - jaxws - jaxrpc ... - nbproject - src - test
One another natural requirement is that project's classpath must be refreshed when project is cleaned or built. See the issue 155186. Note: the Maven project works perfectly in this case.
To see some generated sources, let's create a web service client in J2SE project, e.g. for CurrencyConvertor service ( http://www.webservicex.net/CurrencyConvertor.asmx?wsdl ). Then, generated sources, are located in build/generated/wsimport/client directory. From the "API", I'd expect to have an option to specify a sub-folder of generated-sources folder where java sources should be generated. Or may be better, it could be done without any API. The (ANT) project infrastructure will automatically recognize all subfolders of build/generated-sources.
Reassigning to Jesse.
A number of the requested features in fact already work, at least in the case of build/generates/wsimport/client - at least routine code completion, Go to Type, etc. seem to work normally on generated sources. So I assume that you mean that you want these features to continue to work after deleting a bunch of special-case code somewhere in WS support. But I don't know where that code is. Please tell me specific service impls which you would like to be able to delete. You also say that j2seproject should automatically specify a classpath for the generated sources. What should that be? Same as the regular project classpath? And should the generated sources be part of the same ClassPath.SOURCE as the normal source roots? For that matter, should targets like wsimport-client-compile just be deleted, relying on the regular compile target? (It adds ${libs.jaxws21.classpath} to the classpath, but this is in ${javac.classpath} anyway, so I'm not sure why.) That would be consistent with the regular javadoc target including the generated roots. And what should happen to old projects that rely on the current implementation of some of these features? I need a more concrete description of what it is you want to happen. (I do not know much about the Maven project infrastructure, especially not regarding JAX-WS support, so the comparison is not so helpful unless accompanied by specific code pointers and instructions on how to create a running example.)
As far as Maven project: just create a maven project (j2se or web) and create a WS Client for some SOAP service, e.g. : http://www.webservicex.net/CurrencyConvertor.asmx?wsdl (source level must be > 1.4) > A number of the requested features in fact already work, at least in the case of build/generates/wsimport/client - at least routine code completion, There are problems with that: e.g. when a cleaned project, containing WS Clients, is opened and built - the generated sources aren't recognized in classpath. Another issue: javadoc for generated classes isn't generated. > Please tell me specific service impls which you would like to be able to delete. java.api.common org.netbeans.modules.java.api.common.classpath.SourcePathImplementation : (See hard-coded stuff in getResources() method) websvc.core - org.netbeans.modules.websvc.core.jaxws.projects.JaxWsArtifactsClassPathProvider - org.netbeans.modules.websvc.core.jaxws.projects.JaxWsSourceForBinaryQueryImpl (remove both classes) > You also say that j2seproject should automatically specify a classpath for the generated sources. What should that be? Same as the regular project classpath? And should the generated sources be part of the same ClassPath.SOURCE as the normal source roots? - Yes, regular project classpath, part of ClassPath.SOURCE - however we don't want to see those source roots in wizards. (Maven impl. solved this by creating a specific source group for generated sources) > For that matter, should targets like wsimport-client-compile just be deleted, Right. > And what should happen to old projects that rely on the current implementation of some of these features? I don't think old the code should be removed immediately. It can coexist before the code generation is redirected to another place. I'll be able to make changes in JAX-WS, JAX-RPC, JAXB support.
OK, I think I see what needs to happen; let me know if I am off track. Here is my requirement summary: --- A module wishing to add generated sources to an Ant-based project need only: 1. Register a build extender to generate the sources to build/generated-sources/SUBDIR/ before compile. 2. Add any required classpath items such as libraries in the usual way. (In fact such customizations can be made by a user with no special module at all.) Expected basic features: 1. Generated roots are included in the same ClassPath.SOURCE as true source roots. They should also share the same ClassPath.COMPILE etc. This means that they are available in Go to Type, code completion, Find Usages, etc. 2. Generated roots are included when running <javac> or <javadoc>. 3. Generated roots are not offered as SourceGroup's, i.e. not visible in New File etc. 4. Whenever present, generated roots are displayed in the Projects tab for browsing. 5. Generated roots may be created and deleted while the project is open and everything should refresh. (The Java parser may consider true roots to be in error after a clean, but building should fix it.) --- I am starting on unit tests in java.api.common, where the query changes need to be made. build-impl.xsl in j2seproject et alia will also get patched so that compile and javadoc targets include the gen roots. I should be able to test the functionality interactively (demo) using a j2seproject with a -pre-compile override in build.xml that just generates some source files in an ad-hoc manner, e.g. <echo>. I guess I will just leave the existing build/generated/wsimport/{client,service} stuff (in java.api.common and websvc.core) untouched. Old projects should work the same as before (since "generated" != "generated-sources"); new projects (or projects with features like JAX-WS newly added) would use the new system (e.g. build/generated-sources/jaxws-client) and the old code would be a no-op. Up to you how to deal with automatic upgrades, if those are desired, and when (if ever) to delete the old code.
You're on track - that's exactly what I expect from that "support". Yes, after you're ready I will modify WS modules. Please, specify then what will be responsibility of the user(module writer). Thank You.
Created attachment 76359 [details] Sample project to play with: demonstrates gen src root, CP shared between src roots, CoS vs. dep track
Created attachment 76361 [details] Working patch for review (modifies only j2seproject)
The attached patch makes j2seproject handle generated source roots. The three major components are there: - queries treat direct subdirs of build/generated-sources/ as source roots (part of source CP, but not a SourceGroup), and can handle dynamic creation or removal of genroots - compile & javadoc actions include genroots alongside true roots - each genroot, when it exists, is shown in Projects as "Generated Sources (name)" (merging multiple genroots into one node looked pretty hard) You can try it for yourself with the attached sample project. There is no Java-level API needed; modules such as those providing JAX-WS support need only (a) make sure the project CP is correct, e.g. by adding some libraries during initial setup, and (b) insert a build target as a dep of 'compile' to create the genroot. Let me know what you think and whether I should commit it and when. I can probably add support for some other project types (which?) myself, or leave that to someone else.
Tested. Looks very smart. - CC in project works - CC in generated sources works - CC works if project is added to another project deps - javadoc is generated for generated sources - Go to Source action works - setting breakpoint to generated source file works - generated sources folder doesn't appear in wizards I agree, implementing multiple folders for generated sources in logical view is better solution: "Generated Sources (name)" is a good idea. I found actually 2 issues : 1. Compile on Save problem If Compile on Save is set on, the following scenario doesn't work : - Clean project - Run project The error appears : Exception in thread "main" java.lang.ExceptionInInitializerError Caused by: java.lang.RuntimeException: Uncompilable source code - package generated does not exist at test105645b.Main.<clinit>(Main.java:2) (this may require to register SourceForBinaryQuery for generated sources though I am not expert in this area (H.Lahoda is) 2. Backward Compatibility problem : The existing j2se project, or newly created one cannot be built : init: deps-clean: Deleting directory /space/home/mkuchtiak/NetBeansProjects/JavaApplication4/build clean: init: deps-jar: Created dir: /space/home/mkuchtiak/NetBeansProjects/JavaApplication4/build/classes /space/home/mkuchtiak/NetBeansProjects/JavaApplication4/nbproject/build-impl.xml:350: The following error occurred while executing this line: /space/home/mkuchtiak/NetBeansProjects/JavaApplication4/nbproject/build-impl.xml:159: /space/home/mkuchtiak/NetBeansProjects/JavaApplication4/build/generated not found. After these issues are fixed feel free to commit those changes. If you could implement changes for another project types (web.project, j2ee.ejbjarproject, j2ee.clientproject) it would be nice from you. I think, I know what should be implemented from WS support site then. Thank you indeed.
Thanks many times Jesse! > I can probably add support for some other project types (which?) myself, or leave that to someone else. Looks like most of the changes are thankfully in java.api.common, which is used by all the Java-based project types (web, ejb, appclient), so the changes in the other project types are just a matter of the build scipt + one one-line change. Cc'ing David, who is the most knowledgeable about these project types.
The patch seems fine to me. Only one thing: sourceRoot.toString().startsWith(parent) The comparison of paths using String.equals() may cause some problems on Windows in obscure cases.
Regarding the "Generated Sources (name)" nodes: in the initial patch there is no provision for localizing the "name" part - this is just taken from the folder name. I will add a way for a module like JAX-WS support to provide an alternate localizable label for a known root folder name. BTW if there is interest, it would be possible for these nodes to be displayed with a grey label (matching VCS-ignored build dirs in Files) and/or make subnodes (packages & files) not show modification operations such as Delete; currently all I do is remove the New File... submenu from the root node itself. (You would not anyway be permitted to add a Java source to one of these folders. It is possible to add e.g. an HTML file if you use Ctrl-N with the node selected, or type in the build/... path in the text field, though the build/ subtree is not offered in the folder browser in the New File wizard because it is considered unsharable. The New File wizard could be enhanced to make it even harder to add a generic-type file to an unsharable subfolder, but this might be overkill.) The CoS behavior is as designed. If you clean the project, the generated sources are deleted. The Java parser therefore marks the true sources as in error - as it should, since it cannot really know whether they are good or not. It also removes the genroot classfiles from the CoS classpath, since the source root no longer exists. While you _can_ use CoS with a project using a genroot, you must at least generate the sources before you can do anything else - most easily using S-F11. I will check the build problem you mention. Anyway as I understand it, the JAX-WS support disables CoS mode for projects it modifies.
I am unable to reproduce any problem (such as your "build/generated not found") building a preexisting or newly created j2seproject. I did find that test compilation was broken - fixed. I believe the usage of sourceRoot.startsWith is OK. I am comparing URIs, not File paths. APH.resolveFile always normalizes its result, and files converted to URIs for BFSQ should already have been normalized. But I will try to check functionality on XP if I can.
Both problems I mentioned are there. What I did was : - applied the patch (and re-built those 2 modules) - removed userdir - started ide - opened test105645 project 1. CoS problem - set up Compile on Save option - Clean project - Run project from project node (Run file is the same) Exception in thread "main" java.lang.ExceptionInInitializerError Caused by: java.lang.RuntimeException: Uncompilable source code - package generated does not exist at test105645b.Main.<clinit>(Main.java:2) 2. Build problem - created new Java Application - Build project (or Clean & Build) Created dir: /space/home/mkuchtiak/NetBeansProjects/JavaApplication5/build/classes /space/home/mkuchtiak/NetBeansProjects/JavaApplication5/nbproject/build-impl.xml:350: The following error occurred while executing this line: /space/home/mkuchtiak/NetBeansProjects/JavaApplication5/nbproject/build-impl.xml:159: /space/home/mkuchtiak/NetBeansProjects/JavaApplication5/build/generated-sources not found. BTW: to make CoS work, I remember, we had to implement SourceForBinaryQuery for generated sources.
Sure, #1 is reproducible, but as I said it's not a bug. You must generate sources first, then CoS works. #2 looks like a missing erroronmissingdir="false" which I added in a few places since the initial patch.
The CoS problem is a minor problem (not the stopper for commit). > BTW if there is interest, it would be possible for these nodes to be displayed with a grey label - that would be nice > The New File wizard could be enhanced to make it even harder to add a generic-type file to an unsharable subfolder, but this might be overkill.) Agree. That can be reported as another enhancement. New File (HTML) wizard can be invoked from a subfolder of Generated-Sources (Logical view). De facto it's the same as invoking file wizard from build folder (Files view).
> #1 is reproducible, but as I said it's not a bug. You must generate sources first, then CoS works. IMHO it is. Imagine opening existing project (from VCS) and Run it. Anyway, this can be reported separately.
Whenever you have nondefault build steps, CoS cannot be used unmodified, which is why CoS is usually disabled in such projects. Perhaps j2seproject could, before doing CoS quick actions, check if there were any added or overridden deps of the 'compile' target and run them if so - that would make projects with source generation work under CoS even after a fresh checkout. I think you do not want to run source generation targets every time you do a quick action (too slow & noisy), so perhaps the IDE could instead offer to run such targets whenever the project is opened, perhaps only if there is no ${build.dir} yet and CoS is enabled. Best to file it separately for independent consideration.
All seems well on XP. I made the Projects tab nodes appear in gray. Added and (lightly) tested support in web.project, j2ee.ejbjarproject, and j2ee.clientproject. For the last, the new Projects tab nodes do not get displayed because it is not yet using the common API; when it is fixed to do so, the new nodes will appear. core-main #f52a57923868
Looks like this caused major problems for web.project and others; seems they were already using build.generated.dir=build/generated for unrelated purposes. Will need to pick a new property name.
Fixing that in core-main #1500ef87bc68.
Integrated into 'main-golden', will be available in build *200902010201* on http://bits.netbeans.org/dev/nightly/ (upload may still be in progress) Changeset: http://hg.netbeans.org/main/rev/1500ef87bc68 User: Jesse Glick <jglick@netbeans.org> Log: #105645 amendment: use ${build.generated.sources.dir} to avoid conflict with existing ${build.generated.dir} properties.
The Display name of Generated Sources node is not visible (IDE running in JDK1.6). See the picture. In Jdk1.5 - it's OK.
Created attachment 76449 [details] Generated Sources node
I expect ${build.generated.sources.dir} should be used instead of ${build.generated.dir}. When all * supports will be modified to use ${build.generated.sources.dir} the ${build.generated.dir} property can be removed, right ?
build.generated.dir seems to be used by: - old WS* supports, which should eventually be changed as you say - <jspc> in web.project, which I guess would stay For the JDK 6 display problem: core-main #5bd90db5c21c
Thank You a lot, I've modified JAX-WS support to use new functionality: http://hg.netbeans.org/main?cmd=changeset;node=e8bae77a5d1f http://hg.netbeans.org/main?cmd=changeset;node=edb84b36b1fa http://hg.netbeans.org/main?cmd=changeset;node=5b8510b3b0d1 Except of missing node in Logical View, the JAX-WS client in j2ee client application works fine. (I reported an issue 157790)
You may wish to edit/extend the entry in http://wiki.netbeans.org/NewAndNoteworthyMilestone2NB70 with details about the usage of the new system in JAX-* functionality. BTW in #5b8510b3b0d1 you add whitespace to the end of a line. Editor bug?
Re. "j2ee.clientproject is not yet using the common API" - yes, there was issue 131734 requesting that Re. "<jspc> in web.project using build.generated.dir" - yes, that should stay as is; Web project has in its properties option to generate (and compile) Java classes from JSP files and these are generated to build.generated.dir/src build.generated.dir/classes but as far as I know the only purpose is verification that JSPs are compilable - generated java/class files are not used anywhere else
Integrated into 'main-golden', will be available in build *200902030229* on http://bits.netbeans.org/dev/nightly/ (upload may still be in progress) Changeset: http://hg.netbeans.org/main/rev/5bd90db5c21c User: Jesse Glick <jglick@netbeans.org> Log: #105645 amendment: Label.disabledForeground did not work on JDK 6; using controlShadow instead.
These are another changes related to conversion to new functionality : JAXB support : http://hg.netbeans.org/main/rev/d7b4356d8874 http://hg.netbeans.org/main/rev/3f667d8391a0 JAX-RPC support : http://hg.netbeans.org/main/rev/7af00310fde8 http://hg.netbeans.org/main/rev/7432519375f1
How can I disable this feature so that generated sources will not be added to classpath ??
piastucki - if you don't want entries in your classpath, generate sources to some other folder.
Jesse is right. Don't allow IDE to generate java classes into ${build.generated.sources.dir}. Just change this in build script. Nevertheless: Why do you want to generate java classes and don't want to use them ?
The application I work on allows Java code to be customized on a per-customer basis. In order to do that the build script copies Java sources and then overwrites selected classed with customized versions. The result is placed in a newly created folder within the workspace. This is legacy code and there is very little I can change. NB automatically finds Java files in this new folder and I end up with duplicated classes. As a result I need to check the path any time I want to use Go to type etc. The application is set up as free form web application in NB - what can I change it this case?
One solution is to change the names of custom classes to avoid conflict with generated classes. Another solution is to have custom classes in a specific folder, that is not under source root.
> One solution is to change the names of custom classes to avoid conflict with generated classes. If their names are changed then they will not override the classes they are supposed to override. >Another solution is to have custom classes in a specific folder, that is not under source root. Not acceptable as I want to be able to edit and commit custom classes. Just to clarify what the problem is. The build script works like this: 1) create a folder with Java source files in workspace, let's assume it is called src2 2) override some of these files with custom ones (i.e. copy files from src to src2) 3) compile the files in the src2 folder NB will find src2 folder and all Java files inside it. The files mentioned in step 2) are present in both src and src2 folders and thus NB will see the same class twice. I still think the best solution would be to disable this functionality for selected folders/whole project.
This feature only deals with source roots specifically created as direct subdirs of ${build.generated.sources.dir}, e.g. ${basedir}/build/generated-sources/whatever/org/domain/package/Class.java. "NB will find src2 folder" is likely some unrelated issue which I cannot comment on. Use nbusers@netbeans.org, or file a separate issue report if you can attach a complete, self-contained, reproducible test case.
Jesse, where should I specify build.generated.sources.dir in case of free-form web project?
This feature does not apply to freeform projects at all. For a freeform project, you can define a source root normally wherever you like whether it is generated or not. Please use nbusers@netbeans.org for any further questions on this topic.
Thanks for the info. Based on what you said, this functionality appears to be broken as it it DOES apply to free-form projects too. I use 6.7M2 and NB automatically adds generated sources to source folders. Even if I go project->properties and remove this folder from source folders it will be automatically added by NB again.
piastucki - whatever you are talking about (which I cannot quite follow) is likely unrelated to this feature. If you believe there is a defect in NB, and can supply steps to reproduce so we can diagnose it, please file it as a separate issue in the appropriate component (web/freeform).
jglick - the problem IS related to this feature. Steps to reproduce: 1) create a new web free-form project with all sources in src folder in the workspace 2) in Project Properties -> Java Sources select src as the only folder with Java sources 3) create build folder in the workspace 4) copy src folder to build folder 5) NB will add build/src to Java sources automatically e.g.: a) build/src will be displayed in gray in project view b) build/src will be displayed when you go to Porject Properites -> Java Sources 6) As a result of 5) all classes are shown TWICE in NB 7) Going to Porject Properites -> Java Sources and removing build/src does not help as NB will add it again and again
Once again - this issue did not touch the freeform project type at all. The change was made only to certain IDE-managed project types, a different set of modules. Whatever issue you are having is likely unrelated; please file it separately in web/freeform as previously requested.
Yes, I was able to reproduce the issue you described in free-form project. It's really annoying. You cannot get rid of these sources in project customizer. However, it's completely irrelevant to generated sources functionality implemented in this feature. Please, report a new bug for this.
Integrated into 'main-golden', will be available in build *200904020200* on http://bits.netbeans.org/dev/nightly/ (upload may still be in progress) Changeset: http://hg.netbeans.org/main-golden/rev/81c43271b151 User: Jesse Glick <jglick@netbeans.org> Log: Not entirely clear what this used to be testing, but it fails currently; was probably obsoleted by #105645.
It would be nice if the NetBeans Modules Projects would support this as well.