This Bugzilla instance is a read-only archive of historic NetBeans bug reports. To report a bug in NetBeans please follow the project's instructions for reporting issues.
Apologies for filing this as a P1, but I believe its's justified because a lot of _source_ files got _deleted_! Just to get it off my chest ... this has been my worst NB experience ever! I am _VERY_ mad ... because running (or ^C'ing the NB running ...) the profiler somehow ended up deleting a whole bunch of files including .java and .mf files! Now I have to spend, who knows how long, recovering stuff. And, no, LocalHistory->RevertDeleted doesn't help. After running and interrupting the profiler the Projects view ended up falling back on CNB's instead of project names. As I dug deeper I discovered that a whole bunch of files are missing. Whole .java files wre gone! .mf files were gone! This is why the Bundle file couldn't be found for presenting non-pkg project names. Lucky for me I had backed up all the files under lib.terminalemulator tonight but I can't quite say that I lost one evenings worth of work because files which I hadn't modified had vanished as well! I can provide a "diff -r" of before and after upon request. At the moment my first priority is to recover my work :-( I'll only be in position to try an reproduce the problem and demonstrate that it's not "user error" after that ... although i'd like someone to show how the before-after diffs demonstrate a user error. To reproduce ... I had a suite with a combination of NB modules, plain java libraries and library wrappers. It also included JNA-based code. I chose project-context-menu->Profile. There were a few "calibrating CPU" dialogs followed by a lot of errors about something I can't remember. Eventually the profiled NB hung and I somehow cancelled the run. After that the first visible symptom was that a bunch of my open projects showed pkg names instead of project names. This I eventually traced to missing .mf files, _after_ discovering that whole .java files were missing. I ^C'ed out of that NB (hoping to deny it saving it's corrupted project data) and cleaned my userdir before restarting. Unfortunately this means I don't have a precise record of gestures. I googled a bunch of patters and looked through the NB bug database but didn't run into anything similar.
*** Issue 150011 has been marked as a duplicate of this issue. ***
I'm sorry for the disaster but I'm quite sure there is no code in the profiler that would intentionally or unintentionally delete the project sources. The only parts we touch is $temp and build file (which is never deleted, only modified). Unfortunately there is no way to tell what went wrong without the additional data (messages.log especially).
You can check Issue 138317 - maybe you are experiencing the "Too many open files" problem described there. Unfortunately you haven't provided _any_ configuration details so we cannot proceed further with this bugreport.
I"ll be doing some forensics shortly. It was waay too late last night to do any more analysis.
What happenned to me was pretty much the same as described in Issue 138317. And telling me that there is a workaround is rather like shutting the barn door after the horses have feld :-) I'm running on Fedora6 with a fd limit of 1024. I've been trying to figure a pattern for what files were removed but it's not obvious. Some src/ files were removed but also removed were toplevel build.xml, apichanges.xml, manifest.mf files and such. Also removed were files in the "platform", like nbbuild/templates/default.xml. The removal patterns do not coincide with ... - some sort of "rm -rf" ... files are deleted selectively. - src files. One could theorize that profiler removed src files that it was profiling but that is clearly not the case. It seemed like no directories, like pkg directories, were removed ... only their contents were removed. Because files were removed from the platform (which was a trunk clone) I, instead of trying to ferret out exactly what was removed, basically made a new clone. This new clone was not buildable with NB 6.1, so I switched to NB 6.5 which has the profiler excessive opened file bug fixed. Ergo it will take me a bunch of work,over what I have already suffered, to reproduce the problem. However, it occurs to me that you can try an reproduce the problem by artificially reducing the fd limit and, say, adding a stack dump to security-managers delete file interposer. I understand that profiler is unlikley to be directly guilty in this whole affair, but _somebody_ needs to take the lead on following up on this, most nasty, problem. I got the impression that the filer of Issue 138317 just got disgusted and abandoned NB. Perhaps the core team should be nudged to look into this? After all, deletion of files is a pretty serious matter. Because the file deletion happens during the exceptional condition of running out of fd's, it seems that the place to look is in catch/finally clauses which might be doing something like File.delete().
If you provide reproducible test case, we will be happy to investigate. We did try to reproduce it on NB 6.1 with artificially lowered fd limit, but we failed.
Tried to reproduce this issue, but failed. Need more detailed steps.
Looking at the logfiles/comments the profiled code is versioned by mercurial (Ivan please correct me if I'm wrong) so you should try to reproduce it for example for a NetBeans main clone with mercurial plugin enabled in the IDE. Were you able to at least reproduce the Too many files open problem??
Can not reproduce Too many files open problem with NetBeans 6.1 on linux platform, and reduced number of open files to 500. Profiler works well for me when profiling NB modules from NetBeans clone.
I'm sort of resigned to reproducing and even finding the problem myself. 6.5 has the fd leak plugged so it's only reproducible with 6.1 as the _outer_ IDE. I'll also need a "large" chunk of code to profile, ergo an "inner" IDE. The inner one has to be 6.1 as well since 6.1 cannot work with 6.5 AFAIK. Additionally I'd like to instrument the outer IDE's security manager to dump a stack trace on every file delete. But all of this takes preparation ...
Using NetBeans 6.5 RC1. Have just experienced having a huge number of source files deleted by NetBeans, during deployment though, not profiling. Occasionally have seen the too many open files error, but haven't yet increased the limit. This bug exists.
I have just been hit by this bug for the second time in two days, this time under 6.1. Both times that this has happened I've been deploying to JBoss. Perhaps the fault lies in the deployment part of the server integration. I can provide messages.log and uigestures files if they help.
Please provide as much details/resources as possible. Gestures/ IDE logfile / JBoss logfile will be definitely helpful!
Not a profiler bug. Updating summary and assigning to IDE component for further investigation.
To:colinwsbc - Please provide as much details/resources as possible. Gestures/ IDE logfile / JBoss logfile will be definitely helpful!
Created attachment 73280 [details] gestures and messages log files (.tar.gz)
I've managed to reproduce it. Product Version: NetBeans IDE 6.1 (Build 200804211638) Java: 1.6.0_10-rc2; Java HotSpot(TM) Client VM 11.0-b15 System: Linux version 2.6.9-1.667 running on i386; UTF-8; en_US (nb) Steps: 1. set open files: ulimit -n 600 2. start NB from terminal 3. open project "lib.terminalemulator" from "release65" clone and expand project root 4. profile the project. CPU profiling 5. clean and build the project 6. profile the project again 7. (i've created folder which contains 1031 plain empty files) - from with "Favorites" - I've added that directory, expanded it 8. profile project again "apichanges.xml", "arch.xml", "build.xml", "manifest.mf", "ReleaseNotes.ivan.txt" files are removed. Files were removed, invoking hg status shows that "delete" has been performed (status for all files is "!"). "hg rm" hasn't been used (status results in "R" if "hg rm" is being used)
In terminal: there was just: (<unknown>:3857): Gtk-WARNING **: Error loading theme icon for stock: Failed to open file '/usr/share/icons/Bluecurve/48x48/stock/gtk-dialog-warning.png': Too many open files
Reproduced with patched JDK, which prints out stack trace and file name for deleted files. Below is the info about deleting lib.terminalemulator/build.xml and lib.terminalemulator/manifest.mf, full log attached. Delete: /usr/hg-main/release65/lib.terminalemulator/build.xml java.lang.Exception: Stack trace at java.lang.Thread.dumpStack(Thread.java:1206) at java.io.File.delete(File.java:905) at org.netbeans.modules.mercurial.MercurialInterceptor.fileDeletedImpl(MercurialInterceptor.java:164) at org.netbeans.modules.mercurial.MercurialInterceptor.access$100(MercurialInterceptor.java:69) at org.netbeans.modules.mercurial.MercurialInterceptor$1.run(MercurialInterceptor.java:111) at org.openide.util.RequestProcessor$Task.run(RequestProcessor.java:561) at org.openide.util.RequestProcessor$Processor.run(RequestProcessor.java:986) Delete: /usr/hg-main/release65/lib.terminalemulator/manifest.mf java.lang.Exception: Stack trace at java.lang.Thread.dumpStack(Thread.java:1206) at java.io.File.delete(File.java:905) at org.netbeans.modules.mercurial.MercurialInterceptor.fileDeletedImpl(MercurialInterceptor.java:164) at org.netbeans.modules.mercurial.MercurialInterceptor.access$100(MercurialInterceptor.java:69) at org.netbeans.modules.mercurial.MercurialInterceptor$1.run(MercurialInterceptor.java:111) at org.openide.util.RequestProcessor$Task.run(RequestProcessor.java:561) at org.openide.util.RequestProcessor$Processor.run(RequestProcessor.java:986)
Created attachment 73315 [details] message.log with extended logging
Some of the deleted files below: lib.terminalemulator/ReleaseNotes.ivan.txt lib.terminalemulator/apichanges.xml lib.terminalemulator/arch.xml lib.terminalemulator/build.xml lib.terminalemulator/manifest.mf nbbuild/build.properties nbbuild/build.xml nbbuild/cluster-description.properties nbbuild/cluster.properties nbbuild/default-properties.xml nbbuild/default.xml nbbuild/jdk.xml nbbuild/l10n.patterns nbbuild/monitor.xml nbbuild/standard-nbm-license.txt nbbuild/tagref nbbuild/translate-patch nbbuild/translations nbbuild/validate-project-xmls.xml xtest/lib/driver.xml xtest/lib/harness.xml xtest/lib/kill.exe xtest/lib/lib.jnikill.amd64.x86.dll xtest/lib/lib.jnikill.linux.i386.so xtest/lib/lib.jnikill.macosx.ppc.dylib xtest/lib/lib.jnikill.solaris.sparc.so xtest/lib/lib.jnikill.solaris.x86.so xtest/lib/lib.jnikill.win32.x86.dll xtest/lib/lib.memory-measurement.win32.dll xtest/lib/memory-measurement.unix.sh xtest/lib/module_harness.xml
To colinwsbc: could you please let us know if the application being deployed to JBoss is versioned by mercurial or any other versioning system? We need to check if you are experiencing the same problem or a different one with the same symptoms. Thanks!
The projects I were deploying were versioned by Subversion. My initial thoughts on the bug (without reading the source, so feel free to ignore) are that NetBeans is attempting to delete a temporary file, but due to the too many files exception being handled incorrectly, the wrong file is identified (somewhere within ProjectUtils) and deleted instead.
Results from our yesterday's investigation: 1) In NetBeans 6.5 Mercurial plugin no longer deletes files in MercurialInterceptor 2) Filesystems fires FILE_DELETED event - versioning infrastructure listens for those events and updates VCS metadata and some of VCS implementation (like SVN) will also explicitly delete files. We think that firing FILE_DELETED event is wrong and is caused by the fact that java.io.File.listFile() returns null in such situation (too many files open). Master-filesystem implementation incorrectly thinks that particular folder is empty and fires FILE_DELETED event for all known files in such folder. I will attach patch with proposed fix. Reassigning to openide/filesystems
Created attachment 73357 [details] Patch, which correctly handles situation, when File.listFiles() returns null
Thank you for the patch. It seems reasonably and I will integrate it.
The same thing happened to me since NetBeans 6.0 I lost hours of work because of NetBeans deleting my files. I was using the NetBeans Subversion plug-in on a Linux 64bit machine with 64bit OS. If the problem is related to the number of files being opened, I cannot run the profiler because of this exception java.io.FileNotFoundException: /opt/netbeans-6.1/harness/run.xml (Too many open files) It occurs every time on a different file. If I switch to NetBeans 6.5RC2 it works, but it is so slow that my RCP is not usable.
This is the first time I reported the bug I was developing a module for an RCP using subversion http://www.nabble.com/Problem-with-Netbeans-update-td14336721.html#a14336721
Fixed in trunk. QE, please verify. http://hg.netbeans.org/main/rev/f5128636599a
Assuming that the fix works, is there any chance of a back port to 6.1?
yes, if the fix will work then we could backport it into the 6.1 in a patch. Or we could create a jar with the fix that you would manually replace in your IDE. We are trying to reproduce the problem however it seems that it isn't as easy. Is there anybody out there who can reproduce it in 100%? We could prepare a jar for the testing.
I'm happy to test a jar if you can supply it, but it is difficult to predict when the problem will occur. Is it possible to log any "too many open files" errors so that you can tell that the error has happened but has been handled safely?
Created attachment 73381 [details] jar for 6.5, place it into platform9\modules directory. Just for testing purposes! Backup the original jar!
Created attachment 73382 [details] jar for 6.1, place it into platform8\modules directory. Just for testing purposes! Backup the original jar!
I've attached jars with the fix for 6.5 and 6.1. You can use them for testing. You *should not* be able to reproduce it with the patched jars.
I've often found that once I know the exact cause of a bug, to the point that I can fix it, then I can write an artificial testcase to induce it. (For an example see "Testing Race Conditions ch 11 in Yarda's confessions). If I can't write such a testcase then I must not have thoroughly understood the problem.
Integrated into 'main-golden', will be available in build *200811070201* on http://bits.netbeans.org/dev/nightly/ (upload may still be in progress) Changeset: http://hg.netbeans.org/main/rev/f5128636599a User: Jiri Skrivanek <jskrivanek@netbeans.org> Log: #150009 - do not remove children if listFiles() returns null because of I/O failures.
Fixed in release65. http://hg.netbeans.org/release65/rev/164d778712dc
v in 1110 build