This Bugzilla instance is a read-only archive of historic NetBeans bug reports. To report a bug in NetBeans please follow the project's instructions for reporting issues.
Hi I have been using netbeans for almost a year now. There is one issue which really gets to be a slight pain sometimes. When i have to open very big files, nb always popups the warning asking the user whether he wud like to open it or not. if it opens it will cause Out of Memory error. 1> Here i dont like the idea. See, I have to open the file in order to work on it. There is no use showing the popup. 2> I think even id there was a reason to put it in, please look at this way. As we know, it has to do with memory at the end. What if my machine has a good RAM .Here u see , there might be no issue at all to handle the file For example netbeans can detect if possible, the size that wud be avaliable to it via the Runtime class API in java and do an intelligent guess as to whether it can handle the file or not ? What I am trying to say is essentially: 1> I am using netbeans as the IDE of my choice and cant switch to another just to open the big file. 2> Sometimes, this warning is useless My Request: Please remove the popup OR make it configurable somewhere top show the popup or not -swaroop belur
I think we could add button (checkbox) "Don't show again". I am not sure about the component, so Petr feel free to reassign ...
DataEditorSupport in loaders module. Anyway, it's just a heuristic. And no, you don't need to open the file to work with it. There are files that can really take the IDE to its knees and you won't be able to work with it anyway. What's the threshold is hard to decide, it depends on the file size, file type, installed featureset and so on.
Hi This thought just occured to me. I dont know if you guys find it correct ? If the file is too big and say its jav file , can nb offer a way to open the file in say "reduced -feature" mode . For example in text mode. This will be the simplest. Somewhere a right click against the file and in the pop-up window also in some manner -i.e provide a means to open the file in alternate manner Ofcourse you can warn the user, that file will be opened in a reduced feature mode or something like that ? Like may be for java file say 1> skip syntax highlighting check 2> skip code folding check and expand/collapse Somethings like the above and may be nb team can think of more. Some option like the above + say keep simple/basic refactoring options like rename for example , i think will help nb cope with v ery big files? -swaroop belur
I'd be happy to implement anything guys from editor tell me to handle big files. My ideas would include a special EditorKit and mimetype for big files that would allow editing of memory mapped, etc. So editor guys, it is up to you. When you find a way to open big file, let me know I can modify openide/text or openide/loaders.
I am not sure why we can't open big files, besides of the commonly known thruth that big files need a lot of memory of course. But how big are big files and how much memory they need in the IDE? If we are serious about this we should start measuring the memory consumption for some common file types - eg text/plain and text/x-java for the beginning - and look at what features take what amount of memory. I agree that poping up a dialog is not very useful and crashing with OOME is even less useful. I always thought that it would be generally handy if people were able to open files in alternate editors, such as opening a struts config file as a plain xml or a java file as a plain text. If we had that option we could warn a user when opening a big file and suggest opening it in the plain text editor. We could even try to catch a potential OOME if the user chooses to open the file in the file's own fully fledged editor and fall back to the plain text editor automatically. That would of course need the plain text editor to work reliably with files of pretty much any size, which brings us back to the beginning. If the plain text editor is only marginally more memory efficient then say java editor, it will not provide for a good fallback. But there might be a way how to improve it or perhaps how to improve all editors. So, we really need some numbers.
Let me state first that I'm not trying to dismiss this issue. Here I'm just collecting the data necessary for finding the balance between potential implementation complexity(cost) and the benefits from implementing it. OK, before we get technical, let's look at the use cases. We don't like to invest into complicated infrastructure to support next-to-nonexistent use cases, do we? So what kind of large files is the user likely to encounter? The biggest java file I have ever seen was X.java from ProjectX (video processing suite), accounting for 500KB and 17.000 lines. The file is basically ineditable, not because of the editor but because of the lack of structure. There's no technical reason for not splitting the file/code among more classes. In fact, that file was obviously a result of architectural and technical incompetency on steroids, yet our editor handled it. For xml, I believe that there are some large WSDL files out there that may need editing, yet we are likely to hit some other wall (tree view updates) before editor's OOME. But generally, I don't believe humans should write or edit such large files (view them to verify how they were generated, right, but not edit) or at least not commonly. Now to the technical issues. Our editor (plain) has nontrivial (>100B) overhead per line (not per byte) of the text. If you add features (annotations, syntax/semantic highlighting, support for navigation), the figure multiplies (sorry, no exact numbers at hand). Even if we managed to take the document off-memory with smart indexing, the rest of the IDE would consume the memory much faster than how we spared it. Regarding catching the OOME - this is nice idea on the first sight but has a lot of problems. Other thread may be the one that fails and it is even very likely in this case - editor just opens the file OK with a plenty of headroom initially, but all kinds of asynchronous/delayed/on demand stuff kick in and consume much more memory later. Not talking about the usability of the IDE near its memory limit. I tried to agree with the limit proportional to available memory, but found myself writing phrases like "power user", which would, in fact, benefit more from "don't warn again" check box. Also, I realized we can quite easily provide simple _viewer_ for files of virtually any size (output window handle it right with partial mmaps()). The question is whether it would cover enough use cases.
My post collided with Petr's one, so just briefly: currently there is ~200 bytes per line (includes positions, line-element, line-view). Plus the number-of-characters-on-line * 2 bytes-per-char. Questions affecting design: 1. Should big-file editor be just read-only viewer of the file? 2. Should BFE mmap its content? Problem: Swing convention is that Document.getText() returns '\n' line separators only. For mmap-ed file there would have to be a structure that would allow conversion. BTW does anyone know how does the byte->char conversion work for mmap-ed files and if there's any support for an offset conversion work e.g. in utf8 2 bytes may become single character etc.? IMHO there will always be a certain overhead per-line so for a file with too many lines the IDE still may run out of memory. In case we agree on read-only viewer the impl. from output window shoul IMHO be reused.
Looks like this hasn't been the first time we talk about it - issue #28662. Duplicate?
*** Issue 28662 has been marked as a duplicate of this issue. ***
I have duplicated the older issue 28662 to this one since this one imho contains more useful information. One thing I did not realized (but present in the notes of the other issue): In order to support the not-fully-in-memory documents we will likely be forced to update constraints for certain functionality in the editor. For example current word-match Ctrl-K/L scans the contents of the current doc plus all other opened docs. This needs to be performed in fractions of second. So certain functionality would IMHO have to be disabled for not-fully-in-memory docs.
*** Bug 154326 has been marked as a duplicate of this bug. ***
The decision should be up to the user via a pop-up dialog and have the option to save the decision (Do Not ask again checkbox; along with an entry in Tools-->Options-->Editor... entry where it can be changed). Option 1) 'The user can choose' to cancel the task and not open the document. Option 2) 'The user can choose' to ignore the warning and open the document in Full editor mode at their own risk (Thankfully we can already achieve this by setting a system property '-J-Dorg.openide.text.big.file.size=25', Personally I currently use 25mb to reduce the occurrence of this issue). Option 3) User can choose an external editor (command) and netbeans will launch it with the selected file as an argument. Similar to the QuickOpener or CoolEditor plugin. Option 4) Open the file in a plain text editor/viewer, use mmap (to avoid loading the entire file into memory), and disable all editor features that require the entire file to be loaded into memory (thus significantly reducing the time to open/load the text viewing window). Any highlighting... features should be done only on the currently visible portion of the document. Allow find/search, but use progress bars... to allow user to cancel those possibly lengthy operations. IMHO read-only mode is fine, but if enabling editing of a mmap based plain text file can be accomplished without too much overhead, then I see no reason not to add it as a nice bonus feature. Not very many free editors out there handle large text files very well, so It would be great if netbeans could support this as an open source java based solution (that others can use and build on using netbeans RCP framework). It is NOT a RARE USE-CASE to want to open/view a large file, but I agree that it is rare to want to EDIT a large (10mb+) file. At a minimum, netbeans should be able to do this without OOME or crawling to a stop. Netbeans is not just used for editing small java files!!! Netbeans is an RCP client with countless use-cases. I am currently using netbeans for an RCP and will likely have to end up working around this issue by writing my own memory mapped text viewer for opening large text files (I am likely not the only one that will end up having to reinvent the wheel to do this, and wish there was already some built-in support for this). Many of the editor... packages are private/final... so I will likely have to build my own custom version of netbeans from source....(pain...)... create my own delegate for OpenAction...; Many other users have come across this issue for almost 10 years now and this issue is still not addressed. For every one person that reports this, there are likely hundreds that come across it, but don't report.
I would like to add my voice to this issue as well. This is very annoying. The various solutions already presented seem reasonable. In the meantime isn't there anyway around it such as opening it for viewing only? My file is 6MB big. I tried getting a ViewCookie for a file with a .log extension and it was not supported.
I'd like to add my vote to better handling of large files. Specifically, I regularly have to handle large XML files - on the order of 10-100mb - and that brings NB to its knees on an i7 machine with 16GB of memory. Since the CPU is pegged at near 100%, I'm guessing it's not a memory issue, but NB trying to 'grok' the XML structure. Why do I look at large XML files? Usually it's to find a single data item and check its surrounding data, in order to figure out why my app is having trouble with ingesting/processing the file. Today, I had to kill the NB process when I tried to open a 33MB XML file. I had hoped to use NB to more easily traverse it's "structure" (e.g. see the end tag of a highlighted begin tag) - since I needed to determine what parts of this huge XML I was interested in and write a SAX parser fort those parts. Someone mentioned a "viewer" - that would be cool by me - as long as that viewer can apply formatting to the file (e.g. indentation for XML files) and let me jump between the beginning of a code block or xml tag and its end.
Common editor team, please give Jaroslav Tulach(see comment above) the go ahead to implement the special memory mapped EditorKit and mimetype, and to modify openide/text and/or openide/loaders as needed. The approach of changing(overriding) the mimetype(to say 'text/large') for large(user configurable by system property) text files, and using a special EditorKit to view/edit such files seems like the best approach. Having a special Editor and mimetype would make opening large files work quick and smooth since the text editing features of original mime type will be disabled (i.e. parsing, syntax highlighting, auto-complete... and general loading into memory will all be disabled). Just provide a simple plain vanilla text editor for large text files. This would be a really cool/useful feature. And make the IDE much less annoying to use, when you want to view a large text/log file or xml quickly.
Our typical use case: - Large log files. - More seldom XML data files. - Mostly not in the project tree. I.e. opened via the command line. It would be enough to have the pop-up offering the simplified editor as an alternative. Simple editing would be nice, though. E.g. to insert markers, comments or to truncate the file and cut out the relevant parts, or to modify small portions of a large input file for testing purposes. A button, to re-open the truncated file in a fully fledged editor would be a breeze. Hope, this helps understanding use-cases.