This Bugzilla instance is a read-only archive of historic NetBeans bug reports. To report a bug in NetBeans please follow the project's instructions for reporting issues.
I spent a the last two days profiling and optimizing the critical path of loading the module system, since it is the biggest startup-time eater. I also read through all the code that runs during module system init, and cleaned up things that smelled bad - so it's a mix of macro- and micro- optimizations. Also I implemented a couple of things discussed on nb-perf. The attached patches contains all of the changes. All of them are probably not appropriate for 6.0 this late in the game, but some are, and they are easily isolated. All times are from testing on my MacBook - 2ghz dual core, JDK 1.5. What I'd suggest is first applying all of them to get a sense of the potential here; then apply and commit those that are do not seem risky. Major changes: - A major hot spot in startup is parsing module XML files. These are now parsed on first start and cached (just a serialized ArrayList) in the userdir. There is a 1/2 second penalty to creating the cache (but we make it up elsewhere) on first start, and 1-4 seconds saved on later startups, depending whether the bits are warm in the disk cache, and how many modules are in the installation. On subsequent starts, if any module.xml files are newer than the cache, the XML files are read and the cache is rewritten. With testing we could probably do this for 6.0. - Asynchronous loading of projects - (probably not for 6.0) - OpenProjectList now returns a dummy "Loading Projects" project as the only open project, and a warmup task loads the actual project list after the main window is shown (note this required changes in the Start Page component so it updates recent projects after project data is loaded [the patch includes a few threading fixes - it was assuming changes in the open project list were on the event thread]). This will need some tuning to get exactly right, since if there are a large number of projects, the delay before they are all opened can be substantial. But the rest of the UI is available, and showing the main window is not delayed until projects are opened anymore, which is a substantial improvement in just getting the main window on-screen. Minor changes: - Replaced a bunch of uses of synchronized classes with unsynchronized equivalents - Stack, StringBuffer, etc. - Rewrote a bunch of the parsing code in core/startup for module XML files and other config data to use String.split() instead of StringTokenizers, and in some cases eliminated gratuitous looping over the same data. - There are a bunch of sanity checks which are useful early warnings if you are writing a module, but do not need to run in a production environment. Also, unless you are writing module manifests by hand, this data is auto-generated by apisupport and unlikely to be bad. The two below showed up as hot spots during startup: - Dependency.checkCodeName() is called for every dependency every module declares during startup, and it does a bunch of string parsing. That code will now only run if assertions are enabled. - Checks that package dependencies do not include a version number - this also will run only if assertions are enabled - Test for duplicates in dependencies now only runs if assertions are enabled - Rewrote Dependency.create() to use String.split(), and more importantly to be much more readable and shorter - Deleted code from DataObjectPool that checks an always-empty map on every call to DataObject.find() - Deleted the finalizer from o.o.fs.FileLock - whatever that code once did, it was not currently doing anything useful; also deleted the unused Throwable created in its constructor - looks like debug code that was never deleted - Changed XMLFilesystem loading code not to use synchronized java.util.Stack - Changed FileObject.getFileObject() to use String.split(), not StringTokenizer - Tightened up a bunch of code in org.netbeans.Module that was adding Collections.emptySet() to the set of every module's dependencies many times - Did some similar changes (use String.split(), move sanity checks into assertions) in core/bootstrap/...Module I'll try to attach some real numbers using the perf tests soon - I need to make an identical build without the optimizations to compare apples with apples. Tests against a similar build (but w/ slightly different module list) look very promising.
Created attachment 49617 [details] Async project loading (also should apply the welcome screen patch w/ this one)
Created attachment 49618 [details] core/bootstrap patches
Created attachment 49619 [details] core/startup patches
Created attachment 49620 [details] core patches
Created attachment 49621 [details] filesystems patches
Created attachment 49622 [details] datasystems patches
Created attachment 49623 [details] Module system patches
Created attachment 49624 [details] Welcome screen patches (so it updates recent projects once project data is loaded)
Tim, Thanks for your optimizations. Some of them are worthwhile, especially the module list initialization, which really stands out in the startup profile with our current module set. We shouldn't target some of your microoptimizations, though, especially locking related, as this is better solved by JDK. Some of the patches are in fact unrelated to the startup optimization and you'd better file them as separate issues for individual maintainers. I'll go through your list later next month.
> We shouldn't target some of your microoptimizations, though, especially locking related, as this is better solved by JDK. Re the replacement of Stack and StringBuffer - aren't the optimizations that reduce the cost of synchronization in the JDK mainly in JDK 6? For the case of Mac OS X, the last rumor I heard is that the only way to get JDK 6 on Mac may be via buying an OS upgrade. So if they do make a difference on JDK 5 on Mac, and are otherwise harmless (addition of one class - an ArrayList subclass the replicates the pop() and push() methods of Stack), it seems like it would do no harm to apply them. If I'm wrong about most of that work being in JDK 6, of course, we can forget it. I'm in Prague this week, so perhaps we can sit down and go through some of these.
Notes: 1. Idiom: boolean asserts = false; assert asserts = true; if (asserts) { // more expensive checking } 2. Logging in FileLock is useful. Replace with: private FileLock() {} public static FileLock create() { boolean asserts = false; assert asserts = true; return asserts ? new LoggingFileLock() : new FileLock(); } private static class LoggingFileLock extends FileLock { protected void finalize() { if (!valid()) {Logger.log(.....);} // or throw AssertionError, etc. } } 3. Dependency optimizations: can probably be replaced by clever regexps. (%2 == 0 trick is to catch e.g. "foo..bar".) DependencyTest should already check all the corner cases - make sure it still passes! 4. SpecificationVersion would need svuid. 5. Do not catch NoClassDefFoundError. Ignore this patch.
Mostly future, but I'm trying to het the module list cache to 6.0
I was experimenting with module list cache and the results are not as good as expected (on Linux, at least). The problem is that the up-to-date check alone is quite expensive, as it needs to locate the module files (for up-to-date check) in 17 folders (clusters), and stat()ing a short file is nearly as expensive as reading it. Anyway, I'm closing this as a duplicate of issue 110011 (slow cold start), with reference to this issue as source of inspiration. *** This issue has been marked as a duplicate of 110011 ***