This Bugzilla instance is a read-only archive of historic NetBeans bug reports. To report a bug in NetBeans please follow the project's instructions for reporting issues.

Bug 117179 - [60cat] Package discovery is slow and does not get cached between invocations
Summary: [60cat] Package discovery is slow and does not get cached between invocations
Status: RESOLVED FIXED
Alias: None
Product: java
Classification: Unclassified
Component: Project (show other bugs)
Version: 6.x
Hardware: All All
: P2 blocker (vote)
Assignee: Milan Kubec
URL:
Keywords: PERFORMANCE
: 117162 117169 117180 119670 (view as bug list)
Depends on:
Blocks:
 
Reported: 2007-09-30 20:25 UTC by wobster
Modified: 2008-02-18 12:56 UTC (History)
6 users (show)

See Also:
Issue Type: DEFECT
Exception Reporter:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description wobster 2007-09-30 20:25:35 UTC
I have a project with about 2000 classes in it. The first time I expand the sources node in the project tree, the parser
loops through all 2000 classes looking for the package structure. This process is always slow and the results don't seem
to be cached between invocations of NetBeans. 

It would seem that the modification times could be checked on the files or directories to see if anything has changed
between invocations and if nothing has changed, just display the exact same package structure as shown before.
Comment 1 Antonin Nebuzelsky 2007-10-01 08:49:37 UTC
*** Issue 117180 has been marked as a duplicate of this issue. ***
Comment 2 Jan Lahoda 2007-10-01 08:58:52 UTC
I believe that Milan is maintaining the PackageView(Children).
Comment 3 Tomas Zezula 2007-10-01 09:23:48 UTC
Right
Comment 4 Tomas Zezula 2007-10-01 10:31:06 UTC
From the dumps attached to issue #117162 it seems that it's caused by excludes.
Comment 5 Milan Kubec 2007-10-02 12:32:30 UTC
Jesse, do you have any idea how to make includes/excludes faster? Related issues are issue #117169 and issue #117162.
Comment 6 Jesse Glick 2007-10-02 15:21:52 UTC
No, I don't know of any particular way to make things faster, unless profiling turns up some microoptimizations.

Persistent caching might help but this seems hazardous. Exactly which "timestamps" you need to record is very likely
dependent on operating system and perhaps even filesystem type. Even if you got it correct, it is not obvious to me that
looking up the timestamps after a restart would be any faster than traversing the package structure is now. I would
recommend spending effort instead on making an uncached scan faster.

It is possible PackageViewChildren would run faster if specialized to use java.io.File in case the source group root
were found to be a plain disk directory (not e.g. ZIP archive). This would certainly save some CPU time (the FileSystem
layer adds a lot of overhead); it might or might not save I/O time, which I would guess is the blocking factor for most
people.

I'm not sure what the relationship to includes/excludes is supposed to be. Even without excludes you still need to show
a package tree in basically the same way. Checking against the filter adds a bit of overhead but I tried to keep this
minimal; normally it is a match of a subpath string against a precompiled regexp. If java.io.File-based scanning is to
be used it is possible that an optional extension interface to SourceGroup would be needed which matches containership
against a String relative path rather than FileObject.
Comment 7 Milan Kubec 2007-10-30 11:51:55 UTC
*** Issue 117162 has been marked as a duplicate of this issue. ***
Comment 8 Milan Kubec 2007-10-30 11:55:11 UTC
*** Issue 117169 has been marked as a duplicate of this issue. ***
Comment 9 Milan Kubec 2007-10-30 14:31:19 UTC
*** Issue 119670 has been marked as a duplicate of this issue. ***
Comment 10 Milan Kubec 2007-10-30 14:34:01 UTC
Increasing priority because of duplicate issue #119670. It seems that the delay may happen either if there is lot of
files in package (thousands) or user expands large package that is on network drive. 
Comment 11 Milan Kubec 2007-10-30 14:42:52 UTC
Other thread dumps are in the duplicate issue #119670.
Comment 12 Milan Kubec 2007-10-31 11:41:23 UTC
Profiling showed that the problem can be indeed in includes/excludes, more precisely in SourcesHelper.contains() method
calling SharabilityQuery. Jesse could you provide some hint about the impl. and if it can be somehow optimized?

I don't think we can fix this now and given the fact that it affects only very large packages (thousands of files) and
also when the package is on network drive, we could probably ask for waiver for 6.0.
Comment 13 Jesse Glick 2007-10-31 20:00:57 UTC
I already disabled the nested project check in SH.SourceRoot.Group.contains for typed source roots; perhaps the call to
SQ could also be done only for untyped roots, under the assumption that most unsharable subentries are going to be only
in the generic source group, while unsharable subentries under a typed root is technically legal but in practice unlikely.
Comment 14 Jesse Glick 2007-10-31 20:03:01 UTC
The call to ProjectManager.isProject could probably also be done conditionally only for untyped source roots, though
this might not provide significant benefit.
Comment 15 Milan Kubec 2007-11-01 13:35:53 UTC
I forgot to put the right kw. I will try those optimizations.
Comment 16 Milan Kubec 2008-02-07 08:31:04 UTC
wobster, please could you try the same use case on new builds of NetBeans 6.1? Some optimizations were done on
filesystem layer that could potentially improve the performance of the operation. Thanks.
Comment 17 Milan Kubec 2008-02-18 12:56:49 UTC
Fixed by calling SharabilityQuery only for untyped source roots as suggested by Jesse. Times of node expansion are now
better.

http://hg.netbeans.org/main/rev/bff2c106def1