I add some comments from other users in mailing list:
[nbusers] Netbeans Startup Indexing Scanning & Compiling. 23.04.2008 and 24.04.2008 (German Date)

------

There's also another strange thing I'm seeing. When the furious scan starts, I see that only one of the CPU is used (I 
have a dual core) - that is, there should be plenty of power with the other one at least for the most trivial editor 
tasks; in spite of this, NetBeans is completely unresponsive. Might it be the memory swap? For what I'm seeing at 
first sight, looking at the page swaps counted by "top", it doesn't seem so.

------

Thanks Jirka,

The project tree I am using netbeans on is quite large so I think that its reasonable to take a lot of time in 
scanning and compiling.

But I do not find reasonable the fact that its compiling projects that are already compiled and no changes have 
happened. The first thing that comes into my head is that the projects being unchanged and compiled are just 
dependencies of changed projects. Does the option "Compile Projects On Classpath" affect startup too?

------

An additional improvement should be to disable mysql scanning at startup. See at var/log/messages.log

INFO [org.netbeans.modules.db.mysql.InstallationSupport]: Looking for MySQL installation /usr/bin/gksu/opt/lampp/lampp
startmysql
INFO [org.netbeans.modules.db.mysql.InstallationSupport]: Looking for MySQL installation
/Applications/MAMP/bin/startMysql.sh
INFO [org.netbeans.modules.db.mysql.InstallationSupport]: Looking for MySQL installation /usr/bin/gksu/usr/bin/svcadm -v
enable svc:/application/database/mysql:version_50
INFO [org.netbeans.modules.db.mysql.InstallationSupport]: Looking for MySQL installation /usr/bin/gksu/etc/init.d/mysql
start
INFO [org.netbeans.modules.db.mysql.InstallationSupport]: Looking for MySQL installation C:/Program Files/MySQL/MySQL
Server 5.0/bin/mysqld-nt.exe
INFO [org.netbeans.modules.db.mysql.InstallationSupport]: Looking for MySQL installation C:/Program Files/MySQL/MySQL
Server 5.1/bin/mysqld-nt.exe

Indexing slowness is a *critical* problem.  IDE is being used for Ruby development and becomes completely unresponsive 
on a regular basis.  I've been sitting here for a half-hour frozen after adding a new project.  This was a non-issue 
in 6.01.  I am going to be forced to roll back; I can't use 6.1 in it's current state.

Socionetbeans, please file separate bug for Ruby projects. This issues is about Java Projects.

wbabachan, can you test it with trunk builds? Or wait for beta2. This should be improved by fix of issue 140680

Generally, shouldn't it be enough to compute a digest based on the public API signature of source files? As I understand
the purpose of scanning it should be necassary for dependent files only when this signature has changed. So modifying
e.g. internals of a method would not trigger a rescan which could avoid a lot more of unneeded scanning overhead. Or am
I missing something?

Frank-Michael

Comments from wobster and jlahoda from issue 118490:

------- Additional comments from wobster Thu Aug 28 03:29:41 +0000 2008 -------

I need to have the data directory on the class path. It does not matter whether I put it on build classpath or the
runtime classpath, NetBeans still insists on indexing this directory. If there is a way that I can add a resource/data
directory on the classpath without having NetBeans index it, please let me know.

I have verified that every time I start NetBeans it loops through many of the directories on my classpath and does more
reindexing. If these were networked drives with deep directories, the reindexing would create a very long startup time
for NetBeans.

I will try to get a log of NetBeans in the act of reindexing my directories. 

Eclipse doesn't seem to do this, why does NetBeans need to? (Not that I like Eclipse...I just use Together Control
Center which is based on Eclipse.)

------- Additional comments from jlahoda Fri Aug 29 10:29:51 +0000 2008 -------

I performed the following experiment:
-I have created a directory which contained ~330000 text files.
-I have added this directory to the Runtime classpath of an J2SE project (Project Properties/Libraries/Run). No indexing
was done, even the cache directory was not created (the directory was not touched).
-the directory was indexed when I added the directory on the Compile classpath (Project Properties/Libraries/Compile),
which is expected (otherwise the CC would not show classes that would be part of such roots).

So, how do you add the directory to the CP? What project type do you use?

I have a few comments:
-even though a few bugs, when the IDE was scanning unnecessarily, were fixed recently (#147438, #146800, #145404+related
bugs), it is still possible that there are similar, unfixed, cases. Logs created by:
-J-Dorg.netbeans.modules.java.source.usages.RepositoryUpdater=0
are a good starting point for finding out why the scan runs and if that can be avoided (although this log may not be
enough in some cases). In some cases, log with FINE instead of '0' (which is much smaller) might be enough.
-it is expected that jars on the (internal) classpath would not change very often - if there is a jar on the classpath,
and the jar can be translated to corresponding source root(s) (e.g. the jar is an output of a project, where the
corresponding source roots are the project's source roots), the jar itself is ignored, and only the sources are used. So
the most often updated jars (project outputs) should not force recompile. This might be a bit more difficult for Maven
projects, though.
-regarding computing MD5 sum of all signatures from a jar file ("public" API is unfortunately not enough - there are
some tricky cases where even inaccessible element can change the actual output of the compiler - different errors
reported, etc.). This is of course possible, but will slow down the initial scan (because of necessity to compute the
MD5 sum of the content). So, I do not think it is good to implement this without data indicating it will improve the
situation (e.g. how common is the case of updating jar that does not have an authoritative source attached?).

So, if anybody is able to reproduce an unnecessary rescan/recompile in a recent trunk/6.5 build, please catch the log
(as above) and attach it here, so we can investigate what is going on.

Thanks.

A suggestion regarding use of MD5 (SHA-1, CRC32, ...) checksums for *.class entries and/or complete JAR files in the
classpath: do not do this by default, but if a binary entry is observed by the IDE to change frequently (e.g. at least
once per day), then silently begin computing its checksum and using that to possibly skip rescans. Then you would only
pay the overhead of calculating the checksum in cases where it is likely to actually save work.

Note that the CPU cost of computing a checksum is pretty small (especially if you use a non-cryptographic algorithm such
as CRC32), and the I/O cost (loading a disk cache entry) is probably covered because you would have had to examine its
contents anyway.

No requested logs were added for a long time, I'm closing this issue for now. 
It the problem still persists in 6.5 feel free to reopen and provide requested information.
Thanks