Jarda, 
this is yours, isn't it ?

Jesse, I expect these failures to be due to recent changes in ordering you have made. Can you look at the issue?

1. The behavior is random. Sometimes it passes, sometimes it fails (with different results).

2. The test was failing from time to time long before I did anything with relative ordering attributes.

3. NamedFSServicesLookupTest consistently passes for me.

Expected println's from a passing run:

df.children=[InstanceDataObject[MultiFileObject[inst/ordering/A.instance]],
InstanceDataObject[MultiFileObject[inst/ordering/C.instance]],
InstanceDataObject[MultiFileObject[inst/ordering/X.instance]],
InstanceDataObject[MultiFileObject[inst/ordering/B.instance]]]
l.allInstances=[500, 700, 1000, 1500]
[org.openide.filesystems.Ordering] THREAD: Folder recognizer MSG: Relative ordering attribute A.instance/C.instance on
inst/ordering is deprecated in favor of numeric position attributes
[org.openide.filesystems.Ordering] THREAD: Folder recognizer MSG: Relative ordering attribute C.instance/X.instance on
inst/ordering is deprecated in favor of numeric position attributes
[org.openide.filesystems.Ordering] THREAD: Folder recognizer MSG: Relative ordering attribute X.instance/B.instance on
inst/ordering is deprecated in favor of numeric position attributes

Sometimes I get:

df.children=[InstanceDataObject[MultiFileObject[inst/ordering/A.instance]],
InstanceDataObject[MultiFileObject[inst/ordering/C.instance]],
InstanceDataObject[MultiFileObject[inst/ordering/X.instance]],
InstanceDataObject[MultiFileObject[inst/ordering/B.instance]]]
l.allInstances=[700, 1000, 1500]

Note that the FolderLookup here is missing A.instance entirely.

Most often I get:

df.children=[InstanceDataObject[MultiFileObject[inst/ordering/A.instance]],
InstanceDataObject[MultiFileObject[inst/ordering/B.instance]],
InstanceDataObject[MultiFileObject[inst/ordering/C.instance]],
InstanceDataObject[MultiFileObject[inst/ordering/X.instance]]]
l.allInstances=[500, 1500, 700, 1000]
[org.openide.filesystems.Ordering] THREAD: Folder recognizer MSG: Relative ordering attribute A.instance/C.instance on
inst/ordering is deprecated in favor of numeric position attributes

Note that the DataFolder.children is wrong. Curiously, however, Ordering reports a warning only about A/C, which implies
to me that the folder attrs C/X and X/B were not present when it did the ordering (which is consistent with an order of
A-B-C-X).

I have verified that (1) the folder is really created by the test (not leftover from a previous run or whatever), (2)
the test has written all three ordering attrs by the time it calls Lookups.forPath.

My only guess is that the problem is with testOrderingAttributes and testNumericOrdering clashing over a folder, so I
have tried to use a different folder in the latter, which seems to help. BTW the suite should really set a new
Repository in setUp to prevent any interference between test cases.

Checking in NamedFSServicesLookupTest.java;
/shared/data/ccvs/repository/core/startup/test/unit/src/org/netbeans/core/startup/layers/NamedFSServicesLookupTest.java,v
 <--  NamedFSServicesLookupTest.java
new revision: 1.4; previous revision: 1.3
done

Still sometimes failing.

Sometimes passes, sometimes fails randomly in ways I explained before (see comments from Tuesday). I suspect there is
simply something asynchronous in loaders or settings which makes this test inherently unreliable.

I patched NamedFSServicesLookupTest:

        System.setProperty("XXX", "true");
        System.err.println("XXX turning on logging for testOA");
        try {
        Lookup l = Lookups.forPath("inst/ordering");
// as before
        } finally {
            System.setProperty("XXX", "false");
        }


and also Ordering:

        boolean xxx = Boolean.getBoolean("XXX");
        if (xxx) System.err.println("XXX children=" + children);
        Iterator<FileObject> it = children.iterator();
// as before...
            Object val = parent.getAttribute(attr);
            if (xxx) System.err.println("XXX " + parent + "(" + attr + ") -> " + val);
            if (!Boolean.TRUE.equals(val)) {
// as before...

Now NFFSLT prints, as expected:

XXX turning on logging for testOA
XXX children=[MultiFileObject@11946c2[inst/ordering/X.instance], MultiFileObject@1f2cea2[inst/ordering/A.instance],
MultiFileObject@10fd7f6[inst/ordering/B.instance], MultiFileObject@32060c[inst/ordering/C.instance]]
XXX MultiFileObject@1124746[inst/ordering](A.instance/C.instance) -> true
XXX MultiFileObject@1124746[inst/ordering](C.instance/X.instance) -> true
XXX MultiFileObject@1124746[inst/ordering](X.instance/B.instance) -> true

Sometimes RIOT prints the same. Usually it prints just

XXX turning on logging for testOA

and fails. Meaning that FileUtil.getOrder is not even being called. I tried to determine why, but quickly became lost in
the maze of callbacks and aynchronous tasks created in FolderList and FolderInstance. All I can figure out is that
FolderList.getChildrenList is called on inst/ordering _before_ Lookups.forPath is called, which doesn't make any sense
to me because there should have been no calls to Datasystems on that folder before then.

Why you used System.err and not LOG.fine? I've added more logging.

IDE:-------------------------------------------------
IDE: [21.8.07 11:10] Committing started
Checking in core/startup/test/unit/src/org/netbeans/core/startup/layers/NamedFSServicesLookupTest.java;
/shared/data/ccvs/repository/core/startup/test/unit/src/org/netbeans/core/startup/layers/NamedFSServicesLookupTest.java,v  
<--  NamedFSServicesLookupTest.java
new revision: 1.5; previous revision: 1.4
done
Checking in openide/fs/src/org/openide/filesystems/Ordering.java;
/shared/data/ccvs/repository/openide/fs/src/org/openide/filesystems/Ordering.java,v  <--  Ordering.java
new revision: 1.5; previous revision: 1.4
done
IDE: [21.8.07 11:10] Committing finished

Test fails again.

Created attachment 47041 [details]
Test report

IDE:-------------------------------------------------
IDE: [22.8.07 15:37] Committing "NamedFSServicesLookupTest.java" started
Checking in NamedFSServicesLookupTest.java;
/shared/data/ccvs/repository/core/startup/test/unit/src/org/netbeans/core/startup/layers/NamedFSServicesLookupTest.java,v  
<--  NamedFSServicesLookupTest.java
new revision: 1.6; previous revision: 1.5
done
IDE: [22.8.07 15:37] Committing "NamedFSServicesLookupTest.java" finished