When using Debug->Attach Debugger, the connection appears to be using the NB JDK, not the Project's Platform JDK, which is what I would expect.

This problem exists on NB 8.1/9.

If I build a jar using JDK 8u101 as the Java platform, then start a process running as:

% D:/java/bootdirs/jdk1.8.0_101/bin/java \
  -agentlib:jdwp=transport=dt_socket,server=y,suspend=y,address=5900 \
  -jar "C:/Users/brw.ORADEV/Documents/NetBeansProjects/FileTest/dist/FileTest.jar"
Listening for transport dt_socket at address: 5900

then try to attach the debugger using:

Debugger:    Java Debugger (JPDA)
Connector:   SocketAttach (Attaches by socket to other VMs)
Transport:   dt_socket
Host:        *My_hostname*
Port:        5900
Timeout[ms]: <blank>

where "My_hostname" is the local name of my laptop.  It works just fine.

But if I switch to using JDK 9 as the JDK (same built jar), and now use the JDK 9 Java platform on my NB project (Project: Properties->Libraries->Java Platform->JDK 9 exploded build)

% D:/java/ws/dev/build/windows-x86_64-normal-server-fastdebug/jdk/bin/java \
  -agentlib:jdwp=transport=dt_socket,server=y,suspend=y,address=5900 \
  -jar "C:/Users/brw.ORADEV/Documents/NetBeansProjects/FileTest/dist/FileTest.jar"
Listening for transport dt_socket at address: 5900

Using the same debugger attachment menu as above, it won't connect, and I get the INFO/exception:

INFO [org.netbeans.api.debugger.jpda.AttachingDICookie]: Attaching Connector = com.sun.jdi.SocketAttach (defaults: timeout=, hostname=My_hostname, port=), arguments = {timeout=timeout=, hostname=hostname=My_hostname, port=port=5900}

java.net.ConnectException: Connection refused: connect
        at java.net.DualStackPlainSocketImpl.connect0(Native Method)
        at java.net.DualStackPlainSocketImpl.socketConnect(DualStackPlainSocketI
mpl.java:79)
        at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.ja
va:350)
        at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocket
Impl.java:206)
        at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java
:188)
        at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:172)
        at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
        at java.net.Socket.connect(Socket.java:589)
        at com.sun.tools.jdi.SocketTransportService.attach(SocketTransportServic
e.java:222)
        at com.sun.tools.jdi.GenericAttachingConnector.attach(GenericAttachingCo
nnector.java:116)
        at com.sun.tools.jdi.SocketAttachingConnector.attach(SocketAttachingConn
ector.java:90)
[catch] at org.netbeans.api.debugger.jpda.AttachingDICookie.getVirtualMachine(At
tachingDICookie.java:231)
        at org.netbeans.modules.debugger.jpda.actions.StartActionProvider.doStar
tDebugger(StartActionProvider.java:188)
        at org.netbeans.modules.debugger.jpda.actions.StartActionProvider.access
$200(StartActionProvider.java:80)
        at org.netbeans.modules.debugger.jpda.actions.StartActionProvider$1.run(
StartActionProvider.java:165)
        at org.openide.util.RequestProcessor$Task.run(RequestProcessor.java:1443
)
        at org.netbeans.modules.openide.util.GlobalLookup.execute(GlobalLookup.j
ava:68)
        at org.openide.util.lookup.Lookups.executeWith(Lookups.java:303)
        at org.openide.util.RequestProcessor$Processor.run(RequestProcessor.java
:2058)

This is definitely a JDK 8u101 stack trace. 
So "My_hostname" as a hostname works when the Project JDK platform is 8, but fails when it is 9.

HOWEVER, when I change HOST to *localhost*, it works fine in both cases.