This Bugzilla instance is a read-only archive of historic NetBeans bug reports. To report a bug in NetBeans please follow the project's instructions for reporting issues.

Bug 77122 - Random deployment failures with http connector code in the stack or thread dump
Summary: Random deployment failures with http connector code in the stack or thread dump
Status: RESOLVED FIXED
Alias: None
Product: serverplugins
Classification: Unclassified
Component: Sun Appserver 9 (show other bugs)
Version: 5.x
Hardware: All Windows XP
: P2 blocker (vote)
Assignee: Vince Kraemer
URL:
Keywords:
: 77118 86529 (view as bug list)
Depends on:
Blocks:
 
Reported: 2006-05-31 04:35 UTC by judytang
Modified: 2007-02-06 09:29 UTC (History)
1 user (show)

See Also:
Issue Type: DEFECT
Exception Reporter:

Attachments
j2ee1.4 sample app (10.17 KB, application/octet-stream)
2006-05-31 04:35 UTC, judytang 		Details
server.log (105.10 KB, text/plain)
2006-06-14 00:31 UTC, judytang 		Details
NB log (45.86 KB, text/plain)
2006-06-14 00:31 UTC, judytang 		Details
e-mail from another tester (5.66 KB, text/plain)
2006-12-07 17:59 UTC, Vince Kraemer 		Details
latest patch (19.30 KB, text/plain)
2007-01-12 05:36 UTC, Vince Kraemer 		Details
Add an attachment (proposed patch, testcase, etc.)

Note You need to log in before you can comment on or make changes to this bug.
Description judytang 2006-05-31 04:35:16 UTC 
I am using 5/30 IDE dialy build with JDK 5.0, SDK build 4 on Win XP

Redeploy the archived stateless ejb app project, it fails sometimes

Steps to recreate:

(1) Create the achived project for the attached stateless ejb app  

(2) Deploy the archived project, it worked fine

(3) Deploy several times after, it fails some times says the deploy failed in
IDE console, but in the server.log, there is no error. This situation occurs
randomly, but I did see several times during my testing.
Comment 1 judytang 2006-05-31 04:35:52 UTC 
Created attachment 30689 [details]
j2ee1.4 sample app
Comment 2 Vince Kraemer 2006-05-31 05:54:22 UTC 
do you have any log available? like the output that made you think that the
deploy failed....
Comment 3 Vince Kraemer 2006-05-31 23:38:18 UTC 
I haven't been able to reproduce this issue
Comment 4 Vince Kraemer 2006-06-08 00:12:58 UTC 
Could not duplicate after 7 June 2006 check-ins.
Re-open with more details if these re-appear
Comment 5 judytang 2006-06-14 00:31:04 UTC 
Created attachment 31021 [details]
server.log
Comment 6 judytang 2006-06-14 00:31:40 UTC 
Created attachment 31022 [details]
NB log
Comment 7 judytang 2006-06-14 00:45:47 UTC 
I am using 6/12 IDE dialy build with JDK 5.0, SDK build 4 on Win XP

After deploy the j2ee 1.4 stateless sample app, the server.log shows the
following exception, however the following successful message is issued.  Is
this a bug some where.

CORE5022: All ejb(s) of [stateless-simple] were unloaded successfully!
 
-------------------- server.log ------------

WEB0100: Loading web module [stateless-simple:stateless-simple.war] in virtual
server [server] at [/helloworld]
StandardWrapperValve[RemoteJmxConnectorServlet]: Servlet.service() for servlet
RemoteJmxConnectorServlet threw exception
java.io.StreamCorruptedException: unexpected block data
        at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1313)
        at java.io.ObjectInputStream.readArray(ObjectInputStream.java:1628)
        at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1293)
        at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1912)
        at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1836)
        at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1713)
        at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1299)
        at java.io.ObjectInputStream.readObject(ObjectInputStream.java:339)
        at
com.sun.enterprise.admin.jmx.remote.server.servlet.RemoteJmxConnectorServlet.readRequestMessage(RemoteJmxConnectorServlet.java:192)
        at
com.sun.enterprise.admin.jmx.remote.server.servlet.RemoteJmxConnectorServlet.processRequest(RemoteJmxConnectorServlet.java:108)
        at
com.sun.enterprise.admin.jmx.remote.server.servlet.RemoteJmxConnectorServlet.doPost(RemoteJmxConnectorServlet.java:180)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:727)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
        at
org.apache.catalina.core.ApplicationFilterChain.servletService(ApplicationFilterChain.java:397)
        at
org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:278)
        at
org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:566)
        at
org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:536)
        at
org.apache.catalina.core.StandardContextValve.invokeInternal(StandardContextValve.java:240)
        at
org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:179)
        at
org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:566)
        at com.sun.enterprise.web.WebPipeline.invoke(WebPipeline.java:73)
        at
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:182)
        at
org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:566)
        at
com.sun.enterprise.web.VirtualServerPipeline.invoke(VirtualServerPipeline.java:120)
        at org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:939)
        at
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:137)
        at
org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:566)
        at
org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:536)
        at org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:939)
        at org.apache.coyote.tomcat5.CoyoteAdapter.service(CoyoteAdapter.java:231)
        at
com.sun.enterprise.web.connector.grizzly.ProcessorTask.invokeAdapter(ProcessorTask.java:667)
        at
com.sun.enterprise.web.connector.grizzly.ProcessorTask.processNonBlocked(ProcessorTask.java:574)
        at
com.sun.enterprise.web.connector.grizzly.ProcessorTask.process(ProcessorTask.java:844)
        at
com.sun.enterprise.web.connector.grizzly.ReadTask.executeProcessorTask(ReadTask.java:287)
        at
com.sun.enterprise.web.connector.grizzly.ReadTask.doTask(ReadTask.java:212)
        at com.sun.enterprise.web.connector.grizzly.TaskBase.run(TaskBase.java:252)
        at
com.sun.enterprise.web.connector.grizzly.WorkerThread.run(WorkerThread.java:75)
|#]
CORE5022: All ejb(s) of [stateless-simple] were unloaded successfully!
Comment 8 Vince Kraemer 2006-07-19 17:01:53 UTC 
*** Issue 77118 has been marked as a duplicate of this issue. ***
Comment 9 Vince Kraemer 2006-08-04 01:26:56 UTC 
Have not been able to reproduce the issue

using 20060803 source with GlassFish v1 U1 Build 5.

did multiple deploys all successful according to NetBeans output window,
server's log and tested access to application

Provide more details and reopen if still an issue.
Comment 10 judytang 2006-08-22 02:58:56 UTC 
In NB 8/21 build, when I deploy the 5th time, IDE hangs.  Following is the
thread dump.


"Inactive RequestProcessor thread [Was:TimedWeakReference/org.netbeans.modules.p
rojectapi.TimedWeakReference]" daemon prio=2 tid=0x1e1f6898 nid=0xb6c in Object.
wait() [0x1581f000..0x1581fb68]
        at java.lang.Object.wait(Native Method)
        at org.openide.util.RequestProcessor$Processor.run(RequestProcessor.java
:885)
        - locked <0x049ff2b8> (a java.lang.Object)

"Thread-11" prio=4 tid=0x1e403b48 nid=0x6d4 waiting on condition [0x15bdf000..0x
15bdf9e8]
        at java.lang.Thread.sleep(Native Method)
        at org.netbeans.core.execution.DefaultSysProcess$1.run(DefaultSysProcess
.java:105)
        at java.lang.Thread.run(Thread.java:595)

"Thread-9" daemon prio=4 tid=0x150335c0 nid=0x680 waiting on condition [0x15b3f0
00..0x15b3fae8]
        at java.lang.Thread.sleep(Native Method)
        at org.netbeans.modules.j2ee.sun.ide.j2ee.LogViewerSupport.run(LogViewer
Support.java:137)

"MDR event dispatcher" daemon prio=2 tid=0x151b98a8 nid=0xe50 in Object.wait() [
0x15b9f000..0x15b9fb68]
        at java.lang.Object.wait(Native Method)
        - waiting on <0x03bd05e8> (a java.util.LinkedList)
        at java.lang.Object.wait(Object.java:474)
        at org.netbeans.mdr.util.EventNotifier$EventsDelivery.run(EventNotifier.
java:263)
        - locked <0x03bd05e8> (a java.util.LinkedList)
        at java.lang.Thread.run(Thread.java:595)

"DestroyJavaVM" prio=5 tid=0x00038a38 nid=0xc48 waiting on condition [0x00000000
..0x0007fae8]

"AWT-EventQueue-1" prio=7 tid=0x15084e68 nid=0x9c8 in Object.wait() [0x1564f000.
.0x1564fd68]
        at java.lang.Object.wait(Native Method)
        at java.lang.Object.wait(Object.java:474)
        at java.awt.EventQueue.getNextEvent(EventQueue.java:345)
        - locked <0x03a37920> (a java.awt.EventQueue)
        at java.awt.EventDispatchThread.pumpOneEventForHierarchy(EventDispatchTh
read.java:189)
        at java.awt.EventDispatchThread.pumpEventsForHierarchy(EventDispatchThre
ad.java:163)
        at java.awt.EventDispatchThread.pumpEvents(EventDispatchThread.java:157)

        at java.awt.EventDispatchThread.pumpEvents(EventDispatchThread.java:149)

        at java.awt.EventDispatchThread.run(EventDispatchThread.java:110)

"TimerQueue" daemon prio=5 tid=0x150d0948 nid=0xe80 in Object.wait() [0x1569f000
..0x1569fbe8]
        at java.lang.Object.wait(Native Method)
        at javax.swing.TimerQueue.run(TimerQueue.java:233)
        - locked <0x03630858> (a javax.swing.TimerQueue)
        at java.lang.Thread.run(Thread.java:595)

"Java2D Disposer" daemon prio=10 tid=0x14fe4860 nid=0x9b0 in Object.wait() [0x15
60f000..0x1560fce8]
        at java.lang.Object.wait(Native Method)
        at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:116)
        - locked <0x0352cfb8> (a java.lang.ref.ReferenceQueue$Lock)
        at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:132)
        at sun.java2d.Disposer.run(Disposer.java:107)
        at java.lang.Thread.run(Thread.java:595)

"AWT-Windows" daemon prio=7 tid=0x14fda980 nid=0x1dc runnable [0x155af000..0x155
afd68]
        at sun.awt.windows.WToolkit.eventLoop(Native Method)
        at sun.awt.windows.WToolkit.run(WToolkit.java:269)
        at java.lang.Thread.run(Thread.java:595)

"AWT-Shutdown" prio=5 tid=0x1506cc08 nid=0xa18 in Object.wait() [0x1556f000..0x1
556f9e8]
        at java.lang.Object.wait(Native Method)
        - waiting on <0x0352d0a0> (a java.lang.Object)
        at java.lang.Object.wait(Object.java:474)
        at sun.awt.AWTAutoShutdown.run(AWTAutoShutdown.java:259)
        - locked <0x0352d0a0> (a java.lang.Object)
        at java.lang.Thread.run(Thread.java:595)

"Timer-0" daemon prio=5 tid=0x14f94158 nid=0xdec in Object.wait() [0x154bf000..0
x154bfa68]
        at java.lang.Object.wait(Native Method)
        - waiting on <0x034b1fd8> (a java.util.TaskQueue)
        at java.util.TimerThread.mainLoop(Timer.java:509)
        - locked <0x034b1fd8> (a java.util.TaskQueue)
        at java.util.TimerThread.run(Timer.java:462)

"CLI Requests Server" daemon prio=5 tid=0x14fd2bb0 nid=0x914 runnable [0x1543f00
0..0x1543fb68]
        at java.net.PlainSocketImpl.socketAccept(Native Method)
        at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)
        - locked <0x034b2110> (a java.net.SocksSocketImpl)
        at java.net.ServerSocket.implAccept(ServerSocket.java:450)
        at java.net.ServerSocket.accept(ServerSocket.java:421)
        at org.netbeans.CLIHandler$Server.run(CLIHandler.java:867)

"Active Reference Queue Daemon" daemon prio=2 tid=0x14d60bc8 nid=0xbc0 in Object
.wait() [0x14f6f000..0x14f6fbe8]
        at java.lang.Object.wait(Native Method)
        at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:116)
        - locked <0x034b2290> (a java.lang.ref.ReferenceQueue$Lock)
        at org.openide.util.Utilities$ActiveQueue.run(Utilities.java:3002)
        at java.lang.Thread.run(Thread.java:595)

"Low Memory Detector" daemon prio=5 tid=0x00a945f8 nid=0x248 runnable [0x0000000
0..0x00000000]

"CompilerThread0" daemon prio=10 tid=0x00036518 nid=0x9b8 waiting on condition [
0x00000000..0x14cafa4c]

"Signal Dispatcher" daemon prio=10 tid=0x00a926e0 nid=0x744 waiting on condition
 [0x00000000..0x00000000]

"Finalizer" daemon prio=9 tid=0x00a899e0 nid=0xd00 in Object.wait() [0x14c2f000.
.0x14c2fa68]
        at java.lang.Object.wait(Native Method)
        at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:116)
        - locked <0x034b2438> (a java.lang.ref.ReferenceQueue$Lock)
        at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:132)
        at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)

"Reference Handler" daemon prio=10 tid=0x00a88550 nid=0x78 in Object.wait() [0x1
4bef000..0x14befae8]
        at java.lang.Object.wait(Native Method)
        at java.lang.Object.wait(Object.java:474)
        at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116)
        - locked <0x034b1f50> (a java.lang.ref.Reference$Lock)

"VM Thread" prio=10 tid=0x00a85b88 nid=0xbf4 runnable

"VM Periodic Task Thread" prio=10 tid=0x00a925c8 nid=0x554 waiting on condition
Comment 11 Vince Kraemer 2006-08-22 07:15:29 UTC 
I don't see anything in the thread dump that seems unusual.  Was the server log
similar to the 13 June 2006 server log?
Comment 12 judytang 2006-08-22 18:04:03 UTC 
Hi Vince,

I am using win XP.  I think this is the same kind problem like the one you saw
in my office last time.  It does not happen all the time, but it does happen
some time.  May be you can drop by my office again so we can work on this
together and also a few other bugs that I can show you how I run into them.  Thanks.
Comment 13 judytang 2006-08-31 03:57:04 UTC 
In 8/28 NB build, I tried to deploy a stateless app which is not built from the
archive project, after 3 successful redeployment, the 4th time and 5th time also
failed with the following stack trace in the appserver server.log. Which
indicate this redeployment failure may not be specific to the archived project.


WEB0100: Loading web module [ServletStateless:ServletStateless-war.war] in
virtual server [server] at [/ServletStateless-war]
StandardWrapperValve[RemoteJmxConnectorServlet]: Servlet.service() for servlet
RemoteJmxConnectorServlet threw exception
java.io.StreamCorruptedException
        at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1331)
        at java.io.ObjectInputStream.readArray(ObjectInputStream.java:1633)
        at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1298)
        at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1917)
        at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1841)
        at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1718)
        at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1304)
        at java.io.ObjectInputStream.readObject(ObjectInputStream.java:349)
        at
com.sun.enterprise.admin.jmx.remote.server.servlet.RemoteJmxConnectorServlet.readRequestMessage(RemoteJmxConnectorServlet.java:192)
        at
com.sun.enterprise.admin.jmx.remote.server.servlet.RemoteJmxConnectorServlet.processRequest(RemoteJmxConnectorServlet.java:108)
        at
com.sun.enterprise.admin.jmx.remote.server.servlet.RemoteJmxConnectorServlet.doPost(RemoteJmxConnectorServlet.java:180)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:727)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
        at
org.apache.catalina.core.ApplicationFilterChain.servletService(ApplicationFilterChain.java:397)
        at
org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:278)
        at
org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:566)
        at
org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:536)
        at
org.apache.catalina.core.StandardContextValve.invokeInternal(StandardContextValve.java:240)
        at
org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:179)
        at
org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:566)
        at com.sun.enterprise.web.WebPipeline.invoke(WebPipeline.java:73)
        at
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:182)
        at
org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:566)
        at
com.sun.enterprise.web.VirtualServerPipeline.invoke(VirtualServerPipeline.java:120)
        at org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:939)
        at
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:137)
        at
org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:566)
        at
org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:536)
        at org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:939)
        at org.apache.coyote.tomcat5.CoyoteAdapter.service(CoyoteAdapter.java:231)
        at
com.sun.enterprise.web.connector.grizzly.ProcessorTask.invokeAdapter(ProcessorTask.java:667)
        at
com.sun.enterprise.web.connector.grizzly.ProcessorTask.processNonBlocked(ProcessorTask.java:574)
        at
com.sun.enterprise.web.connector.grizzly.ProcessorTask.process(ProcessorTask.java:844)
        at
com.sun.enterprise.web.connector.grizzly.ReadTask.executeProcessorTask(ReadTask.java:287)
        at
com.sun.enterprise.web.connector.grizzly.ReadTask.doTask(ReadTask.java:212)
        at com.sun.enterprise.web.connector.grizzly.TaskBase.run(TaskBase.java:252)
        at
com.sun.enterprise.web.connector.grizzly.WorkerThread.run(WorkerThread.java:75)
Comment 14 Vince Kraemer 2006-09-19 16:25:23 UTC 
I have seen this enough to know how.  Only seem to be able to reproduce on dual
processor machines...
Comment 15 Vince Kraemer 2006-09-22 23:27:22 UTC 
It looks like the server's deployment manager is depending on code that isn't
thread safe, we need to synchronize our code to help eliminate this issue. 
FindBugs had a number of suggested fixes in SunDeploymentManager.  The DF was
also letting the user create multiple instances of our DM object.  We really
just need one per URI.
Comment 16 Vince Kraemer 2006-09-22 23:45:07 UTC 
committed fix to release55_dev
Comment 18 judytang 2006-09-25 19:20:07 UTC 
Hi Vince,  I just installed 9/25 release55_dev build, when I redeploy the
stateless ejb app for the 6th time, the deployment failed with the following 
fuild failed message in the job output, but the server.log has no error at this
point of time.  This is one of the errors we saw before.  You can log into my
machine to take a look at it. Thanks !

null
See the server log for details.
BUILD FAILED (total time: 0 seconds)
Comment 19 Vince Kraemer 2006-09-27 16:30:11 UTC 
judy's note indicates that this is not resolved yet.
Comment 20 judytang 2006-09-27 20:13:47 UTC 
Thanks Vince, Ludo, Peter for working very hard on this complicate threading 
issue. Based on the risk and time, I am ok if 77122 does not get fixed in 5.5,
since it seems my fast multiple win XP machine is the only machine gets this
problem so far.
Comment 21 Vince Kraemer 2006-09-28 05:55:28 UTC 
opened https://glassfish.dev.java.net/issues/show_bug.cgi?id=1197 to track the
glassfish side of this issue...
Comment 22 Vince Kraemer 2006-09-29 08:12:30 UTC 
Evaluation:

The various thread-dumps and stack traces indicate that communication between
the server and the IDE is across a channel that may not be completely thread
safe. While the channel has been fairly stable on single processor machines,
systems that are multiprocessor or multicore (like the Intel Duo) are starting
to tax our luck.

I had attempted to synchronize the DeploymentManager communication.  That
introduced some deadlocks. And it did not fully resolve the issue, though it did
improve the deployment robustness for the filer. (when it did not deadlock)

Currently, the filer has a deployment failure rate of about 20%.

Redeploying or rerunning usually resolves the deployment failure.

The issue is being investigated by the GlassFish engineering team.

The plugin uses a number of server entry points that communicate across the JMX
remote channel.  Implementing the fix won't be hard, but will introduce a large
number of changes, that would have severe effects. 

The risk would be extremely high and few users have reported this issue.
Comment 23 Vince Kraemer 2006-10-05 01:44:57 UTC 
*** Issue 86529 has been marked as a duplicate of this issue. ***
Comment 24 Vince Kraemer 2006-12-06 20:16:36 UTC 
code changes to address this integrated into 6.0 codebase

http://serverplugins.netbeans.org/servlets/ReadMsg?list=cvs&msgNo=2262
http://serverplugins.netbeans.org/servlets/ReadMsg?list=cvs&msgNo=2267
Comment 25 Vince Kraemer 2006-12-07 17:59:07 UTC 
Created attachment 36563 [details]
e-mail from another tester
Comment 26 Vince Kraemer 2007-01-12 05:36:52 UTC 
Created attachment 37300 [details]
latest patch
Comment 27 Vince Kraemer 2007-01-12 05:38:05 UTC 
my reviewer caught a couple issues with code.

Additional diffs against the trunk are here:
http://serverplugins.netbeans.org/servlets/ReadMsg?list=cvs&msgNo=2307

diff of merged changes against release551 attached.
Comment 28 _ pcw 2007-01-12 17:59:21 UTC 
Looks good Vince.
Comment 29 Vince Kraemer 2007-01-16 07:28:27 UTC 
release551 check in...

http://serverplugins.netbeans.org/servlets/ReadMsg?list=cvs&msgNo=2311,