This Bugzilla instance is a read-only archive of historic NetBeans bug reports. To report a bug in NetBeans please follow the project's instructions for reporting issues.

Bug 62529 - Cannot redeploy webapp due to ThreadDeath
Summary: Cannot redeploy webapp due to ThreadDeath
Status: RESOLVED INVALID
Alias: None
Product: serverplugins
Classification: Unclassified
Component: Tomcat (show other bugs)
Version: 5.x
Hardware: All Windows XP
: P2 blocker (vote)
Assignee: Sherold Dev
URL: http://issues.apache.org/bugzilla/sho...
Keywords: TOMCAT
: 66925 (view as bug list)
Depends on:
Blocks:
 
Reported: 2005-08-18 00:53 UTC by _ gtzabari
Modified: 2005-11-23 13:39 UTC (History)
1 user (show)

See Also:
Issue Type: DEFECT
Exception Reporter:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description _ gtzabari 2005-08-18 00:53:59 UTC
dev build 200508111800
JDK 1.5.0_04

When developing webapps, I get a failure on every second webapp redeploy and
eventually I get "out of PermGen space" and I have to manually kill the Tomcat
process.

Upon further research I've discovered
http://issues.apache.org/bugzilla/show_bug.cgi?id=26372 and discussed it with
one of the Tomcat staff members and they seem to indicate that Netbeans is
incorrectly misinterpreting a "ThreadDeath" exception as a failue to redeploy.
Specifically, here is the output I see when triggering a redeploy using Netbeans:

Incrementally deploying http://localhost:8084/DesktopBeautifierServer
Completed incremental distribution of http://localhost:8084/DesktopBeautifierServer
Incrementally redeploying http://localhost:8084/DesktopBeautifierServer
reload?path=/DesktopBeautifierServer
FAIL - Encountered exception java.lang.ThreadDeath
C:\Documents and Settings\Gili\My
Documents\blueeye\trunk\desktopbeautifier\Server\netbeans4.2\nbproject\build-impl.xml:446:
Deployment failed.

  The tomcat guy says that the ThreadDeath thing is completely normal and does
*not* indicate a failure to redeploy. It simply means that Tomcat killed the
original webapp because it didn't close quickly enough. This does not prevent
the new webapp instance from coming up...

  Does this make sense? Can we really fix Netbeans' handling so it does not fail
under this condition? I've flagged this issue as P2 because it is a serious
productivity concern to me and occurs very frequently.
Comment 1 zikmund 2005-08-18 13:19:51 UTC
From Tomcat documentation it is clear that the answer should be OK or FAIL. FAIL
with ThreadDeath is not fatal error, therefore it should be fixed on Tomcat side
(IMHO) to report something like "OK - but WARNING ..." when user does 'redeploy'
or 'undeploy'. Gili could you please communicate it with Tomcat? Thanks.

Beacause it will take a lot of time to fix it in Tomcat, I'd recommend to do a
workaround in NetBeans for this. Stepan will consider it in next weeks.
Comment 2 _ gtzabari 2005-08-18 16:28:47 UTC
Ok, I filed http://issues.apache.org/bugzilla/show_bug.cgi?id=36250 with Tomcat

On a related note, I just found out about the existance of
http://cargo.codehaus.org/ today. Maybe it would be easier for Netbeans to just
use it to interact with Tomcat (assuming it has all the features we need)?
Comment 3 Sherold Dev 2005-09-16 09:53:09 UTC
F.

A warning message will be displayed if the java.lang.ThreadDeath exception
occur, but the deployment will not fail now.

Gili, could you please verify it, since I can't reproduce the exception.

Checking in Bundle.properties;
/cvs/tomcatint/tomcat5/src/org/netbeans/modules/tomcat5/Bundle.properties,v  <--
 Bundle.properties
new revision: 1.28; previous revision: 1.27
done
Checking in TomcatManagerImpl.java;
/cvs/tomcatint/tomcat5/src/org/netbeans/modules/tomcat5/TomcatManagerImpl.java,v
 <--  TomcatManagerImpl.java
new revision: 1.42; previous revision: 1.41
done
Comment 4 _ gtzabari 2005-10-21 07:06:37 UTC
Guys, now I get:

21-Oct-2005 1:49:51 AM org.apache.catalina.loader.WebappClassLoader loadClass
INFO: Illegal access: this web application instance has been stopped already. 
Could not load org.apache.log4j.spi.ThrowableInformation.  The eventual
following stack trace is caused by an error thrown for debugging purposes as
well as to attempt to terminate the thread which caused the illegal access, and
has no functional impact.

It seems to me this isn't much better than the original problem, as the redeploy
still fails. Can we reopen this issue?
Comment 5 Sherold Dev 2005-10-21 14:45:02 UTC
When does this "INFO: Illegal access" message occur?

What do you think by reopening this issue? Do you suggest to revert the fix?
Comment 6 _ gtzabari 2005-10-21 20:19:56 UTC
The INFO message occurs exactly the same way the original "FAIL: ThreadDeath"
messages occured so I suspect it is the same issue.

I don't propose we revert the fix, but I am saying the fix isn't complete. The
current situation is still much better than it was prior to the fix, but we
still can't seem to redeploy applications properly.

BTW, with the INFO message, once it happens I am forced to restart the Tomcat
server because if I keep on trying to run the webapp it'll keep on failing with
the same message. The result is that I can only redeploy my applications a small
number of times before I am forced to restart the server again.

The reason I haven't picked up on this issue shortly after our fix is that I
haven't touched webapp development for that long. I just recently got back into it.
Comment 7 _ gtzabari 2005-10-23 20:25:32 UTC
Ok, upon further investigation it seems this new issue *is* related to the
original ThreadDeath. If you read
http://issues.apache.org/bugzilla/show_bug.cgi?id=27371 you will notice other
users reporting errors with "Illegal access: this web application instance has
been stopped already".

My guess (and this is only a guess) is that when one gets "FAIL: ThreadDeath"
the redeploy does fails (contrary to what the Tomcat guys say) and the
application remains in a stopped state. Now, because it is in a stopped state,
if you try accessing any files on it, you get "Illegal access: this web
application instance has been stopped already". This would likely explain why I
am seeing bug #66925 (though in truth the stack-trace there does not indicate it
is caused by the webapp being stopped).

Anyway, to recap: we should try modifying the Netbeans behavior such that if
"FAIL: ThreadDeath" occurs, it automatically executes webapp STOP and START
operations (as opposed to a redeploy). This is slower but it'll likely work. I
believe a lot of the mysterious problems I am currently seeing are caused by the
webapp being in a bad state. What do you think? Can I reopen this issue to that
end? :)
Comment 8 _ gtzabari 2005-10-23 20:29:00 UTC
The associated log4j issue is:
http://issues.apache.org/bugzilla/show_bug.cgi?id=36616
Comment 9 _ gtzabari 2005-10-23 21:44:16 UTC
When "INFO: Illegal access: this web application instance has been stopped
already" occurs, if I try Runtime | Tomcat | Webapps | right click on my webapp
| "Start" then I get:

23-Oct-2005 4:41:25 PM org.apache.catalina.core.StandardContext start
INFO: Container
org.apache.catalina.core.ContainerBase.[Catalina].[localhost].[/DesktopBeautifierServer]
has already been started

So, I can't start the webapp nor stop it. Undeploying it and hitting F6 (run)
doesn't help either, I get the same INFO error. The only way I found of fixing
this thus far is to fully stop the entire Tomcat container and start it up
again. This is very slow. I welcome your ideas as I'm pretty much stuck :(
Comment 10 _ gtzabari 2005-10-23 22:27:47 UTC
BTW, as far as I can tell, the original fix for ThreadDeath does not work at
all. For example, a second ago I got:

WARNING: Tomcat Manager application reported a java.lang.ThreadDeath exception
while performing last operation. This does not necessarily mean a real problem.
Please, refer to http://www.netbeans.org/issues/show_bug.cgi?id=62529 for more
information.
FAIL - Encountered exception java.lang.ThreadDeath

FAIL - Encountered exception java.lang.ThreadDeath

but the redeploy still failed so I am formally reopening this issue.

A possible solution comes to mind:

Netbeans should set reloadable=true in web.xml instead of using the manager
interface to redeploy (since the latter is buggy). Let Tomcat do all the work.
The question then is how does Netbeans know when Tomcat has picked up the
changes? We don't want to trigger the browser HTTP request until we know the
changes have been loaded. I believe Eclipse does this (they call it "Class
Hotswapping") so it would make sense to investigate what they do.

I still haven't verified whether the ThreadDeath problem occurs if we STOP/START
instead of redeploy.
Comment 11 Sherold Dev 2005-10-24 09:04:00 UTC
*** Issue 66925 has been marked as a duplicate of this issue. ***
Comment 12 Sherold Dev 2005-10-24 10:37:25 UTC
From my point of view, it looks like the "FAIL - Encountered exception
java.lang.ThreadDeath" message really means an issue, which we can't easily
workaround and thus it should not be overlooked.

I am sorry, but I have to mark this issue as INVALID and revert the previous
fix, since it is not a NetBeans issue as can be seen from the Tomcat bug.

Please, be aware that this is not a NetBeans issue and thus NetBeans are not the
right place to fix it.
Comment 13 _ gtzabari 2005-10-24 10:51:14 UTC
But without a decent workaround it becomes impossible to use develop using
Netbeans and Tomcat. This might not be strictly a Netbeans issue, but consider this:

1) There is no known way for me to fix this problem. The Netbeans people blame
Tomcat, the Tomcat people blame log4j, and the log4j people are not replying to
bug reports.

2) If I was developing this outside Netbeans, I'd set reloadable=true and simply
update the files in the webapp directory (as opposed to use manager/reload as
Netbeans does). I am fairly sure this approach *will* work as that's what I used
to use before using Netbeans for web development. The downside of doing this
without Netbeans is that it is a lot of work to manually copy files over every
time you "Run".

What about the two remaining workarounds I mentioned? Can we please try to
webapp STOP/START instead of manager/redeploy? If that does not work, we could
always have Netbeans copy files into the webapp directory with redeployable=true
when RUN is invoked. The last option is guaranteed to work but does not
guarantee that the browser window will show up-to-date classes.

If one of the workarounds work, we could add an option into the Netbeans server
configuration which allows us to pick manager/reload mode vs one of the other
redeployment modes.
Comment 14 _ gtzabari 2005-10-24 13:48:07 UTC
Ah ha! :) I just noticed something interesting.

http://issues.apache.org/bugzilla/show_bug.cgi?id=26372 indicates that the bug
lies in log4j whereas http://issues.apache.org/bugzilla/show_bug.cgi?id=27371
indicates that the bug lies in Tomcat. The former is a Tomcat BugZilla issue,
the latter is a Log4j Bugzilla issue. The latter is a newer issue.

I sent an email to Yoav and Brett for more information. Hopefully they'll have
more information available on possible workarounds or fixes.
Comment 15 Sherold Dev 2005-10-24 13:48:14 UTC
> 1) There is no known way for me to fix this problem. The Netbeans people blame
> Tomcat, the Tomcat people blame log4j, and the log4j people are not replying 
> to bug reports.

I got your point but you have to admin that NB are innocent in this case.

> 2) If I was developing this outside Netbeans, I'd set reloadable=true and
> simply update the files in the webapp directory (as opposed to use 
> manager/reload as Netbeans does).

Did you tell this to Tomcat guys? I guess, if this worked and deployment via the
Tomcat Manager would not, they should admit that it is a bug and fix it.

> What about the two remaining workarounds I mentioned? Can we please try to
> webapp STOP/START instead of manager/redeploy

The problem is that I am not able to reproduce this issue and even if I was, it
would be hard to verify that the work-around works really correctly on 100 per cent.

> If that does not work, we could always have Netbeans copy files into the 
> webapp directory with redeployable=true when RUN is invoked.

Changing the way how we deploy does not seem to be much systematic to me, if
there is a bug in the Tomcat Manager it should be fixed.