This Bugzilla instance is a read-only archive of historic NetBeans bug reports. To report a bug in NetBeans please follow the project's instructions for reporting issues.
This may be a false alarm, but I've been getting bounces from mail addressed to @netbeans.org addresses since about 3pm CET (2pm GMT). So far they're only coming for mails originating from netbeans.info, which sends a warning if unsuccessfull after just 2hrs. I may not have seen any from mail I sent directly simply bcs Sun's servers have a longer timeout before sending a warning (6hrs I think). Here's a sample bounce : -- Your message has been received by rocko.futurequest.net but has been undeliverable to the following recipients for at least 2 hours. The mail system will continue to attempt to deliver your message to these recipients for a total of 2 days. You do not need to resend your message at this time. Recipient(s): webmaster@netbeans.org --- Below this line is a copy of the original message. I also note no new mail on the nbusers@ list since 3:42pm CET, which seems unusual (but is not impossible). Can Collab pls verify that there are no mail problems right now ? Thanks.
Hmm, well the automated notification of this issue arrived after a few minutes, so seems this is not a general problem. Any idea why mail originating from netbeans.info, sent to netbeans.org addresses, would be getting delayed ?
I know about several people complaining about it too. I for one sent 3 e-mails to netcat@netbeans.org list some ~14 hours ago and they did not reach the list yet. For NetCAT program this is really crucial problem because of impossibility to react to bug waiver requests in time (24 hours). Please investigate what's wrong as soon as possible.
Created attachment 27190 [details] one of returned mails
Created attachment 27191 [details] attaching again as plain text
Obviously is a real problem here, I have 993 bounces of mail from nb.info to nb.org, also several bounces of mails I sent directly to nb.org addresses. Others have reported the same (some of them attached here). I will attach one of my direct bounces.
Created attachment 27195 [details] Bounced msg sent to webmaster@netbeans.org
Created attachment 27196 [details] Warning e-mail received after ~14 hours
Hi Jack I am working on the issue right away and would update you as soon as possible. Regards Karthik-Helpdesk
The engineers are working on the issue and would update you as soon as possible. Regards Karthik-Helpdesk
Hi Jack The engineers have checked and have updated that at the moment the mail queue for netbeans.org is not having a major problem at all. About the "bounced" messages provided - this is not a hard bounce. It is just a notification that mail transfer agent was not able to deliver the message for more than 3 hours. Let me know if the situation looks better. Regards Karthik-Helpdesk
A quick update: I heard reports that the mail gateways were hammered by a Virus/Worm attack which could have triggered a mail delay too.
*** Issue 69231 has been marked as a duplicate of this issue. ***
Karthik - pls inform the engineers that we very strongly disagree with the "not having a major problem" assessment. For close to 48 hours now we have been getting bounce warnings. I don't think ignoring them, or sitting around and waiting for them to resolve themselves is a good idea. This is a P1. Pls keep us posted on : 1) what is causing this 2) what you are doing to fix it 3) when we can expect mail delivery to return to normal
Ani - I missed your update, many thanks for the info. Do we know for sure that is the cause ? Any further new ? Thanks.
Created attachment 27241 [details] Warning could not send message for past 4 hours.eml
Created attachment 27242 [details] Warning could not send message for past 4 hours.eml
Hi Jack I have updated my engineers about this and will keep you posted as soon as possible.I apologise for the inconvenience faced. Regards Karthik-Helpdesk
None of mails in past 24 H reaches netcat mailing list. I recieved no delivery failure ,and i supposed that my mailes reached the list but when i check the mailing list archive from web front-end i found that none of my mails are there ... Maybe the Mail gateway is running some M... Operating System and it is under a worm attack ;-)
Created attachment 27275 [details] netcatWarning could not send message for past 4 hours.eml
Some mails are now hard bouncing, having waited too long in the queue (netbeans.info mail server timeout is 2 days). Email is critical. Still no real update here. What is going on? Please update immediately with : 1) what is causing this 2) what you are doing to fix it 3) when we can expect mail delivery to return to normal
There is now a very large mail queue on our netbeans.info server, mail to netbeans.org addresses that is backed up. The ISP of that machine has contacted us and informed us that queue is already causing delays and problems for other users of that system.
*** Issue 69351 has been marked as a duplicate of this issue. ***
My few attempts to unsubsribe some mailing lists were not processed. If they had been delivered I would recieve confirmation but I did not get any they are rather lost than delayed. My mails to nbdev@netbeans.org were not delivered too. This is really serious problem that already lasts for more than 5 days.
No update here in days - the last official word we have is "there is no problem". This issue is almost week old. Mails have now been *hard* bouncing for a few days. Please update ASAP with : 1) what is causing this 2) what you are doing to fix it 3) when we can expect mail delivery to return to normal
i did not post anymore to netcat list to check the problem is it fixed ? Today there were no weekly report for netcat members , and i come here to check the issue .
A quick update on this issue: So, here is the current scenario. Spamms/virus had saturated most of the available smtp connections to Sun's smtp server, asmx1.sfo machine. We can only determine if the email is spam or not after we establish the smtp connection. These smtp connections get maxed out soon. As a workaround it would be possible to increase the # of smtp connections allowed to asmx1.sfo. However there is a capacity problem with the spam filter (Brightmail) currently in place, which can't handle unlimited number of connections to the filter process. CN operations are trying to minimize the spread of the spam/virus at this point by: 1) Adding known virus subjects to the mail server so that it will reject those mails; 2) Limiting the # of sendmail processes so that we are allowing mail to flow slowly but surely. The Brightmail server as it exists now neither has neither clustering facility nor the Virus scanning ability. The upgrade to the next version of Brightmail promises to reduce the occurrences of such mail delays due to spam/virus attacks. CN is currently reviewing those options with the SUN account team. AFAIK, there is a significant improvement in the mail flow. If that is not what you are seeing, please let us know.
Does this mean our emails are lost, or are they just delayed? Should we resend them in the hopes they will find a connection and get through to the mailing list?
To answer that, I checked with operations the backlog (if any). They told me that for netbeans.org: 1498 remote queue length 23 not-yet-preprocessed queue length BTW, do you see the mails archived in the mailing lists though? It may not make it to the recepient for some other reasons, but it should definitely make it to the mailing list archive (unless it was rejected as Spam/Virus).
I've sent five emails and a couple of subscribe/unsubscribe requests to the mail list in the last two days, none of which show up in the archives. The last to appear in the archives is http://openide.netbeans.org/servlets/ReadMsg?list=dev&msgNo=18204 Subject lines from missing emails since then are What docs did other folks use? How to include native code in library wrapper? Tutorial Idea Re: [openide-dev] documentation Can't post Time of the first missing email is 28/11/2005 6:19PM EST
Hi kitfox, Can you please verify the archive now, as we are able to view the recent mails in the below archive list http://openide.netbeans.org/servlets/BrowseList?listName=dev&by=date&from=2005-11-01&to=2005-11-30&first=1&count=317 Please let us know if you have questions. Thanks, Kavitha-Helpdesk
Hi, Well, I can see more recent emails too. However, none of my emails are making it onto the mailing list since the one I mentioned above. Subscribe and unsubscribe requests are also being lost. I thought that perhaps that only internal email service had been restored. Any ideas why my emails are being lost? Is this my end or your end?
Okay, now I'm having emails rejected. I had the following bounced to my email: This message was created automatically by mail delivery software. A message that you sent could not be delivered to one or more of its recipients. This is a permanent error. The following address(es) failed: dev@openide.netbeans.org SMTP error from remote mailer after end of data: host asmx1.sfo.collab.net [64.125.133.81]: 550 5.7.1 "Test"... Virus signature in Subject rejected ------ This is a copy of the message, including all the headers. ------ Return-path: <mark@kitfox.com> Received: from node-9120.tor.pppoe.execulink.com ([69.63.51.163] helo=[192.168.0.100]) by smtp1.golden.net with esmtp (Exim 4.50) id 1EhIt9-0002j3-4J for dev@openide.netbeans.org; Tue, 29 Nov 2005 22:44:11 -0500 Message-ID: <438D2008.3030409@kitfox.com> Date: Tue, 29 Nov 2005 22:44:08 -0500 From: Mark McKay <mark@kitfox.com> User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.7.12) Gecko/20050915 X-Accept-Language: en-us, en MIME-Version: 1.0 To: dev@openide.netbeans.org Subject: Test Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit Test
Looks like all my other emails got through all at once. Is the problem fixed? Or is the email program batching them all together to deliver once a day?
Ani, thank you for the updates. Make no mistake - mails *are* being irretrievably lost. Mails from me to various nb.org addresses have been (on and off) "hard bouncing", ie sitting in the queue until they expire, since 27/11. Eg : -->%-- The original message was received at Tue, 22 Nov 2005 09:55:03 -0800 (PST) from [129.157.72.190] ----- The following addresses had permanent fatal errors ----- <webmaster@netbeans.org> ----- Transcript of session follows ----- <webmaster@netbeans.org>... Deferred: Connection refused by asmx1.sfo.collab.net. Message could not be delivered for 3 days Message will be deleted from queue Original-Envelope-Id: 0IQD000QSB3P0T@bohemia2-mail1.czech.sun.com Reporting-MTA: dns; nwkea-mail-2.sun.com Arrival-Date: Tue, 22 Nov 2005 09:55:03 -0800 (PST) Final-Recipient: RFC822; webmaster@netbeans.org Action: failed Status: 4.4.7 Last-Attempt-Date: Sat, 26 Nov 2005 18:31:47 -0800 (PST) Subject: Re: WHY Posting FRENCH tutorials From: Jack Catchpoole <Jack.Catchpoole@sun.com> Date: Tue, 22 Nov 2005 18:54:58 +0100 To: "rodneyg11@hotmail.com" <rodneyg11@hotmail.com> CC: webmaster@netbeans.org --%<-- Note : "Message will be deleted from queue". So it is gone, and never reached it's intended recipient. I can't tell if that msg made it to the web archive or not - the web archive search doesn't find either the original msg (which did get through, so definitely should be there), nor my reply (which is the one that hard bounced above). Mark, the msg you include here was bounced as it looks like a virus. The error you include says : SMTP error from remote mailer after end of data: host asmx1.sfo.collab.net [64.125.133.81]: 550 5.7.1 "Test"... Virus signature in Subject rejected This is unrelated to our current problems, it would be rejected under any circumstances. Collab - how is the situation now ? I am still getting a constant stream of "msg rejected, will keep trying" warnings (soft bounces). Is the queue clearing itself, or can you take some action ?
Here are some further updates: The problem was that we had the number of smtp connection to our mail server limited to 100, and most of these connections were used by spam/virus. That is why the nb.org server was not able to make connection to our server to make the delivery. As of now, operations team have increased the # of smtp connections on our mail server (asmx1.sfo) to 200. The spam filter on the server seems to be holding with this setting, but they will keep an eye on things to make sure it doesn't crash. Sending a test message at netbeans.org came through within 5 minutes. The increased number of smtp connections seems to be helping now. So at this point the NB.org users can do a couple of things. They can resend their messages they sent 3 or 5 days ago (depends on what their holding period is on their server-- the server from which they sent the original e-mails -- NOT the nb.org server), or they can wait until their mail server tries again to connect to our mail servers to deliver the message (which might take longer to get to the list/people than resending the message). If they resend they would see duplicates once the messages they sent a few days ago finally makes to the server. The reason for this is, for certain mail servers, once it fails to deliver a message, it takes longer and longer break to try to resend the message, where a new message is in the queue, it tries to send it right away. The situation seems to be under control now; please call/let us know ASAP if the situation is not improving on your end.
Only 100 delay notifications in my inbox this morning (compared to eg ~700 yesterday and thousands the days before), so yes, seems to be improving. Let's keep this issue open until completely resolved.
CN Support will start testing the Brightmail appliances from to-morow. There will be 3 of those and these will considerably improve the current situation and also any such Spam/Virus related issues in the future.
Correction! That was CN operations who will start the work tomorrow. Hopefully they will be in operation by next week.
*** Issue 51173 has been marked as a duplicate of this issue. ***
Bounces and delays seemed almost gone, but as of 6pm CET I am getting a lot of delay notifications again.
Jack, The new Brightmail server/appliances are scheduled to be rolled out for Netbeans.org next week. They are already setup and being deployed for smaller SUN sites. The initial testing has been promising. Also, the introduction of these appliances will diminish the Virus/Spam attacks; with more than 1 MTA available for the mail gateway, you should see significant improvements. In the interim with the current setup the outgoing messages seem to be delivered with minimal delays. The in-bound mails, b'coz of the heavy spams/viruses, are taking longer to process.
*** Issue 69988 has been marked as a duplicate of this issue. ***
netbeans is now placed behind the new anti-spam filter from Brightmail.
Haven't seen or heard of mail delays in a while; I think this is fixed.
This was a temporary issue and is fixed now. Marking this as verified. Anytime this is noticed again can be dealt with in a new issue. Marking as verified
We recently moved out from Collabnet's infrastructure