I'd like to note that this is not restricted to anon login.
I have had the same thing happen frequently using 
:pserver:theatlie...
It seems to be worst in the afternoons.  Quite often I can
get a good cvs update first thing in the morning (Pacific),
but when I come to check in my changes mid-afternoon, it
fails, and I have to wait until the next day to check stuff
in.

This can be quite frustrating.

Rochelle,
I was told by Pete McNab that it is high probably internal network
issue. So that's probably issue for our internal support guys.

For CollabNet folks I want to say that I was not able to reproduce it
on SPARC/Solaris8 and PC/Linux (located in Prague).

This happens to me when I try to use my laptop which is NOT
connected to Sun's network.  The laptop is behind a NAT/Firewall
box, connected to the internet via cable modem.

I filed an internal serviceticket with the Sun IT for this too, but
when I saw a post on the newsgroup today from someone who was not a
Sun employee, I figured this was a more widespread problem and filed
this issue.  By the way, it appears that this works okay on all
machines in SWAN, but not in iPlanet (Solaris, Windows).

I've verified today (from an external site) that I can login as
anoncvs (cvs 1.11.1p1 / Linux RedHat 7.2). But because of other bug of
the server I can't verify whether other operations are OK.

Updating summary, assigning to support.

This looks like an ssh-tunnel related issue, but I need a bit more
verification. Are you folks using ssh tunnels?  If so, what version of
ssh are you running (both the folks who are having success, and the
folks who aren't)?  Also, is everyone who is having a problem behind
some kind of firewall/NAT system?

Meanwhile, I'm following up internally (PCN9021; assigned to
instantiations engineering).

I've asked our IT these questions and will update this issue as soon
as I have the answers.

Taska: For connection from the Sol8&Linux machine I used SOCKS5 server
in Europe.
I did that from official build machine for NetBeans (Sol8/SPARC) and
from my workstation in Prague(PC/Linux).

I'll do a test from my Win98 notebook in bay area using dialup
internet account.

I forgot to say that I was successful from the two machines.

One question for Terry Heatlie:

You wrote you were on cable modem, but weren't connected through VPN
to Sun/iPlanet network? If yes, than it will become Sun/iPlanet
internal network issue and I guess we would close this issue as invalid.

The machine I see this problem from is *NOT* on Sun's network
at all - not even with VPN.  It is effectively directly connected
to the internet (although there is a NAT gateway between the
machine and the cable modem).  I don't see any other connectivity
issues from the laptop, so I'm pretty sure it isn't a firewall
issue.

Terry, thank you for you answer. I don't agree with you, that's not
Sun's network (internal) problem, when I'm able to connect w/o any
problems and Rochelle not. 

Terry, it looks like you work from home. In such case you should have
installed firewall on your computer and might be inproperly set.
Would you please check whether you're able to do:
cvs -d :pserver:anonymous@cvs-mirror.mozilla.org:/cvsroot login
[pasword is anonymous]
cvs -d :pserver:anonymous@cvs-mirror.mozilla.org:/cvsroot co -c

If you'll have the same troubles connecting to mozilla.org then the
problem is not on netbeans.org.

Rochelle, would you please do the same test with mozilla.org and
provide us with results?

Terry, Rochelle I still don't know whether you're using ssh-tunnel to
connect to cvs.netbeans.org.

Taska: I tried that from Win98 notebook when dialup connected to
meer.net in SF and I was successful with login&logout using anoncvs &
rbalada accounts. Also cvs co standard worked for me.

Thanks Rudolf, Terry, et al. for your updates.  These seems to be a
complex issue (some people receiving errors, but not others; people
receiving errors at some times, but not other times), so the more
data, the better.

Mike Williams (mwilliams@agentissoftware.com) has also reported this
problem and is not using a Sun network at all.  I can do the mozilla
cvs login and co -c without any errors.

I could access mozilla.org just fine.
I can access netbeans.org just fine *SOME OF THE TIME*.
But (especially in the afternoons) often times I have the problem.
I am not using ssh-tunnel to connect.

Are there any diagnostics you would like me to perform the
next time the problem happens?  sniff the network, etc?
Most likely I will not be able to access netbeans.org again
this afternoon.  It is working right now, but then it usually
does in the morning (pacific time).

Just FYI, the server is currently emitting "out of temp space"
errors.  Probably unrelated to this issue...

I have not been able to connect even once since this problem started.

It seems like there are multiple problems here.
Rochelle has a consistent failure to connect at all times, and
is on-swan.  Sounds like a Sun-internal problem.

Others of us have intermittent (but somewhat predictable) problems
not on-swan.  I don't see how this can be a Sun-internal problem.

Actually, if you're on SWAN, you need to connect using socks.  I'm on
the iplanet network where the configuration is different (no socks).

Just note that when bay area (PDT) has early morning, Prague (core of
NetBeans developers) has early afternoon (9 hours diff.). One of the
reasons might be an overlay in higher use of cvs.netbeans.org when
Prague business hours ends and PDT business hours starts.

I was wondering about that, Rudolf, but it seems that the people who
are having no problems in the morning, but problems in the afternoon,
are in California, not Prague.

(Administrative Note: internal issue is PCN9201, not PCN9021. 
Adjusting Status Whiteboard)

I just got the failure...

maggie$ cvs update
cvs [update aborted]: recv() from server cvs.netbeans.org: EOF
maggie$ date
Tue Apr 16 11:19:59 PDT 2002

cvs.netbeans.org is pingable - roughly 30 ms per packet.

I have captured a tcpdump file of the traffic between my machine
and cvs.netbeans.org if anyone wants to look at it.

I have verified that I can still access cvs-mirror.mozilla.org
at this time.

Seems to me like the cvs server is out of tmp space:
Another data point:

socks_cvs -d :pserver:ethanrider@cvs.netbeans.org:/cvs update
can't create temporary directory /tmp/cvs-serv12936
No space left on device

Space on /tmp has nothing to do with proxies, SWAN or whatnot.
Pretty critical issue for me right now... hopefully it will
be fixed ASAP.

For /tmp full issues, see PCN22395.  This issue is about the "EOF"s.

Terry you say:
>I have captured a tcpdump file of the traffic between my machine
>and cvs.netbeans.org if anyone wants to look at it.
Our instantiations engineer has actually told me that yes, this would
be very helpful.  Could you attach it to the issue please?  Thanks!

Created attachment 5434 [details]
tcpdump binary capture of attempted cvs update

I just spoke to our IT and here are the answers:
ssh is not used, we use a firewall but no NAT

Our operations group was wondering if the EOF errors were coming from
the same root as the /tmp errors (issue 22395).  Those have been
resolved now.  Who is still seeing the EOF error?  Are you seeing it
on login?  Update? Other commands?  From within Sun? Elsewhere? What
time of day?  Thanks :).

I can still see it (8:30pm Pacific time) from iPlanet network.

Happening here from my non-swan machine also.  8:30 pm pacific.

9:47pm PDT I have no problems from win98 dialup connected notebook.
NetBeans and FFJ build production machines in different locations on
SWAN are also OK.

~11pm PDT notebook w98 dialup connection to meer.net in SF.
IP 209.157.137.96

cvs co standard
...
cvs server: Updating debuggercore/api
cvs [server aborted]: received termination signal
cvs [server aborted]: received broken pipe signal
CVS.EXE [checkout aborted]: end of file from server (consult above
messages if a
ny)

D:\builds\nb_all>echo %CVSROOT%
:pserver:anoncvs@cvs.netbeans.org:/cvs

Thanks for the update, Rudolf.  I've updated our instantiations
engineer, who is working on this actively today.

We're pulling logs from the server to look at this.  Current data is
making us think that this is a problem with CVS, not with any aspect
of networking.

FYI, I'm still seeing the problem here

  $ cvs -t update
  cvs update: 
notice: main loop with 
CVSROOT=:pserver:anoncvs@cvs.netbeans.org:/cvs
  cvs [update 
aborted]: recv() from server cvs.netbeans.org: EOF

Like Rochelle, 
I haven't been able to get an update since the problem began.  I'm using CVS 
1.11 on cygwin.

Wow. Thank you Mike- that is a really interesting error.  I'll show it
to our experts.  So far, we've almost entirely seen only "EOF"s, but
this error has significantly more information.

Mike, Rochelle: you both seem to have been getting errors consistently
for awhile (Mike says "since the problem began").  I have several
questions for each of you (I think you've already answered some, but
not others):
1) Where are you geographically?
2) Are you at a Sun campus, or elsewhere?
3) What is your connection to the internet?
4) What is between you and the internet? (firewall, NAT, etc.)
5) Are you using an SSH tunnel?
6) What login are you using for cvs?
7) What cvs client are you using?
8) Are you only seing this error on update? Or at login? or some other
time?
9) When was the last time you were successfully able to execute these
cvs commands?
10) Are you available tomorrow sometime to do coordinated testing with us?

Thanks,
Taska

Hi Taska,

Like Mike, I have not been successful since the errors started.  I
can't remember exactly when that was, but it's about 3 weeks, I
think.  Here are the answers to your questions:
1) CA
2) Sun campus or working from home
3) iplanet network - direct connection to internet or VPN
4) firewall, no NAT
5) no
6) anoncvs
7) command line on solaris
8) doing -nq update to see what has changed in repository
9) about 3 weeks
10) sure

In answer to Taska's questions ...

I'm located in Melbourne, 
Australia.  Not on a Sun
campus.

We're connected to the net thru 
Telstra.  We have a
simple firewall using Linux iptables, but that 
doesn't
appear to be in the way: I can ping cvs.netbeans.org
and 
telnet to port 2401. Traceroute goes thru
reach.com, verio.net and 
pnap.net to collab.net.

I'm connecting to cvs.netbeans.org using 
pserver,
without any kind of tunnelling (that I know about).
I'm 
using
CVSROOT=":pserver:anoncvs@cvs.netbeans.org:/cvs"

My 
CVS client is Cygwin cvs-1.11, running on WindowsXP.

I get the error 
for all connections: update, login,
everything.

From looking at 
the timestamps my CVS/Entries files, I
believe my last successful 
update was on April 2.

I'm available for the next 6 hours or so to test 
and
provide feedback, if that helps ... not in tomorrow,
though.

Rochelle and Mike,
Thank you for the additional information. We're still 
working on this one.

Rochelle (and Mike if you're available),
Sorry for the late notice but I'd love to test this with 
you and have operations monitor this afternoon if you're 
available. Let me know what time if it's possible. If 
there are any other users who are having the same problem 
and could test at the same time, that would be great.
Thanks,
Kristen

I'll be available for about the next two hours as long as I can do
some other work too =).  Let me know what to try.

Rochelle,
Great! Could you send me your ipaddress before we start so 
I can convey it to operations. Operations mentioned this 
may be tricky to obtain since you're behind the firewall 
but if you can get it that would be helpful. Which 
username are you logging in with? I'll confirm with you 
that ops is ready to go and then we can start. 
To test: We'll just need you to try to reproduce the 
error. Feel free to do other work as well. :)

Operations is observing as Rochelle reproduces the error. 
They have some initial data.
Rochelle is getting the following error:
cvs [update aborted]: recv() from server cvs.netbeans.org: 
EOF
She used this command:
cvs -d :pserver:anoncvs@cvs.netbeans.org:/cvs -nq update -
d -P

Additional information from troubleshooting:

cvs --version:
Concurrent Versions System (CVS) 1.11 (client/server)
With '{d}' (track deleted revisions) loginfo extension 
(russt).

Copyright (c) 1989-2000 Brian Berliner, david d `zoo' 
zuhn, Jeff Polk, and other authors

CVS may be copied only under the terms of the GNU General 
Public License, a copy of which can be found with the CVS 
distribution kit.

Specify the --help option for further information about CVS


OS: Solaris 8

The "With '{d}' (track deleted revisions) loginfo 
extension (russt)."
means that some people at sun patched the binaries for 
something.  We've
been using it the whole time though, and the other people 
in sun (who
don't see this problem) are still using the same version 
as me.

Rochelle

Initially Rochelle was seeing the error when updating an 
existing tree so we asked her to create
a new directory, cd into it, and then run 'cvs -d
:pserver:anoncvs@cvs.netbeans.org:/cvs  co .'

She received the following errors after trying twice:

I tried twice.  The first time I got:
cvs [checkout aborted]: recv() from server 
cvs.netbeans.org: EOF

The second time I got:
cvs.netbeans.org: Connection timed out
cvs [checkout aborted]: end of file from server (consult 
above messages
if any)

This confirmed for us that it was not a problem with the 
repository. Operations is going to discuss this with our 
cvs engineers and would like to do some more testing. 

Thanks,
Kristen

Hi, This problem has been blocking us from downloading sources
for new modules from cvs. Could you give us any temporary
workaround ?

Thank you

Isao Yanagimachi
Compaq Computer Corporation

Dear Isao Yanagimachi,
Thank you for your update.  We're trying to figure out what is going
on, but not everyone seems to be getting the error, and those folks
who are, are not getting it consistently.  Would you mind answering
the first 9 of the 10 questions I list above?  Thanks!
Taska

1) Where are you geographically?
Palm Springs, California, USA.

2) Are you at a Sun campus, or elsewhere?
elsewhere

3) What is your connection to the internet?
cable modem (RoadRunner from Time-Warner)

4) What is between you and the internet? (firewall, NAT, etc.)
linux-based firewall/NAT

5) Are you using an SSH tunnel?
no

6) What login are you using for cvs?
theatlie

7) What cvs client are you using?
linux command line.
maggie$ cvs -version

Concurrent Versions System (CVS) 1.11.1p1 (client/server)
[...]

8) Are you only seing this error on update? Or at login?
or some other time?
Mostly I do updates.  I haven't noticed it doing
any other operation.

9) When was the last time you were successfully 
able to execute these cvs commands?

It worked on Friday morning (I think - anyway late
last week).  It is an intermittent problem, but
a frequent one.

Hi,

Isao and I work together and since I was in the middle of 
adding
myself to the CC list, I'll answer the questions for both 
of us:

1) Northeastern US. Specifically, Nashua, New Hampshire, 
USA.
2) Elsewhere (Compaq Computer Corp.)
3) Direct (T1, T3?). It does take a few hops to actually 
get out of
Compaq
>4) Firewall only
>5) No
>6) anoncvs
>7) CVS V1.10.5 client for Windows running on Win2K
>8) Can't get past login
>9) Looks like between 3-4 weeks. It's not intermittent, 
neither of
us have been able to access the server for a while now.

Let us know if there's anything else you need to know.

Thanks,

Larry

I've been experimenting this problem since April 28th.  Details 
follow, hope them help:

I have tried connecting during the morning, afternoon and at night.

cvs -version output:
     Concurrent Versions System (CVS) 1.11 (client)

command being executed:
     cvs -d :pserver:anoncvs@cvs.netbeans.org:/cvs login
     cvs -d :pserver:jcdiaz@cvs.netbeans.org:/cvs login

output:
     cvs [login aborted]: recv() from server cvs.netbeans.org: EOF

     I have also tried to connect using the IDE internal cvs, same 
message, so I doubt is a problem with the specific client.

Network information
----------------------
My PC is connected to a Linux firewall which is connected to the net 
via a cable modem, of course this guys also have a firewall.

I am willing to cooperate in any way that I can.

Can we have an update on this?  I haven't been able to connect in over
a month.....

*** Issue 21858 has been marked as a duplicate of this issue. ***

Today we have had our CVS specialists working all day with help from
the operations engineers on this problem.  They have been researching
a great variety of possible root causes, but they have not found
anything yet.  We have also still not been able to reproduce this
ourselves.

If you are having troubles reproducing the problem,
then please work with Rochelle as you did before.
She seems to be able to reliably reproduce the
problem.

We have a testing log which Rochelle helped us provide previously, and
unfortunately it shows us nothing out of the ordinary.  Just ordinary
interactions until suddenly the client gives an error.

Our CVS specialists are going to test this further tonight and over
the weekend, trying a variety of different CVS clients from a variety
of locations on the internet.  We've also looked all over the web for
other sightings of these symptoms, and have only found references to
the "recv()" error during certain situations where the server was
misconfigured- and then, the system was consistently giving the error.

I've sent an email to Rochelle requesting her to try to login to cvs
on our staging server, and asked her to report her results here.

This problem is turning out to be difficult to diagnose.

Does Collab need users like Rochelle, Terry, Mike, 
Isao, Larry, Juan Carlos, etc. to experiment with
different clients?  Does Collab want to provide users
with clients that contain debugging info?  Are all the
users experiencing the problem using the same CVS client?
(Looks like most are using 1.11 but at least one person
was using 1.10.5)

This is the kind of problem that must be attacked from
all angles.  If Rochelle's log "showed nothing out of the
ordinary", then what ideas were investigated in the two
weeks since then?  This problem won't go away by ignoring
it.

I must ask the standard debugging qustions of "what changed".
Did Collab install any CVS related changes just prior to this
problem starting to occur?  Did the users change CVS clients?

The "change" was moving to the Solaris-hosted SourceCast, 
and using the SunScreen firewall package.

There does not seem to be any common element among the 
clients, the network they are using, or the platforms the 
clients are on. Obviously, the server and SunScreen are 
the only common elements. The specific network activity 
that we're seeing is that the client sends the 
authentication request and 5 seconds later, the server 
simply closes the connection without a response. The 
client then reports the EOF.

Since we are unable to reproduce the problem ourselves 
(which then means we can fully analyze everything through 
repeated testing), we need to coordinate with people are 
are experiencing the problem. Even better would be if we 
could somehow get onto a box (via ssh) that is seeing the 
problem. Then we can run the client and monitor the server 
at the same time without the need for extensive 
coordination.

One particular item to note: people MUST be running at 
least CVS version 1.10.7. Prior versions had a bug that 
manifested itself as the EOF problem.

Taska, when I try to cvs login to the staging server I get the same 
error: cvs [login aborted]: recv() from server <server name>: EOF

Thank you, Rochelle and Greg, for the updates.  Being able to test
this on staging will help greatly.

We are currently using SunScreen on our servers, based on contract
requirements with Sun.  Now that we've replicated this problem on the
staging server, we're going to investigate whether the problem is
SunScreen related.

Our Operations/AppsDev folks are still investigating.

Rochelle:
Our operations folks would like for you to send us some "snoop
traces".  They have given me some instructions to pass on:

1. As root:
snoop -o/path/to/snoopfile.out host netbeans.org
2. As the normal user, do the offending CVS command.
3. As root, ctrl-c or kill the snoop process
4. Send us the resulting snoop output (gzip it if it's large, but it
shouldn't
be)

And they say:
>I'd love to see what the client sees when they try to connect,
>specifically if they get a RST, which we don't seem to be sending

Thanks.

Rochelle:

Also, could you please re-try the staging server? We have 
done some reconfiguration on it to see whether that will 
remedy the problem. Snoop output for the staging server 
would also be useful.

From an earlier message, it appears you're using CVS 
version 1.11. Could you please confirm that? (cvs --
version will show the version number of the client).

Thanks!

Thanks for the update, Greg.  Just to make sure everything is clear,
Rochelle:
Could you send us:
1) An update on which CVS client you're using;
2) Snoop traces from an attempt with live netbeans.org;
3) Snoop traces, and description of any change in symptoms, from an
attempt with the staging server.
Thanks!

Created attachment 5746 [details]
Snoop output

You asked:
>I'd love to see what the client sees when they try to connect,
>specifically if they get a RST, which we don't seem to be sending

What is RST? I can't answer the question if I don't know what it is.

Rochelle- I don't know either, but the ops folks will definitely know
it when they see it in the snoop output.  Thanks for the output; I'm
going to send it on to the operations group now.

An "RST" is a low-level TCP/IP concept. If we saw one in 
the snoop output, it would be quite relevant. However, we 
see a normal TCP/IP pattern in the snoop output you 
posted. It is also similar to what Terry posted: the 
client sends a request, and the server simply closes the 
connection 5 seconds later.

Was your snoop output for the main server, or the staging 
server?

Thanks!

With the staging server and no snoop, I now get this message:
cvs login (Logging in to <server name>)
CVS password: 
cvs [login aborted]: connect to <server name>:2401 failed: Connection
refused

My cvs version is 1.11.

I'll try the staging server with snoop now and attach the result.

(oh, I answered my own question: the previous snoop output 
was against the main server; I can see that in there)

Rochelle: the connection refused was because the staging 
server was down. OPS is bringing it back up. Could you 
please try again in a few minutes?

Thanks for your assistance! We hope to get to the bottom 
of this, and your help is appreciated.

Created attachment 5747 [details]
staging output from snoop

The first snoop output was from the production server, the second from
the staging server.  Is it possible I have the wrong anoncvs
password?  I now get connection refused - I thought it was blank, but
please verify.

Okay, now I get the regular error from the staging server =):
cvs login
(Logging in to anoncvs@<servername>)
CVS password: 
cvs [login aborted]: recv() from server <servername>: EOF

I'll try again with snoop.

Created attachment 5748 [details]
snoop output from staging server - take 2

Rochelle- Hmm.  We're terribly disappointed that our configuration
changes had no effect :/.  Here's a slightly different kind of
request:  Could you try logging in to:

anoncvs@anoncvs.openoffice.org

And tell us what you get?  Thanks!

(No password, same as anoncvs@cvs.netbeans.org.)

FYI, "cvs login" to 

    
:pserver:anoncvs@anoncvs.openoffice.org:/cvs

works for me, but 
login to 

    :pserver:anoncvs@cvs.netbeans.org:/cvs

is still 
broken.

I can login to anoncvs@anoncvs.openoffice.org without a problem =).

Mike: that should be broken, there is no anoncvs@cvs.openoffice.org.
Rochelle: thanks for the update.  At least we know it's not all
collabnet servers for you :)

Taska: Mike said cvs.netbeans.org, not cvs.openoffice. :-)

Rochelle: we've gone one more test for you. Could you 
please do the test against the staging server one more 
time? We've got some analysis stuff running there now, 
which should capture a ton of detail for when you try to 
log in.

There is no need for you to run snoop. Just a simple 'cvs 
login' against the staging server.

Thanks!

Done - I hope it helped.

Hmm. We aren't seeing anything. Did you log into 
solstage1.sfo.collab.net ? Please make sure your CVSROOT 
variable points to the right place, and try the 'cvs 
login' again.

Thanks much!

I just did it again.  What I did both times was:
cvs -d <cvsroot for staging server> login

Thanks for the help Rochelle. Unfortunately, your attempts 
did not "reach" the point where we were trying to analyze 
what was happening. We've moved the analysis code to a 
different point, and would like you to try logging in one 
more time.

(just once please, it generates a *huge* log of data)

Again, on the staging server, please.

Thanks!

Done - I hope you got what you needed =).

Thanks, Rochelle, for your ongoing help.  Rochelle has helped us test
this a number of times today.  We are poring over a variety of logs now.

We believe we've isolated the problem to the "tcp 
wrappers" installed on the server. Rochelle's IP address 
reverses properly, but the resulting hostname does not 
resolve back to that IP address. The wrappers are set in 
a "paranoid" mode and, thus, disallow her operation.

We are now rebuilding the wrappers to verify our 
assumption, and will run another test.

Okay, we're done rebuilding the wrappers.  Rochelle, could you try
again?  Thanks!!

It worked!

Hoooorah!!!  I have let our engineers know.

Thanks for the assistance, Rochelle. OPS will update the 
production box asap, and we'll get this issue closed after 
that occurs.

We've updated the production.  Rochelle (et al), could you verify that
it's working on the production server now?  Thanks!!

Alright, I can login again! I successfully logged-in
from within NetBeans using the CVS Client on both a
Windows and an OpenVMS system. I could also login from the
Windows command line using the CVS V1.10.5 client.

Looks good so far. My thanks to everyone that worked on
this problem.

Larry

It's working - thanks!

I'm extremely pleased to see that this problem has been fixed.

Thanks to Greg, Taska, Rochelle and everyone else involved in
tracking down, and fixing, this problem.

Is it safe to assume that the "TCP wrappers" changed in 
early April and that caused the problem to appear?

Yes, tcpwrappers change was the root cause. 

I have problems using anonymous cvs from various 
providers using various network configurations.

allthough I can ping cvs.netbeans.org I can't login - 
I get a timeout exception

Jakob, it works for me ... so perhaps what you're seeing is a different 
problem?

accepting.

I am also able to reach anoncvs@cvs.netbeans.org without a problem.
Can you provide some details of how you are trying to connect?

Aren't you connecting from ECN enabled linux machine?
check /proc/sys/net/ipv4/tcp_ecn
if it is enabled (the file contains "1"), try to disable it:
echo 0 > /proc/sys/net/ipv4/tcp_ecn

The firewall around cvs don't like ECN

Jakob,

Are you still having a problem accessing cvs? 

This seems to be affecting only one user who has not responded
further. A solution for this user has been suggested. Closing.

closing..

We recently moved out from Collabnet's infrastructure