Bug 141461

Could you try to run with turned logging on:

  http://wiki.netbeans.org/FaqRubyNBLogging

We will see what command is exactly run and might try it from the command line and compare to narrow the problem. Thanks.

PS: again, best with latest build, due to today's overhaul:

  http://wiki.netbeans.org/RubyInstallation#section-RubyInstallation-HowDoIGetTheContinuousBuilds

Created attachment 65649 [details]
Log for built-in jruby gem fetching with debugging turned on

Created attachment 65650 [details]
Log for built-in jruby and native Ruby gem fetching with debugging turned on

I've attached the logs for both checking the JRuby Gems and the Native Gems.

I noticed that when I checked the native gems, the system appeared to check the gems twice.  Therefore you should see
the gem list command in the log twice.

Additionally, when I brought the Gem Panel up, the JRuby platform was selected.  I immediately switched the platform,
but the jruby search continued to completion before searching for the native gems.

From the attached log, the culprit is that the process:

  /usr/local/bin/ruby -I/usr/local/lib /usr/local/bin/gem list --both --details --all

fails. Thus NetBeans gem update fails as well. Could you check whether the above command line works for you by running
it and subsequently after it:

  echo $?

should shows 0.

If it works for you from CLI, please double check whether it works for you from NetBeans at the *same* time as well.

NetBeans just delegates to the gem tool and checks its exit code. It seems that gem tool cannot contact
http://gems.rubyforge.org/yaml at that time.

I see you've filed issue 141696 for the redundant gem fetching problem, so let's discuss it there.

http://www.netbeans.org/issues/show_bug.cgi?id=141696 is actually a different issue dealing with thread/subprocess
termination within the Gem dialog.

As for this issue, I've attached the log from executing the command under the native ruby environment.

As you indicated, the $? result is 0.

Created attachment 65826 [details]
Native Ruby gem list execution log.

And still has the same problem under the NetBeans I suppose, which runs exactly the same process. That's strange. Could
you check the proxy setting, whether it is not wrong?

And attach result of:

  /usr/local/bin/gem env

Starting with 200807280848, within the IDE I'm getting different results whether I run using the "built-in" jruby or the
native ruby environments.

Using the Built-in jruby, everything appears to be working correctly; however, under the native ruby, I don't get any
results.

I'm attaching a screenshot (2560x1024) of my desktop.  NetBeans is on the right.

Created attachment 65840 [details]
Animated Gif showing Difference between jruby and ruby on RubyGems dialog

Yes, that's odd, simply does not work for you -> P1.

So, it's little different now - you do not get any dialog. Could you attach the logging again. If I can't read anything
from it, I'll attach more logging. I've added some more logging few hours ago, but you likely just get the new build
from Hudson.

http://wiki.netbeans.org/RubyInstallation#section-RubyInstallation-HowDoIGetTheContinuousBuilds

Created attachment 65842 [details]
Latest log showing execution of ruby gem fetching.

Adding a log from 200807281401 showing both gem fetches (built-in jruby and native ruby)

Created attachment 65847 [details]
Full log from startup to gem fetch not displaying results.

I've send you binary patch, let's continue in private email for awhile if you don't mind (for others: I'll be just
adding more and more logging info into binary patch, until it reveals the culprit). Let's update status here then.

From the log you've sent me it seems that you are using proxy on command line, but not in NetBeans so you end up with
unsuccessful update:

  ERROR:  http://gems.rubyforge.org/ does not appear to be a repository
  ERROR:  While executing gem ... (Gem::RemoteFetcher::FetchError)
        SocketError: getaddrinfo: Name or service not known (http://gems.rubyforge.org/yaml)

Also if you grab newer build you will see, in case of problems like your one, whole output of gem tool, which should be
more helpful, then all-in-one-line.
Please check your proxy setting e.g. in Menu -> Tools -> Options, General category and let me know whether it has fixed
the problem.

I don't think it's in the proxy settings unless the IDE is/isn't doing something.  Here's why...

I've been executing the process from networks that both have and do not have a proxy setup.  The proxies are the first
thing I've checked (and rechecked, and rechecked, ad nauseum...).

Regardless of the Proxy settings, it still isn't working for me.  Even in the latest nightly builds.

Currently I'm sitting on my home network.  On this network, the environment settings for proxies are unset: (i.e.)

proxy_host=
proxy_port=
http_proxy=
https_proxy=
ftp_proxy=
rsync_proxy=

According to my options page, my proxy settings are set to "Use System Proxy Settings"

The results displayed in the Ruby Gems dialog for the built-in jruby gems show 15 gems installed and 3360 new gems
available.

The results displayed in the Ruby Gems dialog for the native ruby gems show 0 gems installed and 0 new gems available.

When I execute the command for the native ruby gems on the command line, I see the following list

$ /usr/local/bin/ruby /usr/local/bin/gem list

*** LOCAL GEMS ***

actionmailer (2.1.0, 2.0.2)
actionpack (2.1.0, 2.0.2)
activerecord (2.1.0, 2.0.2)
activeresource (2.1.0, 2.0.2)
activesupport (2.1.0, 2.0.2)
capistrano (2.4.3)
cgi_multipart_eof_fix (2.5.0)
daemons (1.0.10)
facets (2.4.1)
fastthread (1.0.1)
gem_plugin (0.2.3)
goldberg_generator (0.2.2)
haml (2.0.1)
highline (1.4.0)
hobo (0.7.5)
hobofields (0.7.5)
hobosupport (0.7.5)
hpricot (0.6)
htmlentities (4.0.0)
httpclient (2.1.2)
linecache (0.43)
mocha (0.9.0)
mongrel (1.1.5)
mongrel_cluster (1.0.5)
mysql (2.7)
net-scp (1.0.1)
net-sftp (2.0.1)
net-ssh (2.0.3)
net-ssh-gateway (1.0.0)
piston (1.4.0)
railroad (0.5.0)
rails (2.1.0, 2.0.2)
railscheck (0.2.0)
rake (0.8.1)
RedCloth (3.0.4)
ruby-debug-base (0.10.1)
ruby-debug-ide (0.2.0)
soap4r (1.5.8)
sources (0.0.1)
sqlite3-ruby (1.2.2, 1.2.1)
test-spec (0.9.0)
w3c_validators (0.9.3)
warbler (0.9.10)
will_paginate (2.2.2)

so I know the command is able to execute successfully.

I then went back to the Ruby Gems dialog and changed the proxy settings to "No Proxy".  Saved and re-executed the Fetch
for native gems 2 times.  Both times resulted in 0 gems installed and 0 new gems available.

In your earlier posts you indicate that the processing of the commands should be the same; however, based on the
behavior difference between the two commands, I would assume that the capture command which grabs the output from the
commands and brings it into the parser is not working properly.

I don't know how the jruby process is handling the output (since it is using Java I can only assume it is processing on
the same streams as System.out), whereas the native gem command may be using System.err or another stream which isn't
being captured properly.

> When I execute the command for the native ruby gems on the command line, I
> see the following list
> 
> $ /usr/local/bin/ruby /usr/local/bin/gem list
> 
> *** LOCAL GEMS ***
> 
> actionmailer (2.1.0, 2.0.2)
> actionpack (2.1.0, 2.0.2)
> activerecord (2.1.0, 2.0.2)
[...]
> so I know the command is able to execute successfully.

Local gems works from IDE as well (IDE just rejects to shows local gems until remote gems are fetched as well (which
might be considered another bug)).

I suppose that 'gem list --both' shows you '*** REMOTE GEMS ***' as well.

> I then went back to the Ruby Gems dialog and changed the proxy settings to
> "No Proxy".  Saved and re-executed the Fetch for native gems 2 times.  Both
> times resulted in 0 gems installed and 0 new gems available.

When you click 'Reload' it stil does nothing, even does not show the error
dialog? Just silently does nothing?

> In your earlier posts you indicate that the processing of the commands should
> be the same; however, based on the behavior difference between the two
> commands, I would assume that the capture command which grabs the output from
> the commands and brings it into the parser is not working properly.

That's not the case. Since in the log all output was seen by IDE. The *only* problem was that the gem command end up
with exit code 1 due to:

  ERROR:  http://gems.rubyforge.org/ does not appear to be a repository
  ERROR:  While executing gem ... (Gem::RemoteFetcher::FetchError)
        SocketError: getaddrinfo: Name or service not known (http://gems.rubyforge.org/yaml)

in the log you attached/sent me.

> I don't know how the jruby process is handling the output (since it is using
> Java I can only assume it is processing on the same streams as System.out),
> whereas the native gem command may be using System.err or another stream
> which isn't being captured properly.

That should not be the case. stderr is redirected to stdout and parsed at once.

Could you do what you have described above again (JRuby and Ruby gems fetching) with the latest build and attach the log
again. I've added there logging. So far you sent me logs from 'older' builds where new logging were not presented.

Thanks.

I just ran using the nightly build from 200807300201 from behind my firewall.  As before, the built-in gems listed
properly.  As before, the native gems are not listed.

Running the gem command from the log on the command line generates a proper list of all gems and descriptions.

The log is attached.

Created attachment 66084 [details]
Log File from run to obtain gem list from native gems

Could you attach output of:

  /usr/local/bin/gem env

run from command line. I'll then add more logging, since everything seems ok now, without error, with exit code 0.

I've added very detailed logging, which is in the builds #3146 and later. Log as usually is needed + 'gem env' output
would be handy as well. Thanks.

Integrated into 'main-golden', available in build *200807311401* on http://bits.netbeans.org/dev/nightly/
Changeset: http://hg.netbeans.org/main/rev/3f295f6ca35e
User: Martin Krauskopf <mkrauskopf@netbeans.org>
Log: Super fine logging for issue #141461

I've installed Hudson Build #3154 and re-executed with logging set appropriately (the log is attached).

The telling lines are :

FINEST [org.netbeans.modules.ruby.platform.gems.GemManager]: === Output End ===
FINEST [org.netbeans.modules.ruby.platform.gems.GemManager]: Parsed 0 local gems
FINEST [org.netbeans.modules.ruby.platform.gems.GemManager]: Parsed 0 remote gems
FINEST [org.netbeans.modules.ruby.platform.gems.GemPanel]: Update of GemManager[platform:RubyPlatform[id:Ruby,
label:Ruby 1.8.6-p269, /usr/local/bin/ruby, info: RubyPlatform$Info[GEM_HOME:/usr/local/lib/ruby/gems/1.8, GEM_PATH:
/home/esmith/.gem/ruby/1.8:/usr/local/lib/ruby/gems/1.8, gemVersion: 1.2.0 (1.2.0), lib: /usr/local/lib/ruby/1.8]]] finished

following a full listing of the 3K+ gems that I should be seeing in the list.

Created attachment 66189 [details]
Log from gem fetching operation using native ruby.  Hudson Build 3154

And the output from gem env:

$ gem env
RubyGems Environment:
  - RUBYGEMS VERSION: 1.2.0
  - RUBY VERSION: 1.8.6 (2008-07-07 patchlevel 269) [i686-linux]
  - INSTALLATION DIRECTORY: /usr/local/lib/ruby/gems/1.8
  - RUBY EXECUTABLE: /usr/local/bin/ruby
  - EXECUTABLE DIRECTORY: /usr/local/bin
  - RUBYGEMS PLATFORMS:
    - ruby
    - x86-linux
  - GEM PATHS:
     - /usr/local/lib/ruby/gems/1.8
     - /home/esmith/.gem/ruby/1.8
  - GEM CONFIGURATION:
     - :update_sources => true
     - :verbose => true
     - :benchmark => false
     - :backtrace => false
     - :bulk_threshold => 1000
  - REMOTE SOURCES:
     - http://gems.rubyforge.org/

So now, it's finally clear. Your output of the gem tool run from within NetBeans contains all needed except the GEMS lines:

   *** LOCAL GEMS ***
   [...]

   *** REMOTE GEMS ***
   [...]

which NetBeans uses to detect start of local, respective remote, gems. I've took a look at RubyGems sources and at July
1st, drbrain (Eric Hodel) did the following change:

  $ svn di -r 1834:1835 lib/rubygems/commands/query_command.rb
  Index: lib/rubygems/commands/query_command.rb
  ===================================================================
  --- lib/rubygems/commands/query_command.rb	(revision 1834)
  +++ lib/rubygems/commands/query_command.rb	(revision 1835)
  @@ -70,9 +70,11 @@
      end
  
      if local? then
  -      say
  -      say "*** LOCAL GEMS ***"
  -      say
  +      if ui.outs.tty? then
  +        say
  +        say "*** LOCAL GEMS ***"
  +        say
  +      end
  
        specs = Gem.source_index.search name
  
  @@ -84,9 +86,11 @@
      end
  
      if remote? then
  -      say
  -      say "*** REMOTE GEMS ***"
  -      say
  +      if ui.outs.tty? then
  +        say
  +        say "*** REMOTE GEMS ***"
  +        say
  +      end
  
        all = options[:all]

So you have to run pretty recent version of RubyGems (my 1.2.0 does not contain the change yet). I'll take a look at
possible solutions.

Definitelly thanks for helping with catching this, since it will be discovered by more people soon.

Are you using custom build of RubyGems from trunk?

Very Interesting Indeed!!

Due to the latency of the ruby support in Ubuntu 8.04, I downloaded the latest stable versions of Ruby (1.8.6) and
RubyGems (1.2.0) as source and built/installed them myself.

Apparently, since I checked things out on the 9th, I have version 1840 which has the changes.

My first instinct is to call drbrain up and get him to put the lines back in.

My second instinct is to see if we can make the output system think it's a tty.  Not an easy prospect due to multiple
platforms.

Finally, my step back and punt option would be to run the two commands (gem list -dl and gem list -dr) individually. 
Check their response lengths for 0 and if the response has a length > 0 go ahead and parse it.  

I know it's not as optimal as running one single command, but looking through the source code and the help, at this
point I'm not sure there is another easy option.

> My first instinct is to call drbrain up and get him to put the lines back in.

We had the same idea ;) I've did so immediately yesterday.

  http://permalink.gmane.org/gmane.comp.lang.ruby.gems.devel/2199

> My second instinct is to see if we can make the output system think it's a tty.

Also though about this. NetBeans uses ProcessBuilder#start:

  http://download.java.net/jdk7/docs/api/java/lang/ProcessBuilder.html#start()

I do not know whether there is some way to force behaves such way. I guess not.

> Finally, my step back and punt option would be to run the two commands

Another natural choice. Running process under JRuby, might be under Rubinius as well, is (much) slower then with MRI so
this is last resort.

Another option is to call RubyGems API directly, which might be little bit risky now, since we would have to keep
backward compatibility with RubyGems ... likely 0.9.x, which is unfortunately still official the version bundled with
Ubuntu 8.04, so a lot of users might have it.

I'll wait for Eric's response on rubygems-dev and then come with something.

PS: workaround is simple now, just patch, remove the conditions "if ui.outs.tty? then" which should be harmless.

Setting priority to P2. This is because NB 6.5 Beta must not have any P1 opens, so this would be a blocker. As a P2 it
is still blocker for 6.5 final.
I've installed RubyGems trunk, so I'll fix this either this or next week, otherwise I would not be able to use Gem
Manager myself :)

I have a fix (two runs). I'll commit yesterday since I'll be off-line now and would not be able to handle possibly
broken builds.

Two runs are not that big issue in this case, since the most time-consuming part is remote fetching anyway. I've
refactored the code that it will be easily tweakable wrt. to one-run if RubyGems teams revert LOCAL and REMOTE token
when running with '-b' for RubyGems 1.2.1.

Fixed as described.

Changeset: bd6fc867efe3

Commit message:
Use two runs instead of one for fetching both - remote and locals. See
dicussion in the issue for details. Will be easy to tweak further if
RubyGems team (waiting for their feedback) will revert LOCAL and REMOTE
tokens. 

Integrated into 'main-golden', available in build *200808051401* on http://bits.netbeans.org/dev/nightly/
Changeset: http://hg.netbeans.org/main/rev/bd6fc867efe3
User: Martin Krauskopf <mkrauskopf@netbeans.org>
Log: #141461: Gem Fetching Failed for RubyGems >1.2.0
Use two runs instead of one for fetching both - remote and locals. See
dicussion in the issue for details. Will be easy to tweak further if
RubyGems team (waiting for their feedback) will revert LOCAL and REMOTE
tokens.

.