'which' probes output should be hidden in Rex::Interface::Exec::Base::can_run etc...

[uralm1]

Describe the bug

How to reproduce it

On pretty old slackware 13.37 system (without dmidecode for example):
#rex -T
which: no dmidecode in (/bin:/sbin:/usr/bin:/usr/sbin:/usr/local/bin:/usr/local/sbin:/usr/pkg/bin:/usr/pkg/sbin)
Tasks
testrun Test run

or (without lsb_release)

#rex -T
which: no lsb_release in (/bin:/sbin:/usr/bin:/usr/sbin:/usr/local/bin:/usr/local/sbin:/usr/pkg/bin:/usr/pkg/sbin)
which: no dmidecode in (/bin:/sbin:/usr/bin:/usr/sbin:/usr/local/bin:/usr/local/sbin:/usr/pkg/bin:/usr/pkg/sbin)
which: no lsb_release in (/bin:/sbin:/usr/bin:/usr/sbin:/usr/local/bin:/usr/local/sbin:/usr/pkg/bin:/usr/pkg/sbin)
which: no lsb_release in (/bin:/sbin:/usr/bin:/usr/sbin:/usr/local/bin:/usr/local/sbin:/usr/pkg/bin:/usr/pkg/sbin)
which: no lsb_release in (/bin:/sbin:/usr/bin:/usr/sbin:/usr/local/bin:/usr/local/sbin:/usr/pkg/bin:/usr/pkg/sbin)
Tasks
testrun Test run

Expected behavior

which file probes should hide its output.
Where: Rex::Interface::Exec::Base::can_run, and (probably) in Rex::Interface::Exec::Local or in i_run() etc...

Circumstances

• Rex version: 1.13.4
• Perl version: 5.12.3 oops...but it doesn't matter
• OS running rex: Slackware64 13.37 running in lxc container on Debian 10 :)
• OS managed by rex: local
• How rex was installed: cpan

Debug log

not needed

[uralm1]

Old which version prints 'no cmd in ...' message to stderr, so proposed fix is:
Rex::Interface::Exec::Base.pm

--- Base.pm.orig	2021-12-30 10:06:17.479354544 +0500
+++ Base.pm	2021-12-30 10:06:55.644286280 +0500
@@ -68,7 +68,7 @@
     }
 
     my @output = Rex::Helper::Run::i_run "$check_with_command $command",
-      fail_ok => 1;
+      fail_ok => 1, stderr_to_stdout => 1;
 
     next if ( $? != 0 );
     next if ( grep { /^no $command in/ } @output ); # for solaris

[uralm1]

linked with PR #1497, same problem

[uralm1]

also which command is deprecated now: https://www.mail-archive.com/debian-bugs-dist@lists.debian.org/msg1814961.html

[ferki]

Thanks for your report, @uralm1! 👍

I agree that it's most possibly related to a specific build or version of the which command, that is not returning the expected exit code and/or message.

I couldn't reproduce it so far, though. Could you share the exact version of the which command that produces this behavior? If you have a link to its source code or repository, that would also be interesting. Alternatively I might be able to run a Slackware 13.37 VM or container if you have a link for a good image.

so proposed fix is:

Thanks for the code suggestion, we might end up with this approach once I can reproduce the problem (and ideally also write tests for it).

linked with PR #1497, same problem

They do seem to be the same type of problem. Not sure if exactly the same, though, but it's good to have them cross-linked for future reference.

also which command is deprecated now: https://www.mail-archive.com/debian-bugs-dist@lists.debian.org/msg1814961.html

Yeah, I've read some of the original discussions at the time about deprecating it in Debian. I'm not sure if replacing which portably for all the currently supported shells (ash, bash, csh, ksh, sh, tcsh, zsh) are easier than fixing this corner case now, though. We'll probably need to address this separately in its own issue sooner or later, though.

[uralm1]

I found this spam output issue takes place only when CMDB is used. Moreover, Rexfile should use modules=packages.

I created bash script that build docker image and demonstrates this issue:
https://gist.github.com/uralm1/dd66c8816f89e0db2484f3183f08aa3b

Spam output from PR #1497 (from dmidecode) can be reproduced on any modern linux and it also takes place when CMDB activates hardware inventory.

[uralm1]

About which deprecation: I forget that recommended command -v is bash builtin. Reimplementing which in perl is a variant for future.

[ferki]

I created bash script that build docker image and demonstrates this issue: https://gist.github.com/uralm1/dd66c8816f89e0db2484f3183f08aa3b

Thanks! 👍 It seems like these kind of instructions were missing in the original report, so I've wasted time trying to guess the details unsuccessfully :( If you can compile a more minimal version (like a Dockerfile, or just the Rexfile), please keep up posted!

Now I see the "Describe the bug" info ended up under "How to reproduce it", and the reproduction steps themselves are posted now. I've converted the issue templates into forms, so these kind of crucial information is harder to be mixed up or left out in the future.

OS running rex: Slackware64 13.37 running in lxc container on Debian 10 :)
OS managed by rex: local

Please note that managing Slackware is not supported currently. There were two PR attempts to add support: #1387 and #1485, but they did not make their way into extension modules yet by their authors (none of them seemed complete though, and were sadly abandoned).

Also, Slackware 13.37 went end of life on 2018-07-05. I don't think it's worth putting a lot of effort into trying to support an OS version that is unsupported even by their own upstream. In fact, we actively try to avoid it. I believe it's much more responsible to encourage migrating to a supported OS version.

Now that we invested a few hours in it, I'll try to take one more look later at the reproduction through the docker example to learn more. Let's see if this issue here is something we can solve independently of Slackware support or not, and if we are facing a more wide-spread general issue or not.

I found this spam output issue takes place only when CMDB is used.

Oh, that sounds like crucial new information. Querying the CMDB triggers detailed discovery of the remote environment (which is when dmidecode, lsb_release, and similar would be run to determine system properties, which are in turn exposed as built-in CMDB data). It uses can_run and thus which on non-Windows managed endpoints to detect the presence and path of those tools, so that might explain why it only shows up with CMDB.

All my local attempts to reproduce the bug were also using CMDB, though, so it should be somehow specific to the environment you described, not to CMDB directly.

Moreover, Rexfile should use modules=packages.

I don't understand that part. Could you elaborate more, please?

Spam output from PR #1497 (from dmidecode) can be reproduced on any modern linux and it also takes place when CMDB activates hardware inventory.

As I commented there, I can't reproduce that behavior on Gentoo by always using the latest dmidecode. Let's not mix issues just yet, though. If we prove they are duplicates, we will close one of them later.

That PR was also coming out of the blue without describing all the circumstances. If you know more about how to reproduce that too, please consider opening a full bug report. I would love to learn more about these corner cases in order to find an optimal implementation to solve the root cause itself.

Debug log

not needed

There might be additional info in there, though. I might ask for a debug output if I still can't reproduce it based on your examples later.

About which deprecation: I forget that recommended command -v is bash builtin.

Yes, that would work on managed endpoints with bash, but Rex should support many other shells too (ash, csh, ksh, sh, tcsh, zsh). Rex also uses where instead of which on Windows.

Reimplementing which in perl is a variant for future.

That's certainly an option, though I'd avoid it if possible.

It either needs a Perl-core-only one-liner approach to be runnable via i_run, or requires uploading a perl script as a string into a file on the managed endpoint, and running that with i_run. Even then, perl might be only partially available on some managed systems (e.g. Red Hat family OSes doesn't install all core modules...), or might not be available at all.

Therefore I generally would prefer to rely on platform-specific direct solutions whenever possible.

Let's focus on tracking down the root cause of this issue before deciding on how to solve it :)

[ferki]

Hmm, as noted on the gist, that container image segfaults for me both with podman and docker, so I can't use that example.

I'm trying to download a Slackware 13.37 VM image or ISO, and check with that.

[ferki]

Soo, in a fresh Slackware 13.37 install, I can reproduce it while using the following Rexfile (triggering CMDB query, without actual CMDB files though, but for hardware discovery that should be irrelevant):

use Rex -feature => ["1.4"];
use Rex::CMDB;
set cmdb => { type => "YAML", path => ["cmdb/config.yml"] };
user get cmdb("user");
task tt => sub { say "tt" };

package Mod2;
use Rex -feature => ["1.4"];
task tt2 => sub { say "tt2" };

Then rex -T prints:

# rex -T
which: no lsb_release in (/bin:/sbin:/usr/bin:/usr/sbin:/usr/local/bin:/usr/local/sbin:/usr/pkg/bin:/usr/pkg/sbin)
which: no lsb_release in (/bin:/sbin:/usr/bin:/usr/sbin:/usr/local/bin:/usr/local/sbin:/usr/pkg/bin:/usr/pkg/sbin)
which: no lsb_release in (/bin:/sbin:/usr/bin:/usr/sbin:/usr/local/bin:/usr/local/sbin:/usr/pkg/bin:/usr/pkg/sbin)
which: no lsb_release in (/bin:/sbin:/usr/bin:/usr/sbin:/usr/local/bin:/usr/local/sbin:/usr/pkg/bin:/usr/pkg/sbin)
Tasks
 tt

 Mod2:tt2

Having package Mod2; in the same Rexfile seems irrelevant, I can reproduce without that too.

Now that I understand the issue better, I can also reproduce it locally.

It seems Rex::Interface::Exec::IOReader->io_read() is being unable to drain stderr properly for a local command execution via open3. I have to think more about how to test the behavior better 🤔 I might also need to open a separate bug about that.

Adding stdout_to_stderr => 1 could work, and I believe that would be required even if io_read would work with stderr correctly. Another workaround could be setting Rex::Config->set_no_tty(0) (which is effectively also redirecting stderr to stdout in this case, as well as avoiding the io_read code path).

edit: we could also consider setting set_no_tty(0) for i_run calls 🤔 Not sure about all the implications yet though, we need clear test cases first.

[uralm1]

Moreover, Rexfile should use modules=packages.
I don't understand that part. Could you elaborate more, please?

All are tested with rex -dT.
This Rexfile does NOT print which errors (it calls which in can_run() with errors, but this errors are hidden):

use Rex -feature=>['1.4'];
use Rex::CMDB;
set cmdb => {type=>"YAML", path=>["cmdb/config.yml"]};
user get cmdb("user");
task tt => sub {say "tt"};

But this prints:

use Rex -feature=>['1.4'];
use Rex::CMDB;
set cmdb => {type=>"YAML", path=>["cmdb/config.yml"]};
user get cmdb("user");
task tt => sub {say "tt"};
package Mod2;
use Rex -feature=>["1.4"];
task tt2 => sub {say "tt2"};

This prints too (+Mod2.pm in external package file):

use Rex -feature=>['1.4'];
use Rex::CMDB;
require Mod2;
set cmdb => {type=>"YAML", path=>["cmdb/config.yml"]};
user get cmdb("user");
task tt => sub {say "tt"};

Don't understand why yet.