Discovery Troubleshooting Information

When opening a ticket regarding devices not being discovered, the following information is required:

Windows Devices

From Host Result Command
Target Host IP Information ipconfig/all
Firewall Information Screenshot of windows firewall status
WBEM Test local How to Use wbemtest.exe to validate Windows Credentials – Video
wbemtest.exe doc
Service List Screenshot of all services that are running (for 3rd party firewall check)
Source Host (some other box on the customer’s network) IP Information – This box should be on the same subnet as the RN150 Ifconfig or ipconfig /all
Ping to Target host Ping x.x.x.x (must match one of the IPs on the target host)
WBEM Test Remote How to Use wbemtest.exe to validate Windows Credentials – Video
wbemtest.exe doc

Linux Devices

From Host Result Command
Target Host IP Information Ifconfig
Firewall Information Sudo iptables –L –n
Source Host (some other box on the customer’s network) IP Information – This box should be on the same subnet as the RN150 Ifconfig or ipconfig /all
Ping to Target host Ping x.x.x.x (must match one of the IPs on the target host)
SNMP Walk of All MIBS UCD: snmpwalk -v2c -cstring ipaddress ucdavis
UCD-DisckIO: snmpwalk -v2c -cstring ipaddress ucddiskiomib
Host resource: snmpwalk -v2c -cstring ipaddress host
TCP: snmpwalk -v2c -cstring ipaddress tcp
If-mib: snmpwalk -v2c -cstring ipaddress interfaces

SSH Troubleshooting Introduction

It is highly recommended to test SSH access and operation prior to issuing a Discovery scan, to ensure that the hosts are configured correctly for the SSH Collection Module. This can be performed from the SSH credentials section on the RN150 Virtual Appliance, which will validate the ability of the RN150 Virtual Appliance to connect to the target hosts at the network level, and will test the native operation of the SSH Collection Module. However, it may be desired to perform more manual tests from another host in the environment, allowing the enablement of debugging options in SSH for deeper analysis.

Testing from the RN150 Virtual Applaince is covered above in the credential entry sections. This section will cover the manual tests that can be performed for deeper analysis of the configuration.

To test SSH access to a target host, first log in to another host in the environment that provides an SSH client. This will be referred to as the "local" host, while the host that is being connected to is referred to as the "target". In the following examples, the target is assumed to have an IP address of 10.0.0.2, and the username of the account we are connecting to is "risc".

The general structure of the SSH command, in manpage format, is as follows:

    ssh [flags] [authentication-flags] <username>@<host-IP> "<command>"

A breakdown of this command:

Component Format Description
ssh
literal The ssh client binary
[flags]
optional General flags to control the ssh client behavior
[authentication-flags]
optional Controls explicit authentication behavior
<username>
replace

The username of the account we are connecting with

<host-IP>
replace The IP address of the target host
"<command>"
replace The command to issue on the target host, contained in single or double quotes

Troubleshooting Command Reference

Normally, the ssh client is invoked without the final argument shown above, which performs a login to the target host into an interactive shell session. When invoked with the optional command argument, this causes the ssh client to only invoke that command in a non-interactive fashion. This is an important component of the troubleshooting process, as the use of the command argument will replicate the behavior of the SSH Collection Module. It also provides a test for the behavior described above; an interactive session will allocate a TTY terminal device, while the non-interactive execution of a single command will not. This allows testing the sudo configuration, which will often require a TTY device associated with the user session invoking it.

When performing troubleshooting to determine the cause of an authentication failure experienced by the SSH Collection Module, the ‘-v’ flag can be given to the ssh client to cause it to emit extra verbose debugging output. This flag can be given multiple times to cause the ssh client to emit more information. It is highly recommended to use this flag when performing troubleshooting.

Depending on the type of authentication being utilized, the authentication-flags field may need to be used to cause the client to specifically use the type of authentication desired, as well as the specific authentication credentials to be used.

If using password or keyboard-interactive authentication, this can be specified as, for example:

    -o PreferredAuthentications="password"

If using publickey authentication, this would be:

    -o PreferredAuthentications="publickey"

If using publickey authentication, to specify which key the client utilizes for authentication attempts, the following could be used, for example:

    -i /home/risc/key

To perform a simple connection test using password authentication to the target host to validate the connection, authentication, and ability to issue a simple command:

    ssh -v -o PreferredAuthentications="password" risc@10.0.0.2 "true"

To test the same with publickey authentication:

    ssh -v -o PreferredAuthentications="publickey" -i /home/risc/key risc@10.0.0.2 "true"

Testing the availability and configuration of sudo is of primary importance, as this is often the main reason for a failure to bring a host into inventory. This example assumes we are connecting to a Linux host. Note the use of single quotes to avoid having the $PATH variable expanded prior to issuing the command:

    ssh -v risc@10.0.0.2 'sudo env PATH=${PATH}:/sbin ifconfig -a'

Troubleshooting Command Suite

The following is a series of commands that will validate the enablement of the SSH Collection Module for a given host. When a support case is opened, a support engineer will likely request the results of the follow series of commands. In this case, it is not expected that legitimate customer data that results from a successful command be sent to support, but any error messages produced will be expected to be provided. These error messages may be cleaned to the extent that usernames or other data deemed private may be removed, however extensive redaction of data may result in difficulty providing insight into the failure.

In these examples, each ssh command is followed by a shell command to print the exit status of the previously issued command. Replace the user and IP elements in angle brackets with the username being utilized and the IP address of the target host.

For Linux targets:

        ssh -v <user>@<IP> 'true'
        echo $?
        ssh -v <user>@<IP> 'sudo true'
        echo $?
        ssh -v <user>@<IP> 'uname -s'
        echo $?
        ssh -v <user>@<IP> 'sudo env PATH=${PATH}:/sbin ifconfig -a'
        echo $?
        ssh -v <user>@<IP> 'sudo df -P'
        echo $?
        ssh -v <user>@<IP> 'sudo netstat --inet --inet6 -n -p –a -t'
        echo $?

For AIX targets:

        ssh -v <user>@<IP> 'true'
        echo $?
        ssh -v <user>@<IP> 'sudo true'
        echo $?
        ssh -v <user>@<IP> 'uname -s'
        echo $?
        ssh -v <user>@<IP> 'lsdev'
        echo $?
        ssh -v <user>@<IP> 'lsattr -E -l sys0 -a realmem'
        echo $?
        ssh -v <user>@<IP> 'df -Pk'
        echo $?
        ssh -v <user>@<IP> 'df -Pk'
        echo $?
        ssh -v <user>@<IP> 'lsof -i -nP'
        echo $?
        ssh -v <user>@<IP> 'netstat -an -f inet'
        echo $?

Error Messages

Understanding the error messages received when testing the credential, through the RN150 Virtual Appliance credential testing facility or through manual testing, is key to pinpointing the problem. Due to the nature of the ssh client/server interaction, it is common to receive non-error related text in the error message. This includes any banner text returned to the client by the server, as well as a message indicating that the server’s host keys have been added to the list of known hosts. These elements of the error can be ignored.

The following is a non-exhaustive list of common error messages, and their meaning:

Error Text Context Description of behavior Actions
Successfully connected, but sudo validation failed credential testing Connection and authentication was successful, but the user account was unable to utilize sudo
  • Review sudo configuration
Unsupported Operating System: … credential testing The target OS is not yet supported by the SSH Collection Module
  • Contact your RISC Networks sales representative or support
Connection refused connection

The ssh client connection was not successful. This could mean that:

  • the ssh server is not running
  • the ssh server is not listening on TCP 22
  • the local firewall on the host blocked the attempt
  • network ACLs blocked the attempt
  • Ensure ssh server is running on host
  • Ensure ssh server is listening on TCP 22 on host
  • Ensure local firewall is not blocking
  • Ensure network ACLs are not blocking
No route to host connection The RN150 has no route to the target IP address
  • Review network configuration on the RN150
  • Review VLAN configurations
  • Review routing to that subnet
Permission denied (…) authentication

The credentials offered by the client were refused by the server. This could mean that:

  • the username is not a valid user on the target
  • the user is not permitted
  • the password is invalid for the user
  • the key was corrupted or an invalid or unknown type
  • the key was not valid for the user.
  • file permissions on files related to authentication (keys, configuration files, etc) were not correct

This message is followed by a list, in parentheses, of authentication methods announced by the server

  • Ensure credential was entered correctly in the RN150
  • Ensure username exists on the target
  • Review ssh access controls
  • Review authentication configuration
  • Ensure private key is not encrypted
  • Review file permissions on target
Connection reset by peer any

The target ssh server suddenly aborted the connection. This is a generic error that has a large number of possible causes.

Often, this is a particular form of refused connection, where the initial stages of the connection were successful but additional access controls determined that the connection cannot continue.

This could also be due to one of the following:

  • The TCP session was lost
  • The client and server were unable to negotiate compatible session keys and MACs
  • The server only supports version 1 of the SSH protocol
  • Server is unable to handle an additional connection, or to fork a process for the requested command
  • Ensure that hosts.deny, hosts.allow, or similar are configured to permit the connection
  • Ensure that the ssh server supports version 2 of the protocol
  • Review server logs on the target
sudo: sorry, you must have a tty to run sudo privileged command execution The sudo configuration requires a TTY device for the user session
  • Review sudo configuration documentation above
  • Apply configuration parameters to sudo
sudo: no tty present and no askpass program specified privileged command execution The sudo configuration does not require a TTY device, but requires a password. Due to the necessity of having a TTY device in order to provide a password prompt, this results in sudo refusing to operate
  • Review the sudo configuration
  • Remove the password restriction for the user account

sudo: >>> /etc/sudoers: syntax error near line #<<<
sudo: parse error in /etc/sudoers near line #
sudo: no valid sudoers sources found, quitting
sudo: unable to initialize policy plugin

privileged command execution The sudo configuration is syntactically invalid and was refused
  • Correct sudo configuration syntax
  • Test sudo locally on target