5 Replies Latest reply: Dec 10, 2012 6:30 AM by 800381 RSS

    Cannot send snapshot over SSH

    user9368043
      I have two Solaris 11.0 machines - a server (NAS) and a backup machine. I have a script on NAS that creates a new ZFS snapshot and sends it to backup over SSH. The script is executed every day using CRON.

      The snapshot gets created successfully. When trying to send it over to backup, however, using the following command:

      sudo zfs send -i $first $second | ssh $bs_username@$bs_host sudo /usr/sbin/zfs receive -F backup/$1;

      I sometimes get the following error message:

      Permission denied (gssapi-keyex,gssapi-with-mic,publickey,keyboard-interactive).

      The error is probably related to the SSH connection, as the following command fails as well:

      ssh $bs_username@$bs_host sudo /usr/bin/pfexec /usr/sbin/init 5

      Note that the backup process sometimes succeeds, sometimes fails. It has never yet succeeded when launched by CRON (possibly an issue with permissions to launch the script?), but it has both failed and succeeded when running manually.

      On the backup machine, permissions to sudo without providing a password (for pfexec and zfs at least) was given to the $bs_username user account.

      If you'll find that I have omitted some important details, please let me know.
      Thank you for any advice.

      Dusan
        • 1. Re: Cannot send snapshot over SSH
          muvvas
          can u try with ssh in debug mode
          • 2. Re: Cannot send snapshot over SSH
            user9368043
            I tried running the script manually in debug mode and got the following two outputs (if I remember correctly, the first one was run without and the other one with sudo).

            1.

            Sun_SSH_2.0, SSH protocols 1.5/2.0, OpenSSL 0x1000005f
            debug1: Reading configuration data /etc/ssh/ssh_config
            debug1: Rhosts Authentication disabled, originating port will not be trusted.
            debug1: ssh_connect: needpriv 0
            debug1: Connecting to avatar1 [10.10.11.25] port 22.
            debug1: Connection established.
            debug1: ssh_kmf_check_uri: /home/rychnd/.ssh/identity
            debug1: Identity file/URI '/home/rychnd/.ssh/identity' pubkey type UNKNOWN
            debug1: ssh_kmf_check_uri: /home/rychnd/.ssh/id_rsa
            debug1: ssh_kmf_key_from_blob: blob length is 277.
            debug1: Identity file/URI '/home/rychnd/.ssh/id_rsa' pubkey type ssh-rsa
            debug1: ssh_kmf_check_uri: /home/rychnd/.ssh/id_dsa
            debug1: Identity file/URI '/home/rychnd/.ssh/id_dsa' pubkey type UNKNOWN
            debug1: Logging to host: avatar1
            debug1: Local user: rychnd Remote user: rychnd
            debug1: Remote protocol version 2.0, remote software version Sun_SSH_2.0
            debug1: match: Sun_SSH_2.0 pat Sun_SSH_2.*
            debug1: Enabling compatibility mode for protocol 2.0
            debug1: Local version string SSH-2.0-Sun_SSH_2.0
            debug1: use_engine is 'yes'
            debug1: pkcs11 engine initialized, now setting it as default for RSA, DSA, and symmetric ciphers
            debug1: pkcs11 engine initialization complete
            debug1: Creating a global KMF session.
            debug1: My KEX proposal before adding the GSS KEX algorithm:
            debug1: Failed to acquire GSS-API credentials for any mechanisms (No credentials were supplied, or the credentials were unavailable or inaccessible

            *)*
            debug1: SSH2_MSG_KEXINIT sent
            debug1: SSH2_MSG_KEXINIT received
            debug1: My KEX proposal I sent to the peer:
            debug1: KEX proposal I received from the peer:
            debug1: kex: server->client aes128-ctr hmac-md5 none
            debug1: kex: client->server aes128-ctr hmac-md5 none
            debug1: Host key algorithm 'ssh-rsa' chosen for the KEX.
            debug1: Peer sent proposed langtags, ctos: de-DE,en-US,es-ES,fr-FR,it-IT,ja-JP,ko-KR,pt-BR,zh-CN,zh-TW,i-default
            debug1: Peer sent proposed langtags, stoc: de-DE,en-US,es-ES,fr-FR,it-IT,ja-JP,ko-KR,pt-BR,zh-CN,zh-TW,i-default
            debug1: We proposed langtags, ctos: en-US
            debug1: We proposed langtags, stoc: en-US
            debug1: Negotiated lang: en-US
            debug1: SSH2_MSG_KEX_DH_GEX_REQUEST sent
            debug1: expecting SSH2_MSG_KEX_DH_GEX_GROUP
            debug1: Remote: Negotiated main locale: en_US.UTF-8
            debug1: Remote: Negotiated messages locale: en_US.UTF-8
            debug1: dh_gen_key: priv key bits set: 127/256
            debug1: bits set: 1567/3191
            debug1: SSH2_MSG_KEX_DH_GEX_INIT sent
            debug1: expecting SSH2_MSG_KEX_DH_GEX_REPLY
            debug1: ssh_kmf_key_from_blob: blob length is 277.
            debug1: ssh_kmf_key_from_blob: blob length is 277.
            debug1: ssh_kmf_key_from_blob: blob length is 277.
            debug1: Host 'avatar1' is known and matches the RSA host key.
            debug1: Found key in /home/rychnd/.ssh/known_hosts:1
            debug1: bits set: 1629/3191
            debug1: ssh_rsa_verify: signature correct
            debug1: set_newkeys: setting new keys for 'out' mode
            debug1: SSH2_MSG_NEWKEYS sent
            debug1: expecting SSH2_MSG_NEWKEYS
            debug1: set_newkeys: setting new keys for 'in' mode
            debug1: SSH2_MSG_NEWKEYS received
            debug1: done: ssh_kex2.
            debug1: send SSH2_MSG_SERVICE_REQUEST
            debug1: got SSH2_MSG_SERVICE_ACCEPT
            debug1: Authentications that can continue: gssapi-keyex,gssapi-with-mic,publickey,password,keyboard-interactive
            debug1: Next authentication method: gssapi-keyex
            debug1: Next authentication method: gssapi-with-mic
            debug1: Failed to acquire GSS-API credentials for any mechanisms (No credentials were supplied, or the credentials were unavailable or inaccessible

            *)*
            debug1: Next authentication method: publickey
            debug1: Trying private key: /home/rychnd/.ssh/identity
            debug1: ssh_kmf_check_uri: /home/rychnd/.ssh/identity
            debug1: Trying public key: /home/rychnd/.ssh/id_rsa
            debug1: Server accepts key: pkalg ssh-rsa blen 277 lastkey 80aee58 hint 1
            debug1: ssh_kmf_key_from_blob: blob length is 277.
            debug1: ssh_kmf_check_uri: /home/rychnd/.ssh/id_rsa
            debug1: read PEM private key done: type RSA
            debug1: Authentication succeeded (publickey)
            debug1: fd 5 setting O_NONBLOCK
            debug1: fd 6 setting O_NONBLOCK
            *debug1: channel 0: new [client-session]*
            debug1: send channel open 0
            debug1: Entering interactive session.
            debug1: ssh_session2_setup: id 0
            debug1: channel request 0: env
            debug1: channel request 0: env
            debug1: channel request 0: env
            debug1: channel request 0: env
            debug1: channel request 0: env
            debug1: channel request 0: env
            debug1: channel request 0: env
            debug1: channel request 0: env
            debug1: Sending command: sudo /usr/sbin/zfs receive -F backup/rpool/ROOT
            debug1: channel request 0: exec
            debug1: channel 0: open confirm rwindow 0 rmax 32768
            debug1: Remote: Channel 0 set: LANG=en_US.UTF-8
            debug1: Remote: Channel 0 set: LC_CTYPE=
            debug1: Remote: Channel 0 set: LC_COLLATE=
            debug1: Remote: Channel 0 set: LC_TIME=
            debug1: Remote: Channel 0 set: LC_NUMERIC=
            debug1: Remote: Channel 0 set: LC_MONETARY=
            debug1: Remote: Channel 0 set: LC_MESSAGES=
            debug1: Remote: Channel 0 set: LC_ALL=
            debug1: channel 0: read<=0 rfd 5 len 0
            debug1: channel 0: read failed
            debug1: channel 0: close_read
            debug1: channel 0: input open -> drain
            debug1: channel 0: ibuf empty
            debug1: channel 0: send eof
            debug1: channel 0: input drain -> closed
            debug1: channel 0: rcvd eof
            debug1: channel 0: output open -> drain
            debug1: channel 0: obuf empty
            debug1: channel 0: close_write
            debug1: channel 0: output drain -> closed
            debug1: client_input_channel_req: channel 0 rtype exit-status reply 0
            debug1: channel 0: rcvd close
            debug1: channel 0: almost dead
            debug1: channel 0: gc: notify user
            debug1: channel 0: gc: user detached
            debug1: channel 0: send close
            debug1: channel 0: is dead
            debug1: channel 0: garbage collecting
            debug1: channel_free: channel 0: client-session, nchannels 1
            debug1: fd 0 clearing O_NONBLOCK
            debug1: fd 1 clearing O_NONBLOCK
            debug1: Transferred: stdin 0, stdout 0, stderr 0 bytes in 4.0 seconds
            debug1: Bytes per second: stdin 0.0, stdout 0.0, stderr 0.0
            debug1: Exit status 0

            2.

            Sun_SSH_2.0, SSH protocols 1.5/2.0, OpenSSL 0x1000005f
            debug1: Reading configuration data /etc/ssh/ssh_config
            debug1: Rhosts Authentication disabled, originating port will not be trusted.
            debug1: ssh_connect: needpriv 0
            debug1: Connecting to avatar1 [10.10.11.25] port 22.
            debug1: Connection established.
            debug1: ssh_kmf_check_uri: /root/.ssh/identity
            debug1: Identity file/URI '/root/.ssh/identity' pubkey type UNKNOWN
            debug1: ssh_kmf_check_uri: /root/.ssh/id_rsa
            debug1: Identity file/URI '/root/.ssh/id_rsa' pubkey type UNKNOWN
            debug1: ssh_kmf_check_uri: /root/.ssh/id_dsa
            debug1: Identity file/URI '/root/.ssh/id_dsa' pubkey type UNKNOWN
            debug1: Logging to host: avatar1
            debug1: Local user: root Remote user: rychnd
            debug1: Remote protocol version 2.0, remote software version Sun_SSH_2.0
            debug1: match: Sun_SSH_2.0 pat Sun_SSH_2.*
            debug1: Enabling compatibility mode for protocol 2.0
            debug1: Local version string SSH-2.0-Sun_SSH_2.0
            debug1: use_engine is 'yes'
            debug1: pkcs11 engine initialized, now setting it as default for RSA, DSA, and symmetric ciphers
            debug1: pkcs11 engine initialization complete
            debug1: Creating a global KMF session.
            debug1: My KEX proposal before adding the GSS KEX algorithm:
            debug1: Failed to acquire GSS-API credentials for any mechanisms (No credentials were supplied, or the credentials were unavailable or inaccessible

            *)*
            debug1: SSH2_MSG_KEXINIT sent
            debug1: SSH2_MSG_KEXINIT received
            debug1: My KEX proposal I sent to the peer:
            debug1: KEX proposal I received from the peer:
            debug1: kex: server->client aes128-ctr hmac-md5 none
            debug1: kex: client->server aes128-ctr hmac-md5 none
            debug1: Host key algorithm 'ssh-rsa' chosen for the KEX.
            debug1: Peer sent proposed langtags, ctos: de-DE,en-US,es-ES,fr-FR,it-IT,ja-JP,ko-KR,pt-BR,zh-CN,zh-TW,i-default
            debug1: Peer sent proposed langtags, stoc: de-DE,en-US,es-ES,fr-FR,it-IT,ja-JP,ko-KR,pt-BR,zh-CN,zh-TW,i-default
            debug1: We proposed langtags, ctos: en-US
            debug1: We proposed langtags, stoc: en-US
            debug1: Negotiated lang: en-US
            debug1: SSH2_MSG_KEX_DH_GEX_REQUEST sent
            debug1: expecting SSH2_MSG_KEX_DH_GEX_GROUP
            debug1: Remote: Negotiated main locale: en_US.UTF-8
            debug1: Remote: Negotiated messages locale: en_US.UTF-8
            debug1: dh_gen_key: priv key bits set: 154/256
            debug1: bits set: 1596/3191
            debug1: SSH2_MSG_KEX_DH_GEX_INIT sent
            debug1: expecting SSH2_MSG_KEX_DH_GEX_REPLY
            debug1: ssh_kmf_key_from_blob: blob length is 277.
            debug1: ssh_kmf_key_from_blob: blob length is 277.
            debug1: ssh_kmf_key_from_blob: blob length is 277.
            debug1: Host 'avatar1' is known and matches the RSA host key.
            debug1: Found key in /root/.ssh/known_hosts:2
            debug1: bits set: 1632/3191
            debug1: ssh_rsa_verify: signature correct
            debug1: set_newkeys: setting new keys for 'out' mode
            debug1: SSH2_MSG_NEWKEYS sent
            debug1: expecting SSH2_MSG_NEWKEYS
            debug1: set_newkeys: setting new keys for 'in' mode
            debug1: SSH2_MSG_NEWKEYS received
            debug1: done: ssh_kex2.
            debug1: send SSH2_MSG_SERVICE_REQUEST
            debug1: got SSH2_MSG_SERVICE_ACCEPT
            debug1: Authentications that can continue: gssapi-keyex,gssapi-with-mic,publickey,password,keyboard-interactive
            debug1: Next authentication method: gssapi-keyex
            debug1: Next authentication method: gssapi-with-mic
            debug1: Failed to acquire GSS-API credentials for any mechanisms (No credentials were supplied, or the credentials were unavailable or inaccessible

            *)*
            debug1: Next authentication method: publickey
            debug1: Trying private key: /root/.ssh/identity
            debug1: ssh_kmf_check_uri: /root/.ssh/identity
            debug1: Trying private key: /root/.ssh/id_rsa
            debug1: ssh_kmf_check_uri: /root/.ssh/id_rsa
            debug1: Trying private key: /root/.ssh/id_dsa
            debug1: ssh_kmf_check_uri: /root/.ssh/id_dsa
            debug1: Next authentication method: keyboard-interactive
            debug1: Authentications that can continue: gssapi-keyex,gssapi-with-mic,publickey,keyboard-interactive
            debug1: Next authentication method: keyboard-interactive
            debug1: Authentications that can continue: gssapi-keyex,gssapi-with-mic,publickey,keyboard-interactive
            debug1: Authentications that can continue: gssapi-keyex,gssapi-with-mic,publickey,keyboard-interactive
            debug1: No more authentication methods to try.
            Permission denied (gssapi-keyex,gssapi-with-mic,publickey,keyboard-interactive).
            debug1: Calling cleanup 0x807ca90(0x0)
            • 3. Re: Cannot send snapshot over SSH
              user9368043
              So, it seems like that when running via CRON (or with sudo), it looks for the public key in the root home directory.

              How would I have CRON execute the script as a custom user, say rychnd?

              Thanks for your help
              • 4. Re: Cannot send snapshot over SSH
                muvvas
                u can create the cron for user based

                cron.allow ..add the users here
                • 5. Re: Cannot send snapshot over SSH
                  800381
                  You'll also need to disable escape characters for the SSH session:
                  ssh -e none ....
                  If you don't do that, when the SSH process detects the binary data corresponding to the escape character (default is "~"), that character and the next character(s) will not be considered part of the data stream.