0 Replies Latest reply: Aug 23, 2012 4:18 AM by Catch 22 RSS

    OL5 Howto: Installing and using cURL with support for SCP and SFTP

    Catch 22
      h1. Oracle Linux 5 Howto: Installing and using cURL with support for SCP and SFTP

      Author: Dude, 12-12-2011

      History:

      23-Aug-2012, corrected typing mistake: id_rsa.pub was meant to read id_dsa.pub.

      h2. 1) Introduction

      h3. a) What does SCP and SFTP mean?

      SCP (Secure Copy) and SFTP (Secure FTP) are network protocols provided by SSH (Secure Shell) to transfer data between local and remote computer systems. SCP and SFTP rely on SSH to encrypt data transfers and manage user authentication.

      SCP is typically used on Unix platforms and supported by SSH version 1 and 2. SFTP is more platform-independent and was developed as an extension for SSH version 2 to provide file transfer and file management functions. SCP is generally faster than SFTP, but limited to file transfers only, and unlike SFTP, cannot list files in a directory, for instance.

      SFTP should not be confused with FTPS (FTP-TLS/SSL), which use TLS (Transport Layer Security), or its predecessor SSL (Secure Socket Layer) to encrypt segments of network connections.

      h3. b) What is cURL?

      cURL is a is a free multi-platform application that supports several common network protocols to transfer data between local and remote systems. It offers many features, can work without user interaction or feedback, and hence is an ideal companion for command shell scripts. cURL also provides "libcurl", a free multi-protocol C based file transfer library for your use with your own software.

      The following commands will show version, supported protocols and features of cURL and libcurl:
      curl --version
      curl-config --features --protocols 
      h3. c) cURL in Oracle Linux 5

      As of this writing, cURL 7.15.5 is the latest available version for Red Hat Enterprise Linux 5.7 and other RHEL derivatives, including also Oracle Linux 5.7. This version of cURL does not support SCP and SFTP protocols.
      curl 7.15.5 (i686-redhat-linux-gnu) libcurl/7.15.5 OpenSSL/0.9.8b zlib/1.2.3 libidn/0.6.5
      Protocols: tftp ftp telnet dict ldap http file https ftps
      Features: GSS-Negotiate IDN IPv6 Largefile NTLM SSL libz
      curl: (1) Unsupported protocol: sftp
      Support for SCP and SFTP was added in cURL 7.16.1. The latest version of cURL is 7.23.1, however, required software to be able to compile this version and other recent versions of cURL have advanced too far and are not available from standard software distribution channels. Compiling the required shared libraries does not work out of the box and further risks compatibility issues with other applications.





      h2. 2) Compiling and installing cURL including SCP and SFTP support

      According to my own attempts to compile several versions of cURL, version 7.17.1 is the last version that compiles without problems or warnings using the software available from the public Oracle Linux distribution channel.

      The following has been tested using a default installation of Oracle Linux 5.7 x86 and x86_64.

      h3. a) Oracle public software distribution

      Setup access to the Oracle public software repository according to http://public-yum.oracle.com. You only need to enable the distribution channel for the Linux release version you have installed, e.g. ol5_u7_base.

      h3. b) Software prerequisites

      Open a terminal command prompt and copy and paste the following to install required software:
      yum -y install gcc zlib-devel openssl-devel openldap-devel
      yum -y install libidn-devel libgssapi-devel c-ares-devel
      The required "libssh2" C-library is available from EPEL (http://fedoraproject.org/wiki/EPEL):
      wget http://download.fedoraproject.org/pub/epel/5/i386/epel-release-5-4.noarch.rpm
      rpm -Uvh epel-release-5-4.noarch.rpm
      yum -y install libssh2-devel
      h3. c) Download, compile and install cURL

      Login as a regular user, then download and compile cURL:
      wget http://curl.haxx.se/download/curl-7.17.1.tar.gz
      tar zxf curl-7.17.1.tar.gz
      cd curl-7.17.1
      ./configure --enable-ldaps --with-gssapi --enable-ares --enable-ipv6
      Make sure it shows the following at the end:
      curl version: 7.17.1
      Host setup: i686-pc-linux-gnu
      Install prefix: /usr/local
      Compiler: gcc
      SSL support: enabled (OpenSSL)
      SSH support: enabled (libSSH2)
      zlib support: enabled
      krb4 support: no (--with-krb4*)
      GSSAPI support: enabled (MIT/Heimdal)
      SPNEGO support: no (--with-spnego)
      c-ares support: enabled
      ipv6 support: enabled
      IDN support: enabled
      Build libcurl: Shared=yes, Static=yes
      Built-in manual: enabled
      Verbose errors: enabled (--disable-verbose)
      SSPI support: no (--enable-sspi)
      ca cert path: /usr/local/share/curl/curl-ca-bundle.crt
      LDAP support: enabled (OpenLDAP)
      LDAPS support: enabled
      Continue using the "make" utility and install cURL:
      make
      su root
      make install
      h3. d) Verify success

      Type the following to verify the success of the installation:
      curl --version
      curl 7.17.1 (i686-pc-linux-gnu) libcurl/7.17.1 OpenSSL/0.9.8b zlib/1.2.3 c-ares/1.6.0 libidn/0.6.5 libssh2/0.18
      Protocols: tftp ftp telnet dict ldap ldaps http file https ftps scp sftp
      Features: AsynchDNS GSS-Negotiate IDN IPv6 Largefile NTLM SSL libz
      h2. 3) Set up SSH user equivalence for SCP and SFTP

      SSH user equivalence is a secure way to connect to another host without being prompted to enter or hardcode a login password. cURL SCP and SFTP work with key pairs based on the DSA encryption algorithm.

      The following show the commands to to allow "bb@saturn.example.com" to connect to "dude@example.com" without a password.

      h3. a) Create public and private keys

      Login as user "bb" at "saturn.example.com" and copy and paste the following commands:
      mkdir -p ~/.ssh
      chmod 700 ~/.ssh
      rm -f ~/.ssh/id_dsa
      ssh-keygen -t dsa -N "" -f ~/.ssh/id_dsa
      The public key stored inside "id_dsa.pub" needs to be transferred to the remote user account. The "id_dsa" file stores your private key and must not be shared.

      h3. b) Copy public key to remote account

      Create the necessary .ssh directory on the remote host and set required privileges. Enter "yes" to continue connecting in case this is your first time access. When prompted for a password, enter the password of the remote user (dude):
      ssh dude@vm16.example.com "mkdir -p .ssh; chmod 700 .ssh"
      Copy the DSA public key to the remote host account and set required privileges. When prompted for the password, enter the password of the remote user (dude):
      KEY=`cat ~/.ssh/id_dsa.pub`
      ssh dude@vm16.example.com "echo "$KEY" >> .ssh/authorized_keys; chmod 644 .ssh/authorized_keys"
      h3. c) Verify success

      You should no longer be prompted for a password:
      ssh dude@vm16.example.com
      Last login: Sun Dec 11 11:16:36 2011 from saturn.example.com
      h2. 4) cURL Examples:

      Upload "archive.tar" to the home directory of user "dude", prompt for password:
      curl -T archive.tar -u dude scp://vm16.example.com/~/
      Upload "archive.tar" to the home directory of user "dude" with password:
      curl -T archive.tar scp://dude:password@vm16.example.com/~/
      Upload "archive.tar" to the home directory of user "dude" with SSH user equivalence:
      curl -T archive.tar scp://dude@vm16.example.com/~/
      Upload "archive.tar as "uploaded.zip" into "/shared at the remote host:
      curl -T "archive.tar" scp://dude@vm16.example.com/shared/uploaded.zip
      Upload "archive.tar" after renaming the existing "archive.tar" to "archive.tar_old":
      curl -Q "rename archive.tar archive.tar_old" -T archive.tar sftp://dude@vm16.example.com/~/
      Get a listing of files in the /etc directory:
      curl sftp://dude@vm16.example.com/etc///dude@vm16.example.com/etc/
      Display the contents of "/etc/passwd":
      curl scp://dude@vm16.example.com/etc/passwd
      Download "/etc/profile" and save it as "vm16.profile" using # style progress bar:
      curl -# -o vm16.profile scp://dude@vm16.example.com/etc/profile
      Download "archive.tar" and delete it from the remote host after successful transfer:
      curl -Q "-rm archive.tar" -O sftp://dude@vm16.example.com/~/archive.tar
      Download "archive.tar", but limit data transfer rate to 10 KB/s:
      curl --limit-rate 10k -O sftp://dude@vm16.example.com/~/archive.tar//dude@vm16.example.com/~/archive.tar
      Download "archive.tar" using the interface with IP 10.0.1.2:
      curl --interface 10.0.1.2 -O scp://dude@vm16.example.com/~/archive.tar
      Download "archive.tar" in silent mode but show any errors that occurs:
      curl -sS -O scp://dude@vm16.example.com/~/archive.tar
      Download "archive.tar using a HTTP proxy at port 8080:
      curl -x proxy.example.com:8080 -O scp://dude@vm16.example.com/~/archive.tar
      For more information, options and explanations, see the man pages of cURL.
      {code}
      man curl
      info curl
      {code}


      h2. 5) Troubleshooting

      curl: (79) Upload failed: Operation failed
      Verify you use the correct curl syntax. You might be missing the destination or source filename or using an invalid directory location.
      Show what cURL is doing or create a trace log.
      {code}
      curl --verbose sftp://dude@vm16.example.com/~/.ssh
      curl --trace-ascii tracelog.out sftp://dude@vm16.example.com/~/
      cat tracelog.out
      {code}

      After successful "make install" cURL still shows the older previous version.
      The $PATH environment variable might be missing "/usr/local/bin" or not having it in the right place.
      {code}
      echo $PATH
      /usr/kerberos/bin:/usr/local/bin:/bin:/usr/bin:/home/dude/bin

      which curl
      /usr/local/bin/curl
      {code}


      h2. 6) Uninstall cURL and related software

      The commands below will completely uninstall your cURL installation:
      {code}
      su - root
      yum remove gcc zlib-devel openssl-devel openldap-devel
      yum remove libidn-devel libgssapi-devel c-ares-devel
      yum remove libssh2
      su - you
      cd curl-7.17.1
      su root
      make uninstall
      su - you
      curl --version
      curl 7.15.5 (x86_64-redhat-linux-gnu) libcurl/7.15.5 OpenSSL/0.9.8b zlib/1.2.3 libidn/0.6.5
      Protocols: tftp ftp telnet dict ldap ldaps http file https ftps scp sftp
      Features: AsynchDNS GSS-Negotiate IDN IPv6 Largefile NTLM SSL libz
      {code}


      h2. 7) Notes and References

      This article is mainly the result of my own testing and putting various pieces of informations together.

      The following sites were helpful:

      http://curl.haxx.se/changes.html
      http://en.wikipedia.org/wiki/SSH_File_Transfer_Protocol
      http://en.wikipedia.org/wiki/Secure_copy
      http://en.wikipedia.org/wiki/Secure_Shell


      Best of luck!