This discussion is archived
0 Replies Latest reply: Aug 23, 2012 2:18 AM by Dude! RSS

OL5 Howto: Installing and using cURL with support for SCP and SFTP

Dude! Guru
Currently Being Moderated
h1. Oracle Linux 5 Howto: Installing and using cURL with support for SCP and SFTP

Author: Dude, 12-12-2011

History:

23-Aug-2012, corrected typing mistake: id_rsa.pub was meant to read id_dsa.pub.

h2. 1) Introduction

h3. a) What does SCP and SFTP mean?

SCP (Secure Copy) and SFTP (Secure FTP) are network protocols provided by SSH (Secure Shell) to transfer data between local and remote computer systems. SCP and SFTP rely on SSH to encrypt data transfers and manage user authentication.

SCP is typically used on Unix platforms and supported by SSH version 1 and 2. SFTP is more platform-independent and was developed as an extension for SSH version 2 to provide file transfer and file management functions. SCP is generally faster than SFTP, but limited to file transfers only, and unlike SFTP, cannot list files in a directory, for instance.

SFTP should not be confused with FTPS (FTP-TLS/SSL), which use TLS (Transport Layer Security), or its predecessor SSL (Secure Socket Layer) to encrypt segments of network connections.

h3. b) What is cURL?

cURL is a is a free multi-platform application that supports several common network protocols to transfer data between local and remote systems. It offers many features, can work without user interaction or feedback, and hence is an ideal companion for command shell scripts. cURL also provides "libcurl", a free multi-protocol C based file transfer library for your use with your own software.

The following commands will show version, supported protocols and features of cURL and libcurl:
curl --version
curl-config --features --protocols 
h3. c) cURL in Oracle Linux 5

As of this writing, cURL 7.15.5 is the latest available version for Red Hat Enterprise Linux 5.7 and other RHEL derivatives, including also Oracle Linux 5.7. This version of cURL does not support SCP and SFTP protocols.
curl 7.15.5 (i686-redhat-linux-gnu) libcurl/7.15.5 OpenSSL/0.9.8b zlib/1.2.3 libidn/0.6.5
Protocols: tftp ftp telnet dict ldap http file https ftps
Features: GSS-Negotiate IDN IPv6 Largefile NTLM SSL libz
curl: (1) Unsupported protocol: sftp
Support for SCP and SFTP was added in cURL 7.16.1. The latest version of cURL is 7.23.1, however, required software to be able to compile this version and other recent versions of cURL have advanced too far and are not available from standard software distribution channels. Compiling the required shared libraries does not work out of the box and further risks compatibility issues with other applications.





h2. 2) Compiling and installing cURL including SCP and SFTP support

According to my own attempts to compile several versions of cURL, version 7.17.1 is the last version that compiles without problems or warnings using the software available from the public Oracle Linux distribution channel.

The following has been tested using a default installation of Oracle Linux 5.7 x86 and x86_64.

h3. a) Oracle public software distribution

Setup access to the Oracle public software repository according to http://public-yum.oracle.com. You only need to enable the distribution channel for the Linux release version you have installed, e.g. ol5_u7_base.

h3. b) Software prerequisites

Open a terminal command prompt and copy and paste the following to install required software:
yum -y install gcc zlib-devel openssl-devel openldap-devel
yum -y install libidn-devel libgssapi-devel c-ares-devel
The required "libssh2" C-library is available from EPEL (http://fedoraproject.org/wiki/EPEL):
wget http://download.fedoraproject.org/pub/epel/5/i386/epel-release-5-4.noarch.rpm
rpm -Uvh epel-release-5-4.noarch.rpm
yum -y install libssh2-devel
h3. c) Download, compile and install cURL

Login as a regular user, then download and compile cURL:
wget http://curl.haxx.se/download/curl-7.17.1.tar.gz
tar zxf curl-7.17.1.tar.gz
cd curl-7.17.1
./configure --enable-ldaps --with-gssapi --enable-ares --enable-ipv6
Make sure it shows the following at the end:
curl version: 7.17.1
Host setup: i686-pc-linux-gnu
Install prefix: /usr/local
Compiler: gcc
SSL support: enabled (OpenSSL)
SSH support: enabled (libSSH2)
zlib support: enabled
krb4 support: no (--with-krb4*)
GSSAPI support: enabled (MIT/Heimdal)
SPNEGO support: no (--with-spnego)
c-ares support: enabled
ipv6 support: enabled
IDN support: enabled
Build libcurl: Shared=yes, Static=yes
Built-in manual: enabled
Verbose errors: enabled (--disable-verbose)
SSPI support: no (--enable-sspi)
ca cert path: /usr/local/share/curl/curl-ca-bundle.crt
LDAP support: enabled (OpenLDAP)
LDAPS support: enabled
Continue using the "make" utility and install cURL:
make
su root
make install
h3. d) Verify success

Type the following to verify the success of the installation:
curl --version
curl 7.17.1 (i686-pc-linux-gnu) libcurl/7.17.1 OpenSSL/0.9.8b zlib/1.2.3 c-ares/1.6.0 libidn/0.6.5 libssh2/0.18
Protocols: tftp ftp telnet dict ldap ldaps http file https ftps scp sftp
Features: AsynchDNS GSS-Negotiate IDN IPv6 Largefile NTLM SSL libz
h2. 3) Set up SSH user equivalence for SCP and SFTP

SSH user equivalence is a secure way to connect to another host without being prompted to enter or hardcode a login password. cURL SCP and SFTP work with key pairs based on the DSA encryption algorithm.

The following show the commands to to allow "bb@saturn.example.com" to connect to "dude@example.com" without a password.

h3. a) Create public and private keys

Login as user "bb" at "saturn.example.com" and copy and paste the following commands:
mkdir -p ~/.ssh
chmod 700 ~/.ssh
rm -f ~/.ssh/id_dsa
ssh-keygen -t dsa -N "" -f ~/.ssh/id_dsa
The public key stored inside "id_dsa.pub" needs to be transferred to the remote user account. The "id_dsa" file stores your private key and must not be shared.

h3. b) Copy public key to remote account

Create the necessary .ssh directory on the remote host and set required privileges. Enter "yes" to continue connecting in case this is your first time access. When prompted for a password, enter the password of the remote user (dude):
ssh dude@vm16.example.com "mkdir -p .ssh; chmod 700 .ssh"
Copy the DSA public key to the remote host account and set required privileges. When prompted for the password, enter the password of the remote user (dude):
KEY=`cat ~/.ssh/id_dsa.pub`
ssh dude@vm16.example.com "echo "$KEY" >> .ssh/authorized_keys; chmod 644 .ssh/authorized_keys"
h3. c) Verify success

You should no longer be prompted for a password:
ssh dude@vm16.example.com
Last login: Sun Dec 11 11:16:36 2011 from saturn.example.com
h2. 4) cURL Examples:

Upload "archive.tar" to the home directory of user "dude", prompt for password:
curl -T archive.tar -u dude scp://vm16.example.com/~/
Upload "archive.tar" to the home directory of user "dude" with password:
curl -T archive.tar scp://dude:password@vm16.example.com/~/
Upload "archive.tar" to the home directory of user "dude" with SSH user equivalence:
curl -T archive.tar scp://dude@vm16.example.com/~/
Upload "archive.tar as "uploaded.zip" into "/shared at the remote host:
curl -T "archive.tar" scp://dude@vm16.example.com/shared/uploaded.zip
Upload "archive.tar" after renaming the existing "archive.tar" to "archive.tar_old":
curl -Q "rename archive.tar archive.tar_old" -T archive.tar sftp://dude@vm16.example.com/~/
Get a listing of files in the /etc directory:
curl sftp://dude@vm16.example.com/etc///dude@vm16.example.com/etc/
Display the contents of "/etc/passwd":
curl scp://dude@vm16.example.com/etc/passwd
Download "/etc/profile" and save it as "vm16.profile" using # style progress bar:
curl -# -o vm16.profile scp://dude@vm16.example.com/etc/profile
Download "archive.tar" and delete it from the remote host after successful transfer:
curl -Q "-rm archive.tar" -O sftp://dude@vm16.example.com/~/archive.tar
Download "archive.tar", but limit data transfer rate to 10 KB/s:
curl --limit-rate 10k -O sftp://dude@vm16.example.com/~/archive.tar//dude@vm16.example.com/~/archive.tar
Download "archive.tar" using the interface with IP 10.0.1.2:
curl --interface 10.0.1.2 -O scp://dude@vm16.example.com/~/archive.tar
Download "archive.tar" in silent mode but show any errors that occurs:
curl -sS -O scp://dude@vm16.example.com/~/archive.tar
Download "archive.tar using a HTTP proxy at port 8080:
curl -x proxy.example.com:8080 -O scp://dude@vm16.example.com/~/archive.tar
For more information, options and explanations, see the man pages of cURL.
{code}
man curl
info curl
{code}


h2. 5) Troubleshooting

curl: (79) Upload failed: Operation failed
Verify you use the correct curl syntax. You might be missing the destination or source filename or using an invalid directory location.
Show what cURL is doing or create a trace log.
{code}
curl --verbose sftp://dude@vm16.example.com/~/.ssh
curl --trace-ascii tracelog.out sftp://dude@vm16.example.com/~/
cat tracelog.out
{code}

After successful "make install" cURL still shows the older previous version.
The $PATH environment variable might be missing "/usr/local/bin" or not having it in the right place.
{code}
echo $PATH
/usr/kerberos/bin:/usr/local/bin:/bin:/usr/bin:/home/dude/bin

which curl
/usr/local/bin/curl
{code}


h2. 6) Uninstall cURL and related software

The commands below will completely uninstall your cURL installation:
{code}
su - root
yum remove gcc zlib-devel openssl-devel openldap-devel
yum remove libidn-devel libgssapi-devel c-ares-devel
yum remove libssh2
su - you
cd curl-7.17.1
su root
make uninstall
su - you
curl --version
curl 7.15.5 (x86_64-redhat-linux-gnu) libcurl/7.15.5 OpenSSL/0.9.8b zlib/1.2.3 libidn/0.6.5
Protocols: tftp ftp telnet dict ldap ldaps http file https ftps scp sftp
Features: AsynchDNS GSS-Negotiate IDN IPv6 Largefile NTLM SSL libz
{code}


h2. 7) Notes and References

This article is mainly the result of my own testing and putting various pieces of informations together.

The following sites were helpful:

http://curl.haxx.se/changes.html
http://en.wikipedia.org/wiki/SSH_File_Transfer_Protocol
http://en.wikipedia.org/wiki/Secure_copy
http://en.wikipedia.org/wiki/Secure_Shell


Best of luck!

Legend

  • Correct Answers - 10 points
  • Helpful Answers - 5 points