Skip to Main Content
Forums

ODP.NET

Announcement

For appeals, questions and feedback about Oracle Forums, please email oracle-forums-moderators_us@oracle.com. Technical questions should be asked in the appropriate category. Thank you!

Interested in getting your voice heard by members of the Developer Marketing team at Oracle? Check out this post for AppDev or this post for AI focus group information.

Error message with Queued .NET component

406942Oct 20 2003 — edited Oct 21 2003
I am getting the following two error messages when using a .NET component through COM+, setup as a queued component. Basically, the component sets up and executes a single stored procedure on a single database before returning. I am using ODP.NET 9.2.0.2102.

ORA-00164: autonomous transaction disallowed within distributed transaction
ORA-24777: use of non-migratable database link not allowed

The weird part is that if I spawn a thread from within the Interop class called by MSMQ, the stored procedure executes without errors.

Any help would be appreciated.

Christian Nau

Comments

Avi Miller-Oracle
EdStevens wrote:
Any ideas what might be going on?
I'm guessing DNS isn't configured properly in the OL6 VM, which is causing sshd to timeout trying to resolve the incoming connection. Also, OL6 doesn't have any of the firstboot configuration tools in it's minimal/basic install like OL5 did.
1 person found this helpful
EdStevens
Avi Miller wrote:
EdStevens wrote:
Any ideas what might be going on?
I'm guessing DNS isn't configured properly in the OL6 VM, which is causing sshd to timeout trying to resolve the incoming connection. Also, OL6 doesn't have any of the firstboot configuration tools in it's minimal/basic install like OL5 did.
That's possible. I did just now forge ahead to wget the yum repo file, and it took quite a while before telling me it couldn't resolve it. End of the day ... I'll take a look at it in the morning.
Avi Miller-Oracle
EdStevens wrote:
That's possible. I did just now forge ahead to wget the yum repo file, and it took quite a while before telling me it couldn't resolve it. End of the day ... I'll take a look at it in the morning.
Note that 6.3 and 6.4 should have the public-yum-ol6.repo installed by default. I know 6.4 has it, and I believe 6.3 has it as well (though I can't remember exactly!)
Dude!
You might want to check if the delay is perhaps due to some security protocol failure and fallback.

Here is a previous thread that might be useful:
OEL 5.5 - access denied .... but not!
2316926

I suggest to logon using verbose mode to see where the ssh login hangs:

ssh -vvv root@hostname

I don't use PuTTY (I'm on Mac), but I think you have some options in the settings to enable verbose logging to a text file.

So far I have not had any issues with SSH and reverse DNS lookups. From what understand, it's disabled in the default /etc/ssh/sshd_config file (#UseDNS yes).

Edited by: Dude on May 4, 2013 6:28 AM
EdStevens
Dude wrote:
You might want to check if the delay is perhaps due to some security protocol failure and fallback.

Here is a previous thread that might be useful:
OEL 5.5 - access denied .... but not!
2316926

I suggest to logon using verbose mode to see where the ssh login hangs:

ssh -vvv root@hostname

I don't use PuTTY (I'm on Mac), but I think you have some options in the settings to enable verbose logging to a text file.

So far I have not had any issues with SSH and reverse DNS lookups. From what understand, it's disabled in the default /etc/ssh/sshd_config file (#UseDNS yes).
Yeah, well we see the lamer that started THAT thread ... ;-)

But based on that I did a series of tests. Increased my putty logging to include tcp packets, then connected both with and without GSSAPI enabled ... and repeated on one of my OL 5.x vm's.
In all both cases (OL 6.3 vs. 5.6) with GSSAPI enabled, there was an extra set of packets exhanged .. one outgoing and one incoming ... before a packet exchange using the password.
Logs were the same through this: 
Outgoing packet #0x4, type 5 / 0x05 (SSH2_MSG_SERVICE_REQUEST)
  00000000  00 00 00 0c 73 73 68 2d 75 73 65 72 61 75 74 68  ....ssh-userauth
Incoming packet #0x4, type 6 / 0x06 (SSH2_MSG_SERVICE_ACCEPT)
  00000000  00 00 00 0c 73 73 68 2d 75 73 65 72 61 75 74 68  ....ssh-userauth
Outgoing packet #0x5, type 50 / 0x32 (SSH2_MSG_USERAUTH_REQUEST)
  00000000  00 00 00 04 72 6f 6f 74 00 00 00 0e 73 73 68 2d  ....root....ssh-
  00000010  63 6f 6e 6e 65 63 74 69 6f 6e 00 00 00 04 6e 6f  connection....no
  00000020  6e 65                                            ne
Incoming packet #0x5, type 51 / 0x33 (SSH2_MSG_USERAUTH_FAILURE)
  00000000  00 00 00 22 70 75 62 6c 69 63 6b 65 79 2c 67 73  ..."publickey,gs
  00000010  73 61 70 69 2d 77 69 74 68 2d 6d 69 63 2c 70 61  sapi-with-mic,pa
  00000020  73 73 77 6f 72 64 00                             ssword.
{code}

Then, with GSSAPI enabled, we get this
{code}
Event Log: Using SSPI from SECUR32.DLL
Event Log: Attempting GSSAPI authentication
Outgoing packet #0x6, type 50 / 0x32 (SSH2_MSG_USERAUTH_REQUEST)
  00000000  00 00 00 04 72 6f 6f 74 00 00 00 0e 73 73 68 2d  ....root....ssh-
  00000010  63 6f 6e 6e 65 63 74 69 6f 6e 00 00 00 0f 67 73  connection....gs
  00000020  73 61 70 69 2d 77 69 74 68 2d 6d 69 63 00 00 00  sapi-with-mic...
  00000030  01 00 00 00 0b 06 09 2a 86 48 86 f7 12 01 02 02  .......*.H......
Incoming packet #0x6, type 51 / 0x33 (SSH2_MSG_USERAUTH_FAILURE)
  00000000  00 00 00 22 70 75 62 6c 69 63 6b 65 79 2c 67 73  ..."publickey,gs
  00000010  73 61 70 69 2d 77 69 74 68 2d 6d 69 63 2c 70 61  sapi-with-mic,pa
  00000020  73 73 77 6f 72 64 00                             ssword.
Event Log: GSSAPI authentication request refused
After which both configurations continue with 
Outgoing packet #0x7, type 50 / 0x32 (SSH2_MSG_USERAUTH_REQUEST)
  00000000  00 00 00 04 72 6f 6f 74 00 00 00 0e 73 73 68 2d  ....root....ssh-
  00000010  63 6f 6e 6e 65 63 74 69 6f 6e 00 00 00 08 70 61  connection....pa
  00000020  73 73 77 6f 72 64 00 XX XX XX XX XX XX XX XX XX  ssword.XXXXXXXXX
  00000030  XX XX XX                                         XXX
Outgoing packet #0x8, type 2 / 0x02 (SSH2_MSG_IGNORE)
  00000000  00 00 00 b0 82 5d 4d ef a3 dc 88 4e 39 75 ac 11  .....]M....N9u..
  00000010  ca 60 26 f6 c0 31 35 10 03 0d 87 25 92 79 ec 43  .`&..15....%.y.C
  00000020  5e ee ff e7 c0 dc a2 9a 91 be 0b 07 6a 2f 8f 63  ^...........j/.c
  00000030  73 4f 07 e1 21 dd 80 0f 6b d4 84 53 8a 8b 83 0b  sO..!...k..S....
  00000040  35 2a c2 ee 9a 87 77 a4 99 92 99 1d a2 f5 27 a2  5*....w.......'.
  00000050  4e b3 d1 6d db 48 bd fe 90 57 60 46 54 96 d6 74  N..m.H...W`FT..t
  00000060  90 60 e4 b1 c1 68 2b 29 7b cc 42 a7 ed a9 c5 9b  .`...h+){.B.....
  00000070  4b e3 78 bf 74 e2 97 7d cd d0 d6 a2 3e 9d 53 ec  K.x.t..}....>.S.
  00000080  2a fa 14 99 11 25 ef f8 c0 74 e1 4c 77 ed 44 ab  *....%...t.Lw.D.
  00000090  6d 24 20 d8 c2 0e f0 35 7a 20 e8 05 84 ee a2 d6  m$ ....5z ......
  000000a0  be 09 21 1e 9a 7a 85 eb 1a ab 49 da d7 34 a1 26  ..!..z....I..4.&
  000000b0  ff ad e4 29                                      ...)
Event Log: Sent password
Incoming packet #0x7, type 52 / 0x34 (SSH2_MSG_USERAUTH_SUCCESS)
Event Log: Access granted
Both the OL 6.3 and the OL 5.6 exhibited the exact same sequence of packets. On OL 6.3, with GSSAPI enabled, it consistently takes 55+ seconds to get to the password prompt. Without GSSAPI it still takes about 30 seconds. On the OL 5.6, it is near instantaneous with or without GSSAPI.

See my response to Avi Miller regarding follow up on DNS question.
EdStevens
Avi Miller wrote:
EdStevens wrote:
Any ideas what might be going on?
I'm guessing DNS isn't configured properly in the OL6 VM, which is causing sshd to timeout trying to resolve the incoming connection. Also, OL6 doesn't have any of the firstboot configuration tools in it's minimal/basic install like OL5 did.
As always, when we get into OS and networking, I know just enough to be dangerous, but here's what I've put together. I compared and contrasted the key net config files from the OL 5.6 system (which performs as expected) and the OL 6.3, which is the problem child. I'm not sure I understand ... no, I'm quite sure I don't understand -- the significance of the differences. In this listing, I go through the various files, top to bottom, with the ones from the OL 5.6 on the left and the 6.3 on the right.

Keep in mind that these two vm's are on the same host OS under the same VBox setup. Except for IP address, server name, and domain name, everything is 'just shook it out of the box.'. 


---------------------------------- /etc/sysconfig/network ---------------------------------- 

-------- OL 5.6 --------                                           -------- OL 6.3 --------
NETWORKINGyes                                                      NETWORKING=yes
NETWORKING_IPV6=no                                                 HOSTNAME=vblnxsrv02.vbdomain
HOSTNAME=vblnxsrv03.vbdomain


---------------------------------- ifcfg-eth0 ---------------------------------- 
-------- OL 5.7 --------                                           -------- OL 6.3 --------
# Intel Corporation 82540EM Gigabit Ethernet Controller            DEVICE="eth0"
DEVICE=eth0                                                        BOOTPROTO="dhcp"
BOOTPROTO=dhcp                                                     NM_CONTROLLED="yes"
HWADDR=08:00:27:75:FB:5D                                           ONBOOT="yes"
ONBOOT=yes                                                         TYPE="Ethernet"
DHCP_HOSTNAME=vblnxsrv03.vbdomain                                  UUID="77f51d12-27a8-4574-b6e5-d2cd11e3bdc2"
                                                                   HWADDR=08:00:27:D8:F5:3F
                                                                   DEFROUTE=yes
                                                                   PEERDNS=yes
                                                                   IPV4_FAILURE_FATAL=yes
                                                                   IPV6INIT=no
                                                                   NAME="System eth0"


---------------------------------- ifcfg-eth1 ---------------------------------- 
-------- OL 5.7 --------                                           -------- OL 6.3 --------
# Intel Corporation 82540EM Gigabit Ethernet Controller            DEVICE="eth1"
DEVICE=eth1                                                        BOOTPROTO="none"
BOOTPROTO=static                                                   NM_CONTROLLED="yes"
BROADCAST=192.168.56.255                                           ONBOOT="yes"
HWADDR=08:00:27:94:37:86                                           TYPE="Ethernet"
IPADDR=192.168.56.103                                              UUID="75a9f336-55c1-4947-b3ac-6e9e7450a84d"
NETMASK=255.255.255.0                                              IPADDR=192.168.56.102
NETWORK=192.168.56.0                                               PREFIX=24
ONBOOT=yes                                                         GATEWAY=192.168.56.1
                                                                   DNS1=192.168.56.1
                                                                   DEFROUTE=yes
                                                                   IPV4_FAILURE_FATAL=yes
                                                                   IPV6INIT=no
                                                                   NAME="System eth1"
                                                                   HWADDR=08:00:27:60:45:8E
In the above, on the 6.3 side, I find the "PREFIX=24" interesting. When I was going through the installation dialogs and configuring eth1, that was the value that popped into the net mask field before I over-rode it with 255.255.255.0.
---------------------------------- /etc/resolv.conf ---------------------------------- 
-------- OL 5.7 --------                                           -------- OL 6.3 --------
; generated by /sbin/dhclient-script                               ; generated by /sbin/dhclient-script
search <myorganization>.org                                        search <myorganization>.org vbdomain
nameserver 172.nn.n1.3                                             nameserver 172.168.56.1
nameserver 172.nn.n2.1                                             nameserver 172.nn.n2.1
nameserver 172.nn.n1.9                                             nameserver 172.nn.n1.9
As an aside, I just checked ifconfig on both boxes and noticed that the both have the same IP address on eth0 - the NT/dhcp nic. That seems rather odd.
Dude!
If you are using a default sshd server configuration there should be no issue with DNS. I can connect to several versions of Oracle Linux 6 and 5 under VirtualBox and have no login delays. Whether I have /etc/hosts configured and allow reverse DNS for the guest system does not make a difference. I also do not maintain any DNS for any virtual machine on the host system. However, I'm using Mac OS X and not PuTTy.

I think the problem is related to your VirtualBox or TCP/IP setup. Perhaps you have a TCP/IP conflict or setup PORT forwarding on any VirtualBox NAT interface that is making some trouble. I think VirtualBox will not detect a TCP/IP or PORT forwarding conflict when you restore a machine from a snapshot or when a VM is in sleep mode.

I have seen some connection delays in the past when using multiple virtual machines with NAT port forwarding, but I don't recall the details anymore. How does login work if you connect from one machine to another? How does it work if you have only one virtual guest system running. I cannot see how the netmask and ARP protocol can cause a connection delay in your setup.
EdStevens
Dude wrote:
If you are using a default sshd server configuration there should be no issue with DNS. I can connect to several versions of Oracle Linux 6 and 5 under VirtualBox and have no login delays. Whether I have /etc/hosts configured and allow reverse DNS for the guest system does not make a difference. I also do not maintain any DNS for any virtual machine on the host system. However, I'm using Mac OS X and not PuTTy.

I think the problem is related to your VirtualBox or TCP/IP setup. Perhaps you have a TCP/IP conflict or setup PORT forwarding on any VirtualBox NAT interface that is making some trouble. I think VirtualBox will not detect a TCP/IP or PORT forwarding conflict when you restore a machine from a snapshot or when a VM is in sleep mode.

I think I have seen some connection delays when using multiple virtual machines with NAT port forwarding, but I don't recall the details anymore. How does login work if you connect from one machine to another? How does it work if you have only one virtual guest system running. I cannot see how the netmask and ARP protocol can cause a connection delay in your setup.

Edited by: Dude on Apr 30, 2013 5:48 PM
I hadn't tried connecting from another guest vm, but on your suggestion gave that a shot. No joy. ssh from a guest vm running OL 5.6 to my 6.3 guest gives about a 30 second delay. The same as putty from the host desktop without GSSAPI.
Staring at the various config files some more, I found it odd that /etc/resolv.conf had references to my company's dns addresses. Even though my 5.6 vms have the same references, I tried commenting those out on the 6.3 and restarting the network. No change at all.

Really at a loss here. When installing the vm, I made all the same choices as when building a 5.x machine. Any differences in the config files was imposed by the installer. To tie up any loose ends here ..
1) the vbox net adapter is at 192.168.56.1
2) All vms have two NICs. eth0 is configured NAT and dhcp. eth1 is configured host-only and an assigned ip address.
3) The ip address assigned to eth1 is 192.168.56.1nn. For 'nn' I use the last two digits of the server name - vblnxsrv02 that is '02' so I assign an ip of 192.168.56.102.

I just blew away and rebuilt vblnxsrv02, to clear away any vestiges of earlier adjustments. It is now absolutely clean, fresh out of the box. I've not done anything beyond the initial reboot. The key config files look like this: (obvious comments inserted)
[root@vblnxsrv02 ~]# cat /etc/sysconfig/network
NETWORKING=yes
HOSTNAME=vblnxsrv02.vbdomain

[root@vblnxsrv02 ~]# cat /etc/sysconfig/network-scripts/ifcfg-eth0
DEVICE="eth0"
BOOTPROTO="dhcp"
NM_CONTROLLED="yes"                          <- does not exist in 5.6 config
ONBOOT=yes
TYPE="Ethernet"                              <- does not exist in 5.6 config
UUID="d2e4a1d9-f50e-47f7-a8b7-0208dba938af"  <- does not exist in 5.6 config
HWADDR=08:00:27:50:94:AB
DEFROUTE=yes                                 <- does not exist in 5.6 config
PEERDNS=yes                                  <- does not exist in 5.6 config
PEERROUTES=yes                               <- does not exist in 5.6 config
IPV4_FAILURE_FATAL=yes                       <- does not exist in 5.6 config
IPV6INIT=no                                  <- does not exist in 5.6 config
NAME="System eth0"                           <- does not exist in 5.6 config
Also of note, in the eth0 config, the 5.6 server has the following, not found on 6.3

DHCP_HOSTNAME=vblnxsrv01.vbdomain
[root@vblnxsrv02 ~]# cat /etc/sysconfig/network-scripts/ifcfg-eth1
DEVICE="eth1"
BOOTPROTO=none
NM_CONTROLLED="yes"                         <- does not exist in 5.6 config
ONBOOT=yes                                  <- does not exist in 5.6 config
TYPE="Ethernet"                             <- does not exist in 5.6 config
UUID="387da666-e13b-4431-9f13-2645754bf637" <- does not exist in 5.6 config
HWADDR=08:00:27:A2:45:04
IPADDR=192.168.56.102                       <- provided during OS installation
PREFIX=24
GATEWAY=192.168.56.1                        <- provided during OS installation, does not exist in 5.6 config
DNS1=192.168.56.2                           <- provided during OS installation, does not exist in 5.6 config
DOMAIN=vbdomain                             <- provided during OS installation, does not exist in 5.6 config
DEFROUTE=yes
IPV4_FAILURE_FATAL=yes                      <- does not exist in 5.6 config
IPV6INIT=no                                 <- does not exist in 5.6 config
NAME="System eth1"                          <- does not exist in 5.6 config
Also of note, in the eth1 config, the 5.6 server has the following, not found on 6.3

BROADCAST=192.168.56.255
NETMASK=255.255.255.0
NETWORK=192.168.56.0
[root@vblnxsrv02 ~]# cat /etc/resolv.conf
; generated by /sbin/dhclient-script
nameserver 192.168.56.2                     <- provided during OS installation
nameserver 172.nn.n1.1
nameserver 172.nn.n2.9
search vbdomain
[root@vblnxsrv02 ~]#
Also of note in resolv.conf, on the 5.6 machine, the 3 'nameserver' entries all refer to my corporate addresses, none for the local vbox networ, and the 'search' parameter refers to my corporate domain.
Dude!
Perhaps you could try a different approach to diagnose the problem.

A TCP dump is not really useful in this case. Unfortunately my experience with PuTTy logging or debugging is limited. But you could perhaps download the free version of MobaXterm from http://mobaxterm.mobatek.net. It is a single standalone application (15 MB), based on Cygwin, which includes a ssh client, terminal based on putty and a free X Window server, bash, etc.

Then open mobaxterm and type "ssh -vvv root@remote_IP"

You should see at which ssh debug step the login delay happens.
EdStevens
Dude wrote:
Perhaps you could try a different approach to diagnose the problem.

A TCP dump is not really useful in this case. Unfortunately my experience with PuTTy logging or debugging is limited. But you could perhaps download the free version of MobaXterm from http://mobaxterm.mobatek.net. It is a single standalone application (15 MB), based on Cygwin, which includes a ssh client, terminal based on putty and a free X Window server, bash, etc.

Then open mobaxterm and type "ssh -vvv root@remote_IP"

You should see at which ssh debug step the login delay happens.
Here's the last several lines of that output 
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug2: set_newkeys: mode 0
debug1: SSH2_MSG_NEWKEYS received
debug1: Roaming not allowed by server
debug1: SSH2_MSG_SERVICE_REQUEST sent
debug2: service_accept: ssh-userauth
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug2: key: /home/mobaxterm/.ssh/id_rsa (0x0)
debug2: key: /home/mobaxterm/.ssh/id_dsa (0x0)
debug2: key: /home/mobaxterm/.ssh/id_ecdsa (0x0)

This was where the delay occurred.  Up to here it was as fast as the screen could scroll.  Paused at this point.  When it finally picked up, again it was as fast as the screen could scroll.


debug1: Authentications that can continue: publickey,gssapi-keyex,gssapi-with-mic,password
debug3: start over, passed a different list publickey,gssapi-keyex,gssapi-with-mic,password
debug3: preferred hostbased,publickey,password,keyboard-interactive
debug3: authmethod_lookup publickey
debug3: remaining preferred: password,keyboard-interactive
debug3: authmethod_is_enabled publickey
debug1: Next authentication method: publickey
debug1: Trying private key: /home/mobaxterm/.ssh/id_rsa
debug3: no such identity: /home/mobaxterm/.ssh/id_rsa
debug1: Trying private key: /home/mobaxterm/.ssh/id_dsa
debug3: no such identity: /home/mobaxterm/.ssh/id_dsa
debug1: Trying private key: /home/mobaxterm/.ssh/id_ecdsa
debug3: no such identity: /home/mobaxterm/.ssh/id_ecdsa
debug2: we did not send a packet, disable method
debug3: authmethod_lookup password
debug3: remaining preferred: keyboard-interactive
debug3: authmethod_is_enabled password
debug1: Next authentication method: password
root@vblnxsrv02's password:   <<<< finally prompted for pswd
Dude!
Are there any clues in /var/log/secure?

Can you check the result of the following on your server: 
# grep hosts /etc/nsswitch.conf
And change it to "hosts: files dns" if necessary? Then logout and try again.
EdStevens
Dude wrote:
Are there any clues in /var/log/secure?

Can you check the result of the following on your server: 
# grep hosts /etc/nsswitch.conf
And change it to "hosts: files dns" if necessary? Then logout and try again.
Whaddya know! Actually, it was already set to 'files dns', so on a chance I just removed the 'dns'. Immediate problem solved, though I sure don't understand why that was necessary on the 6.3 server and not the 5.6 ones. Now you've done it -- just piled some more on my 'to research' list, to educate myself about nsswitch.conf ...
Dude!
I wouldn't feel comfortable with this solution. The /etc/nsswitch file determines how and in which order your system will obtain name-service information. You will need "dns" in order to browse the internet.

Like I mentioned, non of my virtual machines can do a reverse lookup of my host IP address and I have no login delays. I also think that the ssh server does not attempt a reverse name lookup of the client, unless you configure useDNS in the sshd_config file. The problem might be something else. Can you check or post your /etc/hosts file?
EdStevens
Dude wrote:
I wouldn't feel comfortable with this solution. The /etc/nsswitch file determines how and in which order your system will obtain name-service information. You will need "dns" in order to browse the internet.
I don't feel comfortable with it either. Not because of any technical understanding (obviously something I lack in this area) but simply because it is so far outside of what has been my normal procedure.
Like I mentioned, non of my virtual machines can do a reverse lookup of my host IP address and I have no login delays. I also think that the ssh server does not attempt a reverse name lookup of the client, unless you configure useDNS in the sshd_config file. The problem might be something else. Can you check or post your /etc/hosts file?
Out of the box, the hosts file looks like this: 
[root@vblnxsrv02 ~]# cat /etc/hosts
127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6
again, just on a flyer, I added the 'self reference', which I normally do pretty early in my setup, but a few steps beyond where I am on this one:
192.168.56.102   vblnxsrv02 vblnxsrv02.vbdomain
But that made no difference.

By contrast, a 'straight out of the box' OL 5.6 looks like this 
[root@vblnxsrv03 ~]# cat /etc/hosts
# Do not remove the following line, or various programs
# that require network functionality will fail.
127.0.0.1               vblnxsrv03.vbdomain vblnxsrv03 localhost.localdomain localhost
::1             localhost6.localdomain6 localhost6
Quite a bit of difference in what it assocates with 127.0.0.1 -- especially in putting the actual server name there. In my normal setup, that's something I always change .. so that my final hosts file would look like this: 
# Do not remove the following line, or various programs
# that require network functionality will fail.
127.0.0.1               localhost.localdomain localhost
::1             localhost6.localdomain6 localhost6
192.168.56.103   vblnxsrv03.vbdomain vblnxsrv03
Dude!
What you are seeing about the /etc/hosts file is standard under RHEL 5. The RHEL installation asks for a hostname, but does not know of any TCP/IP address and therefore puts the hostname on the 127.0.0.1 localhost line. This is normally not a problem, although not a common standard and could be RHEL specific.

Btw, the Oracle installer (runInstaller) reads the /etc/hosts file and takes the first entry in the 127.0.0.1 line as the hostname. The hostname becomes part of the directory location for the Oracle dbconsole installation. When people are modifying the /etc/hosts file and remove the hostname from the first position of the 127.0.0.1 line then dbconsole will fail since the directory path does not contain "localhost". The way around it is to set the ORACLE_HOSTNAME variable prior to installing the database. People sometimes reinstall the dbconsole when it fails, but a simple symbolic link would fix the problem.

I don't see anything unusual in your /etc/hosts file. Perhaps you can check if the output of the command "hostname -f" corresponds to your hostname entry in the /etc/hosts file. Can you also check the /etc/ssh/sshd_config file to see the useDNS setting?
EdStevens
Dude wrote:
What you are seeing about the /etc/hosts file is standard under RHEL 5. The RHEL installation asks for a hostname, but does not know of any TCP/IP address and therefore puts the hostname on the 127.0.0.1 localhost line. This is normally not a problem, although not a common standard and could be RHEL specific.

Btw, the Oracle installer (runInstaller) reads the /etc/hosts file and takes the first entry in the 127.0.0.1 line as the hostname. The hostname becomes part of the directory location for the Oracle dbconsole installation. When people are modifying the /etc/hosts file and remove the hostname from the first position of the 127.0.0.1 line then dbconsole will fail since the directory path does not contain "localhost". The way around it is to set the ORACLE_HOSTNAME variable prior to installing the database. People sometimes reinstall the dbconsole when it fails, but a simple symbolic link would fix the problem.
I'd often notice, based on the directory naming of the dbcontrol configuration, that there was some interaction with the hosts file, but had never taken the time to dig into it.
I don't see anything unusual in your /etc/hosts file. Perhaps you can check if the output of the command "hostname -f" corresponds to your hostname entry in the /etc/hosts file. Can you also check the /etc/ssh/sshd_config file to see the useDNS setting?
Very interesting. Before every new test, I restore the vm back to the snapshot taken immediately upon completion of the OS install. That way I'm always working with a know baseline. On this test there was along delay in processing the 'hostname' command. I stacked it with a couple of 'date' commands to show the exact timing involved.
[root@vblnxsrv02 ~]# cat /etc/hosts
127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6
[root@vblnxsrv02 ~]# date;hostname -f;date
Thu May  2 14:00:17 CDT 2013
hostname: Host name lookup failure
Thu May  2 14:01:13 CDT 2013
[root@vblnxsrv02 ~]#
and after looking at the man page for 'hostname' also gave this a shot: 
[root@vblnxsrv02 ~]# date;dnsdomainname ;date
Thu May  2 14:05:16 CDT 2013
dnsdomainname: Host name lookup failure
Thu May  2 14:06:12 CDT 2013
Notice the time delay in both cases .. 56 seconds. About what I was seeing in the logon.

One other variant, with instant response: 
[root@vblnxsrv02 ~]# date;hostname -v;date
Thu May  2 14:14:16 CDT 2013
gethostname()=`vblnxsrv02.vbdomain'
vblnxsrv02.vbdomain
Thu May  2 14:14:16 CDT 2013
[root@vblnxsrv02 ~]#
Edited by: EdStevens on May 2, 2013 2:15 PM
Dude!
Your delay is most likely caused by an invalid entry or unreachable nameserver TCP/IP address in /etc/resolv.conf.

The nameserver TCP/IP address in /etc/resolv.conf is normally your Router/Gateway address, or DNS nameserver if you have one. You can also use some of the public nameserver's if you have Internet, such as 8.8.8.8.

See if this fixes your hostname -f or dnsdomainname delay.

If the commands return "unknown host" make an additional entry in your /etc/hosts file corresponding to the TCP/IP address of your system for reverse name lookup, e.g:

127.0.0.1 vblnxsrv02.vbdomain vblnxsrv02 localhost.localdomain localhost
::1 localhost6.localdomain6 localhost6
192.168.56.5 vblnxsrv02.vbdomain vblnxsrv02
unknown-791521
Ed,
You seem to have sought additional perspectives on this from another forum web site:
http://www.unix.com/red-hat/222789-ssh-logon-delay.html

You linked this thread into there, but perhaps you might have thought to mention that one with a link to it into this thread here.

:)

I've taken care of that for you.
EdStevens
rukbat wrote:
Ed,
You seem to have sought additional perspectives on this from another forum web site:
http://www.unix.com/red-hat/222789-ssh-logon-delay.html

You linked this thread into there, but perhaps you might have thought to mention that one with a link to it into this thread here.

:)

I've taken care of that for you.
You are correct. I wasn't trying to hide anything from anyone here. I just opened that one yesterday, thinking that since it was a totally different forum environment I might get some fresh eyes to look at the issue.
EdStevens
Hmm.. what is the significance of this result of 'route -n' ... I followed it up with 'ifconfig' for reference.

On the 'well behaved' OL 5.6 system: 
[root@vblnxsrv03 ~]# route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
10.0.2.0        0.0.0.0         255.255.255.0   U     0      0        0 eth0
192.168.56.0    0.0.0.0         255.255.255.0   U     0      0        0 eth1
169.254.0.0     0.0.0.0         255.255.0.0     U     0      0        0 eth1
0.0.0.0         10.0.2.2        0.0.0.0         UG    0      0        0 eth0

[root@vblnxsrv03 ~]# ifconfig -a
eth0      Link encap:Ethernet  HWaddr 08:00:27:75:FB:5D
          inet addr:10.0.2.15  Bcast:10.0.2.255  Mask:255.255.255.0
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:9 errors:0 dropped:0 overruns:0 frame:0
          TX packets:23 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:1522 (1.4 KiB)  TX bytes:3267 (3.1 KiB)

eth1      Link encap:Ethernet  HWaddr 08:00:27:94:37:86
          inet addr:192.168.56.103  Bcast:192.168.56.255  Mask:255.255.255.0
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:755 errors:0 dropped:0 overruns:0 frame:0
          TX packets:354 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:71215 (69.5 KiB)  TX bytes:67183 (65.6 KiB)

lo        Link encap:Local Loopback
          inet addr:127.0.0.1  Mask:255.0.0.0
          UP LOOPBACK RUNNING  MTU:16436  Metric:1
          RX packets:34 errors:0 dropped:0 overruns:0 frame:0
          TX packets:34 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:5465 (5.3 KiB)  TX bytes:5465 (5.3 KiB)

[root@vblnxsrv03 ~]#
and on the ill-behaved OL 6.3 
[root@vblnxsrv02 ~]# route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         192.168.56.1    0.0.0.0         UG    0      0        0 eth1
10.0.2.0        0.0.0.0         255.255.255.0   U     0      0        0 eth0
169.254.0.0     0.0.0.0         255.255.0.0     U     1002   0        0 eth0
169.254.0.0     0.0.0.0         255.255.0.0     U     1003   0        0 eth1
192.168.56.0    0.0.0.0         255.255.255.0   U     0      0        0 eth1
[root@vblnxsrv02 ~]# ifconfig -a
eth0      Link encap:Ethernet  HWaddr 08:00:27:50:94:AB
          inet addr:10.0.2.15  Bcast:10.0.2.255  Mask:255.255.255.0
          inet6 addr: fe80::a00:27ff:fe50:94ab/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:2 errors:0 dropped:0 overruns:0 frame:0
          TX packets:11 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:1180 (1.1 KiB)  TX bytes:1346 (1.3 KiB)

eth1      Link encap:Ethernet  HWaddr 08:00:27:A2:45:04
          inet addr:192.168.56.102  Bcast:192.168.56.255  Mask:255.255.255.0
          inet6 addr: fe80::a00:27ff:fea2:4504/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:286 errors:0 dropped:0 overruns:0 frame:0
          TX packets:354 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:25648 (25.0 KiB)  TX bytes:49563 (48.4 KiB)

lo        Link encap:Local Loopback
          inet addr:127.0.0.1  Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING  MTU:16436  Metric:1
          RX packets:7 errors:0 dropped:0 overruns:0 frame:0
          TX packets:7 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:687 (687.0 b)  TX bytes:687 (687.0 b)

[root@vblnxsrv02 ~]#
And this contrast.
From the 5.6 
[root@vblnxsrv03 ~]# netstat -r
Kernel IP routing table
Destination     Gateway         Genmask         Flags   MSS Window  irtt Iface
10.0.2.0        *               255.255.255.0   U         0 0          0 eth0
192.168.56.0    *               255.255.255.0   U         0 0          0 eth1
169.254.0.0     *               255.255.0.0     U         0 0          0 eth1
default         10.0.2.2        0.0.0.0         UG        0 0          0 eth0
[root@vblnxsrv03 ~]# route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
10.0.2.0        0.0.0.0         255.255.255.0   U     0      0        0 eth0
192.168.56.0    0.0.0.0         255.255.255.0   U     0      0        0 eth1
169.254.0.0     0.0.0.0         255.255.0.0     U     0      0        0 eth1
0.0.0.0         10.0.2.2        0.0.0.0         UG    0      0        0 eth0
[root@vblnxsrv03 ~]#
from the 6.3 
[root@vblnxsrv02 ~]# netstat -r
Kernel IP routing table
Destination     Gateway         Genmask         Flags   MSS Window  irtt Iface
default         192.168.56.1    0.0.0.0         UG        0 0          0 eth1
10.0.2.0        *               255.255.255.0   U         0 0          0 eth0
link-local      *               255.255.0.0     U         0 0          0 eth0
link-local      *               255.255.0.0     U         0 0          0 eth1
192.168.56.0    *               255.255.255.0   U         0 0          0 eth1
[root@vblnxsrv02 ~]# route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         192.168.56.1    0.0.0.0         UG    0      0        0 eth1
10.0.2.0        0.0.0.0         255.255.255.0   U     0      0        0 eth0
169.254.0.0     0.0.0.0         255.255.0.0     U     1002   0        0 eth0
169.254.0.0     0.0.0.0         255.255.0.0     U     1003   0        0 eth1
192.168.56.0    0.0.0.0         255.255.255.0   U     0      0        0 eth1
[root@vblnxsrv02 ~]#
Also, while staring at the difference in /etc/resolv.conf, I noticed the first, commented line: 
; generated by /sbin/dhclient-script
So I decided to see what that script does. It was way over my head, but what I immediately noticed was that the 5.6 version is quite a bit different from the 6.3 version. I fact, the 5.6 version was a bit troubling, with this line in the header contents: 
# Updated for Linux 2.[12] by Brian J. Murrell, January 1999.
# No guarantees about this. I'm a novice at the details of Linux
# networking.
Edited by: EdStevens on May 3, 2013 11:47 AM
EdStevens
More follow-up.

On advice from the thread I have at [url http://www.unix.com/red-hat/222789-ssh-logon-delay.html#post302802531]The Unix and Linux Forums I tried two things.

First, I commented out the entire contents of /etc/resolv.conf. That made for an immediate (no restart required) fix, but of course is not persistent since that file gets regenerated at least at ever boot up. But following up from that, I replaced the contents with those from one of my OL5.6 machines. With those settings, the 6.3 was back to the 56-second delay. Very odd.

Second, I modified /etc/ssh/sshd_config, changing 
#UseDNS yes
to 
UseDNS no
That did not provide an immediate fix, but after reboot, (and subsequent resetting of resolv.conf back to its original values) did fix the problem.

So, at this point I have a workable solution, but you know me - I want to know why. I don't like "here, do this - it works". I want to learn something from it. Particularly why it would be different on the 6.3. Since no one seems to be having this issue it must be something I'm doing, but I'm at a loss as to what it would be. I'm thinking the answer lies in the result of the 'route' command I posted earlier, but I don't know enough to understand just what it would be.
Dude!
You DNS delay is very likely the result of a DNS query timeout. This is usually the case if the DNS server is not working properly or not responding.

Have you seen my previous response?

Fixing your /etc/hosts file should fix your "Host name lookup" or Unknown host" failure. It will not fix a DNS issue, but it will fix the DNS timeout when querying the local machine.

I was actually successful to reproduce you DNS delay, but I have not been able to reproduce any ssh login delay regardless of the useDNS setting.

I wonder about the follwowing:

Are you using the VirtualBox host-only adpater with a fixed TCP/IP address and NAT (dhcp) for outgoing connections? I suggest you check the host-only adapter (vboxnet0) network setting in the VirtualBox application preferences.

TCP/IP addresses 192.168.56.100 - 254 in VirtualBox are by default reserved for host-only DHCP. If you configured 192.168.56.103 as a static address you can cause an internal VirtualBox routing issue and a TCP/IP address conflict. I suggest to set the TCP/IP address of the host-only adapter between 192.168.56.100 1 - 99.
Dude!
I was finally able to reproduce the SSH login delay. So by default, the SSH server, unlike I assumed, does indeed perform a reverse DNS lookup. The delay for every invalid or unreachable DNS nameserver in my setup is however only 10 seconds, and there is actually no delay for any unresolved IP as such, which is why I did not notice the problem.

Your 55 seconds delay can perhaps be explained by your DNS server configuration. Setting the useDNS no parameter should resolve you SSH login delay, but will not fix your DNS nameserver configuration.

It is generally a good idea to use any of the reserved domain names for your internal network, like:

.test
.example
.invalid
.localhost

(http://tools.ietf.org/html/rfc2606)

The use of .vboxdomain might be related to your problem.

Any you might still want to check you are using fixed IP numbers according to range defined in your VirtualBox virtual host adapter (DHCP) to avoid TCP/IP conflicts and routing issues.
EdStevens
Dude wrote:
I was finally able to reproduce the SSH login delay. So by default, the SSH server, unlike I assumed, does indeed perform a reverse DNS lookup. The delay for every invalid or unreachable DNS nameserver in my setup is however only 10 seconds, and there is actually no delay for any unresolved IP as such, which is why I did not notice the problem.

Your 55 seconds delay can perhaps be explained by your DNS server configuration. Setting the useDNS no parameter should resolve you SSH login delay, but will not fix your DNS nameserver configuration.

It is generally a good idea to use any of the reserved domain names for your internal network, like:

.test
.example
.invalid
.localhost

(http://tools.ietf.org/html/rfc2606)

The use of .vboxdomain might be related to your problem.

Any you might still want to check you are using fixed IP numbers according to range defined in your VirtualBox virtual host adapter (DHCP) to avoid TCP/IP conflicts and routing issues.
I reassigned the fixed ip address on eth1 (hostonly) to 192.168.56.5 then restarted the machine. No change in behavior.
FWIW, I couldn't find any reference in the VBox docs about reserving a range of addresses for host-only dhcp. Would that even matter, since I am not configuring the host-only adapter (eth1) for dhcp?

Keep in mind that any solution is going to lead to the question "why wasn't that an issue with my OL 5.x machines?" And ideally whatever solution is found - if it is necessarily different from what has been my past practice, will be back-compatible to the 5.x machines.

Just as a quick recap of my configuration ..
VBox adpter on the host os: 
Ethernet adapter VirtualBox Host-Only Network:

   Connection-specific DNS Suffix  . :
   Link-local IPv6 Address . . . . . : fe80::b4f9:a17a:f469:25a4%16
   IPv4 Address. . . . . . . . . . . : 192.168.56.1
   Subnet Mask . . . . . . . . . . . : 255.255.255.0
   Default Gateway . . . . . . . . . :
On all of my vm's (OL 5.x and this OL 6.3),
-- NIC 1 is NAT, goes to eth0, dhcp.
-- NIC 2 is host-only, goes to eth1, is assigned ip address 192.168.56.10n or 192.168.56.20n, where 'n' is simply the one-digit sequence to differentiate.
-- I have always used 'vbdomain' as the default domain for machines running under vbox, 'vmdomain' for machines running under VMworkstation. (I don't use VMware any more, but have retained my naming conventions that made the distinction).

Back on the 'what is different' front, when I create a 5.x machine, nothing in the setup asks me about a dns server. On the 6.3 setup, it asks and I've given it variously nothing, 192.168.56.1, and 192.168.56.2. All really just shots in the dark, as I don't know what I'm doing with that. the ".2" address was a hold-over from VMware, where I found that their adapter apparently ran a dns server at that address (if vmnet8 was at, say 192.168.2.1, dns was at 192.168.2.2). Under VMware, if I didn't get the dns address right, I coulnd't log on at all. With Vbox, it doesn't seem to matter.


One of my 'guiding principles' is that my vm's should be invisible to the outside world -- especially any net administrators. So I was really surprised to see config files that it had somehow found my company's dns servers and configured with their ip addresses.
EdStevens
Ok, I don't know what I did differently, but ..

I just rebuilt the vm from scratch, meticulously documenting every step. First putty session after reboot, and password prompt and following connection were nearly instantaneous. Now I've got to go back and compare all the current config files to what I've previously posted ..

------------

Thought I was on to something with having NOT provided a DNS address at all during setup. Then when comparing notes I noticed the eth0 was configured ONBOOT="no", and as a result, ifconfig showed eth0 wasn't even started. Fixed that, rebooted, and back to the drawing board ...

Edited by: EdStevens on May 6, 2013 3:29 PM
Dude!
The host-only network adapter is configured in the VirtualBox application settings, not the VM settings. You will find it under "Network". There you can add additional virtual adapters and configure DHCP fixed and automatic IP ranges.
EdStevens
Dude wrote:
The host-only network adapter is configured in the VirtualBox application settings, not the VM settings. You will find it under "Network". There you can add additional virtual adapters and configure DHCP fixed and automatic IP ranges.
Yes, the 'hostonly' (likewise NAT) is configured at the VBox console. Double-checking the general (not specific to a given vm) network settings, there is a single host-only net adapter listed, it is at 192.168.56.1, and DHCP server is NOT enabled for it. All that is default. I've never touched it.

The mac address that the vbox manager shows for the NAT adapter matches eth0, which is configured dhcp.
The mac address that the vbox manager shows for the hostonly adapter matches eth1, which is configured with the fixed ip address 192.168.56.102
Dude!
.... there is a single host-only net adapter listed, it is at 192.168.56.1, and DHCP server is NOT enabled for it. All that is default. I've never touched it.
I always thought it was enabled by default. I just installed VirtualBox in Windows Guest machine to check. It is indeed enabled by default:

Server: 192.168.56.100
lower bound: 192.168.56.101
upper bound: 192.168.56.254

If you have it disabled than no problem with your IP address range, otherwise you may run into a problem when configuring a static IP and have any other machine trying to use the same address as a DHCP address.
Amit Shil
How did you replicate this issue?

I have just enabled UseDNS entry in the sshd config and did a restart of my VM and also ensure /etc/resolv.conf has unresolvable entries.. still my SSH works superfast!

sorry, didn't mention VMs havaing OEL 5.6 .. did you replicate teh issue using a 5 variant or 6? i would have thought this should be prevalent irrespective of the OS.!

Regards Amit

Edited by: Amit Shil on 07-May-2013 02:55
Dude!
The /etc/nsswitch.conf file defines the order in which names are resolved. Normally this means first try /etc/hosts than use DNS. If the client IP address can be resolved using the /etc/hosts file on the SSH server, than DNS (/etc/resolve.conf) is not used. This is however usually not the case and the reverse client IP to name mapping will have to be resolved using DNS.

A DNS server provides 3 types of responses.
a) IP or name
b) does not exist
c) no response

A and B do not delay the SSH login. C does because the SSH server is waiting for response. During my tests, the timeout is 10 seconds for each DNS that does not respond.
Amit Shil
Thanks, but how you do simulate a no response? would a entry in the file be enough which if pinged gives a timeout (as there's no internal to external connectivity via the VM)?

Are you suggesting as the VM finds out the DNS servers in my case are non reachable instantly that's why my SSH is working fast?

[root@Gateway ~]# cat /etc/resolv.conf
; generated by /sbin/dhclient-script
nameserver 103.8.46.5
nameserver 121.242.190.210
[root@Gateway ~]# ping 103.8.46.5
connect: Network is unreachable
[root@Gateway ~]# ping 121.242.190.210
connect: Network is unreachable


Regards Amit
Dude!
No, if the DNS servers are not reachable and your client IP is not in /etc/ hosts then there is a delay, provided /etc/nsswitch and useDNS in ssh are configured as default.

Please see the last paragraph of my previous response.
Billy Verreynne
Ed, the sshd daemon is using GSSAPIAuthentication by default - which causes the problem.

Edit +/etc/ssh/sshd_config+ and set the option to no. Then restart the sshd daemon.
EdStevens
Billy Verreynne wrote:
Ed, the sshd daemon is using GSSAPIAuthentication by default - which causes the problem.

Edit +/etc/ssh/sshd_config+ and set the option to no. Then restart the sshd daemon.
No joy. 
[root@vblnxsrv02 ssh]# grep GSSAPI /etc/ssh/sshd_config
# GSSAPI options
GSSAPIAuthentication no
#GSSAPIAuthentication yes
#GSSAPICleanupCredentials yes
GSSAPICleanupCredentials yes
#GSSAPIStrictAcceptorCheck yes
#GSSAPIKeyExchange no
[root@vblnxsrv02 ssh]#
Reboot. Same result.
FWIW, all of my several OL 5.6 systems also have "GSSAPIAuthentication yes" and none exhibit this behavior. I have also disables GSSAPI from my putty connection profiles.
Billy Verreynne
Weird... Have had the exact same symptoms as you describe - ssh connections to RHEL/OL6 systems being very slow during handshaking, before making the eventual connection.

In my case, it was due to GSSAPI authentication. I was pretty sure your problem was a match. +<wry grin>+

Suggest that you isolate potential issues. It could be a ssh client or ssh server induced issue. It could be a network issue. Etc.

What happens if you run ssh on the host, and ssh in as root@localhost? Turn debug tracing on if slow (and please post the results). If not slow, then it is not the server - and potentially either the network, or the client.

&nbsp;
PS. I'm using Kitty at home, with MPutty (Multi Tab Interface for Putty/Kitty). Exact same problem too connecting to a VM in the office, that went away when I disabled GSSAPI in the sshd in the VM.
Dude!
If I remember correctly, the problem with GSSAPI authentication also prints a "access denied" error during the login phase. Putty version 0.61 had this issue.

Regarding this thread, I thought the problem was identified as a DNS issue, and the solution that worked so far was to set useDNS=no in the /etc/ssh/sshd_config file.

Btw, the following is from man resolv.conf
the default timeout is RES_TIMEOUT (currently 5 seconds).
the default number of tries is RES_DFLRETRY (currently 2).
which means a 10 second delay for each unresponsive DNS server in /etc/resolv.conf

I have no idea what happens if the name servers are responding, but not providing a reasonable or ill-formed response. Like I mentioned earlier, maybe using .vboxdomain as the host domain name is not a good idea and it was better to use any of the reserved domain names for private use.

I have not noticed any different behavior between Oracle Linux 5 and 6. Since this is a VirtualBox VM, there could be other issues too. Are there any Firewalls installed on Windows or DNS server side?
Billy Verreynne
Dude wrote:
If I remember correctly, the problem with GSSAPI authentication also prints a "access denied" error during the login phase. Putty version 0.61 had this issue.
Never saw that myself.. but then I do not use Putty and only very seldom use Kitty on Win7 from home. 99% of the time it is Linux OpenSSH clients,
Regarding this thread, I thought the problem was identified as a DNS issue, and the solution that worked so far was to set useDNS=no in the /etc/ssh/sshd_config file.
What struck me is the following that Ed posted: 
debug1: SSH2_MSG_SERVICE_REQUEST sent
debug2: service_accept: ssh-userauth
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug2: key: /home/mobaxterm/.ssh/id_rsa (0x0)
debug2: key: /home/mobaxterm/.ssh/id_dsa (0x0)
debug2: key: /home/mobaxterm/.ssh/id_ecdsa (0x0)
 
This was where the delay occurred...
 
debug1: Authentications that can continue: publickey,gssapi-keyex,gssapi-with-mic,password
debug3: start over, passed a different list publickey,gssapi-keyex,gssapi-with-mic,password
debug3: preferred hostbased,publickey,password,keyboard-interactive
Which means to me that there were some "discussion" between the client and server when handshaking and deciding which authentication issues to use.

Thus my thoughts that it was GSSAPI (as in my case, with all OL6 servers I recently installed).

But it could also be an issue from the client side, with the client going to an enumerated list of auth types it prefers?

Offhand, I do not see why a network resolution issue during this stage would cause an issue - unless both parties agreed on using a 3rd party (e.g. Kerberos server) to authenticate via?
I have no idea what happens if the name servers are responding, but not providing a reasonable or ill-formed response. Like I mentioned earlier, maybe using .vboxdomain as the host domain name is not a good idea and it was better to use any of the reserved domain names for private use.
Never even thought about that as I'm always particular about FQDN and not using defaults. :-)
I have not noticed any different behavior between Oracle Linux 5 and 6. Since this is a VirtualBox VM, there could be other issues too. Are there any Firewalls installed on Windows or DNS server side?
With all the OL5 installations I have done, I have never had a ssh client taking up to 30s to establish a connection and prompt for the password (or using the local rsa public key for authorised hosts auth). The very 1st OL6 server installed (default base install, standard stuff), I ran into ssh delays. And every OL6 server installed since (including a VM installed by colleagues in our labs, which I had to access via the Internet from home using Kitty). In all cases, disabling GSSAPI and bouncing the sshd daemon, worked for me and the delay on connection was no more. Which is why I thought Ed ran into the same issue.

If a ssh root@localhost shows the same delay issues, I would run it in full debug mode - and if that fails to uncover where the delay is, perhaps use strace and dump that to file for investigation.
Dude!
I have too never experienced such a login delay and had my doubts that DNS could be the issue. You may have seen the beginning of this thread where I was providing a link to a previous thread that resolved as a GSSAPI authentication issue.

However, if the authentication process was the culprit, why would disabling DNS on the server side eliminate the delay?

Perhaps the following might finally shed some light on the root cause of the issue:

Don't forget to reset the noDNS parameter in the sshd_config file and /etc/nsswitch.conf to the default

Open a root terminal: 
# service sshd stop
# /usr/sbin/sshd -d -d -d
Then login from the client in another session and watch the server screen for any clues.

If this does not reveal the issue, how about strace: 
# yum install strace
# service sshd stop
# strace /usr/sbin/sshd
To get out of the above mode, simply restart the server or hit Ctr-c followed by service sshd restart

Test:

<pre>
I modified /etc/resolv.conf containing unreachable IP addresses:

# cat /etc/resolv.conf
nameserver 10.0.0.50
nameserver 10.0.0.100

service network restart
service sshd stop
/usr/sbin/sshd -d -d -d

You may see a similar output:

debug2: input_userauth_request: try method publickey
debug1: test whether pkalg/pkblob are acceptable
debug3: mm_key_allowed entering
debug3: mm_request_send entering: type 38
debug3: mm_key_allowed: waiting for MONITOR_ANS_KEYALLOWED
debug3: mm_request_receive_expect entering: type 39
debug3: mm_request_receive entering
debug3: Trying to reverse map address 10.0.0.1.

Delay of 20 seconds
</pre>
Billy Verreynne
Will run some tests when I get a change - pretty busy at the moment.

Remember that SVR4 IPC message queue issue I posted about a while back (and wondering whether Posix IPC message queue would work better)? Reached a point where I had to revisit that code (for running on OL6.4) - and it seems the basic problem have been solved (no more messages dropping). But I ran into a really magnificent error that seems to partially corrupts the code segment when using msgctl(). But except for that, everything else is working fine... Got to love working with kernel APIs. Never know just what to expect. ;-)
EdStevens
Dude wrote:
I have too never experienced such a login delay and had my doubts that DNS could be the issue. You may have seen the beginning of this thread where I was providing a link to a previous thread that resolved as a GSSAPI authentication issue.

However, if the authentication process was the culprit, why would disabling DNS on the server side eliminate the delay?

Perhaps the following might finally shed some light on the root cause of the issue:

Don't forget to reset the noDNS parameter in the sshd_config file and /etc/nsswitch.conf to the default

Open a root terminal: 
# service sshd stop
# /usr/sbin/sshd -d -d -d
Then login from the client in another session and watch the server screen for any clues.
<snip>
trace begins with first message emitted when second session sent connection request 
debug1: userauth-request for user root service ssh-connection method none
debug1: attempt 0 failures 0
debug3: mm_getpwnamallow entering
debug3: mm_request_send entering: type 7
debug3: mm_getpwnamallow: waiting for MONITOR_ANS_PWNAM
debug3: mm_request_receive_expect entering: type 8
debug3: mm_request_receive entering
debug3: monitor_read: checking request 7
debug3: mm_answer_pwnamallow
debug3: Trying to reverse map address 192.168.56.1.


------ delay ---------
------ the address it is trying to reverse map is that of the gateway - the VBox host-only adapter

debug2: parse_server_config: config reprocess config len 563
debug3: mm_answer_pwnamallow: sending MONITOR_ANS_PWNAM: 1
debug3: mm_request_send entering: type 8
debug2: monitor_read: 7 used once, disabling now
debug3: mm_request_receive entering
debug2: input_userauth_request: setting up authctxt for root
debug3: mm_start_pam entering
debug3: mm_request_send entering: type 50
debug3: mm_inform_authserv entering
debug3: mm_request_send entering: type 3
debug3: mm_inform_authrole entering
debug3: mm_request_send entering: type 4
debug2: input_userauth_request: try method none
debug3: Wrote 84 bytes for a total of 2581
debug3: monitor_read: checking request 50
debug1: PAM: initializing for "root"
debug1: PAM: setting PAM_RHOST to "192.168.56.1"
debug1: PAM: setting PAM_TTY to "ssh"
debug2: monitor_read: 50 used once, disabling now
debug3: mm_request_receive entering
debug3: monitor_read: checking request 3
debug3: mm_answer_authserv: service=ssh-connection, style=
debug2: monitor_read: 3 used once, disabling now
debug3: mm_request_receive entering
debug3: monitor_read: checking request 4
debug3: mm_answer_authrole: role=
debug2: monitor_read: 4 used once, disabling now
debug3: mm_request_receive entering

-------  waiting for password ------
EdStevens
Dude wrote:
.... there is a single host-only net adapter listed, it is at 192.168.56.1, and DHCP server is NOT enabled for it. All that is default. I've never touched it.
I always thought it was enabled by default. I just installed VirtualBox in Windows Guest machine to check. It is indeed enabled by default:

Server: 192.168.56.100
lower bound: 192.168.56.101
upper bound: 192.168.56.254

If you have it disabled than no problem with your IP address range, otherwise you may run into a problem when configuring a static IP and have any other machine trying to use the same address as a DHCP address.
Interesting. I was working on my personal laptop last night (all work on this thread has been from my office desktop) and checked that .. exact same version of VBox, and it was as you said. Exact same version of Vbox. I surely don't remember changing that on my office machine, and as little as I monkey with network settings, I'd think I'd remember it. But maybe I'm having another 'senior moment'. ;-)
EdStevens
OK, let's recap and pull together information from several different messages.

Host machine: Win 7 Pro, 64bit
Virtual Box 4.2.10
VBox net adapter: 
Ethernet adapter VirtualBox Host-Only Network:

   Connection-specific DNS Suffix  . :
   Link-local IPv6 Address . . . . . : fe80::b4f9:a17a:f469:25a4%16
   IPv4 Address. . . . . . . . . . . : 192.168.56.1
   Subnet Mask . . . . . . . . . . . : 255.255.255.0
   Default Gateway . . . . . . . . . :
Two virtual machines:
- vblnxsrv56, running OL 5.6 64-bit - normal behavior
- vblnxsrv63, running OL 6.3 64-bit - logon delay of 56 seconds before password prompt

Both vm's configured with two nics:
- nic 1 is NAT
- nic 2 is host-only
Both vm's created with all default values except:
- eth0, mapped to nic 1, configured DHCP
- eth1, mapped to nic 2, fixed IP of 192.168.56.1nn, where 'nn' matches the last two characters of the server name, and also matches the release of OL. (just for ease of keeping track of "who's on first")

Following are contrasting various items of configuration. The value of the shell prompt indicates which server the item is from. I will always list the 5.6 first.

ifconfig - 5.6 
[root@vblnxsrv56 ~]# ifconfig
eth0      Link encap:Ethernet  HWaddr 08:00:27:BD:62:42  
          inet addr:10.0.2.15  Bcast:10.0.2.255  Mask:255.255.255.0
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:16 errors:0 dropped:0 overruns:0 frame:0
          TX packets:30 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:2194 (2.1 KiB)  TX bytes:3696 (3.6 KiB)

eth1      Link encap:Ethernet  HWaddr 08:00:27:C6:D9:6D  
          inet addr:192.168.56.156  Bcast:192.168.56.255  Mask:255.255.255.0
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:1060 errors:0 dropped:0 overruns:0 frame:0
          TX packets:911 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:108054 (105.5 KiB)  TX bytes:147447 (143.9 KiB)

lo        Link encap:Local Loopback  
          inet addr:127.0.0.1  Mask:255.0.0.0
          UP LOOPBACK RUNNING  MTU:16436  Metric:1
          RX packets:34 errors:0 dropped:0 overruns:0 frame:0
          TX packets:34 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:3352 (3.2 KiB)  TX bytes:3352 (3.2 KiB)
ifconfig - 6.3 
[root@vblnxsrv63 ~]# ifconfig
eth0      Link encap:Ethernet  HWaddr 08:00:27:F8:B7:54  
          inet addr:10.0.2.15  Bcast:10.0.2.255  Mask:255.255.255.0
          inet6 addr: fe80::a00:27ff:fef8:b754/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:1 errors:0 dropped:0 overruns:0 frame:0
          TX packets:11 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:590 (590.0 b)  TX bytes:1074 (1.0 KiB)

eth1      Link encap:Ethernet  HWaddr 08:00:27:A0:25:6F  
          inet addr:192.168.56.163  Bcast:192.168.56.255  Mask:255.255.255.0
          inet6 addr: fe80::a00:27ff:fea0:256f/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:641 errors:0 dropped:0 overruns:0 frame:0
          TX packets:1022 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:55066 (53.7 KiB)  TX bytes:128717 (125.7 KiB)

lo        Link encap:Local Loopback  
          inet addr:127.0.0.1  Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING  MTU:16436  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:0 (0.0 b)  TX bytes:0 (0.0 b)
network-scripts/ifcfg-eth0 - 5.6 
[root@vblnxsrv56 ~]# cat /etc/sysconfig/network-scripts/ifcfg-eth0
# Intel Corporation 82540EM Gigabit Ethernet Controller
DEVICE=eth0
BOOTPROTO=dhcp
HWADDR=08:00:27:BD:62:42
ONBOOT=yes
DHCP_HOSTNAME=vblnxsrv56.vbdomain
network-scripts/ifcfg-eth0 - 6.3 
[root@vblnxsrv63 ~]# cat /etc/sysconfig/network-scripts/ifcfg-eth0
DEVICE="eth0"
BOOTPROTO="dhcp"
NM_CONTROLLED="yes"
ONBOOT=yes
TYPE="Ethernet"
UUID="3760d917-8121-429a-a16e-7ec1c7d9b60e"
HWADDR=08:00:27:F8:B7:54
DEFROUTE=yes
PEERDNS=yes
PEERROUTES=yes
IPV4_FAILURE_FATAL=yes
IPV6INIT=no
NAME="System eth0"
network-scripts/ifcfg-eth1 - 5.6 
[root@vblnxsrv56 ~]# cat /etc/sysconfig/network-scripts/ifcfg-eth1
# Intel Corporation 82540EM Gigabit Ethernet Controller
DEVICE=eth1
BOOTPROTO=static
BROADCAST=192.168.56.255
HWADDR=08:00:27:C6:D9:6D
IPADDR=192.168.56.156
NETMASK=255.255.255.0
NETWORK=192.168.56.0
ONBOOT=yes
network-scripts/ifcfg-eth1 - 6.3 
[root@vblnxsrv63 ~]# cat /etc/sysconfig/network-scripts/ifcfg-eth1
DEVICE="eth1"
BOOTPROTO=none
NM_CONTROLLED="yes"
ONBOOT=yes
TYPE="Ethernet"
UUID="787482d6-a888-47d2-a899-0a49379bff25"
HWADDR=08:00:27:A0:25:6F
IPADDR=192.168.56.163
PREFIX=24
GATEWAY=192.168.56.1
DEFROUTE=yes
IPV4_FAILURE_FATAL=yes
IPV6INIT=no
NAME="System eth1"
/etc/sysconfig/network - 5.6 
[root@vblnxsrv56 ~]# cat /etc/sysconfig/network
NETWORKING=yes
NETWORKING_IPV6=no
HOSTNAME=vblnxsrv56.vbdomain
/etc/sysconfig/network - 6.3 
[root@vblnxsrv63 ~]# cat /etc/sysconfig/network
NETWORKING=yes
HOSTNAME=vblnxsrv63.vbdomain
/etc/hosts - 5.6 
[root@vblnxsrv56 ~]# cat /etc/hosts
# Do not remove the following line, or various programs
# that require network functionality will fail.
127.0.0.1vblnxsrv56.vbdomain vblnxsrv56 localhost.localdomain localhost
::1localhost6.localdomain6 localhost6
/etc/hosts - 6.3 
[root@vblnxsrv63 ~]# cat /etc/hosts
127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6
/etc/resolv.conf - 5.6
note: 'acmewidgets.org' and the asterisks mask company identifiable values. I was surprised they even showed up on the vm.
notice that the 5.6 version has only the reference to the company domain, while the 6.3 adds the reference to my vb 'vbdomain'. 
[root@vblnxsrv56 ~]# cat /etc/resolv.conf
; generated by /sbin/dhclient-script
search acmewidgets.org
nameserver ***.***.***.3
nameserver ***.***.***.1
nameserver ***.***.***.9
/etc/resolv.conf - 6.3 
[root@vblnxsrv63 ~]# cat /etc/resolv.conf
; generated by /sbin/dhclient-script
search acmewidgets.org vbdomain
nameserver ***.***.***.3
nameserver ***.***.***.1
nameserver ***.***.***.9
routing table - 5.6 
[root@vblnxsrv56 ~]# route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
10.0.2.0        0.0.0.0         255.255.255.0   U     0      0        0 eth0
192.168.56.0    0.0.0.0         255.255.255.0   U     0      0        0 eth1
169.254.0.0     0.0.0.0         255.255.0.0     U     0      0        0 eth1
0.0.0.0         10.0.2.2        0.0.0.0         UG    0      0        0 eth0
routing table - 6.3
Note the duplication of 169.254.0.0 across both nics 
[root@vblnxsrv63 ~]# route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         192.168.56.1    0.0.0.0         UG    0      0        0 eth1
10.0.2.0        0.0.0.0         255.255.255.0   U     0      0        0 eth0
169.254.0.0     0.0.0.0         255.255.0.0     U     1002   0        0 eth0
169.254.0.0     0.0.0.0         255.255.0.0     U     1003   0        0 eth1
192.168.56.0    0.0.0.0         255.255.255.0   U     0      0        0 eth1
/etc/ssh/sshd_config - 5.6 
[root@vblnxsrv56 ~]# grep DNS /etc/ssh/sshd_config
#UseDNS yes
/etc/ssh/sshd_config - 6.3 
[root@vblnxsrv63 ~]# grep DNS /etc/ssh/sshd_config
#UseDNS yes
/etc/nsswitch.conf - 5.6 
[root@vblnxsrv56 ~]# grep hosts /etc/nsswitch.conf
#hosts:     db files nisplus nis dns
hosts:      files dns
/etc/nsswitch.conf - 6.3 
[root@vblnxsrv63 ~]# grep hosts /etc/nsswitch.conf
#hosts:     db files nisplus nis dns
hosts:      files dns
timing of domain resolution - 5.6 
[root@vblnxsrv56 ~]# date;dnsdomainname ;date
Tue May  7 15:40:49 CDT 2013
vbdomain
Tue May  7 15:40:49 CDT 2013
timing of domain resolution - 6.3 
[root@vblnxsrv63 ~]# date;dnsdomainname ;date
Tue May  7 15:40:47 CDT 2013
dnsdomainname: Host name lookup failure
Tue May  7 15:41:43 CDT 2013
timing of hostname resolution - 5.6 
[root@vblnxsrv56 ~]# date;hostname -v;date
Tue May  7 15:41:52 CDT 2013
gethostname()=`vblnxsrv56.vbdomain'
vblnxsrv56.vbdomain
Tue May  7 15:41:52 CDT 2013
timing of hostname resolution - 6.3 
[root@vblnxsrv63 ~]# date;hostname -v;date
Tue May  7 15:41:49 CDT 2013
gethostname()=`vblnxsrv63.vbdomain'
vblnxsrv63.vbdomain
Tue May  7 15:41:49 CDT 2013
And as discovered, the "fix" seems to be to modify /etc/ssh/sshd_config, to enable "UseDNS no". So I guess the question comes down to why would that be necessary on OL 6.3 (at least on my OL 6.3) and not on my several OL 5.x vm's?

Edited by: EdStevens on May 8, 2013 8:59 AM
Dude!
Answer
debug3: Trying to reverse map address 192.168.56.1.
This is actually normal since the VirtualBox host-only adapter acts on behalf of the client connection. Similarly, with the NAT adapter, the host adapter acts on behalf of the virtual machine.

One thing to remember is that the host-only interface is no good for outgoing connections. Traffic should go out on your VirtualBox NAT interface.

Looking at your previous output:
and on the ill-behaved OL 6.3

[root@vblnxsrv02 ~]# route -n
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
0.0.0.0 192.168.56.1 0.0.0.0 UG 0 0 0 eth1
I see that you have set 192.168.56.1 as the TCP/IP gateway. This is apparently not the case on your OL 5.6 machine.

So I'm wondering, did you perhaps specify 192.168.56.1 as default gateway in the TCP/IP settings of your OL 6.3 virtual machine? You should not specify any gateway on that 192.168.56.102/103 interface. Just leave it blank. Compare it with the setting of your OL 5.6 machine.

At least this would explain why your OL 6.3 machine cannot reach any DNS name server. It is not going to resolve 192.168.56.1 anyway, but at least it won't delay because it cannot connect to any DNS server.

And btw, 192.168.56.100 - 254 are usually reserved for host-only DHCP. You could try to enable the DHCP server on the host-only adapter in your TCP/IP settings on the VM and enable the DCHP server on the host-only adapter of VirtualBox preferences. You will also see it does not set a gateway.
Marked as Answer by EdStevens · Sep 27 2020
Amit Shil
guess it would be worth seeing the routing table of the host machine as well, 192.168.56.0 is on the link local for the host so there should be no need to enter any gateway entry on the guest (either of the machines) .

what about the duplicate entries of 169 network on the 6.3 machine, why the duplicacy - any comments on that? surely Ed wouldn't have touched those entries ..

As there's no such n/w present, it should be better to cleanse the routing table and remove this altogether

Regards Amit
EdStevens
Dude wrote:
debug3: Trying to reverse map address 192.168.56.1.
This is actually normal since the VirtualBox host-only adapter acts on behalf of the client connection. Similarly, with the NAT adapter, the host adapter acts on behalf of the virtual machine.

One thing to remember is that the host-only interface is no good for outgoing connections. Traffic should go out on your VirtualBox NAT interface.

Looking at your previous output:
and on the ill-behaved OL 6.3

[root@vblnxsrv02 ~]# route -n
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
0.0.0.0 192.168.56.1 0.0.0.0 UG 0 0 0 eth1
I see that you have set 192.168.56.1 as the TCP/IP gateway. This is apparently not the case on your OL 5.6 machine.

So I'm wondering, did you perhaps specify 192.168.56.1 as default gateway in the TCP/IP settings of your OL 6.3 virtual machine? You should not specify any gateway on that 192.168.56.102/103 interface. Just leave it blank. Compare it with the setting of your OL 5.6 machine.

At least this would explain why your OL 6.3 machine cannot reach any DNS name server. It is not going to resolve 192.168.56.1 anyway, but at least it won't delay because it cannot connect to any DNS server.
Houston, the Eagle has landed!

As I built these two machines - vblnxsrv56 and vblnxsrv63 - I took screenshots of every step, to give me a complete record of choices I made and possibly map those to the final result. Going back through those, I see that the 5.6 setup never asked for a gateway. When the 6.3 did - simply as a property of configuring eth1 - I never gave it a thought and just entered the address of the VBox hostonly adapter. Probably a mental holdover from the way I configured under VMware.

So I blew away the 6.3 machine and rebuilt it from scratch, this time simply leaving the gateway field blank when configuring eth1. Also left the DNS field blank, which I seem to recall at some point in the distant past -- possibly under VMware -- not being able to do that.

And btw, 192.168.56.100 - 254 are usually reserved for host-only DHCP. You could try to enable the DHCP server on the host-only adapter in your TCP/IP settings on the VM and enable the DCHP server on the host-only adapter of VirtualBox preferences. You will also see it does not set a gateway.
Yes. As mentioned earlier, on this (my office desktop) I have the host-only configured with the DHCP server DIS-abled. Yet my personal laptop, with the exact same version of VBox has it enabled and configured exactly as you say. I just installed that one a couple of weeks ago (new laptop) so I know it is at default settings. The evidence surely says I changed it on the office desktop, but I sure don't remember it.


BTW, if you are curious as to why I'm configuring two NICs the way I am, take a look at [url https://forums.virtualbox.org/viewtopic.php?f=35&t=42385]this. I just this morning found that as I was starting to go through the VBox forum for some ideas. That's a bit encouraging because I had long ago come to that same configuration independently, but for the same reasons cited there. In retrospect, the only reason I have a NAT adapter at all is to allow me to download the packages needed for VBox guest additions and oracle-validated, while not requiring my own IP address. Once I'm past that point, the only communications these vm's should see is between each other and the host os. I guess I could set up my own local yum repository, but that would require its own care and feeding, and in regards to the VBox guest additions that solution would be a bit of a chicken-and-egg situation.

In any event it looks like we can finally put this to bed. I certainly appreciate all the effort you've expended.
1 - 45
Locked Post
New comments cannot be posted to this locked post.

Post Details

Locked on Nov 18 2003
Added on Oct 20 2003
#odp-net
1 comment
230 views