This discussion is archived
1 2 Previous Next 28 Replies Latest reply: Sep 27, 2012 4:48 PM by alan.pae Go to original post RSS
  • 15. Re: I can ping my server but nothing else works
    user171873 Explorer
    Currently Being Moderated
    Did you check the log files for any additional information? For example, the report you've posted shows that the log file for ssh is:

    /var/svc/log/network-ssh:default.log

    You can check the end of that file to see what happened on the last boot. The log files are timestamped so you can distinguish current from older entries. Unfortunately, the information in the log files isn't always terribly useful but sometimes it may help.

    Are you sure your network configuration is correct? The network appears to be up but are routes set up properly so you can get outside your local net, etc.? Can you resolve host names?

    There is also a fault management system. You can use "fmdump" to print out system faults which may have been logged. You can check to see if that reports anything. And there's also /var/adm/messages which could have some diagnostic information.
  • 16. Re: I can ping my server but nothing else works
    962245 Newbie
    Currently Being Moderated
    Hi Dawgman,
    I'm not sure what to check for netmask and nsswitch? Can you tell me what I'm looking for? I've included them. Here.

    My system was working and appearently needed a reboot boot and now it's not. Shouldn't my config files be perfectly fine?
    The problems started around Sept 12. It looks my ssh libcrypto.so.0.9.7 file has disappeared.. can anyone tell me why?


    ssh Log:
    [ Mar  4 22:03:54 Executing start method ("/lib/svc/method/sshd start") ]
    [ Mar  4 22:03:54 Method "start" exited with status 0 ]
    [ Mar 18 22:35:50 Stopping because service disabled. ]
    [ Mar 18 22:35:50 Executing stop method (:kill) ]
    [ Mar 19 01:53:25 Executing start method ("/lib/svc/method/sshd start") ]
    [ Mar 19 01:53:26 Method "start" exited with status 0 ]
    [ Sep 13 16:16:24 Executing start method ("/lib/svc/method/sshd start") ]
    ld.so.1: sshd: fatal: libcrypto.so.0.9.7: open failed: No such file or directory
    Killed
    [ Sep 13 16:16:24 Method "start" exited with status 137 ]
    [ Sep 13 16:16:24 Executing start method ("/lib/svc/method/sshd start") ]
    ld.so.1: sshd: fatal: libcrypto.so.0.9.7: open failed: No such file or directory
    Killed
    [ Sep 13 16:16:24 Method "start" exited with status 137 ]
    [ Sep 13 16:16:24 Executing start method ("/lib/svc/method/sshd start") ]
    ld.so.1: sshd: fatal: libcrypto.so.0.9.7: open failed: No such file or directory
    Killed
    [ Sep 13 16:16:24 Method "start" exited with status 137 ]
    [ Sep 13 16:30:48 Executing start method ("/lib/svc/method/sshd start") ]
    ld.so.1: sshd: fatal: libcrypto.so.0.9.7: open failed: No such file or directory
    Killed
    [ Sep 13 16:30:48 Method "start" exited with status 137 ]
    [ Sep 13 16:30:48 Executing start method ("/lib/svc/method/sshd start") ]
    ld.so.1: sshd: fatal: libcrypto.so.0.9.7: open failed: No such file or directory
    Killed
    [ Sep 13 16:30:48 Method "start" exited with status 137 ]
    [ Sep 13 16:30:48 Executing start method ("/lib/svc/method/sshd start") ]
    ld.so.1: sshd: fatal: libcrypto.so.0.9.7: open failed: No such file or directory
    Killed
    [ Sep 13 16:30:48 Method "start" exited with status 137 ]
    [ Sep 14 12:03:52 Leaving maintenance because disable requested. ]
    [ Sep 14 12:03:52 Disabled. ]
    [ Sep 14 12:10:11 Enabled. ]
    [ Sep 14 12:10:11 Executing start method ("/lib/svc/method/sshd start") ]
    ld.so.1: sshd: fatal: libcrypto.so.0.9.7: open failed: No such file or directory
    Killed
    [ Sep 14 12:10:11 Method "start" exited with status 137 ]
    [ Sep 14 13:16:48 Leaving maintenance because clear requested. ]
    [ Sep 14 13:16:48 Enabled. ]
    [ Sep 14 13:16:48 Executing start method ("/lib/svc/method/sshd start") ]
    ld.so.1: sshd: fatal: libcrypto.so.0.9.7: open failed: No such file or directory
    Killed
    [ Sep 14 13:16:48 Method "start" exited with status 137 ]
    [ Sep 14 13:20:57 Leaving maintenance because disable requested. ]
    [ Sep 14 13:20:57 Disabled. ]
    [ Sep 14 13:21:11 Enabled. ]
    [ Sep 14 13:21:11 Executing start method ("/lib/svc/method/sshd start") ]
    ld.so.1: sshd: fatal: libcrypto.so.0.9.7: open failed: No such file or directory
    Killed
    [ Sep 14 13:21:11 Method "start" exited with status 137 ]
    [ Sep 14 13:31:07 Executing start method ("/lib/svc/method/sshd start") ]
    ld.so.1: sshd: fatal: libcrypto.so.0.9.7: open failed: No such file or directory
    Killed
    [ Sep 14 13:31:07 Method "start" exited with status 137 ]
    [ Sep 14 13:31:07 Executing start method ("/lib/svc/method/sshd start") ]
    ld.so.1: sshd: fatal: libcrypto.so.0.9.7: open failed: No such file or directory
    Killed
    [ Sep 14 13:31:07 Method "start" exited with status 137 ]
    [ Sep 14 13:31:07 Executing start method ("/lib/svc/method/sshd start") ]
    ld.so.1: sshd: fatal: libcrypto.so.0.9.7: open failed: No such file or directory
    Killed
    [ Sep 14 13:31:07 Method "start" exited with status 137 ]
    [ Sep 14 16:09:22 Executing start method ("/lib/svc/method/sshd start") ]
    ld.so.1: sshd: fatal: libcrypto.so.0.9.7: open failed: No such file or directory
    Killed
    [ Sep 14 16:09:22 Method "start" exited with status 137 ]
    [ Sep 14 16:09:22 Executing start method ("/lib/svc/method/sshd start") ]
    ld.so.1: sshd: fatal: libcrypto.so.0.9.7: open failed: No such file or directory
    Killed
    [ Sep 14 16:09:22 Method "start" exited with status 137 ]
    [ Sep 14 16:09:22 Executing start method ("/lib/svc/method/sshd start") ]
    ld.so.1: sshd: fatal: libcrypto.so.0.9.7: open failed: No such file or directory
    Killed
    [ Sep 14 16:09:22 Method "start" exited with status 137 ]
    [ Sep 25 16:26:16 Leaving maintenance because clear requested. ]
    [ Sep 25 16:26:16 Enabled. ]
    [ Sep 25 16:26:16 Executing start method ("/lib/svc/method/sshd start") ]
    ld.so.1: sshd: fatal: libcrypto.so.0.9.7: open failed: No such file or directory
    Killed
    [ Sep 25 16:26:16 Method "start" exited with status 137 ]
    [ Sep 26 10:38:50 Leaving maintenance because clear requested. ]
    [ Sep 26 10:38:50 Enabled. ]
    [ Sep 26 10:38:50 Executing start method ("/lib/svc/method/sshd start") ]
    ld.so.1: sshd: fatal: libcrypto.so.0.9.7: open failed: No such file or directory
    Killed
    [ Sep 26 10:38:50 Method "start" exited with status 137 ]
    [ Sep 26 10:46:15 Leaving maintenance because clear requested. ]
    [ Sep 26 10:46:15 Enabled. ]
    [ Sep 26 10:46:15 Executing start method ("/lib/svc/method/sshd start") ]
    ld.so.1: sshd: fatal: libcrypto.so.0.9.7: open failed: No such file or directory
    Killed
    [ Sep 26 10:46:15 Method "start" exited with status 137 ]


    netmask:
    #
    # The netmasks file associates Internet Protocol (IP) address
    # masks with IP network numbers.
    #
    #      network-number     netmask
    #
    # The term network-number refers to a number obtained from the Internet Network
    # Information Center.
    #
    # Both the network-number and the netmasks are specified in
    # "decimal dot" notation, e.g:
    #
    #           XXX.XX.0.0 255.255.255.0
    #
    XX.XX.XXX.XX 255.255.255.192



    nsswitch.config:
    #
    # /etc/nsswitch.dns:
    #
    # An example file that could be copied over to /etc/nsswitch.conf; it uses
    # DNS for hosts lookups, otherwise it does not use any other naming service.
    #
    # "hosts:" and "services:" in this file are used only if the
    # /etc/netconfig file has a "-" for nametoaddr_libs of "inet" transports.

    # DNS service expects that an instance of svc:/network/dns/client be
    # enabled and online.

    passwd: files
    group: files

    # You must also set up the /etc/resolv.conf file for DNS name
    # server lookup. See resolv.conf(4).
    hosts: files dns

    # Note that IPv4 addresses are searched for in all of the ipnodes databases
    # before searching the hosts databases.
    ipnodes: files dns

    networks: files
    protocols: files
    rpc: files
    ethers: files
    netmasks: files
    bootparams: files
    publickey: files
    # At present there isn't a 'files' backend for netgroup; the system will
    # figure it out pretty quickly, and won't use netgroups at all.
    netgroup: files
    automount: files
    aliases: files
    services: files
    printers:     user files

    auth_attr: files
    prof_attr: files
    project: files
  • 17. Re: I can ping my server but nothing else works
    alan.pae Journeyer
    Currently Being Moderated
    ld.so.1: sshd: fatal: libcrypto.so.0.9.7: open failed: No such file or directory
    So the other ones that failed also have similar log files. Can you peek at them as well.

    libcrypto being missing sure is interesting. How are you backing up the machine?

    alan
  • 18. Re: I can ping my server but nothing else works
    962245 Newbie
    Currently Being Moderated
    "Are you sure your network configuration is correct? The network appears to be up but are routes set up properly so you can get outside your local net, etc.? Can you resolve host names?"
    No I'm not sure it is setup correct. However I didn't change anything so I assumed it would still be work.


    Backups.. there are no backups created by the previous admin (as far as I know.. I have no documentation from the previous admin aside from passwords). I had created a mysql back and also I have all of our site's html code backedup before the system crashed but the server itself is not backedup. I use amazon's EC2 for all the other company servers because of the ease of backup.

    Here is the smtp log (i'm only including the section from where it was successful to where it failed):
    "
    [ Mar  4 20:47:55 Method "stop" exited with status 0 ]
    [ Mar  4 22:03:55 Executing start method
    ("/lib/svc/method/smtp-sendmail start") ]
    [ Mar  4 22:03:55 Method "start" exited with status 0 ]
    [ Mar 18 22:35:50 Stopping because service disabled. ]
    [ Mar 18 22:35:50 Executing stop method
    ("/lib/svc/method/smtp-sendmail stop 57") ]
    [ Mar 18 22:35:55 Method "stop" exited with status 0 ]
    [ Mar 19 01:53:26 Executing start method
    ("/lib/svc/method/smtp-sendmail start") ]
    [ Mar 19 01:53:26 Method "start" exited with status 0 ]
    [ Sep 13 16:16:24 Executing start method
    ("/lib/svc/method/smtp-sendmail start") ]
    [ Sep 13 16:16:24 Method "start" exited with status 0 ]
    ld.so.1: sendmail: fatal: libssl.so.0.9.7: open failed: No such file
    or directory
    ld.so.1: sendmail: fatal: libssl.so.0.9.7: open failed: No such file
    or directory
    [ Sep 13 16:16:24 Stopping because all processes in service exited. ]
    [ Sep 13 16:16:24 Executing stop method
    ("/lib/svc/method/smtp-sendmail stop 70") ]
    [ Sep 13 16:16:25 Method "stop" exited with status 0 ]
    [ Sep 13 16:16:25 Executing start method
    ("/lib/svc/method/smtp-sendmail start") ]
    ld.so.1: sendmail: fatal: libssl.so.0.9.7: open failed: No such file
    or directory
    ld.so.1: sendmail: fatal: libssl.so.0.9.7: open failed: No such file
    or directory"



    pksvr log:
    "Mar 4 22:03:44 Executing start method ("/usr/bin/pkgadm sync") ]
    [ Mar  4 22:03:45 Method "start" exited with status 0 ]
    [ Mar 18 22:35:51 Stopping because service disabled. ]
    [ Mar 18 22:35:51 Executing stop method ("/usr/bin/pkgadm sync") ]
    [ Mar 18 22:35:52 Method "stop" exited with status 0 ]
    [ Mar 19 01:53:17 Method "start" exited with status 0 ]
    [ Sep 13 16:16:09 Executing start method ("/usr/bin/pkgadm sync") ]
    ld.so.1: pkgadm: fatal: libssl.so.0.9.7: open failed: No such file or directory
    [ Sep 13 16:16:09 Method "start" failed due to signal KILL ]
    [ Sep 13 16:16:09 Executing start method ("/usr/bin/pkgadm sync") ]
    ld.so.1: pkgadm: fatal: libssl.so.0.9.7: open failed: No such file or directory
    [ Sep 13 16:16:09 Method "start" failed due to signal KILL ]
    [ Sep 13 16:16:09 Executing start method ("/usr/bin/pkgadm sync") ]
    ld.so.1: pkgadm: fatal: libssl.so.0.9.7: open failed: No such file or directory"



    It seems like all are relying on this missing file.
  • 19. Re: I can ping my server but nothing else works
    alan.pae Journeyer
    Currently Being Moderated
    Ok, so your openssl package appears to be majorly foobared.

    So if it's a Solaris 10 box we can get that re-installed if you have the CD media and if it's Solaris 11 you can just use the pkg command to fix it.

    So we need to know the version number which you can get by posting the contents of /etc/release.

    And in case it's Solaris 10, do you have the installation media? Do you have a support contract in case it needs to be patched? Do you know if it ever was patched?

    alan
  • 20. Re: I can ping my server but nothing else works
    962245 Newbie
    Currently Being Moderated
    I did a search for libcrypto. This is my result:

    # find / -name libcrypto
    /export/home/admin/openssl-1.0.0d/libcrypto.a
    /export/home/admin/openssl-1.0.0d/libcrypto.pc
    /usr/lib/amd64/libcryptoutil.so
    /usr/lib/amd64/libcryptoutil.so.1
    /usr/lib/libcryptoutil.so
    /usr/lib/libcryptoutil.so.1
    /usr/local/ssl/lib/pkgconfig/libcrypto.pc
    /usr/local/ssl/lib/libcrypto.a
  • 21. Re: I can ping my server but nothing else works
    964934 Newbie
    Currently Being Moderated
    Could you please post the output of #svcs -l svc:/network/ssh:default output? I just wanted to check which are the depended services are not started because of which ssh service is in maintenance mode.
  • 22. Re: I can ping my server but nothing else works
    962245 Newbie
    Currently Being Moderated
    Sure, 961931.

    # svcs -l svc:/network/ssh:default
    fmri svc:/network/ssh:default
    name SSH server
    enabled true
    state maintenance
    next_state none
    state_time Wed Sep 26 10:46:15 2012
    logfile /var/svc/log/network-ssh:default.log
    restarter svc:/system/svc/restarter:default
    contract_id
    dependency require_all/none svc:/system/filesystem/local (online)
    dependency optional_all/none svc:/system/filesystem/autofs (online)
    dependency require_all/none svc:/network/loopback (online)
    dependency require_all/none svc:/network/physical (online)
    dependency require_all/none svc:/system/cryptosvc (online)
    dependency require_all/none svc:/system/utmp (online)
    dependency require_all/restart file://localhost/etc/ssh/sshd_config (online)
  • 23. Re: I can ping my server but nothing else works
    962245 Newbie
    Currently Being Moderated
    Sure thing 961931.


    # svcs -l svc:/network/ssh:default
    fmri svc:/network/ssh:default
    name SSH server
    enabled true
    state maintenance
    next_state none
    state_time Wed Sep 26 10:46:15 2012
    logfile /var/svc/log/network-ssh:default.log
    restarter svc:/system/svc/restarter:default
    contract_id
    dependency require_all/none svc:/system/filesystem/local (online)
    dependency optional_all/none svc:/system/filesystem/autofs (online)
    dependency require_all/none svc:/network/loopback (online)
    dependency require_all/none svc:/network/physical (online)
    dependency require_all/none svc:/system/cryptosvc (online)
    dependency require_all/none svc:/system/utmp (online)
    dependency require_all/restart file://localhost/etc/ssh/sshd_config (online)
  • 24. Re: I can ping my server but nothing else works
    962245 Newbie
    Currently Being Moderated
    The first line of my /etc/release is:

    Solaris 10 6/06 s10x_u2wos_09a X86

    I have no idea if the server has been patched. Is there a way to check.

    I don't believe we have a support contract.

    I checked our store of OS install disks and I see a disk "11/06 Solaris 10 Operating System" disk. I don't believe we have ever owned any other Solaris 10 servers so this should be it. However based on the release file I should have expected 6/06, right?

    Also, our server is collocated about 3 hours away. When I log into KVM I am able to access the internet so I suppose I could extract and email the needed file to myself.
  • 25. Re: I can ping my server but nothing else works
    alan.pae Journeyer
    Currently Being Moderated
    Ok, so it looks like someone downloaded, compiled and then installed openssl. Now you have a dilemma.

    You don't appear to have any installation media. Bad.

    You don't appear to have any current machine backups. Bad.

    Somehow your openssl files aren't where they need to be. Bad.

    So you could in theory just run gmake install again and hopefully everything is fine.

    But then you have the dilemma of how did they disappear in the first place?

    I would re-install and not trust the box unless I knew how the files went missing in the first place. Which you can't apparently.

    So you could just re-install openssl and hope for the best which I wouldn't do.

    alan
  • 26. Re: I can ping my server but nothing else works
    alan.pae Journeyer
    Currently Being Moderated
    Yes, you are on Solaris 10 2006 update 6
  • 27. Re: I can ping my server but nothing else works
    962245 Newbie
    Currently Being Moderated
    Thanks Alan.pae.
    So it seems I'm in as a bad a place as I expected to be.

    Is it likely that the deletion of libcrypto was the result of a successful intrusion attempt? I assume this is why you say not to "trust the box".

    The file /export/home/admin openssl-1.0.0.0d was created Mar '11 so I'm guessing that wasn't via an attacker.
    So my options are gmake install openssl again? or not use the box?
  • 28. Re: I can ping my server but nothing else works
    alan.pae Journeyer
    Currently Being Moderated
    Is it likely that the deletion of libcrypto was the result of a successful intrusion attempt? I assume this is why you say not to "trust the box".
    Don't know. It's like when your Mom used to say, "Who took this" and no one answered. I'll assume that someone had access to your box on March 11 such as a contractor or how else did openssl even end up in that directory in the first place?
    The file /export/home/admin openssl-1.0.0.0d was created Mar '11 so I'm guessing that wasn't via an attacker.
    So my options are gmake install openssl again? or not use the box?
    I'm assuming that it was compiled and just left in that directory. If it was then gmake install would do it. If not then you're going to need to download if from somewhere and then move it onto the box, compile it, and then install it.

    Internet facing boxes with "unknowns" I don't like. You may. That's why you get paid, "The big bucks." :-)

    alan
1 2 Previous Next

Legend

  • Correct Answers - 10 points
  • Helpful Answers - 5 points