1 2 Previous Next 28 Replies Latest reply: Sep 27, 2012 6:48 PM by alan.pae Go to original post RSS
      • 15. Re: I can ping my server but nothing else works
        User171873
        Did you check the log files for any additional information? For example, the report you've posted shows that the log file for ssh is:

        /var/svc/log/network-ssh:default.log

        You can check the end of that file to see what happened on the last boot. The log files are timestamped so you can distinguish current from older entries. Unfortunately, the information in the log files isn't always terribly useful but sometimes it may help.

        Are you sure your network configuration is correct? The network appears to be up but are routes set up properly so you can get outside your local net, etc.? Can you resolve host names?

        There is also a fault management system. You can use "fmdump" to print out system faults which may have been logged. You can check to see if that reports anything. And there's also /var/adm/messages which could have some diagnostic information.
        • 16. Re: I can ping my server but nothing else works
          962245
          Hi Dawgman,
          I'm not sure what to check for netmask and nsswitch? Can you tell me what I'm looking for? I've included them. Here.

          My system was working and appearently needed a reboot boot and now it's not. Shouldn't my config files be perfectly fine?
          The problems started around Sept 12. It looks my ssh libcrypto.so.0.9.7 file has disappeared.. can anyone tell me why?


          ssh Log:
          [ Mar  4 22:03:54 Executing start method ("/lib/svc/method/sshd start") ]
          [ Mar  4 22:03:54 Method "start" exited with status 0 ]
          [ Mar 18 22:35:50 Stopping because service disabled. ]
          [ Mar 18 22:35:50 Executing stop method (:kill) ]
          [ Mar 19 01:53:25 Executing start method ("/lib/svc/method/sshd start") ]
          [ Mar 19 01:53:26 Method "start" exited with status 0 ]
          [ Sep 13 16:16:24 Executing start method ("/lib/svc/method/sshd start") ]
          ld.so.1: sshd: fatal: libcrypto.so.0.9.7: open failed: No such file or directory
          Killed
          [ Sep 13 16:16:24 Method "start" exited with status 137 ]
          [ Sep 13 16:16:24 Executing start method ("/lib/svc/method/sshd start") ]
          ld.so.1: sshd: fatal: libcrypto.so.0.9.7: open failed: No such file or directory
          Killed
          [ Sep 13 16:16:24 Method "start" exited with status 137 ]
          [ Sep 13 16:16:24 Executing start method ("/lib/svc/method/sshd start") ]
          ld.so.1: sshd: fatal: libcrypto.so.0.9.7: open failed: No such file or directory
          Killed
          [ Sep 13 16:16:24 Method "start" exited with status 137 ]
          [ Sep 13 16:30:48 Executing start method ("/lib/svc/method/sshd start") ]
          ld.so.1: sshd: fatal: libcrypto.so.0.9.7: open failed: No such file or directory
          Killed
          [ Sep 13 16:30:48 Method "start" exited with status 137 ]
          [ Sep 13 16:30:48 Executing start method ("/lib/svc/method/sshd start") ]
          ld.so.1: sshd: fatal: libcrypto.so.0.9.7: open failed: No such file or directory
          Killed
          [ Sep 13 16:30:48 Method "start" exited with status 137 ]
          [ Sep 13 16:30:48 Executing start method ("/lib/svc/method/sshd start") ]
          ld.so.1: sshd: fatal: libcrypto.so.0.9.7: open failed: No such file or directory
          Killed
          [ Sep 13 16:30:48 Method "start" exited with status 137 ]
          [ Sep 14 12:03:52 Leaving maintenance because disable requested. ]
          [ Sep 14 12:03:52 Disabled. ]
          [ Sep 14 12:10:11 Enabled. ]
          [ Sep 14 12:10:11 Executing start method ("/lib/svc/method/sshd start") ]
          ld.so.1: sshd: fatal: libcrypto.so.0.9.7: open failed: No such file or directory
          Killed
          [ Sep 14 12:10:11 Method "start" exited with status 137 ]
          [ Sep 14 13:16:48 Leaving maintenance because clear requested. ]
          [ Sep 14 13:16:48 Enabled. ]
          [ Sep 14 13:16:48 Executing start method ("/lib/svc/method/sshd start") ]
          ld.so.1: sshd: fatal: libcrypto.so.0.9.7: open failed: No such file or directory
          Killed
          [ Sep 14 13:16:48 Method "start" exited with status 137 ]
          [ Sep 14 13:20:57 Leaving maintenance because disable requested. ]
          [ Sep 14 13:20:57 Disabled. ]
          [ Sep 14 13:21:11 Enabled. ]
          [ Sep 14 13:21:11 Executing start method ("/lib/svc/method/sshd start") ]
          ld.so.1: sshd: fatal: libcrypto.so.0.9.7: open failed: No such file or directory
          Killed
          [ Sep 14 13:21:11 Method "start" exited with status 137 ]
          [ Sep 14 13:31:07 Executing start method ("/lib/svc/method/sshd start") ]
          ld.so.1: sshd: fatal: libcrypto.so.0.9.7: open failed: No such file or directory
          Killed
          [ Sep 14 13:31:07 Method "start" exited with status 137 ]
          [ Sep 14 13:31:07 Executing start method ("/lib/svc/method/sshd start") ]
          ld.so.1: sshd: fatal: libcrypto.so.0.9.7: open failed: No such file or directory
          Killed
          [ Sep 14 13:31:07 Method "start" exited with status 137 ]
          [ Sep 14 13:31:07 Executing start method ("/lib/svc/method/sshd start") ]
          ld.so.1: sshd: fatal: libcrypto.so.0.9.7: open failed: No such file or directory
          Killed
          [ Sep 14 13:31:07 Method "start" exited with status 137 ]
          [ Sep 14 16:09:22 Executing start method ("/lib/svc/method/sshd start") ]
          ld.so.1: sshd: fatal: libcrypto.so.0.9.7: open failed: No such file or directory
          Killed
          [ Sep 14 16:09:22 Method "start" exited with status 137 ]
          [ Sep 14 16:09:22 Executing start method ("/lib/svc/method/sshd start") ]
          ld.so.1: sshd: fatal: libcrypto.so.0.9.7: open failed: No such file or directory
          Killed
          [ Sep 14 16:09:22 Method "start" exited with status 137 ]
          [ Sep 14 16:09:22 Executing start method ("/lib/svc/method/sshd start") ]
          ld.so.1: sshd: fatal: libcrypto.so.0.9.7: open failed: No such file or directory
          Killed
          [ Sep 14 16:09:22 Method "start" exited with status 137 ]
          [ Sep 25 16:26:16 Leaving maintenance because clear requested. ]
          [ Sep 25 16:26:16 Enabled. ]
          [ Sep 25 16:26:16 Executing start method ("/lib/svc/method/sshd start") ]
          ld.so.1: sshd: fatal: libcrypto.so.0.9.7: open failed: No such file or directory
          Killed
          [ Sep 25 16:26:16 Method "start" exited with status 137 ]
          [ Sep 26 10:38:50 Leaving maintenance because clear requested. ]
          [ Sep 26 10:38:50 Enabled. ]
          [ Sep 26 10:38:50 Executing start method ("/lib/svc/method/sshd start") ]
          ld.so.1: sshd: fatal: libcrypto.so.0.9.7: open failed: No such file or directory
          Killed
          [ Sep 26 10:38:50 Method "start" exited with status 137 ]
          [ Sep 26 10:46:15 Leaving maintenance because clear requested. ]
          [ Sep 26 10:46:15 Enabled. ]
          [ Sep 26 10:46:15 Executing start method ("/lib/svc/method/sshd start") ]
          ld.so.1: sshd: fatal: libcrypto.so.0.9.7: open failed: No such file or directory
          Killed
          [ Sep 26 10:46:15 Method "start" exited with status 137 ]


          netmask:
          #
          # The netmasks file associates Internet Protocol (IP) address
          # masks with IP network numbers.
          #
          #      network-number     netmask
          #
          # The term network-number refers to a number obtained from the Internet Network
          # Information Center.
          #
          # Both the network-number and the netmasks are specified in
          # "decimal dot" notation, e.g:
          #
          #           XXX.XX.0.0 255.255.255.0
          #
          XX.XX.XXX.XX 255.255.255.192



          nsswitch.config:
          #
          # /etc/nsswitch.dns:
          #
          # An example file that could be copied over to /etc/nsswitch.conf; it uses
          # DNS for hosts lookups, otherwise it does not use any other naming service.
          #
          # "hosts:" and "services:" in this file are used only if the
          # /etc/netconfig file has a "-" for nametoaddr_libs of "inet" transports.

          # DNS service expects that an instance of svc:/network/dns/client be
          # enabled and online.

          passwd: files
          group: files

          # You must also set up the /etc/resolv.conf file for DNS name
          # server lookup. See resolv.conf(4).
          hosts: files dns

          # Note that IPv4 addresses are searched for in all of the ipnodes databases
          # before searching the hosts databases.
          ipnodes: files dns

          networks: files
          protocols: files
          rpc: files
          ethers: files
          netmasks: files
          bootparams: files
          publickey: files
          # At present there isn't a 'files' backend for netgroup; the system will
          # figure it out pretty quickly, and won't use netgroups at all.
          netgroup: files
          automount: files
          aliases: files
          services: files
          printers:     user files

          auth_attr: files
          prof_attr: files
          project: files
          • 17. Re: I can ping my server but nothing else works
            alan.pae
            ld.so.1: sshd: fatal: libcrypto.so.0.9.7: open failed: No such file or directory
            So the other ones that failed also have similar log files. Can you peek at them as well.

            libcrypto being missing sure is interesting. How are you backing up the machine?

            alan
            • 18. Re: I can ping my server but nothing else works
              962245
              "Are you sure your network configuration is correct? The network appears to be up but are routes set up properly so you can get outside your local net, etc.? Can you resolve host names?"
              No I'm not sure it is setup correct. However I didn't change anything so I assumed it would still be work.


              Backups.. there are no backups created by the previous admin (as far as I know.. I have no documentation from the previous admin aside from passwords). I had created a mysql back and also I have all of our site's html code backedup before the system crashed but the server itself is not backedup. I use amazon's EC2 for all the other company servers because of the ease of backup.

              Here is the smtp log (i'm only including the section from where it was successful to where it failed):
              "
              [ Mar  4 20:47:55 Method "stop" exited with status 0 ]
              [ Mar  4 22:03:55 Executing start method
              ("/lib/svc/method/smtp-sendmail start") ]
              [ Mar  4 22:03:55 Method "start" exited with status 0 ]
              [ Mar 18 22:35:50 Stopping because service disabled. ]
              [ Mar 18 22:35:50 Executing stop method
              ("/lib/svc/method/smtp-sendmail stop 57") ]
              [ Mar 18 22:35:55 Method "stop" exited with status 0 ]
              [ Mar 19 01:53:26 Executing start method
              ("/lib/svc/method/smtp-sendmail start") ]
              [ Mar 19 01:53:26 Method "start" exited with status 0 ]
              [ Sep 13 16:16:24 Executing start method
              ("/lib/svc/method/smtp-sendmail start") ]
              [ Sep 13 16:16:24 Method "start" exited with status 0 ]
              ld.so.1: sendmail: fatal: libssl.so.0.9.7: open failed: No such file
              or directory
              ld.so.1: sendmail: fatal: libssl.so.0.9.7: open failed: No such file
              or directory
              [ Sep 13 16:16:24 Stopping because all processes in service exited. ]
              [ Sep 13 16:16:24 Executing stop method
              ("/lib/svc/method/smtp-sendmail stop 70") ]
              [ Sep 13 16:16:25 Method "stop" exited with status 0 ]
              [ Sep 13 16:16:25 Executing start method
              ("/lib/svc/method/smtp-sendmail start") ]
              ld.so.1: sendmail: fatal: libssl.so.0.9.7: open failed: No such file
              or directory
              ld.so.1: sendmail: fatal: libssl.so.0.9.7: open failed: No such file
              or directory"



              pksvr log:
              "Mar 4 22:03:44 Executing start method ("/usr/bin/pkgadm sync") ]
              [ Mar  4 22:03:45 Method "start" exited with status 0 ]
              [ Mar 18 22:35:51 Stopping because service disabled. ]
              [ Mar 18 22:35:51 Executing stop method ("/usr/bin/pkgadm sync") ]
              [ Mar 18 22:35:52 Method "stop" exited with status 0 ]
              [ Mar 19 01:53:17 Method "start" exited with status 0 ]
              [ Sep 13 16:16:09 Executing start method ("/usr/bin/pkgadm sync") ]
              ld.so.1: pkgadm: fatal: libssl.so.0.9.7: open failed: No such file or directory
              [ Sep 13 16:16:09 Method "start" failed due to signal KILL ]
              [ Sep 13 16:16:09 Executing start method ("/usr/bin/pkgadm sync") ]
              ld.so.1: pkgadm: fatal: libssl.so.0.9.7: open failed: No such file or directory
              [ Sep 13 16:16:09 Method "start" failed due to signal KILL ]
              [ Sep 13 16:16:09 Executing start method ("/usr/bin/pkgadm sync") ]
              ld.so.1: pkgadm: fatal: libssl.so.0.9.7: open failed: No such file or directory"



              It seems like all are relying on this missing file.
              • 19. Re: I can ping my server but nothing else works
                alan.pae
                Ok, so your openssl package appears to be majorly foobared.

                So if it's a Solaris 10 box we can get that re-installed if you have the CD media and if it's Solaris 11 you can just use the pkg command to fix it.

                So we need to know the version number which you can get by posting the contents of /etc/release.

                And in case it's Solaris 10, do you have the installation media? Do you have a support contract in case it needs to be patched? Do you know if it ever was patched?

                alan
                • 20. Re: I can ping my server but nothing else works
                  962245
                  I did a search for libcrypto. This is my result:

                  # find / -name libcrypto
                  /export/home/admin/openssl-1.0.0d/libcrypto.a
                  /export/home/admin/openssl-1.0.0d/libcrypto.pc
                  /usr/lib/amd64/libcryptoutil.so
                  /usr/lib/amd64/libcryptoutil.so.1
                  /usr/lib/libcryptoutil.so
                  /usr/lib/libcryptoutil.so.1
                  /usr/local/ssl/lib/pkgconfig/libcrypto.pc
                  /usr/local/ssl/lib/libcrypto.a
                  • 21. Re: I can ping my server but nothing else works
                    964934
                    Could you please post the output of #svcs -l svc:/network/ssh:default output? I just wanted to check which are the depended services are not started because of which ssh service is in maintenance mode.
                    • 22. Re: I can ping my server but nothing else works
                      962245
                      Sure, 961931.

                      # svcs -l svc:/network/ssh:default
                      fmri svc:/network/ssh:default
                      name SSH server
                      enabled true
                      state maintenance
                      next_state none
                      state_time Wed Sep 26 10:46:15 2012
                      logfile /var/svc/log/network-ssh:default.log
                      restarter svc:/system/svc/restarter:default
                      contract_id
                      dependency require_all/none svc:/system/filesystem/local (online)
                      dependency optional_all/none svc:/system/filesystem/autofs (online)
                      dependency require_all/none svc:/network/loopback (online)
                      dependency require_all/none svc:/network/physical (online)
                      dependency require_all/none svc:/system/cryptosvc (online)
                      dependency require_all/none svc:/system/utmp (online)
                      dependency require_all/restart file://localhost/etc/ssh/sshd_config (online)
                      • 23. Re: I can ping my server but nothing else works
                        962245
                        Sure thing 961931.


                        # svcs -l svc:/network/ssh:default
                        fmri svc:/network/ssh:default
                        name SSH server
                        enabled true
                        state maintenance
                        next_state none
                        state_time Wed Sep 26 10:46:15 2012
                        logfile /var/svc/log/network-ssh:default.log
                        restarter svc:/system/svc/restarter:default
                        contract_id
                        dependency require_all/none svc:/system/filesystem/local (online)
                        dependency optional_all/none svc:/system/filesystem/autofs (online)
                        dependency require_all/none svc:/network/loopback (online)
                        dependency require_all/none svc:/network/physical (online)
                        dependency require_all/none svc:/system/cryptosvc (online)
                        dependency require_all/none svc:/system/utmp (online)
                        dependency require_all/restart file://localhost/etc/ssh/sshd_config (online)
                        • 24. Re: I can ping my server but nothing else works
                          962245
                          The first line of my /etc/release is:

                          Solaris 10 6/06 s10x_u2wos_09a X86

                          I have no idea if the server has been patched. Is there a way to check.

                          I don't believe we have a support contract.

                          I checked our store of OS install disks and I see a disk "11/06 Solaris 10 Operating System" disk. I don't believe we have ever owned any other Solaris 10 servers so this should be it. However based on the release file I should have expected 6/06, right?

                          Also, our server is collocated about 3 hours away. When I log into KVM I am able to access the internet so I suppose I could extract and email the needed file to myself.
                          • 25. Re: I can ping my server but nothing else works
                            alan.pae
                            Ok, so it looks like someone downloaded, compiled and then installed openssl. Now you have a dilemma.

                            You don't appear to have any installation media. Bad.

                            You don't appear to have any current machine backups. Bad.

                            Somehow your openssl files aren't where they need to be. Bad.

                            So you could in theory just run gmake install again and hopefully everything is fine.

                            But then you have the dilemma of how did they disappear in the first place?

                            I would re-install and not trust the box unless I knew how the files went missing in the first place. Which you can't apparently.

                            So you could just re-install openssl and hope for the best which I wouldn't do.

                            alan
                            • 26. Re: I can ping my server but nothing else works
                              alan.pae
                              Yes, you are on Solaris 10 2006 update 6
                              • 27. Re: I can ping my server but nothing else works
                                962245
                                Thanks Alan.pae.
                                So it seems I'm in as a bad a place as I expected to be.

                                Is it likely that the deletion of libcrypto was the result of a successful intrusion attempt? I assume this is why you say not to "trust the box".

                                The file /export/home/admin openssl-1.0.0.0d was created Mar '11 so I'm guessing that wasn't via an attacker.
                                So my options are gmake install openssl again? or not use the box?
                                • 28. Re: I can ping my server but nothing else works
                                  alan.pae
                                  Is it likely that the deletion of libcrypto was the result of a successful intrusion attempt? I assume this is why you say not to "trust the box".
                                  Don't know. It's like when your Mom used to say, "Who took this" and no one answered. I'll assume that someone had access to your box on March 11 such as a contractor or how else did openssl even end up in that directory in the first place?
                                  The file /export/home/admin openssl-1.0.0.0d was created Mar '11 so I'm guessing that wasn't via an attacker.
                                  So my options are gmake install openssl again? or not use the box?
                                  I'm assuming that it was compiled and just left in that directory. If it was then gmake install would do it. If not then you're going to need to download if from somewhere and then move it onto the box, compile it, and then install it.

                                  Internet facing boxes with "unknowns" I don't like. You may. That's why you get paid, "The big bucks." :-)

                                  alan
                                  1 2 Previous Next