1 2 Previous Next 28 Replies Latest reply: Sep 27, 2012 6:48 PM by alan.pae RSS

    I can ping my server but nothing else works

    962245
      Hello Forum,
      I'm a Linux newbie. I'm a new IT manager at my small business. I've inherited a Solaris server with absolutely no documentation. This server hosts a couple of websites and also a mysql database which is accessed via a java program. Everything was running smoothly for about a year after the last employee and we didn't use the server much since it was doing everything we wanted.

      However about yesterday I realized I had some problems:
      1)The web sites it hosted are no longer accessible ( the addresses "Failed to connect to server" ).
      2)There is a java program which connects to a mysql database hosted on the server and when I attempt the run the software I get a "Network Connection Failure: Connection refused" error.
      3) I am able to ping the machine XX.XX.XXX.XX successfully
      4) I am unable to ssh into the machine "ssh: connect to host XX.XX.XXX.XX port 22: Connection refused"

      I had no idea what was wrong. I was hoping a simple reboot was necessary (I've never rebooted a Solaris machine before). I fear my machine was hacked and taken over.

      I requested remote access via a KVM from our co-location hosts. When I logged in I saw some text. So it seems for some reason the server tried to reboot on it's own and there were some issues. I think the import parts are that it said "the following files in / differ from the boot archive: /etc/rtc_config...." and "system/boot-archive:default failed fatally". I took a screen shot if that is necessary for you to view but it doesn't seem that there is a way to attach or embed files in this forum.

      After reading here: http://docs.oracle.com/cd/E19082-01/819-2379/ggngq/index.html
      I used the command: # svcadm clear system/boot-archive

      After that command my system booted up into a gui. I thought my webserver would be running and mysql would be running since I assumed those programs start up on boot.
      However after boot I was still only able to ping.. everything was still failing in the way it failed before.

      Please help me.
      Any ideas on: What cause my problem? Was cleaning the boot-archive the correct move? Where to go from here?

      Edited by: SolarisNewbie on Sep 27, 2012 1:35 PM
        • 1. Re: I can ping my server but nothing else works
          alan.pae
          Depends on how fast you need it up again.

          We can do it the slow way with things such as ps -ef |grep name_of_some_server_process which will take days or you should probably just call someone and then work out some sort of support contract.

          alan
          • 2. Re: I can ping my server but nothing else works
            962245
            Thanks for the response. I have a bs in comp sci so I want/can to learn how to do whatever needs to be done. But I'm confused.. do Solaris Servers normally randomly crash and take days to recover? I have no idea why such an epic failure occurred and I could use some guidance.

            I figure I'll start with getting ssh working then and work up to msyql, tomcat, apache. Thanks for your tip.. I think can get ssh working at least.
            • 3. Re: I can ping my server but nothing else works
              alan.pae
              SolarisNewbie wrote:
              Thanks for the response. I have a bs in comp sci so I want/can to learn how to do whatever needs to be done. But I'm confused.. do Solaris Servers normally randomly crash and take days to recover? I have no idea why such an epic failure occurred and I could use some guidance.
              Depends on how it's done. If we sit here and say try this and see what happens and then wait for the reply it could. If someone knows what they're doing and are sitting directly in front of it then it should be easy enough.
              I figure I'll start with getting ssh working then and work up to msyql, tomcat, apache. Thanks for your tip.. I think can get ssh working at least.
              So the first question would be:

              What is the output of:

              pgrep apache
              pgrep mysql
              pgrep tomcat

              and let us know what each line says.

              alan
              • 4. Re: I can ping my server but nothing else works
                800381
                alan.pae wrote:
                SolarisNewbie wrote:
                Thanks for the response. I have a bs in comp sci so I want/can to learn how to do whatever needs to be done. But I'm confused.. do Solaris Servers normally randomly crash and take days to recover? I have no idea why such an epic failure occurred and I could use some guidance.
                Depends on how it's done. If we sit here and say try this and see what happens and then wait for the reply it could. If someone knows what they're doing and are sitting directly in front of it then it should be easy enough.
                I figure I'll start with getting ssh working then and work up to msyql, tomcat, apache. Thanks for your tip.. I think can get ssh working at least.
                So the first question would be:

                What is the output of:

                pgrep apache
                pgrep mysql
                pgrep tomcat

                and let us know what each line says.

                alan
                Shouldn't that be "pgrep httpd" for Apache?
                • 5. Re: I can ping my server but nothing else works
                  alan.pae
                  I stand corrected.

                  alan
                  • 6. Re: I can ping my server but nothing else works
                    User171873
                    Whenever I boot a Solaris system, the first thing I do after it comes up it run "svcs -xv" to see if everything started properly. If the system is fully up it should print out nothing. If you run it while the system is still coming up you'll see lines indicating that some services aren't yet up. If there is a problem, you'll usually see services in the "maintenance". You can then look at the end of the log file for the services in the maintenance state and maybe get an indication of what is wrong. Clearing up the services in the maintenance state (if any) would be my first step in trying to resolve the problem.

                    Once you've resolved the problem with a service use "svcadm clear <name>" where <name> is the service name to attempt to clear the problem. If you resolved the issue, the service should come up cleanly; otherwise, it will stay in the maintenance state. I should mention that sometimes a service fails due to a transient problem and all you need to do is run "svcadm clear <name>". If you have multiple services in the maintenance state, I'd go from the top to the bottom.
                    • 7. Re: I can ping my server but nothing else works
                      muvvas
                      have you verified the respected application ports are listening

                      netstat -an
                      lsof -i
                      • 8. Re: I can ping my server but nothing else works
                        962245
                        Thank you alan.pae.
                        I'm sorry I couldn't respond sooner but I had other deadlines which took precedence. I appreciate your help.


                        The output of those commands:
                        pgrep apache
                        pgrep mysql
                        pgrep tomcat
                        pgrep httpd

                        are all blank. I hit enter, see no output and the I am returned to the prompt.
                        I.e.
                        "
                        #pgrep apache
                        #pgrep mysql
                        #
                        "


                        I'm assuming that means that those processes never started so you can't see anything about them.
                        • 9. Re: I can ping my server but nothing else works
                          962245
                          822432, Thank you for the excellent advice.

                          I mentioned that I couldn't even boot the system so I ran:
                          svcadm clear system/boot-archive
                          I'm not sure if this removed setting for how everything should boot but this seems to be the case.


                          Here is the result of svcs -xv:
                          # svcs -xv
                          svc:/application/print/server:default (LP print server)
                          State: disabled since Fri Sep 14 16:09:08 2012
                          Reason: Disabled by an administrator.
                          See: http://sun.com/msg/SMF-8000-05
                          See: man -M /usr/share/man -s 1M lpsched
                          Impact: 1 dependent service is not running:
                          svc:/application/print/ipp-listener:default

                          svc:/application/database/mysql5:default (MySQL RDBMS 5.1.31)
                          State: disabled since Fri Sep 14 16:09:09 2012
                          Reason: Disabled by an administrator.
                          See: http://sun.com/msg/SMF-8000-05
                          See: man -M /usr/local/mysql5/man -s 1 mysql
                          Impact: 1 dependent service is not running:
                          svc:/application/web/tomcat6:default

                          svc:/network/rpc/smserver:default (removable media management)
                          State: disabled since Fri Sep 14 16:09:22 2012
                          Reason: Disabled by an administrator.
                          See: http://sun.com/msg/SMF-8000-05
                          See: man -M /usr/share/man -s 1M rpc.smserverd
                          Impact: 1 dependent service is not running:
                          svc:/system/filesystem/volfs:default

                          svc:/system/pkgserv:default (Flush package command database to disk (see pkgadm(
                          1m)).)
                          State: maintenance since Fri Sep 14 16:09:14 2012
                          Reason: Start method failed repeatedly, last died on Killed (9).
                          See: http://sun.com/msg/SMF-8000-KS
                          See: /etc/svc/volatile/system-pkgserv:default.log
                          See: /var/svc/log/system-pkgserv:default.log
                          Impact: This service is not running.

                          svc:/network/ssh:default (SSH server)
                          State: maintenance since Fri Sep 14 16:09:22 2012
                          Reason: Start method failed repeatedly, last exited with status 137.
                          See: http://sun.com/msg/SMF-8000-KS
                          See: man -M /usr/share/man -s 1M sshd
                          See: /var/svc/log/network-ssh:default.log
                          Impact: This service is not running.

                          svc:/network/smtp:sendmail (sendmail SMTP mail transfer agent)
                          State: maintenance since Fri Sep 14 16:09:23 2012
                          Reason: Restarting too quickly.
                          See: http://sun.com/msg/SMF-8000-L5
                          See: man -M /usr/share/man -s 1M sendmail
                          See: /var/svc/log/network-smtp:sendmail.log
                          Impact: This service is not running."


                          I will attempt to start svcadm clearing these functions tomorrow. However I just tried my ssh and it still failed in the same manner.
                          • 10. Re: I can ping my server but nothing else works
                            alan.pae
                            I'm assuming that means that those processes never started so you can't see anything about them.
                            Yup. :-)
                            • 11. Re: I can ping my server but nothing else works
                              alan.pae
                              SolarisNewbie wrote:
                              822432, Thank you for the excellent advice.

                              I mentioned that I couldn't even boot the system so I ran:
                              svcadm clear system/boot-archive
                              I'm not sure if this removed setting for how everything should boot but this seems to be the case.
                              No, your fine.
                              svc:/application/print/server:default (LP print server)
                              svc:/application/database/mysql5:default (MySQL RDBMS 5.1.31)
                              svc:/network/rpc/smserver:default (removable media management)
                              svc:/system/pkgserv:default (Flush package command database to disk (see pkgadm(
                              1m)).)
                              svc:/network/ssh:default (SSH server)
                              svc:/network/smtp:sendmail (sendmail SMTP mail transfer agent)
                              I will attempt to start svcadm clearing these functions tomorrow. However I just tried my ssh and it still failed in the same manner.
                              So Printer, MySql, SMSERVER, pkgserv, SSH and Sendmail are all faulted.

                              So after using

                              svcadm clear FMRI

                              Try

                              pgrep httpd

                              and see if that one returns anything.

                              Run:

                              ifconfig -a

                              and post the output. Sanitize the IP Addresses if you want to.

                              SSH isn't running yet so you'll have to keep using the KVM device for now.

                              alan
                              • 12. Re: I can ping my server but nothing else works
                                962245
                                I tried to svcadm clearthe processes and some weren’t in the maintaince state. After that I pgrep’ed httpd,msyql and tomcat. All of them aren’t running. Returns nothing, simply goes back to the prompt.

                                Below I’ve posted the ifconfig –a and i’ve posted what svc –xv returns after trying the clears.

                                Is it possible that I need to clear some dependent process or manually restart some of these?

                                Thanks for the continued assistance.


                                # ifconfig -a
                                lo0: flags=2001000849<UP,LOOPBACK,RUNNING,MULTICAST,IPv4,VIRTUAL> mtu
                                8232 index 1
                                inet 127.0.0.1 netmask ff000000
                                e1000g0: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index 2
                                inet XX.XX.XX.XX netmask ff000000 broadcast XX.255.255.255
                                ether 0:14:4f:27:38:a8


                                # svcs -xv
                                svc:/application/print/server:default (LP print server)
                                State: disabled since Fri Sep 14 16:09:08 2012
                                Reason: Disabled by an administrator.
                                See: http://sun.com/msg/SMF-8000-05
                                See: man -M /usr/share/man -s 1M lpsched
                                Impact: 1 dependent service is not running:
                                svc:/application/print/ipp-listener:default

                                svc:/application/database/mysql5:default (MySQL RDBMS 5.1.31)
                                State: disabled since Fri Sep 14 16:09:09 2012
                                Reason: Disabled by an administrator.
                                See: http://sun.com/msg/SMF-8000-05
                                See: man -M /usr/local/mysql5/man -s 1 mysql
                                Impact: 1 dependent service is not running:
                                svc:/application/web/tomcat6:default

                                svc:/network/rpc/smserver:default (removable media management)
                                State: disabled since Fri Sep 14 16:09:22 2012
                                Reason: Disabled by an administrator.
                                See: http://sun.com/msg/SMF-8000-05
                                See: man -M /usr/share/man -s 1M rpc.smserverd
                                Impact: 1 dependent service is not running:
                                svc:/system/filesystem/volfs:default

                                svc:/system/pkgserv:default (Flush package command database to disk
                                (see pkgadm(1m)).)
                                State: maintenance since Wed Sep 26 10:46:07 2012
                                Reason: Start method failed repeatedly, last died on Killed (9).
                                See: http://sun.com/msg/SMF-8000-KS
                                See: /etc/svc/volatile/system-pkgserv:default.log
                                See: /var/svc/log/system-pkgserv:default.log
                                Impact: This service is not running.

                                svc:/network/ssh:default (SSH server)
                                State: maintenance since Wed Sep 26 10:46:15 2012
                                Reason: Start method failed repeatedly, last exited with status 137.
                                See: http://sun.com/msg/SMF-8000-KS
                                See: man -M /usr/share/man -s 1M sshd
                                See: /var/svc/log/network-ssh:default.log
                                Impact: This service is not running.

                                svc:/network/smtp:sendmail (sendmail SMTP mail transfer agent)
                                State: maintenance since Wed Sep 26 10:46:25 2012
                                Reason: Restarting too quickly.
                                See: http://sun.com/msg/SMF-8000-L5
                                See: man -M /usr/share/man -s 1M sendmail
                                See: /var/svc/log/network-smtp:sendmail.log
                                Impact: This service is not running.
                                #
                                • 13. Re: I can ping my server but nothing else works
                                  dawgman
                                  What does the log say about the ssh service

                                  /var/svc/log/network-ssh:default.log

                                  have you checked the /etc/netmask and /etc/nsswitch.conf files
                                  • 14. Re: I can ping my server but nothing else works
                                    dawgman
                                    What does the log say about the ssh service

                                    /var/svc/log/network-ssh:default.log

                                    have you checked the /etc/netmask and /etc/nsswitch.conf files
                                    1 2 Previous Next