This discussion is archived
1 2 Previous Next 28 Replies Latest reply: Sep 27, 2012 4:48 PM by alan.pae RSS

I can ping my server but nothing else works

962245 Newbie
Currently Being Moderated
Hello Forum,
I'm a Linux newbie. I'm a new IT manager at my small business. I've inherited a Solaris server with absolutely no documentation. This server hosts a couple of websites and also a mysql database which is accessed via a java program. Everything was running smoothly for about a year after the last employee and we didn't use the server much since it was doing everything we wanted.

However about yesterday I realized I had some problems:
1)The web sites it hosted are no longer accessible ( the addresses "Failed to connect to server" ).
2)There is a java program which connects to a mysql database hosted on the server and when I attempt the run the software I get a "Network Connection Failure: Connection refused" error.
3) I am able to ping the machine XX.XX.XXX.XX successfully
4) I am unable to ssh into the machine "ssh: connect to host XX.XX.XXX.XX port 22: Connection refused"

I had no idea what was wrong. I was hoping a simple reboot was necessary (I've never rebooted a Solaris machine before). I fear my machine was hacked and taken over.

I requested remote access via a KVM from our co-location hosts. When I logged in I saw some text. So it seems for some reason the server tried to reboot on it's own and there were some issues. I think the import parts are that it said "the following files in / differ from the boot archive: /etc/rtc_config...." and "system/boot-archive:default failed fatally". I took a screen shot if that is necessary for you to view but it doesn't seem that there is a way to attach or embed files in this forum.

After reading here: http://docs.oracle.com/cd/E19082-01/819-2379/ggngq/index.html
I used the command: # svcadm clear system/boot-archive

After that command my system booted up into a gui. I thought my webserver would be running and mysql would be running since I assumed those programs start up on boot.
However after boot I was still only able to ping.. everything was still failing in the way it failed before.

Please help me.
Any ideas on: What cause my problem? Was cleaning the boot-archive the correct move? Where to go from here?

Edited by: SolarisNewbie on Sep 27, 2012 1:35 PM
  • 1. Re: I can ping my server but nothing else works
    alan.pae Journeyer
    Currently Being Moderated
    Depends on how fast you need it up again.

    We can do it the slow way with things such as ps -ef |grep name_of_some_server_process which will take days or you should probably just call someone and then work out some sort of support contract.

    alan
  • 2. Re: I can ping my server but nothing else works
    962245 Newbie
    Currently Being Moderated
    Thanks for the response. I have a bs in comp sci so I want/can to learn how to do whatever needs to be done. But I'm confused.. do Solaris Servers normally randomly crash and take days to recover? I have no idea why such an epic failure occurred and I could use some guidance.

    I figure I'll start with getting ssh working then and work up to msyql, tomcat, apache. Thanks for your tip.. I think can get ssh working at least.
  • 3. Re: I can ping my server but nothing else works
    alan.pae Journeyer
    Currently Being Moderated
    SolarisNewbie wrote:
    Thanks for the response. I have a bs in comp sci so I want/can to learn how to do whatever needs to be done. But I'm confused.. do Solaris Servers normally randomly crash and take days to recover? I have no idea why such an epic failure occurred and I could use some guidance.
    Depends on how it's done. If we sit here and say try this and see what happens and then wait for the reply it could. If someone knows what they're doing and are sitting directly in front of it then it should be easy enough.
    I figure I'll start with getting ssh working then and work up to msyql, tomcat, apache. Thanks for your tip.. I think can get ssh working at least.
    So the first question would be:

    What is the output of:

    pgrep apache
    pgrep mysql
    pgrep tomcat

    and let us know what each line says.

    alan
  • 4. Re: I can ping my server but nothing else works
    800381 Explorer
    Currently Being Moderated
    alan.pae wrote:
    SolarisNewbie wrote:
    Thanks for the response. I have a bs in comp sci so I want/can to learn how to do whatever needs to be done. But I'm confused.. do Solaris Servers normally randomly crash and take days to recover? I have no idea why such an epic failure occurred and I could use some guidance.
    Depends on how it's done. If we sit here and say try this and see what happens and then wait for the reply it could. If someone knows what they're doing and are sitting directly in front of it then it should be easy enough.
    I figure I'll start with getting ssh working then and work up to msyql, tomcat, apache. Thanks for your tip.. I think can get ssh working at least.
    So the first question would be:

    What is the output of:

    pgrep apache
    pgrep mysql
    pgrep tomcat

    and let us know what each line says.

    alan
    Shouldn't that be "pgrep httpd" for Apache?
  • 5. Re: I can ping my server but nothing else works
    alan.pae Journeyer
    Currently Being Moderated
    I stand corrected.

    alan
  • 6. Re: I can ping my server but nothing else works
    user171873 Explorer
    Currently Being Moderated
    Whenever I boot a Solaris system, the first thing I do after it comes up it run "svcs -xv" to see if everything started properly. If the system is fully up it should print out nothing. If you run it while the system is still coming up you'll see lines indicating that some services aren't yet up. If there is a problem, you'll usually see services in the "maintenance". You can then look at the end of the log file for the services in the maintenance state and maybe get an indication of what is wrong. Clearing up the services in the maintenance state (if any) would be my first step in trying to resolve the problem.

    Once you've resolved the problem with a service use "svcadm clear <name>" where <name> is the service name to attempt to clear the problem. If you resolved the issue, the service should come up cleanly; otherwise, it will stay in the maintenance state. I should mention that sometimes a service fails due to a transient problem and all you need to do is run "svcadm clear <name>". If you have multiple services in the maintenance state, I'd go from the top to the bottom.
  • 7. Re: I can ping my server but nothing else works
    muvvas Explorer
    Currently Being Moderated
    have you verified the respected application ports are listening

    netstat -an
    lsof -i
  • 8. Re: I can ping my server but nothing else works
    962245 Newbie
    Currently Being Moderated
    Thank you alan.pae.
    I'm sorry I couldn't respond sooner but I had other deadlines which took precedence. I appreciate your help.


    The output of those commands:
    pgrep apache
    pgrep mysql
    pgrep tomcat
    pgrep httpd

    are all blank. I hit enter, see no output and the I am returned to the prompt.
    I.e.
    "
    #pgrep apache
    #pgrep mysql
    #
    "


    I'm assuming that means that those processes never started so you can't see anything about them.
  • 9. Re: I can ping my server but nothing else works
    962245 Newbie
    Currently Being Moderated
    822432, Thank you for the excellent advice.

    I mentioned that I couldn't even boot the system so I ran:
    svcadm clear system/boot-archive
    I'm not sure if this removed setting for how everything should boot but this seems to be the case.


    Here is the result of svcs -xv:
    # svcs -xv
    svc:/application/print/server:default (LP print server)
    State: disabled since Fri Sep 14 16:09:08 2012
    Reason: Disabled by an administrator.
    See: http://sun.com/msg/SMF-8000-05
    See: man -M /usr/share/man -s 1M lpsched
    Impact: 1 dependent service is not running:
    svc:/application/print/ipp-listener:default

    svc:/application/database/mysql5:default (MySQL RDBMS 5.1.31)
    State: disabled since Fri Sep 14 16:09:09 2012
    Reason: Disabled by an administrator.
    See: http://sun.com/msg/SMF-8000-05
    See: man -M /usr/local/mysql5/man -s 1 mysql
    Impact: 1 dependent service is not running:
    svc:/application/web/tomcat6:default

    svc:/network/rpc/smserver:default (removable media management)
    State: disabled since Fri Sep 14 16:09:22 2012
    Reason: Disabled by an administrator.
    See: http://sun.com/msg/SMF-8000-05
    See: man -M /usr/share/man -s 1M rpc.smserverd
    Impact: 1 dependent service is not running:
    svc:/system/filesystem/volfs:default

    svc:/system/pkgserv:default (Flush package command database to disk (see pkgadm(
    1m)).)
    State: maintenance since Fri Sep 14 16:09:14 2012
    Reason: Start method failed repeatedly, last died on Killed (9).
    See: http://sun.com/msg/SMF-8000-KS
    See: /etc/svc/volatile/system-pkgserv:default.log
    See: /var/svc/log/system-pkgserv:default.log
    Impact: This service is not running.

    svc:/network/ssh:default (SSH server)
    State: maintenance since Fri Sep 14 16:09:22 2012
    Reason: Start method failed repeatedly, last exited with status 137.
    See: http://sun.com/msg/SMF-8000-KS
    See: man -M /usr/share/man -s 1M sshd
    See: /var/svc/log/network-ssh:default.log
    Impact: This service is not running.

    svc:/network/smtp:sendmail (sendmail SMTP mail transfer agent)
    State: maintenance since Fri Sep 14 16:09:23 2012
    Reason: Restarting too quickly.
    See: http://sun.com/msg/SMF-8000-L5
    See: man -M /usr/share/man -s 1M sendmail
    See: /var/svc/log/network-smtp:sendmail.log
    Impact: This service is not running."


    I will attempt to start svcadm clearing these functions tomorrow. However I just tried my ssh and it still failed in the same manner.
  • 10. Re: I can ping my server but nothing else works
    alan.pae Journeyer
    Currently Being Moderated
    I'm assuming that means that those processes never started so you can't see anything about them.
    Yup. :-)
  • 11. Re: I can ping my server but nothing else works
    alan.pae Journeyer
    Currently Being Moderated
    SolarisNewbie wrote:
    822432, Thank you for the excellent advice.

    I mentioned that I couldn't even boot the system so I ran:
    svcadm clear system/boot-archive
    I'm not sure if this removed setting for how everything should boot but this seems to be the case.
    No, your fine.
    svc:/application/print/server:default (LP print server)
    svc:/application/database/mysql5:default (MySQL RDBMS 5.1.31)
    svc:/network/rpc/smserver:default (removable media management)
    svc:/system/pkgserv:default (Flush package command database to disk (see pkgadm(
    1m)).)
    svc:/network/ssh:default (SSH server)
    svc:/network/smtp:sendmail (sendmail SMTP mail transfer agent)
    I will attempt to start svcadm clearing these functions tomorrow. However I just tried my ssh and it still failed in the same manner.
    So Printer, MySql, SMSERVER, pkgserv, SSH and Sendmail are all faulted.

    So after using

    svcadm clear FMRI

    Try

    pgrep httpd

    and see if that one returns anything.

    Run:

    ifconfig -a

    and post the output. Sanitize the IP Addresses if you want to.

    SSH isn't running yet so you'll have to keep using the KVM device for now.

    alan
  • 12. Re: I can ping my server but nothing else works
    962245 Newbie
    Currently Being Moderated
    I tried to svcadm clearthe processes and some weren’t in the maintaince state. After that I pgrep’ed httpd,msyql and tomcat. All of them aren’t running. Returns nothing, simply goes back to the prompt.

    Below I’ve posted the ifconfig –a and i’ve posted what svc –xv returns after trying the clears.

    Is it possible that I need to clear some dependent process or manually restart some of these?

    Thanks for the continued assistance.


    # ifconfig -a
    lo0: flags=2001000849<UP,LOOPBACK,RUNNING,MULTICAST,IPv4,VIRTUAL> mtu
    8232 index 1
    inet 127.0.0.1 netmask ff000000
    e1000g0: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index 2
    inet XX.XX.XX.XX netmask ff000000 broadcast XX.255.255.255
    ether 0:14:4f:27:38:a8


    # svcs -xv
    svc:/application/print/server:default (LP print server)
    State: disabled since Fri Sep 14 16:09:08 2012
    Reason: Disabled by an administrator.
    See: http://sun.com/msg/SMF-8000-05
    See: man -M /usr/share/man -s 1M lpsched
    Impact: 1 dependent service is not running:
    svc:/application/print/ipp-listener:default

    svc:/application/database/mysql5:default (MySQL RDBMS 5.1.31)
    State: disabled since Fri Sep 14 16:09:09 2012
    Reason: Disabled by an administrator.
    See: http://sun.com/msg/SMF-8000-05
    See: man -M /usr/local/mysql5/man -s 1 mysql
    Impact: 1 dependent service is not running:
    svc:/application/web/tomcat6:default

    svc:/network/rpc/smserver:default (removable media management)
    State: disabled since Fri Sep 14 16:09:22 2012
    Reason: Disabled by an administrator.
    See: http://sun.com/msg/SMF-8000-05
    See: man -M /usr/share/man -s 1M rpc.smserverd
    Impact: 1 dependent service is not running:
    svc:/system/filesystem/volfs:default

    svc:/system/pkgserv:default (Flush package command database to disk
    (see pkgadm(1m)).)
    State: maintenance since Wed Sep 26 10:46:07 2012
    Reason: Start method failed repeatedly, last died on Killed (9).
    See: http://sun.com/msg/SMF-8000-KS
    See: /etc/svc/volatile/system-pkgserv:default.log
    See: /var/svc/log/system-pkgserv:default.log
    Impact: This service is not running.

    svc:/network/ssh:default (SSH server)
    State: maintenance since Wed Sep 26 10:46:15 2012
    Reason: Start method failed repeatedly, last exited with status 137.
    See: http://sun.com/msg/SMF-8000-KS
    See: man -M /usr/share/man -s 1M sshd
    See: /var/svc/log/network-ssh:default.log
    Impact: This service is not running.

    svc:/network/smtp:sendmail (sendmail SMTP mail transfer agent)
    State: maintenance since Wed Sep 26 10:46:25 2012
    Reason: Restarting too quickly.
    See: http://sun.com/msg/SMF-8000-L5
    See: man -M /usr/share/man -s 1M sendmail
    See: /var/svc/log/network-smtp:sendmail.log
    Impact: This service is not running.
    #
  • 13. Re: I can ping my server but nothing else works
    dawgman Newbie
    Currently Being Moderated
    What does the log say about the ssh service

    /var/svc/log/network-ssh:default.log

    have you checked the /etc/netmask and /etc/nsswitch.conf files
  • 14. Re: I can ping my server but nothing else works
    dawgman Newbie
    Currently Being Moderated
    What does the log say about the ssh service

    /var/svc/log/network-ssh:default.log

    have you checked the /etc/netmask and /etc/nsswitch.conf files
1 2 Previous Next

Legend

  • Correct Answers - 10 points
  • Helpful Answers - 5 points