Forum Stats

  • 3,780,576 Users
  • 2,254,410 Discussions
  • 7,879,386 Comments

Discussions

JSC WLS instance can't be reached via public IP

Timo Hahn
Timo Hahn Senior Principal Technical Consultant - Oracle ACE DirectorMember, Moderator Posts: 37,765 Red Diamond
edited Aug 5, 2016 8:34AM in Java Cloud Service

Hi,

I have two different WLS instanced running in my JCS. One 12.1.3 and one 12.2.1 instance. Both are up and running when looking at the compute cloud service. I can connect via ssh to both of them in a terminal.

Selection_012.jpg
 

My problem is, that the 12.1.3 instance can’t be reached using the public IP for the WLS. Neither the admin console nor the Enterprise Manager can’t be reached. Start of July this worked without a problem, as I used both WLS instances on a conference.


 

The security rules, security lists and security applications entries for both instances are set identical (besides the name). Both http and https ports are set to allow public acceess.


 

Trying to connect to the 12.1.3 instance returns a ‘refused to connect’ error.


 

Anybody any idea why this happens or how to fix this?


 

Thanks,

Timo

Tagged:
Manish Kapur-Oracle

Best Answer

Answers

  • Manish Kapur-Oracle
    Manish Kapur-Oracle Member Posts: 73 Employee
    edited Jul 29, 2016 6:54PM

    Hi Timo, Can you check and make sure your access rules for all the consoles are enabled? Docs for reference here. If they are enabled, can you test to see if any application that you may have deployed on your JCS instance is accessible? If you have verified this and still see the issue, please engage with Oracle Support by opening a Service Request.

  • Timo Hahn
    Timo Hahn Senior Principal Technical Consultant - Oracle ACE Director Member, Moderator Posts: 37,765 Red Diamond
    edited Aug 3, 2016 3:52AM

    The access rules are all setup correctly and enabled. I compared them with the ones for the working instance.

    In the meantime I found out, that the whole weblogic server and the domain I created for version 12.1.3 is gone from the vm. I logged into the console and looked around for some log files which might give some hint about the problem. There is not weblogic server installation or domain to be found on the server. There are some of the needed folders in the file system, but they are empty. It looks as someone has remove all the files from the server.

    No wonder the admin console can#t be reached, it's simply not there.

    Do you know how this could happen?

    The JCS still show the instance up and running.

    Timo

  • Manish Kapur-Oracle
    Manish Kapur-Oracle Member Posts: 73 Employee
    edited Aug 3, 2016 1:11PM

    That is strange. It is difficult to tell how this happened. In JCS VM, there are directories like /u01/jdk, /u01/app/oracle/middleware etc. that are basically mount points for the file systems. These are used by the WebLogic Server installation. I am guessing that there was a problem with mounting the file systems and hence your folders/directories are showing up as empty.  You can check the output of 'mount' command to confirm if this is indeed the case. If they are not mounted, I would suggest opening a SR to debug why the file systems did not get mounted. You may want to restart the service VMs and see if that remounts before you open the SR.

    Timo Hahn
  • Timo Hahn
    Timo Hahn Senior Principal Technical Consultant - Oracle ACE Director Member, Moderator Posts: 37,765 Red Diamond
    edited Aug 4, 2016 11:32AM

    Yes, it's really strange. The mount tells me that no filesystems are mounted to the /u01/jdk, /u01/data ... folders.

    On the running instance I see

    /dev/mapper/vg_binaries-lv_tools on /u01/app/oracle/tools type ext4 (rw)/dev/mapper/vg_backup-lv_backup on /u01/data/backup type ext4 (rw)/dev/mapper/vg_domains-lv_domains on /u01/data/domains type ext4 (rw)/dev/mapper/vg_binaries-lv_mw on /u01/app/oracle/middleware type ext4 (rw)/dev/mapper/vg_binaries-lv_jdk on /u01/jdk type ext4 (rw)

    None of those mappers are visible on the broken instance. Stopping and starting hte server instance doesn't remount the drives.

    My conclusion is that the whole instance is gone.

    I'll open an SR but I have not much hope that they are able to fix this.

    If suppot can'T come up with a fix I delete the instance and create a new one

    Timo

  • Manish Kapur-Oracle
    Manish Kapur-Oracle Member Posts: 73 Employee
    edited Aug 4, 2016 11:45AM

    Don't give up, I think support may be able to help. It could be that the storage which has those file systems got detached and hence the file systems were not mounted. Attaching the storage back and remounting should do the trick if this is the case. You can also try to manually mount and it may work if the storage volume is attached.

    Example: # mount /dev/vg_backup/lv_backup /u01/data/backup

    Anyways, do work with support and hopefully you can recover the instance.

    Timo Hahn
  • Timo Hahn
    Timo Hahn Senior Principal Technical Consultant - Oracle ACE Director Member, Moderator Posts: 37,765 Red Diamond
    edited Aug 4, 2016 12:50PM

    Right, I was a bit too early.

    I enabled boot logging in grub.conf e viola the mount points are back online

    OK, I now dig through the log files. One thing I found in the admins server log are plenty of messages

    ORA-28001: the password has expired

    I can't find the db connection as it'S not specified, but I assume it must be something wis opss or mds schemas used by the server. Looking into the db I see some users

    SP506155984_OPSS  EXPIRED & LOCKEDSP506155984_MDS    EXPIRED & LOCKED

    But it might be another connection.

    The admin server is still not reachable and the log does not contain a message that the server is running. I stopped the instance and started the instance again via the jcs console. Now the mounted filesystems are gone again!

    So, when I restarted the server after a try to install a patch earlier I guess the mounts where lost and I never noticed it because I work mainly in the other instance now.

    I now do a

    sudo /sbin/shutdown -r now

    from the console and see if the mounts are coming back.

    Timo

  • Timo Hahn
    Timo Hahn Senior Principal Technical Consultant - Oracle ACE Director Member, Moderator Posts: 37,765 Red Diamond
    edited Aug 4, 2016 1:07PM

    OK the shutdown did the trick. Conclusion is that stopping the jcs  instance via the jcs console

    Selection_835.jpg

    detaches the needed mounts

    The admins server log shows

    [EL Severe]: ejb: 2016-08-04 16:50:22.862--ServerSession(796424685)--Exception [EclipseLink-4002] (Eclipse Persistence Services - 2.5.2.v20140319-9ad6abd): org.eclipse.persistence.exceptions.DatabaseExceptionInternal Exception: java.sql.SQLException: ORA-28001: the password has expiredError Code: 28001Aug 04, 2016 4:50:22 PM oracle.security.jps.internal.policystore.rdbms.JpsDBDataManager initINFO: Trying to connect to the DB. Please check if the Database is available.

    Not sure how to proceed form here. The server is still not in running mode.

    Timo

  • Manish Kapur-Oracle
    Manish Kapur-Oracle Member Posts: 73 Employee
    edited Aug 4, 2016 5:24PM Accepted Answer

    I guess for security reasons DBCS default policy requires you to change passwords periodically. See My Oracle Support Document 1682942.1 for help addressing the expired password issue.

    Timo Hahn
  • Timo Hahn
    Timo Hahn Senior Principal Technical Consultant - Oracle ACE Director Member, Moderator Posts: 37,765 Red Diamond
    edited Aug 5, 2016 8:34AM

    OK, I got the server back running after following the doc you mentioned.

    Thanks , this was a great help.

    There are still some problems, but the server is back aup

    Timo

    Manish Kapur-Oracle
This discussion has been closed.