1 2 Previous Next 15 Replies Latest reply on Jul 23, 2015 7:00 PM by Stephen9

    SDCLI just hangs

    Stephen9

      (reposting this because I accidentally marked the first one as answered and can't seem to unanswer it...)

       

      We've been playing around with unit testing through SQL developer and absolutely love it, so we're taking it the next step by trying to integrate it into a Jenkins build process.  At the moment I've got an RHEL 6.5 VM with JDK 1.8.0_45 and SQL Developer 4.1.0 installed.  The box doesn't have Oracle DB installed on it yet (we're on 11.2.0.4 by the way), nor does it have Jenkins installed on it.  It's a simple test to see if I can use the SDCLI client to run the unit tests from a separate repository on a separate application db.

       

      However, when I run .../sqldeveloper/sqldeveloper/sdcli help, I get the following:

       

      Oracle SQL Developer

      Copyright (c) 1997, 2015, Oracle and/or its affiliates. All rights reserved.

       

      And it just sits there.  I left it on overnight, and it's still just sitting there, I had to ctrl+c the process so I could resume working in the terminal.

       

      I'm SSH'ed in with X11 forwarding disabled.  $DISPLAY is unset.  I have tried this from another RHEL 6.5 VM (on a different server) and got the same problem, so I don't believe it to be environmental.

       

      Does anyone have any ideas on what to try?  I'm going to try uninstalling SQL Developer and installing a previous version to see if that helps in the meantime.

       

      Thank you for your time!

        • 1. Re: SDCLI just hangs
          thatJeffSmith-Oracle

          attach to the PID and get a jstack so we can see where and what it's hanging on

          1 person found this helpful
          • 2. Re: SDCLI just hangs
            Stephen9

            Here's the jstack of the Java process running:

             

            Full thread dump Java HotSpot(TM) 64-Bit Server VM (25.45-b02 mixed mode):

            "Attach Listener" #32 daemon prio=9 os_prio=0 tid=0x00007faa54006800 nid=0x51f2 runnable [0x0000000000000000]
               java.lang.Thread.State: RUNNABLE

            "DestroyJavaVM" #31 prio=5 os_prio=0 tid=0x00007faa80010800 nid=0x51c2 waiting on condition [0x0000000000000000]
               java.lang.Thread.State: RUNNABLE

            "Flushing caches" #9 prio=1 os_prio=0 tid=0x00007faa2c20c000 nid=0x51e5 in Object.wait() [0x00007faa641cc000]
               java.lang.Thread.State: TIMED_WAITING (on object monitor)
                    at java.lang.Object.wait(Native Method)
                    - waiting on <0x00000000ce1ef1d8> (a org.netbeans.Stamps$Worker)
                    at org.netbeans.Stamps$Worker.run(Stamps.java:872)
                    - locked <0x00000000ce1ef1d8> (a org.netbeans.Stamps$Worker)

            "Thread-6" #30 daemon prio=5 os_prio=0 tid=0x00007faa2c208800 nid=0x51e4 in Object.wait() [0x00007faa64fdc000]
               java.lang.Thread.State: WAITING (on object monitor)
                    at java.lang.Object.wait(Native Method)
                    - waiting on <0x00000000ef9ff790> (a java.util.LinkedList)
                    at java.lang.Object.wait(Object.java:502)
                    at java.util.prefs.AbstractPreferences$EventDispatchThread.run(AbstractPreferences.java:1482)
                    - locked <0x00000000ef9ff790> (a java.util.LinkedList)

            "TimedCache-Timer" #23 daemon prio=5 os_prio=0 tid=0x00007faa2c3ed800 nid=0x51dc in Object.wait() [0x00007faa64edb000]
               java.lang.Thread.State: TIMED_WAITING (on object monitor)
                    at java.lang.Object.wait(Native Method)
                    - waiting on <0x00000000ce91d7a8> (a java.util.TaskQueue)
                    at java.util.TimerThread.mainLoop(Timer.java:552)
                    - locked <0x00000000ce91d7a8> (a java.util.TaskQueue)
                    at java.util.TimerThread.run(Timer.java:505)

            "Framework Event Dispatcher" #19 daemon prio=5 os_prio=0 tid=0x00007faa2c339000 nid=0x51d7 in Object.wait() [0x00007faa652dd000]
               java.lang.Thread.State: WAITING (on object monitor)
                    at java.lang.Object.wait(Native Method)
                    - waiting on <0x00000000ce182af0> (a org.eclipse.osgi.framework.eventmgr.EventManager$EventThread)
                    at java.lang.Object.wait(Object.java:502)
                    at org.eclipse.osgi.framework.eventmgr.EventManager$EventThread.getNextEvent(EventManager.java:400)
                    - locked <0x00000000ce182af0> (a org.eclipse.osgi.framework.eventmgr.EventManager$EventThread)
                    at org.eclipse.osgi.framework.eventmgr.EventManager$EventThread.run(EventManager.java:336)

            "Inactive RequestProcessor thread [Was:On Start/Stop/null]" #16 daemon prio=5 os_prio=0 tid=0x00007faa2c27d000 nid=0x51d4 in Object.wait() [0x00007faa657e0000]
               java.lang.Thread.State: TIMED_WAITING (on object monitor)
                    at java.lang.Object.wait(Native Method)
                    - waiting on <0x00000000ce8e1140> (a java.lang.Object)
                    at org.openide.util.RequestProcessor$Processor.run(RequestProcessor.java:1982)
                    - locked <0x00000000ce8e1140> (a java.lang.Object)

            "State Data Manager" #15 daemon prio=5 os_prio=0 tid=0x00007faa2c25d000 nid=0x51d2 waiting on condition [0x00007faa65cac000]
               java.lang.Thread.State: TIMED_WAITING (sleeping)
                    at java.lang.Thread.sleep(Native Method)
                    at org.eclipse.osgi.internal.baseadaptor.StateManager.run(StateManager.java:297)
                    at java.lang.Thread.run(Thread.java:745)

            "Framework Active Thread" #14 prio=5 os_prio=0 tid=0x00007faa2c239800 nid=0x51d1 in Object.wait() [0x00007faa65ae1000]
               java.lang.Thread.State: TIMED_WAITING (on object monitor)
                    at java.lang.Object.wait(Native Method)
                    - waiting on <0x00000000ce84de80> (a org.eclipse.osgi.framework.internal.core.Framework)
                    at org.eclipse.osgi.framework.internal.core.Framework.run(Framework.java:1862)
                    - locked <0x00000000ce84de80> (a org.eclipse.osgi.framework.internal.core.Framework)
                    at java.lang.Thread.run(Thread.java:745)

            "Inactive RequestProcessor thread [Was:Folder recognizer/org.openide.loaders.FolderList$ListTask]" #13 daemon prio=5 os_prio=0 tid=0x00007faa2c100800 nid=0x51d0 in Object.wait() [0x00007faa65fad000]
               java.lang.Thread.State: TIMED_WAITING (on object monitor)
                    at java.lang.Object.wait(Native Method)
                    - waiting on <0x00000000ce17db00> (a java.lang.Object)
                    at org.openide.util.RequestProcessor$Processor.run(RequestProcessor.java:1982)
                    - locked <0x00000000ce17db00> (a java.lang.Object)

            "RequestProcessor queue manager" #12 daemon prio=5 os_prio=0 tid=0x00007faa2c008000 nid=0x51cf in Object.wait() [0x00007faa665bc000]
               java.lang.Thread.State: TIMED_WAITING (on object monitor)
                    at java.lang.Object.wait(Native Method)
                    - waiting on <0x00000000ce1e9e28> (a java.lang.Class for org.openide.util.RequestProcessor$TickTac)
                    at org.openide.util.RequestProcessor$TickTac.obtainFirst(RequestProcessor.java:2262)
                    - locked <0x00000000ce1e9e28> (a java.lang.Class for org.openide.util.RequestProcessor$TickTac)
                    at org.openide.util.RequestProcessor$TickTac.run(RequestProcessor.java:2238)

            "Active Reference Queue Daemon" #10 daemon prio=1 os_prio=0 tid=0x00007faa80335800 nid=0x51cd in Object.wait() [0x00007faa678d0000]
               java.lang.Thread.State: WAITING (on object monitor)
                    at java.lang.Object.wait(Native Method)
                    - waiting on <0x00000000ce8e1518> (a java.lang.ref.ReferenceQueue$Lock)
                    at java.lang.Object.wait(Object.java:502)
                    at org.openide.util.lookup.implspi.ActiveQueue.removeBetter(ActiveQueue.java:225)
                    at org.openide.util.lookup.implspi.ActiveQueue.access$400(ActiveQueue.java:57)
                    at org.openide.util.lookup.implspi.ActiveQueue$Daemon.run(ActiveQueue.java:180)
                    - locked <0x00000000ce8e1518> (a java.lang.ref.ReferenceQueue$Lock)

            "Service Thread" #7 daemon prio=9 os_prio=0 tid=0x00007faa800c3800 nid=0x51cb runnable [0x0000000000000000]
               java.lang.Thread.State: RUNNABLE

            "C1 CompilerThread1" #6 daemon prio=9 os_prio=0 tid=0x00007faa800be800 nid=0x51ca waiting on condition [0x0000000000000000]
               java.lang.Thread.State: RUNNABLE

            "C2 CompilerThread0" #5 daemon prio=9 os_prio=0 tid=0x00007faa800bb800 nid=0x51c9 waiting on condition [0x0000000000000000]
               java.lang.Thread.State: RUNNABLE

            "Signal Dispatcher" #4 daemon prio=9 os_prio=0 tid=0x00007faa800ba000 nid=0x51c8 runnable [0x0000000000000000]
               java.lang.Thread.State: RUNNABLE

            "Finalizer" #3 daemon prio=8 os_prio=0 tid=0x00007faa80082000 nid=0x51c7 in Object.wait() [0x00007faa6c386000]
               java.lang.Thread.State: WAITING (on object monitor)
                    at java.lang.Object.wait(Native Method)
                    - waiting on <0x00000000ce2e9fd8> (a java.lang.ref.ReferenceQueue$Lock)
                    at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
                    - locked <0x00000000ce2e9fd8> (a java.lang.ref.ReferenceQueue$Lock)
                    at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164)
                    at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:209)

            "Reference Handler" #2 daemon prio=10 os_prio=0 tid=0x00007faa8007f800 nid=0x51c6 in Object.wait() [0x00007faa6c487000]
               java.lang.Thread.State: WAITING (on object monitor)
                    at java.lang.Object.wait(Native Method)
                    - waiting on <0x00000000ce1eb328> (a java.lang.ref.Reference$Lock)
                    at java.lang.Object.wait(Object.java:502)
                    at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:157)
                    - locked <0x00000000ce1eb328> (a java.lang.ref.Reference$Lock)

            "VM Thread" os_prio=0 tid=0x00007faa8007a800 nid=0x51c5 runnable

            "GC task thread#0 (ParallelGC)" os_prio=0 tid=0x00007faa80026000 nid=0x51c3 runnable

            "GC task thread#1 (ParallelGC)" os_prio=0 tid=0x00007faa80028000 nid=0x51c4 runnable

            "VM Periodic Task Thread" os_prio=0 tid=0x00007faa800c6000 nid=0x51cc waiting on condition

            JNI global references: 243

             

             

            Thanks Jeff!

            • 3. Re: SDCLI just hangs
              Brian Jeffries-Oracle

              I can't seem to duplicate this.

              I have a vbox VM with RHEL 6.5

              and a new download/unzip of SQLDeveloper from OTN (4.1.0.19.07)

              Connecting with Putty via ssh,

              [bjeffrie@bjeffrie-linux-vbox ~]$ cd Desktop/sqldeveloper-4.1.0.19.07/sqldeveloper/bin

              [bjeffrie@bjeffrie-linux-vbox bin]$ echo DISPLAY=$DISPLAY

              DISPLAY=

              [bjeffrie@bjeffrie-linux-vbox bin]$ time ./sdcli help

               

               

              Oracle SQL Developer

              Copyright (c) 1997, 2015, Oracle and/or its affiliates. All rights reserved.

               

               

              Type the full pathname of a JDK installation (or Ctrl-C to quit), the path will be stored in /home/bjeffrie/.sqldeveloper/4.1.0/product.conf

              /opt/jdk1.8.0_40

              Available features:

              cart: Database Cart Batch Tasks

              dba: Basic Batch DBA Tasks

              format: Utility Import Task

              migration: Database Migration Tasks

              reports: Basic Batch Reporting Tasks

              unittest: Unit Testing Batch Tasks

              utility: Utility Import Task

               

               

              real    0m23.689s

              user    0m11.601s

              sys     0m1.869s

              [bjeffrie@bjeffrie-linux-vbox bin]$ cat /etc/redhat-release

              Red Hat Enterprise Linux Server release 6.5 (Santiago)

              [bjeffrie@bjeffrie-linux-vbox bin]$


              • 4. Re: SDCLI just hangs
                Stephen9

                Do you mind trying JDK 1.8.0_45?  I haven't found anyone else experiencing the issue via web search (hence why I posted here), so I'm not particularly surprised that it's hard to reproduce.

                It feels like it's still trying to launch the GUI even though $DISPLAY is unset.  Is there something I can try to at least get me started down the path?  I don't really have a direction to follow to troubleshoot at the moment...

                 

                Note that I do get to the step asking me to insert the path to JDK.

                • 5. Re: SDCLI just hangs
                  Brian Jeffries-Oracle

                  I just tried jdk1.8.0_45 & that works as well. A couple random ideas that may help:

                   

                  1. Try running with verbose on "./sdcli -verbose help" - maybe that can give us an idea of where it is stuck.

                  2. I'm a developer with many jdks so none of them are installed from .rpm, the tar.gz versions are simply extracted. Don't really see that making any difference but possibly a difference in how we're executing...

                  3. We use Hudson (origins of Jenkins) extensively for builds & some testing. It might be worthwhile to install Jenkins and set up a simple 'run a batch script' build to try it from there.

                  (edit) remove non-working sqldeveloper-debug.conf idea

                  1 person found this helpful
                  • 6. Re: SDCLI just hangs
                    Stephen9

                    Thanks for the suggestions, here's where I'm at so far:

                     

                    1. With the -verbose flag, I don't get any extra information, just: Oracle SQL Developer Copyright (c) 1997, 2015, Oracle and/or its affiliates. All rights reserved.
                    2. I uninstalled the rpms (I had 1.7 and 1.8 installed) and downloaded + extracted the 1.8.0_45 tar.gz.  Same problem though.
                    3. I'll give Jenkins a shot when I have time.

                     

                    Thanks!

                    • 7. Re: SDCLI just hangs
                      Stephen9


                      I had an idea to see at least some information on where the hangup is.  I'm not an OS guy (I'm a developer as well), so I'm not sure if this helps, but I did a time on sdcli, and after I killed it I got this result:

                       

                      real    5m51.937s

                      user    0m18.740s

                      sys     0m2.166s

                       

                      Is there anything obvious that I might be missing at the OS level?  I'll work on getting Jenkins on the machine in the meantime.

                      • 8. Re: SDCLI just hangs
                        Gary Graham-Oracle

                        Too bad the -verbose flag did not shed any light... I was going to give that a big +1.

                         

                        Edit: Rather than on the command line, try adding  a

                        AddVMOption -verbose

                        line at the end of your sdcli.conf file. At least that's what we have to do in a Windows environment.

                         

                        Just to comment on the JVM thread dump you provided...

                        1. The runnable threads are all non-Java, related to the JVM.

                        2. Each of the Java theads (except State Data Manager) are waiting on a specific monitor object (and also locking the same).

                        3. State Data Manager is just sleeping in a timed wait.

                         

                        Perhaps taking multiple thread dumps with jstack and comparing them might help, but your issue is a bit unusual.

                        1 person found this helpful
                        • 9. Re: SDCLI just hangs
                          Stephen9

                          I added the VM option to the bottom of sdcli.conf, but it still just hangs.  I caused the hang 4 times and took a thread dump in each.  Here are the main differences I found:

                           

                          "Inactive RequestProcessor thread [Was:On Start/Stop/null]" #16 daemon prio=5 os_prio=0 tid=0x00007f6fd8241800 nid=0x604f in Object.wait() [0x00007f6fcbdfe000]
                             java.lang.Thread.State: TIMED_WAITING (on object monitor)
                                  at java.lang.Object.wait(Native Method)
                                  - waiting on <0x00000000ce8e9ab0> (a java.lang.Object)
                                  at org.openide.util.RequestProcessor$Processor.run(RequestProcessor.java:1982)
                                  - locked <0x00000000ce8e9ab0> (a java.lang.Object)

                          in some, not others.


                          "Inactive RequestProcessor thread [Was:TimedSoftReference/org.openide.util.TimedSoftReference]" #13 daemon prio=1 os_prio=0 tid=0x00007f6fd810e800 nid=0x604c in Object.wait() [0x00007f6ff053c000]
                             java.lang.Thread.State: TIMED_WAITING (on object monitor)
                                  at java.lang.Object.wait(Native Method)
                                  at org.openide.util.RequestProcessor$Processor.run(RequestProcessor.java:1982)
                                  - locked <0x00000000ce1378b8> (a java.lang.Object)

                          in some, not others

                           

                          "State Data Manager" #15 daemon prio=5 os_prio=0 tid=0x00007f6fd821d800 nid=0x604e waiting on condition [0x00007f6ff013a000]
                             java.lang.Thread.State: TIMED_WAITING (sleeping)
                                  at java.lang.Thread.sleep(Native Method)
                                  at org.eclipse.osgi.internal.baseadaptor.StateManager.run(StateManager.java:297)
                                  at java.lang.Thread.run(Thread.java:745)

                          in all of them

                           

                          "RequestProcessor queue manager" #12 daemon prio=5 os_prio=0 tid=0x00007fc624015800 nid=0x5609 in Object.wait() [0x00007fc63cf5a000]
                             java.lang.Thread.State: TIMED_WAITING (on object monitor)
                                  at java.lang.Object.wait(Native Method)
                                  at org.openide.util.RequestProcessor$TickTac.obtainFirst(RequestProcessor.java:2262)
                                  - locked <0x00000000ce7ded90> (a java.lang.Class for org.openide.util.RequestProcessor$TickTac)
                                  at org.openide.util.RequestProcessor$TickTac.run(RequestProcessor.java:2238)

                          in some, not others

                           

                          I'm working on Jenkins installation at the moment, but I would like to get it working strictly in the ssh command line as well (so I can develop the tests and test them before deploying to Jenkins).

                          • 10. Re: SDCLI just hangs
                            Gary Graham-Oracle

                            I caused the hang 4 times and took a thread dump in each.

                            In order to compare thread dumps, the actual process should be...

                            1. Cause the hang once.

                            2. Take multiple jstack dumps on that JVM process id, spaced apart by some time interval -- say 30 seconds or a couple minutes.

                             

                            Then use a utility like TDA (Thread Dump Analyzer) to do the diff -- although that is getting a bit dated, and does not parse dumps from more recent JVM versions without some editing.

                            1 person found this helpful
                            • 11. Re: SDCLI just hangs
                              Stephen9

                              I caused the hang once, waited a few minutes, then ran a thread dump every 30 seconds for 3 mins.  I then (ran the sed command to remove the os_prio stuff and) used TDA to diff between all the dumps.  The following is the output:

                               

                               

                              FinalizerDaemon5140009022169088420in Object.wait()[0x00007f564d168000]
                              Active Reference Queue DaemonDaemon5140009025196032426in Object.wait()[0x00007f564c6c1000]
                              State Data ManagerDaemon5140007614308352431waiting on condition[0x00007f5634a52000]
                              Framework Active ThreadTask5140007614169088430in Object.wait()[0x00007f5634b53000]
                              Reference HandlerDaemon5140009022160896419in Object.wait()[0x00007f564d269000]
                              DestroyJavaVMTask5140009021704192414waiting on condition[0x0000000000000000]
                              Thread-6Daemon5140007617187840450in Object.wait()[0x00007f56342ba000]
                              Signal DispatcherDaemon5140009022400512421runnable[0x0000000000000000]
                              Service ThreadDaemon5140009022437376424runnable[0x0000000000000000]
                              C2 CompilerThread0Daemon5140009022404608422waiting on condition[0x0000000000000000]
                              TimedCache-TimerDaemon5140007615973376441in Object.wait()[0x00007f5627bfc000]
                              C1 CompilerThread1Daemon5140009022416896423waiting on condition[0x0000000000000000]
                              Attach ListenerDaemon5140008149225472517waiting on condition[0x0000000000000000]
                              Framework Event DispatcherDaemon5140007615211520437in Object.wait()[0x00007f5627ffe000]

                               

                               

                              Does this mean anything to you?  I'll continue to look at it, but it's a bit foreign to me.  Thanks for your ideas!

                              • 12. Re: SDCLI just hangs
                                Gary Graham-Oracle

                                Sorry, not much there ... it just looks like 14 of the 22 threads are long-running, with "Inactive...", "GC..", "VM..." and a couple of other threads being transient. Good to see you are so resourceful in trying to track down the problem.

                                1 person found this helpful
                                • 13. Re: SDCLI just hangs
                                  Stephen9


                                  I guess I'll work with our DBA to put in a support ticket.  Thanks for your time and advice.  At the least it was a good learning exercise.

                                   

                                  I'll post the resolution if/when we get one.

                                  • 14. Re: SDCLI just hangs
                                    Stephen9


                                    After much back and forth, we did manage to make some headway with Oracle Support.  It turns out SDCLI is having issues on both Mac and Linux although it works fine on Windows (BUG 21299336, but our SR is non-public).

                                     

                                    Oracle Support says that they should have a fix out shortly for this and to watch http://www.oracle.com/technetwork/developer-tools/sql-developer/overview/index.html for the patch.

                                    1 2 Previous Next