8 Replies Latest reply on Feb 26, 2020 2:13 PM by userBDBDMS-Oracle

    cannot deploy topo --- it kept running for a long time and then it failed

    Eric Jing

      Dear sir,

       

      I try to install Oracle no sql db on Oracle linux 8.1 for testing purpose, but failed.

      Could anyone able to help? I still cannot figure out what's wrong with my setting after i tried again and again. Thanks.

       

      Details as follow:

      1. os: oracle linux 8.1 on virtual machine (with 6GB ram configured for the VM)

      2. virtual disks: 4 virtual disks (5GB/5GB/5GB/10GB)

      3. the first 3 disks are for data (I try to configure 3 replications for test), and the 4th vitual disk is for /kv, which contains KVHOME and KVROOT, plus kv/kvroot/admin01(to store the admin db)

      4. after i finished the makebootconfig and securityconfig, i do the follow up configuration according to the online manual, until i encounter below error:

         The command "depoly-topology" kept runnning for very long time, and after 20 mins, it failed.

      5. I doubt that may be it is because of that I do the deployment on the ssd on my laptop,

       

      when i do the formating and mounting on my virtual machine, i use below comand:

      mkfs.ext4 -E stride=128,stripe-width=128 /dev/nvme0n2p1

      mount -o discard -a

       

      and, within /etc/fstab, i wrote:

      /dev/nvme0n2p1 /disk1/data                ext4 noatime,discard,defaults    0 0

       

        • 1. Re: cannot deploy topo --- it kept running for a long time and then it failed
          Eric Jing

          FYI:

          The error log are as follow :

          kv-> show plans

               8 Deploy Zone              SUCCEEDED

               9 Deploy Storage Node      SUCCEEDED

              12 Deploy Admin Service     SUCCEEDED

              13 Deploy Topo              ERROR

           

          --------------------------------------------------------------------------------

          --------------------------------------------------------------------------------

           

          kv-> show plan -id 13

          Plan Deploy Topo

          Id:                    13                           

          State:                 ERROR                        

          Attempt number:        1                            

          Started:               2020-02-24 02:59:32 UTC      

          Ended:                 2020-02-24 03:01:09 UTC      

          Plan failures:        

              Failure 1: 3/DeployNewRN on sn1(localhost:5000) failed.: Problem starting process for rg1-rn1:

          Java HotSpot(TM) 64-Bit Server VM warning: Failed to reserve shared memory. (error = 12)

          Java HotSpot(TM) 64-Bit Server VM warning: Failed to reserve shared memory. (error = 12)

          Java HotSpot(TM) 64-Bit Server VM warning: Failed to reserve shared memory. (error = 12)

          Java HotSpot(TM) 64-Bit Server VM warning: Failed to reserve shared memory. (error = 12)

          Java HotSpot(TM) 64-Bit Server VM warning: Failed to reserve shared memory. (error = 12)

          Java HotSpot(TM) 64-Bit Server VM warning: Failed to reserve shared memory. (error = 12)

          Java HotSpot(TM) 64-Bit Server VM warning: Failed to reserve shared memory. (error = 12)

          Process exiting due to fault

          java.lang.IllegalStateException: Could not install new version since disk limit reached

              at oracle.kv.impl.rep.VersionManager.checkCompatibility(VersionManager.java:123)

              at oracle.kv.impl.rep.RepEnvHandleManager.renewRepEnv(RepEnvHandleManager.java:813)

              at oracle.kv.impl.rep.RepNode.startup(RepNode.java:969)

              at oracle.kv.impl.rep.RepNodeService.start(RepNodeService.java:544)

              at oracle.kv.impl.rep.RepNodeService.start(RepNodeService.java:497)

              at oracle.kv.impl.sna.ManagedRepNode$1.execute(ManagedRepNode.java:266)

              at oracle.kv.impl.fault.ProcessFaultHandler.execute(ProcessFaultHandler.java:138)

              at oracle.kv.impl.sna.ManagedRepNode.start(ManagedRepNode.java:262)

              at oracle.kv.impl.sna.ManagedService.main(ManagedService.java:792)

          Exception creating service rg1-rn1: Could not install new version since disk limit reached (19.5.19): oracle.kv.FaultException: Could not install new version since disk limit reached (19.5.19)

          Fault class name: java.lang.IllegalStateException

          Remote stack trace: java.lang.IllegalStateException: Could not install new version since disk limit reached

              at oracle.kv.impl.rep.VersionManager.checkCompatibility(VersionManager.java:123)

              at oracle.kv.impl.rep.RepEnvHandleManager.renewRepEnv(RepEnvHandleManager.java:813)

              at oracle.kv.impl.rep.RepNode.startup(RepNode.java:969)

              at oracle.kv.impl.rep.RepNodeService.start(RepNodeService.java:544)

              at oracle.kv.impl.rep.RepNodeService.start(RepNodeService.java:497)

              at oracle.kv.impl.sna.ManagedRepNode$1.execute(ManagedRepNode.java:266)

              at oracle.kv.impl.fault.ProcessFaultHandler.execute(ProcessFaultHandler.java:138)

              at oracle.kv.impl.sna.ManagedRepNode.start(ManagedRepNode.java:262)

              at oracle.kv.impl.sna.ManagedService.main(ManagedService.java:792)

           

              at oracle.kv.impl.fault.ServiceFaultHandler.getThrowException(ServiceFaultHandler.java:140)

              at oracle.kv.impl.fault.ProcessFaultHandler.rethrow(ProcessFaultHandler.java:204)

              at oracle.kv.impl.fault.ProcessFaultHandler.execute(ProcessFaultHandler.java:140)

              at oracle.kv.impl.sna.ManagedRepNode.start(ManagedRepNode.java:262)

              at oracle.kv.impl.sna.ManagedService.main(ManagedService.java:792)

          (19.5.19)

          Total tasks:           9                            

          Successful:           2                            

          Failed:               1                            

          Not started:          6                            

           

          Failures:

             Task   3       ERROR at   2020-02-24 03:01:09 UTC: Plan 13 [Deploy Topo] task [DeployNewRN rg1-rn1 on sn1(localhost:5000)]

          Tasks not started

             Task Plan 13 [Deploy Topo] task [UpdateNthRNHelperHost]

             Task Plan 13 [Deploy Topo] task [NewNthRNParameters]

             Task Plan 13 [Deploy Topo] task [BroadcastTopo]

             Task Plan 13 [Deploy Topo] task [BroadcastMetadata]

             Task Plan 13 [Deploy Topo] task [AddPartitions totalPartitions=12]

             Task Plan 13 [Deploy Topo] task [BroadcastTopo]

          • 2. Re: cannot deploy topo --- it kept running for a long time and then it failed
            userBDBDMS-Oracle

            We have a new set of scripts to aid in deployment.  there are on our website.   You answer a couple of questions and it will deploy for you.    On the case above, it looks like you ran out of memory.

            • 3. Re: cannot deploy topo --- it kept running for a long time and then it failed
              Eric Jing

              Thank you for help.

               

              And, I got some questions regarding this:

               

              1. Is there any physical memory & virtual memory requirement for Oracle nosql database in test/development environment? (I mean kvstore database, not kvlite.)

               

              2. below picture is the screenshot of my virtual memory usage via "top", strange thing is that, the physical memory usage is very low, but the virtual memory usage by JVM 8 is very very high. And, the error msg in my previous post shows that the virtual memory is not enough for the JVM.

               

              I just wonder, the virtual memory usage in below picture is normal for oracle JDK 8 or not?Is there a way to fix this issue?

              3. do i have to set MALLOC_ARENA_MAX=1 on oracle linux 8.1 before starting the kvstore, is this a must?

              4. what is the compatible verision or suggested version of Oracle Linux for Oracle nosql db 19.5?

              5. what is the suggested disk space for each replication node in a shard?

               

              FYI, I made the test on a clean virtual machine via vmware. The only thing I installed is oracle JDK 8.

              • 4. Re: cannot deploy topo --- it kept running for a long time and then it failed
                Eric Jing

                and ,same issue occured (failed to reserve shared memory) even if I increase the swap size. (currently, the swap size is 15GB)

                • 5. Re: cannot deploy topo --- it kept running for a long time and then it failed
                  Eric Jing

                  Dear sir,

                  Could you provide me a script file for installing the nosql db on a normal computer with 6~8GB ram?

                  Thank you .

                  • 6. Re: cannot deploy topo --- it kept running for a long time and then it failed
                    userBDBDMS-Oracle

                    As mentioned before, scripts is on our web page, see top of page here:  https://www.oracle.com/database/technologies/nosql-database-server-downloads.html

                     

                    install instructions and pre-reqs are listed in our docs:  https://docs.oracle.com/en/database/other-databases/nosql-database/19.5/admin/installation-prerequisites.html

                     

                    setting of various paramaters such as MALLOC_ARENA_MAX is covered in the docs, admin guide chapter 1: https://docs.oracle.com/en/database/other-databases/nosql-database/19.5/admin/administrators-guide.pdf

                    • 7. Re: cannot deploy topo --- it kept running for a long time and then it failed
                      Eric Jing

                      I increased the disk space for kvstore, and it works well on my laptop test environment now.

                      I will also digest the admin guide to optimize various paramaters for the db.

                      Thank you so much.

                      • 8. Re: cannot deploy topo --- it kept running for a long time and then it failed
                        userBDBDMS-Oracle

                        If you could mark this as answered that would be appreciated.