10 Replies Latest reply on Sep 4, 2017 8:52 AM by Beauty_and_dBest

    EBS on different servers

    Beauty_and_dBest

      Hi ALL,

       

      EBS R12.1.3

      OL6

      11gR2

       

       

      We have two identical EBS R12.1.3 intances (the other is exact clone), residing on two different server locations.

      Both server hardwares have also identical in specs CPU,  Memory, Disk.

      But the other server can be accessed faster and performance is okay,

      while the other has issues, its performance is erratic, sometimes performance is okay but sometimes its hangs on reports ( I do not know if its network connection or its performance disk I/O)?

      I am suspecting its the latency? or disk I/O? How can I compare the stats performance for the two? What commands do I run to get the difference?

       

       

      Please help...

       

       

      Kind regards,

      jc

        • 1. Re: EBS on different servers
          Jibu-Oracle

          Hi,

           

          Since you mentioned the servers are identical, You could verify output of RDA ( <rda> -T hcve ) referring following Note from both servers

          RDA - Health Check / Validation Engine Guide (Doc ID 250262.1)

           

          Compare the versions of Technology stack components like Forms, iAS, Framework, JDK, OJSP, Database, etc

          Utility /Script To Check The Techstack Component Versions (Forms, Http Server, JDK, Framework, Database, etc) (Doc ID 601736.1)

           

          For system wide performance you can refer following Note to review the output which also cover various aspects

          SRDC - Data Collection for E-Business Suite Performance (Doc ID 1920979.1)

           

          I hope this should help to understand more details about the case

           

          Thanks

          Jibu

          • 2. Re: EBS on different servers
            Beauty_and_dBest

            Thanks Jibu,

             

             

            Both EBS instances are also identical since one is an exact clone of the other.

            The only thing I suspect is, the network or the storage which may have different performance?

            Both servers are on different remote site.

             

             

            Kind regards,

            • 3. Re: EBS on different servers
              Jibu-Oracle

              Hi,

               

              If you are particularly doubtful about network or the storage I/O. You could get a snapshot of the same from those servers  for periods which you find performance issue with help of Storage and OS administrator.

              I hope it could help on your analysis. to get a detailed review on those areas

               

              Thanks

              Jibu

              • 4. Re: EBS on different servers
                Beauty_and_dBest

                Thanks Jibu and ALL

                 

                Below is the iostat of the server that has performance issue, Can you help me identify if there are I/O bottlenecks?

                 

                [root@oracle ~]# iostat 10

                Linux 3.8.13-118.14.2.el6uek.x86_64 (oracle.com)       08/31/2017      _x86_64_        (8 CPU)

                 

                avg-cpu:  %user   %nice %system %iowait  %steal   %idle

                           2.28    0.00    0.15    2.41    0.01   95.15

                 

                 

                avg-cpu:  %user   %nice %system %iowait  %steal   %idle

                           0.21    0.00    0.11    0.84    0.01   98.82

                 

                Device:            tps   Blk_read/s   Blk_wrtn/s   Blk_read   Blk_wrtn

                xvdb              2.50         1.60        59.20         16        592

                xvdc              1.50         0.00        35.20          0        352

                dm-0              0.00         0.00         0.00          0          0

                dm-1              2.10         0.00        20.00          0        200

                dm-2              2.20         0.00        39.20          0        392

                 

                avg-cpu:  %user   %nice %system %iowait  %steal   %idle

                           0.36    0.00    0.03    1.51    0.00   98.10

                 

                Device:            tps   Blk_read/s   Blk_wrtn/s   Blk_read   Blk_wrtn

                xvdb              4.90         0.00        88.80          0        888

                xvdc              2.60         2.40        48.00         24        480

                dm-0              0.00         0.00         0.00          0          0

                dm-1              3.30         0.00        32.80          0        328

                dm-2              3.40         0.00        56.00          0        560

                 

                avg-cpu:  %user   %nice %system %iowait  %steal   %idle

                           1.70    0.00    0.06    1.08    0.00   97.16

                 

                Device:            tps   Blk_read/s   Blk_wrtn/s   Blk_read   Blk_wrtn

                xvdb              4.00         0.00        89.60          0        896

                xvdc              6.00         1.60       108.00         16       1080

                dm-0              0.00         0.00         0.00          0          0

                dm-1              1.30         0.00        12.80          0        128

                dm-2              4.90         0.00        76.80          0        768

                 

                avg-cpu:  %user   %nice %system %iowait  %steal   %idle

                           2.12    0.00    0.26    0.79    0.00   96.83

                 

                Device:            tps   Blk_read/s   Blk_wrtn/s   Blk_read   Blk_wrtn

                xvdb              2.60         0.00        65.60          0        656

                xvdc              5.20        24.80        80.00        248        800

                dm-0              0.00         0.00         0.00          0          0

                dm-1              1.20         0.00        15.20          0        152

                dm-2              3.60         0.00        50.40          0        504

                 

                avg-cpu:  %user   %nice %system %iowait  %steal   %idle

                           2.08    0.00    0.14    1.65    0.00   96.13

                 

                Device:            tps   Blk_read/s   Blk_wrtn/s   Blk_read   Blk_wrtn

                xvdb              4.00         1.60        90.40         16        904

                xvdc              7.80         6.40       119.20         64       1192

                dm-0              0.00         0.00         0.00          0          0

                dm-1              1.50         0.00        16.80          0        168

                dm-2              4.90         1.60        73.60         16        736

                 

                avg-cpu:  %user   %nice %system %iowait  %steal   %idle

                           0.48    0.00    0.04    1.20    0.01   98.27

                 

                Device:            tps   Blk_read/s   Blk_wrtn/s   Blk_read   Blk_wrtn

                xvdb              5.60         0.00       118.40          0       1184

                xvdc             10.10        92.00       102.40        920       1024

                dm-0              0.00         0.00         0.00          0          0

                dm-1              1.80         0.00        20.80          0        208

                dm-2              6.10         0.00        97.60          0        976

                Capture.PNG

                 

                 

                Kind regards,

                • 5. Re: EBS on different servers
                  Jibu-Oracle

                  The reported values are not high to reflect any impact on performance

                  May be need to drill down to other areas mentioned in earlier as well

                  • 6. Re: EBS on different servers
                    Prashant Umap

                    Hello,

                     

                    Thanks for the post. I would suggest you to get strace from both the server and compare. (You can take a help from Server Admin).

                    This will show you network traffic/Disk IO comparison.

                     

                    Regards

                    Prashant Umap

                    • 7. Re: EBS on different servers
                      stejac

                      does you storrage is on that machine or on another machine / san

                      • 8. Re: EBS on different servers
                        Aldrich

                        Two thoughts:

                         

                        Thought 1)  Are the server times set correctly on the slow one?  Are they synched with NTP?

                         

                        Thought 2)  Do you have enough entropy?  "cat /proc/sys/kernel/random/entropy_avail" should have a value between 2,000-4,000.

                        • 9. Re: EBS on different servers
                          Beauty_and_dBest

                          Hi Aldirch, stejac, jibu, prasant and ALL,

                           

                           

                           

                          Both server storage have hi-end SAN, I monitored both servers and there are no bottlenecks on I/O.

                           

                          Aldrich you are brilliant, I mean all of you here

                           

                          What is the command used for thought 1? What do you mean by server times? Do you mean timezone?

                           

                          [opc@prod1 ~]$ date

                          Sun Sep  3 13:23:50 PHT 2017

                           

                          [opc@prod1 ~]$ cat  /etc/sysconfig/clock

                          ZONE="Asia/Manila"

                          UTC=true

                           

                          [ec2-user@prod2 ~]$ date

                          Sun Sep  3 13:23:46 PHT 2017

                           

                          [ec2-user@prod2 ~]$ cat  /etc/sysconfig/clock

                          ZONE="Asia/Manila"

                           

                           

                           

                          For thought 2,

                           

                          [root@prod1 ~]# cat /proc/sys/kernel/random/entropy_avail

                          270

                           

                          [root@prod2 ~]#  cat /proc/sys/kernel/random/entropy_avail

                          899

                           

                           

                          Prod2 has larger entropy hence its performance is good?

                          How do I increase entropy for Prod1? Do I just edit it manually?

                           

                          The question is, I did not set those values or It was set by default. How come the prod2 has higher default value?

                           

                           

                          Kind regards,

                          • 10. Re: EBS on different servers
                            Beauty_and_dBest

                            Hi ALL,

                             

                            Correction please....Sorry, I forgot to mention that PROD1 is OL6 while PROD2 is RHEL6

                            Does RHEL6 has larger entropy than OL6?

                             

                             

                             

                            Kind regards,