12 Replies Latest reply on Jul 2, 2012 2:56 PM by user258571

    Archiver Process using paging space on AIX

    845567
      Hi everyone

      Hope that I'm in the right forum to post this threat.
      We are running Oracle database (10g/11g) on AIX 6.1 and we are seeing a high paging space usage although the server memory does not seem to be overcommitted (telling from SGA and PGA settings and usage). The servers are dedicated database servers so no other application (besides system-mgmt) is running.

      What I see on the server is that as soon as there is workload on the database the paging space seems to slowly be eaten up. Looking at the main processes using paging space it always turns out to be the ora_arc and ora_lgwr processes using paging space and I don't have a clue why this is happening.

      Anybody got an idea?

      Thanks
      Marco
        • 1. Re: Archiver Process using paging space on AIX
          12cdb
          Did you check with your OS vendor? they might give you some idea or can help you.
          • 2. Re: Archiver Process using paging space on AIX
            845567
            As I could not find any other similar issues on metalink or in any forum I thought that this just might be an issue of database parameters... something I just did not get so far.
            Especially since it's always about arc and lgwr processes. So in first place I don't consider it to be a OS-issue.
            • 3. Re: Archiver Process using paging space on AIX
              845602
              Hi,
              Are you getting page-out in your system? Page-in is fine.

              How do you determine arch and lgwr using your paging space?

              If you seeing page out during heavy load, my suspect is your PGA.

              check "total PGA allocated" on v$pgastat while you having page out.

              you can also view memory allocation for each process on your instance on v$process

              SELECT PROGRAM, PGA_USED_MEM, PGA_ALLOC_MEM, PGA_FREEABLE_MEM, PGA_MAX_MEM
              FROM V$PROCESS

              If you really notice your system running out memory, review your SGA, whether it is over configured.
              • 4. Re: Archiver Process using paging space on AIX
                845567
                Hi
                Looking at nmon also shows me page in's, yes.
                My suspicion of arc and lgwr processes being the paging users is owing to the output of the command "svmon -U -t 10 -O process=on -O sortentity=pgsp"
                The top processes are always either arc_ oder lgwr_ processes.
                I cannot say when exactly the paging occurs. I noticed that after rebooting three servers last night the usage of paging space started this morning around the time users started to work. It then gradually uses more paging space on the system until using up to 70 oder 80% of the paging space. (where paging space is in between 1 - 1.5 times memory)

                I think that you hit the point in having a closer look at pga sizing as this parameter is definitely undersized in our databases in general (owing to available memory resources). But if this really is the issue, why am I seeing archiver and logwriter processes on top of paging space consumers?

                On the other hand I would say that sga sizing is not too much of an issue.

                Anyway, thanks for your reply!
                • 5. Re: Archiver Process using paging space on AIX
                  845602
                  Hi,

                  Why not you provide your swap space info, paging activity and AWR report with atleast 20 minutes snapshot while you having heavy load.

                  If suspect the arch lgwr, did you check how much pga memory allocated on for these 2 background process in v$process?
                  • 6. Re: Archiver Process using paging space on AIX
                    845567
                    Hi

                    It finally turned out that, adding up the memory requested by all databases (SGA / PGA and server-sessions), there was an over-allocation of physical memory and it showed up in lgwr- and arc-processes keeping paging space allocated.

                    Thanks for your posts!!
                    Marco
                    • 7. Re: Archiver Process using paging space on AIX
                      Markus1004
                      Hi Marco,

                      I have a similar problem. AIX is paging in and ARC-processes are the top memory consumers. However, I don't have an over-allocation of physical memory. The archive-processes allocate ~ 100MB each of additional heap memory (not in shared memory areas). How did you solve the problem? Just added memory? Did you workout, why you're archiver processes consumed most of the memory?

                      Thanks for your response in advance!
                      Markus
                      • 8. Re: Archiver Process using paging space on AIX
                        sb92075
                        Markus1004 wrote:
                        Hi Marco,

                        I have a similar problem. AIX is paging in and ARC-processes are the top memory consumers.
                        All virtual memory Operating Systems are designed to page RAM.
                        It is normal behavior & does not indicate any problem.
                        Some process will ALWAYS be "top memory consumer".
                        You suffer from Compulsive Tuning Disorder.

                        every *NIX OS tends to mis-report memory consumption for Oracle related processes.

                        Simply put, the system is NOT broken & no change is required!
                        • 9. Re: Archiver Process using paging space on AIX
                          Markus1004
                          sb92075 wrote:

                          All virtual memory Operating Systems are designed to page RAM.
                          It is normal behavior & does not indicate any problem.
                          I know that. And as soon as paging takes place, system performance goes down significantly. Based on the application we are running, response times of more than a minute ARE A PROBLEM - even if the database server keeps running.
                          Some process will ALWAYS be "top memory consumer".
                          You suffer from Compulsive Tuning Disorder.

                          every *NIX OS tends to mis-report memory consumption for Oracle related processes.
                          I know that too. At least "+ps+ " mis-reports memory consumption. But if you are using other OS-specific tools, such as "+svmon+ " on AIX, you get detailed information about every memory chunk a process uses. And here you can see exactly(!) how much heap memory the archiver processes have allocated (shared memory areas, program areas, stack space etc. are reported separately). It also shows, how much of the allocated is in Real memory, in Pagespace and how much Virtual memory was allocated for each memory chunk! Apparently, this tool is very informative and a more accurate than any PGA-Information Oracle is keeping internally. But even Oracle reports 40-50 MB pga_alloc_mem for the archiver processes, while all other processes only need about ~1-5 MB.
                          Simply put, the system is NOT broken & no change is required!
                          It is normal behavior & does not indicate any problem.
                          Well, I agree in so far as the system is NOT broken. I never said it was. And if it was, I would have contacted oracle-support. But a change IS required, because paging is causing performance issues and must be avoided!
                          I was just wondering, why the archiver processes need so much more memory than the rest. And I was curious wether simply adding more memory was sufficient in Marco's case, or if he did anything else to avoid paging. I hope for some more in-depth information, that's helping me to reach a decision. That's why I replied on this thread in the first place.

                          - regards
                          Markus
                          • 10. Re: Archiver Process using paging space on AIX
                            user258571
                            Hi,
                            i've the same problem.
                            I've 4 database 11.2 with 4 archiver process on each db.
                            System administrator reports to me high paging in this machine.
                            • 11. Re: Archiver Process using paging space on AIX
                              sb92075
                              842366 wrote:
                              Hi,
                              i've the same problem.
                              I've 4 database 11.2 with 4 archiver process on each db.
                              as POGO once declared, "We have met the enemy & they is US!"

                              If you stop poking yourself in the eye, the pain will stop.

                              why do you have 4 DBs on same system?
                              why do you have 4 archiver processes for each DB?
                              • 12. Re: Archiver Process using paging space on AIX
                                user258571
                                sb92075 wrote:
                                842366 wrote:
                                Hi,
                                i've the same problem.
                                I've 4 database 11.2 with 4 archiver process on each db.
                                as POGO once declared, "We have met the enemy & they is US!"

                                If you stop poking yourself in the eye, the pain will stop.

                                why do you have 4 DBs on same system?
                                why do you have 4 archiver processes for each DB?
                                :D

                                We have 4 databases on the same system because some one, and i don't know why, has created them.
                                We have 4 archiver processes because dbca created them.