Forum Stats

  • 3,825,006 Users
  • 2,260,455 Discussions
  • 7,896,381 Comments

Discussions

what would happen if FRA location became unavailable?

JayG30
JayG30 Member Posts: 15
edited Apr 5, 2019 11:39AM in Recovery Manager (RMAN)

I'll start with admitting I'm not a DBA. I work for a company that has a vendor software that is built on Oracle 10g, running on a Windows Server.

They have RMAN backups with the FRA location to a separate set of local disks. Disk space is not sufficient and backups are failing. Unfortunately adding or replacing additional storage to this server is going to be an issue. I've proposed attaching iSCSI storage from a storage server on our network that they could move their RMAN and FRA backups to. This would then physically allow me to remove, replace, and rebuild some of the local disk storage on the server.

I'm being told that they are concerned about the FRA not being on "local disks". And that if this location was to become disconnected that the database would stop and crash because the FRA location would be unavailable. Is this true? Is the FRA so tightly connected to the live database that it can take it down? I would expect the backup tools to not be able to take down the live database. Our FRA is 100's of GB in size.

Thanks.

Bo AJayG30

Best Answer

  • Dude!
    Dude! Member Posts: 22,829 Black Diamond
    edited Mar 26, 2019 4:52PM Answer ✓

    Archivelogs normally go to FRA. If archivelogs cannot be written, the database will not crash but freeze until the situation is cleared. Using remote storage for FRA based on TCP/IP networking can be troublesome and bears some risks regarding performance and network. No advice an be given, because I don't have a crystal ball to see your environment and requirements.

    FRA should be at least 3 times the size of the database, depending on traffic and size of backups. If you can afford an Oracle Database license, I cannot see where the problem could be to install larger disks. 100 GB by today's standards is peanuts - it's like a floppy disk considering, what are you using? You can't even buy such small disks anymore.

    Bo AJayG30JayG30
«13

Answers

  • Bo A
    Bo A Member Posts: 122 Blue Ribbon
    edited Mar 26, 2019 4:02PM

    There are a couple of questions there. From my experience, If FRA get full, it will bring the database down (hang).

    I did have similar issues with smaller size FRA.

    So what i did was still kept FRA (for archivelogs and flashback logs only, i.e. less space) put all the backup to datadomain which is mounted as NFS. (Simple RMAN configuration command would do the trick). You can easily set FRA to different local disk using "db_create_online_dest" system parameter. Then copy all the archivelogs there. Then "catalog recovery area" (to recognized all the archivelogs under new destination).

    1, Attached another local disk enough to hold archivelogs for a few days. "relocate" FRA to this local disk. (changing db_create_online_dest)

    2, RMAN configuration to send backup to "proposed attaching iSCSI storage from a storage server on our network"  (CONFIGURE CHANNEL DEVICE TYPE DISK FORMAT   '/usr/local/oracle/backup/RMAN/TEST/%d_%U_%I_%T_%t'; in my case /usr/local/oracle/backup is NFS mount for datadomain. Might as well put controlfile autobackup there also temporary)

    3, You can physically rebuild,recreate, expanded volumes on original device "FRA location".

    4, Then set db_create_online_dest to original FRA location, cp backover archivelogs (maybe backup too) and rerun catalog recovery area.

    This is just recommendation. You should discuss this the DBAs there also.

    JayG30JayG30
  • Dude!
    Dude! Member Posts: 22,829 Black Diamond
    edited Mar 26, 2019 4:52PM Answer ✓

    Archivelogs normally go to FRA. If archivelogs cannot be written, the database will not crash but freeze until the situation is cleared. Using remote storage for FRA based on TCP/IP networking can be troublesome and bears some risks regarding performance and network. No advice an be given, because I don't have a crystal ball to see your environment and requirements.

    FRA should be at least 3 times the size of the database, depending on traffic and size of backups. If you can afford an Oracle Database license, I cannot see where the problem could be to install larger disks. 100 GB by today's standards is peanuts - it's like a floppy disk considering, what are you using? You can't even buy such small disks anymore.

    Bo AJayG30JayG30
  • Dude!
    Dude! Member Posts: 22,829 Black Diamond
    edited Mar 26, 2019 5:01PM

    Btw, if your server isn't 20 years old, you can probably replace SAS with SATA drives - controllers are usually compatible, meaning you can put cheaper SATA into SAS, but not SAS into SATA. But if you go for SATA, choose Enterprise grade drives. You can easily get a 2 TB hard drive, that's 200 times larger than what you have now, for a couple of hundred bucks.

  • JayG30
    JayG30 Member Posts: 15
    edited Mar 27, 2019 11:37AM

    The issue with the disk replacement is more complex then just "buy more disks". This is where the DBA's unwillingness to get on the phone with us (even though we pay them monthly support) frustrated me so much dealing with this vendor. Everything has "no" with no further discussion with the DBA's making these comments, all while the disks are filling up and nobody being able to provide any meaningful solution.

    The storage issue is that the server that the VENDOR supplied when everything was originally purchased (and before my employment) made poor decisions. They used a hardware RAID controller that doesn't support online expansion and other features. You also can't mix disk types in a virtual device meaning you can't just move from SAS to SATA. They installed disks in all 8 bays (all of tiny size 15k drives), resulting in 4 x RAID1 arrays. This means that I can't install extra disks and can't upgrade existing disk. Cloning the disks in the array and trying to rebuild it sounds like a recipe for disaster. This leaves full system image backup (it's a windows server mind you), but they managed to setup this machine in a way that prevents system image backups from being performed (it's a Dell server and I have to dig into my notes for the cause but think it has to do with a bad OEM partition or something).

    So any solution will require downtime. Some solutions more downtime then others. And some solutions considerably more work.

    The server in question is at a remote site, requiring planning for me to get to and nobody onsite capable of handling it.

    The site is also a small laboratory that runs around the clock and downtime is very difficult to get.

    And finally, in the next 6 months I'm beginning the complete overhaul of the entire environment. Including moving this particular server and application into a virtual environment, and most likely some form of replication/semi-HA for situations like this.

    So my "idea" was to clear data off the only set of disks I THOUGHT was not "critical" to the operation of the live database (the backup location). Then, that would free me to delete that RAID1 group and replace the disks in question. The live database is currently split across 2 RAID1 partitions (D and E). D is becoming full as well and is made of the smallest disks. So next step would be to move D over to these new larger disks. Then I could destroy the RAID1 configured of these disks and add 2 more large disks (4 larger disks in total). Those disks would create the "new" local disk backup location. Sounds like a lot of work but actually a lot less and less likely for potential disaster then full backup/restore IMO.

    My long term concern however is that this Oracle database requires so much "local disk" storage to essential "hold backups". To me, a backup isn't a backup if it is on the same storage as the primary. A backup should reside on external storage and be replicated offsite to 2 different places for a total of 3 backups. Also, the DBA's asking for TB's worth of "local disk storage" in a virtual environment will ultimately cause me issues. The storage used to run VM's will be high speed enterprise storage that isn't cheap in the slightest (to date our Oracle licensing hasn't been expensive but will likely need to change, but again the "VENDOR" handles all this supposedly). The notion of eating up that valuable storage for backups is crazy to me, especially when ultimately to have to truly backed up you'd need to "duplicate it" off that storage effectively doubling it. Then factor in backups of the ACTUAL VM and not JUST RMAN backups as well.

    The bigger confusing to me from DBA's is this "concern" about "network storage". Am I to believe that all you oracle DBA's don't have customers running in a VMWare environment? Pretty much every VMWare ESXi environment I've seen ends up with compute nodes with no real "local storage" and separate storage nodes. That storage is presented to ESXi over NFS or iSCSI. If storage nodes become "unavailable" all the VM's freeze. Sure, to you inside the VM it might "look" like a local disk, but I'd say 99% of the time your data is transmitting over a network (shared or dedicated SAN).

  • CristianR-Oracle
    CristianR-Oracle Member Posts: 497 Employee
    edited Mar 27, 2019 11:46AM

    You might want to look into Oracle Secure Backup (tapes) as well.

  • JayG30
    JayG30 Member Posts: 15
    edited Mar 27, 2019 11:55AM

    So looking at what they have written to the current "backup disk" in question I see;

    backup_ora

         -DBnameHOT (empty)

         -archive (empty)

         -dpdump (10GB)

         -flash_recovery_area (191GB)

              --DBname

                   ----ARCHIVELOG (33.5GB)

                   ----AUTOBACKUP (29.8MB)

                   ----BACKUPSET (158GB)

                   ----ONLINELOG (50MB)

    Oracle

         -DBname

              --controlfile.ctl

    So by what @boracIe is saying sounds like some of this could be move OFF to external storage without consequence. But some of it SHOULD remain alongside the same storage as the primary database because it basically needs to be just as available as the live database.

    By what he says ARCHIVELOG (33.5GB) would want to remain. I'm not sure what in this constitutes the "flashback logs".

    And I'm guessing the "backups" he was moving to his NFS storage (by datadomain do you mean Dell/EMC?) constitutes the large "BACKUPSET"? How about "dpdump"?

    the other reason I'm interested in this would be with a setup where I can push some of this RMAN backup data off local storage and to a storage server on our network, I would be able to provide a lot more slower storage in our new environment so they could hold MORE recovery points.

    Finally, can you even setup RMAN backups to an NFS storage if you are running the database on Windows? This is why I actually just carved out a iSCSI LUN instead.

  • Dude!
    Dude! Member Posts: 22,829 Black Diamond
    edited Mar 27, 2019 12:18PM

    You can use iSCSI or NFS, but like I said before, it increases the odds, depending on the network? Do you have a spearate network that you can use for storage? If not, you are running at high risk of bad performance and failure. You can probably manage using NFS for the RMAN backups, but it might be a good idea to use local storage for archivelogs.

    How about optimizing and reducing the backup data. Are backups compressed? Are archivelogs deleted after backup? There may be some options left that could buy you more time if you can reduce the requried disk space.

    How about attaching external storage to the server? Does the server have an external SAS/SCSI port? Perhaps you can attach a 1 TB USB drive or use a 128 GB flash stick to offload some data, like older backups.

    JayG30
  • JayG30
    JayG30 Member Posts: 15
    edited Mar 27, 2019 12:42PM

    Good questions. All things I'd HOPE the vendors DBA's have taken into consideration but my experience to date with them gives me little confidence. Part of why I came here to ask some questions so I could understand Oracle backups/RMAN/FRA better since my reading wasn't cutting it. Like I said, I'm not a Oracle DBA. I've had some experience with MySQL and I can write some SQL queries, but noting like this.

    No, unfortunately not external port. I thought about an external DAS enclosure but again downtime and expense when I'm replacing everything.

    Maybe they could just move the archivelogs over to one of our other local disks. Like the E drive which does hold some of the live database, specifically scanned "attachments" (PDF, TIFF, etc). If I remember correctly the recommendation is for it to be on a separate partition, but might be smarter then the network storage as the short term solution?

    Your questions speak to part of my frustration with the vendor. It's their software, which uses Oracle as the database backend. They are paid for support of this. Of course they don't support the "infrastructure" around it (server, network, etc). But they also hold all the credentials. So I can't even check how things are configured. And every time I look into storage usage on this server I find horribly un-optimized setup, people not cleaning up after themselves, etc and then trying to say "just buy more storage". I feel like I'm feeding a drug addict at times, rewarding bad behavior. For instance they turned on debugging for a tool and it chewed up 60+GB of data. Or the 160GB file that looks like some old backup that nobody bothered to remove. Or a folder with literally a million small tiny files that weren't needed but never get deleted.  So, submit a ticket to their support, wait sometimes a month for a general tech to respond, they then try to relay the question onto a DBA who I don't usually get to speak with directly, and then usually end up with a half answer lacking detail or a complete misunderstanding. This is like .01% of my actual job, but it's been taking 40% of my time. Not good.

  • Bo A
    Bo A Member Posts: 122 Blue Ribbon
    edited Mar 27, 2019 1:27PM

    backup_ora

         -DBnameHOT (empty)

         -archive (empty)

         -dpdump (10GB)                                        --this i'm not sure, perhaps datapump area.

         -flash_recovery_area (191GB)                  --crazy small FRA for both backup pieces and other files, must be a small database or rman retention policy must be very narrow

              --DBname

                   ----ARCHIVELOG (33.5GB)           --piggy back on what @Dude! said, you can use local storage and point FRA there

                   ----AUTOBACKUP (29.8MB)         --controlfile and spfile autobackup can be "directed" to NFS for the RMAN backups

                   ----BACKUPSET (158GB)             --backup pieces can be "directed" to NFS for the RMAN backups

                   ----ONLINELOG (50MB)               --In my environment, i kept them in another diskgroup call +REDO, but nothing stopping your from using "local storage" FRA

    Oracle

         -DBname

              --controlfile.ctl

    If you do however decided move the FRA to new "local strorage" and backup pieces to "NFS", make sure you catalog them.

    "can you even setup RMAN backups to an NFS storage if you are running the database on Windows"

    This i don't know because i was lucky enough to have never dealt with Oracle on Windows. This will probably be my pitfall if i ever have to do it one day.

    After all that, you could "physically allow me to remove, replace, and rebuild some of the local disk storage".

    JayG30JayG30
  • Bo A
    Bo A Member Posts: 122 Blue Ribbon
    edited Mar 27, 2019 1:28PM

    flashback logs, if they don't have it they don't have it. Typically, if you have flashback database enabled, flashback logs will live in the FRA also.