This discussion is archived
9 Replies Latest reply: Mar 19, 2012 9:38 AM by 796440 RSS

Is atomic rename still atomic in power cut scenarios

924454 Newbie
Currently Being Moderated
Hi,


I try to rename file atomically using Files.move() which has an ATOMIC_MOVE option.

It says it moves atomically.

Is it certain that it renames (moves) atomically even in power cut scenarios. Or in such situations can we get some invalid files etc.?


Thanks
  • 1. Re: Is atomic rename still atomic in power cut scenarios
    796440 Guru
    Currently Being Moderated
    921451 wrote:
    Is it certain that it renames (moves) atomically even in power cut scenarios. Or in such situations can we get some invalid files etc.?
    I don't know the answer to that question, but I would assume it's "No." Here's why: Even if it is guaranteed, there could arise some other scenario where it you can't count on it. Disk controller dies, head crash, JVM seg-fault, etc. Basically, if you're managing those files through your app, then you or your app is responsible for determining their integrity after a crash. If you've got a reasonable naming scheme or independent records of what the before and after files should be, it's easy enough to check.
  • 2. Re: Is atomic rename still atomic in power cut scenarios
    baftos Expert
    Currently Being Moderated
    921451 wrote:
    Hi,


    I try to rename file atomically using Files.move() which has an ATOMIC_MOVE option.

    It says it moves atomically.

    Is it certain that it renames (moves) atomically even in power cut scenarios. Or in such situations can we get some invalid files etc.?
    On Windows, it normally throws an IOException, but only when the power comes back. I don't know on other OS's:)
  • 3. Re: Is atomic rename still atomic in power cut scenarios
    924454 Newbie
    Currently Being Moderated
    jverd wrote:

    If you've got a reasonable naming scheme or independent records of what the before and after files should be, it's easy enough to check.
    In fact i have a reasonable naming scheme: The file names are constructed from numbers and i only change the defined extensions. But i do not have a record of names of files.

    e.g., rename "123.abc" to "123.qwe"

    In a crash scenario, i list files, i can check if the file names are constructed from numbers, and check their extensions. And also i can make some validity check about the content of the file. (But there are more than one case for a file to be valid. )

    In this scenario do you think i may be sure about the integrity of my files? (or may the o.s. generate an invalid file in the same format as my file name format, or may both of these two files exist (before and after rename), or in rename operation may some of the contents of my file be removed?)

    Thanks
  • 4. Re: Is atomic rename still atomic in power cut scenarios
    EJP Guru
    Currently Being Moderated
    or may the o.s. generate an invalid file in the same format as my file name format
    Why would it do that? You're the one generating files.
    or may both of these two files exist (before and after rename)
    Only if your application so wills it
    or in rename operation may some of the contents of my file be removed?)
    No. Rename only affects the name of the file.
  • 5. Re: Is atomic rename still atomic in power cut scenarios
    796440 Guru
    Currently Being Moderated
    921451 wrote:
    jverd wrote:

    If you've got a reasonable naming scheme or independent records of what the before and after files should be, it's easy enough to check.
    In fact i have a reasonable naming scheme: The file names are constructed from numbers and i only change the defined extensions. But i do not have a record of names of files.

    e.g., rename "123.abc" to "123.qwe"

    In a crash scenario, i list files, i can check if the file names are constructed from numbers, and check their extensions. And also i can make some validity check about the content of the file. (But there are more than one case for a file to be valid. )

    In this scenario do you think i may be sure about the integrity of my files? (or may the o.s. generate an invalid file in the same format as my file name format, or may both of these two files exist (before and after rename), or in rename operation may some of the contents of my file be removed?)

    Thanks
    If the atomic rename fails, due to some catastrophic failure outside of your program, you would either be left with 2 files or with 0 files. The 2 files case is trivial to check. If you don't have a separate record of the base file names, then you won't be able to tell if the failure caused your file to be deleted completely.

    Rename only affects the contents of a file if it has to be copied, such as when moving to a different file system. In this case, you should have been given the exception anyway, so there should be no issue.
  • 6. Re: Is atomic rename still atomic in power cut scenarios
    456795 Newbie
    Currently Being Moderated
    jverd wrote:
    921451 wrote:
    jverd wrote:

    If you've got a reasonable naming scheme or independent records of what the before and after files should be, it's easy enough to check.
    In fact i have a reasonable naming scheme: The file names are constructed from numbers and i only change the defined extensions. But i do not have a record of names of files.

    e.g., rename "123.abc" to "123.qwe"

    In a crash scenario, i list files, i can check if the file names are constructed from numbers, and check their extensions. And also i can make some validity check about the content of the file. (But there are more than one case for a file to be valid. )

    In this scenario do you think i may be sure about the integrity of my files? (or may the o.s. generate an invalid file in the same format as my file name format, or may both of these two files exist (before and after rename), or in rename operation may some of the contents of my file be removed?)

    Thanks
    If the atomic rename fails, due to some catastrophic failure outside of your program, you would either be left with 2 files or with 0 files. The 2 files case is trivial to check. If you don't have a separate record of the base file names, then you won't be able to tell if the failure caused your file to be deleted completely.

    Rename only affects the contents of a file if it has to be copied, such as when moving to a different file system. In this case, you should have been given the exception anyway, so there should be no issue.
    I not sure how you would get 2 or 0 files with a Journaling File System (http://en.wikipedia.org/wiki/Journaling_file_system) or Copy on Write File System (ZFS, Btrfs). Depending on exactly when the power failure (disk controller failure, OS hang, etc) happens you will see the original or the new name. A Journaling File System records ahead of time the move/rename, then does the actual move. If the power failure happens during the recording then the move never takes place. If the power failure happens after the recording in the journal, then on reboot the journal is replayed and the move is then performed.

    On a non-journaled file system you would have corrupted file system structures (inodes,indexes,etc) and your inode or index could be pointing anywhere on or off disk. And/Or you could have a totally corrupt name in the index so you may not even be able to do anything in your code to detect it.
  • 7. Re: Is atomic rename still atomic in power cut scenarios
    796440 Guru
    Currently Being Moderated
    Caffeine wrote:
    jverd wrote:
    921451 wrote:
    jverd wrote:

    If you've got a reasonable naming scheme or independent records of what the before and after files should be, it's easy enough to check.
    In fact i have a reasonable naming scheme: The file names are constructed from numbers and i only change the defined extensions. But i do not have a record of names of files.

    e.g., rename "123.abc" to "123.qwe"

    In a crash scenario, i list files, i can check if the file names are constructed from numbers, and check their extensions. And also i can make some validity check about the content of the file. (But there are more than one case for a file to be valid. )

    In this scenario do you think i may be sure about the integrity of my files? (or may the o.s. generate an invalid file in the same format as my file name format, or may both of these two files exist (before and after rename), or in rename operation may some of the contents of my file be removed?)

    Thanks
    If the atomic rename fails, due to some catastrophic failure outside of your program, you would either be left with 2 files or with 0 files. The 2 files case is trivial to check. If you don't have a separate record of the base file names, then you won't be able to tell if the failure caused your file to be deleted completely.

    Rename only affects the contents of a file if it has to be copied, such as when moving to a different file system. In this case, you should have been given the exception anyway, so there should be no issue.
    I not sure how you would get 2 or 0 files with a Journaling File System (http://en.wikipedia.org/wiki/Journaling_file_system) or Copy on Write File System (ZFS, Btrfs).
    I never said anything about either of those. I'm just laying out the possibilities of what might happen with Java's atomic rename IF a catastrophic failure like a power loss or head crash breaks the underlying FS's atomicity. I think it's safe to assume that the rename isn't truly atomic down to the finest quantum of time, that the atomicity only exists when viewing the multiple steps through the facilities provided by the FS.

    Given that, I can imagine two possible modes of failure:

    1) Create new name/path entry for the same file, then delete the old one. If the failure happens between the two, then both files exist.

    2) Remove the old name (but "remember" it in FS's memory or as a temp file name somewhere), then write the new name. If the failure happens midstream here, we won't see any file.
    Depending on exactly when the power failure (disk controller failure, OS hang, etc) happens you will see the original or the new name.
    So, Java's atomic rename IS guaranteed to give you either the before or after even in those failure situations? I didn't know that, and my comments were addressing what might happen if the failure was severe enough to break the atomicity.
  • 8. Re: Is atomic rename still atomic in power cut scenarios
    456795 Newbie
    Currently Being Moderated
    jverd wrote:
    Caffeine wrote:
    jverd wrote:
    921451 wrote:
    jverd wrote:

    If you've got a reasonable naming scheme or independent records of what the before and after files should be, it's easy enough to check.
    In fact i have a reasonable naming scheme: The file names are constructed from numbers and i only change the defined extensions. But i do not have a record of names of files.

    e.g., rename "123.abc" to "123.qwe"

    In a crash scenario, i list files, i can check if the file names are constructed from numbers, and check their extensions. And also i can make some validity check about the content of the file. (But there are more than one case for a file to be valid. )

    In this scenario do you think i may be sure about the integrity of my files? (or may the o.s. generate an invalid file in the same format as my file name format, or may both of these two files exist (before and after rename), or in rename operation may some of the contents of my file be removed?)

    Thanks
    If the atomic rename fails, due to some catastrophic failure outside of your program, you would either be left with 2 files or with 0 files. The 2 files case is trivial to check. If you don't have a separate record of the base file names, then you won't be able to tell if the failure caused your file to be deleted completely.

    Rename only affects the contents of a file if it has to be copied, such as when moving to a different file system. In this case, you should have been given the exception anyway, so there should be no issue.
    I not sure how you would get 2 or 0 files with a Journaling File System (http://en.wikipedia.org/wiki/Journaling_file_system) or Copy on Write File System (ZFS, Btrfs).
    I never said anything about either of those. I'm just laying out the possibilities of what might happen with Java's atomic rename IF a catastrophic failure like a power loss or head crash breaks the underlying FS's atomicity. I think it's safe to assume that the rename isn't truly atomic down to the finest quantum of time, that the atomicity only exists when viewing the multiple steps through the facilities provided by the FS.

    Given that, I can imagine two possible modes of failure:

    1) Create new name/path entry for the same file, then delete the old one. If the failure happens between the two, then both files exist.

    2) Remove the old name (but "remember" it in FS's memory or as a temp file name somewhere), then write the new name. If the failure happens midstream here, we won't see any file.
    Depending on exactly when the power failure (disk controller failure, OS hang, etc) happens you will see the original or the new name.
    So, Java's atomic rename IS guaranteed to give you either the before or after even in those failure situations? I didn't know that, and my comments were addressing what might happen if the failure was severe enough to break the atomicity.

    I was taking the definition ATOMIC_MOVE from the javadocs: http://docs.oracle.com/javase/7/docs/api/java/nio/file/Files.html#move(java.nio.file.Path, java.nio.file.Path, java.nio.file.CopyOption...)


    ATOMIC_MOVE:     The move is performed as an atomic file system operation and all other options are ignored. If the target file exists then it is implementation specific if the existing file is replaced or this method fails by throwing an IOException. If the move cannot be performed as an atomic file system operation then AtomicMoveNotSupportedException is thrown. This can arise, for example, when the target location is on a different FileStore and would require that the file be copied, or target location is associated with a different provider to this object.


    Since the ATOMIC_MOVE is relying on an atomic file system operation to perform the move I would conclude the OP doesn't need to worry about a power failure for this operation. Of course if his application is writing to those files he will need to do his own journalling to protect the integrity of the files, like databases do, encase of a power failure.
  • 9. Re: Is atomic rename still atomic in power cut scenarios
    796440 Guru
    Currently Being Moderated
    >
    \> I was taking the definition ATOMIC_MOVE from the javadocs: http://docs.oracle.com/javase/7/docs/api/java/nio/file/Files.html#move(java.nio.file.Path, java.nio.file.Path, java.nio.file.CopyOption...)
    >
    >
    ATOMIC_MOVE:     The move is performed as an atomic file system operation and all other options are ignored. If the target file exists then it is implementation specific if the existing file is replaced or this method fails by throwing an IOException. If the move cannot be performed as an atomic file system operation then AtomicMoveNotSupportedException is thrown. This can arise, for example, when the target location is on a different FileStore and would require that the file be copied, or target location is associated with a different provider to this object.


    Since the ATOMIC_MOVE is relying on an atomic file system operation to perform the move I would conclude the OP doesn't need to worry about a power failure for this operation.
    That's the part I was unsure about--how strong the "atomic file system operation" guarantee is. I figured since these are platform-agnostic Java docs, it was better not to assume that the underlying operation would be atomic outside the context of the operation completing in the Java code.

Legend

  • Correct Answers - 10 points
  • Helpful Answers - 5 points