12 Replies Latest reply: Apr 5, 2008 5:53 PM by 807591 RSS

    Split file such as exe

    807591
      To the expert in this forum:
      Here is a brief definition of distributed storage:
      Distributed Storage: Distributed storage is a technique of storing a single data set across multiple hosts. For example, if the information is distributed around a ring of computers and each computer is identical, in order to ensure fault tolerance (in case a particular node breaks down) each object will be stored in the system for multiple times.

      I just wonder if it is possible to spit a file (for example, 1 mp3 file split into 3 different segment) by using some java methods and store these segments into different computers for backup? When the data need to be restored, How can these 3 spit segments group together again into the original complete file?? Are there any java methods for this?

      Thank you for your attentions.

      Edited by: radeon on Apr 5, 2008 2:25 PM
        • 1. Re: Splite file such as exe
          807591
          You could just slice each file's contents into however many byte arrays you want to. This would defeat the purpose of distributed, fault tolerant storage though. Why do you want to split the files up?
          • 2. Re: Splite file such as exe
            807591
            Might be I mis-understood,:(

            For a distributed file system, My expression is ' in stead of a one to one back-up(like mirroring), split a piece of data(file) into a number of small segments and store them into different computers will save the storage capacity and increase the chance of recovery.'

            Is it correct? Please point out my mistake if you see any.
            • 3. Re: Splite file such as exe
              807591
              The problem I see here, is that if you only store one part of each file on one node, then the failure of that node will be catastrophic. I see however no big problem if you allow for redundancy here - i.e. if you store each part on more than one node. But I think you'd be going to too much trouble for too little gain. Just store each file in full on two or three nodes and you're good.
              • 4. Re: Splite file such as exe
                807591
                Thank you for your instant reply!! So you mean this is theoretical possible but will be very difficult/impossible to apply it in in real life?

                Once again, thank you for your answering.
                • 5. Re: Splite file such as exe
                  807591
                  It's not extremely difficult, but way more trouble than what I suggested and I don't really see the gains. Actually I see one case for your idea - if you have very large files, you might be able to retrieve them faster if you get the pieces from several places at once. Does this apply to your case?
                  • 6. Re: Splite file such as exe
                    807591
                    if u mean split an exe file, yes of course it is possible. But to run it, u will need all the splitted files. And you can split any file extensions, not just executables.

                    There are many programs out there to do this, open source and closed source. Most are free.

                    I have the source code for a program made in vb. I can provide link if u want.
                    • 7. Re: Splite file such as exe
                      807591
                      I guess the Zip function in java might help to solve the transmission speed... But what is you mean by huge file size? 100Mb, 1GB, or 10Gb? In which range of size will it be suitable for distributed storage(split the file)?or will see the different? I have to built a file transfer system for my university(for fun!!??) but I am confused.
                      • 8. Re: Splite file such as exe
                        807591
                        My gut feeling would be that you wouldn't gain anything from splitting up 100mb files, but once you approach 500 it might start to be worth it. Zip will not be that much help if you're transmitting binary data, but will be if it's text.
                        • 9. Re: Splite file such as exe
                          807591
                          Thank you for your reply. It would be grateful if you can put the link and allow me to look at.
                          • 10. Re: Splite file such as exe
                            807591
                            Thank you for correcting my mistakes!! I thought Zip class was very powerful as I read an article and it outline 'increase 20% transmission performance'....So it was a April fool joke, right?? What's the point of using zip class if it only help with the txt, doc??!!Once again, thank you for your reply.
                            • 11. Re: Splite file such as exe
                              807591
                              You're welcome. I think it wouldn't be a bad idea to do as VBKing advised and look if you can find an open source solution that does what you want. Even if it's not exactly to your requirements, you might be able to extend it for your needs.
                              • 12. Re: Splite file such as exe
                                807591
                                radeon wrote:
                                Thank you for your reply. It would be grateful if you can put the link and allow me to look at.
                                Here: [Fragit VB6 Code|http://planetsourcecode.com/vb/scripts/ShowCode.asp?txtCodeId=36427&lngWId=1]


                                Please note that it is made in VB6.
                                Also, it is not made by me,, but it works and very useful.