1 2 Previous Next 21 Replies Latest reply: Dec 24, 2008 2:33 PM by 843785 RSS

    Odd behavior using jcraft sftp

    494490
      Please be patient as I am not a java developer but I am the one in the office now trying to diagnose the problem. We have started creating a process where we send XML in a text file to an online job posting site using SFTP. The job posting site monitors the directories on their site and automatically updates the listed jobs. We tested this successfully internally using the jcraft sftp package. The files are not large, around 3Mb. When trying to run the procedure (which is called from an Oracle concurrent program) the process hangs for about 15 minutes (it should complete in under a minute if it was working) and transmits only about 500k of the file. I can switch the parameters to an internal server and it will sftp it correctly. For the external site, the log file in Oracle shows the line 'Here 17' (see below, sorry it does not print the stacktrace in Oracle for some reason) which I think means the process is erroring out in c.put(new FileInputStream(f), f.getName()); If anyone is familiar with the jcraft package, I would greatly appreciate any input as to why it would only transmit a portion of the file than stop. I should tell you that I have sent a smaller file correctly to the external site also, just not the full 3Mb file. I also know that they can handle a file that size because I can SFTP it to them from the command line. Does anyone have any clues as to what I should check?
      import com.jcraft.jsch.*;
      import java.io.*;
      
      public class sendSftp {
      
          public static String send( String ftpDest, String port, String user, String password, String remoteDir, String file ) 
          {
              int ftpPort = 22;
              String step = "here 1";
              try {
                  
                  if ( ( ftpDest != null || !ftpDest.equals("") ) ||
                        ( user != null || !user.equals("") ) ||
                        ( password != null || !password.equals("") ) ||
                        ( file != null || !file.equals("") ) )
                     {
                         return " Failure null input"; 
                         
                     }
                        
                  
                  if ( port != null || !port.equals("") ) ftpPort = Integer.parseInt(port)  ;
                  step = "Here 2";
      
      
                  JSch jsch = new JSch();
                  step = "Here 3";
                  Session session = null;
                  step = "Here 4";
                  Channel channel = null;
                  step = "Here 5";
                  ChannelSftp c = null;
                  step = "Here 6";
      
                      session = jsch.getSession(user, ftpDest, ftpPort);
                      step = "Here 7";
                      session.setPassword(password);
                      step = "Here 8";
      
                      java.util.Properties config = new java.util.Properties();
                      step = "Here 9";
                      config.put("StrictHostKeyChecking", "no");
                      step = "Here 10";
                      session.setConfig(config);
                      step = "Here 11";
                      session.connect();
                      step = "Here 12";
                      channel = session.openChannel("sftp");
                      step = "Here 13";
                      channel.connect();
                      step = "Here 14";
                      c = (ChannelSftp)channel;
                      step = "Here 15";
      
                      if ( remoteDir != null || !remoteDir.equals("") ) c.cd(remoteDir);
                      step = "Here 16";
      
                      File f = new File(file);
                      step = "Here 17";
                      c.put(new FileInputStream(f), f.getName());
                      step = "Here 18";
               return "Success";
               
          }catch (Exception e )
          {
              String err = e.getStackTrace().toString();
              err = step + " Exception: " + err; 
              return err;
          }
          }
      }
      Edited by: sdhalepaska on Dec 24, 2008 12:14 PM
        • 1. Re: Odd behavior using jcraft sftp
          captfoss
          I've never used "the jcraft sftp package", but I do know it's very bad practice to create an object in a function call. Specificlly what I'm refering to is:
          c.put(new FileInputStream(f), f.getName());
          You really should create a variable to hold the FileInputStream, and then pass in the variable. I doubt that'll fix your problem, but it would be much, much better coding practice.

          :-)
          • 2. Re: Odd behavior using jcraft sftp
            494490
            captfoss wrote:
            I've never used "the jcraft sftp package", but I do know it's very bad practice to create an object in a function call. Specificlly what I'm refering to is:
            c.put(new FileInputStream(f), f.getName());
            You really should create a variable to hold the FileInputStream, and then pass in the variable. I doubt that'll fix your problem, but it would be much, much better coding practice.

            :-)
            Thanks for the info. I will make note of that. That line was actually taken from the example on this page: http://timarcher.com/?q=node/57

            I cannot believe that someone would put code on the intarweb that is not up to snuff. ;)
            • 3. Re: Odd behavior using jcraft sftp
              800282
              captfoss wrote:
              ... it's very bad practice to create an object in a function call. ...
              Why is that?
              • 4. Re: Odd behavior using jcraft sftp
                captfoss
                prometheuzz wrote:
                captfoss wrote:
                ... it's very bad practice to create an object in a function call. ...
                Why is that?
                I suppose it's bad practice moreso in C where it's a memory leak and actually hurts something, but I'm pretty sure it's still relatively frowned upon in Java. Makes the code harder to read when you have the new operation in the variable list of a function call, but I'd imagine mostly it's a holdover from C coding style.

                Plus, certain constructors will throw a null pointer exception, and that can be especially confusing if you've got code constructing objects in function calls ;-)
                • 5. Re: Odd behavior using jcraft sftp
                  800282
                  captfoss wrote:
                  prometheuzz wrote:
                  captfoss wrote:
                  ... it's very bad practice to create an object in a function call. ...
                  Why is that?
                  I suppose it's bad practice moreso in C where it's a memory leak and actually hurts something, but I'm pretty sure it's still relatively frowned upon in Java. Makes the code harder to read when you have the new operation in the variable list of a function call, but I'd imagine mostly it's a holdover from C coding style.
                  I don't know C (well enough) to comment on that. Let's stick to Java in this forum.
                  Anyway, in many cases it doesn't make code harder to read, IMO. Take the following examples (not taking autoboxing in account!):
                  Integer a = new Integer(2);
                  Integer b = new Integer(5);
                  Integer sum = add(a, b);
                  and
                  Integer sum = add(new Integer(2), new Integer(5));
                  then I prefer the second example.
                  captfoss wrote:
                  Plus, certain constructors will throw a null pointer exception, and that can be especially confusing if you've got code constructing objects in function calls ;-)
                  I don't understand this remark. What is this about constructors and NPE's?
                  • 6. Re: Odd behavior using jcraft sftp
                    843785
                    captfoss wrote:
                    prometheuzz wrote:
                    captfoss wrote:
                    ... it's very bad practice to create an object in a function call. ...
                    Why is that?
                    I suppose it's bad practice moreso in C where it's a memory leak and actually hurts something, but I'm pretty sure it's still relatively frowned upon in Java. Makes the code harder to read when you have the new operation in the variable list of a function call, but I'd imagine mostly it's a holdover from C coding style.

                    Plus, certain constructors will throw a null pointer exception, and that can be especially confusing if you've got code constructing objects in function calls ;-)
                    On the other hand, creating a reference that you don't need is also frowned upon. I don't see why it would be confusing if the constructor threw an exception, either you can read a stack trace, or you can't. I don't see much wrong with doing it, unless it really does make the code less readable
                    • 7. Re: Odd behavior using jcraft sftp
                      captfoss
                      prometheuzz wrote:
                      I don't understand this remark. What is this about constructors and NPE's?
                      Lemme try to come up with something along the lines of what I was hinting at...
                      FileDescriptor fd = null;
                      c.put(new FileInputStream(fd), f.getName());
                      c.put(new FileInputStream(new FileDescriptor()), "Blah blah blah");
                      Both of those chunks of code will throw a NPE in exactly the same place, and I suppose if you've got access to the stack trace they are equally easy to debug. However, if you're in a situation like the OP is in where you do not have access to the stack trace, one of the errors is obvious from the code and the other is not.

                      Granted, this is a stupid case, but my point was that the constructor to FileInputStream takes in an argument, and will throw a NPE if (1) the argument is null (2) it doesn't like a non-null argument.

                      This certainly could be confusing to a user. Doesn't have to be, sure, but I'd imagine a few people would spend a few minutes scratching their head. First thing I'd expect some people to do? Pull the "new" operations out and write the following code:
                      FileDescriptor fd = new FileDescriptor();
                      
                      if (fd == null) {
                          System.out.println("Screw me #1\n\n");
                      }
                      
                      FileInputStream fis = new FileInputStream(fd);
                      
                      if (fd == null) {
                          System.out.println("Screw me #2\n\n");
                      }
                      
                      c.put(fis , "Blah blah blah");
                      Which is how I'd have written it in the first place (minus the useless debugging statements, of course) because that's how I was taught to do it.

                      Regardless, I'll admit that there's technically nothing wrong with it in Java... but there is something wrong with it in C; so it's considered bad practice by some (especially those who do both ;-) and considered perfectly fine by some (especially those who never use C).
                      • 8. Re: Odd behavior using jcraft sftp
                        843785
                        Both of those chunks of code will throw a NPE in exactly the same place
                        No they won't. The second one cannot throw an NPE at all (unless FileDescriptor isn't the JDK file descriptor, and its default constructor is hard-coded to throw an NPE, of course)

                        Aslo, FileInputStream can indeed throw an NPE, but it will be evident that it's from the constructor of that class, not from where the constructor was invoked.
                        • 9. Re: Odd behavior using jcraft sftp
                          captfoss
                          georgemc wrote:
                          Both of those chunks of code will throw a NPE in exactly the same place
                          No they won't. The second one cannot throw an NPE at all (unless FileDescriptor isn't the JDK file descriptor, and its default constructor is hard-coded to throw an NPE, of course)
                          OOPS, misread the Javadoc. Let's assume it's the non-JDK one and the user doesn't realize that and all sorts of bad things happen.
                          Aslo, FileInputStream can indeed throw an NPE, but it will be evident that it's from the constructor of that class, not from where the constructor was invoked.
                          I admit to that, but it won't be evident from the code itself. I'm imagining a case where the user doesn't get ANY debug info except the crash itself, maybe a line number that comes from print statements, and all they have to go on is the code itself.

                          If I write my own File class, and my own FileInputStream, I could make it so the constructor of the File class throws a NPE if the file path is invalid, and the FileInputStream throws an NPE if the File object I give it is a text file and not a data file, for instance. If I don't get a stack trace, I have no idea where the problem is if I've nested constructors...

                          Again, even this scenario is unlikely in Java (but relatively common with a segmentation fault in C), but it could happen :-)
                          • 10. Re: Odd behavior using jcraft sftp
                            843785
                            captfoss wrote:
                            georgemc wrote:
                            Both of those chunks of code will throw a NPE in exactly the same place
                            No they won't. The second one cannot throw an NPE at all (unless FileDescriptor isn't the JDK file descriptor, and its default constructor is hard-coded to throw an NPE, of course)
                            OOPS, misread the Javadoc
                            Nothing to do with Javadoc, and everything to do with basic stack traces.
                            Aslo, FileInputStream can indeed throw an NPE, but it will be evident that it's from the constructor of that class, not from where the constructor was invoked.
                            I admit to that, but it won't be evident from the code itself. I'm imagining a case where the user doesn't get ANY debug info except the crash itself, maybe a line number that comes from print statements, and all they have to go on is the code itself.
                            You'll at least be told which class the line number applies to, in which case you can see if it's the code from which the constructor is called, or the constructor itself.
                            If I write my own File class, and my own FileInputStream, I could make it so the constructor of the File class throws a NPE if the file path is invalid, and the FileInputStream throws an NPE if the File object I give it is a text file and not a data file, for instance. If I don't get a stack trace, I have no idea where the problem is if I've nested constructors...
                            I've never seen the point of deliberately throwing an NPE in these circumstances. If it's invalid that null gets passed in, then you've got an IllegalArgumentException, or, if I'm firmly in JUnitland, I tend to use either Assert.assertNotNull(), or Assert.fail(). And why would you not have any idea where the problem is? Look again at your snippet:
                            c.put(new FileInputStream(fd), f.getName());
                            If it's fd that's null, the NPE comes from the constructor of FileInputStream. If it's f that's null, the NPE comes from the line posted. Whether you've got a full stack trace, or just the top line of it, that's evident
                            Again, even this scenario is unlikely in Java (but relatively common with a segmentation fault in C), but it could happen :-)
                            I'm unconvinced that porting coding styles between languages for such a tenuous reason is worthwhile!
                            • 11. Re: Odd behavior using jcraft sftp
                              494490
                              I know that this issue may be beyond the scope of this forum, but has anyone had the types of issues I described in the original post?
                              • 12. Re: Odd behavior using jcraft sftp
                                captfoss
                                georgemc wrote:
                                Nothing to do with Javadoc, and everything to do with basic stack traces.
                                Right. But I was reading the Javadoc where it says the constructor will throw a NPE, and I thought it said it would do that if the file descriptor went to a file that didn't exist, rather than a null file descriptor. Hense misreading the JavaDoc because I was temporarily stupid.
                                I'm unconvinced that porting coding styles between languages for such a tenuous reason is worthwhile!
                                It wouldn't really be porting styles so much as carrying over styles. Java is obviously based on C/C++ syntax, so 90% of the "coding style" of Java comes from C++ anyway ... I'd argue that the baseline is "Don't do that" and saying it's okay to do it actually the newest information.

                                Regardless, Java programmers wouldn't see a thing in the world wrong with it...but any decent C++ programmer will look at that and see a memory leak before they realize "Oh yeah, I'm programming in Java right now so I don't care..."

                                This really is a tow may tow / tom auto kinda thing... :-)
                                sdhalepaska wrote:
                                I know that this issue may be beyond the scope of this forum, but has anyone had the types of issues I described in the original post?
                                Yeah, sorry for going off-topic there. You might try my original non-suggestion, which would be to separate out the statement creating the file input stream, and see if you're failing on the c.put or in creating your stream (presumably during the c.put since you're sending some data, you said...)

                                Also, as a reference, I dug up the code for the ChannelSftp class. I'm not sure if you had it before, but here it is...
                                [http://kickjava.com/src/com/jcraft/jsch/ChannelSftp.java.htm]
                                • 13. Re: Odd behavior using jcraft sftp
                                  800308
                                  I know that this issue may be beyond the scope of this forum, but has anyone had the types of issues I described in the original post?
                                  Yeah... your tread got hijacked by some idjit ;-) and a crazy Scottsman ;-).... No real harm done...

                                  -----

                                  h3. So... To the original question

                                  Facts:
                                  1. You can "manually" sftp the same 3MB file to the same host sight
                                  2. The equivalent java program (as listed in the OP) fails, after ?15 minutes?

                                  Surmise:
                                  * You've definately got a problem somewhere in your java implementation, or environment. No news here!

                                  So which is it? Your code? Or the environment it runs in? I suspect it's environmental.

                                  So... I suggest you "exhume" your SFTP code into a simple command line java utility program (which might be useful besides, not the least as a test harness) and try it.

                                  If that works then we can say it's a problem with however the code is executed by oracle... I suspect that it's being time-sliced, and therebye starved of cpu-time... to the point where the "send-->[n]ack merry-go-round" never gets any momentum... to the point where vendor may keep reducing the transfer packet size (a common response to delayed-ack's, which is presumed to be the result of dirty-connection) in the hope of getting things moving along... unfortunately this is exactly the wrong thing to do when the slowness is caused by a cpu-starved client... The net result being that the transfer times out after 15 minutes, with only a small portion of the file transfered.

                                  Where there's will there's a hack!

                                  Cheers. Keith.
                                  • 14. Re: Odd behavior using jcraft sftp
                                    494490
                                    corlettk wrote:
                                    I know that this issue may be beyond the scope of this forum, but has anyone had the types of issues I described in the original post?
                                    Yeah... your tread got hijacked by some idjit ;-) and a crazy Scottsman ;-).... No real harm done...

                                    -----

                                    h3. So... To the original question

                                    Facts:
                                    1. You can "manually" sftp the same 3MB file to the same host sight
                                    2. The equivalent java program (as listed in the OP) fails, after ?15 minutes?

                                    Surmise:
                                    * You've definately got a problem somewhere in your java implementation, or environment. No news here!

                                    So which is it? Your code? Or the environment it runs in? I suspect it's environmental.

                                    So... I suggest you "exhume" your SFTP code into a simple command line java utility program (which might be useful besides, not the least as a test harness) and try it.

                                    If that works then we can say it's a problem with however the code is executed by oracle... I suspect that it's being time-sliced, and therebye starved of cpu-time... to the point where the "send-->[n]ack merry-go-round" never gets any momentum... to the point where vendor may keep reducing the transfer packet size (a common response to delayed-ack's, which is presumed to be the result of dirty-connection) in the hope of getting things moving along... unfortunately this is exactly the wrong thing to do when the slowness is caused by a cpu-starved client... The net result being that the transfer times out after 15 minutes, with only a small portion of the file transfered.

                                    Where there's will there's a hack!

                                    Cheers. Keith.
                                    Believe me, no problem taking over the thread, that is where I have made most of my revelations on this forum. Everything from 80s lyrics to government officials in Basingstoke. OK, here is what our additional testing has shown: the java code originally posted was modified to return more information to the Oracle concurrent program log and it gave us -- Exception: java.net.SocketException: send failed Broken pipe

                                    When we looked for this on the internet in regards to Oracle, it appeared to have something to do with a timeout on the server. Well, no help from our network admins so we tried some of the workarounds. Instead of calling the jcraft package with our small piece of java code, we implemented a shell script instead. This works fine every time. Truthfully I don't have the experience to tell you if it was an environment issue but it would appear to be since we could use our java code internally just fine. Thanks for the suggestions and help and off-topic wanderings.
                                    1 2 Previous Next