This discussion is archived
1 2 3 Previous Next 31 Replies Latest reply: Mar 21, 2012 4:59 PM by jschellSomeoneStoleMyAlias RSS

double primitive or Double is being corrupted(chopping off numbers)

859049 Newbie
Currently Being Moderated
Something is breaking double throughout all applications on our server. This has happened three times in the last month on 3 different servers. I don't know how long the problem has been going on.

System.out((double)35.25) produces 30.0
System.out.(new BigDecimal(35.25).doubleValue) produces 30.0
Any number as a double is chopping off all digits except the first.

Anything using double is broken. I know that we should be using BigDecimal and we are in most places, but our code-base has been around since 2000.

We are using Java 1.6.0_07. ( I know there are later versions, but our applications are numerous and support a large client base. We have not had time to upgrade ). I have checked the changelogs for the newer versions and did not see any fixes related to this problem.
We are running Tomcat 5.5 on a linux box.

There were no reported errors in application or tomcat logs to indicate there was a problem.

I realize this is not much to go on, but can anyone tell me how it is possible to corrupt the JVM?
Is there something I can check when the problem happens to give me some indication of cause?

Restarting tomcat fixes the problem. I don't think this is a tomcat problem though.
  • 1. Re: double primitive or Double is being corrupted(chopping off numbers)
    796440 Guru
    Currently Being Moderated
    user10916752 wrote:
    Something is breaking double throughout all applications on our server. This has happened three times in the last month on 3 different servers. I don't know how long the problem has been going on.

    System.out((double)35.25) produces 30.0
    System.out.(new BigDecimal(35.25).doubleValue) produces 30.0
    No, they don't, as they won't even compile. Can you provide an [url http://sscce.org]SSCCE that demonstrates the actual problem?
    Restarting tomcat fixes the problem. I don't think this is a tomcat problem though.
    This sounds very fishy. Double/double and their conversions to String is a pretty fundamental thing that has been working pretty much since day 1. If corruption is actually occurring, it would have to be a pretty esoteric case. Something of the nature you describe would be huge and would be jumped on immediately if it occurred in general (if it ever made out the door in the first place, which is highly unlikely).
  • 2. Re: double primitive or Double is being corrupted(chopping off numbers)
    859049 Newbie
    Currently Being Moderated
    Sorry, my example was not correct. I cannot reproduce this problem, it is extremely intermittant ( 3 times in a month ). But when the problem does occur, I can reproduce it all day long.

    Correction.
    <%System.out.println((double)35.25);%> embedded in a jsp produced 30.0 in the logs
    <%=((double)35.25)%> embedded in the html of a jsp produces 30.0 on the displayed html/jsp page.

    I can do:
    double myNightmare = 35.25 + 15.0; // output the result and get 50.0
    I can use DecimalFormat on the result and still get 50.0;
    Anything to do with double is corrupted.


    I also agree, very fishy and disturbing. Again, I don't know what is causing the corruption, and cannot reproduce the problem until it becomes corrupt. That is my nightmare.
    We have put into place a process that will alert us as soon as the problem happens using nagios.

    Thanks for you response.

    Edited by: user10916752 on May 2, 2011 12:13 PM
  • 3. Re: double primitive or Double is being corrupted(chopping off numbers)
    796440 Guru
    Currently Being Moderated
    user10916752 wrote:
    Sorry, my example was not correct. I cannot reproduce this problem, it is extremely intermittant ( 3 times in a month ). But when the problem does occur, I can reproduce it all day long.

    Correction.
    <%System.out.println((double)35.25);%> embedded in a jsp produced 30.0 in the logs
    <%=((double)35.25)%> embedded in the html of a jsp produces 30.0 on the displayed html/jsp page.

    I can do:
    double myNightmare = 35.25 + 15.0; // output the result and get 50.0
    I can use DecimalFormat on the result and still get 50.0;
    This is the exact, actual code that you have? Not just an example? You literally put
    <%System.out.println((double)35.25);%>
    in your JSP?

    It's System.out, not a logger? And it's the literal 35.25, not just some double result of a calculation or something?

    If this is indeed the case, then I suspect that the problem is with formatting, not with actual double values.

    Add the following:
    <%System.out.println((double)35.25 == 35.25);%>
    <%System.out.println((double)35.25 == 30.0);%>
    (Also, note that the cast of 35.25 to double is unnecessary.)

    Some code may be calling System.setOut() to some custom PrintStream that "cleverly" formats all doubles to the nearest 10.

    If you get true/false for the above additional print statements, then this is almost certainly the case. If you do not get true/false, then yeah, it would appear there's some serious corruption of the JVM.

    Another thing you can do is when your app starts up, make a local copy of System.out.
    private static final PrintStream myOut = System.out;
    And use myOut.println() in conjunction with your existing println()s. If it gives different results, then the original System.out (which myOut still references) is not the same as what System.out is currently pointing to.

    Restarting fixes this because, of course, we're back to the original System.in, until that replacement call is made again.

    Of course, if you don't literally have exactly
    <%System.out.println((double)35.25);%>
    in your JSP, then you need to more precisely communicate your problem.

    You can also print out the result of Long.toHexString(Double.doubleToLongBits(35.00)) and Long.toHexString(Double.doubleToLongBits(35.25)), and compare those to the expected values 4041a00000000000 and 4041800000000000. That will be further evidence of whether double values are actually being corrupted, or just their formatting.

    Another possibility may be that whatever formatter System.out uses was replaced, or perhaps some code was messing with Locales. I'm not familiar with and haven't dug into the details of all the pieces that come into play when you call System.out.println(double), but that's a path you can pursue.

    Edited by: jverd on May 2, 2011 12:48 PM
  • 4. Re: double primitive or Double is being corrupted(chopping off numbers)
    796440 Guru
    Currently Being Moderated
    import java.io.OutputStream;
    import java.io.PrintStream;
    
    public class EvilOut {
      private static final PrintStream realOut = System.out;
    
      public static void main (String[] args) throws Exception {
    
        System.out.println(35.25);
        realOut.println(35.25);
    
        EvilPrintStream evilOut = new EvilPrintStream (System.out);
        System.setOut (evilOut);
        
        System.out.println(35.25);
        realOut.println(35.25);
      }
    
      static class EvilPrintStream extends PrintStream {
    
        public EvilPrintStream (OutputStream out) {
          super (out);
        }
    
        public void println(double x) {
          super.println (Math.floor (x / 10) * 10);
        }
      }
    }
    
    35.25
    35.25
    30.0
    35.25
    Edited by: jverd on May 2, 2011 12:54 PM
  • 5. Re: double primitive or Double is being corrupted(chopping off numbers)
    859049 Newbie
    Currently Being Moderated
    When the problem happened the last time, I did not know WHAT was going on. We were getting a lot of reports from users that numbers were not correct. They referred to them as rounding errors. So when it happened this last time, I created a jsp on the live box for the purpose of debugging. The code that I show you is for my debugging purposes only. Placing <%System.out.println((double)35.25);%> does produce output in the tomcat log catalina.out. Placing <%=((double)35.25)%> in my debug jsp did produce 30.0 when displayed.

    When I put the same jsp on another server(not having the problem) I got good/expected results (35.25).

    Note that I am only casting to a double because double seemed to be the root cause. During my debugging, I also tried BigDecimal and float. No problems.
    Although if I used myBigDecimal.doubleValue(), it did produce the problem. Anything to do with double was causing the problem.

    In regards to comments on System.out, in lue of all of the reports by users having incorrect numbers ( not necessarily output using System.out ), I don't think that this has any baring. Also, my one debug to embed the number on the jsp <%=((double)35.25)%> does not use System.out. Thanks for the suggestion though.

    As you mentioned, formatting could be at the root of the problem, but would have to be at a very core level. Numbers throughout the applications were being displayed incorrectly. These incorrect numbers are displayed in many areas. PDFs, applet screens, jsp/html. I have found no common formatting code that could be the culprit.

    Unfortunately, the problem does not currently exist. I will have to wait for the corruption to happen again before I can try anything else.
  • 6. Re: double primitive or Double is being corrupted(chopping off numbers)
    859049 Newbie
    Currently Being Moderated
    Thank you for the EvilOut code, and sorry to add more details as I go, but this problem is not limited to 10's. 2,250,375 also becomes 2,000,000.
    It is almost like it is using scientific notation 2.250375E6 and chopping off the decimals.
  • 7. Re: double primitive or Double is being corrupted(chopping off numbers)
    796440 Guru
    Currently Being Moderated
    user10916752 wrote:
    As you mentioned, formatting could be at the root of the problem, but would have to be at a very core level. Numbers throughout the applications were being displayed incorrectly. These incorrect numbers are displayed in many areas. PDFs, applet screens, jsp/html. I have found no common formatting code that could be the culprit.
    Look into the docs and code for println() and see what it does. Track down Locales, Formatters, etc. Just like System.out, if some code replaces one that's used app-wide, then many or all displayed results will be affected.
    Unfortunately, the problem does not currently exist. I will have to wait for the corruption to happen again before I can try anything else.
    Add the code I suggested to look at the actual double values without having to worry about formatting. That way it will be in place next time the problem occurs, and you'll then know whether it's a formatting problem or actual bad values. Then you can decide how to proceed from there.
  • 8. Re: double primitive or Double is being corrupted(chopping off numbers)
    796440 Guru
    Currently Being Moderated
    user10916752 wrote:
    Thank you for the EvilOut code, and sorry to add more details as I go, but this problem is not limited to 10's. 2,250,375 also becomes 2,000,000.
    It is almost like it is using scientific notation 2.250375E6 and chopping off the decimals.
    Yeah, that was just an example of how something like that could happen. But it could still be a rogue formatter, just not one that always rounds to 10s.
  • 9. Re: double primitive or Double is being corrupted(chopping off numbers)
    859049 Newbie
    Currently Being Moderated
    Thanks again for your suggestions and time. I will use your code the next time am able to debug.
  • 10. Re: double primitive or Double is being corrupted(chopping off numbers)
    859049 Newbie
    Currently Being Moderated
    Unfortunately, if it were some rogue code, it would seem that the problem would persist/happen all the time.
    As soon as I restart tomcat( which also restarts the java instance ), the problem goes away.

    Bad problem!
    Thanks again.
  • 11. Re: double primitive or Double is being corrupted(chopping off numbers)
    796440 Guru
    Currently Being Moderated
    user10916752 wrote:
    Unfortunately, if it were some rogue code, it would seem that the problem would persist/happen all the time.
    As soon as I restart tomcat( which also restarts the java instance ), the problem goes away. \
    No, it still fits. The rogue formatter doesn't replace the default one immediately at startup, but eventually some code gets exercised that hadn't been up to that point, the default formatter is replaced, and things are messed up until a restart.

    Note, too, that when I say "rogue" here, I don't mean it's necessarily malicious. Some junior developer may have thought he was being clever and improving "efficiency", not realizing how widespread the effect of his change would be.
  • 12. Re: double primitive or Double is being corrupted(chopping off numbers)
    859049 Newbie
    Currently Being Moderated
    I'll take that into consideration, but at this point it would have to be malicious from an external source. We've not upgraded java for a long time, and no one here with the exception of me would even know how to do what you are suggesting. Also, we make no modifications to java core source.

    Baffled, but you have given me some things to think about. I should have some more ammo for the next round of debugging.
  • 13. Re: double primitive or Double is being corrupted(chopping off numbers)
    796440 Guru
    Currently Being Moderated
    user10916752 wrote:
    I'll take that into consideration, but at this point it would have to be malicious from an external source. We've not upgraded java for a long time, and no one here with the exception of me would even know how to do what you are suggesting. Also, we make no modifications to java core source.
    No new versions of thirdparty libraries? No new version of tomcat? No other apps running under that tomcat?

    Even if all you did was change seemingly unrelated, high-level code that you personally wrote or vetted, it's possible that 10 layers deep, it caused the execution of a code path that had been laying dormant for years. That exact thing happened to me last week. We had a method that was declared to return List<String>, but that just created and populated an unparameterized List of Files. That code hadn't been touched for years. There were warning highlights in the IDE, and I assume warnings on the console at compile time, but apparently nobody paid attention to them amidst all the others. I made one very small change to the order in which certain files are processed at startup time, and suddenly this code which had never been called before (or had perhaps always just returned and empty list), was getting invoked and giving us ClassCastExceptions miles away from where the list was actually populated.
    Baffled, but you have given me some things to think about. I should have some more ammo for the next round of debugging.
    Yeah, it seems weird indeed. Usually the weirder these things seem at first, the more pedestrian the real culprit turns out to be, once found.
  • 14. Re: double primitive or Double is being corrupted(chopping off numbers)
    859049 Newbie
    Currently Being Moderated
    No tomcat, lynux-related or third party libraries updated for quite some time. We deploy application changes all the time. That was the first place that I looked.

    I see what you are saying about code that has been dormant for some time kicking in. The problem that I am having is that I don't know where to begin to look. We have 13 servers in a farm with 5 core applications running, a couple of thousand application options, thousands of users. Three different servers had the problem in the last month. Maybe there is some option out there that is rarely taken that is causing the problem to kick in.

    I am just going to have to continue to dig and dig and dig.
1 2 3 Previous Next

Legend

  • Correct Answers - 10 points
  • Helpful Answers - 5 points