9 Replies Latest reply: May 8, 2013 3:50 PM by jschellSomeoneStoleMyAlias RSS

    Java 7 update 21 a technical, PR, and financial disaster

    1006982
      As CEO of Action Information Systems, I have spent the last few days dealing with the disaster of Javaq 7 update 21. I don't know where this should go but I'm sure someone will tell me...

      Our software is a set of Java applications (nothing web based) that are setup as a client/server architecture. The code was intially released under Java 1.4. We have to tightly integrate with the client's operating system to perform a lot of functions for the users. We have had to use the Runtime Exec() methods in at least 23 points in our code. Those many years ago, we put in the ability to automatically update the client code when the server code changed. This is a huge feature that has allowed us to keep the IT people happy so they don't have to handle client updates all the time.

      The changes to Runtime Exec() has just completely shutdown our software. Our customers, thinking the problem is a security issue, have spent thousands of dollars of their IT time trying to track this down before we even get a phone call. We spent a while trying to help them only to find out it was Java, update 21. Telling them to go back to build 17 requires them to either create a new account with Oracle, which upsets pretty much all of our customers, or they have to find a old version on the net, which freaks out their security people.

      This couldn't have come at a worse time. We have the team working overtime to get our next major release out. I have revenue tied to delivering this release. Now it turns out we have to pull everyone off Version 5 so we can make the changes and retest everything again just for, and I quote the release notes, "... to follow the specification more closely". it is amazing what a simple space character is costing us.

      This one change has shutdown our clients, caused a huge public releations disaster for us, and has put our financial future in jeopardy.

      This code has been running without problems or changes since the early 1.4 days. Without firing up CVS, I don't even know how many years ago that was. Note that the ProcessBuilder didn't even come out until Java 5 so we had no reason to make any changes.

      Apparently, to fix this we are going to have to go to all our customers and tell them they need to uninstall and reinstall all their clients. We had to do this once when we made a protocol change and it was painful even with the few customers we had early on. Now?

      PLEASE. Is there ANY way this change can be backed out and moved to the Java 8 release or some future upgrade so we can plan for this change? If we can figure out how to require the client upgrade when we go to version 5 of our software, I think I can sell that without the huge backlash I'm getting right now.

      Bruce Thompson
      CEO Action Information Systems

      ... back to the phones to try to calm down our customers
        • 1. Re: Java 7 update 21 a technical, PR, and financial disaster
          jtahlborn
          This certainly isn't the place for this discussion. many (most?) people on these forums are not oracle employees. I would suggest working through your oracle support contract to handle this kind of situation.

          As a side note, i assume that your problems are due to using "Runtime.exec(String)" with a single command line string which is now not parsed the same? You mention that ProcessBuilder didn't come out until later, however "Runtime.exec(String[])" (the correct method to use) has existed since 1.4.
          • 2. Re: Java 7 update 21 a technical, PR, and financial disaster
            baftos
            We have had to use the Runtime Exec() methods in at least 23 points in our code. Those many years ago, we put in the ability to automatically update the client code when the server code changed.
            Sh*t happens. Why not fix the 23+ points (it's probably the same issue everywhere) as fast as you can, fake a server code change and they will all get their clients updated.
            Maybe it's not so dramatic from your customers point of view. It looks terrible for you because you are answering all of them, but they don't know about each other.
            • 3. Re: Java 7 update 21 a technical, PR, and financial disaster
              gimbal2
              baftos wrote:
              We have had to use the Runtime Exec() methods in at least 23 points in our code. Those many years ago, we put in the ability to automatically update the client code when the server code changed.
              Sh*t happens.
              And in IT it happens a lot. Any company worth anything knows this and is equipped to deal with it in the "we're not liable" sense of the way.
              • 4. Re: Java 7 update 21 a technical, PR, and financial disaster
                Tolls
                jtahlborn wrote:
                As a side note, i assume that your problems are due to using "Runtime.exec(String)" with a single command line string which is now not parsed the same? You mention that ProcessBuilder didn't come out until later, however "Runtime.exec(String[])" (the correct method to use) has existed since 1.4.
                The classic JavaWorld "When Runtime.exec() Won't" article is from 2000, so predates 1.4.
                http://www.javaworld.com/jw-12-2000/jw-1229-traps.html?page=1

                In essence the code had a Bug In Waiting.
                It's just Sod's Law that it hit around the security change fixes.
                • 5. Re: Java 7 update 21 a technical, PR, and financial disaster
                  gimbal2
                  Tolls wrote:
                  It's just Sod's Law that it hit around the security change fixes.
                  I was just about to say "Don't you mean Murphy?". Luckily I googled it first. You learn something new every day.

                  Its indeed Sod/Murphy, but Oracle did make a minor change to Runtime.exec(), breaking backwards compatibility.

                  http://www.oracle.com/technetwork/java/javase/7u21-relnotes-1932873.html#jruntime

                  That is something Sun would have never done, they even preferred to leave things in a broken state just to maintain the compatibility. I'm not exactly certain about what I find a better strategy to be honest.
                  • 6. Re: Java 7 update 21 a technical, PR, and financial disaster
                    rp0428
                    Like the others I can sympathize with you to a certain point but this is still going to sound harsh. Shoot the messenger if it makes you feel better though. ;)
                    >
                    Those many years ago, we put in the ability to automatically update the client code when the server code changed. This is a huge feature that has allowed us to keep the IT people happy so they don't have to handle client updates all the time.
                    >
                    If that statement means that your IT people DID NOT TEST the updated version before allowing your clients to use it then your IT department has made a MAJOR mistake.

                    There is nothing wrong with 'automatic updates' but ANY update has to be throughly tested before you release your code. That means that your code should be designed to DETECT that an update is necessary (e.g. server code changed) and notify your client (or you) so the client can notify you.

                    Then YOUR company needs to do the appropriate testing to validate that the update will perform properly and only then allow the 'automatic update' to take place. Another issue you didn't mention is whose 'server code' are you talking about. Is the server code yours? If so then you know when it will be changing and can test that 'automatic update' process before releasing the updates to your multiple clients.

                    If the server code is another third-party then what happens when some of your clients update to a new server code version but other clients don't? That means that you now have some clients that are using a different version of YOUR client software than other clients. That can also be problematic.

                    If your process allows your clients to upgrade to new versions of software (i.e. Java) that your company hasn't tested, and if it was your company's decision (IT or otherwise) to allow updates to be made without testing then the responsibility for your predicament falls squarely on the shoulders of the manager that made that decision and, of course, on you and your company.

                    ALL updates needs to be tested (before deployment) using all of the components that are part of the SDLC (software development life-cycle). That may involve your company needing to obtain an updated server version to test with or, alternatively, work with one of your clients to get the testing done.
                    >
                    The changes to Runtime Exec() has just completely shutdown our software. Our customers, thinking the problem is a security issue, have spent thousands of dollars of their IT time trying to track this down before we even get a phone call. We spent a while trying to help them only to find out it was Java, update 21.
                    >
                    Which, of course, means that if your company has performed the testing BEFORE you allowed your customers to update you could have prevented all of this.
                    >
                    Telling them to go back to build 17 requires them to either create a new account with Oracle, which upsets pretty much all of our customers, or they have to find a old version on the net, which freaks out their security people.
                    >
                    Why do your clients need to create a new account with Oracle? If they had an account that allows them to download update 21 they can use it to download any of the older versions.

                    Prior releases of Java (including 7u17) are still available on Oracle's Official download site
                    http://www.oracle.com/technetwork/java/javase/downloads/java-archive-downloads-javase7-521261.html

                    Anyone that wants to download has to agree to the license restrictions but your clients should already have agreed to them.

                    What is it that 'freaks out their security people'? As I said above you and your clients can download Java 7u17 from exactly the same place you downloaded it from to begin with: the Official Oracle site. The only security issues that might affect you are the ones that the 7u21 release addresses.
                    >
                    This code has been running without problems or changes since the early 1.4 days. Without firing up CVS, I don't even know how many years ago that was. Note that the ProcessBuilder didn't even come out until Java 5 so we had no reason to make any changes.
                    >
                    Java 1.4 was introduced Feb 6, 2002 I believe and Java 1.5 about two years later. So 1.4 is over 10 years old and 1.5 close to it.

                    Since most fuctionality has been backward-compatible and you 'had no reason to make any changes' there may, indeed, not have been any substantial reason for your IT group to have proposed updating the Java version that you use.

                    So that part I can sympathize with.

                    But there have been substantial bug fixes between versions 1.4 and 1.5 as well as some major enhancements to specific fuctionality, JDBC for example. So it IS important for IT to always have a plan to migrate to the current version of software used in your products.
                    >
                    PLEASE. Is there ANY way this change can be backed out and moved to the Java 8 release or some future upgrade so we can plan for this change?
                    >
                    I assume that is a rhetorical question since I'm sure you already know the answer to that question.

                    If I were called in to consult on this matter (based ONLY on the info you provided in this thread) my recommendations would be:

                    1. Disable the automatic upgrade functionality of your software immediately. It would be irresponsible to allow this to happen again.

                    2. Meet ASAP with your IT manager with the intent of ordering that no updates are to be performed automatically OR without full SDLC testing.
                    Of course, there may be some work in progress already that needs to be taken into account. You need to make sure you get input from your IT manager to determine the most effective way to put that 'stop work' order into place.

                    3. Instruct your IT team to perform a post-mortem about what exactly happened that caused the mess to begin with. You have hinted at some of the reasons here but there is more to it than that. You have multiple clients. There is likely nothing that prevents one of them from updating to a new server version (if that code is not yours) ahead of other clients or before you have tested.

                    4. Reexamine and document your policy related to the support you provide your clients. Are all clients required to be using the SAME client and server versions? Or can some clients be using differenct client and/or server versions than other clients.

                    5. Intruct your IT team to propose changes needed to your SDLC related to: development of updates, testing of those updates and release of those updates to clients. You need to be proactive and make sure things are properly tested before release. You also need to be realistic. There may be a need for new hardware/software or even people resources to support the testing and deployment that needs to be performed.
                    • 7. Re: Java 7 update 21 a technical, PR, and financial disaster
                      939520
                      As an additional consideration: You may thoroughly test the code on your end before releasing it to customers. However, you can't be sure of your customer's environment and therefore it may still break for some of them. I suggest instead of allowing auto update for all customers at once, you only update a small subset of customers at a time (one initially, to verify your update methodology itself works) and email them to see if it went well (ie, they tested it). Ideally, update the customers that are least important to you first to minimize possible damage to your company's reputation.

                      A better strategy on your customer's end is that they should have a test environment set up (assuming they can afford it) to test such updates before allowing it in thier production environment. Also a roll-back strategy (including database backup) if it still causes problems in production. However, it's probably not wise to mention that to them after the fact.
                      • 8. Re: Java 7 update 21 a technical, PR, and financial disaster
                        EJP
                        I agree entirely with @rp0428. You're blaming others for your own company's errors. Software vendors are entitled to fix bugs any time they like, and they are under substantial market pressure to do so. This will break any code that relies on the bug. That's one of the many things that testing is for.
                        • 9. Re: Java 7 update 21 a technical, PR, and financial disaster
                          jschellSomeoneStoleMyAlias
                          1003979 wrote:
                          Those many years ago, we put in the ability to automatically update the client code when the server code changed. This is a huge feature that has allowed us to keep the IT people happy so they don't have to handle client updates all the time.
                          I suspect you might now understand why that isn't a good idea. Keep in mind that it can happen again.
                          That is why companies certify with one version and no other. And specifically indicate they do not support other versions. You can even program the application so it will not start unless the specific version is in use.

                          You can then choose to manually certify with new versions and then issue updates which allow the software to run with the versions that you know work.

                          >
                          have spent thousands of dollars of their IT time trying to track this down before we even get a phone call.
                          The solution to that is
                          1. Have a automated unit/system testing system.
                          2. Have it running all the time.
                          3. Have the system it runs on do automatic updates.
                          4. Hava failures notify support personel immediately.

                          The above system is SEPERATE from the normal development/QA systems and it only runs with production release(s).


                          Of course all of the above presumes that automated unit/system tests exists. If it is manual then it is going to cost (time and money) quite a bit more.

                          New builds have always had the risk of breaking something and have done so. I have seen more reported since the Sun acquisition by Oracle but I know that Sun once released a version that impacted XML processing in some cases severly.