10 Replies Latest reply on Mar 29, 2008 2:24 AM by jtahlborn

    Java doesn't pick up system's DNS settings change until restarted

    843790
      Hello,

      I have a service running on a few Linux computers. Those computers have a NIC, which is configured with a fixed IP address. So /etc/resolv.conf contains the IP address of the LAN's DNS server. But most of the time, the computers are not plugged into the LAN at all. Instead, they connect themselves periodically to the Internet by establishing a ppp connection. When it happens, the ISP's DHCP server assign them new IP parameters, including their DNS server's IP address. And /etc/resolv.conf gets updated with this address.

      But it seems that java doesn't take care of DNS change taking place during a run. Which means that my program (started at boot with no connectivity at all) tries to connect to some host, and obviously trigger an "UnknownHostException" (at that point it does try to contact the LAN's DNS server). Quite logical. Later, the ppp link become active. But my program still try to contact the LAN's DNS server, despite the new configuration in /etc/resolv.conf. As such, it will forever trigger UnknowHostExceptions, until it gets restarted (it will then pick up the new settings) or it is plugged back into the LAN (it will finally reach the LAN's DNS server).

      This is quite a problem as during one single execution, the machine may encounter several DNS configuration changes, and this problem basically means that it will be impossible for my application to resolve any name at all.

      So is there a way to tell Java to re-read the system wide DNS configuration? Or is there some option to prevent Java to "cache" the DNS server to use?

      To demonstrate my problem, I've written a simple test case, see below.

      To get the problem:

      1) Put a bogus DNS server into your /etc/resolv.conf
      2) Start the test program. Wait for some time: it will trigger UnknownHostExceptions.
      3) Fix the entry in /etc/resolv.conf, and check it actually works (eg ping www.google.be)
      4) Test program will continue to trigger UnknownHostExceptions forever.

      One interesting fact is that someone tried this test on Windows, and didn't suffer from this behaviour, eg the application reacts to DNS system settings changes dynamically. So it looks like a Linux-only problem.

      Thanks in advance for your insight.
      Pierre-Yves
      package com.test.dnsresolver;
       
      import java.net.InetAddress;
      import java.net.UnknownHostException;
       
      public class DnsResolver {
       
          private static String urlString = "www.google.com";
       
          public static void main(String[] args) {
       
              /**
               * Specified in java.security to indicate the caching policy for successful
               * name lookups from the name service. The value is specified as as integer
               * to indicate the number of seconds to cache the successful lookup.
               */
              java.security.Security.setProperty("networkaddress.cache.ttl" , "10");
               
              /**
               * Specified in java.security to indicate the caching policy for un-successful
               * name lookups from the name service. The value is specified as as integer to
               * indicate the number of seconds to cache the failure for un-successful lookups.
               * A value of 0 indicates "never cache". A value of -1 indicates "cache forever".
               */
              java.security.Security.setProperty("networkaddress.cache.negative.ttl", "0");
               
              int loopCounter = 0;
              while (true) {
                  InetAddress resolved;
                  try {
                      resolved = InetAddress.getByName(urlString);
                      System.out.println("Loop " + loopCounter + ": resolved IP address: " + resolved);
                  } catch (UnknownHostException e) {
                      System.out.println("Loop " + loopCounter + ": UnknownHostException");
                      e.printStackTrace();
                  }
                  loopCounter++;
                  try {
                      Thread.sleep(1000);
                  } catch (InterruptedException e) {}
              }
               
          }
       
      }
        • 1. Re: Java doesn't pick up system's DNS settings change until restarted
          843790
          Hello,

          No answer at all so far... I'm wondering, perhaps am I asking for something stupid?

          In the meantime, I've implemented a working workaround to manually resolve all names against the right server at the right time, but I have the feeling I shouldn't have to resort to such hacks. Any comments?

          Thanks in advance.
          • 2. Re: Java doesn't pick up system's DNS settings change until restarted
            jtahlborn
            http://java.sun.com/j2se/1.5.0/docs/guide/net/properties.html
            • 3. Re: Java doesn't pick up system's DNS settings change until restarted
              843790
              Apart from the 3 JNDI DNS service provider settings properties which I didn't know about and are interesting, the informations in this link do not seem relevant to the problem described above: why does Java names resolution break if the system's DNS server to use change while the application is running?
              • 4. Re: Java doesn't pick up system's DNS settings change until restarted
                jtahlborn
                Oh, i didn't notice that your code was already setting the ttl properties. did you test your code setting those properties on the command line? some properties do not have any affect if set at runtime.
                • 5. Re: Java doesn't pick up system's DNS settings change until restarted
                  843790
                  The problem described here is not about DNS results caching. It's about some kind of broken "DNS server to query" caching.

                  Please re-read the whole stuff. I know I probably made it too long, but I wanted to be as precise and complete as possible.

                  Thanks.
                  • 6. Re: Java doesn't pick up system's DNS settings change until restarted
                    jtahlborn
                    I understand what you are saying. what i'm saying, is that your test code does not prove that it is a "dns server caching" problem instead of a simple "dns lookup caching" problem. see the referenced bug report for many complaints that java does not honor these settings when set prgrammatically, as well as comments about using sun specific properties instead of the "generic" ones:

                    http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=6247501

                    if you can prove that the dns lookup is still happening incorrectly after the server change, then it really is a "dns server caching problem". then you might want to try futzing with some of the "*.nameservice.*" properties.
                    • 7. Re: Java doesn't pick up system's DNS settings change until restarted
                      843790
                      Ok, I understand your point.

                      This very same test case can also be used in the "real" situation (which I obviously did), thus proving that it is not a "dns lookup caching" problem.

                      1) Connect the system to the LAN (and only to the LAN, no ppp link at this point). Check that network access and DNS are OK from command line. Start the test case: names are resolved as expected
                      2) Bring up the the ppp link, which will become the default route, and also update /etc/resolv.conf. Do not disconnect the LAN: names are still resolved so far.
                      3) Now unplug the LAN cable. Check from command line that system-wide network access and DNS is still fine (it should be). Check the test-case output: name resolution is now failing. As we get a failure, and we were getting a result earlier, it is safe to tell that positive DNS lookups are not cached.
                      4) Reconnect the LAN cable (not touching the ppp link). Names will be successfully resolved again. As we now get results and were previously getting failure, it is also safe to tell that negative DNS lookups are not cached either

                      My conclusions:
                      a) There is no dns lookup caching taking place in this case, as expected
                      b) For the whole run duration, java tries to resolve names against the LAN's DNS server (the one in use at the beginning), despite the system configuration changing mid-run. This is an obvious deduction as we see that name resolution fails at step 3), but works again at step 4), just by restoring physical network access to this DNS server.

                      Do you better see the point?
                      • 8. Re: Java doesn't pick up system's DNS settings change until restarted
                        jtahlborn
                        Yep.

                        Have you looked at any of the nameservice properties. i think with one you can specify a list of dns servers. you could try specifying the lan and ppp dns servers in that property and see if that works (obviously not ideal solution).

                        i looked into the java code which instantiates the nameserver (at least in jdk 1.6), and it looks like it caches the data from the resolv.conf for 300 seconds (and there is no way to change this config). looks like the only way to get a faster reaction would be to find an alternate NameServer implementation (you can specify your own impl with sun.net.spi.nameservice.provider).
                        • 9. Re: Java doesn't pick up system's DNS settings change until restarted
                          843790
                          Well, the nameservice property allowing to specify my DNS server of choice is interesting (I didn't know about those), but not very usable, as the DNS server to use is not known in advance (it may be whatever the ISP tells me to use at the time where the ppp link gets established). So no real solution there.

                          The fact that it caches /etc/resolv.conf content for 300s is very interesting, but having no possibility to impact on this duration really is a pity. There should be some kind of property to fix this behaviour. So as you say, a custom provider may be the only solution.

                          So far, the hack I use to get this working is based on code similar to this one (it is presented here in a similar form than my test case above). Obviously, reading the /etc/resolv.conf for each dns resolution is not an option in a real environment, but you get the idea.
                          package com.test.dnsresolver;
                          
                          import java.io.BufferedReader;
                          
                          public class DnsResolver {
                               private static final String urlString = "www.google.com";
                               private static final String resolvConf = "/etc/resolv.conf";
                          
                               public static void main(String[] args) {
                          
                                    int loopCounter = 0;
                                    while (true) {
                                         loopCounter++;
                                         try {
                                              Thread.sleep(1000);
                                         } catch (InterruptedException e) {}
                          
                                         // Parse the current DNS server to be used in the config
                                         String nameserver = null;
                                         try {
                                              BufferedReader input =  new BufferedReader(new FileReader(new File(resolvConf)));
                                              String currentLine = null;
                                              while (( currentLine = input.readLine()) != null){
                                                   // Take care of potential comments
                                                   currentLine = currentLine.substring(0, currentLine.indexOf("#") == -1
                                                             ? currentLine.length() : currentLine.indexOf("#") );
                          
                                                   if (currentLine.contains("nameserver")) {
                                                        // It is the line we are looking for
                                                        nameserver = currentLine;
                                                        break;
                                                   }
                                              }
                                         } catch (FileNotFoundException e) {
                                              System.out.println("Loop " + loopCounter + ": FileNotFoundException");
                                         } catch (IOException e) {
                                              System.out.println("Loop " + loopCounter + ": IOException");
                                         }
                          
                                         if (nameserver == null) {
                                              // No "nameserver" line found
                                              System.out.println("Loop " + loopCounter + ": No nameserver found in configration file!");
                                              continue;
                                         }
                                         
                                         // Trim it to just contain the IP address
                                         nameserver = (nameserver.replace("nameserver", "")).trim();
                                         System.out.println("Loop " + loopCounter + ": Going to use DNS server " + nameserver);
                          
                                         // At this point, we know which server to use, now perform the resolution
                                         Hashtable<String, String> env = new Hashtable<String, String>();
                                         env.put("java.naming.factory.initial", "com.sun.jndi.dns.DnsContextFactory");
                                         env.put("java.naming.provider.url",    "dns://" + nameserver);
                          
                                         DirContext ictx;
                                         try {
                                              ictx = new InitialDirContext(env);
                                              Attributes attrs1 = ictx.getAttributes(urlString, new String[] {"A"});
                                              System.out.println("Loop " + loopCounter + ": Manual resolution: +" + attrs1.get("a").get() + "+");
                          
                                         } catch (NamingException e) {
                                              System.out.println("Loop " + loopCounter + ": NamingException");
                                         }
                                    }
                               }
                          }
                          So maybe I should adapt and package this into a proper provider and specify it for sun.net.spi.nameservice.provider. Any link, info or example about how a proper provider should look like?

                          Thanks for your advices!
                          • 10. Re: Java doesn't pick up system's DNS settings change until restarted
                            jtahlborn
                            Well, i looked at the implementation sun provides in openjdk 1.6. if you (or your company) are ok with open source licenses (GPL), you should be able to just copy that one and modify it to suit your needs. my thought was changing the implementation to check the timestamp on the resolv.conf file, and reload it whenever the timestamp changes (File.lastModified()).