This discussion is archived
12 Replies Latest reply: Mar 1, 2010 11:31 AM by 843793 RSS

Testing and debugging annotation processors

843793 Newbie
Currently Being Moderated
I'm developing an annotation processor based on Java 6, that should store it's state in a file. The processor creates a single class based on annotations in several user classes. Thus it needs to know all these classes, even if only one class changes. In order to achieve this, it stores the already gathered information in a file and reloads this file everytime the processor get's initialized. I use processingEnv.getFiler().createResource(...) for saving and processingEnv.getFiler().getResource(...) for loading the state.

My problem is: Until now I could not figure out, what is the best way to test and debug this feature.

I tried the hickory-project. This works fine, if you just want to step through your annotation processor. But testing the above feature seams a bit more difficult, as the file seams to get stored in a RAM-disk, so that the second time I call my test, the file is not there anymore.

I tried to build a sample project in eclipse, that uses my annotation processor. For debugging purposes I tried to use Log4J. But neither a ConsoleAppender nor a FileAppender seam to be recognized. No output. No file. Isn't this possible? Or am I doing something wrong?

Which way do you recommend for testing and debugging this?
  • 1. Re: Testing and debugging annotation processors
    608410 Newbie
    Currently Being Moderated
    yawah wrote:
    I'm developing an annotation processor based on Java 6, that should store it's state in a file. The processor creates a single class based on annotations in several user classes. Thus it needs to know all these classes, even if only one class changes. In order to achieve this, it stores the already gathered information in a file and reloads this file everytime the processor get's initialized. I use processingEnv.getFiler().createResource(...) for saving and processingEnv.getFiler().getResource(...) for loading the state.

    My problem is: Until now I could not figure out, what is the best way to test and debug this feature.

    I tried the hickory-project. This works fine, if you just want to step through your annotation processor. But testing the above feature seams a bit more difficult, as the file seams to get stored in a RAM-disk, so that the second time I call my test, the file is not there anymore.
    Hickory should be able to do what you want. You can acess the ram file system after a compile to assert the content of files. Also, there is a way to create a Compilation initialised witht the ram file system from a previous compilation. Using these two features you should be able test and debug your processor.

    There is a test for hickory's StateSaver class that does just that. You can see the source code here
    [https://hickory.dev.java.net/source/browse/hickory/trunk/code/test/net/java/dev/hickory/incremental/StateSaverTest.java?rev=11&view=markup|https://hickory.dev.java.net/source/browse/hickory/trunk/code/test/net/java/dev/hickory/incremental/StateSaverTest.java?rev=11&view=markup]

    The last method
    public void testTwoCompiles() { ... }
    in that class does two compiles using the same file system and makes asserts about the content of the generated files.


    specifically - reading the ram file system is done by this line
    InputStream in = compilation.getOutputClassLoader().getResourceAsStream("META-INF/services/foo.bar.Service");
    and creating a new Compilation with a ram file system from a previous compilation is done by this line
    compilation = new Compilation(compilation);
    I hope this helps

    Bruce
  • 2. Re: Testing and debugging annotation processors
    843793 Newbie
    Currently Being Moderated
    Thank you.

    This sounds like a very good hint. I will check it out during the next days and come back again.
  • 3. Re: Testing and debugging annotation processors
    843793 Newbie
    Currently Being Moderated
    Setting up a second compilation with the file system of a first compilation works very well:
    Compilation vCompilation = new Compilation();
    InputStream vSourceStream = ClassLoader.getSystemResourceAsStream("testsrcfiles/SimpleClass.testsrc");
    HickoryUtil.addSource(vCompilation, vSourceStream, "testsrcfiles.SimpleClass");
    
    vCompilation.doCompile(new PrintWriter(System.out));
    
    Compilation v2ndCompilation = new Compilation(vCompilation);
    vSourceStream = ClassLoader.getSystemResourceAsStream("testsrcfiles/AnotherSimpleClass.testsrc");
    HickoryUtil.addSource(v2ndCompilation, vSourceStream, "testsrcfiles.AnotherSimpleClass");
         
    v2ndCompilation.doCompile(new PrintWriter(System.out));
    But I still have problems to read the resource file, that I created. I think this is due to the fact, that I write the resource file to the SOURCE_OUTPUT:
    FileObject vStateFile = vFiler.createResource(StandardLocation.SOURCE_OUTPUT, "", RESSOURCE_FILENAME);
    OutputStream vOut = vStateFile.openOutputStream();
    ObjectOutputStream vObjectOut = new ObjectOutputStream(vOut);
    
    vObjectOut.writeObject(myObject);
    The file get's written to: ramfilesystem://jfm0/MyRessource.res

    I tried to load it with
    v2ndCompilation.getOutputClassLoader().getResourceAsStream("/MyRessource.res")
    and with
    v2ndCompilation.getOutputClassLoader().getResourceAsStream("/jfm0/MyRessource.res")
    Neither did work. Thus I tried to load it with
    v2ndCompilation.getGeneratedSource("/MyRessource.res")
    and I'm getting an empty String.

    I don't think the class output to be a good destination for the resource file, is it? Any ideas?
  • 4. Re: Testing and debugging annotation processors
    608410 Newbie
    Currently Being Moderated
    Yawah,


    SOURCE_OUTPUT is for java source files that will be compiled in the next round of processing. It is for things that will be inputs to the compiler. Your resource file is an input to your program. Ultimately you want it in your jar file along with the class files. You don't need the compiler to see it. That is why I think the resource should be created in the CLASS_OUTPUT, and probably why my library is not giving you access to it.
    Compilation.getOutputClassLoader().getResourceAsStream()
    isn't working because it is accessing things visible to the class loader, which is basically CLASS_OUTPUT.
    Compilation.getGeneratedSource()
    is not working for you because (althought it looks in the right place - SOURCE_OUTPUT) its argument is fqn, and although poorly documented (noted) takes the fully qualified name of the class you want the source for. Your resource is not a class and doesn't have an fqn.

    There is no way at present for the Compilation's file system to give you access to a resource file created in SOURCE_OUTPUT. This had not occurred to me but now that I have seen the situation, I think the current behaviour is a good thing because it is making it painful for you to do the wrong thing. Create the resource in CLASS_OUTPUT and I would hope that your struggles will be eased. While you are putting it in SOURCE_OUTPUT you will continue to have little things not be quite right.

    I suspect you are currently not setting the source output directory ( javac -s) and therefore it is defaulting to the same place as CLASS_OUTPUT which means that your resource is magically getting into the jar file. If in future someone changes your build mechanism to keep the generated source separate ( a good idea because you probably don't want generated source files in the jar file) , then the resource will no longer get into the jar file without explicitly being copied there. You wont have these sorts of problems if you write to CLASS_OUTPUT. It is even imaginable that some future version of javac could have an option like -s that specified that SOURCE_OUTPUT was a transient memory based file system. Such an option would really screw up the way you are currently working.

    I think the action plan should be

    jawah: use CLASS_OUTPUT
    bruce: fix Compilation javadocs for getGenerated()
    openJDK: improve FileLocation.CLASS_OUTPUT so that it is the most obvious choice for resource files.


    Bruce
  • 5. Re: Testing and debugging annotation processors
    843793 Newbie
    Currently Being Moderated
    Hello Bruce,

    thank you very much for your help.

    I understand your explanation and would have put the file into the CLASS_OUTPUT, if I wanted it to be available in the jar. But in fact it is the file where the annotation processor stores the information, that it needs, when it gets called the next time. Thus I explicitly do not want it to appear in the jar. - Sorry, "resource file" was probably the wrong term for this file.
    On the other hand - if javac had an option to store the SOURCE_OUTPUT into a transient memory based file system, this would truly screw up my annotation processor.

    Is there a recommended place to put such files?

    Probably I have to invest much more time and have a deeper look into the sources of the hickory project. Hopefully I find this time during the next weekend...

    Greetings

    yawah
  • 6. Re: Testing and debugging annotation processors
    608410 Newbie
    Currently Being Moderated
    Yawah,


    SOURCE_OUTPUT is correct location then.

    If you need to access SOURCE_OUTPUT files from the hickory testing Compilation, I could add a methods to Compilation for accessing "resource" files (not java source files, and not class files - that's already there) in the memory file system.
    FileObject getFile(JavaFileManager.Location location, String packageName, String relativeName) 
    with maybe another arg to say whether when requesting a non existent file, to create it (for when you want to write) or return null (for when you want to check that it is there, and read it)

    alternatively have two methods getExisitingFile(...) that returned null for non existing file, and getFile (as above) that creates a new file if it doesn't already exist.

    Would that be useful to you?

    Bruce
  • 7. Re: Testing and debugging annotation processors
    843793 Newbie
    Currently Being Moderated
    Hello Bruce,

    thank you again for your answer.

    Yes, this would be very helpful for building tests and for debugging. I would prefer the solution that uses two methods. But that is mainly a matter of taste.

    Greetings

    yawah
  • 8. Re: Testing and debugging annotation processors
    608410 Newbie
    Currently Being Moderated
    OK,

    I'll try and get it done soonish - maybe a few days.

    Bruce
  • 9. Re: Testing and debugging annotation processors
    843793 Newbie
    Currently Being Moderated
    That's great. Thank you. :-)
  • 10. Re: Testing and debugging annotation processors
    608410 Newbie
    Currently Being Moderated
    source changes committed, but binaries and javadoc not updated yet.
  • 11. Re: Testing and debugging annotation processors
    843793 Newbie
    Currently Being Moderated
    Thank you. I will have a try at the weekend...
  • 12. Re: Testing and debugging annotation processors
    843793 Newbie
    Currently Being Moderated
    Hello Bruce,

    sorry, it took a little longer - Today I tested the newly provided method. It works perfect.

    Thank you

    yawah