This discussion is archived
2 Replies Latest reply: Nov 2, 2010 8:56 AM by 807565 RSS

Javac AST Symbol Resolving for JavacTask.parse()

807565 Newbie
Currently Being Moderated
Hello!

I am currently working on a source code processor, based on an AST. I use the original javac library. I know that the javac package is not provided as API, but it suffices for a first prototype.

I parse a file (multiple files) with JavacTask.parse() and descent into the AST. Please don't hit me for the solution using Reflection! The interesting lines are on *321-328*, where I try to resolve the Symbols manually, because it seems all .sym properties are set to null.

I don't understand how to resolve the AST previous to descent into it. Is it possible to resolve Symbols with the original library code?


Warning, big Code:
package runtimeLoader;

import java.io.StringWriter;
import java.lang.reflect.Field;
import java.util.HashMap;
import java.util.LinkedList;
import java.util.Stack;

import javax.tools.JavaCompiler;
import javax.tools.JavaFileObject;
import javax.tools.StandardJavaFileManager;
import javax.tools.ToolProvider;

import runtimeLoader.RuntimeLoader.Trace.RuntimeLoaderTrace;

import com.sun.source.tree.CompilationUnitTree;
import com.sun.source.tree.MethodInvocationTree;
import com.sun.source.tree.MethodTree;
import com.sun.source.util.JavacTask;
import com.sun.source.util.TreeScanner;
import com.sun.tools.javac.api.JavacTaskImpl;
import com.sun.tools.javac.api.JavacTrees;
import com.sun.tools.javac.code.Scope;
import com.sun.tools.javac.code.Symbol.MethodSymbol;
import com.sun.tools.javac.code.Symbol.PackageSymbol;
import com.sun.tools.javac.code.Type;
import com.sun.tools.javac.code.TypeTags;
import com.sun.tools.javac.comp.AttrContext;
import com.sun.tools.javac.comp.Env;
import com.sun.tools.javac.comp.Resolve;
import com.sun.tools.javac.tree.JCTree;
import com.sun.tools.javac.tree.JCTree.JCAnnotation;
import com.sun.tools.javac.tree.JCTree.JCAssign;
import com.sun.tools.javac.tree.JCTree.JCBlock;
import com.sun.tools.javac.tree.JCTree.JCCompilationUnit;
import com.sun.tools.javac.tree.JCTree.JCExpression;
import com.sun.tools.javac.tree.JCTree.JCFieldAccess;
import com.sun.tools.javac.tree.JCTree.JCIdent;
import com.sun.tools.javac.tree.JCTree.JCLabeledStatement;
import com.sun.tools.javac.tree.JCTree.JCLiteral;
import com.sun.tools.javac.tree.JCTree.JCMethodDecl;
import com.sun.tools.javac.tree.JCTree.JCMethodInvocation;
import com.sun.tools.javac.tree.JCTree.JCPrimitiveTypeTree;
import com.sun.tools.javac.tree.JCTree.JCStatement;
import com.sun.tools.javac.tree.JCTree.JCVariableDecl;
import com.sun.tools.javac.tree.Pretty;
import com.sun.tools.javac.util.List;
import com.sun.tools.javac.util.ListBuffer;
import com.sun.tools.javac.util.Name;
import com.sun.tools.javac.util.Name.Table;

public class Compile {

     /*
      * parse, pretty, store temp
      * copy tree, process, pretty, store, compile, EXECUTE, TEST
      * process back, pretty, COMPARE EQUAL
      * 
      * 
      * 
      * nested vars
      * nested classes, methods
      * vars there
      */

     static class ReferenceList {
          public Object ref;
          public LinkedList list = new LinkedList();

          public ReferenceList(Object ref) {
               this.ref = ref;
               list.add(ref);
          }
     }

     static class TraceStructure {
          public JCMethodInvocation call = null;
          public JCStatement stat = null;
          public LinkedList<TraceStructure> stats = new LinkedList<TraceStructure>();
     }

     enum UpdateAction {
          UPDATE_CLEANUP,
          UPDATE_PROCESS,
          SCAN_CODE,
          SCAN_SYMBOLS,
          SCAN_RESOLVE,
     }

     static boolean traceHeader = false;
     static boolean traceCall = false;

     static Resolve resolve = null;
     //static TreeScanner treeScanner = null;

     static Stack objectRefStack = new Stack();
     static LinkedList<JCVariableDecl> variableList = new LinkedList<JCVariableDecl>();
     static LinkedList<JCMethodDecl> methodList = new LinkedList<JCMethodDecl>();
     static TraceStructure traceStructure = null;

     public static boolean traceMethod(JCMethodDecl method) {
          boolean traceMethod = false;
          for (JCAnnotation anno : method.mods.annotations) {
               if (anno.annotationType instanceof JCIdent) {
                    if ("RuntimeLoaderTrace".equals(((JCIdent) anno.annotationType).name.toString())) {
                         traceMethod = true;
                         break;
                    }
               }
          }

          return traceMethod;
     }

     public static boolean findMethod(JCMethodDecl method, Object instance) {
          boolean traceMethod = false;
          for (JCAnnotation anno : method.mods.annotations) {
               if (anno.annotationType instanceof JCIdent) {
                    if ("RuntimeLoaderTrace".equals(((JCIdent) anno.annotationType).name.toString())) {
                         traceMethod = true;
                         break;
                    }
               }
          }

          return traceMethod;
     }

     public static void updateTreeRecur(ReferenceList objectRef, UpdateAction action) throws Exception {
          Field[] sourceFieldList = objectRef.ref.getClass().getDeclaredFields();

          HashMap<String, Field> sourceFieldMap = new HashMap<String, Field>();

          if (!traceHeader) {
               for (Field sourceField : sourceFieldList) {
                    Object value = sourceField.get(objectRef.ref);
                    String name = sourceField.getName();

                    if (value instanceof List) {
                         List sourceList = (List) value;
                         ListBuffer targetListBuffer = new ListBuffer();

                         //System.out.println(objectRef.ref.getClass().getName());
                         if (UpdateAction.UPDATE_PROCESS.equals(action)
                                        && objectRefStack.lastElement() instanceof JCMethodDecl
                                        && objectRef.ref instanceof JCBlock
                                        && "stats".equals(name)) {

                              if (traceMethod((JCMethodDecl) objectRefStack.lastElement())) {
                                   Table table1 = ((JCMethodDecl) objectRefStack.lastElement()).getName().table;

                                   variableList.clear();
                                   traceStructure = new TraceStructure();

                                   // Analyze

                                   updateTreeRecur(new ReferenceList(objectRef.ref), UpdateAction.SCAN_CODE);

                                   // Header

                                   targetListBuffer.append(
                                             new CustomLabeledStatement(Name.fromString(table1, "RuntimeLoaderT1"),
                                                       new CustomBlock(0, new ListBuffer().toList())));

                                   targetListBuffer.append(
                                             new CustomLabeledStatement(Name.fromString(table1, "RuntimeLoaderT2"),
                                                       new CustomBlock(0, new ListBuffer().toList())));

                                   // Variables

                                   for (JCVariableDecl variable : variableList) {
                                        //System.out.println(variable.getType().getClass().getName());
                                        if (variable.getType() instanceof JCPrimitiveTypeTree) {
                                             int typetag = ((JCPrimitiveTypeTree) variable.getType()).typetag;
                                             variable.init = new CustomLiteral(typetag, 0);
                                        }
                                        else {
                                             variable.init = new CustomLiteral(TypeTags.BOT, null);
                                        }
                                        targetListBuffer.append(variable);
                                   }

                                   // Trace

                                   targetListBuffer.append(
                                             new CustomLabeledStatement(Name.fromString(table1, "RuntimeLoaderC"),
                                                       new CustomBlock(0, new ListBuffer().toList())));

                              }

                         }

                         for (Object sourceObject : sourceList) {
                              ReferenceList targetObjectRef = new ReferenceList(sourceObject);

                              objectRefStack.push(objectRef.ref);
                              updateTreeRecur(targetObjectRef, action);
                              objectRefStack.pop();

                              if (targetObjectRef.ref != null) {
                                   targetObjectRef.list.set(0, targetObjectRef.ref);
                                   for (Object targetObject : targetObjectRef.list) {
                                        if (targetObject != null) {
                                             targetListBuffer.append(targetObject);
                                        }
                                   }
                              }
                         }

                         sourceField.set(objectRef.ref, targetListBuffer.toList());
                    }
                    else if (value instanceof JCTree) {
                         ReferenceList valueRef = new ReferenceList(value);

                         objectRefStack.push(objectRef.ref);
                         updateTreeRecur(valueRef, action);
                         objectRefStack.pop();

                         if (valueRef.ref == null) {
                              objectRef.ref = null;
                              return;
                         }
                         sourceField.set(objectRef.ref, valueRef.ref);
                    }
               }
          }

          if (UpdateAction.UPDATE_CLEANUP.equals(action)) {
               if (objectRef.ref instanceof JCLabeledStatement) {
                    String label1 = ((JCLabeledStatement) objectRef.ref).label.toString();
                    if ("RuntimeLoaderT1".equals(label1)) {
                         traceHeader = true;
                         objectRef.ref = null;
                         return;
                    }
                    else if ("RuntimeLoaderT2".equals(label1)) {
                         traceHeader = false;
                         objectRef.ref = null;
                         return;
                    }
                    else if ("RuntimeLoaderC".equals(label1)) {
                         objectRef.ref = ((JCLabeledStatement) objectRef.ref).body;
                         if (objectRef.ref instanceof JCBlock) {
                              switch (((JCBlock) objectRef.ref).stats.size()) {
                              case 0:
                                   objectRef.ref = null;
                                   return;
                              case 1:
                                   objectRef.ref = ((JCBlock) objectRef.ref).stats.get(0);
                                   break;
                              }
                         }
                    }
               }

               if (traceHeader) {
                    objectRef.ref = null;
                    return;
               }
          }
          else if (UpdateAction.SCAN_CODE.equals(action)) {
               if (objectRef.ref instanceof JCVariableDecl) {
                    JCVariableDecl object1 = (JCVariableDecl) objectRef.ref;
                    variableList.add(object1);

                    objectRef.ref = new CustomAssign(object1.vartype, object1.init);
               }
               else if (objectRef.ref instanceof JCMethodInvocation) {
                    JCMethodInvocation object1 = (JCMethodInvocation) objectRef.ref;

                    //System.out.println(object1.meth);

                    if (object1.meth instanceof JCIdent) {

                         System.out.println(((JCIdent)object1.meth).sym);
                         /*if(((JCIdent)object1.meth).sym instanceof MethodSymbol) {
                              System.out.println(((MethodSymbol)((JCIdent)object1.meth).sym).code);
                         }*/
                    }
                    else if (object1.meth instanceof JCFieldAccess) {
                         System.out.println(((JCFieldAccess) object1.meth).sym);
                         //System.out.println(((JCFieldAccess) object1.meth).getExpression() + " " + ((JCFieldAccess) object1.meth).name);

                    }
               }
          }
          else if (UpdateAction.SCAN_SYMBOLS.equals(action)) {
               if (objectRef.ref instanceof JCMethodDecl) {
                    JCMethodDecl object1 = (JCMethodDecl) objectRef.ref;

                    
                    //resolve.resolveInternalMethod(object1.pos(), null, object1.type, object1.getName(), object1.);
                    
                    System.out.println(object1.sym);
                    
                    
                    methodList.add(object1);
               }
               if (objectRef.ref instanceof JCMethodInvocation) {
                    JCMethodInvocation object1 = (JCMethodInvocation) objectRef.ref;
                    Name name = null;
                    Type type = null;
                    
                    System.out.println(object1.meth.getClass().getName());

                    if (object1.meth instanceof JCIdent) {
                         name = ((JCIdent)object1.meth).name;
                         type = ((JCIdent)object1.meth).type;

                         //System.out.println(object1.type);
                         //System.out.println(((JCIdent)object1.meth).type);
                    }
                    else if (object1.meth instanceof JCFieldAccess) {
                         name = ((JCFieldAccess)object1.meth).name;
                         type = ((JCFieldAccess)object1.meth).type;
                    }

                    //type = new Type(TypeTags.VOID, null);
                    //type = new Type(TypeTags.VOID, new TypeSymbol());
                    
                    ListBuffer<Type> argtypeListBuffer = new ListBuffer<Type>();
                    AttrContext attrContext = new AttrContext();
                    Env<AttrContext> env = new Env<AttrContext>((JCTree) objectRef.ref, attrContext);

                    System.out.println(type);
                    System.out.println(type.tsym);
                    
                    resolve.resolveInternalMethod(object1.pos(), env, type, name, argtypeListBuffer.toList(), null);
                    
               }
          }
     }

     public static void main(String[] args) throws Exception {
          JavaCompiler compiler = ToolProvider.getSystemJavaCompiler();
          StandardJavaFileManager fileManager = compiler.getStandardFileManager(null, null, null);
          Iterable<? extends JavaFileObject> fileObjects = fileManager.getJavaFileObjects(
                    "C:/Users/user2/workspace/runtimeLoader/src/runtimeLoader/Test.java"
                    //"C:/Users/user2/workspace/runtimeLoader/src/runtimeLoader/Test2.java"
                    );
          JavacTaskImpl javacTaskImpl = (JavacTaskImpl) compiler.getTask(null, fileManager, null, null, null, fileObjects);

          Iterable<? extends CompilationUnitTree> treeList = javacTaskImpl.parse();

          resolve = Resolve.instance(javacTaskImpl.getContext());

          for (CompilationUnitTree sourceTree : treeList) {
               /*treeScanner = new SourceTreeProcessor();
               sourceTree.accept(treeScanner, null);*/
               
               /*System.out.println(
                         JavacTrees.instance(
                                   javacTaskImpl).getScope(
                                             JavacTrees.instance(javacTaskImpl).getPath(
                                                       sourceTree, sourceTree)).getEnv());*/
               
               objectRefStack.clear();
               objectRefStack.push(new Object());
               updateTreeRecur(new ReferenceList(sourceTree), UpdateAction.SCAN_SYMBOLS);
          }

          /*for (CompilationUnitTree sourceTree : treeList) {
               objectRefStack.clear();
               objectRefStack.push(new Object());
               updateTreeRecur(new ReferenceList(sourceTree), UpdateAction.UPDATE_CLEANUP);
               updateTreeRecur(new ReferenceList(sourceTree), UpdateAction.UPDATE_PROCESS);

               StringWriter s = new StringWriter();
               new Pretty(s, false).printExpr((JCTree) sourceTree);

               //System.out.println(s.toString());
          }*/
     }

     /*private static class SourceTreeProcessor extends TreeScanner<Void, Void> {

          @Override
          public Void visitMethodInvocation(MethodInvocationTree node, Void p) {
               //System.out.println(node.getMethodSelect().);
               return super.visitMethodInvocation(node, p);
          }
     }*/

     static class CustomLabeledStatement extends JCLabeledStatement {
          protected CustomLabeledStatement(Name arg0, JCStatement arg1) {
               super(arg0, arg1);
          }
     }

     static class CustomBlock extends JCBlock {
          protected CustomBlock(long arg0, List<JCStatement> arg1) {
               super(arg0, arg1);
          }
     }

     static class CustomAssign extends JCAssign {
          protected CustomAssign(JCExpression arg0, JCExpression arg1) {
               super(arg0, arg1);
          }
     }

     static class CustomLiteral extends JCLiteral {
          protected CustomLiteral(int arg0, Object arg1) {
               super(arg0, arg1);
          }
     }
}
  • 1. Re: Javac AST Symbol Resolving for JavacTask.parse()
    807565 Newbie
    Currently Being Moderated
    Sorry, i am pushing a bit.

    I continue development, and solve the problem temporarily by comparing the literal strings. Because this is highly inaccurate, this issue is very urgent. Please answer shortly, if possible. If you are interested, this project is a first attempt to realize a independent platform for free and open source solution distribution.

    Imagine the case of equal variable names in parallel scope:
    {
         int aNumber = 1;
    }
    {
         int aNumber = 2;
    }
    without correct resolved symbol names, the names are ambiguous and cannot produce the correct result:
    String[] rl_variableList = {"aNumber", "aNumber"};
    int rl_0_aNumber = 0;
    int rl_1_aNumber = 0;
    
    {
         rl_0_aNumber = 1;
    }
    {
         rl_1_aNumber = 2;
    }
  • 2. Re: Javac AST Symbol Resolving for JavacTask.parse()
    807565 Newbie
    Currently Being Moderated
    Hey, it was not as difficult, as I thought:
            Enter enter = Enter.instance(javacTaskImpl.getContext());
    
            ListBuffer<JCCompilationUnit> treeList = new  ListBuffer<JCCompilationUnit>();
            for(CompilationUnitTree tree : javacTaskImpl.parse())
                 treeList.append((JCCompilationUnit) tree);
            
              enter.complete(treeList.toList(), null);
    
              for (CompilationUnitTree sourceTree : treeList) {
    
                        process the trees.....
    near 20% of the symbols are now resolved. thank you for the support!

Legend

  • Correct Answers - 10 points
  • Helpful Answers - 5 points