Revisit Jmx Groovy DSL - Using AstTransformation

In the previous blog entry, we created Jmx Charting DSL. Let visit some aspects of creating DSL using groovy.

In the previous blog example we created DSL using ExpandoMetaClass. Users will define the script starting with node jmx { }. With the use of ExpandoMetaClass we added the dynamic method which in turn delegated to class JmxClosureDelegate. Here is the snippet of the code.

static void runEngine(File dsl){
    Script dslScript = new GroovyShell().parse(dsl.text)
    dslScript.metaClass = createExpandoMetaClass(dslScript.class, {
      ExpandoMetaClass emc ->
        emc.jmx = {
          Closure cl ->
            cl.delegate = new JmxClosureDelegate()
            cl.resolveStrategy = Closure.DELEGATE_FIRST
            cl()
        }
    })
    dslScript.run()
   }

What we essentially did here:

• Using GroovyShell it parses the script file passed as input.
• Defines the ExpandoMetaClass and adds method by name "jmx" having closure as parameter.

So the script file you wrote gets a dynamic method injected into it using ExpandoMetaClass. This happens at runtime.

Getting rid of the Commas
The problem with creating methods with two arguments is that you have to specify commas between them. For example,

     server "nameofserver", {
          ...
     } 

Undoubtedly commas clutter the langauge grammer. To get rid of the commas we used trick provided on Groovy users list.

//To avoid using "," between String and Closure argument  
  def methodMissing(String name, args) {  
    return [name, args[0]]  
  }  

Using AstTransformation
Now lets explore a different possibility. You can achieve similar result using AstTransformation at compile time. The goal remains the same and that is to add method with name "jmx" with closure parameter.

First we will define the annotation.

//import statements skipped for brevity
@Retention(RetentionPolicy.SOURCE)
@Target([ElementType.METHOD])
@GroovyASTTransformationClass(["info.kartikshah.jmx.ast.JmxDslTransformation"])
public @interface UseJmxDsl {}

Transformation Class

This transformation class needs to perform two activities:

• Add method jmx(Closure cl) method
• Invoke the script method being defined

We need to generate AST statements for following snippet of code.

     jmx = {  
          Closure cl ->  
            cl.delegate = new JmxClosureDelegate()  
            cl.resolveStrategy = Closure.DELEGATE_FIRST  
            cl()  
        }  

We will use AstBuilder's buildFromSpec option to generate it. (AstBuilders added with Groovy 1.7 definitely makes generating statement structure relatively easy and clutter free. Not to mention it is also an example of DSL added to the groovy language :-) )

@GroovyASTTransformation(phase = CompilePhase.SEMANTIC_ANALYSIS)
class JmxDslTransformation implements ASTTransformation {
  static int PUBLIC = 1
  static int STATIC = 8

  void visit(ASTNode[] astNodes, SourceUnit sourceUnit) {
    //Add method jmx(Closure cl)
    ClassNode declaringClass = astNodes[1].declaringClass
    MethodNode jmxMethod = makeMethod()
    declaringClass.addMethod(jmxMethod)

    //Insert method call inside run method of the script class
    MethodNode annotatedMethod = astNodes[1]
    List<MethodNode> allMethods = sourceUnit.AST?.classes*.methods.flatten()
    MethodNode runMethod = allMethods.find{ MethodNode method ->
      method.name == "run"
    }
    List existingStatements = runMethod.getCode().getStatements()
    existingStatements.add(0, createMethodCall(annotatedMethod))
  }

  Statement createMethodCall(MethodNode methodNode){
    def statementAst = new AstBuilder().buildFromSpec {
      expression{
         methodCall {
           variable "this"
           constant methodNode.name
           argumentList {}
         }
      }
    }
    Statement stmt = statementAst[0]
    stmt
  }

  MethodNode makeMethod() {
    def ast = new AstBuilder().buildFromSpec {
      method('jmx', PUBLIC | STATIC, Void.TYPE) {
        parameters {
          parameter 'cl': Closure.class
        }
        exceptions {}
        block {
          expression {
            binary {
              property {
                variable "cl"
                constant "delegate"
              }
              token "="
              constructorCall(JmxClosureDelegate.class) {
                argumentList()
              }
            }
          }
          expression {
            binary {
              property {
                variable "cl"
                constant "resolveStrategy"
              }
              token "="
              property {
                classExpression Closure
                constant "DELEGATE_FIRST"
              }
            }
          }
          expression {
            methodCall {
              variable "cl"
              constant "call"
              argumentList {}
            }
          }
        }
      }
    }
    MethodNode jmxMethod = ast[0]
    jmxMethod
  }
}

Why use AstTransformation?
The question is why one would want to use AstTransformation when you can add method during runtime. For the given scenario, it is correct that you want to stick with adding method runtime. But consider scenario where you want to "redefine" meaning of Groovy's syntax. For example like following imaginary script using Statement Labels to add more readability to your DSL syntax.

@info.kartikshah.jmx.ast.UseJmxDsl
runDsl () {
  jmx {
    setup:
      server "service:jmx:rmi://localhost/jndi/rmi://localhost:1090/jmxconnector"
      query "jboss.web:*"
      findAll "j2eeType=Servlet"

    draw:
    chart {
        chartType="Bar"
        attributes={m-> [m.loadTime, m.objectName.find("name=([^,]*)"){it[1]}]}
        labels=["Load Time per Servlet", "Servlet", "Time"]
        options=[false, true, true]
        windowTitle="JBoss Servlet Processing Time"
        width=1200
        height=700
        orientation="HORIZONTAL"
        refreshRate=5000
        show()
      }
  }
}

Spock Framework does similar twist by redefining meaning of existing construct.

With this type of language structure you will end up defining your own set of keywords, supporting parser and few AstTransformation to change the meaning of existing Groovy Syntax.

Blogged with the Flock Browser

Helpful tips - Working with AST Transformation

I have been playing with Groovy AST transformation for some time now. You can walk through some of the earlier posts here. With this entry I want to capture some of the lessons learnt, tips, etc.
 
Using AstViewer Tool (Inspect Ast)
With Groovy 1.6 AST transformation, one of the difficulty was to come up with tree structure for the code you want to inject or change. It involves calling complex set of Statement and Expression APIs. It is ridden with intricate details of language grammar. With groovy 1.7 AstBuilder, you can use buildFromString and buildFromCode which is lot more easier. But still if you have to use buildFromSpec you will need to have some understanding of Statement/Expression API.
 
One of the tool I have found useful is Groovy Console's AST Browser or Eclipse's ASTViewer. Write code you want to generate language structure for and try to replicate using API or using AstBuilder's buildFromSpec. The tool has gotten better from 1.6 to 1.7.
Open Groovy Console and type in following code

            def advice = new String("Hello AST")
            println advice

Go to Script --> Inspect Ast (or press Ctrl+T) and you will see the AST structure needed.

 
Educated Guess in Script
With AstBuilder now you can just start building AST in main method or just as script. This has proven to be useful if you just want to compare what you are generating with what is displayed in AstBuilder.

def nodes = new AstBuilder().buildFromXXX{
}

 
Working with Variables
If you are using variables that gets replaced during runtime you may face problem generating AST. One of the method that I found useful was to replace the variable with static value and see if it works. If it does than the problem is possibly due to the type of the variable.
For example, I wanted to generate

          def advice = new <<variableClassName>>()

where variableClassName will be passed in from annotation value. One option is to try generating it with static value like java.lang.String (or just String), if it works than there is some problem with the type of the variable you are passing.
 
Compile file individually (if using IDE)
Sometimes the tools you are working with just does not seem to give different output even when you are changing things around in your code. Sometimes stale classes just gets called and you never see the output from the updated classes. You may want to compile both transformation class and the class where the transformation gets used. 
 
ByteCode Viewer
Also using Byte Code Viewer like this helps in looking at the classes that are getting generated. You can also write your class how it will look after AST transformation. Use byte code viewer to see the class and compare it with the class that is being generated through AST Transformation. See which statements are properly generated and which are not.
 
Testcases
Another extremely resourceful source for AstBuilder is test cases found here. It contains rich example set of how to use each node in the builder specifically for buildFromSpec option. AstAssert is another excellent utility to compare and test expected and actually generated structure. Looks like AstAssert class is not packaged with groovy-1.7, so you may need to download and drop it in your project.

Divide and Conquer
Treat each section of code for which AST is being generated as separate section. Comment out blocks of code and gradually uncomment them to see which section is actually failing and/or not giving desired output.
 
Hope this helps! Drop in a line to share your tips or problems you face with Groovy's Ast transformation. 

Blogged with the Flock Browser

Revisiting Groovy AST Transformation with AstBuilder

Groovy AST is powerful tool to inject changes into classes at various compile phases. It allows one to operate on AST (Abstract Syntax Tree) to manipulate class structure.

With Groovy 1.7 AstBuilder there are easier ways to write AST transformation. One can build AST nodes using one of following three options

• Build From String
• Build From Spec
• Build from Code

Previously on this blog, we experimented with Groovy 1.6 AST Transformation Example and also AOP style usage  at Groovy AST Transformation - AOP Style. Lets revisit this transformation from earlier post.

Lets see how we can generate the AST tree using these new methods from AstBuilder. Recap this previous post before proceeding ahead.

Looks like we need to rewrite initAdviceCall and createMethodCall to experiment with new methods of AstBuilder.

Build From String
Using buildFromString the task would become very easy. With this options, it is not required to build complicated statement and expression structure. All you need to do is embed code within string and pass it in to buildFromString method. If code you generate is not completely static in nature, you may have to parameterize the value. Here in this example, the declaration of "advice" instantiation depends on the Class value defined in the annotation.  

  def initAdviceCall(classNodeType) {
    def codeString = "def advice = new " + classNodeType.type.typeClass.canonicalName + "()"
    def buildNodes = new AstBuilder().buildFromString(CompilePhase.SEMANTIC_ANALYSIS, true, codeString)
    Statement stmt = buildNodes[0].statements[0]
    stmt
  }

  def createMethodCall(method, Parameter[] parameters) {
    def codeString = "advice.before (" + "'$method.name'" +", [" + parameters.collect {it.name}.join(',') + "])"
    def buildNodes = new AstBuilder().buildFromString(CompilePhase.SEMANTIC_ANALYSIS, true, codeString)
    Statement stmt = buildNodes[0].statements[0]
    stmt
  }

Build From Spec
buildFromSpec option is more verbose. It also requires you to understand internal structure that is required to be build. With use of GroovyConsole's Inspect Ast option, it is not difficult to come up with the AST structure.

  def initAdviceCall (classNodeType) {
    def buildNodes = new AstBuilder().buildFromSpec {
      expression {
        declaration {
          variable "advice"
          token "="
          constructorCall(classNodeType.getType().getTypeClass()){
            argumentList()
          }
        }
      }
    }
    Statement stmt = buildNodes[0]
    stmt
  }

  def createMethodCall(method, Parameter[] parameters) {
    def buildNodes = new AstBuilder().buildFromSpec {
      expression {
        methodCall {
          variable "advice"
          constant "before"
          argumentList {
            constant method.name
            list {
              parameters.each {variable it.name}
            }
          }
        }
      }
    }
    Statement stmt = buildNodes[0]
    stmt
  }

There are some differences between using API directly and using spec builder option. One thing specifically that threw me off was that ConstructorCallExpression takes in ClassNode as parameter, however the constructorCall in builder takes Class. It wouldn't have been much of an annoyance, if I had read GEP-2 which states that any expression taking ClassNode, for builder you just need to pass class instance. Well in this example, value provided by annotation was ClassNode, so I had to get Class and pass it on to constructorCall.

Build From Code
Unlike above two options buildFromCode option did not work for me in above scenario. The problem I faced was I couldn't do def advice = <VariableClassName>() without using reflection. Method would have looked something like this

def initAdviceCall(classNodeType){
new AstBuilder().buildFromCode {
     def advice = classNodeType.getType().getTypeClass().newInstance()
}

Similarly, for createMethodCall, I would have to use reflection for passing in parameters.

More later...

Blogged with the Flock Browser

Groovy AST Transformation - AOP Style

In my last blog about AST Transformation I followed a sample example and created AssertParamsNotNull local transformation. Further, I wanted to create AST Transformation which is generic enough and performs any check before method is executed. In process, goal is to learn about AST Transformation.

BeforeAdvisor AST Transformation
Use case for BeforeAdvisor AST involves:

• Authorization Checking - Security by checking role from context
• Print Parameter values with which the method is called
• Asserts Parameters are not null
• Check various entry-conditions/Pre-Conditions of the method

To make transform generic enough, idea is to inject a method call to before method of advice for each method annotated with @BeforeAdvisor(MyPreConditionAdvice). Advice method can choose to implement any checks/conditions to be performed before actual methods gets executed.

Few of the subtle characteristics of this type of solution are

• It does not allow changing the method parameters. 
  
• It does not allow to stop execution of method. However, you can throw runtime exception.
• Advice needs no arg constructor and must implement method before

Usage and Client
Starting with class that will use this transform. Following defines script-level method with annotation having advice information.

package com.learn.sts.groovy.ast

@com.learn.sts.groovy.ast.BeforeAdvisor(value= com.learn.sts.groovy.MyAdvice)
def sayHello(name, name2)
{
    println "Hello " + name + name2
}

sayHello("World", "Groovy") 

Sample Advice
The advice that we will try to invoke will be something like this. This advice just prints parameter value that method is being invoked with. But you can implement any of the use cases described above.

package com.learn.sts.groovy

public class MyAdvice
{
    def before(String methodName, List listArg)
    {
        println 'Entering Method ' + methodName + ' with params ' + listArg
    }
}

Annotation
Now lets create the Annotation

package com.learn.sts.groovy.ast;

import java.lang.annotation.Retention
import java.lang.annotation.RetentionPolicy
import java.lang.annotation.Target
import java.lang.annotation.ElementType
import org.codehaus.groovy.transform.GroovyASTTransformationClass
@Retention(RetentionPolicy.SOURCE)
@Target([ElementType.METHOD])
@GroovyASTTransformationClass(["com.learn.sts.groovy.ast.BeforeAdvisorASTTransformation"])
public @interface BeforeAdvisor {
    Class value ();
}

One difference to this annotation is that it declares a method value(). This allows us to get value being passed during annotation declaration on method.

AST Transformation

package com.learn.sts.groovy.ast

//Imports section skipped for brevity

@GroovyASTTransformation(phase=CompilePhase.SEMANTIC_ANALYSIS)
public class BeforeAdvisorASTTransformation implements ASTTransformation 
{
    public void visit(ASTNode[] nodes, SourceUnit source)
    {
        AnnotationNode node = (AnnotationNode) nodes[0];
        final Expression classNode = node.getMember("value")
        
        List<MethodNode> allMethods = source.AST?.classes*.methods.flatten()
        List annotatedMethods = allMethods.findAll{ MethodNode method ->
                 method.getAnnotations(new ClassNode(BeforeAdvisor))
                }
        annotatedMethods.each{MethodNode method -> 
             List existingStatements = method.getCode().getStatements()
            Parameter[] parameters = method.getParameters()
            int i = 0;
            existingStatements.add(i++, initAdviceCall(classNode))
            existingStatements.add(i++, createMethodCall(method, parameters))
        }
      }
  
    public Statement initAdviceCall(classNode)
    {
      return new ExpressionStatement(
              new DeclarationExpression(
                      new VariableExpression("advice"),
                      new Token(Types.ASSIGNMENT_OPERATOR, "=", -1, -1),
                      new ConstructorCallExpression(classNode.getType(), new ArgumentListExpression())
                      )
              )
    }
 
  public Statement createMethodCall(method, Parameter[] parameters){
      List parameterExpressionList = new ArrayList()
      parameters.each{ parameter -> parameterExpressionList.add(new VariableExpression(parameter))}
      return new ExpressionStatement(
        new MethodCallExpression(
            new VariableExpression("advice"),
            "before",
            new ArgumentListExpression(new ConstantExpression(method.getName()),
                    new ListExpression(parameterExpressionList)
            )
        )
      )
  }
}

Couple of things to notice here:

First, creation of the AST tree involves

• Creating instance of Advice Class - This is done through method initAdviceCall
  
• Invocation of the method before with methodName and Parameter List - This is done through method createMethodCall

Creating AST structure is not the easiest task. One approach that has worked for me is the write the sample code that you are trying to generate AST for and then use Groovy AST viewer in eclipse.

Second, inspecting the value on ASTNode and getting Advice class value. First node contains information about the annotation and second node is the annotated node.

Lessons Learned

Compilation
It was required to compile Transformations files first and then compile the classes that use AST Transformation. If I compiled all three of them together the AST Transformation did not kick in. I used Eclipse IDE and it forced me to use Compile Groovy File explicitly each time I made change. "Build Automatically" or "Clean" option did not work.

Script Level Methods vs all Class Methods         
In AssertParamsNotNull AST Transformation in previous blog, I used source.getAST()?.getMethods()  what it did is that it only found top level (Script Level) methods. So AST Transformation did not apply to Class Methods, it only got applied to Script Level Method. For this one I changed to source.ast?.classes*.methods.flatten() (Line 12 in BeforeAdvisorASTTransformation). This also become apparent once you see AST structure for a given class using Groovy AST View in eclipse.

Retrive value from annotations
ASTNode being passed to visit method carries information about the annotation. First element (node[0]) contains information about annotation. You can get the values passed in to annotation during the visit method and use it during the transformation. In case above we get class passed in as value and it gets instantiated and before method gets invoked.

Groovy Compilation
When groovy compiler is invoked, any sourcefile.groovy goes under series of transformations.
From Source --> ANTLR Tokens --> ANTLR AST --> Groovy AST --> Bytecode

Using AST Transformation, we manipulate the way groovy AST gets generated. It allows to insert additonal statements.

There are various CompilerPhase that goes along with this process. I couldn't find much documentation on the process. Best information is found on Jochen Theodorou's blog post

Blogged with the Flock Browser