Kristian Kristensen’s Blog

I’ve been looking into IronRuby and how to retrieve the Abstract Syntax Tree (AST) that’s used. In this post I’ll show a simple example of how you can do the same.

First let’s look at what a compiler is and how it works. A compiler basically works in three phases: Lexical Analysis, Syntactic/Semantic Analysis and Code Generation. Given a piece of code the compiler first tokenizes the input. This means splitting the code into its simplest parts and it’s the Lexical Analysis phase. Examples of these parts could be a variable name, a string, an assignment operator (such as = in C#). The output from the Tokenizer is feed into a Parser in the Syntactic/Semantic Analysis phase. The Parser takes the individual parts – the tokens – and creates a generic representation of the code. This representation is the Abstract Syntax Tree (AST). In the last phase (Code Generation) the compiler uses the AST to generate code that can run on the target platform.
Usually each of these phases has multiple sub phases that further helps to validate, optimize, etc. the code.

If we take a modern language such as C# the input will be C# code in a .cs file and the output will be IL code that can run on the Common Language Runtime (CLR).
Now let’s take a look at an example of how the AST might look like for the simple assignment shown in the following code segment:

a = 2 + 3

After being parsed this code might have an AST representation as shown in the following:

  =
 / \
a   +
   / \
   2  3

The parts shown in the tree are called nodes, and there are 5 in total. A variable name, two integers (or literals as they’re called), a plus sign (or more specifically a Binary Expression) and an Assignment (the “=”).

Getting the AST is a first step in trying to analyze the code yourself. Since creating a tokenizer and parser for a language such as Ruby is a non trivial effort, it’s great if you can piggyback on others work. And for looking at Ruby code on .NET IronRuby is obviously perfect.

In the following code I’m using IronRuby 1.0 RC2, which is the latest release.
First we need to add a couple of assembly references, which you can find where you installed IronRuby (usually “c:\ironruby\bin\”). We need IronRuby.dll, IronRuby.Libraries.dll, Microsoft.Dynamic.dll, Microsoft.Scripting.dll, Microsoft.Scripting.Core.dll, Microsoft.Scripting.Core.dll, Microsoft.Scripting.Helpers.dll

Then import the following namespaces:

using IronRuby.Builtins;
using IronRuby.Compiler;
using IronRuby.Compiler.Ast;
using Microsoft.Scripting;
using Microsoft.Scripting.Hosting.Providers;

Next we’ll setup an IronRuby runtime and engine, this is the core infrastructure we need to work with Ruby code. Last we’ll create an instance of the ScriptSource class with a simple piece of Ruby code that calls “puts” with “hello” as a parameter.

var runtime = IronRuby.Ruby.CreateRuntime();
var engine = runtime.GetEngine('rb');
var src = engine.CreateScriptSourceFromString(@"puts 'hello'");

The ScriptSource is actually executeable, so if you called the method Execute() the simple Ruby expression would be executed. This is the basic pattern you’d follow if you wanted to add scripting capabilities to your application.

Using a helper class in the Dynamic Language Runtime (DLR) API’s we can access the SourceUnit instance. We create a new IronRuby Parser and ask it to parse the code found in the SourceUnit instance.

var srcUnit = HostingHelpers.GetSourceUnit(src);
var parser = new Parser();
var srcTreeUnit = parser.Parse(srcUnit, new RubyCompilerOptions(), ErrorSink.Default);

To access the AST we need to implement a Walker. This class implements the Visitor Design Pattern, and using it we can look at each of the nodes in the AST. A simple walker is shown below.

public class MyWalker : Walker
    {
        protected override void Walk(MethodCall node)
        {
            Console.WriteLine("Method call: " + node.MethodName);
            base.Walk(node);
        }

        protected override void Walk(StringLiteral node)
        {
            Console.WriteLine("String Literal: " + node.GetMutableString(RubyEncoding.Default).ToString());
            base.Walk(node);
        }
}

We kickstart the process by calling the Walk() method on our Walker implementation and gives it the SourceUnit as input.

var walker = new MyWalker();
walker.Walk(srcTreeUnit);

For each node type that the AST can contain a method is called on our Walker, and we can then do what we want. In this example I print the current node to the console.

The output of running this code is:

Method call: puts
String Literal: hello

The next step is to implement the rest of the methods on the walker, thereby supporting the full set of AST nodes. Also you’d probably want to build up your own data structure for analysis.
This example showed how you can access the parsed Ruby code using IronRuby. The next step is to actually do something interesting with this knowledge.

As part of my current project we’ve upgraded from Microsoft Team Foundation Server (TFS) 2008 to TFS 2010. Part of the project gungho is to illustrate the status of the current build. Hence we need some sort of programmatic way of accessing the build server history of TFS. This part has obviously changed  from 2008 to 2010, but also from 2010 Beta 2 to the Release Candidate. So I thought I’d detail how you can connect to the build portion of TFS 2010 RC.

First you want to add references to the new TFS SDK dll’s. They’re located in “C:\Program Files\Microsoft Visual Studio 10.0\Common7\IDE\ReferenceAssemblies\v2.0″. You’ll need:

• Microsoft.TeamFoundation.Build.Client
• Microsoft.TeamFoundation.Build.Common
• Microsoft.TeamFoundation.Common
• Microsoft.TeamFoundation.Client

We’ll need the following namespaces included:

using Microsoft.TeamFoundation.Build.Client;
using Microsoft.TeamFoundation.Client;
using Microsoft.TeamFoundation.Framework.Client; 

Then setup the connection to your TFS server and make sure we authenticate:

Uri tfsUri = new Uri("http://tfsserver:8080/tfs/");
var tfssvr = new TfsConfigurationServer(tfsUri, new UICredentialsProvider());
tfssvr.EnsureAuthenticated();

A new feature in TFS 2010 is project collections, that group team projects. Therefore we need to locate the project collection where our team project is located. I know that our collection is called “Default” so I just look for that.

TeamProjectCollection tpc = null;
var collections = tfssvr.GetService<ITeamProjectCollectionService>().GetCollections();
foreach (var teamProjectCollection in collections)
{
    if (teamProjectCollection.Name.Equals("Default"))
    {
        tpc = teamProjectCollection;
    }
}

Next we retrieve a reference to the build server service running inside our project collection. Using this we create reference to the build queue for our specific team project. We can filter this build queue so we only get information about builds in progress and builds that are succeeded. We can also hook up an event handler so we’ll be called whenever there are changes to the build queue; we’ll look at this event handler in a minute. Finally we connect to the build queue, thereby setting up our callback.

IBuildServer buildServer = tfssvr.GetTeamProjectCollection(tpc.Id).GetService<IBuildServer>();

var queue = buildServer.CreateQueuedBuildsView("MyProject");
queue.StatusFilter = QueueStatus.InProgress | QueueStatus.Completed;

queue.StatusChanged += new StatusChangedEventHandler(queue_StatusChanged);

queue.Connect();

The event handler that’ll get called whenever there’s a change in the build queue is located below. We cast the sender to the right interface so we can access the queue. We verify that there’s indeed a change in the build queue, and that we’re not just being pinged. We extract the latest build (the list is sorted by TFS), and then we simply print out some information about the build.

static void queue_StatusChanged(object sender, StatusChangedEventArgs e)
{
  var queue = (IQueuedBuildsView) sender;
  if (e.Changed && queue.QueuedBuilds.Length > 0)
  {
    var curBuild = queue.QueuedBuilds[queue.QueuedBuilds.Length - 1];
    Console.WriteLine("New build detected: " + curBuild.Build.BuildNumber + " is " + curBuild.Build.Status.ToString());
}

The interesting use case for this is of course to hook in your own customer build notification system in the event handler. In my current project we have a Nabaztag bunny, that’ll inform the team when a new build is in progress and when it’s finished. You can see a video of it on Youtube – IronBunny.

Ewald Hofman pointed me in the right direction with his posts on how the SDK works in TFS 2010 Beta 2.

Disclaimer: This code is just an example, and you really shouldn’t put it blindly into production. Basically the usual “Works on my machine”.

I held a TechTalk at the Microsoft office on Wednesday, and have promised to post the slides. So here goes.

Slides @ slideshare.net

The video I showed of the Nabaztag bunny being connected to a TFS server is embedded in that deck. However, you can also view it directly on YouTube.

IronBunny @ YouTube

If there’s interest, I’ll post my demos up here as well.