Experimental Mode Plus - GSoC 2013, Part 1

I’ve been selected for the second time in Google Summer of Code to work for Processing! I'll be working with my last year's mentor Dan Shiffman. In this blog post I’ll be talking about my GSoC project, its goals and major challenges. I’m also pleased to announce that I’m now an official member of the Processing developer team! :)

Project Idea

There are some important features of popular IDEs that I've always missed in the PDE, which make me switch to Eclipse whenever I think of writing a big sketch. Last year I had developed XQMode which now ships with Processing as Experimental Mode (along with Debug Mode). It was appreciated, but it mostly benefits users who are learning to code in Processing. I had always felt that it still isn't good enough to really attract advanced Processing users and make them choose the PDE(Processing Development Environment) over Eclipse for writing sketches. The job seemed half done. Some of the most sought after features were still lacking. This year I’m working to further improve the PDE as a modern code editor and implement more advanced features – intelligent code competition, basic refactoring, live colour editor, quick navigation and a few other features to improve the overall editor experience.

Major Goals

• Intelligent Code Completion
• Refactoring for variables and methods
• Suggestions for missing import statements
• Enhanced code navigation - Ctrl + Click to jump to declaration, Quick Find - as seen in Sublime Text, Eclipse, etc.

(Follow the progress on GitHub)

Quick Navigation

I decided to implement this first hand, since locating the declaration of every variable/method or class was fundamental for supporting code completion. Using Eclipse’s JDT SDK, I obtained the AST of the sketch code. AST stands for Abstract Syntax Tree - a tree representation of the entire code. The main java class is the root node, its methods and fields are child nodes, local variables defined inside methods are child nodes of the method nodes, etc. So by using regular tree traversal techniques, one can find the required node as well as its declaration.

Offset mapping between PDE and Java Code

Processing tries to make Java a bit more beginner friendly by slightly enhancing the Java syntax. So instead of starting hex values (colour codes) with 0xff, a # literal is used. Similarly, for casting to default types, int(), float(),boolean(), etc methods are supported in Processing. This is bit of a syntax trickery; before compiling the sketch code, Processing’s pre-processor finds these enhancements in code and substitutes them with their Java equivalent. So all # literals are replaced with 0xff, int() is replaced with PApplet.parseInt(), etc. But since all this is done behind the scenes, the user doesn’t have to bother. There are some other enhancements involved like scrapping imports, taking care of method scope, etc.

But these pre-processer enhancements create a challenge when using the Eclipse JDT SDK. The AST provide offsets for each node, but all of these offsets are w.r.t to Java version of the sketch code. Whereas the user only sees the sketch code in the PDE. For ex, the following line in Processing code:

color topColor = #ff00ff; int count = int(23.5);

Would be converted to pure Java as:

int topColor = 0xffff00ff; int count = PApplet.parseInt(23.5f);

So, obtaining a clear mapping between Processing code in the editor and Java code(used internally in Experimental Mode) was essential. I’d already implemented PDE code line number to Java code line number mapping in the previous version of Experimental Mode. This time I’d to match offsets in a single line. This proved to be a bit tricky but I worked it out in the end. The important tools for auto-completion were now ready.

Intelligent Code Completion

This is the most challenging  goal of the GSoC project. Some regular code editors (ex, gedit on Ubuntu) provide basic code completion, which consists of scanning the text buffer and creating a prediction list from the most used words. But my goals is to provide code completion that is "intelligent" - i.e., it should be context aware. It should provide meaningful and valid suggestions depending on the current code's context.

This consists of providing completion for two kinds of code:  compiled code (Java libraries in .jar files) as well as local code (present in the editor). Both of these required a different approach. I’ll be talking about local code completion in this post.

The AST of the sketch forms the backbone of code completion. Code completion is accomplished through clever traversal of this tree.

Local Code: I initially started working on code completion for local code to test the feasibility of the idea. Proper usage of Eclipse’s AST features turned out to be very difficult in the beginning because of the myriad of options and the incredible complexity (after all, Eclipse is designed by some of the most brilliant minds of the software industry). But as I worked my way through, I began to get the hang of things and how its classes were related. Initially I started with providing completions for single words like the name of a local variable, method or class; the most basic type of completion. After this, I tried to provide completions for class member access, i.e., supporting the dot operator. For ex: Let’s say I’ve defined a local class named Network. 

Network netw = new Network(); 

netw.display();

network is an object of a local class Network. And did you notice the awesome new toolbar background?

This was not too difficult, I had to first infer the type of the element before the dot operator. If it was an object of a class, get the class type. If it was a method, get its return type. Then look within this type for members – fields and methods.

Then I moved on to supporting dot operator chaining. For ex: 

a.b().c().to.infinity.and.beyond()

This expression needs to be resolved step-by-step, starting with determining the type of a, look for b() within it, find b()’s return type, then find c() within that and so on, to finally obtain the return type of beyond(). Although I had a rough idea of a solution to the problem, writing it down in code proved to be no easy task. It took me quite some time to work out all the edge cases. After some iterations, I finally made a single method - resolveExpression() that got the job done using some clever recursion. But note that this was all just for local code only. Let’s say I want to show completions for ‘network.toString().’, it won’t recognize it yet, since toString() is a member of Object class – a pre-defined class, whose definition is not present in the editor code. The code completion feature was just halfway done. I’ll talk about completion for predefined classes, and some other project goals in my next blog post.

Cheers!

Manindra