17 August 2012

GSoC Update #2 - XQMode for Processing

This post is a continuation of my previous post on my GSoC project. It’s been more than month since the last post. A lot of progress has been made with XQMode. I’ve completed all of my project goals and have added a couple of interesting features too.

XQMode in action

Here’s a brief description of what has been accomplished since the last update:
  • Error Underlining: Just like in eclipse, now errors in code are underlined in red as you type. Warnings are also underlined in yellow. A marker is also shown on the left of an error line which scrolls along with the line.
  • Errors List integrated into the PDE: Earlier the table of errors was being displayed in a separate window. Now the table has been integrated into the PDE window where the console resides. I’ve added a tab like view where one can switch between the console and errors list. The Problem window can still be launched from the XQMode menu.
  • A New Preprocessor:  This change is under the hood. I’m now using my own custom preprocessor that converts Processing code into compilable java code. It doesn’t make use of Processing’s pre-processor. So, confusing internal error messages from the Processing preprocessor are no longer displayed in the console. And this also provides a noticeable decrease in time required to perform error checking.
  • Default libraries that come with Processing are now fully supported
  • Import statements are now checked correctly for errors.
  • Imports from the code folder of a sketch are now correctly detected.
  • User can manually disable error checking as well as warnings.
  • Lots of minor improvements and bug fixes.

XQMode currently needs some modifications to Processing 2.0a8 in order to run. After discussions with Ben Fry, it’s been decided that XQMode will be launched in a few months after the incompatibilities are sorted out.

This was a pretty amazing GSoC for me. I was lucky to not have faced any major technical difficulties or heisenbugs and development was quiet smooth. Daniel Shiffman's (my mentor) guidance has been instrumental in developing XQMode. He used to regularly come up with interesting ideas and tweaks. Other members of the Processing team – Martin Leopold, Florian Jenett and Andres Colubri also provided valuable inputs and helped with internal testing.

It feels great to have made a cool little contribution to Processing! :)

XQMode on Github: https://github.com/Manindra29/XQMode


P.S: Returning readers may notice - I've got my own domain now, mkmoharana.com! :D

2 July 2012

GSoC Update #1 - XQMode for Processing

I’ve got the wonderful opportunity to work for Processing in this year’s GSoC. My mentor for this project is Daniel Shiffman. I’m developing real time syntax checking and compilation error checking functionality for the PDE (Processing Development Environment). These features are implemented as a contributed Mode. Here's a video preview of XQMode (alpha) in action:

But does it blend?

No. But imagine writing lines and lines of code, and to see the result, you hit run. Sadly, there’s some compilation error. The PDE compiler points to some ambiguous line in code, with a weird error message, like “unexpected token void”, “maybe missing a semi-colon ? ” etc. A missing semicolon or curly, or other typos, often have similar results. Well, all these days are gone. XQMode is here to help! Anytime you forget to add a semicolon, bracket, or make a typo, you’re immediately notified – syntax checking happens on the fly. You can correct it right then and there.

And what about compilation errors? Like passing a float variable to a method that needs an int? Or a typo in the function/variable name? XQMode has got you covered there too. With background compilation happening in real time, you get instant feedback! It even warns you about unused variables, methods, etc. leading to less cluttered code.

When you hit run, you can be sure that your code will always compile!

Under the Hood

Some of you might have understood by now this is very similar to the Eclipse Java Editor. That’s somewhat true. I’ve used the Eclipse JDT API for implementing these features. Eclipse JDT (Java Development Tools) is responsible for many cool features of Eclipse like generating AST, syntax checking, code-completion, etc. I’ve used the AST Parser for finding syntax errors and batch compiler for finding compilation errors. The following diagram shows the process workflow:

XQMode Error Checker Workflow
Here pre-processing means converting Processing code into pure java code. XQMode’s pre-processor grabs code from the editor and converts it into pure java, but not compile ready, code. This code is passed through the AST Parser. Eclipse’s AST Parser detects all syntax errors which are shown to the user. If no syntax errors are present, Processing’s pre-processor further converts this code into compile ready pure java code. This is then passed on to the Eclipse compiler which compiles and detects compilation errors which are reported to the user. The reason for using two pre-processors is that Processing’s pre-processor uses the ANTLR parser for conversion. ANTLR syntax error messages can look like Greek to the uninitiated. Therefore additional pre-processing helps in filtering out errors in a more user friendly manner.


Here’s what works:
  • Syntax Checking in the background running in a parallel thread. 
  • Compilation Checking in the background running in the parallel thread (mentioned above). 
  • Import statements for contributed libraries processed on every change. 
  • Errors reported in the Problems window in a tabular format. 
  • Error bar displayed on the right with error markers. 
  • Clicking on an error marker or from the table, scrolls to the error location. 
  • Mouse hover on error marker displays error message. 
  • Error Checking can happen with the Problems Window closed as well. 
  • Ctrl + Click on the Problem Window to pause error checking. 
  • Problem Window is “sticky”. Sticks to the Editor window #likeaboss.

Here’s what doesn’t work:

  • Sketches using OpenGL will not run. For some, this can be a deal breaker. You can edit code, but on hitting run, you’ll get a package not found error. This bug should be fixed in the next update.
  • Some error statements are not reported. For example, “color color = #ffffff” - It would be nice if you can help us find more such anomalies. We hope to fix them in future updates.
  • If an import statement for a contributed library is present in code but the library files are missing from the sketchbook folder, only error messages are shown. This would be improved.
  • Imports (additional jars) from “code” folder of a sketch are not detected.
  • Some error/warning messages may not look user friendly. We’re working to improve on that.

XQMode currently requires some modifications to Processing 2.06a in order to run. XQMode is still in development. The features and UI seen here may change in the final release.

Wish me luck!

Cheers! :)

6 June 2012

Google Summer Of Code 2012 – Processing

The last one month had been hell - the dreaded end semester exams. Thankfully I survived. And something unexpected happened in the mean time - I got selected into Google Summer of Code 2012! I'll be working for Processing. This was the first time I applied for GSoC and it feels great to be selected! :)

Google Summer of Code is a global program organised by Google that offers students stipends (hefty ones, that) to write code for open source projects. And the best part about the program is you don't need to be a computer science student or have really high CGPA to get selected. All you need is passion for coding and open source development.

I'd used Processing in a project during my internship last year. After the project, I decided to get into open source development after being inspired by an article on open source in the Digit magazine. Processing, being a newer community compared to heavy weights like Linux or Mozilla, was easier for me to get started with. Initially I built a simple tool (plugin) called Color Selector Plus. The response from the community was great. Also it's a great feeling to know your code is being used by people from around the globe. And I got in touch with some really impressive individuals who work on cool stuff. 

I then went ahead and downloaded Processing's source code and built it from the svn trunk. Things were complicated at first, but slowly I could figure out more and more details as I played around with the source. During November last year, I started to consider applying for GSoC. And it was a pleasant surprise when I found out that Processing was one of the participating organizations in GSoC! During Jan and Feb, I built another tool for Processing - Sketch Outline. This plugin was quite interesting to work on and a nice learning experience. 

By March, I'd started talking to past students for tips and to potential mentors discussing project ideas. And after many revisions, I finalized my project idea - Sketch Assistant. You can find my proposal here.

My mentor for this project is Daniel Shiffman, a Processing guru and one of the core contributors to Processing.

My project's git repo can be found here.

Looking forward to a summer full of hacking and coding!


Update [10/06/2012] - I'm sharing this video of Philip Johnson, a GSoC 2008 mentor and program administrator, providing an overview of the GSoC 2009 and explaining why you should definitely get involved. Very inspirational.

15 April 2012

Sketch Outline - Processing Tool

Processing is an open source programming language and environment for people who want to create images, animations, and interactionsInitially developed to serve as a software sketchbook and to teach fundamentals of computer programming within a visual context, Processing also has evolved into a tool for generating finished professional work. Today, there are tens of thousands of students, artists, designers, researchers, and hobbyists who use Processing for learning, prototyping, and production. It's highly recommended for learning the art of programming.

I've been a long time user of Processing. It's my favorite language when it comes to quick prototyping and testing out new ideas or concepts. The fact that it's Java based, one can use native Java code too. Processing comes with its own basic IDE designed for beginners. The IDE supports plugin like components called Tools which extend its functionality. I had earlier developed a tool for Processing - Color Selector Plus. I've now developed a second tool - Sketch Outline.

Sketch Outline running in Processing 1.5.1 (OS X Lion) and in Processing 2.04 (Windows 7)

Sketch Outline provides a tree view of the member variables, functions and classes within a sketch and allows easy navigation between them. Clicking on any node scrolls and moves the cursor to its definition in the code. This is kinda similar to the Outline View available in Eclipse and many other IDEs. It helps in easy navigation while writing code and very helpful for larger projects.

I've used java parser for generating the AST(Abstract Syntax Tree). An AST is a tree representation of the abstract syntactic structure of source code written in a programming language. Java parser library is simple to use and was a good learning experience for me regarding code parsers. But it has limited functionality, and hasn't been updated to Java 1.6. I'm currently working on a more advanced version of this tool that would use the Eclipse JDT core components. Eclipse JDT core is the back end of the Eclipse Java IDE, responsible for many of the cool features like generating the outline view, background compilation, real time error reporting, code completion suggestions, etc. For now, I plan to use its AST generation feature(it can recover from errors in syntax and still generate the AST!), and batch compilation for reporting syntax as well as compilation errors in real time. 

You can grab the latest version from here. If you're using version 2.0 of Processing or higher, you can also get it from within the IDE by going to Tools menu → Add Tool. Sketch Outline is also listed on Processing's tools page.

The project is open source and hosted on Google Code. Try it out and feel free to give any suggestions or report any bugs in the forum thread, in the issues section of the project or as a comment. 

Cheers!  :)

7 April 2012

Ninja Air Hockey

Today, I'll be sharing a fun little game I'd made some time back - Ninja Air Hockey. It's your regular air hockey with a twist - You control your bat not with the mouse or keyboard, but with some colored object! Sounds weird? Check out its video in action:

I've decided to make it open source. Ninja Air Hockey was designed using openFrameworks and OpenCV. openFrameworks is an opensource C++ toolkit for creative coding. 

According to its authors, openFrameworks was developed:
(for) folks using computers for creative, artistic expression, and who would like low level access to the data inside of media in order manipulate, analyze or explore. That audience we felt was significantly underserved by the current crop of C++ libraries.

oF is designed to be used on Windows, Mac OS X and Linux and supports XCode, Code::Blocks, Visual Studio and EclipseI had used an older version of openFrameworks - v0062 on Visual Studio. This version is archived here. Here's how you can set it up on Visual Studio:

  • Download the Visual Studio 2010 FAT(v0062), and unzip it. 
  • You need to have OpenCV set up as well. For instructions on how to install OpenCV on Windows, check out the earlier post. Make sure you've Quicktime installed too.
  • Download the source code from google code here.
  • Copy AirHockey folder into apps/addonsExamples.
  • Open AirHockey.sln, wait for the project to load. Press F5 to run. When running for the first time, it can take upto 2-3 mins to start since Visual Studio will build the openFrameworks libraries first.

The all important source files are present in the AirHockey/src folder. And data files are present in the AirHockey/bin/data folder. Using them, you can build this project on the platform and IDE of your choice. Please refer to openFrameworks wiki for instructions on how to set it up on your platform and IDE.

The code is well commented and should be fairly easy to understand. For any questions, feel free to comment. :)


20 March 2012

Arduino Remote Controlled Mouse and Keyboard

Today I'll be sharing a small hack I did about a year back. I'll be showing how you can control your PC's keyboard and mouse using an IR Remote and Arduino. Here's the youtube video of it in action: 

Parts list:
  1. IR based Remote(This could be a remote of a TV, DVD Player, AC, etc) 
  2. TSOP sensor
  3. One 100 ohms resistor
  4. Breadboard
  5. Connecting wires
  6. Arduino
  1. Visual Basic 2010 Express(get it for free here) or Visual Studio
  2. Arduino IDE
The connections are dead simple, all you need to do is connect three wires: 
Vcc is to be connected to Arduino 5V, GND to Arduino GND
TSOP Sensor

Download the complete source code from google code here.

Now, comes the major editing part. We'll be using the IR Remote library for Arduino by Ken Sheriff. This library helps in easy decoding of IR remote signals. You need to install this library(download link and setup instruction on the library page) and I recommend you to go through the examples and get a brief idea about how IR signals are getting decoded before proceeding further.

Since my remote used a non-standard protocol, I obtained the hash code for each button of my remote. Each key when pressed fires a series of precisely timed IR pulses and this pulse train is converted to a unique hash code by this library.

In Remote_Mouse.pde, you'll find that samsung_hash[] contains the hash code values of buttons of the samsung remote I had used. These hex values for buttons are different for every remote. So you'll need to use processKeys() inside loop() initially to obtain the hex values of buttons for your remote. After you obtain them, and update the code with your remote's hash code values, processMouse() can be used.

After you're done, to run the PC application, open MouseSimulator1.sln. I've used the MouseKeyBoard library by codeSummoner to emulate keyboard and mouse from the vb application. The program is well commented and is pretty basic.

Press F5 to run the program. Select your Arduino's com port and click on Start Remote Mouse.

Hope someone finds this useful.

Cheers! :)

3 January 2012

Setting up OpenCV 2.3.1 on Visual Studio 2010

[UPDATE: This post is fairly old now. I wrote a new post on setting up OpenCV in Visual Studio 2012 here.]

OpenCV is an open-source C++ library used for computer vision based applications, and one of the most widely used ones in its field. But setting up this awesome library and getting it running is not a simple task for beginners. This post will show you a method to setup OpenCV on Visual Studio without losing one's sanity. Please note that we won’t be setting up some advanced features of OpenCV like TBB, IPP or GPU support. Those are generally required by advanced users.


Before starting, get the prerequisites:

  • Get the OpenCV 2.3.1 windows super pack from sourceforge. Get it from here.

  • You need Visual C++ 2010 installed for using OpenCV. It’s a part of Visual Studio 2010 bundle. You may download only Visual C++ 2010 or the entire bundle. Students can get the professional edition of Visual Studio for free(instructions). Express edition can be downloaded here. You can work with both professional or express edition. Professional edition is paid while Express edition is free.

  • Download the OpenCV Helper app and some required project files from google code here.

  • STEP ONE: Extract

  • Run the OpenCV super pack.exe, select a directory where OpenCV would be extracted. I’d suggest something simpler like C:\ or any other root. Click Extract.

  • This step is important. A folder named "opencv" must have been created in the location you selected. Make sure you rename it to OpenCV 2.3.1.  

  • snap6

    STEP TWO: Setup Environment Variables
    The next step is to setup Environment Variables. These are some variables specific to the user account that store the location of important files which are required by multiple applications. We’re setting up variables related to OpenCV to make life easier for VC++. You can setup the variables manually(described in a later section) or you can use OpenCVHelper – a program I wrote which does this automatically. It’s present in OpenCV_Helper.zip. Make sure you close Visual Studio before using it.


  • Click on the browse button and select the folder where you extracted OpenCV. Make sure you select the folder containing the OpenCV directories. The extractor extracts it to a directory called opencv in the location selected during extraction.

  • Select 32-bit or 64 bit according to the version of OpenCV you wish to use. If you’re using the Express edition of VC++, use 32-bit since it doesn't support 64-bit.

  • Click on Set Environment Variables. 

  • STEP THREE: “Hello World”
    Now we’re ready to create our first project. Fire up VC++.

  • Click on ‘New Project’ in the Start Page or press Crtl + Shift + N.

  • In the New Project window, select ‘General’ under Visual C++ and select the Empty Project template. Enter a name for your project. Make sure Create Directory for your Solution is checked. Click on OK.

  • Now go to Tools menu, Settings -> Click on Expert Settings. Now go to View menu-> Property Manager. The Property Manager window should be visible now. For Visual Studio users, View -> Other Windows -> Property Manager
  • snap7

  • Right click on Debug, select ‘Add Existing Property Sheet’. Browse to and select the OpenCV 2.3.1 Debug.props file. (located in OpenCV_Helper.zip)

  • Similarly, right click on Release, select ‘Add Existing Property Sheet’. Browse to and select the OpenCV 2.3.1 Release.props file. (located in OpenCV_Helper.zip)

  • Now open Solution Explorer (Crtl + Alt + L), right click on Source Files, select Add –> New Item. Select C++ File, name it HelloWorld, click on Add. Navigate to main.cpp (located in OpenCV_Helper) and copy-paste its contents in this file.

  • That’s it! Hit F5. If you followed the steps correctly the webcam should fire up. If you don’t have a webcam attached, a message would be displayed. Hit Escape to close the window. 

    Setting Up Environment Variables Manually

    In case you want to set up environment variables manually, you can follow these steps:

  • In Windows 7/Vista, open System Properties. Select Advanced System Settings –> Environment Variables. In Windows XP, right click on My Computer –> Properties. Under Advanced tab, select Environment Variables.

  • We’ll need to set four variables. Under User Variables, click on New. If any of the below variables already exist, add a semi-colon ’;’ to the value and append the new value.

    OpenCVFolder path where you extracted OpenCV
    LIBFor 32 bit: %OPENCV%\build\x86\vc10\lib 
    For 64 bit: %OPENCV%\build\x64\vc10\lib
    PathFor 32 bit: %OPENCV%\build\x86\vc10\bin
    For 64 bit: %OPENCV%\build\x64\vc10\bin

  • For anyone further interested, the Property Sheets that we added to our project are a type of configuration files. They tell the compiler where to look for header files and other dependent libraries related to opencv when we use them in our projects. Also since the location of opencv folder has been setup as an environment variable, you can change the location of your opencv folder if you require. You'll only need to update the OPENCV environment variable and restart VC++.

    If you want to setup further features like TBB, Nvidia GPU support, etc. there's a wonderful page describing the steps in detail here.

    Do post a comment if you successfully setup OpenCV or if something went wrong.
    Cheers! :)