Software, Finance, Medicine

Hackathon in Delft: Gone!

Tue, 31 Jan 2012 18:00:05 +0100

I’ve been at it again. In the previous autumn/winter season, I spent a month in Delft, hacking on the Stratego/Spoofax program transformation infrastructure.

Now that another winter is here, I’ve had the good fortune to repeat what is slowly becoming a tradition. This year I even found time to reserve two months for a more extended hackathon.

Many really good things came of out of the stay. A few of the immediate results are:

A command line interpreter for Spoofax that works on Linux/Unix/Windows.
An interactive Spoofax console for Eclipse.
A major revamp of the Spoofax build system that makes both development and bootstrapping a lot easier.
A highly experimental plugin for scripting source code analysis and manipulation inside of Eclipse.

I am also involved in several on-going projects for bringing the necessary parts of the Spoofax infrastructure to the web. Interesting results should start popping up over the new few months.

As is the case for most startups, finding the time to contribute back to the upstream projects we use is always difficult. Whenever this happens, though, the payoff is often substantial.

An Eclipse Console for Spoofax

Mon, 23 Jan 2012 18:34:58 +0100

More good news, everyone! I have found time to integrate the command line shell for Spoofax into Eclipse. You can now have all sorts of tricky conversations with the Spoofax interpreter inside a perfectly innocent-looking Eclipse console:

As you can see, there are still rough edges to be ironed out. One of them is clearly the color palette. Another is the lack of inline rules, which are not supported yet. Neither are dynamic rules (and it is doubtful they ever will be — we are currently exploring other opportunities for dealing with context-sensitive rewrite rules).

You can bring up the actual console by using the console page selector in the top right-hand corner of your console, like you do for the other types of consoles:

Remember that you can always report the issues you come across in YellowGrass and tag me (@karltk).

A Command Line Interpreter for Spoofax

Mon, 23 Jan 2012 18:15:54 +0100

Updated with screenshot! Also take a look at the Eclipse Shell.

Good news, boys and girls! If you happen to be a Stratego and/or Spoofax user, you might appreciate that I finally took some time to piece together an interactive command line interpreter—-a REPL—-for Spoofax.

There are numerous other things you can do in the shell. The help “page” gives a short summary.

1/ok> :help
 :help                       -- print this page
 :forget var1 var2 ... varN  -- forget specific global variables
 :forget strat/(n,m)         -- forget strategy with arity (n, m), e.g :forget zip/(1,0)
 :forget _                   -- forget all global variables
 :arity strategy             -- show available arities for a strategy
 :strategies                 -- show all global strategies
 :vars                       -- show all global variables

Now that I’m a full time Spoofax-user, low hanging productivity fruits suddenly seem a lot more ripe for the picking. I’ve personally been using Stratego Shell for a long time for my interactive needs, but it was getting increasingly out of date. Enter the Spoofax Interpreter, which runs on the JVM.

With a good portion of help from Rob and Eelco, I set up a build job for this thing in the hydra build farm, so that the latest builds will always be available.

There are bound to be a bunch of stuff that doesn’t work. When you find issues, please report them in YellowGrass and tag me (@karltk). I’m especially interested in how well this works on the more obscure developer platforms, such as Windows and OSX.

A sample contribution could be a cmd script that makes this thing easily start on Windows. Hint, hint.

"One downside to the Kindle is that you can no longer throw a bad book across the room in disgust."

Thu, 12 Jan 2012 23:24:49 +0100

“One downside to the Kindle is that you can no longer throw a bad book across the room in disgust.”

- Lisa L

A Year of Kindle

Tue, 03 Jan 2012 23:30:08 +0100

I’ve had my Kindle for exactly a year now. My experience so far: it’s really great! After a multi-year long lull where I didn’t do any reading (except for school and work), I’m now back into a rather steady reading pace.

A few highlights from 2011:

Slaughterhouse Five, Kurt Vonnegut
Too Big to Fail, Andrew Ross Sorkin
The Big Short, Michael Lewis
Liar’s Poker, Michael Lewis
The Prince, Niccolò Machiavelli (Project Gutenberg)
At the Mountains of Madness, Howard Phillips Lovecraft (Gutenberg)
The Call of Cthulhu, Howard Phillips Lovecraft (Gutenberg)
The Art of Money-Getting, Phineas Taylor Barnum (Gutenberg)
Anthem, Ayn Rand (Gutenberg)

It’s the 6” Kindle that I use the most, though I also have a 9.7”. The 6” is so small that I always take it in my backpack when I travel anywhere (which I find myself doing a lot). Compared with all other battery-powered devices I have, the battery life is phenomenal. Easily lasts a month, even with a decent amount of reading.

Of course, the Amazon store is great. Being able to get a new book practically anywhere you go, at a moment’s notice is sweet. For example, I got really bored on the bus in England this summer, so I got a new book straight off of Amazon then and there — no extra charge though I downloaded it over 3G internet in a foreign country.

But for all the greatness of the Amazon store, I must perhaps say that I appreciate the freely available books even more, especially those you get through Project Gutenberg. I’m slowly working my way through old classics like The Age of Reason and The Republic. I never managed to do that with paper books, simply because I didn’t bring the books along all the time. Now, when I have the inclination for some moral philosophy, I actually spend ten minutes or an hour reading further in, say, The Theory of Moral Sentiments.

I concede that there are many reasons why I wouldn’t want to buy a Kindle, but the fact that I read a lot more—and even enjoy the reading itself more—outweighs all those reasons for me.

"Hold snudlene på dåkkår!"

Thu, 15 Dec 2011 16:00:06 +0100

“Hold snudlene på dåkkår!”

- Håvard Sørbø

Hambrosgade, København

Sun, 04 Dec 2011 20:55:26 +0100

Hambrosgade, København

"Men etter noen måneder måtte vi se inn i realitetens blodskutte øyne."

Thu, 07 Jul 2011 14:48:32 +0200

“Men etter noen måneder måtte vi se inn i realitetens blodskutte øyne.”

- John Ivan Wiik

"… long before any human race as we know had shambled out of apedom."

Sat, 02 Apr 2011 12:40:57 +0200

“… long before any human race as we know had shambled out of apedom.”

- Howard Phillips “H. P.” Lovecraft

"Gjør som meg: spis deg tjukk, så fryser du ikke."

Sat, 26 Feb 2011 22:52:13 +0100

“Gjør som meg: spis deg tjukk, så fryser du ikke.”

- Uno Wackelin

"How can NASA send a spacecraft billions of miles through the solar system and somehow end up flying..."

Sat, 26 Feb 2011 18:42:12 +0100

“How can NASA send a spacecraft billions of miles through the solar system and somehow end up flying so close to a comet only a few kilometers in diameter? It’s done with math.”

- Ed Weiler, NASA Science Mission Directorate

"It’s not the end, but really the journey that matters, in the end."

Tue, 11 Jan 2011 20:38:00 +0100

“It’s not the end, but really the journey that matters, in the end.”

- Lennart Kats paraphrasing Ursula K. Le Guin

Merging a Mercurial repository back into Subversion

Thu, 16 Dec 2010 17:03:00 +0100

In a recent project, I forked an upstream SVN repository into a new Mercurial repo, to do some exploratory programming. The exploration proved very fruitful, so I decided to merge the code back into the upstream SVN repo.

Should be simple, right? Wrong. For posterity, here’s a high-level walkthrough that might also be of interest to others.

The Premise

Your case might differ from mine, so YMMV. Here’s my case:

I copied a specific SVN revision (no history) into a freshly created Mercurial repo.
All subsequent development happened on the Mercurial repository (multiple developers, resulting in multiple tips and several merges along the way).
At the end, we wanted to “replay” the revision history from the Mercurial repository back into a new subdir in the upstream SVN repo, under branches/.
I wanted to retain as much of the Mercurial revision history as possible, and make it look like all the development had happened in SVN all the time.

I almost got what I wanted. The only caveat was that I ended up having to collapse parts of the Mercurial revision history: The stretches in the revision history with multiple concurrent branches were collapsed into one revision. We only had a few cases of this, so it wasn’t really a biggie.

Tooling

After some investigation and a few stranded attempts, I ended up with the following Mercurial extensions for the job:

hgsubversion — checked out source, then ran ./setup.py install --user
rebase — part of the default Mercurial distro, but must be enabled (see below).
graphlog — same story as for rebase.
collapse — just checked out source, and pointed Mercurial to it.

For reference, this is the relevant section of my $HOME/.hgrc file:

[extensions]
rebase =
hgsubversion = /home/karltk/.local/lib/python2.6/site-packages/hgsubversion
hgext.graphlog =
collapse = /home/karltk/apps/hgcollapse/hgext/collapse.py

Setting up for merging

I ended up using two local working directories for the job, plus copies of several intermediate stages to get me back in track whenever I made mistakes.

We started with prepping a new branch in SVN, containing the exact revision that the Mercurial repo had been created from. This branch lives in https://path-to-your-svn-repo/branches/new-branch.

Then, the local working copies were simply created with:

hg-repo, initialized with hg clone https://path-to-remove-hg-repo hg-repo
svn-repo, initialized with hg clone https://path-to-remove-svn-repo/branches/new-branch svn-repo

Thanks to the hgsubversion extension, Mercurial can pull from and push to SVN repos. The remote end remains a SVN repo, but locally you work on a Mercurial repository. However, you immediately run into a big feature mismatch between SVN and Mercurial: you cannot push to a remote SVN repository, unless the local Mecurial revision history is completely linear — i.e., no branches nor merges in the local history.

Linearizing the Mercurial revision history

Since SVN only accepts a linear revision history, any non-linearities (shown as parallell lines when you do hg glog) found in the hg-repo must be ironed out. This is where hg collapse comes in. By looking at the branch and merge revisions in the glog, I collapsed parallell areas of the revision history manually. E.g. hg collapse -r 82 -r 89 -f, drops you into the commit editor and allows you to manually merge the commit messages for the collapsed revisions.

Once hg glog showed a linear history, I was ready to merge.

Doing the merging (rebasing!)

Now that we have a linear revision history again, we can go ahead and merge the Mercurial repository into SVN. Except, we cannot merge. We must rebase. A merge would introduce a non-linearity in the revision history, stopping us dead in our tracks.

The procedure I found went somewhat like this:

# cd hg-repo
# hg push -f ../svn-repo
# cd ../svn-repo
# hg glog
# hg rebase -b  -d

When looking for you are looking for the latest revision from the Mercurial repo changes. The is really the latest revision of the SVN repo changes. What we want to do, is to tuck all the Mercurial repo changes onto the last SVN change. And this is exactly why we are using rebase and not merge.

In my case, the was 0 and the was 83.

When I ran hg rebase -b 83 -d 0, there were numerous merge conflicts — because rebase uses merge behind the scenes. I just resolved these normally using hg resolve, and then continued the rebasing using hg rebase -c. After a few more conflicts and resolutions, the processes ended.

During the rebasing process, frequent copies of the entire svn-repo directory tree is advised. In essence, you’re doing destructive updates to your local copy of the svn-repo, and you can only check in your result after all the rebasing is finished. Having a safety net proved quite useful, at least for me:)

After the rebasing was complete, hg glog told me that everything was linear and good. At this point I did

# hg outgoing
# hg push

If you see “Sorry, can’t find svn parent of a merge revision.” when trying hg outgoing or hg push, it probably means that your revision history (as told to you by hg glog) is non-linear. These parts should be hg collapse-ed away.

Hackathon in Delft: Go!

Tue, 02 Nov 2010 20:33:00 +0100

It’s that time of the year again. The glorious month of intensive parser implementation, compiler engineering and language workbenches — the essentials of any IDE — has arrived. I’ve retreated to the TU Delft campus for the month of November to hack on interactive language infrastructure for our startup, and to think big thoughts about IDEs and DSLs in general.

Our startup is a user of both Stratego and Spoofax, so it was only natural to join forces with Eelco Visser and some of his henchmen here in Delft, who are the maintainers of said projects.

For somewhat practical and somewhat nostalgic reasons, I decided to stay at the TU Delft campus. Campus life here already brings back memories of my year living at Cambridgelaan: the access to lightning fast broadband in your dorm room, the four minute walk to the lab, the 24-hour party people (aka Erasmus students), and the freedom to follow your biological clock.

Time for some commits:)

DIY: Fixing the MSN connector for Empathy

Thu, 21 Oct 2010 00:36:13 +0200

I just got bitten by bug 663670: empathy doesn’t connect to msn - blocked connection.

It turns out that Microsoft made changes on the server side two days ago that crashes telepathy-butterfly, the MSN connector used by Empathy.

Fortunately, the fix is simple.

# sudo nano /usr/lib/pymodules/python2.6/papyon/service/description/SingleSignOn/RequestMultipleSecurityTokens.py

Then goto line 24 and replace

CONTACTS = (“contacts.msn.com”, “?s=1&id=24000&kv=7&rn=93S9SWWw&tw=0&ver=2.1.6000.1”)

with

CONTACTS = (“contacts.msn.com”, “MBI”)

After that, hit the “reconnect” button in Empathy and it should now connect to MSN. You don’t have to restart Empathy, even. These are the times I love scripting languages.

"Det er fakta. Men det er ikke rett."

Sat, 16 Oct 2010 02:17:32 +0200

“Det er fakta. Men det er ikke rett.”

- meg selv

"Ingenting i livet er gratis. Ikkje ein gong det besta."

Fri, 15 Oct 2010 20:37:29 +0200

“Ingenting i livet er gratis. Ikkje ein gong det besta.”

- Håvard Sørbø

Prototype of a Scannerless, Generalized Left-to-right Rightmost (SGLR) derivation parser for JavaScript

Wed, 06 Oct 2010 19:57:00 +0200

Need a Scannerless, Generalized Left-to-right Rightmost (SGLR) derivation parser for JavaScript? Then my JSGLR-GWT proof-of-concept might interest you. (If you don’t know what a scannerless GLR parser is, then you are probably not interested in writing modular grammars for your programming language implementation anyway — or you might want to read a few sections down).

A word of warning: I’m not claiming it is written in JavaScript, merely for JavaScript. I simply took our old Java-based SGLR implementation and massaged it quite heavily until GWT could spit out a working version for the JavaScript interpreters.

Performance

The performance is, quite frankly, beyond my wildest expectations. I’ve not made a single attempt at performance tweaking for the JavaScript engines yet, and still I manage to parse through 830 lines of Stratego code in ~3 seconds (Chromium 7.0.519.0). For smaller fragments, even Firefox 3.5 manages to keep up: it manages ~20 lines in ~1s (Chromium does that test in 74ms).

To most people, this probably doesn’t sound very fast at all, but I was much worse off when I initially wrote JSGLR back in 2006 — and that used a term(tree) library that wasn’t hacked together over night. All in all, I think there are plenty of performance tweaking opportunities, and the JavaScript engines are getting faster almost by the minute.

Now, another reason for my optimism, is my use case. I’m interested only interested in JavaScript port of SGLR for interactive use. Think web IDE. My aim is to attain interactive speeds when editing text. A few optimization ideas off the top of my head:

Tweak the parser to only reparse the lines that have changed. This alone might be enough to survive with the current level of performance.
Reduce the extreme amount of garbage created currently by using object pools.
Do the initial parse on the server-side, at least for larger files.

What is a Scannerless GLR parser anyway?

(optional reading)

SGLR does not separate your parser into the usual two steps: (1) lexer/tokenizer/scanner followed by (2) LL- or LR-style parsing of the token stream. Instead, SGLR runs the parser on every character. This is why it’s scannerless.

The next point, GLR, is tricker to explain briefly. Remember that a parser takes a string of text and creates a (parse) tree for it. There might, at some stage during the parsing, be multiple possible trees that might fit the text seen so far. This is called an ambiguity. We don’t yet know what is meant by the text. Only by continue reading, will it (hopefully) become apparent. However, this is where the k comes in. LL(k)/LALR(k) lets you look only k tokens into the future. GLR allows you to look all tokens into the future (for SGLR, I should probably say all characters into the future.).

Typically, LL and LALR-parsers, such as the archaic Yacc and Bison parsers, force you to contort your grammar a lot to meet the k-token lookahead. If you have ever run into the dreaded “shift/reduce” conflict, you know what I’m talking about.

The combination of scannerless and GLR gives us something wonderful: grammars are closed under composition. So, if you write grammars for languages L1 and L2, you can just make a new module L3, include L1 and L2, et voila — you have L3 = L1 ∪ L2.

Trying such a stunt with LALR will almost invariably push you into the “shift/reduce”-conflict corner. Admittedly, there is a significant trade-off or gotcha with the (S)GLR approach. And it is very similar to the trade-offs/gotchas with early vs late binding, or static vs dynamic typing.

If, after processing the entire text, there still are multiple possible parse trees, GLR will give you all of them. So, you catch the ambiguity at runtime. But you can use GLR to say more: the set of grammars you can express with GLR is larger than that of LL(k) and LALR(k).

The trade-off is simple: with LL(k)/LALR(k), you are guaranteed to only have one, unambiguous, tree at the end — but the you might not be able to express the language you want to design. With SGLR, you are free to express the full class of context-free grammars, but the parsing result might be a forest of trees. It’s up to you to prune the forest afterwards, finding the tree you want.

I’m not claiming that SGLR is the best solution for all kinds for parsing needs. With great power comes great responsibility. You might not always need great power, and great responsibilty is usually just a liability, anyway;)

Implementation Details

To interested parties, this is what I did:

I ripped out the dependence on the CWI ATerm library and (yet again) wrote my own. Mostly hacked together based from scratch, but with the Textual ATerm Format (TAF) parser stolen from the BasicStrategoTerm implementation.
Ripped out all references to java.io and replaced with my own stuff, based on strings.
Wrote a tiny RemoteServiceServlet to allow the client side to access the parse tables from server side.
Added a tiny hack to also fetch parse tables as TAFs and load these on the client side.
Various other minor fixes all over the place.

"Ahhh! Thermite; the stuff that dreams are made of!"

Tue, 28 Sep 2010 23:35:26 +0200

“Ahhh! Thermite; the stuff that dreams are made of!”

- Kurt Kristian Lyngholm

Great things in the Eclipse JDT

Sun, 12 Sep 2010 23:50:50 +0200

Since we started with the KolibriFX project six months ago, I have programmed a lot of Scala and Java, plus some Stratego and JavaScript. I love programming in Scala and Stratego because my implementations are shorter and usually a lot easier to reason about. I often get this warm, fuzzy feeling of having captured something tricky in an elegant fashion.

However, in day-to-day programming, Java has one killer feature that makes up for a lot of cruft in the language: the Eclipse JDT. I recently spent a few days rewriting some of our Scala code to Java, and that really brought the point home. I find that, at least for our code-base, the JDT offers productivity benefits that far outweigh the linguistic disadvantage Java suffers compared to Scala.

The points that follow will be an old hat to programmers familiar with the JDT, though they might serve as a reminder that for all its flaws, the JDT is a rather great tool, after all.

Realtime Compilation

This is probably the biggest tool-provided productivity booster I know of. I’ve always been a huge fan of realtime compilation, because it encourages flow. It brings me into “the zone”, “the flow” or whatever you prefer to call it. Whenever I have to wait for the compiler for more than a few seconds, I lose my flow and switch to another desktop “to remain productive”. This invariably forces an expensive mental context switch, and ultimately reduces my productivity, but without reducing my perception of productivity — I’m still doing stuff, so I feel I’m moving forward, except I’m not.

Realtime compiler error messages and warnings, allow me to mindlessly work through the live updated list until it’s empty. At that point, I run my unit tests and continue flowing until they’re all green. I get to decide how quickly I work. The computer (nearly) always follows my pace, so I’m the performance bottleneck. It makes me feel in control. I easily enter the flow, and I stay there. One thing I use as a flow marker is how often I watch the time. With Scala, 5 minutes rarely go by before I check it, since I’m waiting for the compiler to churn through. With the JDT, an hour may pass, easily.

This is not a bash on Scala. Stratego has the same problem, but we’re working on an incremental compiler to improve the problem there. I predict that both Scala and Stratego will eventually have compilers working at interactive speeds, and the same degree of flow will be attained as with the JDT compiler.

Quick Fixes

Beyond realtime compilation, the JDT offers additional inspiration for up-and-coming languages: its refactorings and quick fixes make life with Java easy and nearly enjoyable. Based on my usage the last week, these are my favourite quick fixes (admittedly, this list changes from week to week, based on the nature of the programming tasks):

Assign to Field

class X {
   X(int y|) {
   }
 }

With the cursor placed after y, quick fix suggests the creation of a new field, private final int y, and also adds the line this.y = y to the constructor body.

Extract to Separate File

File X.java:

class X { ... }
class Y| { ... }

With the cursor after Y, quick fix suggests that the class Y be moved to a separate file, Y.java, and it deals with all the imports at the top of both X.java and Y.java.

Generate Getters and Setters

class X {
   private int x;
   private int y;
}

Creates getX/setX and getY/setY. You can control if you want both get and set on a per-variable basis.

Change Visibility to ‘x’.

File foo/X.java:

package foo;
class X {
   Horoscope computeHoroscope();
}

File bar/Y.java:

package bar;
class Y {
   ...
   x.computeHoroscope()|;
}

If you are in another package and invoke, x.computeHoroscope(), quick fix tells you that friendly access is not possible, and suggests adding public to the computeHoroscope() method declaration.

Add/remove Argument to Match.

File IntPair.java:

class IntPair {
   Pair(int x) {
   }
}

File Main.java:

IntPair x = new IntPair(10, 100)|;

The usage doesn’t match any of the known constructors of IntPair. Quick fix is going to suggest that another argument is added to IntPair#IntPair(int), so that it becomes IntPair#IntPair(int,int).

Change type of ‘x’ to match expression.

String currentTime = new Set>();

The type mismatch prompts quick fix to suggest replacing String with Set> for the variable currentTime. You may also sometimes get away with

currentTime| = new Set>();

and ask quick fix to create new local variable with the cursor placed near currentTime.

Infer generic arguments

Map map = new HashMap()|;

Here, quick fix can compute the correct generic arguments for Map ().

Import Type ‘x’.

new HashMap



Quick fix may be used to quickly import java.util.HashMap. It allows you to stop fussing about the endless imports lists, and it actually works flawlessly 99% of the time (sometimes, it may insert the import in a funny place if your code is not parseable at the time you ask for an import).

Wrapping Up

Most of the above features are great when writing fresh code. However, they’re even better when you need to change code — especially the realtime compilation feature. Refactorings that may take hours in Stratego or Scala are usually done in minutes in Java, thanks to the IDE support. Imagine where we may go with that kind of IDE support for the newer languages…