Merging a Mercurial repository back into Subversion

In a recent project, I forked an upstream SVN repository into a new Mercurial repo, to do some exploratory programming. The exploration proved very fruitful, so I decided to merge the code back into the upstream SVN repo.

Should be simple, right? Wrong. For posterity, here’s a high-level walkthrough that might also be of interest to others.

The Premise

Your case might differ from mine, so YMMV. Here’s my case:

• I copied a specific SVN revision (no history) into a freshly created Mercurial repo.
• All subsequent development happened on the Mercurial repository (multiple developers, resulting in multiple tips and several merges along the way).
• At the end, we wanted to “replay” the revision history from the Mercurial repository back into a new subdir in the upstream SVN repo, under branches/.
• I wanted to retain as much of the Mercurial revision history as possible, and make it look like all the development had happened in SVN all the time.

I almost got what I wanted. The only caveat was that I ended up having to collapse parts of the Mercurial revision history: The stretches in the revision history with multiple concurrent branches were collapsed into one revision. We only had a few cases of this, so it wasn’t really a biggie.

Tooling

After some investigation and a few stranded attempts, I ended up with the following Mercurial extensions for the job:

• hgsubversion — checked out source, then ran ./setup.py install --user
• rebase — part of the default Mercurial distro, but must be enabled (see below).
• graphlog — same story as for rebase.
• collapse — just checked out source, and pointed Mercurial to it.

For reference, this is the relevant section of my $HOME/.hgrc file:

[extensions]
rebase =
hgsubversion = /home/karltk/.local/lib/python2.6/site-packages/hgsubversion
hgext.graphlog =
collapse = /home/karltk/apps/hgcollapse/hgext/collapse.py

Setting up for merging

I ended up using two local working directories for the job, plus copies of several intermediate stages to get me back in track whenever I made mistakes.

We started with prepping a new branch in SVN, containing the exact revision that the Mercurial repo had been created from. This branch lives in https://path-to-your-svn-repo/branches/new-branch.

Then, the local working copies were simply created with:

• hg-repo, initialized with hg clone https://path-to-remove-hg-repo hg-repo
• svn-repo, initialized with hg clone https://path-to-remove-svn-repo/branches/new-branch svn-repo

Thanks to the hgsubversion extension, Mercurial can pull from and push to SVN repos. The remote end remains a SVN repo, but locally you work on a Mercurial repository. However, you immediately run into a big feature mismatch between SVN and Mercurial: you cannot push to a remote SVN repository, unless the local Mecurial revision history is completely linear — i.e., no branches nor merges in the local history.

Linearizing the Mercurial revision history

Since SVN only accepts a linear revision history, any non-linearities (shown as parallell lines when you do hg glog) found in the hg-repo must be ironed out. This is where hg collapse comes in. By looking at the branch and merge revisions in the glog, I collapsed parallell areas of the revision history manually. E.g. hg collapse -r 82 -r 89 -f, drops you into the commit editor and allows you to manually merge the commit messages for the collapsed revisions.

Once hg glog showed a linear history, I was ready to merge.

Doing the merging (rebasing!)

Now that we have a linear revision history again, we can go ahead and merge the Mercurial repository into SVN. Except, we cannot merge. We must rebase. A merge would introduce a non-linearity in the revision history, stopping us dead in our tracks.

The procedure I found went somewhat like this:

# cd hg-repo
# hg push -f ../svn-repo
# cd ../svn-repo
# hg glog
# hg rebase -b <latest revision in log> -d <first revision in log>

When looking for <latest revision in log> you are looking for the latest revision from the Mercurial repo changes. The <first revision in log> is really the latest revision of the SVN repo changes. What we want to do, is to tuck all the Mercurial repo changes onto the last SVN change. And this is exactly why we are using rebase and not merge.

In my case, the <first revision in log> was 0 and the <latest revision in log> was 83.

When I ran hg rebase -b 83 -d 0, there were numerous merge conflicts — because rebase uses merge behind the scenes. I just resolved these normally using hg resolve, and then continued the rebasing using hg rebase -c. After a few more conflicts and resolutions, the processes ended.

During the rebasing process, frequent copies of the entire svn-repo directory tree is advised. In essence, you’re doing destructive updates to your local copy of the svn-repo, and you can only check in your result after all the rebasing is finished. Having a safety net proved quite useful, at least for me:)

After the rebasing was complete, hg glog told me that everything was linear and good. At this point I did

# hg outgoing
# hg push

If you see “Sorry, can’t find svn parent of a merge revision.” when trying hg outgoing or hg push, it probably means that your revision history (as told to you by hg glog) is non-linear. These parts should be hg collapse-ed away.

Hackathon in Delft: Go!

It’s that time of the year again. The glorious month of intensive parser implementation, compiler engineering and language workbenches — the essentials of any IDE — has arrived. I’ve retreated to the TU Delft campus for the month of November to hack on interactive language infrastructure for our startup, and to think big thoughts about IDEs and DSLs in general.

Our startup is a user of both Stratego and Spoofax, so it was only natural to join forces with Eelco Visser and some of his henchmen here in Delft, who are the maintainers of said projects.

For somewhat practical and somewhat nostalgic reasons, I decided to stay at the TU Delft campus. Campus life here already brings back memories of my year living at Cambridgelaan: the access to lightning fast broadband in your dorm room, the four minute walk to the lab, the 24-hour party people (aka Erasmus students), and the freedom to follow your biological clock.

Time for some commits:)

DIY: Fixing the MSN connector for Empathy

I just got bitten by bug 663670: empathy doesn’t connect to msn - blocked connection.

It turns out that Microsoft made changes on the server side two days ago that crashes telepathy-butterfly, the MSN connector used by Empathy.

Fortunately, the fix is simple.

# sudo nano /usr/lib/pymodules/python2.6/papyon/service/description/SingleSignOn/RequestMultipleSecurityTokens.py

Then goto line 24 and replace

CONTACTS = (“contacts.msn.com”, “?s=1&id=24000&kv=7&rn=93S9SWWw&tw=0&ver=2.1.6000.1”)

with

CONTACTS = (“contacts.msn.com”, “MBI”)

After that, hit the “reconnect” button in Empathy and it should now connect to MSN. You don’t have to restart Empathy, even. These are the times I love scripting languages.