How I left sourceforge.net

How I left sourceforge.net

Why?

In August 2013, I found this blogpost, and I was disgusted by the practice described there. Long story short: The company which owns sourceforge.net has started to monetarize the project hosting page with unethical means: They started to replace the original installers of top-rated projects with a closed source installer that fools the average user into installing bloat- and spyware.

In my eyes, this is an unbearable breach of my, and anyone else's, trust in SourceForge. Back then, I used to download software not from a third party, but from SourceForge, if possible. Because I has the trust in them that I will only get the original, unmodified code without anything undesireable. I felt safe. Not any more.

What we had

So I, as one of the developers of the MusE Sequencer project (a software for making music, supporting Audio and MIDI), which was hosted on SourceForge, decided that I do not want to support this treacherous company any more, by hosting our software there (generating them more hits).

We had everything hosted on SourceForge:

Luckily, we already had (but have barely used) our own server, serving muse-sequencer.org.

Migrating the repository

I actually did this shortly before, in an unrelated plot: I just was sick of the old, slow, fat SVN, so I decided to convert our repo into a git repository (which was hosted on SourceForge, of course).

Migrating a SVN repository is non-trivial, because of the inherent differences between SVN and git; especially regarding branches. Nevertheless, it isn't that hard.

Mapping users

At first, you will need to create a mapping of your old SVN user names to git-suitable names: 

sourceforge_user1 = John Doe <john.doe@example.com>
jane123 = Jane Doe <jane@example.com>
...
Store this in a text file, let's say authors.txt.

Importing into git

You can use the svn2git tool (which just can be emerged in gentoo, btw). If I recall correctly, 

svn2git --identity-map authors.txt URL_of_svn
just did the job. Note, however, that this command will take a while to complete. You can roughly put 5 minutes per 100 revisions.

This did a pretty good job, however I was not completely content with it. Some tags were missing, a branch was duplicated (for each commit, there was an almost identical twin. Just it had no parent. With some git magic, git rebase and manually adding two tags, I finalized the import. Push it using 

git push -f --tags remoteName refs/heads/*:refs/heads/*

(If you already have a git repo, try git clone --mirror).

Converting the bugtracker(s) into GitHub issues

Unfortunately, there is no direct way to import SourceForge's bugtracker; however, they offer an XML export of your project's data, which can be obtained under sourceforge.net/export

I searched for this "Group ID" for a long time, this is how I finally found it: In the Admin panel for your project, in the sidebar click MySQL Database. Possibly follow the assistant there, until you get the access codes. Your username will look like this: ADMIN User: l98765admin. 98765 is your group ID.

This can be fed into a python script called sf2github, which uses GitHub's REST API to re-create all the tracker items as "issues". The original script can be found on GitHub. However, I had to fix some things as well as add new features to fit our needs. My fork can be found on GitHub (or alternatively at my own cgit); it fixes tag handling (do not create one tag per sourceforge ID, add tags for "resolution" status and more), strips some garbage from the comment fields, and some speed and usability fixes.

What the script can do:

However, there are some limitations: You cannot import

All posts will be owned by whoever was executing the script, and will have the time of execution as timestamp. Caution: GitHub will send out one mail per new issue/comment to each follower of the repo!

Importing will roughly take 3 minutes per 100 bugs+comments.

Migrating the forum

based on this article. Without it, i'd have probably gone insane.

That one was easy, I thought -_-. I never have cursed so loudly...

Exporting the data
Under the admin tool for your project, click hosted apps, then admin hosted apps. Select the phpBB forum, (possibly grant admin to yourself first), then admin, then maintenance. There you can download a phpBB database dump, which can be restored in any other phpBB installation. (You must explicitly download, not store it.)

There are lots of phpBB hosters available, though the preferred solution should be to host the forum on your own server/webspace. We hosted it on our webserver, using Debian Stable (Wheezy).

Setting up phpbb3

Become root, and fire up this: 

apt-get install apache2 mysql-server php5 phpbb3
(You really want MySQL here, even though I prefer PostgreSQL. You won't be able to import your database dump into PostgreSQL without lots of pain.)

Do whatever apt asks you, answer the questions; I've sticked to the default values.

edit /etc/apache2/conf.d/phpbb3.conf and make it say 

Alias /forum/images /var/lib/phpbb3/images
Alias /forum /usr/share/phpbb3/www
in the beginning (i.e., change "phpbb" into "forum". You can set this to whatever you want, but you will need to adapt all the SQL statements below then.

Test your installation by opening yourserver.com/phpbb/ in a browser; a fresh forum should appear.

Importing the data

Place the database backup file under /usr/share/phpbb3/www/store/phpbb3/backup_whatever.sql.gz. Using your web browser, log into the forum as admin, click "Administration Control Panel" at the bottom of the page; then Maintenance and Restore Backup, select the dump you have just moved there, and import the data. And now prepare for some pain. You will end up with a completely unusable forum here, which doesn't do anything right now.

Fixing the breakage
This has been the "fun" part. As root, fire up mysql -p to get a SQL shell. 
# switch to the forum database
use phpbb3;

# fix the forum skin
update phpbb_styles_template set template_name = 'prosilver', template_path = 'prosilver' where template_name = 'sf';
update phpbb_styles_theme set theme_name = 'prosilver', theme_path = 'prosilver' where theme_name = 'sf';
update phpbb_styles_imageset set imageset_name = 'prosilver', imageset_path = 'prosilver' where imageset_name = 'sf';

# fix authentication and captcha
update phpbb_config set config_value = 'db' where config_name = 'auth_method';
insert into phpbb_config values ('captcha_plugin', 'phpbb_captcha_gd',0);
update phpbb_config set config_value='1' where config_name='captcha_gd';

# fix various paths
update phpbb_config set config_value = '/forum/images/avatars/gallery' where config_name = 'avatar_gallery_path';
update phpbb_config set config_value = '/forum/images/avatars/upload' where config_name = 'avatar_path';
update phpbb_config set config_value = '/forum' where config_name = 'cookie_path';
update phpbb_config set config_value = '/forum/images/icons' where config_name = 'icons_path';
update phpbb_config set config_value = '/forum/images/ranks' where config_name = 'ranks_path';
update phpbb_config set config_value = '/forum' where config_name = 'script_path';
update phpbb_config set config_value = '/forum/images/smilies' where config_name = 'smilies_path';
update phpbb_config set config_value = '/forum/images/upload_icons' where config_name = 'upload_icons_path';
update phpbb_config set config_value = '/forum/files' where config_name = 'upload_path';

# CUSTOMIZE THIS
update phpbb_config set config_value = 'YOUR.DOMAIN.COM' where config_name = 'cookie_domain';
update phpbb_config set config_value = 'YOUR.DOMAIN.COM' where config_name = 'server_name';
update phpbb_config set config_value = '80' where config_name = 'server_port';           # probably fine
update phpbb_config set config_value = 'http://' where config_name = 'server_protocol';  # probably fine

Double-check your configuration using 

select * from phpbb_config;

Now, you must delete phpBB's cache. Think before you type! Do so with 

rm /usr/share/phpbb3/www/cache/phpbb3/*
Without doing this, those changes might not show any effect.

Finally, it might be neccessary to upgrade your database to the most recent version; when the forum works fine until you try to log in, but then complains about nonexisting columns etc, do this: 

cd /usr/share/phpbb3/install
php database_update.php

At this point, your forum should be up and running again.

Regaining access to the forum

There's still one problem: You cannot login, because your password isn't accepted. (At least I had this problem). I guess this is because SourceForge used a different means of authentication.

Register yourself a new user, let's say, NewUser. In your MySQL shell, execute this: 

select user_password from phpbb_users where username='NewUser';
Lower/Uppercase does matter!. MySQL will respond like this: 
 
+------------------------------------+
| user_password                      |
+------------------------------------+
| $H$bJ3Nk4J19Diqkd/J3hK8HbhcKn3V1l9 |
+------------------------------------+
1 row in set (0.00 sec)

In order to set this password to your OriginalAccount, do this: 

update phpbb_users set user_password='$H$bJ3Nk4J19Diqkd/J3hK8HbhcKn3V1l9' where username='OriginalAccount';
(Of course you have to use the password string above.)

You should be able to login with OriginalAccount, but with the new password now. (You can delete NewUser).

If it doesn't work, try clearing the cache again.

Done!

Yay! Finally, you have your forum migrated, and up and running again. Log in into the Admin Control Panel and tweak all the new settings (you want to remove lots of "SourceForge"-strings).

If you miss the "download" feature

Look at this.