How I left sourceforge.net
How I left sourceforge.net
In August 2013, I found this blogpost, and I was disgusted by the practice described there. Long story short: The company which owns sourceforge.net has started to monetarize the project hosting page with unethical means: They started to replace the original installers of top-rated projects with a closed source installer that fools the average user into installing bloat- and spyware.
In my eyes, this is an unbearable breach of my, and anyone else's, trust in SourceForge. Back then, I used to download software not from a third party, but from SourceForge, if possible. Because I has the trust in them that I will only get the original, unmodified code without anything undesireable. I felt safe. Not any more.
What we had
So I, as one of the developers of the MusE Sequencer project (a software for making music, supporting Audio and MIDI), which was hosted on SourceForge, decided that I do not want to support this treacherous company any more, by hosting our software there (generating them more hits).
We had everything hosted on SourceForge:
- A small homepage
- our MediaWiki
- a forum (phpBB)
- our mailing lists
- We used their bug- and feature request trackers
- Of course, our repository: We used SVN.
Luckily, we already had (but have barely used) our own server, serving muse-sequencer.org.
Migrating the repository
I actually did this shortly before, in an unrelated plot: I just was sick of the old, slow, fat SVN, so I decided to convert our repo into a git repository (which was hosted on SourceForge, of course).
Migrating a SVN repository is non-trivial, because of the inherent differences between SVN and git; especially regarding branches. Nevertheless, it isn't that hard.
At first, you will need to create a mapping of your old SVN user names to git-suitable names:
sourceforge_user1 = John Doe <email@example.com> jane123 = Jane Doe <firstname.lastname@example.org> ...Store this in a text file, let's say authors.txt.
Importing into git
You can use the svn2git tool (which just can be emerged in gentoo, btw). If I recall correctly,
svn2git --identity-map authors.txt URL_of_svnjust did the job. Note, however, that this command will take a while to complete. You can roughly put 5 minutes per 100 revisions.
This did a pretty good job, however I was not completely content with it. Some tags were missing, a branch was duplicated (for each commit, there was an almost identical twin. Just it had no parent. With some git magic, git rebase and manually adding two tags, I finalized the import. Push it using
git push -f --tags remoteName refs/heads/*:refs/heads/*
(If you already have a git repo, try git clone --mirror).
Converting the bugtracker(s) into GitHub issues
Unfortunately, there is no direct way to import SourceForge's bugtracker; however, they offer an XML export of your project's data, which can be obtained under sourceforge.net/export
I searched for this "Group ID" for a long time, this is how I finally found it: In the Admin panel for your project, in the sidebar click MySQL Database. Possibly follow the assistant there, until you get the access codes. Your username will look like this: ADMIN User: l98765admin. 98765 is your group ID.
This can be fed into a python script called sf2github, which uses GitHub's REST API to re-create all the tracker items as "issues". The original script can be found on GitHub. However, I had to fix some things as well as add new features to fit our needs. My fork can be found on GitHub (or alternatively at my own cgit); it fixes tag handling (do not create one tag per sourceforge ID, add tags for "resolution" status and more), strips some garbage from the comment fields, and some speed and usability fixes.
What the script can do:
- Import bug,feature and patch tracker items. (GitHub doesn't have multiple different trackers, so bugs are tagged as "bug" and so on.
- Import all the comments
- Import tags
- Import open/closed as well as the resolution (Fixed/Wontfix/Worksforme) (resolution added by me)
- Add comment metadata to the comment texts
- Preserve the assignee, if you specify a username map (added by me)
- Preserve the ordering in which the issues were opened (added by me)
However, there are some limitations: You cannot import
- the issues'/comments' creation time
- nor the original posters' names
Importing will roughly take 3 minutes per 100 bugs+comments.
Migrating the forum
based on this article. Without it, i'd have probably gone insane.
That one was easy, I thought -_-. I never have cursed so loudly...
Exporting the dataUnder the admin tool for your project, click hosted apps, then admin hosted apps. Select the phpBB forum, (possibly grant admin to yourself first), then admin, then maintenance. There you can download a phpBB database dump, which can be restored in any other phpBB installation. (You must explicitly download, not store it.)
There are lots of phpBB hosters available, though the preferred solution should be to host the forum on your own server/webspace. We hosted it on our webserver, using Debian Stable (Wheezy).
Setting up phpbb3
Become root, and fire up this:
apt-get install apache2 mysql-server php5 phpbb3(You really want MySQL here, even though I prefer PostgreSQL. You won't be able to import your database dump into PostgreSQL without lots of pain.)
Do whatever apt asks you, answer the questions; I've sticked to the default values.
edit /etc/apache2/conf.d/phpbb3.conf and make it say
Alias /forum/images /var/lib/phpbb3/images Alias /forum /usr/share/phpbb3/wwwin the beginning (i.e., change "phpbb" into "forum". You can set this to whatever you want, but you will need to adapt all the SQL statements below then.
Test your installation by opening yourserver.com/phpbb/ in a browser; a fresh forum should appear.
Importing the data
Place the database backup file under /usr/share/phpbb3/www/store/phpbb3/backup_whatever.sql.gz. Using your web browser, log into the forum as admin, click "Administration Control Panel" at the bottom of the page; then Maintenance and Restore Backup, select the dump you have just moved there, and import the data. And now prepare for some pain. You will end up with a completely unusable forum here, which doesn't do anything right now.
Fixing the breakageThis has been the "fun" part. As root, fire up mysql -p to get a SQL shell.
# switch to the forum database use phpbb3; # fix the forum skin update phpbb_styles_template set template_name = 'prosilver', template_path = 'prosilver' where template_name = 'sf'; update phpbb_styles_theme set theme_name = 'prosilver', theme_path = 'prosilver' where theme_name = 'sf'; update phpbb_styles_imageset set imageset_name = 'prosilver', imageset_path = 'prosilver' where imageset_name = 'sf'; # fix authentication and captcha update phpbb_config set config_value = 'db' where config_name = 'auth_method'; insert into phpbb_config values ('captcha_plugin', 'phpbb_captcha_gd',0); update phpbb_config set config_value='1' where config_name='captcha_gd'; # fix various paths update phpbb_config set config_value = '/forum/images/avatars/gallery' where config_name = 'avatar_gallery_path'; update phpbb_config set config_value = '/forum/images/avatars/upload' where config_name = 'avatar_path'; update phpbb_config set config_value = '/forum' where config_name = 'cookie_path'; update phpbb_config set config_value = '/forum/images/icons' where config_name = 'icons_path'; update phpbb_config set config_value = '/forum/images/ranks' where config_name = 'ranks_path'; update phpbb_config set config_value = '/forum' where config_name = 'script_path'; update phpbb_config set config_value = '/forum/images/smilies' where config_name = 'smilies_path'; update phpbb_config set config_value = '/forum/images/upload_icons' where config_name = 'upload_icons_path'; update phpbb_config set config_value = '/forum/files' where config_name = 'upload_path'; # CUSTOMIZE THIS update phpbb_config set config_value = 'YOUR.DOMAIN.COM' where config_name = 'cookie_domain'; update phpbb_config set config_value = 'YOUR.DOMAIN.COM' where config_name = 'server_name'; update phpbb_config set config_value = '80' where config_name = 'server_port'; # probably fine update phpbb_config set config_value = 'http://' where config_name = 'server_protocol'; # probably fine
Double-check your configuration using
select * from phpbb_config;
Now, you must delete phpBB's cache. Think before you type! Do so with
rm /usr/share/phpbb3/www/cache/phpbb3/*Without doing this, those changes might not show any effect.
Finally, it might be neccessary to upgrade your database to the most recent version; when the forum works fine until you try to log in, but then complains about nonexisting columns etc, do this:
cd /usr/share/phpbb3/install php database_update.php
At this point, your forum should be up and running again.
Regaining access to the forum
There's still one problem: You cannot login, because your password isn't accepted. (At least I had this problem). I guess this is because SourceForge used a different means of authentication.
Register yourself a new user, let's say, NewUser. In your MySQL shell, execute this:
select user_password from phpbb_users where username='NewUser';Lower/Uppercase does matter!. MySQL will respond like this:
+------------------------------------+ | user_password | +------------------------------------+ | $H$bJ3Nk4J19Diqkd/J3hK8HbhcKn3V1l9 | +------------------------------------+ 1 row in set (0.00 sec)
In order to set this password to your OriginalAccount, do this:
update phpbb_users set user_password='$H$bJ3Nk4J19Diqkd/J3hK8HbhcKn3V1l9' where username='OriginalAccount';(Of course you have to use the password string above.)
You should be able to login with OriginalAccount, but with the new password now. (You can delete NewUser).
If it doesn't work, try clearing the cache again.
Yay! Finally, you have your forum migrated, and up and running again. Log in into the Admin Control Panel and tweak all the new settings (you want to remove lots of "SourceForge"-strings).