Switching to git

[kovidgoyal]

Hi all,

Just a heads up. I am switching calibre's source control system to git from bzr. The primary motivation for the move was that bzr was becoming too inefficient at calibre's size. The version history takes ~650MB in bzr and only ~130MB in git. This means that for a new developer the time taken to get calibre's code decreases to a quarter of the current time. For instance, on my internet connection, getting a fresh checkout of calibre's code has gone from ~1 hour to ~10 minutes.

Also, bzr development appears to be dead, see http://stationary-traveller.eu/pages...ospective.html. While that does not matter for bzr itself, it does mean that the support tools and ecosystem will stagnate/decline over time.

Updated instructions for getting the source code are in the manual.

The git repository is hosted on GitHub at https://github.com/kovidgoyal/calibre

Apologies for the inconvenience, however, I felt that this move was best for calibre's future. I did poll the top 5 recent calibre committers, of whom 2 were for it, 2 against and 1 neutral.

If you have any questions or concerns, do let me know and while I'm not a git expert, I will try to help

Below is a basic guide to setting up your own fork of calibre in a way that will allow you to submit pull requests for inclusion into my repository:

Spoiler:

1) Setup git on your machine as described in this article: https://help.github.com/articles/set-up-git

2) Setup ssh keys for authentication to GitHub, as described here: https://help.github.com/articles/generating-ssh-keys

3) Go to https://github.com/kovidgoyal/calibre and click the Fork button.

4) In a Terminal do

git clone git@github.com:<username>/calibre.git

Replace <username> above with your github username. That will get your fork checked out locally.

5) You can make changes and commit them whenever you like. When you
are ready to have me merge your work, do a

git push

and go to https://github.com/<username>/calibre and click the Pull Request to generate a pull request that I can merge.

6) You can update your local copy with code from my repo at any time by doing:

git pull upstream

Just in case, like me, you loved the bzr qdiff tool to view diffs, I've created a small wrapper script that allows you to use it with git:

Spoiler:

#!/usr/bin/env python
# vim:fileencoding=utf-8
from __future__ import (unicode_literals, division, absolute_import,
                        print_function)

__license__ = 'GPL v3'
__copyright__ = '2013, Kovid Goyal <kovid at kovidgoyal.net>'

import sys
import subprocess
import os
import shutil
import atexit
import tempfile


def ipython(ns={}):
    import IPython
    IPython.embed(user_ns=ns)


def walk(d):
    for record in os.walk(d):
        for f in record[-1]:
            f = os.path.join(record[0], f)
            if os.path.islink(f):
                src = os.path.realpath(f)
                os.unlink(f)
                shutil.copy2(src, f)
            yield os.path.relpath(f, d)


def do_diff(left, right):
    rfiles = frozenset(walk(type('')(right)))
    os.chdir(left)
    files = frozenset(walk('.'))
    if not files and not rfiles:
        raise SystemExit(0)

    # Create and populate bzr repo
    subprocess.check_call(['bzr', 'init', '-q'])
    subprocess.check_call(['bzr', 'add', '-q'] + list(files))
    subprocess.check_call(['bzr', 'commit', '-m', 'left', '-q'])

    # Update files in left from right
    for f in files | rfiles:
        src = os.path.realpath(os.path.join(right, f))
        if os.path.exists(src):
            base = os.path.dirname(f)
            if base and not os.path.exists(base):
                os.makedirs(base)
            shutil.copy2(src, f)
        else:  # File was deleted
            os.unlink(f)

    newf = rfiles - files
    if newf:
        subprocess.check_call(['bzr', 'add', '-q'] + list(newf))

    raise SystemExit(subprocess.Popen(['bzr', 'qdiff']).wait())

left, right = sys.argv[1:3]
# ipython(locals())

if not os.path.isdir(left):
    # Single file diff, we make the directory structure ourselves
    left = tempfile.mkdtemp(prefix='difftool_left_')
    right = tempfile.mkdtemp(prefix='difftool_right_')
    atexit.register(shutil.rmtree, left)
    atexit.register(shutil.rmtree, right)
    path = sys.argv[-1]
    base = os.path.dirname(path)
    if base:
        for x in (left, right):
            os.makedirs(os.path.join(x, base))
    shutil.copy2(sys.argv[1], os.path.join(left, path))
    shutil.copy2(sys.argv[2], os.path.join(right, path))

do_diff(left, right)

Save this script somewhere on your path and make it executable (on windows you'd probably have to create a .bat file that calls python with this script). Then in your git config file add the following

[diff]
    tool = bzr
[difftool]
    prompt = false
[difftool "bzr"]
    cmd = /path/to/the/executable/script $LOCAL $REMOTE $MERGED

You can then use it with:

git difftool -d

[hakan42]

Cooooooooool :-)

No, now seriously, while there might be personal preferences in the bzr-vs-git decision, the infrastructure around git and github is simply too vibrant to ignore.

Another question: will you migrate the issues to github too or will they stay on launchpad?

And the translations of the core? Also remaining on launchpad or migrated to something like transifex?

[kovidgoyal]

Launchpad's issue tracker is far superior to GitHub, for one thing you must be able to mark issues private, so that people can attach copyrighted samples. So issue tracking will remain on Launchpad.

Similarly, I see no reason to disrupt the existing translators, the translations will also remain in Launchpad.

EDIT: I've left the issues for the calibre repo on GitHub open for now, but they should only be used by developers wanting to discuss code. The main, user facing bug tracker remains Launchpad.

[kovidgoyal]

I've added a basic guide to the first post on how to get up and running with GitHub if you want to contribute code to the main calibre repo.

[kovidgoyal]

On a personal note, out of the box, I much prefer bzr, it's easier to use and the default workflow is optimized for a project with a relatively small number of regular contributors, like calibre.

However, git, after some customization, scripts/aliases and so on, works just as well. You do have to put in much more upfront time learning and customizing it, but then, given that I use vim, I can hardly complain about that

[DiapDealer]

Git outta here! Seriously?

I prefer bzr too, but I completely understand your reasoning here. Here's hoping for a smooth transition all around.

[Terisa de morgan]

Originally Posted by kovidgoyal

You do have to put in much more upfront time learning and customizing it, but then, given that I use vim, I can hardly complain about that

Tsk, tsk, tsk. Are you complaining about the poor vim (It's the tool which never is absent of my computers, notepad doesn't exist).

[CRussel]

Vim? Love it. It's literally the very first thing I install on any machine I work on. And as much as I wish that it had all the intellisense of the PowerShell ISE, you couldn't get me to swap out for any amount.

[kovidgoyal]

Oh I love vim, I use it for everything, email, editing, even browsing (via pentadctyl). I have even contributed code to several vim plugins. But, out of the box, it requires *a lot* of customization and a very steep learning curve.

[sengian]

Originally Posted by kovidgoyal

Similarly, I see no reason to disrupt the existing translators, the translations will also remain in Launchpad.

Maybe you should rethink this as the translation interface is unusable and will likely remain so for a few years seeing the speed of critical bugs resolution.
I stopped working on translations a year ago mainly for this, and while it is usually possible to work just downloading the PO file and uploading it, you loose a lot of functionality.

[kovidgoyal]

Originally Posted by sengian

while it is usually possible to work just downloading the PO file and uploading it, you loose a lot of functionality.

Like what?

[sengian]

Search tools, which are not as good in poedit or betterpoeditor.
Access to the suggestions of the other projects to help normalize translation
Maybe suggestions (and reviewing), but I am not sure about this one as I always worked on those in Launchpad (they may be marked as Fuzzy in po files)

Those come from the top of my head, I am just saying that the web interface has been unsuable for a while. Working with only po files is not as easy for me for the reasons stated above.
Maybe you could make a poll to see if other translators are happy with the current situation, I am not saying everyone is of the same mind.

[kovidgoyal]

Why is the web interface unusable? I haven't heard any complaints from other translators and judging by https://translations.launchpad.net/c.../+pots/calibre there is plenty of translation activity going on. ALmost 30 languages have been edited in 2014 alone.

[sengian]

Well I don't know how other people are doing but I tried during 2013 to use it a few times and this leaded to OOPS every time I tried a search and so on.
Basically a lot of timeouts and errors.
I tried it again this morning and had an OOPS 2 or 3 times but then it may be better.
To see my point just try to do a search, it is nearly guaranteed to get you an OOPS.
See also this bug reports (I am not the only one complaining) for example:
https://bugs.launchpad.net/launchpad/+bug/736005

Please note, that I am not saying it is impossible to work, just that the platform is just currently working at suboptimal capacity and has the same maintenance problems bazaar had. I was quite happy with it as long as it was reliably working.

[kovidgoyal]

Yeah searches have always been broken in Launchpad translations, for as long as I can remember. Other than that, most things work well.

Certainly if more people complain, I will consider moving to something else.