PATCH: ebook-convert: obviate need for write permission in source directory

[Egnaled Knarf]

I'm using part of the Calibre cli tools to implement a rudimentary library management system capable of converting documents on the fly to several popular formats.

Spoiler:

I can not use all of Calibre for this project because of the abysmal performance when confronted with large collections (> 90000 documents in this case).

While implementing these on-the-fly conversion functions I noticed ebook-convert (and with that Calibre proper as well as far as I can tell) needs write permission in the document source directory for at least conversions from TXT to other formats. Given that it only uses this permission to create a temporary file which is deleted a few moments later and given the risks inherent in allowing write access to the document source directory may I suggest applying this short patch to obviate this requirement? The patch does away with the versioned temporary file in the source directory, replacing it with a proper temporary file as create by Calibre's own TemporaryFile method. This is both cleaner as well as more readable, and works in situations where the user running ebook-convert (in my case www-data) does not have write permission to the library source.

The patch to fix this problem when converting TXT-related documents to other formats is attached to this message. It is of course possible this problem exists elsewhere in Calibre so more patching might be called for. The patch applies to Calibre 0.8.38 (and with some fuzz to earlier versions as well).

[kovidgoyal]

Your patch will break txt files that reference images using relative paths (with markdown, for instance).

[Egnaled Knarf]

Originally Posted by kovidgoyal

Your patch will break txt files that reference images using relative paths (with markdown, for instance).

True, I had not thought of that - probably because I don't have any markdown-formatted texts. This breakage could be resolved by symlinking the contents of the source directory to the temporary directory. I'll go hunting for a suitably complex test case to add this to the patch.

[kovidgoyal]

Symlinking isn't cross platform, and there's no way you can guarantee that the linked to resources are in the same directory as the txt file.

[Egnaled Knarf]

Originally Posted by kovidgoyal

Symlinking isn't cross platform, and there's no way you can guarantee that the linked to resources are in the same directory as the txt file.

Cross-platform - well, it runs on Linux and *BSD and such so it is cross-platform in the Microsoft-sense but I'm starting to see what you're getting at.
Given the (to me at least...) obvious advantages of keeping write access to the library source limited to the absolute minimum I can live with these limitations but the current patch is clearly not ready for general release. Since it does fit my needs I'll just keep patching...

[kovidgoyal]

Why dont you simply copy the source file out of the library to a temp directory and then convert it and copy the result back.

[Egnaled Knarf]

Originally Posted by kovidgoyal

Why dont you simply copy the source file out of the library to a temp directory and then convert it and copy the result back.

While that would have most of the benefits of my method, it has all of the drawbacks with regard to unresolved relative links and it adds an additional file copy.

[kovidgoyal]

There are no relative links for documents in the calibre library, unless the user has been manually putting files in there. And since you only care about unix, create a hard link, that will mean no extra copy unless hte temp dir is mounted ona different partition.