Attachments support

[heretique]

Hi everyone,
I know this is not the place to ask for enhancement requests but I need attachments support in Calibre and I'm willing to spend the time to implement it myself with the help of Calibre's developers community guidance.

I saw an old enhancement request bug here
As Romain mentioned there, plenty of books have associated extra files that would benefit from attachment support. I also use Calibre for storing a lot of research papers that have extra code files.
The solution provided seems more like a hack supported by the way Calibre works. I tried it and one of the side effects is that when I double click my books or papers inside Calibre it would not automatically open them anymore for reading (I store them as PDFs) but it will open the last format added, in this case the ZIP file.
This solution also doesn't seem proper from an UX point of view. Browsing through my library there is no visual hint that a book might have attachments to it.

Calibre, probably, like any other successful open source doesn't take lightly to big changes so any hints from developers on how implementing attachments should be approached would help a lot. I'd like to keep close to core Calibre design and principles and not reinvent the wheel.

Also pointers to relevant documentation might help speed things up.
Thanks!

P.S. I'm not a native English speaker and also don't write much in forums usually so please have patience with me

[kovidgoyal]

I'm afraid the assumption of one format type per book record is very deeply baked in calibre and is not going to change. In any case if you have multiple files of a type in a book double clicking can at best open a dialog to choose which file you want.

Instead what you can do is create a new file format called, say ATTACHMENT. Then when the user tries to open such a format it will present a dialog asking which internal file the user wants to see. And if the user tries to add a second file of an existing format it will automatically become the attachment.

[heretique]

Thanks Kovid for such a quick reply.
Isn't it possible that the attachments be added as metadata? I saw that each book has a folder where the actual book is held (in PDF format for example) and then there's an .opf file containing the metadata and a cover image. Isn't it possible to add attachments to the same folder and then reference them inside the metadata file as attachments?
Is there any documentation that describes underlying Calibre structure and how things work? I mostly found information about how to develop plugins.
If not, could you point me to relevant parts of the source code?

Thanks!

[kovidgoyal]

opf files are not metadata they are simply dsitributed backups for the database. The relevant code for the database operations is in the db directory primarily db.cache and is documented in the manual under database API.

[heretique]

Hey Kovid, I had a deeper look at the way books are stored in the database and I guess the best way of implementing the attachments is through a new table inside the database let's say attachments where each entry will link to a book_id in the books table.
This way we could properly support multiple attachments per book and multiple attachments of the same format (zip, 7z, images). It will require quite some coding to get it functional and probably some subtle UI additions in order to support this but at the same time leave the UI mostly unchanged for the users.
I made a fork of the code and I will start working on this if you think this is a good approach.
It will probably take some time as I'll be doing this only in my spare time.

[kovidgoyal]

Storing binary data in the database is a big no-no. As I outlined in my previous post, the way to do this would be to define a new file type call it .attachment and simply have the UI modified to special case handling of it. So when the user add a second pdf to a book record it automatically gets added to the attachment file (which can internally be a zip container).

[heretique]

I don't want to store binary data into the sqlite db, just paths to where the attachment is added inside the book folder.

The solution you proposed sounds simpler but it has its limitations because you will be able to store only one file of a particular type as attachment (if you have two zip files you can't attach them both, or at least that's my understanding).

With the solution I propose, you store as many attachments as you like for a book, the attachments metadata (path, book_id, description) is stored in the attachments table and the actual attachments can be store inside the book folder or in a sub folder.

There may be some downsides that I don't see, but this solution actually keeps the attachments pretty loosely connected with the books just by id and doesn't affect existing code, maybe except UI part and cleanup code needed when someone moves a book or deletes it, in that case it needs to check and move and/or delete the attachments as well.

[kovidgoyal]

No, you can store as many attachments of a given type as you like. ATTACHMENTS is a zip file. So if you want to store two PDFs in it, you add one.pdf and two.pdf to the attachments file.

The major downside with your proposal is one has to change the db schema, and change a whole bunch of db backend code that moves book files around, import/export code, etc. because now the formats table is not the single source of truth about what files are associated with a book.

[heretique]

I see, it makes sense. I'll have a closer look at your approach.