When creating a library does Calibre create duplicates, rename, or use hard/symlinks?

[HailCardassia]

I need something to manage my very large e-book collection. I tried Calibre years ago and didn't really like it but there aren't really any better options out there and I need a media manager for my ebooks.

I know Calibre builds its own library but I have searched around and haven't really found an answer on how it does this. Does it duplicate the files on your system, rename and move them, or hardlink/symlink them to the new location?

I have certain folder structures that I don't want messed with as well as files that cannot be altered. I also have about 300gb of e-books so I'd rather not have it take up twice the space by Calibre keeping a separate library nor can I delete the would be superfluous original files.

So can I add the files to Calibre and have it automatically create the new library using hardlinks or should I create a hardlink copy of the original database and use that to plug into Calibre?

Alternatively if there is a program that can manage ebooks in a similar way to how Yac Reader library does comics I would greatly prefer that. I haven't been able to find anything that does though.

[BetterRed]

Originally Posted by HailCardassia

I need something to manage my very large e-book collection. I tried Calibre years ago and didn't really like it but there aren't really any better options out there and I need a media manager for my ebooks.

I know Calibre builds its own library but I have searched around and haven't really found an answer on how it does this. Does it duplicate the files on your system, rename and move them, or hardlink/symlink them to the new location?

I have certain folder structures that I don't want messed with as well as files that cannot be altered. I also have about 30gb of e-books so I'd rather not have it take up twice the space by Calibre keeping a separate library nor can I delete the would be superfluous original files.

So can I add the files to Calibre and have it automatically create the new library using hardlinks or should I create a hardlink copy of the original database and use that to plug into Calibre?

Alternatively if there is a program that can manage ebooks in a similar way to how Yac Reader library does comics I would greatly prefer that. I haven't been able to find anything that does though.

The short answer is No.

I'm not sure I understand what your second paragraph is trying to suggest - what do you mean by a 'hardlink copy' of the 'original database' - what is a hardlink copy, apart from just another inode to the same data stream, and what is an 'original database'.

Have you considered any of the Digital Asset Management (DAM) applications, some of them work more like the way you'd prefer. Most come from an image management perspective, so they don't and are never likely to have the e-book specific features of calibre such a integration with ebook readers, ebook specific format shifting, ebook editing. Some of them have limited support for PDF formats. Wikipedia had a page that summarised the major DAM applications - but it's disappeared.

BR

[pdurrant]

Originally Posted by HailCardassia

I know Calibre builds its own library but I have searched around and haven't really found an answer on how it does this. Does it duplicate the files on your system, rename and move them, or hardlink/symlink them to the new location?

Calibre duplicates the files and stores them in its own folder structure. If you want to make use of calibre's excellent metadata management, you will have to find space to allow calibre to make a copy of all your books. And then let calibre manage the book files.

Feel free to keep your original folders untouched. Calibre won't pay them any attention. But I think you'll find that allowing calibre to manage files while you worry about the metadata is a much better way to invest your time in your ebook library.

Storage is cheap. 30GB isn't a lot now.

If you ever do need your books in a particular format in a particular folder structure, you can export a copy from calibre, with all the books updated with the latest metadata.

[HailCardassia]

Originally Posted by pdurrant

Calibre duplicates the files and stores them in its own folder structure. If you want to make use of calibre's excellent metadata management, you will have to find space to allow calibre to make a copy of all your books. And then let calibre manage the book files.

Feel free to keep your original folders untouched. Calibre won't pay them any attention. But I think you'll find that allowing calibre to manage files while you worry about the metadata is a much better way to invest your time in your ebook library.

Storage is cheap. 30GB isn't a lot now.

If you ever do need your books in a particular format in a particular folder structure, you can export a copy from calibre, with all the books updated with the latest metadata.

I will see if I can edit the OP. It was supposed to say 300gb. I guess I left off a zero though it is actually ~327gb of ebooks and PDFs currently and it is constantly growing. Audiobooks and comics are separate.

I cannot afford to double the amount of space my ebooks take up. My 160tb(raw) media server is almost full, I've only got 4 bays left in my chasis, and HDDs aren't cheap especially when every bit of it has to be duplicated offsite and I have to have room for parity disks, etc.

I also cannot have the original files changed. Calibre going in and altering all of my ebook files would create a whole host of problems both with my backups and with the other programs that are currently using the files the way they are now. If there is no way around having duplicate copies taking up double the disk space then I will just have to find an alternative as they are all being used by another program and it would take days to manually go through and relocate every single file again assuming the MD5 isn't altered in the process.

I don't really care about the metadata management. Everything has been tagged and edited already. I just want a ebook manager where I can scroll through, or search, my collections, select something, and have it open in a native reader similar to my music manager (mediamonkey), comic manager(YAC), and video manager(Emby with a Kodi front-end)

Originally Posted by BetterRed

The short answer is No.

I'm not sure I understand what your second paragraph is trying to suggest - what do you mean by a 'hardlink copy' of the 'original database' - what is a hardlink copy, apart from just another inode to the same data stream, and what is an 'original database'.

Have you considered any of the Digital Asset Management (DAM) applications, some of them work more like the way you'd prefer. Most come from an image management perspective, so they don't and are never likely to have the e-book specific features of calibre such a integration with ebook readers, ebook specific format shifting, ebook editing. Some of them have limited support for PDF formats. Wikipedia had a page that summarised the major DAM applications - but it's disappeared.

BR

Well, what I am trying to avoid here is having my ebook collection take up twice the disk space while keeping all of my original files as they are. I thought if Calibre renamed the files and I made hardlinks, or symlinks, of the original files I could point Calibre to the duplicate directory and let it rename those without changing the original files or doubling its size.

I have not considered a DAM application. What I like about Yac for comics is that is preserves my easily navigable file and naming structure making it super easy to find exactly what I want while also having a nice UI and reader built in similar to how Emby+Kodi does my video files.

I would like something that does the same and with an ebook reader integrated as well. I put off Calibre for a long time because I really wasn't a fan when I tried it years ago but after searching periodically during that time I haven't found any better alternatives. I don't need to edit or convert anything.

I will check my offline backups of Wikipedia and see if the DAM application list is in one of them.

Thanks.

[BetterRed]

Originally Posted by HailCardassia

I thought if Calibre renamed the files and I made hardlinks, or symlinks, of the original files I could point Calibre to the duplicate directory and let it rename those without changing the original files or doubling its size.

Calibre's not going to do any of that for you, but you could do in manually. I did an experiment using Win10 1703, out of the box calibre portable, the xplorer² file manager, and Hermann Shinagl's LSE gadget.

I chose Mary Wollestenecraft's "A Vindication of the Rights of Woman" deliberately, because I knew it would introduce the issue of calibre's truncation rules.

First up I used the Add empty book tool to create a book record and an empty epub, here you see the book list and the book folder:



Then I went to where I had the original epub and right button dragged it into the book folder and selected LSE's Create Symbolic Link Here option



Here you see the result



Next I copied the calibre truncated name of the empty epub to the clipboard, and then deleted the empty epub, then I renamed the symlink to the calibre truncated name - here's the result



And here the book is loaded into the viewer via the symlink.



I was also able to use the tool in Metadata edit to extract the cover image from the epub into the book folder. When I tried to edit the book the editor crashed. That's all I tried - which is barely scratching calibre's surface

So what you want can be done, after a fashion, but you will have to do it and sort out any problems when something doesn't work or when a symlink is invalidated because the target was moved or renamed etc etc.

BR

[pdurrant]

Of course, calibre is open source, so you could look into modifying it to do what you want - when importing a book, only make a link to the original book, and to never modify the original book.

[dwig]

Originally Posted by HailCardassia

I need something to manage my very large e-book collection. ...
I have certain folder structures that I don't want messed with as well as files that cannot be altered...

Man, nor AI or computer program, can be a slave to two masters.

You say you want something to manage your library but at the same time say that you've managed it and don't want it altered. The is a contradiction that needs to be very very clearly resolved before you continue your search of a "solution".

Question #1 is "exactly what do you want the new management app to actually do?"

[HailCardassia]

Originally Posted by dwig

Man, nor AI or computer program, can be a slave to two masters.

You say you want something to manage your library but at the same time say that you've managed it and don't want it altered. The is a contradiction that needs to be very very clearly resolved before you continue your search of a "solution".

Question #1 is "exactly what do you want the new management app to actually do?"

I don't see it as a contradiction at all. I want it to do the same thing my other media managers do.

Yac puts all my comics up into an easy to browse and navigate interface and allows me to view my comics, keep up with what I have read, etc.

Emby+Kodi puts all my films and television series into an easily navigable interface, allows me to play my content, and keeps up with what I have and haven't watched.

MediaMonkey takes all my audio files and allows me to access, play, and keep up with all of my audio easily.

None of those applications need to duplicate my files, rename my files, alter my files, or mess with my file structure in any way. I install the programs, point it to the files, and I'm ready to go.

I just want something similar for e-books. Something that will put all my ebooks into a easy to navigate interface, select a book, and read it instead of going ebooks>fiction>author>series>title and then opening the book with a separate reader. I mean, its easy enough with my file structure to find what I want but I would like to have it all in an interface similar to Yac, Kodi, Mediamonkey, etc.

Sadly it seems Calibre is the only decent option out there but it seems to require either having my files moved and changed or creating duplicates taking up twice the space. Neither of which are an option for me as I don't have the disk space to spare for duplicates for something like this and the files are being used by another program and can't feasibly be moved or have its MD5 changed.

Originally Posted by pdurrant

Of course, calibre is open source, so you could look into modifying it to do what you want - when importing a book, only make a link to the original book, and to never modify the original book.

I'll have to look into this a bit more and see how difficult it would be. There doesn't seem to be many other options out there as far as ebook managers.

Originally Posted by BetterRed

Calibre's not going to do any of that for you, but you could do in manually. I did an experiment using Win10 1703, out of the box calibre portable, the xplorer² file manager, and Hermann Shinagl's LSE gadget.

I chose Mary Wollestenecraft's "A Vindication of the Rights of Woman" deliberately, because I knew it would introduce the issue of calibre's truncation rules.

First up I used the Add empty book tool to create a book record and an empty epub, here you see the book list and the book folder:

Attachment 161666

Then I went to where I had the original epub and right button dragged it into the book folder and selected LSE's Create Symbolic Link Here option

Attachment 161667

Here you see the result

Attachment 161668

Next I copied the calibre truncated name of the empty epub to the clipboard, and then deleted the empty epub, then I renamed the symlink to the calibre truncated name - here's the result

Attachment 161669

And here the book is loaded into the viewer via the symlink.

Attachment 161670

I was also able to use the tool in Metadata edit to extract the cover image from the epub into the book folder. When I tried to edit the book the editor crashed. That's all I tried - which is barely scratching calibre's surface

So what you want can be done, after a fashion, but you will have to do it and sort out any problems when something doesn't work or when a symlink is invalidated because the target was moved or renamed etc etc.

BR

I'll have to give this a shot. Once set up none of the files would ever be moved or renamed. It will be a chore doing that for all 300gb worth of ebooks individually but at least its a solution and I could work on it a little bit at a time. Thanks.

[sjfan]

I do the opposite, I let calibre maintain the files internally and create an external shadow tree of symlinks pointed into the calibre directory in the layout I prefer. That’s a little safer if you might want to move your outside tree onto a network drive or something like that.

[BetterRed]

Originally Posted by sjfan

I do the opposite, I let calibre maintain the files internally and create an external shadow tree of symlinks pointed into the calibre directory in the layout I prefer. That’s a little safer if you might want to move your outside tree onto a network drive or something like that.

- and that way, you won't lose any calibre functionality, or have to grapple with calibre's file naming algorithm.

I wouldn't recommend anyone use the technique I outlined in post#5, I just wanted to point out to HailCardassia that calibre does not prevent the functionality of the Operating and File System being utilised in creative ways, but you may lose calibre functionality.

Another approach would be to add empty books (i.e. with no format files) and use a custom column to hold a file URI link to the book format files in a different folder setup. You would have all the metadata based sort and search facilities (i.e. the catalogue) but none of the book format file facilities (conversion, send, save, editing etc) and to view a book you'd have to click a link rather than press 'V'.

BR