Pros & Cons of using the Calibre GUI

[sealbeater]

Originally Posted by BetterRed

However, try asking yourself this: "What will I do with a mega/master folder that I can't do via the calibre GUI?"

BR

Find and move books to my reader in a tenth of the time it would take calibre to load.

edit:

Organize and tag my books from work without the bosses knowing.

Have full control over how I present my own internal library.

[DNSB]

Originally Posted by sealbeater

Find and move books to my reader in a tenth of the time it would take calibre to load.

Hmmm... From the debug log, calibre loads on my laptop in 4.36 seconds, I must admit to severe doubts that you are able to accomplish much in less than half a second.

Just a quick test, I looked for all the books that I own in the 1632 series.

Calibre: open calibre (4.36 seconds), click on search splat, type 1632 in series, hit okay. 26 results on screen in less than 0.1 of a second. Total ~5 seconds.

File Explorer: Opened D drive and browsed to the calibre libraries backup directory. ~3 seconds. No way to browse to the series so typed 1632 in the Windows search box. 20 seconds later, had 25 results. Total time ~23 seconds. Not sure why 1 book didn't show up -- I checked and it's had the series prepend. Please note my D drive is removable and not indexed or the search times would have be lower.

If I did not have the template for save to disk set to prepend the series, I would have been forced to click through multiple directories to find all the books since Eric Flint's who created the series is not the author of all the books -- I list 6 different primary authors and 11 secondary authors with 14 books listing two or more authors and would have had to remember the majority of those names.

If I had wanted to copy any of these books to my ereader,?

Calibre? Plug the ereader in, select the files that weren't already on it, right click, send to device, safely disconnect, reconnect and update metadata, no muss, no fuss.

Doing the same from File Explorer? No indication if the files on on the ereader, multiple directories to deal with. Update metadata and series info (important to those of us who have ereaders that will let us see the series information, sort on it and search on it)? Perhaps you would let me know how you handle that?

For me, calibre wins hands down for speed and convenience. Who really cares if you are not supposed to get your sticky fingers on its internal directory/file structure when you can export your library and create your own file and directory structure there. My purpose in owning an ereader is to read with it not to worry that directory structures aren't created in a way I think would look pretty/elegant.

[sealbeater]

Originally Posted by DNSB

Hmmm... From the debug log, calibre loads on my laptop in 4.36 seconds, I must admit to severe doubts that you are able to accomplish much in less than half a second.

Somehow, I doubt *VERY* highly you have anywhere close to the same number of books that I did when I was using calibre.

Originally Posted by DNSB

File Explorer: Opened D drive and browsed to the calibre libraries backup directory. ~3 seconds. No way to browse to the series so typed 1632 in the Windows search box. 20 seconds later, had 25 results. Total time ~23 seconds. Not sure why 1 book didn't show up -- I checked and it's had the series prepend. Please note my D drive is removable and not indexed or the search times would have be lower.

Your D drive. LOL. Just the fact that you are using windows and a mouse and gui to manipulate your OS is...highly amusing to me. I really don't mean that as disrespect, honestly its just so far from where I am. I am a *nix user. Whatever new books I get, I have a script that I run that postprocesses them, renames, lookups, the whole nine, puts them in a temp dir, I hook up my reader and run another script, all gets copied and file names sanitized and my reader gets unmounted *while calibre would still be loading*.

I don't even have to touch a mouse.

Originally Posted by DNSB

For me, calibre wins hands down for speed and convenience.

Good for you. Personally I have no need for calibre but I'm currently using it again so I can test out using google cloud drive with it. It's not *bad* software tho it has some lame design choices but its' not the end all, be all of ebook management, despite the feelings of people who don't know anything else but windows.

Personally, I found it acceptable even handling the large number of books I loaded into it but the time it would take to load at that point meant that I was obligated to leave it open all the time or wait 15 minutes for it to load. I don't blame calibre, I was impressed it could handle what it did but at that point, it was faster for me to sideload manually but I don't mind at all.

[sealbeater]

Originally Posted by DNSB

If I did not have the template for save to disk set to prepend the series, I would have been forced to click through multiple directories to find all the books since Eric Flint's who created the series is not the author of all the books -- I list 6 different primary authors and 11 secondary authors with 14 books listing two or more authors and would have had to remember the majority of those names.

I would just go "cd library ; find . -name "$series" | parallel cp -rv {} /mnt/bookreader"


I think I would be faster than you.

[sealbeater]

Originally Posted by DNSB

Doing the same from File Explorer? No indication if the files on on the ereader, multiple directories to deal with. Update metadata and series info (important to those of us who have ereaders that will let us see the series information, sort on it and search on it)? Perhaps you would let me know how you handle that?

I think I already answered this.

All of my books have this format:

Anja Jonuleit/Anja Jonuleit - Novemberasche - Kriminalroman (2010)[9783423405638].epub


If it's a series:

Gav Thorpe - [Time of Legends - The Sundering #1] - Malekith (2008) [9788448038373].epub


so I can just go, as stated "cd library ; find . -name "Time of Legends*epub" | parallel cp -rv {} /mnt/bookreader.


I don't care if it's on my bookreader already or not. Why would I?

I can search by any field, ISBN, author, series.

Once you have very many books, I personally have just found it's easier to deal with them that way than wait for calibre.

Of course, small libraries will load calibre quickly but I find command line operations to be faster in general.

[DNSB]

Originally Posted by sealbeater

Somehow, I doubt *VERY* highly you have anywhere close to the same number of books that I did when I was using calibre.

~10,000 books at the moment. I tried opening calibre in a VM with a smaller library and the open time was not all that different. Oddly, though both cases took less than a second. From starting calibre in debug mode:

With the large library on a physical machine,
0.06] Initializing db...
[0.39] db initialized".

With a 7 book library on a VM,
[0.06] Initializing db...
[0.08] db initialized

Originally Posted by sealbeater

Your D drive. LOL. Just the fact that you are using windows and a mouse and gui to manipulate your OS is...highly amusing to me. I really don't mean that as disrespect, honestly its just so far from where I am. I am a *nix user. Whatever new books I get, I have a script that I run that postprocesses them, renames, lookups, the whole nine, puts them in a temp dir, I hook up my reader and run another script, all gets copied and file names sanitized and my reader gets unmounted *while calibre would still be loading*.

I don't even have to touch a mouse.

I get enough command line at work. ESXi, Windows Server Core and 6 or 7 Linux boxes and one lonely Mac OS X system. Note that other than the ESXi servers, the other systems are virtualized. If I move on to the switches and firewalls, I can spend the entire day using Putty for CLI sessions.

While I can make a fire by using flint and steel or a fire bow. I prefer to use a lighter when one is available.

And I still find it humourous that you claim to be able to handle those tasks even using scripts in under 5 seconds. As I said, if nothing else, calibre handles the series information for a Kobo ereader in a much more pleasant fashion than editing the SQLite database. How does your script handle downloading, resizing and inserting covers into the ebook? The same for metadata such as series.

Originally Posted by sealbeater

Good for you. Personally I have no need for calibre but I'm currently using it again so I can test out using google cloud drive with it. It's not *bad* software tho it has some lame design choices but its' not the end all, be all of ebook management, despite the feelings of people who don't know anything else but windows.

Personally, I found it acceptable even handling the large number of books I loaded into it but the time it would take to load at that point meant that I was obligated to leave it open all the time or wait 15 minutes for it to load. I don't blame calibre, I was impressed it could handle what it did but at that point, it was faster for me to sideload manually but I don't mind at all.

Oddly, I find your 15 minutes load time to be about as humourous as your reaction to my using Windows. Comparing that to my 4.36 seconds? Makes me wonder just how out of date your mass storage is.

As for using Google cloud drive with calibre? The first question is what are you calling Google cloud drive? Google Drive for home users? Google Cloud Storage for enterprises? A utility that maps Google Drive as a storage device?

The last time I tested on a computer where Google Drive was used to sync, as long as a single computer is used and synchronization was paused before calibre was started and restarted after calibre exited, it seems to work (a batch file took care of that when I was doing some testing). My library backups are synced to the cloud but I found no real advantage to using cloud storage for my live calibre instance.

[DNSB]

Originally Posted by sealbeater

I think I already answered this.

All of my books have this format:

Anja Jonuleit/Anja Jonuleit - Novemberasche - Kriminalroman (2010)[9783423405638].epub


If it's a series:

Gav Thorpe - [Time of Legends - The Sundering #1] - Malekith (2008) [9788448038373].epub


so I can just go, as stated "cd library ; find . -name "Time of Legends*epub" | parallel cp -rv {} /mnt/bookreader.


I don't care if it's on my bookreader already or not. Why would I?

I can search by any field, ISBN, author, series.

Once you have very many books, I personally have just found it's easier to deal with them that way than wait for calibre.

Of course, small libraries will load calibre quickly but I find command line operations to be faster in general.

Perhaps you should show us the file name for an anthology with more than one author? One that starts pushing the 255 character filename limit. Or a book that belongs in more than one series? Quick example would be Christopher Stasheff's A Warlock in Absentia which is the first book in both The Rogue Wizard and The Warlock's Heirs series.

As for whether or not, it is on your bookreader? I prefer not to accidentally copy over an existing book. YMMV.

As for the library size effect on load times? 9861 books gives .33 seconds for the database, in a VM, 7 books gives .02 seconds for the database.

[kacir]

Originally Posted by sealbeater

I would just go "cd library ; find . -name "$series" | parallel cp -rv {} /mnt/bookreader"
I think I would be faster than you.

Yes, you would be faster. But Calibre is *much* more powerful than that.
With Calibre you can combine various search criteria, use regular expressions, ask Calibre to download the metadata for the book from the net, find duplicates using fuzzy soundex search ...

I have been using Calibre as a conversion tool too. Then I started to use it to manage my books and never looked back.
The main GUI interface is customizable and you can also use powerful commandline tools installed with Calibre.

By the way ... ever since I got Internet connection at home (at about 2002) I have been running my main desktop on FreeBSD and then Linux. I was dualbooting before that.

[sealbeater]

Originally Posted by DNSB

~10,000 books at the moment. I tried opening calibre in a VM with a smaller library and the open time was not all that different. Oddly, though both cases took approximately 0.33 seconds from "[0.06] Initializing db...
[0.39] db initialized".

I'm sorry my man, that's like, not even a 1/3rd of the books that are on my book reader, let alone my library. I don't want to get into how many books I had loaded into calibre but that's not even a percentage point.

Originally Posted by DNSB

I get enough command line at work. ESXi, Windows Server Core and 6 or 7 Linux boxes and one lonely Mac OS X system. Note that other than the ESXi servers, the other systems are virtualized. If I move on to the switches and firewalls, I can spend the entire day using Putty for CLI sessions.

That's cool but no offense, I'm not impressed. You use Windows.

Originally Posted by DNSB

While I can make a fire by using flint and steel or a fire bow. I prefer to use a lighter when one is available.

Originally Posted by DNSB

And I still find it humourous that you claim to be able to handle those tasks even using scripts in under 5 seconds.

And I find it sad that you seem to have missed that it took 15 minutes for calibre to load for me given the amount of books I had loaded into it at the time. My library is not ~10000 books.

Originally Posted by DNSB

As I said, if nothing else, calibre handles the series information for a Kobo ereader in a much more pleasant fashion than editing the SQLite database. How does your script handle downloading, resizing and inserting covers into the ebook? The same for metadata such as series.

I don't use a Kobo so I have no need to edit any SQLite dbs. All of my ebooks have covers. I have no need to resize or insert. The series information is in the filename so I use my ereaders search functionality
to find what I need. I don't have to do anything but load books and read them.

Originally Posted by DNSB

Oddly, I find your 15 minutes load time to be about as humourous as your reaction to my using Windows. Comparing that to my 4.36 seconds? Makes me wonder just how out of date your mass storage is.

Yea, it's my mass storage. That must be what it is. ZFS sucks. It has nothing to do with the fact that 10,000 books isn't even half of a half of a percentage point compared to me.

Well, tell you what, I'm currently loading the A*s into calibre. When that's done, (I'm on record 16033, at the moment), I'll time how long it takes calibre to load and I'll report back as I go though the B* and C* and so on.

Originally Posted by DNSB

As for using Google cloud drive with calibre? The first question is what are you calling Google cloud drive? Google Drive for home users? Google Cloud Storage for enterprises? A utility that maps Google Drive as a storage device?

Google Cloud Storage using rclone.

Originally Posted by DNSB

The last time I tested on a computer where Google Drive was used to sync, as long as a single computer is used and synchronization was paused before calibre was started and restarted after calibre exited, it seems to work (a batch file took care of that when I was doing some testing). My library backups are synced to the cloud but I found no real advantage to using cloud storage for my live calibre instance.

I am doing a unionfs approach, where I have a local encrypted calibre dir mounted to a dir that also has Gcloud mounted, where all local operations will be written to local disk and the encrypted view periodically uploaded to gcloud. I am not uploading the calibre dbs.

I'm actually still in the process of sorted my main library, so once I do that, I'll upload to gcloud just to offset my local storage. My main library is pretty big

I'm just testing calibre to make sure it works as expected, which so far, it has. I may keep it, I may not.

[sealbeater]

Originally Posted by kacir

Yes, you would be faster. But Calibre is *much* more powerful than that.

Doesn't mean much when I'm waiting for it to load. This isn't a criticism of calibre, I'm just saying some people have very large libraries.

Originally Posted by kacir

With Calibre you can combine various search criteria, use regular expressions, ask Calibre to download the metadata for the book from the net, find duplicates using fuzzy soundex search ...

I can do all that from the command line. Faster and better than calibre can.

Originally Posted by kacir

I have been using Calibre as a conversion tool too. Then I started to use it to manage my books and never looked back.
The main GUI interface is customizable and you can also use powerful commandline tools installed with Calibre.

I'm aware, I'm using calibredb to add books now. I just don't need it and honestly, it's not suited for doing operations on large numbers of books. I count large as in exceeding 100k.

Originally Posted by kacir

By the way ... ever since I got Internet connection at home (at about 2002) I have been running my main desktop on FreeBSD and then Linux. I was dualbooting before that.

That's awesome, I've been using *nix exclusively since 1993.

[sealbeater]

Originally Posted by DNSB

Perhaps you should show us the file name for an anthology with more than one author? One that starts pushing the 255 character filename limit. Or a book that belongs in more than one series? Quick example would be Christopher Stasheff's A Warlock in Absentia which is the first book in both The Rogue Wizard and The Warlock's Heirs series.

Good question, let's find out.

Multiple authors:

Stan Berenstain, Jan Berenstain - The Bike Lesson (The Berenstain Bears Beginner Books) (1964) [9780375983276].epub

Pushing the 255 finename limit (ZFS is 256):

./George R. R. Martin, Gardner Dozois, Diana Rowland, Diana Gabaldon, Carrie Vaughn, Joe R. Lansd/George R. R. Martin, Gardner Dozois, Diana Rowland, Diana Gabaldon, Carrie Vaughn, Joe R. Lansd - Dangerous Women (Dangerous Women Anthology) (2013) [9780007549405].mobi

Interestingly, I have different info than you for "A Warlock in Absentia"

Christopher Stasheff - [Rogue Wizard #1] - A Wizard in Absentia (1993) [9
780441515691].epub

but the first in "The Warlocks Heirs":

Christopher Stasheff - [Warlock's Heirs #1] - M'Lady Witch (1994) [0441001130].epub


So, I wonder why the discrepancy. Which of us is wrong?

Originally Posted by DNSB

As for whether or not, it is on your bookreader? I prefer not to accidentally copy over an existing book. YMMV.

Why? What's going to happen if you do? What would it matter?

Originally Posted by DNSB

As for the library size effect on load times? 9861 books gives .33 seconds for the database, in a VM, 7 books gives .02 seconds for the database.

Yes, I'm well aware that the number of books increases load time. That's exactly what I have been saying. Thank you for saying it in a different way. Once again, without going to exact or even close to exact numbers, 10,000 books isn't even half a percentage point to me.

[sealbeater]

Another long mulitple author book.

./Cindy Roland Anderson, Cami Checketts, Kimberley Montpetit, Taylor Hart, Jeanette Lewis, Lucy M/Cindy Roland Anderson, Cami Checketts, Kimberley Montpetit, Taylor Hart, Jeanette Lewis, Lucy M - Christmas in Snow Valley - A Christmas Romance Anthology (Snow Valley Romance Anthologies) (2014) [9781502450319].azw3

[DNSB]

Originally Posted by sealbeater

And I find it sad that you seem to have missed that it took 15 minutes for calibre to load for me given the amount of books I had loaded into it at the time. My library is not ~10000 books.

Given that calibre does not load or re-read your books on startup, the time is that required to read the database. For my larger library, the database clocks in 19.78MB. Resurrecting an old library consisting mostly of a clone of Gutenberg's English epubs (wget worked nicely there) and fan fiction which has ~172 thousand entries and importing it into calibre. I then restarted calibre in debug mode, it increased the db load time to a massive 6 seconds for 342MB and the total time from double-clicking on calibre to being at the GUI and ready to go to a massive 10.7 seconds.

Edit: extrapolating for your ~2,000,000 books, this would bring the database size to ~3.9GB (well below the SQLite maximum of ~140TB). I did some playing with various library sizes and reading the load time at startup seemed to be pretty much linear so that would take about 69 seconds to load so calibre would start in 73 seconds.

Originally Posted by sealbeater

I don't use a Kobo so I have no need to edit any SQLite dbs. All of my ebooks have covers. I have no need to resize or insert. The series information is in the filename so I use my ereaders search functionality to find what I need. I don't have to do anything but load books and read them.

However I do own a Kobo and like to make use of the supplied functionality. I am also interested in where you obtained that mass of ebooks that have no issues with the metadata and covers. I don't find that with most of the ebooks that I purchase.

As for digging through a directory structure to find an ebook to read? Ken Maltby and I had a couple of go-arounds years back comparing my use of search on my Kobo using the stock firmware compared to him locating books using KOReader's file browser. Basically, it took a lot less time and taps to use search compared to digging through multiple pages of directory listings. Perhaps that is why KOReader added the abiltiy to search the calibre database stored on the ereader....

Originally Posted by sealbeater

Yea, it's my mass storage. That must be what it is. ZFS sucks. It has nothing to do with the fact that 10,000 books isn't even half of a half of a percentage point compared to me.

Hmmm.... so 10,000 is not even .5% of your book collection so the total would greater than 2 million books. Out of curiosity, how do you afford so many books? And find the time to read them? Or do you simply collect them from whatever sources you can find?

Originally Posted by sealbeater

Well, tell you what, I'm currently loading the A*s into calibre. When that's done, (I'm on record 16033, at the moment), I'll time how long it takes calibre to load and I'll report back as I go though the B* and C* and so on.

Have fun.

[DNSB]

Originally Posted by sealbeater

Interestingly, I have different info than you for "A Warlock in Absentia"

Christopher Stasheff - [Rogue Wizard #1] - A Wizard in Absentia (1993) [9
780441515691].epub

but the first in "The Warlocks Heirs":

Christopher Stasheff - [Warlock's Heirs #1] - M'Lady Witch (1994) [0441001130].epub


So, I wonder why the discrepancy. Which of us is wrong?

Neither of us is wrong. If you look at Christopher Stasheff's web page and check under the books page, A Wizard in Absentia is listed as the first book in The Rogue Wizard series. If you check the store page and look under ebooks, it's listed as the first book in The Warlock's Heirs series. Considering how long before many of the other books in either series, A Wizard in Bedlam was written and that for continuity purposes, Christopher Stasheff has Magnus' hair colour changed in one of the Warlock books (The Warlock is Missing if my memory serves), the book fits in either series -- Magnus is, after all, Rod and Gwen's oldest son and this book served as a written after the fact bridge between Magnus' as a young man on Gramayre and the character of Gar Pike in A Wizard in Bedlam. You might want to read the 2017 introduction to the re-issue of A Wizard in Absentia for Christopher Stasheff's take on this.

Originally Posted by sealbeater

Why? What's going to happen if you do? What would it matter?

Sending a book that already exists is inefficient and inefficiency offends me.

Originally Posted by sealbeater

Yes, I'm well aware that the number of books increases load time. That's exactly what I have been saying. Thank you for saying it in a different way. Once again, without going to exact or even close to exact numbers, 10,000 books isn't even half a percentage point to me.

And extrapolating the load times, I still end up far less than the 15 minutes you stated. I either have to conclude that Windows is far faster than your operating system ( ) or your mass storage is on the slow side.

[sealbeater]

Originally Posted by DNSB

Given that calibre does not load or re-read your books on startup, the time is that required to read the database. For my larger library, the database clocks in 19.78MB. Resurrecting an old library consisting mostly of a clone of Gutenberg's English epubs (wget worked nicely there) and fan fiction which has ~172 thousand entries and importing it into calibre. I then restarted calibre in debug mode, it increased the db load time to a massive 6 seconds for 342MB and the total time from double-clicking on calibre to being at the GUI and ready to go to a massive 10.7 seconds.

Edit: extrapolating for your ~2,000,000 books, this would bring the database size to ~3.9GB (well below the SQLite maximum of ~140TB). I did some playing with various library sizes and reading the load time at startup seemed to be pretty much linear so that would take about 69 seconds to load so calibre would start in 73 seconds.

Not that many, about a quarter of that. You seem to be calling me a liar. Please, try it and see for yourself. Your math nonewithstanding, I stand as someone who actually has done it. You stand as someone who is guessing.

Originally Posted by DNSB

However I do own a Kobo and like to make use of the supplied functionality.

Sucks for you. I would never ever buy a Kobo. Ewww..

Originally Posted by DNSB

I am also interested in where you obtained that mass of ebooks that have no issues with the metadata and covers. I don't find that most of the ebooks that I purchase.

I bet you are. I'm surprised to hear that, I had no idea you would have issues with metadata and covers. What a dismal experience the e-book consumer game seems to be for most of you.

Originally Posted by DNSB

As for digging through a directory structure to find an ebook to read? Ken Maltby and I had a couple of go-arounds years back comparing my use of search on my Kobo using the stock firmware compared to him locating books using KOReader's file browser. Basically, it took a lot less time and taps to use search compared to digging through multiple pages of directory listings. Perhaps that is why KOReader added the abiltiy to search the calibre database stored on the ereader....

I really don't see how that could be any more irrelevant to me. Please explain why that would matter to me? I use directory structure for both my ereader and my main library, with some post-processing to remove spaces for my ebook reader. I use the search functionality of my reader, I have to, I have over 30k books on it. However that doesn't take away from the value of having a directory structure. Its called the best of both worlds. A known, predictable structure that can be drilled down into at need and search functionality to find the things you want.

Originally Posted by DNSB

Hmmm.... so 10,000 is not even .5% of your book collection so the total would greater than 2 million books.

That's actually about right, with about .5 million sorted and deduped. Its' an ongoing process but I love it, so much fun. I'm not sure what the final number will be, I was loading everything into calibre and doing all my metadata type stuff just like you guys but I found a better way and I decided to blow my calibre library away and start over from scratch. I'm not sure what the exact number of sorted books is now since the reboot but I have started to load into calibre now so I'll be using that to remove foreign language books, merge and so on, however I will be keeping mulitple formats so that will skew the numbers.

I'll be happy to report on calibre's performance as I do so, with the caveant of Calibre having to deal with network traffic as it loads since I'm not maintaining two local copies.

Originally Posted by DNSB

Out of curiosity, how do you afford so many books? And find the time to read them? Or do you simply collect them from whatever sources you can find?

I have a very well paying job and very few fiscal responsibilities.

As for finding the time, I subscribe to the "anti-library" approach.

I do collect from whatever sources I can, I am in progress building a book scanner for some very good 1920s engineering books I want to get into my library.

Originally Posted by DNSB

Have fun.

Thank you, I am having lots of fun and plan on continuing to do so.