Best way to use Kobo ereaders & calibre?

[Grins]

In the “good old days” (until around the end of 2022) it was possible to connect a Kobo reader to calibre (via USB cable) and copy / save 200 or more books or documents (all in EPUB format) onto the reader with single file transfer. It was even possible to have 500 document saved with a single file transfer.

However, the software updates in 2023 made this impossible - worse, there were database errors on the readers, which even led to the complete inoperability of at least 2 devices.

Since then I have been using Pocketbook devices (in the manner described above) and have no longer had any problems.

This thread is not about whether 500 "real" books or documents can be read in a short time (that's not the application here, by the way), but about the stability of the device software.

Now a student of mine wants to use his reader in a similar way as me. He owns a Kobo Clara 2E. Given my experiences, I advised him against it, but he "loves" his Kobo

Therefore my questions: How do you copy EPUB documents from calibre to the Kobo reader via file transfer? Which of the 3 “device interface plugins” do you use? How many books do you copy onto the reading device at a single time? If you divide 500 documents into smaller quantities, how large are these quantities? Do you switch off the device completely once the file transfer and database update has been successful so that the database can completely rebuild itself?

[Cactus Chef]

Originally Posted by Grins

Therefore my questions: How do you copy EPUB documents from calibre to the Kobo reader via file transfer? Which of the 3 “device interface plugins” do you use? How many books do you copy onto the reading device at a single time? If you divide 500 documents into smaller quantities, how large are these quantities? Do you switch off the device completely once the file transfer and database update has been successful so that the database can completely rebuild itself?

There's no hard limit to how much you can transfer in one fell swoop, evidence suggests it simply gets riskier the more books you push. I would recommend capping at 100 per push, then disconnect, let the Kobo update the database, verify that you can open and page through one of the new books, and then starting with the next batch. No need to power off the device.

[rashkae]

Originally Posted by Grins

Therefore my questions: How do you copy EPUB documents from calibre to the Kobo reader via file transfer? Which of the 3 “device interface plugins” do you use? How many books do you copy onto the reading device at a single time? If you divide 500 documents into smaller quantities, how large are these quantities? Do you switch off the device completely once the file transfer and database update has been successful so that the database can completely rebuild itself?

There are only 2 device plugins applicable, and only 1 of them comes with Calibre install. Personally, I use the Kobo Touch Extended driver, (which I have to install), but that is because I prefer the experience of using the Kobo's native 'kepub' format.

The advise to limit the number of books you upload to the kobo is to make it easier to identify and troubleshoot problem files that the kobo will be unable to ingest. It doesn't matter if you upload 500 files, or just the 1 bad one, strange and unpredictable effects can happen. If these are all books you have a high confidence are not malformed (and tested on kobo before), I would go ahead and put hundreds of them on there at a time. When testing new files, small batches is the way to go.

Recent firmware updates have introduced bugs with connecting the USB drive that leaves the filesystem and database in an unclean state (and thus, very vulnerable to corruption.) I haven't experienced this in a long time since the most recent 2 updates, but sporadic reports of this happening are still being posted on this board, and the exact trigger conditions still a mystery and very speculative. The best way I know to make sure you avoid this problem is to reboot the Kobo before connecting to PC, (and do so before opening any books.)

[TheSmitty]

Originally Posted by Cactus Chef

There's no hard limit to how much you can transfer in one fell swoop, evidence suggests it simply gets riskier the more books you push. I would recommend capping at 100 per push, then disconnecting, let the Kobo update the database, verify that you can open and page through one of the new books, and then starting with the next batch. No need to power off the device.

I recently (last week) realized that many of the books I had side-loaded on my Libra 2 were epub format and not kepub which is what I wanted. So I deleted everything off of the device then went into Calibre and sent approximately 800 books from my library in Calibre to the Kobo. It took a while but it managed to do it and once the Kobo was ejected it imported them all without a problem. So, for me it has worked pretty well.

[DNSB]

Originally Posted by Grins

In the “good old days” (until around the end of 2022) it was possible to connect a Kobo reader to calibre (via USB cable) and copy / save 200 or more books or documents (all in EPUB format) onto the reader with single file transfer. It was even possible to have 500 document saved with a single file transfer.

The last batch of files I sent from my computer to a Kobo Sage was 4500 epubs. Note that these epubs are all error checked with epubcheck. I also have sent ~13,000 epubs to a Clara HD (internal storage updated to 128GB) and while importing them took quite a while, no other issues were seen.

Originally Posted by Grins

However, the software updates in 2023 made this impossible - worse, there were database errors on the readers, which even led to the complete inoperability of at least 2 devices.

I've seen the database issues however they have nothing to do with the number of books sent in one batch and, as far as I can tell, are due to the database not being properly closed before the USB connection is established. You will see the BookReader.sqlite and/or the KoboReader.sqlite file will have their <databasename>.sqlite-wal and <databasename>.sqlite-shm files still present if you look at the contents of the .kobo directory.

On my Sage, I do use a reboot option in NickelMenu to ensure the database is closed before connecting to my computer.

I am rather surprised that you would state a database error would result in complete inoperability. At the worst, database corruption would require a factory reset.

Originally Posted by Grins

Since then I have been using Pocketbook devices (in the manner described above) and have no longer had any problems.

Personally, I find Pocketbook devices to be limiting in their metadata management.

Originally Posted by Grins

This thread is not about whether 500 "real" books or documents can be read in a short time (that's not the application here, by the way), but about the stability of the device software.

Now a student of mine wants to use his reader in a similar way as me. He owns a Kobo Clara 2E. Given my experiences, I advised him against it, but he "loves" his Kobo

Therefore my questions: How do you copy EPUB documents from calibre to the Kobo reader via file transfer? Which of the 3 “device interface plugins” do you use? How many books do you copy onto the reading device at a single time? If you divide 500 documents into smaller quantities, how large are these quantities? Do you switch off the device completely once the file transfer and database update has been successful so that the database can completely rebuild itself?

There are only two device interface plugins for Kobo Touch devices. The Kobo Touch driver which comes builtin to calibre and the Kobo Touch Extended driver which is an optional addon and is only useful if you want to convert from epub to kepub on the fly.

As mentioned above, I sent ~13K epubs to my Clara HD in one pass and saw no issues. I don't switch off the device though that does not trigger a database rebuild, it would simply ensure that the database on disk and in RAM copies will be synchronized.

I did run into issues when importing books whether 500 or a single book when I first started using calibre however since I started error checking books as they are added into my calibre library, I haven't seen import issues in quite a few years.

[Lys]

Originally Posted by Grins

Therefore my questions: How do you copy EPUB documents from calibre to the Kobo reader via file transfer? Which of the 3 “device interface plugins” do you use? How many books do you copy onto the reading device at a single time? If you divide 500 documents into smaller quantities, how large are these quantities? Do you switch off the device completely once the file transfer and database update has been successful so that the database can completely rebuild itself?

I have almost 3K books on my Libra H2O. Just the other day I transferred around 150 new fics in one go. No issue now, no issue since I started using Kobo. I'm on latest firmware (and on Calibre 7.5.1). I'm using KoboTouchExtended Driver.

That said, your issue seems more connected to database corruption than anything related to the device itself. I haven't had a DB corruption since forever now, but a couple of reasons you may corrupt the DB are:
1) Connecting to Calibre while you have an ebook open on the device (I always make sure to be on the home page before connecting)
2) Reconnect to Calibre before your Kobo has properly digested the new ebooks in its database.

[DNSB]

Originally Posted by Lys

That said, your issue seems more connected to database corruption than anything related to the device itself. I haven't had a DB corruption since forever now, but a couple of reasons you may corrupt the DB are:
1) Connecting to Calibre while you have an ebook open on the device (I always make sure to be on the home page before connecting)
2) Reconnect to Calibre before your Kobo has properly digested the new ebooks in its database.

3) The database is not properly closed before the USB connection is finalized. This is easily diagnosed by seeing .sqlite-wal and .sqlite-shm files in the .kobo directory. Kobo is aware of this issue and it should be fixed in the next firmware release though my hopes of that happening Real Soon Now have long been dashed. This seems to happen more often when using the Sage or Elipsa.

[Colonel Cathcart]

Originally Posted by DNSB

3) The database is not properly closed before the USB connection is finalized. This is easily diagnosed by seeing .sqlite-wal and .sqlite-shm files in the .kobo directory. Kobo is aware of this issue and it should be fixed in the next firmware release though my hopes of that happening Real Soon Now have long been dashed. This seems to happen more often when using the Sage or Elipsa.

This sounds like a feature could be added to Calibre or, alternatively, to the Kobo plugins to check for the existence of those file when connecting and warn the user. Could that help eliminate completely or reduce the number of DB corruption ever happening?

[rashkae]

Originally Posted by Colonel Cathcart

This sounds like a feature could be added to Calibre or, alternatively, to the Kobo plugins to check for the existence of those file when connecting and warn the user. Could that help eliminate completely or reduce the number of DB corruption ever happening?

If nothing else, doing so would be a great way to ensure a new firmware that fixes the problem entirely gets released within a day, if not even before the Plugin update.

[Grins]

Thank you all for the detailed answers. Until now I have always used the "Kobo Extended Driver", but the error with the database also appears when I use the other driver "Kobo Touch" - regardless of whether EPUB or KEPUB is used.
I also wrote about 2 damaged devices, one of which has now been replaced by the Kobo support. Unfortunately the database errors appears again
And yes, all documents have been checked positively. Many of them were automatically created by LATEX with output to EPUB.

I now see 3 options: 1) waiting for a new firmware release (as everybody else), 2) using NickelMenu (so far no experience on my side with that) and 3) permanently switching to Pocketbook. The latter option seems safest to me at the moment.

Personally, I'm surprised that the database error hasn't been fixed yet. As a ereader manufacturer, I would pay close attention to the discussions in the various MobileRead forums - because it is usually very difficult to get detailed feedback from different customers.

Thanks again everyone for the discussion here!

[Sirtel]

There is a fourth option - stay on an earlier firmware version. Both my Kobos are still on 4.34 and will stay that way until a fix is released. No issues with database corruption so far.

[Aleron Ives]

Yes, I'm staying on 4.35 and haven't had any database corruptions, either. The problem was introduced in 4.37 (4.36 was the Elipsa 2E launch firmware and isn't available on other devices).

[Rid]

I have imported about 600 books at once without an issue, have done this many times. Although I only have done this with a factory reset. I upgrade the firmware before I log into the device or import books.

The reason I have done the factory resets was because of synch issues and database corruption. I seldom do a factory reset anymore since I worked out it was the collections issue causing the synch problem. But every now and then I think a factory reset needs to be done just like any computer to bring it back up to speed again.

I agree with the above comments but maybe it's a good idea to also close and reopen Calibre, since it has it's own database as well...?

[Sirtel]

Originally Posted by Rid

I have imported about 600 books at once without an issue, have done this many times. Although I only have done this with a factory reset. I upgrade the firmware before I log into the device or import books.

The reason I have done the factory resets was because of synch issues and database corruption. I seldom do a factory reset anymore since I worked out it was the collections issue causing the synch problem. But every now and then I think a factory reset needs to be done just like any computer to bring it back up to speed again.

I've never done a factory reset to any of my electronic devices (ereaders, laptops, tablets, phones...), ever. Unless I was going to sell them, of course.

[Rid]

I am probably more anal than you, but the slower speed annoys the sh!) out of me after a while! Pun intended...