Calibre database problem

embryo · 04-10-2015, 08:16 PM

My system is a

Kobo Glo with
KSM 07(update02)
Koreader 2014.11-154
vlasovsoft 2015.03
Nickel 3.8.0
dual-booting Android
Calibre 1.48
Win XP SP3

My library is on the SD card inside a "/.Books/" (system) folder.
There are also two '/Books/" folders with a few more books if I need to read something in Nickel.

The problem:
Calibre does not update the 'metadata.calibre' database on SD card and on board.

I can connect the reader to the PC in 4 ways:

With KSM normally
With KSM using the Nickel ID
With Nickel
With Android

The only way that Calibre finds the books in /.Books/ is with Android.

With the 3 other ways a got the same results.
If the database is created (with the Android way) Calibre can see the books in the reader (can read the database and the addresses).
But if I add a book from explorer and reopen Calibre, it doesn't update the database. It can not see the book in the /.Books/ folder.
If I delete the database files, it finds only the books in the "/Books/" directories but not in the "/.Books/".
I tried with the 'Kobo Touch' driver and 'Kobo Touch Extended'.
The settings are the same on both, and since I don't use kepubs I can use either of them.

I'm certain that there is an obvious thing that I'm missing, but I can't find it.

Update: To make things even more interesting ...
Just when I was finishing the post, I tried to check if it can send the books into the SD card, so I reconnected the reader.
After countless tries with restarts and reboots, changing everything that I could think off, Calibre can not connect anymore to the reader with anything except Android.

It gives me this error:

Code:

calibre, version 1.48.0
ERROR: Error: Error communicating with device

database disk image is malformed

Traceback (most recent call last):
  File "site-packages\calibre\gui2\device.py", line 86, in run
  File "site-packages\calibre\gui2\device.py", line 490, in _books
  File "site-packages\calibre\devices\kobo\driver.py", line 1767, in books
DatabaseError: database disk image is malformed

I tried with a working set of database files (working just 10 minutes ago) and with everything cleared(no Calibre files at both roots).
I can see the reader in the PC just fine.
The only thing I did before was to try to search in the database with Koreader, which worked OK.
I don't think its relevant, but just stating the events.

Anyway, its getting really late.
I give up for today...

davidfor · 04-10-2015, 10:21 PM

I'm not completely sure what you are asking and what you really want. So, I'm going to dump some information. This is based on what you have described and what I know about how calibre and the Kobo firmware works.

And as a short answer: Apart from the last error, I believe it is all working the way it is supposed to.

I think this really depends on how are you connecting calibre to the device? There are several ways and what books are seen by calibre depends on the way the books that will be found.

But firstly, you might have a misunderstanding about the "metadata.calibre" file. This is not really a database of books on the device. It is an aid for calibre to connect the books on the device to the books in the calibre library. Calibre finds the books on the device using the appropriate method for the device. Then it matches those books to the books in the library. Then it writes the metadata.calibre file to help the matching the next time. The metadata.calibre file will also be updated when you send a book to the device, delete a book or use the "Match book to library" function.

The "normal" way for calibre to find books on a Kobo device is the KoboTouch or KoboTouchExtend driver. The driver reads the database on the device. Any book that is in the database will be shown in the device list. If there are books that are on the device and not in the database, calibre knows nothing about them.

Another way is to disable the drivers and "Connect to folder" to treat a folder on the device as a a "folder device". Whit this, calibre scans the folder and subfolders for books. The books that are found are shown in the device list. If there is a metadata.calibre in the folder being used as the root, then it is used to aid matching the books to the library.

By "With Android" I think you mean as an MTP device. In this case, the device presents a folder to the OS and calibre scans that. The folder is probably the root folder of the device and calibre scans all the folders. After the first scan, you can configure the MTP driver to only scan particular folders. And I think this can be configured separately for each Android device.

The last way I can think of is to use "Save to disk" to put the books on the device. This means just treating the device as a MSD and probably needs the drivers disabled or calibre told to ignore the device. In this case, calibre has no idea of what is on the device.

I'm not sure exactly what happens with a system folder called ".books". In Linux, Mac and I think Android, folders whose name start with a dot are usually ignored when scanning. Hence putting books inside one on the Kobo device means that nickel won't find them. But, for XP, it's just a directory but making it a system folder might mean it isn't scanned by calibre when looking for books. From your results, this is probably what is happening.

As I don't use KSM, I'm not sure what you mean by "With KSM normally" and "With KSM using the nickel ID". I suspect that is whether KSM presents the device with the correct vendor and product ids. If so, then if the ids used are correct, then the driver will see the device as a Kobo device and handle it as I have described above. Hence, it won't see books in the .Books folder.

The error message means that the calibre is seeing the ereader as a Kobo device and using the driver. And the database on the device is corrupt. That's the KoboReader.sqlite database that nickel uses is corrupt. If you are going to use nickel, that needs to be fixed.

Still here?

I believe all that is correct, but I might have missed something or messed up the explanation. If you want clarification, ask away.

embryo · 04-11-2015, 06:34 AM

Quote:

Originally Posted by davidfor

I'm not completely sure what you are asking and what you really want. So, I'm going to dump some information. This is based on what you have described and what I know about how calibre and the Kobo firmware works.

First of all, thank you for your time.
About my problem.
"The problem:
Calibre does not update the 'metadata.calibre' database on SD card and on board."
After I read your info, I should rephrase it as:
"Calibre does not see my books on the SD card and on board."

Quote:

And as a short answer: Apart from the last error, I believe it is all working the way it is supposed to.

I think this really depends on how are you connecting calibre to the device? There are several ways and what books are seen by calibre depends on the way the books that will be found.

But firstly, you might have a misunderstanding about the "metadata.calibre" file. This is not really a database of books on the device. It is an aid for calibre to connect the books on the device to the books in the calibre library. Calibre finds the books on the device using the appropriate method for the device. Then it matches those books to the books in the library. Then it writes the metadata.calibre file to help the matching the next time. The metadata.calibre file will also be updated when you send a book to the device, delete a book or use the "Match book to library" function.

The "normal" way for calibre to find books on a Kobo device is the KoboTouch or KoboTouchExtend driver. The driver reads the database on the device. Any book that is in the database will be shown in the device list. If there are books that are on the device and not in the database, calibre knows nothing about them.

To clarify:
The appropriate method for Kobo is to check the KoboReader.sqlite?
It does not scan the SD card and the on board space?

Quote:

Another way is to disable the drivers and "Connect to folder" to treat a folder on the device as a a "folder device". Whit this, calibre scans the folder and subfolders for books. The books that are found are shown in the device list. If there is a metadata.calibre in the folder being used as the root, then it is used to aid matching the books to the library.

By "With Android" I think you mean as an MTP device.

I have configure the User Defined USB driver of Calibre.
I suppose it does use this protocol.

Quote:

In this case, the device presents a folder to the OS and calibre scans that. The folder is probably the root folder of the device and calibre scans all the folders. After the first scan, you can configure the MTP driver to only scan particular folders. And I think this can be configured separately for each Android device.

The last way I can think of is to use "Save to disk" to put the books on the device. This means just treating the device as a MSD and probably needs the drivers disabled or calibre told to ignore the device. In this case, calibre has no idea of what is on the device.

I'm not sure exactly what happens with a system folder called ".books". In Linux, Mac and I think Android, folders whose name start with a dot are usually ignored when scanning. Hence putting books inside one on the Kobo device means that nickel won't find them. But, for XP, it's just a directory but making it a system folder might mean it isn't scanned by calibre when looking for books. From your results, this is probably what is happening.

I didn't make it a system folder in Windows.
I just pointed out that it was treated 'as' system by Linux (Kobo and Android)
I set the Kobo driver to send books to the /.Books/ folder and when I tried it, it created a /_Books/ folder at the root.
So, even Calibre don't like 'dot starting' folders.

Quote:

As I don't use KSM, I'm not sure what you mean by "With KSM normally" and "With KSM using the nickel ID". I suspect that is whether KSM presents the device with the correct vendor and product ids.

I think it does exactly that.

Quote:

If so, then if the ids used are correct, then the driver will see the device as a Kobo device and handle it as I have described above. Hence, it won't see books in the .Books folder.

The error message means that the calibre is seeing the ereader as a Kobo device and using the driver. And the database on the device is corrupt. That's the KoboReader.sqlite database that nickel uses is corrupt. If you are going to use nickel, that needs to be fixed.

Yes. By fixing that the error in Calibre stoped.

Quote:

Still here?

I believe all that is correct, but I might have missed something or messed up the explanation. If you want clarification, ask away.

Well, I must say that it was a very enlightening explanation!
So, now that I understand the things a little better, I must find a way for my system to work.

What I want, is to use the 'SD card/.Books/' as my main library folder.
There are almost 4k books in there that I don't want Nickel to see for obvious reasons.
But if Nickel can not see them, Calibre wont see them also.
I also can't use the User Defined USB driver of Calibre, since I have configure it for the Android part of the device.
I wish there was an option to use a second one.

So I must disable the Kobo drivers and learn how to use the "folder device" thing, or connect only through Android.

Thanks again for your help.

Ken Maltby · 04-11-2015, 10:24 AM

With the Kobo "drivers" in calibre disabled, and using the "Send to folder" function aimed at your .Books folder, calibre should have no problem seeing the folders it will add to .Books and the metadata catalog it creates there.

Using other "Send to device" methods, the metadata catalog and the folders caliber creates will be added to the root of the device (/mnt/onboard and /mnt/sd). Koreader allows you to set two locations where it will look for the metadata catalog.

Luck;
Ken

embryo · 04-11-2015, 11:25 AM

Since I mostly use KSM I just setup the User Defined USB driver of Calibre to see KSM.
Now I can connect and Calibre will find all the books from the internal and external storage.

"Folder Device Interface" almost solved my problem but I had to connect to two folders (Internal - External) so it was not perfect.
The other difficulty is that it didn't create a 'driveinfo.calibre' and a 'metadata.calibre' file but instead a '.driveinfo.calibre' and a '.metadata.calibre'
Koreader could read it OK but then I'd have another pair of files to update and anyway ...

I wish there was a way to have more than one configuration of the User Defined USB driver, but thats just a minor annoyance.

04-10-2015, 08:16 PM	#1
embryo Fanatic Posts: 523 Karma: 64554 Join Date: Aug 2013 Device: Kobo Glo, GloHD	Calibre database problem My system is a Kobo Glo with KSM 07(update02) Koreader 2014.11-154 vlasovsoft 2015.03 Nickel 3.8.0 dual-booting Android Calibre 1.48 Win XP SP3 My library is on the SD card inside a "/.Books/" (system) folder. There are also two '/Books/" folders with a few more books if I need to read something in Nickel. The problem: Calibre does not update the 'metadata.calibre' database on SD card and on board. I can connect the reader to the PC in 4 ways: With KSM normally With KSM using the Nickel ID With Nickel With Android The only way that Calibre finds the books in /.Books/ is with Android. With the 3 other ways a got the same results. If the database is created (with the Android way) Calibre can see the books in the reader (can read the database and the addresses). But if I add a book from explorer and reopen Calibre, it doesn't update the database. It can not see the book in the /.Books/ folder. If I delete the database files, it finds only the books in the "/Books/" directories but not in the "/.Books/". I tried with the 'Kobo Touch' driver and 'Kobo Touch Extended'. The settings are the same on both, and since I don't use kepubs I can use either of them. I'm certain that there is an obvious thing that I'm missing, but I can't find it. Update: To make things even more interesting ... Just when I was finishing the post, I tried to check if it can send the books into the SD card, so I reconnected the reader. After countless tries with restarts and reboots, changing everything that I could think off, Calibre can not connect anymore to the reader with anything except Android. It gives me this error: Code: calibre, version 1.48.0 ERROR: Error: Error communicating with device database disk image is malformed Traceback (most recent call last): File "site-packages\calibre\gui2\device.py", line 86, in run File "site-packages\calibre\gui2\device.py", line 490, in _books File "site-packages\calibre\devices\kobo\driver.py", line 1767, in books DatabaseError: database disk image is malformed I tried with a working set of database files (working just 10 minutes ago) and with everything cleared(no Calibre files at both roots). I can see the reader in the PC just fine. The only thing I did before was to try to search in the database with Koreader, which worked OK. I don't think its relevant, but just stating the events. Anyway, its getting really late. I give up for today...

04-11-2015, 10:24 AM	#4
Ken Maltby Wizard Posts: 4,466 Karma: 6900052 Join Date: Dec 2009 Location: The Heart of Texas Device: Boox Note2, AuraHD, PDA,	With the Kobo "drivers" in calibre disabled, and using the "Send to folder" function aimed at your .Books folder, calibre should have no problem seeing the folders it will add to .Books and the metadata catalog it creates there. Using other "Send to device" methods, the metadata catalog and the folders caliber creates will be added to the root of the device (/mnt/onboard and /mnt/sd). Koreader allows you to set two locations where it will look for the metadata catalog. Luck; Ken Last edited by Ken Maltby; 04-11-2015 at 10:28 AM.

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
Problem with Database?	vivaldirules	Upload Help	3	05-08-2014 07:30 AM
Help! Corrupted database problem	Marok	Library Management	12	06-24-2013 09:01 AM
Calibre Database cp Kindle Database	mitch13	Library Management	1	05-22-2011 07:33 PM
Calibre problem to start (database)	disi	Calibre	1	10-11-2010 06:35 PM
Database problem	zin	Calibre	24	09-22-2010 06:19 AM

04-10-2015, 10:21 PM	#2
davidfor Grand Sorcerer Posts: 24,905 Karma: 47303824 Join Date: Jul 2011 Location: Sydney, Australia Device: Kobo:Touch,Glo, AuraH2O, GloHD,AuraONE, ClaraHD, Libra H2O; tolinoepos	I'm not completely sure what you are asking and what you really want. So, I'm going to dump some information. This is based on what you have described and what I know about how calibre and the Kobo firmware works. And as a short answer: Apart from the last error, I believe it is all working the way it is supposed to. I think this really depends on how are you connecting calibre to the device? There are several ways and what books are seen by calibre depends on the way the books that will be found. But firstly, you might have a misunderstanding about the "metadata.calibre" file. This is not really a database of books on the device. It is an aid for calibre to connect the books on the device to the books in the calibre library. Calibre finds the books on the device using the appropriate method for the device. Then it matches those books to the books in the library. Then it writes the metadata.calibre file to help the matching the next time. The metadata.calibre file will also be updated when you send a book to the device, delete a book or use the "Match book to library" function. The "normal" way for calibre to find books on a Kobo device is the KoboTouch or KoboTouchExtend driver. The driver reads the database on the device. Any book that is in the database will be shown in the device list. If there are books that are on the device and not in the database, calibre knows nothing about them. Another way is to disable the drivers and "Connect to folder" to treat a folder on the device as a a "folder device". Whit this, calibre scans the folder and subfolders for books. The books that are found are shown in the device list. If there is a metadata.calibre in the folder being used as the root, then it is used to aid matching the books to the library. By "With Android" I think you mean as an MTP device. In this case, the device presents a folder to the OS and calibre scans that. The folder is probably the root folder of the device and calibre scans all the folders. After the first scan, you can configure the MTP driver to only scan particular folders. And I think this can be configured separately for each Android device. The last way I can think of is to use "Save to disk" to put the books on the device. This means just treating the device as a MSD and probably needs the drivers disabled or calibre told to ignore the device. In this case, calibre has no idea of what is on the device. I'm not sure exactly what happens with a system folder called ".books". In Linux, Mac and I think Android, folders whose name start with a dot are usually ignored when scanning. Hence putting books inside one on the Kobo device means that nickel won't find them. But, for XP, it's just a directory but making it a system folder might mean it isn't scanned by calibre when looking for books. From your results, this is probably what is happening. As I don't use KSM, I'm not sure what you mean by "With KSM normally" and "With KSM using the nickel ID". I suspect that is whether KSM presents the device with the correct vendor and product ids. If so, then if the ids used are correct, then the driver will see the device as a Kobo device and handle it as I have described above. Hence, it won't see books in the .Books folder. The error message means that the calibre is seeing the ereader as a Kobo device and using the driver. And the database on the device is corrupt. That's the KoboReader.sqlite database that nickel uses is corrupt. If you are going to use nickel, that needs to be fixed. Still here? I believe all that is correct, but I might have missed something or messed up the explanation. If you want clarification, ask away.

04-11-2015, 11:25 AM	#5
embryo Fanatic Posts: 523 Karma: 64554 Join Date: Aug 2013 Device: Kobo Glo, GloHD	Since I mostly use KSM I just setup the User Defined USB driver of Calibre to see KSM. Now I can connect and Calibre will find all the books from the internal and external storage. "Folder Device Interface" almost solved my problem but I had to connect to two folders (Internal - External) so it was not perfect. The other difficulty is that it didn't create a 'driveinfo.calibre' and a 'metadata.calibre' file but instead a '.driveinfo.calibre' and a '.metadata.calibre' Koreader could read it OK but then I'd have another pair of files to update and anyway ... I wish there was a way to have more than one configuration of the User Defined USB driver, but thats just a minor annoyance.

Advert

Advert