Suddenly Calibre doesn't mark some books on a Kobo Glo device

[bernard.ryefield]

On a total of 8346 books on the SD card in my Kobo Glo, 1979 have no green check (not on device) anymore.
The books are on the device and are readable.
The device database has been verified by Kobo Utilities and is OK.
The device driver is KoboTouchExtended 3.0.2.
There's approx. 5.5 Go left on the SD card.

[Terisa de morgan]

Have you checked they are not duplicated in calibre library?

[bernard.ryefield]

Originally Posted by Terisa de morgan

Have you checked they are not duplicated in calibre library?

There are no duplicates.

[davidfor]

Originally Posted by bernard.ryefield

On a total of 8346 books on the SD card in my Kobo Glo, 1979 have no green check (not on device) anymore.
The books are on the device and are readable.
The device database has been verified by Kobo Utilities and is OK.
The device driver is KoboTouchExtended 3.0.2.
There's approx. 5.5 Go left on the SD card.

When I book doesn't get automatically matched to a book in the library, it is usually because the title or author on the device isn't a close enough match to the library. Multiple authors usually prevent the match from happening. Unless you use a plugboard to put the authors on the device correctly, the Kobo devices will probably only show one author. Other reasons are series info included in the title (common for purchased books) or using a title format with the article at the end. Unfortunately, in the end you might need to look at each book.

You can manually match the books in the device list. Right-click on the book and select "Match book to library". Then use the search to find the book, select it and press OK. After you have done this, you need to select the "Update cached metadata on device" option on the menu for the device button. This write the matches the "metadata.calibre" on the device and this is used the next time the device is plugged in to help with the matching. Unfortunately, for 1979 books, that is going to take a long time.

If you have examples of books that don't match but look like they should, send me the details and I'll have a look. There might be something that can be done to improve the matching.

[bernard.ryefield]

Originally Posted by davidfor

When I book doesn't get automatically matched to a book in the library, it is usually because the title or author on the device isn't a close enough match to the library. Multiple authors usually prevent the match from happening. Unless you use a plugboard to put the authors on the device correctly, the Kobo devices will probably only show one author. Other reasons are series info included in the title (common for purchased books) or using a title format with the article at the end. Unfortunately, in the end you might need to look at each book.

I probably should have been more precise. Most of the books not showing with a green check are old books, causing no problem to this day.
My current plugboard for the Kobotouchextended driver is:

{author_sort}/{title} - {authors}

I remove any mention to the series in the titles as I put the appropriate series info in the series field for books in a series.

Originally Posted by davidfor

You can manually match the books in the device list. Right-click on the book and select "Match book to library". Then use the search to find the book, select it and press OK. After you have done this, you need to select the "Update cached metadata on device" option on the menu for the device button. This write the matches the "metadata.calibre" on the device and this is used the next time the device is plugged in to help with the matching. Unfortunately, for 1979 books, that is going to take a long time.

If you have examples of books that don't match but look like they should, send me the details and I'll have a look. There might be something that can be done to improve the matching.

I have already deleted the last books with a green check sent to the device. I will try the "Match book to library" operation on the last books without a green check and report here.

[bernard.ryefield]

Originally Posted by bernard.ryefield

I will try the "Match book to library" operation on the last books without a green check and report here.

No improvement. Matching books one by one ... Any idea how to avoid this strenuous drill?

[davidfor]

Originally Posted by bernard.ryefield

I probably should have been more precise. Most of the books not showing with a green check are old books, causing no problem to this day.
My current plugboard for the Kobotouchextended driver is:

I remove any mention to the series in the titles as I put the appropriate series info in the series field for books in a series.

That isn't a metadata plugboard, that is the save template and is used in naming the files when they are saved to disk on the device.

The metadata plugboard is a separate are in the calibre preferences. They are used to fiddle with the metadata in the books as they are sent to the devices. The most common is to add the series info to the title. This isn't needed for Kobo devices. But, the plugboard I use for authors is:

program: raw_list('authors', ', ')

That has no effect on single authors, but for multiple authors you get: "FN1 LN1, FN2 LN2, FN3 LN3". This is the format that the Kobo devices expect and will interpret generally correctly.

I have already deleted the last books with a green check sent to the device. I will try the "Match book to library" operation on the last books without a green check and report here.

I think I need to know what is happening. Can you run calibre in debug mode and post the log? If you can, I also need some debugging turned on in the driver. What I need is:

1. Pick a book that isn't matching.
2. Open the KoboTouch driver configuration and go third tab.
3. In the "Title to test when debugging" field, put the title of the select book, or at least enough of it to be fairly unique.
4. Save the change.
5. Restart calibre in debug mode by right-clicking on the preferences button and selecting the option there.
6. When calibre restarts, connect the device and wait for the jobs to finish running.
7. Close calibre and it will display the debug log.
8. Post the log or PM it to me if you prefer.

Hopefully I can see what is wrong for that book and we can sort it out.

Originally Posted by bernard.ryefield

No improvement. Matching books one by one ... Any idea how to avoid this strenuous drill?

It really depends on what is wrong. If it is something funky in the metadata, we might be able to fix it in the books or via a metadata plugboard. Then you would need to replace the books on the device. If it is a bug in the matching algorithm, fixing that and waiting for the next calibre update would need to happen. But, it might be a problem that can't be fixed. For example, for books purchased from Kobo and downloaded directly to the device, they frequently have inconsistent titles (eg. Kobo has Isaac Asimov's "The Caves of Steel" as just "Caves of Steel". That doesn't match with the library). We can fix the metadata for this on the device, but they will probably be wiped when you next sync the device.

[bernard.ryefield]

Originally Posted by davidfor

That isn't a metadata plugboard, that is the save template and is used in naming the files when they are saved to disk on the device.

The metadata plugboard is a separate are in the calibre preferences. They are used to fiddle with the metadata in the books as they are sent to the devices. The most common is to add the series info to the title. This isn't needed for Kobo devices. But, the plugboard I use for authors is:

program: raw_list('authors', ', ')

That has no effect on single authors, but for multiple authors you get: "FN1 LN1, FN2 LN2, FN3 LN3". This is the format that the Kobo devices expect and will interpret generally correctly.

I didn't notice any problem with the list of authors on the device before, I will pay more attention to the books with multiple authors in the future.
What is the difference between your plugboard and the default?

Originally Posted by davidfor

I think I need to know what is happening. Can you run calibre in debug mode and post the log? If you can, I also need some debugging turned on in the driver. What I need is:

1. Pick a book that isn't matching.
2. Open the KoboTouch driver configuration and go third tab.
3. In the "Title to test when debugging" field, put the title of the select book, or at least enough of it to be fairly unique.
4. Save the change.
5. Restart calibre in debug mode by right-clicking on the preferences button and selecting the option there.
6. When calibre restarts, connect the device and wait for the jobs to finish running.
7. Close calibre and it will display the debug log.
8. Post the log or PM it to me if you prefer.

Hopefully I can see what is wrong for that book and we can sort it out.

Done and sent.

[davidfor]

Originally Posted by bernard.ryefield

I didn't notice any problem with the list of authors on the device before, I will pay more attention to the books with multiple authors in the future.
What is the difference between your plugboard and the default?

There is no default metadata plugboard. What you had above looks like the file save template. These are used for two very different purposes.

The file save template is used to name the book file when it is put on the device. It can add directories and use parts of the metadata. In one way, Kobo devices don't use the file name. The import the metadata from the book when it is first seen on the device and what you see is that metadata. The file name will only be used if the book has no metadata. The file name is used internally, but not exposed to to the user.

The metadata plugboard is used when calibre sends a book to the device. The metadata in the book is updated during the send. The metadata plugboard allows the metadata to be tweaked. As I think I said above, adding the series info to the title is a popular plugboard. And fiddling with the author is also popular to get sorting on the devices working better.

I have had a very quick look at the logs you sent. For the example book, I can see:

Title               : La lignée
Title sort          : lignée, La
Author(s)           : Guillermo Del Toro [Toro, Guillermo Del & Hogan, Chuck]

I can see two issues with that. The title has unicode characters in it. These probably display correctly, but might fail comparisons to whatever is in the calibre library. I will have to do some experiments on this.

The other problem is the author. What is in the calibre library for the author? The match looks like it is using "Guillermo Del Toro". So, I'm not sure if it is the author or the title that is the problem.

Could you send me a copy of the book? Don't post it as it will the break copyright rules. Though I only need the shell, so stripping the text and sending that would be OK. And I'd also like to see the OPF file from the library. With those files, I should be able to reconstruct this for testing.

[bernard.ryefield]

Originally Posted by davidfor

I have had a very quick look at the logs you sent. For the example book, I can see:

Title               : La lignée
Title sort          : lignée, La
Author(s)           : Guillermo Del Toro [Toro, Guillermo Del & Hogan, Chuck]

I can see two issues with that. The title has unicode characters in it. These probably display correctly, but might fail comparisons to whatever is in the calibre library. I will have to do some experiments on this.

The other problem is the author. What is in the calibre library for the author? The match looks like it is using "Guillermo Del Toro". So, I'm not sure if it is the author or the title that is the problem.

Could you send me a copy of the book? Don't post it as it will the break copyright rules. Though I only need the shell, so stripping the text and sending that would be OK. And I'd also like to see the OPF file from the library. With those files, I should be able to reconstruct this for testing.

In the calibre library the authors field is complete, that is "Guillermo Del Toro & Chuck Hogan". In the SD card view the author's name is only "Guillermo Del Toro". But the folder name and the file name on the device are "Toro, Guillermo Del & Hogan, Chuck" and "lignee, La - Guillermo Del Toro & Chuck Hogan.kepub.epub" respectively. I send you the needed infos .

[davidfor]

Originally Posted by bernard.ryefield

In the calibre library the authors field is complete, that is "Guillermo Del Toro & Chuck Hogan". In the SD card view the author's name is only "Guillermo Del Toro". But the folder name and the file name on the device are "Toro, Guillermo Del & Hogan, Chuck" and "lignee, La - Guillermo Del Toro & Chuck Hogan.kepub.epub" respectively.

Sorry, but you are looking at the wrong thing here. The file name on the device is completely immaterial when matching the book on the device to the book in the library. The KoboTouch driver uses the database on the device, not the file names. When the book is first added to the device, the device extracts the metadata from inside the book and adds this to the database. When connect, the KoboTouch driver reads this database to find out the details of the books on the device. Then it attempts to match that information with what is in the calibre library.

I send you the needed infos .

Looking at this and putting it on one of my devices, the problem is as I first said, the author. The book has two authors, "Guillermo Del Toro & Chuck Hogan". When calibre updates the metadata when sending the book to the device, it puts the authors as two separate dc:creator tags. Unfortunately, the Kobo devices only reads the first one. This means the author in the database is "Guillermo Del Toro", and you will see this when you look at the book on the device.

The solution to this is, as I said above, to use a metadata plugboard to set the author in the book during the send to the device. If this done correctly, the device will see both authors, they will both show on the device, and when the device is connected, calibre will correctly match the book on the device to the book in the library.

[bernard.ryefield]

Originally Posted by davidfor

Looking at this and putting it on one of my devices, the problem is as I first said, the author. The book has two authors, "Guillermo Del Toro & Chuck Hogan". When calibre updates the metadata when sending the book to the device, it puts the authors as two separate dc:creator tags. Unfortunately, the Kobo devices only reads the first one. This means the author in the database is "Guillermo Del Toro", and you will see this when you look at the book on the device.

I confirm there is only one author for this book. However, I have other books with "Guillermo Del Toro & Chuck Hogan" as authors and these authors are all correctly set on the device. Other books with only one author are also concerned with this problem.

[davidfor]

Originally Posted by bernard.ryefield

I confirm there is only one author for this book. However, I have other books with "Guillermo Del Toro & Chuck Hogan" as authors and these authors are all correctly set on the device. Other books with only one author are also concerned with this problem.

And that means that those books have the author set to "Guillermo Del Toro & Chuck Hogan". Or something else has been done to make sure they match, such as the manual matching above or a metadata plugboard has been used. Or the source for the book set the author like this. There are a few reasons for the book matching to work, and unless I can look at it completely, I won't know which it is.

And there are more reasons for the book not to match than just the multiple authors problem. This is just the one I know fails, why it fails, is easy to explain and I know how to fix. I have a lot of books that don't automatically match, but most are those downloaded to the device from Kobo and the title doesn't match what I have in the calibre library. Mostly this is because the title includes the series name or there is some formatting difference. One was the title in one place had the UK spelling and the other the US spelling. Problems like either of these cannot be fixed except manually.

If there are other reasons, I don't know them. Some might be fixable in code. But, I need to see examples. The matches use the title and authors in the library vs in the database on the device. If I have these details, then I can look at them and maybe improve things.

[bernard.ryefield]

Originally Posted by davidfor

And that means that those books have the author set to "Guillermo Del Toro & Chuck Hogan". Or something else has been done to make sure they match, such as the manual matching above or a metadata plugboard has been used. Or the source for the book set the author like this. There are a few reasons for the book matching to work, and unless I can look at it completely, I won't know which it is.

I PM you one of the books with this problem AND the authors set correctly, with the debug log and OPF file.

Originally Posted by davidfor

And there are more reasons for the book not to match than just the multiple authors problem. This is just the one I know fails, why it fails, is easy to explain and I know how to fix. I have a lot of books that don't automatically match, but most are those downloaded to the device from Kobo and the title doesn't match what I have in the calibre library. Mostly this is because the title includes the series name or there is some formatting difference. One was the title in one place had the UK spelling and the other the US spelling. Problems like either of these cannot be fixed except manually.

I suggest we try a step-by-step approach and solve the mystery for the books above. I have some books with only one author and no apparent problem in the title we can try later.

[davidfor]

Originally Posted by bernard.ryefield

I PM you one of the books with this problem AND the authors set correctly, with the debug log and OPF file.



I suggest we try a step-by-step approach and solve the mystery for the books above. I have some books with only one author and no apparent problem in the title we can try later.

At this point, that's about the only thing to do. But, maybe we'll see a pattern and be able to something about that.