new strange " available formats" issue/error

[stumped]

this came about when testing the latest CC amazon beta but it may not be a CC issue.
not sure now to continue the troubleshoot...

last night I wanted a .mobi book from my library for CC testing. library PC was sleeping so I was using CC with my dropbox mirror
i found a book where metatdata says I have epub and mobi formats. but download option only offered me epub. i check the actual dropbox folder and sure enough, all i have is epub, opf, and cover.jpg

so this morning I ran calibre library check - which says all is good. no missing formats - and when I find that same book in calibre, the details pane says that I only have epub.
CC agrees, when connected to content server, but CC insists I also still have mobi when connected to dropbox mirror

I re-ran the file sync, I check that the sync is not set to excluded anything.
my library mirror has the same number of library files and folders as my library source.
5637 files in 2200 folders

but cc connected to cloud still says in book details that I also have the mobi format of this test book- WHY ?
it shows format: EPUB, MOBI
i guess it is reading available formats from the .db file so how come it is getting a different answer

update. i found another instance when CC with cloud metadata details says a book is available in AZW3 and EPUB, but calibre says only in EPUB, as does CC with content server

update: more tests. that was happening with a new install of a CC beta, after uninstalling prev. version
so I tried a different tablet which has the latest non-beta CC I found both those same discrepancies where CC with cloud sees 2 formats for those 2 test books, but only 1 really exists. and CC with content server gets it right.

i cannot go any further without knowing how CC populates the book details result after you connect, select a book and choose metadata from the metadata or download option

also I can't think of any way to see how many such "errors" exist across the library as CC will not count books by formats to give me stats for comparison

[stumped]

update.
another test.
i dug out an old tablet which is running cc 5.4.4.1 ( google). i.e. a version from the previous developers. could not test with dropbox as that old version had the dropbox auth bug, but i could test with box.

i perfectly match the above results i.e. have 2 books where CC with cloud shows a spurious extra format, but CC with content server correctly shows epub only.

test book 1. actual is epub only, cc with content server agrees, CC with cloud mirror says epub and mobi
test book 2 actual is epub only, cc with content server agrees CC with cloud mirror says epub and azw3
the cloud mirror ( via a manual folder inspection) definitely does not have the mobi and the azw files that CC claims to see in book details

[ those additional formats for those may have existed once up on a time, long time ago

so whatever is going on here is not caused by recent cc app updates

i get the same outcomes with 5.4.4.1 as I do from latest amazon beta

[stumped]

update 2. this does look like a CC issue but one that predates the handover to the current devlopers

as proof. I pointed my desktop calibre at the (local drive) x:/dropbox /calibre library [ fully synced]. so that is is using the same files which CC would use when connected to cloud
i had calibre find those 2 test books in that location, also did open path to verify it was looking in the local dropbox folder. calibre correctly says those 2 books exist as epub only in dropbox
yet when any version of CC looks at the same folder , but in the actual cloud, CC metadata reports that the test books have 2 formats available: epub and mobi in one case, epub and azw3 in the other case. in both cases, CC download option only shows epub.

[stumped]

update. i found that I did have a cusom column called format, and that CC was displaying it. that became clear when i renamed it from format to fmt and repeated all tests. that custom column fmt shows is a comma separated list of formats e.f. EPUB, TXT

that clears up most of my issue but I think ( for the devs) there are some inconsistencies between how CC displays book details from ( any) cloud compared to how it displays book details from content server.
i may have skipped a warning message about needing to connect once as wireless device to get up custom columns properly but even so, if CC is going to use custom column content it needs to assemble that correctly on the fly which i don't think it does in cloud connections. that call all go on the low priority f- look later list if no one else reports an issue.

for now I have told CC not to display that custom column in book details.

[chaley]

Originally Posted by stumped

i may have skipped a warning message about needing to connect once as wireless device to get up custom columns properly but even so, if CC is going to use custom column content it needs to assemble that correctly on the fly which i don't think it does in cloud connections.

Calibre does not by default store the value of composite columns in the database. That is what the Save Composite Custom Columns plugin is for.

[stumped]

thanks yes, i vaguely remembered that,
but lets assume i never added that plug in, cc [ with default display settings fro a new install] still tries to use composite column details in book details view when linked to cloud, and that's where the issue seems to lie.

one message did flash up, once, on tablet screen, about missing data in composite columns, but I could not get that message to re-appear

my single composite column, was called format: ( now called fmt had the template #formats.

i think that creates a text string ? of comma separated format names- back in the Calibre details pane, an entry like EPUB, MOBI is clickable but it is a single link , not a pair of links. cc was somehow connecting inaccurate strings to book retails but only in cloud connects. not in content server connects.

and in that respect old version 5.4.4.1 does the same as latest 5.4.1.16, that bit of coding has probably not been touched by the new guys

[chaley]

Originally Posted by stumped

thanks yes, i vaguely remembered that,
but lets assume i never added that plug in, cc [ with default display settings fro a new install] still tries to use composite column details in book details view when linked to cloud, and that's where the issue seems to lie.

Sorry but I can't assume that. The CC cloud connection that I released shows data for composite columns in book details only if that plugin has run sometime in the past. If it has never run then the fields are empty. The plugin stores values in the calibre db. Once they are in the db they are there until the plugin runs again. Thus if you installed the plugin then removed it, old data is left in calibre's db. There is no easy way to delete the data.

one message did flash up, once, on tablet screen, about missing data in composite columns, but I could not get that message to re-appear

You probably checked the box asking that the warning never be shown again.

[stumped]

ok- let me look again at my calibre & see if I can see that plug in. if it is there, that wraps up the explanation. if it was once installed i very much doubt I would have removed it...

i cant see it in user installed plug ins. maybe it did come and go. my calibre has moved to new pc a couple of times but I always copy over the config folder so that plus ins get preserved.
i don't have anything saying library closed plugins, in the plugin types list

the "old data" scenario does fit. those test books almost certainly did have additional formats at some point in the distant past

i seem to have destroyed the evidence now though. I messed with adding an additional txt format to the test books to see where/ if that showed in cc book details. and the ghost format is no longer appearign in the book details

[stumped]

With cc warnings reset, I see
"Failed to fetch some custom metadata from the cloud".
Each time I ask to see book metadata.

So my suggestion is that cc, after giving that warning, should not try to show what that failed fetch may have contained. Instead of displaying what is old and maybe out of date info.