Register Guidelines E-Books Search Today's Posts Mark Forums Read

Go Back   MobileRead Forums > E-Book Software > Calibre > Related Tools

Notices

Reply
 
Thread Tools Search this Thread
Old 11-07-2013, 06:23 PM   #16
itimpi
Wizard
itimpi ought to be getting tired of karma fortunes by now.itimpi ought to be getting tired of karma fortunes by now.itimpi ought to be getting tired of karma fortunes by now.itimpi ought to be getting tired of karma fortunes by now.itimpi ought to be getting tired of karma fortunes by now.itimpi ought to be getting tired of karma fortunes by now.itimpi ought to be getting tired of karma fortunes by now.itimpi ought to be getting tired of karma fortunes by now.itimpi ought to be getting tired of karma fortunes by now.itimpi ought to be getting tired of karma fortunes by now.itimpi ought to be getting tired of karma fortunes by now.
 
Posts: 4,038
Karma: 777825
Join Date: Nov 2008
Device: Sony PRS-950, iphone/ipad (Marvin/iBooks/QuickReader)
Quote:
Originally Posted by cyana View Post
Unfortunately, now all catalogs are generated from the scratch again, when you add just another book (Minimized number of changed files is checked) making it impossible to find out the files that really need to be updated in the cloud. In prior versions, at least some catalogs that stayed untouched were not regenerated thus saving their modified date so that they were left out when you synchronize. At least for me this is a big issue because my only choice is to delete the complete _catalog folder and re-upload it again.
This is certainly not the behaviour I am seeing. If anything less files get changed between runs now than used to be the case. Note however that the first time you run with a new release it always likely that all files WILL be generated due to other changes, but that should only apply to the first run. However I will check things out again to see if I can reproduce your behaviour.

Quote:
And one little bug: the resized book covers are generated in the original calibre folder, but not transferred to the location of the online library.
I will check this out. I never generate resized book covers (I use the options to NOT generate resized covers or thumbnails on my personal libraries) so it could easily have slipped through the net. However can you confirm exactly what revision you are using - there was one where these files only got transferred on the runs after they were first generated, but that should now be cleared.
itimpi is offline   Reply With Quote
Old 11-07-2013, 07:11 PM   #17
cyana
Enthusiast
cyana began at the beginning.
 
cyana's Avatar
 
Posts: 32
Karma: 10
Join Date: Dec 2011
Device: iPad
Hello itimpi,

thank you for your reply.
The version I'm using is 3.3 rev 245

I've generated the new library yesterday (6th Nov.) and uploaded it to the cloud. I added some books today, and ran through the generation as usual and found that all files (besides icons and css sheets) in the target location where overridden and had a new generation date (7th Nov.). Also, a subcatalog generated yesterday was renamed from eface222!... to b202e1b4!... for no obvious reason, and only the eface222!custom folder persisted in the _catalog directory. I did not change anything in the settings between the two runs. The subcatalog contains magazines, and I added 5 books and 5 magazines to the library to see what happens.

thank you very much
cyana

Last edited by cyana; 11-07-2013 at 07:14 PM.
cyana is offline   Reply With Quote
Old 11-08-2013, 04:37 AM   #18
itimpi
Wizard
itimpi ought to be getting tired of karma fortunes by now.itimpi ought to be getting tired of karma fortunes by now.itimpi ought to be getting tired of karma fortunes by now.itimpi ought to be getting tired of karma fortunes by now.itimpi ought to be getting tired of karma fortunes by now.itimpi ought to be getting tired of karma fortunes by now.itimpi ought to be getting tired of karma fortunes by now.itimpi ought to be getting tired of karma fortunes by now.itimpi ought to be getting tired of karma fortunes by now.itimpi ought to be getting tired of karma fortunes by now.itimpi ought to be getting tired of karma fortunes by now.
 
Posts: 4,038
Karma: 777825
Join Date: Nov 2008
Device: Sony PRS-950, iphone/ipad (Marvin/iBooks/QuickReader)
Quote:
Originally Posted by cyana View Post
And one little bug: the resized book covers are generated in the original calibre folder, but not transferred to the location of the online library.
Reproduced this problem. Now looking into why it is occurring.
itimpi is offline   Reply With Quote
Old 11-08-2013, 07:12 PM   #19
itimpi
Wizard
itimpi ought to be getting tired of karma fortunes by now.itimpi ought to be getting tired of karma fortunes by now.itimpi ought to be getting tired of karma fortunes by now.itimpi ought to be getting tired of karma fortunes by now.itimpi ought to be getting tired of karma fortunes by now.itimpi ought to be getting tired of karma fortunes by now.itimpi ought to be getting tired of karma fortunes by now.itimpi ought to be getting tired of karma fortunes by now.itimpi ought to be getting tired of karma fortunes by now.itimpi ought to be getting tired of karma fortunes by now.itimpi ought to be getting tired of karma fortunes by now.
 
Posts: 4,038
Karma: 777825
Join Date: Nov 2008
Device: Sony PRS-950, iphone/ipad (Marvin/iBooks/QuickReader)
Quote:
Originally Posted by cyana View Post
And one little bug: the resized book covers are generated in the original calibre folder, but not transferred to the location of the online library.
Now tracked down the cause of this. Interestingly it was a bug in the caching logic which had cached the fact that the file did not exist at the start of the run in the Calibre library, so did not think there was anything to copy. Next time round the file DID exist at the start of the run so the file got copied.

As far as I can see this file must have existed in both the 3.1 and 3.2 releases so it is a long-standing bug that it is good to get quashed.
itimpi is offline   Reply With Quote
Old 11-09-2013, 05:19 AM   #20
itimpi
Wizard
itimpi ought to be getting tired of karma fortunes by now.itimpi ought to be getting tired of karma fortunes by now.itimpi ought to be getting tired of karma fortunes by now.itimpi ought to be getting tired of karma fortunes by now.itimpi ought to be getting tired of karma fortunes by now.itimpi ought to be getting tired of karma fortunes by now.itimpi ought to be getting tired of karma fortunes by now.itimpi ought to be getting tired of karma fortunes by now.itimpi ought to be getting tired of karma fortunes by now.itimpi ought to be getting tired of karma fortunes by now.itimpi ought to be getting tired of karma fortunes by now.
 
Posts: 4,038
Karma: 777825
Join Date: Nov 2008
Device: Sony PRS-950, iphone/ipad (Marvin/iBooks/QuickReader)
Quote:
Originally Posted by cyana View Post
Hello itimpi,

thank you for your reply.
The version I'm using is 3.3 rev 245

I've generated the new library yesterday (6th Nov.) and uploaded it to the cloud. I added some books today, and ran through the generation as usual and found that all files (besides icons and css sheets) in the target location where overridden and had a new generation date (7th Nov.). Also, a subcatalog generated yesterday was renamed from eface222!... to b202e1b4!... for no obvious reason, and only the eface222!custom folder persisted in the _catalog directory. I did not change anything in the settings between the two runs. The subcatalog contains magazines, and I added 5 books and 5 magazines to the library to see what happens.
I still have not managed to recreate this!

Is there any chance that you could post the synclog.log file that is stored in the Calibre2opds logging folder? That might give an indication of what is happening. Note that this file is over-written each generation run so whatever you send will be from the last run that you did. I would suggest that you ZIP the file before posting which will keep the size down.
itimpi is offline   Reply With Quote
Old 11-09-2013, 02:39 PM   #21
cyana
Enthusiast
cyana began at the beginning.
 
cyana's Avatar
 
Posts: 32
Karma: 10
Join Date: Dec 2011
Device: iPad
Thanks itimpi!

actually, it appeared new to me in 3.3, however I had no real opportunity to test 3.2 extensively.

Will try another test today of the regenerating issue.

best greetings
cyana
cyana is offline   Reply With Quote
Old 11-09-2013, 02:58 PM   #22
cyana
Enthusiast
cyana began at the beginning.
 
cyana's Avatar
 
Posts: 32
Karma: 10
Join Date: Dec 2011
Device: iPad
Seems our messages just crossed
I've added the synclog from the last run - that was the run that seemed to overwrite everything again within the _catalog folder. Actually, the contents of all folders like book_0 were replaced, while the folders itself stayed the same, als you can see here:


thanks
cyana
Attached Files
File Type: zip synclog.zip (50.8 KB, 20 views)
cyana is offline   Reply With Quote
Old 11-09-2013, 04:06 PM   #23
itimpi
Wizard
itimpi ought to be getting tired of karma fortunes by now.itimpi ought to be getting tired of karma fortunes by now.itimpi ought to be getting tired of karma fortunes by now.itimpi ought to be getting tired of karma fortunes by now.itimpi ought to be getting tired of karma fortunes by now.itimpi ought to be getting tired of karma fortunes by now.itimpi ought to be getting tired of karma fortunes by now.itimpi ought to be getting tired of karma fortunes by now.itimpi ought to be getting tired of karma fortunes by now.itimpi ought to be getting tired of karma fortunes by now.itimpi ought to be getting tired of karma fortunes by now.
 
Posts: 4,038
Karma: 777825
Join Date: Nov 2008
Device: Sony PRS-950, iphone/ipad (Marvin/iBooks/QuickReader)
The synclog file is of interest. It shows that many files were left unchanged, but that the files in the 'books' type folders (the book pages) were all changed due to their length changing.

By looking at just those files between runs I was able to identify the problem. There is a checkbox on the Book Details tab about whether tags should be included in cross-references. This value has a fault in the load/save logic for this setting that means the setting gets inverted each run (and thus the book files DO change). That is a relatively new checkbox which would be why this was not showing up previously.

A temporary workaround would be to disable the generation of cross-references. An alternative will be to make sure you set the faulty checkbox to the same value just before doing each generate run. I will shortly make a build available that fixes the problem with loading/saving this setting so the workaround will then no longer be necessary.

Thanks for helping with this.

Last edited by itimpi; 11-09-2013 at 04:29 PM.
itimpi is offline   Reply With Quote
Old 11-09-2013, 04:29 PM   #24
cyana
Enthusiast
cyana began at the beginning.
 
cyana's Avatar
 
Posts: 32
Karma: 10
Join Date: Dec 2011
Device: iPad
I've now looked at the log more closely, in fact it includes files that are not generated at all, e.g.:

NOT COPIED (CRC same): author_434.html

There is no such page, but only author_434_Page_1.html, however this and all other generated author pages like author_xxx_yyy_Page_1 or author_xxx_allbooks_Page_1 don't show up in the log at all.

HTH
thanks for your efforts!
cyana
cyana is offline   Reply With Quote
Old 11-10-2013, 12:13 PM   #25
itimpi
Wizard
itimpi ought to be getting tired of karma fortunes by now.itimpi ought to be getting tired of karma fortunes by now.itimpi ought to be getting tired of karma fortunes by now.itimpi ought to be getting tired of karma fortunes by now.itimpi ought to be getting tired of karma fortunes by now.itimpi ought to be getting tired of karma fortunes by now.itimpi ought to be getting tired of karma fortunes by now.itimpi ought to be getting tired of karma fortunes by now.itimpi ought to be getting tired of karma fortunes by now.itimpi ought to be getting tired of karma fortunes by now.itimpi ought to be getting tired of karma fortunes by now.
 
Posts: 4,038
Karma: 777825
Join Date: Nov 2008
Device: Sony PRS-950, iphone/ipad (Marvin/iBooks/QuickReader)
Quote:
Originally Posted by cyana View Post
I've now looked at the log more closely, in fact it includes files that are not generated at all, e.g.:

NOT COPIED (CRC same): author_434.html

There is no such page, but only author_434_Page_1.html, however this and all other generated author pages like author_xxx_yyy_Page_1 or author_xxx_allbooks_Page_1 don't show up in the log at all.
Strange that - not sure how it could happen. Having said that since that is a log that is only ever used for debugging purposes it is always possible that there is a fault in the logging code as it is not seen as a high priority when testing. It might be interesting to delete the calibre2opds.cache file before running the next generation run in case it is a relic of an earlier naming scheme that has not been cleaned out of the cache for some reason.

Anyway, I have now uploaded rev 248 which hopefully fixes your issue with all the book details files being regenerated every time (as well a number of other small issues). Please let me know if it now behaves as expected for you!
itimpi is offline   Reply With Quote
Old 11-11-2013, 11:44 AM   #26
cyana
Enthusiast
cyana began at the beginning.
 
cyana's Avatar
 
Posts: 32
Karma: 10
Join Date: Dec 2011
Device: iPad
Hello itimpi,

thanks a lot for the new build! I did a run and the number of newly generated files dropped significantly. Around 500 author pages still were regenerated without being affected by added books, the log said that the length changed, so maybe this might have been due to the new version. I added another book to the library and did another run, and this time it worked perfectly!

great work, thank you very much!
cyana
cyana is offline   Reply With Quote
Old 11-11-2013, 12:19 PM   #27
itimpi
Wizard
itimpi ought to be getting tired of karma fortunes by now.itimpi ought to be getting tired of karma fortunes by now.itimpi ought to be getting tired of karma fortunes by now.itimpi ought to be getting tired of karma fortunes by now.itimpi ought to be getting tired of karma fortunes by now.itimpi ought to be getting tired of karma fortunes by now.itimpi ought to be getting tired of karma fortunes by now.itimpi ought to be getting tired of karma fortunes by now.itimpi ought to be getting tired of karma fortunes by now.itimpi ought to be getting tired of karma fortunes by now.itimpi ought to be getting tired of karma fortunes by now.
 
Posts: 4,038
Karma: 777825
Join Date: Nov 2008
Device: Sony PRS-950, iphone/ipad (Marvin/iBooks/QuickReader)
Glad to hear that the change seems to have had the desired effect. It is always nice to have confirmation that the change works in real-world scenarios rather than just the relatively artificial test scenarios.

If you spot any other anomalies, then please point them out.
itimpi is offline   Reply With Quote
Old 11-11-2013, 09:30 PM   #28
cyana
Enthusiast
cyana began at the beginning.
 
cyana's Avatar
 
Posts: 32
Karma: 10
Join Date: Dec 2011
Device: iPad
There is indeed another anomality (Cassanda is my middlename *ggg*)
I've spotted cases, where the book's description shows up in the catalogs, but not in the book page. This appears to happen, when the description is more complex html code rather than plain text:

As you see, the description shows up in the catalogs as plain text, however is totally omitted in the book's page. So it's more out of my curiosity to ask how this could happen?

best greetings
cyana
cyana is offline   Reply With Quote
Old 11-12-2013, 12:31 AM   #29
itimpi
Wizard
itimpi ought to be getting tired of karma fortunes by now.itimpi ought to be getting tired of karma fortunes by now.itimpi ought to be getting tired of karma fortunes by now.itimpi ought to be getting tired of karma fortunes by now.itimpi ought to be getting tired of karma fortunes by now.itimpi ought to be getting tired of karma fortunes by now.itimpi ought to be getting tired of karma fortunes by now.itimpi ought to be getting tired of karma fortunes by now.itimpi ought to be getting tired of karma fortunes by now.itimpi ought to be getting tired of karma fortunes by now.itimpi ought to be getting tired of karma fortunes by now.
 
Posts: 4,038
Karma: 777825
Join Date: Nov 2008
Device: Sony PRS-950, iphone/ipad (Marvin/iBooks/QuickReader)
To be honest no definitive idea!

It might be worth looking at the calibre2opds log file to see if a warning was displayed about the structure of the HTML for that comment. In the book details pages the comment has to be valid HTML, and a warning is logged if the check for this fails. Conversely in the book listing HTML tags have been removed to leave plain text so they can survive malformed HTML comments.

I have been thinking of changing the completion message when there are warnings logged to suggest the user might want to look at the log file. If this turns out to be a case where doing so would have helped I might need to consider moving such a change higher in the priority list!

If you want me to look any further I would need a copy of the metadata.db file from the library containing the book exhibiting this issue.

Last edited by itimpi; 11-12-2013 at 01:04 AM.
itimpi is offline   Reply With Quote
Old 11-12-2013, 05:53 PM   #30
cyana
Enthusiast
cyana began at the beginning.
 
cyana's Avatar
 
Posts: 32
Karma: 10
Join Date: Dec 2011
Device: iPad
Hello itimpi,

thanks for your reply! - I've enclosed the metadatas from an affected book

thank you
cyana
Attached Files
File Type: opf metadata.opf (34.8 KB, 28 views)
cyana is offline   Reply With Quote
Reply

Thread Tools Search this Thread
Search this Thread:

Advanced Search

Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
Calibre2opds v3.2 - OPDS catalogs from your Calibre library itimpi Related Tools 41 10-20-2013 05:01 PM
Calibre2opds v3 - OPDS catalogs from your Calibre library itimpi Related Tools 31 02-10-2013 04:16 PM
Popular public OPDS catalogs kguil Apple Devices 37 01-01-2013 05:00 PM
OPDS catalogs access knfevg PocketBook 7 12-04-2012 04:55 AM
List of Free Online OPDS Catalogs? Madmick General Discussions 1 01-14-2012 12:09 AM


All times are GMT -4. The time now is 07:01 PM.


MobileRead.com is a privately owned, operated and funded community.