Calibre2OPDS - why so many file uploads??

[DMee]

I have been keeping my books and C2O on Dropbox and all working brilliantly.

HOwever I am now in the process of tidying up the metadata which, of course needs me to then regenerate ned C2O files.

I wanted to access these from a second computer and rather than wait for dropbox to sync I decided to copy the dropbox folder to USB drive and copy from that to second (remote location).

Over the weekend I tweaked a few more books and so regenerated C2O again - now I have it set to minimise file changes and when I resync'd data to USB drive it found 175 files had changed - fine, OK

Then I noticed poor old drobox had been working full tilt for 2 days and STILL had 26,000 file to upload???? Huh????

Surely just the 175 should have needed reuploading to DB?

Anyone else seen this - anyone know why??

[itimpi]

Not sure why you were getting so many files. It sounds as though you did something that made DropBox think all the files had changed. The stats in the calibre2opds log will tell you exactly how many files were changed in that particular run - and that is the number that DropBox should sync as a result of that run. In fact the Minimize Changes options was added specifically to help DropBox users and it certainly seems to work fine for me. I wonder if you did a run WITHOUT the Minimize Changes option set - that could have the sort of effect you describe. With the 2.4 release onwards there is also the synclog file generated in the calibre2opds log folder which should give you an exact list of the files that were copied (or not copied) to the new catalog (and thus need syncing). Looking at that might give an indication of what is happening.

If you are using the 2.5 beta 1 release then you may want to also use the feature that has been added to reuse the calibre cover.jpg files for thumbnails and cover images rather than generating new images optimised for that purpose. This means there are less files stored in the library thus saving some space (although there is more network trafic when browsing a library as the cover.jpg files are normally larger than the optimised images that calibre2opds would otherwise generate).

[DMee]

I have Minimise number changed ticked and Do not save Bandwidth unchecked. I do have reprocess ePub data ticked but would have thought this would only affect changed books??

I don't mind about the cover jpgs as loads of room on dropbox and prefer quicker load when browsing

Here's part of the log

30-08-2011 19:24:44,406 [] INFO Runner - The options are taken from the configuration file located at C:\Users\David\.calibre2opds\Lists.profile.xml
30-08-2011 19:24:44,407 [] INFO Runner - 
30-08-2011 19:25:46,954 [] INFO com.gmail.dpierron.calibre.cache.CachedFileManager - STARTED Loading CRC cache from file D:\My Dropbox\Public\####deleted####\_catalog_list\calibre2opds.cache
30-08-2011 19:25:54,884 [] INFO com.gmail.dpierron.calibre.cache.CachedFileManager - Cache Entries Loaded: 168441
30-08-2011 19:25:54,884 [] INFO com.gmail.dpierron.calibre.cache.CachedFileManager - COMPLETED Loading CRC cache from file D:\My Dropbox\Public\####deleted####\_catalog_list\calibre2opds.cache
30-08-2011 19:25:55,357 [] INFO com.gmail.dpierron.calibre.opds.Catalog - 
30-08-2011 19:25:55,358 [] INFO com.gmail.dpierron.calibre.opds.Catalog - Language                                           : en
30-08-2011 19:25:55,359 [] INFO com.gmail.dpierron.calibre.opds.Catalog - Device-specific mode                               : Dropbox
30-08-2011 19:25:55,359 [] INFO com.gmail.dpierron.calibre.opds.Catalog - Destination folder                                 : null
30-08-2011 19:25:55,360 [] INFO com.gmail.dpierron.calibre.opds.Catalog - Wikipedia language                                 : en
30-08-2011 19:25:55,360 [] INFO com.gmail.dpierron.calibre.opds.Catalog - Catalog folder                                     : _catalog_list
30-08-2011 19:25:55,360 [] INFO com.gmail.dpierron.calibre.opds.Catalog - Catalog title                                      : Calibre library - lists
30-08-2011 19:25:55,360 [] INFO com.gmail.dpierron.calibre.opds.Catalog - Number of elements before pagination               : 250
30-08-2011 19:25:55,361 [] INFO com.gmail.dpierron.calibre.opds.Catalog - Number of elements before split                    : 0
30-08-2011 19:25:55,361 [] INFO com.gmail.dpierron.calibre.opds.Catalog - Maximum screen width for mobile option (HTML)      : 960
30-08-2011 19:25:55,361 [] INFO com.gmail.dpierron.calibre.opds.Catalog - Number of books in the "recent" catalog            : 99999
30-08-2011 19:25:55,361 [] INFO com.gmail.dpierron.calibre.opds.Catalog - Included formats                                   : EPUB, PDF, RTF, TXT, PRC, PDB, MOBI, LRF, LRX, FB2
30-08-2011 19:25:55,362 [] INFO com.gmail.dpierron.calibre.opds.Catalog - Minimize Number of Changed Files                   : yes
30-08-2011 19:25:55,363 [] INFO com.gmail.dpierron.calibre.opds.Catalog - Do not save bandwidth                              : no
30-08-2011 19:25:55,363 [] INFO com.gmail.dpierron.calibre.opds.Catalog - Do not generate OPDS catalogs                      : no
30-08-2011 19:25:55,363 [] INFO com.gmail.dpierron.calibre.opds.Catalog - Do not generate HTML catalogs                      : no
30-08-2011 19:25:55,364 [] INFO com.gmail.dpierron.calibre.opds.Catalog - Do not generate OPDS downloads                     : no
30-08-2011 19:25:55,365 [] INFO com.gmail.dpierron.calibre.opds.Catalog - Do not generate HTML downloads                     : no
30-08-2011 19:25:55,365 [] INFO com.gmail.dpierron.calibre.opds.Catalog - Published date as year in book details             : no
30-08-2011 19:25:55,365 [] INFO com.gmail.dpierron.calibre.opds.Catalog - Suppress ratings in the books titles               : yes
30-08-2011 19:25:55,366 [] INFO com.gmail.dpierron.calibre.opds.Catalog - Thumbnail height                                   : 144
30-08-2011 19:25:55,366 [] INFO com.gmail.dpierron.calibre.opds.Catalog - Split tags using                                   : 
30-08-2011 19:25:55,366 [] INFO com.gmail.dpierron.calibre.opds.Catalog - Include books without file                         : no
30-08-2011 19:25:55,366 [] INFO com.gmail.dpierron.calibre.opds.Catalog - Tag that will specify a book language              : Lang:
30-08-2011 19:25:55,367 [] INFO com.gmail.dpierron.calibre.opds.Catalog - Encrypt the filenames                              : no
30-08-2011 19:25:55,367 [] INFO com.gmail.dpierron.calibre.opds.Catalog - Do not show the series in the author catalogs      : no
30-08-2011 19:25:55,367 [] INFO com.gmail.dpierron.calibre.opds.Catalog - Do not generate external links                     : no
30-08-2011 19:25:55,367 [] INFO com.gmail.dpierron.calibre.opds.Catalog - Do not generate cross-reference links              : no
30-08-2011 19:25:55,368 [] INFO com.gmail.dpierron.calibre.opds.Catalog - List of tags that will generate a catalog          : 
30-08-2011 19:25:55,368 [] INFO com.gmail.dpierron.calibre.opds.Catalog - List of tags that will be excluded from the catalog : 
30-08-2011 19:25:55,368 [] INFO com.gmail.dpierron.calibre.opds.Catalog - Maximum length of the summary list                 : 1
30-08-2011 19:25:55,368 [] INFO com.gmail.dpierron.calibre.opds.Catalog - Do not generate the "Ratings" catalog              : yes
30-08-2011 19:25:55,369 [] INFO com.gmail.dpierron.calibre.opds.Catalog - Do not generate the "All books" catalog            : yes
30-08-2011 19:25:55,369 [] INFO com.gmail.dpierron.calibre.opds.Catalog - Copy catalog to database folder                    : yes
30-08-2011 19:25:55,369 [] INFO com.gmail.dpierron.calibre.opds.Catalog - "Browse by Cover" mode (HTML)                      : no
30-08-2011 19:25:55,370 [] INFO com.gmail.dpierron.calibre.opds.Catalog - In the authors list page, skip the list of authors with the same initial : no
30-08-2011 19:25:55,370 [] INFO com.gmail.dpierron.calibre.opds.Catalog - Include the "About calibre2opds" entry             : no
30-08-2011 19:25:55,370 [] INFO com.gmail.dpierron.calibre.opds.Catalog - List of tags that will get an additional level     : science*
30-08-2011 19:25:55,371 [] INFO com.gmail.dpierron.calibre.opds.Catalog - Minimum books to create an additional level        : 20
30-08-2011 19:25:55,371 [] INFO com.gmail.dpierron.calibre.opds.Catalog - Do not split by letter in "Browse by cover" mode   : no
30-08-2011 19:25:55,371 [] INFO com.gmail.dpierron.calibre.opds.Catalog - Cover height                                       : 550
30-08-2011 19:25:55,371 [] INFO com.gmail.dpierron.calibre.opds.Catalog - Include only one ebook file                        : no
30-08-2011 19:25:55,372 [] INFO com.gmail.dpierron.calibre.opds.Catalog - Zip the catalog before copying it to the Nook      : no
30-08-2011 19:25:55,372 [] INFO com.gmail.dpierron.calibre.opds.Catalog - Reprocess the epub metadata                        : yes
30-08-2011 19:25:55,372 [] INFO com.gmail.dpierron.calibre.opds.Catalog - Compatibility level                                : STANZA
30-08-2011 19:25:55,373 [] INFO com.gmail.dpierron.calibre.opds.Catalog - Do not split book titles within an Author          : yes
30-08-2011 19:25:55,373 [] INFO com.gmail.dpierron.calibre.opds.Catalog - Do not split book titles within a Series           : yes
30-08-2011 19:25:55,373 [] INFO com.gmail.dpierron.calibre.opds.Catalog - Database folder                                    : D:\My Dropbox\Public\####deleted#####
30-08-2011 19:25:55,373 [] INFO com.gmail.dpierron.calibre.opds.Catalog - 
30-08-2011 19:25:55,374 [] INFO com.gmail.dpierron.calibre.opds.Catalog - loading and filtering database
30-08-2011 19:26:18,602 [] INFO com.gmail.dpierron.calibre.opds.Catalog - done in 23,169 millisec.
30-08-2011 19:26:18,780 [] INFO com.gmail.dpierron.calibre.opds.Catalog - generating authors catalog
30-08-2011 20:09:07,305 [] INFO com.gmail.dpierron.calibre.opds.Catalog - done in 2,568,524 millisec.
30-08-2011 20:09:07,305 [] INFO com.gmail.dpierron.calibre.opds.Catalog - generating tags catalog
30-08-2011 22:39:15,801 [] INFO com.gmail.dpierron.calibre.opds.Catalog - done in 9,008,496 millisec.
30-08-2011 22:39:15,801 [] INFO com.gmail.dpierron.calibre.opds.Catalog - generating series catalog
30-08-2011 22:39:53,168 [] INFO com.gmail.dpierron.calibre.opds.Catalog - done in 37,367 millisec.
30-08-2011 22:39:53,168 [] INFO com.gmail.dpierron.calibre.opds.Catalog - generating recent books catalog
30-08-2011 22:46:15,405 [] INFO com.gmail.dpierron.calibre.opds.Catalog - done in 382,237 millisec.
30-08-2011 22:46:15,405 [] INFO com.gmail.dpierron.calibre.opds.Catalog - generating books rating catalog
30-08-2011 22:46:15,405 [] INFO com.gmail.dpierron.calibre.opds.Catalog - done in 0 millisec.
30-08-2011 22:46:15,405 [] INFO com.gmail.dpierron.calibre.opds.Catalog - generating all books catalog
30-08-2011 22:46:15,405 [] INFO com.gmail.dpierron.calibre.opds.Catalog - done in 0 millisec.
30-08-2011 22:46:15,405 [] INFO com.gmail.dpierron.calibre.opds.Catalog - generating the thumbnail files
30-08-2011 22:46:21,609 [] INFO com.gmail.dpierron.calibre.opds.Catalog - done in 6,204 millisec.
30-08-2011 22:46:21,609 [] INFO com.gmail.dpierron.calibre.opds.Catalog - generating the cover files
30-08-2011 22:46:25,469 [] INFO com.gmail.dpierron.calibre.opds.Catalog - done in 3,860 millisec.
30-08-2011 22:46:25,469 [] INFO com.gmail.dpierron.calibre.opds.Catalog - reprocessing the ePub metadata
30-08-2011 22:50:06,921 [] INFO com.gmail.dpierron.calibre.opds.Catalog - done in 221,452 millisec.
30-08-2011 22:50:07,904 [] INFO com.gmail.dpierron.calibre.opds.Catalog - copy the library
30-08-2011 22:50:07,904 [] INFO com.gmail.dpierron.calibre.opds.Catalog - done in 222,435 millisec.
30-08-2011 23:00:56,129 [] INFO com.gmail.dpierron.calibre.opds.Catalog - copy the catalog
31-08-2011 03:34:06,363 [] INFO com.gmail.dpierron.calibre.opds.Catalog - 
31-08-2011 03:34:06,630 [] INFO com.gmail.dpierron.calibre.opds.Catalog - File Copying Statistics
31-08-2011 03:34:07,378 [] INFO com.gmail.dpierron.calibre.opds.Catalog -     4710  YES: Target does not exist
31-08-2011 03:34:07,378 [] INFO com.gmail.dpierron.calibre.opds.Catalog -    13163  YES: Length different
31-08-2011 03:34:07,378 [] INFO com.gmail.dpierron.calibre.opds.Catalog -        0  YES: CRC not checked
31-08-2011 03:34:07,378 [] INFO com.gmail.dpierron.calibre.opds.Catalog -    15129  YES: CRC different
31-08-2011 03:34:07,378 [] INFO com.gmail.dpierron.calibre.opds.Catalog -   312411  NO:  CRC same
31-08-2011 03:34:07,378 [] INFO com.gmail.dpierron.calibre.opds.Catalog -        0  NO:  Source older

[itimpi]

Looking at that log, about 30,000 files ended up being copied.

I am guessing that the epub reprocessing did result in many files being changed looking at the elapsed time that it took. In the 2.5 beta 1 version the statistics written to the log have been extended and would have told you how many files were re-processed so you could be more certain. Note that files get re-processed if the .opf file in the book's folder is later than the epub file. Did you perhaps tell Calibre to do a backup as that would update all the .opf files?

[DMee]

Weird. I only updated 80 or so book details. Dunno why so many had ePub data redone??

Turned off reprocess the next time and just reconverted those I had changed ePub to ePub. Still resulted in about 6000 files getting reuploaded to Dropbox?? When I checked it was all the authors, books, tags etc etc needing reupdating too :-(

[itimpi]

The trouble is that when you have a big library then adding even a single book can result in quite a few files being changed as the contents of the multi-page catalog sections get shuffled up. I cannot see any way to avoid that I am afraid. You can suppress the generation of any catalog sections that you do not want to try and help reduce the number.