calibre syncs ENTIRE device every time

[cinco]

I recently made my first post to this forum (about sorting by date within collections on my Sony PRS-300) and was gratified by the responses. I do believe it is possible to sort my collections by date now.

However, here is the ancillary problem, which I have been experiencing for at least a month with all the latest versions of calibre on my PRS-300: EVERY time I sync my Sony Reader, calibre syncs ALL of my documents. (As I mentioned in my previous post, I use my Reader to view PDF files for work.)

So, for example, I have 330 PDF files in calibre right now, and even if I add just one document, it syncs all 330 files. This takes a long time, and it also shows that ALL the files are dated "today." Apart from taking forever to sync my device, it makes it impossible to sort by date, since they all appear to be new documents.

Any ideas why calibre is syncing my entire device every time, even if I am just adding one or two documents? As I said, this has been happening for quite a while, across a number of recent versions of calibre.

Thanks again for everyone's help!

[Manichean]

You would have to be a little clearer about what you mean by "syncing". As far as I know, Calibre has no code to do anything remotely like syncing anything, the only thing I can think of is metadata management, which would update the metadata on your device, though that should only happen if you changed the metadata in the library in the first place, I believe.

[chaley]

Calibre does not sync books. It does sync metadata if metadata management is set to automatic, but that process will not change the file date of the books (or at least, I know of no way that it does). And in any event, the process is identical whether or not a book is added.

I suspect that in this case, the 'sync' happens when the device is ejected. At that point, the reader will scan every book in memory, comparing information from its database to the information in the books. This is a mysterious process, out of control of calibre.

The thing to do now is run calibre in debug mode to see what, if anything, calibre is doing. If you are using windows, then open a command box (start, run, type cmd, press OK). Note the current folder; it will be listed in the prompt. With your 300 not connected, type

calibre-debug -g > caldebug.txt 2>&1

Calibre will start. Plug in your 300 and send 1 book. As soon as everything finishes, quit calibre. You will then get another prompt in the cmd box.

Open a file explorer and go to the folder you noted earlier. There should be a file caldebug.txt in that folder. Look at it to be sure that there isn't any private information in it (names of collections, book title, etc). If not, then post the log here.

[cinco]

Thanks. Here is my log of caldebug.txt:

Microsoft Windows [Version 6.1.7600]
Copyright (c) 2009 Microsoft Corporation. All rights reserved.

C:\Users\CC>calibre-debug -g > caldebug.txt 2>&1

C:\Users\CC>type caldebug.txt
Starting up...
Started up in 1.14700007439
Job: 1 Read metadata (0 of 1) finished
Read metadata (0 of 1)

Added test document to db in: 0.84299993515 seconds
Job: 2 Get device information finished
No details available.
DEBUG: 0.0 PRS505: starting fetching books for card None
DEBUG: 0.0 USBMS: Fetching list of books from device. oncard= None
DEBUG: 0.0 USBMS: dirs are: E:\ database/media/books
DEBUG: 2.2 USBMS: count found in cache: 330, count of files in metadata: 330,
need_sync: False
DEBUG: 2.2 USBMS: Finished fetching list of books from device. oncard= None
DEBUG: 2.2 Building XMLCache... {0: 'E:\\database\\cache\\media.xml'}
DEBUG: 3.7 Done building XMLCache...
DEBUG: 3.7 Updating JSON cache: 0
DEBUG: 3.7 Start build_id_playlist_map
DEBUG: 3.7 after cleaning playlists
DEBUG: 3.7 Finish build_id_playlist_map. Found 329
DEBUG: 3.7 Finished updating JSON cache: 0
DEBUG: 3.7 PRS505: finished fetching books for card None
DEBUG: 3.7 PRS505: starting fetching books for card carda
DEBUG: 3.8 USBMS: Fetching list of books from device. oncard= carda
DEBUG: 3.8 Building XMLCache... {0: 'E:\\database\\cache\\media.xml'}
DEBUG: 4.0 Done building XMLCache...
DEBUG: 4.0 PRS505: finished fetching books for card carda
DEBUG: 4.0 PRS505: starting fetching books for card cardb
DEBUG: 4.0 USBMS: Fetching list of books from device. oncard= cardb
DEBUG: 4.0 Building XMLCache... {0: 'E:\\database\\cache\\media.xml'}
DEBUG: 4.2 Done building XMLCache...
DEBUG: 4.2 PRS505: finished fetching books for card cardb
Job: 3 Get list of books on device finished
No details available.
DEBUG: 6.3 PRS505: started sync_booklists
DEBUG: 6.3 Building XMLCache... {0: 'E:\\database\\cache\\media.xml'}
DEBUG: 6.5 Done building XMLCache...
DEBUG: 6.5 PRS505: collection fields: ['series', 'tags']
Device using plugboard device_db PRS505 None
DEBUG: 6.5 PRS505: use plugboards None
DEBUG: 6.5 Starting update ['series', 'tags']
DEBUG: 6.5 Start build_id_playlist_map
DEBUG: 6.6 after cleaning playlists
DEBUG: 6.6 Finish build_id_playlist_map. Found 329
DEBUG: 6.6 Updating XML Cache: 0
DEBUG: 7.0 Timezone votes: 0 GMT, 330 LTZ, use_tz_var=True
DEBUG: 7.0 Starting update_playlists ['series', 'tags'] 0
DEBUG: 7.0 Starting get_collections: on_connect
DEBUG: 7.0 Renaming rules: {}
DEBUG: 7.0 Sorting rules: [(['pubdate', 'tags'], '#mydate'), (['series'], 'se
ries_index'), (['*'], 'title')]
DEBUG: 7.0 update_playlists: finished building maps
DEBUG: 7.1 Finishing update_playlists
DEBUG: 7.1 In update/ Starting refresh of device_collections
DEBUG: 7.1 Start build_id_playlist_map
DEBUG: 7.1 after cleaning playlists
DEBUG: 7.1 Finish build_id_playlist_map. Found 329
DEBUG: 7.1 Running fix_ids()
DEBUG: 7.2 Finished running fix_ids()
DEBUG: 7.2 Finished update
DEBUG: 10.1 USBMS: starting sync_booklists
DEBUG: 13.6 USBMS: finished sync_booklists
DEBUG: 13.6 PRS505: finished sync_booklists
Job: 4 Send metadata to device finished
No details available.
Device using plugboard pdf PRS505 None
Setting metadata in: test document at: c:\users\cc\appdata\local\temp\calibre
_0.7.33_tmp_dmhyv9\calibre_0.7.33_er_zgv.pdf
Job: 6 Write pdf metadata finished
Write pdf metadata
<</Info 13 0 R/Root 12 0 R/Size 14>>

Job: 6 Write pdf metadata finished
Write pdf metadata
<</Info 13 0 R/Root 12 0 R/Size 14>>

DEBUG: 20.9 USBMS: uploading 1 books
DEBUG: 21.0 USBMS: finished uploading 1 books
Job: 5 Upload 1 books to device:test document finished
No details available.
DEBUG: 21.9 USBMS: adding metadata for 1 books
DEBUG: 21.9 USBMS: finished adding metadata
DEBUG: 23.0 PRS505: started sync_booklists
DEBUG: 23.3 Building XMLCache... {0: 'E:\\database\\cache\\media.xml'}
DEBUG: 23.5 Done building XMLCache...
DEBUG: 23.5 PRS505: collection fields: ['series', 'tags']
Device using plugboard device_db PRS505 None
DEBUG: 23.5 PRS505: use plugboards None
DEBUG: 23.5 Starting update ['series', 'tags']
DEBUG: 23.5 Start build_id_playlist_map
DEBUG: 23.5 after cleaning playlists
DEBUG: 23.5 Finish build_id_playlist_map. Found 329
DEBUG: 23.5 Updating XML Cache: 0
Job: 5 Upload 1 books to device:test document finished
No details available.
DEBUG: 24.0 Use localtime TZ and tz='0' for new book database/media/books/Unkn
own/test document - Unknown_748.pdf
DEBUG: 24.0 Timezone votes: 0 GMT, 330 LTZ, use_tz_var=True
DEBUG: 24.0 Starting update_playlists ['series', 'tags'] 0
DEBUG: 24.0 Starting get_collections: on_connect
DEBUG: 24.0 Renaming rules: {}
DEBUG: 24.0 Sorting rules: [(['pubdate', 'tags'], '#mydate'), (['series'], 'se
ries_index'), (['*'], 'title')]
DEBUG: 24.0 update_playlists: finished building maps
DEBUG: 24.0 Creating playlist: Test
DEBUG: 24.1 Finishing update_playlists
DEBUG: 24.1 In update/ Starting refresh of device_collections
DEBUG: 24.1 Start build_id_playlist_map
DEBUG: 24.1 after cleaning playlists
DEBUG: 24.1 Finish build_id_playlist_map. Found 330
DEBUG: 24.1 Running fix_ids()
DEBUG: 24.2 Finished running fix_ids()
DEBUG: 24.2 Finished update
DEBUG: 25.7 USBMS: starting sync_booklists
DEBUG: 31.6 USBMS: finished sync_booklists
DEBUG: 31.6 PRS505: finished sync_booklists
Job: 7 Send metadata to device finished
No details available.
get_metadata cache: 673 calls, 99.11% misses

C:\Users\CC>

[chaley]

OK, what that says is that it took 6 second to synchronize metadata. Later it took 6 seconds to send the book, then 6 seconds again to sync metadata. The entire process (connect and send) finished in 32 seconds. That seems reasonable to me.

Where is the 'long time' you mentioned? Is it after disconnecting the device (the little arrows are spinning)? It doesn't appear to be while calibre is running.

Edit: the log also shows that calibre thinks all the books have not changed. The 'need_sync=False' in the first two seconds says that all the files agree with calibre's stored information about the file. In this case, calibre does not look further at the books' metadata.

Edit 2: there have been reports that using the PDF metadata writer (which you are) creates files that bother the Sony. It could be that the Sony scan process is choking on the books and changing its metadata DB. You could try disabling the PDF metadata writer plugin (preferences -> advanced -> plugins -> writers)

[cinco]

Thanks. Yes, you're right: the "long time" is after I disconnect the device and the little arrows spin for 10-20 MINUTES! So, I disabled the PDF metadata writer as you suggested (and restarted calibre), but the arrows spun again for about 15 minutes after disconnecting my Reader.

What I am wondering is why ALL the files are dated "today" after each time I disconnect and wait for the arrows to finish spinning.

Thanks again for helping to sleuth this out.

[Perkin]

I don't know if this is related, but maybe someone can add details, but wasn't there a problem with Sony's, when you had epub's and pdf's as well?
(I've got a 300 as well and try and steer clear of pdf's)

[chaley]

Originally Posted by cinco

Thanks. Yes, you're right: the "long time" is after I disconnect the device and the little arrows spin for 10-20 MINUTES!

Does it do this if you connect and disconnect the device without calibre running? If you connect and disconnect the device with calibre running? If the former, then the 300 is deciding all by itself to take the time. If only when calibre is running, then we have a here-to-fore unknown interaction.

So, I disabled the PDF metadata writer as you suggested (and restarted calibre), but the arrows spun again for about 15 minutes after disconnecting my Reader.

I forgot to mention that if a 'bad' pdf has been transferred, it will continue to fail. Changing the plugin would only help (if it helps) if all the books are resent to the device

What I am wondering is why ALL the files are dated "today" after each time I disconnect and wait for the arrows to finish spinning.

Clearly the Sony is deciding that the files have been modified. The question is why. If connecting calibre is involved, then we will need to look at the Sony DB before and after calibre runs to see what changed.

[chaley]

Originally Posted by Perkin

I don't know if this is related, but maybe someone can add details, but wasn't there a problem with Sony's, when you had epub's and pdf's as well?
(I've got a 300 as well and try and steer clear of pdf's)

Nice kitty.

I don't have any memory of such a problem, but that doesn't mean much. As you say, perhaps someone will join in.

[SkyDream]

Originally Posted by cinco

Thanks. Yes, you're right: the "long time" is after I disconnect the device and the little arrows spin for 10-20 MINUTES! .

Sony Readers are scanning all books after you disconnect the device. This will take longer and longer as more and bigger books are added. This is an old everlasting problem of Sony. If you reduce the number of books and those books are not too big the little arrows should spin for a shorter time.

There are several thread about this problem which has nothing to do with calibre.

Maybe you try to go down to 50 books and test how long it take after disconnect.

[chaley]

Originally Posted by SkyDream

Sony Readers are scanning all books after you disconnect the device. This will take longer and longer as more and bigger books are added. This is an old everlasting problem of Sony. If you reduce the number of books and those books are not too big the little arrows should spin for a shorter time.

My 300 scans the file system on disconnect, looking for new books. This process takes around 30 seconds plus 1 second per book books. However, if the date in the Sony DB does not agree with the date on the book file, then the Sony will open the book and scan internal metadata. This operation can take some 4 to 8 seconds per book (epub -- no idea if pdf is worse) (the arrow will frequently stop spinning). For example, if I delete the sony DB, it takes some 2 minutes for my 300 to scan 15 books on my reader.

We (calibre developers) have had no end of problems with sony timestamps. If a sony reader becomes convinced that it is in a timezone different from the PC connecting to it, then the world can become strange. For that reason, calibre refuses to change a date in the DB if the book hasn't been newly added, under the assumption that the Sony has set the date the way it wants. The problems seem to be exacerbated by using both Sony Reader and calibre to talk to the device.

Today, a sony reader should take a small number of seconds per 100 books after disconnecting from calibre. If it is taking a lot longer, then something else is going on, and that something else deserves some attention. Unfortunately, I have only a 300 and I don't have the problem, so it is very hard for me to determine what is happening.

[DoctorOhh]

Originally Posted by chaley

I don't have any memory of such a problem, but that doesn't mean much. As you say, perhaps someone will join in.

The only epub vs pdf conflict was with drm infested books loaned from libraries. Even then it was only the ability to open the other that was hindered.