[GUI Plugin] Goodreads Sync

[auspex]

Originally Posted by Terisa de morgan

I like that, really like it. And I agree options are the bane of life

Options are the bane of life because they exist solely to allow somebody else to do things the wrong way!

[davidfor]

Originally Posted by auspex

That's got to be new. I've never had it disallow longer reviews. That's horrible. It's already impossible to download full reviews without explicitly visiting the site, now they seem to be preventing doing it the other way. Which makes it difficult to keep my own copy of reviews for when they go on another purge.

I used the wrong word. When you do a progress update you can enter a comment. This is limited to 420 characters. The review is separate, and updated when you add the book to the "read" shelf. This appears to be limited as well. But, hopefully 20000 characters is enough.

[auspex]

Ah! Yes, reviews are limited to 20,000 characters on the website.

[davidfor]

Originally Posted by Rellwood

I checked the date and time, they are spot on. I ended up grabbing the .JSON file from another calibre library on another computer, it seems to be working now. I did however, download and install the new beta version you pointed out to me, as well as uninstalled and reinstalled the APNX generator. I don't know if its updated or not, but it's what was available in the "add plugins" arena. I will see how the beta and the reinstalled APNX generators work. Thanks!

The beta versions have no changes that would affect this. The only thing is that I left some extra debug statements in it that show a little more information.

For the APNX plugin, there is an update on the last the last page of the thread that I think fixed this.

This is still coming up when I try to sync...I never had a problem with big libraries before, but it seems like there is some kind of number limit


calibre, version 2.48.0
ERROR: Goodreads Failure: The request contacting Goodreads has failed. Please try again.

URL: http://www.goodreads.com/review/list...1&per_page=200
Response Code: 401
not authorized

Sorry, but, I have no idea what is happening. I don't see any problems, but I don't use the sync much. The only thing I can think of is that Goodreads has added some limits that you might be hitting. Are you doing any searches or anything else with Goodreads before this?

[Krazykiwi]

Originally Posted by davidfor

Sorry, but, I have no idea what is happening. I don't see any problems, but I don't use the sync much. The only thing I can think of is that Goodreads has added some limits that you might be hitting. Are you doing any searches or anything else with Goodreads before this?

Newly started Calibre, no GR connections yet. The plugin is authorised, and I used it earlier today to add GR id's to a dozen or so new books.

My read shelf has 655 books on it, and hasn't synched since actually the same day the 403 errors we were talking about in the metadata plugin thread surfaced. My to-read and to-get shelves have 169 and 182 books on resp. and they sync, as long as I don't attempt to sync my read shelf first.

Spoiler:

calibre, version 2.48.0
ERROR: Goodreads Failure: The request contacting Goodreads has failed. Please try again.

URL: http://www.goodreads.com/review/list...1&per_page=200
Response Code: 401
not authorized

It is authorized, attempting to re-authorize it reports that "This app is already authorized"

[davidfor]

Originally Posted by Krazykiwi

Newly started Calibre, no GR connections yet. The plugin is authorised, and I used it earlier today to add GR id's to a dozen or so new books.

My read shelf has 655 books on it, and hasn't synched since actually the same day the 403 errors we were talking about in the metadata plugin thread surfaced. My to-read and to-get shelves have 169 and 182 books on resp. and they sync, as long as I don't attempt to sync my read shelf first.

Spoiler:

calibre, version 2.48.0
ERROR: Goodreads Failure: The request contacting Goodreads has failed. Please try again.

URL: http://www.goodreads.com/review/list...1&per_page=200
Response Code: 401
not authorized

It is authorized, attempting to re-authorize it reports that "This app is already authorized"

If you look back through the last few pages, you can see that this is happening to a few people. And I don't know why.

Looking at the code just now, can someone who is seeing this error try creating a shelf from the configuration dialog? I'd like to know if this works, or you get a similar error.

And could someone send me their plugin configuration file? The file is "Goodreads Sync.json" and it is in the plugins directory of the calibre configuration directory. But, DO NOT POST it without a bit of editing. The file includes user specific tokens. You don't want to post that in an open forum. I mainly want to see that the file has been created successfully and has the correct structure. You should remove the values for the keys "userSecret" and "userToken".

[davidfor]

While looking at the reported problems with the shelf syncing, I noticed that the lists of books didn't allow any sorting. This beta adds sorting for all columns in most of the lists. Clicking on any of the column headers in the list should sort by that column.

I think I did this for all dialogs and lists, but I might have missed one. If someone sees one that doesn't allow sorting, tell me and I'll fix it.

Unless there is a problem reported (other than the authentication problems) I'll arrange for this to be released over the weekend.

[Krazykiwi]

Originally Posted by davidfor

If you look back through the last few pages, you can see that this is happening to a few people. And I don't know why.

Right that's actually why I mentioned it, I should have said "I can reproduce this", I was hoping the shelf numbers helped (I notice the page is pulling with 200 results per page, the shelves that sync both have less than 200 entries, the one that doesn't has far more).

Creating a shelf from the plugin fails.

Spoiler:

calibre, version 2.48.0
ERROR: Goodreads Failure: The request contacting Goodreads has failed. Please try again.

URL: http://www.goodreads.com/user_shelves.xml
Response Code: 201
<?xml version="1.0" encoding="UTF-8"?>
<user_shelf>
<id type="integer">167547069</id>
<user_id type="integer">638231</user_id>
<name>rereads</name>
<book_count type="integer">0</book_count>
<editable_flag type="boolean">true</editable_flag>
<updated_at type="datetime">2016-01-14T20:17:33+00:00</updated_at>
<created_at type="datetime">2016-01-14T20:17:33+00:00</created_at>
<exclusive_flag type="boolean">false</exclusive_flag>
<sortable_flag type="boolean">false</sortable_flag>
<description nil="true"/>
<sort nil="true"/>
<order nil="true"/>
<per_page type="integer" nil="true"/>
<display_fields>011110100110000110000100000000</display_fields>
<featured type="boolean">false</featured>
<recommend_for type="boolean">true</recommend_for>
<sticky type="boolean" nil="true"/>
</user_shelf>

My GR json attached (I subbed "REMOVED" for anything that looked like a local token.)

[Rellwood]

Right that's actually why I mentioned it, I should have said "I can reproduce this", I was hoping the shelf numbers helped (I notice the page is pulling with 200 results per page, the shelves that sync both have less than 200 entries, the one that doesn't has far more).

Same here....

[davidfor]

Originally Posted by Krazykiwi

Right that's actually why I mentioned it, I should have said "I can reproduce this", I was hoping the shelf numbers helped (I notice the page is pulling with 200 results per page, the shelves that sync both have less than 200 entries, the one that doesn't has far more).

The failure is on the first request, so it doesn't matter how many pages there would be as the first page is never retrieved

Creating a shelf from the plugin fails.

Spoiler:

calibre, version 2.48.0
ERROR: Goodreads Failure: The request contacting Goodreads has failed. Please try again.

URL: http://www.goodreads.com/user_shelves.xml
Response Code: 201
<?xml version="1.0" encoding="UTF-8"?>
<user_shelf>
<id type="integer">167547069</id>
<user_id type="integer">638231</user_id>
<name>rereads</name>
<book_count type="integer">0</book_count>
<editable_flag type="boolean">true</editable_flag>
<updated_at type="datetime">2016-01-14T20:17:33+00:00</updated_at>
<created_at type="datetime">2016-01-14T20:17:33+00:00</created_at>
<exclusive_flag type="boolean">false</exclusive_flag>
<sortable_flag type="boolean">false</sortable_flag>
<description nil="true"/>
<sort nil="true"/>
<order nil="true"/>
<per_page type="integer" nil="true"/>
<display_fields>011110100110000110000100000000</display_fields>
<featured type="boolean">false</featured>
<recommend_for type="boolean">true</recommend_for>
<sticky type="boolean" nil="true"/>
</user_shelf>

OK, that proves it is something in the authentication. Both the shelf syncing and the shelf creation us OAuth for their access.

My GR json attached (I subbed "REMOVED" for anything that looked like a local token.)

That looks OK. The devkeySecret and devkeyToken are hard-coded in the plugin, so they don't really need to be cleared. But, if they were the problem, I think the message would refer to the developer key being wrong. The userToken and userSecret are the user specific values. They appear to be random looking strings of letters and numbers.

Two things I don't think have been mentioned with this problem:

- What OS is being used? It shouldn't matter, but, it should be checked.
- Is anyone using a proxy?

[Krazykiwi]

Originally Posted by davidfor

The failure is on the first request, so it doesn't matter how many pages there would be as the first page is never retrieved

[...]

- What OS is being used? It shouldn't matter, but, it should be checked.
- Is anyone using a proxy?

That's a shame about the numbers, if it would only be so simple.

I also noticed, if I try the read shelf first, it fails, but then I can't sync the others after.

OS here is Windows 10 64 bit. No proxy.

I can set up a calibre and attempt synching on Kubuntu or FreeBSD but that probably won't be until Monday, so hopefully some others chime in before then.

[Rellwood]

@davidfor

Straight up, the plugin wont sync from shelves holding 200+ books.

I managed a workaround by dividing my "read" shelf into multiple "read" shelves (although they are not exclusive) I am easily able to sync from the new shelves that hold 100 books. However....

I am now getting a new error. When I try to link the books in the library to those attempting to be synced I am now getting an error saying that the table widget has too many arguments. I have attached the screenshots.

As far as adding or syncing. I can add all I like, I can only sync from shelves less than 200 books, and then find that there is an error trying to link

[davidfor]

Originally Posted by Rellwood

@davidfor

Straight up, the plugin wont sync from shelves holding 200+ books.

I managed a workaround by dividing my "read" shelf into multiple "read" shelves (although they are not exclusive) I am easily able to sync from the new shelves that hold 100 books. However....

Interesting, but, I'm syncing a 222 book read shelf with no problems. Like you, I don't have all 222 books in my calibre library, but, that is done after the call to Goodreads.

When you split the shelf, did you keep the original "read" shelf or create two new shelves? If you have the original "read" shelf, can you put a few books on it and try to sync it? Do you get an error with any of the other shelves such as "currently-reading" or "to-read"?

I am now getting a new error. When I try to link the books in the library to those attempting to be synced I am now getting an error saying that the table widget has too many arguments. I have attached the screenshots.

As far as adding or syncing. I can add all I like, I can only sync from shelves less than 200 books, and then find that there is an error trying to link

I'll look at it when I get home, but...

Which version of the plugin are you using? The release version or one of the beta's? If it is the last one, I probably broke this when adding the column sorting.

Also, this dialog is doing a search of your library using the name and author from Goodreads. Will there be a match?

And, clicking the "Copy to clipboard" button is usually a better way to get the error information. That has the big advantage to me that I can copy stuff from it for searching the code.

[davidfor]

As Rellwood discovered, there is a problem with the Shelf Sync dialog in the last beta I posted. This is caused by the sorting I added. If you didn't change the sorting, everything should have been OK. But, if you did change the sorting, the search in calibre option could break. And when it didn't, it searched for the wrong book.

The attached beta version fixes this. My quick testing seems to show it is OK. I haven't had a chance to check the other dialogs that I changed. There might be problems with them, but it should only happen if the order of the list is changed.

[Krazykiwi]

David, I just thought I'd mention, syncing is currently working fine, on all shelves and I can no longer reproduce the failures. I've (finally yay) synched all my shelves, including the 600+ one, with no problems.

I just today got around to updating to 1.7.7 on Calibre 2.49 (I suppose I should update that too, but now that it's unbroken, I'm loathe to touch it.) I did not yet test the beta with the sorting, but it doesn't sound like that has anything to do with the issues I was having earlier anyway.