[GUI Plugin] FanFictionDownLoader

[JimmXinu]

@Snowman & @ILB -

From the descriptions, you two have different problems. But I have no idea what's causing either of them. I've tried adding the explicit 'use_new_db = True' setting from testing pre-1.0.0 and using cover grid view, but it makes no difference for me.

Both of you please:

1. Post story URL(s) that exhibit the problem for you.
2. Open a command prompt and run calibre in debug mode (calibre-debug -g). Then download a story that exhibits the problem and post a copy of the debug output from the command prompt window.

[JimmXinu]

Attached is a test version with additional debug output to try and figure out the problems people are having with custom column updates and Count Pages being called.

If you're not having these problems, there's no reason for you to use it.

Question: A user on the google group with the same issues (both of them) is using calibre in portable mode. Are the two users having problems here?

UPDATE Aug 27, 2013 -- Remove obsolete beta versions

[Snowman]

Originally Posted by JimmXinu

Attached is a test version with additional debug output to try and figure out the problems people are having with custom column updates and Count Pages being called.

If you're not having these problems, there's no reason for you to use it.

Question: A user on the google group with the same issues (both of them) is using calibre in portable mode. Are the two users having problems here?

I'm using the standard install, not the portable.

And now that I look again, I see that other columns are not being updated, such as status and last-updated.

Attached is a zip of the output from running the debug version in calibre debug and adding a new book, http://www.fanfiction.net/s/6963092/...ng-Appearances

Many thanks,

snowman

[ILB]

Originally Posted by JimmXinu

Attached is a test version with additional debug output to try and figure out the problems people are having with custom column updates and Count Pages being called.

If you're not having these problems, there's no reason for you to use it.

Question: A user on the google group with the same issues (both of them) is using calibre in portable mode. Are the two users having problems here?

I´m also unsing the standard install.

story:http://archiveofourown.org/works/939556
debug output see enclosed

edit: I´ve got also a column "AO3 collections" which should be filled with a setting in personal.ini.
[archiveofourown.org]
custom_columns_settings:
collections=>#collections_ao3

That column also stays empty.

in regards to "not-custom" columns:
Another column which is not filled is language.
However the column "series" is filled.

[Sefiriot]

First time updating stories after installing calibre 1.0 64-bit standard install on Windows 7 Pro x64.

First off, I noticed that the FFDL plugin seems to have stopped calling Reading List to update my reading lists (I have FFDL set to add new/updated stories to a "Send to Phone" reading list) or custom column metadata-- word count, two other boolean columns for "synced to device" and "(marked as) read" are not being updated.

Lastly, got this error message trying to update an anthology epub, with the series link being http://archiveofourown.org/series/40129.

calibre, version 1.0.0
ERROR: Unhandled exception: <b>ConstraintError</b>:ConstraintError: columns book, tag are not unique

calibre 1.0 [64bit]  isfrozen: True is64bit: True
Windows-7-6.1.7601-SP1 Windows ('64bit', 'WindowsPE')
('Windows', '7', '6.1.7601')
Python 2.7.4
Windows: ('7', '6.1.7601', 'SP1', 'Multiprocessor Free')
Traceback (most recent call last):
  File "site-packages\calibre\gui2\proceed.py", line 118, in do_ask_question
  File "calibre_plugins.fanfictiondownloader_plugin.ffdl_plugin", line 1296, in do_download_merge_update
  File "calibre_plugins.fanfictiondownloader_plugin.ffdl_plugin", line 1111, in update_books_loop
  File "calibre_plugins.fanfictiondownloader_plugin.ffdl_plugin", line 1412, in update_metadata
  File "site-packages\calibre\db\legacy.py", line 458, in set_metadata
  File "site-packages\calibre\db\cache.py", line 55, in ans
  File "site-packages\calibre\db\cache.py", line 1137, in set_metadata
  File "site-packages\calibre\db\cache.py", line 1105, in protected_set_field
  File "site-packages\calibre\db\cache.py", line 1085, in set_field
  File "site-packages\calibre\db\cache.py", line 906, in set_field
  File "site-packages\calibre\db\write.py", line 502, in set_books
  File "site-packages\calibre\db\write.py", line 415, in many_many
  File "site-packages\calibre\db\backend.py", line 291, in executemany
  File "c:\cygwin\home\kovid\sw\build\apsw-3.7.14.1-r1\src\cursor.c", line 231, in resetcursor
ConstraintError: ConstraintError: columns book, tag are not unique

I restarted calibre in debug mode, and proceeded to try again on the same series URL as above, as well as creating three new anthologies from Archive of Our Own series URLs: one series has only one story in it (http://archiveofourown.org/series/54548), the other has multiple stories (http://archiveofourown.org/series/37060).

I have also tried making an anthology manually from FF.Net URLs, as well as update several other existing anthologies. In all instances I consistently get the ConstraintError message. Output from all that is enclosed below in the attachment. All this is from the currently latest released version of the FFDL plugin.

Let me know if you need anything else or if you need me to use your new debugging plugin and repeat the tests as well yeah? Thanks.

[Sefiriot]

Using standard install of calibre x64 v1.0 on Win7 Pro x64. FFDL isn't populating my wordcount column with downloaded wordcount metadata (Count Pages isn't configured, so existing wordcount values aren't from there, I don't think), or updating my Reading Lists automatically, as it should. I'm also getting ConstraintError messages when creating new or updating existing anthology epubs.

calibre, version 1.0.0
ERROR: Unhandled exception: <b>ConstraintError</b>:ConstraintError: columns book, tag are not unique

calibre 1.0 [64bit]  isfrozen: True is64bit: True
Windows-7-6.1.7601-SP1 Windows ('64bit', 'WindowsPE')
('Windows', '7', '6.1.7601')
Python 2.7.4
Windows: ('7', '6.1.7601', 'SP1', 'Multiprocessor Free')
Traceback (most recent call last):
  File "site-packages\calibre\gui2\proceed.py", line 118, in do_ask_question
  File "calibre_plugins.fanfictiondownloader_plugin.ffdl_plugin", line 1296, in do_download_merge_update
  File "calibre_plugins.fanfictiondownloader_plugin.ffdl_plugin", line 1111, in update_books_loop
  File "calibre_plugins.fanfictiondownloader_plugin.ffdl_plugin", line 1412, in update_metadata
  File "site-packages\calibre\db\legacy.py", line 458, in set_metadata
  File "site-packages\calibre\db\cache.py", line 55, in ans
  File "site-packages\calibre\db\cache.py", line 1137, in set_metadata
  File "site-packages\calibre\db\cache.py", line 1105, in protected_set_field
  File "site-packages\calibre\db\cache.py", line 1085, in set_field
  File "site-packages\calibre\db\cache.py", line 906, in set_field
  File "site-packages\calibre\db\write.py", line 502, in set_books
  File "site-packages\calibre\db\write.py", line 415, in many_many
  File "site-packages\calibre\db\backend.py", line 291, in executemany
  File "c:\cygwin\home\kovid\sw\build\apsw-3.7.14.1-r1\src\cursor.c", line 231, in resetcursor
ConstraintError: ConstraintError: columns book, tag are not unique

Did a few tests in calibre's debug mode after, output log is here. let me know if I should try again with the plugin you just put out for debugging.

[JimmXinu]

Okay, now everybody is having the same problem at least. Except me--I still don't see it.

It's failing to update metadata entirely--before it even gets to the custom columns/Count Pages.

Traceback (most recent call last):
  File "calibre_plugins.fanfictiondownloader_plugin.dialogs", line 551, in do_loop
  File "calibre_plugins.fanfictiondownloader_plugin.ffdl_plugin", line 1111, in update_books_loop
  File "calibre_plugins.fanfictiondownloader_plugin.ffdl_plugin", line 1413, in update_metadata
  File "site-packages\calibre\db\legacy.py", line 458, in set_metadata
  File "site-packages\calibre\db\cache.py", line 55, in ans
  File "site-packages\calibre\db\cache.py", line 1137, in set_metadata
  File "site-packages\calibre\db\cache.py", line 1105, in protected_set_field
  File "site-packages\calibre\db\cache.py", line 1085, in set_field
  File "site-packages\calibre\db\cache.py", line 906, in set_field
  File "site-packages\calibre\db\write.py", line 502, in set_books
  File "site-packages\calibre\db\write.py", line 415, in many_many
  File "site-packages\calibre\db\backend.py", line 291, in executemany
  File "c:\cygwin\home\kovid\sw\build\apsw-3.7.17-r1\src\cursor.c", line 231, in resetcursor
ConstraintError: ConstraintError: columns book, tag are not unique

I'm going to bring this to Kovid's attention to see if he has any ideas, but in the meantime, can I get copies of people's personal.ini (remove passwords first) and/or full preferences (FFDL config->Other->View Library Preferences).

[ILB]

Originally Posted by JimmXinu

Okay, now everybody is having the same problem at least. Except me--I still don't see it.

It's failing to update metadata entirely--before it even gets to the custom columns/Count Pages.

Traceback (most recent call last):
  File "calibre_plugins.fanfictiondownloader_plugin.dialogs", line 551, in do_loop
  File "calibre_plugins.fanfictiondownloader_plugin.ffdl_plugin", line 1111, in update_books_loop
  File "calibre_plugins.fanfictiondownloader_plugin.ffdl_plugin", line 1413, in update_metadata
  File "site-packages\calibre\db\legacy.py", line 458, in set_metadata
  File "site-packages\calibre\db\cache.py", line 55, in ans
  File "site-packages\calibre\db\cache.py", line 1137, in set_metadata
  File "site-packages\calibre\db\cache.py", line 1105, in protected_set_field
  File "site-packages\calibre\db\cache.py", line 1085, in set_field
  File "site-packages\calibre\db\cache.py", line 906, in set_field
  File "site-packages\calibre\db\write.py", line 502, in set_books
  File "site-packages\calibre\db\write.py", line 415, in many_many
  File "site-packages\calibre\db\backend.py", line 291, in executemany
  File "c:\cygwin\home\kovid\sw\build\apsw-3.7.17-r1\src\cursor.c", line 231, in resetcursor
ConstraintError: ConstraintError: columns book, tag are not unique

I'm going to bring this ti Kovid's attention to see if he has any ideas, but in the meantime, can I get copies of people's personal.ini (remove passwords first) and/or full preferences (FFDL config->Other->View Library Preferences).

personal.ini enclosed

[Snowman]

Originally Posted by JimmXinu

<snip> can I get copies of people's personal.ini (remove passwords first) and/or full preferences (FFDL config->Other->View Library Preferences).

OK here is my full prefs:

Spoiler:

{
  "updateepubcover": false, 
  "custom_cols": {
    "#updated": "dateUpdated", 
    "#status": "status"
  }, 
  "gc_site_settings": {
    "archiveofourown.org": "", 
    "dramione.org": "", 
    "merlinfic.dtwins.co.uk": "", 
    "dark-solace.org": "", 
    "Default": "", 
    "chaos.sycophanthex.com": "", 
    "bloodties-fans.com": "", 
    "ashwinder.sycophanthex.com": "", 
    "archive.skyehawke.com": "", 
    "erosnsappho.sycophanthex.com": "", 
    "fanfiction.mugglenet.com": "", 
    "castlefans.org": ""
  }, 
  "updatemeta": true, 
  "addtoreadlists": true, 
  "urlsfromclip": false, 
  "collision": "Overwrite if Newer", 
  "suppressauthorsort": false, 
  "countpagesstats": [
    "WordCount"
  ], 
  "deleteotherforms": false, 
  "send_lists": "", 
  "lookforurlinhtml": false, 
  "showmarked": false, 
  "updatedefault": false, 
  "addtolists": false, 
  "read_lists": "Updated", 
  "suppresstitlesort": false, 
  "checkforurlchange": true, 
  "rejecturls": "http://www.fanfiction.net/s/3551453/1/,Harry and the Magic Factory by DisobedienceWriter - Abandoned\nhttp://www.fanfiction.net/s/7372751/1/,Animagi Fury by kevin1984 - Abandoned\nhttp://www.fanfiction.net/s/4352237/1/,A New Hogwarts by Leath1 - Abandoned\nhttp://www.fanfiction.net/s/2986948/1/,Maximus! by padfootsrevenger - Abandoned\nhttp://www.fanfiction.net/s/7572423/1/,Another Life: Year Three by WerewolfDoctor - Abandoned\nhttp://www.fanfiction.net/s/4228543/1/,Harry Potter and the Ties of Blood by WolfMoon - Abandoned\nhttp://www.fanfiction.net/s/2334259/1/,Harry Potter and the Ties of BloodOld by WolfMoon - Abandoned\nhttp://www.fanfiction.net/s/7899997/1/,Viva La Revolution! by Aileen Autarkeia - Withdrawn\nhttp://www.fanfiction.net/s/2216155/1/,Human Magic by Bleudiablo - Withdrawn\nhttp://www.fanfiction.net/s/4205960/1/,Morgana by Bleudiablo - Withdrawn\nhttp://www.fanfiction.net/s/5726087/1/,Peorth Snape: Pack Master by Bleudiablo - Withdrawn\nhttp://www.fanfiction.net/s/5083740/1/,Rootless Tree by Paffy - Withdrawn\nhttp://www.fanfiction.net/s/7165023/1/,A Tale of Two Destinies: The Prisoner by Wendi Jo Harper - Withdrawn", 
  "updatecover": false, 
  "addtolistsonread": false, 
  "gcnewonly": true, 
  "personal.ini": "## This is an example of what your personal configuration might look\n## like.  Uncomment options by removing the '#' in front of them.\n\n[defaults]\n## Some sites also require the user to confirm they are adult for\n## adult content.  Uncomment by removing '#' in front of is_adult.\nis_adult:true\n\n## Don't like the numbers at the start of chapter titles on some\n## sites?  You can use strip_chapter_numbers to strip them off.  Just\n## want to make them all look the same?  Strip them off, then add them\n## back on with add_chapter_numbers.  Don't like the way it strips\n## numbers or adds them back?  See chapter_title_strip_pattern and\n## chapter_title_add_pattern.\n#strip_chapter_numbers:true\n#add_chapter_numbers:true\n\n[epub]\n## include images from img tags in the body and summary of stories.\n## Images will be converted to jpg for size if possible.  Images work\n## in epub format only.  To get mobi or other format with images,\n## download as epub and use Calibre to convert.\ninclude_images:true\n\n## If not set, the summary will have all html stripped for safety.\n## Both this and include_images must be true to get images in the\n## summary.\nkeep_summary_html:true\n\n## If set, the first image found will be made the cover image.  If\n## keep_summary_html is true, any images in summary will be before any\n## in chapters.\n#make_firstimage_cover:true\n\n## Resize images down to width, height, preserving aspect ratio.\n## Nook size, with margin.\n#image_max_size: 580, 725\n\n## Change image to grayscale, if graphics library allows, to save\n## space.\n#grayscale_images: false\n\n\n## Most common, I expect will be using this to save username/passwords\n## for different sites.  Here are a few examples.  See defaults.ini\n## for the full list.\n\n[www.twilighted.net]\n#username:YourPenname\n#password:YourPassword\n## default is false\n#collect_series: true\n\n[www.ficwad.com]\n#username:YourUsername\n#password:YourPassword\n\n[www.twiwrite.net]\n#username:YourName\n#password:yourpassword\n## default is false\n#collect_series: true\n\n[www.adastrafanfic.com]\n## Some sites do not require a login, but do require the user to\n## confirm they are adult for adult content.\n#is_adult:true\n\n[www.thewriterscoffeeshop.com]\n#username:YourName\n#password:yourpassword\n#is_adult:true\n## default is false\n#collect_series: true\n\n[www.fictionalley.org]\nis_adult:true\n\n[www.harrypotterfanfiction.com]\nis_adult:true\n\n[www.fimfiction.net]\nis_adult:true\n#fail_on_password: false\n\n[www.tthfanfic.org]\n#is_adult:true\n## tth is a little unusual--it doesn't require user/pass, but the site\n## keeps track of which chapters you've read and won't send another\n## update until it thinks you're up to date.  This way, on download,\n## it thinks you're up to date.\n#username:YourName\n#password:yourpassword\n\n\n## This section will override anything in the system defaults or other\n## sections here.\n[overrides]\n## default varies by site.  Set true here to force all sites to\n## collect series.\n#collect_series: true\n", 
  "checkforseriesurlid": true, 
  "std_cols_newonly": {
    "publisher": false, 
    "pubdate": false, 
    "tags": false, 
    "series": false, 
    "identifiers": false, 
    "comments": false, 
    "languages": false, 
    "timestamp": false, 
    "authors": false, 
    "title": false
  }, 
  "includeimages": true, 
  "injectseries": true, 
  "custom_cols_newonly": {
    "#wordcount": false, 
    "#updated": false, 
    "#status": false
  }, 
  "adddialogstaysontop": true, 
  "fileform": "epub", 
  "rejectreasons": "Sucked\nBoring\nDup from another site", 
  "keeptags": true, 
  "allow_custcol_from_ini": true, 
  "smarten_punctuation": false, 
  "errorcol": "none", 
  "allow_gc_from_ini": true
}

snowman

[JimmXinu]

@ILB - Thanks. I tried running using your personal.ini but I still don't see the problem.

[Snowman]

Would the metadata.db/db_prefs.json help? mine zip to about 1.3mb

snowman

[JimmXinu]

Originally Posted by Snowman

OK here is my full prefs:

Thanks. I wish I could say it helps, but I'm still in the same position--I can't make it happen on my machines.

[JimmXinu]

Originally Posted by Snowman

Would the metadata.db/db_prefs.json help?

I don't think so. I wouldn't know what to look for in it.

I suppose if you created a new library (using the same structure--there's an option for that) and showed it had the same problem with only one book in it and then sent me a zip of that entire library dir it might help.

[Snowman]

Originally Posted by JimmXinu

I suppose if you created a new library (using the same structure--there's an option for that) and showed it had the same problem with only one book in it and then sent me a zip of that entire library dir it might help.

OK. Done, and attached. I initially copied over one book, cleared the 'status' and 'wordcount' columns, then did a update "overwrite always", and the columns did not get updated, and count pages did not get called. I then deleted and recreated the debug lib again so that you get an untouched version, and copied in the same couple of books.

snowman

[JimmXinu]

@Snowman -- Yes, I get the same error now with that library. With any luck, now I'll be able to figure it out.