Register Guidelines E-Books Search Today's Posts Mark Forums Read

Go Back   MobileRead Forums > E-Book Software > Calibre > Plugins

Notices

Reply
 
Thread Tools Search this Thread
Old 12-30-2011, 03:32 PM   #1
JimmXinu
Plugin Developer
JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.
 
Posts: 1,719
Karma: 509671
Join Date: Dec 2011
Location: Midwest USA
Device: Nook STR w/Glowlight, Kindle 3g, Droid
Post [GUI Plugin] FanFictionDownLoader

Many people use their ebook readers to read FanFiction stories posted on the web on various sites. Many people also use Calibre to manage their ebook collections and readers.

This plugin brings to functionality of FanFiction Downloader to Calibre. The source is available on the project home page, and we monitor our user group for problems. There are additional documentation and FAQs on the project wiki.

Main Features of FanFictionDownLoader Plugin:
  • Download FanFiction stories from over 100 different sites into ebooks. Supported sites list is below.
  • Create new and update existing epub anthologies of several stories. (Uses EpubMerge Plugin)
  • Download in the background for user convenience.
  • Support for downloading images in the story text. (EPUB only--download EPUB and convert to MOBI for Kindle) More details on configuring images in stories and cover images can be found in this post.
  • Support for cover image from three different sources. (EPUB only)
  • Support to resize, make grayscale, and convert images to jpg. (EPUB only)
  • Support to call Generate Cover plugin on calibre metadata create or update. Requires Generate Cover 1.5.1 or newer.
  • Support to call Count Pages plugin on calibre metadata create or update. Requires Count Pages 1.6.0 or newer.
  • Update/Overwrite existing FanFiction stories from story URL in Calibre identity or epub. FFDL ignores Virtual Library and Additional Restrictions while searching for books to update.
  • Update existing epub format ebooks, downloading only new chapters.
  • Optionally keep an Update Log of past updates (epub only).
  • Update calibre metadata from website.
  • Option for plugin to update calibre cover image on metadata update.
  • Option to delete other formats on book update. Handy if you have both a Nook(epub) and Kindle(mobi), for example.
  • Get original URLs from FanFiction stories in your library.
  • Update Reading List plugin lists as an aid to device sync and keeping a list of new books to read. Requires Reading List plugin 1.4.1 or newer to use.
  • Update calibre custom columns with story metadata.
  • Get Story URLs from Web Page
  • Options now stored inside the Library rather than an external JSON file.
  • Keep a list of previously 'Rejected' story URLs and user notes on why. Report if the user tries to download a rejected story again.
  • CLI via calibre-debug --run-plugin
  • Option to call Polish Book's Smarten Punctuation on each epub download/update. (calibre 0.9.39 and newer only.)

Supported Sites:
Spoiler:

Other useful Plugins:
Special Notes:
  • Requires Calibre v1.48.0 or later.

Languages:

This plugin has translations for:
  • English
  • French
  • German
  • Portuguese (Brazil)
  • Spanish

Command Line Interface(CLI)

Calibre offers a command-line interface (CLI) for calling plugins.

This plugin supports that option and allows the user to call the plugin's basic functionality from outside calibre without having to install Python or get the plugin's files separately.

(Note that at this time, FFDL via calibre CLI will use your personal.ini settings from your library, but does not add to or update your library.)

Details:
Spoiler:
Code:
Usage: calibre-debug.exe --run-plugin FanFictionDownLoader -- [options] <storyurl>

Options:
  -h, --help            show this help message and exit
  --library-path=LIBRARY_PATH, --with-library=LIBRARY_PATH
                        Path to the calibre library. Default is to use the
                        path stored in the settings.
  -f FORMAT, --format=FORMAT
                        write story as FORMAT, epub(default), mobi, text or
                        html
  -c CONFIG, --config=CONFIG
                        read config from specified file(s) in addition to
                        calibre plugin personal.ini,
                        ~/.fanficdownloader/personal.ini, and ./personal.ini
  -b START, --begin=START
                        Begin with Chapter START
  -e END, --end=END     End with Chapter END
  -o NAME=VALUE, --option=NAME=VALUE
                        set an option NAME=VALUE
  -m, --meta-only       Retrieve metadata and stop.  Or, if --update-epub,
                        update metadata title page only.
  -u, --update-epub     Update an existing epub with new chapters, give epub
                        filename instead of storyurl.
  --update-cover        Update cover in an existing epub, otherwise existing
                        cover (if any) is used on update.  Only valid with
                        --update-epub.
  --force               Force overwrite of an existing epub, download and
                        overwrite all chapters.
  -l, --list            Get list of valid story URLs from page given.
  -d, --debug           Show debug output while downloading.


Installation Steps:
  1. Download the attached zip file and install the plugin/add to context menu or toolbar/restart Calibre as described in the Introduction to plugins thread.
  2. Click to plugin button, choose your settings, enter URLs to FanFiction stories, and click 'Download'. The plugin will create or update the ebook in Calibre.

Version History:
Spoiler:

Version 2.0.8 - 20 Oct 2014
Additional fix for site updates to: ficwad.com

Version 2.0.7 - 20 Oct 2014
Fixes for site updates to: fimfiction.net (thanks, facedeer)
Fixes for site updates to: ficwad.com
Fixes for site updates to: archiveofourown.org
Known issue: Metadata collection is not as complete for 'Base eFiction' adapters.
Known problem: Password protected FimFiction.net stories aren't working. FimF changed API access.
Known problem: specific metadata 'eroticatags' for literotica.com doesn't work on all stories.

Version 2.0.6 - 06 Oct 2014
New Site: lotrfanfiction.com
New Site: csi-forensics.com, thanks scout78.
New Site: samandjack.net, thanks scout78.
Fixes for sheppardweir.com, thanks scout78.
Fixes for Base eFiction.
Known issue: Metadata collection is not as complete for 'Base eFiction' adapters.
Known problem: Password protected FimFiction.net stories aren't working. FimF changed API access.
Known problem: specific metadata 'eroticatags' for literotica.com doesn't work on all stories.

Version 2.0.5 - 23 Sep 2014
- Fix for AO3 get story list URLs that already have a '?' in them.
- Fix for squidge.org/peja using a story URL for 'Site Map'. (Messed up series and series numbers.)
- Fixes for storiesonline.net site update, from davidfor.
- Add 'extratags' to AllMetadata so it's available for custom columns.
- Only support from calibre v1.48.0 and up for plugin.
- Known problem: Password protected FimFiction.net stories aren't working. FimF changed API access.
- Known problem: specific metadata 'eroticatags' for literotica.com doesn't work on all stories.

Version 2.0.4 - 09 Sep 2014
- New site bdsm-geschichten.net, thanks John Doe.
- New site: sheppardweir.com, thanks scout78.
- New eFiction base adapter courtesy of John Doe. This should make it easier for new developers to add eFiction type sites. But it doesn't support all type of metadata.
- New site using eFiction base adapter: fannation.shades-of-moonlight.com, thanks John Doe.
- New site using eFiction base adapter: themaplebookshelf.com, thanks John Doe.
- Fix numChapters in adapter_literotica.py.
- Fixes for nhamagicalworldsus changes.
- Fix for changes to fanfiktion.de.
- Add/update translations in Spanish, French, Portuguese (Brazil), German.
- Fix so autoconvert won't delete FFDL's own output.
- Add cookie and page caching between stories and between front and background processing for selected sites: fanfiction.net, tthfanfic.org, archiveofourown.org, fanfiktion.de, fimfiction.net, portkey.org and buffynfaith.net.

Version 2.0.3 - 13 Aug 2014
Fix for AO3 authorUrls and authorIds.

Version 2.0.2 - 13 Aug 2014
Fix for AO3 authors all coming out Anonymous.

Version 2.0.1 - 04 Aug 2014
- New German language site bdsm-geschichten.net, thanks to doe5716.
- New site tolkienfanfiction.com, thanks to doe5716.
- Add other languages for literotica.com, thanks to doe5716.
- Fixes for bloodshedverse & spikelover having html tags in title, etc.
- Fix for non-split list replace_metadata.
- Fix for anthology books, don't put author in default anthology comments if only one author.
- Fix to plugin identifiers:"~ur(i|l)..." search string for finding existing books.
- Known problem: Password protected FimFiction.net stories aren't working. FimF changed API access.
- Known problem: specific metadata 'eroticatags' for literotica.com doesn't work on all stories.

Version 2.0.0 - 23 Jul 2014
Now supporting Qt5 and Calibre 2.0 (currently in beta).
Translations updates to French and German and new translation Spanish, courtesy of the volunteers at Transifex.
Apply cover_exclusion_regexp to specific cover images, too
Add site specific reviews to wraithbait.com
Allow ffnet story specific covers by default
Add "Reject Without Confirmation" feature (plugin only)
Add Spanish translation (plugin only)
Change default encoding for bloodshedverse.com to Windows-1252
Allow chapterless Fimf stories
Fix for login needed for efpfanfic.net 'red' rated stories
Known problem: Password protected FimFiction.net stories aren't working. FimF changed API access.
Known problem: specific metadata 'eroticatags' for literotica.com doesn't work on all stories.

Version 1.8.26 - 25 Jun 2014
Fixes for some metadata problems on various sites.
Known problem: specific metadata 'eroticatags' for literotica.com doesn't work on all stories.

Version 1.8.25 - 21 Jun 2014
New site: fictionmania.tv -- Thanks, cryzed!
Fix for site recognizer to handle with/without www. using https URLs.
Fix for some utf8 descriptions on fimf.
Site specific metadata 'eroticatags' for literotica.com.
Fix for 'a' flag on custom_columns_settings not working as intended.
Changes to '\,' to better implement split list feature to avoid infinite recursions.
Fix for calibre-injected series--don't treat series as a list, it isn't.
Auto-add http: to URLs starting with //.
Switch to transifex.com for translations.

Version 1.8.24 - 14 Jun 2014
Now supporting over 100 different sites! Thanks, cryzed, for pushing us over the top.
New site: nocturnal-light.net -- Thanks, cryzed!
New site: fanfic.hu (Hungarian language) -- Thanks, cryzed!
New site: fanfiction.csodaidok.hu (Hungarian language) -- Thanks, cryzed!
Updates for sites: bloodshedverse.com, voracity2.e-fic.com -- Thanks, cryzed!
Fix for making anthology books when source books don't have number of words.
New '\,' feature to 'split' metadata items in replace_metadata.
Example:
replace_metadata:
category=>Captain America \(Movies\)=>Captain America\,Marvel
The category list will then contain both "Captain America" and "Marvel".

Version 1.8.23 - 07 Jun 2014
New site: spikeluver.com -- Thanks, cryzed!
New site: bloodshedverse.com -- Thanks, cryzed!
Updates for site: voracity2.e-fic.com -- Thanks, cryzed!

Version 1.8.22 - 04 Jun 2014
New site: voracity2.e-fic.com -- Thanks, cryzed!

Version 1.8.21 - 23 May 2014
Fix for dark-solace.org Rating.
Fix for literotica.com URLs.

Version 1.8.20 - 13 May 2014
Fix for AO3 story not found.
Fix for storiesonline.net changing urls.
Fix for fictionpad.com removing 'dislikes' in some(all?) cases.
Fix for dark-solace.org metadata parsing.

Version 1.8.19 - 05 May 2014
Allow https URLs for fimfiction.net.
Both allow https URLs and change canonical URLs for squidge.org/peja to https.
Fix for some stories' summaries on onedirectionfanfiction.com.
Add include/exclude metadata feature. And wiki page for it.
Slightly improved connection refused handling.

Version 1.8.18 - 19 Apr 2014
Added workaround for an issue with fimfiction.net and blockquotes. Can be turned off by adding fix_fimf_blockquotes:false to the [www.fimfiction.net] section in personal.ini if it causes problems.

Version 1.8.17 - 09 Apr 2014
Fix for a problem with literotica.com multi-page chapters on Kobo readers.

Version 1.8.16 - 29 Mar 2014
Add Pairing for hpfanficarchive.com

Remove some outdated checks against calibre versions that aren't supported now.

Version 1.8.15 - 16 Mar 2014
Fix for fictionpress.com https urls.

Add options to turn off author notes on AO3 chapters.

Version 1.8.14 - 09 Mar 2014
Fix for potionsandsnitches.net, extra text after date.

Change Job Log output so URLs are more copy/pasteable.

Configurable connection timeout as per FaceDeer's recommendation.

Fix for seriesUrl check.

Fix for AO3 https urls.

On auto-convert, delete target format else convert doesn't run.

Version 1.8.13 - 28 Feb 2014
Fix for https vs http on ffnet (Story does not exist) problem.

Fix for KeyError:'site' problem.

Version 1.8.12 - 24 Feb 2014
Fixes for FimFiction change, thanks FaceDeer.

Set Boolean custom columns from custom_columns_settings, looking for true/t/yes/y/1 vs false/f/no/n/0.

Add a feature to reduce sleep between ffnet requests in initial metadata collection for small download sets.

Make ffnet look-for-next chapter feature optional and default to off.

Version 1.8.11 - 15 Feb 2014
Fix for fanfiction.net cover download. This is turned off by default in CLI and plugin due to how many stories in ffnet show the author's picture. In section [www.fanfiction.net] add never_make_cover: false to turn covers on.

Add feature to automatically inject cover from Generate Cover into epub using calibre's Polish feature.

Don't do auto-convert when Updating Calibre Metadata Only.

Version 1.8.10 - 5 Feb 2014
Fix French translation typo that prevents download.
Add auto-convert on download/update feature.
Fix divide by zero error in replace_br_with_p on chapters with no content.
Change to storiesonline.net by davidfor to address overzealous culling in text.

Version 1.8.9 - 22 Jan 2014
Leading zeros in version number makes it octal. Remove and bump to 1.8.9.

Version 1.8.08 - 21 Jan 2014
Allow literal strings in custom_columns_settings to set columns.

Version 1.8.07 - 13 Jan 2014
Change default encoding for finestories.com.
Change default encoding and improve metadata for storiesonline.net, thanks davidfor.
Fixes for hpfanficarchive.com changes.
Fix for 'update' with no updates and 'mark' enabled.
Now with French translation, thanks Ptitprince.

Version 1.8.06 - 23 Dec 2013
Fix for latest ffnet change.
Additional sleep added to ffnet metadata downloads.
New site: literotica.com. Thanks to de3sw2aq1.
Tweak to htmlheuristics for literotica.com (unicode vs str).

Version 1.8.05 - 12 Dec 2013
Fix issue with AO3 single chapter stories introduced in 1.8.04

Version 1.8.04 - 12 Dec 2013
(v1.8.03 was a testing only version)
Raise minimum calibre version due to changes in identifier:url searching
Alternate URL efiction.trekiverse.org for trekiverse.org/efiction
Add do_update_hook to AO3 for updated chapters in the middle
Reorder defaults.ini files to put user/pass is_adult first in all sections
Graceful fail for 'k'HTML when no 'k'Url

Version 1.8.02 - 05 Dec 2013
New Site: trekiverse.org/efiction
Fix for higher rated stories with not logged in on tthfanfic.org
Add prequels/sequels &amp; fix groups site specific metadata for fimfiction.net
Now including German translation, thanks ILB!
Fix for identifier searchs and Overwrite if Newer with ffnet timestamps.

Version 1.8.01 - 18 Nov 2013
Fix for stories with more than 999 chapters.
Fixes/Enhancements for replace_br_with_p heuristic processing feature. Thanks, Asbj°rn Grandt.
Fix for dc:language when none given and defaulting to English.
Fix for 'Default' Generate Cover setting with Internationalization/Localization.

Version 1.8.00 - 18 Nov 2013
Internationalization/Localization code added. If people volunteer to do translations, they'll be able to use FFDL in their own languages.
More fanfiction.net fixes. Don't included Updated/Published as characters, don't include excess Misc/Movies/etc categories on crossovers, start using https instead of http.
Improved error messaging for ficwad & squidge.org/peja

Version 1.7.54 - 13 Nov 2013
Fix for fanfiction.net more changes.

Version 1.7.53 - 11 Nov 2013
Fix for fanfiction.net changes.
Fixes/Enhancements for replace_br_with_p heuristic processing feature. Thanks, Asbj°rn Grandt.

Version 1.7.52 - 09 Nov 2013
Fixes for storiesonline.net, thanks davidfor.
Fixes/Enhancements for replace_br_with_p heuristic processing feature. Thanks, Asbj°rn Grandt.

Version 1.7.51 - 30 Oct 2013
New site: storiesonline.net
Add replace_br_with_p heuristic processing feature. Thanks, Asbj°rn Grandt.
Handling for fictionpad.com empty chapters. "This chapter has no text."
Handling for fanfiction.net's 'Pairings'.

Version 1.7.50 - 23 Oct 2013
Fix for ffnet changes.

Version 1.7.49 - 19 Oct 2013
Fixes for dark-solace.org
Tweaks to Config dialog for small screens

Version 1.7.48 - 13 Oct 2013
Fix for an HTML change on fanfiction.net

Version 1.7.47 - 13 Oct 2013
New site: fictionpad.com
Debug/information output reduced and changed to logging
New Basic option "Mark added/updated books when finished?"

Version 1.7.46 - 29 Sept 2013
Fixes for indeath.net - Thanks Besnef!
Better Non-BtVS/AtS detection for tthfanfic.org
do_update_hook for fimf, find newest chapter and update from there when true.

Version 1.7.45 - 21 Sept 2013
Fix for whofic.com.
Don't include empty string dc:subject tags in epub.
Set custom column only if there's a value (mostly for int/float columns).

Version 1.7.44 - 14 Sept 2013
Remove dateutil dependency
Fix for certain portkey.org stories.
Fix for PI for int/float custom columns setting empty.

Version 1.7.43 - 11 Sept 2013
Fix dates for fimfiction.net
Additional groups as site specific data for fimfiction.net.
Add default_value_status (etc) feature.

Version 1.7.42 - 09 Sept 2013
Longer sleep for ffnet, config User-agent

Version 1.7.41 - 08 Sept 2013
Fix for AO3 stories without any series.

Version 1.7.40 - 07 Sept 2013
Better doc section override order in ini files.
Fix for calibre pre-1.0.0.
Additional series as site specific data for AO3.
Fixes for changes to harrypotterfanfictioncom.
Add User-agent="FFDL/1.7" for all adapters for fanfiction.net changes.
(Remove from specific adapters.)

Version 1.7.39 - 31 Aug 2013
Remove work around for last_modified vs author link update issue in calibre 1.0.0--fixed in 1.1.0.
New Feature: add_genre_when_multi_category setting
Work around calibre's auto-split when author contains 'and' 'with' ',' or '&'.

Version 1.7.38 - 24 Aug 2013
Work around for last_modified vs author link update issue in calibre 1.0.0.
Better error reporting of metadata update problems.

Version 1.7.37 - 24 Aug 2013
Work around minor bug (1216398) in calibre 1.0.0. Tags 'A' and 'a' (for example) conflict and prevent metadata update.

Version 1.7.36 - 20 Aug 2013
Make check for Story URL change configurable.
Add configurable options to keep Title/Author the same in Title/Author Sort.

Version 1.7.35 - 06 Aug 2013
Remove some debugging output.
Minor optimizations for adapter lookup.
Get story url lists from sites with adapter, get urls from non-fic sites in CLI.
Warning on update if existing epub has no recognizable chapters.

Version 1.7.34 - 28 Jul 2013
Fixes for fanfiktion.de, storiesofarda.com, hpfanficarchive.com, and dark-solace.org/elysian sites.

Version 1.7.33 - 24 Jul 2013
Fix for smarten_punctuation setting in pre-0.9.39 calibre.

Version 1.7.32 - 23 Jul 2013
Fixes for fanfiktion.de & thehexfiles.net, add entries to teststory valid list.

Version 1.7.31 - 18 Jul 2013
Drag and drop onto the FFDL plugin button
Smarten Punctuation
sort_ships option
join_string_<entry> option
keep_in_order_<entry> option
replace_metadata <entry>_LIST options

Version 1.7.30 - 09 Jul 2013
New Site: scarhead.net - Thanks Ida!
Improved error handling when ffnet is having problems.
Phoenixsong can have individual chapters req login. Add force_login parameter.
GUI improvements to Basic config and Add/Update dialogs.

Version 1.7.29 - 28 Jun 2013
New Site: fanfic.potterheadsanonymous.com - Thanks Ida!
New Site: www.simplyundeniable.com - Thanks Ida!
Allow domain fanfic.mugglenet.com for mugglenet.com

Version 1.7.28 - 22 Jun 2013
Minor fixes to plugin
Fix for siye.co.uk stories with HTML in the title

Version 1.7.27 - 10 Jun 2013
Fixes for nha.magical-worlds.us and scarvesandcoffee.net.
Fall back category parsing for fanfiction.net when story has a broken crossover category link.
Restrict url list search on scarvesandcoffee.net.
Add 'Download as New Book?' dialog after 'Change Story URL?', fix author URLs when new author.

Version 1.7.26 - 30 May 2013
New site: nickandgreg.net - Thanks, Ida!
Add Read & Review counts to dramione.org and grangerenchanted.com
Fix for numeric site specific values into float/int custom columns for anthologies.

Version 1.7.25 - 26 May 2013
Add keep_style_attr option.
Add [teststory] feature.

Version 1.7.24 - 28 Apr 2013
Catch errors on db.set_cover().
Don't strip lead/trail whitespace from replace_metadata, add feature \s->' ' in replace_metadata replacements.
After adding to reject list, delete only rejects, not those removed from list.
Offer user chance to skip update when the story URL would be changed.

Version 1.7.23 - 12 Apr 2013
New site: netraptor.org
New site: asr3.slashzone.org
New site: tokra.fandomnet.com
Remove defunct site: www.jlaunlimited.com
Fix author URLs for several sites with leading 'dir' in URL.
Fix for no chapter name for one chapter stories on TtH.
Improved error handling for mobi issues.
Add 'url' to chapter custom formats and class="skip_on_ffdl_update" for updates.
Add anthology_title_pattern feature.

Version 1.7.22 - 08 Apr 2013
Fix for AO3 series URL.
Include notes with chapter text for thewriterscoffeeshop.com.
Remove support for defunct site: prisonbreakfic.net
Remove support for defunct site: svufiction.com

Version 1.7.21 - 05 Apr 2013
Fixes for change to site dark-solace.org/elysian.
Allow ini site section names both with and without www. IE, [www.fanfiction.net] and [fanfiction.net] will both work now. If both are included, the section without www overrides the section with it.
Fix for finding URLs during drag & drop of text emails.
Fix for persistent AddDialog size.

Version 1.7.20 - 01 Apr 2013
Another fix for fanfiction.net changes.

Version 1.7.19 - 28 Mar 2013
Another fix for fanfiction.net changes.

Version 1.7.18 - 28 Mar 2013
Fix for failure to correctly load old JSON settings.
Allow partial download with <url>[1-5] in addition to <url>[1,5].
Add author to chapter TOC for multi-author stories on TtH and WraithBait. (AO3 doesn't reliably report chapter author.)

Version 1.7.17 - 26 Mar 2013
Another fix for fanfiction.net changes.
Fix for collision setting when re-opening the add dialog.
Shorten button name to FanFictionDL.

Version 1.7.16 - 25 Mar 2013
Fix for fanfiction.net changes.

Version 1.7.15 - 25 Mar 2013
Make Add from URLs and Create Anthology for URLs modeless. That means you can open the Add/Create dialogs to put URLs in while still using the calibre library to look for duplicates, etc.
Add seriesUrl metadata. All sites that FFDL collects series info for now also collect a series URL. The default series on title_page is now a link. Used for...
Check for existing Series Anthology books (by seriesUrl) on story add/update. Similar to Reject List check, but offers to skip if you already have an anthology book for that story's series. (It doesn't offer to update the anthology for you.)
Add user/pass for dokuga.com.
Improvements to Reject List feature--allow sorting of reject list by url, title, author (when present). Improve readability of the reject dialogs.

Version 1.7.14 - 18 Mar 2013
Fix for filling int/float custom columns with favs/etc with commas.
Fix for HTML entities in complex replace_metadata setups.
Fixes for fimfiction.net site changes.

Version 1.7.13 - 15 Mar 2013
Remove BBCode--fimfiction.net adapter doesn't use it anymore.
Add "add_to_" feature to ini config. Allow higher priority sections to *add* to any ini param rather than replace it.
Add calibre CLI.
Add "marked:ffdl_success/failed" to added/updated/failed books.
Add "Show added/updated" pref using above.
Preserve log page in epub across overwrites as well as updates.
Drag and drop emails from Thunderbird on FFDL URL box to add URLs.

Version 1.7.12 - 04 Mar 2013
Use existing anthology title on update, even when a series.
Add anthology=true metadata for anthology covers, etc.
Page scrape story descriptions again for fimfiction.net due to API issues.
Fix login failure for tth.

Version 1.7.11 - 27 Feb 2013
www.ficwad.com -> ficwad.com. Ficwad changed their canonical domain. You'll need to change [www.ficwad.com] to [ficwad.com] in your personal.ini.

Version 1.7.10 - 19 Feb 2013
Include author notes in chapters on fimfiction.net.
Add Epub Anthology feature. - Requires EpubMerge 1.3.1+ Plugin.

Version 1.7.9 - 06 Feb 2013
New site: www.henneth-annun.net -- Thanks Ida!
New site: www.psychfic.com -- Thanks Ida!
Now accepting www.skyehawke.com/archive URLs for archive.skyehawke.com stories.

Version 1.7.8 - 04 Feb 2013
Fix author URLs for multiple author stories.

Version 1.7.7 - 04 Feb 2013
Improve error handling, add retry for restrictedsection.org.
Strip leading 0s from story ids for canonical ids in AO3.
Include hyphenation test case in test1 adapter.
Make get list url from page fill add dialog, not view log. (PI only)
Better fix for Nook hyphenation issue--Nook now honors adobe-hyphenate: none.
Remove extra newlines introduced by writer_epub on updates.
Fix broken lastupdate tags feature. Defaulted off for PI.
Optimize adding to Reading Lists on add/update. (PI only)

Version 1.7.6 - 28 Jan 2013
Fix for a change in fimfiction.net's handling of password protected stories.
Fix for thewriterscoffeeshop.com's changed date format.

Version 1.7.5 - 15 Jan 2013
New site: buffynfaith.net (Thanks Dan!)

Version 1.7.4 - 14 Jan 2013
New site: imagine.e-fic.com (Thanks Ida!)
Improved error reporting in background jobs.

Version 1.7.3 - 05 Jan 2013
New site: www.dotmoon.net (Thanks Ida!)
New site: efiction.esteliel.de (Thanks Ida!)
New site: pommedesang.com (Thanks Ida!)
New Spanish language site supported: www.potterfics.com (Thanks Dan!)
New site: www.restrictedsection.org (Yes, this site has been frozen since March 2009.)
More workarounds for fimfiction.net's API issues.

Version 1.7.2 - 02 Jan 2013
New Spanish language site www.potterfics.com (Thanks Dan!)
Remove site www.yourfanfiction.com--it's been gone for awhile.
Correct AO3 extra metadata freefromtags to freeformtags. freefromtags will continue to work for those already using it.
Workaround for fimfiction.net's API issue with non-viewable chapters given.
Add feature to set reason for several Reject URLs at once. PI only.

Version 1.7.1 - 15 Dec 2012
Allow % escapes in ffnet story URLs and use larger ffnet story cover images.
Force bgcolor in JPG output images when source images have transparency.
Add 'collections' to AO3 specific metadata.
Clear Reject List cache when switching libraries.

Version 1.7.0 - 26 Nov 2012
Set Language to Italian for www.efpfanfic.net.
Allow replace_metadata to effect Language metadata.
Fix fimfiction.net cover images
Default calibre language to English for new stories not otherwise marked
New Rejected URL list feature

Version 1.6.19 - 14 Nov 2012
Fixes for www.efpfanfic.net.
Allow 'On Hiatus' status for fimfiction.net.
Recognize calibre identifier uri as well as url.

Version 1.6.18 - 07 Nov 2012
Update potionsandsnitches.net date format.
Switch ponyfictionarchive.net from BeautifulStoneSoup to BeautifulSoup.
Precompile metadata_replace regexps for performance.
Change summary td to div in thehexfiles.net for table titlepages.
Tidy some more poor HTML on fictionalley.org
Honor begin/end chapters on update. For testing purposes.

Version 1.6.17 - 31 Oct 2012
New site: www.efpfanfic.net (Italian language site)

Version 1.6.16 - 28 Oct 2012
New site: indeath.net (thanks besnef)
New site: www.jlaunlimited.com (thanks besnef)
Fixes for changes to fanfiktion.de formatting.
Fix for bad author on twiwritenet with some skins.
Change from www.ncisfiction.com to www.ncisfiction.net due to the ncisfiction.com domain expiring.

Version 1.6.15 - 23 Oct 2012
Fix for some stories on hpfandom.net
Fix for some stories on fimfiction.net
Add conditional replace_metadata feature
Extra metadata for fimfiction

Version 1.6.14 - 17 Oct 2012
Changes to logging to make more logging visible in PI and the same across web/CLI/PI.
Don't number chapter when only one chapter even if add_chapter_numbers is on.
Fix a bug where cover could still be partially set to an image in the cover_exclusion_regexp list.
Add include_images support for HTML format.
Preserve id and class attributes on the HTML tags. (This is for users who want more control from output_css. I don't *think* it will hurt anything for other users...)
Change grangerenchanted.com to use more relaxed HTML parser due to problems with some stories.
Change HTML format output to include title/author in file name that appears inside zip file.


Version 1.6.13 - 08 Oct 2012
Fix for identifiers URL search.

Version 1.6.12 - 03 Oct 2012
New site www.qaf-fic.com (thanks, Ida)
Fix grangerenchanted.com, sg1heliopolis.com & themasque.net.
Add add_chapter_numbers: toconly option.
Better handling of 'Anonymous' stories on AO3.
Allow empty descriptions(summary).
Make just about all the output formatting customizable. See http://code.google.com/p/fanficdownl...erCustomOutput

Version 1.6.11 - 26 Sep 2012
Fix bbcode'd summary issue with fimfiction.net.
Fix get urls from web page bugs.
Fix bug with custom_column_settings 'a' when no data.
Add fanfictiondownloader_macmenuhack.txt 'feature' for Mac users with crash issue.
Improved include_in_* handling.
Change 'version' metadata to plugin version.
Add replace_metadata by metadata key feature.
Additional site-specific metadata for AO3.
Fix 'www.' handling for input URLs.
Fix bug with dokuga.com.
Fix extratags bug.

Version 1.6.10 - 21 Sep 2012
New sites bloodties-fan.com, thehookupzone.net (thanks Marillapm)
Allow individual site adapters to add extra metadata entries to the standard ones. (key word: extra_valid_entries)
Allow custom labels on title/log pages for extra metadata entries. (key words: themes_label, cliches_label, etc)
Allow those extra metadata to be included in titlepage/logpage, or added to existing metadata entries. (key words: extra_titlepage_entries, extra_logpage_entries, include_in_genre, etc)
Allow new, purely composite, metadata entries to be made from existing and extra metadata entries. (key words: include_in_*)
Allow existing metadata entries to be redefined. If one site has what you think should be category and genre reversed, for example, you could swap them. (key words: include_in_category, include_in_genre, etc)
Allow calibre custom columns to be populated from the new extra metadata entries. (key word: custom_columns_settings)
New configuration options to add to titlepage_entries, logpage_entries or include_subject_tags for specific sites (instead of replace it). (key words: extra_titlepage_entries, extra_logpage_entries, extra_subject_tags)
Allow the format for Published, Updated and Packaged dates on titlepage/logpage to be configured in personal.ini. (key words: dateCreated_format, datePublished_format, dateUpdated_format)
Cover support for dramione.org.
A fix for squidge.org/peja Rating.
Loosen url check on AO3 to allow more copy/paste URLs.
Fix author on castlefans.org due to site/skin changes.

Version 1.6.9 - 14 Sep 2012
New site merlinfic.dtwins.co.uk (thanks, Ida)
New option features strip_chapter_numbers and add_chapter_numbers to remove and add numbers to chapter titles.
Fix BBEncoded summaries for fimfiction.net, improved handling for HTML in summaries in general. Default keep_summary_html:true to now.
Added code to prevent crashing when include_images accidentally turned on for non-epub formats.

Version 1.6.8 - 06 Sep 2012
Add 'Relationships' metadata for sites that support it, or are dedicated to a 'ship.
Add 'extracategories', 'extragenres', etc to ini to avoid hard coding them in the adapters.
Special code to *not* include Config & About menu options for Mac users. (Can still configure plugin from calibre's configuration.) Causes some Macs to crash.
Fix for dracoandginny.com adapter for different skins.
Fix for &amp; in tags.
Scroll custom column config tab for users with lots of custom columns.


Version 1.6.7 - 27 Aug 2012
Allow shortcuts for config & about menu items to prevent crashes on some unpatched Macs.
Allow https urls (replace to http).
Fix for extra tags inside metadata on ksarchive.com

Version 1.6.6 - 25 Aug 2012
New site: www.sinful-desire.org
Add adobe-text-layout: optimizeSpeed; to epub CSS to prevent hyphenation on Nook STR(it loses parts of the hyphenated word sometimes.)

Version 1.6.5 - 23 Aug 2012
New site: www.dracoandginny.com
New site: www.scarvesandcoffee.net
New site: www.thepetulantpoetess.com
New site: www.wolverineandrogue.com

Version 1.6.4 - 18 Aug 2012
Fix for Flesch-Kincaid Grade Level Count Page setting.
Workaround for bad html in some story summaries on yourfanfiction.com.

Version 1.6.3 - 06 Aug 2012
Optionally keep an Update Log of past updates (epub only). See personal.ini include_logpage option.
Fix for nha.magical-worlds.us stories with more than 9 chapters.
'Get Story URLs from Web Page' now uses AO3 user/pass if saved in personal.ini.
'Collection' style AO3 story URLs now work.

Version 1.6.2 - 01 Aug 2012
Fix for fanfiction.net to prevent genre(s) appearing as a character on certain stories.

Version 1.6.1 - 01 Aug 2012
Remove extra html body from ancient ffnet chapters.
Add 'newonly' feature for standard and custom columns in plugin.

Version 1.6.0 - 27 Jul 2012
Options now stored inside the Library rather than an external JSON file.

Version 1.5.46 - 22 Jul 2012
Correct integration of grangerenchanted.com, hlfiction.net & nha.magical-worlds.us

Version 1.5.45 - 22 Jul 2012
New Site: grangerenchanted.com
New Site: hlfiction.net
New Site: nha.magical-worlds.us
Minor fixes for finestories.com, finestories.com, thewriterscoffeeshop.com.

Important Plugin Change: FFDL now uses a different, calibre system dialog before updating the library now. This dialog cannot be automatically skipped like the old one. The way it worked before risked corrupting your library if something else was trying to modify it at the same time.

Version 1.5.44 - 15 Jul 2012
New Site: finestories.com
New Site: www.hpfanficarchive.com
New Site: svufiction.com
New Site: www.twilightarchives.com
New Site: www.wizardtales.net
New Feature: Multiple authors per story. Currently supported for AO3, TtH and wraithbait.com. More will be added as we become aware that they use multiple author.
New Plugin Feature: Option to call 'Count Pages' plugin on new/update. Requires Count Pages 1.6.0 or better.
Fixes for latest fanfiction.net change.


Version 1.5.43 - 08 Jul 2012
New Site: dark-solace.org
New Site: pretendercentre.com
New Site: themasque.net
Fixes for a couple obscure problems with calling Generate Cover.

Version 1.5.42 - 06 Jul 2012
Fix for latest fanfiction.net change to prevent id:12345 appearing as a character.

Version 1.5.41 - 04 Jul 2012
Fix for thehexfiles.net on iBooks
Fix for img urls with spaces
Fix for 'Adding/Updating BAD books' status message

Version 1.5.40 - 02 Jul 2012
New Site: ncisfic.com
New Site: national-library.net
Added Support for www.fanfiktion.de and archiveofourown.org username and password.
Set Updated Date to Published Date when not found (fix for recent FFNet changes.)
Fix for fallback metadata labels, allow other text in titlepage_entries
New 'Error Column' feature - On update/overwrite fail, save error msg to a custom column.
Note: As part of the archiveofourown.org fix, the configuration section for AO3 is now [archiveofourown.org], not [www.archiveofourown.org]. Please change your personal configuration accordingly, if you have customized your settings for archiveofourown.org.
Note that www.fanfiktion.de doesn't just require you to say 'yes, I'm an adult'. To see adult content there outside certain hours, you have to have an official government German, Austrian or Swiss ID card or passport number.

Version 1.5.39 - 28 Jun 2012
Another fix for fanfiction.net's latest changes--prevent 'Status: Complete' from appearing as a character.

Version 1.5.38 - 28 Jun 2012
New Site: www.fanfiktion.de
New Site: ponyfictionarchive.net
New Site: sg1-heliopolis.com
Fix for poor summary html on thewriterscoffeeshop.com

Version 1.5.37 - 28 Jun 2012
Fixes for fanfiction.net's latest changes.

Version 1.5.36 - 18 Jun 2012
(1.5.35 skipped)
- Better error handling for bad image_max_size parameter.
- Fix issue with fanfiction.net covers and non-epub formats.
- Fix series# with javascript and different warning#s. geturls w/javascript.

Version 1.5.34 - 16 Jun 2012
Another fix for changed metadata on ffnet.

Version 1.5.33 - 16 Jun 2012
New sites:
- samdean.archive.nu
- www.yourfanfiction.com
- www.destinysgateway.com
- www.thealphagate.com
- stargate-atlantis.org
- www.ncisfiction.com

That's 51 different supported sites now. Thanks, Ida!

New features:
- Add 'Get Story URLs from Web Page' feature to plugin and CLI(-l).
- Plugin - Add some title output lists in the jobs detail window.
- Add cover image to fanfiction.net, but default 'never_make_cover: true' for ffnet.

Fixes:
- Fixes for ffnet characters metadata.
- Fix error in ashwinder from using a different skin.

Version 1.5.32 - 11 Jun 2012
Fix for AO3 'anonymous' stories.
Fix for ashwinder.sycophanthex.com.
Allow www.squidge.org/~peja as well as www.squidge.org/peja.

Version 1.5.31 - 09 Jun 2012
Fix for SIYE metadata for stories in a series.
Fix for ashwinder.sycophanthex.com login.
Fix for onedirectionfanfiction.com login.

Version 1.5.30 - 08 Jun 2012
Add login/is_adult for onedirectionfanfiction.com

Version 1.5.29 - 07 Jun 2012
New Site: onedirectionfanfiction.com by Ida
New Site: www.prisonbreakfic.net by Ida
New Site: www.storiesofarda.com by Ida

Version 1.5.28 - 06 Jun 2012
Fix new/overwrite missing 'begin' KeyError. Plugin only.

Version 1.5.27 - 05 Jun 2012
Fix for fanfiction.net's latest changes.
Allow chapter range after URLs in Add New.
Preserve META-INF/calibre_bookmarks.txt on epub update.
Option to only create/overwrite cover on new, not update.
Make thehexfiles.net accept www.thehexfiles.net too.
New site: www.ik-eternal.net by Ida.

Version 1.5.26 - 29 May 2012
New Site: www.dokuga.com by Ida.
New Option: Under [txt] wrap_width to control word wrap width in text output.

Version 1.5.25 - 27 May 2012
Fix for ffnet genres, characters, status.

Version 1.5.24 - 25 May 2012
New Site: thehexfiles.net by Ida
New Option: Inject calibre series if no series on site.

Version 1.5.23 - 20 May 2012
New Site: ashwinder.sycophanthex.com by Ida
New Site: chaos.sycophanthex.com by Ida
New Site: erosnsappho.sycophanthex.com by Ida
New Site: lumos.sycophanthex.com by Ida
New Site: occlumency.sycophanthex.com by Ida
New Site: dramione.org by Ida
New Site: www.phoenixsong.net by Ida
New Site: www.walkingtheplank.org by Ida

Version 1.5.22 - 18 May 2012
New Site: www.checkmated.com by Ida

Version 1.5.21 - 18 May 2012
New feature - Option to search epub text for story URL when not found in metadata.
New feature - Option to only run GC on new books, not updates.

Version 1.5.20 - 12 May 2012
(1.5.19 skipped)
New Site: www.libraryofmoria.com by Ida
New Site: www.wraithbait.com by Ida
New Site: www.squidge.org/peja (WWOMB) by Jim (This does not support other sections of www.squidge.org, or the other files under www.squidge.org/peja that aren't in the Fiction instance.)

Version 1.5.18 - 08 May 2012
(1.5.17 skipped)
Allow calls to Generate Cover plugin on create/update.

Version 1.5.16 - 25 Apr 2012
Fix for Yet Another numeric entity issue.

Version 1.5.15 - 22 Apr 2012
Fix for 'Hide this banner' title issue on Archive of Our Own.
Changed implementation of fimfiction to use provided API, thanks to Althaine.

Version 1.5.14 - 17 Apr 2012
Change "Include images in EPUBs?" to effect [epubs], not [defaults].
Kludge for incomplete story image URLs on fimfiction.net

Version 1.5.13 - 11 Apr 2012
New Site: archive.skyehawke.com -- Thanks to Ida Leter for implementing this.
Fix for Python GC'ing menu objects if they don't have an explicit Python reference saved.

Version 1.5.12 - 09 Apr 2012
Set calibre's author link field from authorUrl. Thanks to jackr for pointing out it's existence, and davidfor for showing where to find it.

Version 1.5.11 - 24 Mar 2012
Another work around for when SGMLParser's poor entity handling meets poor HTML.

Version 1.5.10 - 19 Mar 2012
New site: ksarchive.com, thanks to Jade Aislin for implementing this
Fixes for mugglenet.com--added user/pass, better handling for summary and is_adult levels.
Allow default_cover_image to use ${title}, etc; add cover_exclusion_regexp feature to exclude images from being covers.

Version 1.5.9 - 16 Mar 2012
Remove site gayauthors.org following complaints from the site administration.
New Site: nfacommunity.com
New Site: www.midnightwhispers.ca. Thanks to Sam for implementing these.
Make 'Update Calibre Only' add a new book if no matching book found.
Minor improvement to column updates for new stories.
Improve image support for a couple of obscure cases.
portkeyorg.org only - Write 'Chapter does not exist' chapter when chapter is in list, but does not exist.
Add is_adult check to twiwrite for when user on site is not set to adult.
Change td in portkey chapter text to div for nook.

Version 1.5.8 - 06 Mar 2012
New Site: fanfiction.mugglenet.com
New Site: fanfiction.portkey.org
New Site: thequidditchpitch.org
New Site: www.hpfandom.net
Also improved category handling of www.tthfanfic.org.

Version 1.5.7 - 01 Mar 2012
Support for downloading images in the story text.
Support for cover image from three different sources.
Support to resize, make grayscale, and convert images to jpg.
Option for plugin to update calibre cover image on metadata update.
Major re-write of EPUB update to support images.
Improved handling of ini section priority.
New 'Stay on Top' option for 'Add New from URL(s)' dialog for easier drag-n-drop.
Add replace_hr option to replace <hr> tags with '* * *'
Support for fimfiction.net's locked story passwords, including an option to declare locked stories 'failed'.
Add an 'Include images in EPUBs' checkbox to duplicate some personal.ini settings.
nook_img_fix option - wrap img tags with div if not already wrapped with div or p for a nook problem. Defaults to true.

Version 1.4.6 - 24 Feb 2012
New Site: gayauthors.org -- Thanks to Ida Leter for implementing this.
Fix for URL copy/paste from Chrome.
Improvements to metadata for AO3.

Version 1.4.5 - 22 Feb 2012
Fix for ffnet minor format change.

Version 1.4.4 - 21 Feb 2012
Minor fixes to ffnet and fictionalley.org.

Version 1.4.3 - 15 Feb 2012
New Russian language site: ficbook.com -- Thanks to Ida Leter for implementing this.
Language metadata now collected for ficbook.com and fanfiction.net.
Fix Character metadata on fanfiction.net.

Version 1.4.2 - 13 Feb 2012
Fix a bug with HTML entity processing.

Version 1.4.1 - 12 Feb 2012
New feature - Allow user to set custom CSS in personal.ini for HTML and EPUB output.
New feature - Allow user to set custom regular expressions in personal.ini to modify metadata.
New feature - Use Accept-Encoding=gzip to speed download. (Not sites will use it--it's common for sites to block gzip based on User-Agent.)
Add progress bars while collecting URLs from stories for list and for updates.
Make ini edit Courier and 1pt larger than default font.

Version 1.3.4 - 04 Feb 2012
Fixed a bug that caused stories with a footer to crash for AO3. (Thanks, Ida)
Removed sleep from ficwad adapter.
Fixed menu/hotkey update when switching libraries.

Version 1.3.3 - 31 Jan 2012
Better date kludge for fimfiction.net--they don't give years for updated/published.

Version 1.3.2 - 30 Jan 2012
New site: archiveofourown.org -- Thanks for Ida Leter for implementing this.
Change fimfiction.net to use genre instead of category.

Version 1.3.1 - 28 Jan 2012
New feature: Fill Custom Columns with Story Metadata. Look in Config to set it up.
New feature: Collect and update Series column.
Improved drag-and-drop of URLs onto the Add New URL text box. Dragging URLs in adds now instead of replaces, and copies the URL, not the pretty text.
Added a 'show this again' checkbox to the dialog for reviewing updates before applying them. When unchecked, updates will be applied immediately if there were no problems.
Added a progress bar while applying updates, which can be lengthy.
Don't disable menu items(like update) when there are no books selected. It broke the keyboard shortcut if the menu hadn't rebuilt since books were selected.
Make the FanFictionDownLoader button (which does add or update, depending on whether books are selected) available for keyboard shortcut.
Fix date change for adastra.
Fix URL pattern, metadata collection for SIYE.

Version 1.2.3 - 22 Jan 2012
Fix for tz aware vs naive dates.
'Fix' for ffnet look ahead for 1 chapter stories.

Version 1.2.2 - 19 Jan 2012
Fix to 'Keep tags'.
Fix to extratags.
'Fix' for ffnet updates coming out before the web site updates everywhere.

Version 1.2.1 - 17 Jan 2012
Fixes for date updating.
Fix for English name months for users running non-English locale computers.

Version 1.2.0 - 16 Jan 2012
New feature - Options to update Reading List plugin lists for device sync and keeping a list of new books to read.
Plugin configuration now tied to library. Different libraries can have different settings.
Shortcut button in plugin config to dialog for assigning hotkeys.
List of Characters now parsed and saved for most supported sites.

Version 1.1.3 - 10 Jan 2012
Fixed lastupdate feature in include_subject_tags.
Separated Plugin defaults.ini from web/CLI defaults.ini to remove unusable options.
Config option to keep existing tags when updating metadata. Complete/In-Progress will still be updated, and Last Updated tags will be updated if lastupdate in include_subject_tags.

Version 1.1.2 - 09 Jan 2012
Restructure of background processing, improved update of existing books, added SIYE, other fixes.

Version 1.0.3 - 30 Dec 2011
Initial Release on mobileread.com forums.
Attached Files
File Type: zip FanFictionDownLoaderPlugin-2.0.08.zip (829.5 KB, 288 views)

Last edited by JimmXinu; 10-20-2014 at 10:08 PM. Reason: Update to latest version.
JimmXinu is offline   Reply With Quote
Old 12-31-2011, 12:24 AM   #2
Aleyst
Zealot
Aleyst plays well with othersAleyst plays well with othersAleyst plays well with othersAleyst plays well with othersAleyst plays well with othersAleyst plays well with othersAleyst plays well with othersAleyst plays well with othersAleyst plays well with othersAleyst plays well with othersAleyst plays well with others
 
Posts: 107
Karma: 2800
Join Date: Jul 2010
Location: Australia
Device: iPad2, Kindle 4 NT
Thanks for this. It's pretty handy.
Aleyst is offline   Reply With Quote
 
Enthusiast
Old 01-06-2012, 12:47 AM   #3
JimmXinu
Plugin Developer
JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.
 
Posts: 1,719
Karma: 509671
Join Date: Dec 2011
Location: Midwest USA
Device: Nook STR w/Glowlight, Kindle 3g, Droid
FanFictionDownLoader 1.1.0

Posting FanFictionDownLoader plugin 1.1.0, hoping to get some beta testers before pushing it out.

Internally, 1.1 is an almost complete re-write of the plugin from 1.0.

I was advised by more experienced plugin developers that the way I was handling background processing was not particularly safe. Given the number of unexpected crashes I kept getting working with 1.0, I saw their point.

1.1 uses Calibre's mechanism to launch separate background processes to download stories rather than using a background thread.

There is now a drop down menu from the plugin tool bar button for Add/Update/Configure/About.

Also, 'update' truly updates the book(s) selected now. Before the plugin gathered URLs from the selected books, but after that treated the URL as if the user had entered it. If the name or author on the book had been changed, either in Calibre or on the website, it would not update the existing book in Calibre, but treat it as a different book.

The plugin now downloads the stories, but does *not* add/update them in your library until all are downloaded and you confirm you want the updates. Again, this is because background processing updating the foreground is dangerous in Calibre.

There is (hopefully) better download error/update reporting now, but it could still use some more work.

There are more options now for how to update existing stories, whether by directly updating or by doing 'Add New' with a URL to a story that already exists in your library.

Under "Add New from URL(s)", there's an option to choose handling for "If Story Already Exists?". This is also sometimes called collision handling.

"Skip" and "Add New Book" should be self-explanatory.

"Update EPUB if New Chapters" and "Update EPUB Always" allow you to incrementally update a story you already have, but only if you use EPUB. "Update EPUB Always" will update the title page and metadata for the story even if there are no new chapters.

"Overwrite Always" will download the story and overwrite the chosen format. "Overwrite if Newer", only if the website story is newer.

"Update Calibre Metadata Only" does not download the story, only enough metadata to update the metadata Calibre keeps.

Update Existing's "Update Mode" is identical.

Under Configuration, there are some changes and new options.

With the split between "Add New" and "Update Existing", the options to take URLs from selected and prioritize selected vs clipboard are removed. There's still an option to take URLs from clipboard or not.

You can also make the menubar button always bring up "Add New" by unchecking "Default to Update when books selected?".

Jim

(Attachment removed after 1.1.1 posted.)

Last edited by JimmXinu; 01-08-2012 at 03:26 PM.
JimmXinu is offline   Reply With Quote
Old 01-06-2012, 02:47 AM   #4
Aleyst
Zealot
Aleyst plays well with othersAleyst plays well with othersAleyst plays well with othersAleyst plays well with othersAleyst plays well with othersAleyst plays well with othersAleyst plays well with othersAleyst plays well with othersAleyst plays well with othersAleyst plays well with othersAleyst plays well with others
 
Posts: 107
Karma: 2800
Join Date: Jul 2010
Location: Australia
Device: iPad2, Kindle 4 NT
Sounds interesting, I'm just updating the main calibre program now, I'll update the plugin and have a mess around.
Aleyst is offline   Reply With Quote
Old 01-08-2012, 03:21 PM   #5
JimmXinu
Plugin Developer
JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.
 
Posts: 1,719
Karma: 509671
Join Date: Dec 2011
Location: Midwest USA
Device: Nook STR w/Glowlight, Kindle 3g, Droid
FanFictionDownLoader 1.1.1

1.1.1 contains everything from 1.1.0 plus:
  • Fixes for problems found with Add New.
  • Fix to 'delete other formats' when html.
  • Hide 'update epub' options when format not epub.
  • fanfic.castletv.net changed to castlefans.org/fanfic.
  • Fix for twiwrite adult-only stories.

I think this is ready to be pushed out as the new version, but I'd appreciate it if I could get a tester or two to try it first.

Jim

Last edited by JimmXinu; 01-21-2012 at 11:59 AM. Reason: Remove obsolete beta versions
JimmXinu is offline   Reply With Quote
Old 01-08-2012, 03:37 PM   #6
scoopaz
Nosy Parker
scoopaz began at the beginning.
 
scoopaz's Avatar
 
Posts: 32
Karma: 10
Join Date: May 2010
Location: Valley of the Sun
Device: EZ Reader Pro, Nook Classic, Touch, Color, PRS-650, Asus Transformer
Exclamation

Hi Jim,

Went to give you plugin a shot since I'm a big collector of fanfiction and use the web version extensively. When trying to download a fanfic from fanfiction.net I received this error message:

Traceback (most recent call last):
File "site-packages\calibre\gui2\__init__.py", line 340, in dispatch
File "calibre_plugins.fanfictiondownloader_plugin.ffdl_ plugin", line 544, in download_list_completed
File "calibre_plugins.fanfictiondownloader_plugin.ffdl_ plugin", line 589, in _add_or_update_book
File "site-packages\calibre\library\database2.py", line 3175, in create_book_entry
File "site-packages\calibre\library\database2.py", line 3171, in _add_newbook_tag
AttributeError: 'set' object has no attribute 'append'

I'm using the latest version of Calibe (0.8.34) on a Windows 7 machine.

Please help! I would love to have this plugin working.

Lisa
scoopaz is offline   Reply With Quote
Old 01-08-2012, 03:46 PM   #7
JimmXinu
Plugin Developer
JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.
 
Posts: 1,719
Karma: 509671
Join Date: Dec 2011
Location: Midwest USA
Device: Nook STR w/Glowlight, Kindle 3g, Droid
Quote:
Originally Posted by scoopaz View Post
Went to give you plugin a shot since I'm a big collector of fanfiction and use the web version extensively. When trying to download a fanfic from fanfiction.net I received this error message:

I'm using the latest version of Calibe (0.8.34) on a Windows 7 machine.

Please help! I would love to have this plugin working.
@scoopaz, I'm downloading 0.8.34 now--I was still on 0.8.33.

What's the URL of the story that gave the error? Does it happen for all, or just one story?

Jim
JimmXinu is offline   Reply With Quote
Old 01-08-2012, 04:00 PM   #8
scoopaz
Nosy Parker
scoopaz began at the beginning.
 
scoopaz's Avatar
 
Posts: 32
Karma: 10
Join Date: May 2010
Location: Valley of the Sun
Device: EZ Reader Pro, Nook Classic, Touch, Color, PRS-650, Asus Transformer
Hi Jim,

thanks for the quick response!. The URL for the story I initially tried is :

http://www.fanfiction.net/s/7623520/1/The_Good_Brother

so I tried a few more on fanfiction.net, and got the same error. Went to www.whofic.com just to confirm with a generic (non-adult) story:

http://www.whofic.com/viewstory.php?sid=29654
and received the error again.

Lisa
scoopaz is offline   Reply With Quote
Old 01-08-2012, 04:01 PM   #9
JimmXinu
Plugin Developer
JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.
 
Posts: 1,719
Karma: 509671
Join Date: Dec 2011
Location: Midwest USA
Device: Nook STR w/Glowlight, Kindle 3g, Droid
@scoopaz, I've figured it out.

It's a bug in Calibre, not the plugin.

There's a problem in the routine that creates new book entries with "Tags to apply when adding a book" set.

I'll report it in the calibre bug tracking system.

For now, if you remove your "Tags to apply when adding a book", it will work.

Even if you don't want to do that for regular use, please give it a try and confirm that it solves your problem.

Jim
JimmXinu is offline   Reply With Quote
Old 01-08-2012, 04:05 PM   #10
scoopaz
Nosy Parker
scoopaz began at the beginning.
 
scoopaz's Avatar
 
Posts: 32
Karma: 10
Join Date: May 2010
Location: Valley of the Sun
Device: EZ Reader Pro, Nook Classic, Touch, Color, PRS-650, Asus Transformer
More info from the FFdownloader Job

Here's the data from the FFdownloader job:

Download FanFiction Book
Failed to initialize plugin: u'C:\\Users\\scoopaz\\AppData\\Roaming\\calibre\\p lugins\\Goodreads Covers.zip'
Failed to initialize plugin: u'C:\\Users\\scoopaz\\AppData\\Roaming\\calibre\\p lugins\\Goodreads Metadata.zip'
FanFictionDownLoader v1.1.1
Logfile for book ID None (The Good Brother)
urlhttp://www.fanfiction.net/s/7623520/1/) idNone)
Failed to initialize plugin: u'C:\\Users\\scoopaz\\AppData\\Roaming\\calibre\\p lugins\\Goodreads Covers.zip'
Failed to initialize plugin: u'C:\\Users\\scoopaz\\AppData\\Roaming\\calibre\\p lugins\\Goodreads Metadata.zip'
write to c:\users\scoopaz\appdata\local\temp\calibre_0.8.34 _tmp_1ijjwz\fanfictiondownloader_czx5j7\new-None-lzgitu.epub
File(c:\users\scoopaz\appdata\local\temp\calibre_0 .8.34_tmp_1ijjwz\fanfictiondownloader_czx5j7\new-None-lzgitu.epub) Updated(2012-01-08) more recently than Story(2012-01-07) - Skipping


Lisa
scoopaz is offline   Reply With Quote
Old 01-08-2012, 04:12 PM   #11
scoopaz
Nosy Parker
scoopaz began at the beginning.
 
scoopaz's Avatar
 
Posts: 32
Karma: 10
Join Date: May 2010
Location: Valley of the Sun
Device: EZ Reader Pro, Nook Classic, Touch, Color, PRS-650, Asus Transformer
More issues...

Hi Jim,

Just saw your post - removed my auto tag when importing and gave it another shot:

The good news: The ebook was created and imported into my library with no errors:

The bad news: I received a different error when trying to open the created epub:

calibre, version 0.8.34
ERROR: Could not open ebook: File is not a zip file

Traceback (most recent call last):
File "site-packages\calibre\gui2\viewer\main.py", line 64, in run
File "threading.py", line 483, in run
File "site-packages\calibre\ebooks\oeb\iterator.py", line 205, in __enter__
File "site-packages\calibre\customize\conversion.py", line 204, in __call__
File "site-packages\calibre\ebooks\epub\input.py", line 138, in convert
File "site-packages\calibre\utils\zipfile.py", line 751, in __init__
File "site-packages\calibre\utils\zipfile.py", line 786, in _GetContents
File "site-packages\calibre\utils\zipfile.py", line 801, in _RealGetContents
BadZipfile: File is not a zip file

In Calibre the size of the file shows as 0.00 kb

Lisa
scoopaz is offline   Reply With Quote
Old 01-08-2012, 04:21 PM   #12
JimmXinu
Plugin Developer
JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.
 
Posts: 1,719
Karma: 509671
Join Date: Dec 2011
Location: Midwest USA
Device: Nook STR w/Glowlight, Kindle 3g, Droid
Quote:
Originally Posted by scoopaz View Post
Just saw your post - removed my auto tag when importing and gave it another shot:

The good news: The ebook was created and imported into my library with no errors:

The bad news: I received a different error when trying to open the created epub:

File "site-packages\calibre\utils\zipfile.py", line 801, in _RealGetContents
BadZipfile: File is not a zip file

In Calibre the size of the file shows as 0.00 kb
Lisa,

Well, that's not supposed to happen... But on the other hand, it is farther along and therefore progress.

Please try it again making sure that you're adding from the URL and have 'epub' and 'Add New' selected. If it still makes a zero length epub, please post the details from the Download job.

Thanks,

Jim
JimmXinu is offline   Reply With Quote
Old 01-08-2012, 04:32 PM   #13
scoopaz
Nosy Parker
scoopaz began at the beginning.
 
scoopaz's Avatar
 
Posts: 32
Karma: 10
Join Date: May 2010
Location: Valley of the Sun
Device: EZ Reader Pro, Nook Classic, Touch, Color, PRS-650, Asus Transformer
info on bad ebook file

Hi Jim,

OK, Made sure it was set to add new ebook, and add new from URL and no go.

Here is the results from the job:

Download FanFiction Book
Failed to initialize plugin: u'C:\\Users\\scoopaz\\AppData\\Roaming\\calibre\\p lugins\\Goodreads Covers.zip'
Failed to initialize plugin: u'C:\\Users\\scoopaz\\AppData\\Roaming\\calibre\\p lugins\\Goodreads Metadata.zip'
FanFictionDownLoader v1.1.1
Logfile for book ID None (The Good Brother)
urlhttp://www.fanfiction.net/s/7623520/1/) idNone)
Failed to initialize plugin: u'C:\\Users\\scoopaz\\AppData\\Roaming\\calibre\\p lugins\\Goodreads Covers.zip'
Failed to initialize plugin: u'C:\\Users\\scoopaz\\AppData\\Roaming\\calibre\\p lugins\\Goodreads Metadata.zip'
write to c:\users\scoopaz\appdata\local\temp\calibre_0.8.34 _tmp_o2hwch\fanfictiondownloader_9vuddg\new-None-ivsafd.epub
File(c:\users\scoopaz\appdata\local\temp\calibre_0 .8.34_tmp_o2hwch\fanfictiondownloader_9vuddg\new-None-ivsafd.epub) Updated(2012-01-08) more recently than Story(2012-01-07) - Skipping

I tried uninstalling the Goodreads plugin and tried again with no luck. could something be wonky with my configuration?


Lisa
scoopaz is offline   Reply With Quote
Old 01-08-2012, 04:42 PM   #14
kiwidude
calibre/Sigil Developer
kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.kiwidude ought to be getting tired of karma fortunes by now.
 
Posts: 4,228
Karma: 1334002
Join Date: Oct 2010
Location: London, UK
Device: Kindle Paperwhite 3G, iPad 3, iPad Air
@scoopaz - the two Goodreads plugin messages are due to you having installed two plugins a long time ago that got superceded by the single Goodreads plugin. They have nothing to do with any behavior with the FanFictionDownloader plugin.

It is just calibre telling you that it is not possible to load those plugins (because they arent compatible with recent calibre versions, hence they were replaced with a single "Goodreads" plugin).

To get rid of the error messages, see the couple of posts here. You can either do it via the command line or as dwnathny says just delete the relevant plugin zip files from the calibre plugins folder. Just make sure you do the right ones - as per your error message text they are "Goodreads Covers" and "Goodreads Metadata" that are the plugins you should remove, and not the "Goodreads" one.
kiwidude is offline   Reply With Quote
Old 01-08-2012, 04:56 PM   #15
JimmXinu
Plugin Developer
JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.
 
Posts: 1,719
Karma: 509671
Join Date: Dec 2011
Location: Midwest USA
Device: Nook STR w/Glowlight, Kindle 3g, Droid
And I was getting ready to blame it all on Goodreads plugins. :-)

Actually, Lisa, your last log posting gave me the answer.

I've replaced the zip file on the "FanFictionDownLoader 1.1.1" post above with a fixed one. Please give that one a try.

Thanks,

Jim
JimmXinu is offline   Reply With Quote
Reply

Thread Tools Search this Thread
Search this Thread:

Advanced Search

Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
[GUI Plugin] Count Pages kiwidude Plugins 668 Yesterday 10:13 AM
[GUI Plugin] Open With kiwidude Plugins 232 10-09-2014 12:38 AM
[GUI Plugin] Find Duplicates kiwidude Plugins 508 09-02-2014 07:00 PM
[GUI Plugin] Resize Cover kiwidude Plugins 34 07-13-2014 07:15 PM
[GUI Plugin] Plugin Updater **Deprecated** kiwidude Plugins 159 06-19-2011 12:27 PM


All times are GMT -4. The time now is 11:49 PM.


MobileRead.com is a privately owned, operated and funded community.