View Full Version : [GUI Plugin] FanFictionDownLoader


Pages : [1] 2 3 4 5 6 7 8 9 10 11 12 13 14 15

JimmXinu
12-30-2011, 04:32 PM
Many people use their ebook readers to read FanFiction stories posted on the web on various sites. Many people also use Calibre to manage their ebook collections and readers.

This plugin brings to functionality of FanFiction Downloader (http://fanfictiondownloader.appspot.com/) to Calibre. The source is available on the project home page (http://code.google.com/p/fanficdownloader/), and we monitor our user group (http://groups.google.com/group/fanfic-downloader) for problems. There are additional documentation and FAQs on the project wiki (http://code.google.com/p/fanficdownloader/w/list).

Main Features of FanFictionDownLoader Plugin:

Download FanFiction stories from over 100 different sites into ebooks. Supported sites list is below.
Create new and update existing epub anthologies of several stories. (Uses EpubMerge Plugin)
Download in the background for user convenience.
Support for downloading images in the story text. (EPUB only--download EPUB and convert to MOBI for Kindle) More details on configuring images in stories and cover images can be found in this post (http://www.mobileread.com/forums/showthread.php?p=1982785#post1982785).
Support for cover image from three different sources. (EPUB only)
Support to resize, make grayscale, and convert images to jpg. (EPUB only)
Support to call Generate Cover (http://www.mobileread.com/forums/showthread.php?t=124219) plugin on calibre metadata create or update. Requires Generate Cover 1.5.1 or newer.
Support to call Count Pages (http://www.mobileread.com/forums/showthread.php?t=134000) plugin on calibre metadata create or update. Requires Count Pages 1.6.0 or newer.
Update/Overwrite existing FanFiction stories from story URL in Calibre identity or epub. FFDL ignores Virtual Library and Additional Restrictions while searching for books to update.
Update existing epub format ebooks, downloading only new chapters.
Optionally keep an Update Log of past updates (epub only).
Update calibre metadata from website.
Option for plugin to update calibre cover image on metadata update.
Option to delete other formats on book update. Handy if you have both a Nook(epub) and Kindle(mobi), for example.
Get original URLs from FanFiction stories in your library.
Update Reading List (http://www.mobileread.com/forums/showthread.php?t=134856) plugin lists as an aid to device sync and keeping a list of new books to read. Requires Reading List plugin 1.4.1 or newer to use.
Update calibre custom columns with story metadata.
Get Story URLs from Web Page
Options now stored inside the Library rather than an external JSON file.
Keep a list of previously 'Rejected' story URLs and user notes on why. Report if the user tries to download a rejected story again.
CLI via calibre-debug --run-plugin
Option to call Polish Book's Smarten Punctuation on each epub download/update. (calibre 0.9.39 and newer only.)


Supported Sites:


archive.skyehawke.com
archiveofourown.org
ashwinder.sycophanthex.com
asr3.slashzone.org
bdsm-geschichten.net
bloodshedverse.com
bloodties-fans.com
buffynfaith.net
chaos.sycophanthex.com
csi-forensics.com
dark-solace.org
dramione.org
efiction.esteliel.de
erosnsappho.sycophanthex.com
fanfic.castletv.net
fanfic.hu
fanfic.potterheadsanonymous.com
fanfiction-junkies.de
fanfiction.csodaidok.hu
fanfiction.mugglenet.com
fanfiction.portkey.org
fanfiction.tenhawkpresents.com
fannation.shades-of-moonlight.com
fhsarchive.com
fictionmania.tv
fictionpad.com
ficwad.com
finestories.com
grangerenchanted.com
hlfiction.net
imagine.e-fic.com
indeath.net
ksarchive.com
literotica.com
lotrfanfiction.com
lumos.sycophanthex.com
merlinfic.dtwins.co.uk
national-library.net
ncisfic.com
netraptor.org
nfacommunity.com
nha.magical-worlds.us
nocturnal-light.net
occlumency.sycophanthex.com
onedirectionfanfiction.com
pommedesang.com
ponyfictionarchive.net
pretendercentre.com
samandjack.net
samdean.archive.nu
scarhead.net
sg1-heliopolis.com
sheppardweir.com
spikeluver.com
stargate-atlantis.org
storiesonline.net
test1.com
thehexfiles.net
thehookupzone.net
themaplebookshelf.com
themasque.net
thequidditchpitch.org
tokra.fandomnet.com
tolkienfanfiction.com
trekiverse.org
voracity2.e-fic.com
www.adastrafanfic.com
www.checkmated.com
www.destinysgateway.com
www.dokuga.com
www.dotmoon.net
www.dracoandginny.com
www.efpfanfic.net
www.fanfiction.net
www.fanfiktion.de
www.ficbook.net
www.fictionalley.org
www.fictionpress.com
www.fimfiction.net
www.harrypotterfanfiction.com
www.henneth-annun.net
www.hpfandom.net
www.hpfanficarchive.com
www.ik-eternal.net
www.libraryofmoria.com
www.mediaminer.org
www.midnightwhispers.ca
www.ncisfiction.net
www.nickandgreg.net
www.phoenixsong.net
www.potionsandsnitches.net
www.potterfics.com
www.psychfic.com
www.qaf-fic.com
www.restrictedsection.org
www.scarvesandcoffee.net
www.simplyundeniable.com
www.sinful-desire.org
www.siye.co.uk
www.squidge.org/peja
www.storiesofarda.com
www.thealphagate.com
www.thepetulantpoetess.com
www.tthfanfic.org
www.twcslibrary.net
www.twilightarchives.com
www.twilighted.net
www.twiwrite.net
www.walkingtheplank.org
www.whofic.com
www.wizardtales.net
www.wolverineandrogue.com
www.wraithbait.com



Other useful Plugins:

FFDL can call Generate Cover (http://www.mobileread.com/forums/showthread.php?t=124219) to generate custom covers for downloaded stories.
FFDL can call Count Pages (http://www.mobileread.com/forums/showthread.php?t=134000) to generate page counts for downloaded stories.
FFDL can call Reading List (http://www.mobileread.com/forums/showthread.php?t=134856) to sync devices and keep a list of new books to read.
FFDL can call EpubMerge (http://www.mobileread.com/forums/showthread.php?t=1973346) to create and update Epub anthologies of several stories in one book.
Kindle 2, 3, 4, Touch Device Interface MBP Update Mod (http://www.mobileread.com/forums/showthread.php?t=186376) preserves your place in MOBI format books on Kindle when they are updated.


Special Notes:

Requires Calibre v1.48.0 or later.


Languages:

This plugin has translations for:

English
French
German
Portuguese (Brazil)
Spanish


Command Line Interface(CLI)

Calibre offers a command-line interface (CLI) for calling plugins.

This plugin supports that option and allows the user to call the plugin's basic functionality from outside calibre without having to install Python or get the plugin's files separately.

(Note that at this time, FFDL via calibre CLI will use your personal.ini settings from your library, but does not add to or update your library.)

Details:

Usage: calibre-debug.exe --run-plugin FanFictionDownLoader -- [options] <storyurl>

Options:
-h, --help show this help message and exit
--library-path=LIBRARY_PATH, --with-library=LIBRARY_PATH
Path to the calibre library. Default is to use the
path stored in the settings.
-f FORMAT, --format=FORMAT
write story as FORMAT, epub(default), mobi, text or
html
-c CONFIG, --config=CONFIG
read config from specified file(s) in addition to
calibre plugin personal.ini,
~/.fanficdownloader/personal.ini, and ./personal.ini
-b START, --begin=START
Begin with Chapter START
-e END, --end=END End with Chapter END
-o NAME=VALUE, --option=NAME=VALUE
set an option NAME=VALUE
-m, --meta-only Retrieve metadata and stop. Or, if --update-epub,
update metadata title page only.
-u, --update-epub Update an existing epub with new chapters, give epub
filename instead of storyurl.
--update-cover Update cover in an existing epub, otherwise existing
cover (if any) is used on update. Only valid with
--update-epub.
--force Force overwrite of an existing epub, download and
overwrite all chapters.
-l, --list Get list of valid story URLs from page given.
-d, --debug Show debug output while downloading.


Installation Steps:

Download the attached zip file and install the plugin/add to context menu or toolbar/restart Calibre as described in the Introduction to plugins thread (http://www.mobileread.com/forums/showthread.php?t=118680).
Click to plugin button, choose your settings, enter URLs to FanFiction stories, and click 'Download'. The plugin will create or update the ebook in Calibre.


Version History:

Version 2.1.00 - 10 Dec 2014
The big change this version is the inclusion of new HTML parsing libraries BS4, html5lib and a couple supporting packages. Not all sites are parsed using the new parser, but several are including fanfiction.net and AO3.

New Site: fanfiction-junkies.de--thanks scout78
Fixes for FimFiction.net changes--thanks FaceDeer.
Update for csiforensics.com adapter--make site specific instead of Base eFiction--thanks scout78.
Add new site fanfiction-junkies.de--thanks scout78.
Rename castlefans.org to fanfic.castletv.net
Known issue: Specific metadata 'eroticatags' for literotica.com doesn't work on all stories.
Known issue: Metadata collection is not as complete for 'Base eFiction' adapters.


Version 2.0.10 - 18 Nov 2014
New Site: fhsarchive.com -- eFiction Base adapter.
Fixes for storiesonline.net site changes--'codes' are now 'sitetags', thanks Jeff.
Fix for literotica.com HTML.
Fix for AO3 fetch after login.
Fix for User-Agent with saved cookie jar.
Fix for ffnet adapter for 'get urls from page'.
Fix for images in FimFiction.net stories.
Fix handling of new books and custom_column_settings.
Fix for fimf not working with manual is_adult (caching issue).
Fix for calibre 2.10 keyboard shortcuts change.
Known issue: Specific metadata 'eroticatags' for literotica.com doesn't work on all stories.
Known issue: Metadata collection is not as complete for 'Base eFiction' adapters.

Version 2.0.9 - 03 Nov 2014
Fix for changed login behavior for the fanfiktion.de adapter (Thanks, cryzed).
Changes for potterfics.com(Spanish) to login for adult rated stories.
Updates for Fimfiction.net changes, fixes for image inclusion and password protected stories (Thanks Facedeer and cryzed).
Change thewriterscoffeeshop.com to twcslibrary.net. (you'll need to change your personal.ini from [www.thewriterscoffeeshop.com] to [www.twcslibrary.net] if you saved your TWCS login.)
Fix for ficwad.com adult rated stories to properly log in.
Improved error reporting for regular expressions in personal.ini.

Version 2.0.8 - 20 Oct 2014
Additional fix for site updates to: ficwad.com

Version 2.0.7 - 20 Oct 2014
Fixes for site updates to: fimfiction.net (thanks, facedeer)
Fixes for site updates to: ficwad.com
Fixes for site updates to: archiveofourown.org
Known issue: Metadata collection is not as complete for 'Base eFiction' adapters.
Known problem: Password protected FimFiction.net stories aren't working. FimF changed API access.
Known problem: specific metadata 'eroticatags' for literotica.com doesn't work on all stories.

Version 2.0.6 - 06 Oct 2014
New Site: lotrfanfiction.com
New Site: csi-forensics.com, thanks scout78.
New Site: samandjack.net, thanks scout78.
Fixes for sheppardweir.com, thanks scout78.
Fixes for Base eFiction.
Known issue: Metadata collection is not as complete for 'Base eFiction' adapters.
Known problem: Password protected FimFiction.net stories aren't working. FimF changed API access.
Known problem: specific metadata 'eroticatags' for literotica.com doesn't work on all stories.

Version 2.0.5 - 23 Sep 2014
- Fix for AO3 get story list URLs that already have a '?' in them.
- Fix for squidge.org/peja using a story URL for 'Site Map'. (Messed up series and series numbers.)
- Fixes for storiesonline.net site update, from davidfor.
- Add 'extratags' to AllMetadata so it's available for custom columns.
- Only support from calibre v1.48.0 and up for plugin.
- Known problem: Password protected FimFiction.net stories aren't working. FimF changed API access.
- Known problem: specific metadata 'eroticatags' for literotica.com doesn't work on all stories.

Version 2.0.4 - 09 Sep 2014
- New site bdsm-geschichten.net, thanks John Doe.
- New site: sheppardweir.com, thanks scout78.
- New eFiction base adapter courtesy of John Doe. This should make it easier for new developers to add eFiction type sites. But it doesn't support all type of metadata.
- New site using eFiction base adapter: fannation.shades-of-moonlight.com, thanks John Doe.
- New site using eFiction base adapter: themaplebookshelf.com, thanks John Doe.
- Fix numChapters in adapter_literotica.py.
- Fixes for nhamagicalworldsus changes.
- Fix for changes to fanfiktion.de.
- Add/update translations in Spanish, French, Portuguese (Brazil), German.
- Fix so autoconvert won't delete FFDL's own output.
- Add cookie and page caching between stories and between front and background processing for selected sites: fanfiction.net, tthfanfic.org, archiveofourown.org, fanfiktion.de, fimfiction.net, portkey.org and buffynfaith.net.

Version 2.0.3 - 13 Aug 2014
Fix for AO3 authorUrls and authorIds.

Version 2.0.2 - 13 Aug 2014
Fix for AO3 authors all coming out Anonymous.

Version 2.0.1 - 04 Aug 2014
- New German language site bdsm-geschichten.net, thanks to doe5716.
- New site tolkienfanfiction.com, thanks to doe5716.
- Add other languages for literotica.com, thanks to doe5716.
- Fixes for bloodshedverse & spikelover having html tags in title, etc.
- Fix for non-split list replace_metadata.
- Fix for anthology books, don't put author in default anthology comments if only one author.
- Fix to plugin identifiers:"~ur(i|l)..." search string for finding existing books.
- Known problem: Password protected FimFiction.net stories aren't working. FimF changed API access.
- Known problem: specific metadata 'eroticatags' for literotica.com doesn't work on all stories.

Version 2.0.0 - 23 Jul 2014
Now supporting Qt5 and Calibre 2.0 (currently in beta).
Translations updates to French and German and new translation Spanish, courtesy of the volunteers at Transifex.
Apply cover_exclusion_regexp to specific cover images, too
Add site specific reviews to wraithbait.com
Allow ffnet story specific covers by default
Add "Reject Without Confirmation" feature (plugin only)
Add Spanish translation (plugin only)
Change default encoding for bloodshedverse.com to Windows-1252
Allow chapterless Fimf stories
Fix for login needed for efpfanfic.net 'red' rated stories
Known problem: Password protected FimFiction.net stories aren't working. FimF changed API access.
Known problem: specific metadata 'eroticatags' for literotica.com doesn't work on all stories.

Version 1.8.26 - 25 Jun 2014
Fixes for some metadata problems on various sites.
Known problem: specific metadata 'eroticatags' for literotica.com doesn't work on all stories.

Version 1.8.25 - 21 Jun 2014
New site: fictionmania.tv -- Thanks, cryzed!
Fix for site recognizer to handle with/without www. using https URLs.
Fix for some utf8 descriptions on fimf.
Site specific metadata 'eroticatags' for literotica.com.
Fix for 'a' flag on custom_columns_settings not working as intended.
Changes to '\,' to better implement split list feature to avoid infinite recursions.
Fix for calibre-injected series--don't treat series as a list, it isn't.
Auto-add http: to URLs starting with //.
Switch to transifex.com for translations.

Version 1.8.24 - 14 Jun 2014
Now supporting over 100 different sites! Thanks, cryzed, for pushing us over the top.
New site: nocturnal-light.net -- Thanks, cryzed!
New site: fanfic.hu (Hungarian language) -- Thanks, cryzed!
New site: fanfiction.csodaidok.hu (Hungarian language) -- Thanks, cryzed!
Updates for sites: bloodshedverse.com, voracity2.e-fic.com -- Thanks, cryzed!
Fix for making anthology books when source books don't have number of words.
New '\,' feature to 'split' metadata items in replace_metadata.
Example:
replace_metadata:
category=>Captain America \(Movies\)=>Captain America\,Marvel
The category list will then contain both "Captain America" and "Marvel".

Version 1.8.23 - 07 Jun 2014
New site: spikeluver.com -- Thanks, cryzed!
New site: bloodshedverse.com -- Thanks, cryzed!
Updates for site: voracity2.e-fic.com -- Thanks, cryzed!

Version 1.8.22 - 04 Jun 2014
New site: voracity2.e-fic.com -- Thanks, cryzed!

Version 1.8.21 - 23 May 2014
Fix for dark-solace.org Rating.
Fix for literotica.com URLs.

Version 1.8.20 - 13 May 2014
Fix for AO3 story not found.
Fix for storiesonline.net changing urls.
Fix for fictionpad.com removing 'dislikes' in some(all?) cases.
Fix for dark-solace.org metadata parsing.

Version 1.8.19 - 05 May 2014
Allow https URLs for fimfiction.net.
Both allow https URLs and change canonical URLs for squidge.org/peja to https.
Fix for some stories' summaries on onedirectionfanfiction.com.
Add include/exclude metadata feature. And wiki page (https://code.google.com/p/fanficdownloader/wiki/InExcludeMetadataFeature) for it.
Slightly improved connection refused handling.

Version 1.8.18 - 19 Apr 2014
Added workaround for an issue with fimfiction.net and blockquotes. Can be turned off by adding fix_fimf_blockquotes:false to the [www.fimfiction.net] section in personal.ini if it causes problems.

Version 1.8.17 - 09 Apr 2014
Fix for a problem with literotica.com multi-page chapters on Kobo readers.

Version 1.8.16 - 29 Mar 2014
Add Pairing for hpfanficarchive.com

Remove some outdated checks against calibre versions that aren't supported now.

Version 1.8.15 - 16 Mar 2014
Fix for fictionpress.com https urls.

Add options to turn off author notes on AO3 chapters.

Version 1.8.14 - 09 Mar 2014
Fix for potionsandsnitches.net, extra text after date.

Change Job Log output so URLs are more copy/pasteable.

Configurable connection timeout as per FaceDeer's recommendation.

Fix for seriesUrl check.

Fix for AO3 https urls.

On auto-convert, delete target format else convert doesn't run.

Version 1.8.13 - 28 Feb 2014
Fix for https vs http on ffnet (Story does not exist) problem.

Fix for KeyError:'site' problem.

Version 1.8.12 - 24 Feb 2014
Fixes for FimFiction change, thanks FaceDeer.

Set Boolean custom columns from custom_columns_settings, looking for true/t/yes/y/1 vs false/f/no/n/0.

Add a feature to reduce sleep between ffnet requests in initial metadata collection for small download sets.

Make ffnet look-for-next chapter feature optional and default to off.

Version 1.8.11 - 15 Feb 2014
Fix for fanfiction.net cover download. This is turned off by default in CLI and plugin due to how many stories in ffnet show the author's picture. In section [www.fanfiction.net] add never_make_cover: false to turn covers on.

Add feature to automatically inject cover from Generate Cover into epub using calibre's Polish feature.

Don't do auto-convert when Updating Calibre Metadata Only.

Version 1.8.10 - 5 Feb 2014
Fix French translation typo that prevents download.
Add auto-convert on download/update feature.
Fix divide by zero error in replace_br_with_p on chapters with no content.
Change to storiesonline.net by davidfor to address overzealous culling in text.

Version 1.8.9 - 22 Jan 2014
Leading zeros in version number makes it octal. Remove and bump to 1.8.9. :smack:

Version 1.8.08 - 21 Jan 2014
Allow literal strings in custom_columns_settings to set columns.

Version 1.8.07 - 13 Jan 2014
Change default encoding for finestories.com.
Change default encoding and improve metadata for storiesonline.net, thanks davidfor.
Fixes for hpfanficarchive.com changes.
Fix for 'update' with no updates and 'mark' enabled.
Now with French translation, thanks Ptitprince.

Version 1.8.06 - 23 Dec 2013
Fix for latest ffnet change.
Additional sleep added to ffnet metadata downloads.
New site: literotica.com. Thanks to de3sw2aq1.
Tweak to htmlheuristics for literotica.com (unicode vs str).

Version 1.8.05 - 12 Dec 2013
Fix issue with AO3 single chapter stories introduced in 1.8.04

Version 1.8.04 - 12 Dec 2013
(v1.8.03 was a testing only version)
Raise minimum calibre version due to changes in identifier:url searching
Alternate URL efiction.trekiverse.org for trekiverse.org/efiction
Add do_update_hook to AO3 for updated chapters in the middle
Reorder defaults.ini files to put user/pass is_adult first in all sections
Graceful fail for 'k'HTML when no 'k'Url

Version 1.8.02 - 05 Dec 2013
New Site: trekiverse.org/efiction
Fix for higher rated stories with not logged in on tthfanfic.org
Add prequels/sequels &amp; fix groups site specific metadata for fimfiction.net
Now including German translation, thanks ILB!
Fix for identifier searchs and Overwrite if Newer with ffnet timestamps.

Version 1.8.01 - 18 Nov 2013
Fix for stories with more than 999 chapters.
Fixes/Enhancements for replace_br_with_p heuristic processing feature. Thanks, Asbjørn Grandt.
Fix for dc:language when none given and defaulting to English.
Fix for 'Default' Generate Cover setting with Internationalization/Localization.

Version 1.8.00 - 18 Nov 2013
Internationalization/Localization code added. If people volunteer to do translations, they'll be able to use FFDL in their own languages.
More fanfiction.net fixes. Don't included Updated/Published as characters, don't include excess Misc/Movies/etc categories on crossovers, start using https instead of http.
Improved error messaging for ficwad & squidge.org/peja

Version 1.7.54 - 13 Nov 2013
Fix for fanfiction.net more changes.

Version 1.7.53 - 11 Nov 2013
Fix for fanfiction.net changes.
Fixes/Enhancements for replace_br_with_p heuristic processing feature. Thanks, Asbjørn Grandt.

Version 1.7.52 - 09 Nov 2013
Fixes for storiesonline.net, thanks davidfor.
Fixes/Enhancements for replace_br_with_p heuristic processing feature. Thanks, Asbjørn Grandt.

Version 1.7.51 - 30 Oct 2013
New site: storiesonline.net
Add replace_br_with_p heuristic processing feature. Thanks, Asbjørn Grandt.
Handling for fictionpad.com empty chapters. "This chapter has no text."
Handling for fanfiction.net's 'Pairings'.

Version 1.7.50 - 23 Oct 2013
Fix for ffnet changes.

Version 1.7.49 - 19 Oct 2013
Fixes for dark-solace.org
Tweaks to Config dialog for small screens

Version 1.7.48 - 13 Oct 2013
Fix for an HTML change on fanfiction.net

Version 1.7.47 - 13 Oct 2013
New site: fictionpad.com
Debug/information output reduced and changed to logging
New Basic option "Mark added/updated books when finished?"

Version 1.7.46 - 29 Sept 2013
Fixes for indeath.net - Thanks Besnef!
Better Non-BtVS/AtS detection for tthfanfic.org
do_update_hook for fimf, find newest chapter and update from there when true.

Version 1.7.45 - 21 Sept 2013
Fix for whofic.com.
Don't include empty string dc:subject tags in epub.
Set custom column only if there's a value (mostly for int/float columns).

Version 1.7.44 - 14 Sept 2013
Remove dateutil dependency
Fix for certain portkey.org stories.
Fix for PI for int/float custom columns setting empty.

Version 1.7.43 - 11 Sept 2013
Fix dates for fimfiction.net
Additional groups as site specific data for fimfiction.net.
Add default_value_status (etc) feature.

Version 1.7.42 - 09 Sept 2013
Longer sleep for ffnet, config User-agent

Version 1.7.41 - 08 Sept 2013
Fix for AO3 stories without any series.

Version 1.7.40 - 07 Sept 2013
Better doc section override order in ini files.
Fix for calibre pre-1.0.0.
Additional series as site specific data for AO3.
Fixes for changes to harrypotterfanfictioncom.
Add User-agent="FFDL/1.7" for all adapters for fanfiction.net changes.
(Remove from specific adapters.)

Version 1.7.39 - 31 Aug 2013
Remove work around for last_modified vs author link update issue in calibre 1.0.0--fixed in 1.1.0.
New Feature: add_genre_when_multi_category setting
Work around calibre's auto-split when author contains 'and' 'with' ',' or '&'.

Version 1.7.38 - 24 Aug 2013
Work around for last_modified vs author link update issue in calibre 1.0.0.
Better error reporting of metadata update problems.

Version 1.7.37 - 24 Aug 2013
Work around minor bug (1216398) in calibre 1.0.0. Tags 'A' and 'a' (for example) conflict and prevent metadata update.

Version 1.7.36 - 20 Aug 2013
Make check for Story URL change configurable.
Add configurable options to keep Title/Author the same in Title/Author Sort.

Version 1.7.35 - 06 Aug 2013
Remove some debugging output.
Minor optimizations for adapter lookup.
Get story url lists from sites with adapter, get urls from non-fic sites in CLI.
Warning on update if existing epub has no recognizable chapters.

Version 1.7.34 - 28 Jul 2013
Fixes for fanfiktion.de, storiesofarda.com, hpfanficarchive.com, and dark-solace.org/elysian sites.

Version 1.7.33 - 24 Jul 2013
Fix for smarten_punctuation setting in pre-0.9.39 calibre.

Version 1.7.32 - 23 Jul 2013
Fixes for fanfiktion.de & thehexfiles.net, add entries to teststory valid list.

Version 1.7.31 - 18 Jul 2013
Drag and drop onto the FFDL plugin button
Smarten Punctuation
sort_ships option
join_string_<entry> option
keep_in_order_<entry> option
replace_metadata <entry>_LIST options

Version 1.7.30 - 09 Jul 2013
New Site: scarhead.net - Thanks Ida!
Improved error handling when ffnet is having problems.
Phoenixsong can have individual chapters req login. Add force_login parameter.
GUI improvements to Basic config and Add/Update dialogs.

Version 1.7.29 - 28 Jun 2013
New Site: fanfic.potterheadsanonymous.com - Thanks Ida!
New Site: www.simplyundeniable.com - Thanks Ida!
Allow domain fanfic.mugglenet.com for mugglenet.com

Version 1.7.28 - 22 Jun 2013
Minor fixes to plugin
Fix for siye.co.uk stories with HTML in the title

Version 1.7.27 - 10 Jun 2013
Fixes for nha.magical-worlds.us and scarvesandcoffee.net.
Fall back category parsing for fanfiction.net when story has a broken crossover category link.
Restrict url list search on scarvesandcoffee.net.
Add 'Download as New Book?' dialog after 'Change Story URL?', fix author URLs when new author.

Version 1.7.26 - 30 May 2013
New site: nickandgreg.net - Thanks, Ida!
Add Read & Review counts to dramione.org and grangerenchanted.com
Fix for numeric site specific values into float/int custom columns for anthologies.

Version 1.7.25 - 26 May 2013
Add keep_style_attr option.
Add [teststory] feature.

Version 1.7.24 - 28 Apr 2013
Catch errors on db.set_cover().
Don't strip lead/trail whitespace from replace_metadata, add feature \s->' ' in replace_metadata replacements.
After adding to reject list, delete only rejects, not those removed from list.
Offer user chance to skip update when the story URL would be changed.

Version 1.7.23 - 12 Apr 2013
New site: netraptor.org
New site: asr3.slashzone.org
New site: tokra.fandomnet.com
Remove defunct site: www.jlaunlimited.com
Fix author URLs for several sites with leading 'dir' in URL.
Fix for no chapter name for one chapter stories on TtH.
Improved error handling for mobi issues.
Add 'url' to chapter custom formats and class="skip_on_ffdl_update" for updates.
Add anthology_title_pattern feature.

Version 1.7.22 - 08 Apr 2013
Fix for AO3 series URL.
Include notes with chapter text for thewriterscoffeeshop.com.
Remove support for defunct site: prisonbreakfic.net
Remove support for defunct site: svufiction.com

Version 1.7.21 - 05 Apr 2013
Fixes for change to site dark-solace.org/elysian.
Allow ini site section names both with and without www. IE, [www.fanfiction.net] and [fanfiction.net] will both work now. If both are included, the section without www overrides the section with it.
Fix for finding URLs during drag & drop of text emails.
Fix for persistent AddDialog size.

Version 1.7.20 - 01 Apr 2013
Another fix for fanfiction.net changes.

Version 1.7.19 - 28 Mar 2013
Another fix for fanfiction.net changes.

Version 1.7.18 - 28 Mar 2013
Fix for failure to correctly load old JSON settings.
Allow partial download with <url>[1-5] in addition to <url>[1,5].
Add author to chapter TOC for multi-author stories on TtH and WraithBait. (AO3 doesn't reliably report chapter author.)

Version 1.7.17 - 26 Mar 2013
Another fix for fanfiction.net changes.
Fix for collision setting when re-opening the add dialog.
Shorten button name to FanFictionDL.

Version 1.7.16 - 25 Mar 2013
Fix for fanfiction.net changes.

Version 1.7.15 - 25 Mar 2013
Make Add from URLs and Create Anthology for URLs modeless. That means you can open the Add/Create dialogs to put URLs in while still using the calibre library to look for duplicates, etc.
Add seriesUrl metadata. All sites that FFDL collects series info for now also collect a series URL. The default series on title_page is now a link. Used for...
Check for existing Series Anthology books (by seriesUrl) on story add/update. Similar to Reject List check, but offers to skip if you already have an anthology book for that story's series. (It doesn't offer to update the anthology for you.)
Add user/pass for dokuga.com.
Improvements to Reject List feature--allow sorting of reject list by url, title, author (when present). Improve readability of the reject dialogs.

Version 1.7.14 - 18 Mar 2013
Fix for filling int/float custom columns with favs/etc with commas.
Fix for HTML entities in complex replace_metadata setups.
Fixes for fimfiction.net site changes.

Version 1.7.13 - 15 Mar 2013
Remove BBCode--fimfiction.net adapter doesn't use it anymore.
Add "add_to_" feature to ini config. Allow higher priority sections to *add* to any ini param rather than replace it.
Add calibre CLI.
Add "marked:ffdl_success/failed" to added/updated/failed books.
Add "Show added/updated" pref using above.
Preserve log page in epub across overwrites as well as updates.
Drag and drop emails from Thunderbird on FFDL URL box to add URLs.

Version 1.7.12 - 04 Mar 2013
Use existing anthology title on update, even when a series.
Add anthology=true metadata for anthology covers, etc.
Page scrape story descriptions again for fimfiction.net due to API issues.
Fix login failure for tth.

Version 1.7.11 - 27 Feb 2013
www.ficwad.com -> ficwad.com. Ficwad changed their canonical domain. You'll need to change [www.ficwad.com] to [ficwad.com] in your personal.ini.

Version 1.7.10 - 19 Feb 2013
Include author notes in chapters on fimfiction.net.
Add Epub Anthology feature. - Requires EpubMerge 1.3.1+ Plugin.

Version 1.7.9 - 06 Feb 2013
New site: www.henneth-annun.net -- Thanks Ida!
New site: www.psychfic.com -- Thanks Ida!
Now accepting www.skyehawke.com/archive URLs for archive.skyehawke.com stories.

Version 1.7.8 - 04 Feb 2013
Fix author URLs for multiple author stories.

Version 1.7.7 - 04 Feb 2013
Improve error handling, add retry for restrictedsection.org.
Strip leading 0s from story ids for canonical ids in AO3.
Include hyphenation test case in test1 adapter.
Make get list url from page fill add dialog, not view log. (PI only)
Better fix for Nook hyphenation issue--Nook now honors adobe-hyphenate: none.
Remove extra newlines introduced by writer_epub on updates.
Fix broken lastupdate tags feature. Defaulted off for PI.
Optimize adding to Reading Lists on add/update. (PI only)

Version 1.7.6 - 28 Jan 2013
Fix for a change in fimfiction.net's handling of password protected stories.
Fix for thewriterscoffeeshop.com's changed date format.

Version 1.7.5 - 15 Jan 2013
New site: buffynfaith.net (Thanks Dan!)

Version 1.7.4 - 14 Jan 2013
New site: imagine.e-fic.com (Thanks Ida!)
Improved error reporting in background jobs.

Version 1.7.3 - 05 Jan 2013
New site: www.dotmoon.net (Thanks Ida!)
New site: efiction.esteliel.de (Thanks Ida!)
New site: pommedesang.com (Thanks Ida!)
New Spanish language site supported: www.potterfics.com (Thanks Dan!)
New site: www.restrictedsection.org (Yes, this site has been frozen since March 2009.)
More workarounds for fimfiction.net's API issues.

Version 1.7.2 - 02 Jan 2013
New Spanish language site www.potterfics.com (Thanks Dan!)
Remove site www.yourfanfiction.com--it's been gone for awhile.
Correct AO3 extra metadata freefromtags to freeformtags. freefromtags will continue to work for those already using it.
Workaround for fimfiction.net's API issue with non-viewable chapters given.
Add feature to set reason for several Reject URLs at once. PI only.

Version 1.7.1 - 15 Dec 2012
Allow % escapes in ffnet story URLs and use larger ffnet story cover images.
Force bgcolor in JPG output images when source images have transparency.
Add 'collections' to AO3 specific metadata.
Clear Reject List cache when switching libraries.

Version 1.7.0 - 26 Nov 2012
Set Language to Italian for www.efpfanfic.net.
Allow replace_metadata to effect Language metadata.
Fix fimfiction.net cover images
Default calibre language to English for new stories not otherwise marked
New Rejected URL list feature

Version 1.6.19 - 14 Nov 2012
Fixes for www.efpfanfic.net.
Allow 'On Hiatus' status for fimfiction.net.
Recognize calibre identifier uri as well as url.

Version 1.6.18 - 07 Nov 2012
Update potionsandsnitches.net date format.
Switch ponyfictionarchive.net from BeautifulStoneSoup to BeautifulSoup.
Precompile metadata_replace regexps for performance.
Change summary td to div in thehexfiles.net for table titlepages.
Tidy some more poor HTML on fictionalley.org
Honor begin/end chapters on update. For testing purposes.

Version 1.6.17 - 31 Oct 2012
New site: www.efpfanfic.net (Italian language site)

Version 1.6.16 - 28 Oct 2012
New site: indeath.net (thanks besnef)
New site: www.jlaunlimited.com (thanks besnef)
Fixes for changes to fanfiktion.de formatting.
Fix for bad author on twiwritenet with some skins.
Change from www.ncisfiction.com to www.ncisfiction.net due to the ncisfiction.com domain expiring.

Version 1.6.15 - 23 Oct 2012
Fix for some stories on hpfandom.net
Fix for some stories on fimfiction.net
Add conditional replace_metadata feature
Extra metadata for fimfiction

Version 1.6.14 - 17 Oct 2012
Changes to logging to make more logging visible in PI and the same across web/CLI/PI.
Don't number chapter when only one chapter even if add_chapter_numbers is on.
Fix a bug where cover could still be partially set to an image in the cover_exclusion_regexp list.
Add include_images support for HTML format.
Preserve id and class attributes on the HTML tags. (This is for users who want more control from output_css. I don't *think* it will hurt anything for other users...)
Change grangerenchanted.com to use more relaxed HTML parser due to problems with some stories.
Change HTML format output to include title/author in file name that appears inside zip file.


Version 1.6.13 - 08 Oct 2012
Fix for identifiers URL search.

Version 1.6.12 - 03 Oct 2012
New site www.qaf-fic.com (thanks, Ida)
Fix grangerenchanted.com, sg1heliopolis.com & themasque.net.
Add add_chapter_numbers: toconly option.
Better handling of 'Anonymous' stories on AO3.
Allow empty descriptions(summary).
Make just about all the output formatting customizable. See http://code.google.com/p/fanficdownloader/wiki/FanFictionDownLoaderCustomOutput

Version 1.6.11 - 26 Sep 2012
Fix bbcode'd summary issue with fimfiction.net.
Fix get urls from web page bugs.
Fix bug with custom_column_settings 'a' when no data.
Add fanfictiondownloader_macmenuhack.txt 'feature' for Mac users with crash issue.
Improved include_in_* handling.
Change 'version' metadata to plugin version.
Add replace_metadata by metadata key feature.
Additional site-specific metadata for AO3.
Fix 'www.' handling for input URLs.
Fix bug with dokuga.com.
Fix extratags bug.

Version 1.6.10 - 21 Sep 2012
New sites bloodties-fan.com, thehookupzone.net (thanks Marillapm)
Allow individual site adapters to add extra metadata entries to the standard ones. (key word: extra_valid_entries)
Allow custom labels on title/log pages for extra metadata entries. (key words: themes_label, cliches_label, etc)
Allow those extra metadata to be included in titlepage/logpage, or added to existing metadata entries. (key words: extra_titlepage_entries, extra_logpage_entries, include_in_genre, etc)
Allow new, purely composite, metadata entries to be made from existing and extra metadata entries. (key words: include_in_*)
Allow existing metadata entries to be redefined. If one site has what you think should be category and genre reversed, for example, you could swap them. (key words: include_in_category, include_in_genre, etc)
Allow calibre custom columns to be populated from the new extra metadata entries. (key word: custom_columns_settings)
New configuration options to add to titlepage_entries, logpage_entries or include_subject_tags for specific sites (instead of replace it). (key words: extra_titlepage_entries, extra_logpage_entries, extra_subject_tags)
Allow the format for Published, Updated and Packaged dates on titlepage/logpage to be configured in personal.ini. (key words: dateCreated_format, datePublished_format, dateUpdated_format)
Cover support for dramione.org.
A fix for squidge.org/peja Rating.
Loosen url check on AO3 to allow more copy/paste URLs.
Fix author on castlefans.org due to site/skin changes.

Version 1.6.9 - 14 Sep 2012
New site merlinfic.dtwins.co.uk (thanks, Ida)
New option features strip_chapter_numbers and add_chapter_numbers to remove and add numbers to chapter titles.
Fix BBEncoded summaries for fimfiction.net, improved handling for HTML in summaries in general. Default keep_summary_html:true to now.
Added code to prevent crashing when include_images accidentally turned on for non-epub formats.

Version 1.6.8 - 06 Sep 2012
Add 'Relationships' metadata for sites that support it, or are dedicated to a 'ship.
Add 'extracategories', 'extragenres', etc to ini to avoid hard coding them in the adapters.
Special code to *not* include Config & About menu options for Mac users. (Can still configure plugin from calibre's configuration.) Causes some Macs to crash.
Fix for dracoandginny.com adapter for different skins.
Fix for &amp; in tags.
Scroll custom column config tab for users with lots of custom columns.


Version 1.6.7 - 27 Aug 2012
Allow shortcuts for config & about menu items to prevent crashes on some unpatched Macs.
Allow https urls (replace to http).
Fix for extra tags inside metadata on ksarchive.com

Version 1.6.6 - 25 Aug 2012
New site: www.sinful-desire.org
Add adobe-text-layout: optimizeSpeed; to epub CSS to prevent hyphenation on Nook STR(it loses parts of the hyphenated word sometimes.)

Version 1.6.5 - 23 Aug 2012
New site: www.dracoandginny.com
New site: www.scarvesandcoffee.net
New site: www.thepetulantpoetess.com
New site: www.wolverineandrogue.com

Version 1.6.4 - 18 Aug 2012
Fix for Flesch-Kincaid Grade Level Count Page setting.
Workaround for bad html in some story summaries on yourfanfiction.com.

Version 1.6.3 - 06 Aug 2012
Optionally keep an Update Log of past updates (epub only). See personal.ini include_logpage option.
Fix for nha.magical-worlds.us stories with more than 9 chapters.
'Get Story URLs from Web Page' now uses AO3 user/pass if saved in personal.ini.
'Collection' style AO3 story URLs now work.

Version 1.6.2 - 01 Aug 2012
Fix for fanfiction.net to prevent genre(s) appearing as a character on certain stories.

Version 1.6.1 - 01 Aug 2012
Remove extra html body from ancient ffnet chapters.
Add 'newonly' feature for standard and custom columns in plugin.

Version 1.6.0 - 27 Jul 2012
Options now stored inside the Library rather than an external JSON file.

Version 1.5.46 - 22 Jul 2012
Correct integration of grangerenchanted.com, hlfiction.net & nha.magical-worlds.us

Version 1.5.45 - 22 Jul 2012
New Site: grangerenchanted.com
New Site: hlfiction.net
New Site: nha.magical-worlds.us
Minor fixes for finestories.com, finestories.com, thewriterscoffeeshop.com.

Important Plugin Change: FFDL now uses a different, calibre system dialog before updating the library now. This dialog cannot be automatically skipped like the old one. The way it worked before risked corrupting your library if something else was trying to modify it at the same time.

Version 1.5.44 - 15 Jul 2012
New Site: finestories.com
New Site: www.hpfanficarchive.com
New Site: svufiction.com
New Site: www.twilightarchives.com
New Site: www.wizardtales.net
New Feature: Multiple authors per story. Currently supported for AO3, TtH and wraithbait.com. More will be added as we become aware that they use multiple author.
New Plugin Feature: Option to call 'Count Pages' plugin on new/update. Requires Count Pages 1.6.0 or better.
Fixes for latest fanfiction.net change.


Version 1.5.43 - 08 Jul 2012
New Site: dark-solace.org
New Site: pretendercentre.com
New Site: themasque.net
Fixes for a couple obscure problems with calling Generate Cover.

Version 1.5.42 - 06 Jul 2012
Fix for latest fanfiction.net change to prevent id:12345 appearing as a character.

Version 1.5.41 - 04 Jul 2012
Fix for thehexfiles.net on iBooks
Fix for img urls with spaces
Fix for 'Adding/Updating BAD books' status message

Version 1.5.40 - 02 Jul 2012
New Site: ncisfic.com
New Site: national-library.net
Added Support for www.fanfiktion.de and archiveofourown.org username and password.
Set Updated Date to Published Date when not found (fix for recent FFNet changes.)
Fix for fallback metadata labels, allow other text in titlepage_entries
New 'Error Column' feature - On update/overwrite fail, save error msg to a custom column.
Note: As part of the archiveofourown.org fix, the configuration section for AO3 is now [archiveofourown.org], not [www.archiveofourown.org]. Please change your personal configuration accordingly, if you have customized your settings for archiveofourown.org.
Note that www.fanfiktion.de doesn't just require you to say 'yes, I'm an adult'. To see adult content there outside certain hours, you have to have an official government German, Austrian or Swiss ID card or passport number.

Version 1.5.39 - 28 Jun 2012
Another fix for fanfiction.net's latest changes--prevent 'Status: Complete' from appearing as a character.

Version 1.5.38 - 28 Jun 2012
New Site: www.fanfiktion.de
New Site: ponyfictionarchive.net
New Site: sg1-heliopolis.com
Fix for poor summary html on thewriterscoffeeshop.com

Version 1.5.37 - 28 Jun 2012
Fixes for fanfiction.net's latest changes.

Version 1.5.36 - 18 Jun 2012
(1.5.35 skipped)
- Better error handling for bad image_max_size parameter.
- Fix issue with fanfiction.net covers and non-epub formats.
- Fix series# with javascript and different warning#s. geturls w/javascript.

Version 1.5.34 - 16 Jun 2012
Another fix for changed metadata on ffnet.

Version 1.5.33 - 16 Jun 2012
New sites:
- samdean.archive.nu
- www.yourfanfiction.com
- www.destinysgateway.com
- www.thealphagate.com
- stargate-atlantis.org
- www.ncisfiction.com

That's 51 different supported sites now. Thanks, Ida!

New features:
- Add 'Get Story URLs from Web Page' feature to plugin and CLI(-l).
- Plugin - Add some title output lists in the jobs detail window.
- Add cover image to fanfiction.net, but default 'never_make_cover: true' for ffnet.

Fixes:
- Fixes for ffnet characters metadata.
- Fix error in ashwinder from using a different skin.

Version 1.5.32 - 11 Jun 2012
Fix for AO3 'anonymous' stories.
Fix for ashwinder.sycophanthex.com.
Allow www.squidge.org/~peja as well as www.squidge.org/peja.

Version 1.5.31 - 09 Jun 2012
Fix for SIYE metadata for stories in a series.
Fix for ashwinder.sycophanthex.com login.
Fix for onedirectionfanfiction.com login.

Version 1.5.30 - 08 Jun 2012
Add login/is_adult for onedirectionfanfiction.com

Version 1.5.29 - 07 Jun 2012
New Site: onedirectionfanfiction.com by Ida
New Site: www.prisonbreakfic.net by Ida
New Site: www.storiesofarda.com by Ida

Version 1.5.28 - 06 Jun 2012
Fix new/overwrite missing 'begin' KeyError. Plugin only.

Version 1.5.27 - 05 Jun 2012
Fix for fanfiction.net's latest changes.
Allow chapter range after URLs in Add New.
Preserve META-INF/calibre_bookmarks.txt on epub update.
Option to only create/overwrite cover on new, not update.
Make thehexfiles.net accept www.thehexfiles.net too.
New site: www.ik-eternal.net by Ida.

Version 1.5.26 - 29 May 2012
New Site: www.dokuga.com by Ida.
New Option: Under [txt] wrap_width to control word wrap width in text output.

Version 1.5.25 - 27 May 2012
Fix for ffnet genres, characters, status.

Version 1.5.24 - 25 May 2012
New Site: thehexfiles.net by Ida
New Option: Inject calibre series if no series on site.

Version 1.5.23 - 20 May 2012
New Site: ashwinder.sycophanthex.com by Ida
New Site: chaos.sycophanthex.com by Ida
New Site: erosnsappho.sycophanthex.com by Ida
New Site: lumos.sycophanthex.com by Ida
New Site: occlumency.sycophanthex.com by Ida
New Site: dramione.org by Ida
New Site: www.phoenixsong.net by Ida
New Site: www.walkingtheplank.org by Ida

Version 1.5.22 - 18 May 2012
New Site: www.checkmated.com by Ida

Version 1.5.21 - 18 May 2012
New feature - Option to search epub text for story URL when not found in metadata.
New feature - Option to only run GC on new books, not updates.

Version 1.5.20 - 12 May 2012
(1.5.19 skipped)
New Site: www.libraryofmoria.com by Ida
New Site: www.wraithbait.com by Ida
New Site: www.squidge.org/peja (WWOMB) by Jim (This does not support other sections of www.squidge.org, or the other files under www.squidge.org/peja that aren't in the Fiction instance.)

Version 1.5.18 - 08 May 2012
(1.5.17 skipped)
Allow calls to Generate Cover plugin on create/update.

Version 1.5.16 - 25 Apr 2012
Fix for Yet Another numeric entity issue.

Version 1.5.15 - 22 Apr 2012
Fix for 'Hide this banner' title issue on Archive of Our Own.
Changed implementation of fimfiction to use provided API, thanks to Althaine.

Version 1.5.14 - 17 Apr 2012
Change "Include images in EPUBs?" to effect [epubs], not [defaults].
Kludge for incomplete story image URLs on fimfiction.net

Version 1.5.13 - 11 Apr 2012
New Site: archive.skyehawke.com -- Thanks to Ida Leter for implementing this.
Fix for Python GC'ing menu objects if they don't have an explicit Python reference saved.

Version 1.5.12 - 09 Apr 2012
Set calibre's author link field from authorUrl. Thanks to jackr for pointing out it's existence, and davidfor for showing where to find it.

Version 1.5.11 - 24 Mar 2012
Another work around for when SGMLParser's poor entity handling meets poor HTML.

Version 1.5.10 - 19 Mar 2012
New site: ksarchive.com, thanks to Jade Aislin for implementing this
Fixes for mugglenet.com--added user/pass, better handling for summary and is_adult levels.
Allow default_cover_image to use ${title}, etc; add cover_exclusion_regexp feature to exclude images from being covers.

Version 1.5.9 - 16 Mar 2012
Remove site gayauthors.org following complaints from the site administration.
New Site: nfacommunity.com
New Site: www.midnightwhispers.ca. Thanks to Sam for implementing these.
Make 'Update Calibre Only' add a new book if no matching book found.
Minor improvement to column updates for new stories.
Improve image support for a couple of obscure cases.
portkeyorg.org only - Write 'Chapter does not exist' chapter when chapter is in list, but does not exist.
Add is_adult check to twiwrite for when user on site is not set to adult.
Change td in portkey chapter text to div for nook.

Version 1.5.8 - 06 Mar 2012
New Site: fanfiction.mugglenet.com
New Site: fanfiction.portkey.org
New Site: thequidditchpitch.org
New Site: www.hpfandom.net
Also improved category handling of www.tthfanfic.org.

Version 1.5.7 - 01 Mar 2012
Support for downloading images in the story text.
Support for cover image from three different sources.
Support to resize, make grayscale, and convert images to jpg.
Option for plugin to update calibre cover image on metadata update.
Major re-write of EPUB update to support images.
Improved handling of ini section priority.
New 'Stay on Top' option for 'Add New from URL(s)' dialog for easier drag-n-drop.
Add replace_hr option to replace <hr> tags with '* * *'
Support for fimfiction.net's locked story passwords, including an option to declare locked stories 'failed'.
Add an 'Include images in EPUBs' checkbox to duplicate some personal.ini settings.
nook_img_fix option - wrap img tags with div if not already wrapped with div or p for a nook problem. Defaults to true.

Version 1.4.6 - 24 Feb 2012
New Site: gayauthors.org -- Thanks to Ida Leter for implementing this.
Fix for URL copy/paste from Chrome.
Improvements to metadata for AO3.

Version 1.4.5 - 22 Feb 2012
Fix for ffnet minor format change.

Version 1.4.4 - 21 Feb 2012
Minor fixes to ffnet and fictionalley.org.

Version 1.4.3 - 15 Feb 2012
New Russian language site: ficbook.com -- Thanks to Ida Leter for implementing this.
Language metadata now collected for ficbook.com and fanfiction.net.
Fix Character metadata on fanfiction.net.

Version 1.4.2 - 13 Feb 2012
Fix a bug with HTML entity processing.

Version 1.4.1 - 12 Feb 2012
New feature - Allow user to set custom CSS in personal.ini for HTML and EPUB output.
New feature - Allow user to set custom regular expressions in personal.ini to modify metadata.
New feature - Use Accept-Encoding=gzip to speed download. (Not sites will use it--it's common for sites to block gzip based on User-Agent.)
Add progress bars while collecting URLs from stories for list and for updates.
Make ini edit Courier and 1pt larger than default font.

Version 1.3.4 - 04 Feb 2012
Fixed a bug that caused stories with a footer to crash for AO3. (Thanks, Ida)
Removed sleep from ficwad adapter.
Fixed menu/hotkey update when switching libraries.

Version 1.3.3 - 31 Jan 2012
Better date kludge for fimfiction.net--they don't give years for updated/published.

Version 1.3.2 - 30 Jan 2012
New site: archiveofourown.org -- Thanks for Ida Leter for implementing this.
Change fimfiction.net to use genre instead of category.

Version 1.3.1 - 28 Jan 2012
New feature: Fill Custom Columns with Story Metadata. Look in Config to set it up.
New feature: Collect and update Series column.
Improved drag-and-drop of URLs onto the Add New URL text box. Dragging URLs in adds now instead of replaces, and copies the URL, not the pretty text.
Added a 'show this again' checkbox to the dialog for reviewing updates before applying them. When unchecked, updates will be applied immediately if there were no problems.
Added a progress bar while applying updates, which can be lengthy.
Don't disable menu items(like update) when there are no books selected. It broke the keyboard shortcut if the menu hadn't rebuilt since books were selected.
Make the FanFictionDownLoader button (which does add or update, depending on whether books are selected) available for keyboard shortcut.
Fix date change for adastra.
Fix URL pattern, metadata collection for SIYE.

Version 1.2.3 - 22 Jan 2012
Fix for tz aware vs naive dates.
'Fix' for ffnet look ahead for 1 chapter stories.

Version 1.2.2 - 19 Jan 2012
Fix to 'Keep tags'.
Fix to extratags.
'Fix' for ffnet updates coming out before the web site updates everywhere.

Version 1.2.1 - 17 Jan 2012
Fixes for date updating.
Fix for English name months for users running non-English locale computers.

Version 1.2.0 - 16 Jan 2012
New feature - Options to update Reading List plugin lists for device sync and keeping a list of new books to read.
Plugin configuration now tied to library. Different libraries can have different settings.
Shortcut button in plugin config to dialog for assigning hotkeys.
List of Characters now parsed and saved for most supported sites.

Version 1.1.3 - 10 Jan 2012
Fixed lastupdate feature in include_subject_tags.
Separated Plugin defaults.ini from web/CLI defaults.ini to remove unusable options.
Config option to keep existing tags when updating metadata. Complete/In-Progress will still be updated, and Last Updated tags will be updated if lastupdate in include_subject_tags.

Version 1.1.2 - 09 Jan 2012
Restructure of background processing, improved update of existing books, added SIYE, other fixes.

Version 1.0.3 - 30 Dec 2011
Initial Release on mobileread.com forums.

Aleyst
12-31-2011, 01:24 AM
Thanks for this. It's pretty handy.

JimmXinu
01-06-2012, 01:47 AM
Posting FanFictionDownLoader plugin 1.1.0, hoping to get some beta testers before pushing it out.

Internally, 1.1 is an almost complete re-write of the plugin from 1.0.

I was advised by more experienced plugin developers that the way I was handling background processing was not particularly safe. Given the number of unexpected crashes I kept getting working with 1.0, I saw their point.

1.1 uses Calibre's mechanism to launch separate background processes to download stories rather than using a background thread.

There is now a drop down menu from the plugin tool bar button for Add/Update/Configure/About.

Also, 'update' truly updates the book(s) selected now. Before the plugin gathered URLs from the selected books, but after that treated the URL as if the user had entered it. If the name or author on the book had been changed, either in Calibre or on the website, it would not update the existing book in Calibre, but treat it as a different book.

The plugin now downloads the stories, but does *not* add/update them in your library until all are downloaded and you confirm you want the updates. Again, this is because background processing updating the foreground is dangerous in Calibre.

There is (hopefully) better download error/update reporting now, but it could still use some more work.

There are more options now for how to update existing stories, whether by directly updating or by doing 'Add New' with a URL to a story that already exists in your library.

Under "Add New from URL(s)", there's an option to choose handling for "If Story Already Exists?". This is also sometimes called collision handling.

"Skip" and "Add New Book" should be self-explanatory.

"Update EPUB if New Chapters" and "Update EPUB Always" allow you to incrementally update a story you already have, but only if you use EPUB. "Update EPUB Always" will update the title page and metadata for the story even if there are no new chapters.

"Overwrite Always" will download the story and overwrite the chosen format. "Overwrite if Newer", only if the website story is newer.

"Update Calibre Metadata Only" does not download the story, only enough metadata to update the metadata Calibre keeps.

Update Existing's "Update Mode" is identical.

Under Configuration, there are some changes and new options.

With the split between "Add New" and "Update Existing", the options to take URLs from selected and prioritize selected vs clipboard are removed. There's still an option to take URLs from clipboard or not.

You can also make the menubar button always bring up "Add New" by unchecking "Default to Update when books selected?".

Jim

(Attachment removed after 1.1.1 posted.)

Aleyst
01-06-2012, 03:47 AM
Sounds interesting, I'm just updating the main calibre program now, I'll update the plugin and have a mess around. :D

JimmXinu
01-08-2012, 04:21 PM
1.1.1 contains everything from 1.1.0 plus:


Fixes for problems found with Add New.
Fix to 'delete other formats' when html.
Hide 'update epub' options when format not epub.
fanfic.castletv.net changed to castlefans.org/fanfic.
Fix for twiwrite adult-only stories.


I think this is ready to be pushed out as the new version, but I'd appreciate it if I could get a tester or two to try it first.

Jim

scoopaz
01-08-2012, 04:37 PM
Hi Jim,

Went to give you plugin a shot since I'm a big collector of fanfiction and use the web version extensively. When trying to download a fanfic from fanfiction.net I received this error message:

Traceback (most recent call last):
File "site-packages\calibre\gui2\__init__.py", line 340, in dispatch
File "calibre_plugins.fanfictiondownloader_plugin.ffdl_p lugin", line 544, in download_list_completed
File "calibre_plugins.fanfictiondownloader_plugin.ffdl_p lugin", line 589, in _add_or_update_book
File "site-packages\calibre\library\database2.py", line 3175, in create_book_entry
File "site-packages\calibre\library\database2.py", line 3171, in _add_newbook_tag
AttributeError: 'set' object has no attribute 'append'

I'm using the latest version of Calibe (0.8.34) on a Windows 7 machine.

Please help! :eek: I would love to have this plugin working.

Lisa

JimmXinu
01-08-2012, 04:46 PM
Went to give you plugin a shot since I'm a big collector of fanfiction and use the web version extensively. When trying to download a fanfic from fanfiction.net I received this error message:

I'm using the latest version of Calibe (0.8.34) on a Windows 7 machine.

Please help! :eek: I would love to have this plugin working.

@scoopaz, I'm downloading 0.8.34 now--I was still on 0.8.33.

What's the URL of the story that gave the error? Does it happen for all, or just one story?

Jim

scoopaz
01-08-2012, 05:00 PM
Hi Jim,

thanks for the quick response!. The URL for the story I initially tried is :

http://www.fanfiction.net/s/7623520/1/The_Good_Brother

so I tried a few more on fanfiction.net, and got the same error. Went to www.whofic.com just to confirm with a generic (non-adult) story:

http://www.whofic.com/viewstory.php?sid=29654
and received the error again.

Lisa

JimmXinu
01-08-2012, 05:01 PM
@scoopaz, I've figured it out.

It's a bug in Calibre, not the plugin.

There's a problem in the routine that creates new book entries with "Tags to apply when adding a book" set.

I'll report it in the calibre bug tracking system.

For now, if you remove your "Tags to apply when adding a book", it will work.

Even if you don't want to do that for regular use, please give it a try and confirm that it solves your problem.

Jim

scoopaz
01-08-2012, 05:05 PM
Here's the data from the FFdownloader job:

Download FanFiction Book
Failed to initialize plugin: u'C:\\Users\\scoopaz\\AppData\\Roaming\\calibre\\p lugins\\Goodreads Covers.zip'
Failed to initialize plugin: u'C:\\Users\\scoopaz\\AppData\\Roaming\\calibre\\p lugins\\Goodreads Metadata.zip'
FanFictionDownLoader v1.1.1
Logfile for book ID None (The Good Brother)
url:(http://www.fanfiction.net/s/7623520/1/) id:(None)
Failed to initialize plugin: u'C:\\Users\\scoopaz\\AppData\\Roaming\\calibre\\p lugins\\Goodreads Covers.zip'
Failed to initialize plugin: u'C:\\Users\\scoopaz\\AppData\\Roaming\\calibre\\p lugins\\Goodreads Metadata.zip'
write to c:\users\scoopaz\appdata\local\temp\calibre_0.8.34 _tmp_1ijjwz\fanfictiondownloader_czx5j7\new-None-lzgitu.epub
File(c:\users\scoopaz\appdata\local\temp\calibre_0 .8.34_tmp_1ijjwz\fanfictiondownloader_czx5j7\new-None-lzgitu.epub) Updated(2012-01-08) more recently than Story(2012-01-07) - Skipping


Lisa

scoopaz
01-08-2012, 05:12 PM
Hi Jim,

Just saw your post - removed my auto tag when importing and gave it another shot:

The good news: The ebook was created and imported into my library with no errors:

The bad news: I received a different error when trying to open the created epub:

calibre, version 0.8.34
ERROR: Could not open ebook: File is not a zip file

Traceback (most recent call last):
File "site-packages\calibre\gui2\viewer\main.py", line 64, in run
File "threading.py", line 483, in run
File "site-packages\calibre\ebooks\oeb\iterator.py", line 205, in __enter__
File "site-packages\calibre\customize\conversion.py", line 204, in __call__
File "site-packages\calibre\ebooks\epub\input.py", line 138, in convert
File "site-packages\calibre\utils\zipfile.py", line 751, in __init__
File "site-packages\calibre\utils\zipfile.py", line 786, in _GetContents
File "site-packages\calibre\utils\zipfile.py", line 801, in _RealGetContents
BadZipfile: File is not a zip file

In Calibre the size of the file shows as 0.00 kb :chinscratch:

Lisa

JimmXinu
01-08-2012, 05:21 PM
Just saw your post - removed my auto tag when importing and gave it another shot:

The good news: The ebook was created and imported into my library with no errors:

The bad news: I received a different error when trying to open the created epub:

File "site-packages\calibre\utils\zipfile.py", line 801, in _RealGetContents
BadZipfile: File is not a zip file

In Calibre the size of the file shows as 0.00 kb :chinscratch:
Lisa,

Well, that's not supposed to happen... But on the other hand, it is farther along and therefore progress.

Please try it again making sure that you're adding from the URL and have 'epub' and 'Add New' selected. If it still makes a zero length epub, please post the details from the Download job.

Thanks,

Jim

scoopaz
01-08-2012, 05:32 PM
Hi Jim,

OK, Made sure it was set to add new ebook, and add new from URL and no go.

Here is the results from the job:

Download FanFiction Book
Failed to initialize plugin: u'C:\\Users\\scoopaz\\AppData\\Roaming\\calibre\\p lugins\\Goodreads Covers.zip'
Failed to initialize plugin: u'C:\\Users\\scoopaz\\AppData\\Roaming\\calibre\\p lugins\\Goodreads Metadata.zip'
FanFictionDownLoader v1.1.1
Logfile for book ID None (The Good Brother)
url:(http://www.fanfiction.net/s/7623520/1/) id:(None)
Failed to initialize plugin: u'C:\\Users\\scoopaz\\AppData\\Roaming\\calibre\\p lugins\\Goodreads Covers.zip'
Failed to initialize plugin: u'C:\\Users\\scoopaz\\AppData\\Roaming\\calibre\\p lugins\\Goodreads Metadata.zip'
write to c:\users\scoopaz\appdata\local\temp\calibre_0.8.34 _tmp_o2hwch\fanfictiondownloader_9vuddg\new-None-ivsafd.epub
File(c:\users\scoopaz\appdata\local\temp\calibre_0 .8.34_tmp_o2hwch\fanfictiondownloader_9vuddg\new-None-ivsafd.epub) Updated(2012-01-08) more recently than Story(2012-01-07) - Skipping

I tried uninstalling the Goodreads plugin and tried again with no luck. could something be wonky with my configuration?


Lisa

kiwidude
01-08-2012, 05:42 PM
@scoopaz - the two Goodreads plugin messages are due to you having installed two plugins a long time ago that got superceded by the single Goodreads plugin. They have nothing to do with any behavior with the FanFictionDownloader plugin.

It is just calibre telling you that it is not possible to load those plugins (because they arent compatible with recent calibre versions, hence they were replaced with a single "Goodreads" plugin).

To get rid of the error messages, see the couple of posts here (http://www.mobileread.com/forums/showthread.php?p=1825733#post1825733). You can either do it via the command line or as dwnathny says just delete the relevant plugin zip files from the calibre plugins folder. Just make sure you do the right ones - as per your error message text they are "Goodreads Covers" and "Goodreads Metadata" that are the plugins you should remove, and not the "Goodreads" one.

JimmXinu
01-08-2012, 05:56 PM
And I was getting ready to blame it all on Goodreads plugins. :-)

Actually, Lisa, your last log posting gave me the answer.

I've replaced the zip file on the "FanFictionDownLoader 1.1.1" post above with a fixed one. Please give that one a try.

Thanks,

Jim

scoopaz
01-08-2012, 06:06 PM
Hi everybody,

Kiwidude: Thanks for the info - I'll go ahead and get that fixed per your link.

Jim: Uninstalled the old plugin completely and installed the updated one - Success! The story downloaded successfully and opened up without any errors.

:thanks: I suspect I will be abusing this plugin regularly :thumbsup:


Lisa

JimmXinu
01-08-2012, 07:25 PM
Jim: Uninstalled the old plugin completely and installed the updated one - Success! The story downloaded successfully and opened up without any errors.

Excellent! Thanks for helping figure this out.

Jim

JimmXinu
01-09-2012, 08:14 PM
It's a bug in Calibre, not the plugin.

There's a problem in the routine that creates new book entries with "Tags to apply when adding a book" set.
Turns out this was indeed my fault. I have a fix for it now.

I'll be releasing a new version (1.1.2), so it should be available for auto-update soon.

Jim

JimmXinu
01-10-2012, 10:36 PM
Somebody on the user group requested an option to keep existing tags on updatemeta.

"Existing tags will be kept and any new tags added.
Completed and In-Progress tags will be still be updated, if known.
Last Updated tags will be updated if lastupdate in include_subject_tags."

I also fixed the Last Update tags feature.

LMx
01-12-2012, 06:49 PM
Awesome plugin! thank you very much for your time doing this

my girlfriend is using it often and loves it

however she uses 2 other big websites that are still not supported by this plugin

is there any plans in the future to add support for

http://www.livejournal.com
and/or
http://archiveofourown.org/ ?


also, there were some links that weren't added to the library so i'm trying to help her
here is an example log:

Download FanFiction Book
FanFictionDownLoader v1.1.3
Logfile for book ID None (A Bag Full of Mysteries)
url: (http://www.fanfiction.net/s/6420141/1/) id: (None)
write to c:\users\______\appdata\local\temp\calibre_0.8.34_ tmp__p6hx7\fanfictiondownloader_ayilc3\new-None-dhpynm.epub
Exception: {u'comment': u"'utf8' codec can't decode byte 0xe7 in position 50: invalid continuation byte", u'publisher': u'www.fanfiction.net', u'good': False, u'pubdate': datetime.datetime(2010, 10, 23, 0, 0), u'author': u'tinybee', u'url': u'http://www.fanfiction.net/s/6420141/1/', u'timestamp': datetime.datetime(2012, 1, 12, 22, 15, 44, 429000), u'author_sort': u'tinybee', u'calibre_id': None, u'is_adult': False, u'comments': u'A series of Harry Potter/Crossover one shots. Pairings will be SLASH Harry/? from numerous crossovers. List of what I can do inside. NO LONGER TAKING PROMPTS.', u'username': 'NoneGiven', u'outfile': u'c:\\users\\______\\appdata\\local\\temp\\calibre _0.8.34_tmp__p6hx7\\fanfictiondownloader_ayilc3\\n ew-None-dhpynm.epub', u'added': False, u'title': u'A Bag Full of Mysteries', u'password': '', u'tags': [u'X-overs', u'Harry Potter', u'In-Progress', u'Romance', 'FanFiction', u'Drama'], u'icon': u'dialog_error.png'}:'utf8' codec can't decode byte 0xe7 in position 50: invalid continuation byte
Traceback (most recent call last):
File "calibre_plugins.fanfictiondownloader_plugin.jobs", line 131, in do_download_for_worker
File "calibre_plugins.fanfictiondownloader_plugin.fanfic downloader.writers.base_writer", line 215, in writeStory
File "calibre_plugins.fanfictiondownloader_plugin.fanfic downloader.adapters.base_adapter", line 191, in getStory
File "calibre_plugins.fanfictiondownloader_plugin.fanfic downloader.adapters.adapter_fanfictionnet", line 208, in getChapterText
File "calibre_plugins.fanfictiondownloader_plugin.fanfic downloader.adapters.base_adapter", line 167, in _fetchUrl
UnicodeDecodeError: 'utf8' codec can't decode byte 0xe7 in position 50: invalid continuation byte

if mobi format is selected it also gives an error, although a different "FailedToDownload: Error downloading Chapter: http://www.fanfiction.net/s/6420141/24/! Missing required element!"

txt and html formats were added without problems

is it a bug or a fanfiction.net problem? thanks in advance :)

JimmXinu
01-12-2012, 08:18 PM
Awesome plugin! thank you very much for your time doing this

Glad you like it. :)

is there any plans in the future to add support for

http://www.livejournal.com
and/or
http://archiveofourown.org/ ?

livejournal isn't really a fanfic site. From what I remember of it, which admittedly isn't much, it's very free form with little common formatting and story demarcation.

Unless it's changed a lot, it's much more challenging than I, for one, am prepared to tackle for site I don't read myself.

However, if anyone else wants to write an adapter for it, we gladly welcome code contributions.

http://archiveofourown.org has it's own ebook(epub/mobi/etc) downloads. For actual content, they've done a better job than we have--theirs contain images, too. They haven't, however, done as good a job of including metadata for Calibre. I'll think about what I might be able to do there.


http://www.fanfiction.net/s/6420141/24/!
txt and html formats were added without problems

is it a bug or a fanfiction.net problem? thanks in advance :)
This story downloaded fine for me in both epub and mobi. Unless you continue to have problems, I'm going to assume that it was an ffnet glitch that came and went.

Jim

LMx
01-12-2012, 10:15 PM
thank you for the quick reply

indeed, it seems to have been a ffnet glitch as i tried it now again and had no problems

good luck with the improvement of the plugin :)

davidfor
01-12-2012, 10:50 PM
Jim,

I'm not a fanfiction reader, but my wife is, so I had a play with your plugin. I tested it with a couple of the links posted here to see how it went and I am very impressed (and amused by the short Dr Who story).

One thing that occurred to me, do you keep the bookmarks that the Calibre viewer puts into the epub? One of the bookmarks is the current reading location, I can see that if the story is being updated with new chapters, if the bookmarks are lost it could get very annoying. If you haven't seen them, the bookmarks are in the file "calibre_bookmarks.txt" in the "META-INF" folder of the epub.

And I just had a look at the story again. On the title page, you have "Categoryq". Is that a typo?

David

JimmXinu
01-12-2012, 11:08 PM
I'm not a fanfiction reader, but my wife is, so I had a play with your plugin.
Oddly, you're the third guy I've heard from with a similar comment about using it for an SO. My wife just shakes her head tolerantly at me when I talk about fanfic. :)
One thing that occurred to me, do you keep the bookmarks that the Calibre viewer puts into the epub? One of the bookmarks is the current reading location, I can see that if the story is being updated with new chapters, if the bookmarks are lost it could get very annoying. If you haven't seen them, the bookmarks are in the file "calibre_bookmarks.txt" in the "META-INF" folder of the epub.

I did not know it did that. I wonder why Calibre puts it there? It has some other, non-invasive method for mobis...

A quick test appears to show that yes, that file is discarded on epub update.

It shouldn't be terribly difficult to retain it when doing update downloads. Overwrites, of course, will still destroy it.

And I just had a look at the story again. On the title page, you have "Categoryq". Is that a typo?
Ah. Oops.

That's a little embarrassing actually--I changed that on purpose to make sure that the correct file was being used and must have forgot to change it back.

Jim

JimmXinu
01-12-2012, 11:47 PM
It shouldn't be terribly difficult to retain it when doing update downloads. Overwrites, of course, will still destroy it.

Changes to retain calibre_bookmarks.txt through epub updates and fix Categoryq are in 1.1.4. It's uploaded and will start showing up in the automatic plugin list in a while.

Jim

davidfor
01-13-2012, 01:45 AM
Oddly, you're the third guy I've heard from with a similar comment about using it for an SO. My wife just shakes her head tolerantly at me when I talk about fanfic. :)


My wife knows that headshake well. :)


I did not know it did that. I wonder why Calibre puts it there? It has some other, non-invasive method for mobis...


That's interesting. My understanding was that Calibre converted everything to epub to view it. I assumed that it didn't remember the current page if it did this. But, a quick test showed it did.


That's a little embarrassing actually--I changed that on purpose to make sure that the correct file was being used and must have forgot to change it back.

Ah, the deliberate change to help debug something. I know it well. I have had a user see this in my tests and then I spent hours explaining that it wasn't a real change and I was going to remove it.

JimmXinu
01-13-2012, 01:58 AM
That's interesting. My understanding was that Calibre converted everything to epub to view it. I assumed that it didn't remember the current page if it did this. But, a quick test showed it did.
I've never really looked into how the book viewer part works. The full-up conversion process generally takes longer than the viewer starting up, though.

I've just glad you knew about the calibre_bookmarks.txt file. I might never have figured out that problem without it. Seems a pretty Evil thing for Calibre to do, quite honestly. I wonder where it remembers it's place in mobis or other formats?

Jade Aislin
01-13-2012, 01:12 PM
I love this plugin. The only reason I use calibre is to sort the fanfic I read. I’ve barely scratched the surface of the stories I read at FFN and can't wait to try downloading from TTH.

I do have a question however. How do I get the characters to show up? I have been trying to figure out how to get the characters to show up in the title page, but they never do. I’d prefer the characters to show up as tags, but am unsure if it is possible.

I have characters listed in the titlepage_entries section. The FFN story I am using as a test does have two characters, but I do not see it showing up.

One of the urls I am using is: http://www.fanfiction.net/s/7661640/1/

When I tried putting other items like storyID or siteabbrev on the title page I did not have a problem. It is only the characters.

JimmXinu
01-13-2012, 01:31 PM
I do have a question however. How do I get the characters to show up? I have been trying to figure out how to get the characters to show up in the title page, but they never do. I’d prefer the characters to show up as tags, but am unsure if it is possible.
You're doing it right, it sounds like. And when there are characters they will show up as tags by default.

But we only have code for parsing and saving characters for some sites and ffnet isn't one of them. (Currently only FimFiction.net and SIYE do.)

It doesn't look too horrible to add character parsing to ffnet. I'll look into it.

Jim

Jade Aislin
01-13-2012, 02:35 PM
You're doing it right, it sounds like. And when there are characters they will show up as tags by default.

But we only have code for parsing and saving characters for some sites and ffnet isn't one of them.


That's good to know. At least I know I'm not doing something wrong.


It doesn't look too horrible to add character parsing to ffnet. I'll look into it.



I hope you are able to get the characters from FFnet. I've been using a complex method for getting my stories into calibre.

1. use fanfictiondownloader to download story list sorted by character.
2. add stories to calibre.
3. manually input custom fields (character, story complete)
4. repeat steps 1-3 for new character
5. find duplicate stories.
6. Merge duplicate stories so that it shows both characters.
7. Repeat for any additional characters.

This is very time consuming. If you can get the characters in the tags, it would eliminate a lot of the work I have to do.


On another note, is it possible to change the font of the comments? I love that the summary is put into the comments, but have trouble reading it on my tablet. I have tried to bulk edit, but could not make any changes as there are no easy html tags to change. Increasing the font in the comments field while the story is downloaded would be a big help.

JimmXinu
01-13-2012, 02:50 PM
...
This is very time consuming. If you can get the characters in the tags, it would eliminate a lot of the work I have to do.

I'm working on it--but I'm remembering why I didn't before: it's a pain to do correctly. ffnet isn't easy to parse.

On another note, is it possible to change the font of the comments? I love that the summary is put into the comments, but have trouble reading it on my tablet. I have tried to bulk edit, but could not make any changes as there are no easy html tags to change. Increasing the font in the comments field while the story is downloaded would be a big help.

The comments field in calibre's metadata? I'm just shoving plain text into that w/o HTML. calibre's adding a bit of HTML around each paragraph with class="description". I wonder if one could change the def of "description" somewhere.

Jim

JimmXinu
01-13-2012, 05:02 PM
(remove reply to misfiled post)

JimmXinu
01-13-2012, 07:22 PM
@Jade Aislin - I've put up a new version (1.1.5) with parsing and recording of characters on fanfiction.net.

It will be a bit before it shows up in calibre as the latest version.

Jim

Jade Aislin
01-13-2012, 09:14 PM
Wow, you're fast. I tried it and got the characters in the tags. Now I only have to take a few things from the tags and put them into my custom columns. Much easier.

Of course this means I have to sort through all the stories from different fandoms I like on FFN. . .this could take awhile.

But I am definately using your plugin. Thanks.

jackr
01-13-2012, 10:59 PM
First off, amazing plugin :D. It really makes life easier for reading fanfics on my Nook. In fact, it was this plugin that got me to try Calibre.

Anyways, I have quite the list of fanfics now and some of them are flagged as adult material. I wanted to know if its possible to incorporate some sort of 'Yes I am an adult' setting into your plugin so I don't have to sit through the process of checking all the fics for updates and the like.

JimmXinu
01-13-2012, 11:16 PM
@jackr - Glad you like it.

Regarding is_adult: Have you looked at the 'personal.ini' configuration? When you configure the plugin, at the bottom, there's a button to view the defaults for the many settings.

What you want is to add this to your personal.ini settings:

[defaults]
## Some sites also require the user to confirm they are adult for
## adult content. In commandline version, this should go in your
## personal.ini, not defaults.ini.
is_adult:true


Looking at it, I see that it's actually not easy to pick that setting out of the defaults file--I'll make it clearer.

You can also set user/pass for different sites that require log in.

Jim

jackr
01-13-2012, 11:39 PM
Thanks for the fast response! I did not look at the personal.ini :o. But it goes to show that if you thought of it, it's probably already covered. :)

This might be the wrong area for it, but is it possible to set up a column to display stats like how many new chapters have been updated, etc?

JimmXinu
01-13-2012, 11:52 PM
Thanks for the fast response! I did not look at the personal.ini :o. But it goes to show that if you thought of it, it's probably already covered. :)
I doubt we've thought of everything, but the FanFictionDownLoader part (as opposed to the plugin part) has been around for a couple years now and I've been working on it for over a year.
This might be the wrong area for it, but is it possible to set up a column to display stats like how many new chapters have been updated, etc?
It can't right now. The closest thing immediately available I know of is the Count Pages (http://www.mobileread.com/forums/showthread.php?t=134000) plugin. But I'm still on a learning curve with advanced calibre use myself.

Setting custom columns from a story's metadata (num chapters, words, etc) is something that could be done in future. I'll add it to the wish list.

Jim

Jade Aislin
01-14-2012, 08:59 AM
I'm loving this now that characters show up in the tags for FFN. It will be nice to use the plugin for updates. I even got the last update to show up in the tags.

I realize that it puts two different formats (year/month and year/month/day), but is there a way to specify which one to use? Or a way to set the format myself, I prefer month/day/year). If there isn't, I won't worry about it. This plugin is still better than any manual input I would have to do otherwise.

cklls
01-14-2012, 10:22 AM
Jim - Loving this plug-in! I'm an avid consumer (and sometimes author) of fanfiction and have happily used your web-based version in the past. Having it built in to Calibre is just... heavenly! If you're open to suggestion on other sites to support, I'd like to request www.dramione.org be added if/when you can. It's a small archive, but rapidly growing in both readers and stories. I'm a staff member there, and would love to be able to use this tool to help with editing and validation.

JimmXinu
01-14-2012, 11:29 AM
@Jade Aislin - You can turn them on and off from personal.ini, but that's it. That feature's actually from one of the other developers--I keep it turned off. :)

@cklss - Which sites get supported depends on the size and activity of the site, how hard it would be to implement, and the interest and free time available to the developer. :)

http://dramione.org is smaller than we usually support, plus right now I'm concentrating on the plugin. So I'm not planning to support it.

However, FFDL (http://code.google.com/p/fanficdownloader/) is an open source project--we welcome other people contributing if you, or anyone, wants to write an adapter to support it.

Jim

cklls
01-14-2012, 12:38 PM
@cklss - Which sites get supported depends on the size and activity of the site, how hard it would be to implement, and the interest and free time available to the developer. :)

http://dramione.org is smaller than we usually support, plus right now I'm concentrating on the plugin. So I'm not planning to support it.

However, FFDL (http://code.google.com/p/fanficdownloader/) is an open source project--we welcome other people contributing if you, or anyone, wants to write an adapter to support it.

Jim

Thanks, Jim. Not at all surprised that it's too small for your attention. :) I'll mention it to our tech person to see if she's got time to fiddle with FFDL. Apart from clicking on the link, I honestly wouldn't know where to begin! The last "code" I wrote was probably close to thirty years ago, "programming" a bunny to print from DOS commands. HTML emphasis codes (italics, bold, etc.) are about as far as I go! LOL

JimmXinu
01-14-2012, 01:41 PM
IMO, it's not hugely difficult. I'd suggest looking at the adapter for castlefans.org (http://code.google.com/p/fanficdownloader/source/browse/fanficdownloader/adapters/adapter_castlefansorg.py) as an example.

Jim

Jade Aislin
01-14-2012, 08:04 PM
@Jade Aislin - You can turn them on and off from personal.ini, but that's it. That feature's actually from one of the other developers--I keep it turned off. :)


That's alright. I'm just happy it's there at all. I can work with what it gives me.

However, FFDL (http://code.google.com/p/fanficdownloader/) is an open source project--we welcome other people contributing if you, or anyone, wants to write an adapter to support it.

Jim
Hmm. . .I might try to get some other sites I visit adapted, http://www.masterapprentice.org/ (a star wars site) and http://dokuga.com/fanfiction or http://dokuga.com/spark (two inuyasha sites). All of them have between 4,000 to 5,000 stories. Unfortunately, it has been over ten years since I did any programing and I had decided then that I wasn't very good at it. But I'll give that link you gave cklss a look.

JimmXinu
01-15-2012, 02:19 PM
@Jade Aislin - You'd asked about changing the appearance of the comments field:

The comments field in calibre's metadata? I'm just shoving plain text into that w/o HTML. calibre's adding a bit of HTML around each paragraph with class="description". I wonder if one could change the def of "description" somewhere.

I happened to find where this is set while looking for something else.

The general answer is in Customizing calibre, Overriding icons, templates, etcetera (http://manual.calibre-ebook.com/customize.html#overriding-icons-templates-etcetera).

First, find your calibre config dir by doing Prefs->Misc->Open calibre conf dir. Go into resources, make a directory templates.

Then, copy C:\Program Files (x86)\Calibre2\resources\templates\book_details.cs s (or equivalent) to the templates dir you just made.

Edit your new book_details.css file and add:
.description {
font-size: 25px;
}

...or whatever size you like.

I believe that will do it for you.

Jim

Jade Aislin
01-15-2012, 04:06 PM
That worked perfectly! Once I figured out where to place it. Now I don't have to go through thousands upon thousands of epubs to individually edit their summaries. I've been trying to figure out what to do for months.

Thank you.:2thumbsup:thanks:

JimmXinu
01-16-2012, 04:37 PM
I'm again looking for some people to beta test the attached newest version before I put it out for everyone.

This version contains:

New feature - Options to update Reading List plugin lists for device sync and keeping list(s) of new books to read.
Plugin configuration now tied to library. Different libraries can have different settings.
Shortcut button in plugin config to dialog for assigning hotkeys.
List of Characters now parsed and saved for most supported sites.


The big change is the integration with Reading List (http://www.mobileread.com/forums/showthread.php?t=134856).

Reading List defines lists of stories that it can automatically send to your reading devices and update tags and custom columns.

Once you've created Reading Lists for sending to your devices, you can configure FanFictionDownLoader 1.2.0 to automatically add new/updated stories to those lists.

Reading Lists can also update custom columns automatically. Using that with a Metadata Plugboard, you can put a notation in the title of new books to read when they're sent to your device.

As a convenience, the plugin offers a menu option to remove stories from the 'to read' lists after you've read them and optionally add 'read' stories to the 'send to device' lists again.

This is an illustrated example (http://code.google.com/p/fanficdownloader/wiki/FanFictionDownLoaderPluginWithReadingList) of how I thought it would be used.

There's also a decent chance somebody will point out a better way to do this. :)

Thanks,
Jim

soleta
01-19-2012, 12:33 AM
Thank you so much for this plugin - it's honestly one of the ones I use most often. Because of that, I was wondering if there's a way I'm missing to tell the plugin not to add tags to the file, or to use calibre's default 'tags to apply when adding a book' setting?

todaysname
01-19-2012, 07:59 AM
Wonderful plugin, I used it all the time up until the latest update. Past updates worked well.

Now every time I open the plugin to either add a new url or update a new story, Calibre has to quit unexpectedly. I'm not a technical person at all. If there is some data that you need about the crash, let me know where to look and I will post that information.

JimmXinu
01-19-2012, 12:51 PM
Thank you so much for this plugin - it's honestly one of the ones I use most often. Because of that, I was wondering if there's a way I'm missing to tell the plugin not to add tags to the file, or to use calibre's default 'tags to apply when adding a book' setting?
Glad you like it.

There's meant to be a way, but when I went to try it, I found a couple bugs that I need to fix before it will work properly. Stay tuned.

Jim

JimmXinu
01-19-2012, 04:21 PM
Version 1.2.2 is up. After a while, calibre will notice it and prompt you to update.

@soleta - Once you have 1.2.2, you can go into plugin config->personal.ini and add a line:
[defaults]
include_subject_tags:

(Don't add another [defaults] line if you already have one.)

That should keep the plugin from writing any tags to calibre. I'd turn on 'Keep Existing Tags' on the 'Basic' config tab, too.

Jim

soleta
01-20-2012, 11:43 AM
Thank you, thank you! That's perfect. :D

richreads
01-22-2012, 11:52 PM
Hi.
Love the plugin.
I just started using it to add stories from fanfiction.com.
It does add the story but then I get an error when I sort by date.
If I shut down calibre and restart it i can sort by date without the error reappearing.
The error also disappears (with restarting calibre) if I remove the story that was just added.
I've added a book manually with out using the plugin and did not get the error so it seems to be the plugin.
I tried with two different authors and received a message both times.

Here is the error message.
calibre, version 0.8.36
ERROR: Unhandled exception: <b>TypeError</b>:can't compare offset-naive and offset-aware datetimes

Traceback (most recent call last):
File "site-packages\calibre\gui2\library\models.py", line 319, in sort
File "site-packages\calibre\gui2\library\models.py", line 327, in _sort
File "site-packages\calibre\library\caches.py", line 995, in sort
File "site-packages\calibre\library\caches.py", line 1015, in multisort
File "site-packages\calibre\library\caches.py", line 1031, in __cmp__
TypeError: can't compare offset-naive and offset-aware datetimes

JimmXinu
01-23-2012, 12:09 AM
Hi.
Love the plugin.
I just started using it to add stories from fanfiction.com.
It does add the story but then I get an error when I sort by date.

Glad you like it. :)

As for the bug--it's fixed in 1.2.3. Thanks for reporting it.

Jim

JimmXinu
01-23-2012, 12:15 AM
FYI, anytime there's a new version of a plugin, there's a delay before it shows up on the official list of plugins that calibre watches. (It's managed manually by kiwidude and Kovid.)

But you don't have to wait for it. You can go into your User Plugins, view your Installed plugins, choose FanFictionDownLoader and, even though it says there's no new version, click 'Install'. Restart calibre and you'll have the latest version.

Jim

richreads
01-23-2012, 12:37 PM
thanx for the quick update - that fixed it.

martinjh99
01-25-2012, 12:41 AM
Latest version of the plugin and Calibre 0.8.36 Windows 7 64bit...

Download and install the plugin from both the plugin list in Calibre and manually downloaded and installed from the first post in this thread.

It doesn't show up in the Toolbar options so i can add it and when i click on the plugin to configure it it always says that I need to restart Calibre when I have already re-started Calibre.

Am I doing anything wrong here or is it a problem with soomething...?

theducks
01-25-2012, 12:14 PM
Latest version of the plugin and Calibre 0.8.36 Windows 7 64bit...

Download and install the plugin from both the plugin list in Calibre and manually downloaded and installed from the first post in this thread.

It doesn't show up in the Toolbar options so i can add it and when i click on the plugin to configure it it always says that I need to restart Calibre when I have already re-started Calibre.

Am I doing anything wrong here or is it a problem with soomething...?

You either missed the configuration step, or you placed it on the wrong (for you) toolbar. There are 2 versions of the same toolbar: Normal and Device connected. If you want it on Both, you need to add it to both.

Spending some time with Preferences Toolbar will set things straight. :thumbsup:

martinjh99
01-25-2012, 12:20 PM
You either missed the configuration step, or you placed it on the wrong (for you) toolbar. There are 2 versions of the same toolbar: Normal and Device connected. If you want it on Both, you need to add it to both.

Spending some time with Preferences Toolbar will set things straight. :thumbsup:

Thanks - actually seems to have worked now I am in a library with books I downloaded from website...

Cheers anyway!

JimmXinu
01-25-2012, 12:28 PM
Hmmm. Not sure what happened there; what you have in your library shouldn't make any difference. But I'm glad it got sorted out.

martinjh99
01-25-2012, 01:55 PM
Yeah thanks Jimmy nice little plugin - Was looking for something to make it easier to read longer stories from Fanfiction.net :D

Cheers Mate

JimmXinu
01-27-2012, 06:10 PM
I'm again looking for some people to beta test the attached newest version before I put it out for everyone.

This version contains:

New feature: Fill Custom Columns with Story Metadata. Look in Config to set it up.
New feature: Collect and update Series column.
Improved drag-and-drop of URLs onto the Add New URL text box. Dragging URLs in adds now instead of replaces, and copies the URL, not the pretty text.
Added a 'show this again' checkbox to the dialog for reviewing updates before applying them. When unchecked, updates will be applied immediately if there were no problems.
Added a progress bar while applying updates, which can be lengthy.
Don't disable menu items(like update) when there are no books selected. It broke the keyboard shortcut if the menu hadn't rebuilt since books were selected.
Make the FanFictionDownLoader button (which does add or update, depending on whether books are selected) available for keyboard shortcut.
Fix date change for adastra.
Fix URL pattern, metadata collection for SIYE.

I appreciate your feedback.

Adding Series support turned out to be easier than I expected and there's not been a lot of testing on this yet, so I rolled it into 1.3.1.

(Look at the personal.ini config tab if you don't want the series column to update, but you do want other calibre Metadata updated.)

Thanks,
Jim

JimmXinu
01-28-2012, 12:56 AM
This version contains:

New feature: Fill Custom Columns with Story Metadata. Look in Config to set it up.
New feature: Collect and update Series column.



Second new feature added to the most recent beta now. FYI.

x85849
01-28-2012, 04:36 PM
New feature: Fill Custom Columns with Story Metadata. Look in Config to set it up.

Thank you for this feature. I just started using this plugin and was trying to figure a way to fill the rating info into a custom column in calibre. Lo and behold you updated the plugin with just that feature. Thank you.

The copy to custom column feature seems to work well with the stories I pulled from fanfiction.net. At first it did not appear to work with the rating setting, however I realized that the issue was that my custom column was using a "text, but with a fixed set of permitted values" field and I did not have all of the story ratings in my list. I created a new column of just text and it imported the rating just fine after that.

JimmXinu
01-28-2012, 04:39 PM
@x85849 - You're quite welcome. :)

Jade Aislin
01-28-2012, 07:06 PM
And here I thought I'd have to fill out my custom columns manually from the data in the tags. I love this new feature. The series feature is nice too, at least for those sites that have series. It's too bad that FFN doesn't. Still, I am very pleased with this plugin.

Good work!

jackr
01-30-2012, 02:09 AM
Amazing work! Solved a number of things on my to do list :D.

I don't know if the actual site adapter is the problem, but fimfiction.net doesn't split "My Little Pony" off into Category and the genre tags off into the Genre section.

And pardon me if this is a noob question :o, but can the progress windows be placed inside the jobs section instead? I have a large list of fanfics now that I regularly check for updates, and I can't go off and do something else in Calibre until the metadata downloading is finished.

JimmXinu
01-30-2012, 01:56 PM
Amazing work! Solved a number of things on my to do list :D.
Glad you like it. :)

I don't know if the actual site adapter is the problem, but fimfiction.net doesn't split "My Little Pony" off into Category and the genre tags off into the Genre section.
I didn't write the fimfiction.net adapter myself, but that should be easy to fix.
And pardon me if this is a noob question :o, but can the progress windows be placed inside the jobs section instead? I have a large list of fanfics now that I regularly check for updates, and I can't go off and do something else in Calibre until the metadata downloading is finished.
Metadata download runs in the foreground so it can stop and ask for user/pass if needed and check the existing book vs the site. I'll think about it, but I don't see that changing.

IMO, you should use the email notifications from the sites to inform you when your stories are updated. Using this to poll them all everyday seems rather wasteful to me and inflates the stories' hit counts.

Jim

JimmXinu
01-30-2012, 10:45 PM
Version 1.3.2 - 30 Jan 2012
New site: archiveofourown.org -- Thanks for Ida Leter for implementing this.
Change fimfiction.net to use genre instead of category.

jackr
01-31-2012, 12:57 AM
Thank-you for the fix :D. I wish I found this sooner :o, but I found another issue. I am not sure what is going on, but for my fimfiction.net fics that had updates in Jan, the year is set at 2011 instead of 2012. Not sure if the fimfiction forgot to roll over the year or not :P.

JimmXinu
01-31-2012, 04:52 PM
I am not sure what is going on, but for my fimfiction.net fics that had updates in Jan, the year is set at 2011 instead of 2012. Not sure if the fimfiction forgot to roll over the year or not :P.
Oh, it's worse than that. fimfiction doesn't give the year anywhere so we left it hardcoded to 2011. We can at least do better than that.

Somebody might want to suggest to the fimfic people that they add a year before Jun, when their oldest story is dated.

JimmXinu
01-31-2012, 05:00 PM
@jackr - There's a new version, 1.3.3, that handles the fimfic dates better, but not perfectly. Now it assumes "this year", unless that date would be in the future, then it subtracts a year.

Until fimfiction.net adds the year, that's as good as it's going to get.

jackr
02-01-2012, 06:46 AM
Thanks for the update! I sent a request to the creator of fimfiction asking if a year could be added.

leistnerm
02-02-2012, 01:14 AM
Any chance of getting any of the following sites supported?

http://www.fanficauthors.net/
http://www.hpfanficarchive.com/
http://thequidditchpitch.org/
http://www.phoenixsong.net/
http://restrictedsection.org/
http://www.mugglenet.com/
http://checkmated.com/
http://mvsfiction.com/
http://sugarquill.net/

... or any documentation on how you go about creating a new site adapter? I can scrape web pages easily enough in C# or similar languages, but don't have much experience with Python.

JimmXinu
02-02-2012, 01:32 PM
Any chance of getting any of the following sites supported?
...
It's unlikely that I'm going to write adapters for them. But we're happy to accept other people's code to support new sites.
... or any documentation on how you go about creating a new site adapter? I can scrape web pages easily enough in C# or similar languages, but don't have much experience with Python.
There's been a recent discussion about "Writing adapters" on the Google group:

http://groups.google.com/group/fanfic-downloader/t/7d3091e8dd61e961

I suggest you start there.

Jim

leistnerm
02-02-2012, 11:20 PM
Jim -- Thanks for the link. That definetly helps point me in the correct direction. I've been looking over the code to get an idea of how it's structured and how python works. So far it seems pretty straight forward.

I did notice one thing that I'll probably work on, but just in case I can't figure it out, I'll let you know too ;) It doesn't appear you are getting urls using gzip or deflate from the server. I have a similar progam I wrote late last year that DL's any story on FF.net over 25k words and matching a few other criteria automatically. When I started using gzip over just normal requests it started running at least an order of magnitude faster.

I did manage to find a few pages on how to do this in python, but not with the method you are using to retireve urls. So I'll probably play around with that for a starter... something easy before I try to do a whole file myself.

http://stackoverflow.com/questions/3947120/does-python-urllib2-will-automaticly-uncompress-gzip-data-from-fetch-webpage
http://techknack.net/python-urllib2-handlers/

Anyways, thought I'd at least metion my previous experience with a progam doing something similar to this and the speed increase I got.

Looking forward to hopefully being about to eventually contribute to this already great plugin.

Mark

JimmXinu
02-02-2012, 11:57 PM
It doesn't appear you are getting urls using gzip or deflate from the server. I have a similar progam I wrote late last year that DL's any story on FF.net over 25k words and matching a few other criteria automatically. When I started using gzip over just normal requests it started running at least an order of magnitude faster.
That's a very good point.

For a very long time, our focus for this code was primarily on the web version that runs on Google AppEngine. I'm pretty sure I remember confirming that AppEngine did "Content-Encoding: gzip" automatically on the ebook files when they were downloaded from the web app.

But I don't remember ever thinking about it on our client side. It's certainly worth a look, thanks for the suggestion.

BTW, ff.net in particular is slower than it could be--there's a hard coded 0.5sec sleep between hits (plus the slow_down_sleep_time, if set). We have first hand evidence from one of the other developers that ff.net will restrict your IP if you hit them "too fast". See http://www.fanfiction.net/tos/ section 4E.

Jim

leistnerm
02-03-2012, 12:17 AM
That's a very good point.

For a very long time, our focus for this code was primarily on the web version that runs on Google AppEngine. I'm pretty sure I remember confirming that AppEngine did "Content-Encoding: gzip" automatically on the ebook files when they were downloaded from the web app.

Jim

Hmm. I've been pegging them pretty hard for a while and not been blocked. I guess I should be careful on that. And after further digging, it looks like it has code to support gzip, in the __decode method/function/whatever python calls them. Perhaps the slowdown is just due to that delay.

Anyways thanks for your time :)

JimmXinu
02-03-2012, 01:35 AM
Using the GZipProcessor from http://techknack.net/python-urllib2-handlers/, it's actually really easy to implement.

Some fairly cursory testing seems to indicate that there's no real improvement for ff.net. And several of our supported sites won't send us gzip (looks like it's due to User-Agent). I did see ~30% improvement with TtH, though.

That's as much effort as I'm going to put into it tonight. :)

ilovejedd
02-03-2012, 11:53 AM
Just tried it last night. Excellent plugin. Wish it was available 3 years ago when I started converting fanfics to ebooks.

iatheia
02-05-2012, 11:38 PM
Any chance of getting any of the following sites supported?

http://www.fanficauthors.net/
http://www.hpfanficarchive.com/
http://thequidditchpitch.org/
http://www.phoenixsong.net/
http://restrictedsection.org/
http://www.mugglenet.com/
http://checkmated.com/
http://mvsfiction.com/
http://sugarquill.net/

... or any documentation on how you go about creating a new site adapter? I can scrape web pages easily enough in C# or similar languages, but don't have much experience with Python.

I can take a look at a few more populated ones in a bit.

Ida.

Ayiana
02-05-2012, 11:59 PM
First off, thank you so much for crafting this plugin. In the past I've spent untold hours downloading fics manually, adding them to Calibre, and then filling in all the fields. This plugin, if I can get it to work right, is the answer to a prayer.

Here's what's going on, though. I installed the plugin, customized my preferences so that the downloader icon should appear in my menubar whether or not my Kindle is connected, and restarted Calibre. When Calibre first starts, everything looks fine. Then the ffdownloader icon vanishes, so I'm assuming it's crashing? If I click on it before it crashes, I get this error message:

calibre, version 0.8.38
ERROR: Unhandled exception: <b>IndexError</b>:list index out of range

Traceback (most recent call last):
File "calibre_plugins.fanfictiondownloader_plugin.dialog s", line 241, in do_loop
IndexError: list index out of range

I've tried uninstalling and reinstalling the plugin several times, even with reboots/restarts in between, to no avail. Any ideas?

JimmXinu
02-06-2012, 11:17 AM
Here's what's going on, though. I installed the plugin, customized my preferences so that the downloader icon should appear in my menubar whether or not my Kindle is connected, and restarted Calibre. When Calibre first starts, everything looks fine. Then the ffdownloader icon vanishes, so I'm assuming it's crashing?
I don't think I've ever seen the button disappear after it was on screen--and believe me, I crashed it a lot during development. :)
If I click on it before it crashes, I get this error message:

calibre, version 0.8.38
ERROR: Unhandled exception: <b>IndexError</b>:list index out of range

Traceback (most recent call last):
File "calibre_plugins.fanfictiondownloader_plugin.dialog s", line 241, in do_loop
IndexError: list index out of range
The only way I know to get that error is to click the plug in with no books selected (or choose 'Add New from URL(s)'), then *not* enter any URLs and click OK.

If you could start calibre in debug mode (calibre-debug -g) and post the debug output, that would be helpful.

Jim

Ayiana
02-06-2012, 11:53 AM
Well, I'm utterly baffled now.

I uninstalled the plugin and shut down my computer last night right after I posted here. Reinstalled this morning, set my preferences, and ... everything's fine. No crash. No error message. Color me confused but relieved.

And thank you again. This plugin is a fantastic addition to Calibre's functionality.

JimmXinu
02-07-2012, 12:15 AM
@Ayiana - Glad you like it. Hopefully it stays fixed.

Laia
02-07-2012, 02:13 PM
Hi Jim,
I downloaded the link "FanFictionDownLoaderPlugin-1.3.4.zip" but it's a .zip, where I had to click to open the program?

Thank you

davidfor
02-07-2012, 09:45 PM
I downloaded the link "FanFictionDownLoaderPlugin-1.3.4.zip" but it's a .zip, where I had to click to open the program?

Save the file to disk. Then open Calibre and it's preferences. Click on "Plugins" (near the bottom-left of the window) and then click on "Load plugin from file". In the file picker window, find the pligin zip file, select it and press "Open". Then follow the rest of the prompts to install and configure.

JimmXinu
02-07-2012, 09:50 PM
@Laia - What davidfor said is correct.

Or, you don't need to download it yourself--calibre can do it for you.

For full instructions on installing plugins, see:
http://www.mobileread.com/forums/showthread.php?t=118680

jjm6383
02-07-2012, 10:23 PM
Hi,

I really like this plugin. It's making my life easier.

I haven't been a big user of FanFiction Downloader. I use Deffnet. With Deffnet, you could set a simple CSS file, so stories came out lightly formated. Can you do this with the plugin? Or could this be added to the plugin at a later time?

Thanks,
Joyslin

JimmXinu
02-08-2012, 07:13 PM
I haven't been a big user of FanFiction Downloader. I use Deffnet. With Deffnet, you could set a simple CSS file, so stories came out lightly formated. Can you do this with the plugin? Or could this be added to the plugin at a later time?
Glad you like it.

Do you use the HTML output? epub, mobi and obviously txt are more limited in what they can use for CSS. And the story text HTML is taken from the source sites as untouched as possible. Deffnetizer only deals with ffnet, which for all its sins is extremely consistent in it's HTML. Trying to wrap custom html and CSS around all the different sites would be... interesting, I fear.

I'll add user CSS to the wish list, but my free time has largely dried up again for a while, so don't hold your breath. :)

anesthezea
02-08-2012, 10:47 PM
On some stories (not all) I get the following error:

'NoneType' object has no attribute 'parent'

Show Details:

calibre, version 0.8.37
ERROR: Unhandled exception: <b>IndexError</b>:list index out of range

Traceback (most recent call last):
File "calibre_plugins.fanfictiondownloader_plugin.dialog s", line 241, in do_loop
IndexError: list index out of range

Help?

JimmXinu
02-08-2012, 10:55 PM
You're the second person to report an error occurring on that line.

The only way I know to get that error is to click the plug in with no books selected (or choose 'Add New from URL(s)'), then *not* enter any URLs and click OK.

(Which is a case I should start checking for, but haven't yet.)

If you could start calibre in debug mode (calibre-debug -g) and post the debug output, that would be helpful.

Or if it happens consistently on the same stories, knowing the URLs for those stories would help me a lot.

anesthezea
02-09-2012, 01:17 AM
ETA: I updated the plugin (didn't realize it was one version out of date) and the "parent" error seems to have been fixed. Thank you!

JimmXinu
02-11-2012, 04:05 PM
Testing on this version has been finished.

I'm again looking for some people to beta test the attached newest version before I put it out for everyone.

This version contains:

New feature - Allow user to set custom CSS in personal.ini for HTML and EPUB output.
New feature - Allow user to set custom regular expressions in personal.ini to modify metadata.
New feature - Use Accept-Encoding=gzip to speed download. (Not sites will use it--it's common for sites to block gzip based on User-Agent.)
Add progress bars while collecting URLs from stories for list and for updates.


Here's an example of what the default output_css parameter for epub looks like:
## output background color--only used by html and epub (and ignored in
## epub by many readers). Included below in output_css--will be
## ignored if not in output_css.
background_color: ffffff

## Allow customization of CSS. Make sure to keep at least one space
## at the start of each line and to escape % to %%. Also need
## background_color to be in the same section, if included in CSS.
output_css:
body { background-color: #%(background_color)s;
text-align: justify;
margin: 2%%; }
pre { font-size: x-small; }
sml { font-size: small; }
h1 { text-align: center; }
h2 { text-align: center; }
h3 { text-align: center; }
h4 { text-align: center; }
h5 { text-align: center; }
h6 { text-align: center; }
.CI {
text-align:center;
margin-top:0px;
margin-bottom:0px;
padding:0px;
}
.center {text-align: center;}
.cover {text-align: center;}
.full {width: 100%%; }
.quarter {width: 25%%; }
.smcap {font-variant: small-caps;}
.u {text-decoration: underline;}
.bold {font-weight: bold;}

Here's an example of what some replace_metadata lines might look like.

## Use regular expressions to find and replace (or remove) metadata.
## For example, you could change Sci-Fi=>SF, remove *-Centered tags,
## etc. See http://docs.python.org/library/re.html (look for re.sub)
## for regexp details.
## Make sure to keep at least one space at the start of each line and
## to escape % to %%, if used.
replace_metadata:
(.*)-Centered=>
Magical Girl Lyrical Nanoha=>Nanoha
Puella Magi Madoka Magica.*=>Madoka
(Friend)(ship)=>\2\1
(.*)Great(.*)=>\1Moderate\2
(?s)(.*)suck(.*)=>\1kcus\2

"(.*)-Centered=>" removes all Name-Centered tags entirely (TTH).
"Puella Magi Madoka Magica.* =>Madoka" changes ffnet's "Puella Magi Madoka Magica/魔法少女まどか★マギカ" to just Madoka.
"(Friend)(ship)=> \2\1" changes Friendship to shipFriend. Not useful, but demonstrates regular expression group substitution.
"(.*)Great(.*)=>\1Moderate\2" shows changing one word in the middle of a tag, such as a series name.
"(?s)(.*)suck(.*)=>\1kcus\2" (?s) tells the regexp engine to allow . to make newlines, too. Needed to affect many descriptions/summaries.

The replacement pattern must match the entire string for a piece of metadata to be changed. "Friend" will not match category "Friendship", but "F.*" will. Replacements are applied to all metadata, so be careful to be specific.

You can have different lists of replacements for different sites. One use could be to normalize the rating tags different sites use to all be the same.

JaneD
02-13-2012, 08:51 PM
I love this plugin. Such a convenience to be able to load stories directly into calibre!

My only problem with it is that I seem to be getting some visible code on italics, elipses or dashes, and sometimes at the end of sentences. For example, on this story on AO3: http://archiveofourown.org/works/338120/chapters/547213.

Thank you so much for this wonderful plugin! I particularly like how it will grab an update on the latest chapter of a story without having to re-enter the story url. Brilliant. :thumbsup:

JimmXinu
02-13-2012, 09:39 PM
@JaneD - Glad you like it.

Those "&#xA0;" etc entities showing up are from a bug in the downloader--thanks for pointing it out.

I'll have a fixed version up later this evening.

JaneD
02-14-2012, 11:58 AM
Just reloaded the story and it works perfectly. Thank you for the quick reply and update!

Jade Aislin
02-14-2012, 03:55 PM
I love the feature that puts tags into custom columns, especially with FFN stories. However, I have recently noticed that not all of the characters are getting put into the correct column. Most of the stories will put the characters in, but any story that has Harry Potter as a category does not grab any characters. The characters do not get put into my tags either. I'm not sure if there is any other category having this problem.

Here is one of the urls that should have characters: http://www.fanfiction.net/s/5930194/1/

Edit: Looking on my list of stories, I realized that all the stories that are not retrieving the characters are crossovers. Each story has to categories or fandoms. I do not know why the plugin is having trouble retrieving the character information with these stories, but it is not just stories in the Harry Potter category.

JimmXinu
02-14-2012, 10:29 PM
@Jade Aislin--The hoops I have to jump through to parse out characters on ffnet are significant. It doesn't surprise me that it was flawed.

But I think I have it figured out. Fix will be in the next version.

Jade Aislin
02-15-2012, 12:42 AM
@Jade Aislin--The hoops I have to jump through to parse out characters on ffnet are significant. It doesn't surprise me that it was flawed.

But I think I have it figured out. Fix will be in the next version.

That's great. I really appreciate this feature and the effort you went through to make it as I don't need to pre-sort the stories by character.

Truthfully, I may not have noticed it if I hadn't been thinking about using the new feature that allows the user to edit metadata. I already tried it by modifying some long categories and some categories that have commas in them which made the pluging think there were two categories instead of one (ex: House, M.D.). It worked great, but I was thinking about using the feature to change the character based on the category so that it reads as category.character instead of just the character. Even if it can't be done, it's an easy matter to manually change the characters if I already have the character and categories.

JimmXinu
02-15-2012, 11:05 AM
... but I was thinking about using the feature to change the character based on the category so that it reads as category.character instead of just the character. Even if it can't be done, it's an easy matter to manually change the characters if I already have the character and categories.
Crossover stories can have multiple categories, making it prohibitively difficult to do in the downloader.

So I think you're on your own for that one. :)

85eren
02-19-2012, 07:42 AM
I love this plugin. Thank you.

I don't know if I'm doing something wrong, but none of the files I've downloaded using the plugin, have chapter breaks on the Locations bar on the bottom of the page, when I put them on the kindle.
What am I doing wrong?

iatheia
02-19-2012, 10:06 AM
I love this plugin. Thank you.

I don't know if I'm doing something wrong, but none of the files I've downloaded using the plugin, have chapter breaks on the Locations bar on the bottom of the page, when I put them on the kindle.
What am I doing wrong?

That is because mobi as a proprietary format needs to die in fire. Advise is to use epub, and then convert it to mobi using Calibre, then it sort of looks normal.

JimmXinu
02-19-2012, 11:30 AM
That is because mobi as a proprietary format needs to die in fire.
:thumbsup: Exactly!

I spent the better part of a week once trying to get those chapter marks to work in our mobi output and failed. I had removed mobi output support entirely at one point (before the plugin), but the web site users asked for it back, even if it is flawed.

JaneD
02-19-2012, 02:50 PM
That is because mobi as a proprietary format needs to die in fire. Advise is to use epub, and then convert it to mobi using Calibre, then it sort of looks normal.

This is what I do, and it looks perfect on my kindle, chapter breaks, TOC & all. Just remember to convert again every time you update a chapter! :thumbsup:

JimmXinu
02-19-2012, 02:57 PM
This is what I do, and it looks perfect on my kindle, chapter breaks, TOC & all. Just remember to convert again every time you update a chapter! :thumbsup:
I highly recommend the Reading List plugin for sending books to kindle. Using Reading List, calibre automatically converts to mobi and sends the files to my kindle when I plug it in.

The FFDL plugin is even integrated with Reading List (http://code.google.com/p/fanficdownloader/wiki/FanFictionDownLoaderPluginWithReadingList).

85eren
02-20-2012, 04:11 PM
Thank you for all your help. Getting the file in epub and converting to mobi worked great.

jackr
02-20-2012, 06:18 PM
Is it possible to grab an image (if the site has images for their fanfictions) and apply it automatically to the cover? And if there is no cover, an option to have Calibre generate a cover?

I can survive by copy-pasting the images, but if fanfic images get popular on other sites, it would be a nice feature to have :D.

JimmXinu
02-20-2012, 07:17 PM
@jackr - I'd like someday to d/l and include images, but it doesn't now. Which site(s) are you talking about having cover images?

Have you looked at the Generate Cover (http://www.mobileread.com/forums/showthread.php?t=124219) plugin?

jackr
02-20-2012, 11:43 PM
Namely fimfiction.net. A good portion of the fics there have cover art for them. Ex: http://www.fimfiction.net/story/8933/ and http://www.fimfiction.net/story/2766/.

I have played with Generate Cover a little bit, makes adding the word 'Complete' in the center of every cover for completed fics a snap for Nook browsing. :D

Your new EpubMerge plugin looks great too; I will need to remind myself to only use it on the device however. :)

mhomann
02-21-2012, 07:24 AM
I have come across a fanfic that gives me a book with 40 empty chapters in calibre:
http://www.fictionalley.org/authors/barb/replay.html

what do i do about that?

JimmXinu
02-21-2012, 04:42 PM
I have come across a fanfic that gives me a book with 40 empty chapters in calibre:
http://www.fictionalley.org/authors/barb/replay.html

what do i do about that?
Reporting it here is good. New version with a fix for that is coming shortly.

cklls
02-22-2012, 01:13 PM
Hi Jim - Just tried to download a completed story from ff.net and got this error message:

coercing to Unicode: need string or buffer, NoneType found

The url of the story is:

http://www.fanfiction.net/s/6141793/1/In_Perfect_Light

Any ideas?

JimmXinu
02-22-2012, 01:15 PM
ffnet changed their format slightly--just enough to break things, really.

Update to the latest version of the plugin and it should be good again.

cklls
02-22-2012, 01:21 PM
They do that often, the buggers. ;) I got the push update a couple of hours ago. Is there another one?

JimmXinu
02-22-2012, 01:22 PM
@cklls - v1.4.5 fixes the problem for me. If it doesn't fix it for you, please let me know and I'll look at it again.

cklls
02-22-2012, 01:32 PM
Just installed that version and tried again. (I had 1.4.4 this morning.) No go - same error.

EDIT: Never mind... I forgot to restart Caliber again. With restart, it works. Thanks!!!

JimmXinu
02-22-2012, 01:36 PM
Oh good--you had me worried for a moment. :)

cklls
02-22-2012, 01:39 PM
:thanks: Don't worry, be happy! It's all good. :)

paegan
02-22-2012, 10:07 PM
Ok first off kudos to you on such a KICK @$$ plugin!:thumbsup: The only other plugin I've found almost as useful is the aforementioned Cover Generator (it's fun to download fonts matching the series to make a halfway decent lookin book cover)

I was a little confused for a bit because, while looking up references to your plugin, I hit upon this site : http://www.fanfictiondownloader.net/ and thought you had made this program and couldn't understand why the calibre plugin didn't have as much features.

I realise now that's not the case but I don't know if you want to get into contact with that author to see if he/they are willing to share info on how they do things. For now I only use it to grab fics from sites you don't support (mostly pervy/adult oriented fic sites:p) but it leaves a lot to be desired in regards to metadata crawling and toc generation. Yours is FAR superior in those areas and also includes the story update function which is a godsend.

My only possible suggestion would be to add support for more sites even without full functionality i.e. toc or metadata. I personally would not mind basic support for the oddball site at least just to grab a fic in its entirety without having to depend on an autopager or worse CTRL+C/V.

Keep up the good work and hope to see lots more in the future...

paegan
02-22-2012, 10:15 PM
Almost forgot...
Dunno if this falls within the scope of your plugin but as an additional feature for the book id's if it would be possible to use customised ids to reference fanfic sites e.g.

http://www.fanfiction.net/s/5149431/1/Stranger_Than_Fanfiction
would be something like:
ffn:5149431

or http://ficwad.com/story/181839
would be fwad:181839

Hopefully you get the idea I'm tryin to put accross

JimmXinu
02-22-2012, 10:19 PM
http://www.fanfiction.net/s/5149431/1/Stranger_Than_Fanfiction
would be something like:
ffn:5149431

or http://ficwad.com/story/181839
would be fwad:181839

Hopefully you get the idea I'm tryin to put accross
Meaning you want to see those somewhere? Because siteabbrev and storyId are already a couple of the metadata fields the plugin makes available.

If you're asking to be able to enter those instead of URLs, I suppose it's possible, but I don't see the point--it's a lot easier, IMO, to copy or drag the URL.

paegan
02-22-2012, 10:52 PM
Meaning you want to see those somewhere? Because siteabbrev and storyId are already a couple of the metadata fields the plugin makes available.

If you're asking to be able to enter those instead of URLs, I suppose it's possible, but I don't see the point--it's a lot easier, IMO, to copy or drag the URL.

Hmmm tried it... dosent work exactly like I hoped it would but meh, it's wasn't a necessity just a nice bit of polish (although if some1 else writes a plugin that does do that I do hope that you'd support it :D)

And to toss another feature idea @ you (which kinda obliquely relates to the custom ids thing):

Story mirrors i.e. the same story from multiple sources eg.
http://ficwad.com/story/182760 and
http://www.fanfiction.net/s/7820904/1/The_bLife_b_bBond_b

Now feel free to tell me where to get off cuz I kno this would be insanely difficult to implement eg. having to deal with choosing which link to update from, which like to crawl for metadata, etc. etc.

Nothing ventured, nothing gained ...

JimmXinu
02-22-2012, 11:03 PM
Hmmm tried it... dosent work exactly like I hoped it would but meh, it's wasn't a necessity just a nice bit of polish (although if some1 else writes a plugin that does do that I do hope that you'd support it :D)
I still don't understand what it is you want here. If you want 'ffnet:1324' to appear in calibre, put siteabbrev and storyId each in a custom column and use a third composite column to display them together. I use that to have a wordcount/chapters column, like: 12,123/7.
Story mirrors i.e. the same story from multiple sources eg.
http://ficwad.com/story/182760 and
http://www.fanfiction.net/s/7820904/1/The_bLife_b_bBond_b
I don't see that there's anything particularly useful to be done there. I've seen the same story on different sites with different names, or chaptered differently, or different author name, etc... You can't make any useful assumptions.

JimmXinu
02-24-2012, 06:47 PM
New Version up with support for a new site: gayauthors.org. Thanks to Ida Leter for implementing this.

JimmXinu
02-27-2012, 03:18 PM
Testing for 1.5 is finished. It's been released for general use.

These changes are all for EPUB format ebooks.

The big changes are:

Support for downloading images in the story text.
Support for cover image from three different sources.
Support to resize, make grayscale, and convert images to jpg.
Option for plugin to update calibre cover image on metadata update.
Major re-write of EPUB update to support images.
Improved handling of ini section priority.
New 'Stay on Top' option for 'Add New from URL(s)' dialog for easier drag-n-drop.
Add replace_hr option to replace <hr> tags with '* * *'
Support for fimfiction.net's locked story passwords.


Images
There's several new config parameters you can add to your personal.ini to turn on these features.

All of them should go under the [epub] section or the epub section for individual sites, such as [www.ficwad.com:epub].

The first turns images on in general. Without this, no images will be collected and it defaults to false.
include_images:true

The next allows the summary section in the title page to preserve its HTML from the source site. That way, images in the summary will be included. More on that in a moment. Summary metadata given to calibre still has HTML stripped for consistency.
keep_summary_html:true

As a convenience, and the draw the attention of more casual users, I added an 'Include images in EPUBs' checkbox to the Basic config tab. This is equivalent to adding:
[defaults]
include_images:true
keep_summary_html:true
make_firstimage_cover:true
...to the top of personal.ini. Your settings in personal.ini will override this.

Covers
There are three places cover images can come from.

First is an explicit cover image from the source site. Today, fimfiction.net is the only one--please let me know (with example URLs) if any of the other sites do, too.

Second, the downloader can use the first image it finds in the story with this paramter. If keep_summary_html is also true, it can come from the summary.
make_firstimage_cover:true

Third, you can set a specific image that will be used if one of the other two isn't found.
[www.adastrafanfic.com:epub]
default_cover_image:file:///C:/Users/username/Desktop/nook/images/icon.png
[www.fictionalley.org:epub]
default_cover_image:http://www.somesite.com/someimage.gif

Finally, if you want images inside the EPUB, but never want an image cover, you can use:
never_make_cover:true

Image options
All images saved to EPUB are converted to jpeg to save size. Further, you can set what size to scale the images down to fit in. Aspect ratio is preserved and images are never scaled up, only down. You can also have them converted to grayscale for additional(minor) space savings.
image_max_size: 580, 725
grayscale_images: true
The default size (580,725--width,height) is for my nook STR. If people tell me what appropriate sizes are for other book readers, I'll include them in the comments.

calibre cover
The plugin has a new option on the Basic tab, "Update Cover when Updating Metadata". When checked, the calibre cover image will be updated everytime the other calibre metadata is updated (assuming EPUB). If there is a cover image in the EPUB, it will be used. If not, calibre will generate the usual 'from EPUB format' type cover, usually showing the title page.

Internal changes
The code for updating existing EPUBs to add new chapters has been completely rewritten. You shouldn't see any change from the outside. Other than supporting images, that is. :)

Priority of personal.ini sections has been more formalized. When downloading an EPUB from ficwad, the sections will take priority in this order, from least to most specific:

[defaults]
[www.ficwad.com]
[epub]
[www.ficwad.com:epub]
[overrides]

Which is why, if you set options in [epub], to override them for ficwad, you have to set them in [www.ficwad.com:epub]

For convenience of cutting and pasting, here's all the new image parameters together:

[epub]
## include images from img tags in the body and summary of
## stories. Images will be converted to jpg for size if possible.
include_images:true

## If not set, the summary will have all html stripped for safety.
## Both this and include_images must be true to get images in the
## summary.
keep_summary_html:true

## If set, the first image found will be made the cover image. If
## keep_summary_html is true, any images in summary will be before any
## in chapters.
make_firstimage_cover: true

## If set, the epub will never have a cover, even include_images is on
## and the site has specific cover images.
#never_make_cover: false

## If set, and there isn't already a cover image from the adapter or
## from make_firstimage_cover, this image will be made the cover.
## It can be either a 'file:' or 'http:' url.
## Note that if you enable make_firstimage_cover in [epub], but want
## to use default_cover_image for a specific site, use the site:format
## section, for example: [www.ficwad.com:epub]
#default_cover_image:file:///C:/Users/username/Desktop/nook/images/icon.png
#default_cover_image:http://www.somesite.com/someimage.gif

## Resize images down to width, height, preserving aspect ratio.
## Nook size, with margin.
image_max_size: 580, 725

## Change image to grayscale, if graphics library allows, to save
## space.
#grayscale_images: true

fimfiction.net locked story passwords
The downloader will now prompt for password on fimfiction.net's locked stories. Because each locked story has a different password, it's not useful to store a password in personal.ini. You can, however, use the fail_on_password option to prevent the plugin from prompting for story passwords.

[www.fimfiction.net]
fail_on_password: true


I appreciate hearing from both people with problems and people without, thanks.

v1.5.1 adds error handing for image loading, stay-on-top
v1.5.2 fixes a problem d/l'ing chapter text in GA.
v1.5.3 fixes a fimficnet problem, addes replace_hr & never_make_cover options.
v1.5.4 more fimfic fixs
v1.5.5 fimfic story password support & fail_on_password option
v1.5.6 add 'Include images in EPUBs' checkbox
v1.5.7 add 'nook_img_fix' - wrap img tags with div if not already wrapped with div or p for a nook problem. Defaults to true.

Jim

jackr
02-28-2012, 04:19 AM
I am very excited with these new updates :D. I am trying the beta out and so far everything seems to work except for when I use "Update Calibre Metadata Only". When I use that option, the cover gets overwritten with a screen-shot of the summary page.

And I noticed that it is set to inject the first image found into the epub, which I like the idea of for sending it to the Nook w/o having to do the reconvert method. But I think having an option as to whether the epub gets injected with the cover or not would be a good idea, if it is possible*

That's one thing I never understood about covers when I was working on an ebub in Sigil once. I was able to assign a cover image and keep it from being within the html pages. But it wasn't kosher, even though the Nook displayed the cover just fine.

JimmXinu
02-28-2012, 01:11 PM
I am very excited with these new updates :D. I am trying the beta out and so far everything seems to work except for when I use "Update Calibre Metadata Only". When I use that option, the cover gets overwritten with a screen-shot of the summary page.
Hmm. I think that's more a matter of labeling than an actual problem. The 'Update Cover' option tells calibre to take the cover from the epub. If the epub hasn't been updated and doesn't have a cover image, it does the usual default. Finding the 'first image' would require downloading the whole story again in the case of a story with no images--not exactly what we want for 'update calibre only'. :)
And I noticed that it is set to inject the first image found into the epub, which I like the idea of for sending it to the Nook w/o having to do the reconvert method. But I think having an option as to whether the epub gets injected with the cover or not would be a good idea, if it is possible*
If you have make_firstimage_cover off and don't have default_cover_image set, the epub won't have a cover in it. Is that sufficient, or do you mean something else?
That's one thing I never understood about covers when I was working on an epub in Sigil once. I was able to assign a cover image and keep it from being within the html pages. But it wasn't kosher, even though the Nook displayed the cover just fine.
:blink: Unfortunately, covers are not specified by the epub standards, so it's a ugly as different readers have different ideas. What I've implemented follows the 'best practices' I've been able to find.

Thanks for the feedback, it's very helpful!

jackr
02-28-2012, 11:46 PM
If you have make_firstimage_cover off and don't have default_cover_image set, the epub won't have a cover in it. Is that sufficient, or do you mean something else?

I tried removing make_firstimage_cover from personal.ini, but a cover is still injected into the epub.

What I mean about the cover is having it as an option so that only Calibre updates the cover. Before when I downloaded a new fanfiction and I either generated a new cover or pasted in an image, that cover wasn't injected into the epub. But maybe that is what you mean and I just can't get it to work yet :o.

Once again though, amazing work :). If I could send more karma pts, I would have. I will go hunting for some stories with images in them to test next.

JimmXinu
02-28-2012, 11:57 PM
I tried removing make_firstimage_cover from personal.ini, but a cover is still injected into the epub.
Is it a story from fimfiction.net? That site has specific cover images, so it gets set whenever the cover image exists and include_images is on. That is a case where a 'don't make a cover' option would be useful, I suppose.

If not, can you give me the story URL? And maybe a copy of your personal.ini--minus user/passwords, of course.
Once again though, amazing work :). If I could send more karma pts, I would have. I will go hunting for some stories with images in them to test next.
Thanks, glad you like it. :cool:

jackr
02-29-2012, 02:39 AM
Yes it was fimfiction stories that I tested with. My personal.ini is just this:
[defaults]
is_adult:true
#Accept-Encoding=gzip

[epub]
include_images:true
keep_summary_html:true
make_firstimage_cover:true
I am not to clear on whether accept-encoding needs a colon or equals, so I have it on standby for now.

An idea that just popped in my head as a possibility for a custom column option is an error code or some sort for stories that have errors when updating. I've come across a couple that no longer exist, and just a bit ago one that had some message about news or some sort for it (its error icon was the red one with a white x, I will check later to see what the exact error was).

Edit:
The error code I got was:local variable 'newsrc' referenced before assignmentfor http://www.fimfiction.net/story/5947/.

Another error code I've gotten is:list index out of rangefor http://www.fimfiction.net/story/9948/.

As well as several:'NoneType' object has no attribute 'find' for fimfiction stories that have either disappeared from the site or have had password locks placed on them by their authors.

JimmXinu
02-29-2012, 01:19 PM
@jackr -

Re: cover image--the plugin gives fimfiction stories a cover whenever story has a cover image on the site and include_images is true. I'll add an option to suppress cover generation in general for such cases.

Re: Accept-Encoding--there's no such option in personal.ini, you can delete it.

Re: http://www.fimfiction.net/story/5947/ --downloaded fine for me with v.1.5.2.

Re: http://www.fimfiction.net/story/9948/ --Failed for me, too. I'll work on it.

Re: disappeared from the site or have password locks --can you give me an example URL for each?

Thanks for your feedback, it's very helpful.

JimmXinu
02-29-2012, 01:31 PM
Re: http://www.fimfiction.net/story/9948/ --Failed for me, too. I'll work on it.
Failed because there's no chapter in that story--it reports 0 words, too. I'll fix the downloader so it doesn't fail outright, but I'm not sure downloading an empty story is useful. :)

JimmXinu
02-29-2012, 02:52 PM
Beta version is now up to v1.5.3:

http://www.mobileread.com/forums/showthread.php?p=1982785#post1982785

v1.5.1 adds error handing for image loading, stay-on-top
v1.5.2 fixes a problem d/l'ing chapter text in GA.
v1.5.3 fixes a fimficnet problem, adds replace_hr & never_make_cover options.

Hopefully this is just about ready to go.

jackr
02-29-2012, 06:14 PM
I didn't notice the update to the beta, as well as the empty story (I just loaded it and noticed it had an image, didn't look at chapter list) X-).

Where is Accept-Encoding? I know you mentioned this feature a while back, but I couldn't find it elsewhere in the plugin. I probably misread your post about it :o.

http://www.fimfiction.net/story/9439/ is an example for a story that no longer exists.
And http://www.fimfiction.net/story/6884/ for a password locked story.
Both of these get the 'NoneType' object error.

I ran the plugin (v.1.5.3), and came up with a lot more errors than in 1.5. A majority of these new errors are:local variable 'datePublished' referenced before assignmentAlso the story that you mentioned worked in 1.5.2 has en error for me in 1.5.3. (http://www.fimfiction.net/story/5947/)

I've uploaded an image showing all the errors I got this time around. http://i39.tinypic.com/16k57o4.png

JimmXinu
02-29-2012, 07:28 PM
@jackr - Accept-Encoding=gzip is an HTTP protocol parameter used down inside the system when it's getting web pages from sites. It's an internal thing that you don't need to worry about.

Missing fimfic stories now handled correctly. It still errors, but at least it's a StoryDoesNotExist error now.

Wow--password per story? That's a new one. I did not know fimfic did that. I've tried to implement it using the user/pass mechanism. If it doesn't work, can you give me a story and it's password? That would help (PM would be fine).

Ah. Oops. The datePublished error is a mistake--the 'fix' for empty stories broke stories with an earlier start date. My bad. Wait, no it didn't--it revealed another, different bug. So, yay, I guess? :shrug:

As for http://www.fimfiction.net/story/5947/, not sure what to say--it still works for me. I suggest doing an 'Overwrite always' on that one and see if that works. Or an 'Add New', if you're paranoid.

v1.5.4 is up on the original beta post.
http://www.mobileread.com/forums/showthread.php?p=1982785#post1982785

Thanks for your help figuring these out.

jackr
02-29-2012, 08:32 PM
Yup overwriting the story fixed the issue, whatever it was :D. The password lock is rather new for me too. I will see if I can place a dummy story on the site and lock it, as I don't know any passwords to the ones that are locked.

Update:
I created a locked story on fimfiction, I will PM you with the details.

JimmXinu
02-29-2012, 08:52 PM
Interesting. If you see the newsrc problem again, I'll probably need a copy of the offending epub to track it down. Hopefully it was just an issue with the different beta versions that won't crop up again.

Locked story works for me, but I want to change the password dialog a bit so people aren't using their account user/pass.

I assume the other issues were all resolved.

Again, thanks for your help.

jackr
02-29-2012, 09:30 PM
Ah the error messages are much more legible now :D. I didn't encounter the newsrc error again.

As for individual story passwords, of the locked stories I have in my Calibre, I downloaded before the author locked them. By the looks of it, fimfiction is set up so an author can lock away their story if they feel it needs revision or whatever else. I think having a setting to disable password prompt for stories that don't have a defined password would be excellent.

JimmXinu
02-29-2012, 10:21 PM
Good suggestion.

v1.5.5 is up on the original beta post.
http://www.mobileread.com/forums/showthread.php?p=1982785#post1982785

jeaterod
03-01-2012, 02:28 AM
can you add http://www.adultfanfiction.net to the downloadable?

JimmXinu
03-01-2012, 03:51 PM
@jeaterod - Sorry, I'm not touching a site with that much legal warning on it.

http://anime2.adultfanfiction.net/form_adult.php

JimmXinu
03-01-2012, 04:02 PM
Okay, hopefully this really is the last beta version for this round.

v1.5.6 add 'Include images in EPUBs' checkbox

http://www.mobileread.com/forums/showthread.php?p=1982785#post1982785

If nobody's reported any problems with it by this evening(4-5 hours from now), I'll put it out for everyone.

Thanks to everyone who's been helping test this.

jackr
03-01-2012, 05:39 PM
This has only happened occasionally, and it might be something to do more with Calibre itself perhaps, but when I go to update fanfics and the process is on the 'Downloading metadata for stories' window, sometimes the process will stop and disappear when I go an load other windows.

I think it might be happening because I have set Calibre to be appear as an icon next to my clock. I noticed that the plugin will sometimes freeze and flash away and back between each story for metadata downloading. So maybe during that brief window, Calibre decides it can go in the background? :blink:

It's not a big thing, has only happened about 2 or 3 times now. And it seems to occur most during the beginning of the download process.

JimmXinu
03-01-2012, 06:12 PM
@jackr - I've never tried the system tray feature of calibre before.

I bet that you tried to minimize calibre to the system tray.

If I click the calibre system tray icon to hide calibre while it's downloading metadata, it aborts the plugin as if I'd clicked the close button on the progress bar.

I see the same behavior from Generate Cover, so I expect any in-progress action that can be cancelled, will be. Copying from one library to another doesn't get cancelled, but neither does it cancel when you click it's close button.

So I'm going to call it a general calibre behavior. I think being able to abort the download is more important than hiding to system tray.

jackr
03-01-2012, 08:07 PM
I had a feeling it was more of a Calibre thing. And I probably did try to minimize without thinking. :) So far the plugin is working solid.

Is it possible to combine the cover with the jacket inside the epub? Like having it between the title/author and other metadata/summary? Or would that not work with the jacket containing just metadata info?

JimmXinu
03-01-2012, 08:13 PM
EPUB best practices say that the cover image has to be just an image. There's already a very nice Generate Cover (http://www.mobileread.com/forums/showthread.php?t=124219) plugin that will create an image with title, etc on it. And you can choose '{Current Cover}' to combine with the existing cover image.

Does that help you any?

jeaterod
03-01-2012, 08:22 PM
aww thats bs legal warning tho... guess ill just have to keep using flagfic to get downloads from that site

jackr
03-01-2012, 08:27 PM
I will find a way to make do, thanks for the reply :). I hope that epub evolves a little in the future, I just find this idea so appealing for some reason :o: http://i42.tinypic.com/qxm3uw.png

JimmXinu
03-01-2012, 08:59 PM
Hmm. I guess I didn't understand what you were asking for.

Inserting the cover image into the title page isn't unreasonable. Not in tonight's release, but I'll look into it.

JimmXinu
03-01-2012, 09:29 PM
FYI, v1.5.7 has been released.

jeaterod
03-02-2012, 06:07 AM
is there a way for me to tweek the plugin so that when it get the chapter number the numbers 1-9 become 01-09?

JimmXinu
03-02-2012, 02:10 PM
The plugin doesn't number the chapters. Any chapter numbers are coming from the source site. So, no, not really.

cklls
03-02-2012, 03:48 PM
Just wanted to say thanks again, Jim. This tool keeps getting better and better, and makes my obsession easier to support. Um... yeah. ;) :blush:

JimmXinu
03-02-2012, 04:51 PM
Glad I could be an enabler for you, cklls!

Wait... :uhoh:

:D

jeaterod
03-02-2012, 07:37 PM
so you cant tell the plugin to look at it and if the value is lower than 10 add 0 to the front?
....

JimmXinu
03-02-2012, 07:53 PM
"Prologue"
"Chapter 1"
"Mission Briefing"
"Chapter One"
"Chapter I"
"Ch 2"
"3rd Chapter"
"7. Girl Talk"
"9. Chapter 9"
"‘Problems in Engineering'"

These are all chapter names from various stories on sites. It's a free form text value, not just a number. And it's not uncommon for file0001 to be 'Prologue' and file0002 'Chapter 1'. Or '2. Chapter 1', depending on the site.

I could give you an option like replace_metadata in personal.ini to modify chapter titles with a regular expression replacement, but you'd have to work out the expression to your own satisfaction.

jeaterod
03-02-2012, 08:44 PM
oh sorry i dont mean in the story it self i mean in the library colomns

JimmXinu
03-02-2012, 08:58 PM
Oh, you don't mean chapter numbers, you mean the number of chapters.

You want to use an integer custom column and then set the format to: {0:02d}

jeaterod
03-02-2012, 09:03 PM
ok ill try that

EDIT:
awesome thanks it worked great

animealam
03-04-2012, 10:33 PM
First of all, thanks for the plugin, it's awesome.:2thumbsup:thumbsup:

I read some of the info you have for adding a site, but frankly python is way over my head. I was going to ask for Adultfanfiction.net, but saw you won't add it sadly. It is completely adult oriented, hence the over abundance of warnings. :(

Anyway, could you maybe add HP Fandom.net (http://www.hpfandom.net/eff/index.php)?
It's an old eFiction based site (1.1 I think) and it's a large archive, actually one of the largest all Harry Potter out there.

Then there is Walking the Plank (http://www.walkingtheplank.org/archive/).

Hopefully you'd be able to add them.

JimmXinu
03-04-2012, 10:43 PM
First of all, thanks for the plugin, it's awesome.:2thumbsup:thumbsup:
You're welcome. :)
Anyway, could you maybe add HP Fandom.net?
It's an old eFiction based site (1.1 I think) and it's a large archive, actually one of the largest all Harry Potter out there.

Then there is Walking the Plank, and The Silver Snitch II, both are eFiction sites.
First, I suggest including the URL when requesting sites.

Second, unless it's something that's personally interesting to me, I'm working on other things than new sites lately. But we do also have other developers who might be interested.

iatheia
03-05-2012, 12:57 AM
First of all, thanks for the plugin, it's awesome.:2thumbsup:thumbsup:

I read some of the info you have for adding a site, but frankly python is way over my head. I was going to ask for Adultfanfiction.net, but saw you won't add it sadly. It is completely adult oriented, hence the over abundance of warnings. :(

Anyway, could you maybe add HP Fandom.net (http://www.hpfandom.net/eff/index.php)?
It's an old eFiction based site (1.1 I think) and it's a large archive, actually one of the largest all Harry Potter out there.

Then there is Walking the Plank (http://www.walkingtheplank.org/archive/).

Hopefully you'd be able to add them.

Eh, I may look at the first one probably next week. Not quite right now. (though, the warning on the front page is not particularly inspiring)
The second one is a bit on the small side, though.

Zauberlehrling!
03-05-2012, 05:52 AM
Hi, just tried the plugin for the first time and have a question already.
Is there any chance to get http://ksarchive.com (Fanfiction Site for Kirk/Spock Stories) included into that plugin, please?

jackr
03-05-2012, 08:00 PM
An idea crossed my mind for password locked stories. If the story is locked, the plugin goes to check a password section in personal.ini to see if it has a password there, and if so it tries to use the password. That way if there are a few locked stories that a reader is privy to, they can still access those stories without needing to wait for the password prompt for all the other locked stories that they aren't privy to.

Something like:[Password]
fimfiction.net/story/796/:*password
fimfiction.net/story/6884/Moonlight-Sonata:12345

And if there is a password listed, but it doesn't match, then maybe have so the error code reflects that.

Just a thought that cropped up when I was working on a story that I have locked. I was going to try out an image inside the story, but was too lazy to go remove the fail_on_password for it. :D

Jade Aislin
03-05-2012, 10:44 PM
I had started to work on an adapter for ksarchive.com, however I hit a snag when I realized that it has two different warnings, which none of the other sites have. I have almost no experience with python and was not sure how to continue. Unfortunately, because of life and school, I have had little time to work on it either.

JimmXinu
03-05-2012, 10:48 PM
Actually, adapter_tenhawkpresentscom.py has two different warnings--depending on how you mean 'warning'. Have you looked in that one?

Jade Aislin
03-05-2012, 10:51 PM
No, but I will look at it tomorrow. School calls tonight. If I remember, ksarchive had a warning = 1 for teen rated stories and a warning = 2 for adult rated stories. I'll look at the adapter_tenhawkpresentscom.py. Hopefully it will solve my problem.

JimmXinu
03-05-2012, 10:53 PM
Something like that. It requires warning=3 for some stories and warning=4 and log-in for others.

JimmXinu
03-06-2012, 01:01 AM
I'm looking for some testers to help find the first round of bugs with four new sites.

fanfiction.mugglenet.com
fanfiction.portkey.org
thequidditchpitch.org
www.hpfandom.net

These are the four largest/most active of the sites that have been requested lately.

Another site that's been requested a few times is adultfanfiction.net. None of the current developers are interested in writing code to support it because of the ominous legal warnings and requirements to access the site.

JimmXinu
03-06-2012, 11:23 PM
Nobody's complained--must work fine. Advanced to general release.

SheDo2
03-08-2012, 12:49 AM
I just started using FanFictiondownloader plugin and it's great... Could you please add a couple of my favorite sites...
http://www.academyofbards.org/

http://www.xenafiction.net/

JimmXinu
03-08-2012, 01:05 AM
@SheDo2 - Probably not.

Those two sites, at first glance, are really just websites that happen to contain stories.

What makes it possible to reliably convert stories to ebooks is the fact that the supported sites use story management software to present the stories in a consistent way.

I suggest googling for 'web page to epub' or something along those lines. There are a lot of different solutions for websites, I believe.

iatheia
03-08-2012, 01:06 AM
I just started using FanFictiondownloader plugin and it's great... Could you please add a couple of my favorite sites...
http://www.academyofbards.org/

http://www.xenafiction.net/


Took a quick look... both of these websites have very, very ugly html formatting, with pretty much every story having a free-form layout. There isn't really a place to easily collect even something as simple as a table of contents for a multi-chaptered work, not to mention metadata. Hell, you can't even get author url from any of the stories!
While not _impossible_, I wouldn't really get your hopes up.

SheDo2
03-08-2012, 08:53 AM
Thanks for the reply. I use iSiloX as a drag and drop HTML converter but it only converts to iSilo format and Calibre doesn't convert iSilo. Oh well back to going around my elbow to get to my a$$...

Thanks for lookin at my request.

Gin

davidfor
03-08-2012, 10:32 PM
Thanks for the reply. I use iSiloX as a drag and drop HTML converter but it only converts to iSilo format and Calibre doesn't convert iSilo. Oh well back to going around my elbow to get to my a$$...

If you are using Firefox, have a look at the extension GrabMyBooks (http://www.grabmybooks.com/). It allows you to build epubs from web pages. I tried it a while ago out of curiosity, but I don't remember how good it was.

Jade Aislin
03-09-2012, 03:32 AM
Something like that. It requires warning=3 for some stories and warning=4 and log-in for others.

I looked at tenhawkspresentscom.py. It looks like it has one regular warning and one warning for stories that need a registered user. I looked at the other sites that have adapters. Now I'm completely confused. From what I could tell, all the sites had at least one warning that needed the user to be registered when there was more than one warning.

ksarchive does not need the user to be registered. You only need to go through the javascript pop up for two different ratings.

I do not understand python at all and can't seem to get beyond editing the basic things like it is outlined in the castlefanorg.py. That is how I was able to get the lower rated stories to work before. Unfortunately, I have too many other things going on in my life at the moment and don't think I will be able to continue working on this.

I had tried to use
if "text from javascript" in data:
addurl= (the warning # depeneding on the text)

But it still didn't seem to be adding the warning to the url. I also was not sure where to put the lines as I ran into errors in both the data and url not being defined.



I do want an adapter for ksarchive.com. But I have no idea what I am doing. Unless someone else does one similar, I do not think I can continue.

- Jade Aislin :blink:

JimmXinu
03-09-2012, 12:23 PM
@Jade Aislin - That's fine. It's on our list of sites to consider and will likely get supported sooner or later.

summerdream81
03-09-2012, 03:50 PM
Hello,

I'm new to using the plugin and I really love it. Thanks for all the great work !!

And now I have, maybe a little dumb questions :

How exactly do I configure personal.ini ?
I tried to add it in .config>calibre>plugin>fanfictiondownloader.zip but somehow I still have to put in my logins manually?
How do I have to format it exactly ? do the ## and # from your example stay ?

And when I try to download from twiwrite I only get errors?
'NoneType' object has no attribute 'string'

I tried this story http://www.twiwrite.net/viewstory.php?sid=785&ageconsent=ok&warning=1
and some others and its always the same error


Maybe someone can help?

TY summerdream81

JimmXinu
03-09-2012, 04:03 PM
@summerdream81 - I think you're trying too hard. :)

Drop down the FFDL plugin menu, choose 'Configure Plugin', choose the personal.ini tab. Edit away. Or, do 'Preferences' > 'Plugins' > 'User Interfact Action plugins' > FanFictionDownLoader

.ini config file format is pretty basic. [sections] and keyword:value lines. The authoritative format source is: http://docs.python.org/library/configparser.html

# and ## are both comment lines and maybe removed if you like. As a convention, I try to use # in front of keywords that FFDL knows that can be turned on by removing the # and ## for comments about them.

As for http://www.twiwrite.net/viewstory.php?sid=785, it downloaded fine for me.

summerdream81
03-09-2012, 04:35 PM
@JimmXinu
Wow thank you for the fast repliy.
Ahh so simpel !! thats great, now the login is working smoothly. TY very much.

But I still get the same error every time I try to download from twiwrite?

JimmXinu
03-09-2012, 04:49 PM
@summerdream81 - I can probably help if you get me some more information.

Run 'calibre-debug -g' from a command line, try downloading from twiwrite again and copy-paste the command line output here. It should contain more error info.

summerdream81
03-09-2012, 05:16 PM
Okay here is the debug protokol :

calibre Fehlerdiagnoseprotokoll
calibre 0.8.41
Linux-2.6.38-8-generic-x86_64-with-debian-squeeze-sid
Linux
('Linux', '2.6.38-8-generic', '#42-Ubuntu SMP Mon Apr 11 03:31:24 UTC 2011')
Python 2.7.1
Linux: ('debian', 'squeeze/sid', '')
Starting up...
Key u'G' for shortcut Bücher erwerben is already used by An Reader übertragen, ignoring
Key u'G' for shortcut Bücher erwerben is already used by An Reader übertragen, ignoring
Key u'G' for shortcut Bücher erwerben is already used by An Reader übertragen, ignoring
Started up in 16.53 seconds with 2173 books
Key u'G' for shortcut Bücher erwerben is already used by An Reader übertragen, ignoring
Key u'G' for shortcut Bücher erwerben is already used by An Reader übertragen, ignoring
FanFictionDownLoader v1.5.8
url:http://www.twiwrite.net/viewstory.php?sid=785
Exception: {u'comment': u"'NoneType' object has no attribute 'string'", u'good': False, u'author': u'Unknown', u'url': u'http://www.twiwrite.net/viewstory.php?sid=785', u'author_sort': u'Unknown', u'calibre_id': None, u'added': False, u'title': u'Unknown'}:'NoneType' object has no attribute 'string'
Traceback (most recent call last):
File "calibre_plugins.fanfictiondownloader_plugin.dialog s", line 262, in do_loop
File "calibre_plugins.fanfictiondownloader_plugin.ffdl_p lugin", line 428, in get_metadata_for_book
File "calibre_plugins.fanfictiondownloader_plugin.fanfic downloader.adapters.base_adapter", line 225, in getStoryMetadataOnly
File "calibre_plugins.fanfictiondownloader_plugin.fanfic downloader.adapters.adapter_twiwritenet", line 131, in extractChapterUrlsAndMetadata
AttributeError: 'NoneType' object has no attribute 'string'
Fetched metadata for 1 of 1

I hope this helps

JimmXinu
03-09-2012, 06:47 PM
I think I've figured it out.

Try going into your account on twiwrite.net, Edit Preferences and change "I am of the age of consent in my locale..." to Yes.

I'll add an 'is adult' check to that adapter for that case, but it wouldn't be right away.

crochetgeek2010
03-09-2012, 10:13 PM
Any chance of adding http://www.squidge.org? thanks Laura

JimmXinu
03-11-2012, 03:13 AM
@crochetgeek2010 - Unlikely. It's not just a story site, it's several collections of differently formatted, differently organized stories and pages.

It's one of those sites that's bad enough that I doubt anyone who doesn't read it themselves is going to be interested in writing the adapter.

Zauberlehrling!
03-11-2012, 11:41 AM
I had started to work on an adapter for ksarchive.com, however I hit a snag when I realized that it has two different warnings, which none of the other sites have. I have almost no experience with python and was not sure how to continue. Unfortunately, because of life and school, I have had little time to work on it either.

Oh super, thanks for trying to "crack" this archive. I know what you mean with two different warnings. I don't know why they have it. Is it possible for you to keep trying?

cordy
03-11-2012, 12:21 PM
I had started to work on an adapter for ksarchive.com, however I hit a snag when I realized that it has two different warnings, which none of the other sites have. I have almost no experience with python and was not sure how to continue. Unfortunately, because of life and school, I have had little time to work on it either.

Hello, I'm no master, and I don't know if the coding I put together will be approved or not, but I tried to work around a site with 3 warnings, locally I'm able to get the fictions downloaded. If you want I can send you the bit of code I used ... maybe together we will be able to work around the two warnings on your website.

Sam.

jackr
03-15-2012, 03:23 AM
I forget if I asked about it before, but could you include in the Custom Columns section an option for outputting the various errors for fanfic's? I feel like I did bring it up in an earlier post, but a brief search made me think I probably forgot to ask. :o

I know one benefit I would get from it would be color coding stories that do get errors as well as filtering them out if I need to run an overwrite always on a couple stories, as well as just keeping general tabs on the various changes; like one fic recently that got a chapter count error because the author revamped their story and compacted chapters together, etc.

JimmXinu
03-15-2012, 11:11 PM
@jackr--That sounds familar, so I expect you did ask before.

This isn't something I'm likely to implement. The internal structure would have to be revamped considerable, plus there's the issue of what to do with errors on new stories that don't have existing books.

IMO, you shouldn't be using update to check for new chapters. That's what the update emails from the sites are for.

JimmXinu
03-16-2012, 07:37 PM
FYI.

Version 1.5.9 - 16 Mar 2012
Remove site gayauthors.org following complaints from the site administration.
New Site: nfacommunity.com
New Site: www.midnightwhispers.ca. Thanks to Sam for implementing these.
Make 'Update Calibre Only' add a new book if no matching book found.
Minor improvement to column updates for new stories.
Improve image support for a couple of obscure cases.
portkeyorg.org only - Write 'Chapter does not exist' chapter when chapter is in list, but does not exist.
Add is_adult check to twiwrite for when user on site is not set to adult.
Change td in portkey chapter text to div for nook.

cordy
03-19-2012, 10:07 AM
Hello,

I have a little problem, I'm running the 0.8.43 calibre portable edition with the plugin activated.

It worked great, until this WE. I didn't change anything, I upgraded to the new version. At first it didn't work, but then I thought maybe it's because the non portable version on the home computer wasn't up to date. I did the update.

And it didn't change a thing, I still had the problem.

When I use the plugin, it grabs the metadata, but doesn't dl the fic.

I tweaked both installs at home, and I was finally able to get it to work. But now, I'm only on the portable version, and the plugin do the same.... grabs the metadata, but not the story itself.

When I look into the "show job details" I have this :

Starting job: Downloading FanFiction Stories
Downloading None
Failed Downloading None: 'title'

doesn't seem too good.... does it?

The fiction I'm trying to grab is here : http://www.fanfiction.net/s/5846760/1/My_Fathers_Keeper


Oh and another problem ... I can't configure the plugin this is the error I have....
Can't understand why

calibre, version 0.8.40
ERROR: Unhandled exception: <b>TypeError</b>:__init__() takes exactly 1 argument (2 given)

Traceback (most recent call last):
File "calibre_plugins.fanfictiondownloader_plugin.dialog s", line 138, in config
File "site-packages\calibre\customize\__init__.py", line 150, in do_user_config
File "calibre_plugins.fanfictiondownloader_plugin.__init __", line 68, in config_widget
TypeError: __init__() takes exactly 1 argument (2 given)


Thanks for any idea.
Sam

JimmXinu
03-19-2012, 01:17 PM
@cordy - I don't think you truly have the latest version of either calibre or the plugin.

The error you show at the end says calibre 0.8.40, not 0.8.43. But that shouldn't make any difference for this problem.

The plugin writes it's version to the calibre job log and has for a number of versions now. It should look like this:
Download FanFiction Book
FanFictionDownLoader v1.5.9
Adding jobs for URLs:

Plus, the line number in the plugin your exception post references is blank, again indicating version mismatch.

If your plugin version is that old, you're probably bumping into the format change ffnet made a while back.

I'm not familiar with portable installs, so I can't help you with that part.

Brown Betty
03-19-2012, 03:31 PM
Hey, thanks for all of your work on this amazing plugin.

I was pleased to see tth added to the list of supported sites, but trying to download http://www.tthfanfic.org/wholestory.php?no=18987#chapter1 fails silently, for me.

Running calibre-debug -g from the terminal produces almost no output (can this be right?) but here is what it does give:

Starting up...
Started up in 3.37 seconds with 822 books
FanFictionDownLoader v1.5.9


And nothing appears in the job log.

JimmXinu
03-19-2012, 04:58 PM
The plugin silently discards URLs (and other text) that it doesn't recognize as story URLs.

To properly chapter a story, you have to use the regular story URL, not the wholestory URL. http://www.tthfanfic.org/Story-18987/jedibuttercup+Handle+With+Care.htm works fine.

JimmXinu
03-19-2012, 11:41 PM
FYI, New version out with these changes:

- New site: ksarchive.com, thanks to Jade Aislin for implementing this.

- Fixes for mugglenet.com--added user/pass, better handling for summary and is_adult levels.

- Allow default_cover_image to use ${title}, etc; add cover_exclusion_regexp feature to exclude images from being covers.

As usual, there may be a delay before calibre notices the new version. You can get the latest by going in and telling calibre to install it again.

cordy
03-20-2012, 04:41 AM
@cordy - I don't think you truly have the latest version of either calibre or the plugin.

The error you show at the end says calibre 0.8.40, not 0.8.43. But that shouldn't make any difference for this problem.

The plugin writes it's version to the calibre job log and has for a number of versions now. It should look like this:
Download FanFiction Book
FanFictionDownLoader v1.5.9
Adding jobs for URLs:

Plus, the line number in the plugin your exception post references is blank, again indicating version mismatch.

If your plugin version is that old, you're probably bumping into the format change ffnet made a while back.

I'm not familiar with portable installs, so I can't help you with that part.

Thanks for helping me... I battled against the plugin for 10 minutes and you where right, it was the very first version installed with the latest overwritten. In fact it was installed twice, and the worked against each other.

Now it's fixed, and your plugin works great on the portable calibre edition.

Thanks
Sam

jackr
03-25-2012, 01:34 AM
Haven't gotten a response back from the site owner of Fimfiction.net after sending them a message a couple weeks ago; I will try sending another soon.

I know that the posts (if there are any) do get dated with a year, but that of course is sketchy for basing the year on. And the stories from the author's creation dashboard (or whatever is a good word for that) have a year on them for "Last Modified", but that doesn't help any if it is not there in the target html. :(

Oh well, it's ultimately a small thing. Anyways, keep up the awesome work! :thanks:

JimmXinu
03-25-2012, 12:24 PM
@jackr - I expect they'll start thinking about it when the site gets old enough that the story dates become ambiguous.

cryzed
03-25-2012, 06:51 PM
Hey there JimmXinu, I just found out about your plugin for Calibre and it is really looking good so far :)! The thing is, I wrote my own solution for FanFiction.Net and FictionPress.com quite a while ago, it's called Lemon (https://bitbucket.org/cryzed/lemon/overview).

After inspecting your adapter files for the respective sites I thought maybe you could use the two APIs I wrote for them: FanFiction.Net API (https://bitbucket.org/cryzed/fanfiction.net-api/overview) and FictionPress.com API (https://bitbucket.org/cryzed/fictionpress.com-api/overview). They don't introduce any new dependencies besides html5lib (using html5lib in correlation with BeautifulSoup is a good idea anyhow) and even easily parse information that you are currently missing, e.g. the review count (which I would really like to see in the final file if possible).

And I realize, of course, that this is only personal preference but could you adapt the default template to something like this (http://www.imagebanana.com/view/fs0ptlz5/screen_shot65231.gif)/this (http://www.imagebanana.com/view/w3g4o9uv/screen_shot65174.gif)/this (http://www.imagebanana.com/view/oumsjyqz/screen_shot65193.gif)? The source for the templates used by Lemon can be found here (https://bitbucket.org/cryzed/lemon/src/8840a93d0bd8/lemon/fanfiction_net/template.py) and here (https://bitbucket.org/cryzed/lemon/src/8840a93d0bd8/lemon/fictionpress_com/template.py). Only if you agree, of course, but quite honestly (no offense): I really think that the templates used by Lemon are an improvement over the ones currently used in the plugin.

Man, I sound like a smug asshole -- but this really isn't my intention, I just want to finally deprecate Lemon for something better and I would be all over your Calibre plugin if you made these small changes.

JimmXinu
03-26-2012, 12:14 AM
@cryzed -

I'm not really inclined to change FFDL to use your APIs for ffnet and fpcom. If you were offering code for a new site, or significant new functionality, it would be a different matter.

As for review count, I don't see that the value is worth the effort. It's going to be out of date most of the time. FFDL has 25+ different adapters for different sites it would have to be added to. And if FFDL collects review count, why not the recs/favorites/hits counts that many sites have, too?

I've added it to the 'wishlist' of future features, but I don't see it happening any time soon.

(BTW, to counter the argument that review counts could be updated everyday: I strongly recommend everyone use email notices for new chapters, and to not poll all your stories every day for new chapters. It inflates hit counts and wastes bandwidth on the fanfic site. We can only do this as long as we don't annoy the sites enough to block us.)

As for changing the appearance of the output, looking at your screen shots, the only differences between Lemon and FFDL output I noticed are that Lemon doesn't put commas in the word count and it centers the TOC entries. And, predictably, I like the output from FFDL better--because it's the way I like it. :)

Have you looked at FFDL's personal.ini settings? Specifically, the titlepage_entries, include_tocpage, and output_css options? There's already a fair degree of customization available. (Fair warning--FFDL mobi output doesn't support output_css. In fact, FFDL mobi output is only one step above deprecated. I strongly recommend epub and conversion.)

If you have specific requests, I might be able to add some additional markup to the title and TOC page output for the CSS to influence.

Thanks for your interest.

nomad062
03-26-2012, 04:12 AM
I'm playing around using Ubuntu and of course one of the first things I downloaded was Calibre - the second thing I wanted was your FFDL plugin. Awesome plugin. Saves me several steps now that I can do it in 1 shot.

At any rate when I went to download your plugin it states that I need Calibre 8.30 or later yet my Calibre says its 8.8. Any idea? or do I need to post something with Calibre?

PeterT
03-26-2012, 10:06 AM
Do an install of calibre from the official web site (http://calibre-ebook.com/download_linux), and not from the Ubuntu repository.

nomad062
03-26-2012, 12:38 PM
Thanks for the info!

JimmXinu
03-26-2012, 01:24 PM
What he said.

0.8.30 is specified because that's the earliest version I've tested it with. There's probably at least a few earlier versions it would work with, but 0.8.8 is too far back--there were significant changes to plugin support around 0.8.18.

nomad062
03-26-2012, 02:07 PM
I guess I was misreading the 8.8 I thought it was more recent than 8.3 but its all fixed now so thank you both for the quick response

Laurin

cryzed
03-26-2012, 03:02 PM
@JimmXinu all right, I did it myself then -- while I was it I also implemented support for special title entries that allow a degree of customization (e.g. br-tags).

I still think that your code in its current form is really bad: in nearly all your files you have unused partial or even full module imports (sometimes duplicated), nothing that resembles PEP-8, you do some really puzzling things with the encoding/decoding of UTF-8 strings in your code and rely on chardet to guess encodings at times (although I didn't discover where), you use BeautifulSoup but apparently have no clue how to navigate its structure, RegEx is a solution for everything in your code while only making things more complicated (e.g. using it to strip '#' at the end of a string instead of just splitting the string), many of your methods do unexpected things, you use decorators incorrectly (e.g. adapaters.adapter_fanfictionnnet.FanFictionNetSite Adapter.getAcceptDomains), you have many methods that have a "get"-prefix and simply return a constant or unchanged value... and that's only after looking at 3-4 files of your project. That's why I wanted you to use my API modules, many, if not all things, you currently do in the adapter for FanFiction.Net and FictionPress.com could have been simplified.

Things that have me stumped are also that somehow when adding new items to the validEntries list and a corresponding label to the titleLabels list in your base_writer.py code the label is not used like apparently intended in BaseStoryWriter.writeTitlePage and the "NO_TITLE_ENTRY" is used as a template, so one has to actually manually define a "<title_entry>_label" entry in the configuration files -- why do the "fall back labels" exist then in the first place? (I'm sure this one's my fault) Additionally modifying the EPUB_TOC* constants in the EpubWriter class somehow has no effect on the resulting table of contents that is actually created, why is that?

Of course, I realize that my code is surely not perfect either but reading your code is really hard and puzzling -- and that's not because the things you do are complicated: the overall structure is just bad. This time there's no way that I could not have sounded like an asshole, sorry.

JimmXinu
03-27-2012, 11:31 PM
@cryzed - I'm not going to argue with you about it. I'm not a python expert, nor have I claimed to be. This is a hobby.

If you'd like to contribute any of your changes, please set up a clone (http://code.google.com/p/fanficdownloader/source/clones) to check your changes into and the other developers on the project and I will review them.

jackr
04-09-2012, 06:00 AM
I was busy tweaking the look of the Book Details pane today when I learned about how links can be assigned to author names. If you are looking for any new ideas at the moment, having the author link direct to the authors page would be great. :D

davidfor
04-09-2012, 08:41 AM
I was busy tweaking the look of the Book Details pane today when I learned about how links can be assigned to author names. If you are looking for any new ideas at the moment, having the author link direct to the authors page would be great. :D

In the tag browser, right click on the "Authors" category and select "Manage Authors". The third column in the list is what you want. I don't know how long that has been there, but, I only discovered this in the last couple of weeks.

JimmXinu
04-09-2012, 11:45 AM
I didn't know that was there either--I'll certainly try and populate it with the author url.

Thanks to both of you for pointing that out.

FFreader920
04-09-2012, 04:15 PM
I've been using the FFDL web app for some time now and was very interested in this plug-in. However, I can't get it to work for me at all. Opening the app, clicking a menu, touching anything immediately shuts the plug-in down (along with Calibre). I posted in another forum earlier and was directed here. Here's part of the error. I've got more if you want. Any thoughts? I updated Calibre version and just DL'd plug-in update.

Thanks for your help.

---
Exception Type: EXC_BAD_ACCESS (SIGSEGV)
Exception Codes: KERN_INVALID_ADDRESS at 0x0000000000000008

VM Regions Near 0x8:
-->
__TEXT 0000000100000000-0000000100002000 [ 8K] r-x/rwx SM=COW /Applications/calibre.app/Contents/MacOS/calibre

Application Specific Information:
objc[1124]: garbage collection is OFF
Performing @selector(qtDispatcherToQAction:) from sender NSMenuItem 0x110e395a0

---

JimmXinu
04-09-2012, 04:24 PM
@FFreader920 - Actually, I was trying to direct you up a level or two where the calibre devs (vs plugin devs) hang out. I don't know any more here than I did on the mail list, sorry.

FFreader920
04-09-2012, 05:16 PM
@FFreader920 - Actually, I was trying to direct you up a level or two where the calibre devs (vs plugin devs) hang out. I don't know any more here than I did on the mail list, sorry.
No problem -- and thank you! I will head there.

JimmXinu
04-10-2012, 12:02 PM
@FFreader920 - Following Kovid's suggestion, I've made a version that keeps explicit references to the menu items. Please give the attached version a try and see if it solves your problem.

See the Plugin Intro thread (http://www.mobileread.com/forums/showthread.php?t=118680) if you need help manually installing it.

FFreader920
04-11-2012, 02:33 AM
@FFreader920 - Following Kovid's suggestion, I've made a version that keeps explicit references to the menu items. Please give the attached version a try and see if it solves your problem.

See the Plugin Intro thread (http://www.mobileread.com/forums/showthread.php?t=118680) if you need help manually installing it.

Wow -- so ... that absolutely worked! I just downloaded a couple of stories and updates. Thank you so much. :thanks: It's like magic!

JimmXinu
04-11-2012, 11:50 AM
I'm glad we could get that sorted. I hope to put it out as the next official version later today.

JimmXinu
04-11-2012, 01:17 PM
FYI, v1.5.13 is out.

New site: archive.skyehawke.com plus a fix for FFreader920's crashing problem.

As usual, there may be a delay before calibre notices the new version. You can get the latest by going in and telling calibre to install it again.

jackr
04-14-2012, 11:29 PM
It looks like fimfiction.net has undergone some changes again. When updating, I am losing the cover images on most of the stories. So far only one has kept itself after an update.

http://www.fimfiction.net/story/6975/ This one still works.

http://www.fimfiction.net/story/4843/ And this is an example of one that doesn't.

I noticed on one fic that the author included a source link beneath their image. I never seen this before, so maybe that has something to do with it? :blink:

http://www.fimfiction.net/story/19066/ Has a source link.

And last but not least, is there a better place that I should go for adapter-site issues? I don't want to clutter this thread with posts about issues if they would be better addressed elsewhere. :o

JimmXinu
04-15-2012, 12:10 AM
@jackr - The project user group (http://groups.google.com/group/fanfic-downloader) is arguably a better place for adapter issues, as opposed to plugin specific issues. However, it's not always clear when an issue is plugin specific or not. If you found the problem using the plugin, I'm not likely to complain if you post it here.

This particular problem with fimfiction.net is happening because fimfiction.net is leaving off the 'http:' on story image URLs. IE, '//static.fimfiction.net/images/...' instead of 'http://static.fimfiction.net/images/...'. The browser are apparently prepared to live with this level of broken and fill in the 'http:'.

(http://www.fimfiction.net/story/6975/ only appears to work because there is a similar image in the first chapter that's being picked up for the cover.)

I've put a kludge in the adapter to deal with this and attached a beta version for you to try. I won't be able to put out an official version until Monday.

You could try reporting it as a problem to the site admins, but since they don't seem to think dates need the year...

jackr
04-15-2012, 02:01 AM
Thanks, works perfectly so far :D. According to a post on their main page, fimfiction has undergone a server change and that there are some bugs to work out. I will post a comment on their site about that partial url issue.

Amalthia
04-16-2012, 03:26 AM
Hi, I love this plugin and I just had one question. Is there a way to disable the confirmation box that pops up asking if you're an adult for the archive of our own archive? It's not so bad when I'm downloading one story but when I try to download five or six at a time it's not so much fun.

I'd appreciate any and all help,
thanks!
Amalthia

EDIT: Figured it out. Apparently I needed to remove the # sign from the url, username, password, and the adult:true line.

JimmXinu
04-17-2012, 04:14 PM
New version released:

Change "Include images in EPUBs?" to effect [epubs], not [defaults].
Kludge for incomplete story image URLs on fimfiction.net

jackr
04-19-2012, 05:52 AM
Fimfiction.net is down for maintenance by the looks of it. But I have found out that if I run an update on any stories that contain them, the plugin hangs and I have to force quit Calibre. I ran a test on a single fimfiction story and the time it takes to determine whether it can get the story is long; I counted 84 seconds that the plugin waits for to get a response.

The error code I get after the wait is this:<urlopen error [Errno 10060] A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond>

I am not sure if it will help any with the plugin, and thankfully sites are rarely down, but I am posting this in case it is something that can be tweaked so that the plugin doesn't hang on a story more than it has to.

JimmXinu
04-19-2012, 01:16 PM
@jackr - Because the server doesn't return a connection refused, the downloader has to first wait for the connection to time out. Then, to deal with slow & flakey servers, it sleeps a little while and tries again a couple times before really failing.

It's a trade off between failing quickly when it's really down and waiting & trying when it's just flakey. I'm inclined to leave it as it is.

Ajax
04-21-2012, 08:19 PM
Finally I can download FictionPress books and not be forced to read online. Thank you so much!

JimmXinu
04-21-2012, 08:21 PM
@Ajax - Glad you like it.

Just M.E.
04-21-2012, 11:14 PM
I really love your plugin; it's made my fanfiction collecting so much easier! However I've run into a slight problem.

In the last few days, the Archive of Our Own website changed their layout to include different banners. Even though there is an option to hide the banner, whenever I use the plugin to grab a story, the title always comes up as 'Hide this banner', whether I am updating or getting a new story.

Is this something that can be changed in the plugin, or can you suggest a workaround other than editing every title manually?

Thanks in advance.

M.E.

JimmXinu
04-21-2012, 11:23 PM
@Just M.E. - FFDL will need to be updated.

Can you test the attached version and make sure it works for your stories? I don't read Archive of Our Own much.

Just M.E.
04-21-2012, 11:36 PM
@Just M.E. - FFDL will need to be updated.

Can you test the attached version and make sure it works for your stories? I don't read Archive of Our Own much.
Perfect, Thank you so much!

M.E.

JimmXinu
04-22-2012, 12:20 PM
v1.5.15 posted with changes:

Fix for 'Hide this banner' title issue on Archive of Our Own.
Changed implementation of fimfiction to use provided API, thanks to Althaine.

As usual, there may be a delay before calibre notices the new version. You can get the latest by going in and telling calibre to install it again.

pookakitten
04-24-2012, 05:05 AM
I have a story from a site that disagrees with my Kindle Touch. I have it set to download to mobi, but when I read to a location, it won't let me read further. Some of the text is superimposed on itself when I reach the part that won't let me advance. Any text after this point is not visible. This happens near location 525 (in chapter 6) on the Touch. I am using ver. 0.8.44 of calibre and the most current version of the plugin. The story works properly on the Touch if I reconvert it in calibre. I may have had a similar problem with another file from AO3.

Link to story: http://www.archiveofourown.org/works/299460

Thanks for the great software.

iatheia
04-24-2012, 01:33 PM
That's exactly it - download it in epub - it is much superior to mobi in every single way. Mobi is... wonky and is not supported properly.

JimmXinu
04-24-2012, 03:10 PM
Regrettably, that's true. FFDL's mobi output is far less than stellar. I spent most of a week once trying to improve it and ended up with something worse than before.

I download to epub and convert to mobi for my own kindle and that's what I recommend for everyone.

pookakitten
04-24-2012, 04:58 PM
Thanks for the heads up. I'll do that in the future.

pookakitten
04-25-2012, 08:52 PM
Hey, some more AO3 wackiness. I'm using the latest version of both calibre and the plugin. The other AO3 stories I have work just fine. Just these 2 black sheep.

I've enclosed a screenshot with the error message and links that are acting up.

JimmXinu
04-26-2012, 12:05 AM
v1.5.16 posted with change:

Fix for Yet Another numeric entity issue for pookakitten's problem.

As usual, there may be a delay before calibre notices the new version. You can get the latest by going in and telling calibre to install it again.

JimmXinu
05-02-2012, 10:53 PM
Anybody interested testing a new feature? Namely, automatically calling Generate Cover(GC) on FFDL download/updates. I know somebody asked for it once. :)

There's a new config tab in FFDL for Generate Cover and three ways to set it up:

Easiest: Select GC setting for 'Default' to use for all downloads/updates.
Still Easy: Select different GC settings to use for different sites. Choose a Default to fall back to or now, as you please.
Somewhat Hard: Define templates and regular expressions for different GC settings in personal.ini. Allows for GC setting by category or genre (or author, or whatever) across different sites, or with different settings per site.


By leaving FFDL's basic 'Update Cover' option on and using GC's '{Current Cover}' feature, you can also get GC to use a cover image from the story automatically.

Here's an example of the personal.ini config:
## If you have the Generate Cover plugin installed, you can use the
## generate_cover_settings parameter to intelligently decide which GC
## setting to run. There are three parts 1) a template of which
## metadata part(s) to look at, 2) a regular expression to match the
## template, and 3) the name of the GC setting to use, which must
## match exactly.
## Make sure to keep at least one space at the start of each line and
## to escape % to %%, if used.
## template => regexp to match => GC Setting to use.
generate_cover_settings:
${category} => Buffy:? [tT]he Vampire Slayer => BuffyCover
${category} => Star Trek => StarTrekCover

(Trying to do a GUI version of generate_cover_settings is a lot of work that I'm only going to invest if there's a lot of interest.)

(zip file removed after official release)

Aleyst
05-03-2012, 01:18 AM
OOH! *happy dance* I'm so testing this!
Cheers!

---
Tested on FF.net stories, works a treat using Category.
Any way to get it to work via the "Character" part?

JimmXinu
05-08-2012, 12:52 PM
v1.5.18 posted with new feature:
(1.5.17 skipped)

Automatically call Generate Cover(GC) on FFDL download/update.

There's a new config tab in FFDL for Generate Cover and three ways to set it up:

Easiest: Select GC setting for 'Default' to use for all downloads/updates.
Still Easy: Select different GC settings to use for different sites. Choose a Default to fall back to or now, as you please.
Somewhat Hard: Define templates and regular expressions for different GC settings in personal.ini. Allows for GC setting by category or genre (or author, or whatever) across different sites, or with different settings per site.


By leaving FFDL's basic 'Update Cover' option on and using GC's '{Current Cover}' feature, you can also get GC to use a cover image from the story automatically.

Here's an example of the personal.ini config:
## If you have the Generate Cover plugin installed, you can use the
## generate_cover_settings parameter to intelligently decide which GC
## setting to run. There are three parts 1) a template of which
## metadata part(s) to look at, 2) a regular expression to match the
## template, and 3) the name of the GC setting to use, which must
## match exactly. Use this parameter in [defaults], or by site eg,
## [www.ficwad.com]
## Make sure to keep at least one space at the start of each line and
## to escape % to %%, if used.
## template => regexp to match => GC Setting to use.
generate_cover_settings:
${category} => Buffy:? [tT]he Vampire Slayer => BuffyCover
${category} => Star Trek => StarTrekCover

(Trying to do a GUI version of generate_cover_settings is a lot of work that I'm only going to invest if there's a lot of interest.)

As usual, there may be a delay before calibre notices the new version. You can get the latest by going in and telling calibre to install it again.

cklls
05-09-2012, 01:49 PM
Thank you once again! Love the new Cover options. :)

JimmXinu
05-09-2012, 03:44 PM
Testing support for a new site:

"The WWOMB" http://www.squidge.org/peja/cgi-bin/index.php

This is a bit different than others--there's several unrelated areas under the domain squidge.org. This is support only for stories in the eFiction instance under squidge.org/peja.

This necessitated some changes to the internals and will require more if we ever want to add support for additional sites under squidge.org.

Let me know what you think.

anesthezea
05-09-2012, 05:59 PM
I got an email this morning from the mod of the 852 Prospect Archive (I'm a member of the TS mailing list) and it seems there are plans in place to move the entire archive over to Archive of Our Own. I'm not sure when this will take place, but I'm looking forward to it! :)

I thought I'd pass word along since 852 is on your list of possible sites to include in future.

JimmXinu
05-09-2012, 06:42 PM
Thanks! I've made a notation to the list.

bb63
05-11-2012, 09:45 PM
hi all. i'm very new to calibre and to the plug in concept!

love all the work you guys do here!!!

are there any plans to add live journal to this?

JimmXinu
05-11-2012, 09:55 PM
@bb63 - Glad you like it.

As for live journal, nope. From what I've seen it really isn't meant to be a story system. But if you can give me several URLs to live journal stories, I might reconsider it.

There are other tools available. Another user said: "...Squeebook.net is an online tool that grabs/formats stories from livejournal/dreamwidth journals." The GrabMyBooks FireFox plugin is another possibility.

Firedancer885
05-12-2012, 04:43 AM
Hi,

I just found the plugin but fell in love with it already. I have a support question/request.
I downloaded a great deal of stories from fanfiction net via the deffnetizer program or the ficsavers.com website. Both of those put the story url at the beginning of the story text in id:xxxxxxx link format. Is there a way to make the "get URLs from selected books" option recognize those links as "url:" identifiers? I don't want to troll through all of my nearly 150 epubs for those urls.

JimmXinu
05-12-2012, 12:41 PM
@Firedancer885 - Hmm. Not a bad idea. Can you provide a (preferably smallish) example of each?

Firedancer885
05-12-2012, 04:57 PM
@Firedancer885 - Hmm. Not a bad idea. Can you provide a (preferably smallish) example of each?

Ok, I've run into a snag. I thought the ficsavers.com site uses the same template as the deffnetizer program but it doesn't. The ficsavers.com template uses a botched up url :( It's "story name" by "author", and the story name is the link to the story. Only some slashes are missing along with the chapter number :(

Harrowed uses the id:xxxxxxx template, The Leap uses "story name" by "author". I'd be thankful for any help.

ps: And then I realize not all of the ficsavers.com stories have botched up urls. :( Sorry :(

JimmXinu
05-12-2012, 05:53 PM
I've looked at deffnetizer and ficsavers.com a bit. Neither outputs epub directly, so apparently you've already converted from html to epub.

The deffnetizer epub you posted does contain a proper story URL, but that depends on the output template used. The first two templates I tried, "Default Template" and "Minimal" do not include the story URL anywhere.

The ficsaver.com epub you posted, as you say, contains a broken version of the story URL. Although it looks like that may only be a problem with ffnet stories.

All told:

The user would have to have converted the story to epub;
Then the plugin would have to scan at least the first three html files (cover page, ToC/splash page and first real page) for URLs, while;
Checking for both valid story URLs and at least one form of broken URL;
And even then, it wouldn't work for half of the included deffnetizer output templates.

So, while possible, it would be a bit of a pain and not work all the time anyway. At this point, I'll add it to the 'wishlist' of features I might add some day, but it's not going to be a priority.