[GUI Plugin] FanFictionDownLoader

[redscarlett]

I have my updated files on a reading list, but I want to sync them to my Nook in a WIP folder, rather than by author, etc. Can this be done, and if so, how please? I'm crap with plugboards and the like, so I can't figure it out myself :S

[JimmXinu]

Originally Posted by redscarlett

I have my updated files on a reading list, but I want to sync them to my Nook in a WIP folder, rather than by author, etc. Can this be done, and if so, how please? I'm crap with plugboards and the like, so I can't figure it out myself :S

Not that I'm aware of. That's why we mess about with plugboards.

[redscarlett]

Originally Posted by JimmXinu

Not that I'm aware of. That's why we mess about with plugboards.

Is there something similar that could be achieved by plugboards that you can think of please? Even a way for them to stick out would be helpful - I sometimes have so many updated, that I can't remember which ones, and it's really irritating when I forget!
Thanks

[riothamus]

Just trying to download from harrypotterfanfiction.com. I'm getting an error when I try. This happens at least by any story by this author.

Status: Bad
Title: Unknown
Author: Unknown
Comment: 'NoneType' object has no attribute 'strftime'
URL: http://www.harrypotterfanfiction.com...hp?psid=267932

[JimmXinu]

Originally Posted by redscarlett

Is there something similar that could be achieved by plugboards that you can think of please? Even a way for them to stick out would be helpful - I sometimes have so many updated, that I can't remember which ones, and it's really irritating when I forget!
Thanks

It's been discussed recently: https://www.mobileread.com/forums/sho...postcount=1748

[JimmXinu]

Originally Posted by riothamus

Just trying to download from harrypotterfanfiction.com. I'm getting an error when I try. This happens at least by any story by this author.

Status: Bad
Title: Unknown
Author: Unknown
Comment: 'NoneType' object has no attribute 'strftime'
URL: http://www.harrypotterfanfiction.com...hp?psid=267932

Looks like the harrypotterfanfiction.com site has changed.

Unfortunately, I won't be able to address that until next week, sorry.

[seabream]

Originally Posted by redscarlett

Thanks - I'll have a play with it! Any resources you recommend for a noob? A starting point?
I've tried the method, but the stories don't come up as not being on my Nook. Any ideas or shall I just do it manually?
Thanks again!

Well the first thought is to double check all the settings, but I'm going to presume that's occurred to you. I do know that it should be possible for it to work, since other people use something like this, but without more information about what your setup looks like it's hard to say why it isn't working for you, so let's check some other things.

Right, so you import or update a story, it appears in Calibre, or the new chapters are added, but the "New" column stays empty. Is that what you're saying (there's more than one possible interpretation)? What about the "Send to Nook" Reading List list? Does the fic show up there? (To view it, click the Reading List icon in the toolbar, or right click in the main library area and select Reading List, then click View List > Send to Nook (This is assuming that the Reading List menu is in one or both of those places - it is removable. If it isn't in either one, go to Calibre preferences > Toolbar > Click to choose toolbar or menu to customize. Then add it to whichever area you want it)) This list is the important part for automatically syncing. The "New" and "On Nook" columns give you information, and if you did the plugboard bit, the "New" column is what Calibre checks when it's deciding which of the books it is syncing should have 000 prepended to their titles, but in terms of _whether_ a book syncs or not, the values in those columns shouldn't matter. Where it looks in is the "Send to Nook" list. If is showing up there, try syncing the list to your Nook. If the books appear on your Nook, then your main purpose has been served. Check to see that the On Nook column updated properly. The next thing to do would be to compare what you did differently with your setup for the "New" list and FFDL auto update of that list that might have resulted in it not working when the other list did.

If it isn't showing up in the "Send to Nook" list, you can still manually add books to the list after you update/import them. That way you don't need to remember them for the next time you connect your Nook. To add manually, select the book or books you want to add, then go to Reading List > Add to list > Send to Nook.

Some simple things to check for in the setup for column updating that might have resulted in the "New" column not being updated properly:
- Make sure the names match exactly. The names are case sensitive, so if the Reading List list is capitalised in Reading List, it needs to be capitalised in the FanFictionDownloader settings menu that tells FFDL to automatically update it when you get a new story, or update an existing one. If the value you have in the set of permitted fixed Values box in the Create a Custom Column window is capitalised, it also needs to be in Reading List settings that say to add it to the custom column.
- Make sure that you've selected the right Custom Column to update in the Reading List settings.

If you're stuck trying to get FFDL to automatically add stories to reading lists with its Reading List settings, there's another way you can try. Fair warning, this is more involved that the one in the tutorial example, and has some different limitations than the Reading List method. This is a rather messy way of doing this, and if you can find a way to get the automatic adding via Reading List to work, you would likely be better served.

- Go to the preferences for Reading List and in List type, under Population Options, set it to "Auto populated from search".
- In the text box in Population Options, write a search string that looks for a value you're using to indicate a story you want to sync. e.g.: tags:"=Sync to Nook" (this means look for Sync to Nook, indicated by the quotation marks after the colon, in the tags. If Reading List sees this string, it will add the associated story to the list. So next we add that tag to incoming stories.
- Go to FFDL preferences > personal.ini and under [defaults] add the line

extratags:Sync to Nook

This will add the tag "Sync to Nook" whenever you get a fic or update one using FFDL. Note: This will not add the tag to stories you update if in FFDL's settings > Standard Columns > Tags, you have "New only" checked, for obvious reasons. If you don't use the tags field, that doesn't really matter. If you do though, you may need to clean up your tags after update, which could be more work than just manually adding the fic to appropriate Reading List list. If you do use this method, and do put things in Tags manually, in FFDL preferences > Basic > Updating Calibe Options, check "Keep Existing Tags when Updating Metadata?", or you will lose any manually inserted/edited tags that don't match the story's tags.

Note: In the workaround option, I'm using an extratags: line. In the default personal.ini there are similarly

extracategories:Harry Potter
extracharacters:Draco Malfoy,Hermione Granger
extraships:Draco Malfoy/Hermione Granger

under [dramione.org]
What determines the acceptable values of "*" in extra*: lines, I'm not sure yet. Is it any valid entry for the level [default,format,site,override] the line is in, or just the ones defined without recourse to extra_valid_entries? Because if the former, I can set it up to put the Sync tag in a non-tag entry type, put that into a custom column, have it autoupdate without Reading List, and still have the "New only" checked for Tags. When I get around to it, I can test it and let you know, but I'm not sure when that will be. Having a site specific ability to add tag-like metadata in certain places could come in handy, so probably I'll look at it eventually, but I've got other things on the go.

As to places to start looking with customisation, well, I usually start with the documentation. Calibre's manual gives a good outline of what it can do, and you can get an idea of what you might want to try. For FFDL, a lot of the metadata management documentation is in the default personal.ini. You can find that in FFDL's settings > personal.ini > View Defaults (plugin-defaults.ini). More detailed descriptions of specific things people have done with FFDL are in this thread itself. You can save the thread in a searchable text document so that it's all in one place and easily searchable by clicking "Thread Tools" at the top of the thread, and "Download this Thread".

Specifically, for your question about getting books to stick out and to go to specific folders, go to Calibre Preferences > "Sending books to devices", look at the default template, and the list of standard template variables. These are also usable in Metadata Plugboards. Pay particular attention to Calibre's template language section in the Calibre manual. Take a look through the Reading List plugin thread, which is shorter than this one. You could also look at the "Technigues to use plugboards, custom columns and templates" post in Calibre > Library Management https://www.mobileread.com/forums/sho...d.php?t=118563 but I'd recommend reading the documentation first or the template strings won't make sense. Experiment (obviously, back up your library and write down the settings you're changing, particularly the default template somewhere, so that you can go back to it). What you are interested in is doable. I don't have a Nook myself, so if you're doing things with shelves, I can't actually test anything, and stuff I write frequently doesn't work exactly as I intend it at first, so you're probably better off giving it a try yourself.

Good luck!

[redscarlett]

Originally Posted by seabream

Well the first thought is to double check all the settings, but I'm going to presume that's occurred to you. I do know that it should be possible for it to work, since other people use something like this, but without more information about what your setup looks like it's hard to say why it isn't working for you, so let's check some other things.

Right, so you import or update a story, it appears in Calibre, or the new chapters are added, but the "New" column stays empty. Is that what you're saying (there's more than one possible interpretation)? What about the "Send to Nook" Reading List list? Does the fic show up there? (To view it, click the Reading List icon in the toolbar, or right click in the main library area and select Reading List, then click View List > Send to Nook (This is assuming that the Reading List menu is in one or both of those places - it is removable. If it isn't in either one, go to Calibre preferences > Toolbar > Click to choose toolbar or menu to customize. Then add it to whichever area you want it)) This list is the important part for automatically syncing. The "New" and "On Nook" columns give you information, and if you did the plugboard bit, the "New" column is what Calibre checks when it's deciding which of the books it is syncing should have 000 prepended to their titles, but in terms of _whether_ a book syncs or not, the values in those columns shouldn't matter. Where it looks in is the "Send to Nook" list. If is showing up there, try syncing the list to your Nook. If the books appear on your Nook, then your main purpose has been served. Check to see that the On Nook column updated properly. The next thing to do would be to compare what you did differently with your setup for the "New" list and FFDL auto update of that list that might have resulted in it not working when the other list did.

If it isn't showing up in the "Send to Nook" list, you can still manually add books to the list after you update/import them. That way you don't need to remember them for the next time you connect your Nook. To add manually, select the book or books you want to add, then go to Reading List > Add to list > Send to Nook.

Some simple things to check for in the setup for column updating that might have resulted in the "New" column not being updated properly:
- Make sure the names match exactly. The names are case sensitive, so if the Reading List list is capitalised in Reading List, it needs to be capitalised in the FanFictionDownloader settings menu that tells FFDL to automatically update it when you get a new story, or update an existing one. If the value you have in the set of permitted fixed Values box in the Create a Custom Column window is capitalised, it also needs to be in Reading List settings that say to add it to the custom column.
- Make sure that you've selected the right Custom Column to update in the Reading List settings.

If you're stuck trying to get FFDL to automatically add stories to reading lists with its Reading List settings, there's another way you can try. Fair warning, this is more involved that the one in the tutorial example, and has some different limitations than the Reading List method. This is a rather messy way of doing this, and if you can find a way to get the automatic adding via Reading List to work, you would likely be better served.

- Go to the preferences for Reading List and in List type, under Population Options, set it to "Auto populated from search".
- In the text box in Population Options, write a search string that looks for a value you're using to indicate a story you want to sync. e.g.: tags:"=Sync to Nook" (this means look for Sync to Nook, indicated by the quotation marks after the colon, in the tags. If Reading List sees this string, it will add the associated story to the list. So next we add that tag to incoming stories.
- Go to FFDL preferences > personal.ini and under [defaults] add the line

extratags:Sync to Nook

This will add the tag "Sync to Nook" whenever you get a fic or update one using FFDL. Note: This will not add the tag to stories you update if in FFDL's settings > Standard Columns > Tags, you have "New only" checked, for obvious reasons. If you don't use the tags field, that doesn't really matter. If you do though, you may need to clean up your tags after update, which could be more work than just manually adding the fic to appropriate Reading List list. If you do use this method, and do put things in Tags manually, in FFDL preferences > Basic > Updating Calibe Options, check "Keep Existing Tags when Updating Metadata?", or you will lose any manually inserted/edited tags that don't match the story's tags.

Note: In the workaround option, I'm using an extratags: line. In the default personal.ini there are similarly

extracategories:Harry Potter
extracharacters:Draco Malfoy,Hermione Granger
extraships:Draco Malfoy/Hermione Granger

under [dramione.org]
What determines the acceptable values of "*" in extra*: lines, I'm not sure yet. Is it any valid entry for the level [default,format,site,override] the line is in, or just the ones defined without recourse to extra_valid_entries? Because if the former, I can set it up to put the Sync tag in a non-tag entry type, put that into a custom column, have it autoupdate without Reading List, and still have the "New only" checked for Tags. When I get around to it, I can test it and let you know, but I'm not sure when that will be. Having a site specific ability to add tag-like metadata in certain places could come in handy, so probably I'll look at it eventually, but I've got other things on the go.

As to places to start looking with customisation, well, I usually start with the documentation. Calibre's manual gives a good outline of what it can do, and you can get an idea of what you might want to try. For FFDL, a lot of the metadata management documentation is in the default personal.ini. You can find that in FFDL's settings > personal.ini > View Defaults (plugin-defaults.ini). More detailed descriptions of specific things people have done with FFDL are in this thread itself. You can save the thread in a searchable text document so that it's all in one place and easily searchable by clicking "Thread Tools" at the top of the thread, and "Download this Thread".

Specifically, for your question about getting books to stick out and to go to specific folders, go to Calibre Preferences > "Sending books to devices", look at the default template, and the list of standard template variables. These are also usable in Metadata Plugboards. Pay particular attention to Calibre's template language section in the Calibre manual. Take a look through the Reading List plugin thread, which is shorter than this one. You could also look at the "Technigues to use plugboards, custom columns and templates" post in Calibre > Library Management https://www.mobileread.com/forums/sho...d.php?t=118563 but I'd recommend reading the documentation first or the template strings won't make sense. Experiment (obviously, back up your library and write down the settings you're changing, particularly the default template somewhere, so that you can go back to it). What you are interested in is doable. I don't have a Nook myself, so if you're doing things with shelves, I can't actually test anything, and stuff I write frequently doesn't work exactly as I intend it at first, so you're probably better off giving it a try yourself.

Good luck!

That's incredibly useful - thank you so much!! I always feel awkward asking for help in case I just look stupid, but everyone on this forum is so helpful that it makes things so much nicer and easier
I've sorted the reading list thing by the way - it was a capitalisation problem, so my own stupid fault!
again!!!!!

[JimmXinu]

Originally Posted by seabream

...
"No Archive Warnings Apply", and "Author Chose Not To Use Archive Warnings" aren't printing, the rest are.
...
They are neither equivalent, nor null information, and I would like to collect it if I can, please.

You are correct, FFDL has been silently discarding the warnings "No Archive Warnings Apply" and "Author Chose Not To Use Archive Warnings". I'll start including them and users who don't want them can remove them with metadata rules.

[redscarlett]

Seems like I'm in-undating you today! Can someone help me correct this please? I want the '- -' to go if there is no series and/or series number, but when I try it out, the hyphens are still there. This is the template i'm using -

{series:||}/{title} - {series_index:0>4.1f|[|] - } - {authors}

Thank you!

[JimmXinu]

Turns out I had some time to kill today after all.

The attached version has the following changes that need testing:

• Fixes for harrypotterfanfiction.com
• Better comments in the ini files for section ordering
• Include AO3 warning tags "No Archive Warnings Apply" and "Author Chose Not To Use Archive Warnings"
• AO3 site specific tags for additional series(*)
• Fix for a FFDL bug when running pre-calibre 1.*

(*) There's a bug in calibre 1.* with setting custom series columns.

[archiveofourown.org]
extra_titlepage_entries: fandoms,freeformtags,ao3categories,comments,kudos,hits,bookmarks,series00,series01,series02,series03,series00Url,series01Url,series02Url,series03Url

custom_columns_settings:
 series01=>#text_series

[JimmXinu]

Originally Posted by redscarlett

Seems like I'm in-undating you today! Can someone help me correct this please? I want the '- -' to go if there is no series and/or series number, but when I try it out, the hyphens are still there. This is the template i'm using -

{series:||}/{title} - {series_index:0>4.1f|[|] - } - {authors}

This isn't an FFDL specific question. But I believe this will do what you want.

{series:||}/{title}{series_index:0>4.1f| - [|] - }{authors}

I haven't tested it.

[redscarlett]

Originally Posted by JimmXinu

This isn't an FFDL specific question. But I believe this will do what you want.

{series:||}/{title}{series_index:0>4.1f| - [|] - }{authors}

I haven't tested it.

Oooops!! Sorry! I just had this tag up, and I completely forgot

I've tested it, and it doesn't work, but thank you for trying, even when I posted it in completely the wrong place

[FaceDeer]

So, I noticed another oddity in my FiMFiction collection after doing a bunch of metadata cleanup; a story that I knew was quite old was listed as having been first published only earlier this month. Specifically, http://www.fimfiction.net/story/4464/ was listed as first published on August 25 2013 when the page says the first chapter went up almost a year ago on October 29 2012. I searched around and found some others that were way off, too, eg http://www.fimfiction.net/story/14392/ (FFDL says August 2013, page says March 2012) or http://www.fimfiction.net/story/2702/ (FFDL says August 3 2013, page says July 19 2012).

It looks like FiMFiction's API results have gone strange again. The API's metadata matches what FFDL reports, at least if you consider it as a UNIX timestamp (not sure what else it could be). So I went into adapter_fimfictionnet.py and figured out how to fix my own problem so it wouldn't be a hassle. I felt kind of embarrassed about how my previous bug report turned out to be a bug with Calibre itself rather than FFDL.

import dateutil.parser as dparser
...
        oldestChapter = datetime.now()
        #Scan all chapters to find the oldest, on FiMFiction it's possible for authors to insert
		#new chapters out-of-order or change the dates of earlier ones by editing them
        for chapterDate in soup.findAll('span', {'class':'date'}):
            rawChapterDate = chapterDate.contents[1].strip()
            chapterDate = dparser.parse(rawChapterDate)
            if chapterDate < oldestChapter:
                oldestChapter = chapterDate
        self.story.setMetadata("datePublished", oldestChapter)

This appears to correctly dig out the oldest chapter posting date from the story's description page and use that as the date published. I updated the metadata on 800 FiMFiction stories in my collection with no errors or apparent weirdness.

And then, while I was high on Python, I went and added an extra bit of site-specific metadata I've been thinking about for a while:

import HTMLParser
...
        rawGroupList = soup.find('ul', {'id':'story_group_list'})
		if rawGroupList is not None:
			for groupName in rawGroupList.findAll('a', {'href':re.compile('^/group/')}):
				groupString = HTMLParser.HTMLParser().unescape(groupName.string)
				if not isinstance(groupString,basestring):
					groupString = unicode(groupString)
				self.story.addToList("groups", groupString)

(don't know if that HTML unescape step is really necessary, there was a bunch of escaped HTML in the group titles of the test story I was playing with and I figured better safe than sorry).

Plus of course the addition of "groups" to plugin-defaults.ini's list of extra metadata for this site. This was tested with those 800 stories too and I didn't get any screwy group titles out of it or unhandled errors.

I hope these snippets are up to standard, and make up for my lazy bug reporting a few days back.

[seabream]

Hi JimmXinu,

Re: Silent discard of AO3 warnings, and adding site specific tags for additional series on AO3 - Thank you kindly.
Testing in progress.

Re: logpage_entries
There's still something strange happening. Even after I removed the logpage_entries line from [epub] and put it in defaults, it wasn't showing up as specified. So I decided to try a few other things in my working personal.ini which didn't work. No matter what I put in the AO3 section, for logpage_entries (edited to remove the extra_logpage_entries part, I misremembered the content of the personal.ini used to test that fic), the logpage had the same entries in the same order. In case it was a problem with some other line in my personal.ini, I then substituted the default personal.ini (I'm still using FFDL 1.7.33 on Calibre 0.9.40. I usually don't update Calibre unless there's a feature or bugfix that I actually use given Kovid's statement about host bandwidth and how for the most part people can update once a year. Also, in this case, I wanted to wait for the major new version bugs to be ironed out. Looking at the version history, it doesn't look like there should be changes in how logpage_entries is handled.).

After changing include_logpage: false to true, in [epub], I removed all the "logpage_entries:" lines and, for simplicity's sake, put "logpage_entries: author" under the "titlepage_entries: " line in [defaults]. And because this was going to get repetitive, I switched to using test1.com?sid=100* increments. And I still got the same logpage. Is there something prioritising defaults.ini (which I presume is where the logpage_entries entries are coming from for the first two, since there are no logpage_entries lines in personal.ini that contain those entries) over the logpage_entries parameter in [defaults] and [sites] in personal.ini? If I put a logpage_entries line in the [epub] section, it is expressed, but when there isn't one there, (or in overrides), the others should work right? I'm kind of baffled.

For reference, copy and pasted log data where "logpage_entries: author" is in [X], and is the only "logpage_entries: *"line in personal.ini:

Spoiler:

[defaults]
Update Log

Packaged: 2013-09-05 04:22:14 Published: 2013-03-15 Updated: 2013-04-15 Chapters: 4 Words: 1,456 Status: In-Progress Series: The Great Test [4] Title: Testing New Feature 1007 Author: Bob Smith Category: Crossover, Furbie, Harry Potter, Magical Girl Lyrical Nanoha, Puella Magi Madoka Magica/é*”法少女ã�¾ã�©ã�‹â˜…マギカ Genre: Comedy, Fantasy, Noir, Sci-Fi Rating: Tweenie Warnings: Swearing, Violence

[test1.com]
Update Log

Packaged: 2013-09-05 04:23:33 Published: 2013-03-15 Updated: 2013-04-15 Chapters: 4 Words: 1,456 Status: In-Progress Series: The Great Test [4] Title: Testing New Feature 1008 Author: Bob Smith Category: Crossover, Furbie, Harry Potter, Magical Girl Lyrical Nanoha, Puella Magi Madoka Magica/é*”法少女ã�¾ã�©ã�‹â˜…マギカ Genre: Comedy, Fantasy, Noir, Sci-Fi Rating: Tweenie Warnings: Swearing, Violence

[epub]
Update Log

Author: Bob Smith

I performed the same test with the test version of FFDL 1.7.39 (or prospective 1.7.40 I suppose), using modified copies of the default personal.ini that came with it (By the way, the order of precedence comments for overrides are clearer, thank you.) and got the same results for those three. I decided to also test [test1.com:epub] and [overrides]. They both worked. I've attached the debug log (removing my username from the filepath where appropriate) for running the import using "Copy of defaults from 1,7,40 logpage in defaults" also attached, in case that might be useful.


Further notes on comments:
You may wish to clarify when one should/can use "add_to_*" vs. "extra_*".

Also, given the comment below,

Originally Posted by JimmXinu

You don't need to put these extra_valid_entries & labels in your personal.ini. They're already in the built-in defaults.ini. In fact, by doing this you are removing comments, kudos, hits, etc.

you may want to have a note about the interaction between what you put in your personal.ini and what's in defaults.ini, as well as what is actually necessary to put in personal.ini. Until reading the above comment, I'd been operating on the premise that extra_valid_entries, etc… were required in personal.ini, specifically because they were in the sample version of it. It's not obvious if you're using the GUI version what defaults.ini does, and what bearing it has on what you put in personal.ini. If it weren't for this thread, I don't know that I'd know that I should use add_to_extra_valid_entries as opposed to just extra_valid_entries for various custom column entries, since extra has the connotation of 'adding to' already, for example.

Actually, that's another question. Is the default.ini used by FFDL the same as the one that comes up as a sample of personal.ini? Unzipping an old version of FanFictionDownLoader.zip that's still in my downloads folder, I find plugin-defaults.ini, and plugin-example.ini, but they aren't the same. The latter is considerably shorter.