09-20-2012, 10:21 PM | #856 |
Plugin Developer
Posts: 6,284
Karma: 3966249
Join Date: Dec 2011
Location: Midwest USA
Device: Kindle Paperwhite(10th)
|
Hmm. Well, what I was thinking about wouldn't have caught that anyway--that's a regexp issue, not just an ini file problem. I've added that notation to the wishlist item, though.
|
09-21-2012, 02:00 PM | #857 |
Plugin Developer
Posts: 6,284
Karma: 3966249
Join Date: Dec 2011
Location: Midwest USA
Device: Kindle Paperwhite(10th)
|
Version 1.6.10 - 21 Sep 2012
New sites (thanks Marillapm):
This version contains some metadata features that have been requested, plus quite a bit of internal refactoring to allow them. New Features:
For now, I've only added a few site specific metadata that I know have been asked for. I repeat: only a few site specific metadata listed below have been added so far. New 'Extra' Metadata:
dramione.org is the prototype for this feature; ini code: Spoiler:
Assuming this works well, I'll take requests on extra metadata to add for other sites. I'm not, however, planning to add new metadata across the board. It's easy to say 'add reviews for all sites', but to actually add it to all 75 adapters is not. Jim |
09-21-2012, 04:36 PM | #858 |
Enthusiast
Posts: 40
Karma: 98
Join Date: Jan 2011
Device: Kindle 3G
|
Hey, Jimm. I'm getting a strange error, here. I've downloaded Hundreds of stories over the past week, but for these 4 stories (And only these 4), I get the error "'NoneType' object has no attribute 'group'"
Here are the story urls: http://www.fimfiction.net/story/6270/Cheater http://www.fimfiction.net/story/6012/Had-a-Rough-Day http://www.fimfiction.net/story/5827...-What-They-Say http://www.fimfiction.net/story/5824/Twilight-Mornings I thought at first that it possibly had something to do with the author's name? Perhaps the period character is the cause? But, I am able to download this story: http://www.fimfiction.net/story/15325/Something-Secret Which is by the same author as the first 4. This is what I see when I try to download Mr.V's stories: I will say that about a month ago, I was able to download Mr.V's stories without problems, so I'm wondering if perhaps a recent change to the site, or perhaps to the plugin, jostled something in the code. :P If you could look into the problem, I'd be grateful. (BTW, the changes you made for the bbcode/html description issue and the fimfiction covers issue have worked wonderfully. I've had no issues since.) |
09-21-2012, 05:17 PM | #859 | |
Plugin Developer
Posts: 6,284
Karma: 3966249
Join Date: Dec 2011
Location: Midwest USA
Device: Kindle Paperwhite(10th)
|
Quote:
I've made a tweak to the bbcode parser that I think fixes it. Please give this version a try. Last edited by JimmXinu; 10-02-2012 at 10:49 AM. Reason: Remove obsolete beta versions |
|
09-22-2012, 01:17 AM | #860 | |
Enthusiast
Posts: 40
Karma: 98
Join Date: Jan 2011
Device: Kindle 3G
|
Quote:
You rock. EDIT: Yep, it's working fine now. So it was a plugin update that messed things up! (I am so sorry for that I appreciate what you do oh god please don't be mad) Last edited by crackshot91; 09-22-2012 at 01:20 AM. |
|
09-22-2012, 11:03 AM | #861 |
Enthusiast
Posts: 48
Karma: 1056
Join Date: Jul 2012
Location: Central Canada
Device: Sony PRS-T1, RIM Playbook & Z10, Samsung Note, MacOSX
|
1.6.10 feature test
As one of the people who requested the inclusion of the new categories for FanFiction.net, thanks for the new functionality. I respect the amount of work that it must have taken, and appreciate the flexibility in how they have been implemented.
Tested on a Macbook Pro running Mac OS X 10.7.4, Calibre 0.8.64 and the beta of FFDL provided in post #844 (updated with notes on 1.6.10), with fics from Fanfiction.net using the following settings in personal.ini: Spoiler:
All of it works as described with multiple fics. However, Get Story URLs from Web Page seems to now be broken for Fanfiction.net. Attempting results in: Traceback (most recent call last): File "calibre_plugins.fanfictiondownloader_plugin.ffdl_ plugin", line 251, in get_urls_from_page NameError: global name 'url' is not defined Attempted with: http://www.fanfiction.net/u/1728624/ and http://www.fanfiction.net/u/2469838/twitchy-witch By accident, I tried it with a non-link and got the same result. I don’t actually read at dramione.org, but since you apparently haven’t gotten many replies on it, I decided to test that too. I made the custom columns listed in custom_columns_settings and used the following basically default settings mentioned in the described personal.ini: Spoiler:
Picking a random popular author with short stories (ThatRomantic) I tried And What If They Were Lovers ? http://dramione.org/viewstory.php?sid=533 and Never Crossed Their Lips http://dramione.org/viewstory.php?sid=532 which worked except for Genres. I eventually realized the mistake, but you may want to add to the notes that in order for include_in_genre:genre,themes to work you need to add genre=>#genre under custom_column_settings too, because from the description, at first glance, it looks as though one doesn’t need to do anything in addition. For reference: ## For another example, you could, by uncommenting this line, include ## themes in with genre metadata. #include_in_genre:genre, themes Testing with a few more stories by the same author: And What If Karma Caught Up With Them? http://dramione.org/viewstory.php?sid=535 and Battlefield: http://dramione.org/viewstory.php?sid=1479 and I got this from the Updates completed, final status window: Comments: sequence item 0: expected string or Unicode, NoneType found If I commented #include_in_genre:genre, themes the comment went away. Commented or not, Hermiones, Dracos, Timeline and Themes don’t populate with the above two stories. Genre doesn’t give the composite column, but does populate, strangely, with “FanFiction”, which isn’t included with the themes in the Genre column for the story that worked. (I have Genre set to populate with Extra Tags in the GUI Custom Column tab in the FFDL preferences, but when I put include_in_genre:genre,themes,extra_tags and try http://dramione.org/viewstory.php?sid=533 “FanFiction” still doesn’t show, so I’m not sure where it’s coming from) Not related to new features, but noticed that for Never Crossed Their Lips http://dramione.org/viewstory.php?sid=532 which of the stories I checked is the only story with an image, even though as shown below I have make_firstimage_cover:false, both under [epub] and [defaults], the epub uses the banner as the title page anyway. Spoiler:
While I was writing this up, 1.6.10 was released, so I retested on dramione and FanFiction.net. The parts that didn’t work still didn’t. I didn’t check all of the parts that did, but I’ll let you know if I notice anything new. Also not new to this version, but something I hadn’t tried before. If I call for the FFDL version in the log or title page, with 1.6.10 it shows version 4.4. Same with 1.6.09. Is this intentional? I can confirm that the relationships meta information is working for me for AO3. I can’t find the setting for adding a keyboard shortcut for FFDL configuration. Calibre > Preferences > Keyboard > FanFictionDownLoader only has the following options [Add New FanFiction Book(s) from URL(s), FanFictionDownloader, Get Story URLs from Web Page, Get URLs from Selected Books, Update Existing FanFiction Book(s)] and it doesn’t seem to be anywhere in the FFDL preferences. Is it somewhere else? Also, I’m not sure exactly when this is was but at least from 1.6.3, fics from FanFiction.net are downloading with left justified text (as displayed on the website) showing up as justified in the epub. I can change it with Convert Books, but that makes everything left justified. Question about tag substitution. Though I haven’t actually implemented it, in principle I know how to automate tag substitution on import in personal.ini globally, or on a site by site basis with replace_metadata and regexp. But is there a way to have FFDL take a given tag and use different rules for replacement depending on the source category and/or destination column? e.g.: input type: Characters, entry “Han S.” output to Tags as “char Star Wars: Han Solo” and to Characters as “Star Wars.Han Solo”? Designating which metadata category a rule applies to is mostly needed for fandoms where the fandom name is also that of a main character, or a main character’s name tends to show up in story titles, but also because I don’t want to automatically change author names when in fan works, they can contain strings that may also appear in just about any other category. I can see how to do something sort of like it after the fact using Bulk Metadata Edit, though it’s two steps, but I’d need to redo it as I add to the library, and as the library gets larger, running it would take progressively longer, and I’d need to do it separately for every single tag I want to replace (which total in the four digits by now). Reason being that I use the custom column with heirarchical tags in Calibre, but not all my mobile readers support tags (in three cases, not at all, and in a limited way with the Sony i.e.: it gets very slow above a few hundred tags, and it’s not always accurate, FB Reader on the Samsung is good, but it’s not my phone), which means that I’m using the output of Calibre’s Generate Catalog, which doesn’t include tags from custom columns. By the way, generally feel free to P.M. me if you have things you want tested. I don’t check the forum every day, or week, and I won’t guarantee that I’ll have time to test immediately, but I’m happy to help when there’s time. Last edited by seabream; 09-22-2012 at 11:13 AM. Reason: typos and adding spoiler tags for length |
09-22-2012, 12:38 PM | #862 |
Plugin Developer
Posts: 6,284
Karma: 3966249
Join Date: Dec 2011
Location: Midwest USA
Device: Kindle Paperwhite(10th)
|
|
09-22-2012, 02:21 PM | #863 | ||||||||||||||
Plugin Developer
Posts: 6,284
Karma: 3966249
Join Date: Dec 2011
Location: Midwest USA
Device: Kindle Paperwhite(10th)
|
Quote:
Quote:
Quote:
Quote:
Quote:
'include_in_genre' adds the theme list to the genre list. The idea is that if you already have #genre column defined and have set FFDL to populate from 'genre' on the Custom Columns tab (not custom_column_settings), that will add themes to it, too. If you uncomment include_in_genre:genre, themes, both genre and themes appear after the Genre heading on the title page and in custom columns set to populate from 'genre' on the Custom Columns tab. I'm happy to change the documenting comment to be more clear, do you have any suggestions? Quote:
Quote:
Quote:
I'm open to suggestions if this doesn't seem like the correct behavior. Quote:
Quote:
Quote:
Quote:
Spoiler:
Quote:
Quote:
Many thanks for your help. Hopefully I can get out a version addressing these issues today yet. |
||||||||||||||
09-22-2012, 04:07 PM | #864 |
Plugin Developer
Posts: 6,284
Karma: 3966249
Join Date: Dec 2011
Location: Midwest USA
Device: Kindle Paperwhite(10th)
|
This version fixes the 'Get Story URLs from Web Page' issue, and the problem when using custom_columns_settings to set a column with 'a' flag and the story doesn't have any of that particular metadata.
This also introduces a different way to deal with the problem of Mac users crashing calibre. To recap: A couple of Mac users have experienced calibre crashes after installing FFDL PI. Long story short, I removed the Configure Plugin and About Plugin menu items for all Mac users to fix it for one user. It has since occurred to me that I have no idea how many Mac users FFDL PI actually has and how many of them have this problem. Am I inconveniencing a bunch of people for a problem only one user has? So I came up with a different way remove those menu items for those users it crashes for. Create a file named 'fanfictiondownloader_macmenuhack.txt' in your calibre/plugins directory. If some Mac user could post an example of where that typically is on Mac, I'd appreciate it. (Theoretically, it should be possible to figure out why it only happens to FFDL and fix/workaround it. But without having a Mac that's doing it to experiment on--or at least a user with one willing to try dozens of versions--I don't see that happening.) Please let me know how this version works for you. Thanks, Jim Last edited by JimmXinu; 10-02-2012 at 06:25 PM. Reason: Remove obsolete beta versions |
09-23-2012, 05:56 PM | #865 |
Plugin Developer
Posts: 6,284
Karma: 3966249
Join Date: Dec 2011
Location: Midwest USA
Device: Kindle Paperwhite(10th)
|
No feedback on the previous test version yet, hopefully that means it's working fine.
This version improves the handling of include_in_* config so that include_in_* lines can now include each other. It also adds the ability to restrict replace_metadata lines to only modify specific metadata by entry/key word. Code:
replace_metadata: genre=>Sci-Fi=>SF genre=>Comedy=>Humor characters,ships=>Harry P\.=>Harry Potter By using extra_valid_entries, I believe you should be able to do anything you would be able to if I had implemented replacement by destination column. Code:
[somesite] extra_valid_entries:editedchars editedchars_label:Characters by site include_in_editedchars:characters replace_metadata: editedchars=>^(.+)$=>somesite:\1 extra_titlepage_entries: editedchars custom_columns_settings: editedchars=>#hierchars,r Last edited by JimmXinu; 10-02-2012 at 06:25 PM. Reason: Remove obsolete beta versions |
09-24-2012, 03:26 AM | #866 | |||||||
Enthusiast
Posts: 48
Karma: 1056
Join Date: Jul 2012
Location: Central Canada
Device: Sony PRS-T1, RIM Playbook & Z10, Samsung Note, MacOSX
|
@863, 864, 865
Sorry, I’m not generally going to have as quick a response time as you do. It’s nice, but it’s not something I can manage all the time myself.
Quote:
Quote:
Quote:
Re-tested Get URLs from Web Page, and it works properly too. The menu hack works, as far as removing the menu items goes. It’s not perfect since it also disables shortcut keys for those items if they’ve been set prior. Either way, it’s not crashing yet, which I do indeed remember, so that’s all good. The Calibre plugin folder is at ~/Library/Preferences/calibre/plugins where ‘~’ is the user’s home folder, which is generally (though not always) called their username, and located at Primary Hard Drive/Users/ Note that in OS X 10.7 and up, users’ Library folders (as distinct from the system Library folder) are hidden. One can get there in Finder by clicking Go > Go to Folder… in the menu bar and entering ~/Library/Preferences/calibre/plugins in the dialogue box. The default keyboard shortcut for Go to Folder is Command + Shift + G Quote:
Quote:
Quote:
Quote:
You mentioned taking metadata requests: Could you add support for isolating what AO3 calls “Category” on the story page itself, and “Relationships, pairings, orientations” in the symbols key (F/F, Gen, F/M, Multi, Other, M/M) as its own metadata source? Currently, it’s lumped in with extra tags, but that means that I can end up with a field that says, for example: “Angst, Insanity, truly bad and wrong, F/M,” which is less useful for sorting purposes. There was a brief period where I could call it as “Category” some months ago, but now that pulls Fandom on AO3. I don’t want to a custom column set to “Text, with fixed set of permitted values”, because there are other classes of relationships I want to include from other sites without adding additional columns. If it’s not too greedy, I’d also like to request counts for kudos, comments, bookmarks, and hits on AO3. |
|||||||
09-24-2012, 09:37 AM | #867 |
Junior Member
Posts: 8
Karma: 10
Join Date: Jul 2012
Device: none
|
Jim seem to be having problems with yourfanfiction.com FFDL seem to no longer be able to recognize new urls. It will updated the ones form book already made but when you try to put in a new one it does nothing.
|
09-24-2012, 09:53 AM | #868 |
Enthusiast
Posts: 48
Karma: 1056
Join Date: Jul 2012
Location: Central Canada
Device: Sony PRS-T1, RIM Playbook & Z10, Samsung Note, MacOSX
|
@865, testing restricted replace_metadata
Currently just tested with ffnet
FFDL Version now shows up properly in titlepage and logpage replace_metadata is beautiful (after some trial and error figuring out how to configure things). Thank you so much. Once I actually put all the various tags in, this will save so much time and reduce data entry errors from manual substitution. For anyone who might want some examples, here’s an illustrative configuration. Obviously add categories as desired. EDIT: for some reason, when posted, there are random spaces inserted here and there in the spoilered block. They don't appear in the edit window, so I don't know how to get rid of them, so if you use the code, proofread it. Spoiler:
I’ll get back to you on test results with AO3 when I’ve got them, but I don't expect big differences. Last edited by seabream; 09-24-2012 at 11:59 AM. Reason: added note about strange spacing. |
09-24-2012, 11:03 AM | #869 |
Plugin Developer
Posts: 6,284
Karma: 3966249
Join Date: Dec 2011
Location: Midwest USA
Device: Kindle Paperwhite(10th)
|
|
09-24-2012, 01:11 PM | #870 |
hopeless n00b
Posts: 5,111
Karma: 19597086
Join Date: Jan 2009
Location: in the middle of nowhere
Device: PW4, PW3, Libra H2O, iPad 10.5, iPad 11, iPad 12.9
|
Been on a fanfic reading binge/marathon so didn't realize all these changes are being made to FFDL. Will try to get around to testing when I find time. Thank you very much!
|
Thread Tools | Search this Thread |
|
Similar Threads | ||||
Thread | Thread Starter | Forum | Replies | Last Post |
[GUI Plugin] Resize Cover | kiwidude | Plugins | 95 | 03-16-2024 11:55 PM |
[GUI Plugin] Open With | kiwidude | Plugins | 402 | 03-16-2024 11:44 PM |
[GUI Plugin] Find Duplicates | kiwidude | Plugins | 1096 | 03-16-2024 11:28 PM |
[GUI Plugin] Count Pages | kiwidude | Plugins | 1744 | 03-02-2024 05:39 PM |
[GUI Plugin] Plugin Updater **Deprecated** | kiwidude | Plugins | 159 | 06-19-2011 12:27 PM |