Register Guidelines E-Books Search Today's Posts Mark Forums Read

Go Back   MobileRead Forums > E-Book Software > Calibre > Plugins

Notices

Reply
 
Thread Tools Search this Thread
Old 01-09-2019, 01:37 PM   #3211
Atherton25
Connoisseur
Atherton25 has learned how to buy an e-book online
 
Posts: 66
Karma: 94
Join Date: May 2016
Device: Kindle Oasis 2
Alright. I did some testing, and I get dups every time when I do this.
So I have several virtual libraries setup. These are all permanent, but I checked with a temp virtual library just simple filtering out something that makes all my FFF stories unique (I have a tag I add called FFF so it makes filtering out FFF stuff easier).

JimmXinu, I'm not sure how you performed your testing, but this is essentially my testing steps

1. Since this was a test to see if it would cause dups I just found a very short story on one of the sites (during testing I tried 3 different stories, one on Fanfiction.net, one on hpfanficarchive.com and one on ficwad) for the last test I did, I grabbed https://ficwad.com/story/278903 a 1k word, 1 chapter story on ficwad. Clicked the FFF icon, Downloaded the story.
2. Once the story was in my library I searched for "not tags:FFF" to filter it out (along with all my other FFF stories)
3. Went to virtual library button and clicked on 'current search' to use the current search as the filter for my temporary library.
4. Now, in the new virtual library I clicked on FFF button, added the story url. Though I don't think it matters, the selection for the dropdown is:
output: epub
if story exists: update epub if new chapters
Update Metadata checked, Update Epub Cover checked.
(I also checked a the drop down options and selected overwrite if newer in one of the tests in one of the tests)
5. Clicked okay and in a few seconds it wanted to add a new story to my library. Once I closed out of the temporary library I had duplicates. 4 in one case, as I tried a couple different options and refreshed the Virtual library to make sure the old one was filtered out before trying the new test.


Now, the way I see it (and this is what I meant when I brought it up the other day), this is a feature of calibre and there's not a lot that FFF can do about it. When in a Virtual Library what exists outside that library is considered a different library and since the filters I used removes anything downloaded with FFF, once I turn on the Virtual Library option those stories no longer exist in my library (on a temporary basis) so FFF can't see them.

It's not a bad thing, it just means that if I'm doing other work and using a Virtual Library tab, I shouldn't try and use the update email feature or add new stories that I might already have. ... or be watchful that I don't accidentally click one of my tabs before using those features, which has happened several times.

Adding more information that probably doesn't matter, some of my libraries are based on custom columns (I have a 'library' column too for different things like fanfiction/webfiction/fiction/nonfiction/etc) and I used the same thing to filter out a FFF story and it still did the same thing.

Edit:
Just for reference. And this probably doesn't matter, but I wanted to make sure. I'm using on Win7, using the latest version of Calibre (x64), the latest version of FFF. I have used the cli once or twice in the past, but nearly always use the gui version.

I've also had the same problem on my laptop in the past (Win10) a couple of times. Most of the time if I realize I've done it (generally because what should be a quick update is taking too long) I just stop the download, go into my full library and restart downloading correctly.

Last edited by Atherton25; 01-09-2019 at 01:45 PM.
Atherton25 is offline   Reply With Quote
Old 01-09-2019, 02:08 PM   #3212
JimmXinu
Plugin Developer
JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.
 
JimmXinu's Avatar
 
Posts: 3,532
Karma: 589322
Join Date: Dec 2011
Location: Midwest USA
Device: Kindle Voyage
Okay, NOW I think I see what's going on.

Atherton25, please look in your FFF settings, do you have "Search by Title/Author(s) for If Story Already Exists?" on or off?

If you have it off, please check it on and try your test again.

What I'm seeing is that the identifier search is 'honoring' Virtual Library and the title/author search doesn't. And because I always have "Search by Title/Author" on (it's on by default), I--and most users--never saw this.
Code:
## honors VL??
self.gui.current_db.search_getting_ids('identifiers:"~ur(i|l):~^%s$"'%re.sub(r'https?\\\:','https?(\:|\|)',re.escape(url)),None)

## ignores VL??
mi = MetaInformation(book['title'],book['author'])
identicalbooks = self.gui.current_db.find_identical_books(mi)
I'm going to go look in the Calibre code at these methods to see what's different between them.
JimmXinu is offline   Reply With Quote
Old 01-09-2019, 02:18 PM   #3213
Atherton25
Connoisseur
Atherton25 has learned how to buy an e-book online
 
Posts: 66
Karma: 94
Join Date: May 2016
Device: Kindle Oasis 2
Quote:
Originally Posted by JimmXinu View Post
Okay, NOW I think I see what's going on.

Atherton25, please look in your FFF settings, do you have "Search by Title/Author(s) for If Story Already Exists?" on or off?

If you have it off, please check it on and try your test again.
You're right, it fixed it.

But, I keep that unchecked for a reason, since I've run into sites that have 2 different versions of a story and generally I want both. Plus, this issue doesn't really bother me much, I can simply search for duplicates with duplicate finder (by url) and delete them when I do this. It was fathertyme that had brought it up. At least it helped (possibly) identify an issue he and others might be having. I'll try and remember to note even more calibre options next time if there's a possible issue.



Moving on, I know this has been brought up before in some form (probably by me) but I've had a lot of new issues/stories that I've had to make edits in my personal.ini for. My current personal.ini has 90+ (and growing) specific story's now in it to 'fix' this, I'm going to ask about it.

The issue actually stems is mostly due to the way the wuxia sites do things, and even worse on some than others. Generally due to translation, sometimes the title/author can vary a bit from site to site, and sometimes they even just use the Chinese/Japanase name rather than a translation. Also the description can be really screwy (on royal road as well), adding a lot of superfluous (update schedules for example, but a while back I had one wuxia story that had like 5 pages in the description that was not needed).

As far as I know (based on past questions) the only way to keep an edited version of a new title/author/description in the FFF metadata (specifically on the title page) is to manually add it into the personal.ini.

Am I incorrect? If I'm not, how difficult would it be to have an option to grab the calibre metadata instead of the website metadata? I know I can edit it in calibre and use options to keep the calibre data the same, but it never reflects inside the book title page for those specific options. If it's too difficult just ignore me, I'm probably the only person that gets so annoyed by this issue partly because I really like clean, proper metadata, partly because some wuxia sites don't always give a real description and I have to get it from another site (generally novelupdates.com) for something that actually makes sense.

On a side note, somehow getting a downloaded story to interconnect with novelupdates and pulling metadata from their would be awesome but likely way too much work.
Atherton25 is offline   Reply With Quote
Old 01-09-2019, 02:28 PM   #3214
JimmXinu
Plugin Developer
JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.
 
JimmXinu's Avatar
 
Posts: 3,532
Karma: 589322
Join Date: Dec 2011
Location: Midwest USA
Device: Kindle Voyage
New Test Version Posted

2019-01-09
- Ignore current Virtual Library when checking for existing story ID.

Note that this could be change to the existing behavior for some Virtual Library users. I could potentially make this optional, but I'd much rather it just be consistent.

Quote:
Originally Posted by Atherton25 View Post
You're right, it fixed it.

But, I keep that unchecked for a reason, since I've run into sites that have 2 different versions of a story and generally I want both. Plus, this issue doesn't really bother me much, I can simply search for duplicates with duplicate finder (by url) and delete them when I do this. It was fathertyme that had brought it up. At least it helped (possibly) identify an issue he and others might be having. I'll try and remember to note even more calibre options next time if there's a possible issue.
Many thanks to Atherton25 for helping figure this out.

Now I'll go read the rest of that last post.
JimmXinu is offline   Reply With Quote
Old 01-09-2019, 02:43 PM   #3215
Atherton25
Connoisseur
Atherton25 has learned how to buy an e-book online
 
Posts: 66
Karma: 94
Join Date: May 2016
Device: Kindle Oasis 2
Quote:
Originally Posted by JimmXinu View Post
New Test Version Posted

2019-01-09
- Ignore current Virtual Library when checking for existing story ID.

Note that this could be change to the existing behavior for some Virtual Library users. I could potentially make this optional, but I'd much rather it just be consistent.
Yeah... I don't know why people would keep 2 version of the exact same story, but there are some other possibilities.

I do know that some authors repost a 'revised' version under the same url sometimes, or remove chapters (particularly in the case of royalroad and people using kindle unlimited).

I generally just remove the url identifier and add [old] to the title or something, then add the story again in it's new/revised form. *But* I did for a while use sorting to allow me to update the new chapters only (before I figured removing the URL and unchecking title/author allowed me to do a normal update via url rather than highlighting)

This was before vlibraries, where I could have just auto filtered them out before trying to update. It is possible some people use that method for one reason or another.

Last edited by Atherton25; 01-09-2019 at 02:47 PM.
Atherton25 is offline   Reply With Quote
Old 01-09-2019, 02:53 PM   #3216
Atherton25
Connoisseur
Atherton25 has learned how to buy an e-book online
 
Posts: 66
Karma: 94
Join Date: May 2016
Device: Kindle Oasis 2
Bah, for information purposes if anyone's interested and can't figure it out themselves. For anyone trying to 'fix' authors on webnovel.com so it doesn't add editor,

replace_metadata:
author=>^(.+?)Translator(.*)=>\1

placed in your personal.ini will fix this, removing the translator name completely, as well as editor if there is one in the metadata. It could be further modified to add translator as a second author name, but I just keep original author personally. Don't know if it helps anyone, but I figured I'd post it for those with little to no knowledge of regex.

Last edited by Atherton25; 01-09-2019 at 02:59 PM.
Atherton25 is offline   Reply With Quote
Old 01-09-2019, 03:12 PM   #3217
JimmXinu
Plugin Developer
JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.
 
JimmXinu's Avatar
 
Posts: 3,532
Karma: 589322
Join Date: Dec 2011
Location: Midwest USA
Device: Kindle Voyage
Quote:
Originally Posted by Atherton25 View Post
... As far as I know (based on past questions) the only way to keep an edited version of a new title/author/description in the FFF metadata (specifically on the title page) is to manually add it into the personal.ini. ...
It's a little involved, but I think you can do at least part of this.
  1. First, you'll need to identify (and/or create) the columns you want to hand edit.
  2. If they are columns that are auto-populated, make sure under the FFF Standard Columns and/or Custom Columns tabs that they are checked 'New Only' so FFF doesn't change them.
  3. Next, go to the personal.ini FFF tab. Towards the bottom, make sure "Pass Calibre Columns into FanFicFare on Update/Overwrite" is checked.
  4. Just under that, click the "Show Calibre Column Names" button. That will show you a list of your custom and standard Calibre columns and their FFF metadata entry names. Copy the names for the columns you care about somewhere for reference. Eg, you might want calibre_std_title, calibre_std_comments & calibre_cust_characters.
  5. Now go into personal.ini and add those (calibre_std_comments, etc) to titlepage_entries as you would like. Either by outright replacing the default titlepage_entries or using add_to_titlepage_entries, extra_titlepage_entries or even add_to_extra_titlepage_entries in appropriate sections. (I won't write an essay about the differences unless you need it.)
Example:
Code:
[defaults]
calibre_std_comments_label:My Comments
add_to_titlepage_entries:,calibre_std_title,calibre_std_comments,calibre_cust_characters
Some issues:
  • During the download of a new story, there won't be any values for calibre_std_comments, etc, so they will be empty.
  • The label for each of these (calibre_std_comments_label, etc) are automatically set to match the Calibre column name. You can also set them in personal.ini for your self.
  • Trying to change the title page to use your calibre_std_title entirely (see wiki) would possible, but difficult--again, due to not having calibre_std_title available on initial download. You'd probably want/need to keep the site title even if you reduced its size.

Quote:
Originally Posted by Atherton25 View Post
... I generally just remove the url identifier and add [old] to the title or something, then add the story again in it's new/revised form. *But* I did for a while use sorting to allow me to update the new chapters only (before I figured removing the URL and unchecking title/author allowed me to do a normal update via url rather than highlighting) ...
I do the same; remove identifier, change title.

And I can vaguely imagine someone deliberately using VL to get 'duplicates'. But because the id and title/author searches were inconsistent and people were reporting it as a bug, not a feature, I'm going to assume that consistent searches that can't make duplicates are the best way to go.

If somebody says they were using it that way, I'll revisit it then.
JimmXinu is offline   Reply With Quote
Old 01-09-2019, 03:22 PM   #3218
Atherton25
Connoisseur
Atherton25 has learned how to buy an e-book online
 
Posts: 66
Karma: 94
Join Date: May 2016
Device: Kindle Oasis 2
Quote:
Originally Posted by JimmXinu View Post
It's a little involved, but I think you can do at least part of this.
Excellent. Thank you very much.

I'll see about implementing this. I know I've seen (and done) some strange workarounds before. As long as they work, they work.

It's probably better than doing the (now semi constant) updates to the ini file for new books. Especially because people on wuxiaworld.co don't even put in a full description half the time. The just grab what's available and half the time it's cut off mid-point.
Atherton25 is offline   Reply With Quote
Old 01-12-2019, 01:41 AM   #3219
Mara_Skywalker
Junior Member
Mara_Skywalker began at the beginning.
 
Posts: 2
Karma: 10
Join Date: Aug 2018
Device: laptop balabolka
I have two problems with fanficfare:

What can I do to download fanfictions or quests from forums like sufficient velocity, spacebattles, questionable questing, alternate history that have neither threadmarks nor links in the first post?

I would like to download fics that have story/quest only threads like this one:

https://forums.sufficientvelocity.co...-archive.3519/

but on all the forums like this.

And I have problems with quests that are too long, for example, if I want to download this here:

https://forums.sufficientvelocity.co...r.27579/reader

It does not download all threadmarks, only the ones visible, so a lot of posts in the middle are not downloaded. Do you understand what I mean?

Would it be so hard to give the possibility to just download all the posts in a story only thread, by choosing that manually?

And to somehow also make it possible to download the quests, that have too many threadmarks?

Thank you for your help!

I do not know anything about programming, or I would try to make something myself.

I need the whole quest/story in epub format to open it with the freeware balabolka, that is capable of reading me the stories aloud because I really cannot read that long without my eyes starting to hurt.
Mara_Skywalker is offline   Reply With Quote
Old 01-12-2019, 12:37 PM   #3220
JimmXinu
Plugin Developer
JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.
 
JimmXinu's Avatar
 
Posts: 3,532
Karma: 589322
Join Date: Dec 2011
Location: Midwest USA
Device: Kindle Voyage
Quote:
Originally Posted by Mara_Skywalker View Post
What can I do to download fanfictions or quests from forums like sufficient velocity, spacebattles, questionable questing, alternate history that have neither threadmarks nor links in the first post?
This is an issue that's come up before. See those posts for why FanFicFare doesn't implement whole-thread downloading.

Quote:
Originally Posted by Mara_Skywalker View Post
And I have problems with quests that are too long, for example, if I want to download this here:

https://forums.sufficientvelocity.co...r.27579/reader

It does not download all threadmarks, only the ones visible, so a lot of posts in the middle are not downloaded. Do you understand what I mean?
Back in Sept SB & SV changed to 'hide' ranges of threadmarks behind '...' links. Code was added to FFF then to read past them. Do you have the current version of FFF installed?

I ran the chapter collection part of FFF on the story given and found 3,359 chapters, which is reasonably close to what the site reports(see below). I'll start it downloading and check it later.

FYI, SB/SV have an unfortunately tendency to not list all of the threadmarks for some categories when the list is small, or to report threadmark numbers that are off. There isn't much I can do when the site gives out bad data.

UPDATE: It took a couple hours with the slow_down_sleep_time:10 setting I threw in, but I downloaded that story successfully.

If you still have issues or want a copy of the epub I downloaded, let me know.

Last edited by JimmXinu; 01-12-2019 at 02:40 PM.
JimmXinu is offline   Reply With Quote
Old 01-13-2019, 07:15 AM   #3221
Kammal
Junior Member
Kammal began at the beginning.
 
Posts: 2
Karma: 10
Join Date: Jul 2014
Device: Kindle
Sufficient Velocity is giving me this error since yesterday, no idea why, I can access it just fine in my browser.

Spacebattles is working fine.

Quote:
HTTP Error 503: Service Temporarily Unavailable

Last edited by Kammal; 01-13-2019 at 09:14 PM.
Kammal is offline   Reply With Quote
Old 01-13-2019, 12:24 PM   #3222
JimmXinu
Plugin Developer
JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.
 
JimmXinu's Avatar
 
Posts: 3,532
Karma: 589322
Join Date: Dec 2011
Location: Midwest USA
Device: Kindle Voyage
Quote:
Originally Posted by Kammal View Post
Sufficient Velocity is giving me this error since yesterday, no idea why, I can access it just fine in my browser.

Spacebattles is working fine.
I'm not seeing this error.

But it says it right in the error: "HTTP Error 503: Service Temporarily Unavailable"

It could be that the site is simply having problems, just as the error says or that you (or your network) is behind a proxy of some kind that is having problems.

Regardless, there's nothing I can do to help you with server errors.
JimmXinu is offline   Reply With Quote
Old 01-13-2019, 09:14 PM   #3223
Kammal
Junior Member
Kammal began at the beginning.
 
Posts: 2
Karma: 10
Join Date: Jul 2014
Device: Kindle
Quote:
Originally Posted by JimmXinu View Post
I'm not seeing this error.

But it says it right in the error: "HTTP Error 503: Service Temporarily Unavailable"

It could be that the site is simply having problems, just as the error says or that you (or your network) is behind a proxy of some kind that is having problems.

Regardless, there's nothing I can do to help you with server errors.
I found out a way around it, if I use a VPN from my country I still get the same error, but if I use a VPN from Canada, it works just fine.
Really weird, on my browser it works just fine without a VPN.
Kammal is offline   Reply With Quote
Old 01-15-2019, 03:10 AM   #3224
fanfictioneer
Junior Member
fanfictioneer began at the beginning.
 
Posts: 8
Karma: 10
Join Date: Oct 2018
Device: Kindle
I am trying to download this link: https://forums.sufficientvelocity.co...-litrpg.51514/

However, it seems it requires you to be logged in to view the thread at all, and FFF states that the story does not exist (I do have a section for forums.sufficientvelocity.com in my personal.ini). Is the adapter not logging in before checking whether a story exists at all?
fanfictioneer is offline   Reply With Quote
Old 01-15-2019, 01:00 PM   #3225
JimmXinu
Plugin Developer
JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.JimmXinu ought to be getting tired of karma fortunes by now.
 
JimmXinu's Avatar
 
Posts: 3,532
Karma: 589322
Join Date: Dec 2011
Location: Midwest USA
Device: Kindle Voyage
New Test Version Posted

2019-01-15
- Add always_login setting to base_xenforo for SV login-required story with 404 result.


I've never seen a user/pass required story on SV before.

We already have code for detecting stories that need login on QQ by the 403 - forbidden return code. But this is coming up 404 - file not found.

Because we still need to be able to detect and handle not-found, I've added always_login as an option for base_xenforo. But you should only set it for stories that need it.

Example:
Code:
[https://forums.sufficientvelocity.com/threads/small-medium-litrpg.51514/]
always_login:true

[forums.sufficientvelocity.com]
username:UserName
password:XXXXXXXXXX
JimmXinu is offline   Reply With Quote
Reply

Thread Tools Search this Thread
Search this Thread:

Advanced Search

Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
[GUI Plugin] Open With kiwidude Plugins 332 08-31-2018 09:54 AM
[GUI Plugin] KindleUnpack - The Plugin DiapDealer Plugins 412 04-24-2018 07:07 PM
[GUI Plugin] Marvin XD Philantrop Plugins 126 01-29-2017 01:48 PM
[GUI Plugin] KiNotes -axel- Plugins 0 07-14-2013 07:39 PM
[GUI Plugin] Plugin Updater **Deprecated** kiwidude Plugins 159 06-19-2011 01:27 PM


All times are GMT -4. The time now is 09:14 PM.


MobileRead.com is a privately owned, operated and funded community.