[GUI Plugin] FanFictionDownLoader

[JimmXinu]

@Jade Aislin - Yes, it's the {#character1:subitems(-1,0)} composite column. Using that, I'm able to duplicate the problem.

A composite of {#characters}, {#genre} still worked, however, so it has to do with the subitems function call.

So this is either a problem of "you're not using it right", or calibre not interpreting the subitems template function in the composite column right away.

If you believe it's the latter, you'll need to take it up with the calibre devs. I can help demonstrate the problem, if needed.

[JimmXinu]

Or you might take it up with kiwidude in the GC forum--he may have more insight.

[JimmXinu]

@Jade Aislin - I think I have a work around for the GC issue with complex custom columns.

Please try the attached version and see if works better for you.

[chaley]

Originally Posted by JimmXinu

@Jade Aislin - Yes, it's the {#character1:subitems(-1,0)} composite column. Using that, I'm able to duplicate the problem.

A composite of {#characters}, {#genre} still worked, however, so it has to do with the subitems function call.

So this is either a problem of "you're not using it right", or calibre not interpreting the subitems template function in the composite column right away.

If you believe it's the latter, you'll need to take it up with the calibre devs. I can help demonstrate the problem, if needed.

I am going to jump in here after reading something on the generate cover thread. I have no idea what this plugin does, but as I wrote the custom column and template code in calibre, perhaps I can help.

If I understand the last couple of posts correctly, in some cases a composite column appears to have delayed evaluation. I interpret this as saying that at point A the value is wrong, where later it becomes correct.

This behavior can happen if point A is when new books are added but before calibre's data cache is refreshed, and point B is after the cache is refreshed. The cache for a changed book tends to be refreshed when the calibre GUI displays the line for that book. It can also be explicitly refreshed.

Because of this characteristic, if a plugin does the sequence:

1) add a book.
2) add metadata to the book.
3) mi = get_metadata()
4) do something with the mi structure

then the composite columns in the mi will not have the correct values. Why? It gets the values from the data cache, and the data cache is not yet up-to-date. The same thing will be true if 1) and 2) are instead "change metadata".

If the plugin is doing the above or something like it, then between steps 2 and 3 (I think) it should do "db.refresh_ids([list of book ids])", where "db" is the database object that you should already have. The list of IDs should contain all the ids that you will pass to get_metadata.

[chaley]

Originally Posted by JimmXinu

@Jade Aislin - I think I have a work around for the GC issue with complex custom columns.

Please try the attached version and see if works better for you.

Looking at this code, I suspect that your change was adding:

            # force a refresh if generating cover so complex composite
            # custom columns are current and correct
            self.gui.library_view.model().refresh_ids([book_id])

This will work, but it might have unintended consequences. I suspect that the view will repaint, and a bunch of internal signals will be shot off including searching. I think it would be better to call db.refresh_ids([book_id]). This will refresh the cache without involving the gui. You can (and, I see, do) refresh the gui later, just before you are finished.

Note that the stale data problem can affect other fields as well. Get_metadata reads most of the standard fields from the cache, such as series, publisher, dates. After metadata updates, these will be stale until a db.refresh_ids is done.

[JimmXinu]

Originally Posted by chaley

I am going to jump in here after reading something on the generate cover thread. I have no idea what this plugin does, but as I wrote the custom column and template code in calibre, perhaps I can help.

Great, thanks!

Originally Posted by chaley

...
If the plugin is doing the above or something like it, then between steps 2 and 3 (I think) it should do "db.refresh_ids([list of book ids])", where "db" is the database object that you should already have. The list of IDs should contain all the ids that you will pass to get_metadata.

Basically, that is what's happening. And I arrived at the same solution after some experimentation:

self.gui.library_view.model().refresh_ids([book_id])

If that's the official solution, I can certainly live with it.

But it would have been a lot easier to find if the behavior were consistent, either one way or the other.

A simple custom column and a composite column with a simple template ("{#characters}, {#genre}"): both work without the refresh.

But a composite column with a template function ("{#genre:uppercase()}") doesn't work without the refresh.

[JimmXinu]

Originally Posted by chaley

This will work, but it might have unintended consequences. I suspect that the view will repaint, and a bunch of internal signals will be shot off including searching. I think it would be better to call db.refresh_ids([book_id]). This will refresh the cache without involving the gui. You can (and, I see, do) refresh the gui later, just before you are finished.

Good point--I just copied it from my 'all done' code and didn't realize there was a difference. Thanks for catching that.

[chaley]

Originally Posted by JimmXinu

If that's the official solution, I can certainly live with it.

Yes, it is the solution.

But it would have been a lot easier to find if the behavior were consistent, either one way or the other.

A simple custom column and a composite column with a simple template ("{#characters}, {#genre}"): both work without the refresh.

But a composite column with a template function ("{#genre:uppercase()}") doesn't work without the refresh.

You raise an interesting point. I might be able to improve the performance of get_metadata by fetching the custom column values from the cache instead of from the db. There are some possible gotchas, but it is certainly worth checking.

And using the cache would make the behavior consistent.

[JimmXinu]

Version 1.5.43 - 08 Jul 2012
New Site: dark-solace.org
New Site: pretendercentre.com
New Site: themasque.net
Fixes for a couple obscure problems with calling Generate Cover.

[Zauberlehrling!]

I can't say thank you enough for everyone who works on that plugin and makes it work.

I came across an interesting thing. There is one story on ksarchive.com what can't be downloaded. Maybe someone wants to try and will see the error as I do.

Link is: http://www.ksarchive.com/viewstory.p...4065&warning=1

The message I get can be seen in attached picture.

[Firedancer885]

Originally Posted by Zauberlehrling!

I came across an interesting thing. There is one story on ksarchive.com what can't be downloaded. Maybe someone wants to try and will see the error as I do.

Link is: http://www.ksarchive.com/viewstory.p...4065&warning=1

I'm using the latest plugin, but have the same problem with this. I can read the story on the website, but can't download it. Something to do with the rating perhaps?

I have an account, but even after I put my name and password into personal.ini, I got the same error.

[Jade Aislin]

That seems to have fixed the problem with the composite column.

Thanks.

[JimmXinu]

Originally Posted by Zauberlehrling!

I can't say thank you enough for everyone who works on that plugin and makes it work.

I came across an interesting thing. There is one story on ksarchive.com what can't be downloaded. Maybe someone wants to try and will see the error as I do.

Link is: http://www.ksarchive.com/viewstory.p...4065&warning=1

The message I get can be seen in attached picture.

For no apparent reason, the author included an extra--and empty--category link in the story description.

<a href="browse.php?type=categories&amp;catid=1"></a>

Attached is a fixed version you can use, and it will be in the next version, but I'm not going to do a new version today for just one story.

Attached version updated for tweak to 'get urls from page' feature.

Attached version with another ksa fix for poor summary html.

Attached version with fix for wraithbait story with blockquote tags.

Attached version with fix for August dates on dramione.org.

[Zauberlehrling!]

Great, it works perfectly.

[Zauberlehrling!]

Originally Posted by Zauberlehrling!

Great, it works perfectly.

And here comes something new. Again a story what can't be read.
I don't know if that is caused by the same thing.

http://www.ksarchive.com/viewstory.php?sid=4124

See again attachment.