[GUI Plugin] Search the Internet

[Wiggo]

FWIW

Originally Posted by Wiggo

However. As I don't have the nerve to do a step-by-step analysis at the moment, I'll just have to live with it.

Today is my first day of holiday.

Totally fresh FF portable folder, nothing changed, all possible Mozilla leftovers in ProgramData and AppData deleted, new portable as new standard browser.

Still the same error - I give up

[Comfy.n]

Originally Posted by Wiggo

FWIW

Today is my first day of holiday.

Same!

Originally Posted by Wiggo

Totally fresh FF portable folder, nothing changed, all possible Mozilla leftovers in ProgramData and AppData deleted, new portable as new standard browser.

Still the same error - I give up

probably too much work for a holiday, but a fresh Windows user profile could be the final frontier

[unicornula]

Originally Posted by kiwidude

The substitution of periods has been in for a few versions now. I am ok with taking that back out - and forcing users to do for instance: {author:re(\., )}

However the / in 10.1038/nrg1711 is absolutely going to get url encoded and not passed through as is, becoming: 10.1038%2Fnrg1711

Can you give me of an example website search url expression you are using with the DOI value above?

The period was working fine until this version. I use it dozens of times every week (or even in a day), so it's something I'd notice right away.

The slash is never a problem. Example:
With doi:10.3390/plants10040726 in the ids
URL in the plugin:
https://scholar.google.com/scholar?hl=en&as_sdt=1%2C5&q={identifiers:select(d oi)}&btnG=

Gets me:
https://scholar.google.com/scholar?h...10040726&btnG=
in my browser -- with the slash.

There's only one search where I put it in quotes, and there the slash is replaced with %2F, but that's fine. Example (same DOI):
https://lens.org/lens/search/scholar/list?q=ids.doi:%22{identifiers:select(doi)}%22&pre view=true

Gets me:
https://www.lens.org/lens/search/sch...2&preview=true
which works.

[kiwidude]

Release Notes
https://github.com/kiwidude68/calibr...ternet-v1.11.1

Thanks unicornula for the examples, this version should let you use the latest plugin version again. I tried both the variations you posted and they seemed to work fine.

[unicornula]

Originally Posted by kiwidude

Release Notes
https://github.com/kiwidude68/calibr...ternet-v1.11.1

Thanks unicornula for the examples, this version should let you use the latest plugin version again. I tried both the variations you posted and they seemed to work fine.

That works! Thanks.

[unicornula]

Spoke too soon! Now I have problems with the slash.

When I use:
https://www.altmetric.com/details/doi/{identifiers:select(doi)}

With the new version, I get %2F sent to the browser in the DOI in place of the slash, which doesn't work. I reverted to v1.10.5 and it sends the slash.

[kiwidude]

Originally Posted by unicornula

Spoke too soon! Now I have problems with the slash.

When I use:
https://www.altmetric.com/details/doi/{identifiers:select(doi)}

With the new version, I get %2F sent to the browser in the DOI in place of the slash, which doesn't work. I reverted to v1.10.5 and it sends the slash.

Thats why I asked for examples around the slash above - the ones you gave me work with the encoding to %2F. As for what I can do about it - not much - passing an unencoded slash is usually a terrible idea for an api call when part of the data. And the plugin now encodes every token in the url.

[chaley]

Originally Posted by kiwidude

As for what I can do about it - not much - passing an unencoded slash is usually a terrible idea for an api call when part of the data. And the plugin now encodes every token in the url.

You perhaps could parse the url then only encode the query string arguments, leaving the path alone. This would avoid the unexpected difference between

URL: http:/foo.com/a/b/c/d
result: http:/foo.com/a/b/c/d

and this, where the column #b contains the characters 'b/c':

URL: http:/foo.com/a/{#b}/d
result: http:/foo.com/a/b%2Fc/d

[kiwidude]

Hi @chaley - perhaps I am missing something but the second example is exactly the situation unicornula has where they dont want the token encoded vs the current plugin behaviour where all tokens are encoded? It is a strange edge case…

I think your idea about querystring args is interesting however, just the examples werent quite right. So if I parse the url as you suggest, and any querystring tokens (i.e. those following a questionmark) continue to get encoded, but any tokens before that do not.

I will have to review all the example searches that come in the plugin as a sample to see whether that might work - not home to check.

[chaley]

Originally Posted by kiwidude

Hi @chaley - perhaps I am missing something but the second example is exactly the situation unicornula has where they dont want the token encoded vs the current plugin behaviour where all tokens are encoded? It is a strange edge case…

I wasn't clear. I was trying to demonstrate why encoding templates in the path could lead to problems. If only query strings are encoded then the second case won't occur because the template in the path portion of the URL is not encoded.

I don't recall ever seeing encoding anywhere but the query string, but I certainly can't say that it doesn't/shouldn't happen.

I have considered adding a url_encode() template function so the user can make the choice. Unfortunately this would work only if the user can guarantee that no further encoding will take place. Double encoding a URL won't work at all. I hesitate to add a function that can lead to this sort of problem.

[kiwidude]

Originally Posted by chaley

I don't recall ever seeing encoding anywhere but the query string, but I certainly can't say that it doesn't/shouldn't happen.

Unfortunately when I went through the predefined shortcuts I found several examples where this is the case and it breaks the suggested fudge I was going to put in. One example is:

https://www.literature-map.com/{author}

So the only way I think this could be made to work is adding another option for every url to control whether the encoding takes place at all for it.

Can I be bothered adding this for one edge case of one website though - right now probably not especially since it is not a website I am ever going to use personally. Because to do it properly is a ton of work in terms of UI, config etc. The hackiest quick way i can think of would be to add a new option to the encoding dropdown that offers utf-8/latin-1 with some new options like utf-8-no-enc or similar. But I might regret that one day...

[kiwidude]

Release Notes
https://github.com/kiwidude68/calibr...ternet-v1.11.2

Some fixes for calibre 7, a big rationalisation of the default shortcuts culling dead sites and switching to https. This also has a complete hack in there to not encode any tokens that contain 'select:identifiers'.

[unicornula]

Originally Posted by kiwidude

Thats why I asked for examples around the slash above - the ones you gave me work with the encoding to %2F. As for what I can do about it - not much - passing an unencoded slash is usually a terrible idea for an api call when part of the data. And the plugin now encodes every token in the url.

It works fine up until the current version -- it had always been done terribly until now? That's why I didn't have an example -- it had always worked until it was broken with 1.11. Never had any problems with periods or slashes.

When the point of the plugin is to launch URLs and slashes are part of URLs, it seems odd to suddenly remove them on version 1.11, but I guess I'm just going to have to use 1.10.5 until it no longer works.

[kiwidude]

Originally Posted by unicornula

When the point of the plugin is to launch URLs and slashes are part of URLs, it seems odd to suddenly remove them on version 1.11, but I guess I'm just going to have to use 1.10.5 until it no longer works.

You posted after 1.11.2 released - have you tried that? I tried it with the three DOI examples you gave me.

In some circumstances the data has to be encoded, in others it does not or causes a problem for specific characters by doing so. The plugin before 1.11 had a flaw in it which meant only a small number of metadata fields were encoded - meaning custom columns could not be passed with it, hence the change made.

The other idea I had overnight on this is I likely rely on a calibre url encode function, which in theory I could write my own version of. Or turn %2F back to a / after calling it. However if that data is part of the querystring portion of a url then it maybe that breaks on some websites, would have to test it. A slash doesnt make sense to be part of a search term for anything other than an identifier (eg slashes would normally not be included in a title search comparison). But I have the same issues of supporting custom columns and other template expressions if I wanted to exclude passing it, because in this one case in the world of a DOI identifier the slash is required. It is a very unfortunate (ie poor in the web world) identifier standard they chose.

[kiwidude]

Release Notes
https://github.com/kiwidude68/calibr...ternet-v1.11.3

A workaround for a calibre bug which has been fixed but just in case people are working with the non-fixed calibre versions. Only would impact anyone attempting to use {id} or {has_cover} template attributes in their urls.