calibre vs proxy vs no-proxy

[waces]

Hello forum,

I checked the previous posts so far but didn't found any solution so far.
My environment have the following. When I'm in the office network I use a proxy deployed to the clients via pac file. When I'm off the office network the connection (defined in the pac file) change to direct.
But, when I'm off the network calibre cannot connect to the internet (Even there is _no_ proxy settings).
As not surprising, but on the office network calibre also not working.
When I try the command : calibre-debug -c "from calibre import get_proxies; print get_proxies()" the response is {} which means no proxy connection. Even there is.
Is there any special settings I missed to configure the proxy usage (it's probably not a weird question all the companies and most of the free wifis have proxy configured)

[kovidgoyal]

PAC proxies are not supported by calibre. PAC proxies are only intended for browsers, they are javascript files.

You want to set a proxy in calibre, use the http_proxy environment variable or your system proxy settings (non-PAC proxy settings).

[waces]

Originally Posted by kovidgoyal

PAC proxies are not supported by calibre. PAC proxies are only intended for browsers, they are javascript files.

You want to set a proxy in calibre, use the http_proxy environment variable or your system proxy settings (non-PAC proxy settings).

Thanks. That's not good (for me€. I assume calibre simply use the proxy setting from os settings (aka ie settings€. If the pacfile not kick in (like direct connection) see mo reason why calibre should not work.
By the way, the system proxy settings not working (it's direct connection in case of the proxy server not available)

Will search for the envir.variab. you mentioned

[waces]

up the thread.
i checked in various scenarios and even the pac file redirects the network traffic to "DIRECT" calibre cannot access the internet. My guess Calibre use the system wide settings, so see no reason why it's not working, but any ideas are more than welcome

[waces]

up the thread. it's getting ridiculous to not having answer to this very simple question.

[theducks]

Either Kovids answer covered it (so no one else saw a need) or it is
NOT SIMPLE, an no one has a answer

[waces]

Originally Posted by theducks

Either Kovids answer covered it (so no one else saw a need) or it is
NOT SIMPLE, an no one has a answer

corrects me if i'm wrong (frankly not dig myself deep into the metadata query, so feel free to correct me) but the data query is a simply http get via port 80 so see no reason why it cannot work. Pac file only route the traffic to a proxy (if available) or use "direct" connection (with no proxy) if the proxy server not available. So, in this case as the client not able to connect to the proxy server on off-corporate network the traffic will simply bypass the pac file settings. As of it I see no reason to fail. But, as I mentioned, I more than happy if someone directs me to the right direction and probably make me able to find a quick fix for the issue

[davidfor]

Originally Posted by waces

corrects me if i'm wrong (frankly not dig myself deep into the metadata query, so feel free to correct me) but the data query is a simply http get via port 80 so see no reason why it cannot work. Pac file only route the traffic to a proxy (if available) or use "direct" connection (with no proxy) if the proxy server not available. So, in this case as the client not able to connect to the proxy server on off-corporate network the traffic will simply bypass the pac file settings. As of it I see no reason to fail. But, as I mentioned, I more than happy if someone directs me to the right direction and probably make me able to find a quick fix for the issue

Are you sure about that for the PAC file usage? I would have expected that if the PAC file was not available, or the proxy it specified was not available, you wouldn't be able to access the URL using the client. If you were able to, it could be considered a security issue in a lot of corporate environments. I'm not sure on this as I've never set one up.

To use a PAC file, the client must:

- Have a way to configure the URL to retrieve the PAC file.
- Retrieve the PAC file.
- Each time the client wants to access a URL, it has to call the Javascript function in PAC file passing the URL and host name.
- Then interpret the results of the call to decide whether to access the URL directly or via the returned proxy.
- Handle when the proxy server is not available.

None of that is hard, but it takes time develop, test and maintain. And the client has to be able to execute Javascript. The latter makes PAC file support for most clients undesirable for most projects. With that, I can understand why Kovid doesn't want to support them.

[waces]

Yes, I'm pretty sure as I designed the proxy infrastrucuture
The way PAC file is in use like it's download the latest pac file from the predefined (Autoconfig) URL _if_ available. If not will use the latest cached version. So, in my example if the Proxy server holding the PAC file unavailable (eg the client is off-office network) the very last line of the PAC file take place which is:
// DEFAULT RULE: All other traffic, use below proxies, in fail-over order.
return "DIRECT";
So, the traffic will flow without any proxy server just like in a no-PAC file environment. (Btw if the machine is on the office network and can access the proxy server calibre still not able to connect to the internet, so it's _NOT_ a pac file access related issue)
Also using pac file is not a new thing, those kind of proxy config exists for many years so had plenty of time to develop this quite simple thing into calibre

Originally Posted by davidfor

Are you sure about that for the PAC file usage? I would have expected that if the PAC file was not available, or the proxy it specified was not available, you wouldn't be able to access the URL using the client. If you were able to, it could be considered a security issue in a lot of corporate environments. I'm not sure on this as I've never set one up.

To use a PAC file, the client must:

- Have a way to configure the URL to retrieve the PAC file.
- Retrieve the PAC file.
- Each time the client wants to access a URL, it has to call the Javascript function in PAC file passing the URL and host name.
- Then interpret the results of the call to decide whether to access the URL directly or via the returned proxy.
- Handle when the proxy server is not available.

None of that is hard, but it takes time develop, test and maintain. And the client has to be able to execute Javascript. The latter makes PAC file support for most clients undesirable for most projects. With that, I can understand why Kovid doesn't want to support them.

[davidfor]

Originally Posted by waces

Yes, I'm pretty sure as I designed the proxy infrastrucuture
The way PAC file is in use like it's download the latest pac file from the predefined (Autoconfig) URL _if_ available. If not will use the latest cached version.

Which of course requires that the PAC file had been retrieved at some time. And that would mean I would need to configure calibre at work and start it there to retrieve the PAC file. An unusual situation, but it would catch someone following a set of instructions.

So, in my example if the Proxy server holding the PAC file unavailable (eg the client is off-office network) the very last line of the PAC file take place which is:
// DEFAULT RULE: All other traffic, use below proxies, in fail-over order.
return "DIRECT";
So, the traffic will flow without any proxy server just like in a no-PAC file environment. (Btw if the machine is on the office network and can access the proxy server calibre still not able to connect to the internet, so it's _NOT_ a pac file access related issue)

That doesn't agree with the more complicated samples I found. They all have conditional checks of the URL against a pattern with a return for that pattern. And not all included a non-proxied option for when it fell through. Plus, what you say sounds like the PAC file execution is checking the proxy availability, not returning the calculated proxy to the client to try.

One of the examples is http://findproxyforurl.com/example-pac-file/. In that example, it uses DIRECT for internal addresses and different proxies for all other sites. And the default when there is no earlier match is to use one of two proxy servers. To me that says that for everything except local and intranet addresses, the client uses a proxy. There is nothing in it that suggests a direct connection can be used for a general internet site. If this example is invalid, then based on the google search results, I would think that a lot of sites are using bad PAC files.

The example on the Wikipedia page ends with:

return "PROXY proxy.example.com:8080; DIRECT";

I can see that as "use proxy.example.com:8080 if it responds, otherwise go direct". But, I expected that as meaning the client is to do the test. But, what you say suggests that the actual result returned to the client is the proxy server if it is available, or "DIRECT" if it is not.

Also using pac file is not a new thing, those kind of proxy config exists for many years so had plenty of time to develop this quite simple thing into calibre

Yes, I know it is not new. And there has been plenty of time for it to be added to calibre. But, that's the point. Someone has to take the time to code it and add it. That means they are not doing something else. And as this is only the second mention of PAC files I can find in this forum, it isn't a highly desired feature. There are a lot of other things to do that people are asking for, so they will be done first.

But, I'm sure that if someone wants to code this, then Kovid will accept a patch.

[waces]

Agree, but if it's just a proxy response time issue (which should occure when the client wait for the server defined in the PAC file to respond first and then move to Direct) it should work in the environment when the pac file response is prompt. But it's not the case. Just tested and even the server defined in the pac file available calibre cannot connect through it. And - as I checked the log file on the proxy server itself - unfortunately there was no request initiated by calibre (and alos checked the target hosts defined in the meta query available via the proxy, so there is no url blocking, and also no blocking on the standard http port required for query the meta data)

Yes, you're right but when the pac file reports back direct it means 2 thing. 1 the defined proxy server is not available and 2 the connection will goes through without defined proxy settings. But it doesn't mean there is no data flow in case of no-proxy situations. (If you want to deny all the traffic in case the user cannot access the proxy you need to remove the "direct" part from the pac file. In that case all the communication will be blocked if the proxy is unavailable. But it's a very rare situation.)

Originally Posted by davidfor

Which of course requires that the PAC file had been retrieved at some time. And that would mean I would need to configure calibre at work and start it there to retrieve the PAC file. An unusual situation, but it would catch someone following a set of instructions.

That doesn't agree with the more complicated samples I found. They all have conditional checks of the URL against a pattern with a return for that pattern. And not all included a non-proxied option for when it fell through. Plus, what you say sounds like the PAC file execution is checking the proxy availability, not returning the calculated proxy to the client to try.

One of the examples is http://findproxyforurl.com/example-pac-file/. In that example, it uses DIRECT for internal addresses and different proxies for all other sites. And the default when there is no earlier match is to use one of two proxy servers. To me that says that for everything except local and intranet addresses, the client uses a proxy. There is nothing in it that suggests a direct connection can be used for a general internet site. If this example is invalid, then based on the google search results, I would think that a lot of sites are using bad PAC files.

The example on the Wikipedia page ends with:

return "PROXY proxy.example.com:8080; DIRECT";

I can see that as "use proxy.example.com:8080 if it responds, otherwise go direct". But, I expected that as meaning the client is to do the test. But, what you say suggests that the actual result returned to the client is the proxy server if it is available, or "DIRECT" if it is not.



Yes, I know it is not new. And there has been plenty of time for it to be added to calibre. But, that's the point. Someone has to take the time to code it and add it. That means they are not doing something else. And as this is only the second mention of PAC files I can find in this forum, it isn't a highly desired feature. There are a lot of other things to do that people are asking for, so they will be done first.

But, I'm sure that if someone wants to code this, then Kovid will accept a patch.

[waces]

happy new year fellow calibre users. and up the thread

[BetterRed]

Moderator Notice
You have been given an answer on this issue by the author of calibre, Kovid Goyal. He is not interested in implementing and maintaining support for PAC proxies, and he has told you why.

There is nothing stopping you from downloading the source and writing the code to support PAC proxies for your own own use, if you lack the skills to write the code, find someone who has them and commission them to do the work.

Thread bumping to no good purpose is frowned at MR. So you should regard this post as a warning.

BR

[waces]

Sorry for the bumping will take care of it next time.
Unfortunately the answer from the developer was not a real answer about the design decision why calibre not support the common method using system API. Or if it's by design why there is no option to overwrite it.
Normally if an application need to connect the internet it use the system default variables (calibre not, as it can't use this for whatever reasons).
The question was more about the "why" and not the "how".

Thanks for pointing to the source code I created a temporary workaround using the source but I think it would be better to implement this in an official way.

Originally Posted by BetterRed

You have been given an answer on this issue by the author of calibre, Kovid Goyal. He is not interested in implementing and maintaining support for PAC proxies, and he has told you why.

There is nothing stopping you from downloading the source and writing the code to support PAC proxies for your own own use, if you lack the skills to write the code, find someone who has them and commission them to do the work.

Thread bumping to no good purpose is frowned at MR. So you should regard this post as a warning.

BR

[BetterRed]

Originally Posted by waces

... but I think it would be better to implement this in an official way.

Have you considered submitting a patch at github, AFAIK that is the 'official way'?

BR