Any suggestions? Intermittant problems with Content Server/NGINX

haertig · 02-27-2018, 03:20 PM

I am having some problems with the Calibre content server. Sometimes it works, and other times it doesn't. Then it "fixes itself" after a time and starts working again.

I now believe my problem may be in nginx itself, which I use as a reverse proxy for the content server. Possibly due to low system resources?

When I start to have problems, the nginx error.log fills up with lots of messages like the ones illustrated below. There are lots more of these errors, I just showed a small example. The errors are for fetch attempts of different things from Calibre. The errors that I am currently trying to track down and repair are the "upstream server temporarily disabled while connecting to upstream" ones. I'm not sure exactly what this means, but I think it has something to do with nginx turning access to itself off temporarily, due to too many errors of some kind.

Is this some type of limited resource issue, possibly? I am running both nginx and the Calibre content server on a Raspberry Pi3. This is certainly not powerful hardware. I can't predict when Calibre will work for me or when it won't. That appears to be random. When it works, it works well. This Raspberry Pi is not running anything else besides nginx and Calibre content server (it runs an rsync in the middle of the night as well, but that only takes a second and is not happening when I'm experiencing problems).

Code:

2018/02/27 12:48:02 [error] 341#341: *2425 connect() failed (111: Connection refused) while connecting to upstream, client: 10.192.0.1, server: myserver.mydomain.net, request: "GET /calibre/get/thumb/303/calibre?sz=300x400 HTTP/2.0", upstream: "http://[::1]:8080/calibre/get/thumb/303/calibre?sz=300x400", host: "myserver.mydomain.net", referrer: "https://myserver.mydomain.net/calibre/"
2018/02/27 12:48:02 [warn] 341#341: *2425 upstream server temporarily disabled while connecting to upstream, client: 10.192.0.1, server: myserver.mydomain.net, request: "GET /calibre/get/thumb/303/calibre?sz=300x400 HTTP/2.0", upstream: "http://[::1]:8080/calibre/get/thumb/303/calibre?sz=300x400", host: "myserver.mydomain.net", referrer: "https://myserver.mydomain.net/calibre/"
2018/02/27 12:48:02 [error] 341#341: *2425 connect() failed (111: Connection refused) while connecting to upstream, client: 10.192.0.1, server: myserver.mydomain.net, request: "POST /calibre/interface-data/set-session-data HTTP/2.0", upstream: "http://[::1]:8080/calibre/interface-data/set-session-data", host: "myserver.mydomain.net", referrer: "https://myserver.mydomain.net/calibre/"
2018/02/27 12:48:02 [warn] 341#341: *2425 upstream server temporarily disabled while connecting to upstream, client: 10.192.0.1, server: myserver.mydomain.net, request: "POST /calibre/interface-data/set-session-data HTTP/2.0", upstream: "http://[::1]:8080/calibre/interface-data/set-session-data", host: "myserver.mydomain.net", referrer: "https://myserver.mydomain.net/calibre/"

Here is my nginx configuration (a lot of this nginx config, especially the ssl parts, were "borrowed" from a different nginx installation I run for NextCloud on a different server):

Code:

user www-data;
worker_processes 4;
error_log /var/log/nginx/error.log warn;
#pid /var/run/nginx.pid;

events {
	worker_connections 1024;
	multi_accept on;
	use epoll;
}

http {
	types_hash_max_size 4096;
	server_names_hash_bucket_size 64;
	include /etc/nginx/mime.types;
	limit_req_zone $binary_remote_addr zone=wp_ddos:20m rate=2r/m;
	default_type application/octet-stream;
	log_format main '$remote_addr - $remote_user [$time_local] "$request" '
	'$status $body_bytes_sent "$http_referer" '
	'"$http_user_agent" "$http_x_forwarded_for" '
	'"$host" sn="$server_name" '
	'rt=$request_time '
	'ua="$upstream_addr" us="$upstream_status" '
	'ut="$upstream_response_time" ul="$upstream_response_length" '
	'cs=$upstream_cache_status' ;
	access_log /var/log/nginx/access.log main;
	sendfile on;
	send_timeout 3600;
	tcp_nopush on;
	tcp_nodelay on;
	open_file_cache max=500 inactive=10m;
	open_file_cache_errors on;
	keepalive_timeout 65;
	reset_timedout_connection on;
	server_tokens off;
	resolver 10.192.0.1;
	resolver_timeout 10s;

	server {
		listen 80;
		return 301 https://$host$request_uri;
	}
	
	server {
		listen 443 ssl http2;
		server_name myserver.mydomain.net localhost 10.192.0.27 default_server;
		root /var/www;

		ssl_certificate /etc/letsencrypt/live/myserver.mydomain.net/fullchain.pem;
		ssl_certificate_key /etc/letsencrypt/live/myserver.mydomain.net/privkey.pem;
		ssl_client_certificate /etc/ssl/certs/nginx/ca.crt;
		ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
		ssl_ciphers 'ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-DSS-AES128-GCM-SHA256:kEDH+AESGCM:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA:ECDHE-ECDSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-DSS-AES128-SHA256:DHE-RSA-AES256-SHA256:DHE-DSS-AES256-SHA:DHE-RSA-AES256-SHA:!aNULL:!eNULL:!EXPORT:!DES:!RC4:!3DES:!MD5:!PSK:!AES128';
		ssl_ecdh_curve secp384r1;
		ssl_prefer_server_ciphers on;
		ssl_stapling off;
		ssl_stapling_verify off;
		ssl_session_timeout 24h;
		ssl_session_cache shared:SSL:50m;
		ssl_session_tickets off;
		ssl_verify_client optional;

		location /calibre {
			if ($ssl_client_verify != SUCCESS) {
				return 403;
			}
			proxy_buffering off;
			proxy_set_header X-Forwarded-For $remote_addr;
			#proxy_pass http://10.192.0.2:8080$request_uri;
			proxy_pass http://localhost:8080$request_uri;
		}
	}

}

gbm · 02-27-2018, 04:18 PM

Quote:

Originally Posted by haertig

I am having some problems with the Calibre content server. Sometimes it works, and other times it doesn't. Then it "fixes itself" after a time and starts working again.

I now believe my problem may be in nginx itself, which I use as a reverse proxy for the content server. Possibly due to low system resources?

When I start to have problems, the nginx error.log fills up with lots of messages like the ones illustrated below. There are lots more of these errors, I just showed a small example. The errors are for fetch attempts of different things from Calibre. The errors that I am currently trying to track down and repair are the "upstream server temporarily disabled while connecting to upstream" ones. I'm not sure exactly what this means, but I think it has something to do with nginx turning access to itself off temporarily, due to too many errors of some kind.

Is this some type of limited resource issue, possibly? I am running both nginx and the Calibre content server on a Raspberry Pi3. This is certainly not powerful hardware. I can't predict when Calibre will work for me or when it won't. That appears to be random. When it works, it works well. This Raspberry Pi is not running anything else besides nginx and Calibre content server (it runs an rsync in the middle of the night as well, but that only takes a second and is not happening when I'm experiencing problems).

Spoiler:

Code:

2018/02/27 12:48:02 [error] 341#341: *2425 connect() failed (111: Connection refused) while connecting to upstream, client: 10.192.0.1, server: myserver.mydomain.net, request: "GET /calibre/get/thumb/303/calibre?sz=300x400 HTTP/2.0", upstream: "http://[::1]:8080/calibre/get/thumb/303/calibre?sz=300x400", host: "myserver.mydomain.net", referrer: "https://myserver.mydomain.net/calibre/"
2018/02/27 12:48:02 [warn] 341#341: *2425 upstream server temporarily disabled while connecting to upstream, client: 10.192.0.1, server: myserver.mydomain.net, request: "GET /calibre/get/thumb/303/calibre?sz=300x400 HTTP/2.0", upstream: "http://[::1]:8080/calibre/get/thumb/303/calibre?sz=300x400", host: "myserver.mydomain.net", referrer: "https://myserver.mydomain.net/calibre/"
2018/02/27 12:48:02 [error] 341#341: *2425 connect() failed (111: Connection refused) while connecting to upstream, client: 10.192.0.1, server: myserver.mydomain.net, request: "POST /calibre/interface-data/set-session-data HTTP/2.0", upstream: "http://[::1]:8080/calibre/interface-data/set-session-data", host: "myserver.mydomain.net", referrer: "https://myserver.mydomain.net/calibre/"
2018/02/27 12:48:02 [warn] 341#341: *2425 upstream server temporarily disabled while connecting to upstream, client: 10.192.0.1, server: myserver.mydomain.net, request: "POST /calibre/interface-data/set-session-data HTTP/2.0", upstream: "http://[::1]:8080/calibre/interface-data/set-session-data", host: "myserver.mydomain.net", referrer: "https://myserver.mydomain.net/calibre/"

Here is my nginx configuration (a lot of this nginx config, especially the ssl parts, were "borrowed" from a different nginx installation I run for NextCloud on a different server):

Code:

user www-data;
worker_processes 4;
error_log /var/log/nginx/error.log warn;
#pid /var/run/nginx.pid;

events {
	worker_connections 1024;
	multi_accept on;
	use epoll;
}

http {
	types_hash_max_size 4096;
	server_names_hash_bucket_size 64;
	include /etc/nginx/mime.types;
	limit_req_zone $binary_remote_addr zone=wp_ddos:20m rate=2r/m;
	default_type application/octet-stream;
	log_format main '$remote_addr - $remote_user [$time_local] "$request" '
	'$status $body_bytes_sent "$http_referer" '
	'"$http_user_agent" "$http_x_forwarded_for" '
	'"$host" sn="$server_name" '
	'rt=$request_time '
	'ua="$upstream_addr" us="$upstream_status" '
	'ut="$upstream_response_time" ul="$upstream_response_length" '
	'cs=$upstream_cache_status' ;
	access_log /var/log/nginx/access.log main;
	sendfile on;
	send_timeout 3600;
	tcp_nopush on;
	tcp_nodelay on;
	open_file_cache max=500 inactive=10m;
	open_file_cache_errors on;
	keepalive_timeout 65;
	reset_timedout_connection on;
	server_tokens off;
	resolver 10.192.0.1;
	resolver_timeout 10s;

	server {
		listen 80;
		return 301 https://$host$request_uri;
	}
	
	server {
		listen 443 ssl http2;
		server_name myserver.mydomain.net localhost 10.192.0.27 default_server;
		root /var/www;

		ssl_certificate /etc/letsencrypt/live/myserver.mydomain.net/fullchain.pem;
		ssl_certificate_key /etc/letsencrypt/live/myserver.mydomain.net/privkey.pem;
		ssl_client_certificate /etc/ssl/certs/nginx/ca.crt;
		ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
		ssl_ciphers 'ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-DSS-AES128-GCM-SHA256:kEDH+AESGCM:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA:ECDHE-ECDSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-DSS-AES128-SHA256:DHE-RSA-AES256-SHA256:DHE-DSS-AES256-SHA:DHE-RSA-AES256-SHA:!aNULL:!eNULL:!EXPORT:!DES:!RC4:!3DES:!MD5:!PSK:!AES128';
		ssl_ecdh_curve secp384r1;
		ssl_prefer_server_ciphers on;
		ssl_stapling off;
		ssl_stapling_verify off;
		ssl_session_timeout 24h;
		ssl_session_cache shared:SSL:50m;
		ssl_session_tickets off;
		ssl_verify_client optional;

		location /calibre {
			if ($ssl_client_verify != SUCCESS) {
				return 403;
			}
			proxy_buffering off;
			proxy_set_header X-Forwarded-For $remote_addr;
			#proxy_pass http://10.192.0.2:8080$request_uri;
			proxy_pass http://localhost:8080$request_uri;
		}
	}

}

Have you looked at and tried this:
https://manual.calibre-ebook.com/ser...g-a-url-prefix

Quote:

With nginx, the required configuration is:

Code:

proxy_set_header X-Forwarded-For $remote_addr;
location /calibre/ {
    proxy_buffering off;
    proxy_pass http://127.0.0.1:8080$request_uri;
}
location /calibre {
    # we need a trailing slash for the Application Cache to work
    rewrite /calibre /calibre/ permanent;
}

bernie

haertig · 02-27-2018, 06:47 PM

Thanks. I did see that, and made use of it when I first set up the nginx config. I do have the functionality of the first "location" in my config. But I don't have the second "location" (the "rewrite") however.

That's because I never access the content server as only:
"https://myserver.mydomainnet/calibre"

I always access it as:
"https://myserver.mydomain.net/calibre/#panel=book_list&sort=author_sort.asc,series.asc"

So the rewrite should never be triggered, thus it is unnecessary, as best I understand.

kovidgoyal · 02-27-2018, 10:24 PM

IIRC nginx does that when it thinks the upstream server is sending an invalid response. See the documentation for max_fails and fail_timeout in the nginx docx. Although why it is doing that for you, I haven't a clue.

haertig · 02-28-2018, 12:50 AM

I think I am just going to have to write off my intermittent problems with the content server as being a manifestation of running it on on a Raspberry Pi (ARM), under Arch Linux ARM, using a user-submitted package for installation.

The content server runs perfectly, even when fronted by NGINX (running on the Pi), when the content server is running on an x86 platform.

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
Content Server Problems	quoddle	Devices	18	04-05-2013 11:04 AM
Content server problems	Mythlandia	Library Management	0	06-14-2012 04:18 PM
Content Server problems	jesscat	Calibre	26	05-24-2011 05:47 PM
Problems with server of content	marcostt	Calibre	2	08-25-2009 09:38 AM
Content Server problems in 102	RoninTech	Calibre	7	11-06-2008 05:21 PM

02-27-2018, 06:47 PM	#3
haertig Wizard Posts: 1,895 Karma: 31522252 Join Date: Sep 2017 Device: PW3, Fire HD8 Gen7, Moto G7, Sansa Clip v2, Ruizu X26	Thanks. I did see that, and made use of it when I first set up the nginx config. I do have the functionality of the first "location" in my config. But I don't have the second "location" (the "rewrite") however. That's because I never access the content server as only: "https://myserver.mydomainnet/calibre" I always access it as: "https://myserver.mydomain.net/calibre/#panel=book_list&sort=author_sort.asc,series.asc" So the rewrite should never be triggered, thus it is unnecessary, as best I understand.

02-27-2018, 10:24 PM	#4
kovidgoyal creator of calibre Posts: 45,331 Karma: 27182818 Join Date: Oct 2006 Location: Mumbai, India Device: Various	IIRC nginx does that when it thinks the upstream server is sending an invalid response. See the documentation for max_fails and fail_timeout in the nginx docx. Although why it is doing that for you, I haven't a clue.

02-28-2018, 12:50 AM	#5
haertig Wizard Posts: 1,895 Karma: 31522252 Join Date: Sep 2017 Device: PW3, Fire HD8 Gen7, Moto G7, Sansa Clip v2, Ruizu X26	I think I am just going to have to write off my intermittent problems with the content server as being a manifestation of running it on on a Raspberry Pi (ARM), under Arch Linux ARM, using a user-submitted package for installation. The content server runs perfectly, even when fronted by NGINX (running on the Pi), when the content server is running on an x86 platform.

Advert

Advert