I've got COPS working with Synology and without any special adaptions. The key probably lies within the fact that I still used an older firmware version of DSM. My Synology was running DSM4.0 and this morning I installed the latest DSM 4.1-2647
As you can see in the release notes
the php module of the Synology box was upgraded in DSM 4.1-2636 (see item 13).
This might explain why you got it working with X-sendfile and I could not get it to work.
1) Install DSM 4.1-2636 or higher
2) Use X-Sendfile
3) Add external path to Open_basedir
4) Add URL_Rewrite = 1
5) Add .htaccess for url rewriting
When I find the time I will update the Wiki. PS: Fancybox issues are independent from this. Is it somehow possible to resize larger cover images to a smaller size in order to have them fit the screen on the reader in stead of showing half of it?