COPS github :
https://github.com/seblucas/cops
It could be that your database is just to big for either the used server hardware (CPU and Memory and HDD speed) or the software (PHP, SQlite3, COPS, etc).
Though I have tested with pretty big databases, never with one this
BIG
To test if its the database you can try one or both of these tests:
A - just copy the files
metadata.db and
metadata_db_prefs_backup.json out of your ebooks directory and into a new directory ebooktest.
Then point COPS to this directory:
$config['calibre_directory'] = './ebooktest/'; [Did we forget the ; in the previous posts?]
Then try to access it wit a browser (preferably FIrefox as I know it just works with COPS).
B - make a small new calibre library with just a few book and ftp it to a new directory ebooktest on your server. Further as A.
Just to be 100% sure (I know for 99.9% sure you did it right, but it would be stupid not to check) did you change the preferences with
chmod -R 775 ebooks ? And do this for the new directories as well.