Extra-large Database sizes?

vibbix · 04-25-2012, 09:05 PM

My library is entirely in .MOBI, and consists of over 8000 books, with Metadata reaching 20MB... It takes about 5 to 10 minutes for Calibre to boot, as it starts with approx. 100MB of ram on my comp, and requires 180MB+ to start working. The large database size also makes it hard to use Calibre2OPDS, which I want to host on my NAS, giving me a "Out of memory" error in the command prompt(it still allows me to go into C2O, but trying to generate the library and it refuses). The library is already stored on the NAS, and even without the ethernet/USB2.0 bottleneck, it takes several minutes to boot up Calibre. Is there anyway to manually create (Via terminal/command prompt?) a Calibre2OPDS HTML library that can actually store my database?

theducks · 04-25-2012, 10:07 PM

Moderator Notice
Moved to (OPDS) Catalog area

While Calibre has not published a definitive: Minimum Equipment required to run Spec, it sure sounds like you are trying to run on a sub-standard system.
A lowly P4 with 2G RAM opens a 4000+ book Library in about 20 seconds.

PeterT · 04-25-2012, 11:25 PM

When you say "The library is already stored on the NAS" what type of NAS is it?

How are you connecterd to the NAS?

How are the drives attached to the NAS?

What happens if you put everything on a local hard drive?

itimpi · 04-26-2012, 01:30 AM

What is the spec of your machine, particularly in terms of RAM? I use calibre2opds with my library on a NAS and about 16k books with an average of about 4 formats per book. My metadata.db file is over 70MB. Calibre2Opds tends to peak at around 350MB RAM usage when handling this.

In practise I would have thought that 512MB RAM was a minimum for both Calibre and Calibre2Opds, and 1GB was more realistic. This is very low for most modern machines which tend to have considerably more RAM.

vibbix · 04-26-2012, 03:09 PM

6GB of ram, 2x 640Gb HDD's at 7200RPM, Core i5 2410m, GT555m graphics(Dell XPS 17)... the NAS is probobally the what is slowing it down. Its a WD 1TB hard drive(usb 2.0), attached to my router(Netgear n600)... I can get 20mb while downloading through uSB, closer to 4000k while in NAS... I want to store the data on the NAS so I can access it outside the LAN...

itimpi · 04-26-2012, 03:15 PM

It does not sound to me as if your machine spec is a problem. I can understand why the NAS might slow down Calibre2Opds, but not why it would make it fail to run. I use Calibre2Opds against my NAS with a much bigger library. In my case Calibre2Opds can take a few hours to complete the run, but I do not have the problem you report of it failing to run.

In terms of speeding Calibre up, you might want to look for the Calibre-Portable.bat file that is held in the resources sub-folder of the Calibre install location. It shows you how you can control where all the Calibre files are located. I use it to keep the calibre metadata.db file on my local hard disk with all the Book files on the NAS. I then periodically copy the metadata.db file into the NAS Calibre library location.

vibbix · 04-26-2012, 04:58 PM

It allows opens the client, it gives me a SQL out of ram exception.... It wont let me install Calibre2OPDS, so i have to manually run the batch script to let it run... I will try the database move....

PeterT · 04-26-2012, 07:41 PM

I hate to say it but expecting reasonable performance through a USB 2.0 attached drive to a consumer router is a bit much. Routers are limited in both CPU and memory performance.

From a review of this particular router they show the following speeds for reads / writes

Code:

FAT32 Write (MBytes/s)	12.5	
FAT32 Read (MBytes/s)	13.3	
NTFS Write (MBytes/s)	4.3	
NTFS Read (MBytes/s)	12.0

shows that NTFS writes are relatively slow.

itimpi · 04-27-2012, 09:44 AM

Quote:

Originally Posted by vibbix

It wont let me install Calibre2OPDS, so i have to manually run the batch script to let it run... I will try the database move....

If you are using 64-bit Java on Windows 64-bit then there is a known issue with the installer not running properly. 32-bit Java on 64-bit Windows is OK for the installer. However if you follow the instructions for doing the manual install from the ZIP file then the main Calibre2Opds program is happy with 64-bit Java on 64-bit Windows.

I have never seen the SQLite error that you mentioned so am not sure what could be causing that.

When you talk about running the batch script, exactly what do you mean? It was not clear if you meant running the rungui.cmd file to start Calibre2Opds in graphical mode, or the run.cmd file to start in batch (non-graphical) mode. If the latter then note that you have to have used the graphical mode successully at least once before the batch mode will run as it is the graphical mode that sets up the required configuration parameters.

04-25-2012, 09:05 PM	#1
vibbix Junior Member Posts: 3 Karma: 10 Join Date: Apr 2012 Device: Kindle Touch	Extra-large Database sizes? My library is entirely in .MOBI, and consists of over 8000 books, with Metadata reaching 20MB... It takes about 5 to 10 minutes for Calibre to boot, as it starts with approx. 100MB of ram on my comp, and requires 180MB+ to start working. The large database size also makes it hard to use Calibre2OPDS, which I want to host on my NAS, giving me a "Out of memory" error in the command prompt(it still allows me to go into C2O, but trying to generate the library and it refuses). The library is already stored on the NAS, and even without the ethernet/USB2.0 bottleneck, it takes several minutes to boot up Calibre. Is there anyway to manually create (Via terminal/command prompt?) a Calibre2OPDS HTML library that can actually store my database?

04-26-2012, 07:41 PM	#8
PeterT Grand Sorcerer Posts: 12,143 Karma: 73448616 Join Date: Nov 2007 Location: Toronto Device: Nexus 7, Clara, Touch, Tolino EPOS	I hate to say it but expecting reasonable performance through a USB 2.0 attached drive to a consumer router is a bit much. Routers are limited in both CPU and memory performance. From a review of this particular router they show the following speeds for reads / writes Code: FAT32 Write (MBytes/s) 12.5 FAT32 Read (MBytes/s) 13.3 NTFS Write (MBytes/s) 4.3 NTFS Read (MBytes/s) 12.0 shows that NTFS writes are relatively slow.

Thread Tools	Search this Thread
Show Printable Version Email this Page	Search this Thread: Advanced Search

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
PRS-950 Extra large font on a Kobo book	Alleyoop	Sony Reader	49	03-12-2012 06:33 AM
Extra large left margin?	peartree	Amazon Kindle	5	02-23-2012 12:01 PM
Splitting large file sizes ...	Hangman	Calibre	1	04-05-2010 10:01 PM
Fictionwise extra 10% large purchase micropay at $50 instead of $100??	texasnightowl	Deals and Resources (No Self-Promotion or Affiliate Links)	4	12-21-2009 04:27 PM
Extra Large Print Kindle ebooks	wallcraft	Amazon Kindle	1	12-20-2008 10:53 AM

04-25-2012, 10:07 PM	#2
theducks Well trained by Cats Posts: 29,762 Karma: 54401244 Join Date: Aug 2009 Location: The Central Coast of California Device: Kobo Libra2,Kobo Aura2v1, K4NT(Fixed: New Bat.), Galaxy Tab A	Moderator Notice Moved to (OPDS) Catalog area While Calibre has not published a definitive: Minimum Equipment required to run Spec, it sure sounds like you are trying to run on a sub-standard system. A lowly P4 with 2G RAM opens a 4000+ book Library in about 20 seconds.

04-25-2012, 11:25 PM	#3
PeterT Grand Sorcerer Posts: 12,143 Karma: 73448616 Join Date: Nov 2007 Location: Toronto Device: Nexus 7, Clara, Touch, Tolino EPOS	When you say "The library is already stored on the NAS" what type of NAS is it? How are you connecterd to the NAS? How are the drives attached to the NAS? What happens if you put everything on a local hard drive?

04-26-2012, 01:30 AM	#4
itimpi Wizard Posts: 4,552 Karma: 950151 Join Date: Nov 2008 Device: Sony PRS-950, iphone/ipad (Marvin/iBooks/QuickReader)	What is the spec of your machine, particularly in terms of RAM? I use calibre2opds with my library on a NAS and about 16k books with an average of about 4 formats per book. My metadata.db file is over 70MB. Calibre2Opds tends to peak at around 350MB RAM usage when handling this. In practise I would have thought that 512MB RAM was a minimum for both Calibre and Calibre2Opds, and 1GB was more realistic. This is very low for most modern machines which tend to have considerably more RAM.

04-26-2012, 03:09 PM	#5
vibbix Junior Member Posts: 3 Karma: 10 Join Date: Apr 2012 Device: Kindle Touch	6GB of ram, 2x 640Gb HDD's at 7200RPM, Core i5 2410m, GT555m graphics(Dell XPS 17)... the NAS is probobally the what is slowing it down. Its a WD 1TB hard drive(usb 2.0), attached to my router(Netgear n600)... I can get 20mb while downloading through uSB, closer to 4000k while in NAS... I want to store the data on the NAS so I can access it outside the LAN...

04-26-2012, 03:15 PM	#6
itimpi Wizard Posts: 4,552 Karma: 950151 Join Date: Nov 2008 Device: Sony PRS-950, iphone/ipad (Marvin/iBooks/QuickReader)	It does not sound to me as if your machine spec is a problem. I can understand why the NAS might slow down Calibre2Opds, but not why it would make it fail to run. I use Calibre2Opds against my NAS with a much bigger library. In my case Calibre2Opds can take a few hours to complete the run, but I do not have the problem you report of it failing to run. In terms of speeding Calibre up, you might want to look for the Calibre-Portable.bat file that is held in the resources sub-folder of the Calibre install location. It shows you how you can control where all the Calibre files are located. I use it to keep the calibre metadata.db file on my local hard disk with all the Book files on the NAS. I then periodically copy the metadata.db file into the NAS Calibre library location.

04-26-2012, 04:58 PM	#7
vibbix Junior Member Posts: 3 Karma: 10 Join Date: Apr 2012 Device: Kindle Touch	It allows opens the client, it gives me a SQL out of ram exception.... It wont let me install Calibre2OPDS, so i have to manually run the batch script to let it run... I will try the database move....