12-07-2020, 08:21 AM
|
#519
|
Wizard
Posts: 1,199
Karma: 1995558
Join Date: Aug 2015
Device: Kindle
|
Version 1.5.4 released
Version 1.5.2- Fix: bool column set to undefined if cell value is empty, 'undefined' or 'n/a'
Version 1.5.3- Update: Add uuid to identifiers to be used in matching, this will not be done if the user has a custom identifier named uuid.
- Update: Add option to disable tidying titles and author fields for csv import.
- Fix: Separate populating csv table from populating preview table. Now csv table is populated whenever csv file is changed without having to press the preview button.
Version 1.5.4- Update: (experimental, so backup first) add ability to run wizard from cmdline using calibre-debug -r "Import List"
Quote:
Tip for linux users for import speed gains:
The speed of importing books with this plugin have two issues: - The main bottleneck here is HDD read/write speed. This is the addressed in the tip below by temporarily copying metadata.db file to a ramdisk.
- The second issue is deterioration of the speed over time (also happens with ramdisk). If you import a large number of books e.g. thousands, you will notice the speed gradually decreasing. This does not happen only with the Import List plugin, but also with any other operation that add book records to calibre, you can test this by adding thousands of books using calibre's add empty book tool, or by using the copy to library feature with a large number of books.
I do not know the cause of this. It is most likely a database performance issue. Maybe someone who knows better about databases can shed more light.
Anyway, to address this, the import list wizard (when invoked from calibre-debug) has an option to re-initialize the database every nth books (default is 2000) to prevent the performance drop. Note that this does not apply to plugin when used from within calibre gui, this optional feature is only available if the wizard is invoked from command line. This is all experimental so caution is required when using this cmdline mode.
So, here are the steps: - Make a new dir in /dev/shm (ramdisk) and copy your metadata.db file to that dir. For the purpose of the tutorial, I will suppose you made a directory called MyLibrary. So the path to the newly copied metadata.db should be /dev/shm/MyLibrary/metadata.db
- Start the plugin from calibre-debug using the following command:
Code:
calibre-debug -r "Import List"
- A dialog will pop up. First choose the location of calibre library you want to import books into, this is the original location of your library folder.
- In the same dialog, check the option "override database path", and press the button to choose the metadata.db you just copied i.e. /dev/shm/MyLibrary/metadata.db
- Check the option to reinitialize the database and leave the number in the spinbox as it is i.e. 2000
- Press OK and the usual plugin wizard should start where you can import your books and witness the speed boost.
- After you finish this process, you should copy the metadata.db file from /dev/shm/MyLibrary/metadata.db back its original location.
A few warnings regarding the above tip- /dev/shm is a ramdisk and not a persistent storage. You should only use it by making copy of you metadata.db file just before importing. And after the import process is finished, you should move that copy immediately to another location on the HDD. Since /dev/shm is ramdisk, as soon as you restart your computer (or have a power cut), all the data living there will be lost.
- Don't under any circumstances copy or move your whole library to /dev/shm, we only COPY the metadata.db file. Why? in addition to its ethereal nature, ramdisk is as the name implies, part of your system RAM, so anything you copy to a ramdisk will subtract from the available RAM. The metadata.db is at worst a few hundred megabytes which is OK for today's RAM capacities. Still, it wouldn't hurt to check your available RAM compared to the size of the metadata.db file before copying.
- The re-initialization step is experimental, so keep testing it while always having a backup in case something goes wrong. Also note that re-initializing the database takes time that is proportionate to the size of the library. In a library containing tens of thousands of books it may take up to a minute. So re-initializing every 2000 books is a reasonable default value, take that into account if you want to modify it.
|
|
|
|