Register Guidelines E-Books Search Today's Posts Mark Forums Read

Go Back   MobileRead Forums > E-Book Software > Calibre > Library Management

Notices

Reply
 
Thread Tools Search this Thread
Old 05-12-2018, 04:33 AM   #1
raafal
Junior Member
raafal began at the beginning.
 
Posts: 5
Karma: 10
Join Date: May 2018
Device: none
Unhappy problem with "import all calibre data"

Using calibre 3.23 (linux 32bit) I successfully exported all calibre data.
I try to import that data in the new installation (linux 64bit). Unfortunately the process stops 2/3 way.
In the gui mode the progress window just freezes at some percentage and book title. I also tried:
Code:
calibre-debug -c "from calibre.utils.exim import test_import; test_import('/backup-path', '/calibre-library-path')"
the listing stops at the same book title, the last line shows "Killed". The import restores all the metadata, but folders for the latest book are just empty. Also, import did not restore installed plugins and calibre settings.

Is there a way to import all the data into calibre or at least unpack the whole archieve to get hold of the missing books? Because otherwise I'd lose 6000+ of them

Last edited by raafal; 05-12-2018 at 05:25 AM.
raafal is offline   Reply With Quote
Advert
Old 05-12-2018, 06:13 AM   #2
Divingduck
Wizard
Divingduck ought to be getting tired of karma fortunes by now.Divingduck ought to be getting tired of karma fortunes by now.Divingduck ought to be getting tired of karma fortunes by now.Divingduck ought to be getting tired of karma fortunes by now.Divingduck ought to be getting tired of karma fortunes by now.Divingduck ought to be getting tired of karma fortunes by now.Divingduck ought to be getting tired of karma fortunes by now.Divingduck ought to be getting tired of karma fortunes by now.Divingduck ought to be getting tired of karma fortunes by now.Divingduck ought to be getting tired of karma fortunes by now.Divingduck ought to be getting tired of karma fortunes by now.
 
Posts: 1,039
Karma: 1293081
Join Date: Nov 2010
Location: Germany
Device: Sony PRS-650
Mass import a complete library isn't a good solution. Use instead the calibre export/import functionality
https://manual.calibre-ebook.com/faq...ter-to-another

Or, if you like to do it manually, copy the complete library folder to the new place and tell calibre where to find the library. Same for the configuration directory if you want to have your old configuration setup too.
https://manual.calibre-ebook.com/faq...backup-calibre
Divingduck is offline   Reply With Quote
Old 05-12-2018, 06:32 AM   #3
raafal
Junior Member
raafal began at the beginning.
 
Posts: 5
Karma: 10
Join Date: May 2018
Device: none
Maybe I wasn't clear
The problem I described is with calibre export/import functionality
  1. First, in calibre, I used "export/import all calibre data/export all your calibre data" to export the whole library (as a result I got a folder with 298 files in the format part-xxxx.calibre-data, no error messages there)
  2. Then I tried to import it in the new installation, two methods:
    1. in the calibre, "export/import all calibre data/import previously exported data", got stuck 2/3 way through, no changes in the progress window, no disk activity
    2. in the terminal with calibre-debug, all metadata was imported, but only 2/3 books, the latest entries are just empty directories, during the import the terminal listed all the titles imported, the last entry is the word "Killed"

my question is: what can I do to import library successfully, or, if this is not possible, how can I unpack the exported data (those part-xxxx.calibre-data files) to retrieve books manually?
raafal is offline   Reply With Quote
Old 05-12-2018, 06:49 AM   #4
BetterRed
null operator
BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.
 
Posts: 10,154
Karma: 9778562
Join Date: Mar 2012
Location: Sydney Australia
Device: none
@raafal - I'm pretty sure the spanned archive (part-0001.calibre-data etc) can be unpacked using the calibre-debug command, but it doesn't seem to be documented - something like '--unpack-calibre-data' I imagine. Hopefully someone will chime in with the right answer.

If you have the library and configuration directories on the 32bit linux system you could probably just copy them to your 64bit linux system - file systems permitting of course.

BR
BetterRed is offline   Reply With Quote
Old 05-12-2018, 07:29 AM   #5
raafal
Junior Member
raafal began at the beginning.
 
Posts: 5
Karma: 10
Join Date: May 2018
Device: none
If I had the original installation, there would be no problem... but I'm just stuck with this archive I can not import

so I hope you are right there's an undocumented way to just unpack it
raafal is offline   Reply With Quote
Advert
Old 05-12-2018, 08:58 PM   #6
BetterRed
null operator
BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.
 
Posts: 10,154
Karma: 9778562
Join Date: Mar 2012
Location: Sydney Australia
Device: none
I'm 80% sure Kovid has posted a command, or a script, that will decompress an export/import spanned archive. I've done a couple of searches but failed to find anything.

Kovid's taking a holiday right now, but his holidays are usually short.

BR
BetterRed is offline   Reply With Quote
Old 05-13-2018, 06:28 AM   #7
Divingduck
Wizard
Divingduck ought to be getting tired of karma fortunes by now.Divingduck ought to be getting tired of karma fortunes by now.Divingduck ought to be getting tired of karma fortunes by now.Divingduck ought to be getting tired of karma fortunes by now.Divingduck ought to be getting tired of karma fortunes by now.Divingduck ought to be getting tired of karma fortunes by now.Divingduck ought to be getting tired of karma fortunes by now.Divingduck ought to be getting tired of karma fortunes by now.Divingduck ought to be getting tired of karma fortunes by now.Divingduck ought to be getting tired of karma fortunes by now.Divingduck ought to be getting tired of karma fortunes by now.
 
Posts: 1,039
Karma: 1293081
Join Date: Nov 2010
Location: Germany
Device: Sony PRS-650
Quote:
Originally Posted by raafal View Post
Maybe I wasn't clear
The problem I described is with calibre export/import functionality
Sorry, my fault. I didn't read careful enough.

I did a little check today in two VM's. It seems that the import via GUI is working (as well across Windows/Linux with a smaller 3k and a bit bigger 50k library).
But I had a problem to do this via command line tools with Windows where I am not sure if this is a problem with my installation. It will not accept an export folder via cmd. Anyway, that won't really help you.
Divingduck is offline   Reply With Quote
Old 05-13-2018, 06:37 AM   #8
raafal
Junior Member
raafal began at the beginning.
 
Posts: 5
Karma: 10
Join Date: May 2018
Device: none
@Divingduck no problem
your tests in VM's, 3k and 50k lib, you mean number of books or size of archives? my case: 304GB, 35k books
raafal is offline   Reply With Quote
Old 05-13-2018, 08:29 AM   #9
Divingduck
Wizard
Divingduck ought to be getting tired of karma fortunes by now.Divingduck ought to be getting tired of karma fortunes by now.Divingduck ought to be getting tired of karma fortunes by now.Divingduck ought to be getting tired of karma fortunes by now.Divingduck ought to be getting tired of karma fortunes by now.Divingduck ought to be getting tired of karma fortunes by now.Divingduck ought to be getting tired of karma fortunes by now.Divingduck ought to be getting tired of karma fortunes by now.Divingduck ought to be getting tired of karma fortunes by now.Divingduck ought to be getting tired of karma fortunes by now.Divingduck ought to be getting tired of karma fortunes by now.
 
Posts: 1,039
Karma: 1293081
Join Date: Nov 2010
Location: Germany
Device: Sony PRS-650
Number of books
Divingduck is offline   Reply With Quote
Old 05-13-2018, 09:16 AM   #10
raafal
Junior Member
raafal began at the beginning.
 
Posts: 5
Karma: 10
Join Date: May 2018
Device: none
@Divingduck I see, maybe the sheer size of it makes it a problem somehow, I don't know
@BetterRed, thanks

long story short, I managed to import the library

longer version:
  1. the archive was on the disk sitting in the icy box docking station, so it wasn't automatically checked for errors by fsck – I run fsck and it reported and corrected several minor inconsistencies (inode extent tree could be shorter)
  2. a couple of failed import attempts (both gui (just freeze) and calibre-debug ("Killed" message))
  3. moved the archive to the system disk
  4. more failed imports (gui and calibre-debug)
  5. finally, the success
I'm out of the woods now but I still don't know why the import failed so many times and then, magically, succeeded – so I'd like to learn that other decompress method, it might come in useful one day also a more permanent solution would be nice
raafal is offline   Reply With Quote
Old 05-13-2018, 07:07 PM   #11
BetterRed
null operator
BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.BetterRed ought to be getting tired of karma fortunes by now.
 
Posts: 10,154
Karma: 9778562
Join Date: Mar 2012
Location: Sydney Australia
Device: none
Quote:
Originally Posted by raafal View Post

[--snip--]

I'm out of the woods now but I still don't know why the import failed so many times and then, magically, succeeded


Perhaps Kovid or one of the other Linux guru's can shed some light.

Quote:
Originally Posted by raafal View Post
so I'd like to learn that other decompress method, it might come in useful one day also a more permanent solution would be nice
Maybe I'm misremembering.

BR

Last edited by BetterRed; 05-13-2018 at 07:09 PM.
BetterRed is offline   Reply With Quote
Old 05-15-2018, 03:35 PM   #12
kovidgoyal
creator of calibre
kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.kovidgoyal ought to be getting tired of karma fortunes by now.
 
kovidgoyal's Avatar
 
Posts: 32,918
Karma: 10034422
Join Date: Oct 2006
Location: Mumbai, India
Device: Various
The correct way to import using the command line is

calibre-debug --import-calibre-data

As for why it failed, it sounds like the OOM killer (out of memory) Try increasing swap.
kovidgoyal is offline   Reply With Quote
Reply

Thread Tools Search this Thread
Search this Thread:

Advanced Search

Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
Problem with TTS on OnyxBoox M92 "invalid TTS data" Marion Juin Onyx Boox 1 03-27-2013 05:56 PM
No data in "In Library" and "On Device" columns after upgrade ily426 Library Management 8 04-03-2011 02:53 PM
"cannot import name JSONConfig" on launching calibre .42 hakan42 Calibre 4 02-27-2010 03:37 AM


All times are GMT -4. The time now is 02:16 PM.


MobileRead.com is a privately owned, operated and funded community.