Command line instruction to export all data ?

Bertrand · 07-07-2018, 11:32 AM

Hi

(Windows 10, calibre 3.27)

I've used the option "Export all your data" (not sure of the name in english) from the Library menu to transfer my library and all the settings, extensions, etc., to my laptop.
It's a great tool, it worked very well and I now have an exact copy on my laptop.

Is there a way to do the export via the CLI ?
I'd like to run this option via a batch file + Windows Task Scheduler.

(I already do this with "calibredb export" but if I'm not mistaken, it's only to export books, not the whole calibre configuration.)

I didn't find anything in the doc.

Thanks.

kovidgoyal · 07-07-2018, 02:10 PM

calibre-debug --export-all-calibre-data

Bertrand · 07-08-2018, 02:48 AM

Thanks, but when prompted for a path, I keep getting an error about the syntax.

Code:

> calibre-debug --export-all-calibre-data
Enter path to an empty folder (all exported data will be saved inside it): c:\test

I tried different variations, c:/test, "c:\test" etc but no luck.

Code:

Python function terminated unexpectedly
  [Error 123] La syntaxe du nom de fichier, de répertoire ou de volume est incorrecte: u'c:\\test\r' (Error Code: 1)
Traceback (most recent call last):
  File "site.py", line 101, in main
  File "site.py", line 78, in run_entry_point
  File "site-packages\calibre\debug.py", line 325, in main
  File "site-packages\calibre\utils\exim.py", line 391, in run_exporter
  File "os.py", line 157, in makedirs
WindowsError: [Error 123] La syntaxe du nom de fichier, de répertoire ou de volume est incorrecte: u'c:\\test\r'

kovidgoyal · 07-08-2018, 03:26 AM

Strange, I dont see anything that could cause that. You can always create the emty directory yourself first the calibre will not try to create it.

kovidgoyal · 07-08-2018, 03:27 AM

Never mind, I see it is because there is an extra carriage return at the end of the directory name -- probably something weird about the console in your system? Are you running it in cygwin or a normal windows command prompt?

Bertrand · 07-08-2018, 03:57 AM

I'm using ConEmu (https://conemu.github.io/).
But I tried with a normal windows command prompt and the end of the error message is different now.

Folder created in File Explorer before running the command :

Code:

C:\>calibre-debug --export-all-calibre-data
Enter path to an empty folder (all exported data will be saved inside it): c:\test

Python function terminated unexpectedly
  [Error 123] La syntaxe du nom de fichier, de rTraceback (most recent call last):
  File "site.py", line 101, in main
  File "site.py", line 78, in run_entry_point
  File "site-packages\calibre\debug.py", line 325, in main
  File "site-packages\calibre\utils\exim.py", line 391, in run_exporter
  File "os.py", line 157, in makedirs
WindowsError: [Error 123] La syntaxe du nom de fichier, de rpertoire ou de volume est incorrecte: u'c:\\test\r'
close failed in file object destructor:
sys.excepthook is missing
lost sys.stderr

Folder not created before running the command, same result :

Code:

C:\>calibre-debug --export-all-calibre-data
Enter path to an empty folder (all exported data will be saved inside it): c:\test2

Python function terminated unexpectedly
  [Error 123] La syntaxe du nom de fichier, de rTraceback (most recent call last):
  File "site.py", line 101, in main
  File "site.py", line 78, in run_entry_point
  File "site-packages\calibre\debug.py", line 325, in main
  File "site-packages\calibre\utils\exim.py", line 391, in run_exporter
  File "os.py", line 157, in makedirs
WindowsError: [Error 123] La syntaxe du nom de fichier, de rpertoire ou de volume est incorrecte: u'c:\\test2\r'
close failed in file object destructor:
sys.excepthook is missing
lost sys.stderr

kovidgoyal · 07-08-2018, 04:33 AM

I have no idea why your console is doing that, but in any case, since \r is not legal in filenames, this should work around it: https://github.com/kovidgoyal/calibr...ba5bea9cb26db4

Bertrand · 07-08-2018, 08:37 AM

Thanks Kovid.
I'll check again at the next calibre update.

(I just tested on my laptop (Windows 10 fresh install) and I have the same error)

Divingduck · 07-10-2018, 10:14 AM

@Kovid,
I just recognize the old version had not worked for me too and I had forgot to report that some when in May. I just see the remark in my script

I run your new modification from source. It is a bit better as I can now pass a backup directory, but if I select only one of the libraries no library will be backup.

Spoiler:

Code:

C:\Users\Armin\Documents\Calibre\Calibre-Debug>calibre-debug --export-all-calibre-data c:\Users\Armin\Documents\Calibre\Calibre-Debug\backup
Enter path to an empty folder (all exported data will be saved inside it): c:\Users\Armin\Documents\Calibre\Calibre-Debug\backup
        Compiling form c:\Users\Armin\git\calibre\src\calibre\gui2\filename_pattern.ui
        Compiling form c:\Users\Armin\git\calibre\src\calibre\gui2\catalog\catalog_bibtex.ui
        Compiling form c:\Users\Armin\git\calibre\src\calibre\gui2\catalog\catalog_epub_mobi.ui
        Compiling form c:\Users\Armin\git\calibre\src\calibre\gui2\catalog\catalog_tab_template.ui
        Compiling form c:\Users\Armin\git\calibre\src\calibre\gui2\convert\azw3_output.ui
        Compiling form c:\Users\Armin\git\calibre\src\calibre\gui2\convert\comic_input.ui
        Compiling form c:\Users\Armin\git\calibre\src\calibre\gui2\convert\debug.ui
        Compiling form c:\Users\Armin\git\calibre\src\calibre\gui2\convert\docx_input.ui
        Compiling form c:\Users\Armin\git\calibre\src\calibre\gui2\convert\epub_output.ui
        Compiling form c:\Users\Armin\git\calibre\src\calibre\gui2\convert\fb2_input.ui
        Compiling form c:\Users\Armin\git\calibre\src\calibre\gui2\convert\fb2_output.ui
        Compiling form c:\Users\Armin\git\calibre\src\calibre\gui2\convert\font_key.ui
        Compiling form c:\Users\Armin\git\calibre\src\calibre\gui2\convert\heuristics.ui
        Compiling form c:\Users\Armin\git\calibre\src\calibre\gui2\convert\htmlz_output.ui
        Compiling form c:\Users\Armin\git\calibre\src\calibre\gui2\convert\look_and_feel.ui
        Compiling form c:\Users\Armin\git\calibre\src\calibre\gui2\convert\lrf_output.ui
        Compiling form c:\Users\Armin\git\calibre\src\calibre\gui2\convert\metadata.ui
        Compiling form c:\Users\Armin\git\calibre\src\calibre\gui2\convert\mobi_output.ui
        Compiling form c:\Users\Armin\git\calibre\src\calibre\gui2\convert\page_setup.ui
        Compiling form c:\Users\Armin\git\calibre\src\calibre\gui2\convert\pdb_output.ui
        Compiling form c:\Users\Armin\git\calibre\src\calibre\gui2\convert\pdf_input.ui
        Compiling form c:\Users\Armin\git\calibre\src\calibre\gui2\convert\pdf_output.ui
        Compiling form c:\Users\Armin\git\calibre\src\calibre\gui2\convert\pmlz_output.ui
        Compiling form c:\Users\Armin\git\calibre\src\calibre\gui2\convert\rb_output.ui
        Compiling form c:\Users\Armin\git\calibre\src\calibre\gui2\convert\regex_builder.ui
        Compiling form c:\Users\Armin\git\calibre\src\calibre\gui2\convert\rtf_input.ui
        Compiling form c:\Users\Armin\git\calibre\src\calibre\gui2\convert\search_and_replace.ui
        Compiling form c:\Users\Armin\git\calibre\src\calibre\gui2\convert\single.ui
        Compiling form c:\Users\Armin\git\calibre\src\calibre\gui2\convert\snb_output.ui
        Compiling form c:\Users\Armin\git\calibre\src\calibre\gui2\convert\structure_detection.ui
        Compiling form c:\Users\Armin\git\calibre\src\calibre\gui2\convert\toc.ui
        Compiling form c:\Users\Armin\git\calibre\src\calibre\gui2\convert\txt_input.ui
        Compiling form c:\Users\Armin\git\calibre\src\calibre\gui2\convert\txt_output.ui
        Compiling form c:\Users\Armin\git\calibre\src\calibre\gui2\convert\xexp_edit.ui
        Compiling form c:\Users\Armin\git\calibre\src\calibre\gui2\convert\xpath_wizard.ui
        Compiling form c:\Users\Armin\git\calibre\src\calibre\gui2\device_drivers\configwidget.ui
        Compiling form c:\Users\Armin\git\calibre\src\calibre\gui2\dialogs\catalog.ui
        Compiling form c:\Users\Armin\git\calibre\src\calibre\gui2\dialogs\choose_format_device.ui
        Compiling form c:\Users\Armin\git\calibre\src\calibre\gui2\dialogs\choose_library.ui
        Compiling form c:\Users\Armin\git\calibre\src\calibre\gui2\dialogs\comicconf.ui
        Compiling form c:\Users\Armin\git\calibre\src\calibre\gui2\dialogs\comments_dialog.ui
        Compiling form c:\Users\Armin\git\calibre\src\calibre\gui2\dialogs\confirm_delete_location.ui
        Compiling form c:\Users\Armin\git\calibre\src\calibre\gui2\dialogs\conversion_error.ui
        Compiling form c:\Users\Armin\git\calibre\src\calibre\gui2\dialogs\delete_matching_from_device.ui
        Compiling form c:\Users\Armin\git\calibre\src\calibre\gui2\dialogs\device_category_editor.ui
        Compiling form c:\Users\Armin\git\calibre\src\calibre\gui2\dialogs\drm_error.ui
        Compiling form c:\Users\Armin\git\calibre\src\calibre\gui2\dialogs\edit_authors_dialog.ui
        Compiling form c:\Users\Armin\git\calibre\src\calibre\gui2\dialogs\jobs.ui
        Compiling form c:\Users\Armin\git\calibre\src\calibre\gui2\dialogs\match_books.ui
        Compiling form c:\Users\Armin\git\calibre\src\calibre\gui2\dialogs\metadata_bulk.ui
        Compiling form c:\Users\Armin\git\calibre\src\calibre\gui2\dialogs\password.ui
        Compiling form c:\Users\Armin\git\calibre\src\calibre\gui2\dialogs\quickview.ui
        Compiling form c:\Users\Armin\git\calibre\src\calibre\gui2\dialogs\smartdevice.ui
        Compiling form c:\Users\Armin\git\calibre\src\calibre\gui2\dialogs\tag_categories.ui
        Compiling form c:\Users\Armin\git\calibre\src\calibre\gui2\dialogs\tag_editor.ui
        Compiling form c:\Users\Armin\git\calibre\src\calibre\gui2\dialogs\tag_list_editor.ui
        Compiling form c:\Users\Armin\git\calibre\src\calibre\gui2\dialogs\template_dialog.ui
        Compiling form c:\Users\Armin\git\calibre\src\calibre\gui2\lrf_renderer\config.ui
        Compiling form c:\Users\Armin\git\calibre\src\calibre\gui2\lrf_renderer\main.ui
        Compiling form c:\Users\Armin\git\calibre\src\calibre\gui2\preferences\adding.ui
        Compiling form c:\Users\Armin\git\calibre\src\calibre\gui2\preferences\behavior.ui
        Compiling form c:\Users\Armin\git\calibre\src\calibre\gui2\preferences\columns.ui
        Compiling form c:\Users\Armin\git\calibre\src\calibre\gui2\preferences\email.ui
        Compiling form c:\Users\Armin\git\calibre\src\calibre\gui2\preferences\look_feel.ui
        Compiling form c:\Users\Armin\git\calibre\src\calibre\gui2\preferences\metadata_sources.ui
        Compiling form c:\Users\Armin\git\calibre\src\calibre\gui2\preferences\misc.ui
        Compiling form c:\Users\Armin\git\calibre\src\calibre\gui2\preferences\plugboard.ui
        Compiling form c:\Users\Armin\git\calibre\src\calibre\gui2\preferences\plugins.ui
        Compiling form c:\Users\Armin\git\calibre\src\calibre\gui2\preferences\save_template.ui
        Compiling form c:\Users\Armin\git\calibre\src\calibre\gui2\preferences\saving.ui
        Compiling form c:\Users\Armin\git\calibre\src\calibre\gui2\preferences\search.ui
        Compiling form c:\Users\Armin\git\calibre\src\calibre\gui2\preferences\sending.ui
        Compiling form c:\Users\Armin\git\calibre\src\calibre\gui2\preferences\template_functions.ui
        Compiling form c:\Users\Armin\git\calibre\src\calibre\gui2\preferences\toolbar.ui
        Compiling form c:\Users\Armin\git\calibre\src\calibre\gui2\store\basic_config_widget.ui
        Compiling form c:\Users\Armin\git\calibre\src\calibre\gui2\store\web_store_dialog.ui
        Compiling form c:\Users\Armin\git\calibre\src\calibre\gui2\store\config\chooser\adv_search_builder.ui
        Compiling form c:\Users\Armin\git\calibre\src\calibre\gui2\store\config\chooser\chooser_widget.ui
        Compiling form c:\Users\Armin\git\calibre\src\calibre\gui2\store\config\search\search_widget.ui
        Compiling form c:\Users\Armin\git\calibre\src\calibre\gui2\store\search\adv_search_builder.ui
        Compiling form c:\Users\Armin\git\calibre\src\calibre\gui2\store\search\search.ui
        Compiling form c:\Users\Armin\git\calibre\src\calibre\gui2\store\stores\mobileread\adv_search_builder.ui
        Compiling form c:\Users\Armin\git\calibre\src\calibre\gui2\store\stores\mobileread\cache_progress_dialog.ui
        Compiling form c:\Users\Armin\git\calibre\src\calibre\gui2\store\stores\mobileread\store_dialog.ui
        Compiling form c:\Users\Armin\git\calibre\src\calibre\gui2\viewer\config.ui
        Compiling form c:\Users\Armin\git\calibre\src\calibre\gui2\wizard\device.ui
        Compiling form c:\Users\Armin\git\calibre\src\calibre\gui2\wizard\finish.ui
        Compiling form c:\Users\Armin\git\calibre\src\calibre\gui2\wizard\kindle.ui
        Compiling form c:\Users\Armin\git\calibre\src\calibre\gui2\wizard\library.ui
        Compiling form c:\Users\Armin\git\calibre\src\calibre\gui2\wizard\send_email.ui
        Compiling form c:\Users\Armin\git\calibre\src\calibre\gui2\wizard\stanza.ui
Compiled 91 forms
Export the library E:/Test-DB [y/n]: y
Export the library E:/Calibre Normen-DB [y/n]: n
Export the library E:/Calibre Magazine [y/n]: n
Export the library E:/Calibre MobilleRead-DB [y/n]: n
Export the library E:/Calibre Bibliothek [y/n]: n
Export the library E:/Calibre Galileo-DB [y/n]: n
Export the library c:/users/armin/documents/calibretest-embroidery_db [y/n]: n
Export the library E:/Calibre Embroidery [y/n]: n
Export the library E:/Calibre CountPage-DB [y/n]: n
Export the library E:/Calibre Test-DB [y/n]: n
Export the library C:/Users/Armin/Documents/Python/Annot-DB [y/n]: n
Export the library E:/Calibre JK-DB [y/n]: n
No libraries selected for export

I have a little additional request. Is it possible to add some arguments for scripting? I like to pass a backup directory and in addition only dedicated libraries or "all" as parameters. No parameter means the original behavior as it is now. This would be very helpful.

kovidgoyal · 07-10-2018, 12:48 PM

There you go: https://github.com/kovidgoyal/calibr...836c2d918bb6a1

Divingduck · 07-10-2018, 06:10 PM

Thank you very much.
I test from command line and select some of the libraries. This worked very well.
Same for scripting. The only thing what I was not able to do: all libraries as script.

For those who like to test:

calibre-debug --export-all-calibre-data [backup_dir] [library1] [library2] ...

Needs absolute paths for the export directory and libraries and quotation marks if path or library names include spaces.

Sample script:
calibre-debug --export-all-calibre-data c:\path\to\backup_dir C:\path\to\first\library\library_name "C:\path with space\to\second\library name" [...]

kovidgoyal · 07-10-2018, 10:43 PM

https://github.com/kovidgoyal/calibr...836c2d918bb6a1

Divingduck · 07-11-2018, 05:56 AM

Sweet. This works perfect. Just finished a backup about 68GB on my test system.

Question, do you need to modify --import-calibre-data as well? I've trouble to run this via cmd on my test system. There seems to be the same behavior for passing a backup directory.

Spoiler:

It would be nice having the scripting similar to the exporter

kovidgoyal · 07-11-2018, 06:21 AM

I will fix the import input, but I dont see the point of making it scriptable. It's not like it needs to be run regularly.

Divingduck · 07-11-2018, 06:29 AM

Yes, I understood.

The additional request was more for having a automated test scenario to verify a backup w/o the need for checking this personally.

07-07-2018, 11:32 AM	#1
Bertrand Zealot Posts: 124 Karma: 10 Join Date: Dec 2008 Location: France Device: None	Command line instruction to export all data ? Hi (Windows 10, calibre 3.27) I've used the option "Export all your data" (not sure of the name in english) from the Library menu to transfer my library and all the settings, extensions, etc., to my laptop. It's a great tool, it worked very well and I now have an exact copy on my laptop. Is there a way to do the export via the CLI ? I'd like to run this option via a batch file + Windows Task Scheduler. (I already do this with "calibredb export" but if I'm not mistaken, it's only to export books, not the whole calibre configuration.) I didn't find anything in the doc. Thanks.

07-11-2018, 05:56 AM	#13
Divingduck Wizard Posts: 1,166 Karma: 1410083 Join Date: Nov 2010 Location: Germany Device: Sony PRS-650	Sweet. This works perfect. Just finished a backup about 68GB on my test system. Question, do you need to modify --import-calibre-data as well? I've trouble to run this via cmd on my test system. There seems to be the same behavior for passing a backup directory. Spoiler: Code: C:\Users\Armin\Documents\Calibre\Calibre-Debug>calibre-debug --import-calibre-data Enter path to folder containing previously exported data: c:\Users\Armin\Documents\Calibre\Calibre-Debug\backup is not a folderocuments\Calibre\Calibre-Debug\backup C:\Users\Armin\Documents\Calibre\Calibre-Debug> It would be nice having the scripting similar to the exporter

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
Is there a way to execute a non-root command without access to command line	Galunid	Kindle Developer's Corner	4	05-15-2018 08:50 PM
Export Calibre Data	GeckoFriend	Library Management	4	03-27-2018 04:15 PM
Export Category Editor data	DedTV	Library Management	6	06-03-2013 05:06 AM
Any command line to edit epub meta data?	bthoven	Calibre	1	03-25-2010 08:26 AM
Any command line to edit epub meta data?	bthoven	ePub	2	03-25-2010 05:15 AM

07-07-2018, 02:10 PM	#2
kovidgoyal creator of calibre Posts: 45,609 Karma: 28549044 Join Date: Oct 2006 Location: Mumbai, India Device: Various	calibre-debug --export-all-calibre-data

07-08-2018, 03:26 AM	#4
kovidgoyal creator of calibre Posts: 45,609 Karma: 28549044 Join Date: Oct 2006 Location: Mumbai, India Device: Various	Strange, I dont see anything that could cause that. You can always create the emty directory yourself first the calibre will not try to create it.

07-08-2018, 03:27 AM	#5
kovidgoyal creator of calibre Posts: 45,609 Karma: 28549044 Join Date: Oct 2006 Location: Mumbai, India Device: Various	Never mind, I see it is because there is an extra carriage return at the end of the directory name -- probably something weird about the console in your system? Are you running it in cygwin or a normal windows command prompt?

07-08-2018, 04:33 AM	#7
kovidgoyal creator of calibre Posts: 45,609 Karma: 28549044 Join Date: Oct 2006 Location: Mumbai, India Device: Various	I have no idea why your console is doing that, but in any case, since \r is not legal in filenames, this should work around it: https://github.com/kovidgoyal/calibr...ba5bea9cb26db4

07-08-2018, 08:37 AM	#8
Bertrand Zealot Posts: 124 Karma: 10 Join Date: Dec 2008 Location: France Device: None	Thanks Kovid. I'll check again at the next calibre update. (I just tested on my laptop (Windows 10 fresh install) and I have the same error)

07-10-2018, 12:48 PM	#10
kovidgoyal creator of calibre Posts: 45,609 Karma: 28549044 Join Date: Oct 2006 Location: Mumbai, India Device: Various	There you go: https://github.com/kovidgoyal/calibr...836c2d918bb6a1

07-10-2018, 06:10 PM	#11
Divingduck Wizard Posts: 1,166 Karma: 1410083 Join Date: Nov 2010 Location: Germany Device: Sony PRS-650	Thank you very much. I test from command line and select some of the libraries. This worked very well. Same for scripting. The only thing what I was not able to do: all libraries as script. For those who like to test: calibre-debug --export-all-calibre-data [backup_dir] [library1] [library2] ... Needs absolute paths for the export directory and libraries and quotation marks if path or library names include spaces. Sample script: calibre-debug --export-all-calibre-data c:\path\to\backup_dir C:\path\to\first\library\library_name "C:\path with space\to\second\library name" [...]

07-10-2018, 10:43 PM	#12
kovidgoyal creator of calibre Posts: 45,609 Karma: 28549044 Join Date: Oct 2006 Location: Mumbai, India Device: Various	https://github.com/kovidgoyal/calibr...836c2d918bb6a1

07-11-2018, 06:21 AM	#14
kovidgoyal creator of calibre Posts: 45,609 Karma: 28549044 Join Date: Oct 2006 Location: Mumbai, India Device: Various	I will fix the import input, but I dont see the point of making it scriptable. It's not like it needs to be run regularly.

07-11-2018, 06:29 AM	#15
Divingduck Wizard Posts: 1,166 Karma: 1410083 Join Date: Nov 2010 Location: Germany Device: Sony PRS-650	Yes, I understood. The additional request was more for having a automated test scenario to verify a backup w/o the need for checking this personally.

Advert

Advert