MobileRead Forums

MobileRead Forums (https://www.mobileread.com/forums/index.php)
-   Calibre (https://www.mobileread.com/forums/forumdisplay.php?f=166)
-   -   Command line instruction to export all data ? (https://www.mobileread.com/forums/showthread.php?t=303171)

Bertrand 07-07-2018 11:32 AM

Command line instruction to export all data ?
 
Hi

(Windows 10, calibre 3.27)

I've used the option "Export all your data" (not sure of the name in english) from the Library menu to transfer my library and all the settings, extensions, etc., to my laptop.
It's a great tool, it worked very well and I now have an exact copy on my laptop.

Is there a way to do the export via the CLI ?
I'd like to run this option via a batch file + Windows Task Scheduler.

(I already do this with "calibredb export" but if I'm not mistaken, it's only to export books, not the whole calibre configuration.)

I didn't find anything in the doc.

Thanks.

kovidgoyal 07-07-2018 02:10 PM

calibre-debug --export-all-calibre-data

Bertrand 07-08-2018 02:48 AM

Thanks, but when prompted for a path, I keep getting an error about the syntax.

Code:

> calibre-debug --export-all-calibre-data
Enter path to an empty folder (all exported data will be saved inside it): c:\test

I tried different variations, c:/test, "c:\test" etc but no luck.

Code:

Python function terminated unexpectedly
  [Error 123] La syntaxe du nom de fichier, de répertoire ou de volume est incorrecte: u'c:\\test\r' (Error Code: 1)
Traceback (most recent call last):
  File "site.py", line 101, in main
  File "site.py", line 78, in run_entry_point
  File "site-packages\calibre\debug.py", line 325, in main
  File "site-packages\calibre\utils\exim.py", line 391, in run_exporter
  File "os.py", line 157, in makedirs
WindowsError: [Error 123] La syntaxe du nom de fichier, de répertoire ou de volume est incorrecte: u'c:\\test\r'


kovidgoyal 07-08-2018 03:26 AM

Strange, I dont see anything that could cause that. You can always create the emty directory yourself first the calibre will not try to create it.

kovidgoyal 07-08-2018 03:27 AM

Never mind, I see it is because there is an extra carriage return at the end of the directory name -- probably something weird about the console in your system? Are you running it in cygwin or a normal windows command prompt?

Bertrand 07-08-2018 03:57 AM

I'm using ConEmu (https://conemu.github.io/).
But I tried with a normal windows command prompt and the end of the error message is different now.

Folder created in File Explorer before running the command :
Code:

C:\>calibre-debug --export-all-calibre-data
Enter path to an empty folder (all exported data will be saved inside it): c:\test

Python function terminated unexpectedly
  [Error 123] La syntaxe du nom de fichier, de rTraceback (most recent call last):
  File "site.py", line 101, in main
  File "site.py", line 78, in run_entry_point
  File "site-packages\calibre\debug.py", line 325, in main
  File "site-packages\calibre\utils\exim.py", line 391, in run_exporter
  File "os.py", line 157, in makedirs
WindowsError: [Error 123] La syntaxe du nom de fichier, de rpertoire ou de volume est incorrecte: u'c:\\test\r'
close failed in file object destructor:
sys.excepthook is missing
lost sys.stderr


Folder not created before running the command, same result :
Code:

C:\>calibre-debug --export-all-calibre-data
Enter path to an empty folder (all exported data will be saved inside it): c:\test2

Python function terminated unexpectedly
  [Error 123] La syntaxe du nom de fichier, de rTraceback (most recent call last):
  File "site.py", line 101, in main
  File "site.py", line 78, in run_entry_point
  File "site-packages\calibre\debug.py", line 325, in main
  File "site-packages\calibre\utils\exim.py", line 391, in run_exporter
  File "os.py", line 157, in makedirs
WindowsError: [Error 123] La syntaxe du nom de fichier, de rpertoire ou de volume est incorrecte: u'c:\\test2\r'
close failed in file object destructor:
sys.excepthook is missing
lost sys.stderr


kovidgoyal 07-08-2018 04:33 AM

I have no idea why your console is doing that, but in any case, since \r is not legal in filenames, this should work around it: https://github.com/kovidgoyal/calibr...ba5bea9cb26db4

Bertrand 07-08-2018 08:37 AM

Thanks Kovid.
I'll check again at the next calibre update.

(I just tested on my laptop (Windows 10 fresh install) and I have the same error)

Divingduck 07-10-2018 10:14 AM

@Kovid,
I just recognize the old version had not worked for me too and I had forgot to report that some when in May. I just see the remark in my script :o

I run your new modification from source. It is a bit better as I can now pass a backup directory, but if I select only one of the libraries no library will be backup.

Spoiler:

Code:

C:\Users\Armin\Documents\Calibre\Calibre-Debug>calibre-debug --export-all-calibre-data c:\Users\Armin\Documents\Calibre\Calibre-Debug\backup
Enter path to an empty folder (all exported data will be saved inside it): c:\Users\Armin\Documents\Calibre\Calibre-Debug\backup
        Compiling form c:\Users\Armin\git\calibre\src\calibre\gui2\filename_pattern.ui
        Compiling form c:\Users\Armin\git\calibre\src\calibre\gui2\catalog\catalog_bibtex.ui
        Compiling form c:\Users\Armin\git\calibre\src\calibre\gui2\catalog\catalog_epub_mobi.ui
        Compiling form c:\Users\Armin\git\calibre\src\calibre\gui2\catalog\catalog_tab_template.ui
        Compiling form c:\Users\Armin\git\calibre\src\calibre\gui2\convert\azw3_output.ui
        Compiling form c:\Users\Armin\git\calibre\src\calibre\gui2\convert\comic_input.ui
        Compiling form c:\Users\Armin\git\calibre\src\calibre\gui2\convert\debug.ui
        Compiling form c:\Users\Armin\git\calibre\src\calibre\gui2\convert\docx_input.ui
        Compiling form c:\Users\Armin\git\calibre\src\calibre\gui2\convert\epub_output.ui
        Compiling form c:\Users\Armin\git\calibre\src\calibre\gui2\convert\fb2_input.ui
        Compiling form c:\Users\Armin\git\calibre\src\calibre\gui2\convert\fb2_output.ui
        Compiling form c:\Users\Armin\git\calibre\src\calibre\gui2\convert\font_key.ui
        Compiling form c:\Users\Armin\git\calibre\src\calibre\gui2\convert\heuristics.ui
        Compiling form c:\Users\Armin\git\calibre\src\calibre\gui2\convert\htmlz_output.ui
        Compiling form c:\Users\Armin\git\calibre\src\calibre\gui2\convert\look_and_feel.ui
        Compiling form c:\Users\Armin\git\calibre\src\calibre\gui2\convert\lrf_output.ui
        Compiling form c:\Users\Armin\git\calibre\src\calibre\gui2\convert\metadata.ui
        Compiling form c:\Users\Armin\git\calibre\src\calibre\gui2\convert\mobi_output.ui
        Compiling form c:\Users\Armin\git\calibre\src\calibre\gui2\convert\page_setup.ui
        Compiling form c:\Users\Armin\git\calibre\src\calibre\gui2\convert\pdb_output.ui
        Compiling form c:\Users\Armin\git\calibre\src\calibre\gui2\convert\pdf_input.ui
        Compiling form c:\Users\Armin\git\calibre\src\calibre\gui2\convert\pdf_output.ui
        Compiling form c:\Users\Armin\git\calibre\src\calibre\gui2\convert\pmlz_output.ui
        Compiling form c:\Users\Armin\git\calibre\src\calibre\gui2\convert\rb_output.ui
        Compiling form c:\Users\Armin\git\calibre\src\calibre\gui2\convert\regex_builder.ui
        Compiling form c:\Users\Armin\git\calibre\src\calibre\gui2\convert\rtf_input.ui
        Compiling form c:\Users\Armin\git\calibre\src\calibre\gui2\convert\search_and_replace.ui
        Compiling form c:\Users\Armin\git\calibre\src\calibre\gui2\convert\single.ui
        Compiling form c:\Users\Armin\git\calibre\src\calibre\gui2\convert\snb_output.ui
        Compiling form c:\Users\Armin\git\calibre\src\calibre\gui2\convert\structure_detection.ui
        Compiling form c:\Users\Armin\git\calibre\src\calibre\gui2\convert\toc.ui
        Compiling form c:\Users\Armin\git\calibre\src\calibre\gui2\convert\txt_input.ui
        Compiling form c:\Users\Armin\git\calibre\src\calibre\gui2\convert\txt_output.ui
        Compiling form c:\Users\Armin\git\calibre\src\calibre\gui2\convert\xexp_edit.ui
        Compiling form c:\Users\Armin\git\calibre\src\calibre\gui2\convert\xpath_wizard.ui
        Compiling form c:\Users\Armin\git\calibre\src\calibre\gui2\device_drivers\configwidget.ui
        Compiling form c:\Users\Armin\git\calibre\src\calibre\gui2\dialogs\catalog.ui
        Compiling form c:\Users\Armin\git\calibre\src\calibre\gui2\dialogs\choose_format_device.ui
        Compiling form c:\Users\Armin\git\calibre\src\calibre\gui2\dialogs\choose_library.ui
        Compiling form c:\Users\Armin\git\calibre\src\calibre\gui2\dialogs\comicconf.ui
        Compiling form c:\Users\Armin\git\calibre\src\calibre\gui2\dialogs\comments_dialog.ui
        Compiling form c:\Users\Armin\git\calibre\src\calibre\gui2\dialogs\confirm_delete_location.ui
        Compiling form c:\Users\Armin\git\calibre\src\calibre\gui2\dialogs\conversion_error.ui
        Compiling form c:\Users\Armin\git\calibre\src\calibre\gui2\dialogs\delete_matching_from_device.ui
        Compiling form c:\Users\Armin\git\calibre\src\calibre\gui2\dialogs\device_category_editor.ui
        Compiling form c:\Users\Armin\git\calibre\src\calibre\gui2\dialogs\drm_error.ui
        Compiling form c:\Users\Armin\git\calibre\src\calibre\gui2\dialogs\edit_authors_dialog.ui
        Compiling form c:\Users\Armin\git\calibre\src\calibre\gui2\dialogs\jobs.ui
        Compiling form c:\Users\Armin\git\calibre\src\calibre\gui2\dialogs\match_books.ui
        Compiling form c:\Users\Armin\git\calibre\src\calibre\gui2\dialogs\metadata_bulk.ui
        Compiling form c:\Users\Armin\git\calibre\src\calibre\gui2\dialogs\password.ui
        Compiling form c:\Users\Armin\git\calibre\src\calibre\gui2\dialogs\quickview.ui
        Compiling form c:\Users\Armin\git\calibre\src\calibre\gui2\dialogs\smartdevice.ui
        Compiling form c:\Users\Armin\git\calibre\src\calibre\gui2\dialogs\tag_categories.ui
        Compiling form c:\Users\Armin\git\calibre\src\calibre\gui2\dialogs\tag_editor.ui
        Compiling form c:\Users\Armin\git\calibre\src\calibre\gui2\dialogs\tag_list_editor.ui
        Compiling form c:\Users\Armin\git\calibre\src\calibre\gui2\dialogs\template_dialog.ui
        Compiling form c:\Users\Armin\git\calibre\src\calibre\gui2\lrf_renderer\config.ui
        Compiling form c:\Users\Armin\git\calibre\src\calibre\gui2\lrf_renderer\main.ui
        Compiling form c:\Users\Armin\git\calibre\src\calibre\gui2\preferences\adding.ui
        Compiling form c:\Users\Armin\git\calibre\src\calibre\gui2\preferences\behavior.ui
        Compiling form c:\Users\Armin\git\calibre\src\calibre\gui2\preferences\columns.ui
        Compiling form c:\Users\Armin\git\calibre\src\calibre\gui2\preferences\email.ui
        Compiling form c:\Users\Armin\git\calibre\src\calibre\gui2\preferences\look_feel.ui
        Compiling form c:\Users\Armin\git\calibre\src\calibre\gui2\preferences\metadata_sources.ui
        Compiling form c:\Users\Armin\git\calibre\src\calibre\gui2\preferences\misc.ui
        Compiling form c:\Users\Armin\git\calibre\src\calibre\gui2\preferences\plugboard.ui
        Compiling form c:\Users\Armin\git\calibre\src\calibre\gui2\preferences\plugins.ui
        Compiling form c:\Users\Armin\git\calibre\src\calibre\gui2\preferences\save_template.ui
        Compiling form c:\Users\Armin\git\calibre\src\calibre\gui2\preferences\saving.ui
        Compiling form c:\Users\Armin\git\calibre\src\calibre\gui2\preferences\search.ui
        Compiling form c:\Users\Armin\git\calibre\src\calibre\gui2\preferences\sending.ui
        Compiling form c:\Users\Armin\git\calibre\src\calibre\gui2\preferences\template_functions.ui
        Compiling form c:\Users\Armin\git\calibre\src\calibre\gui2\preferences\toolbar.ui
        Compiling form c:\Users\Armin\git\calibre\src\calibre\gui2\store\basic_config_widget.ui
        Compiling form c:\Users\Armin\git\calibre\src\calibre\gui2\store\web_store_dialog.ui
        Compiling form c:\Users\Armin\git\calibre\src\calibre\gui2\store\config\chooser\adv_search_builder.ui
        Compiling form c:\Users\Armin\git\calibre\src\calibre\gui2\store\config\chooser\chooser_widget.ui
        Compiling form c:\Users\Armin\git\calibre\src\calibre\gui2\store\config\search\search_widget.ui
        Compiling form c:\Users\Armin\git\calibre\src\calibre\gui2\store\search\adv_search_builder.ui
        Compiling form c:\Users\Armin\git\calibre\src\calibre\gui2\store\search\search.ui
        Compiling form c:\Users\Armin\git\calibre\src\calibre\gui2\store\stores\mobileread\adv_search_builder.ui
        Compiling form c:\Users\Armin\git\calibre\src\calibre\gui2\store\stores\mobileread\cache_progress_dialog.ui
        Compiling form c:\Users\Armin\git\calibre\src\calibre\gui2\store\stores\mobileread\store_dialog.ui
        Compiling form c:\Users\Armin\git\calibre\src\calibre\gui2\viewer\config.ui
        Compiling form c:\Users\Armin\git\calibre\src\calibre\gui2\wizard\device.ui
        Compiling form c:\Users\Armin\git\calibre\src\calibre\gui2\wizard\finish.ui
        Compiling form c:\Users\Armin\git\calibre\src\calibre\gui2\wizard\kindle.ui
        Compiling form c:\Users\Armin\git\calibre\src\calibre\gui2\wizard\library.ui
        Compiling form c:\Users\Armin\git\calibre\src\calibre\gui2\wizard\send_email.ui
        Compiling form c:\Users\Armin\git\calibre\src\calibre\gui2\wizard\stanza.ui
Compiled 91 forms
Export the library E:/Test-DB [y/n]: y
Export the library E:/Calibre Normen-DB [y/n]: n
Export the library E:/Calibre Magazine [y/n]: n
Export the library E:/Calibre MobilleRead-DB [y/n]: n
Export the library E:/Calibre Bibliothek [y/n]: n
Export the library E:/Calibre Galileo-DB [y/n]: n
Export the library c:/users/armin/documents/calibretest-embroidery_db [y/n]: n
Export the library E:/Calibre Embroidery [y/n]: n
Export the library E:/Calibre CountPage-DB [y/n]: n
Export the library E:/Calibre Test-DB [y/n]: n
Export the library C:/Users/Armin/Documents/Python/Annot-DB [y/n]: n
Export the library E:/Calibre JK-DB [y/n]: n
No libraries selected for export



I have a little additional request. Is it possible to add some arguments for scripting? I like to pass a backup directory and in addition only dedicated libraries or "all" as parameters. No parameter means the original behavior as it is now. This would be very helpful.

kovidgoyal 07-10-2018 12:48 PM

There you go: https://github.com/kovidgoyal/calibr...836c2d918bb6a1

Divingduck 07-10-2018 06:10 PM

Thank you very much.
I test from command line and select some of the libraries. This worked very well.
Same for scripting. The only thing what I was not able to do: all libraries as script.


For those who like to test:

calibre-debug --export-all-calibre-data [backup_dir] [library1] [library2] ...

Needs absolute paths for the export directory and libraries and quotation marks if path or library names include spaces.

Sample script:
calibre-debug --export-all-calibre-data c:\path\to\backup_dir C:\path\to\first\library\library_name "C:\path with space\to\second\library name" [...]

kovidgoyal 07-10-2018 10:43 PM

https://github.com/kovidgoyal/calibr...836c2d918bb6a1

Divingduck 07-11-2018 05:56 AM

Sweet. This works perfect. Just finished a backup about 68GB on my test system.

Question, do you need to modify --import-calibre-data as well? I've trouble to run this via cmd on my test system. There seems to be the same behavior for passing a backup directory.

Spoiler:
Code:

C:\Users\Armin\Documents\Calibre\Calibre-Debug>calibre-debug --import-calibre-data
Enter path to folder containing previously exported data: c:\Users\Armin\Documents\Calibre\Calibre-Debug\backup
 is not a folderocuments\Calibre\Calibre-Debug\backup
C:\Users\Armin\Documents\Calibre\Calibre-Debug>



It would be nice having the scripting similar to the exporter :)

kovidgoyal 07-11-2018 06:21 AM

I will fix the import input, but I dont see the point of making it scriptable. It's not like it needs to be run regularly.

Divingduck 07-11-2018 06:29 AM

Yes, I understood.

The additional request was more for having a automated test scenario to verify a backup w/o the need for checking this personally. :)


All times are GMT -4. The time now is 09:38 PM.

Powered by: vBulletin
Copyright ©2000 - 3.8.5, Jelsoft Enterprises Ltd.
MobileRead.com is a privately owned, operated and funded community.