Converting a large number of e-books from fb2 to epub.

[vicar82]

Hi to all!

My library consists of more than 420K e-books in fb2 format.
I set up converting my entire collection to epub format.
It took 1 week.

My server parameters:
Intel Xeon E5-2670 v2 *2
768G - RAM
RaidZ - Intel dc s3710S 1.6 TB * 4
Calibre running in docker container

After I started the conversion, I saw that the work was in progress and the files were being converted and the CPU was busy.
Conversion completed.
In my library, epub books did not appear in the folder, but in the folder /tmp/calibre_7.19.0_tmp_9cx3yqx2 I found 2,389,531 new files that are very similar to the result of format conversion::

4jig00o_.epub                 ar1vfnfc.fb2                  gyqzezv7.opf                  ipc_result_1_320681_x1t0o2yi.pickle  lvn5rlai.opf                         s3j3c434.fb2                  yb6bfbgp_worker_redirect.log
4jigbfpi.fb2                  ar1vz29g.epub                 gyq_zys3.epub                 ipc_result_1_320682_md_73bdi.pickle  lvn5wfka.fb2                         s3j413o8.jpeg                 yb6c19bc.opf
4jigsbv6.jpeg                 ar1x7nje.fb2                  gyr01hgw.opf                  ipc_result_1_320683_8txijuno.pickle  lvn_8c30.jpeg                        s3j45v59.epub                 yb6dhdjh.epub
4j__ijb8_worker_redirect.log  ar1yk9mb_worker_redirect.log  gyr0_nxo.jpeg                 ipc_result_1_320684_snz0gyfj.pickle  lvn8olfn.epub                        s3j5gqet.fb2                  yb_6evrf.jpeg
4jijxyyn.opf                  ar1ylgt4.opf                  gyr1j51h_worker_redirect.log  ipc_result_1_320685_h_j4syt2.pickle  lvnaf95a.opf                         s3j6fywf.fb2                  yb6_exk0.epub
4jikr6m3.epub                 _ar_1z5v.epub                 gyr2at0d_worker_redirect.log  ipc_result_1_320686_ttp0wtjf.pickle  lvnat15v.opf                         s3j7avaq.fb2                  yb6fy8u4.epub
4jimuw82.fb2                  ar1zmf9k.fb2                  gyr7nkr9_worker_redirect.log  ipc_result_1_320687_2nf7q59_.pickle  l_vnb_2k_worker_redirect.log         s3j8bol5.epub                 yb6g7vf9.jpeg
4jin5hlw.epub                 ar22djdr.fb2                  gyr_7vti_worker_redirect.log  ipc_result_1_320688_0t90dzlg.pickle  lvnc0qxs.fb2                         s3j8iw_h.jpeg                 yb6gtmbp.epub
4jinrrqk.opf                  ar2414q7.jpeg                 gyr7wlf4.jpeg                 ipc_result_1_320689_5r9sc71n.pickle  lv_n_c8z.fb2                         s3jankij.opf                  yb6guhcm.fb2
4jioiche.jpeg                 _ar252x1.fb2                  gyr8aton.jpeg                 ipc_result_1_320690_z2jaeij0.pickle  lvnd0gjv.jpeg                        s3jb1k8q.jpeg                 yb6gyhf4.fb2
4jipbvob.epub                 ar26057s.opf                  gyr8sxvl.opf                  ipc_result_1_320691_gefirdv7.pickle  lvnd957h.opf                         s3jdsxdh.opf                  _yb6lpia_worker_redirect.log
4jipj0b5_worker_redirect.log  ar2_7gql.epub                 gyr9yvds_worker_redirect.log  ipc_result_1_320692_d2gx7ez4.pickle  lvndck41.epub                        s3jexwa9_worker_redirect.log  yb6l_r95.fb2
4jipzmze.epub                 ar27ha_w.fb2                  gyra4q8v.epub                 ipc_result_1_320693_fd13n8j4.pickle  lvne3260.fb2                         s3jfv2wy.epub                 yb6m6c3a_worker_redirect.log
4jiqi5n0.epub                 ar27hjux.opf                  gyrbd3_c.jpeg                 ipc_result_1_320694_u7hr2heq.pickle  lvneaq2_.epub                        s3jglwai.epub                 yb6mqcc2.epub
4jir1q3v.epub                 ar29bl2z_worker_redirect.log  gy_rbgtz_worker_redirect.log  ipc_result_1_320695_sir165ae.pickle  lvngblzb.opf                         s3jhrazi_worker_redirect.log  yb6mqlh4_worker_redirect.log
_4jirec3.fb2                  ar29_ify.epub                 gyrbk428.opf                  ipc_result_1_320696_6atd5jxg.pickle  lvnhdmse.fb2                         s3j__hrw_worker_redirect.log  yb6nebq__worker_redirect.log

Currently, there is only one process running inside the calibre docker container:

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
644 abc 20 0 17.8g 11.9g 113664 R 100.7 3.1 7d+2h MetadataBackup

This process takes up 100% of one core. It does not write anything and almost does not read anything.
Questions:
What is this process?
What is it doing now?
And when will it end?

I did not find any logs to understand what is happening now

[kovidgoyal]

MetadataBackup is a background process you dont need to concern yourself with it. It has nothing to do with conversion.

[theducks]

ANY change to the Library (in your case, Adding a format) fires of a backup of the metadata entry in the Library. (there is a small file in each books folder: metadata.opf, which could be used to restore a damaged DB).
That is why it is of 'no concern'. It is a background CYA calibre process It runs at a very low priority.

[DNSB]

Originally Posted by vicar82

In my library, epub books did not appear in the folder

Not sure but are you saying that after the conversion, the epub books did not appear in your library? I'm not sure what you meant by folder unless you are referring to the folders inside your calibre library where you fb2 books were stored.

[vicar82]

Originally Posted by DNSB

Not sure but are you saying that after the conversion, the epub books did not appear in your library? I'm not sure what you meant by folder unless you are referring to the folders inside your calibre library where you fb2 books were stored.

Not exactly. Conversion was only the first step.
Calibre first places all the files in the /tmp folder and does the conversion there. As a result of the first step, this folder contains fb2 files, epub files, ofp files and a number of files with service information.
The files have already been converted, but not yet in the calibre library.

But the next step is to transfer the prepared files from the /tmp folder to the calibre library.
This is where I was very disappointed.
The process is slow. Very slow!

Number of fb2 files: 412722
After 17 hours of calibre work, 8905 epub files appeared in my library
According to the speed of data transfer, it will take 787 hours to complete the work! That's 32 days!
Just transfer the already converted data from the tmp folder to the library folder + Rename the files.

I understand that calibre uses python and SQLite as a database.
I understand that it is free software. And thanks to kovidgoyal for calibre.
I understand that calibre is simply not designed for that many books.

But 32 days, Karl!!
This is the slowest database I've ever encountered.

[vicar82]

7 days for conversion
and
32 days for copying

I have a lot of RAM - just take it!
Enterprise SSD storage, which is designed to work with DB
20 cores and 40 CPU threads
Why is everything so slow?!

I even wonder, will the subsequent deletion of all fb2 files from the calibre library also take 32 days?

[JSWolf]

The thing to do is to start converting a small number of eBooks to make sure everything works. Then when you have things working and your settings as you want, then you can go for a larger amount of eBooks to convert.

[vicar82]

Originally Posted by JSWolf

The thing to do is to start converting a small number of eBooks to make sure everything works. Then when you have things working and your settings as you want, then you can go for a larger amount of eBooks to convert.

That's the problem, that converting a small number of books - 100, 200, or 500 - does not cause any problems and everything goes smoothly and quickly. The problem is precisely in the large number of books.
In this case, the calibre GUI becomes unresponsive.
And all that remains for me is to wait and monitor and count files via the cli.

I thought that the longest and most difficult thing is converting books, since it is a cpu-intensive process.
But I could not even imagine that the biggest problem is filling the calibre library with already prepared and converted data.

[PeterT]

Have you looked at the options in preferences for how many conversions to run at a time?

Preferences->Miscellaneous->Max simultaneous jobs

Sent from my Pixel 7a using Tapatalk