|
Ok, found a major culprit for memory usage. The way I have been doing cover thumbnail creation is... suboptimal.
Basically I was being "clever" by spawning thumbnail generation into it's own goroutine. Couple of problems. Book download is generally faster than cover generation, so the number of active "create thumbnail" goroutines climbs, each consuming memory. And the second problem is I never limited the number of goroutines that could be spawned.
I've changed the code to wait for the goroutine to finish before continuing on to the next book. Makes the transferring book process take longer, but causes less problems.
I'm going to continue seeing if I can reduce the thumbnail creation memory usage further.
|