There's definitely ways to get the pdf size down, but it probably requires using some other packages. Check this thread on diybookscanner.org:
My workflow for almost djvubind-equivalent PDFs...
I think you could skip the OCR part in that workflow (or possibly leverage the existing ocr text somehow), but convert the SVG to Black & White TIFF, stick that in the pdf, and then run pdfsizeopt.
I'm guessing this process would give you a 10-20 meg pdf. edit: didn't realize pdfsizeopt is linux/mac only...