I tried it with another book (an excerpt, actually, of 42 pages) and it came out fine. (Other than I really do need to crop the pages before sending it to DX.)
Specs: I used the Opticbook 3600 and Opticbook software, scanned at 300 dpi in B&W (not grayscale) to .tiff, then used Acrobat 8 Pro to create a PDF. The final file shows up as 2MB.
I wish I could help, but I had no hitches along the way, so don't have any suggestions for you. Sorry.
Good luck with fixing the problems.