I deleted the books from calibre (just from the list of books in my library)... but that didn't take them off the ereader.
Right, deleting a book from your Calibre library won't delete them from your reader. As far as I know, Calibre does not have any kind of 'sync' functionality.
While the manual method you used is perfectly acceptable, if you already have Calibre open, it might be easier to remove books that way so you don't have to look through the folder structure on the device for a particulare book.
When you plug in your Kobo, give Calibre a second to detect the device, and then detect all the books on it (you'll see the Job icon spinning while it's doing this).
You should see a READER tab appear beside your library tab, as you see in the screenshot below. Click on that, and delete books from that list to remove them from the device.