I just went through my columns to find the ones you mentioned, and then realized the "modified" column is what I wanted - why the default "date" one was appearing to show by date added (since recently), I'm not sure, but now I can sort properly!
Thanks for explaining.