How come you cant go the exact route you went out but in reverse? That's what I do. Ladder N1 takes you back up into the ghost cellar.
Are you saying you cannot exit the ghost cellar back up to the main hall of teh tower?
You need to put a certain object in the alcove near ladder N1 Spoiler