Ah, not all people can use magic, only those who have been baptised with a Charter mark. [Although non-magic user wouldn't be able to see it...or maybe they simply didn't notice it.]

The Wall is to keep the Free magic monsters and the Dead in the north where we can properly deal with them.