Parking Brake Cable adjustment

I'm in the process of installing a new parking brake.
I attached the top first and figured I could remove the cotter pin on the lever down below, remove the pin, screw the end onto the cable, put the pin back in to check it and so on.
Problem is I took out the cotter pin and that pin don't come out. I gotta believe it is supposed to but it sure is tight.
It does have some rust but moves OK side to side, but not up and down?

If you understand what I am talking about maybe I could get a little guidance.

 

If yo uare talkin gabout the pin i nthe clevis it does come but alot of them have bent over the years or are justed rust up inside and don't want to come out. It may be better to remove the lever itself from the transfercase and set it on a bench where you can hammer the pin out, with a brass drift of course.