Initially, any picking up was fought. He escapes out to an enclosed back porch sometimes, where it's cluttered, can be dangerously hot or cold and is generally uncomfortable and unsafe. We try to block him, but aren't always successful. (We think he thinks that's where we keep the mice.). Initially, we had to leave him out there so the dog didn't join him. Eventually he'd get panicked enough to escape back inside when we opened the door; a thing he'd only do if we opened the door in such a way that he couldn't see us. It was as if he was sure we'd hurt him. Now, I go out and call him and he creeps to me, calling for me to pick him up and carry him inside, away from the scary place. Of course, he never learns to stop doing this.
He used to have trouble with affection, too. All contact led to a fight. Now, he still doesn't want us to initiate it and he won't jump up on his own. Instead, he stands or sits and talks to me - anything from a pitiable mew to a demanding yell. I then have to ask him up by pantomime, holding my hands out and moving my fingers as if scratching both cheeks. Then, and only then, will he jump up. I can settle him on my chest or lap, as long as I'm not holding him there. Petting and purring ensue for a little while until he decides to go sleep between my knees, protected but not confined. Every morning and most evenings.
He's still too scared to relinquish control. But the rules keep changing for the better.