Coming off of 40mg Propranolol for Anxiety

Hi all I've been on 40mg Propranolol for half a year now. I've been taking one tablet every morning for my anxiety and it helped me at my worst but I want to come off of it now. I spoke to my GP and he said I could come off of it abruptly if I wanted but he said just to help my anxiety about it he prescribed me 3 boxes of 10mg tablets and to wean off of it like that, starting at 20mg for a few weeks, then 10 and then stop. I'm nervous to come off of it as I don't want any withdrawal symptoms and if I'm anxious about coming off of it then it could mimic symptoms. I'm considering getting a few other opinions from other GPs and seeing what they say. I'm just super nervous about it as I really do want to stop taking them but I feel like I can't.