I'm losing it with my toddler boy...HELP!?

I'm a 23 yr old single mother. I have a 30 mos old boy who has been in the terrible 2's phase since last year. At first, I did follow up on anything he did for the fear of him getting hurt. Plus, I figure he doesn't understand much since he doesn't talk yet. I started to call agencies because I feel like I need some parenting cles so I don't lose my cool. One of the agencies send a child visitor and we focus on his behaviour and such. Once he turned 2, he started to get a visit from a speech therapist, occupational therapist, family counselor, behaviour specialist. Every visit goes great and they tell me how smart my son is. Btw, his speech is still not so great but there has been improvement. Once they leave all hell breaks loose. He runs, jumps, climbs, yells, throws and gets every single one of his toys out. He also grabs his favourite blanket lays on the kitchen or bathroom floor and lays on top of it. I go and tell please pick your nite nite(blanket) and go play with your toys in the room. He looks at me as I'm saying this and lays back down and rolls on the floor. I also baby sit an infant; every time the infant is napping I let my son to be quiet and to play quietly. Five minutes later he's yelling or throwing stuff in the same room that the infant is sleeping. I have ed him, yelled at him(that's how bad I'm losing it) to the point that I feel like just grabbing my son and throwing him against the wall or punching him sooo hard. I know that my boy understands. Am I crazy to think this way? I really need someone to give tips or advice. and Yes, I'm still on a waiting list on parenting education.