Just give it some time, you'll get used to it. I was riding an SG 162 SL I got from Big Canuck, and I believe he's a bigger guy than I am. It was my first metal board and I certainly felt like I was in over my head, but after putting in some consistent riding, I could make it do what I wanted and loved the damn thing.
I've only relegated it to back-up-board after picking up a 162 Kessler from Michael Lambert on a killer deal when he retired after the Sochi Olympics. I rode it a bit at the end of the season and then haven't been able to get out over the last two winters. I felt a bit like you, the speed is scary, and no doubt amplified by the fact that it was built for an Olympic rider who is about the same size as my weekend-warrior ass.
I've only been out a couple of times this winter and definitely felt like it was too much board for me after three runs, but started to have fun after that and have no doubt I'll stop skidding off speed once I get the rust of inactivity off.
Side-note, part of that amazing deal included an APEX V2 plate, and I have to say, that thing is like cheating. I'd usually start to feel my thighs burning up after 5-6 runs, but I can ride all day on a plate without wanting to quit. Mind you, I'm not recommending you strap on an expensive isolation plate; I think snow-feel is crucial to improving your riding and I went for years without one. That said, when you tame the beast, it's something to think about.