It isn't the first hour or two that gets you. It is the following several hours in those temps that will tell you if your gear works well.
^ This is the most important advice so far. I'll add that if you're shivering or your hands/feet are numb, you're in danger. That's the point where both judgment and fine motor control start to fade.
Some observations from nine years of winter riding:
- A neck warmer is a requirement, balaclava or vest can help
- Heated gloves are usually better than heated grips. If your bike spends its nights in moisture, replace the shitty barrel connectors with SAE
- If you're shivering or your hands/feet are numb, you're in danger
- For longer trips, some military surplus thermal pant liners are wonderful
- Your tires will never reach their design operating temperature, so don't ask too much of them
It's very possible your helmet just isn't up to the task. You might consider getting one designed for smowmobiling, they're designed with cold temperatures in mind.
Whether it's worth it is up to you. If you also have a car and there's any chance of black ice / snow, it would be foolish not to take it instead.