I avoid it, rather wait till their old enough to ride their own bike and are old enough to manage the risk. I've been involved in crashes caused by other vehicles I couldn't avoid, crap happens no matter how careful you think you are.
I didn't start riding on the street till my 20's, I don't encourage friends to ride unless they make that decision themselves on their own. Its a high risk activity no doubt, I'm a motorcycle nut that rides year round in all weather thousands of miles and I'm fully aware of the risks. I wouldn't ride with children unless it was a day ride and that would be in FULL riding gear.
Think about your kid getting hurt because you hit some gravel mid corner, that would make me ride ultra careful and what fun would that be? Just like riding with a passenger, if they are fine with your riding style its all good. But, in reality I always will ride in a conservative manner with a passenger because I don't want to take chances with someone's well being in a careless manner. Rather take kids fishing, kayaking, canoeing, hiking, something that involves exercise and less risk.
But, I'm not a cruiser rider running around in shorts and flip flops either and my riding gear has allowed me to walk away from an unexpected face plant while I was just riding in a relaxed manner a mile from home.
I don't shelter kids in general, but I also don't encourage them to do high risk stuff either.