Photography is my other passion. Only you can decide whether a DSLR is too big or not. Personally I think babies and kids are the perfect use for them. They're not too expensive anymore, you can get a Canon T4i or Nikon D3200 or Pentax K-30 from stores liquidating them for like $450 or less with lens. They are the previous model but are very mature, capable cameras. Throw them in full-auto mode and they're as easy to use as a point-n-shoot.
If you don't want the size of a DSLR but you want the same image quality, have a look at mirrorless interchangeable lens cameras like the Sony NEX series (now called the a5000 and a6000) or the Fuji XE-2, XM-1, or XA-1. These can fit in a small bag or even a large jacket pocket. The trade-off here is the autofocus is not usually up to the task of kids running around or playing sports like soccer. Personally I wouldn't bother with Panasonic or Olympus as they use a smaller sensor which I am prejudiced against.
If you really want a point-n-shoot that you can easily bring everywhere without a bag, the hands-down best is the Sony RX100-II, or you can save a bit with the previous but still current model, the RX100. The next best thing would be the Canon S120, but I wouldn't go any cheaper than that. Avoid "superzoom" cameras, and point-n-shoot cameras with lenses with f-numbers that start higher than f/2.8. For example the RX100's lens starts at f/1.8 and goes to f/4.9. f/1.8 is nice and bright at the wide end.