"When you think about the time you get with your kids and put it in that finite focus of summers instead of years, you feel the weight of it," he says Tricia Despres is a writer for PEOPLE digital, ...