The idea that somebody who you don't trust has your child with them is a terrifying thought,” the mother of one tells PEOPLE ...