The Smith boys were lucky to share their childhood with the Peters Brothers