Carson's time at DA

Just caught this in season 6 episode 2. There's reference to Carson being "50 years at downton as a boy and man." If it's 1925 in this episode, 50 years past would be 1875. How young did hall boys start in this era in england? Daisy started service at 13. Or might he have started as early as 6 or 7, being his father was a butler? If this has been on a thread already, I must have missed it.