The Keepers of the House by Shirley Ann Grau
The Howland family have lived on the old homestead since the early 1800's. In 1965 the current occupant is Abigail. Her grandfather, William, had brought home a young black woman named Margaret to help take care of Abigail. But Margaret also became his mistress and had children by him.
Abigail's husband is a budding politician. So William and Margaret's secret is a devastating blow to his campaign. And to the community at large.
"The Keepers of the House" is a powerful book about race relations and hyprococy in the south. It asks the question "Do you say what you mean or say what you think people want to hear." It's pertinant in todays PC dominated world.