Slavery was a very poor economic model and many studies have determined that chattel slavery ended up costing southern society economically. In fact, it is one of the reasons, the South fell behind the rest of the country, which relied on "free" low-cost abundant immigrant labor.
Productivity per slave was low, and lifetime costs were high. Interestingly, after the Civil War when many slaves were now free and contracted with prior slave owners as "sharecroppers" productivity soared... and wealth in the South expanded.
Slavery became an institution baked into the culture of the South. And there were some wealthy individual plantation owners, but not that many. And collectively, slavery produced a relatively poor society.
As an interesting case study, let's compare the wealthiest slave owner in the South with the wealthiest industrialist in the North to get a feel for where the "real white math" was taking place.
The wealthiest slave owner whose last name was Duncan owned 2200 slaves. Adjusted for inflation his net worth is estimated to have been around $6.8 billion in today's dollars. That is a lot of money.
But compare that to the wealthiest industrialist of the 19th century, Andrew Carnegie, whose net worth adjusted for inflation is estimated at $418 billion. Sixty times higher than the wealthiest slaveowner.
Clearly, employing white immigrants was more profitable than owning slaves.
The wealth of America if anything was built on the backs of immigrants more than slaves. Slavery wed the South to a largely agricultural economy, while the endless supply of cheap labor in the form of immigrants and the rich natural resources of the upper Midwest fed an industrial economy that began to rival England. Industrialists also benefited from almost no regulation and low taxes. The result was a society that produced massive wealth in the North arguably more concentrated than in the South.
Slavery was a horrible institution. Sadly, the "elite" in the South enjoyed a "good life" at the expense of both slaves and poor whites and had the power to preserve the society that while it did not work in general, worked for the wealthy slaveowners.
Racism
Racism brings with it the same problems as slavery. I agree with the analysis by Citigroup that racism costs the economy.
I have never been an advocate of affirmative action as it is a form of racism. But as an executive, I made it crystal clear, that anyone that allows "race", gender, even gender identity over competency to influence their hiring decision would be fired.
When you put skin color ahead of skill, you are stealing from your employer by costing them the best person for the job.
Neither slavery nor racism makes economic sense, but nevertheless many people are influenced by their bigotry to make bad choices.