When the topic of American slavery is brought up, some people will say that you cannot apply modern ethics to historical figures. How common were anti-slavery beliefs before abolition?

I know it’s not always fair to judge historical figures by our standards today, but surely there were plenty of abolitionists before 1865. When certain people say that you can’t judge the founding fathers for slavery, I find it silly. It’s like saying you can’t judge people for being pro-segregation when it was legal; sure, it was normalized, but there were plenty of people that disagreed with it. Is it irrational to judge slave owners, just because there were abolitionists at the time? What percentage of the population was against slavery throughout American history? Hopefully I’m making sense.