It is this simple.
From your point-of-view, you have done one little thing. "I was only parking there for a minute." "I only blocked that door for a little while." "I only said something unkind once."

You are part of a collective and when you act within that collective from the privilege positions you have, it is not little.
Your one act may be drop in the ocean, but the ocean is real and I am drowning in it.
And that is why I am so angry.