Sorry, but 30+ years of offensive, immoral, harmful behavior that has done immeasurable harm to the free software movement doesn't get a pass.
He was not removed for his opinions. He was removed for his decades of harmful immoral/amoral behavior that harmed the FSF and MIT.
That was the latest (final) in hundreds or thousands of documented cases of inappropriate public behavior.
For people who care about, support, or build Free, Libre, and Open Source Software (FLOSS).