Why does everyone look to me? Don't I have a right to my time and future, too?

No one is born into slavery these days and that includes yourself. If you really don't want to be everyone's caregiver both in present and in future, you don't have to be.
It may require some hard decisions on your part that will not be easy for you to make or easy for your family to accept. In fact, they may not accept it at all and it could end with your family being estranged from you. Estrangement is better than slavery. It's better than sacrificing your life indefinitely and your son's as well because you can't plan any kind of a life.
I understand when people become caregivers out of love and respect or for the sake of financial and material benefit. Or for all the reasons. People's reasons are their own, but in your case it sounds like things have gone too far. You and your son have to establish boundaries with the people you take care of. If those people will not or cannot respect those boundaries you and your son will have to move out and leave them behind.
Speak plainly but kindly to the family member who thinks you will bring up her grandkids. Tell her that you will not be doing it. Those kids did not come from eggs that hatched in a nest one day. It took two people to bring them about into creation and those two also came from people. Tell her to start looking into other branches of their family or families. Do not add an apology either because there's nothing to apologize for.
She might not take you seriously but in order for you to "get the kids" in the event that something happens to her, it will have to be in writing. In legal document that you sign before a lawyer and a witness agreeing that you'll become their custodial guardian. If you refuse to agree to it legally, you will not be responsible for those kids if something happens to her. If you are their only functioning relative the state may try to persuade you, but you don't have to accept.
Like I said sometimes hard decisions about caregiving can lead to estrangement. It's your life though. You don't get back the years sacrificed to caregiving. Think about that.

I can't imagine having the thought of taking care of kids with special needs hanging over your head is pleasant to live with.

I would have a talk with family member raising her special needs grand kids about her making arrangements now for these kids in the event she is incapacitated and cant take care of them.

Yes it sounds terrible for you to not take them if she cant do it but it is not wrong if you can't do it. Raising someone elses children is hard and with special needs even harder and a life long commitment even past 18 depending on their needs and limitations.

Better she knows now so she can make other arrangements forvthe care if the kids.

You do have a right to have a life and happiness too. It's not all about everyone else and what they want.

You're not overreacting at all, you probably have burnout. One of the frequent responders on this forum says that elders won't look for another solution if you continue to be their only solution. Your child has priority. You have priority. Your LOs have assumed you into caregiving and you've allowed it. No one if forcing you to do it, right? They have other options -- it's just that they don't want to go through the hassle or discomfort and uncertainty of change or pay for it or manage it. Like other people do. All the time. And, let's be honest: you don't like the feeling of guilt, even though backing away is not doing anything wrong or immoral.

Your LOs are not in any unique needs situation. Start researching alternative help for your people. Contact your local area's Agency on Aging, caregiving agencies, social services for their county, etc. If they have financial resources maybe take them to nice care facilities to tour. Wanting to stay living on a farm that they can't maintain themselves but then expecting someone else to take on that burden is what's immoral.

You seem to struggle knowing where your boundaries are. I suggest talking to a therapist who would be objective and can help you identify your life goals, define clear, strong personal boundaries and learn how to defend them so that you can stop being a voluntary doormat. The woman with the special needs grandkid? Tell her, sorry -- you are retiring from the caregiving world. She'll get over it, and then she'll find someone else to assume into that role. So will the other 3. Your child especially needs to have boundaries modeled so that they aren't taken advantage of by others or make poor decisions that have negative impact. Talking to therapist will be worth every penny. Some work on a sliding fee.

Your 3 people are full grown adults who are totally able to come to grips with a change in their care plan. You are not responsible for their happiness. Your child and you are priority. Keep repeating this to yourself. Blessings!