Does going to a prestigious university matter? ?

The name of your university loses importance as soon as you get your first job. After that, employers care more about what you've done on the job than what school you attended. If you are unhappy where you are, transfer. That's more important in the long run.