I know what you're saying but it's less about the person taking the shot and more about the quality of chances that a team are creating.
If a team is creating a high xG score but the chances are falling to the wrong players, then that's probably indicative of something - players in the wrong positions, poor choices, etc. Similarly, if a team is overperforming on xG it maybe points towards players who are exceptionally good or a team who can't maintain their goal scoring. In short, it's used to analyse other things.
Like any other statistic, you need other numbers to make sense of it. No one looks at total number of shots or possession and thinks of it as being a definitive statistic. Expected goals is the same.