I think if being scattered feels right to you, then you should go with it! I gave up trying to become a “specialist” in one thing a long time ago.
I realized that it’s just not me. I’m a generalist. My strength comes from knowing a little about a lot of different things. Maybe that means I cannot become a world-renown programmer or a world-class athlete, but I can become a programmer who is also an athlete. My strength comes from having exposure to both and therefore having a perspective that cannot be provided by someone who only does programming or just focuses on being an athlete.
In short, I say embrace what you feel true to yourself and only use the opinions of others as things to consider during your journey. Own your path, don’t borrow it. 🙂