The Silmarillion is certainly more for readers who want to know more. However, you can enter the legendarium from there. That's what I did, actually.

There's a reason why it feels the way you describe it: it was never meant to be read in the format it is. The Silmarillion is basically a gigantic synopsis, working notes, if you will.

The Children of Húrin is a perfect example. When we read it in The Silmarillion, we just see the skeleton of the story, one that Tolkien wrote to see whether the story worked as itself and as part of the larger legendarium. When we read the fragment of the poem he actually wrote, it is a completely different feeling.

Unfortunately, most readers don't know that the Silmarillion is a synopsis and not a novel, and they judge it as if it were a novel.