World Building - tougher than it sounds?

Every time I sit down to world build, I get frustrated: All I want to do is rush off and work on the PLOT of the story. I know how things are imagined in MY head. Why not let the reader imagine things for themselves, too?

I had planned to spend all of July building up the setting of my work in progress, and suddenly found myself almost two thirds of the way through the month without having done much. I finally gave up and decided I was going to rush ahead and make up the plot.

Within the first five minutes of plotting, I wanted to write a scene where one of my characters exclaims "By Jove!"

While I never intended to leave the word Jove in my novel, it did raise a question.
Jove is a derivative form of swearing by god's name. I asked myself, "What would my characters swear by? What would they bless by? Would they do the traditional Christian thing and say 'Thank Goodness!' even though they're on an alternate version of Earth?"

Suddenly I was world building and didn't even realize it. Funny how the right question will unlock a wealth of information.

I went from making up swear words and friendly exclamations to inventing what sorts of religions these people might follow. Soon after that I was drawing a map of my world just so I could point out to myself where all the religious places were.
Then I felt compelled to name all the new countries I had just drawn (or at least name them something other than colors of the rainbow which I had temporarily bestowed upon them).
Coming up with names for countries led me to realize that country names in our world today are from ancient words meaning things like "Horse people" or "People of the white mountain" (this is my simplification of it).
So I came up with a general genetics map of who has black hair, red hair, white hair and where they live, and which areas are more mixed than others. I also named the countries after things that they're known for, like industry, desert-land, and bountiful harvests. Plugging these words into Google translator and choosing a different language until I got something I liked was tons of fun.

By the end of the first day of me "giving up" on world building, I had a map of my entire world, names for all my countries, some slang words and their etymologies, I knew where everyone lived, what they believe and how they practice it, and, of course, which gods they swear by. I knew the political tensions between the countries, what certain places war over, what the weather is like in different parts on my map, and I had even created three different mythologies which a few characters in my story will draw upon.

I don't know how I could have ever found World Building hard before. Before I started asking questions, (ok, so just the one question) I wasn't interested. I didn't think there was enough to my world to talk about. I didn't want to bore my readers. But now, because I got curious and asked a question, I have a wealth of information.
Not all of it, or even most of it, will make it into my book, but it IS useful. It helps me imagine what I need to in order to build the world around my main character.

So here's to being curious, even about imaginary places!