Does it really pay to build a tight house if I like to have my windows open any time it's between 62-92 degrees? I'm not a fan of AC unless it's very humid, then I just want it for sleeping.

I don't know where you live, but what about when its 12 degrees outside? A tight house for you would simply mean that it would cost you less to heat or cool your home when you did want the windows closed.
A tight house with a good ERV system could help eliminate your need for AC.

It always makes sense to build a tight house if at any time you have to heat or cool it. Plus your house will live on after you are there and other people might have different cooling requirements. Don't put another inefficient house in the world. Over the lifetime of a house the energy savings are HUGE.

I suppose if you live in one of a few areas on the California coast that you never really need heat or A/C, then you could build as leaky a house as you want.

Also a less insulated house may get hotter, in hot weather, than outside air temperature even with the windows open.

If you have a constant temperature of 68F- 75F, perfect humidity of 55% (anything higher than 65% will grow mold and mildew), have a little wind for drafts, no natural disasters and basically no pollens outside, you should go ahead and build as cheap as possible. A structure with a tent covering. But we all know Paradise does not exist, even though it is advertised everywhere.

I would build as green as possible, taking energy usage into consideration. If you can't afford a 2500sqf house because you want/need to implement green building with a price tag attached that would only allow you to build a 2100sqf house, imho, I would build the 2100sqf house with all the "green".

When I asked "is it worth it", I did not mean financially. Once the temperature is over 65 degrees, my windows are open. Pollen is no concern, as I'm not allergic. We had no central air conditioning in the house I grew up in. The windows were open in summer no matter what the humidity, except in the bedrooms which had window air conditioners which were used only when it was too hot and humid to sleep. Mold did not seem to be a problem, outside of the grout in the shower.
What I was wondering about is if I'm going to have my windows open 3 seasons of the year, does it make all that much difference, since opening the windows defeats the purpose of building tight.

hmmm....I think you have answered your question with your last sentence.

Telling us where you are located (climate) would help understand the need (or lack of) for "tight" building the fourth season.

I am always amazed at the number of posts asking for forum member's opinions, without giving the most basic info.
Green building is VERY climate dependent.  No one would recommend the same strategy in Florida and Alaska.
If you want informed opinions, provide the info needed.

NY state, Hudson Valley.

Ah, yes.  You have a real "heating" season.  Built tight for the heating season and any future owners.

"Build tight, ventilate right"

I don't know if it pays to build tight, but it certainly costs to build leaky. Durability, health, energy efficiency, and comfort are all thrown away in a leaky house.   When you close the windows during the heating season, do you want your 'fresh air' coming through leaks (and filtered through dead squirrels) or do you want fresh air coming from a planned pathway delivering air that's clean and healthful to breathe? And there's always the point that heating costs money and leaky houses waste a lot of heat. Air transports more moisture than diffusion does, so you can create moisture problems in insulated but leaky houses. Moisture can cause mold and rot which ruins indoor air quality and causes maintenance headaches.

I was in a discussion about Passive Houses in the US the other day with some of the editors at Fine Homebuilding; PH are extremely tight and well insulated. So much so that they don't need heating or cooling systems. One of the editors asked what you do if the power goes out and the mechanical ventilation doesn't work? "Open the window."

Sometimes the simplest concepts are hard to grasp because they're foreign.
Like Rich said, Build tight, ventilate right.