I've got an engineered floor layout (OpenJoist 2000) for house I'm getting ready to build and I'm a bit confused as to what type of design options I have (if any) in the particulars of the floor layout.

Specifically there is a complicated area of the layout to accommodate an open stair and railing system from the main floor to the second floor and then to the attic/loft.  Part of the cut-out is such that a post will be required to support a floor corner.

In the floor system design there are 3 LVL beams that support around the cut-out which are are used to hang the joists on.  What I'd like to do is have the LVLs dropped below the floor with the joists resting on top.  This would allow me to expose the LVLs (I would replace with finished GluLams instead) and would allow me to get rid of the post by cantilevering the floor joists over the lowered beam.

So the question is whether this type of option is even available with an engineered floor system, or am I really now looking for a custom floor system?

When you say 'I've got", do you mean you have already purchased the materials or just had the layout done, and are ready to purchase?.
If you do not already own the materials, then your supplier should able to reconfigure the layout to meet your needs. If you own the materials, then the process may or may not be possible. I am not up to speed on the Open Joist system, but cantilevering requires special bracing at the load points and each system has its own set of criteria for length of cantilever.
First check with your supplier for the system specs.

Right now I've just got the initial layout, have not purchased material yet.  Cantilever length is just under 3 feet past the beam on which it is cantilevered.  I had a quick look at the framing guidelines for OJ2000, they show a scenario where the joists are cantilevered over a bearing wall and have extra bracing over the webs above the bearing wall.

There is also another scenario shown where you simply create the cantilevered platform with dimensioned lumber attached to the joist webs, such that the joists themselves would stop at the bearing wall (beam).

Maximum beam span would be 19 feet, which has already been designed into the floor system.  All I'm looking to do is lower them below the joists.

Go back and discuss the layout with whomever did it. There are many instances where LVLs can be used to carry the stair header and eliminate the post. Although you have lvls in your design, can't tell how they are using them? And why cantilever 3' when it sounds like the stair header opening would be at this 3' position?

Without a pic, can't go further than that, but I will say it is not difficult to eliminate the post as long as someone works thru the floors to be sute that each stair opening on EACH floor is self-supporting.

Yeah, hard to explain in words...  I've attached 2 pictures, one of the stair layout and the other with the floor layout I am thinking to solve it.

If i understand your pic correctly, the cantilever should be not problem. I would have the floor designer run with the cantilever as shown. I have done this many times with floor trusses and I joists, but have never used oj 2000's, but shouldn't be a problem at all. I like to try and eliminate the posts whenver possible, this gives much more freedom.

Good Luck

Bunt

What supports your platform tread and weight of your 6R stringers?

The stair stringer will be steel with welded joints (open stair treads).  It can attach on the right side against the beam that is part of the wall, but will also be supported in large part by the cantilevered platform...  at least that is the plan.