What would do a better job with sound deadening, R-13 fiberglass batts, or 3" open cell foam?

Firstly, I did a quick search on the r value of open cell foam and it seems to be similar to fiberglass whereas closed cell is just under twice the r value of open cell. Due to that I am not sure how much difference you would see between open cell and fiberglass. Sound travels via vibration and insulation does help reduce the transmitted vibrations. Low frequencies are very hard to deaden. Real soundproofing (like in studios) is often created by having two walls with an airspace between them (both insulated). If you are trying to eliminate sound through a single wall, the best you can do is to try to get the highest r value possible.

R-value has no bearing at all on sound transmission, but when all else is equal, more higher density gives better results. (R15 batts beat R13 batts by a hair due to the higher density.)

2lb foam is higher R and 4x as dense as open cell foam but not exactly "equal"- it's significantly more mechanically rigid than half-pound foam- don't count on the mass effect canceling out the better vibe-transimission of the more rigid material.

But to answer the question directly.

In a 2x4 studwall, half-inch gypsum both sides half pound foam has an STC of about 37.

Fiberglass batts in the same assembly yields an STC of 38 (slightly better- not much to write home about).

Blown cellulose gives you an STC of about 41 (better still.)

Metal stud 2x4 & blown cellulose gives an STC of about 45.

Sound abatement is a perennial subject on these forums. If you use the search function you can find a lot info on it from recent threads.