what are the downsides of plastic or metal insulated panels?

I'd think this would be an affordable way for a modern boxy simpler structure with good insulation value. Iron frame with the metal sandwich panels.

Metal might rust and is easily dented if a protective coating is not applied. Perhaps if you want to use stucco or other materials it might be harder to apply? (not sure since I've never seen it done yet on this type of SIP). Plastics might degrade due to heat. As with all plastics, the concern is leaching. However if hardiboard or similar rock style siding is applied properly exposure to the sun is limited. Just look at commercial cold storage food facilities. Plenty of examples can be found commercially.

The other thing with type of SIP is that if you're planning plumbing or electrical, chases may have to be cut at OEM level. Building codes vary so if you're looking to have externally mounted piping and wires. It's a very inexpensive choice. Did I just drive the price of OSB down? hehehe!

aziansenzation

We use steel sips that are galvalume skins so there is no fear of rusting. You need to be careful obtaing refrigeration panels like zeroloc as they may not be suitable for structural applications and only for cladding.
You need to buy from a supplier that sells specifically for structural capabilities www.SipDepot.com
We replaced a residential roof on a house when the home owner did not verify span capabilities of the panel and found out too late. that they were only rated at 8ft. and were spanning 20 ft.
Stucco lath goes to metal just as easy as frame
There is no leaching with EPS
Cement board siding ad attched per mfg. spec with screws
With any sips its good to plan them on interior walls if possible
Steel SIPs are not chased for wiring they are furred out with metal Hi-Hat furring
No you did not drive down the price of OSB as steel SIPs are slightly higher

Even when skinned a puncture say a (metal screw used for mounting) exposes the metal causing a thermal break. And of course the screws itself are good conductors and may result in condensation formation.  So as to the possibility, yes there is because I've seen it in commercial plants where there's high degree of "abuse". As not to advertise any specific retailer I just chose panels that are easily obtainable here locally, which is ZeroLoc. And they are for structure. The logistics for me will be just too expensive to ship from sipdepot in Florida let alone pass canadian regulations/tariff as to importation rules.

As for leaching, I meant the leaching from the plastic sheathing and not from the urethane or EPS. The health concern is from bisphenol A or other harmful chemicals from plastic outgassing when the sun hits any type of plasticized material. (not only walls but toys, food containers etc.).

Don't get me wrong, I like metal SIPs.

And of course you already understood the OSB price joke ;)