Here are some curve radius guidelines based on the lengths of your longest pieces of rolling stock.
2X - Some model equipment may be able to track reliably on 2X their length, but this is generally considered pushing it.
3X - Making your curve radius at least 3X the length of your longest cars gets reliable tracking around curves, but looks toylike.
4X - If you make your curve radius at least 4X, your longest cars will look much better on curves.
5X - If you make your curve radius at least 5X, your longest cars will couple easily with minimal manual fiddling of the couplers.
This measurement is based on the length of your longest car (coupler to coupler).
For example, an HO forty foot box car is about 6" long, coupler to coupler. Here are the curve radius guidelines if your longest cars are forty footers:
FORTY FOOT CARS
2X - 12"
3X - 18"
4X - 24"
5X - 30"
If you go below an 18" radius, your cars may not track well on curves, although some equipment may still track okay down to a 12" radius. You need at least a 24" radius for your cars to not look toy-like on curves. To have reliable coupling on curves, you need a 30" radius, which would be a good minumum radius standard for a yard on a curve, for example.
However, few will have only forty foot cars as the longest cars on their layout. Here are some guidelines in HO for other more common "longest" car lengths:
FIFTY FOOT CARS
2X - 15"
3X - 22.5"
4X - 30"
5X - 37.5"
SIXTY FOOT CARS
2X - 18"
3X - 27"
4X - 36"
5X - 45"
EIGHTY FOOT CARS
2X - 24"
3X - 36"
4X - 48"
5X - 60"
So to summarize, some cars or locos may track okay at 2X, but at 3X cars of that length will all track well, use 4X if you want things to look decent, and use 5X if you want reliable coupling on curves (like say a yard on a curve).
Here's another trick you can take advantage of, too. Curves, when viewed from the inside, don't look as sharp. So for inside curves, you can drop the radius to 3.5x instead of 4x (say 27" instead of 30" for 50-foot equipment) and the curve will still look pretty good.
But outside curves (like at the end of a peninsula) look sharp and the equipment doesn't look as good, so the curves there should be as broad as possible. If you can manage it, you should use the 4x rule (at least) for those curves.
If all the cars are of similar length, you can drop to perhaps 2.5x, but if you have a mix of longer and shorter cars, you'd better go at least 3x of the longer equipment if you don't want tracking problems, or 5x of your longest equipment if you want the most reliable coupling. Widening the gauge to maximum may help with rigid wheelbase steam locos.
You may also be able to do 3x on longer equipment and get reasonable coupler performance if the couplers have been altered to allow them to swivel at least 50% of the width of the car.
To be absolutely sure what works best with your equipment, do some testing on your own, using these guidelines as a starting point.