Here are two different links that talk of when to go and how difficult the hiking can be. My perspective is this: We hiked in to the lower lake and made a base camp not far from the trail and not far from the lake as so we could filter water and fish till dark. During the day we would hike to the surrounding lakes for more fishing and for vistas.
The upper and lower red castle lakes were better fishing and better scenery while the other side of the castle formations were more fun to explore and there is a pass there that is not too hard to get up too, and you can see the entire other side of the pass and valleys that lie there. The view was not as scenic and the side we camped on was worth checking out. The lower lake is about 8 miles from the trailhead if I remember correctly so the hike can be done in and out in a day if you hurry.
I recommend camping at the lower lake and taking at least a day to check out the other areas. If one is to go at the right time of year. We missed it on this trip, but were right on the other times I’ve been, the hike to the lake features plentiful amounts of moose. One trip we counted 35, it was neat. The trail passes through China Meadows and this is the hot spot for moose to hang out in. I can’t remember the time of year that we saw them all, but I’m sure the included links have better information than I can provide. For those not to avid on hiking lots, there is a campground at the trail head and China Meadows is only a couple of miles from there. This is a protected wilderness area so the only way in is by foot or horseback.