When I draw 3D maps using Bryce and Photoshop, I deliberately don't plan them out too well. I have a vague idea of where I'd like things to go and let the twists and vagaries of fate push my maps in new and interesting directions.
For this map, I open up a new image in Photoshop 2048 x 2048 pixels (since that's the Bryce scale for the terrain I'll be using).
The first thing I do is draw a black circle on a black background.
Specifically for this, I've added a thin outer glow (about 20 pixels) around the outside of the circle and a thicker inner glow (about 50 pixels) on the inside of the circle. Both of these glows are done with white. The circle is duplicated and applied as a screened layer directly over itself. The duplicated version of the circle has its outer glow inner glow expanded to 250.
When converted to terrain this will vaguely give me a dish shape...flat in the middle with gentle slopes that get steeper toward the outer rim. The circular "mountain range" around the world will then sharply drop away.
Next we need to add a bit of interest to the terrain inside the circle.
I do this by building up a few layers. This is done with a 50% application of the airbrush tool, each time applying modification to a new layer. For each of these examples, I've kept the circle layers intact underneath and have just shown the specific layer I'm working with superimposed over the top.
Layer 1: First I have to idea that I want a range of mountains that leads to the central part of the world (where the reflective sun mirror is). That range splits off from a longer range over to the "west". The western range has some shadowy lands behind it that might be obscured from the sunlight. Over to the east, to counterbalance things, I throw in another range.
Layer 2: Next I consider where I vaguely want water to go. I know full well that the water won't exactly match these patterns I'm drawing up, but I don't care. Dark spots are low, that's where the water lies. I build up shapes with the airbrush tool, I cut them away with the eraser. It doesn't matter that I go out of the lines, I can clean that up later. I reinforce the idea that the central part of the world is higher by making sure that there are no water areas there.
Layer 3: With a much wider airbrush (600 pixels across), I get a few more sweeping motions across the world to build up more general forms. I keep following the vague motions that have been laid down in previous layers. The world is generally taking shape.
Now comes the trick...
Layer 4: Photoshop has a pair of tools called "Clouds" and "Difference Clouds". The first creates amottled pattern across the layer where it is applied, the second uses an algoritm to apply clouds on top of clouds a depict the differeces between the two layers. Applying "Difference Clouds" once over a cloud layer produces the effect shown above, applying it many times gives a more complicated pattern.
These layers can be compiled on top of one another, turning them all on, but making them only 30% opaque. You can experiment with "screening" the layers or leaving them as "normal".
You end up with something like this...
It's getting pretty close to the final terrain map that we'll be using.
Next you can either manually erase everything outside the circle, or duplicate the bottom-most circle layer, change the circle colour from black to white and bring it to the top-most layer. Then multiply the layer over everything else. This has the effect of blacking everything outside the circle while leaving the interesting terrain texture in the middle.
Once this image is exported across to Bryce and converted to terrain, you can see how the shapes of the ranges fall across the landscape...then pull it back into Photoshop to erase away parts of the landscape or build them up to get a shape closer to what you had in mind.