Atlas layout is the process of fitting together all of the textures for that atlas so that they take up as little space as possible. This process should be mostly automatic, but it is not a solved problem, so occasionally a little tweaking here and there help to produce more optimal results.

To avoid the necessity of multiple sets of texture coordinates, all of the layers of an atlas (diffuse, normal, specular, etc.) are laid out identically. This may sometimes lead to some unused space in the non-diffuse textures.

There are two layout algorithms that can be used: one that works better when there are many textures of similar sizes and that works better when the textures are of widely varying sizes. Which one works best in a particular situation may require some experimentation.

When using separate texture and billboard atlases (that is, when “Layout:Merge atlases” is unchecked), layout for the texture atlas should be largely automatic. You don't need to specify a size for it because the Compiler will attempt to make the smallest possible size it can.

For the billboard atlas, however, you do specify the size. Then this sized atlas will be filled as much as possible with billboard textures that are sized relative to the size of the trees themselves (including any modification of tree or billboard size scalars). If “Layout:Billboards:Use extra space” is checked, then smaller billboards may be made bigger just to use up any wasted space in the atlas.

When the billboard and texture atlases are merged, the available properties change somewhat. You now specify the minimum size of the combined atlas. If the textures cannot be fit into this size, then it will be doubled until they can be. Once the textures fit, the billboards will be fit into the remaining space. Be sure to set the combined atlas size big enough so that the billboard textures get enough space to be of adequate resolution.