Ultra-efficient tree texturing and modeling

Software used:
-Maya
-Photoshop

Things I assume you know how to do:

-Intermediate Maya knowledge

-Intermediate Photoshop knowledge

-Basic rendering abilities

-----------------------------------------------------------------------------

Leaves

Create a single square card to which you can apply a single leaf texture:

Create natural clusterings that can be attached and moved and rotated and combined to make full-sized branches:

Here are some examples of finished branches:

Sprigs and groups of leaves can achieve different results than single leaves:

Here are many different branch examples where the only thing that has been changed is the texture applied:

Some examples of leaf textures used in the above samples:

Do top-view TGA Production-quality renderings of these clusters at very high resolution (so that each leaf cluster gets around 1500-2000px sq. devoted to it. Use the same method that you used previously to pull them from the background - load the selection from the alpha channel, contract selection 1 pixel, then paste 15 layers onto the same blank canvas, merge these layers, and save. You now have the branch fully opaque and isolated.

Do the same with the arrangement of leaves as you did with the branches - try to leave enough space to allow for triangular or quad cards:

Also - try to pick clusters that will be usable in more than one way, with more than one card.

In Photoshop, paint the most efficient card outlines you can draw around the leaf clusters. Ideally, they will be 3 or 4 sided. Overlapping is fine (ideal) in the non-opaque areas:

Save this as a JPG and apply it to a perfect square in Maya. Duplicate the square (or create new ones) and float them above this "guide." In "Top" view mode, move their verts around until they match the guide you drew. When all are done (you may need to slide some straight down, beneath the guide, to keep the guide visible if there is a lot of overlapping) attach them, go into "Face" mode, and planar project the entire group.

Go into your UV space and scale/translate the UV data until it perfectly lines up with the lines on the guide. This should be very easy to do precisely.

Separate the cards now that they are properly UV'd, apply the proper Kona Shader texture to them (so they will be properly masked) and line them up, properly oriented:

Now, assemble the leaf cards together into clusters that look natural, and varied:

Branches

I've found that the best method for texturing trees in our game is to use two texture pages, ideally 1024 pixels in dimension, though reducing these to 512 is not out of the question.

Use the Maya Paint Effects tool to create the textures for the 2D branch cards. Start with the Birch Medium brush from the Trees brush folder in the Visor, and immediately "Save Brush Preset..." (found when in "Rendering" drop down, under the Paint Effects tab) and make your own custom brushes, save to the visor:

Choose your custom brush, as seen here:

Apply your own bark texture to the slot in the texturing setting (as seen in the following settings):

Everything in green is stuff you will find useful. The rest I didn't even touch. Ignore the settings that pertain to the leaves - I included them in the screenshot because it is valuable information for general Paint Effects use, but for SOE_Project trees we are doing leaves on separate cards so that we can have better control over bark coverage.

Under "Growth" uncheck leaves and twigs. Only branches should be selected.

Set the Paint Effects canvas size to 2048x2048 (the maximum) and make the background color 50% grey. The reasoning behind the color is to prevent a halo from coming with the branch when you pull it off of the background - choose whatever color is most appropriate if 50% grey doesn't suit your needs to this effect.

In Maya Paint Effects, paint one or two branches, sizing up the brush so that any given branch just about fills the canvas. Go to Save Image and save each nice looking branch as a .TGA - this will come with a great alpha channel. To get some variety in the branches you create in Paint Effects, change the values a bit that pertain to number of branches or thickness or length.

In photoshop, CTRL-Click the layer thumbnail of the alpha channel and go back to the RGB view, you will see that the branch is selected based on the alpha channel selection you just click-loaded. Contract your selection 1 pixel, and "cut" the branch off of the BG. Now create a new document in photoshop, and paste the branch from the clipboard over and over until the feathered, antialiased edges appear solid (paste about 15 times to achieve this.)

Save this PSD and do the next branch the same way until you have 6 or so nice branches, nicely isolated on their own layers.

When you have assembled your master PSD, merge the branch layers together and place them into your texture page. By keeping them separate from the background until now, it will allow you to create a perfect alpha channel for them in the "OpacityMask" folder - all you will need to do it CTRL-click your branch layer and "fill" the selection with white in your "OpacityMask" folder (onto a black background of course).

Your PSD folders should use textures like this:

When assembling your branch page, try to leave enough space around each branch so that your poly cards don't need to be more than 3 or 4 sided:

Apply your branch texture to cards the same way that you do with the leaves (in the leaf section, above), by floating cards above a guide with lines drawn over it representing the most efficient use of polygonal cards. Keeping them low-poly will allow you to make more granular, organic branch assemblies, because you can use many smaller cards assembled at slightly different angles, instead of one big 2D card. This also allows you to avoid large ugly low pixel-density branches.

Assemble branches in multiple parts so that the branch does not become invisible from any one angle:

Place enough 2D branches onto your solid trunk to have a nice "skeleton" of a tree. It will not be as fine and granular as a bare tree in real life looks, but all we need is to be able to see hints of branch bark peeking through the foliage here and there. An example of an assembled "naked" tree:

Assembly

Begin to place the leaf clusters that you have assembled onto the branches. The process of "leafing" a tree will take a few stages, but first and foremost, you want to cover up 90% of the branches with clusters so that you can only see hints of bark here and there when looking at the tree from ground level:

Once the branches are well covered in leaf clusters, you can begin to fill the remaining negative space. Frequently rotate around in the viewport to make sure that you are not leaving any awkward gaps or holes. Try to leave the appropriate amount of negative space for the particular species you are trying to emulate:

It may be easier to do by scaling the clusters down (keep them separate until the very end so that you can do scaling to "thicken" or "thin" the tree) to 30% size or something small, just so that you can eyeball the evenness of the distribution better:

Once you feel as though the distribution between the branches looks even, scale them back up:

Finally, once the tree is looking as you intend it to, you need to make all of the normals face the same direction. This allows it to vert-light uniformally. Select your canopy (the entire tree minus the trunk and 2D branches)

Go to: Display>Polygons>Backface Culling - this will make your faces whose normals are facing away from the camera disappear.

Then, go to an angle in the viewport that bisects the cross-quads at a perfect 45 degree angle.

Go into "face mode" and marquee-elect your leaf faces (I find it easiest to keep the canopy separate from the trunk until the very end so that I can make adjustments like this) - this will select all faces whose normals are facing the camera, and leave the other faces alone.

Now hit Normals>Reverse and the entire canopy will have normals all facing the same 2 directions.

See the following examples: these are the same tree, but one has had its normals reoriented to face the same direction, and the other has normals that were un-adjusted:

As always, I hope this was helpful in your 3D modeling endeavors.

-Andy Zibits