Everything You Need To Know About Vertex Masking in UE5

On Patreon, some of you guys messaged me asking if I could create a tutorial on the best method to implement a material system where we can mask specific color data (for example, red) and apply animated materials to certain vertex colors. A real-life usage scenario would be having a vegetation atlas where you don’t want trunks to move, want leaves to move heavily, and have the grass on the surface move lightly, all using the same material. So, I decided to write this tutorial for you. The logic discussed here can be applied to moving water, tents, or any other moving surface that can use a vertex painting mask input (e.g., Simple Layer Water Material, SimpleGrassWind, etc.).

But why would I need to use vertex painting on my meshes? Wouldn’t it be simpler to create multiple materials and assign them to the corresponding meshes?

Simpler doesn’t always mean more performant. In some of my older tutorials, I talked about texture atlases and trimsheets. The reason we rely on such methods is that we aim to create as few materials as possible when making our assets. Why? Because every material we create and assign comes with a draw call, which is one of the most important aspects of any video game when it comes to achieving great performance. If you’re interested in optimization methods in UE5, let me know on Patreon so I can create a dedicated tutorial for it. But in this tutorial, I’ll be discussing vertex masking methods in Unreal.

Let’s Begin Vertex Painting Our Mesh

    Before we begin:

  • Switch to ‘Face Orientation’ mode in Blender and ensure that your normals are facing the correct direction. This is extremely important, as incorrect normals can cause strange behavior when the material is animated in UE5.

I have a ‘space pool’ asset in this area, and I created some cubes where I’ll apply a water texture and use the ‘SimpleGrassWind’ node to animate the water. Later, I will configure Unreal to animate the green-marked areas while leaving the red-painted areas untouched.

I have isolated my cubes (the ones to which I applied my water texture) and painted them green in ‘Vertex Paint’ mode:

Lazy Loaded Image

Then, I painted the parts that I don’t want to animate in red.

Lazy Loaded Image

Pure Red, Green, and Blue Hex Codes

    Here are the hex code references for pure red, green, and blue to paint your vertices with:

  • Red: #FF0000 (RGB = 1, 0, 0)
  • Green: #00FF00 (RGB = 0, 1, 0)
  • Blue: #0000FF (RGB = 0, 0, 1)

Important Node: In this case, if something is pure green, the animation will be fully applied to it. The shades of green will determine the intensity of the effect. For example, if you paint your mesh a color that falls between red and green, that vertex-painted area won’t be affected by the ‘SimpleGrassWind’ node as much as the vertex painted pure green.

Troubleshooting: If you can’t paint in the vertex paint node, make sure that your normals are facing the correct direction.

Lazy Loaded Image

Your advertisement can be placed here for $200 permanently. Suggested resolutions: 1800x663 or 1400x1400. Contact us.

Is there a ‘fill color’ option in vertex paint mode? Manually painting takes a lot of time!

There isn’t an option called ‘Fill Color,’ but such functionality exists and is a bit hidden in Blender. For example, I want to paint some cubes pure green:

Switch to ‘Edit Mode’ and select the vertices you wish to fill with a color:

Lazy Loaded Image

Then switch to ‘Vertex Painting’ mode and activate the ‘Vertex Selection’ tool:

Lazy Loaded Image

Set the color you want to use for vertex painting, then select ‘Paint > Set Vertex Colors’:

Lazy Loaded Image

Result:

Lazy Loaded Image

Important note: When you select multiple unconnected meshes and try to implement this method, it works only for the first selected mesh. I thought this limitation was worth mentioning!

Blender Exporting, Unreal Importing, Limitations, and Moving Material Implementation

  • When you’re exporting your mesh from Blender, make sure to select the ‘Linear’ option from the ‘Vertex Colors’ menu, as we don’t want this color information to be gamma corrected (which changes the shades of the colors).
Lazy Loaded Image

  • If you have already imported your mesh into Unreal and use the ‘Reimport’ or ‘Reimport with a new file’ option to replace your mesh with a new one, you’ll notice that your vertex color information is not imported. I am not sure if this is a bug or an engine limitation, but the asset must be completely removed from Unreal and then imported manually/from scratch.
  • When importing your mesh into Unreal, set the ‘Vertex Color Import Option’ to ‘Replace’ (underneath ‘Advanced’ category) in the ‘FBX Import Options ’ that pop up. By default, it may be set to ‘Ignore’ or ‘None.’

Troubleshooting

How to display the vertex colors of a mesh in UE5?

To ensure that the vertex color information of your mesh is imported correctly, you can double-click on your mesh inside Unreal’s Content Browser, then in the upcoming window, can click on ‘Show’ and then ‘Vert Colors’ option:

Lazy Loaded Image

Some vertex-painted parts of my mesh display incorrect color info in Unreal Engine 5, even though everything looks as it should in Blender!

    This is usually caused by one of the following things:

  • Please ensure that the normals are facing the correct direction.
  • Sometimes Blender creates two color attributes when you vertex paint. Make sure the exported vertex colors are displayed correctly here:
Lazy Loaded Image

Vertex Masking in Unreal

I’ll be using an example from one of my old projects to show you how the masking is implemented. To mask the color, you need to take the Vertex Color node (I want to animate the green parts), multiply it with the ‘SimpleGrassWind’ node (or a similar one), and connect it to the ‘World Position Offset’ pin of the master shader node.

Lazy Loaded Image

Conclusion

I believe this tutorial should answer all of the questions you might have. However, if you encounter any problems, please feel free to contact me. If you have suggestions for more tutorials or would like to see more from me (I try to write as detailed tutorials as possible, and it takes quite a lot of time, so I’d appreciate your support), you can reach out to me on Patreon. Until then, happy game developing!