Writing Custom Shaders For Toolbag 3

Note: This tutorial is intended for users of Toolbag 3.03, which has some minor changes to the custom shader system and is currently in beta. Join the Toolbag 3 User Group on Facebook to try out the beta.

By Jeff Russell

Toolbag 3 provides a framework for users to customize the material shader system. This can be a powerful tool for users to extend Toolbag’s rendering capabilities, and for these improvements to be shared with others. The purpose of this article is to provide enough of an introduction to get a developer or technical artist started. Some basic knowledge of shader development and languages will be helpful; users looking to install custom shaders should see our Toolbag Add-On Library instead.

For those of you who might prefer a video introduction, our friend Chris Perrella has prepared a tutorial demonstrating the basics of custom shader creation and use. We suggest you give it a look before reading further:


Toolbag’s material shaders are located in the installation directory, inside data/shader/mat. User shaders belong in data/shader/mat/custom; any files placed there will appear in the shader dropdown in the “Custom” material module as the tutorial video illustrates. It’s worth exploring the contents of the mat directory as it contains source for the built-in shader modules in Toolbag, all of which are written in the same language and structure available to user shaders. Of particular note are mat.frag and state.frag, which will be discussed in more detail below. There are also two example shaders included with Toolbag in the custom directory, and these provide good examples of common shading operations in Toolbag.

Please note also that Toolbag’s console window (accessible through the Help -> Dev menu, or by pressing Ctrl + ~ on Windows and Cmd + Shift + C on Mac) will display any syntax errors during shader load or reload. This can be very helpful for development and debugging.


Marmoset Toolbag uses a somewhat customized shader language, which is a kind of union of HLSL and GLSL syntax conventions. In many cases, both sets of keywords work (e.g. both vec3 and float3 are valid), and in other cases custom macros are required. This syntax allows Toolbag shaders to compile and run in Direct3D, OpenGL, and more recently Apple’s Metal. If you follow these conventions, your shader should work correctly on a wide variety of operating systems and hardware.

One common example of this unique syntax, which may seem unusual to those of you used to HLSL, relates to textures. Textures are declared with the USE_TEXTURE2D macro, and are accessed with GLSL-like commands. Here is a simple example of texture usage in a Toolbag shader:


void myFunc( inout FragmentState s )
   vec4 val = texture2D( myTexture, s.vertexTexCoord );

A full description of the shading language is beyond the scope of this article. Reading through other files for examples is likely to be the most productive route to learning the ins and outs. Some basic reference for GLSL may also be helpful as Toolbag’s shading language most closely resembles this.


Marmoset material shaders are organized as a collection of subroutines for various predefined ‘slots’, all of which are called from the main shader file, mat.frag. Custom shaders may extend or replace any of these subroutines, and by doing so significantly alter the rendering behavior of the material. For example, a custom shader my supply a new “Surface” subroutine to alter the standard normal mapping functions. The following shader extends the Surface subroutine to adjust the normal to point in a different direction. This will affect all subsequent shading processing, including diffuse and specular lighting:

#include “../state.frag”

void CustomNormalThing( inout FragmentState s )
   //run the existing Surface function, if it is defined.
   //this allows regular normal mapping etc. to run first.

   #ifdef Surface

   //alter the surface normal to be messed up. whee!
   s.normal = -s.normal.yxz;

//define Surface as our new function
#ifdef Surface
   #undef Surface
#define Surface CustomNormalThing

Other material subroutines in Toolbag can be replaced and extended in similar fashion. Here is a full listing of the available subroutines, in the order in which they run:

Subroutine Description
Premerge Preparatory work at the start of a shader.
Surface Specifies the material surface; typically a normal mapping function.
Microsurface Sets gloss/roughness for shading.
Albedo Sets diffuse albedo/color.
Reflectivity Sets specular reflectivity.
DiffusionEnv Computes diffuse environment lighting.
Diffusion Computes diffuse lighting (called once per light).
ReflectionEnv Computes specular environment lighting.
Reflection Computes specular lighting (called once per light).
Occlusion Applies occlusion, typically ambient occlusion (called after lighting).
Emissive Emissive light, typically glow.
Transparency Any transparency work happens here.
Merge Performs the final merge of above values into the render color outputs.

All of these subroutines take a FragmentState structure by reference (hence the inout tag). This structure holds all intermediate values used by a Marmoset material shader, and is an excellent place to start reading. This structure is defined in state.frag, and several of its key members are listed here for reference:

FragmentState Variables
Type Name Purpose
vec3 vertexPosition Position in 3D space.
vec3 vertexEye Unit vector from the position to the camera.
float vertexEyeDistance Direct distance from the position to the camera.
vec2 vertexTexCoord Mesh texture coordinates.
vec2 vertexTexCoordSecondary Secondary mesh texture coordinates; often absent/unused.
vec4 vertexColor Mesh color; will be white if absent.
vec3 vertexNormal Mesh normal vector.
vec3 vertexTangent Mesh tangent vector.
vec3 vertexBitangent Mesh bitangent vector.
vec2 screenTexCoord Coordinates for screen position in [0,1]. A full-screen texture can be sampled with this.
float screenDepth Post-projection depth value. Not the same as vertexPosition.z.
uint sampleCoverage A bit mask for sample coverage; typically only used for voxelization pass.
int instanceID Instance number; typically only used for voxelization pass.
vec4 albedo Albedo or diffuse color in rgb, opacity optionally in alpha.
vec3 normal Surface normal direction as a unit vector.
float gloss Surface roughness as a scalar on [0,1].
vec3 reflectivity Colored specular reflectivity.
vec3 fresnel Fresnel strength.
vec3 diffuseLight Sum of all diffuse lighting.
vec3 specularLight Sum of all specular lighting.
vec3 emissiveLight Sum of all emissive lighting.
vec4 generic0…3 Four generic values for special use by some subroutines.
vec4 diffuseGI Global diffuse illumination in RGB, mask in A. Not always present.
vec4 specularGI Global specular illumination in RGB, mask in A. Not always present.
vec4 output0…7 Final render color outputs. Usually only modified in the Merge subroutine.

Some lighting functions, like Diffusion and Reflection, also receive a LightParams structure describing the current light being rendered. For a full definition of this parameter, read other/lightParams.frag.


Most custom shaders will need to pass constant parameters to control their operation, and most users will want to see these values exposed with a graphical interface. Fortunately, Toolbag provides exactly such a mechanism. Toolbag will automatically parse your custom shader code looking for ‘uniform’ variables declared in global scope, and create UI based on their type. All float, integer, vector, and scalar types are supported, along with textures. Shader authors may supply comments to give additional information about how the interface should be constructed. A quick example:

uniform float uVal;
uniform float uVal2; //name “My Parameter” min 0 max 3.141592 default 2.0

In this snippet, two float scalars are declared as uniform parameters, and both will appear in the UI. The difference is that the second one, tagged with formatters, will be displayed as “My Parameter” instead of just “uVal2”, will have a slider ranging from 0 to pi instead of 0 to 1, and will be given a default value of 2.0. Decorating uniform parameters with these formatting options makes for friendlier interfaces to custom shaders. A full list of formatting options follows:

Uniform Options
Directive Options Description
name “Name” Uses “Name” as the display name.
default v0 v1 … A default value for numerical parameters. May be a vector or single value.
min v0 v1 … A minimum value for numerical parameters. May be a vector or single value.
max v0 v1 … A maximum value for numerical parameters. May be a vector or single value.
color Displays 3D vectors as color pickers instead of text boxes.
bool This option presents integer parameters as checkboxes instead of sliders.
srgb Sets a texture to use the sRGB color space where possible.
labels “Name0”, “Name1”, … Displays an enumerated list of options for integer parameters instead of a slider. Must be the last element in the comment line.


Toolbag performs a number of passes while rendering, depending on settings and the needs of the final frame. Each pass involves drawing some or all of the scene geometry with different shader options. Material subroutines are not always needed in every pass, or are sometimes only appropriate for certain passes. If you notice your custom shader is interfering with effects like GI, shadows, or others, you may want to adjust the passes with which your shader interacts (custom shader code is run in all passes by default). This can be done by checking #define values. For example, let’s say you wanted some code to only apply to the lighting pass (and not, for example, shadowing, wireframe, or other passes):

   //code for light pass goes here

A full listing of pass defines follows:

Pass Description
MATERIAL_PASS_PREPASS Geometry pre-pass; fills position & normal buffer before main render.
MATERIAL_PASS_LIGHT Main lighting pass, includes light from skybox and light sources.
MATERIAL_PASS_VOXELIZATION Voxelization/lighting for GI. Only runs when GI is enabled.
MATERIAL_PASS_WIREFRAME Wireframe pass; not usually of interest.
MATERIAL_PASS_SHADOWMAP Shadow map pass, runs when meshes are rendered into shadow textures.
MATERIAL_PASS_EXPORT Marmoset Viewer or other texture export pass (not present in renders).
MATERIAL_PASS_SCATTER Subsurface scatter prepass; writes values critical to scatter effects.

Vertex & Tessellation

In addition to custom fragment shader routines, it is also possible to write shaders for the vertex shader (.vert), as well as hull and domain tessellation shaders (.hull, .dom). The process for these stages is somewhat similar, however there are some obvious differences in the state structure and available parameters. If you want to go beyond altering shading and adjust the geometry itself, you will need to write a vertex or tessellation shader module. This can be done by adding a new file with a proper extension but the same name as your .frag file. Toolbag will load multiple files with matching names to their appropriate stages, which would allow a .frag and .vert pair to be distributed together, for example. The user may select either in the custom shader UI and both will be loaded.


Shader development is always an involved task. Patience, persistence, and curiosity are rewarded in this endeavor. But if you’ve been poking around and reading example code and you’re still stuck on something, you might try the Toolbag User Group on Facebook. There are several knowledgeable users there including shader developers and some of our own staff. Come and say hello, and show us anything cool you’ve made. Happy shading!

4 Channel Detail Normal Map Shader for Marmoset Toolbag

Tech art wizard Chris Perrella created a Detail Normal RGBA shader that provides four slots for tiling micro detail normal maps and a splat mask to control where each detail map is shown. This shader is especially useful for cloth, skin and environment assets like rocks or terrain. You can grab the shader on Gumroad.

The NAC-22: Universal Asset Modeling for StemCell

Ben Bickle sat down with to discuss how he approaches modeling for TurboSquid’s new StemCell system. Ben detailed his process from concept to completion, including how he used Toolbag 3’s baking tools to drastically speed up his texture baking process. Be sure to check out the rendering section as well, where Ben explains his methodology for adding context and believability to his final Toolbag renders.

Toolbag Artist Highlight | Ep. 149

Another Monday, another fresh set of renders from the Toolbag community!

    1. Ray Le captivated us with this stunning render of the G.E Soldier from the Glass Egg Character Portfolio Project.
    2. Demin Dmitry created a dangerously fun Shark-man Captain.
    3. Baj Singh is back with an all new set of characters from the Total War: Warhammer series, including Krell.
    4. Matthew Kean created a fanciful real-time sci fi character.

Thanks for checking out the latest featured artwork rendered in Toolbag. Stay tuned for more next week

Hexels Feature Friday | Ep. 106

Fresh off the grid, it’s your weekly dose of Hexels Feature Friday!

  1. Welcome to Hanamura by France Mansiaux.
  2. Pixel lemon cake by chlumy.
  3. Mo Money Mo Problems by inomnom.
  4. Floating Isometric Tree by Keiiros.

Join the Hexels User Group on Facebook to download the all-new pixel art beta build.  If you own Hexels on Steam, find the instructions for installing the beta on the Hexels Steam Community.