Toolbag

Writing Custom Shaders For Toolbag 3

Note: This tutorial is intended for users of Toolbag 3.03, which has some minor changes to the custom shader system and is currently in beta. Join the Toolbag 3 User Group on Facebook to try out the beta.

By Jeff Russell

Toolbag 3 provides a framework for users to customize the material shader system. This can be a powerful tool for users to extend Toolbag’s rendering capabilities, and for these improvements to be shared with others. The purpose of this article is to provide enough of an introduction to get a developer or technical artist started. Some basic knowledge of shader development and languages will be helpful; users looking to install custom shaders should see our Toolbag Add-On Library instead.

For those of you who might prefer a video introduction, our friend Chris Perrella has prepared a tutorial demonstrating the basics of custom shader creation and use. We suggest you give it a look before reading further:

Location

Toolbag’s material shaders are located in the installation directory, inside data/shader/mat. User shaders belong in data/shader/mat/custom; any files placed there will appear in the shader dropdown in the “Custom” material module as the tutorial video illustrates. It’s worth exploring the contents of the mat directory as it contains source for the built-in shader modules in Toolbag, all of which are written in the same language and structure available to user shaders. Of particular note are mat.frag and state.frag, which will be discussed in more detail below. There are also two example shaders included with Toolbag in the custom directory, and these provide good examples of common shading operations in Toolbag.

Please note also that Toolbag’s console window (accessible through the Help -> Dev menu, or by pressing Ctrl + ~ on Windows and Cmd + Shift + C on Mac) will display any syntax errors during shader load or reload. This can be very helpful for development and debugging.

Language

Marmoset Toolbag uses a somewhat customized shader language, which is a kind of union of HLSL and GLSL syntax conventions. In many cases, both sets of keywords work (e.g. both vec3 and float3 are valid), and in other cases custom macros are required. This syntax allows Toolbag shaders to compile and run in Direct3D, OpenGL, and more recently Apple’s Metal. If you follow these conventions, your shader should work correctly on a wide variety of operating systems and hardware.

One common example of this unique syntax, which may seem unusual to those of you used to HLSL, relates to textures. Textures are declared with the USE_TEXTURE2D macro, and are accessed with GLSL-like commands. Here is a simple example of texture usage in a Toolbag shader:

USE_TEXTURE2D(myTexture);

void myFunc( inout FragmentState s )
{
   vec4 val = texture2D( myTexture, s.vertexTexCoord );
}

A full description of the shading language is beyond the scope of this article. Reading through other files for examples is likely to be the most productive route to learning the ins and outs. Some basic reference for GLSL may also be helpful as Toolbag’s shading language most closely resembles this.

Subroutines


Marmoset material shaders are organized as a collection of subroutines for various predefined ‘slots’, all of which are called from the main shader file, mat.frag. Custom shaders may extend or replace any of these subroutines, and by doing so significantly alter the rendering behavior of the material. For example, a custom shader my supply a new “Surface” subroutine to alter the standard normal mapping functions. The following shader extends the Surface subroutine to adjust the normal to point in a different direction. This will affect all subsequent shading processing, including diffuse and specular lighting:

#include “../state.frag”

void CustomNormalThing( inout FragmentState s )
{
   //run the existing Surface function, if it is defined.
   //this allows regular normal mapping etc. to run first.

   #ifdef Surface
      Surface(s);
   #endif

   //alter the surface normal to be messed up. whee!
   s.normal = -s.normal.yxz;
}

//define Surface as our new function
#ifdef Surface
   #undef Surface
#endif
#define Surface CustomNormalThing

Other material subroutines in Toolbag can be replaced and extended in similar fashion. Here is a full listing of the available subroutines, in the order in which they run:

Subroutine Description
Premerge Preparatory work at the start of a shader.
Surface Specifies the material surface; typically a normal mapping function.
Microsurface Sets gloss/roughness for shading.
Albedo Sets diffuse albedo/color.
Reflectivity Sets specular reflectivity.
DiffusionEnv Computes diffuse environment lighting.
Diffusion Computes diffuse lighting (called once per light).
ReflectionEnv Computes specular environment lighting.
Reflection Computes specular lighting (called once per light).
Occlusion Applies occlusion, typically ambient occlusion (called after lighting).
Emissive Emissive light, typically glow.
Transparency Any transparency work happens here.
Merge Performs the final merge of above values into the render color outputs.

All of these subroutines take a FragmentState structure by reference (hence the inout tag). This structure holds all intermediate values used by a Marmoset material shader, and is an excellent place to start reading. This structure is defined in state.frag, and several of its key members are listed here for reference:

FragmentState Variables
Type Name Purpose
vec3 vertexPosition Position in 3D space.
vec3 vertexEye Unit vector from the position to the camera.
float vertexEyeDistance Direct distance from the position to the camera.
vec2 vertexTexCoord Mesh texture coordinates.
vec2 vertexTexCoordSecondary Secondary mesh texture coordinates; often absent/unused.
vec4 vertexColor Mesh color; will be white if absent.
vec3 vertexNormal Mesh normal vector.
vec3 vertexTangent Mesh tangent vector.
vec3 vertexBitangent Mesh bitangent vector.
vec2 screenTexCoord Coordinates for screen position in [0,1]. A full-screen texture can be sampled with this.
float screenDepth Post-projection depth value. Not the same as vertexPosition.z.
uint sampleCoverage A bit mask for sample coverage; typically only used for voxelization pass.
int instanceID Instance number; typically only used for voxelization pass.
vec4 albedo Albedo or diffuse color in rgb, opacity optionally in alpha.
vec3 normal Surface normal direction as a unit vector.
float gloss Surface roughness as a scalar on [0,1].
vec3 reflectivity Colored specular reflectivity.
vec3 fresnel Fresnel strength.
vec3 diffuseLight Sum of all diffuse lighting.
vec3 specularLight Sum of all specular lighting.
vec3 emissiveLight Sum of all emissive lighting.
vec4 generic0…3 Four generic values for special use by some subroutines.
vec4 diffuseGI Global diffuse illumination in RGB, mask in A. Not always present.
vec4 specularGI Global specular illumination in RGB, mask in A. Not always present.
vec4 output0…7 Final render color outputs. Usually only modified in the Merge subroutine.

Some lighting functions, like Diffusion and Reflection, also receive a LightParams structure describing the current light being rendered. For a full definition of this parameter, read other/lightParams.frag.

Parameters


Most custom shaders will need to pass constant parameters to control their operation, and most users will want to see these values exposed with a graphical interface. Fortunately, Toolbag provides exactly such a mechanism. Toolbag will automatically parse your custom shader code looking for ‘uniform’ variables declared in global scope, and create UI based on their type. All float, integer, vector, and scalar types are supported, along with textures. Shader authors may supply comments to give additional information about how the interface should be constructed. A quick example:

uniform float uVal;
uniform float uVal2; //name “My Parameter” min 0 max 3.141592 default 2.0

In this snippet, two float scalars are declared as uniform parameters, and both will appear in the UI. The difference is that the second one, tagged with formatters, will be displayed as “My Parameter” instead of just “uVal2”, will have a slider ranging from 0 to pi instead of 0 to 1, and will be given a default value of 2.0. Decorating uniform parameters with these formatting options makes for friendlier interfaces to custom shaders. A full list of formatting options follows:

Uniform Options
Directive Options Description
name “Name” Uses “Name” as the display name.
default v0 v1 … A default value for numerical parameters. May be a vector or single value.
min v0 v1 … A minimum value for numerical parameters. May be a vector or single value.
max v0 v1 … A maximum value for numerical parameters. May be a vector or single value.
color Displays 3D vectors as color pickers instead of text boxes.
bool This option presents integer parameters as checkboxes instead of sliders.
srgb Sets a texture to use the sRGB color space where possible.
labels “Name0”, “Name1”, … Displays an enumerated list of options for integer parameters instead of a slider. Must be the last element in the comment line.

Passes


Toolbag performs a number of passes while rendering, depending on settings and the needs of the final frame. Each pass involves drawing some or all of the scene geometry with different shader options. Material subroutines are not always needed in every pass, or are sometimes only appropriate for certain passes. If you notice your custom shader is interfering with effects like GI, shadows, or others, you may want to adjust the passes with which your shader interacts (custom shader code is run in all passes by default). This can be done by checking #define values. For example, let’s say you wanted some code to only apply to the lighting pass (and not, for example, shadowing, wireframe, or other passes):

#if defined(MATERIAL_PASS_LIGHT)
   //code for light pass goes here
#endif

A full listing of pass defines follows:

Pass Description
MATERIAL_PASS_PREPASS Geometry pre-pass; fills position & normal buffer before main render.
MATERIAL_PASS_LIGHT Main lighting pass, includes light from skybox and light sources.
MATERIAL_PASS_VOXELIZATION Voxelization/lighting for GI. Only runs when GI is enabled.
MATERIAL_PASS_WIREFRAME Wireframe pass; not usually of interest.
MATERIAL_PASS_SHADOWMAP Shadow map pass, runs when meshes are rendered into shadow textures.
MATERIAL_PASS_EXPORT Marmoset Viewer or other texture export pass (not present in renders).
MATERIAL_PASS_SCATTER Subsurface scatter prepass; writes values critical to scatter effects.

Vertex & Tessellation


In addition to custom fragment shader routines, it is also possible to write shaders for the vertex shader (.vert), as well as hull and domain tessellation shaders (.hull, .dom). The process for these stages is somewhat similar, however there are some obvious differences in the state structure and available parameters. If you want to go beyond altering shading and adjust the geometry itself, you will need to write a vertex or tessellation shader module. This can be done by adding a new file with a proper extension but the same name as your .frag file. Toolbag will load multiple files with matching names to their appropriate stages, which would allow a .frag and .vert pair to be distributed together, for example. The user may select either in the custom shader UI and both will be loaded.

Questions?

Shader development is always an involved task. Patience, persistence, and curiosity are rewarded in this endeavor. But if you’ve been poking around and reading example code and you’re still stuck on something, you might try the Toolbag User Group on Facebook. There are several knowledgeable users there including shader developers and some of our own staff. Come and say hello, and show us anything cool you’ve made. Happy shading!

4 Channel Detail Normal Map Shader for Marmoset Toolbag

Tech art wizard Chris Perrella created a Detail Normal RGBA shader that provides four slots for tiling micro detail normal maps and a splat mask to control where each detail map is shown. This shader is especially useful for cloth, skin and environment assets like rocks or terrain. You can grab the shader on Gumroad.

The NAC-22: Universal Asset Modeling for StemCell

Ben Bickle sat down with 80.lv to discuss how he approaches modeling for TurboSquid’s new StemCell system. Ben detailed his process from concept to completion, including how he used Toolbag 3’s baking tools to drastically speed up his texture baking process. Be sure to check out the rendering section as well, where Ben explains his methodology for adding context and believability to his final Toolbag renders.

Toolbag Artist Highlight | Ep. 148

Check out this week’s fantastical display of real-time renders.

  1. Lukasz Siudzinski made a stellar fan art piece based on Dishonored 2’s Delilah.
  2. Chris Kuhner had the opportunity to create the new Goddess, Artio, for the game Smite.
  3. Petr Sokolov developed a glorious character from the medieval times.
  4. Shayleen Hulbert created a troublemaker by the name of Güilly.

Thanks for checking out the latest featured artwork rendered in Toolbag. Stay tuned for more next week.

Python Scripting For Toolbag

This tutorial is for Toolbag 3.03, which has not been released yet, however, a beta version can be downloaded via the Marmoset Toolbag User Group on Facebook.

By Jeff Russell

We’re proud to announce that as of version 3.03, Marmoset Toolbag has Python scripting support! This feature allows for the creation and distribution of user scripts and plugins to add custom behavior to Toolbag. Python is a powerful language, and its inclusion into Toolbag greatly enhances the possibilities of the software.

This page serves as a basic introduction to developing with Python with Toolbag. If you’re a software developer or a technical artist, you’ll probably be up and running quickly. If you’re new to programming, you might need to spend some time with the Python language and tools first before you’re able to make much headway.

Python

Python is a powerful, simple, and widely-used language, well suited for scripting and plugin development. It is beyond the scope of this article to teach the Python language itself, but if you are unfamiliar you can fairly quickly get your bearings in the official Python tutorial.

As of this writing Toolbag uses Python 3.6. You may find other reference material elsewhere on the internet for earlier versions of Python, notably 2.7 which still sees active use. Most concepts from Python 2 are the same in Python 3, but there are some differences so you should be careful when checking reference material.

Hello World

Hello World Python Example
Let’s create and run a simple example script. Open your user plugins folder by selecting “Show User Plugin Folder” in the “Plugins” menu, located under “Edit”. Create a file named “tutorial.py” in this folder with a text editor of your choice. Place the following line of code in it:

print( "I am using Python in Toolbag! What a time to be alive." )

Next, either relaunch Toolbag or refresh the plugin menu with the “Refresh” option to make your script visible. Then open the console (Ctrl+~ on Windows, Cmd+Shift+C on Mac, or just via Help-> Dev-> Console). Finally, select your plugin from the plugin menu to run it. You should see the above phrase printed in your console. Congratulations! You’ve just run a simple python program in Toolbag.

The ‘mset’ Module

The real business of writing plugins for Toolbag resides in the ‘mset’ module, a set of built-in functions and classes that gives you access to Toolbag’s main features. Full reference for this module is available online, or in your installation of Toolbag (select “Python API Documentation” from the “Help” menu).

In order to use the mset module in your code, you will need to first import it as you would any other, with the ‘import’ directive. Here is a simple program that imports the mset module and prints the names of all objects in the current scene:

# Necessary for using the Toolbag interface
import mset

# Obtain a list of all toolbag objects
sceneObjects = mset.getAllObjects()

# Print them out
print( "Scene Objects:" )
for object in sceneObjects:
   print( " - " + object.name )

As you can see, the relevant call here is to the mset function getAllObjects, which returns a list of every object in the scene. Toolbag objects correspond to visible icons & names in the Toolbag outliner on the lefthand side of the UI. Essentially, an “Object” in the Toolbag sense refers to an editable entity that the user can see and control through the graphical interface. There are many different types of objects in Toolbag: Cameras, Skies, Meshes, Lights, and more.

In Python terms, every Toolbag object is a subclass of the SceneObject class. This base class provides some common methods and fields for naming, child objects, visibility, and more that every object will have.

Here is a slightly more advanced example that searches for a specific object by name and alters some of its settings, in this case some post effect settings for the main camera:

import mset

# find our camera by name
mycam = mset.findObject("Main Camera")

# make sure it is a CameraObject instance
if isinstance(mycam, mset.CameraObject):
   # up the contrast!
   pfx = mycam.getPostEffect()
   pfx.contrast += 0.5
   mycam.setPostEffect(pfx)

Most objects types in Toolbag have settings that can be manipulated in a similar fashion. The mset module also has functions to create and destroy objects, import and create meshes and materials, as well as loading and saving scenes, taking screenshots and video, and much more. We’ve made an effort to expose just about everything Toolbag does through the Python interface, and we hope you find it to be as powerful a tool as we have. You should explore the reference to see everything that is possible.

User Interface

User Interface Python Example
The mset module also provides a simple set of classes for creating your own user interface. There are classes for windows, buttons, sliders, checkboxes, text fields, and more. Here is a quick example script that creates a window with one button:

import mset

#create a window
mywindow = mset.UIWindow("My Window")

#this function will be called when we click the button
def doSomething():
   print("You pressed a button!")
   mset.shutdownPlugin() # exit the plugin

#create a button, make it call doSomething when clicked
mybutton = mset.UIButton("My Button")
mybutton.onClick = doSomething

#add the button to the window
mywindow.addElement( mybutton )

This demonstrates a few important things. First and foremost, we see how straightforward it is to create UI elements and add them to a window of our own making. Second, if you run this script you will notice that it does not terminate right away. The UI stays up indefinitely until the user presses the button. Plugins that have active UI will remain “running” until a call to exit is made (or all UI is closed). Accordingly, note that the button onClick member has been hooked to the ‘doSomething’ function above, which contains a call to mset.shutdownPlugin. This call, as you may have guessed, shuts down the currently running plugin and closes any user interface.

File Locations

Plugins reside in one of two locations, either of the “User” or “Default” directories. The user directory is in an OS-specific location with full permissions for your user login, and may be accessed with the “Show User Plugin Folder” option in the plugin menu. This is a good place to continue to put your own plugins, as it will always have permissions on your machine and will survive any removal or reinstallation of Toolbag.

The “Default” plugin directory contains a short list of example plugins we have provided, in a location tied to the Toolbag installation. This is a good place to install plugins for all users on your machine if you are a system administrator.

Once you’ve added a script in one of the plugin directories, you may reboot Toolbag or refresh the plugin menu to add it to the plugin list. Once you see your script in the list, you may run it by selecting it from the menu.

Scripts may be structured as either a single .py file, or a named folder containing multiple .py files and subfolders. Plugins organized in folders in this way should contain one file named __main__.py, which will serve as the main entry point for the code. This file can import and reference other files in its directory as needed.

Lastly, Toolbag may run a python script anywhere on your machine at launch time, if the path to the script file is passed as a boot argument. In this way Toolbag can run as a configurable tool for use in automated systems or other scripting tasks. In windows this looks something like:

toolbag.exe C:\Path\To\myscript.py

And on macOS:

open Marmoset\ Toolbag.app/ --args /Path/To/myscript.py

More Examples

Strange Attractors Python Plugin
More advanced examples are included with Toolbag in the Default plugin directory, and we encourage you to give them a look over once you feel comfortable with the basics. These plugins demonstrate some more advanced concepts such as mesh creation, material importing, and screenshot automation.

Toolbag Scripting is Growing

We would love to hear how well the Toolbag scripting tools are working for you. We are looking to expand and improve the Toolbag python API in the near future, and your suggestions would be valuable. If you find something you think is missing or have an idea for something useful, please let us know! We can be reached at support@marmoset.co any time for any queries or suggestions.