Neural Video Style Transfer Leveraging PDG

‚ÄĻ Head back to the blog

What is texture-synthesis-houdini?

ūüé® Texture Synthesis Houdini is an open source Houdini / PDG plugin for EmbarkStudios’ texture-synthesis.

  Github

What is texture synthesis?

Texture Synthesis is the process of algorithmically constructing a large digital image from a small digital sample image by taking advantage of its structural content. It is an object of research in computer graphics and is used in many fields, amongst others digital image editing, 3D computer graphics and post-production of films.

Texture synthesis can be used to fill in holes in images (as in inpainting), create large non-repetitive background images and expand small pictures.
(Wikipedia, 28.03.2021)

You can find more about the specific process of Embarkstudios’ tool in the “More Like This, Please!” talk by Anastasia Opara.

How can we use it?

We can create new images based on existing ones. We could also edit existing ones based on the same or other images. We could then use the images to use as 3d data. The possibilities are endless.

Couple of examples can be found on Embark’s repository. SRC

Texture Tiling
 Single example generation
 Multi example generation
Guided synthesis
Style Transfer
Inpaint

What is the Houdini part in texture-synthesis-houdini?

I created a PDG wrapper around texture-synthesis that gives EmbarkStudios’ tool a great base to work with.

I’ve also added an .hda that makes it possible to use Houdini 3d attribute data as Texture input for the Texture Synthesis Process.

For example you can use float attributes (left) as a mask for inpainted texture synthesis (right).

Implementation

The way the .hda is implemented lets users easily plug in upstream data into the tool. For example you can use upstream outputs fith the file/image tag for multi example generation.

Provided Examples

Texture-synthesis github examples

With textures-synthesis-houdini you can replicate anything that you can do with the commandline version. For example I’ve provided a .hipfile that recreates all of the examples on the very thorough documentation of texture-synthesis.

  1. Single example generation
  2. Multi example generation
  3. Guided Synthesis
  4. Style Transfer
  5. Inpaint
  6. Inpaint Channel
  7. Tiling texture
  8. Repeat texture synthesis transform on a new image
  9. Combining texture synthesis ‘verbs’

Due to the PDG integration you can easily use the wedging features to drive parameters of the texture synthesis. So in the provided example I’m using wedges to drive the “strength” of the style transfer. Additionally I’m using the wedges to drive seeds for the inpainting example.

Tiling PBR textures

Texture-synthesis-houdini also comes with another cool example, that let’s you easily tile PBR textures. It makes the diffuse map tileable and uses the transformations of the diffuse map to apply them to all other maps.

The PDG process

Output

Filling texture holes

You can also use texture-synthesis to fill texture holes. Like for a 3d scanning process where you weren’t able to reach some part of the scanned object. You could fill the 3d geometry holes and then generate the textures for it based on a mask on your geometry to sample the textures from.

This is exactly what I’m doing to Houdini’s beloved pighead.

I’m painting a float mask to mark areas where I want to sample textures from using an Attribute Paint node.

Then I’m generating a target mask based on input groups.

Finally I’m writing out these 3d masks as textures with the provided Texture from Attribute TOP¬†and run the texture-synthesis-process using the pighead texture, the source and the target mask.

BeforeBefore

Driving the CLI

I’ve additionally applied some tricks to quickly add all the options to the node interface as well as convert them back to commands within my PDG processor.

I created a small node where I can quickly copy&paste the CLI options to which will then add them with a specific syntax to a targeted node. At the time of texture-synthesis creation it wasn’t quite finished yet, but it served me well.
Example for all the texture-synthesis options:

On pressing Add/Override it will add all the options to the specified node including types, defaults, ranges and help (if specified).

These options are then read from the PDG processor with a small function and passed onto the command creation process.

def getArgsFromFolder(pdg_node, foldername, toggle_prefix="__TGGL__", arg_prefix="__ARG__", as_string = False, work_item=None):
    args = {}
    
    node = pdg_node.topNode()
    folder = node.parm(foldername)
    
    parms = self.topNode().parms()
    folder_parms = [parm for parm in parms if foldername in [ f.lower() for f in parm.containingFolders()]]
    folder_toggle_parms = [parm for parm in folder_parms if parm.name().startswith(toggle_prefix)]

    for toggle in folder_toggle_parms:
        if pdg_node[toggle.name()].evaluate(work_item):
            name = toggle.name().replace(toggle_prefix,"")
            value_parm = node.parm("__ARG__"+name)
            value_type = value_parm.parmTemplate().type()
            key = value_parm.description()
            
            if value_type != hou.parmTemplateType.Toggle:
                value = pdg_node["__ARG__"+name].evaluate(work_item)
            else:
                value = ""
            args[key] = value

    if as_string:
        args_string = " ".join(["{key} {value}".format(key=key,value=value) for key, value in args.items()])
        return args_string
    else:
        return args

Distribution

I’ve also added a Github action to the repo that makes the plugin deployment really easy for the different platforms. See here

The action runs on tags being pushed to GH. The action includes serveral steps:

  1. Initialize windows, macOS and linux-ubuntu environments
  2. Install rust toolchain
  3. Install build dependencies
  4. Pull the texture-synthesis-houdini repository including the submodule texture-synthesis
  5. Build texture-synthesis
  6. Package texture-synthesis-houdini
  7. Publish Github Release
  8. Success!

https://github.com/manuelkoester/texture-synthesis-houdini/releases

Shoutout to Anastasia and her team at Embark for creating and open sourcing such an awesome suit of tools for everyone to use!

Hope this plugin helps people using Embark’s work. ūüôā

Neural Video Style Transfer Leveraging PDG