Vfx Graph

'Vfx Graph' is an implementation of a node-based system for making visuals. It includes the VfxGraph data structure and many VfxNode type implementations. It comes with a real-time editing interface for use with 'AV Graph'. Together, the AV Graph editor and Vfx Graph create a real-time synthesis environment for visuals. Vfx Graph is the result of a personal on-going process of explorations and collaborations with media artists and a desire to create a powerful tool for creative coders for use in their practice. A primary focus for Vfx Graph and the AV Graph project in general is to let coders harness the power of the data flow paradigm directly from within their apps. As such, Vfx Graph is designed in a way that the coder has full control over when and how graphs are used and instanced. Another focus for Vfx Graph is the attention being paid to the performance and quality of the implementations of both the various algortihms it embeds and the drawing of the visuals.

Nodes and sockets

Vfx Graph defines many node types representing the basic ingredients for making visuals. Broadly speaking, nodes can be categorized as either math, logic, media, drawing, vision or communication types. Examples of math nodes are nodes which perform basic operations like adding two numbers together or oscillator nodes such as the sinewave, sawtooth or triangle generators. Examples of logic nodes are the sample and hold node and the various trigger nodes. Media is represented primarily by the picture and video nodes. The drawing subset contains various nodes for drawing primitives, working with surfaces and running shaders. Vision nodes encapsulate algorithms to perform for instance dot detection within images. Finally, communication is handled by event nodes to communicate with C++ and OSC send and receive nodes to communicate with external processes and apps.

In addition to the various node types, Vfx Graph defines a few types which can be used to connect input to output sockets. These types are:

The bool, int, float, string and color types pretty much speak for themselves. The trigger type is used to connect a node generating events to nodes reacting to events. Examples of trigger inputs are play!, pause! and resume! on the sound node and switch! on the logic.switch node. Examples of trigger outputs are the event! and beat! triggers on a timeline node and wentUp! and wentDown! on a trigger.treshold node.

The image type represents a GPU-accessible image (a texture) and is output by the picture, video, image.downsample and various other nodes.

The image_cpu type represents a CPU-accessible image (a bitmap) and is output by the picture.cpu, video, spectrum.1d, spectrum.2d and various other nodes.

The channels type represents one or more arrays of floating point data. The channels type is used to share for instance the results of the dot detector (x, y, radius) and pointcloud (x, y, z) nodes.

Images

One of the main ingredients for making visuals are, of course, images. Images may come from a variety of sources; they may be static images, loaded from for instance jpegs or pngs. They may be a still from a video, streamed from an mpeg or mp4 file. Or they may be captured in real-time using a webcam or a kinect sensor.

Vfx Graph defines two types of images:

The distinction between these two types of images is made for performance reasons. CPU-accessible images are ideal for use in visions algorithms or other tasks which require the CPU to access its pixels. GPU-accessible images are suited for drawing and access from within shaders.

For vision algorithms it's enticing to use video or real-time capture data as input. Nodes which output images, suchs as the video and webcam nodes, will typically expose both a CPU-accessible image as well as a GPU-accessible image as outputs.

The flow of data is considered one-way between CPU and GPU. Data is generated and processed on the CPU and streamed to the GPU, which in turn will process pixels and run shaders and output the results to the screen. To stream CPU-accessible images to the GPU, the image.toGpu node is used. The flow of data from the GPU back to the CPU is explicitly denied for now, as it typically causes the CPU to stall as it waits for the GPU to flush its drawing, causing the frame rate to drop significantly.

Channels

Common node types

todo : rename this section to nodes to explore and list only a few interesting nodes

Channels

channels.merge channel.select channel.slice channel.swizzle channel.toGpu channels.fromFile

Math and numbers

sample.delay impulse.response map.ease map.range math.add, math.sub, math.mul, math.sin, math.cos, math.abs, math.min, math.max, math.saturate, math.negate, math.sqrt, math.pow, math.exp, math.mod, math.fract, math.floor, math.ceil, math.round, math.sign, math.hypot, math.pitch, math.semiton

Oscillators

noise.simplex2d osc.sine, osc.saw, osc.square, osc.triangle, osc.random physical.mover physical.spring

Logic

logic.switch sampleAndHold timeline trigger.onchange trigger.timer trigger.treshold

Drawing

color display draw.primitive fsfx surface transform.2d image.downsample image.scale

Image sources

picture picture.cpu video webcam

Vision and image operations

deepbelief images.dots image.dotTracker image_cpu.crop image_cpu.delay image_cpu.downsample image_cpu.equalize image.toGpu pointcloud spectrum.1d spectrum.2d yuvToRgb

Sensors and actuators

kinect1 kinect2 leap mouse osc.receive osc.send touches xinput

Creating your own nodes

VfxNodeBase VFX_NODE_TYPE VFX_ENUM_TYPE

Speed considerations

Hacking Vfx Graph