Vfx Graph
'Vfx Graph' is an implementation of a node-based system for making visuals. It includes the VfxGraph data structure and many VfxNode type implementations. It comes with a real-time editing interface for use with 'AV Graph'. Together, the AV Graph editor and Vfx Graph create a real-time synthesis environment for visuals. Vfx Graph is the result of a personal on-going process of explorations and collaborations with media artists and a desire to create a powerful tool for creative coders for use in their practice. A primary focus for Vfx Graph and the AV Graph project in general is to let coders harness the power of the data flow paradigm directly from within their apps. As such, Vfx Graph is designed in a way that the coder has full control over when and how graphs are used and instanced. Another focus for Vfx Graph is the attention being paid to the performance and quality of the implementations of both the various algortihms it embeds and the drawing of the visuals.
- Nodes and sockets
- Images
- Channels
- Common node types
- Creating your own nodes
- Speed considerations
- Hacking Vfx Graph
Nodes and sockets
Vfx Graph defines many node types representing the basic ingredients for making visuals. Broadly speaking, nodes can be categorized as either math, logic, media, drawing, vision or communication types. Examples of math nodes are nodes which perform basic operations like adding two numbers together or oscillator nodes such as the sinewave, sawtooth or triangle generators. Examples of logic nodes are the sample and hold node and the various trigger nodes. Media is represented primarily by the picture and video nodes. The drawing subset contains various nodes for drawing primitives, working with surfaces and running shaders. Vision nodes encapsulate algorithms to perform for instance dot detection within images. Finally, communication is handled by event nodes to communicate with C++ and OSC send and receive nodes to communicate with external processes and apps.
In addition to the various node types, Vfx Graph defines a few types which can be used to connect input to output sockets. These types are:
- bool
- int
- float
- string
- color
- trigger
- image
- image_cpu
- channels
- any
The bool, int, float, string and color types pretty much speak for themselves. The trigger type is used to connect a node generating events to nodes reacting to events. Examples of trigger inputs are play!, pause! and resume! on the sound node and switch! on the logic.switch node. Examples of trigger outputs are the event! and beat! triggers on a timeline node and wentUp! and wentDown! on a trigger.treshold node.
The image type represents a GPU-accessible image (a texture) and is output by the picture, video, image.downsample and various other nodes.
The image_cpu type represents a CPU-accessible image (a bitmap) and is output by the picture.cpu, video, spectrum.1d, spectrum.2d and various other nodes.
The channels type represents one or more arrays of floating point data. The channels type is used to share for instance the results of the dot detector (x, y, radius) and pointcloud (x, y, z) nodes.
Images
One of the main ingredients for making visuals are, of course, images. Images may come from a variety of sources; they may be static images, loaded from for instance jpegs or pngs. They may be a still from a video, streamed from an mpeg or mp4 file. Or they may be captured in real-time using a webcam or a kinect sensor.
Vfx Graph defines two types of images:
- CPU-accessible images.
- GPU-accessible images.
For vision algorithms it's enticing to use video or real-time capture data as input. Nodes which output images, suchs as the video and webcam nodes, will typically expose both a CPU-accessible image as well as a GPU-accessible image as outputs.
The flow of data is considered one-way between CPU and GPU. Data is generated and processed on the CPU and streamed to the GPU, which in turn will process pixels and run shaders and output the results to the screen. To stream CPU-accessible images to the GPU, the image.toGpu node is used. The flow of data from the GPU back to the CPU is explicitly denied for now, as it typically causes the CPU to stall as it waits for the GPU to flush its drawing, causing the frame rate to drop significantly.
Channels
Common node types
todo : rename this section to nodes to explore and list only a few interesting nodesChannels
channels.merge channel.select channel.slice channel.swizzle channel.toGpu channels.fromFile
Math and numbers
sample.delay impulse.response map.ease map.range math.add, math.sub, math.mul, math.sin, math.cos, math.abs, math.min, math.max, math.saturate, math.negate, math.sqrt, math.pow, math.exp, math.mod, math.fract, math.floor, math.ceil, math.round, math.sign, math.hypot, math.pitch, math.semiton
Oscillators
noise.simplex2d osc.sine, osc.saw, osc.square, osc.triangle, osc.random physical.mover physical.spring
Logic
logic.switch sampleAndHold timeline trigger.onchange trigger.timer trigger.treshold
Drawing
color display draw.primitive fsfx surface transform.2d image.downsample image.scale
Image sources
picture picture.cpu video webcam
Vision and image operations
deepbelief images.dots image.dotTracker image_cpu.crop image_cpu.delay image_cpu.downsample image_cpu.equalize image.toGpu pointcloud spectrum.1d spectrum.2d yuvToRgb
Sensors and actuators
kinect1 kinect2 leap mouse osc.receive osc.send touches xinput
Creating your own nodes
VfxNodeBase VFX_NODE_TYPE VFX_ENUM_TYPE
Speed considerations
Hacking Vfx Graph