1. frameCount, millis(), modulo % and noise
Let me start with a few functions that in themselves and in combination are extremely useful. Since they’re often used together, it’s easier to combine them in one tip and one code example. But know there are countless other uses for these functions. frameCount is a simple number that increases by one on each frame. Apart from it’s informative quality it can generate change in your sketch. Millis() is something similar. It’s the number of milliseconds from the start of the sketch. Because it does not depend on the frameRate it is more accurate. This makes it very useful for precise timing (also see tip 15). Modulo is a somewhat non-intuitive function that gives the remainder when one number is divided by another. When you see it in action, it’s easier to understand. Finally, noise is an implementation of perlin noise. It’s semi-random nature makes it a great tool for achieving harmonious changes in movement, color, shape or anything else. All of these functions are highly recommended. For a very basic use of each, check out the following code example (and the examples that come with Processing, such as Basics > Math).

float x, y, r, g, b, radius;
int timer;

void setup() {
  size(500, 500);
  background(255);
  noStroke();
  smooth();
}

void draw() {
  // use frameCount to move x, use modulo to keep it within bounds
  x = frameCount % width;

  // use millis() and a timer to change the y every 2 seconds
  if (millis() - timer >= 2000) {
    y = random(height);
    timer = millis();
  }

  // use frameCount and noise to change the red color component
  r = noise(frameCount * 0.01) * 255;

  // use frameCount and modulo to change the green color component
  g = frameCount % 255;

  // use frameCount and noise to change the blue color component
  b = 255 - noise(1 + frameCount * 0.025) * 255;

  // use frameCount and noise to change the radius
  radius = noise(frameCount * 0.01) * 100;

  color c = color(r, g, b);
  fill(c);
  ellipse(x, y, radius, radius);
}

2. math, logical and relational operator shortcuts
Any program will use operators. So since you will be using them a lot, it’s good to know shortcuts that will keep your code shorter and save you some typing. Let me walk through them quickly. First, you can write x = x + 10 shorter as x += 10. Exactly the same can be done with all other mathematical operators like substraction, multiplication and division. Second, when you want to add (or substract) 1, there is an even shorter way to write it: x++. Third, when checking if something is true or false, there is no need to fully write it out, so: if(something == true) is the same as: if (something). Fourth, to negate a statement (aka if NOT something), you can put an exclamation mark in front of it: if (!something). Check out the following code example.

int x = 0;
x = x + 10;
x += 20;
x++;
println("x = " + x);

int y = 90;
y /= 3;
y++;
y *= 2;
println("y = " + y);

boolean feelingPositive = true; // change this to false to go Doh!

if (feelingPositive) { println("Yeah!"); }
if (!feelingPositive) { println("Doh!"); }

3. math with ints
A very common mistake is doing math operations on integers and expecting a floating number outcome. Obviously, this won’t happen. When you divide an int by another int, the result will be an int. If you want a floating number outcome, then one of your numbers should be a float. Or you have to specifically tell the program that you want a floating number outcome. Check out the following code example and notice the difference.

println(1 / 3);
println(1.0 / 3);
println((float) 1 / 3);

4. frameRate
Processing’s frameRate is by default capped at 60 fps. You could say that’s the default top speed of the sketch. If you want to make it run faster, you can set the maximum frameRate manually. To get the actual frameRate, you can use frameRate. I often find it useful to display the frameRate in the sketch window title. Check out the following code example.

float x, y, radius;

void setup() {
  size(500, 500);
  background(255);
  smooth();
  frameRate(200);
}

void draw() {
  x = random(width);
  y = random(height);
  radius = random(200);
  fill(random(255));
  ellipse(x, y, radius, radius);
  frame.setTitle(int(frameRate) + " fps");
}

5. loading specific files from an external input directory
Many applications use file input, for example an image or a data file. In most cases one can place these in the data directory and they become available to load into the program. However when a program uses a lot of input files, for example a large sequence of images or text files, this method is not very practical. Using an external directory is then more efficient. First, you won’t need to keep copies for every sketch (each sketch will use the same files from a central location). Furthermore, when working with many files you don’t want to type out all the individual filenames or be forced to name them sequentially. So you need a function to load the filenames, whatever they are. Finally, you might not want to load all the files in a directory. Maybe you only want the images or perhaps only the text files. In short, you will need the ability to filter by file extension. Fortunately you can do all of this with relatively little code. Check out the following code example.

String[] filenames;
String fullPath = "C:/WINDOWS/Web/Wallpaper"; // use forward slashes

void setup() {
  filenames = loadFilenames(fullPath);
  println(filenames);
  exit();
}

String[] loadFilenames(String path) {
  File folder = new File(path);
  FilenameFilter filenameFilter = new FilenameFilter() {
    public boolean accept(File dir, String name) {
      return name.toLowerCase().endsWith(".jpg"); // change this to any extension you want
    }
  };
  return folder.list(filenameFilter);
}

6. timestamp
When saving output to a file you often need a way to distinguish between different sessions. A great way to do this is by using a unique timestamp. Either for one file or one sequence. There are many ways in Processing and/or Java to create a timestamp. One way is to use the built-in Processing functions for year, month, day, hour, minute and second. Then format the final timestamp to keep the number of characters fixed (no matter if it’s made out of single or double digits). Check out the following code example.

String timestamp;

void setup() {
  timestamp = year() + nf(month(),2) + nf(day(),2) + "-"  + nf(hour(),2) + nf(minute(),2) + nf(second(),2);
  println(timestamp);
  exit();
}

7. fast image sequence saving
If you want to save your visual output to a file you can use the save() or saveFrame() funtions. In addition you could use a timestamp or frameCount to distinguish between files. The saveFrame function automatically uses the frameCount. When you put this function at the end of draw it will save each frame to an image file. The bigger your output, the more it slows down your sketch. While there is no magic solution, the fastest image format to save an image sequence is… TGA. Bet you didn’t see that one coming. Since this is an uncompressed format it saves much faster than for example JPG and it also provides 100% quality. The price you pay is the size of the output files. Of course there are situations when even this will not work out and specialist software such as fraps is needed.

float x, y, radius;

void setup() {
  size(500, 500);
  background(255);
  smooth();
}

void draw() {
  x = random(width);
  y = random(height);
  radius = random(200);
  fill(random(255));
  ellipse(x, y, radius, radius);

  saveFrame("output-####.tga"); // will save each frame with the frameCount
}

8. slow fade & simulated motion blur
Most Processing sketches use a background call at the beginning of the draw() loop to clear the screen each frame. A handful use the technique of aggregate drawing, whereby every frame adds imagery much like the canvas of a painter. But what if you want something in between: slowly fading shapes or motion trails. For these cases, you can use a semi-transparent rectangle. Check out the following code example.

void setup() {
  size(500, 500);
  noStroke();
  smooth();
}

void draw() {
  fill(255, 10); // semi-transparent white
  rect(0, 0, width, height);
  
  fill(random(255));
  ellipse(mouseX, mouseY, 100, 100);
}

9. combining 2D and 3D drawing
What if you want a 2D background behind a 3D sketch? Or put a 2D display on top of a 3D sketch? If you try to do this, you will quickly find that the 3D settings make your life difficult. The solution is to reset everything to basic 2D temporarily by placing the necessary code at the beginning or end of the draw loop() respectively. Different situations, will require slightly different solutions. But useful techniques are disabling the depth test with hint, resetting the camera(), turning off the lights, and sometimes using pushMatrix and popMatrix. Check out the following non-runnable code snippet.

void draw() {
  // 3D code

  hint(DISABLE_DEPTH_TEST);
  camera();
  noLights();
  // 2D code
  hint(ENABLE_DEPTH_TEST);
}

10. change the transparency of an image
If you want to change the transparency of an image, you could of course do this through pixel manipulations. However for uniform changes in the transparency, it’s quicker to use tint(). Check out the following code example.

PImage img;

void setup() {
  size(500, 500);
  img = loadImage("input.jpg");
}

void draw() {
  background(255);
  tint(255, 125);
  image(img, mouseX, mouseY);
}

11. mirror images vertically or horizontally
As with the previous tip, if you want to change the orientation of an image, you could do this through pixel manipulations. However for a uniform orientation change another option is to use scale(). In a bigger sketch you can put it between push-popMatrix calls. Check out the following code example.

PImage img;

void setup() {
  size(500, 500);
  img = loadImage("input.jpg");
}

void draw() {
  background(255);
  
  scale(-1, 1); // this will flip everything! (read: also the coordinates)
  image(img, mouseX, mouseY);
}

12. take full screen screenshots
If you only check out one code example, make sure it’s this one, it’s the most fun! It’s a quick ‘hack’ to get a full screen screenshot. Really full screen, so not just the sketch’s window. Click the mouse to toggle the quality (and speed). Check it out!

import java.awt.Robot;
import java.awt.Rectangle;
import java.awt.AWTException;

PImage screenshot;
boolean smoothOn;
int x,y;

void setup() {
  size(int(screen.width*0.85),int(screen.height*0.85));
  frame.removeNotify();
  frame.setUndecorated(true);
}

void draw() {
  if (smoothOn) { smooth(); } else { noSmooth(); }
  screenshot();
  image(screenshot, 0, 0, width, height);
}

void screenshot() {
  try {
    Robot robot = new Robot();
    screenshot = new PImage(robot.createScreenCapture(new Rectangle(0,0,screen.width,screen.height)));
  } catch (AWTException e) { }
}

void mouseMoved(){
  x = int((float) mouseX/width * (screen.width - width));
  y = int((float) mouseY/height * (screen.height - height));
  x = int(x * 0.02 + frame.getLocation().x * 0.98);
  y = int(y * 0.02 + frame.getLocation().y * 0.98);
  frame.setLocation(x,y);
}

void mousePressed() {
  smoothOn = !smoothOn;
}

13. keyboard text and number input
Allowing the user to input text can be a great addition to your sketch. For example to change text on the screen or to input search terms. While Processing provides the basic tools to realise this via the keyPressed() function, there are several tweaks required to make keyboard input an intuitive experience for the user. This mainly relates to keys that one does not want to use (such as CTRL, ALT and others) and keys that allow removing some or all characters in a string (BACKSPACE and DELETE respectively). Check out the following code example.

String myText = "Type something";

void setup() {
  size(500, 500);
  textAlign(CENTER, CENTER);
  textSize(30);
  fill(0);
}

void draw() {
  background(255);
  text(myText, 0, 0, width, height);
}

void keyPressed() {
  if (keyCode == BACKSPACE) {
    if (myText.length() > 0) {
      myText = myText.substring(0, myText.length()-1);
    }
  } else if (keyCode == DELETE) {
    myText = "";
  } else if (keyCode != SHIFT && keyCode != CONTROL && keyCode != ALT) {
    myText = myText + key;
  }
}

14. a simple menu and/or different screens
When you want to create a simple menu or perhaps a program that moves through different screens, the best way to do it is by using a switch. This is clearer and faster than the alternative if-else structure in these circumstances. A switch will allow the user to easily move back and forth through different screen, for example via keyboard or mouse input. To keep things clearly readable, my advise is to refer to seperate functions for each screen instead of putting all the code in the draw() loop itself. Check out the following code example.

int currentScreen;

void setup() {
  size(500, 500);
  noStroke();
  smooth();
}

void draw() {
  switch(currentScreen) {
  case 0: drawScreenZero(); break;
  case 1: drawScreenOne(); break;
  case 2: drawScreenTwo(); break;
  default: background(0); break;
  }
}

void mousePressed() {
  currentScreen++;
  if (currentScreen > 2) { currentScreen = 0; }
}

void drawScreenZero() {
  background(255, 0, 0);
  fill(255);
  ellipse(100, 100, 400, 400);
}

void drawScreenOne() {
  background(0, 255, 0);
  fill(0);
  rect(250, 40, 250, 400);
}

void drawScreenTwo() {
  background(0, 0, 255);
  fill(255, 255, 0);
  triangle(150, 100, 150, 400, 450, 250);
}

15. controlling the sketch through millis() or frameCount
Using millis() can be useful to start actions exactly when you want. Or to do the opposite: (temporarily) disable actions. Or to do something for a certain amount of time. You get the point. This was touched upon in the very first tip, here we take a little further. Check out the following code example to see how millis() and frameCount can be used to control what happens in your sketch.

boolean blinker;
int timer = -3000;

void setup() {
  size(700,400);
  textFont(createFont("Arial",30));
}
 
void draw() {
  background(255);
  fill(0);
  text("you are in frame: " + frameCount,50,50);
  text(millis()/1000 + " seconds have passed since the start", 50,100);
  text("this text will be here forever",50,150);
  if (frameCount < 500) {  
    text("this text will be here for 500 frames",50,200);
  }
  if (frameCount > 800) {  
    text("this text will be here from 800 frames onwards",50,200);
  }
  if (millis() < 8000) {
    text("this text will be here the first 8 seconds",50,250);
  }
  if (millis() > 12000) {  
    text("this text will be here from 12 seconds onwards",50,250);
  }
  if (frameCount % 12 == 0) { blinker = !blinker; }
  if (blinker) { 
    text("this text will blink",50,300);
  }  
  if (millis() - timer < 3000) {  
    text("this text will be here 3 secs after pressing a key",50,350);
  }
}
 
void keyPressed() {
  timer = millis();
}

16. high resolution output
Sometimes you only need to draw to the screen. Sometimes you want to save that output. Sometimes you need to have that output in a very high resolution, for example when you want to print it. There are different ways to generate high resolution output. One can use the PDF export options. Or one could use one of the many different ‘hacks’ to create a high resolution copy of the regular draw() loop. Here is an example of one such technique.

void setup() {
  size(500, 500);
}

void draw() {
  background(255);
  smooth();
  strokeWeight(10);
  fill(255, 0, 0);
  ellipse(width/2, height/2, 200, 200);
}

void keyPressed() {
  if (key == 's') {
    save("normal.png");
    saveHiRes(10);
    exit();
  }
}

void saveHiRes(int scaleFactor) {
  PGraphics hires = createGraphics(width*scaleFactor, height*scaleFactor, JAVA2D);
  beginRecord(hires);
  hires.scale(scaleFactor);
  draw();
  endRecord();
  hires.save("hires.png");
}

17. checking if the mouse is over a circle or rect
It may be useful to check if the mouse is hovering over a circle or rectangle. For example if you want to make a button or text input field. Of course the principles can be applied to collision detection as well. To check if the mouse is over a circle one can use the dist() function in Processing in combination with the circle’s radius. To check if the mouse is over a rectangle one can check the mouse coordinates against the sides of the rectangle. Check out the following code example.

void setup() {
  size(500, 500);
  noStroke();
  smooth();
}

void draw() {
  background(255);
  
  int rX = 40;
  int rY = 60;
  int rW = 150;
  int rH = 170;
  if (mouseOverRect(rX, rY, rW, rH)) { fill(0, 0, 255); }
  else { fill(255, 0, 0); }
  rect(rX, rY, rW, rH);
  
  int cX = 340;
  int cY = 280;
  float cR = 250;
  if (mouseOverCircle(cX, cY, cR)) { fill(0, 255, 0); }
  else { fill(255, 0, 0); }
  ellipse(cX, cY, cR, cR);
}

boolean mouseOverCircle(int x, int y, float radius) {
  return (dist(mouseX, mouseY, x, y) < radius*0.5);
}

boolean mouseOverRect(int x, int y, int w, int h) {
  return (mouseX >= x && mouseX <= x+w && mouseY >= y && mouseY <= y+h);
}

18. change the PRESENT mode background color
This does not refer to the background of the sketch (window), but rather to the background color that is used in the PRESENT mode (and for full screen applications). By default I believe it’s set to grey. But it’s possible to change this color to any RGB color you want. Remember, to see this in action you have to use Sketch > Present (CTRL+SHIFT+R). Check out the following code example.

void setup() {
  size(500, 500);
  frame.setBackground(new java.awt.Color(255,0,0));
}

void draw() {
  background(frameCount % 255);
}

19. disable the ESCAPE key
For a full screen application, one might want to disable the use of the ESCAPE key to prevent end-users from escaping the application, whether deliberately or by accident. Keep ‘em tight, right? Well, you can! Check out the following code example.

void setup() {
  size(500, 500);
  noStroke();
  smooth();
}

void draw() {
  background(255);
  fill(0);
  ellipse(width/2, height/2, 300, 300);
}

void keyPressed() {
  if (key == ESC) { key = 0; println("Trapped! Muhaha! ;-)"); }
}

20. per-vertex color fills within a beginShape-endShape
Processing’s P2D, P3D and OPENGL render modes allow for individual fill colors per vertex in a beginShape-endShape. This allows you to easily create very smooth gradients. Check out the code.

void setup() {
  size(500,500,P2D);
  noStroke();
  smooth();
}

void draw() {
  background(0);
  beginShape();
  fill(255,0,0);
  vertex(100,100);
  fill(0,255,0);
  vertex(400,100);
  fill(0,0,255);
  vertex(mouseX,mouseY);
  endShape();
}

21. breakShape
Disclaimer! breakShape has been undocumented, mysterious and unsupported ever since I started using Processing. But it’s still there hidden in the source code. Maybe this will change with Processing 2.0. But for now let me just mention this function that can create holes in a beginShape-endShape.

void setup() {
  size(400, 400);
  smooth();
}
 
void draw() {
  background(255);
  fill(255, 0, 0);
  ellipse(width/2, height/2, 200, 200);
  fill(255, 255, 0);
  beginShape();
  vertex(0, 0);
  vertex(width, 0);
  vertex(width, height);
  vertex(0, height);
  vertex(0, 0);  
  breakShape();
  vertex(mouseX-100, mouseY-100);
  vertex(mouseX-100, mouseY+100);
  vertex(mouseX+100, mouseY+100);
  vertex(mouseX+100, mouseY-100);
  endShape(CLOSE);
}

22. using arctangent to determine angle
The atan2() function can be used to compute the angle between two points, which -as you can imagine- may come in handy in many cases. Here is a code example.

int x, y;

void setup() {
  size(500, 500);
  x = width/2;
  y = height/2;
  textSize(20);
  textAlign(CENTER, CENTER);
  noStroke();
  smooth();
}

void draw() {
  background(255);
  float angle = atan2(mouseY-y, mouseX-x);
  translate(x, y);
  fill(0);
  ellipse(0, 0, 100, 100);
  fill(255);
  text(int(degrees(angle)), 0, 0);
  rotate(angle);
  fill(0, 0, 255);
  ellipse(50, 0, 30, 30);
}

23. constraining a point within a circle
There are always more ways than one to do something. I wanted to show a bit of vector math. Because once you start to use more mathematical operations, it pays off to learn about vectors. Either the basic PVector class that comes with Processing or if you wanna get serious (or lazy) the more elaborate Vec2D/Vec3D classes of Toxiclibs. The example below was an answer to specific question on the forum, how to constrain a point within a circle? As mentioned, there are multiple ways to do it, but using vectors makes it pretty easy. Check it out.

float x, y;
float easing = 0.05;
PVector circle = new PVector(250, 250);
int radius = 300;
 
void setup() {
  size(500, 500);
  x = y = width/2;
  noStroke();
  smooth();
}
 
void draw () {
  background(51);
  fill(255);
  ellipse(circle.x, circle.y, radius, radius);
 
  PVector m = new PVector(mouseX, mouseY);
  if (dist(m.x, m.y, circle.x, circle.y) > radius/2) {
    m.sub(circle);
    m.normalize();
    m.mult(radius/2);
    m.add(circle);
  }

  fill(0, 0, 255);
  ellipse(m.x, m.y, 6, 6);
 
  x = x + (m.x - x) * easing;
  y = y + (m.y - y) * easing;
 
  fill(255, 0, 0);  
  ellipse(x, y, 24, 24);
}

24. renderers (choose wisely)
At the moment Processing has four different renderers: JAVA2D, P2D, P3D and OPENGL. The advantage of different renderers is that each has it’s strengths and weaknesses. So it’s sensible to determine which renderer will work best for your situation. The first obvious difference is 2D versus 3D. JAVA2D is basically the best quality. P2D is the fastest for pixel manipulations. P3D is useful for basic 3D sketches. OPENGL is best for more elaborate 3D sketches. Choose the one that works best for your specific sketch. From Processing 2.0 onwards there will be only 2 renderers: JAVA2D and OPENGL. The main reason is that the developers want to focus their energy on two renderers (one for 2D, one for 3D) instead of having to divide their attention over four. In the short term there is some cost (the loss of the speedy P2D, some libraries will break until updated to work with 2.0). However in the long run there are greater benefits (higher quality renderers for both 2D and especially 3D).

25. libraries + java
I guess one of the greatest qualities a programmer can have is to be as lazy as possible! Kidding aside, besides useful code example there is a huge reservoir of existing code that can be directly used within Processing. First there are over 100 libraries that extend the capabilities of Processing in areas like sound, video, computer vision, data import/export, math, geometry, physics and many more things. Don’t reinvent the wheel! If that wasn’t enough, realise this: Processing is java. Yes, that’s right. Processing is java. Which means that all java code (and java libraries) can be used within Processing. A world of possibilities.

BONUS TIP: the knowledge base
One of Processing’s greatest strengths, besides it’s low entry barrier, is it’s extensive knowledge base. Use these invaluable resources to your advantage. Check the examples that come with processing and the ones that are available on openprocessing. Check out the wiki. And most importantly, search the forums. The current forum and all that came before. I speak from experience when I say they are a huge source of knowledge. And most questions HAVE been asked before. If all of this doesn’t get you what you need, you can always ask a question on the forum yourself.

Final note!
This post is meant to help you with all the little tips and tricks I have learned while using Processing. This is NOT the grand opening of a special helpdesk. The best way to get help is first to help yourself (aka figure it out or use the knowledge base for existing answers to your question) and second to ask the question on the Processing forum. I hope this post was helpful. Again, feel free to share thoughts or your own tips & tricks. Good luck!
 

Just a quick heads up. I wrote an augmented reality tutorial for CreativeApplications.net.

You can check out the full blogpost including three code example HERE.
 

Let me start with the most basic code snippet, where you input a String and the corresponding mp3 is downloaded to the sketch directory. I haven’t tested all of them, but I believe the languages that you can use are English (en), Spanish (es), French (fr), German (de) and Italian (it). This functionality is also ready-to-use in the code snippet. The first String is the text(-to-speech), the second String the language. So when you run this code you will end up with an mp3 of your spoken text, pretty cool.

void setup() {
  googleTTS("Google text to speech is awesome!", "en"); // en, es, fr, de, it
  exit();
}

void googleTTS(String txt, String language) {
  String u = "http://translate.google.com/translate_tts?tl=";
  u = u + language + "&q=" + txt;
  u = u.replace(" ", "%20"); // replace spaces by %20
  try {
    URL url = new URL(u);
    try {
      URLConnection connection = url.openConnection();
      // pose as webbrowser
      connection.setRequestProperty("User-Agent", "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; .NET CLR 1.0.3705; .NET CLR 1.1.4322; .NET CLR 1.2.30703)");
      connection.connect();
      InputStream is = connection.getInputStream();
      // create a file named after the text
      File f = new File(sketchPath + "/" + txt + ".mp3");
      OutputStream out = new FileOutputStream(f);
      byte buf[] = new byte[1024];
      int len;
      while ((len = is.read(buf)) > 0) {
        out.write(buf, 0, len);
      }
      out.close();
      is.close();
      println("File created for: " + txt); // report back via the console
    } catch (IOException e) {
      e.printStackTrace();
    }
  } catch (MalformedURLException e) {
    e.printStackTrace();
  }
}

Now let’s see if you can actually playback that mp3 in Processing during runtime (as was the original question). Can we? Yes we can! The following makes use of the Minim library that automatically comes with Processing. So all these code snippets run in Processing, without installing additional libraries or other files. The way it works is that you can type the String during runtime. Use the keyboard as you normally would, backspace deletes a character, delete clears the whole String etc. The number of characters is limited to 100, because I believe that is the limit of the webservice. Anyway, once you have the String you want, press enter. Then the file is downloaded and played back, this code is based on the playback example that comes with Minim. For more advanced sketches, the use of threads is of course recommended. In the code example you can keep typing and playing different Strings if you want. An interesting trick is to comment or remove the line that closes the player (line 36). If you do that, you can layer sound upon sound. Also useful to remember is that – given the way the code works – all of the spoken texts are saved to the harddisk. So once you quit the program, you will still have all the mp3′s inside the sketch directory.

You can just copy-paste this code in Processing’s PDE and press RUN. Hope it’s useful to someone!

import ddf.minim.*;
AudioPlayer player;
Minim minim;

String s = "Type here";

void setup() {
  size(1280, 720);
  minim = new Minim(this);
  textAlign(CENTER, CENTER);
  textSize(50);
  stroke(255);
}

void draw() {
  background(0);
  text(s, 0, 0, width, height);
  if (player != null) {
    translate(0, 250);
    for(int i = 0; i < player.left.size()-1; i++) {
      line(i, 50 + player.left.get(i)*50, i+1, 50 + player.left.get(i+1)*50);
      line(i, 150 + player.right.get(i)*50, i+1, 150 + player.right.get(i+1)*50);
    }
  }
}

void keyPressed() {
  if (keyCode == BACKSPACE) {
    if (s.length() > 0) {
      s = s.substring(0, s.length()-1);
    }
  } else if (keyCode == DELETE) {
    s = "";
  } else if (keyCode == ENTER) {
    googleTTS(s, "en");
    if (player != null) { player.close(); } // comment this line to layer sounds
    player = minim.loadFile(s + ".mp3", 2048);
    player.loop();
    s = "";
  } else if (keyCode != SHIFT && keyCode != CONTROL && keyCode != ALT && s.length() < 100) {
    s += key;
  }
}

void googleTTS(String txt, String language) {
  String u = "http://translate.google.com/translate_tts?tl=";
  u = u + language + "&q=" + txt;
  u = u.replace(" ", "%20");
  try {
    URL url = new URL(u);
    try {
      URLConnection connection = url.openConnection();
      connection.setRequestProperty("User-Agent", "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; .NET CLR 1.0.3705; .NET CLR 1.1.4322; .NET CLR 1.2.30703)");
      connection.connect();
      InputStream is = connection.getInputStream();
      File f = new File(sketchPath + "/" + txt + ".mp3");
      OutputStream out = new FileOutputStream(f);
      byte buf[] = new byte[1024];
      int len;
      while ((len = is.read(buf)) > 0) {
        out.write(buf, 0, len);
      }
      out.close();
      is.close();
      println("File created: " + txt + ".mp3");
    } catch (IOException e) {
      e.printStackTrace();
    }
  } catch (MalformedURLException e) {
    e.printStackTrace();
  }
}

void stop() {
  player.close();
  minim.stop();
  super.stop();
}

The Streaming Festival 6th edition 2011 (1-18 dec 2011)

The Streaming Festival is an art event for independent artists exhibiting unconventional audiovisual art from all over the world.

• www.streamingfestival.com
• My Artist Profile

onedotzero_adventures in motion 2011 (23-27 nov 2011)

onedotzero is a London-based moving image and digital arts organisation which commissions, showcases and promotes innovation across all aspects of moving image, digital and interactive arts.

Eternalism has been selected for the code warriors programme. Apparantly after the uk festival there will also be a world tour, so perhaps I’ll get to see it somewhere. That would be awesome.

• www.onedotzero.com
• onedotzero_adventures in motion 2011
• onedotzero_adventures in motion 2011 – Screening programmes

As most Processing experiments, the Striate Cortex Series started with a small investigative sketch. Triggered by a book I was reading called Cutting Edges: Contemporary Collage, I set out to take the technique of collage into digital generative territories. The code I’ve written, which so far has led to two short films, actually consists of two seperate sketches at the moment. Whether or not the result can be called collage, it’s definitely a unique and completely dynamic mixture of still images. In fact when viewing the final films, one hardly notices that these are just moving pixels from a series of static images. All the images used in this project have been released under the Attribution 2.0 Generic Creative Commons licence. For full attribution see this document.

The big picture
In this post I’ll give a global description of the whole project and dive a little deeper into the code of the second, smaller app. I might do a more detailed write-up of the first app with code examples later on. But it’s a bit much to stuff all those different topics into a single post. Also, in the forthcoming Processing 2.0 release there will be changes in the XML classes, so I’d rather not post code now that will break in the (near) future. Let me start with an overview of the steps from start to finished films covering both apps. Steps 1 to 11 take place in the first app, the subsequent steps happen in the second app. And on a sidenote, in real-life happened a few months later.

Simplified steps
1. The user types one or more search terms and presses enter to start the search.
2. According to these search terms and several other criteria (creative commons) the program searches the Flickr database through the Flickr API.
3. An XML is generated by the Flickr API with all the information requested.
4. The program reads the XML, interprets the information and checks the dimensions of the images.
5. 100 thumbnails are downloaded for the approved images (see screenshot above).
6. As the thumbnails are downloaded they appear in the main view.
7. The user can select these thumbnails and/or search using new search terms.
8. As thumbnails are selected they are sorted to the left and the high resolution images are downloaded.
9. When the high resolution image has been downloaded, pixels start moving in the background.
10. The user can influence the speed, direction and volatility of the pixel movement using mouse controls.
11. The user can output to an image sequence. Information regarding source images is saved to a text file.
12. The user chooses two images sequences and can set their respective offset.
13. One image sequence runs in the background, one in the front. The front sequence runs through a cartesian-to-polar transformation.
14. The radius of the front sequence runs on trigonometry.
15. The transparency of both sequences is tweaked to give the final film a transcendental feeling.

The first app
As the number of steps suggests, the bulk of the coding went into this app. An important characteristic of this sketch is that a lot of the external communication (XML, downloading images) is done using concurrent threads. This allows the main application to keep running, without having to wait for a response from Flickr or 100 thumbnails being downloaded. Once the external communication is finished and the high resolution images are in memory, the application turns these static images into moving collages through manipulation of the pixel arrays. The main focus of my learning process for this app was in fact the whole communication with Flickr thing which required the solutions mentioned.

The second app
Although smaller in scope and complexity, the second app gave that necessary creative push to the output, making it visually interesting enough to release. This code actually came about when working on another topic from my (seemingly ever growing) to-do-list: cartesian-to-polar transformation. While such transformations are not that hard, trying to do them within the pixel array can be a challenge. The reason is that the pixel array is inherently cartesian. So there has to be not only transformation but also some kind of re-interpretation. Let me give working code examples of what I started with and then shortcut to where I ended up.

// Example 1: input-based cartesian-to-polar

PImage inputBased;

void setup() {
  PImage input = loadImage("input.jpg");
  size(input.width, input.height, P2D);
  inputBased = inputBasedPolar(input, 0.5, 0.25);
}

void draw() {
  image(inputBased, 0, 0);
}

PImage inputBasedPolar(PImage input, float factor, float density) {
  PImage output = createImage(input.width, input.height, RGB);
  for (float y=0; y<input.height; y+=density) {
    float r = y * factor;
    for (float x=0; x<input.width; x+=density) {
      float q = map(x, output.width, 0, 0, TWO_PI)-HALF_PI;
      int polarX = int(r * cos(q)) + output.width/2;
      int polarY = int(r * sin(q)) + output.height/2;
      polarX = constrain(polarX, 0, output.width-1);
      polarY = constrain(polarY, 0, output.height-1);
      int outputIndex = polarX + polarY * output.width;
      int inputIndex = int(x) + int(y) * input.width;
      output.pixels[outputIndex] = input.pixels[inputIndex];
    }
  }
  return output;
}

I started with input-based cartesian-to-polar transformation. As the name suggests, this starts from the input xy and via calculations ends up at the output xy. Although this code works, there is a major inefficiency at the core due to the different density moving from cartesian to polar. Resulting in gaps (transparent pixels) in the output image. There are at least two solutions. The first is to increase the density. Although this makes the output look good, it also increases the inefficiency of the program. The second solution is to use color interpolation to ‘fill the gaps’. I’ve implemented both solutions. However as said, both are relatively resource-intensive due to their inefficiencies. So if speed is important (which it is, when working with images sequences) one might look at an alternative option: output-based cartesian-to-polar transformation.

// Example 2: output-based cartesian-to-polar

PImage outputBased;

void setup() {
  PImage input = loadImage("input.jpg");
  size(input.width, input.height, P2D);
  outputBased = outputBasedPolar(input, 0.5);
}

void draw() {
  image(outputBased, 0, 0);
}

PImage outputBasedPolar(PImage input, float factor) {
  PImage output = createImage(input.width, input.height, RGB);
  color black = color(0);
  for (int y=0; y<output.height; y++) {
    for (int x=0; x<output.width; x++) {
      int my = y-output.height/2;
      int mx = x-output.width/2;
      float angle = atan2(my, mx) - HALF_PI ;
      float radius = sqrt(mx*mx+my*my) / factor;
      float ix = map(angle,-PI,PI,input.width,0);
      float iy = map(radius,0,height,0,input.height);
      int inputIndex = int(ix) + int(iy) * input.width;
      int outputIndex = x + y * output.width;
      if (inputIndex <= input.pixels.length-1) {
        output.pixels[outputIndex] = input.pixels[inputIndex];
      } else {
        output.pixels[outputIndex] = black;
      }
    }
  }
  return output;
}

Output-based cartesian-to-polar transformation works the other way around. It starts from the output xy and via calculations ends up at the input xy. This is much more efficient, since calculation is only done for exactly the amount of pixels that you need. No more, no less. Not only is it more efficient (read: faster), it also looks better. So truly a win-win option. To really make it lightning fast, one can pre-calculate a lot of the necessary math. This information is stored in so called lookup tables (LUTs). When you apply this technique to my code, the whole cartesian-to-polar transformation can be done in real-time with no noticable effect on the framerate. Awesome! To specify it even more: the inputBased code runs below 5 fps, the regular outputBased around 20 fps and the outputBasedLUT above 200 fps. I think those numbers speak for themselves. The price you pay for pre-computation is of course loss of flexibility (there are some workarounds to counter this though). So it depends on your needs which route you take. But it’s always good to have different options, right?

// Example 3: output-based cartesian-to-polar using LUTs

PImage input, outputBasedLUT;
int[][] LUT;

void setup() {
  input = loadImage("input.jpg");
  size(input.width, input.height, P2D);
  calculateLUT(0.5, input.width, input.height);
}

void draw() {
  // lookup tables are so fast that - when using them -
  // the pixel manipulation can be run in realtime without
  // any noticable cost to the sketch's frameRate
  outputBasedLUT = outputBasedLUT(input);
  println(int(frameRate));
  image(outputBasedLUT, 0, 0);
}

void calculateLUT(float factor, int w, int h) {
  LUT = new int[w][h];
  int pL = w * h - 1;
  for (int y=0; y<h; y++) {
    for (int x=0; x<w; x++) {
      int my = y-h/2;
      int mx = x-w/2;
      float angle = atan2(my, mx) - HALF_PI ;
      float radius = sqrt(mx*mx+my*my) / factor;
      float ix = map(angle,-PI,PI,w,0);
      float iy = map(radius,0,h,0,h);
      int inputIndex = int(ix) + int(iy) * w;
      if (inputIndex <= pL) {
        LUT[x][y] = inputIndex;
      } else {
        LUT[x][y] = -1;
      }
    }
  }
}

PImage outputBasedLUT(PImage input) {
  PImage output = createImage(input.width, input.height, RGB);
  color black = color(0);
  for (int y=0; y<output.height; y++) {
    for (int x=0; x<output.width; x++) {
      int outputIndex = x + y * output.width;
      int inputIndex = LUT[x][y];
      if (inputIndex == -1) {
        output.pixels[outputIndex] = black;
      } else {
        output.pixels[outputIndex] = input.pixels[inputIndex];
      }
    }
  }
  return output;
}

The future
I have a few projects lined up that I intend to finish and release. But once I have time to return to this one, there are a few things I’d like to work on. Improve the pixel manipulations of the first app (they are highly unoptimized right now). Clean up the code and add some features such as including the search terms in the text file (don’t know how I missed that one). And most importantly, try to see if it’s possible to integrate all of the code from these two apps into a single application that is capable to perform all the steps from start to finish efficiently (and perhaps be suitable for public release). Note that these two videos were basically made from test footage while writing & playing around with code. Now that I have a clearer view of where to end up, optimizing the code may be easier. I hope my blog post gave you a bit of insight into the making-of these two videos. See you around!

For high resolution screenshots from the Striate Cortex Series check out the Flickr set.

Hey everyone! Working on some projects, more soon! Right now I’d like to share two Toxiclibs code examples I made to learn this excellent library by Karsten Schmidt. Both were made in response to questions posted in the Processing forum. The first example deals with picking and dragging shapes, the second with making shapes explode. As it turns out, Toxiclibs has many useful functions to realise these things quickly, clearly and efficiently. The first example makes use of the basic functionalities. The second example uses more adventurous options such as physics-based particles and Voronoi tessellation. Creating these examples also allowed me to brush up on my vector skills since Toxiclibs has an advanced Vec2D class with many additional options for vector math manipulations. So I definitely learned a few things from making these examples and I hope they’re useful for others as well.

Example 1: Creating, Picking & Dragging Shapes
This is a pretty basic example but it contains a few interesting facets. With the Toxiclibs library, specifically the containsPoint function, it’s very easy to check if a certain point is within a shape. This is useful for many different things, but picking is definitely one of them. This function exists for all the Shape2D shapes, so it can be used quite broadly. In this example you can create, pick and drag Polygon2D shapes. When creating a shape, the left mouse button only adds a point, while the right mouse button finalizes the shape once there are more than two points. So if you only want to make triangles, just keep using the right mouse button. With regard to keys: space clears everything, d deletes the shape under the mouse and x deletes the last point. There are a lot of little tricks and solutions to make everything work as intended, check the full commented code to find out. You can copy-paste and run it in Processing’s pde directly.

import toxi.geom.*;
import toxi.processing.*;

ArrayList <Polygon2D> polygons = new ArrayList <Polygon2D> ();
ArrayList <Vec2D> points = new ArrayList <Vec2D> ();
int draggedPolygon = -1;
ToxiclibsSupport gfx;
boolean onPolygon;
Vec2D mouse;

void setup() {
  size(1280,720);
  gfx = new ToxiclibsSupport(this);
  noStroke();
  smooth();
}

void draw() {
  background(255);
  mouse = new Vec2D(mouseX,mouseY);

  // (re)set onPolygon to false
  onPolygon = false;

  // draw all the polygons
  for (Polygon2D p : polygons) {
    if (p.containsPoint(mouse)) {
      // if the mouse is over a polygon...
      // set onPolygon to true and color it red
      onPolygon = true;
      fill(255,0,0);
    } else {
      fill(0);
    }
    gfx.polygon2D(p);
  }

  // draw all the points
  fill(0);
  for (Vec2D p : points) {
    ellipse(p.x,p.y,5,5);
  }
}

void mousePressed() {
  // if the mouse is NOT on a polygon
  if (!onPolygon) {
    // add a point at mouseX,mouseY
    points.add(mouse);
    // if the right mouse button is pressed
    // and there are more than 2 points
    if (mouseButton == RIGHT && points.size() > 2) {
      // create a polygon from the points
      polygons.add(new Polygon2D(points));
      // clear the points
      points.clear();
    }
  }
}

void mouseDragged() {
  // if no polygon is selected
  if (draggedPolygon == -1) {
    // check if the mouse is on a polygon
    for (int i=0; i<polygons.size(); i++) {
      if (polygons.get(i).containsPoint(mouse)) {
        // if so, set this to be the selected polygon
        draggedPolygon = i;
      }
    }
  // if a polygon is selected
  } else {
    // set change amount to the movement of the mouse
    Vec2D change = new Vec2D(mouseX-pmouseX,mouseY-pmouseY);
    // get the selected polygon
    Polygon2D p = polygons.get(draggedPolygon);
    // add the change to all of it's individual points
    for (Vec2D v : p.vertices) {
      v.addSelf(change);
    }
  }
}

void mouseReleased() {
  // on mouse release reset to 'no polygon selected'
  draggedPolygon = -1;
}

void keyPressed() {
  // clear all points and polygons
  if (key == ' ' && !mousePressed) {
    points.clear();
    polygons.clear();
  }
  // delete the polygon under the mouse
  if (key == 'd' && !mousePressed) {
    for (int i=polygons.size()-1; i>=0; i--) {
      if (polygons.get(i).containsPoint(mouse)) {
        polygons.remove(i);
      }
    }
  }
  // remove the last point (if points > 0)
  if (key == 'x') {
    if (points.size() > 0) {
      points.remove(points.size()-1);
    }
  }
}

Example 2: Exploding Circles (Voronoi style)
The next example is in many ways more advanced. First, it deals with physics and verletParticles to create a lot of different-sized circles all within a system of counteracting forces, where the force of each particle is in line with the radius of that circle. Most of this is handled internally by the Toxiclibs library. So that’s step one. The next step is the creative destruction of a circle. Once the breaking commences, the “circle” is divided in many segments using Voronoi tessellation. Currently the only polygon clipping available in the Toxiclibs library, is rectangular. Therefore I’ve used some vector tricks to clip points outside the radius of the circle. The different fragments are loaded into an arrayList of Polygon2D shapes that move away from their initial position. To make things visually more interesting, the fragments move away at different speeds and more importantly their direction depends on the relative angle to the point-of-impact. In addition, the point-of-impact influences the Voronoi tessellation. Close to the impact there are multiple, smaller fragments, while further away the fragments get bigger. All in all this creates a fairly realistic and interesting destruction.

A few things keep the sketch running at a decent framerate. First of all, only breaking shapes are made out of Voronoi fragments. Secondly, the transparency of the breaking shape starts decreasing after a few frames. Once it is completely invisible, it’s removed entirely. This makes the code fairly efficient, although it is always possible to further optimize of course. For convenience, I’ve placed everything in one class. The showcase sketch starts with a single shape in the center of the screen. Click somewehere on the circle to make it explode. As stated before, the point-of-impact determines the tessellation and the direction of the moving fragments. For every shape that is removed, two shapes return! So things will get crowdy real soon. Therefore I’ve maximized the amount of circles on the screen. Below is the full commented code for both the main program and the BreakCircle class. Copy-paste them both in Processing’s pde and you will be able to run this example.

Main Program

import toxi.geom.*;
import toxi.geom.mesh2d.*;
import toxi.physics2d.*;
import toxi.physics2d.behaviors.*;
import toxi.util.datatypes.*;
import toxi.processing.*;

ArrayList <BreakCircle> circles = new ArrayList <BreakCircle> ();
VerletPhysics2D physics;
ToxiclibsSupport gfx;
FloatRange radius;
Vec2D origin, mouse;

int maxCircles = 90; // maximum amount of circles on the screen
int numPoints = 50;  // number of voronoi points / segments
int minSpeed = 2;    // minimum speed of a voronoi segment
int maxSpeed = 14;   // maximum speed of a voronoi segment

void setup() {
  size(1280,720);
  smooth();
  noStroke();
  gfx = new ToxiclibsSupport(this);
  physics = new VerletPhysics2D();
  physics.setDrag(0.05f);
  physics.setWorldBounds(new Rect(0,0,width,height));
  radius = new BiasedFloatRange(30, 100, 30, 0.6f);
  origin = new Vec2D(width/2,height/2);
  reset();
}

void draw() {
  removeAddCircles();
  background(255,0,0);
  physics.update();

  mouse = new Vec2D(mouseX,mouseY);
  for (BreakCircle bc : circles) {
    bc.run();
  }
}

void removeAddCircles() {
  for (int i=circles.size()-1; i>=0; i--) {
    // if a circle is invisible, remove it...
    if (circles.get(i).transparency < 0) {
      circles.remove(i);
      // and add two new circles (if there are less than maxCircles)
      if (circles.size() < maxCircles) {
        circles.add(new BreakCircle(origin,radius.pickRandom()));
        circles.add(new BreakCircle(origin,radius.pickRandom()));
      }
    }
  }
}

void keyPressed() {
  if (key == ' ') { reset(); }
}

void reset() {
  // remove all physics elements
  for (BreakCircle bc : circles) {
    physics.removeParticle(bc.vp);
    physics.removeBehavior(bc.abh);
  }
  // remove all circles
  circles.clear();
  // add one circle of radius 200 at the origin
  circles.add(new BreakCircle(origin,200));
}

BreakCircle class

class BreakCircle {
  ArrayList <Polygon2D> polygons = new ArrayList <Polygon2D> ();
  Voronoi voronoi;
  FloatRange xpos, ypos;
  PolygonClipper2D clip;
  float[] moveSpeeds;
  Vec2D pos, impact;
  float radius;
  int transparency;
  int start;
  VerletParticle2D vp;
  AttractionBehavior abh;
  boolean broken;

  BreakCircle(Vec2D pos, float radius) {
    this.pos = pos;
    this.radius = radius;
    vp = new VerletParticle2D(pos);
    abh = new AttractionBehavior(vp, radius*2.5 + max(0,50-radius), -1.2f, 0.01f);
    physics.addParticle(vp);
    physics.addBehavior(abh);
  }

  void run() {
    // for regular (not broken) circles
    if (!broken) {
      moveVerlet();
      displayVerlet();
      checkBreak();
    // if the circle is broken
    } else {
      moveBreak();
      displayBreak();
    }
  }

  // set position based on the particle in the physics system
  void moveVerlet() {
    pos = vp;
  }

  // display circle
  void displayVerlet() {
    fill(255);
    gfx.circle(pos,radius*2);
  }

  // if the mouse is pressed on a circle, it will be broken
  void checkBreak() {
    if (mouse.isInCircle(pos,radius) && mousePressed) {
      // remove particle + behavior in the physics system
      physics.removeParticle(vp);
      physics.removeBehavior(abh);
      // point of impact is set to mouseX,mouseY
      impact = mouse;
      initiateBreak();
    }
  }

  void initiateBreak() {
    broken = true;
    transparency = 255;
    start = frameCount;
    // create a voronoi shape
    voronoi = new Voronoi();
    // set biased float ranges based on circle position, radius and point of impact
    xpos = new BiasedFloatRange(pos.x-radius, pos.x+radius, impact.x, 0.333f);
    ypos = new BiasedFloatRange(pos.y-radius, pos.y+radius, impact.y, 0.5f);
    // set clipping based on circle position and radius
    clip = new SutherlandHodgemanClipper(new Rect(pos.x-radius, pos.y-radius, radius*2, radius*2));
    addPolygons();
    addSpeeds();
  }

  void addPolygons() {
    // add random points (biased towards point of impact) to the voronoi
    for (int i=0; i<numPoints; i++) {
      voronoi.addPoint(new Vec2D(xpos.pickRandom(), ypos.pickRandom()));
    }
    // generate polygons from voronoi segments
    for (Polygon2D poly : voronoi.getRegions()) {
      // clip them based on the rectangular clipping
      poly = clip.clipPolygon(poly);
      for (Vec2D v : poly.vertices) {
        // if a point is outside the circle
        if (!v.isInCircle(pos,radius)) {
          // scale it's distance from the center to the radius
          clipPoint(v);
        }
      }
      polygons.add(new Polygon2D(poly.vertices));
    }
  }

  void addSpeeds() {
    // generate random speeds for all polygons
    moveSpeeds = new float[polygons.size()];
    for (int i=0; i<moveSpeeds.length; i++) {
      moveSpeeds[i] = random(minSpeed,maxSpeed);
    }
  }

  // move polygons away from the point of impact at their respective speeds
  void moveBreak() {
    for (int i=0; i<polygons.size(); i++) {
      Polygon2D poly = polygons.get(i);
      Vec2D centroid = poly.getCentroid();
      Vec2D targetDir = centroid.sub(impact).normalize();
      targetDir.scaleSelf(moveSpeeds[i]);
      for (Vec2D v : poly.vertices) {
        v.set(v.addSelf(targetDir));
      }
    }
  }

  // draw the polygons
  void displayBreak() {
    // after 12 frames, start decreasing the transparency
    if (frameCount-start > 12) { transparency -= 7; }
    fill(255,transparency);
    for (Polygon2D poly : polygons) {
      gfx.polygon2D(poly);
    }
  }

  void clipPoint(Vec2D v) {
    v.subSelf(pos);
    v.normalize();
    v.scaleSelf(radius);
    v.addSelf(pos);
  }
}

Eternalism is a philosophical view on reality that proposes time is just another dimension. As opposed to the view of the world as a three-dimensional space modulated by the passage of time. An interesting facet of eternalism is that it implies future events are “already there” and as such that there is no objective flow of time. One can argue about the truth of such a theory in the real world. With regard to the world of film, it is most certainly true. When the viewer starts the film, the ending is already there. So the perceived spatial dimension and the perceived temporal dimension, are truly equal. This creates the possibility to view everything in a non-conventional way.

The manipulation of time
Wasn’t this blog about Processing? Yes it is! So let me get right to point. Time bending experiments in Processing, the possibilities are endless! Visually that is. I’ve been working on a series of sketches that mess with the flow of time in some way, shape or form. Ranging from manipulations at the frame level, all the way down to the very pixels that images are made of. I have no idea what you would normally call these effects. But for myself I used names like TimeMatrix-based VideoSequencer and pixelTimeDisplacement (preliminary screenshot below). More than just cool names, they’re actually a fairly decent representation of the functionality. Perhaps later I will release, write or show material related to these experiments. The focus of this blog post however will be on the technique I used to make the video above.

Slit-scanning
The term slit-scanning has it’s roots in the history of the technique. Back in the days, the process would supposedly be as follows. A moveable slide, into which a slit had been cut, was inserted between the camera and the subject to be photographed. Making one single slit-scan photograph would thus require a whole series of shots. You can imagine this was extremely time-consuming. Over the years, technological advances have not only made it cheaper, easier and faster to create slit-scan photography, but also allowed for creation of and experimentation with slit-scan video art. A great catalogue of the latter is maintained by Golan Levin. So check out this page to see some inspiring examples. You can also find a simple slitscanning sketch for Processing there, which works very similar to the old-school techniques I just described.

Existing code
We have computers now to do all the heavy lifting (yay for the digital age 8)). And there are quite a few code examples out there to get started with slit-scanning. Although a good starting point, none of those were unfortunately completely suited for what I had in mind. Which was full HD slitscanning of existing image sequences. The two biggest problems for me with existing sketches: they were fairly slow to process frames or incapable of processing large files altogether due to memory restrictions. So these were the two points I focused on when developing different prototype sketches. What I ended up with ain’t perfect, but it’s good enough for the job at hand. Also, the learning process itself is still my most important goal. Of course solid software and pretty pictures are a big bonus! The final sketch works in batches and uses the maximum amount of memory it can use for a single batch. So it works regardless of the size and number of images. A small video can be processed in a matter of seconds in a single batch. If you’re processing a longer sequence of full HD images, it can take around two hours, depending on your computer and memory limit. However in my opinion this is pretty reasonable. Plus the added advantage of batches, is that after the first batch (10 minutes even with full HD) you’ll already be able to see the first series of resulting images. And decide if you wanna see the rest.

SlitsP5
In the spirit of open-source I like to share the code I wrote. For lack of a better name, let’s refer to it as SlitsP5. You can download it from here. Of course I release my code under the usual disclaimer. That it was created as a personal experiment by yours truly. So there are no guarantees it actually works, haha. For your convenience I placed almost all of the code in a class, so it becomes really simple to use it. Aside from the aforementioned class, the code example below is the only thing you need to run it (this is all included in the download, no need to copy-paste). As you can see there aren’t many settings, just state the directory and the direction of the slit-scanning (X or Y) and that’s basically it. Some settings hidden in the class itself are a boolean for saving (true by default) and one for additional console output (false by default). I put them there, cause they’re unlikely to be used anyway. I’ve experimented with video and image sequences for both input and output. But in the end I found video unreliable (fps, codecs and whatnot), so I switched fully to image sequences for both input and output. For me there are a number of advantages to this like accuracy, flexibility and quality. The downside being the need to separately convert between video and image sequence. Of course it’s possible to rework this sketch to include direct video input and output.

SlitsP5 mySlitscan;

String directory = "inputDirectory";
char direction = 'X';
int memBarrier = 320000000;

void setup() {
 size(500,500);
 mySlitscan = new SlitsP5();
}

void draw() {
 mySlitscan.run();
}

Setting the memBarrier
With regard to settings, the only really important thing is to correctly set the memBarrier. You only need to do this once. The memBarier is the maximum amount of pixels your computer can hold in the memory at one time (given the Processing/Java/WinXP memory limitations). So this is different per computer. For me it was about 360 million, so I set the memBarrier at a secure 320 million. A little less won’t have a huge impact, but if you set it too high, you’ll get an out-of-memory error right at the start. I’m pretty sure there is a neater way to manage memory, but since you only need to do this once, it’s not a whole lot of work. So how do you find your memBarrier? Well, for your convenience another sketch – called memBarrierTest – is included in the download. Run this program until it crashes. Yes, crashes. Then scroll up a little in the console and see the last line that says something like “Number of pixels in memory: 363.250.000”. That’s the number you can use for the SlitsP5 sketch. Just to be safe you can even set it a little lower.

How to use SlitsP5
Using SlitsP5 in four steps (assuming a correct memBarrier is set):

1. Place the input image sequence in a subdirectory in your sketch directory.
2. State the name of the directory in the sketch (images will be read automatically).
3. State the direction you want to slit-scan (X or Y).
4. Run. You will see information and a progress indicator in the console as well as the main window. Output images are saved in a subdirectory of the same name as the input directory with the addition of either an X or an Y, depending on your slit-scan direction.

Tips, tricks & troubleshooting
A collection of useful suggestions to help you on your way:

• SlitsP5 automatically reads JPG, PNG and TGA images from the input directory. If you want to use other image formats, it’s fairly easy to add them to the class (as long as they can be read by Processing in the first place). You can add other filetypes in the loadFilenames() function.
• Remember that the number of frames will be your new X or Y dimension. For example if you have 100 input images of 200 x 200 pixels and you slit-scan over the X. You will end up with… 200 images of 100 x 200 pixels. That is slit-scanning. A spatial dimension switches places with a temporal dimension. So if you wanna end up with full HD width (1920 pixels) you will need to input exactly 1920 frames.
• Footage with big cuts and/or compilations doesn’t work well. As you can imagine, sudden changes stand out in slit-scanned material. Slow moving objects on the other hand create much smoother output.
• If you get an outOfMemory error right at the start, you need to correctly set the memBarrier!

Inspiration
For anyone interested in slit-scanning, I can recommend the 1988 experimental short film The Fourth Dimension by Zbigniew Rybczyński, a filmmaker and innovator in the technical field. Aesthetically the video I made was also influenced by the work of Pipilotti Rist, an amazingly talented video artist. I was – unexpectedly – introduced to her wonderful art on my visit to the Fundació Joan Miró in Barcelona. Last but not least an obvious source of inspiration were the works of previous Processing tinkerers such as Don Whitaker, Lukasz Karluk and Matt Ditton. Don Whitaker also happens to be the man behind Mother Nature Videos. This is one of the few online collections of free quality HD stock footage, which is perfect material for slit-scan experimentation.

Looking back, looking forward
It’s becoming kind of a blog post tradition to reflect upon the whole creative & educational process. With each project, I just feel that my grasp of programming in general and Processing in particular is growing. Which is of course great. I do notice that despite my best efforts, my to-do-list is even growing faster! The more you know, the more you realise you don’t know. There are numerous areas of interest that I’d like to investigate. I just try to go from one to the next intuitively, so we’ll see what happens… I hope this post, video and/or source code is useful to someone out there. And please feel free to talk back in the comments below.

You can download SlitsP5 from here.

Been working on some really interesting projects. The first of these has just been released via a guestpost on CreativeApplications.net. It’s a graphical user interface I made for the excellent Hemesh library by Frederik Vanhoutte. The video above (you can turn HD on) showcases some of the first things I created with it. It’s more than just a gui, because the shapes can now be rendered directly in Sunflow via Christopher Warnow’s SunflowApiApi.

Check out the full article on CreativeApplications.net.
Check out the source code on Google.
Check out the results on Flickr.

Stay tuned for my other projects. I’ll be releasing those via this blog in the new year!

I’ve been using the java-based global illumination renderer Sunflow together with Processing for quite some time now. The first ever video I made with Processing (only 10 months ago!) was actually one that employed Sunflow through the P5Sunflow library. Two months ago I went a little further to create a short motion graphics experiment for the 5 second project. All my Sunflow projects until that time worked through the P5Sunflow library. The main advantage of this library is that it automatically ‘translates’ a sketch. This means you don’t have to make any changes to your original sketch, except change the renderer. This is ideal. Unfortunately however, this library was never fully developed. So although it supports geometry, it does not allow for more advanced Sunflow functionalities such as shaders, lighting and Sunflow’s built-in primitives.

SunflowP5 alternatives
For the last few months I’ve been looking at different alternatives to bring full Sunflow functionality to Processing. I checked out some of the P5Sunflow clones like Sunflow4P5. See the discussion in this thread on the old forum. Didn’t work out at all. I then downloaded Sunflow itself and examined the possibility of exporting geometry from Processing, which is subsequently rendered in “the real” Sunflow. This is possible. You can export stuff from Processing as regular 3D models or try to export it in a Sunflow-ready format. Many 3D applications already have plugins to export to Sunflow format and it’s a relatively simple format. Interestingly enough, a very recent attempt to bridge the gap between Processing and Sunflow seems to focus on this technique. Like a hybrid between full translation and using the Sunflow .sc format or something. See this thread in the new forum for more details about this library-in-development called JoonsRenderer. Looks really interesting, although I haven’t tried it yet. It wasn’t around at the time I started my search. Getting back to the story, my attempt to export geometry and then manually render in Sunflow, worked. But it was very time-intensive and for my end goal (animation) not a viable solution at all. So I decided to look into different alternatives.

SunflowAPIAPI
I then found a java-wrapper called SunflowAPIAPI, created by Christopher Warnow back in 2008. It functions like an easier-to-use API for Sunflow’s functionalities. After some figuring out, I got it to work directly from within Processing. I’ve been spending some time (between vacations, my day job and other personal projects!) on learning this wrapper and the possibilities of Sunflow in general. The three big advantages of using SunflowAPIAPI are that it allows for (a) full Sunflow functionality (b) directly from within Processing and (c) can be tweaked at will; as the code is directly accessible from Processing’s IDE. The big disadvantage, especially compared to SunflowP5, is that it requires special syntax. So it will not translate existing sketch code automatically. Instead it requires some manual translation. How much work this will be, depends on the sketch. The syntax itself is not difficult. In fact it’s fairly straight-forward, as it completely mirrors the terminology used in Sunflow. So it all makes sense, but you gotta consider the translation-issue beforehand If you’re aiming for an efficient workflow.

Workflow alternatives
I’ve found two main workflow options. Either work fully in SunflowAPIAPI-ready code and render to Sunflow in preview mode. The advantage of this method is that it’s 1:1. You write it, you see it etcetera. The disadvantage is, that Sunflow is SLOW even for low-resolution, low-quality preview renders. The second method is to program your sketch in OPENGL and have a ‘mirror copy’ of all your code for the SunflowAPIAPI. The advantage is that OPENGL works in real-time. Then at the press of a button you could start rendering to Sunflow. The disadvantage is that all your code is basically doubled and you have to sync geometry, camera’s etcetera. I’ve used both methods. It really depends on your needs and the complexity of the scene, which one you’d wanna use.

How to install SunflowAPIAPI
For people interested in using this API, let me elaborate on the steps you need to make it work.

1. Create a new empty sketch.
2. Compile a version of Sunflow or download the latest version 0.073 here.
3. Copy this file “sunflow-and-janino.jar” into the “/code” subdirectory of your sketch.
4. Go to the Google repository for SunflowAPIAPI.
5. Checkout the code via a Subversion client or directly download the files via browse.
6. The main file you will need is “SunflowAPIAPI.java”. Copy this file into the main sketch directory (besides your “sketchname.pde”).
7. Now when you code your sketch – using the correct syntax – it should render using Sunflow. A basic code example is provided below.

How to use SunflowAPIAPI (code example)
Once you have followed all of the above steps, all that remains is the content of the sketch. The examples provided with the API are useful, but you have to make some small changes for them to work in Processing. Here is an example for learning purposes, which only has the extremely bare essentials needed to make a simple test render. Code example:

import java.awt.Color;
import com.briansteen.SunflowAPIAPI;

int sceneWidth = 500;
int sceneHeight = 500;

void setup() {
 SunflowAPIAPI sunflow = new SunflowAPIAPI();
 sunflow.setWidth(sceneWidth);
 sunflow.setHeight(sceneHeight);
 sunflow.setCameraPosition(0,2.5,-5);
 sunflow.setCameraTarget(0,0,0);
 sunflow.setThinlensCamera("thinLensCamera", 50f, (float)sceneWidth/sceneHeight);
 sunflow.drawPlane("ground", new Point3(0,-3.5,0), new Vector3(0,1,0));
 sunflow.drawSphereFlake("mySphereFlake", 20, new Vector3(0,1,0), 1);
 sunflow.setPathTracingGIEngine(64);
 sunflow.render(sketchPath + "/SunflowTestRender.png");
 exit();
}

The above code snippet should, if all goes well, result in something like the image below…

Trying out different Sunflow functionalities
All right, so like I said before, this API basically gives you full Sunflow functionality. Christopher’s been adding some of the last remaining functions lately. And I’m also a project committer myself now (yay! ), so in the latest revision I added one little thing, the sphereFlake (as shown in the above code example). I’ve been playing with shaders, lighting options and some of Sunflow’s built-in primitives, like Hair and the JuliaFractal. The former is quite useful, the latter less interesting than it might seem at first sight. I’m of course grateful to Christopher for helping me out along the way and for developing the wrapper to begin with! Perhaps now that I’ve learned how to use it, it will be easier to apply it to actual projects. I’ll definitely be committing interesting and/or useful code additions, if I have any, into the repository. A video which mashes up two of my preliminary test render sessions can be found on Vimeo here.

Connecting Toxiclibs & HEMesh with Sunflow
Once I knew the API could run from inside Processing, I started working on connections with other libraries. So far I’ve successfully come up with methods to get stuff from Toxiclibs into Sunflow, from HEMesh into Sunflow and from external 3D model files into Sunflow. All of them go through Sunflow’s triangleMesh shape. This shape, as the name suggests, is made out of vertices and triangulated faces. Since most 3D shapes are already in  – or can be translated into – this format, it becomes relatively easy to get complex geometry from Processing (libraries) into Sunflow.

Tips & Tricks
Please benefit from some of the lessons I learned the hard way

- The regular render option does not allow saving in Processing. You have to render to a file directly. Of course it’s possible to immediately afterwards load and draw the rendered image in Processing. But you could just as easily go to the output directory to see the rendered result. Depends on your needs.

- The way the API works is somewhat different from regular draw cycles. The API creates a 3D scene, which stays in memory! When nothing changes except for example camera movement, you could use this. Only create the scene once and subsequently just change the camera’s position. If however you have moving/changing objects you will need to re-create them. Initially I did this by removing objects/shaders and then creating them again. Since not removing them results in errors because the names are already in use. This is however not the most efficient way to do it. A better way is to re-create the full Sunflow object on every draw cycle. The old scene goes into the garbage collector. And you don’t have to worry about removing stuff or names already in use. Of course this makes rendering animations much more convenient.

- Syncing the camera between OPENGL en Sunflow is a challenge, which I’ve only figured out partially. I’m using the peasyCam library by the way. Getting the camera position and lookAt points is easy, because they are 1:1 related between renderers. The only thing to consider is the reverse Y due to OPENGL’s coordinate system. PeasyCam’s getPosition and getLookAt functions can be used to get the relevant coordinates, which can be directly inserted into Sunflow’s setCameraPosition and setCameraTarget functions. The problem is that this is not the full transformation, but only two-thirds. I’m guessing the third parameter is the camera’s rotation, probably around it’s own center point. Like I said, haven’t fully figured this one out yet, so suggestions are welcome…

Future developments
As with all my experiments, the learning process is more important than the final result. And I must say I’ve learned a great many things again. Sunflow stuff like lighting, shading and cool camera tricks. But also coding stuff like how to using the original Sunflow’s source code to your advantage and using subversion to keep track of code revisions. And I finally overcame my long-held reservations to working with advanced libraries like Toxiclibs and HEMesh, only to find out that they’re not as scary as they look from the outside, haha Another step forward, but just a first step really. This is only the beginning. I hope to push all of this further. Been reading about fractal geometry, which poses interesting challenges and possibilities. Also I’ve barely scratched the surface of the mentioned libraries so my intention is to dive into those much more and see what comes out at the other end. That’s it for now! See you around and please feel free to share any comments below!

Links related to this post:

1. Vimeo Video
2. Flickr Set

The time between posts is definitely not a sign of Processing inactivity, rather the exact opposite! Been so busy with several experiments that I just haven’t found the time to post vids and/or write about my code adventures. But here is a little write-up for my latest animation called Subliminal Glitch. I’ve been interested in glitch art for a while now. A discussion in the forum finally gave me the right push to start experimenting with glitch art in Processing. Aesthetically my final animation is probably closer to datamoshing and compression artifacts, than to glitch art resulting from replaced bytes in an image. Of course, technically it’s not pure malfunction glitching, but rather a re-creation or mimicking of the aesthetics of the glitch using different techniques and functions in Processing. There are many existing Processing functions that can be useful for this purpose, for example get(), set(), filter(), blend(), tint() and the pixels array in combination with bitshifting.

My animation has two main layers. First, the ‘normal’ image. This is mainly made up of the typography and a particle system that has color-based collision. Simply stated, the particles stay either inside or outside the letters and they bounce of ‘em when they get to close. The second part, is the glitch system. I used object-oriented programming to write it, which means a glitchObject can easily be added to and run on top of another (read: any other) sketch to create these effects. 

The glitchObject consists of a series of glitch-functions, which are sequentially (and thus cumulatively) applied to the image. I just started writing the first glitch and gradually added more and more. Slowly increasing the complexity and unpredictability of the result. Funnily enough, things started to get interesting fairly quickly. Of course, you could add as many glitches as you want. In this preliminary experiment, eleven glitches are applied. Of these, seven are getSetGlitches (randomly copying parts of the image to somewhere else), two are elementGlitches (adding horizontal and diagonal lines) and two are colorGlitches (filters for inverting and dilating the image). So the bulk is in the get() and set() functions right now. I know these functions are not ideal. But it was just the easiest and fastest way to write it for me in an experimental phase. Therefore the program is completely unoptimized and in fact very inefficient. To improve the speed, you’d probably want to replace all the get-sets and color effects by pixel arrays and bitshifting. However these can be a little less intuitive. At least for me they’re still quite enigmatic, so that would need some investigating on my part before I can really use them.

I’m running the application at a full HD resolution of 1920×1080. Right now framerate isn’t optimal, but this really results in an interesting side effect when recording to a movie. Because when the movie is played, it blasts through at an enormous pace! For the video I wanted something energetic anyway, so I coupled the visuals with a mixtape. I then increased the pitch and speed up to 160%! Of course this video absolutely kills Vimeo compression. But luckily this suits the subject perfectly.

Links related to this post:

1. Vimeo Video
2. Flickr Set