MidtermProject

Title

MIDI Capture

Description

Motivation

I have a strong interest in music and manipulating sound. I wanted to do a project that would spin off those interests and combine that with something interactive involving video capture methods in Processing.

Interactive paradigm

This program uses the brightness tracking in the Video library of Processing to track the brightest pixel. Using a flashlight, the user controls the frequency and amplitude of drum synthesized MIDI notes. Simultaneously, a predetermined array of notes plays at random and creates random ellipses that appears on the screen. This creates an improvisational live "duet" between the participant and the program. The colors of the ellipses both controlled by the user and the program are generated at random to add a dynamic visual experience.

Technical Description

The flashlight held by the user will be tracked by the brightness tracking method in Processing's video library. The top half of the screen is designated to the area where the user can control the MIDI notes. Left to right controls the amplitude, while Up and Down control the frequency. While the bottom half of the screen is designated to the random ellipses that create a sort of rhythmic dance on the screen while playing an improvised melody for the user to interact with. I used the Soundcipher library to control the MIDI sounds. The speed of each drawing of the ellipses as well as the playing of the notes is determined by the frame rate is by default is set at 6. The user can change the frame rate to faster or slower by pressing the up or down button on the keyboard.

The flow in of information will go as follows: As the user moves the flashlight across the screen, it will change the MIDI note's parameters (amplitude and frequency) while the random ellipses play a melody on the bottom half of the screen. The ending result is an interactive MIDI sound dialogue between the user and the program.

Documentation

Code

/**

*MIDI Capture
* by Leilani Martin
* 
* Tracks the brightest pixel in a live video signal and plays 
a musical note at that pixel, a live duet between a predetermined/random MIDI
melody and the drum notes controlled by the user.
*/

import processing.video.*; import arb.soundcipher.*;

//soundcipher data SoundCipher sc = new SoundCipher(this);

//MIDI scale for square melody int[] scale = {57, 60, 60, 62, 64, 67, 67, 69, 72}; int rate = 7;

Capture video; void setup() {

 size(640, 480);
 //160 
 // Uses the default video input, see the reference if this causes an error
 video = new Capture(this, width, height, 30);
 noStroke();
 smooth();
 frameRate(6);

}

//Handles Frame Rate void keyPressed(){

 if (key == CODED){
   if (keyCode == UP){
     frameRate(rate * 2);
   }
   else if(keyCode == DOWN){
     frameRate(rate - 1);
   }
 }

}

//Captures the video and finds the brightest pixel void draw() {

 if (video.available()) {
   video.read();
   image(video, 0, 0, width, height); // Draw the webcam video onto the screen
   int brightestX = 0; // X-coordinate of the brightest video pixel
   int brightestY = 0; // Y-coordinate of the brightest video pixel
   float brightestValue = 0; // Brightness of the brightest video pixel
   // Search for the brightest pixel: For each row of pixels in the video image and
   // for each pixel in the yth row, compute each pixel's index in the video
   video.loadPixels();
   int index = 0;
   for (int y = 0; y < video.height; y++) {
     for (int x = 0; x < video.width; x++) {
       // Get the color stored in the pixel
       int pixelValue = video.pixels[index];
       // Determine the brightness of the pixel
       float pixelBrightness = brightness(pixelValue);
       // If that value is brighter than any previous, then store the
       // brightness of that pixel, as well as its (x,y) location
       if (pixelBrightness > brightestValue) {
         brightestValue = pixelBrightness;
         brightestY = y;
         brightestX = x; 
       }
       index++;
     }
   }
   
  //Capture to MIDI note (Drum sounds)
 ellipse(brightestX, brightestY, 25, 25);
 fill(random(0, 255), random(0, 255), random(0, 255));
 stroke(random(0, 255));
 strokeWeight(3);
 sc.instrument(sc.DRUM); 
 sc.playNote(brightestY, brightestX, 3000);
 
 //Line dividing the screen
 line(0, 160, 700, 160);
 stroke(200);  
 
 //Random ellipse melody
  if (random(1) < 0.8) {
        sc.instrument(2); 
        sc.playNote(scale[(int)random(scale.length)], (int)random(80)+40, 0.2);
        stroke(color(random(256), random (256), random(256)));
        ellipse(random(0, 1000), random(300, 500), 100, 100);
 }

} }