1. Simple Explanation
Essentially, my box is a bunch of knobs that allow me to communicate with a computer in whatever way I want. I can set the knobs to control different things in different pieces of software, such as those for making music and visuals. The idea has always been to provide tangible feedback to the user in real time. With a simple USB connection, the entire device can power itself and exchange information with a computer. The layout has been kept simple so I do not have to think very much about what my hands are doing while I’m using it; this frees me up to consider other creative aspects of my projects.
2. Video
3. Detailed Explanation
Teensy Code (C++)
Let’s start with the hardware. 20 potentiometers in a 4×5 grid are directly mounted to a flat sheet of acrylic. LEDs above each row indicate which step (column) of the grid is being powered and measured. All the wires on the back of the panel are grouped by type (signal, power, LED), and connect via pin headers to the Teensy below. The Teensy rests in a custom-soldered bed of pin headers atop a PCB, for easy troubleshooting. For now, to reduce the strain on any wires, an acrylic box suspends the panel 4″ above the Teensy on the PCB.
The acrylic panel was laser-cut to the exact dimensions necessary to accommodate the potentiometers. The shafts of the pots protrude through the front of the panel, and are fastened by a washer and nut. Knurled knobs atop the shafts add aesthetic appeal.
Every pot is connected to ground through a network of wires that serve as the grounding bus. The power for each row is supplied by a digital-on signal from one of the digital output pins on the teensy. (Each column/step of the power rails are electrically connected to each pot, but are separate from the others.) This configuration requires fewer wires and used pins on the Teensy. Four ribbons of five wires each attach to the middle lugs of each potentiometer, per column. On the Teensy side, detachable male pin headers allow these ribbons to be disconnected for repairs.
As will be explained in a moment, the project has two modes: one where it responds to MIDI coming from a different source (like a daw), and another where the Teensy spits out MIDI to be used by a physical or software instrument. A SPST switch on the board next to the Teensy allows for switching between these modes (however, I suggest stopping whatever relevant activities are happening in the computer before switching modes).
Now for the software. The Teensy source code accesses the Serial and MIDI libraries. All the input and output digital pins are configured in the setup() function.
Let’s begin with declaration of variables. stepLength is set to the default time value for which a single step will hold, in milliseconds. ledPins[] is an array that holds the references for the output pins for the LEDs. currentStep holds the value for the step in the sequence, and is updated after each stepTime (for outputMode()) or when the Teensy receives an incoming MIDI signal (for inputMode()). lastStep is used for comparisons in order to properly trigger later events (like sending Serial data). switchPin holds the reference for the center pin on the SPST switch that’s used to go between modes. potMatrixPins[][] holds all the references for the analog pins. Meanwhile, potMatrix[][] holds the values read on each step. rowPowerPins[] holds the values for the digital output pins that light up the LEDs on the panel. currentMIDI[] and lastMIDI[] are used by outputMode() to send out MIDI, convert between MIDI and serial (byte) values, and compare between two adjacent steps.
I intentionally kept the loop() function pretty minimal, and broke out everything into a bunch of other functions. Depending on which way the mode switch is thrown, loop() will continuously call either inputMode() or outputMode(). These respective modes will call their own sub-functions.
First, inputMode(). It simply reads incoming MIDI and calls the event handler. The actual event handler (myNoteOn()) powers up the current step, invokes readPotValues() to read the values, and transmitSerial() to transmit bytes of Serial. Then myNoteOff() cuts the power to the pots and increments stepCount.
outputMode() is not broken out like the structure for the input mode, because the Teensy is directly in control of whatever’s happening; it’s not waiting to read incoming values from elsewhere. Its code is nearly identical to that of myNoteOn() and myNoteOff() combined. The only differences are that it calls the function txMIDI() to transmit MIDI values to a connected device and that it delays the program function for previously read-and-mapped stepLength values.
Processing Code (Java)
Now, let’s discuss the code in Processing, a visual scripting language coded in Java. Late in the project, I migrated my code from the traditional Processing IDE to Eclipse, which is much better for more complicated software development. The default IDE is what’s known as a PApplet, which extends the functionality of Java with new classes and methods, while simplifying some of the syntax. In order for Eclipse to run Processing sketches, I had to import essential libraries that are buried within the “package contents” of the Processing app. These .jar files are self-contained with the functionality I need to work with Serial and to make anything display on the screen.
One of the tradeoffs of working in Eclipse is that the syntax becomes more complicated. Java really cares about the cope of variables and methods, so I have to designate them as public or private, in addition to their return types. I also have to specify that my class inherits the functionality of the PApplet class. Inheritance is an enormously important concept in higher-level programming. Essentially, one class is able to take on the methods and variables of the “parent” class, and can even extend the functionality. This makes the “child” or subclass a subtype of the parent’s class type. Writing my class as inheriting the PApplet’s methods allows me to call some of its methods for my own uses. If you know how to do this, it’s definitely worth using Eclipse.
The main() method is analogous to the loop() function in the C++ code. It’s what runs while the program is operating, cooperating with all the other methods to make everything work. PApplet has its own overriden main method, which I use inside my class’ own main method to have the PApplet handle everything I write in my class.
One extra method I have to create is settings() at the top of the class. This is where I set the size of the canvas on which I will draw the visuals. setup() takes care of instantiating Serial communication and a PImage object called fader, which will be used to loop delayed projections of previous frames atop one another. incomingSerial, mappedSerial, lastStep, and currentStep are exactly what they sound like; they store the recorded incoming serial values. current is an array that holds all the bytes of the pot values from all steps in the cycle.
Now for the other methods. getSerial() recieves the incoming Serial that’s sent out by the Teensy. interpolate() creates smooth transisitions among the frames for the objects on screen. It calls linInt(), which compares the target and current values, and then advances the values slightly towards their goal. This is actually a recreation of the lerp() function that Processing already has. The problem with lerp(), however, is that it is not friendly to the standard double-type values that Eclipse interprets decimal numbers as (Processing likes floats). So, I figured out the logic behind lerp() and rewrote the method to take doubles, just to make my life a little easier.
All these methods serve the primary method called draw(), which is analogous to the main() method in Java and the loop() method in the Arduino C++ code. Every frame, it first clears the screen, takes in the serial values from the Teensy (via getSerial()), linearly interpolates them between the last stored value for the previous step and the current incoming serial, and then draws ellipses to the screen.
The PImage object “fader” captures what’s on the screen in the previous frame and displays it again in the following, before the new ellipses are drawn. When this is continuously repeated, the result is an echoing or trailing effect of multicolored ellipses dancing on screen, either to the timing of the Teensy (in outputMode()) or from the incoming MIDI from a DAW (in inputMode()).
In a nutshell, that’s all the code that I’ve written for this project! The Teensy’s compiled C++ sends information out, while the processing Java code receives all this and makes something appealing out of it.
4. Source Code
Arduino (Teensy) Code (C++):
#include <MIDI.h> // invoking MIDI library. // Serial included by default, as well. int stepLength = 250; // default, in milliseconds. int ledPins[4] = {0, 1, 2, 3}; // not including ground. int currentStep; // 0-3 (4 steps) int lastStep; // for comparison purposes int switchPin = 28; // for changing modes // specific analog reading row pins int potMatrixPins[4][5] = { {A0, A1, A2, A3, A4}, {A5, A6, A7, A8, A9}, {A13, A14, A15, A16, A17}, {A18, A19, A20, A21, A22}, }; // all the attachments of pots to the analog pins. int potMatrix[4][5]; // stored values from analogRead()s int rowPowerPins[4] = {12, 11, 10, 9}; // not including ground int currentMIDI[5]; // outputed/read MIDI per step int lastMIDI[5]; // for comparison purposes void setup() // configuring the Teensy for proper functionality. { pinMode(switchPin, INPUT); // Sets up pin that changes between inputMode() and outputMode() for (int i = 0; i < 4; i++) // Setting up pins for lighting up LEDs { pinMode(ledPins[i], OUTPUT); } for (int i = 0; i < 4; i++) // Setting up pins for powering columns (steps) of pots. { pinMode(rowPowerPins[i], OUTPUT); } Serial.begin(115200); // Establishing baud rate for Serial usbMIDI.setHandleNoteOn(myNoteOn); // Establishing which function is called (myNoteOn()) when a note-on MIDI signal is detected usbMIDI.setHandleNoteOff(myNoteOff); // Establishing which function is called (myNoteOff()) when a note-off MIDI signal is detected. } void loop() // continuously runs during program execution { if (digitalRead(switchPin)) // If the switch is set to on/HIGH, go into input mode. { inputMode(); } else // If the switch is set to off/LOW, go into output mode. { outputMode(); } } // six values: step #, pot values 1-5. // The first should correspond to the step (in this case, the row of the array). // The others should correspond to the analog read values for the 5 respective pots. void transmitSerial() { Serial.write(currentStep); // Transmission of first value for (int i = 0; i < 5; i++) // it's 5, not 6!!! { Serial.write(potMatrix[currentStep][i]); // Transmitting pot values. } }
// mapping range of Serial values to MIDI note values. void calculateMIDI() { for (int j = 0; j < 5; j++) { lastMIDI[j] = currentMIDI[j]; // Priming for later comparisons. currentMIDI[j] = map(potMatrix[currentStep][j], 0, 255, 24, 108); // Conversion from Serial to MIDI. } } void txMIDI() { // We need to send out usable MIDI values to other places, // so we first have to "convert" from Serial values to MIDI values. calculateMIDI(); if (lastMIDI[0] != currentMIDI[0]) // detection of change in MIDI note { // Turn off previous note usbMIDI.sendNoteOff(lastMIDI[0], 0, 1); // Control change usbMIDI.sendControlChange(10, currentMIDI[2], 1); // Turn on next note usbMIDI.sendNoteOn(currentMIDI[0], 100, 1); } } // Event handler for when incoming MIDI note detected void myNoteOn(byte channel, byte note, byte velocity) { // turn on both LED indicator and power to column of pots. digitalWrite(rowPowerPins[currentStep], HIGH); digitalWrite(ledPins[currentStep], HIGH); // Once pots are powered, read analog values. readPotValues(); // Send them out (to processing or elsewhere). transmitSerial(); } // Event handler for when incoming MIDI off signal detected void myNoteOff(byte channel, byte note, byte velocity) { // Cut power to LED indicator and respective column/step of pots. digitalWrite(rowPowerPins[currentStep], LOW); digitalWrite(ledPins[currentStep], LOW); // increment currentStep value (remain within 0-3) currentStep ++; currentStep %= 4; } // Not yet used; for later implementation. // Void myControlChange(byte channel, byte control, byte value)
// Use external (incoming) MIDI values for control void inputMode() { usbMIDI.read(); // call incoming MIDI-on event handler (myNoteOn()) // Not yet used; for later implementation. // usbMIDI.setHandleControlChange(myControlChange); } // Use internal pot values for MIDI control. void outputMode() { // Turning on power. to LEDs and pots. digitalWrite(rowPowerPins[currentStep], HIGH); digitalWrite(ledPins[currentStep], HIGH); // Read pot values for the row. // If you make this a double for-loop, // you will always be reading zero values on the other rows! readPotValues(); //Serial data to be sent to Processing (or elsewhere). // Transmit MIDI to DAW or other compatible instrument. //******** Where the magic happens. transmitSerial(); txMIDI(); stepLength = map(potMatrix[currentStep][4], 20, 255, 1, 750); // 20 seems to be as low as they go. delay(stepLength); // Turning off power to LEDs and step/column of pots. digitalWrite(rowPowerPins[currentStep], LOW); digitalWrite(ledPins[currentStep], LOW); // Iterate step (within range 0-3). currentStep ++; currentStep %= 4; } // Analog-read the pot values for the current powered-up step. void readPotValues() { for (int j = 0; j < 5; j++) { potMatrix[currentStep][j] = 1023 - analogRead(potMatrixPins[currentStep][j]); // map them from 10-bit values to 8-bit values (bytes). potMatrix[currentStep][j] = map(potMatrix[currentStep][j], 0, 1023, 0, 255); } }
Processing (Java) Code:
import processing.core.*; // Underlying code to make the Processing PApplet run.
import processing.serial.*; // Getting Processing to communicate with the Teensy.
//Testing01 is a subclass of the type PApplet, so it inherits all of the parent's functions.
public class Testing01 extends PApplet
{
// The main() method required to make a Java program run (at least in
// Eclipse).
public static void main(String[] args)
{
// Calling the overridden main() method and passing it the argument of
// the current class.
PApplet.main("Testing01");
}
// Will capture the incoming Serial data from the Teensy.
// Create serial object from Serial library.
Serial mySerial;
// Required method to create canvas.
public void settings()
{
size(1080, 720); // Setting size of canvas.
}
// The standard function in Processing to establish the fundamentals of the
// program.
// Similar to in C++ for Arduino.
public void setup()
{
background(0); // Clear frame; make completely black.
// constructor (this, the serial port, the baud rate).
mySerial = new Serial(this, Serial.list()[1], 115200);
fader = get(0, 0, width, height);
}
// serial-related declarations//
int[] incomingSerial = new int[6]; // Stores incoming Serial bytes for
// current frame.
int[][] mappedSerial = new int[4][6]; // Stores Serial bytes that have been
// converted to pixel values.
int lastStep; // Same as in C++
int currentStep; // Same as in C++
// Contains current (mapped) values, for comparison with interpolated
// values.
int[][] current = new int[4][5]; // does not include step count
/// Visual-related declarations.
PImage fader; // Image that will be scaled for delayed-feedback effect.
public void draw() // Analogous to loop().
{
// Clears frame.
background(0);
// Reduces alpha (transparency) for incoming projection of image of
// previous frame.
tint(255, 255, 255, incomingSerial[5]);
image(fader, 0, 0); // Projection of image of previous frame.
// Disables transparency settings for projection of elements in current
// frame.
noTint();
rectMode(CORNER); // Establishes top left corner as pixel (0,0).
// Retrieval of Serial from Teensy.
getSerial();
// Interpolates (blends) values of current ellipse dimensions with those
// just read over Serial.
interpolate();
// For reference/comparison.
lastStep = currentStep;
currentStep = incomingSerial[0];
// Drawing ellipses of current frame to canvas.
for (int i = 0; i < 4; i++) { // Setting ellipse color. fill(incomingSerial[1], incomingSerial[2], incomingSerial[3]); // Draws the actual ellipse (per iteration of loop). ellipse(current[i][0], current[i][1], current[i][2], current[i][3]); } // Overwrites previously captured image with updated capture of the most recent frame. fader = get(0, 0, width, height); } // Retrieve Serial from Teensy. void getSerial() { // Ensuring all 6 Serial values have been transmitted at a time. if (mySerial.available() >= 6)
{
for (int i = 0; i < 6; i++)
{
// The int() casting is apparently unnecessary
// Commit new Serial values to memory.
incomingSerial[i] = (mySerial.read());
// Mapped values are arbitrary.
// In this case, I'm using the dimensions of the canvas as the limits of the mapping.
mappedSerial[currentStep][i] = (int) (map((float) (incomingSerial[i]), (float) (0.0), (float) (255.0),
(float) (0.0), (float) (height)));
}
// Debug
// Prints Serial values to console.
for (int i = 0; i < 6; i++)
{
print(incomingSerial[i] + " ");
}
println(); // New line.
}
}
// Smoothly iterates between current positions/dimensions of objects on screen and their eventual targets,
// as defined by the incoming Serial data.
public void interpolate()
{
for (int i = 0; i < 4; i++)
{
for (int j = 0; j < 5; j++)
{
// Don't interpolate if the difference between the current position and target position
// is less than 3 pixels.
if (abs(current[i][j] - mappedSerial[i][j + 1]) < 3) current[i][j] = mappedSerial[i][j + 1];
// If the difference is greater than that, linearly interpolate.
else current[i][j] = linInt(current[i][j], mappedSerial[i][j + 1], 0.1);
}
}
}
// Linear interpolation.
// Method that combines map() and lerp() functionality
// into something easier for me to use in the Eclipse environment.
public int linInt(int a, int b, double c)
{
double delta = b - a; // Find difference between two points.
delta *= c; // Multiply difference by some factor.
return a + (int) (delta); // Add this scaled difference to first point.
}
}
Franklin, a guardian at the eLab, read my request, and set aside his time to help. I read out the measurements I acquired with my digital calipers of various important dimensions, including the diameter of the potentiometer shafts, the length and width of the stabilizing metallic protrusion on the sides of the pots, and the distance between these two items. I then determined how far apart I wanted the pots to be placed in relation to one another. Here is a list of the important measurements we found:
