Kind of as reply to Tom’s mini howto for using SQLite with Processing, but also since I’ve been dabbling with it myself recently, here’s an alternative take on using an embedded database from within Processing (or more generally in Java)…

Mainly due to Java’s strong focus on server side development, over the past few years there have been several large scale community efforts to create Java native database engines, which don’t rely on underlying C code, are high performant and portable: the essence of the Java way. The other benefit is that such DB engines can be embedded and distributed with your application without requiring any further installation. One such development effort is Apache Derby, a project which started in 1996, swapped owners several times (amongst them IBM) and then became an incubator project at Apache in 2004. Sun also joined the project and has been bundling it as library (under the name JavaDB) as part of the JDK (v6+) since December 2006. So in other words if you have Java6 installed you also should have Derby. But even if you don’t (for example Mac users), you can download Derby from here and unzip it to any folder on your hard drive.

The following deals with setting up Processing to work with Derby:

1. Create a new folder structure /derby/library within Processing’s /libraries folder
2. Copy the file derby.jar from Derby’s /lib folder into the newly created folder

Before we can start using the database now we first need to create a new database. Derby comes with its own commandline client “ij” which is located in the /bin directory of the main Derby install dir (If you’re going to use this tool more often it would make sense to add this /bin directory to your system path).

Launch ij from the commandline and then create a new database with this command:

ij> connect 'jdbc:derby:/path/to/database;create=true';

Databases are just folders and can be stored anywhere. For example on Windows the path /dev/derby/mydb would refer to C:\dev\derby\mydb…

Next create some a simple table in the new database and add some data:

ij> create table cities (cityID integer not null primary key,name varchar(32) not null);insert into cities values(1,’london’);insert into cities values(2,’berlin’);insert into cities values(3,’san francisco’);

Given that all worked fine so far we can finally move on to a small Processing demo to query our exciting dataset:

import org.apache.derby.*;import java.sql.*;

String driver = “org.apache.derby.jdbc.EmbeddedDriver”;String connectID = “jdbc:derby:/derby/testdb”;Connection conn;

void setup() {  size(100,100);  try {    query();  }  catch(SQLException e) {    e.printStackTrace();  }}

void query() throws SQLException {  try{    Class.forName(driver);   }   catch(java.lang.ClassNotFoundException e) {    e.printStackTrace();  }

  try {    conn = DriverManager.getConnection(connectID);    Statement st=conn.createStatement();    ResultSet results=st.executeQuery(”SELECT * FROM cities”);    while (results.next()){      println(”City: “+results.getString(”name”));    }    results.close();  }  catch (Exception e)  {    e.printStackTrace();  }  finally {    // always make sure we close the connection    if (conn!=null) conn.close();  }}

Now, this is obviously an absolute bare bones demo, however Florian Jennet wrote a little database library last year to hide all these excessive try/catch clauses. Unfortunately he’s also hardcoded the database connection string to only work with MySQL and there’s no source supplied with the library so one could change it and easily add support for other JDBC drivers… (nudge!

Speaking of databases, here’s another food for thought. SQLite, MySQL, Apache Derby et al are all based on the relational database model. Java on the other hand is object oriented and as you can see it takes a relative large effort to exchange data between both worlds without any further help. To ease these tasks there’re various powerful object relationship mappers available acting as translators between the worlds of objects and that of SQL. Last but not least, native object oriented databases are yet another alternative approach here which is far more aligned with the language. db4o is such an object database and allows you to store and query complex object hierarchies in a most natural (in a Java context) way. For example with db4o you can store and restore the entire state of an application - with just a single line of code. This in turn not just saves you a lot of time and headaches, but also enables building more complex, data intensive applications. If you’re interested, the db4o site has a very easy to follow tutorial…

New software from Protohaus added to the exhibition.

SQLite is a great standalone SQL database engine - not ideal for every situation (particularly large websites), but more than good enough to have already made its way into desktop projects from Apple, and more recently Adobe and Google.

My colleague Mike recently used it as a way to distribute data from a 511.org transit website scraping project he’s been doing. I wanted to see if I could download his data (gathered and processed using python) and access it using Processing.

Web searches for SQLite and Java (java wrapper sqlite, etc.) turn up lots of matches, including this promising tutorial from Tim Anderson, but sadly the most prominent matches didn’t work for me. It’s a real pain to get up and running and installed in a reliable way, mainly due to the need to compile SQLite natively for each platform and then talk to the native code using Java.

Enter this amazing project, a JDBC library for SQLite written in pure java that uses NestedVM to compile the C code for SQLite into something any Java VM can use. It’s frightening to think about the amount of misdirection, abstraction and interpretation going on here, but it worked first time and plenty fast enough for my purposes.

Take a look after the jump for the code I ended up with. I hope people find it useful:

// download the pure java lib from http://www.zentus.com/sqlitejdbc/
//   ... then put the jar file in the code folder (drag and drop it onto this sketch)

// I read http://www.pysquared.com/files/Java/JavaSQLiteExample/ for a basic primer too

import java.sql.*;

void setup() {

  size(100,100);
  noLoop(); 

  try {

    // It's ugly, even for a java idiom, but you need to do this so that
    //  DriverManager doesn't throw an error looking for sqlite.
    Class.forName("org.sqlite.JDBC");

    // create a database, or open an existing database, in the sketch data folder:
    // I have no idea if this works in an applet yet
    String dbFileName = dataPath("transit.db");
    println("opening: " + dbFileName);

    // this normally expects a URL, hence the funky jdbc:sqlite: protocol before the file path
    Connection conn = DriverManager.getConnection("jdbc:sqlite:"+dbFileName);

    // http://java.sun.com/j2se/1.4.2/docs/api/java/sql/Connection.html
    // NB: the docs say "If the same SQL statement is executed many times, it may
    // be more efficient to use a PreparedStatement object."
    Statement stmt = conn.createStatement();

    // http://java.sun.com/j2se/1.4.2/docs/api/java/sql/Statement.html
    // executeQuery returns "true if the first result is a ResultSet object;
    //                       false if it is an update count or there are no results"
    ResultSet rs = stmt.executeQuery("SELECT * FROM stops LIMIT 10"); // delete "LIMIT 10" to get everything!

    // http://java.sun.com/j2se/1.4.2/docs/api/java/sql/ResultSet.html
    // "A ResultSet cursor is initially positioned before the first row;
    //  the first call to the method next makes the first row the current row"
    while (rs.next()) {
      String provider   = rs.getString("provider");
      String route_name = rs.getString("route_name");
      String schedule_name = rs.getString("schedule_name");
      String stop_location = rs.getString("stop_location");
      String stop_time = rs.getString("stop_time");
      String schedule_url = rs.getString("schedule_url");

      println("provider: " + provider);
      println("route_name: " + route_name);
      println("schedule_name: " + schedule_name);
      println("stop_location: " + stop_location);
      println("stop_time: " + stop_time);
      println("schedule_url: "+ schedule_url);
    }

    conn.close();
  }
  catch(Exception e) { // ClassNotFoundException, SQLException
    e.printStackTrace();
  }

}

void draw() {
  // nothing yet!
}

Cast: ghedo

As we delve deeper into the realms of applied generative design and deal with a whole population of possible design outcomes, we often find ourselves preferring certain outcomes more than others and want to narrow down our explorations. So the identity of each such design plays an important role. Identity in this context can be defined by the set of input parameters used, but we also need to ensure the processing of these parameters is deterministic, meaning that even though we often use (pseudo)randomness as part of the algorithm, the outcome should be replicable for each set of parameters.

Most (if not all) pseudo-random generators use the concept of a random seed which subsequently produces a unique (and deterministic) sequence of “random” numbers. In Processing you can use both randomSeed() and noiseSeed() to achieve this. Now while using numbers is all fine, and technically speaking, all digital media is just numbers - there’re use cases where it’d be nicer to use e.g. text as seed directly. For example, the 20,000 designs of the Lovebytes fluffies are all based on their generated character name only. There’re about 10 other parameters, but these too are chosen based on the random sequence seeded by the name.

One way of turning a string into a number is by using message digests, like the popular MD5 or SHA1 algorithms. A message digest takes any number of bytes as input and calculates a fixed length hash. MD5 results in a number 128 bits long and SHA1 160 bits. This is more data than we can cope with since most common random number generators only accept up to 64 bits as input. In Java/Processing this is equivalent to the long type.

The following function takes a string as input, computes the hash and then returns the first 8 bytes as long integer to be used as random seed. Because it doesn’t use the full hash it’s possible in theory to end up with the same result for different inputs. However, I’ve not yet managed to come across a collision with the relative short strings (names, sentences, phrases) used in my work.

import java.security.*;

/** * Calculates the message digest of the given string and * returns the first 8 bytes packed into a long * * @param msg string to form hash from * @param digest message digest ID (e.g. “MD5″ or “SHA1″) * @return zero if failed, else partial digest as type long  */long getLongHash(String msg, String digest) {  long result=0;  try {    MessageDigest md = MessageDigest.getInstance(digest);    md.update(msg.getBytes());    byte[] buffer=md.digest();    for(int i=0,bits=56; i<8; i++) {      long val=(buffer[i]<0 ? 0×100+buffer[i] : buffer[i]);      result|=val< 
And again, use it like that:
long seed=getLongHash("Hello world!","MD5"); // "SHA1" as alternative
noiseSeed(seed);
Btw. The default Java Random generator does not guarantee to produce a deterministic sequence across all platforms. This means as long you’re using Processing’s default random() or noise() functions you’re only guaranteed the same sequence as long as you stay on either Windows or OSX or Linux. Last year Marius wrote a Processing wrapper for the famous Mersenne Twister generator, however this one can only be used as alternative and in isolation. Processing’s noise() function is hardcoded to use the default Java generator…
Tags: generative, design, randomness, md5, sha1, seed, determinism, code, lovebytes

 

I’ve started working on a new code-driven graphics project in processing using toxi’s vector library which has turned out quite fitting so far. It’s basically a series of pull/push points in space which direct a number of ribbons.

more soon…

This month I completed a custom-software installation for COS. It’s all built with processing and is running until the sale ends (mid-August) in the stores in London, Berlin and Stuttgart. This screenshot is from the close-to-finished graphics.

…and here is a shot of the final graphics running in-store at the Regent Street store in London. The projection has two main render-modes: 1) a dotted grid cycling through press-photos from the launch of the brand and 2) a typewriter style renderer showing text being typed (with spelling mistakes - which are corrected) and hessitating pauses to simulate a person writing it.

Another try with the surfaceLib. Now different textures.

Author: eskimoblood
Keywords: processing.org 3d surface visual
Added: July 28, 2007

Created in Processing. More info at http://www.nofi.org/2007/04/10/processing-random-spheres-01/

Cast: nofi