Creative functional programming with Ramsey Nasser

Foreword: I first encountered Ramsey as I watched his StrangeLoop video from 2014. He was talking about Arcadia; a way to write high performance video games in Clojure. A project he and Tims Gardner were working on. For me this was the smashing together of two of my favourite technologies and so I was amazed and excited. Over the last four years Ramsey has continued to do weird and wonderful things with code. Sometimes Clojure, sometimes other mainstream languages. Sometimes languages he's created himself for the entire purpose of visualisation and why on Earth not?

(source: Ramsey Nasser Twitter @ra)

1 In case people aren't familiar with your work, you appear to focus mainly on 'creative' coding: games, visualisations, media. What is it about these kinds of projects that you enjoy and where does that interest come from?

I've found "creative coding" to be a source of consistent work, a ton of fun, seriously challenging, and a great way to connect with other human beings. In New York City there's a robust freelance industry that revolves around building interactive game-like experiences that has been my primary source of revenue for the past half decade. I tend to get work subcontracting Augmented or Virtual Reality work for design firms in the city who are working with a larger client, as many of my friends and colleagues do. I've had the privilege of working on installations, out-of-box-experiences, prototypes, and a lot more. For a sense of what I mean, here's a project I made a few years ago.

A big part of the allure for me is that at the end of an aesthetic project you have something to look at or play with, which I find super rewarding compared to other kinds of programming. It won't come as a surprise that I actually enjoy playing games and interacting with digital art, so being able to make the things I love is great incentive. There's also a wonderful community of people who do similar work in New York and around the world that have come to be my second family and my circle of friends. Playing each other's games and interacting with each other's art is part of how we support each other and keep in touch.

As a programmer, I find the challenges that present themselves in these areas to be the most interesting and extreme I've faced. My friend Brandon Bloom once said (and I quote this just about everywhere!) "making a video games means solving every hard problem in computer science sixty times a second." And that's totally true. Making games forces me to channel my absolute best networking, graphics, software architecture, AI, and optimization techniques every time. On top of the technical challenges, you're trying to make something where success is subjective. You could have all the best code in the world and still have made a terrible game or a boring piece of art. This forces you to not only be technically proficient, but also build systems that are nimble enough to change with user feedback. You basically never really know what you're doing and you're constantly course correcting, and I find that thrilling.

Finally, games and art are kind of a social cheat code that I use to connect with other people. The kind of stuff I get excited about -- compilers and language theory -- can be pretty alienating to anyone not engaged in that kind of work. But people understand games and digital art and can connect with it. Watching a stranger play a game I made never gets old, nor does watching people at a party dance a long to visuals I am livecoding. I find making experiences for other humans fulfilling in a way that programming towards any other end doesn't just stack up.

2 Arcadia is an ambitious effort to bring Clojure to the popular game engine, Unity 3D. You're clearly a big fan of Clojure - why is that? And why do you believe Clojure is a good language for game development?

Clojure walks a line of functional purity and pragmatism better than any language I have used. I find myself able to build very large systems that still fit in my head in Clojure in a way that I've found difficult in other languages, and its interactive development flow is by a wide margin the best I have experienced. It is simple and minimal in very specific ways that take a lot of the ceremony out of rapidly building large projects.

For game development this is a godsend. It's very common to want to tweak some logic in a part of a game while you are playing with it or debugging a problem. In traditional game development pipelines this means quitting the game, making the code changes, restarting the game, and manually returning to the game state you were at previously if you can. If you're testing on a mobile device or console, this can take up to several minutes. Unity makes this somewhat better by simulating a live coding workflow, but it is full of caveats and only works on desktop. With Arcadia, swapping out a function while a game is running is a single keystroke in my editor and it works without caveats on desktop and Android phones. The difference is night and day. On top of that, being able to manage my game's state using persistent data and functional programming means entire classes of bugs fall away and make the development process a lot smoother.

3 I know Arcadia has been used in a few game jams, but are you aware of any serious development efforts using it? What prevents more adoption, do you think?

The current version has been mostly among hobbyists and indie developers. There is a wonderful community of Arcadia users who hang out on our Gitter channel and make great games and support each other in a way that makes me very happy and proud.

My collaborator Tims Gardner and I were contracted this summer to work on an exploratory prototype of an Arcadia-based mobile game framework for a game development studio. Beyond that, I'm not aware of any commercial use yet. For a long time the project has been in the "rough and ready" state that a lot of unfunded open source software exists in. The documentation hasn't been great, there were unresolved design and technical issues pending, and getting up and running usually meant asking questions to the community. None of this is severe enough to deter indies or hobbyists, but for a commercial studio to stake a bet on new technology I think a lot of these realities have been non-starters.

The good news is that we actually received funding this summer and have been working on Arcadia full time since September! It's the FOSS dream and we're very grateful. Our first order of business is to stabilize and document Arcadia itself, which will remain open source. We're closing in on a beta release as I write this. Following that we're planning on building proprietary tools on top of Arcadia and Clojure specifically geared towards commercial game development. We're hoping that the mix of an open source foundation with proprietary products on top can keep us afloat and able to support the project and community better than we have in the past.

4 You've actually been working on your own Lisp for some time now. Can you tell us a little about that and why you decided to target the CLR?

Yes! It's called Jn. It's a Lisp that is inspired by Clojure, Julia and Extempore. It comes out of my experience working on ClojureCLR and the MAGIC compiler I am building for Arcadia. Jn is mainly an opportunity for me to experiment with language design and compiler construction in a low-stakes environment and without backwards compatibility concerns. ClojureCLR and MAGIC are fun to work on, but the fact that there is a larger project and community -- and now a company -- riding on them makes me more conservative and less willing to try out weird ideas that might turn out to be wastes of time. A personal language project is the perfect place for that!

The goal is to make a high-performance, live coded, primarily functional Lisp for the kind of creative coding I do. It gets its syntax and data structures from Clojure, though it is likely to be more typed and less centered on lazy semantics. Its functions work similarly to those in Julia, where every function is a generic function with an open set of overloaded methods chosen by multiple dispatch. It has other tricks up its sleeve, too, like inline bytecode. Here's how + could be defined for integers:

(defn + [^System/Int32 a ^System/Int32 b]

  (cil* System/Int32

        (ldarg-0)

        (ldarg-1)

        (add)))

cil* is a special form that expands into raw Common Intermediate Language bytecode. It's a powerful way to build the language's fundamentals in the language itself and an escape hatch if you ever need to do anything wild and custom. Just don't break the stack!

I targeted the CLR because it's the VM I know best, for sure, but it's also the VM that has all the properties I'd want in a target. The CLR is cross platform, has a decent multithreaded garbage collector, C interop, an LLVM JIT backend, an ecosystem of existing code, and a rich bytecode language that is amenable to really interesting analysis and cross-compilation. Given that my end goal is going to be something aesthetic, high performance access to, e.g. OpenGL is crucial, and the CLR does a good job at that.

5 For those of us who aren't cool enough to know, what are "algoraves" and what's "live coding" in this context? How do the tools you've built help you do this/perform? Can you explain a bit about "Thixels" and where this fits in?

(source: Ramsey Nasser Twitter @ra)

The Live Coding movement is a trend across the world of people using programming systems that generate music and visuals as instruments on stage in front of an audience at parties called "Algoraves." I'm one of the organizers of livecode.nyc, the New York City community. We meet once every two weeks and put on at least one Algorave a month!

Different communities have different rules, and we're pretty lax in New York, but generally people are writing code on stage to generate live music in environments like Sonic PI, SuperCollider, Overtone, Tidal Cycles and others. There's a strong aesthetic to showing your code as you perform, so your audience can see what you're doing as they party. Accompanying the music and its code is often a visualist, also working in their own live coded environment to generate concert visuals. The tools here are more varied, and many artists I know actually make their own.

Thixels is my own live coding visual tool that is programmed in Clojure and emulates the look of the fantasy console Pico 8. Though it is Clojure, it's not a terribly functional system, and it feels more like a Processing-style drawing tool than anything else. The extreme constrains inherited from Pico 8 -- 128x128 pixels and 16 colors, a "cozy design space" in the words of Pico 8's inventor Joseph White -- are actually a ton of fun to play with and are read really well by an audience.

Here's what it looks like (with comments included to explain the cryptic terse API that is optimized for rapid live typing -- I certainly do not comment my code on stage!):

(defn draw [t]

  ;; background

  (cyc 25     ;; every 25 frames, run one of these expressions

       (bg 4) ;; color the background with color 4

       (bg 2)

       (bg 3)

       nil    ;; dont do anything, the background will not be cleared!

       (bg 6)

       nil

       nil)

  ;; rectangles

  (forl [i 0 10] ;; loop i from 0 to 10

        (let [i (- 10 i)

              i (* 4 i)]

          (frect (- 64 i) ;; draw some filled rectangles

                 (- 64 i)

                 (+ 64 i)

                 (+ 64 i)

                 (+ 8 (mod (+ (* 4 t) i) 8))))) ;; with these colors

  ;; wavy pattern

  (forl [i 0 128] ;; loop i from 0 to 128

        (rrow i   ;; roll row i by the number of pixels in the next expression

              (* 4 (sin (* (cyc (/ 25 1) 0.5 0.075 0.01)

                             (+ (* 8 t) i)))))))

The result is this animation:

What time to be alive when writing Lisp at parties is considered cool!

6 If people are looking to develop games or visualisations using Clojure or your own set of tools, where would you recommend they start?

There are a few options for games, like play-clj, but I can't comment on anything but Arcadia as I haven't really used anything else. The community is very kind and supportive, and the tools will only get better now that we're working on them full time. Check us out for sure!

If people want to get into Thixels, clone the repo and give it a whirl! I develop and test on Linux, but it should work on OSX as well. PRs/feedback welcome!

7 What's next on the horizon for Ramsey Nasser?

More language stuff, I hope! By 2016 I realized that I really loved working on compilers, but didn't know how to make it "my job." Through some hard work and a lot of luck, it looks I have the opportunity to do just that for the next few years. Arcadia has me focusing on Clojure and MAGIC, I am working on Jn in stolen moments, and I am also building a language called OSKAR for the animator Larry Cuba. I am extremely grateful for this pivot and I am learning a lot each day!

I am having a great time doing live coded visuals, but I would love to figure out a way to do music as well. I've been trying to reproduce the sound of synthesizer wedding music from the Levant where I grew up in a live coded environment, basically a Lisp REPL into this.

Something I would like to do more of is share my process and what I am learning with others. I don't know if that means writing and blogging more, live streaming my compiler work, or what, but I would really like to be able to put these skills in other people's hands so that they can share in the joy of language design live coding If I figure something out I will let you know!

R