Kyle McDonald is an artist who works in the open with code. He is a contributor to arts-engineering toolkits like openFrameworks, and spends a significant amount of time building tools that allow artists to use new algorithms in creative ways.
His recent experiment was to modify Andrej Karpathy’s “NeuralTalk” code to run from a webcam feed. McDonald recorded this live while walking near the bridge at Damstraat and Oudezijds Voorburgwal in Amsterdam, while visiting for IDFA DocLab.
NeuralTalk is trained on the MS COCO dataset, which guides the kind of captions that are generated mscoco.org/dataset/#captions-challenge2015 MS COCO contains 100k image-caption pairs covering a wide variety of situations. But in a brief walk, you will really only run into a few of those situations.
All processing is done on McDonald’s 2013 MacBook Pro with the NVIDIA 750M and only 2GB of GPU memory. The artist’s walking around with his laptop open pointing it at things, hence the shaky footage and people staring at themselves. The openFrameworks code for streaming the webcam and reading from disk is available at gist.github.com/kylemcdonald/b02edbc33942a85856c8