After discussion my minor breakthrough yesterday, it was suggested that I try adding in some face capture to enhance the uncanny effect. I set up a quick and dirty test again and gave it a go.
The majority of the work is done in VDMX again, using a plugin which handles face capture (how convenient). For this I just used the webcam on my laptop and this plugin, then set the output of the plugin to syphon out so that I could use it as an input in MadMapper. From here I created four new quads and mapped them onto the face areas of the miniature sculptures. I also added a mask to the input as straight out of madmapper the video is a rectangle which includes some background so I crudely masked it out.
This test was more just a proof of concept, just to see if I could do it - turns out I can. However, without some major optimisation I’m not sure this is going to be possible given my current computing situation. As you might be able to see in the taskbar of the image below, my CPU was running at 99 degrees… and that really isn’t okay. Even more so, looking at the processes, VDMX combined with madmapper was using around 400% of the CPU. Now I’m now mathematician bit im pretty sure thats more than 100%. As a result, I’m not comfortable with leaving my computer running like this for extended periods of time, as it is likely to cause some damage.
On the brightside there are some easy improvements to be made as, I didn’t really take much care while putting together this prototype (I wanted it done as fast as possible). Firstly I can optimise the videos, both in terms of resolution and the codec used. At the moment the video for each of the sculptures is aproximately 720p, but given their shape and size, it’s pretty unnecessary. Also, when projecting glitchy things having a high resolution isn’t essential, the low-res pixellated look can just add to the ‘glitchy aesthetic’. Also I was using MPEG-4 codec for the videos as this renders out quickly and has a low file size. I’ve read that it is recommended to use photojpeg or Apple Intermediary codecs as they are less resource intensive when used in VDMX.
Below is a short video with some footage from the test. I reused some of my existing glitchy footage and just had the captured face overlaying it. On this small scale it is difficult to encounter any added uncanniness, but with some imagination the effect could be visualised. In hindsight, I think it would be worth glitching the face too, see what effect that would have. I imagine at full scale it would look quite impressive.
If I was to continue chasing this idea (after fixing obvious optimisation issues of course), there are a few things I will need to consider. Firstly it is the placement of the camera and lighting. For the projection to look most impressive I need a dark room, but for the camera to detect faces I need light. Also I need the camera to be in front of the sculptures so that it can detect the faces of audience members looking at the sculptures so the positioning will need to be considered. Also, again, the poses of the sculptures comes under consideration. Perhaps it could be interesting to have the standing sculpture pointing towards where the camera is (or even secretly holding the camera) so that once it captures the face of the viewer the work suddenly changes and the viewer is now pointing at themselves. While this is happening, we have two onlookers (assuming keeping the tripartite installation) who also exhibit the viewers face. So we have the viewer watching the viewer point at themselves - quite an experience. All this while glitching and distorting, resulting in some wild, confusing, uncanny encounter. I will also need to think how best the glitches can ehance the uncanny experience when taking the face capturing into consideration. This can only come from more experimentation.
I find this idea really interesting but I am apprehensive due to technological limitations. It is an interesting way of adding interactivity to the work by forcing the viewer to be a part of it. They give no consent to having their face stolen, they just happen to be standing in the right place at the right time and the project works it’s magic. It is something that is unexpected when encountering sculpture in a gallery context, which is traditionally a very passive experience.