Update on Visualizing 2001: A Space Odyssey

One of my major goals in this project is to streamline the process of creating a visualization by freeing myself from ImageJ. I’m currently working on a Python script which nominally accomplishes this goal, and I figure this is as good a time as any to give you all an update.

My shotorg script makes use of several free software tools. Perhaps the most essential is shotdetect, a program written by Google software engineer Johan Mathe that computes the time at which each shot begins and exports the data to an xml file. From there it’s just a matter of using the Beautiful Soup Python library to read the data (specifically the millisecond at which each shot begins and how many milliseconds it lasts) and feed arguments to ffmpeg, which will rip the specified frames from the film and place them in their own directory.

Running the organize function on 2001: A Space Odyssey according to my current settings creates 596 folders and 27570 png files totaling 133.9 MB. This is the point in the process when I would normally turn to ImageJ; however, I’ve found that ImageMagick is able to do almost everything I want faster, simpler, and more reliably than ImageJ. I can also run ImageMagick from the command line, which makes it much easier to integrate into my Python script. The two visualization functions I’ve written so far, visualize and loop, transform each shot into a 1-row “film strip” montage and an animated gif, respectively. I wanted to make the visualize function go even further, composing each film strip into a larger montage of each shot in the film arranged chronologically from top to bottom, but doing so causes ImageMagick to run out of memory long before completing its task. Luckily I was able to run the function on several chunks of the film and finish the composition in GIMP.

This visualization’s original version was 112.1 MB with a resolution of 33000×21204, and each individual frame was only 80×36! I think the key element that sets a visualization like this apart from a bar chart (which it certainly resembles) is its interactivity. You can zoom in on any point in the graph until an individual frame fills your entire screen. However, this feature is for the most part neutralized when the image is resized for the web, and my computer can barely handle 100 MB images when I view them locally!

I’ll write more about how useful I think this process is in my next post. Meanwhile, if you’re interested in duplicating my results, feel free to download my horrifically crude and inelegant Python script and customize it for your computer. Creating visualizations for single scenes is super easy, just follow these steps:

ffmpeg -ss (start time) -t (duration of clip) -i (yourvideo) -r (frame rate) -s (size of ripped frames) -f image2 out%03d.png

This will start ripping frames from your video file at a specified point, naming them out000.png, out002.png, etc. (change %03d for more initial zeros). For example, entering

ffmpeg -ss 01:23:45 -t 00:00:05 -i /home/user/videos/my_video.mp4 -r 12 -s 1920x1080 -f image2 out%02d.png

will result start ripping 1920×1080 resolution frames from my_video.mp4 starting at 1:23:45 for 5 seconds, saving them as out00.png to out60.png. Now we can make a montage:

montage *.png -tile (colxrow) -geometry +0+0 -background black out.png

or a gif:

convert -delay 15 -loop 0 *.png out.gif


2001: A Space Odyssey as an Animated Barcode

2so reslice

Here you see a “movie barcode” created from 600 186×84 thumbnails which represent the first frame of every shot in 2001: A Space Odyssey. I originally stumbled upon this monstrosity by randomly playing around with some of ImageJ’s built-in functions. This particular one is called “Reslice;” it transforms a stack of images into something like the above, with the option of exporting it as a single frame or an animated gif. At first, I was genuinely baffled by what I had created—now I think I can explain it as follows.

The gif is 600 pixels wide, so it seems like a reasonable assumption that each 1px column represents a frame from the original film. The gif’s height is 186px, which corresponds to the width of the original thumbnails. And finally, the animation has 84 frames, which corresponds to height of the original thumbnails.

So effectively what we’re seeing is a series of 1px-wide slices of the film arranged chronologically from left to right. Over time, we see 84 lengthwise slices of each frame, cutting (I think) from top to bottom. Think of it as a throbbing film-strip style montage. I think this is an interesting way of reducing frames to sheer color, almost as if I were making 84 different color palettes per frame and juxtaposing them in time. Not sure how useful this will be as an analytic tool, but you have to admit it’s damn mesmerizing!

Eyes Wide Shut Graph


This is my first attempt at mapping frames onto an x-y axis. In this case, the x axis is median hue while the y axis is median saturation; all values were determined by Software Studies’s ImagePlot macro for ImageJ. I think the main limitation of this method is the sheer quantity of images I’m graphing. As you can see from my earlier post, the vast majority of frames in Eyes Wide Shut seem to be composed of very warm yellow colors; in this visualization, such frames would appear on the far left of the graph. However, since ImageJ processes the images in numerical order, chronologically early frame are often covered over by later ones. If you look closely, for instance, you can see that every frame from the ballroom scene has been obscured by later scenes (mostly the prostitute scene and the bedroom argument). The solution is obviously fewer images, but I’m not sure if I should simply decimate them or try to pick out representation frames from each shot.

Aside from that, I feel like this initial result is really promising. The graph really lets you visualize the extreme distance between Bill’s domestic life and his nocturnal wanderings, epitomized by the orgy scene. You can actually see the progression starting at the warm yellows of incandescent light bulbs and soft Christmas lights which illuminate Bill’s apartment. Next, the deep blues which characterize the scenes which most unsettle Bill and drive him into the New York underworld: Nick Nightingale’s description of his next job, Alice’s description of her “nightmare,” and Bill’s fantasy of his wife with the naval officer. Almost all of the reds occur in the costume store (although oddly juxtaposed with the final scene at the toy store), which serve as the gateway to voluptuous pinks and violets. It’s interesting to note that, at the bottom of the saturation spectrum, Bill continues to go about his normal life—scenes from the office, from the hospital, walking home, etc.—completely oblivious of the turmoil above.

I’m excited to see how the rest of the visualizations come out. Obviously there is still tweaking to be done—I’m not sure what to make of the three or four vertical lines—and I’d like to be able to represent individual scenes in some way. Progress is slow, as each visualization takes close to an hour to complete. Nevertheless, we’ll soldier on; watch this space for further updates!