bunnyhero dev

iOS 4 added a lot to AVFoundation, including classes and APIs that give you much more control over the iPhone camera. One of the things you can now do with the camera is read the video frame data in real time.

In this post, I’ve created a simple demo that simulates a Twilight-style vampire. In the Twilight series, vampires aren’t hurt by daylight; instead, they sparkle. Yes, sparkle.

Here are a couple of screenshots from the app:

And here’s a low-quality video of the vampire simulator in action.

The app detects the amount of light shining on the phone by doing very simple image analysis of the incoming video frames from the camera. The brighter the image seen by the camera, the more sparkles it draws on the vampire.

So how does this all work?

AVCaptureSession

The AVCaptureSession object is the centre of the new video and audio input/output universe. An AVCaptureSession can have multiple inputs and outputs. Inputs include video sources (the cameras) and audio sources (microphone). Outputs can be things like a file, or (in this example) an object that captures every frame of video data as it comes from the camera.

The following method demonstrates how to create an AVCaptureSession and add a video input from the front-facing camera (we’re using the front camera because it is more likely to be facing upward, to measure the ambient light) and an output to an object that will let us inspect every video frame that comes from the input.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70

- (void)start
{
    NSError *error = nil;
 
    AVCaptureDevice *captureDevice = [self frontFacingCameraIfAvailable];
    AVCaptureDeviceInput *videoInput = [AVCaptureDeviceInput deviceInputWithDevice:captureDevice error:&error];
    if ( ! videoInput)
    {
        NSLog(@"Could not get video input: %@", error);
        return;
    }
 
    //  the capture session is where all of the inputs and outputs tie together.
 
    captureSession = [[AVCaptureSession alloc] init];
 
    //  sessionPreset governs the quality of the capture. we don't need high-resolution images,
    //  so we'll set the session preset to low quality.
 
    captureSession.sessionPreset = AVCaptureSessionPresetLow;
 
    [captureSession addInput:videoInput];
 
    //  create the thing which captures the output
    AVCaptureVideoDataOutput *videoDataOutput = [[AVCaptureVideoDataOutput alloc] init];
 
    //  pixel buffer format
    NSDictionary *settings = [[NSDictionary alloc] initWithObjectsAndKeys:
                              [NSNumber numberWithUnsignedInt:kCVPixelFormatType_32BGRA],
                              kCVPixelBufferPixelFormatTypeKey, nil];
    videoDataOutput.videoSettings = settings;
    [settings release];
 
    //  we don't need a high frame rate. this limits the capture to 5 frames per second.
    videoDataOutput.minFrameDuration = CMTimeMake(1, 5);
 
    //  we need a serial queue for the video capture delegate callback
    dispatch_queue_t queue = dispatch_queue_create("com.bunnyherolabs.vampire", NULL);
 
    [videoDataOutput setSampleBufferDelegate:self queue:queue];
    [captureSession addOutput:videoDataOutput];
    [videoDataOutput release];
 
    dispatch_release(queue);
 
    [captureSession startRunning];
}
 
- (AVCaptureDevice *)frontFacingCameraIfAvailable
{
    //  look at all the video devices and get the first one that's on the front
    NSArray *videoDevices = [AVCaptureDevice devicesWithMediaType:AVMediaTypeVideo];
    AVCaptureDevice *captureDevice = nil;
    for (AVCaptureDevice *device in videoDevices)
    {
        if (device.position == AVCaptureDevicePositionFront)
        {
            captureDevice = device;
            break;
        }
    }
 
    //  couldn't find one on the front, so just get the default video device.
    if ( ! captureDevice)
    {
        captureDevice = [AVCaptureDevice defaultDeviceWithMediaType:AVMediaTypeVideo];
    }
 
    return captureDevice;
}

The code is (I hope) pretty straightforward. A few things to note:

• (line 49) The frontFacingCameraIfAvailable method gets the AVCaptureDevice that represents the front-facing camera, if it exists. AVCaptureDevice makes it easy to enumerate all of the capture sources on the device for any given media type (e.g. video, audio). We simply query each device in the list to see if it’s on the front or the back of the phone.
• Since all we’re doing is measuring the average brightness of the picture, there’s no need to waste resources by grabbing full HD video at 30 frames per second. The sessionPreset property of AVCaptureSession (line 20) governs the resolution of the capture, and the minFrameDuration property of AVCaptureVideoDataOutput (line 35) limits the frame rate.
• (line 28) We tell the AVCaptureVideoDataOutput object what format we want the frames in. Available options are Y’CbCr and RGB. RGB data is simple to understand, so we ask for that with the kCVPixelFormatType_32BGRA constant.
• (lines 38-44) AVCaptureVideoDataOutput sends the captured video frames to its delegate. The “queue” argument in setSampleBufferDelgate:queue: is a Grand Central Dispatch queue. I’m not going to go into detail about GCD queues here. Luckily, creating a suitable queue can be done with a single line of code. It can be released immediately after setting the delegate because the AVCaptureVideoDataOutput retains the queue.

Once the capture session starts (with startRunning), every frame of video seen by the camera is sent to the delegate object by callings its captureOutput:didOutputSampleBuffer:fromConnection: method. My implementation of this delegate callback is below.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33

- (void)captureOutput:(AVCaptureOutput *)captureOutput
didOutputSampleBuffer:(CMSampleBufferRef)sampleBuffer
       fromConnection:(AVCaptureConnection *)connection
{
    CVImageBufferRef imageBuffer = CMSampleBufferGetImageBuffer(sampleBuffer);
    if (CVPixelBufferLockBaseAddress(imageBuffer, 0) == kCVReturnSuccess)
    {
        UInt8 *base = (UInt8 *)CVPixelBufferGetBaseAddress(imageBuffer);
 
        //  calculate average brightness based on a naive calculation
 
        size_t bytesPerRow      = CVPixelBufferGetBytesPerRow(imageBuffer); 
        size_t width            = CVPixelBufferGetWidth(imageBuffer); 
        size_t height           = CVPixelBufferGetHeight(imageBuffer); 
        size_t pixelCount       = width * height;
        UInt32 totalBrightness  = 0;
 
        for (UInt8 *rowStart = base; height; rowStart += bytesPerRow, height --)
        {
            size_t columnCount = width;
            for (UInt8 *p = rowStart; columnCount; p += 4, columnCount --)
            {
                UInt32 value = (p[0] + p[1] + p[2]);
                totalBrightness += value;
            }
        }
        CVPixelBufferUnlockBaseAddress(imageBuffer, 0);
        Game *theGame = self.game;
        dispatch_async(dispatch_get_main_queue(), ^{
            [theGame updateSparklesWithBrightness:(float)totalBrightness/(255 * 3 * pixelCount)];
        });
    }
}

Note that I have not tried to optimize this code at all. This code loops through every pixel in the frame and sums the red, green and blue components. When it’s done, it calculates the average by dividing by the number of pixels and components. It also converts the 8-bit pixel value (0-255) it to a floating-point value in the range of 0 to 1.0. It uses GCD to call the main app on the main thread with the brightness value. The method updateSparklesWithBrightness: in my app adds or removes sparkle graphics based on how bright the camera image is.

Next week I’ll present the rest of the app, which uses the Sparrow framework to display and animate the sparkles.

Tagged avfoundation, idevblogaday

§ 7 Responses to Turn your iPhone into a vampire with AVFoundation and iOS 4"

• Bhagirath says:

  2010 November 25 at 2:06 am

  Hi bunnyhero
  its good blog, can you please send me source code?

• bunnyhero says:

  2010 December 6 at 4:21 pm

  Right now the only source code I’m sharing is what’s already in the blog post. Sorry about that.

• Dev Kanchen says:

  2010 December 24 at 4:55 am

  While I’m not a fan of the Twilight series, LOVE the idea and the implementation. :) Kudos.

  P.S: I haven’t check the rest of the blog yet, but is the app out? Or was this a proof-of-concept/hobby project?

• bunnyhero says:

  2011 January 14 at 6:33 pm

  I haven’t finished the app yet. It was more of a proof-of-concept thing but I might add a few features and submit it to the store anyway :)

• Prabhjot Singh says:

  2011 July 5 at 3:05 am

  hi, any idea how we can add normal camera controls on this view? Like Flash, flip etc

• CoreGr says:

  2011 July 6 at 3:41 am

  Thanks very much for your post. It has helped me to know how to access the front camera :)

• Eric says:

  2011 November 15 at 8:40 pm

  Really cool code :) I was looking for a way to determine brightness from the camera and viola!