iOS 4 added a lot to AVFoundation, including classes and APIs that give you much more control over the iPhone camera. One of the things you can now do with the camera is read the video frame data in real time.
In this post, I’ve created a simple demo that simulates a Twilight-style vampire. In the Twilight series, vampires aren’t hurt by daylight; instead, they sparkle. Yes, sparkle.
Here are a couple of screenshots from the app:
-
- Vampire in low light
-
- Vampire in bright light
And here’s a low-quality video of the vampire simulator in action.
The app detects the amount of light shining on the phone by doing very simple image analysis of the incoming video frames from the camera. The brighter the image seen by the camera, the more sparkles it draws on the vampire.
So how does this all work?
AVCaptureSession
The AVCaptureSession object is the centre of the new video and audio input/output universe. An AVCaptureSession can have multiple inputs and outputs. Inputs include video sources (the cameras) and audio sources (microphone). Outputs can be things like a file, or (in this example) an object that captures every frame of video data as it comes from the camera.
The following method demonstrates how to create an AVCaptureSession and add a video input from the front-facing camera (we’re using the front camera because it is more likely to be facing upward, to measure the ambient light) and an output to an object that will let us inspect every video frame that comes from the input.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 | - (void)start { NSError *error = nil; AVCaptureDevice *captureDevice = [self frontFacingCameraIfAvailable]; AVCaptureDeviceInput *videoInput = [AVCaptureDeviceInput deviceInputWithDevice:captureDevice error:&error]; if ( ! videoInput) { NSLog(@"Could not get video input: %@", error); return; } // the capture session is where all of the inputs and outputs tie together. captureSession = [[AVCaptureSession alloc] init]; // sessionPreset governs the quality of the capture. we don't need high-resolution images, // so we'll set the session preset to low quality. captureSession.sessionPreset = AVCaptureSessionPresetLow; [captureSession addInput:videoInput]; // create the thing which captures the output AVCaptureVideoDataOutput *videoDataOutput = [[AVCaptureVideoDataOutput alloc] init]; // pixel buffer format NSDictionary *settings = [[NSDictionary alloc] initWithObjectsAndKeys: [NSNumber numberWithUnsignedInt:kCVPixelFormatType_32BGRA], kCVPixelBufferPixelFormatTypeKey, nil]; videoDataOutput.videoSettings = settings; [settings release]; // we don't need a high frame rate. this limits the capture to 5 frames per second. videoDataOutput.minFrameDuration = CMTimeMake(1, 5); // we need a serial queue for the video capture delegate callback dispatch_queue_t queue = dispatch_queue_create("com.bunnyherolabs.vampire", NULL); [videoDataOutput setSampleBufferDelegate:self queue:queue]; [captureSession addOutput:videoDataOutput]; [videoDataOutput release]; dispatch_release(queue); [captureSession startRunning]; } - (AVCaptureDevice *)frontFacingCameraIfAvailable { // look at all the video devices and get the first one that's on the front NSArray *videoDevices = [AVCaptureDevice devicesWithMediaType:AVMediaTypeVideo]; AVCaptureDevice *captureDevice = nil; for (AVCaptureDevice *device in videoDevices) { if (device.position == AVCaptureDevicePositionFront) { captureDevice = device; break; } } // couldn't find one on the front, so just get the default video device. if ( ! captureDevice) { captureDevice = [AVCaptureDevice defaultDeviceWithMediaType:AVMediaTypeVideo]; } return captureDevice; } |
The code is (I hope) pretty straightforward. A few things to note:
- (line 49) The
frontFacingCameraIfAvailablemethod gets theAVCaptureDevicethat represents the front-facing camera, if it exists.AVCaptureDevicemakes it easy to enumerate all of the capture sources on the device for any given media type (e.g. video, audio). We simply query each device in the list to see if it’s on the front or the back of the phone. - Since all we’re doing is measuring the average brightness of the picture, there’s no need to waste resources by grabbing full HD video at 30 frames per second. The
sessionPresetproperty ofAVCaptureSession(line 20) governs the resolution of the capture, and theminFrameDurationproperty ofAVCaptureVideoDataOutput(line 35) limits the frame rate. - (line 28) We tell the
AVCaptureVideoDataOutputobject what format we want the frames in. Available options are Y’CbCr and RGB. RGB data is simple to understand, so we ask for that with thekCVPixelFormatType_32BGRAconstant. - (lines 38-44)
AVCaptureVideoDataOutputsends the captured video frames to its delegate. The “queue” argument insetSampleBufferDelgate:queue:is a Grand Central Dispatch queue. I’m not going to go into detail about GCD queues here. Luckily, creating a suitable queue can be done with a single line of code. It can be released immediately after setting the delegate because theAVCaptureVideoDataOutputretains the queue.
Once the capture session starts (with startRunning), every frame of video seen by the camera is sent to the delegate object by callings its captureOutput:didOutputSampleBuffer:fromConnection: method. My implementation of this delegate callback is below.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 | - (void)captureOutput:(AVCaptureOutput *)captureOutput didOutputSampleBuffer:(CMSampleBufferRef)sampleBuffer fromConnection:(AVCaptureConnection *)connection { CVImageBufferRef imageBuffer = CMSampleBufferGetImageBuffer(sampleBuffer); if (CVPixelBufferLockBaseAddress(imageBuffer, 0) == kCVReturnSuccess) { UInt8 *base = (UInt8 *)CVPixelBufferGetBaseAddress(imageBuffer); // calculate average brightness based on a naive calculation size_t bytesPerRow = CVPixelBufferGetBytesPerRow(imageBuffer); size_t width = CVPixelBufferGetWidth(imageBuffer); size_t height = CVPixelBufferGetHeight(imageBuffer); size_t pixelCount = width * height; UInt32 totalBrightness = 0; for (UInt8 *rowStart = base; height; rowStart += bytesPerRow, height --) { size_t columnCount = width; for (UInt8 *p = rowStart; columnCount; p += 4, columnCount --) { UInt32 value = (p[0] + p[1] + p[2]); totalBrightness += value; } } CVPixelBufferUnlockBaseAddress(imageBuffer, 0); Game *theGame = self.game; dispatch_async(dispatch_get_main_queue(), ^{ [theGame updateSparklesWithBrightness:(float)totalBrightness/(255 * 3 * pixelCount)]; }); } } |
Note that I have not tried to optimize this code at all. This code loops through every pixel in the frame and sums the red, green and blue components. When it’s done, it calculates the average by dividing by the number of pixels and components. It also converts the 8-bit pixel value (0-255) it to a floating-point value in the range of 0 to 1.0. It uses GCD to call the main app on the main thread with the brightness value. The method updateSparklesWithBrightness: in my app adds or removes sparkle graphics based on how bright the camera image is.
Next week I’ll present the rest of the app, which uses the Sparrow framework to display and animate the sparkles.
Hi bunnyhero
its good blog, can you please send me source code?
Right now the only source code I’m sharing is what’s already in the blog post. Sorry about that.
While I’m not a fan of the Twilight series, LOVE the idea and the implementation. :) Kudos.
P.S: I haven’t check the rest of the blog yet, but is the app out? Or was this a proof-of-concept/hobby project?
I haven’t finished the app yet. It was more of a proof-of-concept thing but I might add a few features and submit it to the store anyway :)
hi, any idea how we can add normal camera controls on this view? Like Flash, flip etc
Thanks very much for your post. It has helped me to know how to access the front camera :)
Really cool code :) I was looking for a way to determine brightness from the camera and viola!