Turn your iPhone into a vampire with AVFoundation and iOS 4

August 15th, 2010 § 7 comments

iOS 4 added a lot to AVFoundation, including classes and APIs that give you much more control over the iPhone camera. One of the things you can now do with the camera is read the video frame data in real time.

In this post, I’ve created a simple demo that simulates a Twilight-style vampire. In the Twilight series, vampires aren’t hurt by daylight; instead, they sparkle. Yes, sparkle.

Here are a couple of screenshots from the app:

And here’s a low-quality video of the vampire simulator in action.

The app detects the amount of light shining on the phone by doing very simple image analysis of the incoming video frames from the camera. The brighter the image seen by the camera, the more sparkles it draws on the vampire.

So how does this all work?

AVCaptureSession

The AVCaptureSession object is the centre of the new video and audio input/output universe. An AVCaptureSession can have multiple inputs and outputs. Inputs include video sources (the cameras) and audio sources (microphone). Outputs can be things like a file, or (in this example) an object that captures every frame of video data as it comes from the camera.

The following method demonstrates how to create an AVCaptureSession and add a video input from the front-facing camera (we’re using the front camera because it is more likely to be facing upward, to measure the ambient light) and an output to an object that will let us inspect every video frame that comes from the input.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
- (void)start
{
    NSError *error = nil;
 
    AVCaptureDevice *captureDevice = [self frontFacingCameraIfAvailable];
    AVCaptureDeviceInput *videoInput = [AVCaptureDeviceInput deviceInputWithDevice:captureDevice error:&error];
    if ( ! videoInput)
    {
        NSLog(@"Could not get video input: %@", error);
        return;
    }
 
    //  the capture session is where all of the inputs and outputs tie together.
 
    captureSession = [[AVCaptureSession alloc] init];
 
    //  sessionPreset governs the quality of the capture. we don't need high-resolution images,
    //  so we'll set the session preset to low quality.
 
    captureSession.sessionPreset = AVCaptureSessionPresetLow;
 
    [captureSession addInput:videoInput];
 
    //  create the thing which captures the output
    AVCaptureVideoDataOutput *videoDataOutput = [[AVCaptureVideoDataOutput alloc] init];
 
    //  pixel buffer format
    NSDictionary *settings = [[NSDictionary alloc] initWithObjectsAndKeys:
                              [NSNumber numberWithUnsignedInt:kCVPixelFormatType_32BGRA],
                              kCVPixelBufferPixelFormatTypeKey, nil];
    videoDataOutput.videoSettings = settings;
    [settings release];
 
    //  we don't need a high frame rate. this limits the capture to 5 frames per second.
    videoDataOutput.minFrameDuration = CMTimeMake(1, 5);
 
    //  we need a serial queue for the video capture delegate callback
    dispatch_queue_t queue = dispatch_queue_create("com.bunnyherolabs.vampire", NULL);
 
    [videoDataOutput setSampleBufferDelegate:self queue:queue];
    [captureSession addOutput:videoDataOutput];
    [videoDataOutput release];
 
    dispatch_release(queue);
 
    [captureSession startRunning];
}
 
- (AVCaptureDevice *)frontFacingCameraIfAvailable
{
    //  look at all the video devices and get the first one that's on the front
    NSArray *videoDevices = [AVCaptureDevice devicesWithMediaType:AVMediaTypeVideo];
    AVCaptureDevice *captureDevice = nil;
    for (AVCaptureDevice *device in videoDevices)
    {
        if (device.position == AVCaptureDevicePositionFront)
        {
            captureDevice = device;
            break;
        }
    }
 
    //  couldn't find one on the front, so just get the default video device.
    if ( ! captureDevice)
    {
        captureDevice = [AVCaptureDevice defaultDeviceWithMediaType:AVMediaTypeVideo];
    }
 
    return captureDevice;
}

The code is (I hope) pretty straightforward. A few things to note:

  • (line 49) The frontFacingCameraIfAvailable method gets the AVCaptureDevice that represents the front-facing camera, if it exists. AVCaptureDevice makes it easy to enumerate all of the capture sources on the device for any given media type (e.g. video, audio). We simply query each device in the list to see if it’s on the front or the back of the phone.
  • Since all we’re doing is measuring the average brightness of the picture, there’s no need to waste resources by grabbing full HD video at 30 frames per second. The sessionPreset property of AVCaptureSession (line 20) governs the resolution of the capture, and the minFrameDuration property of AVCaptureVideoDataOutput (line 35) limits the frame rate.
  • (line 28) We tell the AVCaptureVideoDataOutput object what format we want the frames in. Available options are Y’CbCr and RGB. RGB data is simple to understand, so we ask for that with the kCVPixelFormatType_32BGRA constant.
  • (lines 38-44) AVCaptureVideoDataOutput sends the captured video frames to its delegate. The “queue” argument in setSampleBufferDelgate:queue: is a Grand Central Dispatch queue. I’m not going to go into detail about GCD queues here. Luckily, creating a suitable queue can be done with a single line of code. It can be released immediately after setting the delegate because the AVCaptureVideoDataOutput retains the queue.

Once the capture session starts (with startRunning), every frame of video seen by the camera is sent to the delegate object by callings its captureOutput:didOutputSampleBuffer:fromConnection: method. My implementation of this delegate callback is below.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
- (void)captureOutput:(AVCaptureOutput *)captureOutput
didOutputSampleBuffer:(CMSampleBufferRef)sampleBuffer
       fromConnection:(AVCaptureConnection *)connection
{
    CVImageBufferRef imageBuffer = CMSampleBufferGetImageBuffer(sampleBuffer);
    if (CVPixelBufferLockBaseAddress(imageBuffer, 0) == kCVReturnSuccess)
    {
        UInt8 *base = (UInt8 *)CVPixelBufferGetBaseAddress(imageBuffer);
 
        //  calculate average brightness based on a naive calculation
 
        size_t bytesPerRow      = CVPixelBufferGetBytesPerRow(imageBuffer); 
        size_t width            = CVPixelBufferGetWidth(imageBuffer); 
        size_t height           = CVPixelBufferGetHeight(imageBuffer); 
        size_t pixelCount       = width * height;
        UInt32 totalBrightness  = 0;
 
        for (UInt8 *rowStart = base; height; rowStart += bytesPerRow, height --)
        {
            size_t columnCount = width;
            for (UInt8 *p = rowStart; columnCount; p += 4, columnCount --)
            {
                UInt32 value = (p[0] + p[1] + p[2]);
                totalBrightness += value;
            }
        }
        CVPixelBufferUnlockBaseAddress(imageBuffer, 0);
        Game *theGame = self.game;
        dispatch_async(dispatch_get_main_queue(), ^{
            [theGame updateSparklesWithBrightness:(float)totalBrightness/(255 * 3 * pixelCount)];
        });
    }
}

Note that I have not tried to optimize this code at all. This code loops through every pixel in the frame and sums the red, green and blue components. When it’s done, it calculates the average by dividing by the number of pixels and components. It also converts the 8-bit pixel value (0-255) it to a floating-point value in the range of 0 to 1.0. It uses GCD to call the main app on the main thread with the brightness value. The method updateSparklesWithBrightness: in my app adds or removes sparkle graphics based on how bright the camera image is.

Next week I’ll present the rest of the app, which uses the Sparrow framework to display and animate the sparkles.

Tagged ,

§ 7 Responses to Turn your iPhone into a vampire with AVFoundation and iOS 4"

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong> <pre lang="" line="" escaped="" highlight="">

What's this?

You are currently reading Turn your iPhone into a vampire with AVFoundation and iOS 4 at bunnyhero dev.

meta

viagra cialis trial pack
cialis generic 20mg
cialis 10 mg where to buy
viagra sales
generic cialis prices
generic propecia
viagra ireland
generic viagra 50 mg
levitra india
calis
inderal overnight
buy strong viagra
finpecia
cialis viagra cocktail
does generic viagra work
viagra cheap prices
us cialis online pharmacy
viagra pro online
buy viagra without prescription
sun pharmaceuticals india
uninsured cost of cialis
viagra gold
cheap sildenafil
generic cialas
brand cialis 20
cialis online nz
buy viagra usa
cialis soft tabs canada
tadalafil 20mg
viagra levitra cialis offers
.buy viagra
viagra price in india
viagra for sale australia
discount viagra
buy nolvadex
cialis 5mg
viagra levitra which is best
genaric viagra
ordering viagra onl
cialis no perscription
is levitra sold at walmart
order viagra from canada
viagra patent date
sildenafil citrate
100mg viagra
where to buy viagra in calgary
viagra super active
generic cialis
cialis 36 canada
where to buy genuine viagra online
genaric propecia
viagra video
cialis onlilne
cialis no prescrition
need prescription for viagra in cyprus?
cheep viagra
buy proscar
viagra for sale online
sildenafil online canada
european cialis
buy wholesale viagra
cialisonline
cheapest viagra
no risk pharmacy
levitra 10mg buy
levitra order
propecia
viagra canada free sample
ca i buy viagra online
cialis 20 mg
cialis men
cheap cialis generic online
what works as well as viagra
aftermarket viagra
cialis 40 mg
viagra suppliers canada
sildenafil citrate overnight delivery
redtube viagra
generic viagra 50mg
online viagra
where to get viagra online
Sildenafil citrate 50mg
Viagra 50mg pills
very cheap propecia
viagra over the counter walgreens
viagra prices walmart
viagra no prescription online
cialis 20mg price
viagra canada no prescription
viagra canada
viagra on line
cialis for daily use disount
Generic viagra
viagra no prescription
best viagra deals from India
canada customs and cialis
buy levitra online
buy viagra on line
cialis online
generic viagra
where to buy viagra in toronto
viagra professional
i need cialis
buy cialis
cialis online canadian pharmacy
viagra generic
Sildenafil Citrate
cialis fastest shipping
tadalafil dosage
cheap generic viagra pills
viagra prices
supreme suppliers cialis
viagra 4 pack
ed. trusted medstore in cialis
where can i buy finasteride
levitra 100mg uk
propecia generic price
status 365 pills
buy viagra no prescription
cialis doesnt work
cialis overnight
lilly cialis 20mg
buy cialis black
viagra free trial offer
levitrawithnoprescription
purchase propecia
cialis paypal
Dapoxetine
Cialis 50mg price
viagra profesionals
black cialis
generic cailis
voucher cialis index
low dose cialis
best viagra online toronto
generic viagra
levitra buy
cialis without prescriptions
viagra quebec
viagra without a prescription
viagra costs walmart
nolvadex buy
tadafil cialis
cheap levitra online india
viagra express
overnightpharmacy4u
nolvadex
sildenafil citrate
Sildenafil Citrate
cialis for sale canada
finasteride sale
cialis black
canada generic viagra
how to buy viagra from canada
1 mg propecia
is super active cialis a scam
cheapest cialis 20mg
cialis vs viagra
generic levitra overnight delivery
levitra generic
buy real viagra with a echeck
100mg Viagra
cialis 200 mg price
levitra canada
viagra 100 tablets
HTMLGIANT LEVITRA
GENERIC CIALIS WITH DAPOXETINE 80MG
generic viagra soft tabs
Discount Viagra
ordering cialias
where to buy Propecia
30 day free cialis
ciali sell from usa
viagra online without a prescription
buy dapoxetine online
levitra professional 100mg
overnight pharmacy4u
finasteride 1mg
Cialis
cheap vigra
viagra for women for sale
Tadalafil canada
Sildenafil Citrate 50mg
dapoxetine
generic viagra canada
Viagra online
generic viagra online
noriskpharmacy
buy tadalafil
cialis online canada
cialis 20mg
buy cialis online canada
cialis soft tabs
cheaP viagra
generic cialis 20mg
buy cialis using paypal
reasons to use viagra
viagra
pfizer viagra price
Viagra Oklahoma city
buy cialis from india
where to buy viagra
cialis for sale
generic viagra price
30 day free trial of cialis
generic viagra paypal
compralviagra
cialis price
cealis generic
cheap viagra online canada
cialis 10mg
36 hour cialis
viagra no prescription canada pfizer
viagra online 365
viagra cialis combo
cialis no prescription
purchase generic viagra
sildenafil tablets
buy viagra
walmartcanadapharmacy
propecia generic finasteride
generic viagia
ed pro trial pack
generis cialis
buy cialis 20mg
Propecia 1mg
cialis professional
indo viagral
original viagra for sale
buy propecia
Finasteride purchase
cialis daily
shipping viagra to australia
10mg cialis
viagra with dapoxetine overnight
where can I buy cialis 20mg
brand levitra
viagra sale
viagra 100mg
Cialis 5mg Tablets
cialis 20mg prices
cialis pills canada
cialis 10mg vs 20mg
viagra for sale in ireland
cialis-buy-info/
buy viagra in canada fast
sildenafil citrate generic
cheap cialis overnight
generic cialis cipla
tadalafil soft gelatin capsules
viagra patent expiration date
generic viargra
buy viagra from germany
cialis black reviews
order viagra
VIAGRA FOR DAILY USE
Viagra uk
viagra, no prescription
Order viagra
supreme suppliers viagra
propecia
viagra jelly sachet
buying viagra online in montreal
viagra and dapoxetine
viagra no prescription
buy viagra online sydney
viragra
buy viagra Australia
cialis without prescription
cheap sildenafil citrate
overnight generic viagra
VIAGRA UK
buy generic cialis
generic viagra blogs
viagra online netherlands
original cialis
cialis paypal payment
dsicount viagra
viagra in united states
cialis 200 mg no prescription
20mg cialis
kamagra for sale
viAGRA ONLINE
cialis with dapoxetine
canadian levitra vs usa levitra
causes of irectal dysfuction
drugstore1st
viagra coupons
viagra usa
CHEAP VIAGRa
buy finasteride
generic viagra from india
tadalafil daily use
ebay + cialis
pfizer viagra without prescription
Finasteride 1mg
viagra pills for sale
PROPECIA
cialis overnight shipping
viagra mail order
amazon viagra
cialis promise program
discount cialis 20mg
levitra
viagra next day delivery usa
viagra softtabs
centerpill
viagra coupons and discounts
100 ml viagra for sale
order cialis 20mg
Cheap viagra
ganeric viagra
buy Dapoxetine
bying viagra uk
ggeneric viagra for sale
canadian pharmacies selling viagra
Viagra greate britain
viagra coupon
how can i buy viagra online in canada
price of tadalafil 20 mg
cialis 200mg
buy cialis online in canada
viagra in uk
viagra canada pharmacy
paypal online pharmacy
cialis comparison
viagra nz
pills house
viagra no prescription overnight
Generic Cialis
where can I buy propecia?
10 mg cialis for sale
drugstore 1st
viagra price comparison
viagra no prescrition
canadian generic viagra
buy super viagra
cialis viagra levitra canada
viagra uk
montreal online pharmacy
buy kamagra
viagra for boys
Cheap Viagra
levitra online canadian
finasteride 1 mg
cialis no prescription canada
online pharmacy for cialis
Generic Viagra
canadian cialis online
brand name viagra canada
Buy Viagra online
cialis generic versus brand name
cost of viagra
levitra coupon
viagra no prescription cialis
cipla canada
generic viagra
viagra without a script
tadalafil canada
buy cialis generic
buy cheap cialis
drugstore 365 org order levitra online
viagra india
viagRA online
Finasteride
canada pharmacy generic cialis
best viagra source
viagra
newestpills
buy levitra
cheapest cialis tadalafil 20 mg
viagrasuperforte
tadalafil tablets
sildenafil 100mg
viagra loest prise
cilias 20mg
levitra 20mg
viagra canada online
cheap viagra
Cialis online
order levitra
black tab cialis
cialis usa online
NEXIUM FOR SALE
viagra melbourne
buy oral viagra online
buy generic propecia
viagra alcohol effects
planet pharma warehouse pvt ltd
Cheap viagra
Cialis generic alternative
cialis
viagra online without prescription
cialis super active review
buy viagra online ireland
200 mg cialis deals online
viagra by mail
buy cialis 20 mg
viagra overnight no prescription
cialis generic 10 mg