The PS3Eye Multicam Driver is here!!!

•September 4, 2009 • 32 Comments

Finally, here is the long awaited PS3Eye multicam driver

After a lot of testing and optimization of the internal driver code, I finally am ready to release the first version of the PS3Eye multicam driver.
For your reference, below is the diagram of the internal components of the driver. For a single camera configuration the ‘Image Stitching’ component is bypassed.

The driver exports simple API that allows for full control of PS3Eye cameras.
I chose a C style export functions this time to allow for easier integration with different programming languages such as C#.
Here is the ‘PS3EyeMulticam.h’ file that you would normally include in your code:

////////////////////////////////////////////////////////////////////////////////////////////
//
// This library allows you to use multiple PS3Eye cameras in your own applications.
//
// For updates, more information and downloads visit: 
// http://www.alexpopovich.com or
// https://alexpopovich.wordpress.com/
//
// Copyright 2008,2009 (c) AlexP.  All rights reserved.
//
////////////////////////////////////////////////////////////////////////////////////////////
#pragma once
#include <windows .h>

#ifdef PS3EYEMULTICAM_EXPORTS
#define PS3EYEMULTICAM_API extern "C" __declspec(dllexport)
#else
#define PS3EYEMULTICAM_API extern "C" __declspec(dllimport)
#endif

////////////////////////////////////////////////////////////////////////////////////////////
// PS3EyeMulticam API
////////////////////////////////////////////////////////////////////////////////////////////

typedef enum{ QVGA, VGA }Resolution;

// Camera information
PS3EYEMULTICAM_API int PS3EyeMulticamGetCameraCount();

// Library initialization
PS3EYEMULTICAM_API bool PS3EyeMulticamOpen(int camCnt, Resolution res, int frameRate);
PS3EYEMULTICAM_API void PS3EyeMulticamClose();

// Capture control
PS3EYEMULTICAM_API bool PS3EyeMulticamStart();
PS3EYEMULTICAM_API void PS3EyeMulticamStop();

// Settings control
PS3EYEMULTICAM_API bool PS3EyeMulticamLoadSettings(char* fileName="settings.xml");
PS3EYEMULTICAM_API bool PS3EyeMulticamSaveSettings(char* fileName="settings.xml");
PS3EYEMULTICAM_API void PS3EyeMulticamShowSettings();

// Processed frame image data retrieval
PS3EYEMULTICAM_API bool PS3EyeMulticamGetFrameDimensions(int &width, int &height);
PS3EYEMULTICAM_API bool PS3EyeMulticamGetFrame(PBYTE pData, int waitTimeout=2000);

////////////////////////////////////////////////////////////////////////////////////////////

Included in the driver archive is a full source code of a simple ‘PS3EyeMulticamTest’ application that allows you to run and test the multicamera driver. This code could also serve as a good starting point for writing your own OpenCV image processing apps.
The PS3EyeMulticamTest app uses all of the API functions defined in ‘PS3EyeMulticam.h’ file.
Here is the ‘PS3EyeMulticamTest.cpp’ file:

//
// PS3EyeMulticamTest.cpp
//
// This is a quick test application for the 
// PS3EyeMulticam driver
//
// To build it, in VS2008 create en empty window app project
// and add all the files in this folder to it.
//
#include "stdafx.h"

#define WINDOW_NAME		"Capture Window"
bool bRunning = false;
int w, h;
 
// Capture thread
DWORD WINAPI Capture(LPVOID)
{
	IplImage *grayImage = cvCreateImage(cvSize(w, h), IPL_DEPTH_8U, 1);
	PS3EyeMulticamStart();
	while(bRunning)
	{
		PBYTE pBuffer = NULL;
		cvGetImageRawData(grayImage, &pBuffer);
		PS3EyeMulticamGetFrame(pBuffer);
		cvShowImage(WINDOW_NAME, grayImage);
	}
	PS3EyeMulticamStop();
	cvReleaseImage(&grayImage);
	return 0;
}

void Program()
{
	// Query driver for number of connected PS3Eye cameras
	int camCnt = PS3EyeMulticamGetCameraCount();
	printf("Found %d PS3Eye cameras\n", camCnt);
	if(camCnt == 0)
	{
		MessageBox(NULL,"No PS3Eye camera(s) found","PS3EyeMulticamTest", MB_ICONEXCLAMATION);
		return;
	}
	// Open cameras
	if(!PS3EyeMulticamOpen(camCnt, VGA, 30))
	{
		MessageBox(NULL,"Could not open PS3Eye camera(s)","PS3EyeMulticamTest", MB_ICONEXCLAMATION);
		return;
	}
	// Load the default camera settings 'settings.xml' file
	// If the file is not found it will be created
	PS3EyeMulticamLoadSettings("settings.xml");

	// Get output image dimensions
	PS3EyeMulticamGetFrameDimensions(w, h);
	printf("Image dimensions (%d, %d)\n", w, h);

	// Create OpenCV main window
	cvNamedWindow(WINDOW_NAME, CV_WINDOW_AUTOSIZE);
	cvMoveWindow(WINDOW_NAME, 0, 0);
	cvResizeWindow(WINDOW_NAME, w, h);

	// Start PS3Eye image capture thread
	HANDLE hThread = CreateThread(NULL, 0, Capture, NULL, 0, 0);
	if(hThread == NULL)
	{
		MessageBox(NULL,"Could not create capture thread","PS3EyeMulticamTest", MB_ICONEXCLAMATION);
		PS3EyeMulticamClose();
		return;
	}
	bRunning = true;
	// The '<esc>' key will exit the program
	// The 's' key will open camera settings dialog
	while(bRunning)
	{
		int key = cvWaitKey(0);
		switch(key)
		{
			case 's':
			case 'S':
				// Display settings
				PS3EyeMulticamShowSettings();
			break;
			case 0x1b:
				printf("Exiting...\n");
				// Stop the capture thread
				bRunning = false;
				// Wait for thread to exit
				WaitForSingleObject(hThread, 3000);
				printf("Thread exited\n");
			break;		
		}
	}
	// Save camera settings
	printf("Saving settings\n");
	PS3EyeMulticamSaveSettings();
	printf("Closing OpenCV window\n");
	cvDestroyWindow(WINDOW_NAME);
	PS3EyeMulticamClose();
}

// This is the program entry if you link with /SUBSYSTEM:CONSOLE option
int main(int argc, char *argv[])
{
	Program();
	return 0;
}

// This is the program entry if you link with /SUBSYSTEM:WINDOWS option
int WINAPI WinMain(HINSTANCE hInstance, HINSTANCE hPrevInstance, LPSTR lpCmdLine, int nShowCmd)
{
	Program();
	return 0;
}

Last but not least, here you will find the PS3EyeMulticam.rar file.
Enjoy!

~Alex

Open Source PS3Eye DirectShow driver component

•July 20, 2009 • 18 Comments

After many requests from PS3Eye users about the PS3Eye DirectShow component I decided to release it as an open source component. Let me explain. When I wrote the PS3Eye DirectShow component I didn’t even dream that it is going to be used in so many applications. Naturally, it is very hard for me to test and debug this component under various circumstances and in all use cases. Many of you are skilled programmers and so frequently I got requests for the source code. While initially reluctant to release it due to the nature of the code (driver component code versus a normal application code), I truly believe in open source and that by releasing the code as an open source to the community, in time, it will only get better.

I do believe that openly sharing knowledge (source code) will eventually make this world a better place (I did it myself many times). While this is true, the work put in throughout the design and development process has a definite value. In order to satisfy both sides of the issue, I came up with the win-win model that I think is the best suited in this particular case – the communal donation. While giving a freedom to each one of you to donate any amount you think its appropriate, you will ultimately contribute to the code eventually being open source. In this way the source code will become a property of the community as a whole (which by the definition is an open source code).

Based on the research, design, development efforts and time spent on this project I estimate that its total value is $15,000.

Below you’ll find a link to ChipIn widget that will allow you to donate and see the progress in real time. When the bar hits 100% I will release the full source code of the PS3Eye DirectShow component.

NOTE: Since the original chipin event has ended so I created a new one.
http://widget.chipin.com/widget/id/a717a49af8cdfabd

The old one:
http://widget.chipin.com/widget/id/a717a49af8cdfabd

Note: Due to WordPress limitation I was not able to embed ChipIn widget into the post.

AlexP’s Forum is up!

•July 20, 2009 • 5 Comments

I created a forum where you can find the latest news and join the discussion about various topics featured on my blog. Join the forum here.

Blog’s new home…

•March 30, 2009 • 2 Comments

I finally got around registering my own domain. This blog is therefore moved there.

I will not post any updates of my posts here  so please update your links. In addition to that on my new site I added a forum that you can join. My new site is at: http://www.alexpopovich.com/blog/

See you all there!!!

MacBook Rev. F Audio Skipping in Vista Analisys and Solution – Part 2

•December 22, 2008 • 16 Comments

In my previous post I talked about reducing the DPC latency in order to prevent frequent audible clicks and audio skipping. The solution was to fix the CPU affinity in the KbdMgr.exe file. This resulted in great reduction of DPC latency of the system. 

The Problem

However, I noticed that every 50-60 seconds the DPC latency spiked to about 16ms. Since most professional audio software requires absolute low latency at all times, this DPC spikes will result in audible skips in those high-performance audio programs. I decided to look into this more closely.

latencyrevisited1

As it turns out (by the process of elimination) this spikes are caused by the Broadcom wireless driver. As seen in devince manager:

broadcom-devmgr1

Here are the driver details:

broadcom-driver

The reason why this spikes happen every 60 seconds or so is that, when enabled, Vista is actively scanning and detecting available wireless network and managing wireless network profile. The service responsible for this is called ‘WLAN AutoConfig Service’ which is similar to ‘Wireless Zero Configuration service’ in XP. 

Since this service is required in order for us to establish a wireless conection, we cannot just disable it. 

The Solution

The quickest thing is to create two cmd files and allow user to manually turn on and off the WLAN autoconfig feature. Here is what I have done. I created two files and placed them on my desktop. After establishing a wireless connection to my access point, I double-click on the ‘WlanAutocfgOff.cmd’. This turns off the WLAN autoconfig feature, but keeps me connected to my access point. In order to later be able to connect to it again (eg. after Windows restart), I just double-click on ‘WlanAutocfgOn.cmd’ and wait until I get connected. Then after establishing a wireless connection to my access point, I double-click on the ‘WlanAutocfgOff.cmd’to stop DPC spikes from happening again. 

File ‘WlanAutocfgOn.cmd’:

netsh wlan set autoconfig enabled=yes interface="Wireless Network Connection"

 
File ‘WlanAutocfgOff.cmd’:

netsh wlan set autoconfig enabled=no interface="Wireless Network Connection"

As you can see each file consists of single command.

After running ‘WlanAutocfgOff.cmd’, the DPC spikes will be gone (max latency ~1ms), and you’ll be able to use your pro audio app of choice without any skipping and stuttering.

latencyrevisited2

MacBook Rev. F Audio Skipping in Vista Analisys and Solution – Part 1

•October 30, 2008 • 40 Comments

Last week I got the latest Apple’s 13.3″ MacBook Rev. F. (aluminum unibody model)

This is an amazing looking laptop, and is one of the fastest performing laptops out there. I decided to have dual boot OS and installed Vista and Apple’s BootCamp.
Here is MacBook’s performance index in Vista:

As you can see, the overall rating is pretty high and the graphics card has pretty high rating (5.4) as well. One nice thing is that at the back Apple gave us easy access to the hard drive on the back. This makes a very easy upgrade. The thing that I found is that this is the only laptop that can fit new 500GB SATA laptop drives which are thicker (12.5mm) than standard 250GB laptop drive (9.5mm). So I managed to easily upgrade mine to 500GB 5400rpm Hitachi laptop drive.

The Problem

After playing with MacBook for a while on Vista, I noticed that when playing music, frequently there are audible clicks and audio skipping. This was even more audible when I was listening streamed radio stations over shoutcast network. On top of this, I think that there is significant reduction of the MacBook battery run time as well (comparing to OSX). I decided to investigate this more closely.

In short, I found, the reason why this happens is because of the exceptionally high DPC latency.

Why Audio Drop-outs Occur?

(Taken from http://www.thesycon.de/deu/latency_check.shtml)

Processing of streaming data in real-time is a very challenging task for Windows based applications and device drivers. This is because by design Windows is not a real-time operating system. There is no guarantee that certain (periodic) actions can be executed in a timely manner.

Audio or video data streams transferred from or to an external device are typically handled by a kernel-mode device driver. Data processing in such device drivers is interrupt-driven. Typically, the external hardware periodically issues interrupts to request the driver to transfer the next block of data. In Windows NT based systems (Windows 2000 and better) there is a specific interrupt handling mechanism. A device driver cannot process data immediately in its interrupt routine. It has to schedule a Deferred Procedure Call (DPC) which basically is a callback routine that will be called by the operating system as soon as possible. Any data transfer performed by the device driver takes place in the context of this callback routine, named DPC for short.

The operating system maintains DPCs scheduled by device drivers in a queue. There is one DPC queue per CPU available in the system. At certain points the kernel checks the DPC queue and if no interrupt is to be processed and no DPC is currently running the first DPC will be un-queued and executed. DPC queue processing happens before the dispatcher selects a thread and assigns the CPU to it. So, a Deferred Procedure Call has a higher priority than any thread in the system.

Note that the Deferred Procedure Call concept exists in kernel mode only. Any user-mode code (Windows applications) runs in the context of a thread. Threads are managed and scheduled for execution by the dispatcher.

While there is a pre-emptive multitasking for threads, DPCs are executed sequentially according to the first in, first out nature of a DPC queue. Thus, a sort of cooperative multitasking scheme exists for Deferred Procedure Calls. If any DPC runs for an excessive amount of time then other DPCs will be delayed by that amount of time. Consequently, the latency of a particular DPC is defined as the sum of the execution time of all DPCs queued in front of that DPC. In order to achieve reasonable DPC latencies, in the Windows Device Driver Kit (DDK) documentation Microsoft recommends to return from a DPC routine as quick as possible. Any lengthy operation and specifically loops that wait for a hardware state change (polling) are strongly discouraged.

Unfortunately, many existing device drivers do not conform to this advice. Such drivers spend an excessive amount of time in their DPC routines, causing an exceptional large latency for any other driver’s DPCs. For a device driver that handles data streams in real-time it is crucial that a DPC scheduled from its interrupt routine is executed before the hardware issues the next interrupt. If the DPC is delayed and runs after the next interrupt occurred, typically a hardware buffer overrun occurs and the flow of data is interrupted. A drop-out occurs.

The Solution

After running the DPC Latency Checker program, I found that the excessive DPC latency was indeed the cause of all the audio skipping. 

Here is the DPC Latency with my MacBook idling:

At moments the latency was as high as 30.5ms!

After running the Process Explorer program I was able to track down the program that was causing the problem. It was the Apple’s KbdMgr.exe. This program is responsible for managing the keyboard, handling the special keys and displaying the OSD (on-screen-display) when you, for example, change screen brightness. 

For a program that is mostly idle the KbdMgr.exe consumes quite a lot of CPU cycles.

You can see that KbdMgr has eight threads, but only one of them constantly consumes high number of CPU cycles. After inspecting the API calls that thread (TID 1352) makes, I found that the main cause for the high CPU cycles is the bad KbdMgr’s multiprocessor implementation (deadlock situation betwen CPU cores in User/Kernel mode). In another words, by simply limiting KbdMgr to a CPU core #1 (CPU affinity) and optionally setting KdbMgr’s priority to Idle, the DPC latency of my MacBook improved dramatically:

As you can see the latency went from whopping high 30.5ms down to 1ms. The result was smooth audio playback with no skipping, much more responsive machine and improved laptop’s battery life.

Here is the fix: KbdMgr.rar
The new fixed version will automatically run on CPU #1 with low priority and will reduce the DPC latency of MacBook.

Installation:
1. Go to Task Manager and kill KbdMgr.exe process
2. Download and extract the contents of the KbdMgr.rar file to a temporary directory
3. Backup the original KbdMgr.exe file located in C:\Program Files\Boot Camp directory
4. Copy new KbdMgr.exe file to C:\Program Files\Boot Camp directory
5. Restart your machine

PS3Eye Camera SDK for use in your own applications!!!

•October 20, 2008 • 69 Comments

Many of you asked me how can you use this camera in your applications. 
Besides the existing implementation of PS3Eye DirectShow filter, I developed the PS3EyeLib as well. 
This SDK library lets you get full access to the PS3Eye camera and allows you to configure the camera and grab video frames.

Lets look at the PS3EyeLib API: 

typedef enum
{
    R320x240
,
    
R640x480,
}PS3EYE_RESOLUTION;        

typedef struct tFormat
{
    PS3EYE_RESOLUTION res
;
    
int width;
    
int height;
    
DWORD rate;
    
LONGLONG period;
    
char formatTxt[32];
}PS3EYE_FORMAT;

class PS3EYELIB_EXPORT IPS3EyeLib
{
public:
    static 
IPS3EyeLib *Create();

    virtual ~IPS3EyeLib();

    virtual void SetFormat(int formatIdx) = 0;
    static 
PS3EYE_FORMAT *GetFormats();
    static 
int GetNumFormats();
    
virtual void StartCapture() = 0;
    
bool GetFrame(PBYTE bufint bitsPerPixelbool flip);
    
virtual void StopCapture() = 0;

    virtual void DumpEEPROM(LPCTSTR filename) = 0;
    
virtual void DumpRAM(LPCTSTR filename) = 0;
    
virtual void LED(bool on) = 0;

    virtual bool IsCapturing() = 0;
    
virtual UINT GetWidth() = 0;
    
virtual UINT GetHeight() = 0;
    
virtual PS3EYE_RESOLUTION GetRes() = 0;
    
virtual DWORD GetRate() = 0;
    static 
int GetFormatIndex(int widthint heightLONGLONG period);
    static 
int GetFormatIndex(int widthint heightint rate);
    
virtual LPCTSTR GetCameraVersion() = 0;
};

 
Here is the example of PS3EyeLib usage in you application: 

// Create PS3EyeLib object
IPS3EyeLib *pCam=IPS3EyeLib::Create();        

// Query supported video formats
for(int i=0i<IPS3EyeLib::GetNumFormats(); i++)
{
    int width
heightrate;
    
char *description;
    
width=IPS3EyeLib::GetFormats()[i].width;
    
height=IPS3EyeLib::GetFormats()[i].height;
    
rate=IPS3EyeLib::GetFormats()[i].rate;
    
description=IPS3EyeLib::GetFormats()[i].formatTxt;
    
// Display available formats ...
}

// Decide on the format to use (Example: 320x240 * 30fps)
// Select this format
pCam->SetFormat(IPS3EyeLib::GetFormatIndex(320,240,30));
// Allocate image buffer (we are going to capture 24bit RGB images)
// The supported color depths are 16, 24 and 32
PBYTE pBuffer=new BYTE[(320*240*24)/8];
// Start capturing
pCam->StartCapture();
// Process frames until 'done'
bool done=false;
while(!
done)
{
    
// This function will block until a new frame is available
    // It will then fill the buffer with frame image data
    
if(pCam->GetFrame(pBuffer24false))
    {
        
// Process/display video frame here
        // ...
        // If your program is done set 'done=true;'
    
}
}
// Free the image buffer
delete [] pBuffer;
// Stop capturing
pCam->StopCapture();
// Free the 
delete pCam;

 

The latest setup file (PS3EyeSetup.2.0b81021), besides the PS3Eye driver, DirectShow filter and test app, includes the following files in the SDK directory:

  • IPS3EyeLib.h – Camera API include file 
  • PS3EyeLib.lib – Camera API library file 
  • PS3EyeLib.dll – Camera API dynamic library file

List of fixes/additions in the current release:

– Created DirectShow camera property page (selectable resolution and frame rate) 
– Support for RGB-16/24/32 color output format 
– Implemented both 32-bit and 64-bit versions of the PS3Eye driver 
– Fixed the PS3EyeCamera.inf file so that drivers (32-bit and 64-bit) install correctly
– Improved capture performance and responsiveness on Vista OS

 
If you find the software useful or if you feel like supporting this project, please feel free to click the donation link below.

 

And of course, the files: PS3EyeSetup (v2.0b81021) and PS3Eye TouchLib (rev400) build.

Enjoy!

 
Follow

Get every new post delivered to your Inbox.