Face Match Face Off

Face Match Face Off

face recognition

Recently, a client stopped by and asked if Armedia had, or knew of any open source, face matching software that could scan a video and determine if a suspect’s face appeared anywhere in the video.

Our team was intrigued.

In today’s world, how available and how accurate was this type of software? This request launched a line of investigation that not only solved our client’s problem but proved to be insightful and rewarding.

A quick Internet search for open source products that could match a probe face in a video produced limited results, so we expanded our search parameters slightly to include low-cost cloud services.  The products we decided to investigate were:

We assigned a developer to each product to prototype a solution, given only the following minimum requirements from our client:

  • The solution shall identify (i.e., indicate time off-sets) all occurrences of a suspect’s face in a video.
  • The solution shall output a CSV file for each suspect indicating in which videos they were detected, and at what time off-sets.
  • The solution shall output a CSV file for each video containing the name and time off-set at which each suspect was identified in the video.
  • The solution shall accept all probe images as a group.
  • The solution shall accept all videos as a group.
  • The solution shall perform better than real-time (i.e., a 5-minute video should be processed in less than 5 minutes).
  • The solution shall have adjustable operating parameters to affect the sensitivity of detection and computer resource usage.

Early in the project, the pyAnnote developer determined that although the accuracy and output format of the pyAnnote software was impressive, it was intolerably slow (roughly 2-3 times real time).  Therefore, she abandoned pyAnnote as a possible solution and moved on to investigate the Python Face Recognition software, taking a different tack than the developer already investigating the same software.

With these broad requirements, developers were free to architect creative solutions. Therefore, each project took on its own character and individual goals.

  • Face to Face – a desktop, Java-based UI, which leveraged Amazon’s Rekognition API in addition to their S2 storage infrastructure. This project could scale as demand required.
  • Azure Multi-Imaged Face Identifier (AMIFI) – a desktop, C#-based UI, which leveraged Microsoft’s Face Cognitive Service. This project could accept multiple samples of a face to build a more comprehensive model of the suspect.
  • Distributed Face Recognition (DFaR) – a massive, enterprise-level solution utilizing multiple servers, GPUs, load balanced queueing, etc. This project leveraged Python Face Recognition had no GUI and was designed to be integrated into a larger forensic process.
  • Face Recognizer – the other project using Python Face Recognition took the opposite approach and built a solution that ran on a laptop with no special hardware or network connections.

The developers and the project managers met weekly for 3 months to demonstrate progress, discuss problems, and to solicit advice from one another.  These meetings were dubbed “face offs” as they took on a tone of friendly competition. The developers went all out to produce the most accurate and fastest performing solution.

Two interesting findings resulted from these collaborative/competitive meetings.

  1. None of these software products could match face images in videos directly. They were designed to compare and match faces in still images.  As a result, all the developers converted the videos into sequential series of still images (i.e., frames) using FFmpeg and compared the frames to the probe image.
  2. The conversion of videos to frames resulted in thousands (sometimes tens of thousands) of JPG files. Often, the majority of these images contained no faces and incurred processing and resource usage penalties unnecessarily.  This was especially the case with cloud solutions where storage and processing incurred real costs.  To address this problem, all of the developers filtered the images using Dlib.  Only the frames in which Dlib indicated a face was present were allowed to pass through the identification process.

Perhaps the most interesting result was that all of these prototypes and underlying technologies achieved nearly the same level of accuracy (see Precision-Recall chart in Figure 1).  Open source, community-driven technology held its own against the technologies developed by the global R&D powerhouse companies.

Although the results were nearly identical, that doesn’t mean they were sound.  Looking at the Precision-Recall chart in Figure 1, you see that the data points cluster around 0.6 precision and 0.45 recall.  This means that 60% of the time the software identified a suspect it was correct, while it only identified 45% of the suspects it was presented.  Ideally, both precision and recall values should be high for an accurate, well-performing solution.

It is important to note that our test videos and real-life videos were low quality, grainy, surveillance videos.  We would expect better results using higher quality videos.

precision-recall chart

Armedia satisfactorily identified 19 suspects in nearly 6,000 surveillance videos with confidence values and time off-sets so manual confirmation could be made.

The Face Match Face Off is a good example of the type of research and development Armedia brings to client engagements. Our team is well versed and well equipped to research, develop, prototype, and deploy technologies in the areas of computer vision, machine learning, big data, signals processing, and computer forensics.  The Face Match Face Off not only solved a real-world problem for our client, it provided them with information about four leading face recognition technologies, enabling more informed decision making in future technology investments.

Predictive Analytics and The Most Important Thing

Predictive Analytics and The Most Important Thing

It turns out I was wrong…which happens at an alarmingly increasing rate these days—though I chalk that to a thirst to challenge myself…errr, my story!

So, for a while now, I had convinced myself that I knew what the most important thing was about successfully doing predictive analytics: accuracy of data and the model (separating the noise). Veracity, as they say. In working with a few clients lately though, I no longer think that’s the case. Seems the most important thing is actually the first thing: What is the thing you want to know? The Question.

As technologists, we often tend to over-complicate and possibly over-engineer. And it’s easy to make predictive analytics focus on the how; the myriad of ways to integrate large volumes and exotic varieties of data, the many statistical models to evaluate for fit, the integration of the technology components, the visualization techniques used to best surface results, etc. All of that has its place. But ultimately, first, and most importantly, we need to articulate the business problem and the question we want answers for.

What do we want to achieve from the analytics? How will the results help us make a decision?

Easy as that sounds, in practice it is not particularly easy to articulate the business question. It requires a real understanding of the business, its underlying operations, data and analytics and what would really move the meter. There is a need to marry the subject matter expert (say, the line of business owner) with a quant or a data scientist and facilitate the conversation. This is where we figure out the general shape and size of the result and why it would matter; also, what data (internal and external) feeds into it.

Articulating The Question engages the rest of the machinery. Answers are the outcome we care about. The process and the machinery (see below for how we do it) give us repeatability and ways to experiment with both asking questions and getting answers.

Armedia Predictive Analytics Process

Armedia Predictive Analytics process for getting from The Question to answers

The New Data Visualization Part 2 – You Can See the Forest and the Trees


In the first article in this 3-part series The New Data Visualization – This is Not Your Father’s Pie Chart we discussed the three main categories of data and the standard methods used to best portray each of them. If you haven’t read that one yet I’d like to suggest you CLICK HERE and do that first as it will give you a good foundation through which to understand this article.

When beginning to craft a data visualization solution, it’s important to understand the scope of your task. Often times, a very large amount of data will be the starting point. What we want to do first then is use this data to give us clues about where we need to focus our time. We aren’t going query on this initial dump, we’re going to put it into a tool that will show us what we have.

A good way to illustrate this tactic is via a network diagram. Also referred to as node/link diagrams, these are charts that show us how entities – the nodes – are “linked” to other entities in our data set.

Look at this screenshot from an open source tool called Gephi

Data Visualization - Facebook Networking


Federal Cloud Computing Challenges Part 3: Data Management


In a Statement by Dr. David McClure, Associate Administrator, Office of Citizen Services and Innovative Technology, at an April 2011 Senate Subcommittee Hearing, McClure stated one of the biggest challenges federal agencies face in migrating to the cloud is data management. Data management in cloud computing is something that needs to be critically analyzed and strategized before solutions can be implemented. So lets take a look at some of the Data Management challenges that exist in Federal Cloud Computing Solutions:


Cloud Computing Challenges: Data Management


First of all, it is important to understand that the IT needs of global organizations pale in comparison with those of the US federal government. Quite simply, the US Federal Government is enormous – composed of more than 2.1 million full-time federal employees, each of who use at least one IT system and 2,094 federal government data centers composed of thousands of servers.