Last time we left off wondering if there is some formula or set of rules we could use
to determine if a set of 81 pixels showed a finger or something else.
Before we begin searching for a solution, let’s simplify our problem a bit,
by reducing the number of possible intensity values for each pixel from 256 to just two.
We’ll replace every pixel value greater than or equal to 92 with a 1,
and every pixel value less than 92 with a 0.
By reducing the possible values for each pixel, we’ve significantly simplified our problem,
while preserving the structure of our data.
Our 9×9 grids around each pixel now only contain 1s and 0s.
We can now really begin to think about solutions.
Since we have a pretty good idea of what fingers look like,
we should be able to write some code that captures this understanding.
We can do this by creating our own 9×9 grid that shows what we expect a finger to look like.
We can make this into a rule in Python by creating a 9×9 NumPy array
with 1s where the finger is and 0s where it’s not.
If a 9×9 grid matches our pattern, it’s a finger!
If it doesn’t, it’s not.
Now that we have a rule, let’s test it.
We’ll search for matches to our pattern
and we’ll indicate matches by coloring the central pixel of the matching grid on our original images.
We see some matches, but we’ve missed most finger pixels.
Why did our rule miss so many pixels?
One reason is that our rule is way too specific.
It only returns a match if the finger is exactly centered in the 9×9 grid of pixels we’re considering.
We would really like our finger-finding algorithm to return a match for any pixel that belongs to a finger,
even if that pixel is not right in the center of the finger.
We can fix this problem by simply adding more rules.
We could manually enter more arrays of 0s and 1s, as we did with our first rule,
but to save time, we’ll write a Python loop that shifts our finger pattern around for us.
This results in 30 new patterns that should help us identify more finger pixels.
Our finger-finding algorithm now consists of searching for matches to any of our 30 patterns.
If we color our images with our new matches, we see that our new approach does find more finger pixels.
However, we’re still missing quite a few.
And, perhaps a little bit more alarmingly,
our algorithm is labeling some non-finger pixels as fingers.
This brings us to an important point.
There is more than one way in which our algorithm can be wrong.
We could be wrong because we fail to identify finger pixels,
or because we mislabel pixels that actually don’t belong to fingers.
A popular way to represent this idea is with a confusion matrix.
For a given image and algorithm to identify fingers, each pixel could belong to a finger or not,
and may be labeled by our algorithm as a finger or not.
This creates four total possibilities.
Two where our algorithm is correct and two where it’s not.
This distinction between the two types of errors, false positives and false negatives,
is not such a big deal for our application.
But there are other types of problems where this distinction is critical.
If we’re predicting heart attacks, a false positive means you’re prepared for a heart attack and nothing happens,
while a false negative means you give the all-clear and the patient does have a heart attack.
The difference between false positives and false negatives is far thus important to us,
but we’ll still keep track of both of these to see how we’re doing.
Speaking of how we’re doing... How ARE we doing?
We can see that we’re making errors of each type.
But we would really like to know exactly how many.
To count our errors, we need correct labels to compare our labels to.
We could label images by hand, by manually marking each pixel as belonging to a finger or not.
This would work just fine, but it’s of course slow.
Instead, we’ll use Leap Motion hand-tracking software and some custom Python code
to determine correct finger labels.
Now that we have the correct labels for each pixel, we can measure exactly
how well our finger-finding rules are performing.
We can see that our algorithm is correctly identifying some finger pixels, but it’s still missing most of them.
At this point we have an important decision to make.
Should we continue with our current strategy, or reevaluate?
We certainly could improve our current approach.
We’ve accounted for different locations of fingers in our 9×9 grid,
but we haven’t yet considered different rotations or sizes of fingers.
To help us decide which strategy is best,
next time we’ll visit artificial intelligence in the 1980s.