We have seen interpolation in the news concerning a recent court case. Here we cover what interpolation does to an image, not only because of the recent news but also because face recognition uses interpolation to better recognize a face – something we have covered in the past.
Interpolation means to take pixels in an image and calculate what their color might be when the image is resized. Many image processing libraries use bicubic interpolation to increase details on the face, which is the same bicubic interpolation talked about in the case. In the case of the Rittenhouse trial, we don’t know for sure if evidence provided to the jury used bicubic interpolation though.
For you to understand bicubic interpolation we must step back to linear interpolation. We perform linear interpolation all the time on graphs by extrapolating data between two datapoints. We can run into problems though if the data points are too far apart as seen below.
We can compensate for this error in the graph using cubic interpolation by inspecting not just the nearest data points but the next nearest data points. We take four data points total to determine the number we wish to find.
Pictures aren’t one-dimensional but two-dimensional, so we must account for both directions. This leads us to bicubic interpolation. To increase the resolution of a picture, we take the 16 pixels (4 pixels x 4 pixels) closest to the area we wish to expand and determine what color to put in the middle.
Problems can arise if we try to expand the picture too much. Whenever performing interpolation, we could add artifacts to the picture, so we limit on how far we expand a picture. This makes using bicubic interpolation on a picture to determine what happened in that picture unreliable in a trial setting. It can also cause problems in face recognition, but we can get around this by inspecting multiple areas on a face. If one area of the face become corrupted because of interpolation we can still match other areas of the face to determine if a face will match. This works because you don’t have a very high chance of getting one right, but you do have a high chance of getting at least one of many right. Like flipping a coin, if on one flip I bet heads, I have a 50% chance of winning the bet. If I bet out of 10 flips, I get at least one heads I have a 1023/1024 chance of winning that bet.
We must understand the interpolation algorithms we use before we start using them in critical areas like court trials. One single mistake from interpolation of an image in a trial can cause significant harm. Also, those operating face recognition software must also understand interpolation even though it doesn’t have as much of an impact.
You bring up scientific error and imply it could have real world impacts. Without an example of where scientific error with bicubic interpolation would lead to a mistake in the picture leading to false conclusions, I treat this article as an introduction. Show a blow up of a photo that turns a knife into a gun or the face of the accused changing into Gerald Ford and it would prove your argument.
Steganography is the practice of hiding text in a picture and rarely changes the picture to be noticeable to the human eye. Your line of reasoning says we could develop an algorithm to find likely probable images where text has been hidden because color bits don’t match the neighbors.
Hi David, you’re correct that this article simply introduces bicubic interpolation and how it COULD lead to real world mistakes. Since the time of writing we now do have a real world example. The defense brought up a picture that used bicubic interpolation to enhance it and claimed it showed Rittenhouse pointing a gun at someone, other didn’t see Rittenhouse point a gun someone from the same enhanced picture. Finally, witnesses said Rittenhouse didn’t point the gun at the time of the picture and the jury seemed to agree. This shows the To add to it, the creators of the program used to create the image in question have said they don’t recommend the use of their software in a trial.