Automatic Sensor Dust Removal

Interchangeable lens cameras are amazing feats of engineering, but as photographers trying to capture that perfect image you are sometimes saddened to realize that there are big black smudges scattered throughout your digital image. These black spots or smudges are caused by small specs of dust and debris that settle on the image sensor and occlude the image being taken.

There are many ways to clean the sensor, using air blowers and special cleaners, but most modern cameras have a sensor vibration function that tries to shake the sensor to dislodge the dust particles.

All these things are great, but they can’t fix an image that was already taken. For that, we need to use image-editing software like Photoshop or Gimp to touch up the image spots manually. This is pretty simple if you only have a few images to re-touch, but what if you had hundreds?

This got me thinking, what if I could use image processing combined with the retouching capabilities of current photo-editing algorithms to automatically touch up photos that had been marred by dust spots.

OpenCV

I was inspired by this example from PyImageSearch to use OpenCV to try to correct the sensor dust occlusions that occur. In the example, an image to fix is combined with a pre-computed mask that tells OpenCV where to apply the inpainting.

Image inpainting with OpenCV and Python

Install the opencv library for python using pip command below.

pip install opencv-contrib-python 

I created a proof of concept for correcting the sensor dust blemishes on an image. I used Inkscape to create the required mask image that places white dots over the locations that contain dust, blanking out the rest of the image. These white spots will be where the image inpainting algorithm performs the operation.

• 

• 

Now when we perform the inpainting we can see those dust spots disappear! There are some tuning aspects that we can try as there are still some irregular spots but they are much less noticeable than the dark dust spots.

You can take a look at the code using my Jupyter Notebook that will walk you through the example.

Loading

Sorry, something went wrong. Reload?

Sorry, we cannot display this file.

Sorry, this file is invalid so it cannot be displayed.

view raw ocv_sensor_dust_inpainting.ipynb hosted with ❤ by GitHub

Now that I know how to correct the spots I need to deviate from the example and do pre-processing on an image to automatically generate the mask based on detected dust locations.

How to Detect Sensor Dust

In order to detect sensor dust on images, we need to use object detection methods to find and locate the dust. Once we know the pixel locations that make up that dust occlusion, we can use that data to create a binary mask that can then be passed into our inpainting algorithm for removal.

Lets start with this as the origional image that we want to correct.

Image Processing Algorithm

I spent a good amount of time experimenting with various features of OpenCV from thresholds and filters to contours and more. But finally came up with a decent solution that can pick up a good portion of the dust particles while rejecting background noise in images without dust issues.

First, we need to apply some thresholds to the image to bring out the dust particles and separate the sparse sky from the noisy foreground. To bring out the dust particles we will be using an adaptive mean threshold. The adaptive mean threshold operates on the image in sections and applies a threshold for each pixel based on its neighboring pixels.

Then to help fill in the non-sparse parts of the image we can use an adjusted mean global threshold. This will find the mean of all the pixels in the image and reduce it slightly to increase coverage. This will bring up and fill in the dense portions of the image.

The thresholded images will then be filtered with a magnification kernel. This magnification kernel is a matrix of all ones multiplied by a gain factor. For my high-resolution images, I used a 10×10 matrix with a gain of 10. This combination gave a good dust magnification and rejection fill without covering over multiple dust spots.

This gain kernel was applied multiple times to the adaptive thresholded image to pull those small dust dots into squares. The application of this filter kernel multiple times is possible because dust spots will converge into small solid white squares but denser parts of the image will be filled in to become a contiguous region.

I then summed the filtered adaptive and global threshold images to create the main detection image. This allows for the dust to stand out while the other parts of the image merge into each other so we can more easily reject them.

Now to extract only the small squares we can then calculate the contours of this image. This basically creates borders around the white pixel areas. With the contours defined we can classify them as shapes based on geometry.

Once we classified them we can use the classified squares and rectangles to create our mask. We do this by creating an all-black pixel image, with the same resolution as our original, then drawing filled polygons in the shape of the contours in that image.

With that done, we can correct our original image using the same procedure as our basic inpainting example.

Jupyter Notebook

You can take a look at the full algorithm in the jupyter notebook I created showing the code with the same steps as the above algorithm.

Loading

Sorry, something went wrong. Reload?

Sorry, we cannot display this file.

Sorry, this file is invalid so it cannot be displayed.

view raw auto_sensor_dust_inpainting.ipynb hosted with ❤ by GitHub

If you want to run the code on your own machine you can clone my GitHub repo that contains some test images for you to play with. I have not tried different image resolutions so the algorithm might fail for lower resolution images.

GitHub: Automatic Sensor Dust Removal

Performance

It looks like the detection and inpainting are doing an ok job, but I have not run any empirical performance metrics. This would require going through all my images to locate the dust and manually remove it to create a baseline. I will probably do this for a few images but not the whole set.

Future Improvements

My next goal is to integrate this algorithm into a GUI application using DearPyGui where the user can import multiple image files and tune the parameters of the algorithm to best suit their images. This requires a much more robust algorithm with error checking and handling which are not currently available in the notebook algorithm.

I also want to see if AI or Artificial Intelligence could improve the algorithm performance. AI has really taken over all areas concerning images, from object detection and classification to auto-correction, enhancement, and many more.