Complex cropping and resizing idea to make images square

Detailed Description: I’ve seen the autocrop threads here and initially wanted to comment my method there, but the text turned out to be too large, deserving its own thread.

When I have tall/wide images I found the following pipeline to be useful for making them square.

As you know, there are a few ways to change the image aspect ratio, and neither is perfect: cropping loses data, letterboxing introduces bars, resizing squeezes contents. One could also try to fix the bars issue by outpainting the image to the bars area.

So what I found is that while using any one of those methods to do all the work affects the image too much, if you split the aspect change between all of them, each one only distorts the image slightly and in different ways, so the result looks better.

I’m doing the following manually when preparing the dataset:

Find the pixel difference between width and height: D=|W-H|. Divide D by amount of steps (e.g. 4).

Then I crop by D/4, resize by D/4, apply seam carving/content-aware scale by D/4, and finally add bars and outpaint them, which brings the image to 1:1 ratio. Since last two operations are computationally heavy it’s better to do them at the end when the image is already smaller from cropping/resizing. To further reduce cost, it’s better to resize large images before last step to target resolution by larger side (so they will be 512x(H-D/4) and outpainting doesn’t have a lot of pixels to process).

Out of these operations only initial crop requires human input, or you can just use center-crop, since the loss will be negligible (D/4).

Possible improvements to my method:

1. As you might know, there are smart-crop algos that try to preserve faces (or generally more complex parts of the scene, sacrificing simple areas such as sky), so they can be used instead of requiring human input.

2. While seam-carving algo can only reduce the size, content-aware scale can also stretch it, which increases the amount of steps to 5, further reducing the impact of every step. Although I’m not sure if it reduces the distortions of those two methods together.

3. Instead of a naive approach of using the absolute D/steps value for change each step, it might be better to calculate the change every step so it remains a constant fraction of image size rather than a constant number of pixels (since as the image gets smaller, the constant D/4 become a larger change with each step in relation to it). I believe the formula would be M =(W/H)^(1/STEPS), where M is the multiplier below 1 to get the new size each step from the previous one.

One should also consider the cases where the source images are already close to 512×512 or even smaller than that by one side.

I don’t know how difficult it would be to do the last two steps on client-side, however I saw seam-carving implementation in Javascript and it worked reasonably well in browser. For outpainting you could probably use the cheapest nearest-neighbour or blur methods, maybe even border mirroring or pixel-repeating. Anything beats having white bars.

Of course, if you have more ideas about the ways of image resizing, you can add them into this pipeline, more steps = less distortion.