Advice needed?

I am working on building a solution to help find pairs of shoes for a company. Inference runs on a dataset of 851 shoe images - top down. The goal is 100% recall (false positives can be tolerated). The dataset is sparse and is expected to have ~ 40 pairs. The rest is trash. My current setup is: REMBG (silueta) cleans up the background Embed the cleaned images using a deep learning model (tf_efficientnetv2_s.in21k_ft_in1k) b