An app for climbing shoe recommendation

One of the most important factors of a good climbing shoe is a good fit. Unfortunately, human feet vary greatly. Feet vary not only in the length, but also in the length/width and length/height ratios, toe lengths:

and deviations like bunions, hammer toes and so on. In the case of walking shoe, a single "size" measure is enough to guarantee a good enough fit, i.e.: the shoe doesn't slip but it also doesn't hurt anywhere. But a single measure is enough for walking shoe only because in walking shoe we tolerate wast spaces between the feet and the shoe (e.g.: between the toes and the shoe). In the climbing shoe, each such empty space results into degradation of the climbing performance because our feet do not have a good contact with the rock at that particular "empty" spot. Hence, power climbers generally prefer as snug fit, as they can handle.

Ideally, an experienced shop assistant should be able to recommend a well fitting climbing shoe based on the look at the client's foot. Shoe, which can provide a snug fit on the client's feet without causing deformities. But my experience is that the salesperson is commonly (and naturally) biased toward the shoes that fit him/her well. Only exceptionally you encounter an expert, who can overcome the bias. But these experts generally (and naturally) work for some brand and if this brand does not make shoe for your type of feet, you are out of luck.

Hence my proposal: an app in a phone, which would take a photo of your feet (a self-photo when you are standing barefoot on the floor), perform some rudimentary calculations (like length-to-width ratio, the ratio of individual toe lengths to the feet length,...), provide some result illustrations in order to persuade the users that your app actually does something (e.g.: overlay the user's foot outline to the prototypical foot) and display shoe ranked from best fit to the worst fit.

There are three obstacles in order to get it working:

1. Data collection
2. Monetization
3. Machine learning  

Data collection

First, you have to have some data in order to feed the recommendation system. The best option would be to contact some climbing shoe manufacturer, present them your aim and ask them for their shoe profiles. I do not think that the biggest manufacturer's like La Sportiva are going to be supportive (they may see it as too risky). But the small manufacturer's may see it as a good opportunity for shoving progressiveness and improving their visibility without risking too much. Plus, thanks to the fact that they are small, they can make their decision quickly. And finally, there are many small manufacturers and they vary wildly - it would be surprising if neither of them was eccentric enough to provide you with the data (or allow you to take photos of the shoe lasts...).

But the initial measurements are just the beginning. You also have to track what people actually buy and what do they return. And based on that alter the recommendations.

Monetization

Second, if you want the app to be successful in the long term, it has to make money. Or it will eventually die due to technological obsolescence (and lack of your motivation to keep it alive on your side). In this case, the monetization model is simple: a shop/brand that can provide good recommendation will make better sales. The reasoning is simple: whenever a customer has doubts which product to buy, they prefer to postpone their action. And whenever they postpone their action, you risk that they will eventually perform the action (the purchase) somewhere else or that they do not perform the action at all (they stick with their current shoe or they even completely abandon climbing). Hence, the app should allow purchase and return realization in order to collect data and make money.

Machine learning

The app must be simple and fast to use. Hence, it is a good idea to not require any measurement with a ruler - a single photo of the foot/feet should be enough.
But how to process the photo? I would simply train a convolutional neural network to identify, which pixels belong to a feet and which to the background. I would collect training images of the feet from internet and build the ground truth masks either in Photoshop (for tough photos) or with local thresholding methods (Savuola thresholding for photos with clean background). Since everything relies on getting a good outline of the feet, I would also instruct the users to get the photo of their feet on white monolithic background like a paper or a wall. The paper has the advantage that you can estimate the size of the paper (A4 or letter) based on the paper side ratio and use it for feet size estimation (toe to heel) and perspective correction (as the camera does not always have to be at the same position and always point in the same direction) based on the knowledge that the paper should have right angles.

Once we have an outline of the foot (let's say of the right foot), we should normalize the outline. This is important for visualization (the overlay the user's feet to the prototypical feet) and for feature extraction. I would simple use affine projection of an ideal foot outline to the obtained outline in OpenCV (or whatever is your favourite tool).

Once we have the user's normalized foot outline, we can compare overlay of the user's foot to the inner shape of the shoe. Ideally, there should be a perfect overlap. And the measure of the overlap can be used for ranking of the shoe.

Latter on, once we collect enough data from sells and return records, we could even retrain the convolutional network to directly rank the shoes based on the foot photo. The idea is, that the photo may contain more information than the outline alone. And that the neural network could be better in deciding, which parts of the feet tolerate overly tight/loose fits and which not so much.

Evaluation

At the beginning, the goal will be to get repeatable results. I.e.: when we snap two photos of the same foot, we expect to get the same foot outline and the same shoe ranking. On the other end, when we snap two wildly different foots, we expect different outlines and different shoe ranking.

Latter on, we can simply maximize profit (sale margin minus the returns).

 

Edit

It looks like that there is already at least one company that takes body measurements thru camera: Menro, which makes smart suits. As a reference of the body size, they take A4 paper with 2 corners blacked to get a good contrast against a (likely white painted) wall.