Better debugging UI for CTE

One of the great advancements in SQL was introduction of Common Table Expressions (CTE), which made creation of long queries without creating intermediate tables on disk (or temporary tables if the database supports it) convenient.

However, debugging of long CTEs is difficult. To find the bug, I currently use bisection. I modify the query to show the first rows of some intermediate CTE somewhere in the middle of the query. If the CTE result looks OK, the bug is somewhere in the second part of the query. If the result does not look OK, I know that the bug is in the first half of the query. And this process repeats recursivelly, until I pinpoint the single offending CTE.

This approach works well on quickly running queries. However, once each query execution takes long time, the debugging drags as it requires multiple executions to just find the offending CTE, let alone to fix the CTE.

A possible mitigation is to enhance the query editor. Just like some query editors allow you to fold CTEs, the editor should present these information for each CTE:

1. the CTE's row count
2. an icon, which on click opens the CTE result

These informations should appear in the editor progressively, as they get available. With this approach, identification of the offending CTE should require, at most, 1 complete query execution. If you are lucky, just by looking at the row count of the intermediate CTEs, you spot the issue while the query is still running.

Technical limitations:

1. The database ought to support temporary tables. This is needed for a simple implementation of the CTEs' result preview.
2. While in the debugging mode, optimisation techniques that work across CTEs, like predicate pushdown, will not be applicable. This is acceptable as programmers expect a slow-down in the debug mode. However, it means that there has to be an additional "Debug" button next to the ordinary "Run" button. These two modes can't be mixed.

Future enhancements:
Do we want to store whole CTE results or only the top 1000 rows? If we store whole tables, we can use that to warm start the query execution, once we modify the query. The disadvantage of storing the whole results is that even a single query in the debugging mode can exhaust disk space and consequently prematurelly terminate, while in the ordinary run mode the query may finnish without any issue. Hence, the limit mode will likely have to be implemented. Nevertheless, for testing of the core idea, the limit/warm-start functionality is not needed.

Summary evaluation for Wikipedia

Wikipedia articles, at least in English, tend to be overgrown - they contain a lot of information of mixed importance. However, we do not always have time to go thru all the content. It helps that articles are structured to have the most important things in the first sentence/paragraph. However, the importance is not really differentiated within the body. If you have to read the body, you get swamp. I use two tricks to deal with that: 1. Switch to a different language. The idea is that articles in different languages are smaller. However, they still contain the most important information. 2. Use a historical version of the article. The idea is that the most important information was entered before the less important information. People are obsessed these days with text-generative AI. Hence a proposal to use AI for shortening of English articles. Do you need a short description? Generate just a single sentence. Was it not enough? Generate the rest of the paragraph. Need even more? Write a subtopic, which interests you. How to evaluate the quality of the summaries? A. Machine translate all different language variants of the article into English and check the information overlap between the summary and the language variants. Ideally, the overlap will be large. This exploits trick #1. B. Check the overlap between the summary and historical versions of the article. Ideally, the information in the summary will be present even in the old versions of the article. This exploits trick #2. Limitations: 1. Some important information is known only from some date. For example, election results are not available before the results are announced. This can be corrected by observing how quickly given information spreads across different language versions. If the information spreads quickly, it is likely important information, even though it is young information. 2. Language variants are highly correlated because they copy from each other. However, it is reasonable to assume that, for example, English and Spanish are more correlated than, for example, Tuu and Thai, simply because fewer people speak both Tuu and Thai than English and Spanish. If the compensation of these differences is necessary, estimate a correlation matrix on the data and use it to weight the signal.

Replication crisis and the proposed solution

Remarkably, only 12 percent of post-replication citations of non-replicable findings acknowledge the replication failure. [1]

Roommate submitted his thesis for publication and one reviewer told him "oh, you cited this result from ~30y ago but it actually has a gap in the proof that no one's figured out how to fix yet." (People learn this stuff via the number theory gossip grapevine apparently?) [2]

Google Scholar is in a great position to reduce the "replication crisis", by alerting the users that the listed article is known to have some defect. 

Principally, it could work like "disputed" on Twitter or Facebook:

Is it the best UX to show a modal window? Most likely not:

1. We want to inform the visitors not just about failed replications, but also about successful replications and small rectifications (like adding a missing condition to a claim or fix of a troublesome typo).
2. We do not want to unnecessarily interrupt the visitor’s flow - maybe the visitor is already familiar with the issues of the article or they just don't care about them.

So what? The information about the presence and the overall conclusion of the replicas could be represented with a double-ended bar chart sparkline similar to how Google Translate shows frequency the translation pair (note the red-gray bar graph at the bottom):

When there is a lot of negative evidence, the red bar graph on the left from the black divider is long. When there is a lot of positive evidence, the green bar graph on the right from the black line is long (not present in this case). 

How to get it started? Let people mark articles as a replication of other articles. 

Why people would bother? 

1. It is a great opportunity for the authors of replication studies to piggyback (collect citations) on the original, likely popular, articles. 
2. After a lot wasted time, you might find out that a claim in paper A does not hold. And that there is paper B that has already spot the issue. It's just that you were not aware of paper's B existence. In the rage, you might be willing to spend a minute and complain to the world that paper A has some issue, as noted by paper B. 

How to collect feedback? The "piggybacking" articles could be explicitly ranked (up-voted/down-voted) like on StackOverflow. While an explicit feedback is not in Google's style, it is important to realize that Google Scholar is for a niche community and niche communities seem to benefit from the explicit feedback as there isn't enough implicit signal (observe success of StackOverflow, Reddit, Hacker News,...). A nice side effect of that would be an increased engagement due to Ikea effect (People values things, on which they have spend some effort, more than things that they got for free. In this case, people would value Google Scholar more, because they have spent time marking articles as a "rectification" of other orticles). 

And what about machine learning? Of course, over the time, Google would collect enough training data, explicit feedback, and implicit feedback, that the pairing of the articles could get fairly reliably predicted. But to get there, Google has to first get the training data.

An app for climbing shoe recommendation

One of the most important factors of a good climbing shoe is a good fit. Unfortunately, human feet vary greatly. Feet vary not only in the length, but also in the length/width and length/height ratios, toe lengths:

and deviations like bunions, hammer toes and so on. In the case of walking shoe, a single "size" measure is enough to guarantee a good enough fit, i.e.: the shoe doesn't slip but it also doesn't hurt anywhere. But a single measure is enough for walking shoe only because in walking shoe we tolerate wast spaces between the feet and the shoe (e.g.: between the toes and the shoe). In the climbing shoe, each such empty space results into degradation of the climbing performance because our feet do not have a good contact with the rock at that particular "empty" spot. Hence, power climbers generally prefer as snug fit, as they can handle.

Ideally, an experienced shop assistant should be able to recommend a well fitting climbing shoe based on the look at the client's foot. Shoe, which can provide a snug fit on the client's feet without causing deformities. But my experience is that the salesperson is commonly (and naturally) biased toward the shoes that fit him/her well. Only exceptionally you encounter an expert, who can overcome the bias. But these experts generally (and naturally) work for some brand and if this brand does not make shoe for your type of feet, you are out of luck.

Hence my proposal: an app in a phone, which would take a photo of your feet (a self-photo when you are standing barefoot on the floor), perform some rudimentary calculations (like length-to-width ratio, the ratio of individual toe lengths to the feet length,...), provide some result illustrations in order to persuade the users that your app actually does something (e.g.: overlay the user's foot outline to the prototypical foot) and display shoe ranked from best fit to the worst fit.

There are three obstacles in order to get it working:

1. Data collection
2. Monetization
3. Machine learning  

Data collection

First, you have to have some data in order to feed the recommendation system. The best option would be to contact some climbing shoe manufacturer, present them your aim and ask them for their shoe profiles. I do not think that the biggest manufacturer's like La Sportiva are going to be supportive (they may see it as too risky). But the small manufacturer's may see it as a good opportunity for shoving progressiveness and improving their visibility without risking too much. Plus, thanks to the fact that they are small, they can make their decision quickly. And finally, there are many small manufacturers and they vary wildly - it would be surprising if neither of them was eccentric enough to provide you with the data (or allow you to take photos of the shoe lasts...).

But the initial measurements are just the beginning. You also have to track what people actually buy and what do they return. And based on that alter the recommendations.

Monetization

Second, if you want the app to be successful in the long term, it has to make money. Or it will eventually die due to technological obsolescence (and lack of your motivation to keep it alive on your side). In this case, the monetization model is simple: a shop/brand that can provide good recommendation will make better sales. The reasoning is simple: whenever a customer has doubts which product to buy, they prefer to postpone their action. And whenever they postpone their action, you risk that they will eventually perform the action (the purchase) somewhere else or that they do not perform the action at all (they stick with their current shoe or they even completely abandon climbing). Hence, the app should allow purchase and return realization in order to collect data and make money.

Machine learning

The app must be simple and fast to use. Hence, it is a good idea to not require any measurement with a ruler - a single photo of the foot/feet should be enough.
But how to process the photo? I would simply train a convolutional neural network to identify, which pixels belong to a feet and which to the background. I would collect training images of the feet from internet and build the ground truth masks either in Photoshop (for tough photos) or with local thresholding methods (Savuola thresholding for photos with clean background). Since everything relies on getting a good outline of the feet, I would also instruct the users to get the photo of their feet on white monolithic background like a paper or a wall. The paper has the advantage that you can estimate the size of the paper (A4 or letter) based on the paper side ratio and use it for feet size estimation (toe to heel) and perspective correction (as the camera does not always have to be at the same position and always point in the same direction) based on the knowledge that the paper should have right angles.

Once we have an outline of the foot (let's say of the right foot), we should normalize the outline. This is important for visualization (the overlay the user's feet to the prototypical feet) and for feature extraction. I would simple use affine projection of an ideal foot outline to the obtained outline in OpenCV (or whatever is your favourite tool).

Once we have the user's normalized foot outline, we can compare overlay of the user's foot to the inner shape of the shoe. Ideally, there should be a perfect overlap. And the measure of the overlap can be used for ranking of the shoe.

Latter on, once we collect enough data from sells and return records, we could even retrain the convolutional network to directly rank the shoes based on the foot photo. The idea is, that the photo may contain more information than the outline alone. And that the neural network could be better in deciding, which parts of the feet tolerate overly tight/loose fits and which not so much.

Evaluation

At the beginning, the goal will be to get repeatable results. I.e.: when we snap two photos of the same foot, we expect to get the same foot outline and the same shoe ranking. On the other end, when we snap two wildly different foots, we expect different outlines and different shoe ranking.

Latter on, we can simply maximize profit (sale margin minus the returns).

 

Edit

It looks like that there is already at least one company that takes body measurements thru camera: Menro, which makes smart suits. As a reference of the body size, they take A4 paper with 2 corners blacked to get a good contrast against a (likely white painted) wall.

Hacker in the browser and other hacking ideas

If I was a hacker with the ability to modify web pages in the browser, I would modify the top of Wikipedia and ask people to donate. People accustomed to donations would donate again. But this time to my pocket. If it was synced with the real campaign it would be awesome.

Or if I was Microsoft and detected that two people communication over Skype (or other messenger) are in a relationship, I would suggest the male counterpart to send roses to his fiancé. It would be just a small bunch of roses. But the delivery would be guaranteed in the next 30 minutes for the current estimated location of the fiancé.

And to be even more devilish I would use the fiancé as the vector. I would show a pop-up to the fiancé saying: “Do you want to test your boyfriend? Show him an advertisement with one of the following buckets!” She would be presented with three simple options like snowdrops, violas and sunflowers. She would be thinking about the ad while chatting with the boyfriend and in the end she would click on one of the buckets because who would not like to test the love of the beloved one? And to make sure that the boyfriend is not going to disappoint her (man are notoriously unreliable) she would hinge the boyfriend to buy the flowers until she actually gets the bucket (women can get really persistent when they decide to get something). The poor boyfriend would be then forced to buy the overprized bucket because who the hell should bear all that morning of the fiancé about some stupid inedible vegetable? Sooner he gets over this the better.

The boyfriend would be stressed to buy the specific bucket. But for men it’s hard to multitask (search for a cheaper delivery while chatting with his fiancé). And it would be ridiculously hard to find a delivery with that specific flower (Why the hell didn’t she want roses?!) and even harder to find deliver able to deliver the bucket in less than 30 minutes. Simply Machiavellous.

Camera sensor

Recently I have noticed that someone patented a layout of sensors at camera chip that was better tuned to sensitivity of eye. Particularly the patented chip was combining colorless sensors with color sensors. This combination makes sense since human eye is more sensitive to luminance than color. Furthermore the resulting pictures are less noisy because colorless sensors do not filter light.

Nevertheless, the presented design doesn’t exactly follow the sensitivity of human eye. Hence I predict that sooner or later someone will patent the right proportion of sensors without specifying the exact geometric shape. And indeed it’s possible that the exact layout of the sensors will be random (while preserving the right proportions).

Nest Thermostat

Since we have self learning thermostats I predict that sooner or later we will have self learning electrical kettles. When you wake up, the water for the morning coffee will be preheated. So when you push the button you don't have to wait an eternity for the morning cup of coffee. And in the evening when it detects you are returning home it preheats water. It doesn't matter whether you use the hot water for a cup of tea or for dinner. It will be there ready on push of button. But in the mean time the kettle is not going to keep the water warm since it knows no one is home. Comfort and efficiency blended together. PS: did you notice how hard it is to estimate how much of water to pour into the kettle so you always pour a bit more of water than is necessary just to be sure you don't end up with too little water? If you have your favorite cup the intelligent kettle can learn it and preheat the exact amount of water. And it can get so far that you push the button and based on the sensors it fills any cup just the way you like - to the border or a centimeter below. Your choice.

Maturity

Se státem zadávanými maturitami je problém, že studenti neví, zda si mají vybrat těžší variantu, a potom třeba propadnout, a nebo raději nic neriskovat a dělat jednodušší variantu. Další problém je, že složitost maturit občas ulétne a propadne víc jak 10% žáků.

Jako řešení se přímo nabízí lehčí a těžší variantu sjednotit. Slabší žák v zadaném čase zvládne správně vyřešit jen jednodušší příklady, zatímco schopnější žák zvládne vyřešit i ty těžší. Celkově by obtížnost maturity byla nastavena tak, aby průměrné skóre bylo kolem 50% a nikdo nedal ani 0% ni 100%. Protože kdyby žáci skórovali 100%, tak se neví, kdo je dobrý na 100%, nebo třeba na 160%. Oba dosáhnou stejného skóre. A obdobně to platí u 0% hranice. A možná že u dolní hranice si přejeme zachovat rozlišovací schopnost více, než u horní. Protože horní hranice jen určuje, kdo bude dělat zkoušky na vysokou školu a kdo bude přijat bez přijímaček. Ale dolní hranice rozlišuje, kdo dá maturitu a kdo nikoliv.

Celkově by se jednalo o zjednodušení. Žák by si variantu dopředu nevolil, učitelé by rozdávali jen jedno řešení a Cermat by vypracovával jen jedno zadání.Většina lidí by to tedy přivítala. Jediný problém by bylo, že někteří žáci by mohli mít problém se srovnat s tím, že u maturity nedosáhli 100%, jako obvykle dosahovali ve škole. Tohle je problém pro marketing, aby vysvětlil, že nikdo nemá dosáhnout 100% a jen promile se k tomu má přiblížit.

Druhá část řešení maturit je hodnotit výsledky relativně, a nikoliv absolutně. Z Cermatu několikrát přímo zaznělo, že maturitní testy byly koncipovány tak, aby 8% žáků je nedalo. Ale z novin víme, že jim to každoročně ulítne a někde propadne mnohem víc žáků. Proto by mělo být na rovinu řečeno, že 8% žáků s nejslabším skórem maturitu nedá. Cermatu by odpadli skandály, že jim to někde ulítne, a žáci by si nestěžovali, že jich 50% vyletělo. Problém by opět mohl být jak žákům, kteří jsou uvyklí na absolutní hodnocení, vysvětlit, že relativní hodnocení je pro ně lepší. Ale to je opět záležitost pro marketing, aby ukázalo, že v novém systému se nesoupeří mezi spolužáky, ale mezi školami. A třeba se může říci, že hranice pro průchod se snížila z 8% na 6%. Což může být pravda a přitom se nemusí nic fakticky změnit, protože většinou se stejně u maturit ukázalo, že průměrně projde víc žáků, než se plánuje, protože Cermat se jistí, aby se často nestávaly takové extrémy, jako že 50% žáků vyletí.

Soup bowl from a brick

I have been examined on creativity and asked "For what can you use a brick?". And I thought about using a brick as a soup bowl in a restaurant. Just imagine the eyes of a surprised guest when you tell him "here is your soup" and all he can see is a plate with a steaming brick. The same brick from which the restaurant is built.

How to create a bowl from a brick:

1. You need a high quality brick, which doesn't crumble. It can have decorative imprints on it's sides.
2. Use lathe to carve a bath into the brick. It's possible to carve a logo into the sides of the bowl.
3. Use hydrophobic impregnation to prevent soaking of water by the brick during the washing. 
4. Glaze the inside of the bath so the customer has a sense of cleanliness.

Greatest easily replicable inventions of 20th century

By simple I mean things you can replicate in a reasonably short time just yourself, given you know ahead how to make them.

1. Diffie-Hellman asymmetric cipher
2. Game theory applied to prisoners dilemma, Nash equilibrium
3. Prospect theory. Demonstrate it on a questionnaire
4. Penicilin. Use bread
5. Nylon. Use it on Velcro - demonstrate it on shoe for kids. Use on pantyhose and toothbrush
6. Ballpoint Pen
7. Zip
8. Slinky
9. Rubic cube
10. Sticky notes
11. Contact lenses
12. Two-ply toilet paper 
13. Dunning-Krugger effect (Wikipedia)
14. Bubble wrap 
15. Elastic rope 
16. Jenga (Wikipedia) 
17. Twister game 
18. Crosswords
19. Controlled studies in medicine 
20. Fastener

Frecency

Jak já nesnáším Firefoxí frecency algoritmus! Po několik let navštěvuji svůj oblíbený blog, začínající na písmeno B. Vždycky v urlbaru napíšu B, odentruji a jsem na blogu. Jenže jednoho dne jsem potřeboval napsat článek, řekněme o Bunkrech. A tak jsem během během jednoho dne navštívil hypotetické stránky bunkry.cz. A když jsem měl svůj článek hotov, přál jsem si odpočinout. Zadal jsem v prohlížeči B, odentroval a ... byl jsem zpět na bunkrech.

Jsem přesvědčen, že Firefox by měl uživatelům nabídnout nejen možnost změnit, jak dlouho si přejí uchovávat historii, ale i ostatní parametry, jako je rychlost učení se novým trendům. A to vše pomocí táhla krátká paměť - dlouhá paměť. Dlouhá paměť si bude udržovat historii po delší dobu a bude konzervativnější na změny, krátkodobá paměť zase pravý opak.

Dalším zlepšením by bylo zvýšit váhu ohvězdičkovaným stránkám s prodlužující se pamětí, protože jinak důležité, ale málo často navštěvované ohvězdičkované stránky, se ztrácejí v záplavě stránek, které člověk navštěvuje denně, protože ty časem naakumulují obrovskou váhu.

Awesome Bar je skvělá věc, ale není tak awesome, jak by mohl být. Snad příště.