Pandas violates composability

Python Pandas library violates composability. What is a composability? It is the ability to take a part of code and use it as an input for another code.

For example, SQL is composable - SQL takes relations and produces a relation. Lisp is composable - Lisp takes lists and produces a list. Matlab is composable - classical Matlab code takes arrays and produces an array.

But Pandas is not composable. A Pandas method, which takes a DataFrame, may return anything, be it:

• Categorical
• DataFrame
• Group
  
• List 
  
• Mask (a Numpy vector)
• Index
• IntervalIndex
• Scalar (be it a Python data type, Numpy data type or Pandas data type)
  
• Series
• Slice
  
• TimeDeltaIndex

And these are just the one I got after one minute at Pandas API.

Now you may argue: "What's the problem? Python uses duct typing". The issue is, that many Pandas methods, which accept a DataFrame, refuse to work on anything but a DataFrame. And that's the better case. In the worse case, the method accepts it, but produces something else than you would get if you passed the data as a DataFrame.

To make it even worse, Pandas mixes the ways how do you do things. Sometimes, you have to use method of the instance. Sometimes you have to access an attribute of the instance. And sometimes you have to use a class method.

If Pandas was using class methods everywhere, like Numpy, it would be possible to hide the differences between the different data structures. But you don't hide a missing attribute or method of an instance.

Composability is not a small thing. For example, GQL, a new language for graph databases, has composability as the governing principle. And people seem to, knowingly or not, gravitate toward composability. If nothing else, datatable, an alternative to Pandas in Python, ditches Series data structure, as it is considered unnecesary.

Matlab

I took me a while to realize why I reproduce results from articles in Matlab (and not in Java, Python or R).

Here are the reasons:

1. Matlab has succinct syntax for matrix operations, which is similar enough to equations in articles (this is where Java gets beaten).
2. It is extremely simple to get an example data from the article into the Matlab code - you just copy-paste the data from the table in the article, surround them in square brackets and that's it. You have a matrix. There is no need to separate the numbers with commas (but you can - they are optional). There is no need to separate the individual rows with brackets. It just works. This is where Python and R gets beaten.

Who would guess that great copy-paste support would affect the choice of the language...

Another nice thing is that Matlab (and R in RStudio) autocompletes paths to files and and directories - once again, it is a small thing. But it helps to avoid typos.

Keyboard shortcuts on MacOS

MacOs does some things right. And some not so much.

Positives:

1. A dedicated (and simple!) shortcut for displaying the configuration of an application. On windows, you generally have to fish for "setting" or "preferences" in the application menu. Not so on MacOS - it is always command+",".
2. A variety of shortcuts for different screenshots.

Negatives:

1. No dedicated keyboard shortcut for find and replace. On Windows, it is always ctrl+"r". On MacOS, it is application specific.  Sometimes it is command+alt+"f", sometimes command+shift+"f" and sometimes there is simply no keyboard shortcut at all and you have to use mouse to tell the app to perform the find and replace and not just find! If you wonder, both, F5 and command+"r" perform refresh.
2. No keyboard shortcut for displaying context menu - you have to use right click on the mouse.

A proper handling of nulls

I am familiar with two approaches how to deal with nulls in programming languages: either completely avoid them or embrace them. Languages for logic programming generally avoid the nulls (a great apologetic is given by Design and Implementation of the LogicBlox System). But languages that permit them should provide following facilities:

1) meta information about why the value is missing
2) nullable & non-nullable variables

For example, SAS got it right 40 years ago. In SAS, a missing value is represented with a dot. That by itself is not great whenever you need to print out the code or the data, because you never know whether that dot really represents a missing variable or it is just an imperfection of the paper or the printer. But it permits to easily define the reason why the value is missing:
    .                 // Generic missing value
    .refusedToAnswer  // Missing value with a metadatum
    .didntKnow        // Missing value with a different metadatum Hence, generic algorithms can threat all missing values the same way. But if you want to treat them differently, for example because refusedToAnswer can have a vastly different meaning in a questionary than didntKnow, you can do it.

Furthermore, SAS provides optional non-null constraints on attributes, just like SQL. The only ward on SAS's implementation is that it raises exceptions only during the runtime, not during the compilation time as, for example, Kotlin does. Note also that nullability must be configurable for all variables. For example in C#, nullability is configurable only for value types, not class types. And this omission is a source of many null-pointer exceptions.

There is just one thing where I am not sure which approach is better. If we have a function sum, it can:

1. Accept nullable variables and use some default strategy to deal with nulls (as SQL does).
2. Accept nullable variables and blow during the runtime when null is encountered, unless some strategy is defined (R takes this approach).
3. Have a dedicated sumnan function, which accepts nullable variables and takes a parameter, which determines, how nulls should be treated. Non-nulable variables get accepted by sum function (something like this is used in MATLAB, minus the type control).

The first approach is convenient to use. But potentially dangerous, because the user may never realize that a null leaked into the data and the conclusions are wrong.

The second approach is safer, because the user at least learns that something is wrong immediately when null is passed to the function without passing a strategy how to deal with nulls. The disadvantage is that it is a runtime check, not a static check. Nevertheless, programmers that are concerned about safety may use a lint to identify calls to functions with nullable variables without defining the strategy what to do with nulls.

The third approach is easier to validate by the compiler than the second approach. But the naming convention can be difficult to enforce.

Do you have some thought about this issue?

What is the correct measure for classification?

There are multiple measures that we can use to describe performance of a classifier. To name just a few: classification accuracy (ACC), area under receiver-operating curve (ROC-AUC) or F-measure.

Each metric is suitable for a different task. And there are two questions we have ask to select an appropriate metric:

1. Is the decision threshold known ahead and fixed?
2. Do we need calibrated predictions?  

If a single threshold is known ahead and fixed, for example, because we know the misclassification cost matrix, threshold-based metrics like classification accuracy or F-measure are appropriate. However, if we do not know the cost matrix or if the cost matrix is expected to evolve in time, it is desirable to use ranking-based metrics like ROC-AUC, that consider all possible thresholds with equal weight.

Of course, sometimes we may want sometimes in the middle, because we have some knowledge about the distribution of the values in the cost matrix. Example measures that consider just a range of thresholds or weight the thresholds unequally are partial Gini index, lift or H-measure. If you have some information about the working region of the model, you should use a measure that takes that knowledge into account (see the correlations of measures in the second referenced article).

Another question to ask is whether we need calibrated results. If all we need is to rank results and pick top n% of samples, ranking measures like ROC-AUC or lift are appropriate, following the  Vapnik's philosophy: "Do not solve harder problem than you have to". On the other end, if we need to asses risk, calibrated results are desirable. Measures that evaluate quality of calibration are Brier score or logarithmic-loss.

And of course, there are measures that are somewhere in the middle, like threshold-base measures, that evaluate quality of calibration at a single threshold, while ignoring the quality of calibration at the rest of thresholds.

A metric may also feature following desirable properties:

1. Provides information about how much better is the model in comparison to a baseline for easier interpretation of the quality of the model.
2. Works not only with binary, but also with polynomial labels.
3. Is bounded for easier comparison of models across different datasets.
4. Works also with unbalanced classes. 

But that would be for a longer post. Relevant literature:

1. Getting the Most Out of Ensemble Selection
2. Benchmarking state-of-the-art classification algorithms for credit scoring: An update of research

Reserved words. Or no reserved words.

Recently I have heard that Prolog is awesome because it uses only 6 reserved words. However, is less better?

I argue that there is a sweet spot. You can write your code in bit code. But sooner or later you will realize that you are repeating the same sequence again and again. So you start to giving names to the repeating sequences. Congratulation. You have just reinvented assembler. Later on you realize that it would be nice to simplify some constructs. And you end up with something like C.

On the other end there are languages which have many keywords. Like SAS. There are so many keywords to remember, that you have to use documentation all the time. Hence this extreme is neither ideal.

OK. Too little or too much reserved words is bad. But where is the sweet spot? Let's introduce a parallel between computer and human languages. They both function for communication. And some computer languages, like Prolog or SQL, were modeled after human languages. Hence we can translate the problem from what is the optimal count of reserved words in computer languages to the count of unique sound atoms in human languages. And in many cases we can approximate this count with the length of alphabet. Here is (an incomplete) list of alphabets:

27 Hebrew
27 Latin
28 Arabic
28 Hindi
29 Cyrillic
48 Kanji

There are some extremes in the coding. For example, Chinese are using many characters. However, one character doesn't have to always represent an atomic sound. On the other end we can represent a language with Morse code. However, it doesn't appear to be a preferred way of communication - otherwise we would not bother to convert human voice to bits, send it over space and transform back to sound when we are talking over cellphones when the cellphones has a button, which directly generates signal in bit form.

Since the distribution of lengths of alphabets is left bounded, it is asymmetric. Hence it is better to use median for the mean value. And based on millenniums of evolution we are safe to say that the optimal count of reserved words in a programming language is 28.

Of course you can argue, that they are accents. Or changes in pronunciation when one character follows another. But you could say something similar about the programming languages. Hence let it ignore for the simplicity.

How do the computer languages compare between each other? Let's see:

• ANSI COBOL 85: 357
• SystemVerilog: 250 + 73 reserved system functions = 323
• C#: 79 + 23 contextual = 102
• F#: 64 + 8 from ocaml + 26 future = 98
• C++: 82
• Java: 50
• PHP: 49
• Ruby 42
• Python 3.x: 33
• C: 32
• Python 2.7: 31
• Go: 25
• Smalltalk: 6 pseudo-variables
• iota: 2

Based on this comparison Python made it right.

For discussion about the topic see http://lambda-the-ultimate.org/node/4295.

Comparison of MATLAB and R

Advantage of R:

• Easy setting of default parameters (inheritance from functional languages). Not that it is incredibly difficult to set a default value in MATLAB, but it's verbose and error prone.
• Named parameters (again, inheritance from functional languages). In MATLAB, when you pass many parameters with string values to a function, it's unclear at glance, what is parameter name and what is parameter value. In R, it's immediately clear.
• Mixed tables (combination of string and numerical columns). Incredibly useful for real world messy data sets. A partial remedy to this problem is 'Tables' in the late versions of MATLAB.
• Possibility to name rows and columns.This is awesome because you don't have to remember that you want column 181, all you have to remember is the name of the column. Also, it has the advantage that metadata are together with the data. Hence if you perform selection, projection or transformation of the data, the metadata are automatically in sync with the data. No work is left on the user. In MATLAB, you have to use 'Struct'. Or 'Tables' in the late versions of MATLAB.
• Negative indexes for dropping of particular columns/rows.

Advantage of MATLAB:

• There is a fewer competing packages for MATLAB than for R. Hence in MATLAB you are spared of deciding, which library is the best.
• Spare matrices are integral part of MATLAB. Hence all algorithms benefiting from spare matrices are using the same representation of spare matrices. In R, each library is using it's own representation.

Difference between Machine Learning and Artificial Inteligence

In my biased opinion, the difference between Machine Learning (ML) and Artificial Intelligence (AI) is in the way, how do they solve problems. AI seeks an optimal solution, while ML seeks a usable solution.

And this difference reflects in used tool sets. A typical tool for AI is logic, which is traditionally binary. On the other end probability, a common tool in ML, allows any value between 0 and 1.

The difference reflects in individual algorithms as well. A* is a representative algorithm from AI. It is an elegant algorithm that guaranties, that the returned is optimal. In contrast, neuron network, an algorithm from ML, doesn’t guaranty optimality of the solution at all. But it can tackle much wider range of problems than A*. And that is the reason, why ML is currently more popular and successful than AI. Despite all the hopes, it turned out we are unable to optimally solve many problems like voice or object recognition. All we can hope for is a good enough solution. And that is exactly the thing, where ML beats AI. ML is all about “how to get a usable solution”, while AI is about “how to get an optimal solution”.  And when this optimal solution is unreachable, AI just gives up, while ML gives  at least something.

Comparison of SAS data step and SQL

The default tool for ETL in SAS is data step. However, SAS also offers support for SQL. When to use which?

The main advantages of data step are:

1. Drop keyword. Let's imagine that you want to remove one column from a table with 2000 columns. In SQL you would have to name all columns you want to keep. But in data step it is enough to just name the column you don't want to include. Awesome.
2. Wildcards. If you want to select all columns beginning with "pred_", all you have to do in data step is to write "pred_:" (note the column). In SQL you would have to write name of each predictor.
3. Speed. SQL in SAS is not implemented overly effectively. 
4. LAG command. In SQL you have to perform a slow and cumbersome join to get the corresponding functionality. 

The main advantages of SQL:

1. Group by command. Simply because data step doesn't offer such functionality.
2. Order by command. Again you can't sort directly in data step.  
3. Metadata. Queries on the metadata are so addictive!

The difference between statistics, machine learning, data mining and data science.

Originally there was just statistics - a method how to summarize huge populations into two numbers, average and variance. And with a bit of exaggeration whole statistics is operating with just these two numbers. With these two numbers you can compute significance, confidence intervals, correlation, regression and many other. Back in time it was amazing success - you could operate with millions of records on a single piece of paper.

But with dawn of computers people became less limited in the amount of computation that was deemed practical and they started to think in big. What if we worked with whole population distribution? Or if we run these old trivial statistical tests on this huge pile of data, wouldn’t we find something? These two questions stood at the beginning of two fields, machine learning, evaluating the former question, and data mining, evaluating the later question.

Statisticians with access to computers started to pull nasty tricks like bootstrapping to narrow confidence intervals. And traditional statistics felt threatened. And they started to accuse computer statisticians from cheating. Latter on computer statisticians persuaded traditional statisticians about validity of the approach and statisticians accepted bootstrapping as a useful tool. But disagreements like this led to divergence of machine learning from statistics.

Similarly guys and gals from data mining were targets of many attacks because data mining allowed production of scientific articles with amazing pace – what would have taken whole life of a respected scientist could have been done in less than 5 minutes with a stupid computer. Unfortunately, in this case the despect was deserved because many results of data mining were false positives. Later on data miners learned to use Bonferroni correction and validate results to decrease the rate of false positives, but damage was done. Both, statisticians and machine learners, started to look down upon data miners as kids that learned a few tricks, which they apply without any deeper understanding.

With rise of Internet access to data simplified and the biggest time burden shifted from data collection to data procession. The methods invented by machine learners were hopelessly slow on data from Internet and phones and even methods employed by data miners were too slow to be executed on whole datasets. This change of paradigm led to return to the roots of statistics where people first created hypotheses and then they studied the data to prove or invalidate the hypothesis. But because focus shifted from correctness of methods (they were proved many times since then) to efficient computation of trivial algorithms, new group of statisticians with computer science background emerged. Nowadays we call this group as data scientists.

PS: A quick guide how to differentiate different fields based on the keywords:

1. State space -> artificial inteligence
2. Significance -> statistics
3. Maximum a posteriori (MAP) -> machine learning
4. Algorithmic efficiency (O-notation) -> theoretical computer science
5. Cross-validation -> data mining

Comparison of variable selection methods

Since I didn't know which variable selection method to use, I performed a trivial test on Sonar dataset. Sonar dataset has 60 attributes. But I arbitrarily decided to reduce the number of attributes to 10. Then I measured classification accuracy with ten fold cross-validation. And to get an idea how feature selection methods are dependent on the classifiers I tried three different models: naive Bayes, k-NN and classification tree:

Based on the comparison the best method to use is SVM attribute selection. However, this method requires parameter tunning. The next best variable selection method is Chi2. The disadvantage of this method is that it favors attributes with many levels. Hence the performance of Chi2 could be severely hindered on diverse set not like Sonar set. The last method from the top three is information gain ratio. The advantage of this method is that it can handle attributes with diverse number of levels not like Chi2.

Comparsion of SAS Enterprise Miner and RapidMiner

Enterprise Miner (EM)
+ Nice interactive visualizations. Whenever you click on a label, the corresponding data in the chart gets highlighted. Still, the plots could support interactive exploration similar to KNIME, where when you highlight a sample in one plot, it gets highlighted in all other plots.
+ Ingenious default settings. You plug the operator and it works. In RapidMiner you have often preprocess the data and fiddle with the settings to get usable result in a reasonable time.
+ You can selectively execute a portion of a flow. While in RapidMiner you can execute only whole flow. The selective execution in EM is a great feature, which accelerates development, because if you make a mistake at the end of the flow, you don't have to recalculate everything from the scratch. Instead of that, EM (by default) recalculates only branches affected by the change. Unfortunately, this feature takes a high toll. While RapidMiner can process data "on the fly", EM has to process data in steps - each operator in the flow has to finish before the subsequent operators can start. Furthermore, in EM you have to have fast and big storage for intermediate calculations.    
- Uninformative error messages. You often have to blindly test many different things until you find the root of the problem.
- It's full of bugs. Not that RapidMiner would be free of bugs. But after a day of bug hunting in RapidMiner you can at least fix it in the source code by yourself. In the case of SAS you either have to be better hacker, then am I, or you have to go through frustrating process of communication with the infamous SAS support.
- EM ecosystem lacks ETL operators - instead of them, you are supposed to use SAS code operator. Personally, I prefer to perform all the transformations in Enterprise Guide. And then I just load the final table(s) into EM.
- EM doesn't know undo command. Instead of undo, you have to confirm each action, since each change is irreversible. This approach goes against the best UI practice to limit amount of modal windows and make each action easily reversible.  
- EM doesn't know save command. Beware of blind confirming of modal windows! It happened to me several time that instead of deleting a single operator whole flow was deleted because I miss-clicked the operator and selected the drawing board instead.
- Setup SAS ecosystem takes some time. Count something around one month.
- Keep SAS ecosystem running takes a lot of energy. Count two days of repairs for each working day.
- Import of data into EM is tedious as you have first define a library and then walk through a long wizard. Count 5 minutes for even the simplest data set. But at least example data sets from SASHELP are fast to load.
- Moving of EM projects from one directory to another is trivial but unintuitive. First, you must make sure that the name of the project directory is the same as the project name. Second, you open moved projects via "New project" and selecting project directory as the destination directory.   
- Metadatabase. So far all it does is that it doubles maintenance time. I am sure there must be some benefit of having metadatabase but so far I do not see it. Hence I strongly suggest everyone to install a standalone version instead of the server version unless you have some good reason to do it otherwise.

RapidMiner (RM)
+ Fully sufficient ecosystem. You can do all ETL within RapidMiner.
+ You can truly visually program in RM since you have loop operators and conditions.
+ You have source codes.
+ The drawing board is editable during the runtime. Hence you can prepare a new experiment while RapidMiner is still performing calculations on the last experiment.
- Every time you have to rerun whole flow even though you want to make just a small refinement.
- It's too easy to setup the flows/operators that they would take eternity to calculate. For comparison, everything in SAS is optimized for big data and you don't get trapped in computational black holes too often.
- You can't stop running node. You can only tell RapidMiner to prevent execution of subsequent nodes. Hence forced application restarts are common.
~ Since each connection between operators in RM fulfills distinct function it's common that you have to wire two operators with more than one connection (for example one connection for training data and second connection for testing data), making the flow look overcrowded. In EM it's always enough to make a single connection between two operators as each connection can transfer any type of data (training, testing, validation...). Hence flows in EM looks tidier.
~ On the other end RM allows grouping of operators into a single operator allowing "divide and conquer" strategy. While absence of operator nesting in EM makes small EM flows easy to understand, bigger flows look messier in EM than in RM.
- SVD and PCA are ridiculously slow and memory consuming in comparison to SAS versions.
- Graphical presentation of hierarchical clustering is just a joke in comparison to output from Orange, KNIME or MATLAB.
- I miss Partial Least Square regression and full Bayes (not just Naive Baves). But you can get them from WEKA plugin.
- ROC plot should show reference line for simpler visual evaluation.