To calculate all the p-values at once, you can use calculate_pvalues function (code below): df = pd.DataFrame({‘A’:[1,2,3], ‘B’:[2,5,3], ‘C’:[5,2,1], ‘D’:[‘text’,2,3] }) calculate_pvalues(df) The output is similar to the corr() (but with p-values): A B C A 0 0.7877 0.1789 B 0.7877 0 0.6088 C 0.1789 0.6088 0 Details: Column D is automatically ignored as it … Read more
Correlated features do not affect classification accuracy per se. The problem in realistic situations is that we have a finite number of training examples with which to train a classifier. For a fixed number of training examples, increasing the number of features typically increases classification accuracy to a point but as the number of features … Read more
Kinda finished I guess.. code can be found here at github. Documentation is included in the file. The yellow/magenta color (for positive/negative correlation) is configurable, as well as the fontsize of the labels and the angles at which the labels are plotted, so you can get fancy if you want and not distribute them evenly … Read more
You pretty much figured out all the pieces, just need to combine them: >>> df.groupby(‘ID’)[[‘Val1′,’Val2′]].corr() Val1 Val2 ID A Val1 1.000000 0.500000 Val2 0.500000 1.000000 B Val1 1.000000 0.385727 Val2 0.385727 1.000000 In your case, printing out a 2×2 for each ID is excessively verbose. I don’t see an option to print a scalar correlation … Read more
As Jeff mentioned in the comments, the problem resulted from my columns having the object dtype. For future reference, even if the object looks numeric, check the dtype and make sure it is numeric (e.g. do foo.astype(float)) before computing the correlation matrix.