pandas columns correlation with statistical significance

To calculate all the p-values at once, you can use calculate_pvalues function (code below): df = pd.DataFrame({‘A’:[1,2,3], ‘B’:[2,5,3], ‘C’:[5,2,1], ‘D’:[‘text’,2,3] }) calculate_pvalues(df) The output is similar to the corr() (but with p-values): A B C A 0 0.7877 0.1789 B 0.7877 0 0.6088 C 0.1789 0.6088 0 Details: Column D is automatically ignored as it … Read more

What is the difference between numpy.linalg.lstsq and scipy.linalg.lstsq?

If I read the source code right (Numpy 1.8.2, Scipy 0.14.1 ), numpy.linalg.lstsq() uses the LAPACK routine xGELSD and scipy.linalg.lstsq() usesxGELSS. The LAPACK Manual Sec. 2.4 states The subroutine xGELSD is significantly faster than its older counterpart xGELSS, especially for large problems, but may require somewhat more workspace depending on the matrix dimensions. That means … Read more

scipy, lognormal distribution – parameters

The distributions in scipy are coded in a generic way wrt two parameter location and scale so that location is the parameter (loc) which shifts the distribution to the left or right, while scale is the parameter which compresses or stretches the distribution. For the two parameter lognormal distribution, the “mean” and “std dev” correspond … Read more

How to force zero interception in linear regression?

As @AbhranilDas mentioned, just use a linear method. There’s no need for a non-linear solver like scipy.optimize.lstsq. Typically, you’d use numpy.polyfit to fit a line to your data, but in this case you’ll need to do use numpy.linalg.lstsq directly, as you want to set the intercept to zero. As a quick example: import numpy as … Read more

Fit sigmoid function (“S” shape curve) to data using Python

After great help from @Brenlla the code was modified to: def sigmoid(x, L ,x0, k, b): y = L / (1 + np.exp(-k*(x-x0))) + b return (y) p0 = [max(ydata), np.median(xdata),1,min(ydata)] # this is an mandatory initial guess popt, pcov = curve_fit(sigmoid, xdata, ydata,p0, method=’dogbox’) The parameters optimized are L, x0, k, b, who are … Read more

savefig without frames, axes, only content

EDIT Changed aspect=”normal to aspect=”auto’ since that changed in more recent versions of matplotlib (thanks to @Luke19). Assuming : import matplotlib.pyplot as plt To make a figure without the frame : fig = plt.figure(frameon=False) fig.set_size_inches(w,h) To make the content fill the whole figure ax = plt.Axes(fig, [0., 0., 1., 1.]) ax.set_axis_off() fig.add_axes(ax) Then draw your … Read more