You can use the Series method str.startswith (which takes a regex): In [11]: s = pd.Series([‘aa’, ‘ab’, ‘ca’, np.nan]) In [12]: s.str.startswith(‘a’, na=False) Out[12]: 0 True 1 True 2 False 3 False dtype: bool You can also do the same with str.contains (using a regex): In [13]: s.str.contains(‘^a’, na=False) Out[13]: 0 True 1 True 2 … Read more
Here’s one way to do it in Pandas-way You could groupby on Auction_ID and take rank() on Bid_Price with ascending=False In [68]: df[‘Auction_Rank’] = df.groupby(‘Auction_ID’)[‘Bid_Price’].rank(ascending=False) In [69]: df Out[69]: Auction_ID Bid_Price Auction_Rank 0 123 9 1 1 123 7 2 2 123 6 3 3 123 2 4 4 124 3 1 5 124 2 … Read more
Based on Haipeng Su’s answer, I added a function that does this: dataframe = dataframe.applymap(lambda x: x.encode(‘unicode_escape’). decode(‘utf-8’) if isinstance(x, str) else x) Basically, it escapes the unicode characters if they exist. It worked and I can now write to Excel spreadsheets again!
I tried everything you mentioned in a new environment using conda and I had another issue related to the version of ipywidgets (a bug found in Github with comments saying that got solved after using last version). I solved the problem I had installing last version of ipywidgets. Here is my process: Create a new … Read more
this is a workaround spark_df.limit(5).toPandas().head() although, I do not know the computational burden of this query. I am thinking limit() is not expensive. corrections welcome.
Note for pandas versions < 0.20.0: I will be using df.melt(…) for my examples, but you will need to use pd.melt(df, …) instead. Documentation references: Most of the solutions here would be used with melt, so to know the method melt, see the documentation explanation. Unpivot a DataFrame from wide to long format, optionally leaving … Read more
You can mark their attributes as dynamically generated using generated-members option. E.g. for pandas: generated-members=pandas.*
With a multi-index we have to specify the column using a tuple in order to drop a specific column, or specify the level to drop all columns with that key on that index level. Instead of saying drop column ‘c’ say drop (‘a’,’c’) as shown below: df.drop((‘a’, ‘c’), axis = 1, inplace = True) Or … Read more
Pandas already has a built-in method to convert the table on the web to a dataframe: table = soup.find_all(‘table’) df = pd.read_html(str(table))[0]