Graphing/visualization - Data Analysis with Python 3 and Pandas
Welcome to part 2 of the data analysis with Python and Pandas tutorials, where we're learning about the prices of Avocados at the moment. Soon, we'll find a new dataset, but let's learn a few more things with this one. Where we left off, we were graphing the price from Albany over time, but it was quite messy. Here's a recap:
import pandas as pd
df = pd.read_csv("datasets/avocado.csv")
albany_df = df[df['region']=="Albany"]
albany_df.set_index("Date", inplace=True)
albany_df["AveragePrice"].plot()
So dates are funky types of data, since they are strings, but also have order, at least to us. When it comes to dates, we have to help computers out a bit. Luckily for us, Pandas comes built in with ways to handle for dates. First, we need to convert the date column to datetime objects:
df = pd.read_csv("datasets/avocado.csv")
df['Date'] = pd.to_datetime(df['Date'])
albany_df = df[df['region']=="Albany"]
albany_df.set_index("Date", inplace=True)
albany_df["AveragePrice"].plot()
Alright, the formatting looks better in terms of axis, but that graph is pretty wild! Could we settle it down a bit? We could smooth the data with a rolling average.
To do this, let's make a new column, and apply some smoothing:
albany_df["AveragePrice"].rolling(25).mean().plot()
Hmm, so what happened? Pandas understands that a date is a date, and to sort the X axis, but I am now wondering if the dataframe itself is sorted. If it's not, that would seriously screw up our moving average calculations. This data may be indexed by date, but is it sorted? Let's see.
albany_df.sort_index(inplace=True)
What's this warning above? Should we be worried? Basically, all it's telling us is that we're doing operations on a copy of a slice of a dataframe, and to watch out because we might not be modifying what we were hoping to modify (like the main df). In this case, we're not trying to work with the main dataframe, so I think this warning is just plain annoying, but whatever. It's just a warning, not an error.
albany_df["AveragePrice"].rolling(25).mean().plot()
And there we have it! A more useful summary of avocado prices for Albany over the years.
Visualizations are cool, but what if we want to save our new, smoother, data like above? We can give it a new column in our dataframe:
albany_df["price25ma"] = albany_df["AveragePrice"].rolling(25).mean()
albany_df.head()
Perfect example of why tail is useful sometimes...
albany_df.tail()
That warning sure is annoying though isn't it. What could we do? A common idea is to silence it all of the warnings. What if we instead...
albany_df = df.copy()[df['region']=="Albany"]
albany_df.set_index('Date', inplace=True)
albany_df["price25ma"] = albany_df["AveragePrice"].rolling(25).mean()
This way, we are explicit. Pandas is happy, we're happy. You can ignore pandas warnings, but I would strongly advise against silencing them. Now you know you can do it, but I am not going to show you how :)
Another subtle thing you might have glossed over is the requirement for us to sort things how we intend before we start performing operations and calcs. Many times, you wont be visualizing columns before you make them. You may actually never visualize them. Imagine if we wrote the above code before we sorted by date, basically just assuming things were ordered by date. We'd have produced bad data.
Then imagine maybe we're doing some machine learning or other statistical analysis on that data. Well, chances are, our MA column is not only fairly useless, it's also being informed often of future data!
It's very easy to make mistakes like this. Check your code early and check it often through printing it out and visualizing it where possible!
Alright, we want more cool stuff, what else can we do?
Let's graph prices in the different regions. We hard-coded the Albany region, but hmm, we don't know all of the regions. What do we do?! If we every just wanted to get a "list" from one of our columns, we could reference just that column, like:
df['region']
Then convert to array with:
df['region'].values
Could go to list like:
df['region'].values.tolist()
# and then maybe get the uniques with:
print(set(df['region'].values.tolist()))
So, very quickly you can take your pandas data and get it out into array/list form and use your own knowledge of python. Or, you could also just use some Pandas method. Just know, if you're trying to do it, it probably has a method!
df['region'].unique()
That was quick and painless!
graph_df = pd.DataFrame()
for region in df['region'].unique()[:16]:
print(region)
region_df = df.copy()[df['region']==region]
region_df.set_index('Date', inplace=True)
region_df.sort_index(inplace=True)
region_df[f"{region}_price25ma"] = region_df["AveragePrice"].rolling(25).mean()
if graph_df.empty:
graph_df = region_df[[f"{region}_price25ma"]] # note the double square brackets!
else:
graph_df = graph_df.join(region_df[f"{region}_price25ma"])
I set the limit to 16 just to show we're getting bogged down. This one really tripped me up. I couldn't quite figure out what was going on. Things were taking exponentially longer and longer, then memory was getting exhausted. That's not what I know and love with Pandas, so what gives? Upon some digging, we find that hmm, dates are still getting duplicated. For example:
graph_df.tail()
Each row should be a separate date, but it's not. Through some debugging, we can discover what's happening, which actually informs us to why our previous data looked so ugly too.
Our avocados have multiple prices: Organic and Conventional! So, let's pick one. I'll go with organic. So we'll just start over pretty much.
import pandas as pd
df = pd.read_csv("datasets/avocado.csv")
df = df.copy()[df['type']=='organic']
df["Date"] = pd.to_datetime(df["Date"])
df.sort_values(by="Date", ascending=True, inplace=True)
df.head()
Now, let's just copy and paste the code from above, minus the print:
graph_df = pd.DataFrame()
for region in df['region'].unique():
region_df = df.copy()[df['region']==region]
region_df.set_index('Date', inplace=True)
region_df.sort_index(inplace=True)
region_df[f"{region}_price25ma"] = region_df["AveragePrice"].rolling(25).mean()
if graph_df.empty:
graph_df = region_df[[f"{region}_price25ma"]] # note the double square brackets! (so df rather than series)
else:
graph_df = graph_df.join(region_df[f"{region}_price25ma"])
graph_df.tail()
Now it's quick! Awesome!
Let's graph!
graph_df.plot(figsize=(8,5), legend=False)
Lots more we could poke around with here, but, in the next tutorial, we'll be visiting a new dataset with new challenges.
-
Introduction - Data Analysis with Python 3 and Pandas
-
Graphing/visualization - Data Analysis with Python 3 and Pandas
-
Groupby - Data Analysis with Python 3 and Pandas
-
Visualizing Correlation Table - Data Analysis with Python 3 and Pandas
-
Combining multiple datasets - Data Analysis with Python 3 and Pandas
-
Machine Learning with Scikit-learn - Data Analysis with Python 3 and Pandas
