Forecasting Time Series Data With Prophet IV

Originally published as Forecasting Time-Series data with Prophet - Part 4 at pythondata.com.

Introduction

This is the fourth in a series of posts about using Forecasting Time Series data with Prophet. The other parts can be found here:

In those previous posts, I looked at forecasting monthly sales data 24 months into the future using some example sales data that you can find here.

In this post, I want to look at the output of Prophet to see how we can apply some metrics to measure ‘accuracy’. When we start looking at ‘accuracy’ of forecasts, we can really do a whole lot of harm by using the wrong metrics and the wrong data to measure accuracy. That said, its good practice to always try to compare your predicted values with your actual values to see how well or poorly your model(s) are performing.

For the purposes of this post, I’m going to expand on the data in the previous posts. For this post we are using fbprophet version 0.2.1. Also – we’ll need scikit-learn and scipy installed for looking at some metrics.

Note: While I’m using Prophet to generate the models, these metrics and tests for accuracy can be used with just about any modeling approach.

Since the majority of the work has been covered in Part 3, I’m going to skip down to the metrics section…

Import necessary libraries

import pandas as pd
import numpy as np
from fbprophet import Prophet
import matplotlib.pyplot as plt
from sklearn.metrics import mean_squared_error, r2_score, mean_absolute_error
 
plt.rcParams['figure.figsize']=(20,10)
plt.style.use('ggplot')

4.5s

Python

Matplotlib must be manually registered with Pandas due to a conflict between Prophet and Pandas.

pd.plotting.register_matplotlib_converters()

0.2s

Python

Read in the data

Read the data in from the retail sales CSV file in the examples folder then set the index to the date column. We are also parsing dates in the data file.

retail_sales.csv

sales_df = pd.read_csv(retail_sales.csv
, index_col='date', parse_dates=True)

0.3s

Python

sales_df.head()

0.3s

Python

date	sales
2009-10-01	338630
2009-11-01	339386
2009-12-01	400264
2010-01-01	314640
2010-02-01	311022

5 items

Prepare for Prophet

As explained in previous prophet posts, for prophet to work, we need to change the names of these columns to ds and y.

df = sales_df.reset_index()

0.2s

Python

df.head()

0.3s

Python

	date	sales
0	2009-10-01	338630
1	2009-11-01	339386
2	2009-12-01	400264
3	2010-01-01	314640
4	2010-02-01	311022

5 items

Let's rename the columns as required by fbprophet. Additionally, fbprophet doesn't like the index to be a datetime... it wants to see ds as a non-index column, so we won't set an index differently than the integer index.

df=df.rename(columns={'date':'ds', 'sales':'y'})

0.2s

Python

df.head()

0.2s

Python

	ds	y
0	2009-10-01	338630
1	2009-11-01	339386
2	2009-12-01	400264
3	2010-01-01	314640
4	2010-02-01	311022

5 items

Now's a good time to take a look at your data. Plot the data using pandas' plot function

plt.figure()
df.set_index('ds').y.plot().get_figure()

0.9s

Python

Running Prophet

Now, let's set Prophet up to begin modeling our data using our promotions dataframe as part of the forecast

Note: Since we are using monthly data, you'll see a message from Prophet saying Disabling weekly seasonality. Run prophet with weekly_seasonality=True to override this. This is OK since we are working with monthly data but you can disable it by using weekly_seasonality=True in the instantiation of Prophet.

model = Prophet(weekly_seasonality=True)
model.fit(df);
model.weekly_seasonality

2.6s

Python

True

We've instantiated the model, now we need to build some future dates to forecast into.

future = model.make_future_dataframe(periods=24, freq = 'm')
future.tail()

0.2s

Python

	ds
91	2017-04-30
92	2017-05-31
93	2017-06-30
94	2017-07-31
95	2017-08-31

5 items

To forecast this future data, we need to run it through Prophet's model.

forecast = model.predict(future)

3.3s

Python

The resulting forecast dataframe contains quite a bit of data, but we really only care about a few columns. First, let's look at the full dataframe:

forecast.tail()

1.2s

Python

We really only want to look at yhat, yhat_lower and yhat_upper, so we can do that with:

forecast[['ds', 'yhat', 'yhat_lower', 'yhat_upper']].tail()

0.6s

Python

	ds	yhat	yhat_lower	yhat_upper
91	2017-04-30	477814.8556124669	472384.57979563996	483243.6921667977
92	2017-05-31	476745.63667196187	470984.0140790574	482503.0393795469
93	2017-06-30	475688.35458719084	469461.0635859311	481449.94429493666
94	2017-07-31	479770.6856162995	473377.5419909371	485777.41729538876
95	2017-08-31	455621.6718395891	449243.7110760838	462451.4513158483

5 items

Plotting Prophet results

Prophet has a plotting mechanism called plot. This plot functionality draws the original data (black dots), the model (blue line) and the error of the forecast (shaded blue area).

model.plot(forecast);

1.0s

Python

Personally, I'm not a fan of this visualization but I'm not going to build my own...you can see how I do that here.

Additionally, Prophet let's us take a at the components of our model, including the holidays. This component plot is an important plot as it lets you see the components of your model including the trend and seasonality (identified in the yearly pane).

model.plot_components(forecast);

2.0s

Python

Now that we have our model, let's take a look at how it compares to our actual values using a few different metrics - R-Squared and Mean Squared Error (MSE).

To do this, we need to build a combined dataframe with yhat from the forecasts and the original y values from the data.

metric_df = forecast.set_index('ds')[['yhat']].join(df.set_index('ds').y).reset_index()

0.4s

Python

metric_df.tail()

0.2s

Python

	ds	yhat
91	2017-04-30	477814.8556124669
92	2017-05-31	476745.63667196187
93	2017-06-30	475688.35458719084
94	2017-07-31	479770.6856162995
95	2017-08-31	455621.6718395891

5 items

You can see from the above, that the last part of the dataframe has NaN for y... that's fine because we are only concerned about checking the forecast values versus the actual values so we can drop these NaN values.

metric_df.dropna(inplace=True)

0.2s

Python

metric_df.tail()

0.3s

Python

	ds	yhat	y
67	2015-05-01	463310.0074438375	462615.0
68	2015-06-01	448897.68864012434	448229.0
69	2015-07-01	453159.7624656404	457710.0
70	2015-08-01	454307.4463012435	456340.0
71	2015-09-01	431775.77250561136	430917.0

5 items

Now let's take a look at our R-Squared value

r2_score(metric_df.y, metric_df.yhat)

0.2s

Python

0.9930150764264917

An r-squared value of 0.99 is amazing (and probably too good to be true, which tells me this data is most likely overfit).

mean_squared_error(metric_df.y, metric_df.yhat)

0.2s

Python

11020348.091546109

That's a large MSE value... and confirms my suspicion that this data is overfit and won't likely hold up well into the future. Remember... for MSE, closer to zero is better.

Now...let's see what the Mean Absolute Error (MAE) looks like.

mean_absolute_error(metric_df.y, metric_df.yhat)

0.2s

Python

2609.1933238624315

Not good. Not good at all. BUT... the purpose of this particular post is to show some usage of R-Squared, MAE and MSE's as metrics and I think we've done that.

I can tell you from experience that part of the problem with this particular data is that its monthly and there aren't that many data points to start with (only 72 data points... not ideal for modeling).

Another approach for metrics

While writing this post, I came across ML Metrics, which provides 17 metrics for Python (python version here).

Let's give it a go and see what these metrics show us.

import ml_metrics as metrics

0.1s

Python

metrics.mae(metric_df.y, metric_df.yhat)

0.2s

Python

2609.1933238624315

Same value for MAE as before... which is a good sign for this new metrics library. Let's take a look at a few more.

Here's the Absolute Error (pointwise...shows the error of each date's predicted value vs actual value)

metrics.ae(metric_df.y, metric_df.yhat)

0.2s

Python

array([1056.4...858.77250561])

Let's look at Root Mean Square Error

metrics.rmse(metric_df.y, metric_df.yhat)

0.2s

Python

3319.690963259398

This new metrics library looks interesting...I'll keep in in my toolbox for future use.

Hopefully this was useful for you to see how to look at some useful metrics for analyzing how good/bad your models are. These approaches can be used for most other modeling techniques other than Facebook Prophet.