# Forecasting Time Series Data With Prophet IV

Originally published as *Forecasting Time-Series data with Prophet - Part 4* at pythondata.com.

## Introduction

This is the fourth in a series of posts about using Forecasting Time Series data with Prophet. The other parts can be found here:

In those previous posts, I looked at forecasting monthly sales data 24 months into the future using some example sales data that you can find here.

In this post, I want to look at the output of Prophet to see how we can apply some metrics to measure ‘accuracy’. When we start looking at ‘accuracy’ of forecasts, we can really do a whole lot of harm by using the wrong metrics and the wrong data to measure accuracy. That said, its good practice to always try to compare your predicted values with your actual values to see how well or poorly your model(s) are performing.

For the purposes of this post, I’m going to expand on the data in the previous posts. For this post we are using `fbprophet`

version 0.2.1. Also – we’ll need `scikit-learn`

and `scipy`

installed for looking at some metrics.

Note: While I’m using Prophet to generate the models, these metrics and tests for accuracy can be used with just about any modeling approach.

Since the majority of the work has been covered in Part 3, I’m going to skip down to the metrics section…

## Import necessary libraries

import pandas as pd import numpy as np from fbprophet import Prophet import matplotlib.pyplot as plt from sklearn.metrics import mean_squared_error, r2_score, mean_absolute_error plt.rcParams['figure.figsize']=(20,10) plt.style.use('ggplot')

Matplotlib must be manually registered with Pandas due to a conflict between Prophet and Pandas.

pd.plotting.register_matplotlib_converters()

## Read in the data

Read the data in from the retail sales CSV file in the examples folder then set the index to the `date`

column. We are also parsing dates in the data file.

sales_df = pd.read_csv(retail_sales.csv, index_col='date', parse_dates=True)

sales_df.head()

date | sales |
---|---|

2009-10-01 | 338630 |

2009-11-01 | 339386 |

2009-12-01 | 400264 |

2010-01-01 | 314640 |

2010-02-01 | 311022 |

## Prepare for Prophet

As explained in previous prophet posts, for prophet to work, we need to change the names of these columns to `ds`

and `y`

.

df = sales_df.reset_index()

df.head()

date | sales | |
---|---|---|

0 | 2009-10-01 | 338630 |

1 | 2009-11-01 | 339386 |

2 | 2009-12-01 | 400264 |

3 | 2010-01-01 | 314640 |

4 | 2010-02-01 | 311022 |

Let's rename the columns as required by `fbprophet`

. Additionally, `fbprophet`

doesn't like the index to be a `datetime`

... it wants to see `ds`

as a non-index column, so we won't set an index differently than the integer index.

df=df.rename(columns={'date':'ds', 'sales':'y'})

df.head()

ds | y | |
---|---|---|

0 | 2009-10-01 | 338630 |

1 | 2009-11-01 | 339386 |

2 | 2009-12-01 | 400264 |

3 | 2010-01-01 | 314640 |

4 | 2010-02-01 | 311022 |

Now's a good time to take a look at your data. Plot the data using pandas' `plot`

function

plt.figure() df.set_index('ds').y.plot().get_figure()

## Running Prophet

Now, let's set Prophet up to begin modeling our data using our `promotions`

dataframe as part of the forecast

Note: Since we are using monthly data, you'll see a message from Prophet saying `Disabling weekly seasonality. Run prophet with weekly_seasonality=True to override this.`

This is OK since we are working with monthly data but you can disable it by using `weekly_seasonality=True`

in the instantiation of Prophet.

model = Prophet(weekly_seasonality=True) model.fit(df); model.weekly_seasonality

We've instantiated the model, now we need to build some future dates to forecast into.

future = model.make_future_dataframe(periods=24, freq = 'm') future.tail()

ds | |
---|---|

91 | 2017-04-30 |

92 | 2017-05-31 |

93 | 2017-06-30 |

94 | 2017-07-31 |

95 | 2017-08-31 |

To forecast this future data, we need to run it through Prophet's model.

forecast = model.predict(future)

The resulting forecast dataframe contains quite a bit of data, but we really only care about a few columns. First, let's look at the full dataframe:

forecast.tail()

We really only want to look at `yhat`

, `yhat_lower`

and `yhat_upper`

, so we can do that with:

forecast[['ds', 'yhat', 'yhat_lower', 'yhat_upper']].tail()

ds | yhat | yhat_lower | yhat_upper | |
---|---|---|---|---|

91 | 2017-04-30 | 477814.8556124669 | 472384.57979563996 | 483243.6921667977 |

92 | 2017-05-31 | 476745.63667196187 | 470984.0140790574 | 482503.0393795469 |

93 | 2017-06-30 | 475688.35458719084 | 469461.0635859311 | 481449.94429493666 |

94 | 2017-07-31 | 479770.6856162995 | 473377.5419909371 | 485777.41729538876 |

95 | 2017-08-31 | 455621.6718395891 | 449243.7110760838 | 462451.4513158483 |

## Plotting Prophet results

Prophet has a plotting mechanism called `plot`

. This plot functionality draws the original data (black dots), the model (blue line) and the error of the forecast (shaded blue area).

model.plot(forecast);

Personally, I'm not a fan of this visualization but I'm not going to build my own...you can see how I do that here.

Additionally, Prophet let's us take a at the components of our model, including the holidays. This component plot is an important plot as it lets you see the components of your model including the trend and seasonality (identified in the `yearly`

pane).

model.plot_components(forecast);

Now that we have our model, let's take a look at how it compares to our actual values using a few different metrics - R-Squared and Mean Squared Error (MSE).

To do this, we need to build a combined dataframe with `yhat`

from the forecasts and the original `y`

values from the data.

metric_df = forecast.set_index('ds')[['yhat']].join(df.set_index('ds').y).reset_index()

metric_df.tail()

ds | yhat | y | |
---|---|---|---|

91 | 2017-04-30 | 477814.8556124669 | |

92 | 2017-05-31 | 476745.63667196187 | |

93 | 2017-06-30 | 475688.35458719084 | |

94 | 2017-07-31 | 479770.6856162995 | |

95 | 2017-08-31 | 455621.6718395891 |

You can see from the above, that the last part of the dataframe has `NaN`

for `y`

... that's fine because we are only concerned about checking the forecast values versus the actual values so we can drop these `NaN`

values.

metric_df.dropna(inplace=True)

metric_df.tail()

ds | yhat | y | |
---|---|---|---|

67 | 2015-05-01 | 463310.0074438375 | 462615.0 |

68 | 2015-06-01 | 448897.68864012434 | 448229.0 |

69 | 2015-07-01 | 453159.7624656404 | 457710.0 |

70 | 2015-08-01 | 454307.4463012435 | 456340.0 |

71 | 2015-09-01 | 431775.77250561136 | 430917.0 |

Now let's take a look at our R-Squared value

r2_score(metric_df.y, metric_df.yhat)

An r-squared value of 0.99 is amazing (and probably too good to be true, which tells me this data is most likely overfit).

mean_squared_error(metric_df.y, metric_df.yhat)

That's a large MSE value... and confirms my suspicion that this data is overfit and won't likely hold up well into the future. Remember... for MSE, closer to zero is better.

Now...let's see what the Mean Absolute Error (MAE) looks like.

mean_absolute_error(metric_df.y, metric_df.yhat)

Not good. Not good at all. BUT... the purpose of this particular post is to show some usage of R-Squared, MAE and MSE's as metrics and I think we've done that.

I can tell you from experience that part of the problem with this particular data is that its monthly and there aren't that many data points to start with (only 72 data points... not ideal for modeling).

## Another approach for metrics

While writing this post, I came across ML Metrics, which provides 17 metrics for Python (python version here).

Let's give it a go and see what these metrics show us.

import ml_metrics as metrics

metrics.mae(metric_df.y, metric_df.yhat)

Same value for MAE as before... which is a good sign for this new metrics library. Let's take a look at a few more.

Here's the Absolute Error (pointwise...shows the error of each date's predicted value vs actual value)

metrics.ae(metric_df.y, metric_df.yhat)

Let's look at Root Mean Square Error

metrics.rmse(metric_df.y, metric_df.yhat)

This new metrics library looks interesting...I'll keep in in my toolbox for future use.

Hopefully this was useful for you to see how to look at some useful metrics for analyzing how good/bad your models are. These approaches can be used for most other modeling techniques other than Facebook Prophet.