Data Transformation

Do different genres of films receive consistently different ratings from critics? As a first step towards answering this question, we might examine the average (a.k.a. the arithmetic mean) rating for each genre of movie.

Let's visualize genre along the y axis and plot average Rotten Tomatoes ratings along the x axis.

There does appear to be some interesting variation, but looking at the data as an alphabetical list is not very helpful for ranking critical reactions to the genres.

For a tidier picture, let's sort the genres in descending order of average rating. To do so, we will add a sort parameter to the y encoding channel, stating that we wish to sort by the average (op, the aggregate operation) Rotten Tomatoes rating (the field) in descending order.

The sorted plot suggests that critics think highly of documentaries, musicals, westerns, and dramas, but look down upon romantic comedies and horror films... and who doesn't love null movies!?

While averages are a common way to summarize data, they can sometimes mislead. For example, very large or very small values (outliers) might skew the average. To be safe, we can compare the genres according to the median ratings as well.

The median is a point that splits the data evenly, such that half of the values are less than the median and the other half are greater. The median is less sensitive to outliers and so is referred to as a robust statistic. For example, arbitrarily increasing the largest rating value will not cause the median to change.

Let's update our plot to use a median aggregate and sort by those values:

We can see that some of the genres with similar averages have swapped places (films of unknown genre, or null, are now rated highest!), but the overall groups have stayed stable. Horror films continue to get little love from professional film critics.

It's a good idea to stay skeptical when viewing aggregate statistics. So far we've only looked at point estimates. We have not examined how ratings vary within a genre.

Let's visualize the variation among the ratings to add some nuance to our rankings. Here we will encode the inter-quartile range (IQR) for each genre. The IQR is the range in which the middle half of data values reside. A quartile contains 25% of the data values. The inter-quartile range consists of the two middle quartiles, and so contains the middle 50%.

To visualize ranges, we can use the x and x2 encoding channels to indicate the starting and ending points. We use the aggregate functions q1 (the lower quartile boundary) and q3 (the upper quartile boundary) to provide the inter-quartile range. (In case you are wondering, q2 would be the median.)

Now let's ask a completely different question: do box office returns vary by season?

To get an initial answer, let's plot the median U.S. gross revenue by month.

To make this chart, use the timeUnit transform to map release dates to the month of the year. The result is similar to binning, but using meaningful time intervals. Other valid time units include year, quarter, date (numeric day in month), day (day of the week), and hours, as well as compound units such as yearmonth or hoursminutes. See the Altair documentation for a complete list of time units.

Looking at the resulting plot, median movie sales in the U.S. appear to spike around the summer blockbuster season and the end of year holiday period. Of course, people around the world (not just the U.S.) go out to the movies. Does a similar pattern arise for worldwide gross revenue?

Yes!

Think back to our comparison of U.S. gross and worldwide gross. Doesn't worldwide revenue include the U.S.? (Indeed it does.) How might we get a better sense of trends outside the U.S.?

With the calculate transform we can derive new fields. Here we want to subtract U.S. gross from worldwide gross. The calculate transform takes a Vega expression string to define a formula over a single record. Vega expressions use JavaScript syntax. The datum. prefix accesses a field value on the input record.

We can see that seasonal trends hold outside the U.S., but with a more pronounced decline in the non-peak months.

The filter transform creates a new table with a subset of the original data, removing rows that fail to meet a provided predicate test. Similar to the calculate transform, filter predicates are expressed using the Vega expression language.

Below we add a filter to limit our initial scatter plot of IMDB vs. Rotten Tomatoes ratings to only films in the major genre of "Romantic Comedy".

How does the plot change if we filter to view other genres? Edit the filter expression to find out.

Now let's filter to look at films released before 1970.

They seem to score unusually high! Are older films simply better, or is there a selection bias towards more highly-rated older films in this dataset?

We have already seen aggregate transforms such as count and average in the context of encoding channels. We can also specify aggregates separately, as a pre-processing step for other transforms (as in the window transform examples below). The output of an aggregate transform is a new data table with records that contain both the groupby fields and the computed aggregate measures.

Let's recreate our plot of average ratings by genre, but this time using a separate aggregate transform. The output table from the aggregate transform contains 13 rows, one for each genre.

To order the y axis we must include a required aggregate operation in our sorting instructions. Here we use the max operator, which works fine because there is only one output record per genre. We could similarly use the min operator and end up with the same plot.

The window transform performs calculations over sorted groups of data records. Window transforms are quite powerful, supporting tasks such as ranking, lead/lag analysis, cumulative totals, and running sums or averages. Values calculated by a window transform are written back to the input data table as new fields. Window operations include the aggregate operations we've seen earlier, as well as specialized operations such as rank, row_number, lead, and lag. The Vega-Lite documentation lists all valid window operations.

One use case for a window transform is to calculate top-k lists. Let's plot the top 20 directors in terms of total worldwide gross.

We first use a filter transform to remove records for which we don't know the director. Otherwise, the director null would dominate the list! We then apply an aggregate to sum up the worldwide gross for all films, grouped by director. At this point we could plot a sorted bar chart, but we'd end up with hundreds and hundreds of directors. How can we limit the display to the top 20?

The window transform allows us to determine the top directors by calculating their rank order. Within our window transform definition we can sort by gross and use the rank operation to calculate rank scores according to that sort order. We can then add a subsequent filter transform to limit the data to only records with a rank value less than or equal to 20.

We can see that Steven Spielberg has been quite successful in his career! However, showing sums might favor directors who have had longer careers, and so have made more movies and thus more money. What happens if we change the choice of aggregate operation? Who is the most successful director in terms of average or median gross per film? Modify the aggregate transform above!

Earlier in this notebook we looked at histograms, which approximate the probability density function of a set of values. A complementary approach is to look at the cumulative distribution. For example, think of a histogram in which each bin includes not only its own count but also the counts from all previous bins — the result is a running total, with the last bin containing the total number of records. A cumulative chart directly shows us, for a given reference value, how many data values are less than or equal to that reference.

As a concrete example, let's look at the cumulative distribution of films by running time (in minutes). Only a subset of records actually include running time information, so we first filter down to the subset of films for which we have running times. Next, we apply an aggregate to count the number of films per duration (implicitly using "bins" of 1 minute each). We then use a window transform to compute a running total of counts across bins, sorted by increasing running time.

Let's examine the cumulative distribution of film lengths. We can see that films under 110 minutes make up about half of all the films for which we have running times. We see a steady accumulation of films between 90 minutes and 2 hours, after which the distribution begins to taper off. Though rare, the dataset does contain multiple films more than 3 hours long!