When is a New Scenario a New Scenario?

On March 14, I published a blog on “Why Version Control Matters to Enrolment Forecasters”. We talked about why versioning your technical architecture matters. Today we’re going to discuss why versions of your scenarios are important.

What is a Scenario?

One of the common debates when building an enrolment forecast is “what is a scenario”?

Well, it depends.

But here’s some perspectives to consider:

• A new scenario is defined by the intakes being different than another scenario.
• A new scenario is defined by one or more of the input variables (intakes, student behaviour, mix of programs, etc.).
• A new scenario is defined by its name (eg: Senate Plan 2023).
• A new scenario is defined by the user saying it is new, even if it is actually identical to another scenario (eg: Andrew’s scenario versus Pat’s scenario).

I’ve found the first case is the most common, but have also seen a mix of all of these. I’d argue the second is the most sound and most scalable.

Conversely, here are some questions to ponder over:

• Is it a new scenario if the system updates it based on a pre-determined time schedule (eg: maybe your forecast updates weekly, even if user-controlled variables don’t change)?
• Is it a new scenario if the user updates it?

In both of these cases, I would argue no. It is an iteration of the same scenario.

When we think about scenario versions, it’s helpful to break them down into different buckets:

1. Changes in user-controlled parameters.
2. Changes in automatically updated parameters (non-user-controlled) parameters.
3. Changes in the underlying model.

Changes in User-Controlled Parameters

These changes are ones your users can control. Ideally they’re through an easy-to-use web interface or application, but they could also be through spreadsheets or manipulation of your base data.

For this, you see examples like:

• “Arts wants to increase their intake by 5%”.
• “Our new investments in student success programs should help increase our retention rates in future years”.
• “The government has changed the rules and we can now increase tuition rates”.

These changes could include new intakes, overriding assumptions around retention rates, full-time/part-time proportions, and perhaps even tuition rates etc.

Automatic Changes

These changes are ones that your users cannot control. They become changes you have to take into account, whether you intended them or not.

The most common cases here relate to the passage of time. With each week / month / term / year that passes, we get new information such as:

• Course enrolments.
• New admits.
• Headcount to FTE.
• Retention rates.
• Full-time/part-time proportions.
• Gender proportions.
• Transition rates between levels or programs.

Of course, all of these can have a real impact. COVID-19 shows us that we can’t always rely on past history to guide us. For this reason, we recommend making sure that your system allows your users to override these assumptions when required.

Underlying Model Changes

Perhaps you’ve found some new information that makes your forecast 10% more accurate (you’re measuring your accuracy, right?).

Once you’ve tested your changes and validated that they work as expected, you move your model to production.

At that point, all your existing scenarios should be run again for the current cycle. This will allow them to benefit from the changes you’ve made.

What Now?

Now that we have some understanding of what makes for different scenarios and different iterations of the same scenarios, it’s worth considering what comes next.

All of your results should be saved somewhere. Often this should be a database, but there are alternatives.

You should have the ability to compare your results from every iteration of every scenario to what actually occured. It’s possible, though unlikely, that “Andrew’s test scenario, iteration 6” actually was the most accurate. If this happened, it would be:

1. good to know and
2. good to try to understand why.

You can’t do this if you don’t track each iteration.

Ideally, these are tracked at the same granular detail you do your forecasting work in. Maybe that’s by campus, national status, faculty, program, and level. If you can track your results in this relatively granular fashion, it will be possible to see if some areas were more accurate than others. Science could have ended up way above prediction because Applied Science overenrolled by 10%.

You can also learn a lot from the meta data of your scenarios. Knowing who is running them, when they’re being run, and what the changes are relative to other scenarios or previous iterations can help you understand how to better serve your audience, and where some of their challenges lie.

At the end of the day, you as the institutional analytics professional are in the business of helping the institution make better decisions informed by data. The more time you can spend on that, relative to preparing data and slide decks in static form, the more valuable you are to your institution.

Summary

While the definition of a new scenario can vary, I find a definition that takes into accounts differences in inputs is helpful. Scenario versions are based on user-controlled parameters, automatically updated parameters, and changes in your data model. Tracking the outputs of your scenarios allows you to go back and reflect on the results, seeing where there are opportunities for improvement.