<< Click to Display Table of Contents >> Navigation:  Details > Decomposition > Details

Decomposition is done via the DECOMP command in Gekko. This section describes DECOMP in Gekko 3.0, in its most advanced version. DECOMP can basically decompose non-linear expressions and equations, that is, for multipliers or time-changes find how much the change in the independent variables change the dependent variable. These changes can also be tracked through multiple equations (a system of equations).

Consider this example:

 

 

When decomposing, there will be a list of equations to decompose (for instance, eqs e1 and e2), and each of these equations may contain "uncontrolled" (free) lists, for instance a list like #a (that might represent, for instance, age groups 1 to 100 year olds). A "controlled" list could be for instance an equation like y = sum(#a, x[#a]); -- this does not unroll any equations, but just produces the single equation y = x['1'] + x['2'] + ... + y['100'];. To return to the example, both of the equations are defined over the "uncontrolled" list #a, so all in all 2 * 100 = 200 equations are unrolled: y['1'] = x1['1'] + x1['1'][-1] + 5; ... y['100'] = x1['100'] + x1['100'][-1] + 5; x1['1'] = 2 * x2['1']; ...  x1['100'] = 2 * x2['100'];.

Before decomposition, Gekko will unroll such equations, calling an evaluation on each of the two FRML's. For instance, regarding the former of the two, a statement similar to temp = y[#a] - (x1[#a] + x1[#a][-1] + 5); will be issued, where temp is a temporary variable. This statement will be unrolled into 100 equations of the form temp = y['1'] - (x1['1'] + x1['1'][-1] + 5); ... temp = y['100'] - (x1['100'] + x1['100'][-1] + 5);, calculating the difference between the left-hand and right-hand sides of the equations (that is, the 'residuals'). While doing this, Gekko will also remember which variables are part of which equations (this info is stored from session to session, as long as the model file is not changed).

When decomposing, the following takes places. First of all, all equations are treated equal, and left-hand side variables are not treated in any special way. So in a sense, the equations are treated as a number of model equations, and in principle all of these equations could have the following form:

 

In this system of equations, the endogenous variables have to be indicated, so this is the reason why the DECOMP command has an endo section. The number of unrolled equations, and the number of endogenous variables must match, otherwise Gekko will issue an error. But other than that, Gekko does not care if there is simultaneity in the equations. You may also have en endogenous variable appear several times on the left-hand side of equations -- Gekko does not distinguish between left-hand of right-hand sides when decomposing.

In real life, Gekko equations are obviously not guaranteed to be linear, so in order to linearize these, small perturbations are added to the variables, so that numerical derivatives can be computed. In the above system, we might imagine that x1 is augmented by a small value, after which the effect on the n functions is computed. Repeating this for x2, x3, ... xn results in a n x n matrix of effects (the Jacobian matrix). Gekko solves the linearized equations by means of inverting the Jacobian matrix.

For each unrolled equation (representing "uncontrolled" lists in the equations), the following is performed in order to calculate the Jacobian matrix:

The unrolled equations are normalized, so for instance y['1'] = 2 * x1['1'] + 3 * x1['1'][-1] + 5; is transformed into temp = y['1'] - (2 * x1['1'] + 3 * x1['1'][-1] + 5);. The temp variable (timeseries) represents the residuals of the equation, and Gekko takes note of the variables (timeseries) that are appearing in the equation (in this case, the array-subseries y['1'] and x1['1']).

The unrolled equations look like this:

 

Next, the following loop takes place (for all unrolled equations):

 

The difficult part of this is perhaps to understand how lags/leads are treated. Let us assume the unrolled and normalized equation temp = y['1'] - (x1['1'] + x1['1'][-1] + 5);. If we assume that the decomp period is 2020-22, we may try to analyze what happens regarding the x1['1'] variable (which appears with a lag). First, a temp0 series is calculated for 2020-22, we may assume that the value is 0 for each year. Next, for the year 2020, x1['1'] is augmented with a small value eps (for instance 1e-9). Then a temp series is calculated for 2020-22, and the difference temp - temp0 (this is a series of values) tells us the effect of changing x1['1'] in 2020. There are two effects:

1.There will be an effect on temp in 2020 because x1['1'] appears in the equation.

2.There will be an effect on temp in 2021 because x1['1'][-1] appears in the equation.

In 2022, there are no effects from changing x1['1'] in 2020. The first effect will be stored under the name "x1['1']" for the year 2020, whereas the second effect will be stored under the name "x1['1'][-1]" for the year 2021 (the lag can be calculated as -1 = 2020 - 2021). When evaluating the temp expression, Gekko takes notice regarding which variables are called, so it is easy to determine that only y['1'] and x1['1'] are called. So Gekko does not look at the temp expression symbolically (as a string), and therefore an expression like temp = dif(y) - dif(x) will identify the implicit presence of lagged variables y[-1] and x[-1] just as easily as the equivalent expression temp = y - y[-1] - (x - x[-1]).

At this point, we have differentiated the unrolled equations, identifying how changes in the variables (possibly with lags or leads) affect these equations. Next, this knowledge is used for decompositions, either decomposing time-changes or multiplier changes (different scenarios).

This is handled in the following loop:

 

The point of the above code is that two matrices mEndo and mExo are constructed, for instance allowing the system of equations to be simultaneous. If it is not simultaneous, the mEndo matrix is diagonal (or triangular), but simultaneity is allowed. For instance, consider the equations y['1'] = 2 * x1['1'] + 3 * x1['1'][-1] + 5; and x1['1'] = 4 * x2['1'];. These are obviously simultaneous in y['1'] and x1['1'], if these variables are denoted as endogenous. In the general case with many equations, iterative methods or matrix inversion can be used to solve the linear system for y['1'] and x1['1']. After matrix inversion, for each endogeous variable like y['1'] and x1['1'], we now know (for each time period) how y['1'] and x1['1'] these are affected by the exogenous variables. In this case, we know how x2['1'] affects y['1'], and how x2['1'] affects y['1'], including indirect effects flowing through the two equations. These effects are stored in the effect matrix.

So the following equations:

 

can obviously be transformed into these by simple substitution:

 

In e1, the endogenous y[#a] is only affected by the exogenous x2[#a]. This re-arrangement can be done because the equations are linear, but decomposition works for non-linear cases, too. In this case, augmenting x2[#a] with 1 will change y[#a] with 8, and x1[#a] with 4 (and there is an effect from x2[#a][-1] on y[#a], too).

Up to now, we have linearized the equations and removed any simultaneity. The next step is to use these linearized dependencies to decompose how the endogenous variables react to exogenous variables in two scenarios:

•Same period: two scenarios. Can we explain why an endogenous variable changes from a baseline scenario to a shock scenario ("multiplier").

•Different periods: period by period decomposition. Can we explain why an endogenous variable changes from one period to the next?

In the "new" DECOMP, it is possible to show effects more flexibly than before, including the possibility of operating on sets like the age groups #a in a way that resembles a pivot table. To to this, the decomposition effects are basically stored in a table similar to the following:

In the first three lines, equation e1 is decomposed in 2010, and it is seen that the equations depends upon y['a'], x2['1'] and x2['1'][-1] (so it depends upon x2 and not x1, because the decomposition is seen as a system of equations). In the next two line, equation e2 is decomposed, where there are effects from x1['1'] and x2['1']. All this concerns age group '1' (1-year olds). In the next five equations, age group '2' is decomposed. All these effect concern the year 2010.

If the decomposed equations are linear, the contributions in each equation in each year from the different variables will be 0 (remember that the equations are normalized with a temp variable on the left-hand side and all other variables on the right-hand side). Therefore, the sum of the complete value column will be 0, too. So if you define a pivot table from this (with the value field as values), without selecting or filtering any fields, it will just start out showing 0.

The pivoting part of the decomposition is handled separately from the decomposition proper, so that the code does not get mixed up. The pivoting part of decomposition uses a FrameLight C# object, which is a homebrewn simple container to do pivoting. The plan is to use a more modern and speedy approach to this, to do the sorting, filtering, aggregation, etc. Perhaps use Apache Arrow format to store the data, and combine it with a "real" inbuilt C# dataframe like this.

 

Code parts in Decomp.cs

DecompStart(). Starting point of decompositions, collecting the DECOMP options etc. The method calls DecompGetFuncExpressionsAndRecalc(), which in turn Calls DecompEvalGams(). This method uses Program.CallEval() to transform an equation (in the form of a string) into into a corresponding C# Func<>. The latter is then used to compute precedents: that is, which variables that are part of an equation (including its left-hand side if any). In our example, precedents could be for instance y['1'], x1['1'], and x1['1'][-1], corresponding to age group '1' in equation e1. These precedents are store for later use. As long as the model is not changed, these are the same (they are even stored on disk as a so-called protobuffer file to speed up decompositions between Gekko sessions).

The

When something in the Gekko decomp GUI is changed (for instance, a field is dragged), WindowDecomp.RecalcCellsWithNewType() is called. This in turns calls DecompMain(), which performs the decomposition and returns a Gekko Table object (the results of which will be shown in the GUI).

DecompMain() starts out calculating the gradients. For each unrolled equation, it calls DecompLowLevel(). This method creates a DecompData object containing these fields (sub-objects) of type DecompDict: cellsQuo, cellsGradQuo, cellsRef, cellsGradRef, cellsContribD, cellsContribM and cellsContribDRef. The DecompDict object is basically a Dictionary<string, Series>, where key-value pairs of names and values (in the form of timeseries) are stored.

For each precedent variable, the field .cellsQuo is set to the name and the raw timeseries data (untouched levels) before anything is changed ("quo" signals "status quo"). If it is a multiplier analysis, .cellsRef is filled with raw level data from the Ref databank. The field .cellsGradQuo is filled with gradients computed by perturbating the precedent variables with a very small value (corresponding to numerical differentiation). For multiplier analysis, .cellsGradRef also contains gradients, just based on data from the Ref databank.

Next, contributions are calculated. For instance, cellsContribD (contributions to time-differences) is calculated from cellsGradQuo (gradient) multiplied by the time difference in cellsQuo. The others, cellsContribM and cellsContribDRef are calculated in a similar way. After these contributions are calculated, for each period they are put into two matrices called mEndo and mExo. Here, mEndo is a m x m matrix, where m is the number of endogenous variables, and mExo is a m x k matrix, where k is the number of exogenous variables. When computing the matrix effect = inv(mEndo) * mExo, the result is a m x k matrix with endogenous variables in the rows and the exogenous variables in the columns. For each row in this matrix, the columns tell us how much the changes in this particular exogenous variable contributed to the changes in the endogenous variable (with any simultaneity resolved). If there is no simultaneity, the mEndo matrix is diagonal and particularly easy to invert. This calculation is done for each period.

After these contributions have been calculated (and simultaneity removed), the effects are transformed into a suitable table for showing in the Gekko GUI. This takes part in DecompPivotToTable(), which produces a table that can be shown directly in the decomposition GUI window. Inside DecompPivotToTable(), a Gekko FrameLight object is filled with the contributions (from the effect matrix), and this FrameLight object corresponds to the table shown in the beginning of this section. This FrameLight is then aggregated and filtered into a Gekko Table object, depending upon how the users prefers to view the data (for instance which variables are shown in rows and columns, which values are ignored, etc.). As noted above, both the FrameLight object and the aggregation/filtering could possibly be done using external components tailored for dataframes, pivoting etc.