There are a few things I cannot understand; I'm assuming Transaction and Amount are fact columns. Then how do they have different values for the same values of Flag and Display ??
Anyway, sub total can be achieved by going to edit view for the table and from columns and measures area under the Layout pane click on the summation icon and select After. This will automatically display a grand total value at the bottom of each fact column.
Now, to get the final result, say Total Transaction you can add a dummy column and edit its formula.
Total Transaction = trans_subtotal1 - trans_subtotal2 and
Total Amount = amt_subtotal1 - amt_subtotal2 + amt_subtotal3
Now if we could say that trans_subtotal1 is the total of transaction value when Flag = 'C' or something like that then you can get this value of subtotal1 by using the formula FILTER(SUM("Subject Area"."Folder".Transaction Column") USING (DESCRIPTOR_IDOF("Subject Area"."Folder"."Flag column") = 'C')). Similarly we can obtain subtotal2 and subtract them to get the total transaction value. But I am sure if we can do this because for the same values of Flag and Display, Transaction has different values. But by using the above formula, the values we obtain for subtotals 2 and 3 will be the same. So calculating Total Amount would be a problem.
The above formula is obtained by going to edit formula and selecting the filter option at the bottom of the edit dialog box.
Hope I made my point clear