We're running Dn:Director v8.1.2 (458) with Match Review.
We have a tiered set of match rules on two seperate sources of products, each level (set) controlled be a seperate process but all working from the same pair of staged data tables.
Depending on decisions made by the users inside each match review level, particular products may go up or down levels for the next pass.
The issue we have is that currently we have to run the process twice, once to retrieve the decisions and determine whether products are promoted/demoted, and then a second time to use these promotions/demotions to establish a refreshed match review for users.
Users do not process a whole level at a time (there might be 5000 products in a level each with any number of possible matches), nor is there any restriction on which levels are dealt with first - the conduct of the matching is entirely down to the users themselves.
Consequently it's not feasible to have a level completed before dealing with the decisions they've made, not to have the levels proccessed in sequence.
Each run of each process takes a fair time, so running each twice within a job is less than ideal.
Is there a means of retrieving the decisions & relationships that are sitting in a match review session without actually running the process that controls it ??
This way we could bring in the decisions & determine the promotion/demotion actions much quicker than by actually running the process.
My feeling (maybe completely wrong !!) is that this is possible using web services, but we've no experience of these and can't find any meaningful documentation on it.
Thanks in advance for anyone's help.
I assume from your mail that decisions are being made externally and imported into Match Review - is that correct?
If you are making decisions within Match Review itself, I am not sure I understand why you need to run the process for a second time?
The decisions are made inside the match review, not externally.
Quick example to hopefully clarify the situation :-
Assume it's a completely new set of matching and data, so all review sessions are empty.
A - process runs for level 1 and suggests 6 different possibles against each of 3 'main' products, these going in to 3 match review groups.
B - within match review users do a match in group 1 , a no-match in group 2, and leave group 3 alone (so they remain as possibles).
The outcome we want to have is that group 1 (match) goes through as a match, group 2 (no-match) never appears in level 1 again but does appear in level 2, and group 3 (not actioned) remains in level 1.
We tried to accomplish this by re-running the process, but the no-matches reappeared at the levels they originally appeared at.
To resolve it, we run the process to get the decisions & mark up the group 2 ones as demoted (thus prevented from appearing at level 1).
Re-running the process after this now throws the product in group 2 down in to the next level as it's excluded from level 1.
Problem we have is that we're looking for the process to act on the decisions present before it offers suggestions.
We tried putting everything in one big advanced match, but it took absolutely ages to run & made it really awkward for 10 users to go in to the single review all at the same time.
What are the levels? Are you using a customized decision workflow?
Normally the behaviour you want is what you will get so it is still unclear why the process is rerun. When you say 'goes through as a match' does this mean that you need to flush the data out of the process at some point?
When you say 'appears in level 1' where do you mean? When users search?
I think we would need to see the configuration of your process and your customdecisions.xml in order to understand the implementation.
In general in Match Review, a decision is made against a relationship (a pair of records) and 'sticks' on the same pair of records on subsequent runs based on a configurable decision key - i.e. as long as the attributes in the configured decision key in both records stay the same, the relationship is re-applied.
The decisions made change the match groups immediately within the review application; this is so that if users also want to make merged output decisions on a match group they can do so without match having to rerun.
There should be no need to rerun match in order to make the decisions effective, except if you want to consume the full match groups and/or merged output downstream of match (for example in a target application).
I am perhaps starting to understand this. You are using two different processes matching the same data. Are the decisions from one process fed into the next? What is the order of execution?
It is not very clear to me why the processes are separate. In theory there is a performance cost rather than gain in doing the matching twice. Are they separated for access (permission) reasons? (In this case, I think Case Management would have been a better choice as the review application.)
There's no customisation on the workflow or decisions, but we do have processing based on the relationship decisions to help the users.
Within a review there might be say 3000 groups.
Users might deal with the 200 'easy' groups and leave the other 2800 for another time.
A 'main' product might have some 50 suggestions, so to save the users putting match against the single 'correct' entry AND 'no match' against the other 49 suggestions, we pull in the relationships, look at each review group as a whole, and assign an 'action' based on the number of pairs in the review group marked as match (1 match & 0 no matches) or no-match (1 no match & 0 match) - anything else is left as undeclared.
The main product in a match pair goes in the destination database (after peer review, which may revoke the decision) and do not appear in matching again - we do not want to use merged review.
The main product in a no-match pair is excluded from all runs of this particular processor, but is made available to the next stage of matching that has 'looser' rules than the first.
We can't use bulk review updates because only a sample of the review will have been completed at any one time.
So the situation remains that the processor needs the results of the review session to identify the match/no-match pairs as described above BEFORE it re-stablishes the review session, this time without the matches (committed in to the destination database) and the no-matches (pushed down in to a 'looser rule' process.
Without being able to retrieve the review relationship decisions, the only way we've been able to achieve this is to run the process twice.
OK so you have a significantly customized configuration where most of the actual application of decisions is based on logic in the groups and occurs outside of Match Review, and you have two processes with separate reviews on the same data.
Sounds like the reason you are using two processes is that you don't want too much noise in the first review, and coupled with that a concern over applying looser matching rules to all records (as opposed to only those that could not easily be matched).
Within this setup, I'm fairly sure there is no way to avoid running the process again as you are post-processing the output data from match and that is only written when the process runs.
Sounds like there is room for some matching accuracy improvements, perhaps aided by product data standardization and matching. I would have thought you could auto-match more of the 'easy' pile of records and keep to one process.
Aside from that I think we will have to ponder if there is another solution to the problem... but have a better understanding of it now. The 'if user matches one, discount the rest' use case is interesting.
A sample DXI would still be enormously helpful to help us understand the configuration better since it is far from standard.
We think you could achieve what you want in a single match process... but doing the 'status' checking of a given main product record differently.
It would likely be a fairly significant change to what you have now, and would need proving/testing.
Brief description of the idea:
- Use a single match process with match rules in two tiers. Match rules in the Level 1 tier are only applied to records of status A ('Main Review'). Match rules in the level 2 tier are only applied to records of status B ('Deep Review'). Use an 'Is Value' comparison to apply the condition (if you do not have this from a solution installation, it can be provided as an extension).
- Records always start in status A (Main Revoew). However, if there is a 'No Match' decision made to a main record in the review it is moved to status B. The moving to status B occurs each time the process runs and uses the output of matching - the relationships output run through a Group and Merge processor to deduplicate it, making sure that any No Match decision is always retained. The output of matching is used to update a status tracking table for the record.
- Before the match process, perform a Lookup and Return against the status tracking table to work out the status of a given main record and tag the record with this status. This means that on subsequent run, a record may be 'demoted' to status B.
A wrinkle is how to maintain the status table correctly since if a record is demoted to status B it will no longer match on the previous match rule. The logic here will need to be either:
1. 'Update' (primary key replace) an external database table with the status of each record when it is either demoted from the first review or promoted from the second.
2. Append, with a date/time and always take the latest status record for a given main record.
Certainly veering into consultancy territory but pretty sure it is possible.
Auto-matching is 100% banned I'm afraid (high level steering group decision), and would only help a little anyway (maybe 100 out of the 3000 main products) - the fields used for matching aren't very close in the vast majority of cases, so auto-match is likely to cause more problems than it solves.
There's actually 6 'levels' in the full cycle (level = process), with the 'suggestion' rules getting looser & looser as we go down - the bottom level is almost (but not quite) a 'show me everything', and leads to an awful lot of suggestions for each main product.
We tried it all in one big advanced match and had lots of trouble, not least the run time - the double pass runs of all 6 take about an hour but that could be halved if we could get at the reviw sessions.
Is there no way that we can retrieve the data in the review session outside of running the owning processes ??
The data must be stored somewhere, and we only want to pull it in to staged data for reading, never writing.
Using one match process will be quicker as long as you avoid applying the looser match rules where you don't have to.
Attempting to look up into the internal tables, which are in any case keyed using hash keys, is not the way to go, no.
Although taking your comments in to account, any chance you could let me know where the review data is actually held so I can take a look at it ??
May not be able to do anything with it, but the small amount of data we want from the review session is visible inside the user's match review screens, so must be out there somewhere.
I think we have an easier way. There is a trigger mechanism that can be called when a decision is made. The trigger logic could write the required decision data (ID and status) to a db table which could then be looked up by process 2... meaning process 2 would still need to be re-run but process 1 would not.
We can provide details on the trigger mechanism if this is the path you go down.
There is no supported method to lookup decisions in a review. You should not attempt this.
The trigger mechanism is a definite possibility, however. I would also recommend considering moving to a single match process, but applying rules selectively to records based on their status.