Exiting a steps with OnExecReadFileError
Hello folks,
So when I encounter the above condition (OnExecReadFileError), I want to be able to exit the given test step, using "Events", IMMEDIATELY and be able to RETURN to the calling step. The reason I need to do this is that in most cases, this step is really a "readFile" step and has an RM attached to it. Right now, all I have at my disposal is "ExitProcedure", "AbortStep", "AbortExecution", or "AbortTest".
When I try "ExitProocedure", this actually allows the evaluation of the RM because I believe this is a deferred action. If I try "AbortStep", the RM is ignored but the testcase\procedure continues - I also tried adding "ExitProcedure" after this but the Abort short-circuits it so it never gets executed. If I try "AbortExecution\AbortTest", both of these punch out of the step\procedure\testcase and avoid the RM but they do not allow a return to the CALLING step which is where I need to go because it is there where I trap the error so I can send myself a report as to what happened.
So I need a new action called "EJECT" or "RETURN" that stops exection immediately and returns to what got it there.
Thoughts and\or ideas?
I'm a little confused as to what you mean by the "calling step". I assume that what you want to do is to skip response mapping for the current step and then immediately exit the procedure.
I believe you will get the behavior you want by adding the following actions to the OnExecReadFileError event -- in the following order:
ExitProcedure
AbortStep
I agree that it is a little counter-intuitive that you have to put these in this order. But because ExitProcedure is a deferred action, you need to queue it up BEFORE you invoke AbortStep -- which stops all further mapping and analysis. So, in this case, if there is a readFile error, it will abort the step immediately, then proceed with the deferred actions, which will cause an exit from the procedure. I just tested this and it appears to work just fine.
Thanks Paul - and yes - you are accurate in terms of what I want to do but you said so with much brevity compared to my original post.
Anyway, what you suggested was actually my first attempt as it makes the most sense and logic but it failed - the RM steps are still evaluated so maybe this is indeed a bug?
Paul - I tried a simpler test and it does seem to work but only if the step that needs to skip the RM is not being called from another step - this actually gets back to a question you had earlier as to what did I meand by CALLING step.
So what I am doing is the following:
1) Main procedure is running
2) I call abother procedure that resides in another testcase
3) The CALLED step (the one being called in step #2 above) has a read file error so it needed to abort and exit
4) The CALLED step still evals the RM
5) The CALLED step exits the procedure after the RM evaluation and returns to the CALLING procedure.
Message Edited by tleding on 03-31-2009 10:09 PM
If you are worried about response mapping error, you can simply override Response mapping error event to not produce an error.
Also - the deferred actions are implemented as part of iTest 3.3. In iTest 3.2, these actions may take effect immediately and therefore may have slightly different behaviour.
Kumar - the first option really isn't a fix - the system is still misbehaving - also, we want errors to be produced if they really exist but not due to a file misread. As you can see, this would lead a person to troubleshoot the wrong thing - the RM errors would indicate a parsing error but in reality, they are being caused by an incorrect filename. This is why I want to use the OnExecReadFileError event as designed - to stop execution and go back the the higher level procedure.
Now as to the deferred action not existing in 3.2, then this is definitely a bug becuase if the ExitProcedure action is not being deferred, then why is iTest still evaluating the RM when it is being told to Exit immediately?
I am wondering if this exists in 3.3 but again, we haven't fully evaluated it yet but I will try to test it this week. Still, can you guys please reproduce in at least 3.2 and see if this is indeed a bug? A sample testcase is attached.
Thanks. We'll look into the bug in 3.2. The logic surrounding event processing (and deferred actions which were added in 3.3) is more complicated than you might think. There are a variety of cases that we're trying to deal with -- and I'm pretty sure that before ExitProcedure became a deferred action it was probably exiting the procedure only AFTER performing response mapping. The "bug" in 3.2, it seems to me, is the fact that there is no way to accomplish what you are quite reasonably hoping to accomplish.
As I said, I did try this with the latest, and it appears to work correctly. I don't get a response mapping error, the subsequent step is not executed, and execution of the procedure exits immediately after the readFile step. We'll look into the situation with 3.2 more carefully and see if there is a workaround for you until you upgrade.
Paul\Kumar - really appreciate your help here and also, not trying to be a stickler but I want to make sure I completely understand what is going on...
So what I think you are saying is that when you try this in v3.3 with a called procedure containing a "readFile" error and using "ExitProcedure" & "AbortStep" events, then the RM is not evaluated. But when I try the exact same thing in v3.2, it is probably expected that it would read the RM because there were some changes made to the Event logic in v3.3. Is this accurate?
Also, are the Event logic changes documented anywhere? What concerns me is that core operations of iTest can change behavior from one minor release to another with no mention - at least in the release notes documentation I have had access to. If my statement above is correct, then this can have far reaching implications for folks with existing systems already built with expected behavior in mind. It's analoagous to a router company changing the route prefix lookup rules for BGP when going from 12.2 to 12.3 software - folks would be freaked...
Message Edited by tleding on 04-01-2009 01:03 PM
Deferred action processing is documented in 3.3 documentation as well as 3.3 release notes. We take backwards compatibility very seriously and only incompatibilities we introduce from time to time are to "fix" the older incorrect behaviour - and only if fixing the older incorrect behaviour will not cause huge amounts of pain to our customers on moving to the new release.
OK - so I'm not asking this properly - here's what I need to understand:
1) What is the proper and expected behavior of Events in v3.2?
2) What is the proper and expected behavior of Events in v3.3?
3) What is the delta between the two?
Also, as an aside - and this is not directed at you or anyone at Fanfare - but when I made the move from v3.1 to v3.2, it was extremely painful which is why I have not moved to v3.3 yet. I spent countless hours (go look at my ticket history and chat with Bala and Sukhpreet - they will tell you) re-doing my testcases because some behaviors were changed - albeit it better but the move did "break\change" some things.
Here's the thing - I test complex software for a living here at Cisco so I totally understand how difficult this gig is - but, I am also probably one of Fanfare's most critical users because of the high expectation level I place upon myself for this job - I know you guys feel the same way but things always get through - even at Cisco. So net-net - I am just in the process of figuring out what those things are...
Message Edited by tleding on 04-02-2009 12:52 PMEvents are processed in exactly the same way in 3.2 and 3.3. What has changed between the two releases is the following (this is explained in detail in 3.3 documentation):
All actions inside events or analysis rules which change the "execution flow" are processed after the step which they are attached to has completed execution. So if you had the following in 3.2:
analysis rule:
when true:
exit execution
declare execution issue
which is kind of strange thing to do here because if you exit execution, do you expect the execution issue to get logged?
In 3.3, since exit execution changes the execution flow is processed after all the non execution flow affecting actions are processed. So declare execution issue will be processed and then step will finish execution and then "exit execution" action will get executed.
ok - so it sounds like the only change re: Events is the ExitProcedure event which I happen to rely upon white heavily because the new paradigm breaks my scripts...
What would be really cool is to have both as follows:
ExitProcedure = Exit AFTER other step processing is coimplete
AbortProcedure = Exit procedure immediately
Thoughts?
We cannot support any "execution flow" changing action to take effect immediately because it causes "unpredictability" in the order of execution. For example, if there are two actions:
AbortExecution
DeclareIssue
Since aborting execution is asynchronous, DeclareIssue may or may not get logged in time. That was the reason we changed the design in 3.3 to make sure that we can get predictable and well defined order in which actions are executed.
In your case, you will not have any impact in the order of execution unless you have corner cases like I mention above and even those should be beneficial. Your main problem in 3.2 which you started this thread for was that response was getting mapped even though you aborted the step. That is a bug and is fixed in 3.3.
Ahhh - I missed that last piece that what I am seeing now in v3.2 was a bug - I didn;t know that was confirmed yet. Paul said he would try it if he had the cycles but I thought it was still just my detemrination.
If - in v3.3 - the RMs are not read after an ExitProcedure is encountered, then we're good to go...
