Preventing iTest from learning propmts automatically

Question

question

ankugarg asked Mar 18, '10 ericnute edited Jul 3, '14

Preventing iTest from learning propmts automatically

Sometimes when iTest gets stuck on something for few seconds(like during reload of switch),then it atomatically learns the prompt(user do not reply to "learn this propmt message), which we don't want it to learn.How can we prevent this beforehand.The properties are set to "auto and idle" rather than "auto or idle"

If you can tell when exactly it learns the propmt automatically,it'll be great help.This behaviour is really inconsistent even in the same testcase being run on same setup.Sometimes it learns the propmt,sometimes it goes fine.

Thanks.

iTest session profile prompt

10 |950

Attachments: Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.

Answer 1 · 2010-03-22T18:21:49Z

dclaar Suspended answered Mar 22, '10 ericnute edited Jul 3, '14

In your session, under Completion, there is "Time for user to respond (sec):" If you set that to a very large number, then iTest won't "learn" the prompts. You should set this--and other things that you want all your sessions to have--in a high-level session profile that all your other session profiles are based on. You will need one for telnet and one for ssh if you use both protocols. Not sure if any of the other protocols require it. I've included a screen shot below. (Yes, it doesn't "learn" the prompt, but I'm pretty sure that the question is "how to I stop iTest from continuing when a command is taking a while?" ### Session Properties, **More** button > Terminal > Replay Completion ![Session Properties > Terminal > Replay Completion page, section "Unknown Prompts During Execution][1] [1]: /storage/attachments/1322-completion.jpg

completion.jpg (131.7 KiB)

2

10 |950

Attachments: Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.

ankugarg commented · Mar 23, 2010 at 05:16 PM

Exactly. My device is going for reload,and there it upgrades image. It gets stuck there for more than 15 minutes(which is perfectly ok from testcase point of view). I was executing in gui only.I tried Doug's idea and it is working fine.

0 ·

dclaar ankugarg commented · Mar 23, 2010 at 05:33 PM

I really think you need to go back and examine this. 30 seconds is a blink. Even for someone who is sitting and constantly watching the test, 30 seconds is not that long. 3-5 minutes might be a lot more reasonable. But even then, it is adding variability to the testing outcome, and again, all the arguments against "auto or idle" apply.

I understand that you want to help the people just starting out in iTest, but adding these "training wheels" causes real problems. Having my test work differently when I run it interactively vs. non-interactively goes completely against reliability and repeatibility, which are key qualities of good testing.

If you are going to have features like this, I'd like to see them only available in "I'm a newbie" mode, which the user should have to switch on, or at least can switch off. Maybe a "help me learn iTest" mode? In this mode, when you click to learn the prompt, iTest should explain--or offer to explain--how to add prompts, etc.

0 ·

Answer 2 · 2010-03-18T18:52:21Z

PreetS answered Mar 18, '10 ericnute converted comment to answer Jul 3, '14

It does not automatically learn the prompt.

The "learn this prompt" is for users to tell iTest to learn it. This is for scenarios where you would look at the test and say , "Hmm... My test case is stuck because of a new prompt, why not learn it right now and move on".

If you do not click "learn new prompt", it will show you a warning with a timer. At that point you can click on "keep waiting" to wait more. If the timer expires then an execution issue is raised and the test continues.

Note that this behavior is not present when you are running in headless mode.

10 |950

Attachments: Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.

Answer 3 · 2010-03-22T19:49:31Z

PaulD answered Mar 22, '10 ericnute converted comment to answer Jul 3, '14

One thing to note....

During an interactive session, the telnet/ssh/serial session may discover something that it thinks may be a new prompt. When the session comes to an end, it will pop up a dialog where you decide whether these "learned" prompts (and other things) should be used to update the session profile or not.

When you are executing a test case in the GUI, you may see the session stall because there is a prompt it is waiting for. When you click "learn prompt" here, you are not directly having any effect on the session profile. You are just telling the session to proceed and to use that prompt for the remainder of that session. Then, when that session ends, you will again get that dialog asking you whether you want to actually update the session profile or not with the learned information. If you cancel that wizard, the learned information is discarded.

So I don't see a good reason to set this "time for a user to respond" to a huge number -- unless you really have a weird case of a device that may have an incredibly unpredictable idle time before giving you a prompt -- where that could be a really really long time. (Note that iTest is smart enough to know how long the device originally took to respond at this point, and won't bother asking you until it thinks that it is unlikely that a prompt is coming.)

7

10 |950

Attachments: Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.

dclaar commented · Mar 22, 2010 at 08:01 PM

The reason for setting this number large is simple:

I start running a test case from the GUI. It goes off and does its thing. I check my email. Or whatever. Sometime during the test, it reboots the DUT. After some short amount of seconds, iTest decides that it has a new prompt, and continues on its merry way, totally messing up my test run. And this indeed appears to be the same use case that the original poster described.

The alternative to setting the number large is to sit and watch every single interactive run to be sure that I catch that very short window and answer the "keep waiting" prompt.

0 ·

PaulD dclaar commented · Mar 22, 2010 at 11:22 PM

We agree. The point is that if iTest had captured the original reboot sequence, it would have discovered that it took a long time, and would have automatically populated that field accordingly -- with a large number.

0 ·

dclaar PaulD commented · Mar 22, 2010 at 11:32 PM

I suppose that's the theory. My experience has been that doesn't necessarily happen. Maybe it is because I have lots of existing tests, and getting that number set is dependent on the test being brand new, maybe created from capture?

Before I set the number large, several people got bitten by this, because they ran "their" test interactively to find out why it failed in the nightly non-interactive run, and it went off in the weeds at the first long operation.

I understand the motivation behind it, but--in my (not so) humble opinion, it really isn't a good idea. Maybe even a bad one. It makes tests fragile and unreliable: Step x took longer than usual this time. Oops, iTest didn't wait.

I'd go so far as to equate it with using "auto or idle", which--as has been said here many times--is something that should be avoided as much as possible.

0 ·

PaulD dclaar commented · Mar 23, 2010 at 12:51 AM

Perhaps you are right. We implemented this to address a long-standing usability problem -- which is where users failed to have prompts configured properly and then failed to understand why their test cases would "hang".

But perhaps we need to revisit this to see whether we've created a different kind of problem.

0 ·

KumarS PaulD commented · Mar 23, 2010 at 05:02 AM

That is not completely true. If you run tests in your nightly regression environments using itestcli or using itestrt, iTest will wait forever and never learn any prompts.

Only if UI session window is present, the UI prompt learning logic is activated.

0 ·

ankugarg KumarS commented · Mar 23, 2010 at 01:50 PM

dclaar has exactly got my problem.Itest indeed sometimes start throwing the commands after waiting for several time interval,when the time interval is indeed genuine.It learns the prompts automatically.

After the testcase has finished executing,you get a pop up asking for the newly learned prompts.There you have check box at the bottom "Don't allow this session profile or test bed to learn prompts automatically " which is checked by default.

Initially my test cases learn that prompt.Then i checked out that box,and deleted those learned prompts from my session profile.Then when i executed that testcase again,iTest started throwing commands at that particular prompt only(May be some timing issue,which it has decided in earlier run).

0 ·

KumarS ankugarg commented · Mar 23, 2010 at 04:06 PM

This designed behaviour happens only if you execute a cli testcase in GUI and you have a step which takes more than 30 seconds to produce a prompt. As Doug said, if you do not like this behaviour, you can change it by specifying bigger timeout in the session profile or testbed properties for the device.

I again want to iterate that in regression (when you use itestcli or itestrt), these properties are ignored and iTest will wait for as long as it takes to get the prompt. This feature was designed for people who do not know what prompts are and help them learn prompts for their devices and make their testcases run more smoothly.

Most likely case where this behaviour in the GUI causes problems is when rebooting the switch/device to which you are connected through a terminal server. In these cases, it is not unreasonable to put a longer wait just for the reboot step.

I would also argue that if this dialog pops up, you should take a look at why the step is waiting so long for a prompt and if it is reasonable for it do so. Most of the time it will be - but sometimes when it is not, it may point to a bug in the device.

0 ·

question