question

JWu avatar image
JWu Suspended asked JWu Suspended posted

Does iTest support simultaneous access to two or more routers? Is there multithreading support?

I would like to find out if iTest supports access to multiple devices simultaneously. I have a situation where I need to access two routers but ideally I would like to access them in parallel such that while my test case is sending commands to router1 my second test case is running at the same time sending commands to router2. Is this possible in iTest?
iTest
10 |950

Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.

dclaar avatar image
dclaar Suspended answered dclaar Suspended posted

Looking back over this, it looks like "threads" don't share the same data space? So they're really more like processes? (In the linux sense).

 

Still, I don't quite understand why:

 

Call foo, store results in bah, run in a thread

wait until info exists local bah

 

Doesn't work, while making it a global does.

 

Doug

Message Edited by dclaar on 01-22-2009 05:47 PM
3 comments
10 |950

Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.

PaulD avatar image PaulD commented ·

Threads do share the same data space.  But like virtually every programming environment, each thread has its own "stack".  Each procedure call gets its own "frame" on that stack.  Local variables are stored in that frame.  (This entire "heap" is implemented as an XML document, where there is a node representing the root of each stack, containing nested nodes for each frame, containing nodes for each variable.)

 

If you know the location of data in another thread, you can access it.  

 

For fun, set a breakpoint in your test that has multiple threads, and take a close look at the Data view.  You'll get a sense for where all this data is.

 

Global variables, by the way, are just nodes immediately under the root node of the entire heap.

 

In answer to why the scenario you're describing doesn't work...

 

Call foo, store results in bah, run in a thread:  This whole step (including the storing of results in bah) happens in a different thread.  So "bah" is going to be in the new thread's stack.

 

When you "wait until info exists local bah", you're going to be waiting in the original thread.

 

If you really want to avoid global variables, then you'd have to look for data stored in the other thread -- and you'll need a complex xpath location to refer to that variable over in the other thread's stack.

 

I suspect that you're going to come back instantly with a comment that we should be storing the results in the calling rather than called thread.  I guess there's an argument for that.  But in terms of "purity", every iTest step is exactly like every other step.  A "call" step is really no different than any other step in some sense.  And if you have any other step that is going to run in a different thread, then it makes sense for its analysis/post-processing to occur in the newly created thread.  So we treat "call" the same way.

 

At the same time, I recognize the need here.  And I think the answer is a new "join" step that makes it easy to wait for another thread to complete -- or to get to a certain point.  We've spec'd that out, but have not implemented it at this point.

0 Likes 0 ·
dclaar avatar image dclaar PaulD commented ·

Wow, that just makes no sense! If I do a call, and store the results, they go into the called threads stack? But the results aren't available until after the thread completes. And when the thread completes, the called thread's context goes away, so the results are gone.

 

That makes my head hurt! I guess I can see it from the analysis point of view: The analysis should happen in the thread. But as "Store Response" is in "Post-processing", and happens after the step is complete (which is why the info global exists thing works), this is just...strange.

 

Is the thread a direct child? In other words, could I do:

 

run thread, store results in ../bah

info exists local bah

Message Edited by dclaar on 01-22-2009 06:23 PM
0 Likes 0 ·
PaulD avatar image PaulD dclaar commented ·

The model builds on the classic Unix "fork".  That means that when the new thread starts it makes a clone of the caller's stack and then creates its own stack frames starting from there.  So if you did store something in ../bah -- that would go into a stack frame in the new thread's stack.  In fact, you can refer to clones of the calling procedure's local variables in the new thread using this approach (e.g., ../bah), but writing these does not help -- as they are not the variables that are in the original thread.

 

This might make your head hurt, but what could make your head hurt even more is what would happen if the calling procedure exits while the called thread is still running.  Where would those "store in" variables go then!?

 

Fundamentally, the post-processing is really part of the step itself.  For example, the analysis of the results of making a call may result in additional actions being taken -- even such as calling yet another procedure.  It is just too complicated to try to think about doing some of this in the original thread. 

0 Likes 0 ·
dclaar avatar image
dclaar Suspended answered dclaar Suspended posted

Cool. I was close; I just didn't get my analyze quite right.

 

I should add that, for safety, the global variable name should either be really, really unique (probably not "s1":smileywink:), or there needs to be an "eval gunset s1" in front of the step that stores the response in s1.

 

Doug

10 |950

Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.

dclaar avatar image
dclaar Suspended answered dclaar Suspended posted

What's the chance of this not completing, due to the increment code being non-atomic?

 

thread 1: gget all_finished

thread 2: gget all_finished

thread 1: incr

thread 2: incr

thread 1: gset all_finished

thread 2: gest all_finished

 

==> all_finished never reaches $length

 

In SVT, you could wait for "prev in session analyzed". This doesn't seem to be in iTest?

 

There really ought to be the ability to wait for a thread or all threads.

 

Doug

6 comments
10 |950

Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.

PaulD avatar image PaulD commented ·

Doug,

 

There was a lively discussion about how much inter-thread communication to build into iTest.  We discovered that the inter-thread implications of things like "prev in session analyzed" from FanfareSVT was not a reliable scheme for building dependable inter-thread synchronization.

 

One way to accomplish inter-thread synchronization in iTest is through the data model.  Essentially, you can use a global variable as a semaphore.  It would work something like this:

 

main thread:

    gset semaphore 0

    <start thread 1>

    <start thread 2>

 

thread 1:

    <various steps A>

    gset semaphore 1

    <additional steps A>

 

thread 2:

    <various steps B>

    gget semaphore:  with an analysis rule that will repeat the step until this returns 1

    <additional steps B>

 

With this setup, all of <various steps A> are guaranteed to be completed before any <additional steps B> begin.

0 Likes 0 ·
dclaar avatar image dclaar PaulD commented ·

So, a global variable per thread? Ugh. OK, I guess I can do that. Seems a bit strange to build in threads but not build in any kind of thread wait.

0 Likes 0 ·
dclaar avatar image dclaar PaulD commented ·

So in SVT, I did this:

    set _routine foo

    set _args "SP1"

S1: run parallel

    set _args "SP2"

S2: run parallel

S1: start after previous in session analyzed

S2: start after previous in session analyzed

    # both are done now, continue

 

parallel:

    call $_routine $_args

 

Now, the advantage of this was that I didn't have to modify any of my routines to make them run in parallel. I guess I can do something similar in iTest; parallel can set the semaphore, and I can have a parallel_done routine.

0 Likes 0 ·
dclaar avatar image dclaar PaulD commented ·
Hmm. When is the "Store the response in a variable which can be used later" variable written? Seems like I could use [info exists] on it.
0 Likes 0 ·
dclaar avatar image dclaar PaulD commented ·

Hmm. Doesn't seem to work.

call sominex, store result in s1; run as thread

call sominex

eval info exists local s1

  analyze

    none

    assert $value==1

      When True OK

      When False

        RepeatStep max:10; delay: 1.0; skip remaining rules

 

sominex:

  sleep 5

 

I loop 10 times waiting for s1, then I die.

 

Doug

0 Likes 0 ·
PaulD avatar image PaulD dclaar commented ·

When you store the response in a variable, it will be stored when the step finishes execution and has been analyzed. 

 

I'm attaching a test case that does what you are suggesting.  Note that each thread has its own local variables.  So you can't use [info exists local s1].  You have to store it in a global variable and check that.  

0 Likes 0 ·
sominex.fftc (3.8 KiB)
AdeelM avatar image
AdeelM answered AdeelM posted

JWu,

 

There is multithreading support available in iTest. It is upto the user to enable it. For each step, if you open the step properties, the General properties have a checkbox for "Start the step (in a new thread) and proceed to the next step".

 

 

 

 

Notice that when user enables this option, there is a purple arrow that appears next to this step.

 

 

 

 

This means that if this property was enabled for step 3 in the test case, then during execution, after iTest completes step 2, step 3 will be launched in a new thread and the execution will immediately proceed to step 4.

 

Main thread        New thread

 

* Step 1

     |

* Step 2

     |

* Step 4           * Step3          <-- New thread created here

     |                  |

* Step 5                |

     |                  |

* ...

 

 

2 comments
10 |950

Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.

JWu avatar image JWu commented ·

Thank you AdeelM.

 

How does iTest handle more than two simultaneous executions? And is there any way to determine how many threads are running concurrently?

0 Likes 0 ·
AdeelM avatar image AdeelM JWu commented ·

You can have a loop calling a procedure that runs in a new thread. This way, each call ends up running in a new thread. There are a few things to keep in mind:

 

  1. If you open a session in the calling procedure you need to make sure that the session name is unique for each iteration
  2. You need to build some logic to wait for all spawned threads to finish execution. The default behavior is that the main thread does not communicate with spawned threads. Typically, at some point in your test case you would like to wait for all spawned threads to finish before proceeding in the main thread and this is where you need to build test case logic.This is done using global variables.

 

Attached is a sample test case. Lets examine this.

 

 

 

Notice that we initialize four IP addresses that we pass to the foreach loop (step 4). Step 4.1 iterates through all IP addresses and calls the ping procedure. This call step is set to run in a new thread, indicated by the purple arrow so each iteration of the loop then spawns a new thread. Also note that we pass session name as an argument. As stated above, this is because each thread that opens a session needs to have a unique session name. In this example, we pass the IP address into the session name arugment (ping -session $each_ip -ip $each_ip).


The foreach loop completes ahead of the four threads that it spawned. This is because nothing in the loop runs in the main thread, so the execution immediately proceeds to step 5.

 

However, the test case does not complete execution until all the spawned thread also complete execution. 

 

Main Thread           Thread 1             Thread 2             Thread 3             Thread 4

           

Step 1

Step 2

Step 3

Step 4                Step 4.1             Step 4.1             Step 4.1             Step 4.1

Step 5                   |                    |                    |                    |

   |                     |                    |                    |                    |

   |                     |                    |                    |                    |

   |                     |                    |                    |                    |

<Wait for all threads to complete execution>

<Finish>

 

It is important to note that if you would like to wait in the main thread for spawned threads to complete execution then you need to build additional logic. This is done by making use of global variables. Take a look at steps 2 and 3 in the main procedure. We initialize a global variable called all_finished. We also create a local variable called length based on the total number of IP addresses. Each time in the  ping procedure we increment the value of the global variable. 

 

 

 

 

Now we want to wait for all spawned threads to finish execution. This is done in step 5 where we check the value of our global variable (all_finished) to reach length. This is done by creating an analysis rule that checks this assertion ($value == $length). If spawned threads have not finished execution this assetion will fail in which case we have set the action to RepeatStep. It is a good practice to limit the maximum number of times to repeat. 

 

 

 

0 Likes 0 ·

Write an Answer

Hint: Notify or tag a user in this post by typing @username.

Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.