itestcli and the GUI

Question

question

dclaar Suspended asked Apr 22, '09 ericnute edited Apr 22, '09

itestcli and the GUI

Here's an interesting anomoly: If you have started the GUI, itestcli can use the workspace, no problem. But if you have started itestcli, the GUI won't let you open the workspace! Is there any workaround for this?

Being as the test reports are only available in the workspace, this causes a serious problem when you need to hop onto your test controller and figure out why last night's test failed, since today's is already running. I guess you could leave the GUI running all the time, but--as I mentioned elsewhere--it gets slow after a while.

I haven't figured out a real good way to mix the whole workspace concept with batch jobs. I'm open to suggestions as to how to do this.

iTest itestcli

10 |950

Attachments: Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.

Answer 1 · 2009-04-22T17:09:04Z

KumarS answered Apr 22, '09 KumarS posted Apr 22, '09

When we designed iTestCli, we designed it for headless regression environments. We did not expect the use model like yours where you fire up itestcli with GUI running.

itestcli fires up iTest to perform its task. iTest works on a workspace. That is why itestcli takes a workspace option which it passes to iTest. If you start itestcli with "-g" option, it will launch iTest in gui mode. If you launch it without it, then iTest will be launched in headless mode (this is the preferred way to run tests in headless regression environments). While an instance of iTest is running on a workspace (headless or gui), another instance of iTest cannot open that workspace.

Your workaround can be to always launch itestcli with "-g" mode because it does not seem like you have a headless regression environment. Another workaround could be to make a copy of workspace and use itestcli with one and iTest with another.

In iTest 3.4, we have added option to recursively export test reports for people who use EXEC.Run a lot. This will obviate the need for opening a GUI to export test reports.

Another thing new in iTest 3.4 is "itestrt" which is a workspace independent way of executing tests. You save your projects as "itar" files (equivalent of Java jar files) and itestrt can execute testcases in these itar files. These itar files contain all your assets (testcases, testbeds, parameters, response maps, etc...). It can also recursively export your test reports. You can use iTest and itestrt simultaneously because itestrt does not work on a workspace - but itar files.

22

10 |950

Attachments: Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.

dclaar commented · Apr 22, 2009 at 05:43 PM

There's a flaw in your design: When you run itestcli, it puts the results into the workspace, and the only way you can get to those results is via the GUI. So it is really not possible to run a completely headless regression, unless of course nothing ever fails! Making a separate copy of the workspace to use with itestcli doesn't solve this problem, unless copying the .reports directory just happens to bring the reports over intact. And even if it did, I'm pretty sure that's not at all supported!

Even with an option to export reports, unless there's also an import option, this is not really going to work that well, since the way iTest is designed, you figure out what went wrong by bringing up the report, looking at the responses, queries, etc. Unless I can see those things, it will be difficult to troubleshoot failures, which is the primary human-centered task in running regression. If all I have is a static html report, all the benefits of the eclipse environment are lost.

0 ·

KumarS dclaar commented · Apr 22, 2009 at 06:25 PM

We understand that and that is why iTest 3.4 has a recursive report export option.

0 ·

dclaar KumarS commented · Apr 22, 2009 at 06:44 PM

And corresponding import?

0 ·

KumarS dclaar commented · Apr 22, 2009 at 07:53 PM

What is this "import"?

0 ·

dclaar KumarS commented · Apr 22, 2009 at 08:30 PM

I run regression via itestcli, and export the reports.

regression go BOOM.

How do I figure out what went wrong?

The power of itest is the whole eclipse integration, and the test reports don't necessarily contain everything--for example the prompts--based on the assumption that you can go into the GUI and look at responses, queries, etc. So how do I get those exported reports back into eclipse? I guess that, since you're asking what import is, that means that I can't look at the results in the GUI, and have to try and figure things out from the static, incomplete report.

0 ·

KumarS dclaar commented · Apr 22, 2009 at 08:43 PM

I see what you are trying to say. Once you get an exported HTML report, you may not be able to know the cause of test failure by just looking at the report. If you export your reports as XML_Raw, you will get the complete information - but that is not easily viewable because of the amount of information as well as the format of the information.

You want the actual "test report" residing in the test report database inside the workspace in which executions were taking place (this is on a different machine than yours). You probably want to export all these reports in their native format (.fftz) and then import them back in a workspace on your machine. This scenario is not supported currently. Even if we did support it, it could take a long time to export all the reports and then import them based on how many executions you have.

Currently, you can copy ".reports" folder from the regression machine's workspace and copy it to your workspace on your machine and open iTest to see all the reports.That will allow you to browse these reports from inside iTest in the rich test report editor.

0 ·

dclaar KumarS commented · Apr 23, 2009 at 06:53 PM

Yeah, not so much. Maybe you can copy .reports if nothing is running, but that kinda defeats the whole purpose.

eclipse.buildId=unknown java.version=1.6.0_06 java.vendor=Sun Microsystems Inc. BootLoader constants: OS=win32, ARCH=x86, WS=win32, NL=en_US Command-line arguments: -os win32 -ws win32 -arch x86 Error Thu Apr 23 11:50:24 PDT 2009 Attempting to recover persistent data after improper shutdown. Renaming database. java.sql.SQLException: Failed to start database 'C:/Documents and Settings/dclaar/Desktop/iTestStuff/main/.report', see the next exception for details. at org.apache.derby.impl.jdbc.SQLExceptionFactory40.getSQLException(Unknown Source) at org.apache.derby.impl.jdbc.Util.newEmbedSQLException(Unknown Source) at org.apache.derby.impl.jdbc.Util.seeNextException(Unknown Source) at org.apache.derby.impl.jdbc.EmbedConnection.bootDatabase(Unknown Source) at org.apache.derby.impl.jdbc.EmbedConnection.<init>(Unknown Source) at org.apache.derby.impl.jdbc.EmbedConnection30.<init>(Unknown Source) at org.apache.derby.impl.jdbc.EmbedConnection40.<init>(Unknown Source) at org.apache.derby.jdbc.Driver40.getNewEmbedConnection(Unknown Source) at org.apache.derby.jdbc.InternalDriver.connect(Unknown Source) at org.apache.derby.jdbc.AutoloadedDriver.connect(Unknown Source) at java.sql.DriverManager.getConnection(Unknown Source) at java.sql.DriverManager.getConnection(Unknown Source) at com.fnfr.foundation.database.Database.connectToDB(Database.java:652) at com.fnfr.foundation.database.Database.access$2(Database.java:589) at com.fnfr.foundation.database.Database$1$1.run(Database.java:89) at org.eclipse.core.internal.jobs.Worker.run(Worker.java:55) Caused by: java.sql.SQLException: Failed to start database 'C:/Documents and Settings/dclaar/Desktop/iTestStuff/main/.report', see the next exception for details. at org.apache.derby.impl.jdbc.SQLExceptionFactory.getSQLException(Unknown Source) at org.apache.derby.impl.jdbc.SQLExceptionFactory40.wrapArgsForTransportAcrossDRDA(Unknown Source) ... 16 more Caused by: java.sql.SQLException: The conglomerate (192) requested does not exist. at org.apache.derby.impl.jdbc.SQLExceptionFactory.getSQLException(Unknown Source) at org.apache.derby.impl.jdbc.SQLExceptionFactory40.wrapArgsForTransportAcrossDRDA(Unknown Source) at org.apache.derby.impl.jdbc.SQLExceptionFactory40.getSQLException(Unknown Source) at org.apache.derby.impl.jdbc.Util.generateCsSQLException(Unknown Source) ... 13 more Caused by: ERROR XSAI2: The conglomerate (192) requested does not exist. at org.apache.derby.iapi.error.StandardException.newException(Unknown Source) at org.apache.derby.impl.store.access.heap.HeapConglomerateFactory.readConglomerate(Unknown Source) at org.apache.derby.impl.store.access.RAMAccessManager.conglomCacheFind(Unknown Source) at org.apache.derby.impl.store.access.RAMTransaction.findExistingConglomerate(Unknown Source) at org.apache.derby.impl.store.access.RAMTransaction.openConglomerate(Unknown Source) at org.apache.derby.impl.sql.catalog.DataDictionaryImpl.getDescriptorViaIndex(Unknown Source) at org.apache.derby.impl.sql.catalog.DataDictionaryImpl.locateSchemaRow(Unknown Source) at org.apache.derby.impl.sql.catalog.DataDictionaryImpl.boot(Unknown Source) at org.apache.derby.impl.services.monitor.BaseMonitor.boot(Unknown Source) at org.apache.derby.impl.services.monitor.TopService.bootModule(Unknown Source) at org.apache.derby.impl.services.monitor.BaseMonitor.startModule(Unknown Source) at org.apache.derby.iapi.services.monitor.Monitor.bootServiceModule(Unknown Source) at org.apache.derby.impl.db.BasicDatabase.boot(Unknown Source) at org.apache.derby.impl.services.monitor.BaseMonitor.boot(Unknown Source) at org.apache.derby.impl.services.monitor.TopService.bootModule(Unknown Source) at org.apache.derby.impl.services.monitor.BaseMonitor.bootService(Unknown Source) at org.apache.derby.impl.services.monitor.BaseMonitor.startProviderService(Unknown Source) at org.apache.derby.impl.services.monitor.BaseMonitor.findProviderAndStartService(Unknown Source) at org.apache.derby.impl.services.monitor.BaseMonitor.startPersistentService(Unknown Source) at org.apache.derby.iapi.services.monitor.Monitor.startPersistentService(Unknown Source) ... 13 more

eclipse.buildId=unknown java.version=1.6.0_06 java.vendor=Sun Microsystems Inc. BootLoader constants: OS=win32, ARCH=x86, WS=win32, NL=en_US Command-line arguments: -os win32 -ws win32 -arch x86 Error Thu Apr 23 11:50:24 PDT 2009 Attempting to recover persistent data after improper shutdown. Renaming log files. java.sql.SQLException: Failed to start database 'C:/Documents and Settings/dclaar/Desktop/iTestStuff/main/.report', see the next exception for details. at org.apache.derby.impl.jdbc.SQLExceptionFactory40.getSQLException(Unknown Source) at org.apache.derby.impl.jdbc.Util.newEmbedSQLException(Unknown Source) at org.apache.derby.impl.jdbc.Util.seeNextException(Unknown Source) at org.apache.derby.impl.jdbc.EmbedConnection.bootDatabase(Unknown Source) at org.apache.derby.impl.jdbc.EmbedConnection.<init>(Unknown Source) at org.apache.derby.impl.jdbc.EmbedConnection30.<init>(Unknown Source) at org.apache.derby.impl.jdbc.EmbedConnection40.<init>(Unknown Source) at org.apache.derby.jdbc.Driver40.getNewEmbedConnection(Unknown Source) at org.apache.derby.jdbc.InternalDriver.connect(Unknown Source) at org.apache.derby.jdbc.AutoloadedDriver.connect(Unknown Source) at java.sql.DriverManager.getConnection(Unknown Source) at java.sql.DriverManager.getConnection(Unknown Source) at com.fnfr.foundation.database.Database.connectToDB(Database.java:632) at com.fnfr.foundation.database.Database.access$2(Database.java:589) at com.fnfr.foundation.database.Database$1$1.run(Database.java:89) at org.eclipse.core.internal.jobs.Worker.run(Worker.java:55) Caused by: java.sql.SQLException: Failed to start database 'C:/Documents and Settings/dclaar/Desktop/iTestStuff/main/.report', see the next exception for details. at org.apache.derby.impl.jdbc.SQLExceptionFactory.getSQLException(Unknown Source) at org.apache.derby.impl.jdbc.SQLExceptionFactory40.wrapArgsForTransportAcrossDRDA(Unknown Source) ... 16 more Caused by: java.sql.SQLException: cannot create log file at directory C:\Documents and Settings\dclaar\Desktop\iTestStuff\main\.report\log. at org.apache.derby.impl.jdbc.SQLExceptionFactory.getSQLException(Unknown Source) at org.apache.derby.impl.jdbc.SQLExceptionFactory40.wrapArgsForTransportAcrossDRDA(Unknown Source) at org.apache.derby.impl.jdbc.SQLExceptionFactory40.getSQLException(Unknown Source) at org.apache.derby.impl.jdbc.Util.generateCsSQLException(Unknown Source) ... 13 more Caused by: ERROR XSLAQ: cannot create log file at directory C:\Documents and Settings\dclaar\Desktop\iTestStuff\main\.report\log. at org.apache.derby.iapi.error.StandardException.newException(Unknown Source) at org.apache.derby.impl.store.raw.log.LogToFile.getLogDirectory(Unknown Source) at org.apache.derby.impl.store.raw.log.LogToFile.getControlFileName(Unknown Source) at org.apache.derby.impl.store.raw.log.LogToFile.boot(Unknown Source) at org.apache.derby.impl.services.monitor.BaseMonitor.boot(Unknown Source) at org.apache.derby.impl.services.monitor.TopService.bootModule(Unknown Source) at org.apache.derby.impl.services.monitor.BaseMonitor.startModule(Unknown Source) at org.apache.derby.iapi.services.monitor.Monitor.bootServiceModule(Unknown Source) at org.apache.derby.impl.store.raw.data.BaseDataFileFactory.bootLogFactory(Unknown Source) at org.apache.derby.impl.store.raw.data.BaseDataFileFactory.setRawStoreFactory(Unknown Source) at org.apache.derby.impl.store.raw.RawStore.boot(Unknown Source) at org.apache.derby.impl.services.monitor.BaseMonitor.boot(Unknown Source) at org.apache.derby.impl.services.monitor.TopService.bootModule(Unknown Source) at org.apache.derby.impl.services.monitor.BaseMonitor.startModule(Unknown Source) at org.apache.derby.iapi.services.monitor.Monitor.bootServiceModule(Unknown Source) at org.apache.derby.impl.store.access.RAMAccessManager.boot(Unknown Source) at org.apache.derby.impl.services.monitor.BaseMonitor.boot(Unknown Source) at org.apache.derby.impl.services.monitor.TopService.bootModule(Unknown Source) at org.apache.derby.impl.services.monitor.BaseMonitor.startModule(Unknown Source) at org.apache.derby.iapi.services.monitor.Monitor.bootServiceModule(Unknown Source) at org.apache.derby.impl.db.BasicDatabase.bootStore(Unknown Source) at org.apache.derby.impl.db.BasicDatabase.boot(Unknown Source) at org.apache.derby.impl.services.monitor.BaseMonitor.boot(Unknown Source) at org.apache.derby.impl.services.monitor.TopService.bootModule(Unknown Source) at org.apache.derby.impl.services.monitor.BaseMonitor.bootService(Unknown Source) at org.apache.derby.impl.services.monitor.BaseMonitor.startProviderService(Unknown Source) at org.apache.derby.impl.services.monitor.BaseMonitor.findProviderAndStartService(Unknown Source) at org.apache.derby.impl.services.monitor.BaseMonitor.startPersistentService(Unknown Source) at org.apache.derby.iapi.services.monitor.Monitor.startPersistentService(Unknown Source) ... 13 more

0 ·

PaulD dclaar commented · Apr 23, 2009 at 09:07 PM

I certainly understand what you're trying to do here. It is worth noting that we have been thinking about these issues. The big question is where the detailed report information should live. We actually did a bunch of work to allow detailed test report information to be post-processed in a whole variety of ways starting in 3.4. I understand your (completely reasonable) request to get the report data back into the local workspace where you can look at it using iTest's report viewer. But we also see another common use case where people want to centralize this detailed report information. So we went to the trouble of doing two things:

1) There is a new API in our SDK in 3.4 that allows anyone to build an Eclipse plugin that gets access to all of the detailed test report information -- all as straight Java interfaces rather than having to deal with XML/HTML stuff. This allows that code to do anything they want with it. In the most common cases, we see customers publishing this data to a server, pushing it into a database, writing it out into customized files, post-processing it -- looking for specific stuff, and so on.

2) We defined a web service that we call "test reporting service" so that the same information could be published using an open interface. We are happy to share this work -- including the web service definition and the plugin implementation that will take test reports and push all of the data to a server supporting that web service. (We even have an implementation of a server of that web service that will push that data into a MySQL database.) We have not yet chosen to productize any of that stuff. But you get the idea that we are trying to recognize a variety of cases where users want to centralize their test reports.

Admittedly, we haven't handled your use case all that well. The normal cases look so simple that you might wonder why we haven't. The challenge comes from having to deal with potentially huge test reports. Imagine a test that runs for a week and produces 100GB of data. We have to worry about this case. Of course, we could impose artificial limits to handle the simple cases, but we always end up getting burnt by someone coming up with another way to use a feature that we weren't really planning for well. (Sound like anyone familiar? :)

We'll keep working. I appreciate that you're still sharing your needs and ideas for how to address them.

0 ·

dclaar PaulD commented · Apr 23, 2009 at 10:11 PM

I think that there are two main reasons to want to look at it in the report viewer:

There's a lot of information missing in the HTML or text reports. (and the XML report is tough to read).
Users are familiar with the report viewer, so there's some efficiency there: We know our way around.

One of the things that I'm noticing is that y'all keep stressing not "having to deal with XML/HTML stuff". And I'm sure you've had a lot of requests along that line. But I'd like to offer a different point of view. I spent a lot of time and effort getting good at dealing with that stuff, using the tools available on a system running SVT, meaning mostly TCL (but also some Visual Basic). Now, instead of being able to leverage that expertise--and code!--I have to rethink it all in java...not a language I've done a lot in. Sometimes it almost seems like you're trying to make it more difficult for existing customers, or at least their existing expertise. I know that's not true, but there are a lot of scripting and programming languages out there (and I actually know more than a few of them), and limiting the SDK to Java means yet one more thing I have to deal with to get things done.

As for my use case--can Derby even handle 100 Gb of data? If I don't go and cleanup my report files every week or two, it already kind of grinds to a halt. I'd hate to see it get that huge.

Here's another idea. The ability to use an external database from within iTest, for both read and write, would be really cool. For example, my headless regression, which is running 24/7, would write to a MySQL database. When a test goes kablooie, anyone with access to the database server could load the report into their copy of iTest and troubleshoot it. That would be pretty darn nifty!

0 ·

PaulD dclaar commented · Apr 23, 2009 at 11:17 PM

Yes. Sweet idea. Definitely consider that.

Regarding Java "versus" XML, and other scripting languages, this is certainly a difficulty. It's hard to find something that satisfies everyone. If I took a poll right now of our users, it would be interesting to see what the responses would look like to the question, "If iTest allowed you to extend it using some standard programming language, what would you like that to be?" (Hey, everyone else reading this, please feel free to chime in and answer!) I predict that we'll hear a lot of scripting languages (Tcl, Perl, Python, Ruby, Javascript, vbscript), some programming languages (VB, C, C#, C++, Java), and some neutral file formats (like XML, XSL, CSV, etc.) As you can tell, iTest has largely focused on three pieces: Java/Eclipse for programming, Tcl for scripting and procedural syntax, and XML for persistence and post-processing. It doesn't mean that you can't integrate others into this fabric. But we're certainly "defaulting" to these in the appropriate areas.

What you properly raise is that we use embedded databases and so SQL also is part of the iTest story but hasn't been exposed much in iTest. (This is changing a bit now with the new database tool that is being released in 3.4.) Perhaps it is time to make the actual database use in iTest itself a more visible, extensible, part of the product. Good conversation.

0 ·

dclaar PaulD commented · Apr 24, 2009 at 04:06 PM

Back to the original topic of this posting: How does all this play into memory usage? To use itestcli, I must specify a workspace. If I'm running more than one itestcli instances against the same workspace, do they each get their own memory, or do they share? Does this answer change if the GUI is up? Yep, I ran out of memory last night, and all my nightly runs appear to have hung.

Addendum: Wow: Thousands of errors, many of them "Test report Database task threw an exception"...what a mess.

Message Edited by dclaar on 04-24-2009 09:13 AM

0 ·

KumarS dclaar commented · Apr 24, 2009 at 04:16 PM

One can have only one iTest process per workspace. I think I mentioned before in some other post that itestcli is simply a little shell of a program which sends requests to iTest to do the job. So if you have multiple instances of itestcli running on a single workspace, behind the scenes, they are talking to a single instance of iTest which has locked itself to that workspace.

The answer is same whether GUI is up or not. GUI will take slightly more memory to render itself - but on the whole, majority of the memory is used by background threads (i.e. executions in this case).

So yes - if you are running "x" number of itestcli processes against an iTest locked to a workspace, you are creating "x" execution engines in iTest and therefore there is more memory usage. You will have tune the "-Xmx" figure in iTest.ini based on what your requirements are.

0 ·

Show more comments

Answer 2 · 2009-04-22T17:06:40Z

AdamB answered Apr 22, '09 AdamB posted Apr 22, '09

One option would be to use the gui flag for itestcli, this would cause it to always open up the gui, (needed for web and swing tests). Then you could just go on to the machine and look at it.

The other option would be to make sure that itestcli had completed, then you should be able to open up the UI.

This stems from the limitation that only one instance of iTest can point at a given workspace.

10 |950

Attachments: Up to 2 attachments (including images) can be used with a maximum of 512.0 KiB each and 1.0 MiB total.

question