Have a load test script that runs fine when there's a delay between ramping up the users but has problems and throwing solve variable errors while trying to run even 2 users concurrently. I even tested the variable through the Substitute and add variable process and open script correctly found the data when tested from that screen, but fails with 2 users running concurrently through OLT.
If the solve variable is due to data problem then my guess is the script should fail even when there's a delay between ramping up the users, but that's not the case. Script runs fine without any solve exceptions when there's a delay in ramping up the users.
Using OLT version 9.3 and the application under test is a simple web application with JBoss application server.
Even if the problem only appears to be occurring with concurrent users there is probably a request parameter or request cookie that has not been dynamically substituted. Does this happen before or after a user login?
1. You should look at the response content of the failed vusers HTML. This is the first thing to look at since there may be some indication in that response from your server. This is the reason why the solve is failing, even it is not the root of the problem.
2. After viewing the response page failure, work backwards to examine and compare the HTTP requests of the two virtual users. Compare all request cookies, and request parameters (query string and post data). Any parameter or cookie that is identical for both virtual users is a possible suspect.
3. How many application servers are there? Is there a load balancer between the virtual users and the application server? If so, repeat your test with just one application server running.
Thanks for the helpful reply and sorry for my very late repsonse.
I went through the error logs on the application server and found that only one user is logging in successfully and the second user login is failing and the credentials are being passed on as 'null' and that's the reason for failure. So the second user is even unable to log into the application and that is my next step to see why that's happening. As I mentioned before running with a time delay is fine with the logins but not concurrently, so, not sure where the disconnect is.
As you suggested, I will go through the HTTP requests, response content and request parameters in a detailed way to see why the login is failing.
Yes, there is a load balancer. And the tests are failing even if I hit just one application server after bypassing the load balancer. So, with or without load balancer the result is the same. I went through the script step by step and substituted the dynamic parameters, unable to figure out what I'm missing. Any additional help is appreciated
Post the HTTP request text (cookies and request parameters) of both the login which fails and the one which works when you attempt two concurrent vusers. Either post here or somewhere temporary like http://pastebin.com which is a site for sharing code. Post as private or unlisted if you want, though you will still need to tell us the pastebin link.
Glenn, here are the request headers and response headers for the successful login and failed login. I saved the virtual user display as an attachment but I don't see a way to attach that in these posts. There are few more requests and responses but these are session based ones, so posted only these, but if you want to see all of those I can do that too. Also, I blanked the hostname as I cannot publish it, hope that's ok.
Thanks for the help
Request headers and response headers for successful login
GET /csl/ HTTP/1.1
User-Agent: Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)
Accept: text/html, image/gif, image/jpeg, */*
That is a fairly simple HTTP transaction with a small number of parameters and cookies, which makes it easy to debug. Both users appear to be properly sending unique data. Unfortunately there is no indication in these client logs for the reason of the failure. Only the final failed Response Header is different (no Content-Length), probably as it redirects to some failure page. The Response Content HTML did not show anything helpful?
I know you said you configured your load balancer to use only one web/application server, but can you instead bypass the LB completely while still scripting to only one web server? It is possible that traffic from the same IP address is causing the problem, and the most likely place for that would be in the LB config. It's not common though to only have this type of IP problem with concurrently timed logins
You say you have HTTP logs from the server side? Can you post those for this same pass/fail of two users?
Glenn, I did not configure the LB, I completely bypassed it by directly hitting the app server. I was trying to edit the server log to show only relevant data but still it is around 1800 lines and not sure copying the whole thing here is a good idea. Is there a way to attach a file here or mail it to you?
Yes that is a JBOSS application server level log. JBOSS for web applications cannot be hosted without a HTTP web server, which is typically (by default) a Tomcat web server. We are interested in the HTTP log files from Tomcat.
The log file you posted is a 2nd tier. You need to see the raw communication of the HTTP Client (which is IE, Firefox, OpenScript, OATS agents, etc..) as it is logged by the first tier (the Tomcat HTTP server)
Sorry Glen, we were having a lot of application issues. And also, the Tomcat server was not configured to generate the access logs that have the info you were looking for. Now it is configured to save the access logs and once our application is in a stable state, I will re-run the test and post the data.