loadrunner Application Overview:- Application- Java based...

  • perf-test.com need your contributions to build up a strong repository of performance engineering resources.

C

Chandrakant Singh

Guest
Application Overview:- Application- Java based Web server - Apache App server - Jboss LoadRunner version :- 12.50 Loadrunner protocol used:- HTTP/HTML Browser emulation:- Simulate a new user on each iteration + Clear cache on each iteration Issue:- Response time of all transactions is < 5 seconds till 15 minutes after the ramp up (180 Vusers ~250 hits/second). Response time starts increasing exponentially after some time with the same user load in steady state. Observations :- 1. CPU and memory on both APP and web servers are under threshold 2. No issues observed with Heap utilization on App servers 3. Performance logs on App server shows that none of the transactions took more than a second ( almost every request was served in milliseconds) 4. All queries were processed in milliseconds on the DB end 5. Enabled MOD_STATUS on the Apache web servers and noticed that during the high response time most of the requests are in state "W" SENDING REPLY (Mode of operation) 6. At the time of high response time the issue is reproducible manually in the browser Note:- Login is in init and there are 3 web_url requests with lot of CSS, js and some image files in "EXTRARES" All your comments and suggestions to identify the bottleneck are welcome Thanks
 
Sort by date Sort by votes
Did you check NIc card on the server for full/half duplex? How many LGS are you using and how is health of LGS during ramp up?
 
Upvote 0 Downvote
Add time-taken to your Web logs. Then, as you degrade is this happening uniformly across all mime types or only on components which have off Web server dependencies (app / dB )?
 
Upvote 0 Downvote
Check GC statistics. Such as # of GC major and minor. Time taken for each GC. Heap utilization may not give you complete picture. Also check connection pool configurations, active sessions. Take a thread dump to see if there are any object level deadlocks which may cause your threads to go in wait state.
 
Upvote 0 Downvote
Take 5 thread dumps each at 3 second interval when you see slowness and see if any of the Web container threads are long running. Monitor your application thread pool and DB connection pool and make sure no request is waiting to get a thread and no thread is waiting to get a DB connection. What is your GC overhead , do you know how many minor and major collections are happening per minute. Mostly this should be case of some of the application threads getting stuck somewhere on some backend and not getting released back to the pool. Thread dumps analysis will give a clue. I am suggesting you this as you observe this issue early in your test when you don't restart the application server indicating resources available like threads are stuck somewhere.
 
Upvote 0 Downvote
Thank u for a proper problem statement, never seen such minute details on the issue. I had same kind of issue on sharepoint server SP2013. After 8-10 minutes of ramp up, response time was crossing 18 seconds(usually 2 seconds),after arch.debugging found that appFabric cacheing.it was getting recycled once threshold session limit is crossed. can we have look into how Jboss session & cacheing is working ? *i m just newbie here, just getting into performence engineering. Above was just attemp to share my pointers on issue
 
Upvote 0 Downvote
Thank you all for your valuable suggestions! I am working on all the provided suggestions and will update you once we identify the root cause.
 
Upvote 0 Downvote
Thank you all for your suggestions! We have found the issue that was causing high response time, it was due to siteminder and the application works perfectly fine after the build without siteminder. We have opened a ticket with CA to get help on this issue.
 
Upvote 0 Downvote
You must log in or register to reply here.
Top Bottom
Back