JBOSS Introduction
JBoss, is an application server authored by JBoss, now developed by Red Hat, Red Hat that provides support for the JBoss open source application server program and related services marketed under the JBoss Enterprise Middleware Suite (JEMS) brand. The JBoss applications server is a J2EE platform for developing and deploying enterprise Java applications, Web applications and services, and portals. J2EE allows the use of standardized modular components and enables the Java platform to handle many aspects of programming automatically.
Common issue in JBoss performance are
JBoss Performance tuning involves also tuning the environment on which jBoss is run, we will start discussing about JVM settings and OS settings on which JBoss can produce best results. Then we'll see some specific JBoss config settings-
Tune the garbage collector –
One strength of the J2SE platform is that it shields the developer from the complexity of memory allocation . However, once garbage collection is the principal bottleneck, it is worth understanding some aspects of this hidden implementation An object is considered garbage when it can no longer be reached from any pointer in the running program. The most straightforward garbage collection algorithms simply iterate over every reachable object. Any objects left over are then considered garbage. The time this approach takes is proportional to the number of live objects.
Description of young ,tenured and permanent generation are listed below in table :-
Use the the command line option -verbose:gc causes information about the heap and garbage collection to be printed at each collection.
For example:-
here is output from a large server application:demonstrated that an application that spends 10% of its time in garbage collection can lose 75% of its throughput when scaled out to 32 processors.
Use Sever VM-
The server JVM is better suited to longer running applications. To enable it simply set the -server option on the command line.
Garbage collection-
There are two categories of garbage collection
->Minor - cleans out the young generation only
-> Major - otherwise know as a full collection cleans out the tenured and young generations
Before a minor collection the JVM looks at the tenured generation and determines if there's ample space to hold any objects from the young generation that overflow into the tenured generation, if not, then it performs a collection on the tenured generation first, this is known as a full collection, the tenured generation is also compacted moving all the objects to one end of the tenured generation space.
There are two collector types
Serial collectorl - pauses the Java application until the collection has finished
Concurrent collector - performs most of the collection while the Java application continues to run, only two shorts phases causes the application to pause.
By default if the server has multiple processors and at least 1GB of memory the JVM turns on UseParallelGC automatically for the serial collector, the JVM uses one collector thread for every processor on the server. The arguments that you can use are the following
You can also adjust the number of threads using the -XXParallelGCThreads option.
To gather garbage collection data you can use a number of parameters
here is a Java program (but you can use what ever you want) that you can use to put the data and into a CSV file that excel can chart.
For Log analyzer –
Resulting file
60176,56377,0.0050069
60473,56675,0.0049446
60771,13067,0.1216777
Plot the excel graph
1. Open the CSV file
2. Select the first column only
3. Click Insert, on the subsequent drop-down menu, select Chart to display the Chart wizard
4. Choose XY (scatter), click next
5. In steps 2 of the chart wizard click series tab, then click add under the Series list box. Type the text after in the name field and in the Y values field select the data in the second column
6. Repeat the pervious step for the third column using Time as the name Click Next
7. on step 3 fill out the fields as desired and click next
The type of graph you are looking for is one that starts performing garbage collections but then progresses into a smooth line (see below diagram).
Set -Xms and -Xmx to the same value -
By default, the virtual machine grows or shrinks the heap at each collection to try to keep the proportion of free space to live objects at each collection within a specific range.Setting -Xms and -Xmx to the same value. This increase predictability by removing the most important sizing decision from the virtual machine.
There are a number of JVM heap sizing arguments that you can supply-
Don't use Huge heaps, use a cluster-
More JVMs/smaller heaps can outperform fewer JVMs/Larger Heaps. So instead of huge heaps, use additional server nodes. Set up a JBoss cluster and balance work between nodes.
Don't choose an heap larger then 70% of your OS memory-
Choose a maximum heap size not more then 70% of the memory to avoid excessive page faults and thrashing.
Tune the hip ratio-
This is one of most important tuning factor: the heap ratio. The heap ratio specifies how the amount of the total heap will be partitionedBy decreasing the space in the young generation and putting that space into the tenured generation (a value of NewRatio larger than the default
value was chosen), there was enough room in the tenured generation to hold all the long lived data and also space to support minor collections.
What is the recommeded heap ratios ? Set the tenured generation to be approximately two times the size of the young generation.
Monitor the free memory with monitors and snapshots
You’ve been using the JMX Console to access your MBeans. In this example we’ll look at MBeans through the Web Console, which adds a few new twists to what you’ve seen so far.
The Web Console is found in the home page just under the JMX-Console. Enter it !Now Expand the System icon and then the JMX MBeans tree.
jboss.system domain: expand it and look for the MBean jboss.system:type=ServerInfo.
Beneath this Mbean you can find all the monitoring options available: you should be able to right click on in and get a few menu options:
Now Supposing you want to draw a graph of the attribute FreeMemory, simply right click onFreeMemory and you'll see a real-time graph of your Free Memory.
JBoss gives you the ability to capture data not only in real time but also over regular intervals.
You can start or stop the data collection process according to your needs. Then you can return later to analyze the collected values.
To create a snapshot open the Web Console and point to the JMX MBean you want to monitor.
Then choose the Create Snapshot menu item on the specific attribute you want to monitor.
You’ll be presented with a simple configuration form. You only need to enter the measurement time period. This is the time between measurements, in milliseconds. To monitor the value every 100ms enter
100 in the corresponding text field and click the Create button.
To start the snapshot of memory usage, click the Start Snapshot button.
Wait a few seconds for JBoss to collect measurements. Now you should see your snapshot available in the upper Monitoring Menu:
Ok, now click on the Snapshot : you might analyze it graphically or display the collected data.
Tune the operating system –
Each operating system sets default tuning parameters differently. For Windows platforms, the default settings are usually sufficient. However, the UNIX and Linux operating systems usually need to be tuned appropriately.
Check the following TCP parameters with yours sysadmin:
/dev/tcp-tcp_time_wait_interval
/dev/tcp-tcp_conn_req_max_q
/dev/tcp-tcp_conn_req_max_q0
/dev/tcp-tcp_ip_abort_interval
/dev/tcp-tcp_keepalive_interval
/dev/tcp-tcp_rexmit_interval_initial
/dev/tcp-tcp_rexmit_interval_max
/dev/tcp-tcp_rexmit_interval_min
/dev/tcp-tcp_smallest_anon_port
/dev/tcp-tcp_xmit_hiwat
/dev/tcp-tcp_recv_hiwat
/dev/ce-instance
/dev/ce-rx_intr_time
Tip: Use the netstat -s -P tcp command to view all available TCP parameters.
Set TCP-related tuning parameters using the ndd command
Example: ndd -set /dev/tcp tcp_conn_req_max_q 16384
Tune /etc/system filesystem
Each socket connection to JBoss consumes a file descriptor. To optimize socket performance,
you may need to configure your operating system to have the appropriate number of file descriptors.
Since in Linux everything is a file, check the file-max parameter
cat /proc/sys/fs/file-max
set fs.file-max=102642 into /etc/sysctl.conf
Raise ulimit with /etc/limits.conf (or ulimit -n for current session)
Increase default socket send/receive buffer
Sysctl –w net.core.rmem_default=262144
(default socket receive buffer)
sysctl -w net.core.wmem_default=262144
(default socket send buffer)
sysctl -w net.core.rmem_max=262144
(max socket receive buffer)
sysctl -w net.core.wmem_max=262144
(max socket send buffer size)
Optimize MTU. The TCP maximum transfer unit is 1512 on the Internet. If you are sending larger packets it's a good idea to increase MTU size in order to reduce packet fragmentation (especially if you have a slow network)
vi /etc/sysconfig/network-scripts/ifcfg-xxx (eth0 for instance)
– add "MTU=9000" (for gigabit ethernet)
– restart the interface (ifdown eth0;ifup eth0)
Use Big Memory Pages
Default page size is 4KB (usually too small!)
Check page size with:
$ cat /proc/meminfo
If you see "HugePage_Total," "HugePages_Free" and "Hugepagesize", you can apply this optimization
Here's how to do it (2GB Heap Size Example)
$ echo 2147483647 > /proc/sys/kernel/shmmax
$ echo 1000 > /proc/sys/vm/nr_hugepages
In Sun's JVM, add this flag: XX:+UseLargePages
Ø Lots of Requests ? check JBoss thread pool –
JBoss thread pool is defined into conf/jboss-service.xml
01.<mbeancode="org.jboss.util.threadpool.BasicThreadPool"
02.name="jboss.system:service=ThreadPool">
03.<attributename="Name">JBoss System Threads</attribute>
04.<attributename="ThreadGroupName">System Threads</attribute>
05.<attributename="KeepAliveTime">60000</attribute>
06.<attributename="MaximumPoolSize">10</attribute>
07.<attributename="MaximumQueueSize">1000</attribute>
08.<attributename="BlockingMode">run</attribute>
09.</mbean>
For most applications this defaults will just work well, however if you are running an application with issues lots of requests to jboss (such as EJB invocations) then monitor your thread pool. Open the Web Console and look for the MBean jboss.system:service=ThreadPool.
Start a monitor on the QueueSize parameter. Have you got a QueueSize which reaches MaximumPoolSize ? then probably you need to set a higher MaximumPoolSize pool size attribute Watchout! Speak at first with your sysadmin and ensure that the CPU capacity support the increase in threads.
Ø Check the Embedded web container
JBoss supports connectors for http, https, and ajp. The configuration file is server.xml and it's deployed in the root of JBoss web container (In JBoss 4.2.0 it's: "JBOSS_HOME\server\default\deploy\jboss-web.deployer")
1.<Connectorport="8080"address="${jboss.bind.address}"
2.maxThreads="250"maxHttpHeaderSize="8192"
3.emptySessionPath="true"protocol="HTTP/1.1"
4.enableLookups="false"redirectPort="8443"acceptCount="100"
5.connectionTimeout="20000"disableUploadTimeout="true"/>
The underlying HTTP connector of JBoss needs to be fine tuned for production settings. The important parameters are:
maxThreads - This indicates the maximum number of threads to be allocated for handling client HTTP requests. This figure corresponds to the concurrent users that are going to access the application. Depending on the machine configuration, there is a physical limit beyond which you will need to do clustering.
acceptCount - This is the number of request threads that are put in request queue when all available threads are used.When this exceeds, client machines get a request timeout response.
compression - If you set this attribute to “force”, the content will be compressed by JBoss and will be send to browser. Browser will extract it and display the page on screen. Enabling compression can substantially reduce bandwidth requirements of your application.
So how do you know if it's necessary to raise your maxThreads number ? again open the web console and look for the MBean jboss.web:name=http-127.0.0.1-8080,type=ThreadPool. The key attribute is currentThreadsBusy. If it's about 70-80% of the the maxThreads you should
consider raising the number of max thread.
Ø Turn off JSP Compilation in production
JBoss application server regularly checks whether a JSP requires compilation to a servlet before executing a JSP. In a production server, JSP files won’t change and hence you can configure the settings for increased performance.
Open the web.xml in deploy/jboss-web.deployer/conf folder. Look for the jsp servlet in the file and modify the following XML fragment as given below:
view source
print?
1.<init-param>
2. <param-name>development</param-name>
3. <param-value>false</param-value>
4.</init-param>
5.<init-param>
6. <param-name>checkInterval</param-name>
7. <param-value>300</param-value>
8.</init-param>
Ø Have you got readonly Entity Beans ? tell it to JBoss
JBoss uses RMI for EJB communication and by default creates a single thread for every incoming request.When the number of requests is very large this could be a bottleneck. However you can switch from the standard jrmp service invoker to the pool invoker.
How to do it ? open standardjboss.xml and find the following fragment:
<invoker-mbean>jboss:service=invoker,type=jrmp</invoker-mbean>
If you want to change the default attributes of your pool then open jboss-service.xml
01.<mbeancode="org.jboss.invocation.pooled.server.PooledInvoker"
02.name="jboss:service=invoker,type=pooled">
03. <attributename="NumAcceptThreads">1</attribute>
04. <attributename="MaxPoolSize">300</attribute>
05. <attributename="ClientMaxPoolSize">300</attribute>
06. <attributename="SocketTimeout">60000</attribute>
07. <attributename="ServerBindAddress">${jboss.bind.address}</attribute>
08. <attributename="ServerBindPort">4445</attribute>
09. <attributename="ClientConnectAddress">${jboss.bind.address}</attribute>
10. <attributename="ClientConnectPort">0</attribute>
11. <attributename="ClientRetryCount">1</attribute>
12. <attributename="EnableTcpNoDelay">false</attribute>
13.</mbean>
There are two key attributes for the PooledInvoker in regards to how many threads are used in processing requests. The first is the NumAcceptThreads attribute. The value for this attribute will determine how many threads are created to listen for incoming requests. These threads will be the ones that call the accept() , method of the server socket (which is a blocking call and will wait there till data is received on the network interface for the server socket).
The MaxPoolSize is the other key factor: it's the size of the pool containing the ServerThreads .How can MaxPoolSize become a bottleneck ? if the accept thread
can not get a worker thread from the pool and the pool size has reached the MaxPoolSize value, it will wait for one to become available (instead of creating a new one)
Ø Have you got readonly Entity Beans ? tell it to JBoss
JBoss offers a way to handle this situation by defining either an entire EJB as being "read-only" or simply as a subset of its methods. When accessing a read-only method (or EJB), while JBoss still prevents concurrent access to the same bean instance, the bean will not be enrolled in the transaction and will not be locked during the whole transaction lifetime. Consequently, other transactions can directly use it for their own work.
view source
print?
01.<enterprise-beans>
02.<entity>
03.<ejb-name>MyEntity</ejb-name>
04.<method-attributes>
05.<method>
06.<method-name>get*</method-name>
07.<read-only>true</read-only>
08.</method>
09.<method-attributes>
10.</entity>
11.</enterprise-beans>
Ø Disable the hot deployer in production-
For performance reason you might want to disable JBoss hot deployment's feature.
Simply open the conf/jboss-service.xml file and find the line:
<!-- A flag to disable the scans -->
<attribute name="ScanEnabled">true</attribute>
Simply change to false and hot deployment will be disabled.
If on the other hand you simply want to reduce scan polling time, look for a few lines before:
<!-- Frequency in milliseconds to rescan the URLs for changes -->
<attribute name="ScanPeriod">5000</attribute>
This will set the scan time (in milliseconds)<
Ø Configure the EJB container to use cache, when possible-
If the EJB container has exclusive access to the persistent store, it doesn’t need to synchronize the in-memory bean state from the persistent store at the beginning of each transaction.
So you could activate the so-called Commit-A option that caches entity bean state between transactions. In order to activate this option :
view source
print?
01.<jboss>
02. <enterprise-beans>
03.
04. <container-configurations>
05. <container-configurationextends=
06. "Standard CMP 2.x EntityBean">
07. <container-name>CMP 2.x and Cache</container-name>
08. <commit-option>A</commit-option>
09. </container-configuration>
10. </container-configurations>
11.
12. <entity>
13. <ejb-name>MyEntity</ejb-name>
14. <configuration-name
15. CMP 2.x and Cache</configuration-name>
16. <method-attributes>
17. <method>
18. <method-name>get*</method-name>
19. <read-only>true</read-only>
20. </method>
21. <method-attributes>
22. </entity>
23.</jboss>
Ø Use Cache invalidation in a Cluster for Commit Option A
Commit option A can boost your Entity Bean but what happens when running in a cluster ? in a cluster configuration more than one JBoss node will access the same database. Furthermore, they will not only read data, but may also update the db store.Consequently, we now have as many points of write access to the database as we have JBoss instances in the cluster.
For these scenarios, JBoss incorporates a handy tool: the cache invalidation framework. It provides automatic invalidation of cache entries in a single node or across a cluster of JBoss instances. As soon as an entity bean is modified on a node, an invalidation message is automatically sent to all related containers in the cluster and the related entry is removed from the cache. The next time the data is required by a node, it will not be found in cache, and will be reloaded from the database.
In order to activate it, add to your Entity Bean the cache-invalidation tag.
view source
print?
01.<entity>
02.<ejb-name>MyEntity</ejb-name>
03.<configuration-name>
04.Standard CMP 2.x with cache invalidation
05.</configuration-name>
06.<method-attributes>
07.<method>
08.<method-name>get*</method-name>
09.<read-only>true</read-only>
10.</method>
11.<method-attributes>
12.<cache-invalidation>True</cache-invalidation>
13.</entity>
JBoss maintains a pool of database connections which are used by the application, in the *-ds.xml file you can specify information about the pool size,
try not to set a too high value as each connection uses systems resources which are wasted if you are not using them.
Option
Default
Description
<min-pool-size>
0
The minimum number of connections maintained to the database
<max-pool-size>
20
The maximum number of connections maintained to the database
<blocking-timeout-millis>
30 sec
The amount of time a thread waits on a connection if all connections are in use and the maximum connections have been allocated
<idle-timeout-minutes>
0
The amount of time the application server waits before deallocating a connection that's no longer needed
There is a way to monitor the database connection pool usage, when a datasource is deployed server creates three MBeans, one of which manages the connection pool shown in table.
connection pool manager MBean
Jboss.jca:name=<dsname>,service=ManagedConnectionPool
Key connection pool MBean properties
ConnectionCount
The number of connection to the database
AvailableConnectionCount
The number of database connections not allocated to a request
MaxConnectionInUseCount
The largest number of database connections ever allocated to requests
InUseConnectionCount
The number of database connections allocated to requests
ConnectionCreatedCount
The total number of connections created by the application server, this this becomes to high then increase <min-pool-size> and <idle-timeout-minutes>
ConnectionDestroyCount
The total number of connections closed by the application server, this this becomes to high then increase <min-pool-size> and <idle-timeout-minutes>
Ø Synchronize at commit time when possible
The sync-on-commit-only element configures a performance optimization that will cause entity bean state to be synchronized with the database only at commit time. Normally the state of all the beans in a transaction would need to be synchronized when an finder method is called or when an remove method is called :
view source
1.<container-configuration>
2.<container-name>Standard Pessimistic CMP 2.x EntityBean</container-name>
3.<call-logging>false</call-logging>
4.<invoker-proxy-binding-name>entity-pooled-invoker</invoker-proxy-binding-name>
5.<sync-on-commit-only>true</sync-on-commit-only>
6.....
7.</container-configuration>
The server/xxx/deployers/jbossweb.deployer/web.xml file contains initialization parameters for the JSP servlet, We have listed some of the parameters that you can change to increase performance
Parameter
Default
Description
development
True
If true then the application server checks to see if the JSP has been updated using the modification text interval, if false changed JSPs are compiled in the background using check interval
checkInterval
0
The number of seconds between checks to see if any JSPs need to be recompiled, if 0 the JSPs are never checked for changes or recompiled Used when development is false.
modificationTestInterval
4
The number of seconds to wait before the JSP is checked for updates, if 0 the JSP is checked each time its accessed. Used when development is true
genStrAsCharArray
False
Generates strings as character arrays, under some circumstances can improve performance
trimSpaces
False
Removes extraneous white space from the resulting HTML text, decreasing the size of the response sent back to the client.
Ø Use Prepared Statement Cache
In JBoss, by default,prepared statements are not cached. To improve performance one can configure a prepared statement cache of an arbitrary size. You can use in your -ds.xml file the <prepared-statement-cache-size> : this is the number of prepared statements per connection to be kept open and reused in subsequent requests
Ø Remove services you don't need
JBoss ships with lots of services, however you'll seldom need to use them all. The service is usually deployed as *-service.xml under the deploy directory. Sometimes it's also deployed as .sar/.rar archive. In order to remove the service, remove the file in the "Server/deploy" column. If needed remove also the relative libs stated under "Server/lib"
Servizio
Server/deploy
Server/lib
Mail service
mail-service.xml
mail-plugin.jar, mail.jar,activation.jar
Cache invalidation service
cache-invalidation-service.xml
J2EE client deployer service
client-deployer-service.xml
Hibernate HAR support
hibernate-deployer-service.xml
jboss-hibernate.jar, hibernate2.jar, cglib-full-2.0.1.jar, odmg-3.0.jar
HSQL DB
hsqldb-ds.xml
hsqldb-plugin.jar, hsqldb.jar
Default JMS Service
jms folder
jbossmq.jar
HTTP Invoker (tunnels RMI through HTTP)
http-invoker.sar
XA Datasources
jboss-xa-jdbc.rar
JMX Console
jmx-console.war
Web Console
management/web-console.war
JSR-77
management/console-mgr.sar
Monitoring mail alerts
monitoring-service.xml
jboss-monitoring.jar
Schedule Manager
schedule-manager-service.xml
scheduler-plugin.jar, scheduler-plugin-example.jar
Sample Schedule service
scheduler-service.xml
If you are removing a core JBoss service like JMS or EAR Deployer then you need to remove it also from the jboss-service.xml :
view source
print?
="org.jboss.management.j2ee.LocalJBossServerDomain"
02.name="jboss.management.local:j2eeType=J2EEDomain,name=Manager">
03.<attributename="MainDeployer">jboss.system:service=MainDeployer</attribute>
04.<attributename="SARDeployer">jboss.system:service=ServiceDeployer</attribute>
05.<attributename="EARDeployer">jboss.j2ee:service=EARDeployer</attribute>
06.<attributename="EJBDeployer">jboss.ejb:service=EJBDeployer</attribute>
07.<attributename="RARDeployer">jboss.jca:service=RARDeployer</attribute>
08.<attributename="CMDeployer">jboss.jca:service=ConnectionFactoryDeployer</attribute>
09.<attributename="WARDeployer">jboss.web:service=WebServer</attribute>
10.<attributename="CARDeployer">jboss.j2ee:service=ClientDeployer</attribute>
11.<attributename="MailService">jboss:service=Mail</attribute>
12.<attributename="JMSService">jboss.mq:service=DestinationManager</attribute>
13.<attributename="JNDIService">jboss:service=Naming</attribute>
14.<attributename="JTAService">jboss:service=TransactionManager</attribute>
15.<attributename="UserTransactionService">jboss:service=ClientUserTransaction</attribute>
16.<attributename="RMI_IIOPService">jboss:service=CorbaORB</attribute>
17.</mbean>
Simply comment the attribute relative to the unwanted service
Ø Tell log4j to shut up !
Wel not really anyway Log4j uses a valuable amount of time/CPU so you had better remove unnecessary logs, for example :
Remove logs from Console
Comment the following line in log4j.xml in order to remove logs on the Console:
view source
print?
1.<root>
2. <!-- <appender-ref ref=CONSOLE"></appender-ref> -->
3. <appender-refref="FILE"></appender-ref>
4. </root>
remove also the relative appender:
<appender name="CONSOLE" class="org.apache.log4j.ConsoleAppender">
....
</appender>
Raise log priority Consider raising the log level to the highest level possible in production. Here only error logs are written:
view source
print?
1.<root>
2. <priorityvalue="ERROR"></priority>
3. <!--<appender-ref ref="CONSOLE"></appender-ref> -->
4. <appender-refref="FILE"></appender-ref>
5. </root>
References
http://www.mastertheboss.com/jboss-performance/jboss-performance-tuning-part-1
http://www.mastertheboss.com/jboss-performance/jboss-performance-tuning-part-2
adisk.co.uk/html_docs/java_app/jboss5/jboss5_tuning.htm
http://blog.yes24.eu/wp-content/uploads/2011/02/JB_JEAP4_3_PerformanceTuning_wp.pdf
JBoss, is an application server authored by JBoss, now developed by Red Hat, Red Hat that provides support for the JBoss open source application server program and related services marketed under the JBoss Enterprise Middleware Suite (JEMS) brand. The JBoss applications server is a J2EE platform for developing and deploying enterprise Java applications, Web applications and services, and portals. J2EE allows the use of standardized modular components and enables the Java platform to handle many aspects of programming automatically.
Common issue in JBoss performance are
- Connection pooling(to maximize throught[put)
- Thread pooling
- Object and component pools
- Logging
- Caching
JBoss Performance tuning involves also tuning the environment on which jBoss is run, we will start discussing about JVM settings and OS settings on which JBoss can produce best results. Then we'll see some specific JBoss config settings-
Tune the garbage collector –
One strength of the J2SE platform is that it shields the developer from the complexity of memory allocation . However, once garbage collection is the principal bottleneck, it is worth understanding some aspects of this hidden implementation An object is considered garbage when it can no longer be reached from any pointer in the running program. The most straightforward garbage collection algorithms simply iterate over every reachable object. Any objects left over are then considered garbage. The time this approach takes is proportional to the number of live objects.
Description of young ,tenured and permanent generation are listed below in table :-
Use the the command line option -verbose:gc causes information about the heap and garbage collection to be printed at each collection.
For example:-
here is output from a large server application:demonstrated that an application that spends 10% of its time in garbage collection can lose 75% of its throughput when scaled out to 32 processors.
Use Sever VM-
The server JVM is better suited to longer running applications. To enable it simply set the -server option on the command line.
Garbage collection-
There are two categories of garbage collection
->Minor - cleans out the young generation only
-> Major - otherwise know as a full collection cleans out the tenured and young generations
Before a minor collection the JVM looks at the tenured generation and determines if there's ample space to hold any objects from the young generation that overflow into the tenured generation, if not, then it performs a collection on the tenured generation first, this is known as a full collection, the tenured generation is also compacted moving all the objects to one end of the tenured generation space.
There are two collector types
Serial collectorl - pauses the Java application until the collection has finished
Concurrent collector - performs most of the collection while the Java application continues to run, only two shorts phases causes the application to pause.
By default if the server has multiple processors and at least 1GB of memory the JVM turns on UseParallelGC automatically for the serial collector, the JVM uses one collector thread for every processor on the server. The arguments that you can use are the following
You can also adjust the number of threads using the -XXParallelGCThreads option.
To gather garbage collection data you can use a number of parameters
here is a Java program (but you can use what ever you want) that you can use to put the data and into a CSV file that excel can chart.
For Log analyzer –
Code:
import java.io.*;
import java.util.regex.*;
public class Analyzer {
public static void main(String[] args) throws Exception {
InputStream fin = new FileInputStream(args[0]);
int iSize = fin.available();
byte mvIn[] = new byte[iSize];
fin.read(mvIn, 0, iSize);
fin.close();
String strText = new String(mvIn);
PrintStream fout = new PrintStream(new FileOutputStream(args[0] + ".cvs"));
fout.println("Before,After,Seconds");
Pattern p = Pattern.compile("\\[?:Full |)GC (\\d*)K->(\\d*)K\\(\\d*K\\), ([\\d.]*) secs\\]");
Matcher m = p.matcher(strText);
while (m.find())
fout.println(m.group(1) + "," + m.group(2) + "," + m.group(3));
fout.close();
}
}Resulting file
60176,56377,0.0050069
60473,56675,0.0049446
60771,13067,0.1216777
Plot the excel graph
1. Open the CSV file
2. Select the first column only
3. Click Insert, on the subsequent drop-down menu, select Chart to display the Chart wizard
4. Choose XY (scatter), click next
5. In steps 2 of the chart wizard click series tab, then click add under the Series list box. Type the text after in the name field and in the Y values field select the data in the second column
6. Repeat the pervious step for the third column using Time as the name Click Next
7. on step 3 fill out the fields as desired and click next
The type of graph you are looking for is one that starts performing garbage collections but then progresses into a smooth line (see below diagram).
Set -Xms and -Xmx to the same value -
By default, the virtual machine grows or shrinks the heap at each collection to try to keep the proportion of free space to live objects at each collection within a specific range.Setting -Xms and -Xmx to the same value. This increase predictability by removing the most important sizing decision from the virtual machine.
There are a number of JVM heap sizing arguments that you can supply-
Don't use Huge heaps, use a cluster-
More JVMs/smaller heaps can outperform fewer JVMs/Larger Heaps. So instead of huge heaps, use additional server nodes. Set up a JBoss cluster and balance work between nodes.
Don't choose an heap larger then 70% of your OS memory-
Choose a maximum heap size not more then 70% of the memory to avoid excessive page faults and thrashing.
Tune the hip ratio-
This is one of most important tuning factor: the heap ratio. The heap ratio specifies how the amount of the total heap will be partitionedBy decreasing the space in the young generation and putting that space into the tenured generation (a value of NewRatio larger than the default
value was chosen), there was enough room in the tenured generation to hold all the long lived data and also space to support minor collections.
What is the recommeded heap ratios ? Set the tenured generation to be approximately two times the size of the young generation.
Monitor the free memory with monitors and snapshots
You’ve been using the JMX Console to access your MBeans. In this example we’ll look at MBeans through the Web Console, which adds a few new twists to what you’ve seen so far.
The Web Console is found in the home page just under the JMX-Console. Enter it !Now Expand the System icon and then the JMX MBeans tree.
jboss.system domain: expand it and look for the MBean jboss.system:type=ServerInfo.
Beneath this Mbean you can find all the monitoring options available: you should be able to right click on in and get a few menu options:
Now Supposing you want to draw a graph of the attribute FreeMemory, simply right click onFreeMemory and you'll see a real-time graph of your Free Memory.
JBoss gives you the ability to capture data not only in real time but also over regular intervals.
You can start or stop the data collection process according to your needs. Then you can return later to analyze the collected values.
To create a snapshot open the Web Console and point to the JMX MBean you want to monitor.
Then choose the Create Snapshot menu item on the specific attribute you want to monitor.
You’ll be presented with a simple configuration form. You only need to enter the measurement time period. This is the time between measurements, in milliseconds. To monitor the value every 100ms enter
100 in the corresponding text field and click the Create button.
To start the snapshot of memory usage, click the Start Snapshot button.
Wait a few seconds for JBoss to collect measurements. Now you should see your snapshot available in the upper Monitoring Menu:
Ok, now click on the Snapshot : you might analyze it graphically or display the collected data.
Tune the operating system –
Each operating system sets default tuning parameters differently. For Windows platforms, the default settings are usually sufficient. However, the UNIX and Linux operating systems usually need to be tuned appropriately.
Check the following TCP parameters with yours sysadmin:
/dev/tcp-tcp_time_wait_interval
/dev/tcp-tcp_conn_req_max_q
/dev/tcp-tcp_conn_req_max_q0
/dev/tcp-tcp_ip_abort_interval
/dev/tcp-tcp_keepalive_interval
/dev/tcp-tcp_rexmit_interval_initial
/dev/tcp-tcp_rexmit_interval_max
/dev/tcp-tcp_rexmit_interval_min
/dev/tcp-tcp_smallest_anon_port
/dev/tcp-tcp_xmit_hiwat
/dev/tcp-tcp_recv_hiwat
/dev/ce-instance
/dev/ce-rx_intr_time
Tip: Use the netstat -s -P tcp command to view all available TCP parameters.
Set TCP-related tuning parameters using the ndd command
Example: ndd -set /dev/tcp tcp_conn_req_max_q 16384
Tune /etc/system filesystem
Each socket connection to JBoss consumes a file descriptor. To optimize socket performance,
you may need to configure your operating system to have the appropriate number of file descriptors.
Since in Linux everything is a file, check the file-max parameter
cat /proc/sys/fs/file-max
set fs.file-max=102642 into /etc/sysctl.conf
Raise ulimit with /etc/limits.conf (or ulimit -n for current session)
Increase default socket send/receive buffer
Sysctl –w net.core.rmem_default=262144
(default socket receive buffer)
sysctl -w net.core.wmem_default=262144
(default socket send buffer)
sysctl -w net.core.rmem_max=262144
(max socket receive buffer)
sysctl -w net.core.wmem_max=262144
(max socket send buffer size)
Optimize MTU. The TCP maximum transfer unit is 1512 on the Internet. If you are sending larger packets it's a good idea to increase MTU size in order to reduce packet fragmentation (especially if you have a slow network)
vi /etc/sysconfig/network-scripts/ifcfg-xxx (eth0 for instance)
– add "MTU=9000" (for gigabit ethernet)
– restart the interface (ifdown eth0;ifup eth0)
Use Big Memory Pages
Default page size is 4KB (usually too small!)
Check page size with:
$ cat /proc/meminfo
If you see "HugePage_Total," "HugePages_Free" and "Hugepagesize", you can apply this optimization
Here's how to do it (2GB Heap Size Example)
$ echo 2147483647 > /proc/sys/kernel/shmmax
$ echo 1000 > /proc/sys/vm/nr_hugepages
In Sun's JVM, add this flag: XX:+UseLargePages
Ø Lots of Requests ? check JBoss thread pool –
JBoss thread pool is defined into conf/jboss-service.xml
01.<mbeancode="org.jboss.util.threadpool.BasicThreadPool"
02.name="jboss.system:service=ThreadPool">
03.<attributename="Name">JBoss System Threads</attribute>
04.<attributename="ThreadGroupName">System Threads</attribute>
05.<attributename="KeepAliveTime">60000</attribute>
06.<attributename="MaximumPoolSize">10</attribute>
07.<attributename="MaximumQueueSize">1000</attribute>
08.<attributename="BlockingMode">run</attribute>
09.</mbean>
For most applications this defaults will just work well, however if you are running an application with issues lots of requests to jboss (such as EJB invocations) then monitor your thread pool. Open the Web Console and look for the MBean jboss.system:service=ThreadPool.
Start a monitor on the QueueSize parameter. Have you got a QueueSize which reaches MaximumPoolSize ? then probably you need to set a higher MaximumPoolSize pool size attribute Watchout! Speak at first with your sysadmin and ensure that the CPU capacity support the increase in threads.
Ø Check the Embedded web container
JBoss supports connectors for http, https, and ajp. The configuration file is server.xml and it's deployed in the root of JBoss web container (In JBoss 4.2.0 it's: "JBOSS_HOME\server\default\deploy\jboss-web.deployer")
1.<Connectorport="8080"address="${jboss.bind.address}"
2.maxThreads="250"maxHttpHeaderSize="8192"
3.emptySessionPath="true"protocol="HTTP/1.1"
4.enableLookups="false"redirectPort="8443"acceptCount="100"
5.connectionTimeout="20000"disableUploadTimeout="true"/>
The underlying HTTP connector of JBoss needs to be fine tuned for production settings. The important parameters are:
maxThreads - This indicates the maximum number of threads to be allocated for handling client HTTP requests. This figure corresponds to the concurrent users that are going to access the application. Depending on the machine configuration, there is a physical limit beyond which you will need to do clustering.
acceptCount - This is the number of request threads that are put in request queue when all available threads are used.When this exceeds, client machines get a request timeout response.
compression - If you set this attribute to “force”, the content will be compressed by JBoss and will be send to browser. Browser will extract it and display the page on screen. Enabling compression can substantially reduce bandwidth requirements of your application.
So how do you know if it's necessary to raise your maxThreads number ? again open the web console and look for the MBean jboss.web:name=http-127.0.0.1-8080,type=ThreadPool. The key attribute is currentThreadsBusy. If it's about 70-80% of the the maxThreads you should
consider raising the number of max thread.
Ø Turn off JSP Compilation in production
JBoss application server regularly checks whether a JSP requires compilation to a servlet before executing a JSP. In a production server, JSP files won’t change and hence you can configure the settings for increased performance.
Open the web.xml in deploy/jboss-web.deployer/conf folder. Look for the jsp servlet in the file and modify the following XML fragment as given below:
view source
print?
1.<init-param>
2. <param-name>development</param-name>
3. <param-value>false</param-value>
4.</init-param>
5.<init-param>
6. <param-name>checkInterval</param-name>
7. <param-value>300</param-value>
8.</init-param>
Ø Have you got readonly Entity Beans ? tell it to JBoss
JBoss uses RMI for EJB communication and by default creates a single thread for every incoming request.When the number of requests is very large this could be a bottleneck. However you can switch from the standard jrmp service invoker to the pool invoker.
How to do it ? open standardjboss.xml and find the following fragment:
<invoker-mbean>jboss:service=invoker,type=jrmp</invoker-mbean>
If you want to change the default attributes of your pool then open jboss-service.xml
01.<mbeancode="org.jboss.invocation.pooled.server.PooledInvoker"
02.name="jboss:service=invoker,type=pooled">
03. <attributename="NumAcceptThreads">1</attribute>
04. <attributename="MaxPoolSize">300</attribute>
05. <attributename="ClientMaxPoolSize">300</attribute>
06. <attributename="SocketTimeout">60000</attribute>
07. <attributename="ServerBindAddress">${jboss.bind.address}</attribute>
08. <attributename="ServerBindPort">4445</attribute>
09. <attributename="ClientConnectAddress">${jboss.bind.address}</attribute>
10. <attributename="ClientConnectPort">0</attribute>
11. <attributename="ClientRetryCount">1</attribute>
12. <attributename="EnableTcpNoDelay">false</attribute>
13.</mbean>
There are two key attributes for the PooledInvoker in regards to how many threads are used in processing requests. The first is the NumAcceptThreads attribute. The value for this attribute will determine how many threads are created to listen for incoming requests. These threads will be the ones that call the accept() , method of the server socket (which is a blocking call and will wait there till data is received on the network interface for the server socket).
The MaxPoolSize is the other key factor: it's the size of the pool containing the ServerThreads .How can MaxPoolSize become a bottleneck ? if the accept thread
can not get a worker thread from the pool and the pool size has reached the MaxPoolSize value, it will wait for one to become available (instead of creating a new one)
Ø Have you got readonly Entity Beans ? tell it to JBoss
JBoss offers a way to handle this situation by defining either an entire EJB as being "read-only" or simply as a subset of its methods. When accessing a read-only method (or EJB), while JBoss still prevents concurrent access to the same bean instance, the bean will not be enrolled in the transaction and will not be locked during the whole transaction lifetime. Consequently, other transactions can directly use it for their own work.
view source
print?
01.<enterprise-beans>
02.<entity>
03.<ejb-name>MyEntity</ejb-name>
04.<method-attributes>
05.<method>
06.<method-name>get*</method-name>
07.<read-only>true</read-only>
08.</method>
09.<method-attributes>
10.</entity>
11.</enterprise-beans>
Ø Disable the hot deployer in production-
For performance reason you might want to disable JBoss hot deployment's feature.
Simply open the conf/jboss-service.xml file and find the line:
<!-- A flag to disable the scans -->
<attribute name="ScanEnabled">true</attribute>
Simply change to false and hot deployment will be disabled.
If on the other hand you simply want to reduce scan polling time, look for a few lines before:
<!-- Frequency in milliseconds to rescan the URLs for changes -->
<attribute name="ScanPeriod">5000</attribute>
This will set the scan time (in milliseconds)<
Ø Configure the EJB container to use cache, when possible-
If the EJB container has exclusive access to the persistent store, it doesn’t need to synchronize the in-memory bean state from the persistent store at the beginning of each transaction.
So you could activate the so-called Commit-A option that caches entity bean state between transactions. In order to activate this option :
view source
print?
01.<jboss>
02. <enterprise-beans>
03.
04. <container-configurations>
05. <container-configurationextends=
06. "Standard CMP 2.x EntityBean">
07. <container-name>CMP 2.x and Cache</container-name>
08. <commit-option>A</commit-option>
09. </container-configuration>
10. </container-configurations>
11.
12. <entity>
13. <ejb-name>MyEntity</ejb-name>
14. <configuration-name
15. CMP 2.x and Cache</configuration-name>
16. <method-attributes>
17. <method>
18. <method-name>get*</method-name>
19. <read-only>true</read-only>
20. </method>
21. <method-attributes>
22. </entity>
23.</jboss>
Ø Use Cache invalidation in a Cluster for Commit Option A
Commit option A can boost your Entity Bean but what happens when running in a cluster ? in a cluster configuration more than one JBoss node will access the same database. Furthermore, they will not only read data, but may also update the db store.Consequently, we now have as many points of write access to the database as we have JBoss instances in the cluster.
For these scenarios, JBoss incorporates a handy tool: the cache invalidation framework. It provides automatic invalidation of cache entries in a single node or across a cluster of JBoss instances. As soon as an entity bean is modified on a node, an invalidation message is automatically sent to all related containers in the cluster and the related entry is removed from the cache. The next time the data is required by a node, it will not be found in cache, and will be reloaded from the database.
In order to activate it, add to your Entity Bean the cache-invalidation tag.
view source
print?
01.<entity>
02.<ejb-name>MyEntity</ejb-name>
03.<configuration-name>
04.Standard CMP 2.x with cache invalidation
05.</configuration-name>
06.<method-attributes>
07.<method>
08.<method-name>get*</method-name>
09.<read-only>true</read-only>
10.</method>
11.<method-attributes>
12.<cache-invalidation>True</cache-invalidation>
13.</entity>
JBoss maintains a pool of database connections which are used by the application, in the *-ds.xml file you can specify information about the pool size,
try not to set a too high value as each connection uses systems resources which are wasted if you are not using them.
Option
Default
Description
<min-pool-size>
0
The minimum number of connections maintained to the database
<max-pool-size>
20
The maximum number of connections maintained to the database
<blocking-timeout-millis>
30 sec
The amount of time a thread waits on a connection if all connections are in use and the maximum connections have been allocated
<idle-timeout-minutes>
0
The amount of time the application server waits before deallocating a connection that's no longer needed
There is a way to monitor the database connection pool usage, when a datasource is deployed server creates three MBeans, one of which manages the connection pool shown in table.
connection pool manager MBean
Jboss.jca:name=<dsname>,service=ManagedConnectionPool
Key connection pool MBean properties
ConnectionCount
The number of connection to the database
AvailableConnectionCount
The number of database connections not allocated to a request
MaxConnectionInUseCount
The largest number of database connections ever allocated to requests
InUseConnectionCount
The number of database connections allocated to requests
ConnectionCreatedCount
The total number of connections created by the application server, this this becomes to high then increase <min-pool-size> and <idle-timeout-minutes>
ConnectionDestroyCount
The total number of connections closed by the application server, this this becomes to high then increase <min-pool-size> and <idle-timeout-minutes>
Ø Synchronize at commit time when possible
The sync-on-commit-only element configures a performance optimization that will cause entity bean state to be synchronized with the database only at commit time. Normally the state of all the beans in a transaction would need to be synchronized when an finder method is called or when an remove method is called :
view source
1.<container-configuration>
2.<container-name>Standard Pessimistic CMP 2.x EntityBean</container-name>
3.<call-logging>false</call-logging>
4.<invoker-proxy-binding-name>entity-pooled-invoker</invoker-proxy-binding-name>
5.<sync-on-commit-only>true</sync-on-commit-only>
6.....
7.</container-configuration>
The server/xxx/deployers/jbossweb.deployer/web.xml file contains initialization parameters for the JSP servlet, We have listed some of the parameters that you can change to increase performance
Parameter
Default
Description
development
True
If true then the application server checks to see if the JSP has been updated using the modification text interval, if false changed JSPs are compiled in the background using check interval
checkInterval
0
The number of seconds between checks to see if any JSPs need to be recompiled, if 0 the JSPs are never checked for changes or recompiled Used when development is false.
modificationTestInterval
4
The number of seconds to wait before the JSP is checked for updates, if 0 the JSP is checked each time its accessed. Used when development is true
genStrAsCharArray
False
Generates strings as character arrays, under some circumstances can improve performance
trimSpaces
False
Removes extraneous white space from the resulting HTML text, decreasing the size of the response sent back to the client.
Ø Use Prepared Statement Cache
In JBoss, by default,prepared statements are not cached. To improve performance one can configure a prepared statement cache of an arbitrary size. You can use in your -ds.xml file the <prepared-statement-cache-size> : this is the number of prepared statements per connection to be kept open and reused in subsequent requests
Ø Remove services you don't need
JBoss ships with lots of services, however you'll seldom need to use them all. The service is usually deployed as *-service.xml under the deploy directory. Sometimes it's also deployed as .sar/.rar archive. In order to remove the service, remove the file in the "Server/deploy" column. If needed remove also the relative libs stated under "Server/lib"
Servizio
Server/deploy
Server/lib
Mail service
mail-service.xml
mail-plugin.jar, mail.jar,activation.jar
Cache invalidation service
cache-invalidation-service.xml
J2EE client deployer service
client-deployer-service.xml
Hibernate HAR support
hibernate-deployer-service.xml
jboss-hibernate.jar, hibernate2.jar, cglib-full-2.0.1.jar, odmg-3.0.jar
HSQL DB
hsqldb-ds.xml
hsqldb-plugin.jar, hsqldb.jar
Default JMS Service
jms folder
jbossmq.jar
HTTP Invoker (tunnels RMI through HTTP)
http-invoker.sar
XA Datasources
jboss-xa-jdbc.rar
JMX Console
jmx-console.war
Web Console
management/web-console.war
JSR-77
management/console-mgr.sar
Monitoring mail alerts
monitoring-service.xml
jboss-monitoring.jar
Schedule Manager
schedule-manager-service.xml
scheduler-plugin.jar, scheduler-plugin-example.jar
Sample Schedule service
scheduler-service.xml
If you are removing a core JBoss service like JMS or EAR Deployer then you need to remove it also from the jboss-service.xml :
view source
print?
="org.jboss.management.j2ee.LocalJBossServerDomain"
02.name="jboss.management.local:j2eeType=J2EEDomain,name=Manager">
03.<attributename="MainDeployer">jboss.system:service=MainDeployer</attribute>
04.<attributename="SARDeployer">jboss.system:service=ServiceDeployer</attribute>
05.<attributename="EARDeployer">jboss.j2ee:service=EARDeployer</attribute>
06.<attributename="EJBDeployer">jboss.ejb:service=EJBDeployer</attribute>
07.<attributename="RARDeployer">jboss.jca:service=RARDeployer</attribute>
08.<attributename="CMDeployer">jboss.jca:service=ConnectionFactoryDeployer</attribute>
09.<attributename="WARDeployer">jboss.web:service=WebServer</attribute>
10.<attributename="CARDeployer">jboss.j2ee:service=ClientDeployer</attribute>
11.<attributename="MailService">jboss:service=Mail</attribute>
12.<attributename="JMSService">jboss.mq:service=DestinationManager</attribute>
13.<attributename="JNDIService">jboss:service=Naming</attribute>
14.<attributename="JTAService">jboss:service=TransactionManager</attribute>
15.<attributename="UserTransactionService">jboss:service=ClientUserTransaction</attribute>
16.<attributename="RMI_IIOPService">jboss:service=CorbaORB</attribute>
17.</mbean>
Simply comment the attribute relative to the unwanted service
Ø Tell log4j to shut up !
Wel not really anyway Log4j uses a valuable amount of time/CPU so you had better remove unnecessary logs, for example :
Remove logs from Console
Comment the following line in log4j.xml in order to remove logs on the Console:
view source
print?
1.<root>
2. <!-- <appender-ref ref=CONSOLE"></appender-ref> -->
3. <appender-refref="FILE"></appender-ref>
4. </root>
remove also the relative appender:
<appender name="CONSOLE" class="org.apache.log4j.ConsoleAppender">
....
</appender>
Raise log priority Consider raising the log level to the highest level possible in production. Here only error logs are written:
view source
print?
1.<root>
2. <priorityvalue="ERROR"></priority>
3. <!--<appender-ref ref="CONSOLE"></appender-ref> -->
4. <appender-refref="FILE"></appender-ref>
5. </root>
References
http://www.mastertheboss.com/jboss-performance/jboss-performance-tuning-part-1
http://www.mastertheboss.com/jboss-performance/jboss-performance-tuning-part-2
adisk.co.uk/html_docs/java_app/jboss5/jboss5_tuning.htm
http://blog.yes24.eu/wp-content/uploads/2011/02/JB_JEAP4_3_PerformanceTuning_wp.pdf