General Tuning Instructions

[charlie]

Below are several suggestions for tuning Wowza Server on your production hardware. If you would like more specific instructions for configuring Wowza Server on your specific hardware configuration, send an email with a detailed description of your hardware (# of CPU/cores, memory, operating system, Java version...) to support@wowzamedia.com:

1. It is generally best to run the most recent Sun Java JDK (http://java.sun.com).
   
   Note: If you are on a Mac with OS X 10.5.x or greater (Leopard) with 64-bit hardware you can run the Java 6 VM. To change your default Java VM in OSX, open the Java preferences application /Applications/Utilities/Java/Java Preferences and drag the Java SE 6 item to the top of the Java Applications list. More information here: Changing the default JVM on OSX Leopard
   
   
2. It is best to run a 64-bit operating system with the 64-bit Java VM. The 64-bit VM enables Java heap sizes greater than 2GB.
   
   Note: Wowza Media Server 2 will run properly on 64-bit Windows platforms.
   
   
3. The most recent development builds can be found here:
   
   http://www.wowzamedia.com/devbuild.html
   
   
4. It is best to run the -server version of the Java VM. The instructions for configuring the server version is described in these two forum threads:
   
   Linux and OSX tuning, running the "server" Java VM (tuning)
   Windows tuning, running the "server" Java VM (tuning)
   
   
5. The default Java heap size is set to 768MB which is great for development but not large enough for high volume production use. If you are running the 64-bit version of the Java VM and have 4GB or more of RAM in your machine, we suggest a Java heap size of between 3000MB to 5000MB. If your are running the 32-bit version of the Java VM or have less than 3GB of RAM in your machine, we suggest a Java heap size of 1500MB. Do not attempt to set a heap size greater than 1800MB when using the 32-bit VM. On many Linux platforms the server will start but will fail with an OutOfMemory error after a short period of time. Memory settings can be adjusted in the following script files (based on platform):
   
   
   • [install-dir]/bin/setenv.sh (Linux, Mac OSX and Solaris)
     
     JAVA_OPTS="-Xmx3000M"
     
     
   • [install-dir]/bin/setenv.bat (Windows)
     
     set JAVA_OPTS=-Xmx3000M
     
     
   • [install-dir]/bin/WowzaMediaServer-Service.conf (Windows)
     
     wrapper.java.additional.1=-Xmx3000M
     
     
6. Garbage collection (GC) tuning in Java is a tricky subject. What works best in one server setup may not work in another. Through trial and error and customer feedback we have several suggest approaches:
   
   Garbage Collection Options:
   
   
   • No Addtional Settings: The default -server tuning can work quite well for many stream situations. So in a lot of cases this tuning works great.
   • The Concurrent Collector: The concurrent collector is designed for applications that prefer shorter garbage collection pauses and that can afford to share processor resources with the garbage collector while the application is running. The suggested additional settings are:
     

     -XX:+UseConcMarkSweepGC -XX:+CMSIncrementalMode -XX:+UseParNewGC

   • Expiremental: Garbage-First garbage collector (G1): In the Sun Java 6 Update 14 or greater release there is a new garbage collector that is in the expirmental stage. It is designed for low pause time high throughput applications such as Wowza Media Server. It is an option to explore. There is more information below. To enable this garbage collector the addtional settings are:
     

     -XX:+UnlockExperimentalVMOptions -XX:+UseG1GC

   
   There is more detailed information here:
   • Java SE 6 HotSpot[tm] Virtual Machine Garbage Collection Tuning
   • Java SE 6 Performance White Paper
   • The G1 Garbage Collector: Coming in Java 6 Update 14
   
   To change the garbage collection settings, modify the following scripts:
   
   
   • [install-dir]/bin/setenv.sh (add at line 10, Linux, Mac OSX and Solaris). For example:
     
     JAVA_OPTS="$JAVA_OPTS -XX:+UseConcMarkSweepGC -XX:+CMSIncrementalMode -XX:+UseParNewGC"
     
     
   • [install-dir]/bin/setenv.bat (add at line 8, Windows). For example:
     
     set JAVA_OPTS=%JAVA_OPTS% -XX:+UseConcMarkSweepGC -XX:+CMSIncrementalMode -XX:+UseParNewGC
     
     
   • [install-dir]/bin/WowzaMediaServer-Service.conf (Windows). For example:
     
     wrapper.java.additional.[n]=-XX:+UseConcMarkSweepGC -XX:+CMSIncrementalMode -XX:+UseParNewGC
     
     Note: Set [n] to the next available number in the list of wrapper.java.additional items. So if the last uncommented-out wrapper.java.additional item is 8 then this item needs to be one item below it with an index of 9.
     
     
   
   
7. For low latency chat applications, it is best to use smaller socket buffer sizes (16000 bytes for read and write). The socket buffer sizes are configured in [install-dir]/conf/VHost.xml:
   
   

   <ReceiveBufferSize>16000</ReceiveBufferSize>
   <SendBufferSize>16000</SendBufferSize>

8. On Linux, increase the maximum number of file descriptors. This will address the "Too many files open" error message. To do this edit the following two files:
   
   /usr/local/WowzaMediaServer/wms.sh
   

   from:
   #ulimit -n 20000
   
   to:
   ulimit -n 20000

   /usr/local/WowzaMediaServer/startup.sh
   

   from:
   #ulimit -n 20000
   
   to:
   ulimit -n 20000

   Note: This may fail on some versions of Linux or if you do not have permission to change the descriptor limit. If after making this change Wowza Server will not start, consult the documentation for your Linux distribution to see how to increase the descriptor limit.
   
   Note: On some versions of Linux there is also a limit in the kernel that may need to be increased. Consult the documentation for your distribution. You may need to add the following line to your /etc/sysctl.conf file:
   

   fs.file-max=20000

   
   
   
9. On Linux, switch to using the Anticipatory elevator (as) elevator algorithm. It is described here:
   
   Choosing an I/O Scheduler for Red Hat
   
   
10. On Linux, mount your disk with the noatime option. This operation will differ based on your Linux distribution. Here is a basic description of the setting:
    
    Using noatime Attribute
    
    
11. To tune your server based on the available CPU resources of your server, use the following guidelines (this assumes a single virtual host, if you are running more than one virtual host contact support@wowzamedia.com for more detailed instructions).
    
    The [total-core-count] refers to the total number of cpu cores in your server. For example if you have dual quad core processors (two quad core processors) the [total-core-count] is:
    
    

    2x4 = 8

    With the number of cores and threads per physical processing growing and growing we suggest a maximum number of threads for each value below:
    
    In the configuration file [install-dir]/conf/VHost.xml:
    

    HostPort/ProcessorCount: 2x[total-core-count] (maximum of 24)
    IdleWorkers/WorkerCount: 2x[total-core-count] (maximum of 24)
    
    NetConnections/ProcessorCount: 2x[total-core-count] (maximum of 24)
    
    RTP/DatagramConfiguration/UnicastIncoming/ProcessorCount: [total-core-count] (maximum of 12)
    RTP/DatagramConfiguration/UnicastOutgoing/ProcessorCount: 2x[total-core-count] (maximum of 24)
    
    RTP/DatagramConfiguration/MulticastIncoming/ProcessorCount: [total-core-count] (maximum of 12)
    RTP/DatagramConfiguration/MulticastOutgoing/ProcessorCount: [total-core-count] (maximum of 12)
    
    HandlerThreadPool/PoolSize: (300x[total-core-count])/5 (maximum of 480)
    TransportThreadPool/PoolSize: (200x[total-core-count])/5 (maximum of 320)

    This assumes you have at least 1GB of memory per core and that in the case you have 4 or more total cores you are running the 64-bit Java VM using the suggested memory settings above.
    
    
12. If you are not doing low latency streaming and you have a client side buffer of 3 or more seconds (NetStream.bufferTime), you can reduce the CPU load on the machine and handle more concurrent sessions by editing [install-dir]/conf/VHost.xml and changing the following two values:
    
    

    IdleWorkers/CheckFrequency: 100
    Client/IdleFrequency: 500

13. If you are having problems with multiple incoming multicast streams interfering with each other on Linux you may need to set the Java property java.net.preferIPv4Stack to true. To do this, edit [install-dir]/bin/setenv.sh and uncomment the following line (line 13):
    
    

    JAVA_OPTS="$JAVA_OPTS -Djava.net.preferIPv4Stack=true"

14. This really isn't tuning but it seems like a good place... When streaming to the Adobe Flash player, it is important to try and avoid RTMPT (tunneling version of RTMP) as much as possible. RTMPT uses a polling mechanism which is very chatty and CPU intensive. We suggest employing a protocol rollover strategy so that only connections that need RTMPT use RTMPT. Is is described here:
    
    http://www.wowzamedia.com/forums/showthread.php?t=576

I hope this helps.

Charlie

[amcorona]

Just wondering if this is a setting we should use if using a hardware raid array or SCSI drives (single disk). How has it improved over CFQ?

And how would you test the performance? I assume one of your engineers did test and that's why this is a suggested setting. Was that testing done on a single disk or RAID array?

[charlie]

I really don't have an answer (I wish I did). We do not have the resources to test a bunch of configs and hardware. We point people to the anticipatory scheduler based on our limited testing with Dell hardware and a very simple SCSI RAID array.

Your best bet is to do some testing. We have a load test tool that can simulate Flash load so you can see how different hardware/software configs perform. That is how we test our performance. You can get the load tool by sending a request to test@wowzamedia.com.

Charlie

[pietjuh1]

Say i've got a dual quad core server.
Do I enter 2x8 or 16 in ProcessorCount?

[rrlanham]

Enter "16"

Richard

[derrick217]

For Mac is Java 6 only supported on Mac OS X 10.5 with 64-bit only? Do you know if Java 6 is supported on Mac OS X 10.6 64-bit only or does it support 32-bit too?

Thanks,

Derrick

[charlie]

Both 32-bit and 64-bit should work fine.

Charlie

[derrick217]

Hi Charlie,

I haven't seen an option to install Java 6 on 10.5.8 on 32-bit. The highest I could get is java version 1.5.0_24. Will that still work for Wowza 2?

Derrick

[charlie]

It will work but it not suggested.

Charlie

[derrick217]

Thanks Charlie for the reply. Do you know of a way to get Java 6 onto Mac OS X 10.5?

Derrick