Minecraft and Datadog Monitoring

DataDog is a nifty monitoring / statistics gathering system. It is something like a akin to a combination of Graphite / Grafana, but with a social aspect so that your team can attach discussions to a given point in time. They have a free tier that retains data for a day, which is handy for visualizing the state of the Minecraft server.

Java applications normally expose their statistics via JMX. I did not see anything Minecraft-specific in my stock instance, but Java itself exposes several counters that are informative.

I created my Datadog account, procured my API key, and installed the agent with:

DD_API_KEY=MyAPIKey  bash -c "$(curl -L https://raw.githubusercontent.com/DataDog/dd-agent/master/packaging/datadog-agent/source/install_agent.sh)"

JMX is not enabled by default for Java processes, so I updated my systemd unit file in /etc/systemd/system/minecraft.service to include the JMX configuration:

[Unit]
Description=Daemon to detect crashing apps
After=syslog.target
RequiresMountsFor=/minecraft

[Service]
ExecStart=/bin/java -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=9010 -Dcom.sun.management.jmxremote.local.only=false -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -javaagent:../newrelic/newrelic.jar -Xmx1024M -Xms1024M -jar /minecraft/bin/minecraft_server.jar nogui
WorkingDirectory=/minecraft/var
User=minecraft
Group=minecraft
PrivateTmp=true
NoNewPrivileges=true
InaccessibleDirectories=/home
ReadOnlyDirectories=/bin /lib /usr /etc /
ReadWriteDirectories=/minecraft/var

[Install]
WantedBy=multi-user.target

This opens JMX on port 9010 and allows remote access, though I don’t need it for this. My AWS security group will prevent any other machines from accessing jmx. I may come back to this later and implement authentication.

Datadog stores its java configuration in /etc/dd-agent/conf.d/jmx.conf, so I added the list of counters to store:

init_config:
      conf:
            - include:
                domain: java.lang
                type: MemoryPool
            - include:
                domain: java.lang
                type: Threading
            - include:
                domain: java.lang
                type: GarbageCollector
            - include:
                domain: java.lang
                type: Threading
            - include:
                domain: java.nio
                type: BufferPool

instances:
     - host: localhost
       port: 9010

The list of counters is available after the instance is configured in the yaml file by running /etc/rc.d/init.d/datadog-agent jmx list_everything:

The output looks like:

       Not Matching: Bean name: java.lang:type=Compilation - Attribute name: TotalCompilationTime  - Attribute type: long
       Not Matching: Bean name: java.lang:type=Compilation - Attribute name: CompilationTimeMonitoringSupported  - Attribute type: boolean
       Not Matching: Bean name: java.lang:type=Compilation - Attribute name: Name  - Attribute type: java.lang.String
       Not Matching: Bean name: java.lang:type=OperatingSystem - Attribute name: OpenFileDescriptorCount  - Attribute type: long
       Not Matching: Bean name: java.lang:type=OperatingSystem - Attribute name: MaxFileDescriptorCount  - Attribute type: long
       Not Matching: Bean name: java.lang:type=OperatingSystem - Attribute name: CommittedVirtualMemorySize  - Attribute type: long
       Not Matching: Bean name: java.lang:type=OperatingSystem - Attribute name: TotalSwapSpaceSize  - Attribute type: long
       Not Matching: Bean name: java.lang:type=OperatingSystem - Attribute name: FreeSwapSpaceSize  - Attribute type: long
       Not Matching: Bean name: java.lang:type=OperatingSystem - Attribute name: ProcessCpuTime  - Attribute type: long
       Not Matching: Bean name: java.lang:type=OperatingSystem - Attribute name: FreePhysicalMemorySize  - Attribute type: long
       Not Matching: Bean name: java.lang:type=OperatingSystem - Attribute name: TotalPhysicalMemorySize  - Attribute type: long
       Not Matching: Bean name: java.lang:type=OperatingSystem - Attribute name: SystemCpuLoad  - Attribute type: double
       Not Matching: Bean name: java.lang:type=OperatingSystem - Attribute name: ProcessCpuLoad  - Attribute type: double
       Not Matching: Bean name: java.lang:type=OperatingSystem - Attribute name: Version  - Attribute type: java.lang.String
       Not Matching: Bean name: java.lang:type=OperatingSystem - Attribute name: AvailableProcessors  - Attribute type: int
       Not Matching: Bean name: java.lang:type=OperatingSystem - Attribute name: SystemLoadAverage  - Attribute type: double
       Not Matching: Bean name: java.lang:type=OperatingSystem - Attribute name: Arch  - Attribute type: java.lang.String
       Not Matching: Bean name: java.lang:type=OperatingSystem - Attribute name: Name  - Attribute type: java.lang.String
       Matching: 4/350. Bean name: java.lang:type=MemoryPool,name=Perm Gen - Attribute name: Usage  - Attribute type: javax.management.openmbean.CompositeData
       Matching: 8/350. Bean name: java.lang:type=MemoryPool,name=Perm Gen - Attribute name: PeakUsage  - Attribute type: javax.management.openmbean.CompositeData
       Matching: 9/350. Bean name: java.lang:type=MemoryPool,name=Perm Gen - Attribute name: UsageThreshold  - Attribute type: long
       Matching: 10/350. Bean name: java.lang:type=MemoryPool,name=Perm Gen - Attribute name: UsageThresholdExceeded  - Attribute type: boolean
       Matching: 11/350. Bean name: java.lang:type=MemoryPool,name=Perm Gen - Attribute name: UsageThresholdCount  - Attribute type: long
       Matching: 12/350. Bean name: java.lang:type=MemoryPool,name=Perm Gen - Attribute name: UsageThresholdSupported  - Attribute type: boolean
       Matching: 13/350. Bean name: java.lang:type=MemoryPool,name=Perm Gen - Attribute name: CollectionUsageThreshold  - Attribute type: long
       Matching: 14/350. Bean name: java.lang:type=MemoryPool,name=Perm Gen - Attribute name: CollectionUsageThresholdExceeded  - Attribute type: boolean
       Matching: 15/350. Bean name: java.lang:type=MemoryPool,name=Perm Gen - Attribute name: CollectionUsageThresholdCount  - Attribute type: long
       Matching: 19/350. Bean name: java.lang:type=MemoryPool,name=Perm Gen - Attribute name: CollectionUsage  - Attribute type: javax.management.openmbean.CompositeData
       Matching: 20/350. Bean name: java.lang:type=MemoryPool,name=Perm Gen - Attribute name: CollectionUsageThresholdSupported  - Attribute type: boolean

The domain is the portion before the first : in the bean name.

The end result is some nice free graphs out of Datadog:

The graphs for host CPU usage, threads active and java memory utilization are most interesting.