Toger Blog

Minecraft Datadog Monitoring

DataDog is a nifty monitoring / statistics gathering system. It is something like a akin to a combination of Graphite / Grafana, but with a social aspect so that your team can attach discussions to a given point in time. They have a free tier that retains data for a day, which is handy for visualizing the state of the Minecraft server.

Java applications normally expose their statistics via JMX. I did not see anything Minecraft-specific in my stock instance, but Java itself exposes several counters that are informative.

I created my Datadog account, procured my API key, and installed the agent with:

1
DD_API_KEY=MyAPIKey  bash -c "$(curl -L https://raw.githubusercontent.com/DataDog/dd-agent/master/packaging/datadog-agent/source/install_agent.sh)"

JMX is not enabled by default for Java processes, so I updated my systemd unit file in /etc/systemd/system/minecraft.service to include the JMX configuration:

Minecraft and NewRelic

NewRelic is an exceptionally useful tool for monitoring java applications, or at least those that deal with web or other transational workloads. I tried hooking it up to MineCraft and it doesn’t report anything. The free version doesn’t let me look at the JVM stats (threads and such) so it appears to be a waste. However, they also provide a general unix server agent that does provide some nifty dashboards. The procedure to install it is:

1
2
3
4
rpm -Uvh https://yum.newrelic.com/pub/newrelic/el5/x86_64/newrelic-repo-5-3.noarch.rpm
yum install newrelic-sysmond
nrsysmond-config --set license_key=YourLicenseKey
/etc/init.d/

The RPM is EL5-era so doesn’t understand SystemD, so I created a unit file:

MineCraft and SystemD

In the first installment I launched a basic Minecraft service on CentOS7. However, a proper service should not be run from the command line, instead it should be controlled by the system service daemon. In years past this would be by writing a ‘sysV init script’ which would try to determine if the process was running, if not launch it and capture its PID for future reference, and capture its output to a file. CentOS-7 has switched away from that model to one called SystemD which makes much of that easier. There is some controversy over the SystemD model (is it UNIX-y? Too monolithic? Taking over everything?) but seems pretty handy for what it needs to do, plus has some nice security features.

So I will create a Minecraft service definition for SystemD. I used http://0pointer.de/blog/projects/systemd-for-admins-3.html to help me with this. The unit file will look like:

MineCraft Server in AWS Done Too Well

A new Server

This series will go through how to host a MineCraft server, and go totally overboard on the configuration/management of it. I’ll be integrating a variety of management/monitoring tools that go far beyond the needs of the average ‘friends & family’ server, because it is fun. I’ll starting the basics and build up from there.

So, first we need a machine. I’m going to use a Amazon AWS machine for this. I’ll be using some AWS-specific features, but I don’t think any of them will be critical. In some cases I’ll show the non-AWS alternative.

Ec2 Classiclink and Elbs

—assiclink-and-elbs.markdown layout: post title: “EC2 ClassicLink and ELBs” date: 2015-01-09 16:08:02 -0800 comments: true

categories: AWS VPC

I was looking at EC2 ClassicLink today. It is going to make VPC migrations much easier. Some things I noticed:

  • Connectivity to external-facing ELBs only works if the ELBs have ‘world’ incoming access, or specifically mention the public IPs of the source. That is because the ELB IP resolves to its public IP.
  • Connectivity to internal-facing ELBs work fine, assuming the source security group has access.
  • My default-VPC IP space overlapped with my EC2 Classic node’s DNS resolver in /etc/resolv.conf, so attaching it to the VPC broke DNS. Update 1/25/15: I retested this and it is working now, so I’m not sure that this is a problem anymore.
  • The ReInvent session says that instance-DNS won’t resolve, so you won’t be able to use ec2-* domain names to reach specific instances — it’ll have to be IPs.

PaperTrailApp and RouterBOARD 750GL

PaperTrailApp (Referral link) is a SaaS RSyslog host that currently offers 100MB/mo of capacity free. In setting up my home network and servers I wanted to have my RouterBOARD 750GL send its logs to PaperTrail. The RouterBoard supports syslog, so configuring it is pretty straightforward — with one exception. My ISP utilizes PPPoE and so I have an additional interface configured in my RouterBOARD for it. It seems to get confused as to what source IP to use when talking to PaperTrail and the syslog packets go into the bitbucket. I resolved this by hardcoding my source IP into the src-address for the syslog. The resulting configuation is:

1
2
3
4
5
6
7
8
# jan/07/2015 20:12:24 by RouterOS 6.22
# software id = 2JL8-DKGT
#
/system logging action
set 3 remote=<IP by resolving your log*.papertailapp.com address> remote-port=<port defined in the console>
add bsd-syslog=yes name=papertrail remote==<IP by resolving your log*.papertailapp.com address> remote-port=<port defined in papertrail console> src-address=<pppoe address> target=remote
/system logging
add action=papertrail topics=!async,!packet,!ddns,!debug

Mesos, Docker and Consul

I was playing with Mesos/Marathon + Docker recently and tried to integrate Consul. Mesos sets the Docker container ‘name’ to a internal value that messes with the Consul Registrator, something like mesos-aa998abcabab. Registrator can accept an environmental variable telling it what service name to register as instead. The format for that is:

Registrator will register the instance with a service name of nginx-xyz

Home Network Plan

I have several daughters, the oldest of which is just becoming familiar with the Internet. Her grandparents also bought her a WiFi tablet for Christmas. I know that keeping an eye on her usage is the best approach, but I wanted technology to help backstop that. And, because it is interesting to do, I want to way overbuild the environment Just Because.

The technologies:

  • Squid
  • DansGuardian
  • ElasticSearch/LogStash/Kibana
  • Logstash-forwarder
  • Raspberry Pi
  • AWS

She is not sophisticated enough to actively undo any of the protections I might put in (yet!!). So for the mean time the config has to be just good enough rather then prevent a determined attacker.

What I have done is set up a Raspberry Pi on the network that runs Squid+DansGuardian, and configure her devices to utilize that for a web proxy. Logstash-forwarder forwards the logs for Squid and DansGuardian to a EC2 t2.micro instance that parses/indexes them, and I’ve set up reports that show me what domains she’s connected to. Finally, I’ve configured the RouterBOARD 750GL to utilize NetFlow logging to send NetFlow data to EC2 as well, so I can track what domains non-http/https traffic uses.

I am utilizing EC2 for log processing as my Pi just isn’t big enough to run squid/dansguardian plus an ELK stack. I might migrate to a SaaS solution at some point.

I expect to utilize the ElasticSearch function to spool daily logs out to S3 and on to Glacier. I don’t have a specific reason to do this as the logs aren’t really meaningful after a week or so, but it is just something to tool around with. The NetFlow logs could be useful at some point if there was a question about the accuracy of my ISP’s metering though.

Future enhancements will be to put the wireless devices / her network jacks into their own VLAN and either transparently proxy them or disallow unproxied content. Further would be to upgrade Squid to the version that can MitM SSL connections and add a trusted cert to her devices.

The layout so far: