Service file for systemd and ongdb
[su_expand height=”200″]
[Unit]
Description=ONgDB Management Service
[Service]
Type=forking
User=ongdb
ExecStart=/etc/init.d/ongdb start
ExecStop=/etc/init.d/ongdb stop
ExecReload=/etc/init.d/ongdb restart
RemainAfterExit=no
Restart=on-failure
PIDFile=/opt/ongdb/data/ongdb-service.pid
LimitNOFILE=60000
TimeoutSec=600
[Install]
WantedBy=multi-user.target
[/su_expand]
Backwards compatibility of systemd service for ongdb
service ongdb start/etc/init.d/ongdbupdate-rc.d ongdb defaultsservice ongdb start-no-waitConfiguring systemd file limits for ongdb
/etc/security/limits.confLimitNOFILE=60000Configuring systemd start/stop timeouts for ongdb
TimeoutSec=600TimeoutStartSec=120TimeoutStopSec=600Managing the ongdb daemon with systemd
Auto restart ongdb on failure with systemd
Restart=on-failureSummary of configuring systemd and ongdb
systemd ships as the default process manager on Ubuntu 15; get the ongdb systemd service file to install and learn about the configuration. systemd has a lot of great features and I encourage you to check it out as it’s a big improvement IMO from SystemV and Upstart. If you’ve started working with systemd to manage your ongdb process, then you
might have stumble through the new setup. Here are a couple things I found helpful and some config that might get you moving along faster. I won’t go into detail on how to install services with systemd but it basically breaks down into these steps:
[su_list icon=”icon: arrow-right”]
- Save service file: [su_highlight] /lib/systemd/system/ongdb.services[/su_highlight]
- Reload config: [su_highlight]sudo systemctl daemon-reload[/su_highlight]
- Enable on startup: [su_highlight]sudo systemctl enable ongdb.service[/su_highlight]
[/su_list]
For those simply looking for a working ongdb.service file one is shown below. The rest of the writeup goes into detail how to install this service file and why some of the properties are set the way they are. This configuration assumes you have linked the ongdb script into /etc/init.d/ongdb and that your ongdb installation is at /opt/ongdb e.g. ONGDB_HOME=/opt/ongdb. The file below should be located at:
/lib/systemd/system/ongdb.services
From all my readings (and experience) systemd is backwards compatible with SystemV. This means that running services such as should continue to work if you have linked the ongdb process into and run the commands to setup /etc/rc[1-6].d. I have found this works well for the scripts that ship with ongdb in version 2.1-3.0. I found that calling should not be used anymore after you get systemd configured. I had some issues with it and it has been removed in 3.0.0-M04 so it’s best not to rely on it.
It wasn’t first obvious to me but systemd does not respect the global security limits at . If you are tuning ongdb using the linux performance guide then all you need to include under the [Service] section of your ongdb.service file. When you configure the limits in systemd, you will see the warning message from ongdb startup disappear. If you are seeing this message below, adding this config is how you fix it:
[su_highlight background=”#ffa999″]WARNING: Max 1024 open files allowed, minimum of 40 000 recommended. See the ONgDB manual.[/su_highlight]
As of 2.3, the default start (was 120s) and stop (was 600s) timeouts on ongdb have been removed. This was done to avoid scenarios where large graphs take a long time to start and under heavy transaction load could experience moments where the transaction log flushing prevents shutdown from occurring cleanly. By default, systemd has timeouts in place for start and stop set to 90s. This is not good for large graph deployments. [su_highlight background=”#9aff99″]The solution is to override the default by setting under the [Service] section.[/su_highlight] I recommend this because most conditions we have experienced where the graph hangs it’s necessary to do a kill regardless because something is running that isn’t shutting down properly (often times we see this with extension code gone wrong). So after 10 minutes it’s likely time to kill the process. I have experienced that ongdb does a good job of recovering transactions in situations of forced shutdown. If you want to execute a kill using different times then separately configure start and stop by setting and . You may need to adjust based on the size of your graph.
Systemd supports processes that spawn sub processes (forking). The default ongdb startup script spawns such a process. In order to get systemd to manage the process that the ongdb startup script spawns, it’s necessary to configure the PIDFile to point to the pid file that ongdb manages. By default this is at $ONGDB_HOME/data/ongdb-service.pid. The big gotcha with the PIDFile setting is the inability to use environment variables. [su_highlight background=”#9aff99″]The solution is to set an absolute path for PIDFile to work properly.[/su_highlight] For systems that would prefer to have systemd load the location of your $ONGDB_HOME from an environment file it’s real bummer.
ONgDB has run very reliably for us but it’s very nice to auto restart ongdb for events that cause the process to fail unexpectedly. The way to achieve this is with which causes the watchdog to monitor the PID we supplied via the PIDFile and if that process fails with an exit code that is non-normal e.g. anything except 0 then it will start the ongdb process again. It is possible to configure the exit codes but any exit code other than 0 on the ongdb process is a candidate for auto-restart.
In summary, systemd has been easy to use and works well with ONgDB. I hope these few tips have been helpful. Please message with questions or comments.
