How to write Redhat cloned distro initialization scripts with Upstart

On Linux systems, initialization (init) scripts manage the state of system services during system startup and shutdown. When the system goes through its runlevels, the System V init system starts and stops services as configured. While this tried-and-true technology has been around since the dawn of Unix, you can now create modern and efficient CentOS 6 init scripts by using Upstart, an event-based replacement for System V init.
Until its latest release, CentOS used the System V init system by default. SysV init scripts are simple and reliable, and guarantee a certain order of starting and stopping.
Starting with version 6, however, CentOS has turned to a new and better init system – Upstart. Upstart is faster than System V init because it starts services simultaneously rather than one by one in a certain order. Upstart is also more flexible and robust, because it is event-based. Upstart generates events at various times, including while going through the system runlevels, similar to the SysV init system. However, Upstart may also generate custom events. For example, with Upstart you can generate an event that requires certain services to be started, regardless of the runlevel. And Upstart not only generates events, it also handles them – so, for example, when it acknowledges the event for starting a service it will do so. This event-based behavior is robust and fast.
Upstart supports SysV init scripts for compatibility reasons; most service init scripts in CentOS 6 continue to be SysV-based. You might someday have to create an init script yourself if you write custom software. If you do, you should write your new init scripts with Upstart in mind so you can benefit from the new init system's faster performance and additional features.

Beginning the Upstart init script

Upstart keeps init scripts in the /etc/init/ directory. A script's name should correspond to the name of the service or job it controls, with a .conf extension. The init script for the Tomcat service, for example, should be named /etc/init/tomcat.conf.
Like SysV init scripts, Upstart init scripts are regular Bash scripts, but extended with some Upstart-specific directives, which are called stanzas in Upstart. In SysV init scripts you commonly see the line . /etc/init.d/functions, which provides access to additional necessary SysV functions. Upstart scripts are more sophisticated and complete; you don't have to include any additional functions or libraries.
Just as in any Bash script, comments in Upstart scripts start with #. Put descriptive comments at the beginning of each script to explain its purpose, and in other places where the code may need explanation. You can use two special stanzas, author and description, for documentation.

Defining when a service starts

After the introductory comments you can define when a service should start and stop using the special stanzas stop on and start on. These two stanzas can be used with a recognized Upstart event such as when the system enters a runlevel.
Usually administrators configure service jobs to start and stop with the server. By convention, in CentOS you should configure a service job to start at runlevels 2, 3, 4, and 5 and stop at runlevels 0, 1, and 6. In an Upstart init script this is written like this:

start on runlevel [2345]
stop on runlevel [06]

This instructs Upstart to start and stop the service whenever the system enters the runlevel in brackets.
Upstart also lets you start or stop services based on other types of events, such as the starting or stopping of other services. For example, suppose you have an Apache web server integrated with a Varnish caching web server, as described in the article Varnish improves web performance and security. In such a scenario you should make sure that Varnish starts whenever Apache starts, so the configuration stanza for Varnish should look like:

start on starting httpd
stop on stopped httpd

The latter stanza is an unique feature of Upstart; Upstart init scripts can stop services at the same time as other services are stopped, while SysV init scripts depend solely on runlevels.
Another difference is that you configure SysV init scripts when to start and stop by placing symlinks to them in the corresponding runlevels' directories in /etc/rcX.d/, where X is the runlevel number. The command chkconfig does this automatically for you in CentOS. While chkconfig continues to manage most of the init scripts in CentOS 6, it does not work with Upstart, and you cannot manage Upstart jobs with it.

Preparing for an Upstart job

To prepare and customize your environment for an Upstart service job you can use a few additional parameters, each on a new line in the job's .conf file:

• respawn – When you use this parameter the service process will be restarted if it dies unexpectedly. Without this Upstart parameter you might have to write a dedicated wrapper program to start a service and ensure its proper and constant operation, such as mysqld_safe for starting MySQL.
• expect fork – Every service job should be expected to fork in as a background process. When you specify this parameter Upstart obtains the new process's PID, which it can use later to send signals to it, such as to shut down or reload configuration.
• kill timeout [seconds] – This is the number of seconds before the process may be forcibly killed. You should specify enough time (say, 120 seconds) so that interruption-sensitive services such as MySQL are able to complete any pending operations and shut down safely.

You can also adjust a few Bash variables in the job's .conf file. For example, you can configure the umask (permissions) with which new files will be created. A secure choice is umask 007, which means that new files will be created with restrictive permissions 770, which allow only the user itself and the members of its user group to manipulate the newly created files.
A special Upstart stanza pre-start allows you to specify a command or inline script to be run right before Upstart actually starts the job. This stanza is suitable for specifying any sanity checks, such the existence of a necessary file. You may also define prerequisite tasks, such as cleaning of the caching directory of a caching proxy. If the pre-start procedure fails – that is, if it exits with a code other than zero – then the whole job fails and the service is not started.
To see how this works, here's a pre-start directive to remove PHP accelerator eAccelerator's previously cached files, as you would want to do when eAccelerator is integrated with Apache: pre-start exec rm /var/cache/eaccelerator/* -rf. A longer example with a whole inline script looks like:

pre-start script
    # check if Apache's binary is executable or fail 
    [ -x /usr/sbin/httpd ]
    # clear the /tmp directory from old sessions
    rm /tmp/sess_*
end script

Several other stanzas are similar to the pre-start stanza:

• post-start – specifies a procedure to run after starting the service. This is usually useful for complex services that may need additional attention after startup, such as MySQL.
• pre-stop – specifies actions used in preparing for the service shutdown. It is rarely used.
• post-stop – may be regarded as an alternative to the pre-start stanza; in some cases it makes more sense to take some actions right after service shutdown instead of waiting for its next start.

Configuring the Upstart start command or script

Once you've configured your environment, the last thing you must do is define the job's command or script using the stanzas script ... end script to write a regular Bash script inline, or just exec to simply execute a command with arguments. Here's an example that uses the script stanza for the rsyslog service in the rsyslog.conf file:

script
    . /etc/default/rsyslog
    exec rsyslogd $SYSLOGD_OPTIONS
end script

The above directive first sources (includes) the content of the file /etc/sysconfig/rsyslog, where the variable SYSLOGD_OPTIONS is defined. This variable is then used to start the rsyslogd service. This is a convenient way to start a service that requires complex or custom configuration, and it's why such script stanzas are suitable for services such as MySQL or Apache.
Alternatively, the exec stanza lets you specify an executable file and any additional arguments it may need. It's suitable for simpler services; for example, you can start the CUPS daemon with the directive exec /usr/sbin/cupsd -F.

How Upstart stops and reloads services

If you're familiar with SysV init scripts, you may wonder how you configure the commands to stop a service or reload its configuration. The answer is that you don't have to; with Upstart you only configure the start command for a service. When Upstart starts a service it keeps track of its PID and the PIDs of the process forks. When Upstart later needs to shut down a service, it does so with the native Unix signals. Upstart first sends a PID the SIGTERM signal to gracefully shut it down. If the process ignores SIGTERM, Upstart sends SIGKILL to forcibly kill it. Similarly, when the configuration needs to be reloaded, Upstart sends the SIGHUP signal. Upstart's simple architecture removes the needs to specify procedures for stopping, restarting, and reloading a service, though if the shutdown procedure for a service requires more than just sending SIGHUP or SIGTERM signals, you can use the pre-stop and post-stop stanzas.
One last and important difference between Upstart and SysV inits is how you manually start and stop jobs. Upstart works with the command /sbin/initctl, the init daemon control tool. It accepts as a first argument stop, start, restart, or reload, and changes the state of the service correspondingly. The second argument is the name of the service. For example, to start MySQL's Upstart job manually you would run the command initctl start mysqld.