Capistrano is a great tool for running batches of shell commands on remote servers. It’s mostly used in the Rails community, but is helpful for deploying non-rails apps, or even running commands unrelated to web apps at all.

All our apps are run on Solaris servers and managed by SMF, and we use a series of manifests to control each individual thin instance (named thin/port4000…thin/port4014). Getting Capistrano to play with SMF (so we can restart the app servers with the new code) is luckily pretty simple. We could create our own spin and reaper scripts, but it’s nicer to keep the commands in the deploy script. We can tell Cap to run our commands like this:

namespace :deploy do
  task :start, :roles => :app do
    run "/usr/sbin/svcadm enable network/thin/*"
  end
end

The stop task is similar, but calling svcadm with disable instead. The majority of the time we will be deploying a new version of the app to production, and it’s the restart task that gets called. Restart can be composed of the stop and start tasks:

namespace :deploy do
  task :restart, :roles => :app do
    stop
    start
  end
end

By default, SMF only allows the root user to control SMF managed processes. For security we disable root logins over ssh, so we need a way for lower privileged users to control our process instances. Role Based Access Control (RBAC) is a clean way of allowing a user to perform specific tasks. To begin, we need to add a new authorisation. Add this line to the end of /etc/security/auth_attr:

solaris.smf.manage.thin:::Manage Thin::

This is simply to tell Solaris there is a new authorisation – we could have named it anything. To be able to control SMF using this authorisation, we need to apply it the instances we want to control. There are two types of authorisation we need to apply to each instance: the action authorisation, and the value authorisation. Action is used to perform an action on an instance (e.g restart), value is used to change a value on an instance (e.g set the enabled/disabled state). We need to do this once for each thin instance we are controlling:

svccfg -s network/thin/port4000 setprop general/value_authorization=astring: 'solaris.smf.manage.thin'
svccfg -s network/thin/port4000 setprop general/action_authorization=astring: 'solaris.smf.manage.thin'

Finally we need to apply this authorisation to the user we want to give control to:

usermod -A solaris.smf.manage.thin admin

Admin will now be able to completely start and stop our thin instances now:

svcadm restart network/thin/port4000
svcadm enable network/thin/port4000
svcadm disable network/thin/port4000
svcadm refresh network/thin/port4000

Solaris can sometimes seem a little foreign to people used to Linux or OS X, but its network performance and admin tools make it an excellent platform for hosting web apps. We decided on using nginx as our static file server, and load balancer, and thin as our rails app server. I won’t go in to details of getting these all up and running on Solaris, there’s plenty of resources already available for that.

nginx

Depending on the way you compile nginx, it may already default to using Solaris eventports. Event port is Solaris’ fast asynchronous blocking IO call, similar to epoll in Linux and kqueue in BSD/OS X. To ensure you use it, add a definition to events:

events {
  worker_connections  2048;
  use eventport;
}

One of the great admin tools in Solaris is SMF (Service Management Facility). It’s similar to launchd on OS X – built in to the kernel, it keeps your app up and running. Unlike monit there’s no lag time between checks on the process; the instant a process dies SMF restarts it. You (unfortunately) use XML files called manifests to tell SMF what to watch, and how to start/stop/refresh it. You can also define dependencies so your app doesn’t start up before network services are ready, for instance. This is the manifest we use for nginx:

<?xml version='1.0'?>
<!DOCTYPE service_bundle SYSTEM '/usr/share/lib/xml/dtd/service_bundle.dtd.1'>
<service_bundle type='manifest' name='nginx'>
  <service name='network/nginx' type='service' version='0'>
    <create_default_instance enabled='true'/>
    <single_instance/>

    <dependency name='fs' grouping='require_all' restart_on='none' type='service'>
      <service_fmri value='svc:/system/filesystem/local'/>
    </dependency>

    <dependency name='net' grouping='require_all' restart_on='none' type='service'>
      <service_fmri value='svc:/network/loopback'/>
    </dependency>

    <exec_method name='start' type='method' exec='/usr/local/nginx/sbin/nginx -c /usr/local/nginx/conf/nginx.conf' timeout_seconds='60'>
      <method_context working_directory='/usr/local/nginx/'>
        <method_credential user='root' group='root'/>
      </method_context>
    </exec_method>

    <exec_method name='stop' type='method' exec=':kill' timeout_seconds='60'>
      <method_context/>
    </exec_method>

    <exec_method name='refresh' type='method' exec='/bin/kill -HUP `cat /usr/local/nginx/logs/nginx.pid`' timeout_seconds='60'>
      <method_context/>
    </exec_method>
  </service>
</service_bundle>

Save this and import it into your system manifest with:

svccfg import nginx.xml

This will check the validity of the XML file, import it, and start the service.

thin

There’s no config files required for thin, instead we supply the config as flags to the process. The SMF manifest we use for thin places the multiple thin processes in their own “thin” hierarchy under network/thin.

<?xml version='1.0'?>
<!DOCTYPE service_bundle SYSTEM '/usr/share/lib/xml/dtd/service_bundle.dtd.1'>
<service_bundle type='manifest' name='thin/port4000'>
  <service name='network/thin/port4000' type='service' version='0'>
    <create_default_instance enabled='true'/>
    <single_instance/>

    <dependency name='fs' grouping='require_all' restart_on='none' type='service'>
      <service_fmri value='svc:/system/filesystem/local'/>
    </dependency>
    <dependency name='net' grouping='require_all' restart_on='none' type='service'>
      <service_fmri value='svc:/network/loopback'/>
    </dependency>
    <dependent name='thin_port4000' restart_on='none' grouping='optional_all'>
      <service_fmri value='svc:/milestone/multi-user'/>
    </dependent>

    <exec_method name='start' type='method' exec='/usr/local/bin/thin -e production -d -p 4000 -c /app/current -l /app/current/log/production.log -P /app/current/tmp/pids/thin4000.pid start' timeout_seconds='60'>
      <method_context working_directory='/var/log'>
        <method_credential user='root' group='root'/>
        <method_environment>
          <envvar name='PATH' value='/usr/bin:/bin:/usr/local/bin'/>
        </method_environment>
      </method_context>
    </exec_method>

    <exec_method name='stop' type='method' exec=':kill' timeout_seconds='60'>
      <method_context/>
    </exec_method>
  </service>
</service_bundle>

One of these manifests is used per thin instance we want to keep up. So, for instance, if we had a second thin server running on port 4001, we would make a copy of this manifest and change the appropriate port numbers throughout the file (both for the name of the service, and in the start method). This way SMF will only restart a single thin process at a time if any fail.

By placing the processes under their own hierarchy, we can reference all processes with one pattern. e.g to restart every thin server, we can issue one command, regardless of how many thin processes we have set up:

svcadm restart `svcs -H -o FMRI network/thin/*`

I recently needed to pass a WSSE authentication header with some SOAP messages sent from a Rails app to a government service. Ruby’s native SOAP library is soap4r. It’s featureful, but under-documented, so it took some fiddling to get WSSE to work.

WSSE is a pretty simple standard for sending a username and password to a service in a SOAP message’s header. The layout is:

<wsse:Security xmlns:wsse="http://schemas.xmlsoap.org/ws/2002/07/secext" soap:mustUnderstand="0">
  <wsse:UsernameToken>
    <wsse:Username>Username</wsse:Username>
    <wsse:Password>Password</wsse:Password>
  </wsse:UsernameToken>
</wsse:Security>

Constructing the WSSE headers

Adding headers in soap4r requires constructing an object that responds to a method that will return the headers to be added. A helper class provided for this is SOAP::Header::SimpleHandler. SimpleHandler allows us to respond with a hash that is converted into XML.

SimpleHandler also expects subclasses to call its initialize method with an enclosing tag. We can use xsd4r (the XML parser/constructor used by soap4r) to create the tag and define its namespace:

XSD::QName.new(NAMESPACE, 'Security')

Our call back method is a hash containing the username and password to be sent:

{"UsernameToken" => {"Username" => USERNAME, "Password" => PASSWORD}}

The entire class looks like this:

require 'soap/header/simplehandler'

class WsseAuthHeader < SOAP::Header::SimpleHandler
  NAMESPACE = 'http://schemas.xmlsoap.org/ws/2002/07/secext'
  USERNAME  = 'username'
  PASSWORD  = 'password'

  def initialize()
    super(XSD::QName.new(NAMESPACE, 'Security'))
  end

  def on_simple_outbound
    {"UsernameToken" => {"Username" => USERNAME, "Password" => PASSWORD}}
  end
end

Adding the WSSE headers

Adding the headers to a soap4r driver is quite easy (this has only been tested with a driver that has been auto-generated from a WSDL file).

require 'soap_driver.rb'
require 'wsse_authentication.rb'
d = Driver.new
...
d.headerhandler << WsseAuthHeader.new()

Postgres has support for 8 byte integers with bigint’s, and ActiveRecord natively supports reading & writing these values in Postgres (both on 32 and 64 bit machines, thanks to Fixnum and Bignum). Using them in AR migrations is actually pretty easy as well, but there’s some fudgery required to use 8 byte ints as an id if we don’t want to patch AR.

To create an 8 byte field:

create_table :example do |t|
  t.integer  :eight_byte_integer, :limit => 8
end

To create a table with an 8 byte id without patching:

create_table :example do |t|
  ... fields
end

change_column :example, :id, :integer, :limit => 8

Note: id sequences are automatically created as bigint’s, so all we’re doing here is allowing the table’s id field to store all the values the sequence will create.

Alternatively, we can patch ActiveRecord to automatically make all primary keys bigserial’s. It’s better to patch a frozen version of Rails (in your vendor folder) than your installed gem.

file: vendor/rails/activerecord/lib/active_record/connection_adapters/postgresql_adapter.rb
@@ -283,7 +283,7 @@

       def native_database_types #:nodoc:
         {
-          :primary_key => "serial primary key",
+          :primary_key => "bigserial primary key",
           :string      => { :name => "character varying", :limit => 255 },
           :text        => { :name => "text" },
           :integer     => { :name => "integer" },