Thursday, 8 November 2007

New Solaris Security Toolkit patch available

Sun has not released an updated package for the Solaris Security Toolkit (aka JASS) for quite a while now and it remains at version 4.2 on the official web site. Instead they have been releasing patches. The latest (122608-06) of which is now available for download from Sunsolve.

The timing of the patch releases usually coincide with each Solaris 10 update release. It is worthwhile keeping up to date with these patches as they cater for changes made to the OS with each update such as new and/or removed SMF services, Secure By Default, etc.

Wednesday, 7 November 2007

An alternate SMF manifest for Splunk

I recently deployed an enterprise licensed version of Splunk. When an enterprise license is installed Splunk requires users to login before they can start splunking. This protects the Splunk administration pages from anonymous users and removes the need to put Splunk behind an apache reverse proxy for protection.

In order to still be able to access Splunk via HTTP (TCP/IP port 80) or HTTPS (TCP/IP port 443) the SMF manifest I originally wrote needed to be modified to allow the splunk user to open TCP/IP ports less than 1024. This was achieved by adding the “net_privaddr” privilege to the method context in the manifest. The manifest also had to be changed so that Splunk no longer bound sockets to the loop-back address only and was directly accessible.

Once you have loaded this manifest and started Splunk you must initially connect to your splunk server via the web interface and configure it to listen on TCP/IP port 80 or enable SSL and have it listen on TCP/IP port 443. Then restart the service with “svcadm restart splunk”.

Download the complete Splunk SMF manifest. Note that it can be used with both the free and enterprise versions of Splunk.

Tuesday, 2 October 2007

5.10 Kernel Patch 120011-14 breaks JASS enable-bart.fin script

A recent kernel patch, 120011-14, delivers a number of the updates that were also delivered in Solaris 10 8/07. If you are applying this patch (it is part of the latest recommended patch bundle) in a JASS/SST driven jumpstart installation of an earlier Solaris 10 release such as 11/06 then you will get an error like this when the enable-bart.fin script is run:
secure.driver: Finish script: enable-bart.fin

Enabling the function: Basic Auditing and Reporting Tool (BART)

[NOTE] Copying /a/var/opt/SUNWjass/BART/rules from /tmp/install_config/Files/var/opt/SUNWjass/BART/rules.
[NOTE] Setting path /a/var/opt/SUNWjass/BART/rules permissions to 444.
[NOTE] Setting path /a/var/opt/SUNWjass/BART/rules owner to 0.
[NOTE] Setting path /a/var/opt/SUNWjass/BART/rules group to 0.
Running bart create -r /a/var/opt/SUNWjass/BART/rules. bart: fatal: open failed: No such file or directory
DEBUG: matched ignored error message
[ERR ] BART create failed; return code 1. Manifest file

[NOTE] Removing saved file, /a/var/opt/SUNWjass/BART/rules.JASS.20071002095649
The cause of the problem lies in how BART is executed in a mini-root environment. The BART executable does not exist in the Solaris 10 mini-root so JASS runs it from the system being installed i.e. /a/usr/bin/bart. The newly patched /a/usr/bin/bart is now linked against the new /lib/ library delivered by this patch, which is also not present in the mini-root of older Solaris 10 releases (though it is in 8/07).

The fix is simple enough:
Index: Finish/enable-bart.fin
--- Finish/enable-bart.fin (revision 666)
+++ Finish/enable-bart.fin (working copy)
@@ -44,6 +44,8 @@
# file is deleted after Solaris is first booted.
logMessage 'Running bart create -r ${JASS_ROOT_DIR}${RULES}.'
${BART} create -R "${JASS_ROOT_DIR}" -r "${JASS_ROOT_DIR}${RULES}" \
| egrep -v '^/var/|^/etc/rc2.d/S82mkdtab' \

Thursday, 20 September 2007

IPMP with IPv6 test addresses

Quite a while back Dave Miner discussed how IP Multipathing can be done on the cheap in Solaris 10 by not using IPv4 test addresses at all and relying on the physical link state or alternatively using IPv6 link local addresses as test addresses. My first attempts at configuring the latter failed and not getting any response to my comment in his blog, I left it for more pressing work.

Recently I got back to this and managed to get it working.

Firstly you need to have /etc/inet/hosts and/or /etc/inet/ipnodes set up correctly with your IP address and hostname. In this example our host is called myhost with two interfaces bge0 and bge1.

/etc/inet/hosts: myhost

Then configure your interfaces.

myhost group production failover up


group production -failover up

group production -failover up

You need to reboot for these configuration changes to take effect or you can pass the contents of the /etc/hostname* files as arguments to ifconfig.

Update: You need to ensure that both the host and the switch are set to autonegotiate for this to work.

Update 2007.09.20: According to this Sun document you do not need to mark an IPv6 test address as deprecated to prevent applications from using the test address. I've updated the configurations above to reflect this.

Wednesday, 5 September 2007

Solaris 10 8/07 is available

Also known as Solaris 10 Update 4. The list of what's new is here. Some highlights are:
  • Python updated to 2.4.4 with 64 bit support and it now lives in /usr/bin. The 8/07 what's new docs don't mention this but it is mentioned in the HW 7/07 what's new.
  • Gnu Zebra replaced with the Quagga routing suite.
  • IP Instances: LAN and VLAN Separation for Non-Global Zones
  • DTrace can now be used in Non-Global zones.
A few things will need a little further investigation:
You can download it now from Sun.

Friday, 10 August 2007

iPhoto Keyword Utilities and iPhoto '08 incompatibility

A warning to iPhoto Keyword Utilities users: you will not be able to import/export keywords to your images if you upgrade to iPhoto '08. The main reason for this is that Keyword Assistant does not work with '08 and the author, Ken Ferry, has stated that he no longer plans to maintain it.

I am talking with Ken to see if he can provide me with source code to the portion of Keyword Assistant that my utilities use. In the mean time I'll be looking at other ways to achieve the same result without Ken's software.

Tuesday, 7 August 2007

Monday, 28 May 2007

One rule to bind them - part II

In part I of this article I tried reducing the number of steps a user was required to perform to get a system installed via jumpstart and addressed a couple of the goals I mentioned in Building a manageable jumpstart infrastructure using the Solaris Security Toolkit.

My ultimate goal was to reduce steps 1 through 5 and selecting from the build menu to a single interaction such as entering the jumpstart client details into a web page or a configuration file. To eliminate the build menu I had to find a way to inform the client which build it was to install onto itself. The /etc/bootparams file doesn't allow custom parameters to be passed to the client. So after some research and experimentation I switched to DHCP with my own custom client options.

I started by working out the minimum amount of information a client required. I settled on the following DHCP option definitions in dhcp_inittab(4) format:
JSBuild         SITE,           128,    ASCII,     1,      0,      sdmi
JSRevision SITE, 130, ASCII, 1, 0, sdmi
JSOSVersion SITE, 131, ASCII, 1, 0, sdmi
A single word such as FIREWALL, DNS, WEBSERVER etc.

The version of the jumpstart environment. I have all of my finish scripts and configurations under source control using subversion. This option is so that I can build hosts from a specific version of my jumpstart environment.

The OS version e.g. Solaris_8, Solaris_9, Solaris_10.
Both JSRevision and JSOSVersion are used on the DHCP server side to craft the paths of a number of jumpstart specific DHCP client options.

These new DHCP options were added to the dhcp inittab in the OS install miniroot:
This was necessary to allow me to run '/sbin/dhcpinfo JSBuild' or '/sbin/dhcpinfo 128' in my begin script.

I chose the ISC DHCPD for my DHCP server as it is extremely easy to configure and very flexible. The dhcpd.conf is:
# DHCP Configuration

pid-file-name "/var/run/";
lease-file-name "/srv/dhcp/dhcpd.leases";

ping-check true;
ddns-update-style none;

default-lease-time 86400;
max-lease-time 86400;

# MY Custom Options
option space MY;
option MY.jumpstart-build code 128 = text;
option MY.jumpstart-revision code 130 = text;
option MY.jumpstart-osversion code 131 = text;

# SUN's Jumpstart DHCP Vendor options
option space SUNW;
option SUNW.root-mount-options code 1 = text;
option SUNW.root-server-address code 2 = ip-address;
option SUNW.root-server-hostname code 3 = text;
option SUNW.root-path-name code 4 = text;
option SUNW.swap-server-address code 5 = ip-address;
option SUNW.swap-file-path code 6 = text;
option SUNW.boot-file-path code 7 = text;
option SUNW.posix-timezone-string code 8 = text;
option SUNW.boot-read-size code 9 = unsigned integer 16;
option SUNW.install-server-address code 10 = ip-address;
option SUNW.install-server-hostname code 11 = text;
option SUNW.install-path code 12 = text;
option SUNW.sysidcfg-path code 13 = text;
option SUNW.jumpstart-cfg-path code 14 = text;
option SUNW.terminal-type code 15 = text;
option SUNW.boot-uri code 16 = text;
option SUNW.http-proxy code 17 = text;

subnet netmask {
deny unknown-clients;
next-server jumpserver;
server-name "jumpserver";
option tftp-server-name "jumpserver";
option domain-name "";
option routers;
option subnet-mask;
option broadcast-address;
option domain-name-servers,;
option ntp-servers,;

option SUNW.root-server-address jumpserver;
option SUNW.root-server-hostname "jumpserver";
option SUNW.posix-timezone-string "Australia/NSW";

# default OS is Solaris 10
option MY.jumpstart-osversion "Solaris_10";
# Solaris
class "SUNW" {
match if substring(option vendor-class-identifier, 0, 4) = "SUNW";

# Serve the correct inetboot file to sun4v hardware platforms.
# Note: T2000 is actually SUNW.Sun-Fire-T200
if option vendor-class-identifier = "SUNW.Sun-Fire-T1000" or
option vendor-class-identifier = "SUNW.Sun-Fire-T200" {
filename = concat ("inetboot.SUN4V.",
config-option MY.jumpstart-osversion, "-1");
} else {
filename = concat ("inetboot.SUN4U.",
config-option MY.jumpstart-osversion, "-1");

option dhcp-parameter-request-list 1,3,6,12,15,42,43,128,129,130;

site-option-space "MY";

vendor-option-space SUNW;

option SUNW.terminal-type "vt100";
option SUNW.root-mount-options "rsize=32768";
option SUNW.install-path = concat("/srv/install/OS/",
config-option MY.jumpstart-osversion);
option SUNW.install-server-address = config-option SUNW.root-server-address;
option SUNW.install-server-hostname = config-option SUNW.root-server-hostname;

# the path to the miniroot
option SUNW.root-path-name = concat(config-option SUNW.install-path,
"/", config-option MY.jumpstart-osversion, "/Tools/Boot");

# the path to correct the version of the jumpstart scripts
option SUNW.jumpstart-cfg-path = concat(config-option SUNW.root-server-hostname,
":/srv/jass/", config-option MY.jumpstart-revision);

# the path to the OS specific sysidcfg file
option SUNW.sysidcfg-path = concat(config-option SUNW.jumpstart-cfg-path,
"/Sysidcfg/", config-option MY.jumpstart-osversion);

# there is always a symlink in /srv/jass to the latest release.
option MY.jumpstart-revision "latest";

# Solaris host declarations
include "/srv/dhcp/hosts/solaris";
A host declaration typically looks like this:
host testfw01 {
hardware ethernet 8:0:20:ab:cd:e1;
option MY.jumpstart-build "FIREWALL";
option MY.jumpstart-revision "2.555";
option MY.jumpstart-osversion "Solaris_9";
I include the host declarations from a separate file so that I can assign group privileges to users to edit that file only or generate it from the information the user enters into a Django web front-end to a database.

Now on the client I can drop down to the ok prompt and type:
ok boot net:dhcp -s
and when it gets to the root prompt run:
# /sbin/dhcpinfo JSBuild
# /sbin/dhcpinfo JSRevision
There you have it. The client now knows which build it is meant to be. You'll notice that I haven't actually saved the user any work yet. In the next part of this series I'll demonstrate how I use this information in the begin script to select a jumpstart profile and pass this information to the JASS driver for post-installation processing.

Monday, 7 May 2007

One rule to bind them - part I

One of the major developments in my jumpstart system's life was the move from having a jumpstart rule per client to a single rule for all clients. Up until then when a user was building a new system they had to:
  1. Login to the jumpstart server.
  2. Obtain root privileges.
  3. Edit /etc/hosts and add an entry for their client.
  4. Edit /etc/ethers and add an entry for their client.
  5. Run JASS add-client script to add the client configuration to /etc/bootparams.
  6. Edit the rules file and add the client to it with the appropriate Begin and Driver scripts for the system build they wanted installed.
  7. Regenerate the rules.ok file with the check script.
  8. Connect to the client system's console and type 'boot net - install'.
  9. Come back and hour or so later to a fully installed system.
This worked well for a few people but handing out root privileges to the growing number of users was not really desirable. Also, if any of the required steps was not done or an error/typo was made then the build would fail and possibly cause other users builds to fail as well. I needed to reduce this to at most 3 steps including steps 8 & 9.

My first attempt resulted in a more sophisticated begin script that eliminated steps 6 & 7. The user would perform steps 1 through 5, then step 8, wait a few minutes while the system booted from the network miniroot, then they would be presented a menu with a selection of different systems builds (firewall, DNS, web server etc) to choose from. The new menu system allowed a great deal of flexibility for our users and our system build developers.

The main advantage was that the jumpstart rules file needn't be maintained on a per host basis, instead a single rule like this was used:
any - Begin/begin = Drivers/driver
What this says is that for any client run Begin/begin script, which will generate (=) the jumpstart profile for installation. Then after installation run the Drivers/driver finish script to perform post-installation tasks such as the standard JASS hardening.

The majority of the magic lies in the Begin/begin script. This script determines which OS version is being installed, presents a menu to the user with the list of builds for that OS, then based on the user's selection generates the jumpstart profile for that build to include the base Solaris installation cluster (Core, User, Developer, Entire, or Entire plus OEM), additional SUNW clusters and packages, and the root file system layout and swap allocation.

The begin script also stores the user selection in a file which is read by the Driver/driver script to determine which build specific driver to execute post-install.

If you look at what I achieved above you'll see I only reduced the number of steps to build a system by two (from 9 to 7) but I laid the groundwork for much more extensive automation. In part II of this article I'll discuss how I developed the system to further reduce the work required by the user and issues caused by human error.

Tuesday, 27 March 2007

Summarizing an IP address range

As I've mentioned previously, I've found IPy python module to be extremely useful for manipulating IP addresses. One such use is a script I've written to summarize an IP address range into the networks that make it up. The script supports both IPv4 and IPv6 addresses.

Example usage:
$ ./

or alternatively as a python module:
>>> from summarize import summarize
>>> summarize('', '')
['', '', '', '', '', '', '', '']

The source for this script is available for download from the Wad of Stuff repository.

Wednesday, 14 March 2007

iPhoto 6.0.6 and Keyword Assistant 1.9.4 released

Those of you using my iPhoto Keyword Utilities who have upgraded to iPhoto 6.0.6 should note that a minor release of Keyword Assistant is out too and is needed to re-enable it.

Monday, 12 March 2007

New Google Code repository for Wad of Stuff

I've created a new Google Code hosted project to house all of the code that I'll be posting in this blog. The source browser and subversion access details are here.

SMF manifest for buildbot

As I mentioned in a previous post, I use buildbot for testing my Solaris installations. I'll be writing more about that in a future article. Until then here is a Solaris 10 SMF manifest that I've developed for starting up a buildbot master or slave. By default it creates one instance called default with a configuration path of /home/buildbot/master.

Before using this manifest you'll need to create your buildbot master or slave directories yourself using buildbot create-master or buildbot create-slave.

To change the path run:
# svccfg -s svc:/site/application/buildbot:default setprop buildbot/config="/path/to/buildbot/dir"

Then enable the service:
# svcadm enable svc:/site/application/buildbot:default

With this manifest you can run as many buildbot masters and slaves as your system can handle.

To add another instance run:
# svccfg -s svc:/site/application/buildbot add instance
# svccfg -s svc:/site/application/buildbot:instance addpg buildbot application
# svccfg -s svc:/site/application/buildbot:instance setprop buildbot/config="/path/to/instance"

Now enable the new buildbot instance:
# svcadm enable svc:/site/application/buildbot:instance

Update: Added the PATH environment variable to the manifest.

Friday, 9 March 2007

SMF manifests for ISC dhcpd, splunk and syslog-ng

I recently had a couple of SMF manifests listed on the Open Solaris SMF Community web site. The manifests are for ISC dhcpd, splunk 2.1/2.2, and syslog-ng.

The splunk and syslog-ng manifests are intended to be used together. The syslog-ng and splunk services depend on a FIFO existing at /var/log/splunk-syslog. In my environment I have syslog-ng accepting syslog messages, filtering them, and writing to the FIFO and splunk reading from it. You may remove this dependency if you wish.

I also have splunk binding to with an environment variable being set in the manifest. Remove the following from the manifest if you wish to have splunk directly accessible:

<envvar name="SPLUNK_BINDIP" value=""/>

I do this to limit access to the splunk admininstrator interface via an apache reverse proxy. The apache configuration snippet to achieve this is included below. Add apache Allow statements to those who you trust to access the administrator interface.

# Reverse Proxy for splunk listening on localhost:8000
ProxyRequests Off
<Proxy *>
Order Deny,Allow
Allow from all
ProxyPass /
ProxyPassReverse /
# Don't allow access to /admin/ which allows people to upload log files
# and tail any file on the server.
<Location /admin/>
Order Deny,Allow
Deny from all
# Add allow statements here

Update: Here are the same ISC dhcpd, splunk and syslog-ng manifests now hosted in my new Google Code repo. I expect these to be more up to date than the hosted copies.

Thursday, 22 February 2007

Understanding snoop(1M) NFSv3 file handles

I wish Peter had written this a month ago when I was trying to troubleshoot a jumpstart client issue in my network.

Monday, 19 February 2007

Building a manageable jumpstart infrastructure using the Solaris Security Toolkit

Luke Kanies has been stirring the hornet's nest lately about why system administration isn't evolving despite the huge growth in the development communities around the applications we manage and use every day. The tools system administrators use get nowhere near the attention, if any, that the applications do. Nor do system administrators often publish the tools they use because they feel they are too specific to the environment they were written for, are one-off hacks, they don't have time, etc, etc. The main aim of this blog is to address my lack of sharing the sysadmin love over the years.

When I started working for my current employer a few years ago, one of my first tasks was to deploy around 20 Check Point firewalls. I had used the Solaris Security Toolkit (aka JASS) many times before as a post-installation hardening tool and set forth building a jumpstart server using JASS to automate the build of these systems. After a couple of weeks I had a build system that could have firewall up and running from bare metal in 20 minutes.

It wasn't long before a colleague had to do a similar roll out of another application so I quickly added it. He was soon to be followed by another colleague...and then another.

Cut to a year later and there were 20 different system builds being done with JASS and it was starting to become a little difficult to manage. The kinds of issues I started facing were:
  1. Modularity - the JASS drivers could be modularized to a degree but it was error prone and difficult for new developers to grasp.
  2. One rule to bind them - having to manage the rules file was a pain. A single rule like "any - Begin/soe.beg = Drivers/soe.driver" that would allow a client system to boot, automatically select the build it was meant to have, and install would be ideal.
  3. Flexibility - varying disk layouts, users, third-party packages, etc on a per build basis.
  4. Testing - When there were 5 individual system builds it was easy for one person to test each one and make sure they built correctly. Once it got to 20 and beyond (now closer to 40) it became very time consuming and error prone. Add to that a few developers checking in changes and you end up with bugs being introduced that could potentially impact many systems.
  5. Builds were slow - initial installation and applying recommended patches were painfully slow.
To solve 1 and 3, I developed a simple configuration layer on top of JASS called slant. Slant removes the need to edit JASS drivers directly. Instead a simple configuration file is created and from it a driver file is generated from a template.

For 2, DHCP based booting rather than bootparamd was needed. A custom DHCP option is provided to the jumpstart client during installation so that it could determine which system build it was installing.

For testing, I discovered a great tool called buildbot that is normally used to automate the compile/test phase of software projects. I've developed a buildbot configuration that allows me to test all of my system builds from bare metal to login prompt and report any errors, warnings, core dumps and more that could possibly happen during an installation.

To address the slow build times I developed an automated flash archive building system that would check for new recommended patch bundles, download them, and trigger a rebuild of the flash archives with the new patches applied. New systems being built look for the presence of a flash archive and use it, otherwise they do a standard (slow!) installation. One of additional problems faced here was making a flash archive that could be built on a V210 and deployed onto a V440 with all the necessary drivers.

All of these developments have culminated in a system that is now used by dozens of people every day with nice a django web front end for provisioning that also generates our DHCP configurations and DNS zones. There is also a jumpstart "appliance" build that is put onto loan or permanent servers for other groups in the organisation to use.

I plan to discuss each of these topics in much greater detail in later articles so stick around.

Friday, 9 February 2007

Sorting IP addresses in python

Here's a quick example of a function to sort a list of IP addresses in python using the decorate-sort-undecorate idiom (aka Schwartzian Transform) and the IPy module.
def sort_ip_list(ip_list):
"""Sort an IP address list."""
from IPy import IP
ipl = [(IP(ip).int(), ip) for ip in ip_list]
return [ip[1] for ip in ipl]

And here is an example of it in use:
>>> l = ['', '', '', '', '']
>>> sort_ip_list(l)
['', '', '', '', '']

Thursday, 8 February 2007

Manipulating IPv4 and IPv6 addresses in python

Eighteen months ago, after many years of programming in perl for most of my systems programming needs, I decided to give python a go after much coaxing by a colleague, Alec Thomas. At the time I had started developing an IP address management system, designed for ISPs/Telcos who need to manage hundreds of address blocks and allocations to customers and internal infrastructure. I did a quick evaluation of both Ruby on Rails and Django and decided on Django for a few reasons:
  • I didn't have to manage my database schema and models separately. Django allowed me to define my data models in a single place and it handled the job of creating the database tables. (This was before Rails had migrations).
  • Django's built in administrative interface was a huge time saver and allowed me to focus on developing my application rather than designing forms.
  • After programming in perl for so long the cleanliness of the python language really appealed to me. Ruby to me just seemed like OO-Perl done properly but with all the $@!#{} perlisms left in.
While developing the application I found an extremely useful python module called IPy (originally developed here), that handles IPv4 and IPv6 addresses and networks.

Here's a sample of how you can use it:
>>> from IPy import IP
>>> ip = IP('')
>>> for x in ip:
... print x
>>> ip2 = IP('0x7f000000/30')
>>> ip == ip2
>>> ip.reverseNames()
['', '',
'', '']

I'll be discussing some tools I've developed with this module at a later date.

Wednesday, 17 January 2007

iPhoto keyword export/import utilities 1.1.0 released

I've just made a new release of my iPhoto IPTC keyword export/import utilities. The main feature update for this release is that it now exports keywords to both the original and modified copies of an image. Head on over and try it out.

I already have plans for the next release to add the export/import of IPTC title and caption as iPhoto title and comment as well as allowing the user to select which of these fields they want to export/import.

Tuesday, 16 January 2007

I've never been to me

Yes, I've been to paradise, but I've never blogged. So here goes.

I've worked for the last 13 years in a variety of jobs that have been predominantly Internet focused, including the early genesis of the Internet in Australia, AARNet, the deployment of AARNet2, and UTS data network.

Whilst building and running these networks I also ran the essential services such as Firewalls, SMTP, DNS, network monitoring, and web, all on Sun Solaris platforms.

These days my interests lie more in the systems area, focusing on developing the infrastructure and tools for deploying and managing large numbers of Solaris servers.

In this blog I intend to write about Solaris configuration tips, advanced jumpstart installation tricks, tools and code snippets I've written, as well as my own personal projects.

Happy New Year!

Matthew Flanagan