Page tree
Skip to end of metadata
Go to start of metadata

How the 2 thumpers are set up at CISD

Initial configuration

  • In the Bios, go to advanced -> IPMI setup and setup the network settings of the ILOM
  • The default Grub config, sends the output to the serial console, so be awake when the menu comes up.
  • Solaris 10 u3 is preinstalled, with a / partition of 10 Gbytes a /var partition of 6 Gbytes. These are on a Solstice mirror on c5t0d0 and c5t4d0

Live upgrade

Fully explained here: also posted here in case the link goes bad.
The metadevice d10, is made up of submirrors d11 and d12.
The following procedure dismantles the mirror d15, performs a live upgrade on submirror d11 and attaches d11 to mirror d10.
The same goes for d35, d31, d32 and new mirror d30

The commands I used. Please don't follow blindly
Mount the DVD and install the Live Upgrade tools:

# lofiadm -a /local0/src/sol-10-u5-ga-x86-dvd.iso /dev/lofi/1
# mount -F hsfs -o ro /dev/lofi/1 /mnt/iso
# /mnt/iso/Solaris_10/Tools/Installers/liveupgrade20 -noconsole -nodisplay

Perform the Live upgrade:

# lucreate -c sol10u4 -n sol10u5 -m /:/dev/md/dsk/d10:ufs,mirror -m /:/dev/dsk/c5t0d0s0:detach,attach,preserve \
 -m /var:/dev/md/dsk/d30:ufs,mirror -m /var:/dev/dsk/c5t0d0s5:detach,attach,preserve
# luupgrade -u -n sol10u5 -s /mnt/iso
# luactivate sol10u5
# init 6

Wait patiently. If all is okay:

# lustatus
# ludelete sol10u4

Now attach the old half of the mirror to the new one:

# metaclear d15;metattach d10 d12
# metaclear d35;metattach d30 d32
# metastat -c d10 d30

Now you should check /kernel/drv/e1000g.conf for jumbo frames as below.
You can always check the status of the boot environments through the lustatus command

Network configuration

e1000g0 and e1000g1 are joined in a link aggregate, and are on the network
e1000g2 and e1000g3 are joined in a link aggregate and joined to the same interfaces on the other thumper in the net
To configure jumbo frames, you need to edit /kernel/drv/e1000g.conf and change MaxFramesize to 3 MaxFrameSize=3,3,3,3,3,3,3,3,3,3,3,3,3,3,3,3;.
Caution: This file may get overwritten by a live upgrade.
In /etc/hostname.aggr2 add mtu 16128 and reboot.
Within the ETHZ network, leave the mtu at the default of 1500, that means aggr1 in this case.

Revision control

System configuration files are under RCS control. see man rcs

ZFS setup

Terminoligy note: RAID-Z2 is striped disks with 2 parity disks, similar to RAID6, but with variable stripe sizes. So the net size is (number of disks - 2)

There are 46 disks to play with, so reading suggests a 4x(9+2) RAID-Z2 configuration to achieve 17 TBytes.

Okay, the sums don't add up. - We also put in 2 hotspares. (see the last line in the command)
Here's the command used to create the ZFS pool. Note: it assumes your boot disks are: c5t0 and c5t4

zpool create dataPool \
raidz2 c0t0d0 c1t0d0 c4t0d0 c6t0d0 c7t0d0 c0t1d0 c1t1d0 c4t1d0 c5t1d0 c6t1d0 c7t1d0 \
raidz2 c0t2d0 c1t2d0 c4t2d0 c5t2d0 c6t2d0 c7t2d0 c0t3d0 c1t3d0 c4t3d0 c5t3d0 c6t3d0 \
raidz2 c7t3d0 c0t4d0 c1t4d0 c4t4d0 c6t4d0 c7t4d0 c0t5d0 c1t5d0 c4t5d0 c5t5d0 c6t5d0 \
raidz2 c7t5d0 c0t6d0 c1t6d0 c4t6d0 c5t6d0 c6t6d0 c7t6d0 c0t7d0 c1t7d0 c4t7d0 c5t7d0 \
spare c6t7d0 c7t7d0

New x4540 setup with SSD disks as log disks:
Firstly format the SSDs with a small (~=1Gbyte) slice for ZIL and the rest for Write cache
My SSDs are c10t3d0 and c9t3d0 - slot 3 and 11

~# zpool create dataPool \
raidz2 c9t0d0 c10t0d0 c11t0d0 c12t0d0 c13t0d0 c14t0d0 c9t1d0 c10t1d0 c11t1d0 c12t1d0 c13t1d0 \
raidz2 c14t1d0 c9t2d0 c10t2d0 c11t2d0 c12t2d0 c13t2d0 c14t2d0 c11t3d0 c12t3d0 c13t3d0 c14t3d0 \
raidz2 c9t4d0 c10t4d0 c11t4d0 c12t4d0 c13t4d0 c14t4d0 c9t5d0 c10t5d0 c11t5d0 c12t5d0 c13t5d0 \
raidz2 c14t5d0 c9t6d0 c10t6d0 c11t6d0 c12t6d0 c13t6d0 c14t6d0 c9t7d0 c10t7d0 c11t7d0 c12t7d0 \
log mirror c10t3d0s0 c9t3d0s0 cache c10t3d0s1 c9t3d0s1 spare c13t7d0 c14t7d0

zfs set mountpoint=/array0 dataPool

When using a Compact Flash boot device.

Move swap to the dataPool

swap -d /dev/zvol/dsk/rpool/swap
zfs create -V 8000M dataPool/swap
ls -l /dev/zvol/dsk/dataPool/swap
swap -a /dev/zvol/dsk/dataPool/swap

Add it to /etc/vfstab

# /dev/zvol/dsk/rpool/swap      -               -               swap    -       no      -
/dev/zvol/dsk/dataPool/swap     -               -               swap    -       no      -

Move Temporary Files to a RAM Disk

Add the following entry to the /etc/vfstab file:

swap - /tmp     tmpfs - yes -
swap - /var/tmp tmpfs - yes - 

Turn access time off on the root filesystem

zfs set atime=off rpool

Send logs to to a loghost and dataPool:

zfs create dataPool/var
zfs create dataPool/var/log
zfs create dataPool/var/adm
mv /var/adm /var/adm-old
mv /var/log /var/log-old
zfs set mountpoint=/var/adm dataPool/var/adm
zfs set mountpoint=/var/log dataPool/var/log
mv /var/log-old/* /var/log/
mv /var/adm-old/* /var/adm/

--- syslog.conf.dist    2009-08-05 08:58:15.060993435 +0200
+++ syslog.conf 2009-08-07 14:54:27.034039964 +0200
@@ -30,8 +30,10 @@
 # that match m4 reserved words.  Also, within ifdef's, arguments
 # containing commas must be quoted.
+*.info;kern.debug;daemon.notice;mail.crit       /var/adm/messages
 *.err;kern.notice;auth.notice                  /dev/sysmsg
-*.err;kern.debug;daemon.notice;mail.crit       /var/adm/messages
+*.err;kern.debug;daemon.notice;mail.crit       @bs-log01
+*.err;kern.debug;daemon.notice;mail.crit       @bs-log02

 *.alert;kern.err;daemon.err                    operator
 *.alert                                                root
@@ -40,9 +42,12 @@

 # if a non-loghost machine chooses to have authentication messages
 # sent to the loghost machine, un-comment out the following line:
-#auth.notice                   ifdef(`LOGHOST', /var/log/authlog, @loghost)
-mail.debug                     ifdef(`LOGHOST', /var/log/syslog, @loghost)
+auth.notice                    ifdef(`LOGHOST', /var/log/authlog, @bs-log01)
+auth.notice                    ifdef(`LOGHOST', /var/log/authlog, @bs-log02)
+mail.debug                     ifdef(`LOGHOST', /var/log/syslog, @bs-log01)
+mail.debug                     ifdef(`LOGHOST', /var/log/syslog, @bs-log02)
+*.info                         ifdef(`LOGHOST', /var/adm/messages, @bs-log01)
+*.info                         ifdef(`LOGHOST', /var/adm/messages, @bs-log02)

Restart syslog

svcadm restart system-log


root@cisd-ruapehu # time dd if=/dev/zero of=20G bs=8192k count=2500
2500+0 records in
2500+0 records out

real    0m37.376s
user    0m0.016s
sys     0m36.512s

That's a mere 560 Mbytes/sec (wink)

root@cisd-ruapehu # time dd if=/dev/zero of=40G bs=8192k count=5000
5000+0 records in
5000+0 records out

real    1m18.844s
user    0m0.031s
sys     1m12.160s

Only 530 Mbytes/sec (wink)
Just couldn't resist adding one more:

root@cisd-ruapehu # time dd if=/dev/zero of=80G bs=8192k count=10000
10000+0 records in
10000+0 records out

real    2m41.484s
user    0m0.064s
sys     2m26.890s

You do the calculation. (smile)

Solaris and ZFS Rock!! (big grin)

Setting up a zfs iSCSI target.

Now done with COMSTAR. See:

On a Linux Initiator, follow these instructions:

You'll need to start iscsid and iscsi

  • No labels