Openldap migration notes

Overview:
In much the same vein as my previous post, I will be migrating an openldap directory from one server to another (zone to zone in actual fact, not that the implementation is any different).
Unlike the previous mysql migration however, the source zone is running snv_95, and at that time Sun wasn't bundling openldap at all; the snv_95 zone is running a blastwave packaged copy of openldap, and it's at version 2.3.39.

The destination zone is running snv_105, which now comes with a native SUNWopenldap(r|u) 2.4.11 implementation, so this should be a nice upgrade along the way.

The source server was a dedicated zone named ldap. The destination will be my database zone (db) that we created earlier. I plan on eventually moving away from using openldap altogether, but for tonight I just simply needed to move it between servers so I could remove the last of the zones from what is actually primarily my file server.

The migration
On the source server, I dumped the entire directory using:
root@ldap ~]#ldapsearch -b dc=griffous,dc=net -h ldap.griffous.net objectClass=* > ldapdump.ldif
This works much in the same way as mysqldump as it happens.

Next I manually editted across my changes from the default in the openldap /etc config files.
/opt/csw/etc/openldap/ldap.conf to the new /etc/openldap/ldap.conf
/opt/csw/etc/openldap/sldap.conf to the new /etc/openldap/sldap.conf (the main file)

I'd just like to highlight again that this openldap directory exists solely to serve my existing mail infrastruture, so I was able to configure it in an identical fashion, including all schemas and copying across password hashes.
Were I starting again with this, I would probably look at setting up full SSL and better security. Tonight we're just moving it, warts and all.

I also had to scp across authldap.schema, since that's a non-default schema.

With the config files modified to suit, start the server with 
root@db:/etc/openldap# svcadm enable ldap/server

I ran into issues at this point, and the smf and dmesg logs were not help at all. I ran the command that smf was using to start the server, with -d 5 (debug) to print out some extra information.
This printed out:
....
<= ldap_dn2bv(cn=manager,dc=griffous,dc=net)=0 <<< cn="Manager,dc="griffous,dc="net">,
/etc/openldap/slapd.conf: line 63: can only be set when rootdn is under suffix
slapd destroy: freeing system resources.
slapd stopped.

My initial suspicion was that it was something to do with capitilisation, but this turned out to not be the case. Line 63 was the rootpassword line, which also threw me for a bit.
After head scratching and looking at the rest of the debug log I spotted the problem further up the log. In the openldap config file I had entered 
database bdb
suffix "dc=griffous.net,dc=net"
rootdn "cn=manager,dc=griffous,dc=net"

(The suffix line is wrong, and should have read "dc=griffous,dc=net"). Whoops, that one was clearly a pebkac mistake.

I fixed that, but still the server wouldn't start! The tail end of the debug this time printed:
unable to open pid file "/var/run/openldap/slapd.pid": 2 (No such file or directory)
slapd destroy: freeing system resources.
slapd stopped.

Sure enough, that directory doesn't exist... Given that this was a new install using default paths, this felt like a fairly systemic issue, so I quickly searched the opensolaris bug repository and found this bug which is exactly my problem. OKayyyyyy, so I'm on to an OS level problem rather than my own now!
I manually created the directory, and ran the command listed in the bug to check on the permissions.
As expected that directory needed to be openldap:openldap, which I quickly changed.

Once more with feeling!
I svcadm cleared the service once more, and finally the server started properly!

The data import

Next I loaded in my data with ldapadd:
root@db:~# ldapadd -h db.griffous.net -D cn=manager,dc=griffous,dc=net -w passwordhere -f ldapdump.ldif
adding new entry dc=griffous,dc=net
adding new entry cn=Manager,dc=griffous,dc=net
adding new entry o=hosting,dc=griffous,dc=net
addding new entry ....
...

Excellent, that bit was trouble free.
In theory this new directory should be ready for production use, but given that rather worrysome bug, I wanted to test it that it survived a reboot.

It didn't.

Under Solaris, /var/run is actually mounted on swap, which means that unless openldap is creating the /var/run/openldap directory when it starts, then it's going to be lost across reboots (as anything in /var/run should be).
It would seem then that openldap is not creating this directory for one reason or another when it starts each time.
The simplest answer to me seemed to be to tell opensldap to store these 2 files in the /var/openldap directory that already used for the BDB backend for directory data.

That way I don't need to mess about with changing openldap startup scripts or playing with the permissions in /var/run, as I would have needed to do if I simply changed the path to /var/ran/* rather than /var/run/openldap/*.

This solved the problem, and openldap now works across reboots.

Testing

I updated the CNAME for ldap.griffous.net to point to the db zone, and carefully watched the ldap traffic while sending myself a few test emails and logging into the server with imap.
Everything seemed fine, so I shutdown the original zone.

If everything stays working for another couple of days, I'll delete the old zone, formally completing the migration.

And that's how you move an openldap directory between hosts!

Quick mysql notes

I was doing a mysql migration from one zone to another machine, in the process upgrading from mysql 5.0.45 (bundled with snv_95) to mysql 5.0.67 (bundled with snv_105), and I'm just making some quick notes for reference.

First off, the migration of the database data is best done with mysqldump. Mysqldump can dump all of the tables live, and it can also be piped into mysql to push the data to the remote system rather than scping the dump across if you prefer.
Obviously you'll want to make sure that your client isn't making changes to the data at the moment of cutover.

Example command:
/usr/mysql/5.0/bin/mysqldump -u root -pshhhhhhh -B dspam | /usr/mysql/5.0/bin/mysql -u test -p -h 192.168.10.5 dspam

Out of the box, mysql is a bit wide open.
mysql> select user,password,host from mysql.user;
| user | password | host |
+-------+-------------------------------------------+-----------------------+ |
|root | | 127.0.0.1 |
|        | | db |
|root | | db |
|        | | localhost |
|root | | localhost |

This tells us that root can log into mysql from 127.0.0.1, db (hostname), or localhost, and that there are no passwords needed....yikes! There are also two 'anonymous' (no user needed) accounts setup.
At least this only represents a local vulnerability....

First lets drop all the extra accounts.
mysql> drop user ''@'localhost';
mysql> drop user ''@'db';
mysql> drop user 'root'@'127.0.0.1'; (doesn't work anyway, should be localhost)
mysql> drop user 'root'@'db'; (doesn't work anyway, should be db.hostname.domain)

That clears out the obvious ones, next update the root@localhost password.
mysql> SET PASSWORD FOR 'root'@'localhost' = PASSWORD('passwordhere');

And recreate the root@fqdn account.
mysql> grant all privileges on *.* to 'root'@'db.griffous.net' identified by 'passwordhere';

At this point I recommend backgrounding your existing mysql session (ctrl-Z), or using a second ssh session into the server to attempt another parrallel mysql login. If you can't get in, the foregrounded mysql connection is your last remaining chance to fix the problem so it's worth testing before ending that session and locking yourself completely!

As far as server configuration goes, the my-small-cnf.cnf configuration file is used by default, which is designed for systems with less that 64MB of RAM. I would change this to at least the my-medium.cnf which uses up to 128MB. (large uses 512MB btw)

Handy commands
Print all users accounts setup on the server.
select User,Password,Host from mysql.user

Show databases:
show databases;
use database; to open it.
show tables; to view tables in that db

/G at the end of all commands prints listings vertically, which can be easier to read in some circumstances.

mysqldump syntax at it's most simple is:
/usr/mysql/5.0/bin/mysqldump -u root -ppassherewithnospace dbnamehere > outputfilename.
I recommend using -B to specific the database name(s) even if you are just dumping 1 single DB, as it will add lines to the script to create the database on the destination if it doesn't already exist.

scp the output file across, and pump it back in again with a simple
mysql -u root <>

Adding users and access takes the following format:
mysql> grant all privileges on dspam.* to 'dspam'@'taraxen1.griffous.net' identified by 'password here';
To grant access to any host, us the % symbol rather than .*. i.e to 'user'@'%'

To grant access to the entire DB, use on dbname.*. Next is the username, and where they are connecting from. It seems that fqdns are favoured.

Removing users is pretty straight forward with the drop statement:
mysql> drop user 'dspam'@'taraxen1';

Show table type (remember that mysql uses different backends per table)
mysql> show table status

Show table structure:
mysql> describe dspam_stats;

Monitor the innodb engine:
mysql> show engine innodb status \G

Show currently connected users:
mysql> show processlist;
+-----+------+-----------+------+---------+------+-------+------------------+
| Id | User | Host | db | Command | Time | State | Info |
501 | root | localhost | NULL | Query | 0 | NULL | 

Solaris ldap naming - Part 3 Client configuration

With your DS now configured correctly, configuring a client to use [non-ssl] ldap is actually pretty straight forward. So much so in fact, I'm going to spend the bulk of this post explaining the bigger picture of how solaris clients interact in your network environment subsequent to making the change from a DNS client to an LDAP client.

The biggest distinction that I had trouble getting my head around was that unlike a windows domain, solaris naming & authentication services are very much distinct concepts, and for that matter identity and authentication can be seperated out too.

While we could just proceed on with implemention, I really feel that it's worth pausing for a moment first to clarify some concepts. I can't have been the only one to be baffled by the differences in implemenation so for the benefit of others I'd like to quickly run through the architecture.
For those that aren't interested, feel free to pagedown a few times!

Naming & resolution architecture
In the windows world, AD takes care of authentication (and identity), policy, profiles, home drives, and other useful information, but not host name resolution. Hosts *are* stored in AD, but this is actually only for computer authentication as part of AD's kerberos implementation; it's not for hostname *resolution*.
Host lookups are done by DNS completely independently of AD.
Now I realise that it is very common for windows DNS zones to be AD integrated, but this is just used as backend for the DNS server's database information.
From a host pespective, AD doesn't tell you what the IP address is of a host, windows clients instead rely on DNS for that information.

In the Solaris world, LDAP also takes care of identity, policy, profile, home drive information etc, and optionally it can also be used authentication too.
The significant difference between the platforms is that ldap can also used for host lookup information, in place of DNS. If you're trying to find a host named serverA.domain.com, and your ldap infrastructure was setup in this manner, you can do an ldap lookup for server!.domain.com and the ldap server would tell you it's IP address based on information in the directory. DNS isn't needed.
Solaris LDAP clients are configured by default to use ldap for host lookups, and DNS isn't used, making ldap your "one stop shop" for all centralised services.

The history for all this predates me in a big way, but I believe it's done this way is because that's the way NIS did it way back when NIS was THE centralised system, before DNS was the cornerstone that it is today. NIS was early 80s remember, and back then networks were small, DNS while more scaleable was more complex to implement and required maintaining an entirely seperate system - hence naming functionality was wrapped into NIS.
I expect that in the interests of easing the transition from NIS(or NIS+) to LDAP, solaris LDAP continues to include/provide this information in place of DNS so it can be a direct drop-in replacement, but with DNS being so ubiquitous I expect that as with this guide, most shops prefer to use native DNS.

Active Directory services summary:
User Identity/Authentication: YES
User settings/profiles: YES
SSO: YES
Host lookups: NO

Solaris LDAP services summary:
User Identity/Authentication: YES (Authentication is optional, kerberos can be used)
User settings/profiles: YES
SSO: No (Kerberos provides SSO functionality)
Host lookups (like DNS): YES (Though I don't think it's commonly setup this way). 

It's that host resolution difference that I expect will catch most people.
To stress the point once more, by default when a client is reconfigured to use ldap, it stops using any dns configuration that may have already been configured for the host. Following an ldapclient init, despite /etc/resolv.conf still being configured correctly, it is no longer used because the updated /etc/nsswitch.conf (now the nsswitch.ldap template) no longer attempts to use dns for 'hosts'. For this reason, it's common practice to modify the nsswitch.ldap template to use dns rather than ldap for host/ipnodes resolution, while leaving ldap for everything else.

With the warnings and background out of the way, lets proceed with actually setting up an ldap client

Configuring a host to be an ldap client:

Here we will use the ldapclient command to reconfigure the host for ldap usage. It does a lot of the heavy lifting for you, though of course you can always modify all the necessary files by hand if you're mad enough.This command reconfigures the host to use ldap for naming and lookup information. If you wish to use dns for host resolution (and you probably do!), edit /etc/nsswitch.conf and change ldap for dns on the hosts: and  ipnodes: lines.
Better yet, change the nsswitch.ldap template beforehand, that way you can jump backwards and forth while testing this without having to correct the file each time.

The command is run on an ordinary host that uses DNS for naming, and it doesn't have any ldap SSL preconfiguration setup, which means that the client needs to be able to contact the DS using a cleartext non-encrypted connection. (This obviously isn't desirable, but getting SSL up and running is a whole post in itself, coming later).

jwheeler@test:~# ldapclient init -a proxyDN=cn=proxyagent,ou=profile,dc=solnet,dc=com -a domainName=solnet.com -a profileName=default -a proxyPassword=proxypasswordhere 192.168.10.4
System successfully configured
jwheeler@test:~#

ldapclient does NOT register user/computer accounts in the directory in the way that an AD join would. Please note that this is not the solaris equilivent of a windows "domain join" operation. In fact no information is loaded into the directory at all, this is an entirelyread only operation that simply configures the client host to look to the directory for information based on the information in your default profile which is contained within the ldap server itself.

Great, my first client is configured. Now what, how do I test it?
For most people following this guide, this will be your first time deploying ldap in your environment, in which case your newly deployed directory will be empty and you'll need to prepulate it with some useful information to test it with. For starters, in the very least you'll want to load some user accounts into ldap, and ideally some automounter information.

ldapaddent is the command used to take local "databases", and loads them into ldap. Databases is the official term to descibe data sources such as your /etc/passwd and /etc/shadow files.
To load in user accounts:
ldapaddent -D “cn=directory manager” -f /etc/passwd passwdNote that this command only loads in the user accounts, and not the passwords, which are in your shadow file. You must also add the shadow file after doing the inital passwd import:
ldapaddent -D “cn=directory manager” -f /etc/shadow shadow
This pair loads your entire passwd database, including all system accounts and root, which you probably won't want. There isn't an option to add only specific accounts, so at this point I would suggest using the DSCC web interface to browse your ldap data and then using your mouse and the ctrl key, do a group select to delete all but the user accounts that you're wanting to work with.To test that your client is correctly connecting to the ldap server, and pulling information we can use the getent tool.This command queries the host's databases using whichever naming service(s) is configured, so it can also be used locally. 
jwheeler@ldapclient:/home# getent passwd | grep jwheeler
jwheeler:x:101:1::/export/home/jwheeler:/bin/bash
jwheeler:x:101:1::/export/home/jwheeler:/bin/bash
 
2 answers were returned because the host is configured to try ldap first, but jwheeler is still contained locally (in /etc/passwd) too. Unfortunately this tool isn't too clear about where the data is coming from, but it should be fairly easy to deduce what's happening with a bit of intuition.With a positive result to that test, I next deleted the jwheeler entries from the local passwd and shadow files (after taking a backup first of course!), and attempted to login to the ldap client host remotely via SSH.
jwheeler@angelous:~$ ssh 192.168.10.10
Password:
Last login: Sun Feb 1 23:59:27 2009 from 192.168.10.60 Sun Microsystems Inc. SunOS 5.11 snv_105 November 2008
jwheeler@test:~$ id uid=101(jwheeler) gid=1(other)
jwheeler@test:~$ grep jwheeler /etc/passwd
jwheeler@test:~$

Excellent, a successful login via LDAP!You'll note that it wasn't single sign on (I was prompted for my passwd, despite already being logged into angelous as jwheeler), as ldap isn't a SSO system.To get an SSH working without a password prompt, you need to either use kerberos, or ensure that the user's public ssh key is already on the remote host in the user's home directory.
This is where the nfs automounter comes to the rescue.

On this ldap test host, the jwheeler homedirectory /export/home/jwheeler already existed, as it had been created and setup as a local user until moments ago, and I didn't get as far as physically deleting the directory from /export/home.

I also have a user called testuser that exists only in ldap, and hasn't ever existed on the test ldap client host, which I'll use here to illustrate the more typical result: 
jwheeler@angelous:~$ ssh testuser@192.168.10.10
Password:
Last login: Sun Feb 1 23:24:29 2009 Could not chdir to home directory /export/home/testuser: No such file or directory
Sun Microsystems Inc. SunOS 5.11 snv_105 November 2008
-bash-3.2$ pwd
/ 

So if the user is a valid ldap user and nothing else, you still will at least log in however since your home directory doesn't exist, it isn't magically created. Enter the Automounter service.... With the automounter service correctly configured it will use magic (information stored in ldap) to mount an nfs mounted homedirectory for your user at login, which is excatly what we'll setup next.

The Automounter Service
The Solaris convention is to use /home for nfs mounted home directories, and to use /export/home for locally mounted home directories.  The automounter service dynamically takes care of managing these nfs mounts for all active users on a system, so nfs mounts for active users are mounted transparently at login, and automatically unmounted after use.

Setting up the remote server for the nfs homedirectories is beyond the scope of this article; I'm sure you know how to create a directory on a server, and share(1M) it out via NFS. A full run down of automounter service and it's maps is also quite a big topic, so I'll only cover the bare essentials in the interests of brevity. 
In it's simplest form though, configuring the automounter is dead easy.

The only key file that you need to modify is /etc/auto_home.
In this file add a single catch-all configuration line to the bottom of the file.
The simplest possible configuration is:
* fqdnnfsservername:/shared/homedirectoryroot/&

The * matches all users, the nfs mount is pretty self explanitory, and the ampersand at the end is substituted for the user's actual username.
For example, mine reads:
* supernova.griffous.net:/Z/nfshome/&

Start the automounter service (svc:/system/filesystem/autofs:default) if it isn't already, and that's all you have to do on the automounter side.
In ldap, you will need to update the homedirectory path for your user account to "/home/usernamehere", instead of "/export/home/usernamehere". I should point out the /home/usernamehere is the default behaviour, so you'll only have to update this for users that you've manually configured for local logins pre-ldap!The updated /etc/auto_home will work nicely now for *this* host, but wouldn't it be even better if you didn't have to manually add that line to this file for all future clients that you convert over to ldap? Fortunately, this can be done!In the /etc/auto_home file, comment out the +auto_home, and in the /etc/auto_master file, comment out +auto_master line. These are just used for NIS, which isn't used in this guide.

Next we load this information into ldap using ldapaddent again:
ldapaddent -D “cn=directory manager” -f /etc/auto_master auto_master
ldapaddent -D “cn=directory manager” -f /etc/auto_home auto_home

With this information present in ldap, the host will merge the information from the ldap versions of auto_* and the local /etc/auto_*. By default the local /etc/auto_* files are empty, so the ldap information is used. Ldap hosts are now setup for nfs automounts with zero configuration!

Next up, SSL configuration

Handy commands for ldap clients:

ldapaddent is the command that you use to populate the directory with information from the local machine. Typical examples would be local user accounts, which is to say the contents or your /etc/[passwd|shadow] files.

I.E: ldapaddent -D “cn=directory manager” -f /etc/passwd passwd 
Note that this doesn't load up the passwords, you'll need to also add the shadow file after doing the inital passwd impart:
ldapaddent -D “cn=directory manager” -f /etc/shadow shadow

I got caught by that one!

getent - This is used for querying your naming service for an entry in a database, for instance a user in the passwd database
$ getent passwd user1
user1:x:1002:10::/export/home/user1:/bin/bash 

ldaplist - By itself, it will print all DNs in the base, so typically you'd narrow it down to the 'database' that you're interesting in. In this case the term database just means a table such as the passwd table. Any entry that appears in your /etc/nsswitch.conf is called a database. To view the user list (passwd equivilent), do ldaplist -l passwd which gives an entry like this:
dn: uid=jwheeler,ou=people,dc=solnet,dc=com
cn: jwheeler
uidNumber: 100
gidNumber: 1
homeDirectory: /home/jwheeler
loginShell: /bin/bash
objectClass: posixAccount
objectClass: shadowAccount
objectClass: account
objectClass: top
uid: jwheeler
userPassword: {crypt}**************
shadowLastChange: 14276
shadowFlag: 0 

ldapclient list will print the current ldap client configuration
jwheeler@test:~# ldapclient list
NS_LDAP_FILE_VERSION= 2.0
NS_LDAP_BINDDN= cn=proxyagent,ou=profile,dc=solnet,dc=com
NS_LDAP_BINDPASSWD= {NS1}*************
NS_LDAP_SERVERS= 192.168.10.4
NS_LDAP_SEARCH_BASEDN= dc=solnet,dc=com
NS_LDAP_AUTH= tls:simple;simple
NS_LDAP_SEARCH_REF=
TRUE NS_LDAP_SEARCH_SCOPE= one 
NS_LDAP_SEARCH_TIME= 30
NS_LDAP_CACHETTL= 43200
NS_LDAP_PROFILE= default
NS_LDAP_CREDENTIAL_LEVEL= proxy
NS_LDAP_BIND_TIME= 10 

Solaris ldap naming - Part 2 concepts and preparation

Before we can move onto actually configuring and loading the DSEE with data, there are some concepts and terminologies for the future ldap clients that need to be understood.

Authentication and Encryption options

There are three basic methods to authenticate with the DS, and many methods to encrypt that conversion. This is a very important distinction, but this wasn't made very clear in the official documentation that I've found to date. To make matters worse the terminology used to describe these concepts are very counter intuative.

Authentication to the Directory (Called "credential level" in solaris-ldap-speak)
This is who your ldap clients will connect to the DS as.There are 3 basic choices:
Anonymous
Pretty self explanitory really, clients won't connect as a any specific user at all, this is called an anonymous bind. Bear in mind that this means that everyone (computers or users) connect to your server at the same access level. Ldap clients need write to the directory to perform tasks like password changes which means that you'll have to grant write access not to just everybody, but more precisely anybody with network access to your directory server.I really hope I don't have to explain how dire this is from a security standpoint!

Proxy
This is not to be confused with directory proxy servers, or http proxy servers. This method simply means that all ldap client will share a single username/password. The key difference here as compared with anonymous is that only those clients that already know this shared username/password can connect. Obviously only your known clients should have that password, not the evil hackers! The documentation refers to the ability to create and use multiple proxy accounts across your enterprise, which seems to be something a bit like a rudementary implementation of group based access. This might be useful for segmenting your ldap domain into major business units with different access to different sections of your directory however I expect that if your network is that big/complex than you'll want the next method.

In short, you never ever want to use anonymous, proxy is the more secure and better choice to start off with.

Per user (Kerberos)
This method assumes that each user/host will have their own account with which to access the directory server. This of course will be familiar ground for anyone that's run any Windows AD network. Each of your user's will have their own account in your directory, so why not have them use this account to access the directory, rather than this silly shared proxy account?

It turns out that it doesn't work that way here. Instead the concept of a kerberos user account is entirely seperate from the LDAP record representing a user account's identity.
As we won't be installing kerberos (yet), proxy is the "credential level" that we will use across the Directory Servers.

Encryption (Called Authentication Method in solaris-ldap-speak)

This deals with how to connect with the directory server, rather than as whom.
By using proxy authentication we've stopped just anyone on the network connecting and accessing our directory server, but unfortunately in it's native implementation the proxy user's username & password go across the wire in clear text. *shudder*

In this modern age of switched networks, (rather than "hey, everyone watch my traffic" hubs) man-in-the-middle attacks are hopefully far less common or as easy to pull off as they once were when Solaris 8/9 were making the move to LDAP, however encryption remains a very good idea and it's standard practice in most commercial implementations, such as the Windows AD world.

Unfortunately, based on my experiences setting it up for this blog, the Solaris implementation and easee of setup of SSL for LDAP is decades behind what is found in the Microsoft domain world.
I recommend starting off with encryption optional while you get everything up and running initially.

Of the 8 choices available, there are 3 main ones of interest. I expect the others are more for legacy implementations.
simple:Simple doesn't use encryption at all. Enough said.

TLS:simple:
This method uses SSL encryption for all communication with the DS. Think https, but ldap.
This is the method that this guide will be using, and it's the method that all the other guides seemed to recommend based on my googling at the time.
As I mentioned earlier, unfortunately it does takes quite bit of work o get SSL up and working correctly on clients but fortunately you can specify multiple methods and an order of preference.  Therefore we are going to select tls:simple with the option to fall back to "simple" (no encryption remember) while getting initially setup.

sasl:GSSAPI:
This method is exclusively for use with kerberos.

Schema:

Another big surprise for first timers will be the fact that out of the box, DSEE can't work with and doesn't understand how to deal with Solaris clients. I'm just really dumbstruck here.
I would understand if the default implementation needed to be extended in some way to handle non-SUN clients, but in actual fact it can't do anything in it's default configuration. There is no integration whatsoever. SUN - FIX THIS!

For a client to use ldap, the directory needs to be prepped with a Solaris client friendly schema. The second shock to newcomers will be that the DSEE server software doesn't actually come with any software, wizards, or buttons to set it up for Solaris clients.

The 'tool' (a shell script in fact!) for setting up a solaris ldap client friendly schema is found on client machines in /usr/lib/ldap/idsconfig, which comes with the SUNWnisu (Network Information System - usr) package.
This package is included as part of the base Solaris install. 

Run this script an any user, and follow the prompts. Most of the defaults will be OK. My configuration summary is belowe, with notes appended where important.

  1 Domain to serve : solnet.com
  2 Base DN to setup : dc=solnet,dc=com (my test domain)
  3 Profile name to create : default
  4 Default Server List : 192.168.10.4 (My DS server)
  5 Preferred Server List : 
  6 Default Search Scope : one
  7 Credential Level : proxy (Important. Make sure you don't use anonymous) 
  8 Authentication Method : tls:simple;simple (Important. SSL if possible, but fall back to no encryption while we get setup) 
  9 Enable Follow Referrals : FALSE
 10 iDS Time Limit : 
 11 iDS Size Limit : 
 12 Enable crypt password storage : FALSE
 13 Service Auth Method pam_ldap : 
 14 Service Auth Method keyserv : 
 15 Service Auth Method passwd-cmd: 
 16 Search Time Limit : 30
 17 Profile Time to Live : 43200
 18 Bind Limit : 10
 19 Service Search Descriptors Menu

Next I ran a bunch of index commands as instructed. In my case I had to change the paths and domains to look something like this for each line:
dsadm reindex -l -t solnet.com.getgrent /var/opt/SUNWdsee/dsins1/ dc=solnet,dc=com

This will create all the additional records and schema information in the directory to service Solaris LDAP clients.

At last you are ready to connect your first client machine, which we will do next.

Solaris ldap naming - Part 1 Directory Server Enterprise Edition

Now that we've decided to go with an LDAP directory server, the question is - which one?

The major players appear to be:
Apache Directory Server
OpenDS
OpenLDAP
Sun Java System Directory Server

I'm pretty sure that Solaris clients can be made to work with pretty much any LDAP server (even Windows 2003 r2 these days), but in the interests of K.I.S.S, I'll stick with the SUN product for this guide. I expect this will be the most common approach taken by most administrators.

The current full name for the Sun directory server is "Sun Java Server Directory Server Enterprise Edition", a REDICULOUS mouthful - come on SUN, seriously!.
I shall refer to it simply as DSEE everywhere going forward.

DSEE needs to be installed in the global zone, or a full-root zone. A sparse-root zone doesn't work unfortunately. Technically you could install the common components only in the global zone, and the rest can be installed in a sparse zone, but that messes with my global zone and is just generally messy For the installation of DSEE, I have created a full-root zone on persephone, my x86 SXCE 105 VMware ESX guest. The zone is called ds1.

Installation steps:

1. Download and install the native pkg version of DSEE. The main product appears to be "java ES 5.0u1" which comes with DS 6.2, and there is an update patch which takes that up to 6.3. I have it on good authority that 6.2 is evil - make sure you download the 6.3 update too!
2. I used ssh X11 forwarding to run the installer via ./installer. I assure you - this is much nicer than installing over the command line! I seleted DSEE 6.2 and it's subcomponents only.
   
   
3. The shared components upgrade screen was a bit of a surprise, and it highlights a bug when installing on recent SXCE builds. It listed some upgrades to Cacao that are actually downgrades. On my system it wanted to "upgrade Cacao 2.2.0.1 to 2.0::PATHCHES:12384". You have to continue anyway, but the unfortunate reality is that you are going to end up with a broken cacao that will need to be reinstalled from original media :(Next I failed the OS resources check because I didn't have SUNWpl5u installed, which turns out to be a perl package.
   This turned into an even bigger problem as it turns out that sxcr 105 doesn't even ship with SUNWpl5u on the DVD any longer (that will be why the resource check was moaning!). I seems that in modern SXCE builds the SUNWperl584core pkg is used instead.
   Fortunatly I still had a sxcr95 DVD around, which did have the package but it turns out that SUNWpl5u can only be installed in a global zone....ack!Boy this is fun isn't it!
   So I had to install the sxcr95 SUNWpl5u package in the sxcr105 global zone to proceed with the install, all the while crossing my fingers hoping that this won't break my perl for all my existing and future zones on persephone.
   
   
4. The install continued, and I chose to configure the server now, including creating a default DS instance. It used the default path "/var/opt/SUNWdsee/dsins1", which didn't seem all that intuative to me, but I left it as is.
   I changed the Suffix to dc=solnet,dc=com to suit my test environment here, but otherwise left everything as default, including running everything as root. (I'll try to come back to that!). The install proceeded smoothly after that.
   
5. Next I tried to install the upgrade patch to get up to DS 6.3. As I'd almost come to expect by now, this too led to cursing. The patchadd folder contained two packages that I didn't have installed on the system, which in turn caused the whole patch to fail. I didn't install the ldap-proxy, but it appears the pactch fails it if can't patch it regardless.This is actually my first time installing a proper solaris patch, since I've always lived a bit on the edge running sxcr or higher. I found that if I deleted the two offending folders from the patchadd folder, it would proceed.
   rm -rf 125278-07/SUNWldap-proxy, and rm -rf 125278-07/SUNWldap-proxy-man did the trick, and the patch installed cleanly.
6. Ok, actually on to configuring the product at last!
   I connected to the webconsole over https://ds1:6789. Normally I disable the webconsole by default due to it's memory usage, check that svc:/system/webconsole:console is enabled.
   You can connected with your standard non-root user initially and the DSCC will give you the following (helpful) error: the Directory Service Control Center requires a one-time initialization process to be run before it can be used. You then log in an root, and after following the DSCC wizard it will helpfully tell you that
   Authentication As root No Longer Required.
7. Once logged in again as my non-root user, I saw that I didn't have any directory servers registered. This came as a bit of a surprise, as I know we created one during the install in /var/opt/SUNWdsee/dsins1!
   
   After a quick bit of digging around, I quickly realised that cacaoadm wasn't working and that I was hitting this bug again. This is the bit that I mentioned earlier about Cacao needing to be re-installed.The fix is to un-install SUNWcacaowsvr, and SUNWcacaort and to revert back to the OS provided versions of these packages (2.2x rather that 2.0x).
   SUNWcacaomn doesn't come with ON, so I left that package at the older 2.0 version.
   
   After being re-installed the dscc and cacao aren't talking properly:
   # /opt/SUNWdsee/dscc6/bin/dsccsetup status
   *** DSCC Application is registered in Sun Java (TM) Web Console
   *** DSCC Agent is not registered in Cacao
   *** DSCC Registry has been created Path of DSCC registry is /var/opt/SUNWdsee/dscc6/dcc/ads
   Port of DSCC registry is 3998
   
   You must reregister the Cacao agent in the dscc using the cacao-reg command
   # /opt/SUNWdsee/dscc6/bin/dsccsetup cacao-reg
   Registering DSCC Agent in Cacao...
   Checking Cacao status...
   Stopping Cacao...
   Enabling remote connections in Cacao ...
   Starting Cacao...
   DSCC agent has been successfully registered in Cacao.
   At this point I thought I'd check in on the resourcse usage of the zone. DSEE and the webconsole are real memory hogs!How much RAM is used by an idle DSEE server (zone) with no data yet?
   
   ZONEID NPROC SWAP RSS MEMORY TIME CPU ZONE
   4 44 388M 568M 21% 0:01:07 0.4% ds1 
   Over half a gig already! cripes!
   
   
8. For reasons beyond me, the default behaviour for both the DSCC registry/management system and your DS instance both won't start by default. You must use the dsadm command to register them both within SMF to get them starting by default with the following command:
   ds1:~# /opt/SUNWdsee/ds6/bin/dsadm enable-service --type SMF /var/opt/SUNWdsee/dsins1/
   Registering 'Directory Server' as 'application/sun/ds' in SMF ...
   Registering '/var/opt/SUNWdsee/dsins1' as 'ds--var-opt-SUNWdsee-dsins1' in SMF ...
   Instance /var/opt/SUNWdsee/dsins1 registered in SMF
   Use 'dsadm start '/var/opt/SUNWdsee/dsins1/'' to activate the service 
   
   And then run the suggested command to actually start you instance.Repeat again for the DSCC registry:ds1:~# dsadm enable-service --type SMF /var/opt/SUNWdsee/dscc6/dcc/ads/
   
   Now your DS instance (and management service) are installed and ready for configuration!Phew!Note to Sun: This really needs some better polish in a BIG way!
9. Bonus step: Have yourself a coffee, there is lots more to come.
   

zfs sparse provisioning with compression - results

I thought I'd quickly share an update with the results after a few days of using this solution.

My ESX servers are hosting 5 VMs, with 30GB,15GB,15GB,10GB,10GB disks respectively for a total of 90GB.
ESX is showing the iscsi volume as 93GB/500GB used. I have a couple of GB on the volume for the ESXi userworld swap , which explains the descrepancy.

ZFS on the other hand is showing 35.0GB/220GB used on the zvol.
What's more, it's also reporting 1.28x compression!

This is a great space saving, so what are we seeing here?

On the ESX/VMFS side, the vmdks are actually being sparsely provisioned onto the storage from what I can tell. VMFS makes sure that it accounts for the full space of each vmdk in it's totals so that it doesn't overcommit itself, but it isn't actually zeroing/preallocating the contents of each vmdk as it makes them, despite what you may think when you hear the term "thick-provisioning".

I actually confirmed this by testing with another zvol with compression off to rule out zfs compression reclaiming any "000000....0" blocks that VMFS may have been writing. Same result as my production compression=on iscsi volume.

On the Solaris side, ZFS is COWing only the blocks that actually get touched. As you would expect most of my VMs are not using the full capacity of their file systems yet, which explains the big difference between ~90GB and the 35GB that ZFS is actually using.

It's also interesting to see that I'm getting 1.28x compression too. It was higher earlier on but I've been doing quite a bit of work on a Solaris vm which is using ZFS-root, with compression=on in the guest, so we can't compress any blocks for that VM a second time.
This means that this compression figure is for 4 windows VMs of the 5, which is worth having in my book!

Scaling this out: 35GB-for-90GB is a space saving of bit over 2.5x.
I thinly provisioned a 500GB volume with around 250GB of actually storage available with was only a 2x overcommit. Maybe I should have aimed higher!

Snapshots

Just a quick note on ESX snapshots. They will burn through your space savings as they are COWing in a totally different area on the disk, raising the "high water mark" that I referenced in my earlier post.
That's not to say that you shouldn't use them, but just be aware that they really are quite costly both in terms of performance to the VM as we all know, but also to your thin provisioning savings :)

zfs sparse provisioning with compression

As I was writing my last post, the possibility of zfs reclaiming/saving some space within the zvol by using compression kept nagging at me. In this post I've explored what happens when compression is used with a sparse-vmfs-iscsi-zvolume

I'm not so much interested in whether the real,used VM data blocks would compress well (I don't really expect there is much to be made in the way of gains here) but I did want to see if there were circumstances where compression would allow some unused or freed blocks above zfs to be reclaimed to the zpool by way of compressing empty blocks to almost nothing.

Testing

I performed the following tests to see what would happen:
Virtual machine FS blocks - Write 10GB of data to the file system in a VM, and then delete it again. Check zvol usage before/after.
VMFS blocks - Delete a the 10GB virtual disk from the iscsi VMFS lun. Check zvol usage before/after.

I added a 10GB disk to a winXP VM, formatted it with NTFS and then ran an iometer benchmark on the volume. I did have an interesting result while iometer was creating it's initial 10GB scratch file on the disk. Iometer must write zeros or something very compressable while populating the file because I was getting fantastic compression. Iometer finished creating it's 10GB file, but ZFS was reporting only 78MB used on the whole zvol. That's over 100x compression/deduplication!

With the scratch file fully created, the benchmark actually started and at this point iometer must have been writing some real data across the disk, as the zfs usage worked it's way back up to the full 10GB disk usage that I would have expected.

After the benchmark had finished the zvol looked like this:
#zfs get used,compressratio,referenced Z/esxiscsi-compressed
NAME PROPERTY VALUE SOURCE
Z/esxiscsi-compressed used 10.9G -
Z/esxiscsi-compressed compressratio 1.00x -
Z/esxiscsi-compressed referenced 10.9G - 

I closed the benchmark, and deleted the 10Gb iobw.tst file and rebooted the VM to ensure that all NTFS/VM caches were flushed.
As I expected, the zvol used space remained at the full 10.9G despite the virtual disk being 'empty'.
The NTFS delete operation won't have zeroed the blocks, instead it will have simply updated it's FS pointers/metadata to reflect that the blocks previously occupied by iobw.tst are now 'free'. There is no way for zfs to know that suddenly that these blocks are unreferenced so it's still faithfully storing them unaware of what's happening up the stack at the NTFS level. It was worth a shot!

Next I deleted the virtual disk from the VM using the VIC. Will vmfs free/zero the blocks??
No, it didn't and while I don't know as much about VMFS semantics I expect this result is a result of the same FS level operations/shortcuts as the NTFS test, but with VMFS this time. The zvol usage at this point was still 10.9GB.

That's 0/2 for my compressability tests.

Reusing blocks?

Next I wondered what would happen if I added and fill another 10GB disk on the now empty VMFS iscsi volume.
If VMFS were to reuse the same blocks that the previous 10GB virtual disk had used, then the vzol will simply update the contents of those blocks again, using no further disk space.

To my surprise this is exactly what happened. After creating and filling another 10GB disk and performing the same iometer procedure the zvol was still only using 10.9GB.
In actual fact it went from 10.9 down to around 100MB before climbing back to 10.9 again due to the iometer scratch file/zeroing behaviour that we observed earlier.

We now know that if guests write zeros blocks on their file systems, zfs compression will reclaim the blocks - though I should point out that this would be pretty unusual in a real environment!

VMFS block allocation policies

This most recent result made me even more curious about how VMFS allocates blocks on disks/LUNs. If the block allocater works in a very simple first to last LBA fashion with a preference to reuse blocks at the start of a volume then there may be some space savings to be realised if you churn VMs a lot as I will in my test setup here.
Modern filesystems have fragmentation and wear levelling considerations to factor in, which typically results in blocks being allocated all over the volume but since vmfs typically deals with few, but very large files rather than the millions of small files that are typically created by any modern OS perhaps the engineers at VMware decided to fill their filesystem across a disk/LUN in much the same fashion as filling a glass from bottom to top. As files are deleted, all new ones are created at beginning again!

To test this I started afresh with an empty VMFS volume. I created a 10GB virtual disk, and filled it up. I then added a second 15GB disk and filled this too.
Here is the ZFS view of that:

root@supernova WinXP-ESX]#zfs get used,compressratio,referenced Z/esxiscsi-compressed Z/esxiscsi
NAME PROPERTY VALUE SOURCE
Z/esxiscsi used 27.3G -
Z/esxiscsi compressratio 1.00x -
Z/esxiscsi referenced 27.3G -
Z/esxiscsi-compressed used 27.2G -
Z/esxiscsi-compressed compressratio 1.00x -
Z/esxiscsi-compressed referenced 27.2G - 

Next I deleted both the 10, and 15GB disks, and created a new 20GB disk.
I then filled the 20GB disk, and checked the zfs usage again.

Sure enough I was still using the previously allocated 25Gb (27.3G reported by zfs).

I could go on to test fragmentation and overlap, but I've learned all I wanted to find out today.

Take aways
Zvol sparse provisioning + iscsi +VMFS works out to be a very efficient and scalable storage system. The total disk usage will only be as much as your highest utilisation point, and with sparse provisioning you don't have to worry about extending your vmfs volume each time you need to add another VM.

I should also point out that I've been taking a worst case scenario of a completed full disk that has had every single block written to. In the real world free space within the VMs isn't initially going to take up space on the zvol until the OS's FS allocates those blocks with data. This will of course happen over time but in the short term your disk savings will be even better than what I've tested here.

ZFS compression doesn't appear to have any obvious benefits in terms of storage savings, but i'll have a play with some real world VMs that don't contain synthetic data and I'll post an update in the future with my compression ratio results.