The good old world of storage…

Typically in organisations there are two distinct roles assigned in the storage department.

1. Storage Admin – the person who provisions and supports primary storage to application or server admins

2. Backup Admin – the person who administers the backup software, tape solution and/or VTL

So where is the role Backup Storage Admin then?
Firstly we need to describe what is changing to open up the field for this new role. Traditionally all backup task and operations have been dragged through some kind of backup application. This consists of a number of components:

1. The Backup server – tracks and catalogs all backups, manages schedules and retention etc
2. Media or Storage node – performs the data movement of the data to be protected
3. Backup client – gathers and informs the server what files need backing up.

All of these components make up the backup product. They need specialist skills and training to understand how to implement them. The application guy doesn’t know the details of how NetBackup, Networker or TSM work, he understands how his application need protecting.

So what needs to change to the way backups are done to allow for this new role?

In our last post we discussed the DD Boost for RMAN feature of the EMC Data Domain. This allows the application guy to use his own backup tools to speak directly to the back end Data Domain storage. This puts the application admin in control of scheduling, retention and management of backups.

Scale this up and think about how this might develop. Imagine you had a VMWare admin who wanted to manage his own backups to a central Backup repository, or a NAS administer that wanted to point his NAS device directly at the backup storage device. Repeat the same question for any number of apps and databases.

No need for a dedicated backup software?
What we have described is that each application uses its own custom data protection method to send backups to a central backup storage device. This means that no longer is the backup admin managing all the database and file modules of a backup product, but is provisioning storage to application teams to be used for backups.  In fact the provisioning of the backup storage could be automated too so that when a service is brought online the backup storage is automatically provisioned, I will avoid using the word cloud at this point

Who will know what is backed up though!?
Good question! There would have to be a way to catalog all these disparate types of backups into a common format that could then be used to report and review backup success. Easier said than done you may think but not an impossibility by any means.

So in summary then, what would the responsibilities of a Backup Storage Admin be:
- Manage pot of storage specifically as a backup target
- Provision backup storage based on requests from application or server teams
- Manage the backup catalog
- Report on capacity trending
- Chargeback storage usage to the application or server teams

So surely this is all a pipe dream, well keep your eye on some of the stuff at EMC World 2012 and what Stephen Manley has to say here

Carrying on the theme from last post I can now talk about the new DD Boost software for Oracle RMAN.

What is DD Boost?
DD Boost is a software add-on to the Data Domain hardware that allows you to do a few funky things, namely:

- Distribute some of the steps in the dedupe process out to the media server or storage node.  This reduces the LAN traffic and weirdly the CPU load on the media server and the DD.

- Backup aware replication – Using the OST standard the DD can be sent commands from an OST compatible application to initiate the replication of backups.  This way the backup app is aware of all the copies of data even though the DD performed the replication process.

- Link Aggregation and Failover – The ports on the DD can be put into a DD Boost group meaning that the DD Boost software client on the host will balance the network traffic between them.  This improves the performance of backups.

So how does this apply to Oracle RMAN?
Up till now you have had a few options to back up Oracle to a DD:

1. Use the SBT_TAPE channel to write the data to a backup app client such as NetBackup or Networker

2. Use NFS or dNFS (this is NFS embedded in the Oracle code to improve performance) to the DD and get RMAN to write directly to the DD.

There was no way to use the SBT_TAPE method so that Oracle could write directly to the DD.  That is what DD Boost for Oracle RMAN provides.

The DD Boost software hijacks the SBT_TAPE channel (as all backup apps do) meaning when the RMAN script runs using that channel it will do the first steps of dedupe on the Oracle server, then send the optimized network traffic directly to the DD.

The other really cool feature is that the RMAN tool can keep in its catalog details of the DD replication copies.  This means that directly from the application you are aware of your level of protection.
So another way to skin the cat…

This is not going to be to everyone’s taste as it moves the control of the Oracle backups over to the DBA.  So it means that the backup admin can monitor what is happening but will have to go through the DBA and RMAN expert to change anything.

A good slide that covers the pro’s and con’s of the different methods is below, enjoy:

more info here – http://www.datadomain.com/pdf/h9687-so-emc-data-domain-rman.pdf

The Problem.
Oracle RMAN allows you to do different types of backups:
1. Full
2. Incremental
3. Image

There is another type that is called an Incremental Merge, this combines the Image and the Incremental to create a new full backup.  This is created by applying the latest incremental backup on top of your previous image.
The problem is that in doing this you have overwritten the previous full backup.  In order to overcome this, typically what happens is that before the merge operation occurs a backup of the previous full is taken to tape directly or via a backup app such as NetBackup, TSM or Networker.  The diagram below shows the process:

The steps are:
1. Perform Oracle Incr Merge to update Image location.
2. Perform Oracle RMAN backup through a backup app of the image
3. The data is received from RMAN and the Oracle Backup Module by the Backup Server.  The server then allocates tape or disk resources to write the data.  If this is a 10TB database then the entire amount is transferred over the network irrespective of if the destination storage is deduped.
4. 24 hours later the next Incr Merge can occur (only if the backup app has completed its backup of the previous day’s version)

The major problems in this config are:

- Bandwidth usage on the network between the Oracle Server and Backup App
- Work done by the dedupe appliance having to ingest / analyze and dedupe the entire database size every time.  This means CPU and memory load taken on the appliance.
- Workload on the Backup App server having to transfer the I/O for the entire database

Data Domain and Fastcopy
So with that back ground the wheeze I learnt about today allows you to eliminate the Backup App step from our example and allow multiple versions of the Incr Merge backups to be instantly available.  This assumes that you have a Data Domain as the backup storage and have a network link between the DD and the Oracle server (either CIFS or NFS).  The feature of DD that allows this miracle to occur is Fastcopy.  This is a command that you can run on any DD to produce a “near-instant” read-writable copy of any file or folder.  This is done by using duplicating the pointers to the segments.  To give you an example of the speed, a 10TB folder can be copied in 3 seconds!
The way that Fastcopy allows you to have multiple Image (Incr Merge) states is show below:

The Steps are:

1.Incr Merge backup is placed in a folder on the Data Domain (presented to the server via NFS or CIFS)

2. Data Domain Fastcopy command is initiated on the “Monday” folder and it is copied to the “Tuesday” folder (however the image inside this folder is the backup from Monday)

3. Oracle then catalogs the new Tuesday folder and finds that it has a second copy of the Monday backup.

4. The next Incremental Merge backup is initiated and Oracle picks the most recent backup with the given tag to apply the merge to.  This will be the backup in the Tuesday folder.

What we now have is multiple image backups (generated using the Incr Merge technique) available to the Oracle DBA.

The great things about this method are:

- Only the incremental backups are ever transferred over the network to the DD
- Less DD workload to ingest and analyze the data as it is only receiving changed blocks
- Database backup load is taken away from the Backup App server for other workloads
- Oracle DBA has full visibility of all the backups available
- Recoverability is improved from all Image type backups (rather than a mix of Full, Incr, Incr Merge)

So there you go! does that sound good to you?  I would be interested to hear from Oracle DBAs that have an opinion, which is all of you I think
 

Some of you may know that I recently made a move to the EMC BRS Division. Leaving behind the years of experience of TSM and the IBM storage portfolio, I decided to see what was going on over the fence in the noisy neighbours garden.

I have never had so much information thrown at me in such a short period of time. I have got a staggering stack of notes on my iPad for all the new tech that I am getting my head around. Products such as:

• Data Domain
• Avamar
• NetWorker
• Atmos

here are a few examples of my frantic note taking!

So really this post is a bit of a place holder to say that I will be posting good techie information on these products when I have been appropriately indoctrinated!

Over and out…

I want to failover my TSM 6 server!…

I have spoken to a number of customers that wanted to be able to failover their TSM 6 servers to another piece of hardware by remapping the LUNs that contain all of the database, log and disk storage pool data.  In TSM 5 this was very simple indeed because of the self contained nature of the database.  TSM 6 presents more of a challenge however.  The challenge comes from the DB2 database that is now used as the backend behind the TSM server.

In fact I was of the opinion that it would be such a faff that it would be easier to do a restore on the 2nd TSM server.  However once I had a go I was pleasantly surprised (and the customer I was with told me off for being so sceptical).

The process that we followed is detailed below:

The Servers:
2 x RHEL Servers both with the TSM 6.2.3.0 code installed (server1 is the primary TSM server, server2 is the DR TSM server)
TSM instance TSMINST1 running on the first machine.
The user that runs the instance is “tsminst1″

SAN disks used:

/tsm/TSMINST1 - home dir for user "tsminst1" folder contains dsmserv.opt, dsmserv.dsk, etc
/tsm/TSMINST1/db - database files
/tsm/TSMINST1/log - active log files
/tsm/TSMINST1/archlog - archive log files

The Process:
- halt TSMINST1 on server1
- Create the user on server2 using a tiny dummy bit of SAN disk to create the directory structure above
- Create a DB2 instance on server2 called TSMINST1

/opt/tivoli/tsm/db2/instance/db2icrt -a SERVER -u tsminst1 tsminst1

- Create the TSMDB1 database under that instance. (use a format or loadformat to do this).  Then stop DB2

 su - tsminst1
 /opt/tivoli/tsm/server/bin/dsmserv loadformat dbdir=/tsm/TSMINST1/db activelogdir=/tsm/TSMINST1/log mirrorlogdir=/tsm/TSMINST1/logm archlogdir=/tsm/TSMINST1/archlog
 db2stop

- Remove the dummy filesystem /tsm/TSMINST1 from server2 and import all the real filesystems from server1 (detatch the filesystems from server1 and import on server2).
- Start the DB2 database (using the db2start command)
- Start the TSMINST1 server

/opt/tivoli/tsm/server/bin/dsmserv

Magical Trevor - Disappearing a TSM server!

And guess what – the TSM server comes up. Happy days!

Now there maybe a better way of doing it than this but I figured that you need to trick DB2 that there was an instance of TSM on server2 and then replace all the files with the real files from the previously running instance.

Where might this be useful?

I can see that this process will be useful if you have two p795 frames that contain a SAP or Oracle server that uses PowerHA and when the TSM server is in the same frame as the active cluster node.  You may want to get TSM to follow the failover of your production server when it moves.  This would mean that the production backups can remain as an inter-frame activity.

Is this the only way?

no, in short there is a HADR DB2 feature that can be implemented to do TSM server mirroring.  See this as article:

http://www.ibm.com/developerworks/wikis/display/tivolistoragemanager/Electronic+vaulting+using+deduplicated+remote+copy+storage+pools

Sorry if that was a bit too techie…

Back in the day…

When I was a lad my elder brother and father were busy tinkering with the internals of a ZX Spectrum.  Unfortunately I was a little too young to be allowed into the same room as these amateur hackers (I would probably have done the annoying little brother thing of grabbing some ZX internals and running off to flush it down the loo).  The hackers enjoyed copying endless pages of code from magazines to perform tasks that only a hardened techie would get excited about.

I am now considerably older and only slightly more grown up, however I feel that I missed a key part of my computing education by not being able to fiddle in the fundamentals of a computer.  (The first hacking I did started on a 286 PC running DOS).

However all is not lost because a project based in Cambridge, UK is building a cheap and small computer that is designed for people to fiddle, furtle and poke it. The project is run by the Raspberry Pi foundation and the stated aim is to:

“…see cheap, accessible, programmable computers everywhere; we actively encourage other companies to clone what we’re doing. We want to break the paradigm where without spending hundreds of pounds on a PC, families can’t use the internet. We want owning a truly personal computer to be normal for children.”

So the aim of the project was to instil the enthusiasm for computing in the “up and coming“ generation.  There has been a huge hoo-ha here in the UK recently with people questioning the value of ICT lessons in schools.  The main issue being that the curriculum is too focused on using Office products rather than the fundamentals of computer science.

The hardware consists of the basic board (about the size of a credit card) with the chip (with RAM stacked on the top), SD slot for storage and OS, USB, HDMI, Power connector, RJ45 network port, Audio jack and a few connector ports.  The aim is that enthusiasts build add-on boards that link to the basic board to extend the functionality.  One of the Raspberry Pi team has already developed a board for operating motors. (http://www.raspberrypi.org/archives/500)

So what would you use it for?

Well other than it being fun to play with you could use it for a tiny media PC.  There is an example of the Pi being used as an AirPlay receiver! (http://www.raspberrypi.org/archives/564).  If you were doing the honourable education thing then it should become the core of some interesting school computing projects.  But make the projects interesting, some examples are:

- Parent warning system (pressure sensors in the landing carpet)
- Bedroom alarm system for unauthorised access to off limit areas
- Smelly breath detector for preventing those awkward discotheque moments
- Mobile DJ rig

But can I afford to have this toy?

So when I went home one evening and told my wife that I knew what my next gadget purchase would be, she rolled her eyes and imagined a 6 month delay to the kitchen re-fit.  However when I told her the price she smiled in a “whatever makes you happy” way.  The price is a staggeringly low $25 for the 128MB RAM model and $35 for the 256MB model! 

So hats off to the Cambridge boffins… go and get a slice of the Pi and support a good educational cause!

check out – http://www.raspberrypi.org/

Flash – AAAAH   aaaah…

 So EMC have released the first of their flash only arrays!  The VNX5500-F is based on the standard VNX chassis hardware but has a load of SSD drives shoved up it.

 The architecture is similar to the IBM STorwize v7000 model of disk system for block and additional “servers” to provide the file protocols.  EMC call the file modules, data movers (alternatively called X-Blade enclosures).

 EMC provide 3.5″ or 2.5″ SSD options for the VNX5500-F.  With the 2.5′ version you can install 25 x 100GB or 200GB SAS SSD’s.  The max number of trays you can attach is 15, meaning you can have 50TB of capacity (46.5TB of formatted storage).

 This capacity range seems to match the other “flash only” products such as the FA-310 from Pure Storage and the Violin 3100 series.  I am sure there would be lots of arguments from these flash purists that they would wipe the floor with the EMC VNX5500-F however EMC’s name does a lot to overcome these misgivings.

 The great thing with the VNX5500-F is that you can actually intermix all the other drive types that you can put in a standard VNX (SAS 10k, 15k and Near-line).  This mix of storage will then allow the tiering of storage as you would typically with an intermixed VNX.

 The other grand thing over some of the other smaller flash vendors is the functionality you get with the VNX such as, Block and File protocols, NDMP, Automatic tiering etc etc.

The future is flash…

 Anyway I am sure that this is the start of the trend that will become the norm. SSD will be the place your production data lives and the array will dynamically move your data through the tiers and then out to tape and into the vault.  A few years off but a very realistic prospect.

October was a busy month for announcements for IBM with the SVC 6.3, v7000 Unified, and TSM Flashcopy Manager (FCM) 3.1 amongst the announcements.  We have mentioned the v7000 unified previously but haven’t covered the FCM announcement.

Flashcopy Manager…

… is a software product in the Tivoli brand that allows you to manage snapshots of your IBM storage (and windows VSS snaps on any hardware) from one console.  It allows you to optionally perform a backup to TSM of the resulting hardware snaphost.  All of this functionality is performed through one interface.

What is in FCM Version 3.1

In the announcement in October IBM included the following new features:

• Advanced data protection and recovery features for VMware vSphere environments
• Enhanced data protection capabilities for Microsoft® Windows®, including support for New Technology File Systems (NTFS) and custom applications, and enhanced user interfaces for Microsoft Exchange and Microsoft SQL Server
• Support for IBM DB2® and Oracle databases (with or without SAP environments) on IBM AIX®, Solaris SPARC, Linux® x64, and HP-UX IA64 platforms
• Support for custom business-critical applications on IBM AIX, Solaris SPARC, Linux x64, HP-UX IA64, and Microsoft Windows platforms
• Transparent integration with IBM storage systems such as IBM System Storage® SAN Volume Controller space efficient FlashCopy target volumes, IBM Storwize® V7000, IBM XIV® Storage System, and IBM System Storage DS8000™
• Can leverage the Microsoft Volume Shadow Copy Services (VSS) framework for integration with non-IBM hardware subsystems
• Database cloning support for UNIX® and Linux clients

The diagram below shows the process of using hardware snapshots and TSM for VE:

In this first diagram we see the VMWare environment, FCM Linux proxy (either VM or Physical) and the TSM server.

The next diagram show the FCM proxy initiating VMWare software snapshots of the vmdk files that are owned by the VMs that we are protecting.  This process is using the standard vStorage API calls that are used by other products such as TSM for VE.

The third diagram shows the FCM software initiating a v7000 hardware snapshot (flashcopy) and removing the VMWare software snapshots so that the performance of the VM is not impacted during the backup.

The line from the snapshot through the FCM proxy to the TSM server shows the backup of the resulting snapshot into TSM.  The format of the backup into TSM uses the Common Data Format, this means that the data in TSM is in the structure that allows you to use the TSM for VE Recovery Agent.  This in turn means that you are able to recover individual files from a hardware snapshot backup that is stored in TSM!

At the moment the following configurations / restrictions apply:

• At the moment the following configurations / restrictions apply:
• FCM proxy is Linux only (RHEL or SUSE) – later will be Windows too
• If the FCM proxy is a VM, it can’t be on a datastore that is being backed up! (surprise)
• No SAN attachment required to the FCM proxy
• RDM and iSCSI attached volumes not supported
• When VMs span data stores the underlying disk system needs to be the same

Money Money Money!
So you are now saying to yourself “all well and good, but that is gonna cost!”.  Well actually if you purchase a IBM Storwize v7000 you can configure it to come with FCM.  In this configuration you can access a bigger discount than if you were purchasing a native v7000!  This incentive means that all v7000 quotes that I have seen recently have actually got this FCM licence bundled in.

Another reason to go get yourself a v7000 for xmas!
For more info:

Announcement:  http://www-01.ibm.com/common/ssi/cgi-bin/ssialias?infotype=AN&subtype=CA&htmlfid=897/ENUS211-363&appname=USN

 Rapid Application Storage: http://public.dhe.ibm.com/common/ssi/ecm/en/tsd03121usen/TSD03121USEN.PDF

Network Attached Storage is the NA(D)S!

So we all think it is a jolly good idea to take our file servers and migrate them to a NAS device.  We get the resilience of our file servers being on a dedicated storage appliance with failover components and RAID.  We are able to consolidate multiple fileservers into one appliance and so gain ease of management and administration.

Backup is always an afterthought

The one problem with consolidating file data onto a NAS is when it comes to backups!  Previously the backup agent on the fileserver scanned the files and backed them up to the backup server.  This may have taken ages because of the scan process, however you could at least take an incremental backup.

With a NAS you can’t put the backup agent onto the appliance hardware and so you are stuck with a number of different options:

1. CIFS / NFS mount the drives to a proxy.

With this option you are using a proxy server as the medium to get the data into the backup server.  You would install your backup agent onto the proxy server and then mount the shares that need protecting onto the proxy.

PROs – You are now able to do a progressive incremental backup (with TSM) or standard Grandfather-father-son backups using the standard backup agent.

CONs – You are dragging the data over the network twice.  The first hop is going from the NAS to the proxy and the second from the proxy to the backup server.  This could increase the backup time beyond the allowed window.  The other disadvantage is that the file systems are often sizable and the scan of the file system can be a very lengthy process.

2. NDMP backups

NDMP is a protocol that is implemented by the NAS vendors and allows a backup to be taken directly from the NAS to the backup server.  This backup can either be over the network to the backup server or other the SAN directly to tape.  The list of files that is included in the backup is transferred in a Table of Contents (TOC) and is stored alongside the data.  The backup is a block based backup and is very rudimentary.

A typical implementation would see a NAS being backed up using a Full backup every weekend and during the week differential (all changes since the last full) backups would be taken.

PROs – no additional server hardware is needed, backups direct to tape can occur, file and full recoveries are available.

CONs – NDMP is clunky and very simplistic, if you think about the size of your NAS, you have to perform a full backup of that during a weekend.  Let’s take an example of a 15TB NAS.  If you want to retain 4 weekly full’s and 12 months of monthly full’s you will need 240TB of backup storage (and capacity licence) on your backup server!  We then come to the recoveries, Full volume recoveries are great as it is a full block based recovery of your data.  File recovery (the one you are most likely to want) means that the TOC needs to be read first before the actual restore can be started.  essentially you are reading a file and folder structure that may have millions of objects in before initiating the restore of the single file you wanted.  This process I have seen run for the best part of a day!

3. 3rd Party integration

NetApp allow backup products to use the differential analysis of the changed data without using the NDMP protocol.  This SNAPDIFF function means that you need a proxy server with CIFS mount of the NAS shares and the standard backup agent set to know which file systems are on the NAS.  The Backup agent also needs to have stored the login details to the NAS so that it can logon and initial a snap.

PROs – brilliant for reducing the backup traffic for very large file volumes.  With this process not as many full backups are required.

CONs – You have to have a Netapp or IBM nSeries NAS

4. No traditional backup – use the force (of the NAS)

This is an option that a good number of people are taking when faced with the problem of how to protect a NAS.  Most NAS devices will have a snap-shotting ability.  These snapshots will be space efficient as they will only track the changes to the data, original data will be accessed via pointers.  The data in a NAS will also be replicated to a second site, and at this site too, snapshots of the data can be taken.  Some NAS vendors allow you to have different snapshot policies on the production and DR NAS heads.  This allows you to keep hourly snaps of the data in production and then daily, weekly, monthly snaps on the DR site.

PROs – Very space efficient, only the block level changes between snaps will be occupying space on the storage.  Very fast recovery, many NAS vendors integrate with VSS so the backups can be accessed by the user by right clicking on the file and looking at the older versions that are available.  Full recovery of volumes is possible using the replicated copy of the data.  You don’t clog up your backup system with repeated full backups of huge chucks of data.

CONs – Your primary storage and backups are in the same place!  You never really get a full copy of the data at a point in time (as the data is partly changed blocks and partly pointers).  You never get the data out to tape (sometimes this is a requirement).  You are locked into the NAS vendor tighter as your primary storage and backups reside within the one solution.

SUMMARY

The primary advantages of NAS’s are all well and good, however you need to consider how to protect the data before you decide on which one to go for.  Points to mull over:

1. How is the new backup method going to affect my backup licensing?
2. How will my backup window be effected?
3. What level of protection do I need (file, full, DR replica)?
4. What retention do I need on my backups (do I really need 125 million years of backups)?
5. Who do I want to perform restores? the user, the admins?

I personally would look to use the functions of the NAS as the first line of defence.  Using the snap-shotting tools gives some brilliant recovery advantages.  I would only look to NDMP as a DR “backs against the wall” recovery method.

IBM have gone all unified!!

In October this year IBM made a huge leap forward in the clash of the unified titans.  Netapp and EMC have already got Unified Storage systems (systems that provide file and block protocols), but IBM have now released the IBM Storwize v7000 Unified.

Recap!
The v7000 was essentially a 2 node IBM SVC plus the disk all in one box.  This was IBM’s attempt at stealing some of the mid market storage sales.  In my opinion the initial release (while lovely and compact and having a brilliant GUI) lacked a few key ingredients.  One of these was proper IP replication.  The majority of these original failings have been ironed out, but still the v7000 lacked the killer file protocol punch.

All unified and lovely
Well now it is there.  The v7000 Unified can provide FC, iSCSI, NFS, CIFS,  HTTPS and SCP protocols.  In addition to this, it isn’t like the usual IBM solutions where you have to log onto 13 different portals to operate it.  the functionality is provided using the updated v7000 software.  A GUI example of this is shown below:

As well as adding the file protocol goodness, IBM have also lobbed in ILM features to allow you to migrate your file data between tiers of storage based on the frequency of file use.  Again this is all driven through the GUI to build your migration policies.  This feature is based on the IBM Active Cloud engine, it allows the file data to be moved between tiers and even written out to TSM as the last tier – HOW COOL IS THAT!!  All of this is configured, yes you guessed it, in the GUI.  I think IBM should go and lie down.

The physical features of the Storwize v7000 Unified are the same as the standard Storwize v7000 but with some additional NAS head type boxes that are connected to the disk controllers using FC and Ethernet cables.  These additional boxes are called the v7000 Unified File Modules.

Some other additional features of the IBM Storwize v7000 Unified:

- Antivirus Software Integration – v7000 Unified is certified for use with the Symantec and McAfee ISV antivirus applications
- File replication – async replication for protection of NAS data across sites
- File level snapshots (up to 256 snapshots per file system).  This allows the recovery of data at a file level!

A practical application of the Storwize v7000 Unified:

QUESTION: Have you got your home directories or file share on a windows server that is a pain to backup and has loads of old data on?  Well all you need to do is:

- Buy an IBM Storwize v7000 Unified
- Migrate your CIFS share data onto it
- Setup ILM policies to migrate data older than 90 days to cheap storage (could be externally virtualised storage)
- Setup ILM policies to migrate data older than 6 months to TSM
- Write a report to tell your boss how good you are
- receive a 110% pay rise…

I must say IBM, you have excelled yourselves!

(screen grabs taken from https://www.ibm.com/developerworks/mydeveloperworks/blogs/ibmnas/entry/marriage_of_san_nas?lang=en with thanks)