Configuring NFS Kerberos using IBM Spectrum Scale

Disclaimer: The content of this post is not approved nor endorsed by IBM.

INTRODUCTION

User can configure Spectrum Scale (SS) System to use NFS service over kerberos. This blog explains how to configure Spectrum Scale System for NFS Kerberos using LDAP file authentication over MIT KDC.

The blog is divided into two sections – overview and details. Initially, an overview of all steps is provided and subsequently the details. Both the sections talk on following 8 sub topics.

1. various machines involved
2. pre-requisites
3. configure Spectrum Scale System for NFS kerberos ldap file authentication
4. configure NFS client for kerberos
5. create kerberos principals for ldap users
6. create kerberos NFS export
7. mount on client and access/read/write kerberos mount
8. required daemons running


Flow for NFS Kerberos configuration

Following steps and diagram depict the summary of NFS Kerberos configuration for Spectrum scale.

1. generate principal and keytab for Spectrum scale protocol nodes
2. Copy keytab to SS protocol node under /var/mmfs/tmp/
3. configure SS auth (mmuserauth/installer)
4. generate keytab for NFS client
5. copy keytab to NFS client
6. update kerberos configs on NFS client
7. generate keytab for LDAP users
8. start nfs-ganesha daemon on SS if not running
9. start rpcgssd and rpcidmapd on NFS client
10. create NFS export on SS
11. Mount kerberos NFS on NFS client
12. access/read/write Kerberos NFS mount without valid ticket – should fail
13. generate kerberos ticket for ldap user on NFS client
14. access/read/write Kerberos NFS mount with valid ticket – should succeed


 Diagram depicting NFS Kerberos configuration

nfskrbsetup

At the end of blog, few useful links are provided which user can refer e.g. KDC setup, kerberos troubleshooting.


OVERVIEW

This section briefly touches the topics that will be covered in this blog to configure Spectrum Scale System for NFS+LDAP+Kerberos using MIT KDC :

Various machines involved

  • LDAP server ==> OpenLDAP: slapd 2.4.23 (Apr 22 2013 05:03:41)
  • KDC (Key Distribution Center) ==> krb5-server-1.12.2-14.el7.x86_64
  • Spectrum Scale System protocol nodes
  • NFS client ==> nfs-utils-1.3.0-0.8.el7.x86_64

Pre-requisites

  • LDAP server is configured (How to setup LDAP server)
  • KDC i.e. Key Distribution Center is configured (How to configure KDC)
  • NFS client is set up as LDAP client (How to set up Linux machine as LDAP client)
  • All machines are time synchronized.
  • All machines involved in Kerberos are DNS configured (i.e. KDC, Spectrum Scale System Protocol Nodes, NFS clients).
  • An easy way out or alternative for this is to update /etc/hosts on all machines with each machine’s FQDN.

Configure Spectrum Scale System for NFS kerberos ldap file authentication

  • generate kerberos keytab file for all protocol nodes in Spectrum Scale System
  • copy kerberos keytab file to /var/mmfs/tmp/ on one of the Spectrum Scale System protocol node where you execute authentication configuration commands
  • execute installer or mmuserauth cli command to configure NFS kerberos file auth over ldap
  • validate Spectrum Scale System is properly configured with file authentication

Configure NFS client for kerberos

  • generate kerberos keytab file for NFS client
  • copy kerberos keytab file to /etc/krb5.keytab
  • update /etc/krb5.conf file
  • add following line into file /etc/sysconfig/NFS to use secured NFS
    • SECURE_NFS=”yes”
  • make sure gssd daemon is running, if not start it using either of following
    • ==> RHEL7 : systemctl restart nfs-secure; sysctemctl restart nfs-idmap
    • ==> RHEL6.5 : service rpcgssd restart; service rpcidmapd restart

Create kerberos principals for ldap users

  • this is required for accessing keberos NFS mounts uisng ldap users
  • remember the passwords we set for ldap user principal (need to provide them while creating ldap user tickets on NFS client)

Create kerberos NFS export

  • create and link fileset
  • create NFS kerberos export
  • create few files using ldap user under NFS kerberos export

Mount on client and access/read/write kerberos mount

  • create NFS mountpoint directory
  • mount NFS export on client using krb5/krb5i/krb5p security flavor
  • try to access kerberos NFS mount using ldap user when there is no valid ticket for that ldap user : SHOULD FAIL
  • create kerberos ticket for ldap user
  • try to access kerberos NFS mount using ldap user that has valid ticket : SHOULD SUCCEED
  • read/write to kerberos NFS mount using ldap user that has valid ticket : SHOULD SUCCEED

Required Daemons

  • NFS server / Spectrum Scale System protocol nodes : nfs-ganesha
  • NFS client : rpcgssd, rpcidmapd

DETAILS

Various machines involved

To explain the procedure, I will use following machines (all these are RHEL7.* machines) :

(1) KDC : kdc.example.com

(2) LDAP Server details :
ldap server : 192.168.122.55
base dn : dc=example,dc=com
bind user name : cn=Manager,dc=example,dc=com
bind user password : password
sample ldap users : ldapuser1, ldapuser2

(3) NFS client : (I’ll set up KDC and NFS client on same machine. They can be on different machines.)
kdc.example.com

(4) Spectrum Scale System Protocol Nodes :
192.168.65.167 protocolnode1.example.com protocolnode1
192.168.65.168 protocolnode2.example.com protocolnode2
192.168.65.169 protocolnode3.example.com protocolnode3
192.168.65.170 protocolnode4.example.com protocolnode4


Prerequisites

Prerequisites are already mentioned briefly in the previous section.
I won’t go into details for pre-requisites because each pre-requisite is a new blog subject itself and mentioning each pre-requisite in detail will divert intention of this blog.
However, I have provided the appropriate links for each pre-requisite which user can follow and fulfil the pre-requisite e.g. ldap server setup, ldap client setup, kdc setup etc..


Configure Spectrum Scale System for NFS kerberos ldap file authentication

generate kerberos keytab file for all protocol nodes

In order to use Spectrum Scale System NFS service over Kerberos, you need to generate a keytab file for protocol nodes in Spectrum Scale System cluster.
We need to create a principal NFS/<node-fqdn> for each protocol node — (execute this command on KDC).
In our example, we have four protocol nodes with names protocolnode1.example.com to protocolnode4.example.com. Hence, I will create one principal for each protocol node i.e. total 4 principals of the form NFS/<protocol-node-fqdn>.
Thereafter, I will add all these principals into one file which will be used as kerberos keytab file for Spectrum Scale System authentication configuration.

# login to KDC and generate keytab file for protocol nodes
[root@glogin02 ~]# ssh kdc.example.com
[root@kdc ~]#
[root@kdc ~]# kadmin.local
kadmin.local:
kadmin.local: addprinc -randkey nfs/protocolnode1.example.com
WARNING: no policy specified for nfs/protocolnode1.example.com@EXAMPLE.COM; defaulting to no policy
Principal “nfs/protocolnode1.example.com@EXAMPLE.COM” created.
kadmin.local:
kadmin.local: addprinc -randkey nfs/protocolnode2.example.com
WARNING: no policy specified for nfs/protocolnode2.example.com@EXAMPLE.COM; defaulting to no policy
Principal “nfs/protocolnode2.example.com@EXAMPLE.COM” created.
kadmin.local:
kadmin.local: addprinc -randkey nfs/protocolnode3.example.com
WARNING: no policy specified for nfs/protocolnode3.example.com@EXAMPLE.COM; defaulting to no policy
Principal “nfs/protocolnode3.example.com@EXAMPLE.COM” created.
kadmin.local:
kadmin.local: addprinc -randkey nfs/protocolnode4.example.com
WARNING: no policy specified for nfs/protocolnode4.example.com@EXAMPLE.COM; defaulting to no policy
Principal “nfs/protocolnode4.example.com@EXAMPLE.COM” created.
kadmin.local:
kadmin.local:
kadmin.local: ktadd -k /tmp/krb5.keytab nfs/protocolnode1.example.com
Entry for principal nfs/protocolnode1.example.com with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/krb5.keytab.
Entry for principal nfs/protocolnode1.example.com with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/krb5.keytab.
Entry for principal nfs/protocolnode1.example.com with kvno 2, encryption type des3-cbc-sha1 added to keytab WRFILE:/tmp/krb5.keytab.
Entry for principal nfs/protocolnode1.example.com with kvno 2, encryption type arcfour-hmac added to keytab WRFILE:/tmp/krb5.keytab.
Entry for principal nfs/protocolnode1.example.com with kvno 2, encryption type camellia256-cts-cmac added to keytab WRFILE:/tmp/krb5.keytab.
Entry for principal nfs/protocolnode1.example.com with kvno 2, encryption type camellia128-cts-cmac added to keytab WRFILE:/tmp/krb5.keytab.
Entry for principal nfs/protocolnode1.example.com with kvno 2, encryption type des-hmac-sha1 added to keytab WRFILE:/tmp/krb5.keytab.
Entry for principal nfs/protocolnode1.example.com with kvno 2, encryption type des-cbc-md5 added to keytab WRFILE:/tmp/krb5.keytab.
kadmin.local:
kadmin.local: ktadd -k /tmp/krb5.keytab nfs/protocolnode2.example.com
Entry for principal nfs/protocolnode2.example.com with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/krb5.keytab.
Entry for principal nfs/protocolnode2.example.com with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/krb5.keytab.
Entry for principal nfs/protocolnode2.example.com with kvno 2, encryption type des3-cbc-sha1 added to keytab WRFILE:/tmp/krb5.keytab.
Entry for principal nfs/protocolnode2.example.com with kvno 2, encryption type arcfour-hmac added to keytab WRFILE:/tmp/krb5.keytab.
Entry for principal nfs/protocolnode2.example.com with kvno 2, encryption type camellia256-cts-cmac added to keytab WRFILE:/tmp/krb5.keytab.
Entry for principal nfs/protocolnode2.example.com with kvno 2, encryption type camellia128-cts-cmac added to keytab WRFILE:/tmp/krb5.keytab.
Entry for principal nfs/protocolnode2.example.com with kvno 2, encryption type des-hmac-sha1 added to keytab WRFILE:/tmp/krb5.keytab.
Entry for principal nfs/protocolnode2.example.com with kvno 2, encryption type des-cbc-md5 added to keytab WRFILE:/tmp/krb5.keytab.
kadmin.local:
kadmin.local: ktadd -k /tmp/krb5.keytab nfs/protocolnode3.example.com
Entry for principal nfs/protocolnode3.example.com with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/krb5.keytab.
Entry for principal nfs/protocolnode3.example.com with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/krb5.keytab.
Entry for principal nfs/protocolnode3.example.com with kvno 2, encryption type des3-cbc-sha1 added to keytab WRFILE:/tmp/krb5.keytab.
Entry for principal nfs/protocolnode3.example.com with kvno 2, encryption type arcfour-hmac added to keytab WRFILE:/tmp/krb5.keytab.
Entry for principal nfs/protocolnode3.example.com with kvno 2, encryption type camellia256-cts-cmac added to keytab WRFILE:/tmp/krb5.keytab.
Entry for principal nfs/protocolnode3.example.com with kvno 2, encryption type camellia128-cts-cmac added to keytab WRFILE:/tmp/krb5.keytab.
Entry for principal nfs/protocolnode3.example.com with kvno 2, encryption type des-hmac-sha1 added to keytab WRFILE:/tmp/krb5.keytab.
Entry for principal nfs/protocolnode3.example.com with kvno 2, encryption type des-cbc-md5 added to keytab WRFILE:/tmp/krb5.keytab.
kadmin.local:
kadmin.local: ktadd -k /tmp/krb5.keytab nfs/protocolnode4.example.com
Entry for principal nfs/protocolnode4.example.com with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/krb5.keytab.
Entry for principal nfs/protocolnode4.example.com with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/krb5.keytab.
Entry for principal nfs/protocolnode4.example.com with kvno 2, encryption type des3-cbc-sha1 added to keytab WRFILE:/tmp/krb5.keytab.
Entry for principal nfs/protocolnode4.example.com with kvno 2, encryption type arcfour-hmac added to keytab WRFILE:/tmp/krb5.keytab.
Entry for principal nfs/protocolnode4.example.com with kvno 2, encryption type camellia256-cts-cmac added to keytab WRFILE:/tmp/krb5.keytab.
Entry for principal nfs/protocolnode4.example.com with kvno 2, encryption type camellia128-cts-cmac added to keytab WRFILE:/tmp/krb5.keytab.
Entry for principal nfs/protocolnode4.example.com with kvno 2, encryption type des-hmac-sha1 added to keytab WRFILE:/tmp/krb5.keytab.
Entry for principal nfs/protocolnode4.example.com with kvno 2, encryption type des-cbc-md5 added to keytab WRFILE:/tmp/krb5.keytab.
kadmin.local:
kadmin.local: [root@kdc ~]#
[root@kdc ~]#
[root@kdc ~]# klist -kt /tmp/krb5.keytab
Keytab name: FILE:/tmp/krb5.keytab
KVNO Timestamp Principal
—- ——————- ——————————————————
2 06/08/2015 21:33:45 nfs/protocolnode1.example.com@EXAMPLE.COM
2 06/08/2015 21:33:45 nfs/protocolnode1.example.com@EXAMPLE.COM
2 06/08/2015 21:33:45 nfs/protocolnode1.example.com@EXAMPLE.COM
2 06/08/2015 21:33:45 nfs/protocolnode1.example.com@EXAMPLE.COM
2 06/08/2015 21:33:45 nfs/protocolnode1.example.com@EXAMPLE.COM
2 06/08/2015 21:33:45 nfs/protocolnode1.example.com@EXAMPLE.COM
2 06/08/2015 21:33:45 nfs/protocolnode1.example.com@EXAMPLE.COM
2 06/08/2015 21:33:45 nfs/protocolnode1.example.com@EXAMPLE.COM
2 06/08/2015 21:34:00 nfs/protocolnode2.example.com@EXAMPLE.COM
2 06/08/2015 21:34:00 nfs/protocolnode2.example.com@EXAMPLE.COM
2 06/08/2015 21:34:00 nfs/protocolnode2.example.com@EXAMPLE.COM
2 06/08/2015 21:34:00 nfs/protocolnode2.example.com@EXAMPLE.COM
2 06/08/2015 21:34:00 nfs/protocolnode2.example.com@EXAMPLE.COM
2 06/08/2015 21:34:00 nfs/protocolnode2.example.com@EXAMPLE.COM
2 06/08/2015 21:34:00 nfs/protocolnode2.example.com@EXAMPLE.COM
2 06/08/2015 21:34:00 nfs/protocolnode2.example.com@EXAMPLE.COM
2 06/08/2015 21:34:06 nfs/protocolnode3.example.com@EXAMPLE.COM
2 06/08/2015 21:34:06 nfs/protocolnode3.example.com@EXAMPLE.COM
2 06/08/2015 21:34:06 nfs/protocolnode3.example.com@EXAMPLE.COM
2 06/08/2015 21:34:06 nfs/protocolnode3.example.com@EXAMPLE.COM
2 06/08/2015 21:34:06 nfs/protocolnode3.example.com@EXAMPLE.COM
2 06/08/2015 21:34:06 nfs/protocolnode3.example.com@EXAMPLE.COM
2 06/08/2015 21:34:06 nfs/protocolnode3.example.com@EXAMPLE.COM
2 06/08/2015 21:34:06 nfs/protocolnode3.example.com@EXAMPLE.COM
2 06/08/2015 21:34:09 nfs/protocolnode4.example.com@EXAMPLE.COM
2 06/08/2015 21:34:09 nfs/protocolnode4.example.com@EXAMPLE.COM
2 06/08/2015 21:34:09 nfs/protocolnode4.example.com@EXAMPLE.COM
2 06/08/2015 21:34:09 nfs/protocolnode4.example.com@EXAMPLE.COM
2 06/08/2015 21:34:09 nfs/protocolnode4.example.com@EXAMPLE.COM
2 06/08/2015 21:34:09 nfs/protocolnode4.example.com@EXAMPLE.COM
2 06/08/2015 21:34:09 nfs/protocolnode4.example.com@EXAMPLE.COM
2 06/08/2015 21:34:09 nfs/protocolnode4.example.com@EXAMPLE.COM
[root@kdc ~]#

copy kerberos keytab file to /var/mmfs/tmp/

# copy Spectrum Scale System krb5.keytab file and store it under /var/mmfs/tmp path on a protocol node where user will run installer or mmuserauth cli command.
[root@node1 ~]# scp kdc.example.com:/tmp/krb5.keytab /var/mmfs/tmp/
root@kdc.example.com’s password:
krb5.keytab 100% 3586 3.5KB/s 00:00
[root@node1 ~]#
[root@node1 ~]# klist -kt /var/mmfs/tmp/krb5.keytab
Keytab name: FILE:/var/mmfs/tmp/krb5.keytab
KVNO Timestamp Principal
—- ——————- ——————————————————
2 06/08/2015 21:33:45 nfs/protocolnode1.example.com@EXAMPLE.COM
2 06/08/2015 21:33:45 nfs/protocolnode1.example.com@EXAMPLE.COM
2 06/08/2015 21:33:45 nfs/protocolnode1.example.com@EXAMPLE.COM
2 06/08/2015 21:33:45 nfs/protocolnode1.example.com@EXAMPLE.COM
2 06/08/2015 21:33:45 nfs/protocolnode1.example.com@EXAMPLE.COM
2 06/08/2015 21:33:45 nfs/protocolnode1.example.com@EXAMPLE.COM
2 06/08/2015 21:33:45 nfs/protocolnode1.example.com@EXAMPLE.COM
2 06/08/2015 21:33:45 nfs/protocolnode1.example.com@EXAMPLE.COM
2 06/08/2015 21:34:00 nfs/protocolnode2.example.com@EXAMPLE.COM
2 06/08/2015 21:34:00 nfs/protocolnode2.example.com@EXAMPLE.COM
2 06/08/2015 21:34:00 nfs/protocolnode2.example.com@EXAMPLE.COM
2 06/08/2015 21:34:00 nfs/protocolnode2.example.com@EXAMPLE.COM
2 06/08/2015 21:34:00 nfs/protocolnode2.example.com@EXAMPLE.COM
2 06/08/2015 21:34:00 nfs/protocolnode2.example.com@EXAMPLE.COM
2 06/08/2015 21:34:00 nfs/protocolnode2.example.com@EXAMPLE.COM
2 06/08/2015 21:34:00 nfs/protocolnode2.example.com@EXAMPLE.COM
2 06/08/2015 21:34:06 nfs/protocolnode3.example.com@EXAMPLE.COM
2 06/08/2015 21:34:06 nfs/protocolnode3.example.com@EXAMPLE.COM
2 06/08/2015 21:34:06 nfs/protocolnode3.example.com@EXAMPLE.COM
2 06/08/2015 21:34:06 nfs/protocolnode3.example.com@EXAMPLE.COM
2 06/08/2015 21:34:06 nfs/protocolnode3.example.com@EXAMPLE.COM
2 06/08/2015 21:34:06 nfs/protocolnode3.example.com@EXAMPLE.COM
2 06/08/2015 21:34:06 nfs/protocolnode3.example.com@EXAMPLE.COM
2 06/08/2015 21:34:06 nfs/protocolnode3.example.com@EXAMPLE.COM
2 06/08/2015 21:34:09 nfs/protocolnode4.example.com@EXAMPLE.COM
2 06/08/2015 21:34:09 nfs/protocolnode4.example.com@EXAMPLE.COM
2 06/08/2015 21:34:09 nfs/protocolnode4.example.com@EXAMPLE.COM
2 06/08/2015 21:34:09 nfs/protocolnode4.example.com@EXAMPLE.COM
2 06/08/2015 21:34:09 nfs/protocolnode4.example.com@EXAMPLE.COM
2 06/08/2015 21:34:09 nfs/protocolnode4.example.com@EXAMPLE.COM
2 06/08/2015 21:34:09 nfs/protocolnode4.example.com@EXAMPLE.COM
2 06/08/2015 21:34:09 nfs/protocolnode4.example.com@EXAMPLE.COM
[root@node1 ~]#

configure NFS Kerberos file auth over MIT KDC

# This can be achieved using two methods – mmuserauth cli command or installer
# Make sure there is no file authentication configured on Spectrum Scale System.
# If file authentication exists already, at first remove it using ‘/usr/lpp/mmfs/bin/mmuserauth service remove –data-access-method file’.

[root@node1 ~]# mmuserauth service list
FILE access not configured
PARAMETERS VALUES
————————————————-

OBJECT access configuration : LOCAL
PARAMETERS VALUES
————————————————-
ENABLE_KS_SSL false
ENABLE_KS_CASIGNING false
KS_ADMIN_USER admin

# try configuring ldap+kerberos for NFS using either of the method mentioned below
# method 1 – mmuserauth cli command
# method 2 – installer

== method 1 – mmuserauth cli command ==

# please see below as an example, if /var/mmfs/tmp/krb5.keytab file is missing for Spectrum Scale System, will get following error

[root@node1 ~]# mmuserauth service create –data-access-method file –type ldap –servers 192.168.122.55 –base-dn dc=example,dc=com –user-name cn=Manager,dc=example,dc=com –password password –enable-kerberos –kerberos-server kdc.example.com –kerberos-realm EXAMPLE.COM –netbios-name spectrumscale
/var/mmfs/tmp/krb5.keytab: [E] File not found at specified location
mmuserauth service create: Command failed. Examine previous error messages to determine cause.
[root@node1 ~]#

# execute mmuserauth cli command to configure ldap+kerberos for NFS on Spectrum Scale System
# the values provided here are provided above in “various machines involved” section

[root@node1 ~]# mmuserauth service create –data-access-method file –type ldap –servers 192.168.122.55 –base-dn dc=example,dc=com –user-name cn=Manager,dc=example,dc=com –password password –enable-kerberos –kerberos-server kdc.example.com –kerberos-realm EXAMPLE.COM –netbios-name spectrumscale
File authentication configuration completed successfully.
[root@node1 ~]#

== method 2 – installer ==

# configure NFS+kerberos using installer
# as a part of installer configuration, execute following step so that installer sets up ldap+kerberos file auth on Spectrum Scale System

[root@spctscl2nod1 installer]# ./spectrumscale auth file ldap
[ INFO ] A configuration template has been created at configuration/authconfig.txt. Please open this file in the text editor of your choice and complete the template.
Would you like to open this file now? [Y/n]: Y
Are you ready to save your changes now? [Y/n]: Y
[ INFO ] Your authentication settings have been merged into the main cluster defintion file (configuration/clusterdefinition.txt).
[root@spctscl2nod1 installer]#

# installer opens config for file ldap auth in an editor,
# user has to fill in the values e.g. ldap server, kerberos server etc and save it
# providing herewith file auth ldap configuration file which I used for setting up my system

# For detailed information on the authentication settings refer to the manual
# page for mmuserauth service create, located in /usr/lpp/mmfs/bin/

[file_auth]
enable_file_auth = True
backend_server = ldap

[file_ldap]
; mandatory settings for file authentication:
servers = 192.168.122.55
netbios_name = spectrumscale
base_dn = dc=example,dc=com
bind_username = cn=Manager,dc=example,dc=com
bind_password = password

; additional settings for file authentication:
;user_dn =
;group_dn =
;netgroup_dn =

; optional settings for file authentication: (initialised with default values)
; If required, please un-comment and specify your environment configuration
;user_name_attribute = cn
;user_id_attribute = uid
;user_id_attribute = mail
;group_objectclass = posixGroup
;user_objectclass = posixAccount

; If required, please un-comment and specify the paths to your certificates
enable_server_tls = False
;path_to_tls_certificate =

enable_kerberos = True
kerberos_server = kdc.example.com
kerberos_realm = EXAMPLE.COM
;path_to_kerberos_keytab =

validate NFS Kerberos file auth

# validation after configuring Spectrum Scale System with ldap+NFS+kerberos
# now you should be able to resolve LDAP users on Spectrum Scale System

[root@node1 ~]# mmuserauth service list
FILE access configuration : LDAP
PARAMETERS VALUES
————————————————-
ENABLE_SERVER_TLS false
ENABLE_KERBEROS true
USER_NAME cn=Manager,dc=example,dc=com
SERVERS 192.168.122.55
NETBIOS_NAME spectrumscale
BASE_DN dc=example,dc=com
USER_DN none
GROUP_DN none
NETGROUP_DN none
USER_OBJECTCLASS posixAccount
GROUP_OBJECTCLASS posixGroup
USER_NAME_ATTRIB cn
USER_ID_ATTRIB uid
KERBEROS_SERVER kdc.example.com
KERBEROS_REALM EXAMPLE.COM

OBJECT access configuration : LOCAL
PARAMETERS VALUES
————————————————-
ENABLE_KS_SSL false
ENABLE_KS_CASIGNING false
KS_ADMIN_USER admin

[root@node1 ~]#
[root@node1 ~]# id ldapuser2
uid=7002(ldapuser2) gid=7002(grp7002) groups=7002(grp7002)
[root@node1 ~]#
[root@node1 ~]# id ldapuser1
uid=7001(ldapuser1) gid=7001(grp7001) groups=7001(grp7001)
[root@node1 ~]#


Configure NFS client for kerberos

generate kerberos keytab file for NFS client

In this example, my KDC and NFS client is on same machine (you need to scp/ftp keytab file if they are on different machines)

[root@glogin02 ~]# ssh kdc.example.com
[root@kdc ~]# kadmin.local
Authenticating as principal root/admin@EXAMPLE.COM with password.
kadmin.local:
kadmin.local: addprinc -randkey nfs/kdc.example.com
WARNING: no policy specified for nfs/kdc.example.com@EXAMPLE.COM; defaulting to no policy
Principal “nfs/kdc.example.com@EXAMPLE.COM” created.
kadmin.local:
kadmin.local: ktadd -k /tmp/nfsclient.keytab nfs/kdc.example.com
Entry for principal nfs/kdc.example.com with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/nfsclient.keytab.
Entry for principal nfs/kdc.example.com with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/nfsclient.keytab.
Entry for principal nfs/kdc.example.com with kvno 2, encryption type des3-cbc-sha1 added to keytab WRFILE:/tmp/nfsclient.keytab.
Entry for principal nfs/kdc.example.com with kvno 2, encryption type arcfour-hmac added to keytab WRFILE:/tmp/nfsclient.keytab.
Entry for principal nfs/kdc.example.com with kvno 2, encryption type camellia256-cts-cmac added to keytab WRFILE:/tmp/nfsclient.keytab.
Entry for principal nfs/kdc.example.com with kvno 2, encryption type camellia128-cts-cmac added to keytab WRFILE:/tmp/nfsclient.keytab.
Entry for principal nfs/kdc.example.com with kvno 2, encryption type des-hmac-sha1 added to keytab WRFILE:/tmp/nfsclient.keytab.
Entry for principal nfs/kdc.example.com with kvno 2, encryption type des-cbc-md5 added to keytab WRFILE:/tmp/nfsclient.keytab.
kadmin.local:
kadmin.local: [root@kdc ~]#

copy kerberos keytab file to /etc/krb5.keytab

[root@kdc ~]# cp /tmp/nfsclient.keytab /etc/krb5.keytab
[root@kdc ~]# ls -ltr /etc/krb5.keytab
-rw——-. 1 root root 794 May 13 12:15 /etc/krb5.keytab
[root@kdc ~]#
[root@kdc ~]# klist -kt
Keytab name: FILE:/etc/krb5.keytab
KVNO Timestamp Principal
—- ——————- ——————————————————
3 05/13/2015 12:14:41 nfs/kdc.example.com@EXAMPLE.COM
3 05/13/2015 12:14:41 nfs/kdc.example.com@EXAMPLE.COM
3 05/13/2015 12:14:41 nfs/kdc.example.com@EXAMPLE.COM
3 05/13/2015 12:14:41 nfs/kdc.example.com@EXAMPLE.COM
3 05/13/2015 12:14:41 nfs/kdc.example.com@EXAMPLE.COM
3 05/13/2015 12:14:41 nfs/kdc.example.com@EXAMPLE.COM
3 05/13/2015 12:14:41 nfs/kdc.example.com@EXAMPLE.COM
3 05/13/2015 12:14:41 nfs/kdc.example.com@EXAMPLE.COM
[root@kdc ~]#

update /etc/krb5.conf file with correct KDC

(or you can simply scp Spectrum Scale System protocol node’s /etc/krb5.conf on NFS client)

[root@kdc ~]# cp /etc/krb5.conf /etc/krb5.conf.backup
[root@kdc ~]# scp inkfilsh1:/etc/krb5.conf /etc/krb5.conf
[root@kdc ~]# diff /etc/krb5.conf.backup /etc/krb5.conf
11c11
< default_realm = NAS.EXAMPLE.COM

> default_realm = EXAMPLE.COM
16,17c16,18
< NAS.EXAMPLE.COM = {
< kdc = 9.3.101.219:88

> EXAMPLE.COM = {
> kdc = kdc.example.com:88
> admin_server = kdc.example.com:749
21,22c22,23
< .nas.example.com = NAS.EXAMPLE.COM
< nas.example.com = NAS.EXAMPLE.COM

> .example.com = EXAMPLE.COM
> example.com = EXAMPLE.COM
[root@kdc ~]#

update /etc/sysconfig/NFS to use secured NFS

# this is RHEL client specific setting, it will vary as per your client platrorm

[root@kdc ~]# grep SECURE_NFS /etc/sysconfig/nfs
SECURE_NFS=”yes”
[root@kdc ~]#

start required kerberos daemons on NFS client

# we start gssd daemon on RHEL client platform

[root@kdc ~]# cat /etc/redhat-release
Red Hat Enterprise Linux Server release 7.1 (Maipo)
[root@kdc ~]#
[root@kdc ~]# systemctl restart nfs-idmap

[root@kdc ~]# systemctl restart nfs-secure

[root@kdc ~]# ps -eaf | grep idmap
root 4320 1 0 00:19 ? 00:00:00 /usr/sbin/rpc.idmapd
root 4327 3813 0 00:19 pts/0 00:00:00 grep –color=auto idmap

[root@kdc ~]# ps -eaf | grep gss
root 4325 1 0 00:19 ? 00:00:00 /usr/sbin/rpc.gssd
root 4337 3813 0 00:19 pts/0 00:00:00 grep –color=auto gss
root 25079 1 0 May13 ? 00:00:05 /usr/sbin/gssproxy -D
[root@kdc ~]#


create kerberos principals for ldap users

# create kerberos principal for few ldap users that will participate in NFS+kerberos operations
# please remember the password you provide here for each ldap user principal
# while creating ldap user kerberos tickets you will be asked for these passwords

[root@glogin02 ~]# ssh kdc.example.com
[root@kdc ~]# kadmin.local
Authenticating as principal root/admin@EXAMPLE.COM with password.
kadmin.local:
kadmin.local: addprinc ldapuser1
WARNING: no policy specified for ldapuser1@EXAMPLE.COM; defaulting to no policy
Enter password for principal “ldapuser1@EXAMPLE.COM”:
Re-enter password for principal “ldapuser1@EXAMPLE.COM”:
Principal “ldapuser1@EXAMPLE.COM” created.
kadmin.local:
kadmin.local: addprinc ldapuser2
WARNING: no policy specified for ldapuser2@EXAMPLE.COM; defaulting to no policy
Enter password for principal “ldapuser2@EXAMPLE.COM”:
Re-enter password for principal “ldapuser2@EXAMPLE.COM”:
Principal “ldapuser2@EXAMPLE.COM” created.
kadmin.local:


create kerberos NFS export

create and link fileset

[root@node1 ~]# mmcrfileset fs0 nfsfset1
Fileset nfsfset1 created with id 2 root inode 413441.
[root@node1 ~]#

[root@node1 ~]# mmlinkfileset fs0 nfsfset1 -J /ibm/fs0/nfsfset1
Fileset nfsfset1 linked at /ibm/fs0/nfsfset1
[root@node1 ~]#

[root@node1 ~]# chmod -R 777 /ibm/fs0/nfsfset1
[root@node1 ~]#

using ldap user; create few files and directories inside fileset

[root@node1 ~]# su smbuser25 -c “echo ganapati > /ibm/fs0/nfsfset1/smbuser25_file1”
[root@node1 ~]#
[root@node1 ~]# su smbuser25 -c “mkdir /ibm/fs0/nfsfset1/smbuser25_dir1”
[root@node1 ~]#
[root@node1 ~]# su ldapuser2 -c “echo bappamoryare > /ibm/fs0/nfsfset1/ldapuser2_file1”
[root@node1 ~]#
[root@node1 ~]# ls -ld /ibm/fs0/nfsfset1
drwxrwxrwx 3 root root 4096 Jun 3 20:52 /ibm/fs0/nfsfset1
[root@node1 ~]#
[root@node1 ~]# ls -ltr /ibm/fs0/nfsfset1
total 0
-rw-r–r– 1 smbuser25 domadms 9 Jun 3 20:51 smbuser25_file1
drwxr-xr-x 2 smbuser25 domadms 4096 Jun 3 20:52 smbuser25_dir1
-rw-r–r– 1 ldapuser2 grp7002 13 Jun 3 20:52 ldapuser2_file1
[root@node1 ~]#

create NFS export

[root@node1 ~]# mmnfs export add /ibm/fs0/nfsfset1 –client \*\(ACCESS_TYPE=RW,SQUASH=no_root_squash,SECTYPE=sys:krb5:krb5i:krb5p\)
192.168.65.140: Redirecting to /bin/systemctl stop nfs-ganesha.service
192.168.65.162: Redirecting to /bin/systemctl stop nfs-ganesha.service
192.168.65.164: Redirecting to /bin/systemctl stop nfs-ganesha.service
192.168.65.163: Redirecting to /bin/systemctl stop nfs-ganesha.service
192.168.65.163: Redirecting to /bin/systemctl start nfs-ganesha.service
192.168.65.140: Redirecting to /bin/systemctl start nfs-ganesha.service
192.168.65.164: Redirecting to /bin/systemctl start nfs-ganesha.service
192.168.65.162: Redirecting to /bin/systemctl start nfs-ganesha.service
[root@node1 ~]#

validate NFS export is created

[root@node1 ~]# mmnfs export list -n /ibm/fs0/nfsfset1

Path Delegations Clients Access_Type Protocols Transports Squash Anonymous_uid Anonymous_gid SecType PrivilegedPort Export_id DefaultDelegation Manage_Gids nfs_Commit
————————————————————————————————————————————————————————————————————-
/ibm/fs0/nfsfset1 none * RW 3,4 TCP NO_ROOT_SQUASH -2 -2 SYS,KRB5,KRB5I,KRB5P false 3 none false false
[root@node1 ~]#
[root@node1 ~]# cat /var/mmfs/ces/nfs-config/gpfs.ganesha.exports.conf
# CHANGELOG ADD #3
#
# end changelog —————————————————————-

EXPORT {
Attr_Expiration_Time=60;
Delegations=none;
Export_id=3;
Filesystem_id=666.666;
MaxOffsetRead=18446744073709551615;
MaxOffsetWrite=18446744073709551615;
MaxRead=1048576;
MaxWrite=1048576;
Path=”/ibm/fs0/nfsfset1″;
PrefRead=1048576;
PrefReaddir=1048576;
PrefWrite=1048576;
Pseudo=”/ibm/fs0/nfsfset1″;
Tag=”/ibm/fs0/nfsfset1″;
UseCookieVerifier=true;
FSAL {
Name=GPFS;
}
CLIENT {
Access_Type=RW;
Anonymous_gid=-2;
Anonymous_uid=-2;
Clients=*;
Delegations=none;
Manage_Gids=false;
nfs_Commit=false;
PrivilegedPort=false;
Protocols=3,4;
SecType=SYS,KRB5,KRB5I,KRB5P;
Squash=NO_ROOT_SQUASH;
Transports=TCP;
}
}

[root@node1 ~]#


NFS client operations

mount NFS export

[root@kdc ~]# ps -eaf | grep gss
root 12662 12510 0 20:54 pts/0 00:00:00 grep –color=auto gss
root 25079 1 0 May13 ? 00:00:04 /usr/sbin/gssproxy -D
[root@kdc ~]#
[root@kdc ~]# systemctl restart nfs-secure
[root@kdc ~]#
[root@kdc ~]# ps -eaf | grep gss
root 12668 1 0 20:54 ? 00:00:00 /usr/sbin/rpc.gssd
root 12670 12510 0 20:54 pts/0 00:00:00 grep –color=auto gss
root 25079 1 0 May13 ? 00:00:04 /usr/sbin/gssproxy -D
[root@kdc ~]#
[root@kdc ~]# showmount -e protocolnode2.example.com
Export list for protocolnode2.example.com:
/ibm/fs0/nfsfset1 *
[root@kdc ~]#
[root@kdc ~]# mkdir -p /mnt/8k5 /mnt/8k5i /mnt/8k5p
[root@kdc ~]# mount -t nfs -o vers=4,sec=krb5 protocolnode2.example.com:/ibm/fs0/nfsfset1 /mnt/8k5 -vvvv
mount.nfs: timeout set for Wed Jun 3 20:56:57 2015
mount.nfs: trying text-based options ‘vers=4,sec=krb5,addr=192.168.65.168,clientaddr=9.3.101.93’
[root@kdc ~]#
[root@kdc ~]# mount -t nfs -o vers=4,sec=krb5i protocolnode2.example.com:/ibm/fs0/nfsfset1 /mnt/8k5i -vvvv
mount.nfs: timeout set for Wed Jun 3 20:56:57 2015
mount.nfs: trying text-based options ‘vers=4,sec=krb5i,addr=192.168.65.168,clientaddr=9.3.101.93’
[root@kdc ~]#
[root@kdc ~]# mount -t nfs -o vers=4,sec=krb5p protocolnode2.example.com:/ibm/fs0/nfsfset1 /mnt/8k5p -vvvv
mount.nfs: timeout set for Wed Jun 3 20:56:57 2015
mount.nfs: trying text-based options ‘vers=4,sec=krb5p,addr=192.168.65.168,clientaddr=9.3.101.93’
[root@kdc ~]#
[root@kdc ~]# mount | grep /ibm/fs0/nfsfset1
protocolnode2.example.com:/ibm/fs0/nfsfset1 on /mnt/8k5 type nfs4 (rw,relatime,vers=4.0,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,port=0,timeo=600,retrans=2,sec=krb5,clientaddr=9.3.101.93,local_lock=none,addr=192.168.65.168)
protocolnode2.example.com:/ibm/fs0/nfsfset1 on /mnt/8k5i type nfs4 (rw,relatime,vers=4.0,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,port=0,timeo=600,retrans=2,sec=krb5i,clientaddr=9.3.101.93,local_lock=none,addr=192.168.65.168)
protocolnode2.example.com:/ibm/fs0/nfsfset1 on /mnt/8k5p type nfs4 (rw,relatime,vers=4.0,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,port=0,timeo=600,retrans=2,sec=krb5p,clientaddr=9.3.101.93,local_lock=none,addr=192.168.65.168)
[root@kdc ~]#

access without valid ticket

# it fails as expected

[root@kdc ~]# su ldapuser1
bash-4.2$
bash-4.2$ klist
klist: Credentials cache keyring ‘persistent:7001:7001’ not found
bash-4.2$
bash-4.2$ cd /mnt/8k5
bash: cd: /mnt/8k5: Permission denied
bash-4.2$
bash-4.2$ ls -ltr /mnt/8k5
ls: cannot access /mnt/8k5: Permission denied
bash-4.2$

creating kerberos ticket for ldap user

# Please note below, while creating kerberos tickets using ldap users,
# I provided the same password which we set while creating kerberos principals for those ldap users

[root@kdc ~]# su ldapuser1
bash-4.2$
bash-4.2$ kinit -l 3600s

Password for ldapuser1@EXAMPLE.COM:
bash-4.2$
bash-4.2$ klist
Ticket cache: KEYRING:persistent:7001:7001
Default principal: ldapuser1@EXAMPLE.COM

Valid starting Expires Service principal
06/03/2015 20:55:56 06/03/2015 21:55:56 krbtgt/EXAMPLE.COM@EXAMPLE.COM
bash-4.2$

access with valid ticket

# it succeeds
# basic krb5 NFSv4 file operations succeed i.e. access, read, write …

bash-4.2$ cd /mnt/8k5
bash-4.2$
bash-4.2$ ls -ltr
total 0
-rw-r–r–. 1 smbuser27 domadms 9 Jun 3 20:51 smbuser25_file1
drwxr-xr-x. 2 smbuser27 domadms 4096 Jun 3 20:52 smbuser25_dir1
-rw-r–r–. 1 ldapuser2 grp7002 13 Jun 3 20:52 ldapuser2_file1
bash-4.2$
bash-4.2$ echo yeaaayeaaa > ldapuser1_client_file1
bash-4.2$
bash-4.2$ ls -ltr
total 0
-rw-r–r–. 1 smbuser27 domadms 9 Jun 3 20:51 smbuser25_file1
drwxr-xr-x. 2 smbuser27 domadms 4096 Jun 3 20:52 smbuser25_dir1
-rw-r–r–. 1 ldapuser2 grp7002 13 Jun 3 20:52 ldapuser2_file1
-rw-r–r–. 1 ldapuser1 grp7001 11 Jun 3 20:56 ldapuser1_client_file1
bash-4.2$
bash-4.2$ cat *
yeaaayeaaa
bappamoryare
cat: smbuser25_dir1: Is a directory
ganapati
bash-4.2$
bash-4.2$ exit
[root@kdc ~]#


required daemons for kerberos

Spectrum Scale System Daemon

gpces service start NFS -N cesNodes

NFS client Daemons

RHEL6.* client : service rpcidmapd restart; service rpcgssd restart
RHEL7.* client : systemctl nfs-idmap restart; systemctl nfs-secure restart


LINKS REFERRED

http://www.certdepot.net/rhel7-configure-kerberos-kdc/
http://www.certdepot.net/rhel7-use-kerberos-control-access-NFS-network-shares/
http://suresh-chandra.blogspot.in/2013/08/configuring-openldap-serverclient-on.html
https://wiki.ncsa.illinois.edu/display/ITS/Kerberos+Troubleshooting+for+Unix#KerberosTroubleshootingforUnix-general

Advertisements

2 thoughts on “Configuring NFS Kerberos using IBM Spectrum Scale

  1. Excellent post, just a quick question. When you create the nfs principles for the protocol nodes, are these for the physical nodes or for each of the pool addresses that the nodes host?

    Like

    1. Sorry for the late reply.

      We provide netbios name in ‘mmuserauth service create’ cli command and for that netbios name we create nfs principals in latest spectrum scale release 4.2.* e.g. If we run ‘mmuserauth service create –data-access-method file –type ldap –netbios-name sscluster –kerberos-realm XYZ.COM … ‘; then we need to create nfs principals as nfs/sscluster@XYZ.COM or nfs/sscluster.xyz.com@XYZ.COM. Ensure to copy this keytab file at /var/mmfs/tmp/krb5.keytab before you execute ‘mmuserauth service create’ cli command for ldap-kerberos.

      Please note that we update /etc/hosts file on nfs client where spectrum scale netbios name points to the ces ips of cluster. This way it is just the pool of ces ips that matter most.

      Like

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s