GPFS troubleshooting

# ./mmlscluster

GPFS cluster information
========================
  GPFS cluster name:         b570gw1-gpfs
  GPFS cluster id:           723401712311966547
  GPFS UID domain:           b570gw1-gpfs
  Remote shell command:      /usr/bin/ssh
  Remote file copy command:  /usr/bin/scp

GPFS cluster configuration servers:
-----------------------------------
  Primary server:    b570gw1-gpfs
  Secondary server:  b570gw2-gpfs

 Node  Daemon node name            IP address       Admin node name             Designation   
-----------------------------------------------------------------------------------------------
   1   b570gw1-gpfs                10.10.10.6       b570gw1-gpfs                quorum-manager
   2   b570gw2-gpfs                10.10.10.8       b570gw2-gpfs                quorum-manager
   3   b570its-gpfs                10.10.10.24      b570its-gpfs                quorum-manager
   4   b570was-gpfs                10.10.10.21      b570was-gpfs               




# /usr/lpp/mmfs/bin/mmlsdisk gw1data
disk         driver   sector failure holds    holds                            storage
name         type       size   group metadata data  status        availability pool
------------ -------- ------ ------- -------- ----- ------------- ------------ ------------
gpfs12nsd    nsd         512      -1 yes      yes   ready         up           system      
gpfs13nsd    nsd         512      -1 yes      yes   ready         up           system      
gpfs14nsd    nsd         512      -1 yes      yes   ready         up           system      
gpfs15nsd    nsd         512      -1 yes      yes   ready         up           system      

# /usr/lpp/mmfs/bin/mmlsdisk gw2data
disk         driver   sector failure holds    holds                            storage
name         type       size   group metadata data  status        availability pool
------------ -------- ------ ------- -------- ----- ------------- ------------ ------------
gpfs16nsd    nsd         512      -1 yes      yes   ready         up           system      
gpfs17nsd    nsd         512      -1 yes      yes   ready         up           system      
gpfs18nsd    nsd         512      -1 yes      yes   ready         up           system      
gpfs19nsd    nsd         512      -1 yes      yes   ready         up           system      



# ./mmlsnsd -f gw1data

 File system   Disk name    NSD servers                                   
---------------------------------------------------------------------------
 gw1data       gpfs12nsd    (directly attached)     
 gw1data       gpfs13nsd    (directly attached)     
 gw1data       gpfs14nsd    (directly attached)     
 gw1data       gpfs15nsd    (directly attached)     



# ./mmfsadm test readdesc gpfs12nsd
reading descriptor from gpfs12nsd ...
    uid 0916D908:4C43E4F2, name "gw1data", config version 15
    created on node 9.22.217.8, Mon Jul 19 01:38:58 2010
    formatting system: architecture=Power/PC, OS=AIX, big-endian
    original format version 1105, cur version 1105 (mgr 1105, helper 1105, mnode 1105)
    read quorum {2,2}, write quorum {2,2} (out of {3,3}) migrate complete
    stripe group state
    desc area size: reserved 2089984 = 2041k, currently used 2220
    deny write|all lock semantic (SMBCIFSSemantic): nfs4
    prefetch buffers 0
    sector size 512, metadataAtomicSectorSize 512, subblocksPerFullBlock 32
    subblock size 32768 (64 sectors), full block size 1048576 (2048 sectors)
    inode file block size 32 subblks, alloc map 32, inode alloc map 32
    inodes: size 512 (1 sectors), inodes per block 2048, disk addrs 32
    indirect blocks: size 32768 (64 sectors), disk addrs 2727
    log files: 4191744 data bytes
    ialloc map: type 'cluster', regns 131, segmnts per regn 1, segmnt size 31784 bytes (8 pages)


# ./mmlsmount all
File system aadata is mounted on 4 nodes.
File system ccdata is mounted on 4 nodes.
File system egwdata is mounted on 4 nodes.
File system fed_data is mounted on 4 nodes.
File system gw1data is not mounted.
File system gw2data is not mounted.
File system logs1 is mounted on 2 nodes.
File system logs2 is mounted on 4 nodes.
File system miscdata is mounted on 4 nodes.
File system mqdata is mounted on 4 nodes.
File system nfsdata is mounted on 4 nodes.
File system presdata is not mounted.
File system tables1 is mounted on 4 nodes.
File system tables2 is mounted on 4 nodes.
File system tsdata is mounted on 4 nodes.


    inodes per ialloc subsegment 4096, segment 126976 (active 12288)
    max allowed replication factor: 2; partial replica allocation flags 0
    disk address size: 12
    default replication factors: metadata 1 (max 2), data 1 (max 2)
    stripe method RoundRobin
    disk I/O parallelism 1
    dir block size 32768 (1 subblks), dir alignment 32
    longIOWarnLen 0.000000 sec
    nNodes 32, avgFileSize 1048576, total no of inodes 1100806, reserved 6
    maxInodes 1100806, surplusInodes 15533050 addReservedInodes 4000 Num 38
    nFullBlocks 1100800, nMFullBlocks 1100800
    unbalanced user files: some, metadata files: some
    improper replication of user files: some, metadata files: some
    exposed user files: some, metadata files: some
    no of unsuspended failure groups: total 4, data 4, metadata 4
    supported ACL semantics: all
    failureGroupBitmapP[0]: Bitmap size 4 1s 1 : 80000000
    failureGroupBitmapP[1]: Bitmap size 4 1s 1 : 40000000
    failureGroupBitmapP[2]: Bitmap size 4 1s 1 : 20000000
    failureGroupBitmapP[3]: Bitmap size 4 1s 1 : 10000000
    dataDisksBitmapP: Bitmap size 4 1s 4 : F0000000
    metadataDisksBitmapP: Bitmap size 4 1s 4 : F0000000
    logPoolId: 0
    quota file inodes: user quota -1, group quota -1 fileset quota -1
    extended attribute file inode: 5
    allocation summary file inode: -1
    fileset metadata file inode: 38
    policy file inode: -1
    Aggressiveness level of tokens prefetch: 0
    EnableDmapi: no
    EnablePnfs: no
    EnableFsetDF: no
    dmSGEventList: 0x00000000
    EnableSANergy yes
    diskConfigVersion 1
    sgnameChgInProgress 0
    inode 0: 1:4098
    snapshot: version 0 nItem 0 max 256 maxId 0 filtered 0 mtime Wed Dec 31 19:00:00 1969
    snapdir ".snapshots" disabled (rootdir only) latest "NotDefined"
    restore: flags 0 Dirty resIdSnapId 0 resIdVersion 1
             resIdStripeId 0916D908:4C43E4F2 resIdTime 0
    Fileset record size: 4096
    Has pcache filesets: no
    storage pools: max 8
       0: name 'system' Valid nDisks 4 nInUse 4 id 0 type 'cluster'
           regns 1121 segs 1 size 32768 FBlks 1100800 MBlks 1100800
           alloc map inode 1 sum inode 39
           data 1 nFGs 4 disks: Bitmap size 4 1s 4 : F0000000
           meta 1 nFGs 4 disks: Bitmap size 4 1s 4 : F0000000
       1: not in use
       2: not in use
       3: not in use
       4: not in use
       5: not in use
       6: not in use
       7: not in use
    tschfsVersion 0
    disks: version 1, no of items 4, in use 4
      catQuorumWeight {0,0}
        1: gpfs12nsd: uid 0916D908:4C43E4EB, status InUse, availability OK,
             created on node xxx.xxx.xxx.xxx, Mon Jul 19 01:38:51 2010
             type 'nsd', sector size 512, failureConfigVersion 0
             quorum weight {1,1}, failure group: id -1, fg index 0
             nSectors 52428800 (0:3200000) (25600 MB), inode0Sector 4098
             alloc region: no of bits 736, seg num 0, offset 48, len 256
             suballocator 0xF1000001F118BEA4 type 0 subSize 736 dataOffset 52
               nRows 5 len/off: 1/0 2/1 3/3 6/6 12/12
             storage pool: 0
             holds data and metadata
             sectors past efficient device boundary: 0
             isFenced: 0
        2: gpfs13nsd: uid 0916D908:4C43E4ED, status InUse, availability OK,
             created on node xxx.xxx.xxx.xxx, Mon Jul 19 01:38:53 2010
             type 'nsd', sector size 512, failureConfigVersion 0
             quorum weight {1,1}, failure group: id -1, fg index 1
             nSectors 524288000 (0:1F400000) (256000 MB), inode0Sector 4098
             alloc region: no of bits 7328, seg num 0, offset 304, len 1904
             suballocator 0xF1000001F118BD74 type 0 subSize 7328 dataOffset 308
               nRows 8 len/off: 1/0 2/1 4/3 8/7 15/15 29/30 58/59 115/117
             storage pool: 0
             holds data and metadata
             sectors past efficient device boundary: 0
             isFenced: 0
        3: gpfs14nsd: uid 0916D908:4C43E4EF, status InUse, availability OK,
             created on node xxx.xxx.xxx.xxx, Mon Jul 19 01:38:55 2010
             type 'nsd', sector size 512, failureConfigVersion 0
             quorum weight {1,1}, failure group: id -1, fg index 2
             nSectors 524288000 (0:1F400000) (256000 MB), inode0Sector 4098
             alloc region: no of bits 7328, seg num 0, offset 2208, len 1904
             suballocator 0xF1000001F118BC44 type 0 subSize 7328 dataOffset 2212
               nRows 8 len/off: 1/0 2/1 4/3 8/7 15/15 29/30 58/59 115/117
             storage pool: 0
             holds data and metadata
             sectors past efficient device boundary: 0
             isFenced: 0
        4: gpfs15nsd: uid 0916D908:4C43E4F1, status InUse, availability OK,
             created on node xxx.xxx.xxx.xxx, Mon Jul 19 01:38:57 2010
             type 'nsd', sector size 512, failureConfigVersion 0
             quorum weight {0,0}, failure group: id -1, fg index 3
             nSectors 1153433600 (0:44C00000) (563200 MB), inode0Sector 4098
             alloc region: no of bits 16096, seg num 0, offset 4112, len 4096
             suballocator 0xF1000001F118BB14 type 0 subSize 16096 dataOffset 4116
               nRows 9 len/off: 1/0 2/1 4/3 8/7 16/15 32/31 63/63 126/126 252/252
             storage pool: 0
             holds data and metadata
             sectors past efficient device boundary: 0
             isFenced: 0
    logs: version 1, no of items 32, maxLogGroupNum 32
     group  1, addr 1:26138624, sectors 8187, inode    6, genNum 0x1
     group  2, addr 2:262221824, sectors 8187, inode    7, genNum 0x1
     group  3, addr 3:262070272, sectors 8187, inode    8, genNum 0x1
     group  4, addr 4:576643072, sectors 8187, inode    9, genNum 0x1
     group  5, addr 1:26134528, sectors 8187, inode   10, genNum 0x1
     group  6, addr 2:262225920, sectors 8187, inode   11, genNum 0x1
     group  7, addr 3:262066176, sectors 8187, inode   12, genNum 0x1
     group  8, addr 4:576638976, sectors 8187, inode   13, genNum 0x1
     group  9, addr 1:26130432, sectors 8187, inode   14, genNum 0x1
     group 10, addr 2:262230016, sectors 8187, inode   15, genNum 0x1
     group 11, addr 3:262062080, sectors 8187, inode   16, genNum 0x1
     group 12, addr 4:576634880, sectors 8187, inode   17, genNum 0x1
     group 13, addr 1:26126336, sectors 8187, inode   18, genNum 0x1
     group 14, addr 2:262234112, sectors 8187, inode   19, genNum 0x1
     group 15, addr 3:262057984, sectors 8187, inode   20, genNum 0x1
     group 16, addr 4:576630784, sectors 8187, inode   21, genNum 0x1
     group 17, addr 1:26122240, sectors 8187, inode   22, genNum 0x1
     group 18, addr 2:262238208, sectors 8187, inode   23, genNum 0x1
     group 19, addr 3:262053888, sectors 8187, inode   24, genNum 0x1
     group 20, addr 4:576626688, sectors 8187, inode   25, genNum 0x1
     group 21, addr 1:26118144, sectors 8187, inode   26, genNum 0x1
     group 22, addr 2:262242304, sectors 8187, inode   27, genNum 0x1
     group 23, addr 3:262049792, sectors 8187, inode   28, genNum 0x1
     group 24, addr 4:576622592, sectors 8187, inode   29, genNum 0x1
     group 25, addr 1:26114048, sectors 8187, inode   30, genNum 0x1
     group 26, addr 2:262246400, sectors 8187, inode   31, genNum 0x1
     group 27, addr 3:262045696, sectors 8187, inode   32, genNum 0x1
     group 28, addr 4:576618496, sectors 8187, inode   33, genNum 0x1
     group 29, addr 1:26109952, sectors 8187, inode   34, genNum 0x1
     group 30, addr 2:262250496, sectors 8187, inode   35, genNum 0x1
     group 31, addr 3:262041600, sectors 8187, inode   36, genNum 0x1
     group 32, addr 4:576614400, sectors 8187, inode   37, genNum 0x1
# 


# ./mmlsconfig
Configuration data for cluster b570gw1-gpfs:
--------------------------------------------
clusterName b570gw1-gpfs
clusterId 723401712311966547
autoload yes
minReleaseLevel 3.3.0.2
dmapiFileHandleSize 32
pagepool 4G
seqDiscardThreshhold 8388608
writebehindThreshhold 1045876
prefetchThreads 250
worker1Threads 150
maxMBpS 1500
maxFilesToCache 2000
adminMode central

File systems in cluster b570gw1-gpfs:
-------------------------------------
/dev/aadata
/dev/ccdata
/dev/egwdata
/dev/fed_data
/dev/gw1data
/dev/gw2data
/dev/logs1
/dev/logs2
/dev/miscdata
/dev/mqdata
/dev/nfsdata
/dev/presdata
/dev/tables1
/dev/tables2
/dev/tsdata


# ./mmmount all -a

b570gw1-gpfs:  mmremote: Remount failed for file system /dev/presdata. Error code 52.
b570was-gpfs:  mmremote: Remount failed for file system /dev/presdata. Error code 52.
mmmount: The following nodes could not be reached:
b570gw2-gpfs