Posted on 2010/03/26 08:47
Filed Under 클러스터란/고성능연산_HPC 조회수: view 432018

HOWTO - Building a Diskless Linux Cluster

Author: Mohamad Sindi
Email:
sindimo@ieee.org

__________________________________________________________________________________________________

This is a detailed procedure of how to install a cluster of diskless Linux nodes. The procedure if fully done via command line tools so it is convenient to script and use for large scale clusters. My operating system of choice was RedHat EL 5.3 but it should work with other distributions as well. RedHat's documentation doesn't seem to have any procedure on doing this using command line, they only illustrate it using a GUI tool which is not convenient at all to use when you have a large number of nodes in a diskless cluster in an enterprise enviornment, so I thought it would be useful to share this with everyone, I hope you find it useful.
__________________________________________________________________________________________________


Outline:

1. Introduction
2. Setting up the Server Node
3. Setting the Diskless Environment via Command Line
4. Booting the Diskless Clients
5. Modifying the Golden Image Node


1. Introduction


This is a step by step procedure to set up a cluster of diskless compute nodes along with a node acting as the server for the diskless environment. The operating system of choice was RedHat EL 5.3 using an x86_64 IBM Infiniband cluster. We cover how to do it in specific details via RedHat’s netboot. We will use the command line version of it since the GUI version might not be an efficient method to use when setting a large number of diskless nodes.

Types of nodes that are used for this demonstration:
• node002 (Image server – IP 10.8.226.72)
• node003 (Diskless client – IP 10.8.226.73)
• node004 (Golden image node installed on disk – IP 10.8.226.74)


2. Setting up the Server Node


First of all, install the server node node002 with all RedHat packages for simplicity (use “@Everything” in kickstart packages section) and disable firewall and selinux.

Install one of the client nodes, e.g. node004, normally on disk and include whatever settings and software you want to have on it, this will be the golden source for our diskless image.

It is very important to make sure that the “busybox-anaconda” package is installed on the client golden image node and on the server node since it’s needed for diskless environments:
[root@node002 ~]# rpm -qa | grep busybox-anaconda
busybox-anaconda-1.2.0-4.el5
[root@node004 ~]# rpm -qa | grep busybox-anaconda
busybox-anaconda-1.2.0-4.el5


Create the root diskless directory (this will be shared with all diskless clients in Read Only mode):
[root@node002 ~]# mkdir -p /diskless/i386/RHEL5-AS/root

Copy your golden image (node004) to the server (make sure to exclude /proc and /sys):
[root@node002 ~]# rsync -a -e ssh --exclude='/proc/*' --exclude='/sys/*' node004:/ /diskless/i386/RHEL5-AS/root/

Verify it has been copied:
[root@node002 ~]# du -hs /diskless/i386/RHEL5-AS/root
1.6G /diskless/i386/RHEL5-AS/root


Make sure tftp is running on the server:
[root@node002 ~]# chkconfig xinetd start
[root@node002 ~]# chkconfig tftp on
[root@node002 ~]# chkconfig --list | grep tftp
tftp: on


Setup your DHCP configuration file, I used the below settings (obviously modify it with your specific IPs, MACs, PXE booting NIC, hostnames, DNS, etc…):

[root@node002 ~]# cat /etc/dhcpd.conf

Code:
authoritative;
ddns-update-style none;

allow bootp;
allow booting;

shared-network eth0 {

subnet 10.8.226.0 netmask 255.255.255.0 {
        option routers		        10.8.226.250; # --- default gateway
	option subnet-mask		255.255.255.0;
        option nis-domain	        "<YOUR-DOMAIN>";
	option domain-name		"<YOUR-SEARCH-DOMAIN>";
	option domain-name-servers	<YOUR-DNS-IP>;
	option time-offset		-18000;	# Eastern Standard Time
	default-lease-time 		21600;
	max-lease-time 			43200;
        next-server 		        10.8.226.72; #<--- TFTP server IP here

	host node003 {
		hardware ethernet 00:21:5E:40:64:5C;
		fixed-address 10.8.226.73;
        	              filename "linux-install/pxelinux.0";
        		}

	host node004 {
		hardware ethernet 00:21:5E:40:63:C8;
		fixed-address 10.8.226.74;
        	              	filename "linux-install/pxelinux.0";
	             }
   }
}
Make sure the DHCP is running:
[root@node002 ~]# chkconfig dhcpd on
[root@node002 ~]# service dhcpd start
[root@node002 ~]# service dhcpd status
dhcpd (pid 28250) is running...


Set up NFS (notice the read-only and read-write settings):
[root@node002 ~]# cat /etc/exports
/diskless/i386/RHEL5-AS/root *(ro,sync,no_root_squash)
/diskless/i386/RHEL5-AS/snapshot *(rw,sync,no_root_squash)
[root@node002 ~]# chkconfig nfs on
[root@node002 ~]# service nfs start
[root@node002 ~]# showmount -e node002

Export list for node002:
/diskless/i386/RHEL5-AS/root *
/diskless/i386/RHEL5-AS/snapshot *



3. Setting the Diskless Environment via Command Line


Now we need to add clients to the diskless environment and create a snapshot image for each diskless client. The system-config-netboot is a RedHat GUI utility to do so. However, what if you have a cluster with hundreds of nodes and you want to add them to the diskless environment, it would be a hassle to use the GUI to do so. There are 2 commands that come with the system-config-netbook package which are called “pxeos” and “pxeboot”. The “pxeos” enables you to create a new operating system under /tftpboot and places the vmlinux and initrd images there. The “pxeboot” enables you to add new hosts to the diskless environment. The snapshot image for each diskless client contains data that the client has read-write access to, for example /var/log/messages and so on.

Below is how to add a new O.S. called “rhel5compute” using “pxeos”:
[root@node002 ~]# pxeos -a -i rhel5compute -p NFS -D 1 -s 10.8.36.26 -L /diskless/i386/RHEL5-AS rhel5compute
Kernel not specified, using 2.6.18-128.el5


To verify that the O.S. was created:
[root@ node002 ~]# pxeos -l
rhel5compute
Description: rhel5compute
Protocol: NFS
isDiskless: True
Server: 10.8.36.26
Location: /diskless/i386/RHEL5-AS


[root@ node002 ~]# ls /tftpboot/linux-install/rhel5compute/
initrd.img vmlinuz


Options for “pxeos”:
-a: Adds new O.S.
-i: Description of O.S.
-p: Protocol to use
-D: If set to 1, then it indicates it’s diskless
-s: NFS server
-L: Location of your O.S., this should contain the vmlinuz and initrd somewhere in it.

The “pxeboot” command adds clients to the diskless environment and creates the HEX file for the host in the /tftpboot directory and it also creates the snapshot directory for the individual nodes being added. You can script it to do all of this easily, below is an example for a 128 node cluster.

First thing create a file with the IPs and hostnames that need to be added, you can use the entries found in your /etc/hosts file if they exist:
[root@node128 ~]# cat nodes

Code:
10.8.136.1	node001
10.8.136.2	node002
10.8.136.3	node003
………………………
10.8.136.128	node128
Now use the script below to add all 128 nodes to your diskless environment:
[root@node128 ~]# cat add-pxe-hosts

Code:
#!/bin/bash
#sindimo

#This script will generate the HEX files in /tftpboot as well as the individual snapshoot directories for nodes. The nodes file has 2 columns, first one is the IP and the second is the hostname

while read line
do 
ip=`echo $line | gawk -F" " '{ print $1 }'`
host=`echo $line | gawk -F" " '{ print $2 }'`

pxeboot -a -O rhel5compute -r 28753 -S $host -e eth0 -N my_nis_domain -s console=ttyS0,115200n8r     $ip

done < nodes
Options for “pxeboot”:
-a: Adds new host
-O: Name of O.S. (You should have already created this with the system-config-netboot or pxeos)
-r: Size of ram disk
-S: Snapshot name of the diskless client.
-e: The interface through which the pxe boot will take place.
-N: NIS domain
-s: Serial console settings to view remote consoles during boot up

Now verify that all 128 hex files for tftpboot were created:
[root@node128 ~]# ls /tftpboot/linux-install/pxelinux.cfg/ | grep 0A | wc -l
128


An example of the content of one of the HEX files for node node001:
[root@node128 ~]# cat /tftpboot/linux-install/pxelinux.cfg/0A088801

Code:
default rhel5compute

label rhel5compute
    kernel rhel5compute/vmlinuz
    append console=ttyS0,9600n8 initrd=rhel5compute/initrd.img root=/dev/ram0 init=disklessrc NFSROOT=10.8.226.72:/diskless/i386/RHEL5-AS ramdisk_size=28753 ETHERNET=eth0 SNAPSHOT=node001 NISDOMAIN=my_nis_domain

Verify that all 128 snapshots were created:
[root@node128 ~]# ls /diskless/i386/RHEL5-AS/snapshot/ | grep node | wc -l
128


Each snapshot image should have a unique name since its data is specific to that client and the client has read-write access to it.
[root@node002 ~]# du -hs /diskless/i386/RHEL5-AS/snapshot/node003
80M /diskless/i386/RHEL5-AS/snapshot/node003
[root@node002 ~]# ls /diskless/i386/RHEL5-AS/snapshot/node003
boot etc home lib root var


Verify that the snapshot directory contains a text file called “files” which lists the files that can be mounted in read-write mode. Do not modify the “files” file manually, if you need to add more read-write files in your diskless system, create a new file called “files.custom” with the same format as “files” and list the desired files in it:

[root@node002 snapshot]# cat /diskless/i386/RHEL5-AS/snapshot/files

Code:
#
# This file contains the list of files/directories to be stored in the 
# snapshot directory for each diskless client.  Please do not edit this file,
# as Red Hat will be updating it with each release. If you wish to 
# add files please create a files.custom in this directory and add entries to it.
#
/boot/kernel.h
/etc/resolv.conf
/etc/resolv.conf.predhclient
/etc/yp.conf
/etc/yp.conf.predhclient
/etc/ntp.conf
/etc/ntp.conf.predhclient
/etc/ntp/step-tickers
/etc/ntp/step-tickers.predhclient
/etc/adjtime
/etc/cups/certs/
/etc/reader.conf
/etc/smartd.conf
/etc/printcap
/etc/aliases.db
/etc/modules.conf
/etc/fstab
/etc/sysconfig/hwconf
/etc/X11/XF86Config
/etc/X11/XF86Config.backup
/etc/X11/xorg.conf
/etc/X11/xorg.conf.backup
/etc/ssh/
/etc/selinux/
/etc/blkid/
/var/empty/sshd/
/var/gdm/
/var/lib/nfs/
/var/lib/random-seed
/var/lib/xkb/
/var/tmp/
/var/lock/
/var/run/
/var/spool/
/var/log/
/var/lib/logrotate.status
/var/cache/man/
/var/lib/misc/
/var/lib/mrtg/
/var/lib/dhcp/
/var/lib/dhclient/
/var/lib/ntp/
/var/lib/setroubleshoot/
/var/yp/binding/
/root/
/home/
Verify that the kernel images have been created:
[root@node002 ~]# ls /tftpboot/linux-install/rhel5compute
initrd.img vmlinuz



4. Booting the Diskless Clients


At this stage the server is ready. Now we need to boot the clients into their diskless environment. Make sure that your client’s BIOS and NIC cards support PXE boot and that that they are setup in the BIOS to PXE boot via eth0 since our DHCP was configured with the MACs of eth0.

You can verify that the tftpboot file for the diskless client is ready by checking the tftpboot directory and file below for node003:
[root@node002 ~]# cat /tftpboot/linux-install/pxelinux.cfg/0A08E249

Code:
default rhel5compute

label rhel5compute
    kernel rhel5compute/vmlinuz
    append  initrd=rhel5compute/initrd.img root=/dev/ram0 init=disklessrc NFSROOT=10.8.226.72:/diskless/i386/RHEL5-AS ramdisk_size=28760 ETHERNET=eth0 SNAPSHOT=node003 NISDOMAIN=my_nis_domain
When a diskless client boots, the DHCP server will assign it an IP and the vmlinux and initrd.img images will be loaded via tftp to the memory of the diskless client and the client will boot up giving you the login prompt. After you log in you can verify that it is running from memory and not from disk (/dev/sda) by running commands such as “df” and “fdisk” and verifying your /etc/fstab file which indicates a diskless file system.

#The fstab file doesn't mount the physical hard disk partitions:
[root@node003 ~]# cat /etc/fstab
# /etc/fstab for diskless clients, written by system-config-netboot
none /dev/pts devpts gid=5,mode=620 0 0
none /dev/shm tmpfs defaults 0 0
none /tmp tmpfs defaults 0 0
/dev/cdrom /media/cdrom iso9660 noauto,owner,kudzu,ro 0 0
/dev/fd0 /media/floppy auto noauto,owner,kudzu 0 0


#The df command shows that the local disk /dev/sda is not used
[root@node003 ~]# df -hl
Filesystem Size Used Avail Use% Mounted on
rootfs 44G 13G 29G 32% /
/dev 5.9G 212K 5.9G 1% /dev
/dev 5.9G 212K 5.9G 1% /dev
none 5.9G 0 5.9G 0% /dev/shm
none 5.9G 796K 5.9G 1% /tmp


#The fdisk command shows that the physical hard disk is there, but not used
[root@node003 ~]# fdisk -l
Disk /dev/sda: 73.4 GB, 73407820800 bytes
255 heads, 63 sectors/track, 8924 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

Device Boot Start End Blocks Id System
/dev/sda1 * 1 25 200781 83 Linux
/dev/sda2 26 1069 8385930 82 Linux swap / Solaris
/dev/sda3 1070 6871 46604565 83 Linux
/dev/sda4 6872 8924 16490722+ 5 Extended
/dev/sda5 6872 8401 12289693+ 83 Linux
/dev/sda6 8402 8923 4192933+ 83 Linux



5. Modifying the Golden Image Node

If later on you need to install more packages on these nodes, you only need to install it on the golden image node then sync your root directory on the server. Another example if you need to update the root password on all nodes, you just need to change it on the golden image and sync.

For example let’s say you want to install the “dstat” package on all nodes.

Go to the golden image node node004 which is running on normal disk and install it there:
[root@node004 ~]# yum install dstat

Now on your NFS server sync again with the golden image:
[root@node002 ~]# rsync -a -e ssh --exclude='/proc/*' --exclude='/sys/*' node004:/ /diskless/i386/RHEL5-AS/root/

Now the diskless client (node003) should be updated on the fly with no need for a reboot. This node has 12GB of physical RAM and only around 250MB seems to be used by the diskless environment:
[root@node003 ~]# dstat -m
------memory-usage-----
used buff cach free
131M 256k 116M 11G



I hope you find this useful

Mohamad Sindi
sindimo@ieee.org
Writer profile
author image
-아랑 -
2010/03/26 08:47 2010/03/26 08:47

트랙백 주소 : 이 글에는 트랙백을 보낼 수 없습니다

  1. Subject : besoin d argent immediatement

    Tracked from besoin d argent immediatement / 2014/11/22 03:39  삭제

    시스템 엔지니어 공동 관리 구역 :: RHEL5기반 Diskless cluster 구축하기

  2. Subject : Dungeon Hunter 4 Cheats

    Tracked from Dungeon Hunter 4 Cheats / 2014/11/22 05:04  삭제

    시스템 엔지니어 공동 관리 구역 :: RHEL5기반 Diskless cluster 구축하기

  3. Subject : nbnatural.com

    Tracked from nbnatural.com / 2014/11/22 09:01  삭제

    시스템 엔지니어 공동 관리 구역 :: RHEL5기반 Diskless cluster 구축하기

  4. Subject : hermes エルメス ケリー35

    Tracked from hermes エルメス ケリー35 / 2014/11/23 14:43  삭제

    시스템 엔지니어 공동 관리 구역 :: RHEL5기반 Diskless cluster 구축하기

  5. Subject : スケートボードサイト スケボーセット 子供用

    Tracked from スケートボードサイト スケボーセット 子供用 / 2014/11/28 08:10  삭제

    시스템 엔지니어 공동 관리 구역 :: RHEL5기반 Diskless cluster 구축하기

  6. Subject : メンズ ファッション 長袖ハイネックシャツ393鹿の子

    Tracked from メンズ ファッション 長袖ハイネックシャツ393鹿の子 / 2014/11/28 09:41  삭제

    시스템 엔지니어 공동 관리 구역 :: RHEL5기반 Diskless cluster 구축하기

  7. Subject : birkenstock ビルケンシュトック arizona

    Tracked from birkenstock ビルケンシュトック arizona / 2014/11/28 10:29  삭제

    시스템 엔지니어 공동 관리 구역 :: RHEL5기반 Diskless cluster 구축하기

  8. Subject : Music

    Tracked from Music / 2014/11/29 01:21  삭제

    시스템 엔지니어 공동 관리 구역 :: RHEL5기반 Diskless cluster 구축하기

  9. Subject : Coach cheap

    Tracked from Coach cheap / 2014/12/01 02:45  삭제

    시스템 엔지니어 공동 관리 구역 :: RHEL5기반 Diskless cluster 구축하기

  10. Subject : top 10 fragrances for men

    Tracked from top 10 fragrances for men / 2014/12/01 11:48  삭제

    시스템 엔지니어 공동 관리 구역 :: RHEL5기반 Diskless cluster 구축하기

  11. Subject : tumi トゥミ シーン別

    Tracked from tumi トゥミ シーン別 / 2014/12/02 11:33  삭제

    시스템 엔지니어 공동 관리 구역 :: RHEL5기반 Diskless cluster 구축하기

  12. Subject : Camisetas Futbol Baratas

    Tracked from Camisetas Futbol Baratas / 2014/12/06 08:11  삭제

    시스템 엔지니어 공동 관리 구역 :: RHEL5기반 Diskless cluster 구축하기

  13. Subject : ラグビー その他

    Tracked from ラグビー その他 / 2014/12/07 07:31  삭제

    시스템 엔지니어 공동 관리 구역 :: RHEL5기반 Diskless cluster 구축하기

  14. Subject : vaginal tightening

    Tracked from vaginal tightening / 2014/12/07 09:54  삭제

    시스템 엔지니어 공동 관리 구역 :: RHEL5기반 Diskless cluster 구축하기

  15. Subject : tighten vagina

    Tracked from tighten vagina / 2014/12/07 13:12  삭제

    시스템 엔지니어 공동 관리 구역 :: RHEL5기반 Diskless cluster 구축하기

  16. Subject : サッカー トータル90シリーズ

    Tracked from サッカー トータル90シリーズ / 2014/12/08 13:25  삭제

    시스템 엔지니어 공동 관리 구역 :: RHEL5기반 Diskless cluster 구축하기

  17. Subject : cheap coach

    Tracked from cheap coach / 2014/12/13 16:38  삭제

    시스템 엔지니어 공동 관리 구역 :: RHEL5기반 Diskless cluster 구축하기

  18. Subject : Snow plowing

    Tracked from Snow plowing / 2014/12/17 20:50  삭제

    시스템 엔지니어 공동 관리 구역 :: RHEL5기반 Diskless cluster 구축하기

  19. Subject : Clash of Clans Gems Generator

    Tracked from Clash of Clans Gems Generator / 2015/01/07 00:36  삭제

    시스템 엔지니어 공동 관리 구역 :: RHEL5기반 Diskless cluster 구축하기

  20. Subject : Darkness Reborn Hack

    Tracked from Darkness Reborn Hack / 2015/01/08 15:01  삭제

    시스템 엔지니어 공동 관리 구역 :: RHEL5기반 Diskless cluster 구축하기

  21. Subject : clash of clans gems

    Tracked from clash of clans gems / 2015/01/09 08:29  삭제

    시스템 엔지니어 공동 관리 구역 :: RHEL5기반 Diskless cluster 구축하기

  22. Subject : raze-123.com

    Tracked from raze-123.com / 2015/01/14 19:15  삭제

    시스템 엔지니어 공동 관리 구역 :: RHEL5기반 Diskless cluster 구축하기

  23. Subject : raze-123.comWalkthroughs for the automated process: For

    Tracked from raze-123.comWalkthroughs for the automated process: For / 2015/01/20 16:38  삭제

    시스템 엔지니어 공동 관리 구역 :: RHEL5기반 Diskless cluster 구축하기

  24. Subject : Ideal Shape Coupon Code

    Tracked from Ideal Shape Coupon Code / 2015/01/25 00:32  삭제

    시스템 엔지니어 공동 관리 구역 :: RHEL5기반 Diskless cluster 구축하기

  25. Subject : wedding

    Tracked from wedding / 2015/02/14 04:03  삭제

    시스템 엔지니어 공동 관리 구역 :: RHEL5기반 Diskless cluster 구축하기

  26. Subject : marco fumagalli dermatologo

    Tracked from marco fumagalli dermatologo / 2015/02/25 20:27  삭제

    시스템 엔지니어 공동 관리 구역 :: RHEL5기반 Diskless cluster 구축하기

  27. Subject : your guitar academy bristol

    Tracked from your guitar academy bristol / 2015/02/27 00:03  삭제

    시스템 엔지니어 공동 관리 구역 :: RHEL5기반 Diskless cluster 구축하기

  28. Subject : your guitar academy kingston

    Tracked from your guitar academy kingston / 2015/03/02 18:41  삭제

    시스템 엔지니어 공동 관리 구역 :: RHEL5기반 Diskless cluster 구축하기

  29. Subject : Auckland SEO

    Tracked from Auckland SEO / 2015/03/05 11:56  삭제

    시스템 엔지니어 공동 관리 구역 :: RHEL5기반 Diskless cluster 구축하기

  30. Subject : your guitar academy sutton

    Tracked from your guitar academy sutton / 2015/03/10 15:01  삭제

    시스템 엔지니어 공동 관리 구역 :: RHEL5기반 Diskless cluster 구축하기

  31. Subject : دانلود اهنگ جدید

    Tracked from دانلود اهنگ جدید / 2015/03/11 18:17  삭제

    시스템 엔지니어 공동 관리 구역 :: RHEL5기반 Diskless cluster 구축하기

  32. Subject : ngewe

    Tracked from ngewe / 2015/03/17 00:31  삭제

    시스템 엔지니어 공동 관리 구역 :: RHEL5기반 Diskless cluster 구축하기

  33. Subject : {ngentot

    Tracked from {ngentot / 2015/03/19 08:37  삭제

    시스템 엔지니어 공동 관리 구역 :: RHEL5기반 Diskless cluster 구축하기

  34. Subject : best action cam 2015

    Tracked from best action cam 2015 / 2015/03/19 16:55  삭제

    시스템 엔지니어 공동 관리 구역 :: RHEL5기반 Diskless cluster 구축하기

  35. Subject : alpharetta tree service

    Tracked from alpharetta tree service / 2015/03/23 22:49  삭제

    시스템 엔지니어 공동 관리 구역 :: RHEL5기반 Diskless cluster 구축하기

  36. Subject : Does HGH Diet Work

    Tracked from Does HGH Diet Work / 2015/03/24 02:57  삭제

    시스템 엔지니어 공동 관리 구역 :: RHEL5기반 Diskless cluster 구축하기

  37. Subject : lottery payments

    Tracked from lottery payments / 2015/03/24 18:56  삭제

    시스템 엔지니어 공동 관리 구역 :: RHEL5기반 Diskless cluster 구축하기

  38. Subject : Belize Travel

    Tracked from Belize Travel / 2015/04/18 20:16  삭제

    시스템 엔지니어 공동 관리 구역 :: RHEL5기반 Diskless cluster 구축하기

  39. Subject : gold jewelry

    Tracked from gold jewelry / 2015/05/03 20:35  삭제

    시스템 엔지니어 공동 관리 구역 :: RHEL5기반 Diskless cluster 구축하기

  40. Subject : Shipwreck Beach Tour

    Tracked from Shipwreck Beach Tour / 2015/05/25 12:56  삭제

    시스템 엔지니어 공동 관리 구역 :: RHEL5기반 Diskless cluster 구축하기

  41. Subject : kraamkado

    Tracked from kraamkado / 2015/05/30 10:36  삭제

    시스템 엔지니어 공동 관리 구역 :: RHEL5기반 Diskless cluster 구축하기

  42. Subject : Dentist

    Tracked from Dentist / 2015/06/18 05:39  삭제

    시스템 엔지니어 공동 관리 구역 :: RHEL5기반 Diskless cluster 구축하기

  43. Subject : bikini luxe review

    Tracked from bikini luxe review / 2015/06/25 22:20  삭제

    시스템 엔지니어 공동 관리 구역 :: RHEL5기반 Diskless cluster 구축하기

  44. Subject : creatine

    Tracked from creatine / 2015/07/04 12:25  삭제

    시스템 엔지니어 공동 관리 구역 :: RHEL5기반 Diskless cluster 구축하기

  45. Subject : customhouse global limited julienne peeler

    Tracked from customhouse global limited julienne peeler / 2015/08/20 11:16  삭제

    시스템 엔지니어 공동 관리 구역 :: RHEL5기반 Diskless cluster 구축하기

  46. Subject : REVIEWED

    Tracked from REVIEWED / 2015/11/16 22:41  삭제

    시스템 엔지니어 공동 관리 구역 :: RHEL5기반 Diskless cluster 구축하기

  47. Subject : ugly Christmas sweaters

    Tracked from ugly Christmas sweaters / 2015/11/20 19:45  삭제

    시스템 엔지니어 공동 관리 구역 :: RHEL5기반 Diskless cluster 구축하기

  48. Subject : mobitrans phone

    Tracked from mobitrans phone / 2016/02/10 23:00  삭제

    시스템 엔지니어 공동 관리 구역 :: RHEL5기반 Diskless cluster 구축하기

  49. Subject : Florida divorce online

    Tracked from Florida divorce online / 2016/02/18 22:11  삭제

    시스템 엔지니어 공동 관리 구역 :: RHEL5기반 Diskless cluster 구축하기

  50. Subject : favorite anal

    Tracked from favorite anal / 2016/02/19 08:32  삭제

    시스템 엔지니어 공동 관리 구역 :: RHEL5기반 Diskless cluster 구축하기

  51. Subject : craft beer club

    Tracked from craft beer club / 2016/02/21 13:09  삭제

    시스템 엔지니어 공동 관리 구역 :: RHEL5기반 Diskless cluster 구축하기

  52. Subject : low cost holidays

    Tracked from low cost holidays / 2016/02/21 15:28  삭제

    시스템 엔지니어 공동 관리 구역 :: RHEL5기반 Diskless cluster 구축하기

  53. Subject : Lawyer

    Tracked from Lawyer / 2016/02/24 18:33  삭제

    시스템 엔지니어 공동 관리 구역 :: RHEL5기반 Diskless cluster 구축하기

  54. Subject : low cost holidays

    Tracked from low cost holidays / 2016/02/28 13:45  삭제

    시스템 엔지니어 공동 관리 구역 :: RHEL5기반 Diskless cluster 구축하기

  55. Subject : vencendo a calvicie

    Tracked from vencendo a calvicie / 2016/03/23 20:14  삭제

    시스템 엔지니어 공동 관리 구역 :: RHEL5기반 Diskless cluster 구축하기

  56. Subject : computer equipment recycling

    Tracked from computer equipment recycling / 2017/05/04 05:47  삭제

    시스템 엔지니어 공동 관리 구역 :: RHEL5기반 Diskless cluster 구축하기

  57. Subject : computer equipment recycling

    Tracked from computer equipment recycling / 2017/05/04 05:47  삭제

    시스템 엔지니어 공동 관리 구역 :: RHEL5기반 Diskless cluster 구축하기

  58. Subject : a good rescue knife

    Tracked from a good rescue knife / 2017/05/14 10:39  삭제

    시스템 엔지니어 공동 관리 구역 :: RHEL5기반 Diskless cluster 구축하기

  59. Subject : a good rescue knife

    Tracked from a good rescue knife / 2017/05/14 10:40  삭제

    시스템 엔지니어 공동 관리 구역 :: RHEL5기반 Diskless cluster 구축하기

  60. Subject : baseball cap for men

    Tracked from baseball cap for men / 2017/11/25 18:05  삭제

    시스템 엔지니어 공동 관리 구역 :: RHEL5기반 Diskless cluster 구축하기

About

by 서진우
Twitter :@muchunalang

Counter

• Total
: 3933869
• Today
: 232
• Yesterday
: 1438