安装部署手册

时空大数据云平台安装部署手册

安装要求

所有节点安装时不可设置交换分区

对于已经存在交换分区的，使用以下命令关闭。

1
2
sed -i '/swap/s/^/#/' /etc/fstab
swapoff -a

存储空间

单机版

单机版会默认使用本机文件系统，要求根分区可用空间不低于 500 GB

集群版 – 标准版、高级版
  • 根分区可用空间不小于 500 GB

平台运行时需要的本地镜像、容器运行时、底层依赖安装等数据需要存储于本地存储，使用根分区，建议空间不低于 500 GB

  • 平台数据需要专用磁盘，大小根据实际需求而定，且数据盘不可分区格式化，建议每台机器数据盘不低于 1TB

平台会安装分布式文件系统用于集群的存储调度，单机版会默认使用本机文件系统。

示例中 sda 用于系统安装，sdb 为数据盘，未设置任何分区及文件系统。

1
2
3
4
5
6
7
8
[root@node1 ~]# lsblk -f
NAME                  FSTYPE      LABEL           UUID                                   MOUNTPOINT
sda                                                                                      
├─sda1                xfs                         c513eeb3-2e87-4972-81a8-d8377bf4bbde   /boot
└─sda2                LVM2_member                 eaauaZ-XuFo-JWm9-Q3Dc-RwcV-Wgqx-1dfY87 
  └─centos_node1-root xfs                         54a6dfc2-db16-41be-8b4a-bcaf5ffbdd71   /
sdb                                                                                      
[root@node1 ~]#
第三方存储 – 待完善

平台目前只支持 NFS 作为第三方存储。如果使用第三方存储，可以不准备上文提到的单独数据盘，但需要跳过 安装步骤 的 1、安装存储。

示例中安装环境规划

IP 主机名 角色 磁盘
192.168.21.101 node1 应用节点 sda - 系统盘
192.168.21.102 node2 应用节点 sda - 系统盘
192.168.21.103 node3 应用节点 sda - 系统盘
192.168.21.104 node4 存储节点 sda - 系统盘/sdb - 数据盘
192.168.21.105 node5 存储节点 sda - 系统盘/sdb - 数据盘
192.168.21.106 node6 存储节点 sda - 系统盘/sdb - 数据盘

安装步骤

0、分发安装包

将安装包拷贝到应用节点的第一台节点和存储节点的第一台节点中，使用如下命令解压

1
2
tar zxvf iSmart-Standard-V1.0.tar.gz
cd iSmart

1、安装存储

第一台存储节点修改 config 下 topology.json 设置存储节点信息及磁盘信息
  • nodes 中的每一条的 .node.hostnames.manager 和 .node.hostnames.storage 代表一个存储节点，修改为集群的存储节点 IP
  • nodes 中的每一条的 devices 代表存储磁盘，即上文提到的未分区和格式化的裸盘，可以设置多块盘
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
{
  "clusters": [
	{
	  "nodes": [
		{
		  "node": {
			"hostnames": {
			  "manage": [
				"192.168.21.104"
			  ],
			  "storage": [
				"192.168.21.104"
			  ]
			},
			"zone": 1
		  },
		  "devices": [
			"/dev/sdb"
		  ]
		},
		{
		  "node": {
			"hostnames": {
			  "manage": [
				"192.168.21.105"
			  ],
			  "storage": [
				"192.168.21.105"
			  ]
			},
			"zone": 1
		  },
		  "devices": [
			"/dev/sdb"
		  ]
		},
		{
		  "node": {
			"hostnames": {
			  "manage": [
				"192.168.21.106"
			  ],
			  "storage": [
				"192.168.21.106"
			  ]
			},
			"zone": 1
		  },
		  "devices": [
			"/dev/sdb"
		  ]
		}
	  ]
	}
  ]
}
存储节点安装文件系统

在存储节点中选择第一台作为操作机，示例中为 192.168.21.104

1
2
chmod +x storage.sh
nodes="192.168.21.104 192.168.21.105 192.168.21.106" ./storage.sh

安装过程会出现命令交互，按照提示确认安装信息、输入各节点密码。 安装完成后，会输出存储集群 ID、docker registry 信息。

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
[root@node1 iSmart]# nodes="192.168.21.101 192.168.21.102 192.168.21.103" ./storage.sh 
 [WARN]  INSTALL INFO IS BELOW, PLEASE CHECK IT CAREFULLY. 

 [INFO]  node number: 3 
 [INFO]  node list: 192.168.21.101 192.168.21.102 192.168.21.103 
 [INFO]  arch: amd64 
 [INFO]  mode: distributed 
 [INFO]  will install glusterfs 

Is OK? y/N[y] y
 [INFO]  has verified, install process continue... 
 [INFO]  input root password of 192.168.21.101 
/usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/etc/geovis_key.pub"
The authenticity of host '192.168.21.101 (192.168.21.101)' can't be established.
ECDSA key fingerprint is SHA256:cGoA2UIfiAwYccUArGV/kprxJASaTx/J5Krd20Y+HRk.
ECDSA key fingerprint is MD5:6c:03:b9:d4:0b:1f:c9:ea:85:7b:ca:2a:8a:f4:34:16.
Are you sure you want to continue connecting (yes/no)? yes
root@192.168.21.101's password: 
 [PUSH]  192.168.21.101 /etc/hosts 
 [INFO]  input root password of 192.168.21.102 
/usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/etc/geovis_key.pub"
The authenticity of host '192.168.21.102 (192.168.21.102)' can't be established.
ECDSA key fingerprint is SHA256:7iYC/hpp3YNsdsXut2j7l2pFBwg6XHmTJGHSAx20TFM.
ECDSA key fingerprint is MD5:1d:f8:c3:25:7b:e5:e4:bc:a0:dd:56:25:6b:31:4f:9f.
Are you sure you want to continue connecting (yes/no)? yes
root@192.168.21.102's password: 
 [PUSH]  192.168.21.102 /etc/hosts 
 [INFO]  input root password of 192.168.21.103 
/usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/etc/geovis_key.pub"
The authenticity of host '192.168.21.103 (192.168.21.103)' can't be established.
ECDSA key fingerprint is SHA256:AANhF+IwnJamQpZEkb4KVi6vS5uZ/17mFU004mMlO8E.
ECDSA key fingerprint is MD5:68:6d:53:6c:42:d6:cf:41:21:97:0d:63:40:12:df:b4.
Are you sure you want to continue connecting (yes/no)? yes
root@192.168.21.103's password: 
 [PUSH]  192.168.21.103 /etc/hosts 
 [INFO]  192.168.21.101 create iso mirror repo 
 [PUSH]  192.168.21.101 offline mirror 
 [INFO]  192.168.21.101 mount mirror 
 [PUSH]  192.168.21.101 offline repo 
 [INFO]  192.168.21.101 makecache 
 [INFO]  192.168.21.102 create iso mirror repo 
 [PUSH]  192.168.21.102 offline mirror 
 [INFO]  192.168.21.102 mount mirror 
 [PUSH]  192.168.21.102 offline repo 
 [INFO]  192.168.21.102 makecache 
 [INFO]  192.168.21.103 create iso mirror repo 
 [PUSH]  192.168.21.103 offline mirror 
 [INFO]  192.168.21.103 mount mirror 
 [PUSH]  192.168.21.103 offline repo 
 [INFO]  192.168.21.103 makecache 
 [INFO]  192.168.21.101 disable firewall and selinux 
 [INFO]  192.168.21.101 set systemd timeout 
 [INFO]  192.168.21.101 install ntpdate 
 [INFO]  192.168.21.101 sync time from internet 
 [INFO]  192.168.21.102 disable firewall and selinux 
 [INFO]  192.168.21.102 set systemd timeout 
 [INFO]  192.168.21.102 install ntpdate 
 [INFO]  192.168.21.102 sync time from internet 
 [INFO]  192.168.21.103 disable firewall and selinux 
 [INFO]  192.168.21.103 set systemd timeout 
 [INFO]  192.168.21.103 install ntpdate 
 [INFO]  192.168.21.103 sync time from internet 
 [INFO]  192.168.21.101 install base packages 
 [INFO]  192.168.21.102 install base packages 
 [INFO]  192.168.21.103 install base packages 
 [INFO]  192.168.21.101 install gluster 
 [INFO]  192.168.21.102 install gluster 
 [INFO]  192.168.21.103 install gluster 
 [INFO]  create gluster cluster 
peer probe: success
peer probe: success
Number of Peers: 2

Hostname: node2
Uuid: 0023ff91-29d6-4371-b77e-9deed9265e39
State: Peer in Cluster (Connected)

Hostname: node3
Uuid: 66b9fc9f-5a6a-4cfc-9e3b-efa7de4f4de9
State: Peer in Cluster (Connected)
 [INFO]  setup heketi 
 [PUSH]  heketi config 
 [PUSH]  heketi private key 
 [INFO]  create heketi user and group 
 [PUSH]  heketi binary 
 [INFO]  start heketi 
 [INFO]  create heketi cluster 
Creating cluster ... ID: 576494a56276e02035002a44b231778a
		Allowing file volumes on cluster.
		Allowing block volumes on cluster.
		Creating node 192.168.21.101 ... ID: f8a11b9028ae899d60a77c399442f181
				Adding device /dev/sdb ... OK
		Creating node 192.168.21.102 ... ID: cb2141f8bb4825fae52de8ae012b83a8
				Adding device /dev/sdb ... OK
		Creating node 192.168.21.103 ... ID: 7d3758c142316367bfb6e7acd89b4597
				Adding device /dev/sdb ... OK
 [INFO]  heketi install success 
 [INFO]  install docker registry 
 [INFO]  192.168.21.101 config docker 
 [PUSH]  certs and docker daemon config 
 [INFO]  docker starting 
 [INFO]  192.168.21.102 config docker 
 [PUSH]  certs and docker daemon config 
 [INFO]  docker starting 
 [INFO]  192.168.21.103 config docker 
 [PUSH]  certs and docker daemon config 
 [INFO]  docker starting 
 [INFO]  create registry dir 
 [INFO]  loading registry 
 [PUSH]  certs 
 [INFO]  run registry 

 [INFO]  ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ 
 [INFO]  STORAGE CLUSTER INSTALLED SUCCESSFULLY. CLUSTER INFO IS HERE 
 [INFO]  Cluster id: 576494a56276e02035002a44b231778a
Nodes:
7d3758c142316367bfb6e7acd89b4597
cb2141f8bb4825fae52de8ae012b83a8
f8a11b9028ae899d60a77c399442f181
Volumes:

Block: true

File: true 
 [INFO]  
 [INFO]  docker registry: hub.geovis.com.cn -> 192.168.21.101 
 [INFO]  registry dir: 192.168.21.101:/data/registry 
 [INFO]  
 [INFO]  ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ 

[root@node1 iSmart]#

2、部署集群

第一台应用节点修改 config 下 cluster.yaml 设置集群节点信息
  • etcd 表示在哪些节点安装 etcd 用于存储集群云平台数据信息
  • control-plane 表示设置哪些节点为主节点，三节点及以上集群中建议设置至少 3 台为主节点
  • worker 表示设置哪些节点为工作节点，三节点集群中建议全部设置为工作节点，六节点及以上集群中建议将除主节点意外的所有节点设置为工作节点
1
2
3
4
5
6
7
8
9
10
11
12
13
14
...
spec:
  hosts:
  - {name: node1, address: 192.168.21.101, internalAddress: 192.168.21.101, user: root, password: "123456"}
  - {name: node2, address: 192.168.21.102, internalAddress: 192.168.21.102, user: root, password: "123456"}
  - {name: node3, address: 192.168.21.103, internalAddress: 192.168.21.103, user: root, password: "123456"}
  roleGroups:
	etcd:
	- node[1:3]
	control-plane:
	- node[1:3]
	worker:
	- node[1:3]
...

示例中将 endpointIps 修改为上文设置的 etcd 部署的所有节点 IP

1
2
3
4
5
6
7
...
  etcd:
	monitoring: true      
	endpointIps: '192.168.21.101,192.168.21.102,192.168.21.103'
	port: 2379             
	tlsEnable: true
...
第一台应用节点修改 config 下 cluster.yaml 集群配置文件的 addons 配置存储插件

存储插件设置分为几种不同的情况，按照实际情况选择

单机版使用本机文件系统

修改 addons 为空

1
addons: []
单机版/集群版使用 NFS

修改 addons 如下

其中 path 修改为正确的路径，nfs.server 修改为 NFS 服务器的 IP 地址，nfs.path 修改为挂载的 NFS 服务器路径。

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
addons:
  - name: ismart
	namespace: kube-system
	sources:
	  chart:
		name: nfs-client-provisioner
		path: /root/iSmart/packages/charts/nfs
		values:
		  - nfs.server=192.168.21.101
		  - nfs.path=/mnt/nfs
		  - image.repository=hub.geovis.com.cn/library/nfs-subdir-external-provisioner
		  - image.tag=v4.0.2
		  - storageClass.name=ismart
		  - storageClass.defaultClass=true
		  - storageClass.allowVolumeExpansion=true
集群版使用平台提供的分布式存储

示例中设置 glusterfs 插件的路径修改为 glusterfs.yaml 正确路径

1
2
3
4
5
6
7
8
9
...
addons:
  - name: ismart
	namespace: kube-system
	sources:
	  yaml:
		path:
		  - /root/iSmart/config/glusterfs.yaml
...

第一台应用节点修改 config 下 glusterfs.yaml 设置 REST 地址

将 resturl 修改为存储节点的第一个节点 IP，schema 及 端口保持不变

1
2
3
...
  resturl: "http://192.168.21.104:48080"
...

第一台应用节点修改 config 下 glusterfs.yaml 设置 clusterid

将 clusterid 修改为存储节点安装完成时提示的 clusterid

1
2
3
...
  clusterid: "cc2ec84c659064cbc276555bfbed24df"
...

第一台应用节点安装智慧中台

NS 用于指定将智慧中台部署于云平台的命名空间，REGISTRY 用于指定安装智慧中台使用的镜像仓库，为存储节点的第一个节点 IP

1
2
chmod +x install.sh
NS=aiplatform REGISTRY=192.168.21.104 ./install.sh

安装过程会出现命令交互，按照提示确认安装信息、输入各节点密码。 

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
[root@node1 iSmart]# NS=ismart REGISTRY=192.168.21.101 ./install.sh 

 [WARN]  INSTALL INFO IS BELOW, PLEASE CHECK IT CAREFULLY. 

 [INFO]  registry: 192.168.21.101 
 [INFO]  node number: 3 
 [INFO]  node list: 192.168.21.101 192.168.21.102 192.168.21.103 
 [INFO]  arch: amd64 
 [INFO]  mode: distributed 

 [INFO]  offline environment install process 
Is OK? y/N[y] y
 [INFO]  has verified, install process continue... 
 [INFO]  input root password of 192.168.21.101 
/usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/etc/ks_key.pub"
root@192.168.21.101's password: 
 [PUSH]  192.168.21.101 /etc/hosts 
 [INFO]  input root password of 192.168.21.102 
/usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/etc/ks_key.pub"
root@192.168.21.102's password: 
 [PUSH]  192.168.21.102 /etc/hosts 
 [INFO]  input root password of 192.168.21.103 
/usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/etc/ks_key.pub"
root@192.168.21.103's password: 
 [PUSH]  192.168.21.103 /etc/hosts 
 [INFO]  192.168.21.101 create iso mirror repo 
 [PUSH]  192.168.21.101 offline mirror 
 [INFO]  192.168.21.101 mount mirror 
 ...
 ...
[root@node1 iSmart]#

安装完成后，会输出集群信息。

3、检查安装结果

4、访问平台