ambari在kylin-v10适配ambari适配遇到的问题ambari修改思路ambari相关代码位置的修改界面生成截图

165 阅读 0 评论 109 点赞

我是靠谱客的博主粗暴羊，这篇文章主要介绍ambari在kylin-v10适配ambari适配遇到的问题ambari修改思路ambari相关代码位置的修改界面生成截图，现在分享给大家，希望可以做个参考。

ambari在arm架构适配思路

ambari适配遇到的问题
ambari修改思路
ambari相关代码位置的修改
- 涉及相关脚本及对应位置
- params.py
- yum_manager.py
- script.py
- os_check.py
- system.py
界面生成截图

ambari适配遇到的问题

此次系统适配的是国产kylin-v10系统(是系统非Apache-kylin)，内核是4.19(仿centos8)，yum的版本4.2(由python2转变为python3,由yum转变为dnf.yum，这就意味着依赖centos8之前的软件，在centos8的yum的api无法使用)。
通过尝试发现
1.agent的 yum_manager.py 的函数yum_check_package_available无法通过yum的api获取以安装的rpm包。
2.agent 无法获取${stack_version}的对应版本。
3.agent无法获取yum源的base_url地址。
4.kylin系统不被ambari所支持。

ambari修改思路

安装是由ambari-agent请求server端的api返回的 command.json去执行相应的job，因此只需要修改返回的value就可以修改因版本不匹配，yum源无法获取等问题。

ambari相关代码位置的修改

此版本适配的是ambari-2.7.3，hadoop版本是3_1_0_0。
由于初版，仅为实现功能，有些地方是写死的，后期需要进一步优化。代码有不足的地方，仅供参考, 欢迎来提改进意见。
相关rpm包已经打好：ambari-agent

涉及相关脚本及对应位置

ambari-agent	ambari-server
	/var/lib/ambari-server/resources/stack-hooks/before-INSTALL/scripts/params.py
/usr/lib/ambari-agent/lib/ambari_commons/repo_manager/yum_manager.py	/usr/lib/ambari-server/lib/ambari_commons/repo_manager/yum_manager.py
/usr/lib/ambari-agent/lib/resource_management/libraries/script/script.py	/usr/lib/ambari-server/lib/resource_management/libraries/script/script.py
/usr/lib/ambari-agent/lib/ambari_commons/os_check.py	/usr/lib/ambari-server/lib/ambari_commons/os_check.py
/usr/lib/ambari-agent/lib/resource_management/core/providers/system.py	/usr/lib/ambari-server/lib/resource_management/core/providers/system.py

params.py

此类是公共类script获取conf的地方

复制代码

#原版
repo_file = default("/repositoryFile",None)
##############################################
#新版
middle_test = default("/repositoryFile",None)
#定义获取repo的key及value
def explain(repo_file):   
  groups_dict={}
  with open (repo_file,"rb") as f:
    for line in f:
      if line.startswith("#") or line.startswith(";") or len(line.strip()) == 0:
        continue
      if line.strip().find('[') == 0 and line.strip().find(']') == len(line.strip()) - 1:
        key = line.strip().lstrip('[').rstrip(']')
        groups_dict[key] = []
      else:
        if line.strip().find('baseurl') != -1:
          groups_dict[key].append(line.strip().split('=')[1])
  return groups_dict

#方便获取repo文件的base_url
middle_list=[]
middle_dict={}
for i in middle_test['repositories']:
  if i['repoName']=="HDP":
    i['baseUrl']="".join(explain('/etc/yum.repos.d/hdp.repo')['HDP-3.1'])
  elif i['repoName']=="HDP-UTILS":
    i['baseUrl']="".join(explain('/etc/yum.repos.d/hdp.repo')['HDP-UTILS-1.1.0.22'])
  elif i['repoName']=="GPL":
    i['baseUrl']="".join(explain('/etc/yum.repos.d/hdp.repo')['HDP-3.1-GPL'])
  middle_list.append(i)
middle_dict['repositories']=middle_list
middle_dict['middle_repoVersion']='false'
middle_dict['repoVersion']="3.1"
repo_file=middle_dict

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
#原版
repo_file = default("/repositoryFile",None)
##############################################
#新版
middle_test = default("/repositoryFile",None)
#定义获取repo的key及value
def explain(repo_file):   
  groups_dict={}
  with open (repo_file,"rb") as f:
    for line in f:
      if line.startswith("#") or line.startswith(";") or len(line.strip()) == 0:
        continue
      if line.strip().find('[') == 0 and line.strip().find(']') == len(line.strip()) - 1:
        key = line.strip().lstrip('[').rstrip(']')
        groups_dict[key] = []
      else:
        if line.strip().find('baseurl') != -1:
          groups_dict[key].append(line.strip().split('=')[1])
  return groups_dict

#方便获取repo文件的base_url
middle_list=[]
middle_dict={}
for i in middle_test['repositories']:
  if i['repoName']=="HDP":
    i['baseUrl']="".join(explain('/etc/yum.repos.d/hdp.repo')['HDP-3.1'])
  elif i['repoName']=="HDP-UTILS":
    i['baseUrl']="".join(explain('/etc/yum.repos.d/hdp.repo')['HDP-UTILS-1.1.0.22'])
  elif i['repoName']=="GPL":
    i['baseUrl']="".join(explain('/etc/yum.repos.d/hdp.repo')['HDP-3.1-GPL'])
  middle_list.append(i)
middle_dict['repositories']=middle_list
middle_dict['middle_repoVersion']='false'
middle_dict['repoVersion']="3.1"
repo_file=middle_dict

yum_manager.py

cmd包安装，repolist校验，校验安装包是否被安装等

复制代码

#由于api无法使用因此cmd去查询是否存在对应安装包
  def yum_check_package_available(self, name):
    """
    Does the same as rpm_check_package_avaiable, but faster.
    However need root permissions.
    """
    #r=shell.subprocess_executor("/usr/bin/yum list installed|awk '{print $1}'")
    #if name.find("stack_version") != -1:
    #  name=name.replace("${stack_version}","3_1_0_0_78")
    #with suppress_stdout():
    #  package_list = r.out.split()
    #for package in package_list:
    #  if package.find(name) != -1:
    #    return True
    #return False
    #11/03修改匹配规则
    r=shell.subprocess_executor("/usr/bin/yum list installed|awk '{print $1}'")
    if name.find("stack_version") != -1:
      name=name.replace("${stack_version}","3_1_0_0_78")
    with suppress_stdout():
      package_list = r.out.split()
    for package in package_list:
      if package[0:package.rfind('.', 1)] == name:
      #if package.find(name) != -1:
        return True
    return False
    
  #install_package函数 由于name的stack_version无法获取因此写死了 不然yum无法安装
name=name.replace("${stack_version}","3_1_0_0_78")

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
#由于api无法使用因此cmd去查询是否存在对应安装包
  def yum_check_package_available(self, name):
    """
    Does the same as rpm_check_package_avaiable, but faster.
    However need root permissions.
    """
    #r=shell.subprocess_executor("/usr/bin/yum list installed|awk '{print $1}'")
    #if name.find("stack_version") != -1:
    #  name=name.replace("${stack_version}","3_1_0_0_78")
    #with suppress_stdout():
    #  package_list = r.out.split()
    #for package in package_list:
    #  if package.find(name) != -1:
    #    return True
    #return False
    #11/03修改匹配规则
    r=shell.subprocess_executor("/usr/bin/yum list installed|awk '{print $1}'")
    if name.find("stack_version") != -1:
      name=name.replace("${stack_version}","3_1_0_0_78")
    with suppress_stdout():
      package_list = r.out.split()
    for package in package_list:
      if package[0:package.rfind('.', 1)] == name:
      #if package.find(name) != -1:
        return True
    return False
    
  #install_package函数 由于name的stack_version无法获取因此写死了 不然yum无法安装
name=name.replace("${stack_version}","3_1_0_0_78")

script.py

复制代码

1
2
3
#此处版本写死了
STACK_VERSION_PLACEHOLDER = "3_1_0_0_78"

os_check.py

复制代码

1
2
3
4
5
6
7
8
9
#版本的获取是通过platform获取的，我们仿照centos7.8修改 如果你有redhat-release， _is_redhat_linux()返回true
 if PYTHON_VER < 26:
        distribution = platform.dist()
      elif _is_redhat_linux():
        distribution = ('centos','7.8.2003','AltArch')
        #platform.dist()
      else:
        distribution = ('centos','7.8.2003','AltArch')

system.py

此函数是校验文本是否相同，如果不相同就将新版本配置文件覆盖，此函数是公共函数，因此仅当匹配到替换源时才交write返回false.

复制代码

def action_create(self):
    path = self.resource.path

if sudo.path_isdir(path):
      raise Fail("Applying %s failed, directory with name %s exists" % (self.resource, path))

dirname = os.path.dirname(path)
    if not sudo.path_isdir(dirname):
      raise Fail("Applying %s failed, parent directory %s doesn't exist" % (self.resource, dirname))

write = False
    content = self._get_content()
    if not sudo.path_exists(path):
      write = True
      reason = "it doesn't exist"
    elif self.resource.replace:
      if content is not None:
        old_content = sudo.read_file(path, encoding=self.resource.encoding)
        Checkrepo=content
        hdprepo="HDP-3.1-repo"
        hdputil="HDP-UTILS-1.1.0.22-repo"
        if isinstance(content,unicode):
          Checkrepo=Checkrepo.encode()
        if Checkrepo.find(hdprepo) != -1 and Checkrepo.find(hdputil) != -1:
          write = False
          old_content = content
        if content != old_content:
          write = True
          reason = "contents don't match"
          if self.resource.backup:
            self.resource.env.backup_file(path)

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
  def action_create(self):
    path = self.resource.path

    if sudo.path_isdir(path):
      raise Fail("Applying %s failed, directory with name %s exists" % (self.resource, path))

    dirname = os.path.dirname(path)
    if not sudo.path_isdir(dirname):
      raise Fail("Applying %s failed, parent directory %s doesn't exist" % (self.resource, dirname))

    write = False
    content = self._get_content()
    if not sudo.path_exists(path):
      write = True
      reason = "it doesn't exist"
    elif self.resource.replace:
      if content is not None:
        old_content = sudo.read_file(path, encoding=self.resource.encoding)
        Checkrepo=content
        hdprepo="HDP-3.1-repo"
        hdputil="HDP-UTILS-1.1.0.22-repo"
        if isinstance(content,unicode):
          Checkrepo=Checkrepo.encode()
        if Checkrepo.find(hdprepo) != -1 and Checkrepo.find(hdputil) != -1:
          write = False
          old_content = content
        if content != old_content:
          write = True
          reason = "contents don't match"
          if self.resource.backup:
            self.resource.env.backup_file(path)