Favor more utilization of bottom screenshot row

When we have the grid size, cut off any extra columns
so long as it doesn't gain a row.

README.md

@@ -0,0 +1,30 @@
+# Confluent
+
+![Python 3](https://img.shields.io/badge/python-3-blue.svg) [![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](https://github.com/xcat2/confluent/blob/master/LICENSE)
+
+Confluent is a software package to handle essential bootstrap and operation of scale-out server configurations.
+It supports stateful and stateless deployments for various operating systems.
+
+Check [this page](https://hpc.lenovo.com/users/documentation/whatisconfluent.html
+) for a more detailed list of features.
+
+Confluent is the modern successor of [xCAT](https://github.com/xcat2/xcat-core).
+If you're coming from xCAT, check out [this comparison](https://hpc.lenovo.com/users/documentation/confluentvxcat.html).
+
+# Documentation
+
+Confluent documentation is hosted on hpc.lenovo.com: https://hpc.lenovo.com/users/documentation/
+
+# Download
+
+Get the latest version from: https://hpc.lenovo.com/users/downloads/
+
+Check release notes on: https://hpc.lenovo.com/users/news/
+
+# Open Source License
+
+Confluent is made available under the Apache 2.0 license: https://opensource.org/license/apache-2-0
+
+# Developers
+
+Want to help? Submit a [Pull Request](https://github.com/xcat2/confluent/pulls).

confluent_client/COPYRIGHT

@@ -1,6 +1,8 @@
-Name: confluent-client
 Project: https://hpc.lenovo.com/users/
 Source: https://github.com/lenovo/confluent
+Upstream-Name: confluent-client
+
+All file of the Confluent-client software is distributed under the terms of an Apache-2.0 license as indicated below: 
 
 Files: *
 Copyright: 2014-2019 Lenovo
@@ -11,7 +13,8 @@ Copyright: 2014 IBM Corporation
            2015-2019 Lenovo
 License: Apache-2.0
 
-File: sortutil.py
+File: sortutil.py, 
+      tlvdata.py
 Copyright: 2014 IBM Corporation
            2015-2016 Lenovo
 License: Apache-2.0
@@ -19,3 +22,56 @@ License: Apache-2.0
 File: tlv.py
 Copyright: 2014 IBM Corporation
 License: Apache-2.0
+
+License: Apache-2.0
+Apache License
+Version 2.0, January 2004
+http://www.apache.org/licenses/
+
+TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
+
+1. Definitions.
+
+"License" shall mean the terms and conditions for use, reproduction, and distribution as defined by Sections 1 through 9 of this document.
+
+"Licensor" shall mean the copyright owner or entity authorized by the copyright owner that is granting the License.
+
+"Legal Entity" shall mean the union of the acting entity and all other entities that control, are controlled by, or are under common control with that entity. For the purposes of this definition, "control" means (i) the power, direct or indirect, to cause the direction or management of such entity, whether by contract or otherwise, or (ii) ownership of fifty percent (50%) or more of the outstanding shares, or (iii) beneficial ownership of such entity.
+
+"You" (or "Your") shall mean an individual or Legal Entity exercising permissions granted by this License.
+
+"Source" form shall mean the preferred form for making modifications, including but not limited to software source code, documentation source, and configuration files.
+
+"Object" form shall mean any form resulting from mechanical transformation or translation of a Source form, including but not limited to compiled object code, generated documentation, and conversions to other media types.
+
+"Work" shall mean the work of authorship, whether in Source or Object form, made available under the License, as indicated by a copyright notice that is included in or attached to the work (an example is provided in the Appendix below).
+
+"Derivative Works" shall mean any work, whether in Source or Object form, that is based on (or derived from) the Work and for which the editorial revisions, annotations, elaborations, or other modifications represent, as a whole, an original work of authorship. For the purposes of this License, Derivative Works shall not include works that remain separable from, or merely link (or bind by name) to the interfaces of, the Work and Derivative Works thereof.
+
+"Contribution" shall mean any work of authorship, including the original version of the Work and any modifications or additions to that Work or Derivative Works thereof, that is intentionally submitted to Licensor for inclusion in the Work by the copyright owner or by an individual or Legal Entity authorized to submit on behalf of the copyright owner. For the purposes of this definition, "submitted" means any form of electronic, verbal, or written communication sent to the Licensor or its representatives, including but not limited to communication on electronic mailing lists, source code control systems, and issue tracking systems that are managed by, or on behalf of, the Licensor for the purpose of discussing and improving the Work, but excluding communication that is conspicuously marked or otherwise designated in writing by the copyright owner as "Not a Contribution."
+
+"Contributor" shall mean Licensor and any individual or Legal Entity on behalf of whom a Contribution has been received by Licensor and subsequently incorporated within the Work.
+
+2. Grant of Copyright License. Subject to the terms and conditions of this License, each Contributor hereby grants to You a perpetual, worldwide, non-exclusive, no-charge, royalty-free, irrevocable copyright license to reproduce, prepare Derivative Works of, publicly display, publicly perform, sublicense, and distribute the Work and such Derivative Works in Source or Object form.
+
+3. Grant of Patent License. Subject to the terms and conditions of this License, each Contributor hereby grants to You a perpetual, worldwide, non-exclusive, no-charge, royalty-free, irrevocable (except as stated in this section) patent license to make, have made, use, offer to sell, sell, import, and otherwise transfer the Work, where such license applies only to those patent claims licensable by such Contributor that are necessarily infringed by their Contribution(s) alone or by combination of their Contribution(s) with the Work to which such Contribution(s) was submitted. If You institute patent litigation against any entity (including a cross-claim or counterclaim in a lawsuit) alleging that the Work or a Contribution incorporated within the Work constitutes direct or contributory patent infringement, then any patent licenses granted to You under this License for that Work shall terminate as of the date such litigation is filed.
+
+4. Redistribution. You may reproduce and distribute copies of the Work or Derivative Works thereof in any medium, with or without modifications, and in Source or Object form, provided that You meet the following conditions:
+
+You must give any other recipients of the Work or Derivative Works a copy of this License; and
+You must cause any modified files to carry prominent notices stating that You changed the files; and
+You must retain, in the Source form of any Derivative Works that You distribute, all copyright, patent, trademark, and attribution notices from the Source form of the Work, excluding those notices that do not pertain to any part of the Derivative Works; and
+If the Work includes a "NOTICE" text file as part of its distribution, then any Derivative Works that You distribute must include a readable copy of the attribution notices contained within such NOTICE file, excluding those notices that do not pertain to any part of the Derivative Works, in at least one of the following places: within a NOTICE text file distributed as part of the Derivative Works; within the Source form or documentation, if provided along with the Derivative Works; or, within a display generated by the Derivative Works, if and wherever such third-party notices normally appear. The contents of the NOTICE file are for informational purposes only and do not modify the License. You may add Your own attribution notices within Derivative Works that You distribute, alongside or as an addendum to the NOTICE text from the Work, provided that such additional attribution notices cannot be construed as modifying the License.
+You may add Your own copyright statement to Your modifications and may provide additional or different license terms and conditions for use, reproduction, or distribution of Your modifications, or for any such Derivative Works as a whole, provided Your use, reproduction, and distribution of the Work otherwise complies with the conditions stated in this License.
+
+5. Submission of Contributions. Unless You explicitly state otherwise, any Contribution intentionally submitted for inclusion in the Work by You to the Licensor shall be under the terms and conditions of this License, without any additional terms or conditions. Notwithstanding the above, nothing herein shall supersede or modify the terms of any separate license agreement you may have executed with Licensor regarding such Contributions.
+
+6. Trademarks. This License does not grant permission to use the trade names, trademarks, service marks, or product names of the Licensor, except as required for reasonable and customary use in describing the origin of the Work and reproducing the content of the NOTICE file.
+
+7. Disclaimer of Warranty. Unless required by applicable law or agreed to in writing, Licensor provides the Work (and each Contributor provides its Contributions) on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied, including, without limitation, any warranties or conditions of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A PARTICULAR PURPOSE. You are solely responsible for determining the appropriateness of using or redistributing the Work and assume any risks associated with Your exercise of permissions under this License.
+
+8. Limitation of Liability. In no event and under no legal theory, whether in tort (including negligence), contract, or otherwise, unless required by applicable law (such as deliberate and grossly negligent acts) or agreed to in writing, shall any Contributor be liable to You for damages, including any direct, indirect, special, incidental, or consequential damages of any character arising as a result of this License or out of the use or inability to use the Work (including but not limited to damages for loss of goodwill, work stoppage, computer failure or malfunction, or any and all other commercial damages or losses), even if such Contributor has been advised of the possibility of such damages.
+
+9. Accepting Warranty or Additional Liability. While redistributing the Work or Derivative Works thereof, You may choose to offer, and charge a fee for, acceptance of support, warranty, indemnity, or other liability obligations and/or rights consistent with this License. However, in accepting such obligations, You may act only on Your own behalf and on Your sole responsibility, not on behalf of any other Contributor, and only if You agree to indemnify, defend, and hold each Contributor harmless for any liability incurred by, or claims asserted against, such Contributor by reason of your accepting any such warranty or additional liability.
+
+END OF TERMS AND CONDITIONS

confluent_client/addattribs.py

@@ -14,11 +14,16 @@ import shutil
 
 shutil.copyfile('doc/man/nodeattrib.ronn.tmpl', 'doc/man/nodeattrib.ronn')
 shutil.copyfile('doc/man/nodegroupattrib.ronn.tmpl', 'doc/man/nodegroupattrib.ronn')
-with open('doc/man/nodeattrib.ronn', 'a') as outf:
-    for field in sorted(attr.node):
-        outf.write('\n* `{0}`:\n  {1}\n'.format(field, attr.node[field]['description']))
-with open('doc/man/nodegroupattrib.ronn', 'a') as outf:
-    for field in sorted(attr.node):
-        outf.write('\n* `{0}`:\n  {1}\n'.format(field, attr.node[field]['description']))
 
+def append_attributes(filename):
+    with open(filename, 'a') as outf:
+        for field in sorted(attr.node):
+            outf.write('\n* `{0}`:\n  {1}\n'.format(field, attr.node[field]['description']))
+            # Optionally write valid values if they exist
+            for key, values in attr.node[field].items():
+                if key.startswith('valid'):
+                    values_formatted = ', '.join("'{0}'".format(v) for v in values)
+                    outf.write(f'\n    Valid values: {values_formatted}\n')
 
+append_attributes('doc/man/nodeattrib.ronn')
+append_attributes('doc/man/nodegroupattrib.ronn')

confluent_client/bin/collate

@@ -21,6 +21,7 @@
 
 import optparse
 import os
+import re
 import select
 import sys
 
@@ -84,6 +85,7 @@ fullline = sys.stdin.readline()
 printpending = True
 clearpending = False
 holdoff = 0
+padded = None
 while fullline:
     for line in fullline.split('\n'):
         if not line:
@@ -92,13 +94,18 @@ while fullline:
             line = 'UNKNOWN: ' + line
         if options.log:
             node, output = line.split(':', 1)
-            output = output.lstrip()
+            if padded is None:
+                if output.startswith(' '):
+                    padded = True
+                else:
+                    padded = False
+            if padded:
+                output = re.sub(r'^ ', '', output)
             currlog = options.log.format(node=node, nodename=node)
             with open(currlog, mode='a')  as log:
                 log.write(output + '\n')
             continue
         node, output = line.split(':', 1)
-        output = output.lstrip()
         grouped.add_line(node, output)
         if options.watch:
             if not holdoff:

confluent_client/bin/confetty

@@ -1,4 +1,4 @@
-#!/usr/bin/python2
+#!/usr/bin/python3
 # vim: tabstop=4 shiftwidth=4 softtabstop=4
 
 # Copyright 2014 IBM Corporation
@@ -654,13 +654,17 @@ def quitconfetty(code=0, fullexit=False, fixterm=True):
 def get_session_node(shellargs):
     # straight to node console
     if len(shellargs) == 1 and ' ' not in shellargs[0]:
-        return shellargs[0]
+        targ = "/nodes/%s/console/session" % shellargs[0]
+        return targ, shellargs[0]
     if len(shellargs) == 2 and shellargs[0] == 'start':
         args = [s for s in shellargs[1].split('/') if s]
         if len(args) == 4 and args[0] == 'nodes' and args[2] == 'console' and \
                 args[3] == 'session':
-            return args[1]
-    return None
+            return shellargs[1], args[1]
+        if len(args) == 5 and args[0] == 'nodes' and args[2] == 'shell' and \
+                args[3] == 'sessions':
+            return shellargs[1], args[1]
+    return None, None
 
 
 def run_inline_command(path, arg, completion, **kwargs):
@@ -917,10 +921,10 @@ def main():
     doexit = False
     inconsole = False
     pendingcommand = ""
-    session_node = get_session_node(shellargs)
+    targ, session_node = get_session_node(shellargs)
     if session_node is not None:
         consoleonly = True
-        do_command("start /nodes/%s/console/session" % session_node, netserver)
+        do_command("start %s" % targ, netserver)
         doexit = True
     elif shellargs:
         do_command(shellargs, netserver)
@@ -948,8 +952,9 @@ def main():
                     except IOError:
                         pass
             if powerstate is None or powertime < time.time() - 10:  # Check powerstate every 10 seconds
+                if powerstate == None:
+                    powerstate = True
                 powertime = time.time()
-                powerstate = True
                 check_power_state()
         else:
             currcommand = prompt()

confluent_client/bin/confluent2hosts

@@ -118,6 +118,7 @@ def main():
     ap.add_argument('-a', '--attrib', help='Pull ip addresses and hostnames from attribute database', action='store_true')
     ap.add_argument('-i', '--ip', help='Expression to generate addresses (e.g. 172.16.1.{n1} or fd2b:246f:8a50::{n1:x})')
     ap.add_argument('-n', '--name', help='Expression for name to add ({node}-compute, etc). If unspecified, "{node} {node}.{dns.domain}" will be used', action='append')
+    ap.add_argument('-f', '--fqdn-first', help='Put the FQDN first in the hosts entries', action='store_true')
     args = ap.parse_args()
     c = client.Command()
     if args.name:
@@ -157,7 +158,7 @@ def main():
                         elif attrib.endswith('.ipv6_address') and val:
                             ip6bynode[node][currnet] = val.split('/', 1)[0]
                         elif attrib.endswith('.hostname'):
-                            namesbynode[node][currnet] = re.split('\s+|,', val)
+                            namesbynode[node][currnet] = re.split(r'\s+|,', val)
         for node in ip4bynode:
             mydomain = domainsbynode.get(node, None)
             for ipdb in (ip4bynode, ip6bynode):
@@ -173,7 +174,13 @@ def main():
                                     break
                             else:
                                 for name in list(names):
-                                    names.append('{0}.{1}'.format(name, mydomain))
+                                    fqdn = '{0}.{1}'.format(name, mydomain)
+                                    if args.fqdn_first:
+                                        # Insert FQDN at the beginning if --fqdn-first flag is set
+                                        names.insert(0, fqdn)
+                                    else:
+                                        # Otherwise, append FQDN at the end (original behavior)
+                                        names.append(fqdn)
                     names = ' '.join(names)
                     merger.add_entry(ipdb[node][currnet], names)
         merger.write_out('/etc/whatnowhosts')

confluent_client/bin/dir2img

@@ -8,30 +8,35 @@ import os
 import subprocess
 import sys
 
-def create_image(directory, image, label=None):
-    ents = 0
-    datasz = 512
-    for dir in os.walk(sys.argv[1]):
-        ents += 1
-        for filen in dir[2]:
+def create_image(directory, image, label=None, esize=0, totalsize=None):
+
+    if totalsize:
+        datasz = totalsize * 1048576
+    else:
+        ents = 0
+        datasz = 512 + (esize * 1048576)
+        for dir in os.walk(sys.argv[1]):
             ents += 1
-            filename = os.path.join(dir[0], filen)
-            currsz = os.path.getsize(filename)
-            # assuming up to 65k cluster
-            currsz = (currsz // 512 +1) * 512
-            datasz += currsz
-    datasz += ents * 32768
-    datasz = datasz // 16384 + 1
+            for filen in dir[2]:
+                ents += 1
+                filename = os.path.join(dir[0], filen)
+                currsz = os.path.getsize(filename)
+                # assuming up to 65k cluster
+                currsz = (currsz // 512 + 1) * 512
+                datasz += currsz
+        datasz += ents * 32768
+    datasz = datasz // 65536 + 1
     with open(image, 'wb') as imgfile:
-        imgfile.seek(datasz * 16384 - 1)
+        imgfile.seek(datasz * 65536 - 1)
         imgfile.write(b'\x00')
     if label:
+        # 4 heads, 32 sectors, means 65k per track
         subprocess.check_call(['mformat', '-i', image, '-v', label,
                                '-r', '16', '-d', '1', '-t', str(datasz),
-                               '-s', '16','-h', '2', '::'])
+                               '-s', '32','-h', '4', '::'])
     else:
         subprocess.check_call(['mformat', '-i', image, '-r', '16', '-d', '1', '-t',
-                                str(datasz), '-s', '16','-h', '2', '::'])
+                                str(datasz), '-s', '32','-h', '4', '::'])
     # Some clustered filesystems will have the lock from mformat
     # linger after close (mformat doesn't unlock)
     # do a blocking wait for shared lock and then explicitly
@@ -56,6 +61,21 @@ if __name__ == '__main__':
             sys.argv[0]))
         sys.exit(1)
     label = None
-    if len(sys.argv) > 3:
-        label = sys.argv[3]
-    create_image(sys.argv[1], sys.argv[2], label)
+    args = sys.argv
+    esize = 0
+    try:
+        earg = args.index('-e')
+        esize = int(args[earg + 1])
+        args = args[:earg] + args[earg +2:]
+    except ValueError:
+        pass
+    totsize = None
+    try:
+        earg = args.index('-s')
+        totsize = int(args[earg + 1])
+        args = args[:earg] + args[earg +2:]
+    except ValueError:
+        pass
+    if len(args) > 3:
+        label = args[3]
+    create_image(args[1], args[2], label, esize, totsize)

confluent_client/bin/nodeapply

@@ -102,9 +102,9 @@ def run():
         cmdv = ['ssh', sshnode] + cmdvbase + cmdstorun[0]
         if currprocs < concurrentprocs:
             currprocs += 1
-            run_cmdv(node, cmdv, all, pipedesc)
+            run_cmdv(sshnode, cmdv, all, pipedesc)
         else:
-            pendingexecs.append((node, cmdv))        
+            pendingexecs.append((sshnode, cmdv))
     if not all or exitcode:
         sys.exit(exitcode)
     rdy, _, _ = select.select(all, [], [], 10)

confluent_client/bin/nodeattrib

@@ -126,13 +126,14 @@ elif options.set:
         argset = argset.strip()
         if argset:
             arglist += shlex.split(argset)
-        argset = argfile.readline() 
+        argset = argfile.readline()
     session.stop_if_noderange_over(noderange, options.maxnodes)
     exitcode=client.updateattrib(session,arglist,nodetype, noderange, options, None)
 if exitcode != 0:
     sys.exit(exitcode)
 
 # Lists all attributes
+
 if len(args) > 0:
     # setting output to all so it can search since if we do have something to search, we want to show all outputs even if it is blank.
     if requestargs is None:

confluent_client/bin/nodebmcpassword

@@ -0,0 +1,121 @@
+#!/usr/libexec/platform-python
+# vim: tabstop=4 shiftwidth=4 softtabstop=4
+
+# Copyright 2015-2017 Lenovo
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+__author__ = 'tkucherera'
+
+from getpass import getpass
+import optparse
+import os
+import signal
+import sys
+try:
+    signal.signal(signal.SIGPIPE, signal.SIG_DFL)
+except AttributeError:
+    pass
+
+path = os.path.dirname(os.path.realpath(__file__))
+path = os.path.realpath(os.path.join(path, '..', 'lib', 'python'))
+if path.startswith('/opt'):
+    sys.path.append(path)
+
+import confluent.client as client
+
+argparser = optparse.OptionParser(usage="Usage: %prog <noderange> <username> <new_password>")
+argparser.add_option('-m', '--maxnodes', type='int',
+                     help='Number of nodes to affect before prompting for confirmation')
+argparser.add_option('-p', '--prompt', action='store_true',
+                     help='Prompt for password values interactively')
+argparser.add_option('-e', '--environment', action='store_true',
+                     help='Set passwod, but from environment variable of '
+                          'same name')
+
+
+
+(options, args) = argparser.parse_args()
+
+
+try:
+    noderange = args[0]
+    username = args[1]
+except IndexError:
+    argparser.print_help()
+    sys.exit(1)
+    
+client.check_globbing(noderange)
+session = client.Command()
+exitcode = 0
+
+if options.prompt:
+    oneval = 1
+    twoval = 2
+    while oneval != twoval:
+        oneval = getpass('Enter pass for {0}: '.format(username))
+        twoval = getpass('Confirm pass for {0}: '.format(username))
+        if oneval != twoval:
+            print('Values did not match.')
+    new_password = twoval
+
+elif len(args) == 3:
+    if options.environment:
+        key = args[2]
+        new_password = os.environ.get(key, os.environ[key.upper()])
+    else:
+        new_password = args[2]          
+else:
+    argparser.print_help()
+    sys.exit(1)
+    
+errorNodes = set([])
+uid_dict = {}
+session.stop_if_noderange_over(noderange, options.maxnodes)
+
+for rsp in session.read('/noderange/{0}/configuration/management_controller/users/all'.format(noderange)):
+    databynode = rsp["databynode"]
+    for node in databynode:
+        if 'error' in rsp['databynode'][node]:
+            print(node, ':', rsp['databynode'][node]['error'])
+            errorNodes.add(node)
+            continue
+        for user in  rsp['databynode'][node]['users']:
+            if user['username'] == username:
+                if not user['uid'] in uid_dict:
+                    uid_dict[user['uid']] = node
+                    continue
+                uid_dict[user['uid']] = uid_dict[user['uid']] + ',{}'.format(node)
+                break
+
+if not uid_dict:
+    print("Error: Could not reach target node's bmc user")
+    sys.exit(1)
+
+for uid in uid_dict:     
+    success = session.simple_noderange_command(uid_dict[uid], 'configuration/management_controller/users/{0}'.format(uid), new_password, key='password', errnodes=errorNodes)   # = 0 if successful
+
+allNodes = set([])
+
+for node in session.read('/noderange/{0}/nodes/'.format(noderange)):
+    if 'error' in node and success != 0:
+        sys.exit(success)
+    allNodes.add(node['item']['href'].replace("/", ""))
+
+goodNodes = allNodes - errorNodes
+
+for node in goodNodes:
+    print(node + ": Password Change Successful")
+
+
+sys.exit(success)

confluent_client/bin/nodeconfig

@@ -303,9 +303,14 @@ else:
                         '/noderange/{0}/configuration/management_controller/extended/all'.format(noderange),
                         session, printbmc, options, attrprefix='bmc.')
         if options.extra:
-            rcode |= client.print_attrib_path(
-                    '/noderange/{0}/configuration/management_controller/extended/extra'.format(noderange),
-                    session, printextbmc, options)
+            if options.advanced:
+                   rcode |= client.print_attrib_path(
+                        '/noderange/{0}/configuration/management_controller/extended/extra_advanced'.format(noderange),
+                        session, printextbmc, options)
+            else:
+                rcode |= client.print_attrib_path(
+                        '/noderange/{0}/configuration/management_controller/extended/extra'.format(noderange),
+                        session, printextbmc, options)
     if printsys or options.exclude:
         if printsys == 'all':
             printsys = []

confluent_client/bin/nodeconsole

@@ -15,6 +15,7 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
+import base64
 import optparse
 import os
 import subprocess
@@ -28,8 +29,31 @@ import confluent.client as client
 import confluent.sortutil as sortutil
 import confluent.logreader as logreader
 import time
+import select
 import socket
 import re
+import tty
+import termios
+import fcntl
+import confluent.screensqueeze as sq
+try:
+    from PIL import Image
+except ImportError:
+    Image = None
+
+try:
+    # sixel is optional, attempt to import but stub out if unavailable
+    import io
+    import sixel
+
+    class DumbWriter(sixel.SixelWriter):
+        def restore_position(self, output):
+            return
+except ImportError:
+    class DumbWriter():
+        def draw(self, imgfile):
+            sys.stderr.write("PySixel not detected, Sixel format display not supported\n")
+
 
 confettypath = os.path.join(os.path.dirname(sys.argv[0]), 'confetty')
 argparser = optparse.OptionParser(
@@ -46,6 +70,11 @@ argparser.add_option('-l', '--log', action='store_true', default=False,
 argparser.add_option('-T', '--Timestamp', action='store_true', default=False,
                      help= 'Dump log in stdout with timestamps')
 
+argparser.add_option('-s', '--screenshot', action='store_true', default=False,
+                     help='Attempt to grab screenshot and render using kitty image protocol')
+argparser.add_option('-i', '--interval', type='float',
+                     help='Interval in seconds to redraw the screenshot. Currently only '
+                          'works for one node')
 argparser.add_option('-w','--windowed', action='store_true', default=False,
                      help='Open terminal windows for each node.  The '
                           'environment variable NODECONSOLE_WINDOWED_COMMAND '
@@ -69,6 +98,179 @@ argparser.add_option('-w','--windowed', action='store_true', default=False,
 
 (options, args) = argparser.parse_args()
 
+oldtcattr = None
+oldfl = None
+
+def get_coords():
+    sys.stdout.write('\x1b[6n') #
+    sys.stdout.flush()
+    gotreply = select.select([sys.stdin,], [], [], 0.250)[0]
+    if gotreply:
+        response = ''
+        while select.select([sys.stdin,], [], [], 0.1)[0] and 'R' not in response:
+            response += sys.stdin.read()
+        coords = response.replace('R', '').split('[')[1].split(';')
+    #sys.stdout.write('\x1b[{}:{}H'.format(*coords))
+
+def direct_console():
+    global oldtcattr
+    global oldfl
+    oldtcattr = termios.tcgetattr(sys.stdin.fileno())
+    oldfl = fcntl.fcntl(sys.stdin.fileno(), fcntl.F_GETFL)
+    tty.setraw(sys.stdin.fileno())
+    fcntl.fcntl(sys.stdin.fileno(), fcntl.F_SETFL, oldfl | os.O_NONBLOCK)
+
+def indirect_console():
+    fcntl.fcntl(sys.stdin.fileno(), fcntl.F_SETFL, oldfl & ~os.O_NONBLOCK)
+    termios.tcsetattr(sys.stdin.fileno(), termios.TCSANOW, oldtcattr)
+
+def determine_tile_size(numnodes):
+    # for now, smash everything to a common aspect ratio.  16:11
+    # is pretty much wrong for everything, making 4:3 a bit too wide
+    # and 16:9 significantly too narrow, but it is serviceable
+    # An improvement could come with us owning the scaling
+    # instead of delegating to Kitty, which says if we specify both,
+    # we get stretching.  In theory we should be able to get aspect correct
+    # from kitty by omitting, but:
+    # then we don't know how much to move the cursor left after draw_image
+    # Konsole won't scale at all with only partial scaling specified
+    cheight, cwidth, pixwidth, pixheight = sq.get_screengeom()
+    # 16:12 is to roughly account for the 'titles' of the tiles
+    ratio = (pixwidth / 16) / (pixheight / 12)
+    bestdeviation = None
+    bestdims = []
+    for i in range(1, numnodes + 1):
+        number = numnodes
+        while number % i != 0:
+            number += 1
+        columns = i
+        rows = number // i
+        deviation = abs(ratio -  (columns / rows))
+        if bestdeviation is None:
+            bestdeviation = deviation
+            bestdims = [columns, rows]
+        elif deviation < bestdeviation:
+            bestdeviation = deviation
+            bestdims = [columns, rows]
+    # ok, the above algorithm can still pick things like
+    # 1 2 3
+    # 4
+    # So we will let it pick the number of rows, and
+    # then see if we can chop columns and still fit
+    while (bestdims[0] - 1) * bestdims[1] >= numnodes:
+        bestdims[0] = bestdims[0] - 1
+    cellswide = cwidth // bestdims[0]
+    cellshigh = cheight // bestdims[1]
+    tilewidth = cellswide * pixwidth / cwidth
+    tileheight = cellshigh * pixheight / cheight
+    if tilewidth > (tileheight * 16 / 11):
+        tilewidth = tileheight * 16 / 11
+        cellswide = int(tilewidth // (pixwidth / cwidth))
+    if tileheight > (tilewidth * 11 /16):
+        tileheight = tilewidth * 11 / 16
+        cellshigh = int(tileheight // (pixheight / cheight))
+    bestdims = bestdims + [cellswide, cellshigh, cellshigh * bestdims[1]]
+    # incur any scrolling we might get. This allows us to accurately
+    # save/restore cursor or even get coordinates without scrolling fouling
+    # the desired target
+    sys.stdout.write('\n' * bestdims[4])
+    sys.stdout.flush()
+    cursor_up(bestdims[4])
+    return bestdims
+
+cursor_saved = False
+def sticky_cursor():
+    global cursor_saved
+# get cursor restore_position
+    if sys.stdin.isatty() and not cursor_saved:
+        try:
+            direct_console()
+            sys.stdout.write('\x1b7')
+            cursor_saved = True
+        finally:
+            indirect_console()
+    elif cursor_saved:
+        try:
+            direct_console()
+            sys.stdout.write('\x1b8')
+        finally:
+            indirect_console()
+
+def cursor_up(count=1):
+    sys.stdout.write(f'\x1b[{count}A')
+def cursor_down(count=1):
+    sys.stdout.write(f'\x1b[{count}B')
+def cursor_right(count=1):
+    sys.stdout.write(f'\x1b[{count}C')
+def cursor_left(count=1):
+    sys.stdout.write(f'\x1b[{count}D')
+def cursor_save():
+    sys.stdout.write('\x1b7')
+def cursor_restore():
+    sys.stdout.write('\x1b8')
+def cursor_hide():
+    sys.stdout.write('\x1b[?25l')
+def cursor_show():
+    sys.stdout.write('\x1b[?25h')
+
+def draw_image(data, width, height):
+    imageformat = os.environ.get('CONFLUENT_IMAGE_PROTOCOL', 'kitty')
+    if imageformat == 'sixel':
+        sixel_draw(data)
+    elif imageformat == 'iterm':
+        iterm_draw(data, width, height)
+    else:
+        kitty_draw(data, width, height)
+
+
+def sixel_draw(data):
+    bindata = base64.b64decode(data)
+    binfile = io.BytesIO()
+    binfile.write(bindata)
+    binfile.seek(0)
+    DumbWriter().draw(binfile)
+
+def iterm_draw(data, width, height):
+    if not height:
+        height = 'auto'
+    if not width:
+        width = 'auto'
+    bindata = base64.b64decode(data)
+    datalen = len(bindata)
+    sys.stdout.write(
+        '\x1b]1337;File=inline=1;width={};height={};size={}:'.format(width,height,datalen))
+    sys.stdout.write(data.decode('utf8'))
+    sys.stdout.write('\a')
+    sys.stdout.flush()
+
+def kitty_draw(data, width, height):
+    if Image:
+        bindata = base64.b64decode(data)
+        binfile = io.BytesIO()
+        binfile.write(bindata)
+        binfile.seek(0)
+        img = Image.open(binfile)
+        outfile = io.BytesIO()
+        img.save(outfile, format='PNG')
+        data = base64.b64encode(outfile.getbuffer())
+    preamble = '\x1b_Ga=T,f=100'
+    if height:
+        preamble += f',r={height},c={width}'
+    #sys.stdout.write(repr(preamble))
+    #sys.stdout.write('\xb[{}D'.format(len(repr(preamble))))
+    #return
+    first = True
+    while data:
+        chunk, data = data[:4096], data[4096:]
+        m = 1 if data else 0
+        if first:
+            sys.stdout.write('{},m={};'.format(preamble, m))
+        else:
+            sys.stdout.write('\x1b_Gm={};'.format(m))
+        sys.stdout.write(chunk.decode('utf8'))
+        sys.stdout.write('\x1b\\')
+        sys.stdout.flush()
+
 pass_through_args = []
 killcon = False
 try:
@@ -106,6 +308,102 @@ if options.Timestamp:
     logreader.dump_to_console(logname)
     sys.exit(0)
 
+def prep_node_tile(node):
+    currcolcell, currrowcell = nodepositions[node]
+    if currcolcell:
+        cursor_right(currcolcell)
+    if currrowcell:
+        cursor_down(currrowcell)
+    sys.stdout.write(node)
+    cursor_left(len(node))
+    cursor_down()
+
+def reset_cursor(node):
+    currcolcell, currrowcell = nodepositions[node]
+    if currcolcell:
+        cursor_left(currcolcell)
+    cursor_up(currrowcell + 1)
+
+nodepositions = {}
+numrows = 0
+def do_screenshot():
+    global numrows
+    cwidth = None
+    cheight = None
+    sess = client.Command()
+    if options.tile:
+        imageformat = os.environ.get('CONFLUENT_IMAGE_PROTOCOL', 'kitty')
+        if imageformat not in ('kitty', 'iterm'):
+            sys.stderr.write('Tiled screenshots only supported with kitty or iterm protocol')
+            sys.exit(1)
+        allnodes = []
+        numnodes = 0
+        for res in sess.read('/noderange/{}/nodes/'.format(args[0])):
+            allnodes.append(res['item']['href'].replace('/', ''))
+            numnodes += 1
+        cols, rows, cwidth, cheight, numrows = determine_tile_size(numnodes)
+        currcol = 1
+        currcolcell = 0
+        currrowcell = 0
+        for node in allnodes:
+            nodepositions[node] = currcolcell, currrowcell
+            if currcol < cols:
+                currcol += 1
+                currcolcell += cwidth
+            else:
+                currcol = 1
+                currcolcell = 0
+                currrowcell += cheight
+    elif options.interval is not None:
+        sys.stdout.write('\x1bc')
+    firstnodename = None
+    dorefresh = True
+    while dorefresh:
+        for res in sess.read('/noderange/{}/console/ikvm_screenshot'.format(args[0])):
+            for node in res.get('databynode', {}):
+                if not firstnodename:
+                    firstnodename = node
+                imgdata = res['databynode'][node].get('image', {}).get('imgdata', None)
+                if imgdata:
+                    if len(imgdata) < 32: # We were subjected to error
+                        sys.stderr.write(f'{node}: Unable to get screenshot\n')
+                        continue
+                    if node in nodepositions:
+                        prep_node_tile(node)
+                        cursor_save()
+                    else:
+                        if options.interval is not None:
+                            if node != firstnodename:
+                                sys.stderr.write('Multiple nodes not supported for interval')
+                                sys.exit(1)
+                            sticky_cursor()
+                        sys.stdout.write('{}: '.format(node))
+                    # one row is used by our own name, so cheight - 1 for that allowance
+                    draw_image(imgdata.encode(), cwidth, cheight - 1 if cheight else cheight)
+                    if node in nodepositions:
+                        cursor_restore()
+                        reset_cursor(node)
+                    else:
+                        sys.stdout.write('\n')
+                    sys.stdout.flush()
+        if options.interval is None:
+            dorefresh = False
+        else:
+            dorefresh = True
+            time.sleep(options.interval)
+    sys.exit(0)
+
+if options.screenshot:
+    try:
+        cursor_hide()
+        do_screenshot()
+    except KeyboardInterrupt:
+        pass
+    finally:
+        cursor_show()
+        cursor_down(numrows)
+        sys.stdout.write('\n')
+    sys.exit(0)
 def kill(noderange):
     sess = client.Command()
     envstring=os.environ.get('NODECONSOLE_WINDOWED_COMMAND')
@@ -243,7 +541,7 @@ if options.windowed:
             elif 'Height' in line:
                 window_height = int(line.split(':')[1])
             elif '-geometry' in line:
-                l = re.split(' |x|-|\+', line)
+                l = re.split(' |x|-|\\+', line)
                 l_nosp = [ele for ele in l if ele.strip()]
                 wmxo = int(l_nosp[1])
                 wmyo = int(l_nosp[2])

confluent_client/bin/nodedeploy

@@ -81,6 +81,12 @@ def main(args):
     if not args.profile and args.network:
         sys.stderr.write('Both noderange and a profile name are required arguments to request a network deployment\n')
         return 1
+    if args.clear and args.profile:
+        sys.stderr.write(
+            'The -c/--clear option should not be used with a profile, '
+            'it is a request to not deploy any profile, and will clear '
+            'whatever the current profile is without being specified\n')
+        return 1
     if extra:
         sys.stderr.write('Unrecognized arguments: ' + repr(extra) + '\n')
     c = client.Command()
@@ -166,8 +172,6 @@ def main(args):
                     ','.join(errnodes)))
             return 1
         rc |= c.simple_noderange_command(args.noderange, '/power/state', 'boot')
-    if args.network and not args.prepareonly:
-        return rc
     return 0
 
 if __name__ == '__main__':

confluent_client/bin/nodeeventlog

@@ -78,8 +78,11 @@ exitcode = 0
 def format_event(evt):
     retparts = []
     if 'timestamp' in evt and evt['timestamp'] is not None:
-        display = dt.strptime(evt['timestamp'], '%Y-%m-%dT%H:%M:%S')
-        retparts.append(display.strftime('%m/%d/%Y %H:%M:%S'))
+        try:
+            display = dt.strptime(evt['timestamp'], '%Y-%m-%dT%H:%M:%S')
+            retparts.append(display.strftime('%m/%d/%Y %H:%M:%S'))
+        except ValueError:
+            display = ''    
     dscparts = []
     if evt.get('log_id', None):
         retparts.append(evt['log_id'] + ':')

confluent_client/bin/nodefirmware

@@ -56,7 +56,7 @@ components = ['all']
 
 argparser = optparse.OptionParser(
     usage="Usage: "
-          "%prog <noderange> [list][update [--backup <file>]]|[<components>]")
+          "%prog <noderange> [list][updatestatus][update [--backup <file>]]|[<components>]")
 argparser.add_option('-b', '--backup', action='store_true',
                      help='Target a backup bank rather than primary')
 argparser.add_option('-m', '--maxnodes', type='int',
@@ -65,14 +65,18 @@ argparser.add_option('-m', '--maxnodes', type='int',
              
 (options, args) = argparser.parse_args()
 upfile = None
+querystatus = False
 try:
     noderange = args[0]
     if len(args) > 1:
         if args[1] == 'update':
             upfile = args[2]
         else:
+            comps = []
             if args[1] == 'list':
                 comps = args[2:]
+            elif args[1] == 'updatestatus':
+                querystatus = True
             else:
                 comps = args[1:]
             components = []
@@ -171,7 +175,16 @@ def show_firmware(session):
 
 try:
     session = client.Command()
-    if upfile is None:
+    if querystatus:
+        for res in session.read(
+                '/noderange/{0}/inventory/firmware/updatestatus'.format(noderange)):
+            for node in res.get('databynode', {}):
+                currstat = res['databynode'][node].get('status', None)
+                if currstat:
+                    print('{}: {}'.format(node, currstat))
+                else:
+                    print(repr(res))
+    elif upfile is None:
         show_firmware(session)
     else:
         update_firmware(session, upfile)

confluent_client/bin/nodel2traceroute

@@ -0,0 +1,176 @@
+#!/usr/libexec/platform-python
+# vim: tabstop=4 shiftwidth=4 softtabstop=4
+
+# Copyright 2017 Lenovo
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+__author__ = 'tkucherera'
+
+import optparse
+import os
+import signal
+import sys
+import subprocess
+
+try:
+    signal.signal(signal.SIGPIPE, signal.SIG_DFL)
+except AttributeError:
+    pass
+path = os.path.dirname(os.path.realpath(__file__))
+path = os.path.realpath(os.path.join(path, '..', 'lib', 'python'))
+if path.startswith('/opt'):
+    sys.path.append(path)
+
+import confluent.client as client
+
+argparser = optparse.OptionParser(
+    usage="Usage: %prog <start_node> -i <interface> <end_node> -e <eface>",
+)
+argparser.add_option('-i', '--interface', type='str',
+                    help='interface to check path against for the start node')
+argparser.add_option('-e', '--eface', type='str',
+                    help='interface to check path against for the end node')
+argparser.add_option('-c', '--cumulus', action="store_true", dest="cumulus",
+                    help='return layer 2 route through cumulus switches only')
+
+(options, args) = argparser.parse_args()
+
+try:
+    start_node = args[0]
+    end_node = args[1]
+    interface = options.interface
+    eface = options.eface
+except IndexError:
+    argparser.print_help()
+    sys.exit(1)
+
+session = client.Command()
+
+def get_neighbors(switch):
+    switch_neigbors = []
+    url = '/networking/neighbors/by-switch/{0}/by-peername/'.format(switch)
+    for neighbor in session.read(url):
+        try:
+            switch = neighbor['item']['href'].strip('/')
+        except:
+            continue
+        if switch in all_switches:
+            switch_neigbors.append(switch)
+    return switch_neigbors
+
+def find_path(start, end, path=[]):
+    path = path + [start]
+    if start == end:
+        return path  # If start and end are the same, return the path
+
+    for node in get_neighbors(start):
+        if node not in path:
+            new_path = find_path(node, end, path)
+            if new_path:
+                return new_path  # If a path is found, return it
+
+    return None  # If no path is found, return None
+
+def is_cumulus(switch):
+    try:
+        read_attrib = subprocess.check_output(['nodeattrib', switch, 'hardwaremanagement.method'])
+    except subprocess.CalledProcessError:
+        return False
+    for attribs in read_attrib.decode('utf-8').split('\n'):
+        if len(attribs.split(':')) > 1:
+            attrib = attribs.split(':')
+            if attrib[2].strip() == 'affluent':
+                return True
+            else:
+                return False
+        else:
+            return False
+
+
+def host_to_switch(node, interface=None):
+    # first check the the node config to see what switches are connected
+    # if host is in rhel can use nmstate package
+    if node in all_switches:
+        return [node]
+    switches = []
+    netarg = 'net.*.switch'
+    if interface:
+        netarg = 'net.{0}.switch'.format(interface)
+    try:
+        read_attrib = subprocess.check_output(['nodeattrib', node, netarg])
+    except subprocess.CalledProcessError:
+        return False
+    for attribs in read_attrib.decode('utf-8').split('\n'):
+        attrib = attribs.split(':')
+        try:
+            if ' net.mgt.switch' in attrib or attrib[2] == '':
+                continue
+        except IndexError:
+            continue
+        switch = attrib[2].strip()
+        if is_cumulus(switch) and options.cumulus:
+            switches.append(switch)
+        else:
+            switches.append(switch)
+    return switches
+
+def path_between_nodes(start_switches, end_switches):
+    for start_switch in start_switches:
+        for end_switch in end_switches:
+            if start_switch == end_switch:
+                return [start_switch]
+            else:
+                path = find_path(start_switch, end_switch)
+                if path:
+                    return path
+                else:
+                    return 'No path found'
+
+
+all_switches = []
+for res in session.read('/networking/neighbors/by-switch/'):
+    if 'error' in res:
+        sys.stderr.write(res['error'] + '\n')
+        exitcode = 1
+    else:
+        switch = (res['item']['href'].replace('/', ''))
+        all_switches.append(switch)
+
+end_nodeslist = []
+nodelist = '/noderange/{0}/nodes/'.format(end_node)
+for res in session.read(nodelist):
+    if 'error' in res:
+        sys.stderr.write(res['error'] + '\n')
+        exitcode = 1
+    else:
+        elem=(res['item']['href'].replace('/', ''))
+        end_nodeslist.append(elem)
+
+start_switches = host_to_switch(start_node, interface)
+for end_node in end_nodeslist:
+    if end_node:
+        end_switches = host_to_switch(end_node, eface)
+        if not end_switches:
+            print('Error: net.{0}.switch attribute is not valid')
+            continue
+        path = path_between_nodes(start_switches, end_switches)
+        print(f'{start_node} to {end_node}: {path}')
+
+# TODO dont put switches that are connected through management interfaces. 
+
+
+    
+
+
+

confluent_client/bin/nodelist

@@ -68,7 +68,7 @@ def main():
             else:
                 elem=(res['item']['href'].replace('/', ''))
                 list.append(elem)
-    print(options.delim.join(list))
+        print(options.delim.join(list))
 
     sys.exit(exitcode)
 

confluent_client/bin/nodeping

@@ -1,4 +1,4 @@
-#!/usr/bin/python2
+#!/usr/bin/python3
 # vim: tabstop=4 shiftwidth=4 softtabstop=4
 
 # Copyright 2016-2017 Lenovo
@@ -42,6 +42,8 @@ def run():
         usage="Usage: %prog [options] noderange")
     argparser.add_option('-f', '-c', '--count', type='int', default=168,
                          help='Number of commands to run at a time')
+    argparser.add_option('-o', '--origname', action='store_true',
+                         help='Use original nodename in print out even if substituted')
     argparser.add_option('-s', '--substitutename',
                          help='Use a different name other than the nodename for ping, with {}, it is the entire name evaluated as an expression, otherwise it is used as a suffix')
     # among other things, FD_SETSIZE limits.  Besides, spawning too many
@@ -83,7 +85,10 @@ def run():
             cmdv = ['ping', '-c', '1', '-W', '1', pingnode]
             if currprocs < concurrentprocs:
                 currprocs += 1
-                run_cmdv(pingnode, cmdv, all, pipedesc)
+                if options.origname:
+                    run_cmdv(node, cmdv, all, pipedesc)
+                else:
+                    run_cmdv(pingnode, cmdv, all, pipedesc)
             else:
                 pendingexecs.append((pingnode, cmdv))
     if not all or exitcode:

confluent_client/bin/nodesensors

@@ -67,7 +67,7 @@ argparser.add_option('-n', '--numreadings', type='int',
 argparser.add_option('-c', '--csv', action='store_true',
                      help='Output in CSV format')
 argparser.add_option('-s', '--skipnumberless', action='store_true',
-                     help='Output in CSV format')
+                     help='Do not show non-numeric sensors')
 (options, args) = argparser.parse_args()
 repeatmode = False
 if options.interval:

confluent_client/bin/stats

@@ -72,6 +72,8 @@ def plot(gui, output, plotdata, bins, fmt):
             tdata = io.BytesIO()
         plt.savefig(tdata)
     if not gui and not output:
+        if fmt == 'environment':
+            fmt = os.environ.get('CONFLUENT_IMAGE_PROTOCOL', 'kitty')
         if fmt == 'sixel':
             writer = DumbWriter()
             writer.draw(tdata)
@@ -108,7 +110,7 @@ aparser = argparse.ArgumentParser(description='Quick access to common statistics
 aparser.add_argument('-c', type=int, default=0, help='Column number to analyze (default is last column)')
 aparser.add_argument('-d', default=None, help='Value used to separate columns')
 aparser.add_argument('-x', default=False, action='store_true', help='Output histogram in graphical format')
-aparser.add_argument('-f', default='sixel', help='Format for histogram output (sixel/iterm/kitty)')
+aparser.add_argument('-f', default='environment', help='Format for histogram output (sixel/iterm/kitty)')
 aparser.add_argument('-s', default=0, help='Number of header lines to skip before processing')
 aparser.add_argument('-g', default=False, action='store_true', help='Open histogram in separate graphical window')
 aparser.add_argument('-o', default=None, help='Output histogram to the specified filename in PNG format')

confluent_client/confluent/client.py

@@ -595,7 +595,7 @@ def print_attrib_path(path, session, requestargs, options, rename=None, attrpref
                 else:
                     printmissing.add(attr)
         for missing in printmissing:
-            sys.stderr.write('Error: {0} not a valid attribute\n'.format(attr))
+            sys.stderr.write('Error: {0} not a valid attribute\n'.format(missing))
     return exitcode
 
 
@@ -668,6 +668,9 @@ def updateattrib(session, updateargs, nodetype, noderange, options, dictassign=N
         for attrib in updateargs[1:]:
             keydata[attrib] = None
         for res in session.update(targpath, keydata):
+            for node in res.get('databynode', {}):
+                for warnmsg in res['databynode'][node].get('_warnings', []):
+                    sys.stderr.write('Warning: ' + warnmsg + '\n')
             if 'error' in res:
                 if 'errorcode' in res:
                     exitcode = res['errorcode']
@@ -702,6 +705,14 @@ def updateattrib(session, updateargs, nodetype, noderange, options, dictassign=N
                     noderange, 'attributes/all', dictassign[key], key)
     else:
         if "=" in updateargs[1]:
+            update_ready = True
+            for arg in updateargs[1:]:
+                if not '=' in arg:
+                    update_ready = False
+                    exitcode = 1
+            if not update_ready:
+                sys.stderr.write('Error: {0} Can not set and read at the same time!\n'.format(str(updateargs[1:])))
+                sys.exit(exitcode)
             try:
                 for val in updateargs[1:]:
                     val = val.split('=', 1)

confluent_client/confluent/screensqueeze.py

@@ -18,8 +18,9 @@ import struct
 import termios
 
 def get_screengeom():
-    return struct.unpack('hh', fcntl.ioctl(sys.stdout, termios.TIOCGWINSZ,
-                                           b'....'))
+    # returns height in cells, width in cells, width in pixels, height in pixels
+    return struct.unpack('hhhh', fcntl.ioctl(sys.stdout, termios.TIOCGWINSZ,
+                                           b'........'))
 class ScreenPrinter(object):
 
     def __init__(self, noderange, client, textlen=4):
@@ -58,7 +59,7 @@ class ScreenPrinter(object):
 
     def drawscreen(self, node=None):
         if self.squeeze:
-            currheight, currwidth = get_screengeom()
+            currheight, currwidth, _, _ = get_screengeom()
             currheight -= 2
             if currheight < 1:
                 currheight = 1
@@ -120,6 +121,7 @@ if __name__ == '__main__':
     c = client.Command()
     p = ScreenPrinter('d1-d12', c)
     p.set_output('d3', 'Upload: 67%')
+    p.set_output('d7', 'Upload: 67%')
 
 
 

confluent_client/confluent/textgroup.py

@@ -98,17 +98,24 @@ class GroupedData(object):
         self.byoutput = {}
         self.header = {}
         self.client = confluentconnection
+        self.detectedpad = None
 
     def generate_byoutput(self):
         self.byoutput = {}
+        thepad = self.detectedpad if self.detectedpad else ''
         for n in self.bynode:
-            output = '\n'.join(self.bynode[n])
+            output = ''
+            for ln in self.bynode[n]:
+                output += ln.replace(thepad, '', 1) + '\n'
             if output not in self.byoutput:
                 self.byoutput[output] = set([n])
             else:
                 self.byoutput[output].add(n)
 
     def add_line(self, node, line):
+        wspc = re.search(r'^\s*', line).group()
+        if self.detectedpad is None or len(wspc) < len(self.detectedpad):
+            self.detectedpad = wspc
         if node not in self.bynode:
             self.bynode[node] = [line]
         else:
@@ -164,7 +171,9 @@ class GroupedData(object):
                     self.byoutput[outdata])))
                 currout += '\n====================================\n'
             currout += outdata
-            currout += '\n\n'
+            if currout[-1] != '\n':
+                currout += '\n'
+            currout += '\n'
             output.write(currout)
         output.flush()
 
@@ -204,7 +213,9 @@ class GroupedData(object):
             else:
                 currout += '\n'.join(colordiff(modaloutput.split('\n'),
                                                  outdata.split('\n')))
-            currout += '\n\n'
+            if currout[-1] != '\n':
+                currout += '\n'
+            currout += '\n'
             if reverse:
                 revoutput.append(currout)
             else:
@@ -219,4 +230,4 @@ if __name__ == '__main__':
         if not line:
             continue
         groupoutput.add_line(*line.split(': ', 1))
-    groupoutput.print_deviants()
+    groupoutput.print_deviants()

confluent_client/confluent_client.spec.tmpl

@@ -1,12 +1,16 @@
 %define name confluent_client
 %define version #VERSION#
+%define fversion %{lua:
+sv, _ = string.gsub("#VERSION#", "[~+]", "-")
+print(sv)
+}
 %define release 1
 
 Summary: Client libraries and utilities for confluent
 Name: %{name}
 Version: %{version}
 Release: %{release}
-Source0: %{name}-%{version}.tar.gz
+Source0: %{name}-%{fversion}.tar.gz
 License: Apache2
 Group: Development/Libraries
 BuildRoot: %{_tmppath}/%{name}-%{version}-%{release}-buildroot
@@ -21,7 +25,7 @@ This package enables python development and command line access to
 a confluent server.
 
 %prep
-%setup -n %{name}-%{version} -n %{name}-%{version}
+%setup -n %{name}-%{fversion}
 
 %build
 %if "%{dist}" == ".el7"

confluent_client/confluent_env.sh

@@ -153,11 +153,11 @@ _confluent_osimage_completion()
 {
     _confluent_get_args
     if [ $NUMARGS == 2 ]; then
-        COMPREPLY=($(compgen -W "initialize import updateboot rebase" -- ${COMP_WORDS[COMP_CWORD]}))
+        COMPREPLY=($(compgen -W "initialize import importcheck updateboot rebase" -- ${COMP_WORDS[COMP_CWORD]}))
         return
     elif [ ${CMPARGS[1]} == 'initialize' ]; then
         COMPREPLY=($(compgen -W "-h -u -s -t -i" -- ${COMP_WORDS[COMP_CWORD]}))
-    elif [ ${CMPARGS[1]} == 'import' ]; then
+    elif [ ${CMPARGS[1]} == 'import' ] || [ ${CMPARGS[1]} == 'importcheck' ]; then
         compopt -o default
         COMPREPLY=()
         return

confluent_client/doc/man/confluent2hosts.ronn

@@ -13,7 +13,7 @@ noderange.  There are two general approaches.
 
 It can be used ad-hoc, using -i and -n to specify the address and name portions respectively.  This accepts the standard confluent expression syntax, allowing for things like 172.30.1.{n1} or {node}.{dns.domain} or {bmc}.
 
-It can also read from the confluent db, using `-a`.  In this mode, each net.<value>.<attribute> group is pulled together into hosts lines.  ipv4_address and ipv6_address fields are associated with the corresponding hostname attributes.
+It can also read from the confluent db, using `-a`.  In this mode, each net.<value>.<attribute> group is pulled together into hosts lines.  ipv4_address and ipv6_address fields are associated with the corresponding hostname attributes. You can use `-f` to put the FQDN first.
 
 ## EXAMPLES
 

confluent_client/doc/man/confluentdbutil.ronn

@@ -38,5 +38,8 @@ the json files (password protected, removed from the files, or unprotected).
   keys do not change and as such they do not require
   incremental backup.
   
+* `-y`, `--yaml
+  Use YAML instead of JSON as file format
+  
   * `-h`, `--help`:
   Show help message and exit

confluent_client/doc/man/nodeapply.ronn

@@ -0,0 +1,30 @@
+nodeapply(8) -- Execute command on many nodes in a noderange through ssh
+=========================================================================
+
+## SYNOPSIS
+
+`nodeapply [options] <noderange>`
+
+## DESCRIPTION
+
+Provides shortcut access to a number of common operations against deployed
+nodes.  These operations include refreshing ssh certificates and configuration,
+rerunning syncflies, and executing specified postscripts.
+
+## OPTIONS
+  
+* `-k`, `--security`
+  Refresh SSH configuration (hosts.equiv and node SSH certificates)
+  
+* `-F`, `--sync`
+  Rerun syncfiles from deployed profile
+  
+* `-P SCRIPTS`, `--scripts=SCRIPTS`
+  Re-run specified scripts, with full path under scripts specified, e.g. post.d/scriptname,firstboot.d/otherscriptname
+ 
+* `-c COUNT`, `-f COUNT`, `--count=COUNT`
+  Specify the maximum number of instances to run concurrently
+
+* `-m MAXNODES`, `--maxnodes=MAXNODES`
+  Specify a maximum number of nodes to run remote ssh command to, prompting
+  if over the threshold

confluent_client/doc/man/nodebmcpassword.ronn

@@ -0,0 +1,28 @@
+nodebmcpassword(8) -- Change management controller password for a specified user
+=========================================================
+
+## SYNOPSIS
+
+`nodebmcpassword <noderange> <username> <new_password>`  
+
+## DESCRIPTION
+
+`nodebmcpassword` allows you to change the management controller password for a user on a specified noderange
+
+## OPTIONS
+
+* `-m MAXNODES`, `--maxnodes=MAXNODES`:
+  Number of nodes to affect before prompting for
+  confirmation
+  
+* `-h`, `--help`:
+  Show help message and exit  
+
+## EXAMPLES:
+
+* Reset the management controller for nodes n1 through n4:
+  `# nodebmcreset n1-n4`  
+  `n1: Password Change Successful`  
+  `n2: Password Change Successful`  
+  `n3: Password Change Successful`  
+  `n4: Password Change Successful`  

confluent_client/doc/man/nodefirmware.ronn

@@ -3,7 +3,7 @@ nodefirmware(8) -- Report firmware information on confluent nodes
 
 ## SYNOPSIS
 
-`nodefirmware <noderange> [list][update [--backup <file>]]|[<components>]`
+`nodefirmware <noderange> [list][updatestatus][update [--backup <file>]]|[<components>]`
 
 ## DESCRIPTION
 
@@ -17,6 +17,9 @@ not be relevant to redfish.  Additionally, the Lenovo XCC makes certain
 information available over IPMI that is not otherwise available (for example
 the FPGA version where applicable).
 
+The updatestatus argument will describe the state of firmware updates on the
+nodes.
+
 In the update form, it accepts a single file and attempts to update it using
 the out of band facilities.  Firmware updates can end in one of three states:
 

confluent_client/doc/man/nodel2traceroute.ronn

@@ -0,0 +1,42 @@
+nodel2traceroute(8) -- returns the layer 2 route through an Ethernet network managed by confluent given 2 end points.
+==============================
+## SYNOPSIS
+`nodel2traceroute [options] <start_node> <end_noderange>`  
+
+## DESCRIPTION  
+**nodel2traceroute** is a command that returns the layer 2 route for the configered interfaces in nodeattrib.
+It can also be used with the -i and -e options to check against specific interfaces on the endpoints. If the 
+--interface or --eface option are not used then the command will check for routes against all the defined 
+interfaces in nodeattrib (net.*.switch) for the nodes.
+
+
+
+## PREREQUISITES
+**nodel2traceroute** the net.<interface>.switch attributes have to be set on the end points if endpoint is not a switch
+
+
+## OPTIONS
+* ` -e` EFACE, --eface=INTERFACE
+   interface to check against for the second end point or end points if using checking against multiple nodes 
+* ` -i` INTERFACE, --interface=INTERFACE
+   interface to check against for the first end point  
+* ` -c` CUMULUS, --cumulus=CUMULUS
+   return layer 2 route through cumulus switches only 
+* `-h`, `--help`:  
+  Show help message and exit      
+
+   
+## EXAMPLES
+ * Checking route between two nodes:  
+  `# nodel2traceroute n244 n1851`  
+  `n244 to n1851: ['switch114']`    
+    
+* Checking route from one node to multiple nodes:  
+  `# nodel2traceroute n244 n1833,n1851`    
+  `n244 to n1833: ['switch114', 'switch7', 'switch32', 'switch253', 'switch85', 'switch72', 'switch21', 'switch2', 'switch96', 'switch103', 'switch115']
+   n244 to n1851: ['switch114']`  
+
+
+
+ 
+

confluent_osdeploy/buildrpm

@@ -1,7 +1,11 @@
 VERSION=`git describe|cut -d- -f 1`
 NUMCOMMITS=`git describe|cut -d- -f 2`
 if [ "$NUMCOMMITS" != "$VERSION"  ]; then
-    VERSION=$VERSION.dev$NUMCOMMITS.g`git describe|cut -d- -f 3`
+    LASTNUM=$(echo $VERSION|rev|cut -d . -f 1|rev)
+    LASTNUM=$((LASTNUM+1))
+    FIRSTPART=$(echo $VERSION|rev|cut -d . -f 2- |rev)
+    VERSION=${FIRSTPART}.${LASTNUM}
+    VERSION=$VERSION~dev$NUMCOMMITS+`git describe|cut -d- -f 3`
 fi
 sed -e "s/#VERSION#/$VERSION/" confluent_osdeploy.spec.tmpl > confluent_osdeploy.spec
 cd ..

confluent_osdeploy/buildrpm-aarch64

@@ -2,7 +2,11 @@ cd $(dirname $0)
 VERSION=`git describe|cut -d- -f 1`
 NUMCOMMITS=`git describe|cut -d- -f 2`
 if [ "$NUMCOMMITS" != "$VERSION"  ]; then
-    VERSION=$VERSION.dev$NUMCOMMITS.g`git describe|cut -d- -f 3`
+    LASTNUM=$(echo $VERSION|rev|cut -d . -f 1|rev)
+    LASTNUM=$((LASTNUM+1))
+    FIRSTPART=$(echo $VERSION|rev|cut -d . -f 2- |rev)
+    VERSION=${FIRSTPART}.${LASTNUM}
+    VERSION=$VERSION~dev$NUMCOMMITS+`git describe|cut -d- -f 3`
 fi
 sed -e "s/#VERSION#/$VERSION/" confluent_osdeploy-aarch64.spec.tmpl > confluent_osdeploy-aarch64.spec
 cd ..

confluent_osdeploy/common/initramfs/opt/confluent/bin/apiclient

@@ -14,6 +14,8 @@ import ssl
 import sys
 import struct
 import time
+import re
+import json
 
 class InvalidApiKey(Exception):
     pass
@@ -89,8 +91,11 @@ def scan_confluents():
                 confluentuuid = line.split(': ')[1]
                 msg += '/confluentuuid=' + confluentuuid
                 break
-    with open('/sys/devices/virtual/dmi/id/product_uuid') as uuidin:
-        msg += '/uuid=' + uuidin.read().strip()
+    try:
+        with open('/sys/devices/virtual/dmi/id/product_uuid') as uuidin:
+            msg += '/uuid=' + uuidin.read().strip()
+    except Exception:
+        pass
     for addrf in glob.glob('/sys/class/net/*/address'):
         with open(addrf) as addrin:
             hwaddr = addrin.read().strip()
@@ -412,8 +417,92 @@ class HTTPSClient(client.HTTPConnection, object):
                     self.node, [self.host], errout=self.errout)
         raise Exception(rsp.read())
 
+def get_current_vmnic_vswitch():
+    uplinkmatch = re.compile(r'^\s*Uplinks:\s*(.*)')
+    switchmatch = re.compile(r'^\s*Name:\s*(.*)')
+
+    vswinfo = subprocess.check_output(['localcli', 'network', 'vswitch', 'standard', 'list']).decode()
+
+    vmnic = None
+    vswitch_name = None
+
+    for info in vswinfo.split('\n'):
+        name_match = switchmatch.match(info)
+        if name_match:
+            vswitch_name = name_match.group(1).strip()
+
+        upinfo = uplinkmatch.match(info)
+        if upinfo:
+            vmnic = upinfo.group(1).strip()
+            if vmnic and 'vusb0' not in vmnic:
+                return vswitch_name, vmnic
+
+    return vswitch_name, vmnic
+
+def get_available_nics():
+    nicinfo = subprocess.check_output(['localcli', 'network', 'nic', 'list']).decode('utf8').split('\n')
+    available_nics = {}
+
+    # Skip headers and separators
+    parsing_started = False
+    for line in nicinfo:
+        if re.match(r'^-+', line):
+            parsing_started = True
+            continue
+        if not parsing_started or not line.strip():
+            continue
+
+        parts = re.split(r'\s{2,}', line.strip())
+        if len(parts) >= 5:
+            nic_name = parts[0]
+            nic_status = parts[4]  # "Link Status" este al 5-lea câmp
+            available_nics[nic_name] = nic_status
+
+    return available_nics
+
+def is_esxi():
+    return os.path.isdir("/etc/vmware")
+
+def fix_vswitch():
+    if is_esxi():
+        start_time = time.time()
+        while True:
+
+            current_vswitch, current_vmnic = get_current_vmnic_vswitch()
+            if current_vswitch is None:
+                raise RuntimeError("Panic: current vswitch is None")
+            if current_vmnic is None:
+                raise RuntimeError("Panic: current vmnic is None")
+
+            available_nics = get_available_nics()
+
+            if current_vmnic and available_nics.get(current_vmnic) == 'Up':
+                break
+
+            new_vmnic = next((nic for nic, status in available_nics.items() if status == 'Up'), None)
+
+            if new_vmnic and new_vmnic != current_vmnic:
+                subprocess.check_call(['localcli', 'network', 'vswitch', 'standard', 'uplink', 'remove',
+                                       '--uplink-name', current_vmnic, '--vswitch-name', current_vswitch])
+                subprocess.check_call(['localcli', 'network', 'vswitch', 'standard', 'uplink', 'add',
+                                       '--uplink-name', new_vmnic, '--vswitch-name', current_vswitch])
+            elif not new_vmnic:
+                if time.time() - start_time > 300:
+                    break
+            time.sleep(5)
+
+        time.sleep(5)
+
 if __name__ == '__main__':
     data = None
+
+    if '-f' in sys.argv:
+        try:
+            fix_vswitch()
+        except Exception as e:
+            print(f"fix_vswitch() error: {e}")
+        sys.argv.remove('-f')
+        sys.exit(0)
     usejson = False
     if '-j' in sys.argv:
         usejson = True
@@ -468,6 +557,7 @@ if __name__ == '__main__':
                 outf.write(chunk)
                 chunk = reader.read(16384)
         sys.exit(0)
+
     client = HTTPSClient(usejson, errout=errout, phmac=phmac, checkonly=checkonly)
     if waitfor:
         status = 201

confluent_osdeploy/common/profile/scripts/autoconsole

@@ -0,0 +1,117 @@
+#!/usr/bin/python3
+
+# This script evaluates whether firmware redirection is likely. It uses three cues:
+# - Does the system offer up SPCR? This would indicate that the firmware is doing serial output.
+#   Otherwise, there's no indication that the firmware cares about serial console.
+# - Is the system EFI?  BIOS implementations may not intercept text draw calls after POST exit,
+#   thus even when BIOS tells us serial port is in use, it may not be doing anything when
+#   grub would be running
+# - Is the serial port connected? In the event that firmware indicates serial port, but
+#   serial port is not reporting DCD, then it doesn't look like a comfortable enough scenario
+
+import fcntl
+import os
+import os.path
+import struct
+import subprocess
+import termios
+
+
+addrtoname = {
+    0x3f8: '/dev/ttyS0',
+    0x2f8: '/dev/ttyS1',
+    0x3e8: '/dev/ttyS2',
+    0x2e8: '/dev/ttyS3',
+}
+speedmap = {
+    0: None,
+    3: 9600,
+    4: 19200,
+    6: 57600,
+    7: 115200,
+}
+
+termiobaud = {
+    9600: termios.B9600,
+    19200: termios.B19200,
+    57600: termios.B57600,
+    115200: termios.B115200,
+}
+
+
+def deserialize_grub_rh():
+    if 'console=ttyS' in open('/proc/cmdline').read():
+        return None # User manually indicated serial config
+        # they own the grub behavior too for now
+    grublines = []
+    with open('/etc/default/grub') as grubin:
+        grublines = grubin.read().split('\n')
+    with open('/etc/default/grub', 'w') as grubout:
+        for grubline in grublines:
+            if grubline.startswith('GRUB_TERMINAL'):
+                grubline = grubline.replace('serial ', '')
+            grubout.write(grubline + '\n')
+    subprocess.check_call(['grub2-mkconfig', '-o', '/boot/grub2/grub.cfg'])
+
+def fixup_ubuntu_grub_serial():
+    # Ubuntu aggressively tries to graphics up
+    # grub. We will counter that for serial
+    # They also aggressively hide UI and
+    # block ability to interject. We will
+    # compromise and lean on nodeboot <node> setup
+    # as a means to give someone reasonable shot at
+    # the short timeout
+    with open('/etc/default/grub') as grubin:
+        grublines = grubin.read().split('\n')
+    with open('/etc/default/grub', 'w') as grubout:
+        for grubline in grublines:
+            if grubline.startswith('GRUB_TIMEOUT_STYLE=hidden'):
+                grubline = 'GRUB_TIMEOUT_STYLE=menu'
+            elif grubline.startswith('GRUB_TIMEOUT=0'):
+                grubline = 'GRUB_TIMEOUT=2'
+            elif grubline.startswith('#GRUB_TERMINAL=console'):
+                grubline = grubline.replace('#', '')
+            grubout.write(grubline + '\n')
+    subprocess.check_call(['update-grub'])
+
+def get_serial_config():
+    if not os.path.exists('/sys/firmware/efi'):
+        return None
+    spcr = open("/sys/firmware/acpi/tables/SPCR", "rb")
+    spcr = bytearray(spcr.read())
+    if spcr[8] != 2 or spcr[36] != 0 or spcr[40] != 1:
+        return None
+    address = struct.unpack('<Q', spcr[44:52])[0]
+    tty = None
+    try:
+        tty = addrtoname[address]
+    except KeyError:
+        return None
+    retval = { 'tty': tty }
+    try:
+        retval['speed'] = speedmap[spcr[58]]
+    except KeyError:
+        return None
+    if retval['speed']:
+        ttyf = os.open(tty, os.O_RDWR | os.O_NOCTTY)
+        currattr = termios.tcgetattr(ttyf)
+        currattr[4:6] = [0, termiobaud[retval['speed']]]
+        termios.tcsetattr(ttyf, termios.TCSANOW, currattr)
+    retval['connected'] = bool(struct.unpack('<I', fcntl.ioctl(
+        ttyf, termios.TIOCMGET, '\x00\x00\x00\x00'))[0] & termios.TIOCM_CAR)
+    os.close(ttyf)
+    return retval
+
+def main():
+    autoconscfg = get_serial_config()
+    if not autoconscfg or not autoconscfg['connected']:
+        return
+    if os.path.exists('/etc/redhat-release'):  # redhat family
+        deserialize_grub_rh()
+    elif os.path.exists('/etc/os-release'):
+        with open('/etc/os-release') as osr:
+            if 'Ubuntu' in osr.read():
+                fixup_ubuntu_grub_serial()
+
+if __name__ == '__main__':
+    main()

confluent_osdeploy/common/profile/scripts/confignet

@@ -86,7 +86,7 @@ def map_idx_to_name():
     for line in subprocess.check_output(['ip', 'l']).decode('utf8').splitlines():
         if line.startswith(' ') and 'link/' in line:
             typ = line.split()[0].split('/')[1]
-            devtype[prevdev] = typ if type != 'ether' else 'ethernet'
+            devtype[prevdev] = typ if typ != 'ether' else 'ethernet'
         if line.startswith(' '):
             continue
         idx, iface, rst = line.split(':', 2)
@@ -192,8 +192,10 @@ class NetplanManager(object):
             if needcfgwrite:
                 needcfgapply = True
                 newcfg = {'network': {'version': 2, 'ethernets': {devname: self.cfgbydev[devname]}}}
+                oumask = os.umask(0o77)
                 with open('/etc/netplan/{0}-confluentcfg.yaml'.format(devname), 'w') as planout:
                     planout.write(yaml.dump(newcfg))
+                os.umask(oumask)
         if needcfgapply:
             subprocess.call(['netplan', 'apply'])
 
@@ -224,7 +226,11 @@ class WickedManager(object):
             self.cfgbydev[devname] = currcfg
             for cfg in open(ifcfg).read().splitlines():
                 cfg = cfg.split('#', 1)[0]
-                kv = ' '.join(shlex.split(cfg)).split('=', 1)
+                try:
+                    kv = ' '.join(shlex.split(cfg)).split('=', 1)
+                except Exception:
+                    # unparseable line, likely having something we can't handle
+                    del self.cfgbydev[devname]
                 if len(kv) != 2:
                     continue
                 k, v = kv
@@ -295,7 +301,14 @@ class WickedManager(object):
 
 
 class NetworkManager(object):
-    def __init__(self, devtypes):
+    bondtypes = {
+            'lacp': '802.3ad',
+            'loadbalance': 'balance-alb',
+            'roundrobin': 'balance-rr',
+            'activebackup': 'active-backup',
+    }
+    def __init__(self, devtypes, deploycfg):
+        self.deploycfg = deploycfg
         self.connections = {}
         self.uuidbyname = {}
         self.uuidbydev = {}
@@ -345,7 +358,8 @@ class NetworkManager(object):
                     bondcfg[stg] = deats[stg]
         if member in self.uuidbyname:
             subprocess.check_call(['nmcli', 'c', 'del', self.uuidbyname[member]])
-        subprocess.check_call(['nmcli', 'c', 'add', 'type', 'bond-slave', 'master', team, 'con-name', member, 'connection.interface-name', member])
+        devtype = self.devtypes.get(member, 'bond-slave')
+        subprocess.check_call(['nmcli', 'c', 'add', 'type', devtype, 'master', team, 'con-name', member, 'connection.interface-name', member])
         if bondcfg:
             args = []
             for parm in bondcfg:
@@ -353,7 +367,7 @@ class NetworkManager(object):
                 args.append(bondcfg[parm])
             subprocess.check_call(['nmcli', 'c', 'm', team] + args)
 
-    def apply_configuration(self, cfg):
+    def apply_configuration(self, cfg, lastchance=False):
         cmdargs = {}
         cmdargs['connection.autoconnect'] = 'yes'
         stgs = cfg['settings']
@@ -367,6 +381,20 @@ class NetworkManager(object):
             cmdargs['ipv4.gateway'] = stgs['ipv4_gateway']
         if stgs.get('ipv6_gateway', None):
             cmdargs['ipv6.gateway'] = stgs['ipv6_gateway']
+        dnsips = self.deploycfg.get('nameservers', [])
+        if not dnsips:
+            dnsips = []
+        dns4 = []
+        dns6 = []
+        for dnsip in dnsips:
+            if '.' in dnsip:
+                dns4.append(dnsip)
+            elif ':' in dnsip:
+                dns6.append(dnsip)
+        if dns4:
+            cmdargs['ipv4.dns'] = ','.join(dns4)
+        if dns6:
+            cmdargs['ipv6.dns'] = ','.join(dns6)
         if len(cfg['interfaces']) > 1:  # team time.. should be..
             if not cfg['settings'].get('team_mode', None):
                 sys.stderr.write("Warning, multiple interfaces ({0}) without a team_mode, skipping setup\n".format(','.join(cfg['interfaces'])))
@@ -379,26 +407,46 @@ class NetworkManager(object):
             for arg in cmdargs:
                 cargs.append(arg)
                 cargs.append(cmdargs[arg])
-            subprocess.check_call(['nmcli', 'c', 'add', 'type', 'bond', 'con-name', cname, 'connection.interface-name', cname, 'bond.options', 'mode={}'.format(stgs['team_mode'])] + cargs)
+            if stgs['team_mode'] in self.bondtypes:
+                stgs['team_mode'] = self.bondtypes[stgs['team_mode']]
+            subprocess.check_call(['nmcli', 'c', 'add', 'type', 'bond', 'con-name', cname, 'connection.interface-name', cname, 'bond.options', 'miimon=100,mode={}'.format(stgs['team_mode'])] + cargs)
             for iface in cfg['interfaces']:
                 self.add_team_member(cname, iface)
             subprocess.check_call(['nmcli', 'c', 'u', cname])
         else:
             cname = stgs.get('connection_name', None)
             iname = list(cfg['interfaces'])[0]
-            if not cname:
-                cname = iname
+            ctype = self.devtypes.get(iname, None)
+            if not ctype:
+                if lastchance:
+                    sys.stderr.write("Warning, no device found for interface_name ({0}), skipping setup\n".format(iname))
+                return 1
+            if stgs.get('vlan_id', None):
+                vlan = stgs['vlan_id']
+                if ctype == 'infiniband':
+                    vlan = '0x{0}'.format(vlan) if not vlan.startswith('0x') else vlan
+                    cmdargs['infiniband.parent'] = iname
+                    cmdargs['infiniband.p-key'] = vlan
+                    iname = '{0}.{1}'.format(iname, vlan[2:])
+                elif ctype == 'ethernet':
+                    ctype = 'vlan'
+                    cmdargs['vlan.parent'] = iname
+                    cmdargs['vlan.id'] = vlan
+                    iname = '{0}.{1}'.format(iname, vlan)
+                else:
+                    sys.stderr.write("Warning, unknown interface_name ({0}) device type ({1}) for VLAN/PKEY, skipping setup\n".format(iname, ctype))
+                    return
+            cname = iname if not cname else cname
             u = self.uuidbyname.get(cname, None)
             cargs = []
             for arg in cmdargs:
                 cargs.append(arg)
                 cargs.append(cmdargs[arg])
             if u:
-                cmdargs['connection.interface-name'] = iname
-                subprocess.check_call(['nmcli', 'c', 'm', u] + cargs)
+                subprocess.check_call(['nmcli', 'c', 'm', u, 'connection.interface-name', iname] + cargs)
                 subprocess.check_call(['nmcli', 'c', 'u', u])
             else:
-                subprocess.check_call(['nmcli', 'c', 'add', 'type', self.devtypes[iname], 'con-name', cname, 'connection.interface-name', iname] + cargs)
+                subprocess.check_call(['nmcli', 'c', 'add', 'type', ctype, 'con-name', cname, 'connection.interface-name', iname] + cargs)
                 self.read_connections()
                 u = self.uuidbyname.get(cname, None)
                 if u:
@@ -419,6 +467,12 @@ if __name__ == '__main__':
     srvs, _ = apiclient.scan_confluents()
     doneidxs = set([])
     dc = None
+    if not srvs:  # the multicast scan failed, fallback to deploycfg cfg file
+        with open('/etc/confluent/confluent.deploycfg', 'r') as dci:
+            for cfgline in dci.read().split('\n'):
+                if cfgline.startswith('deploy_server:'):
+                    srvs = [cfgline.split()[1]]
+                    break
     for srv in srvs:
         try:
             s = socket.create_connection((srv, 443))
@@ -481,15 +535,30 @@ if __name__ == '__main__':
             netname_to_interfaces['default']['interfaces'] -= netname_to_interfaces[netn]['interfaces']
         if not netname_to_interfaces['default']['interfaces']:
             del netname_to_interfaces['default']
+    # Make sure VLAN/PKEY connections are created last
+    netname_to_interfaces = dict(sorted(netname_to_interfaces.items(), key=lambda item: 'vlan_id' in item[1]['settings']))
     rm_tmp_llas(tmpllas)
     if os.path.exists('/usr/sbin/netplan'):
         nm = NetplanManager(dc)
     if os.path.exists('/usr/bin/nmcli'):
-        nm = NetworkManager(devtypes)
+        nm = NetworkManager(devtypes, dc)
     elif os.path.exists('/usr/sbin/wicked'):
         nm = WickedManager()
+    retrynics = []
     for netn in netname_to_interfaces:
-        nm.apply_configuration(netname_to_interfaces[netn])
+        redo = nm.apply_configuration(netname_to_interfaces[netn])
+        if redo == 1:
+            retrynics.append(netn)
+    if retrynics:
+        idxmap, devtypes = map_idx_to_name()
+        if os.path.exists('/usr/sbin/netplan'):
+            nm = NetplanManager(dc)
+        if os.path.exists('/usr/bin/nmcli'):
+            nm = NetworkManager(devtypes, dc)
+        elif os.path.exists('/usr/sbin/wicked'):
+            nm = WickedManager()
+        for netn in retrynics:
+            nm.apply_configuration(netname_to_interfaces[netn], lastchance=True)
     if havefirewall:
         subprocess.check_call(['systemctl', 'start', 'firewalld'])
     await_tentative()

confluent_osdeploy/common/profile/scripts/setupssh

@@ -3,10 +3,22 @@
 [ -f /opt/confluent/bin/apiclient ] && confapiclient=/opt/confluent/bin/apiclient
 [ -f /etc/confluent/apiclient ] && confapiclient=/etc/confluent/apiclient
 for pubkey in /etc/ssh/ssh_host*key.pub; do
+    if [ "$pubkey" = /etc/ssh/ssh_host_key.pub ]; then
+        continue
+    fi
     certfile=${pubkey/.pub/-cert.pub}
     rm $certfile
     confluentpython $confapiclient /confluent-api/self/sshcert $pubkey -o $certfile
 done
+if [ -d /etc/ssh/sshd_config.d/ -a ! -e /etc/ssh/sshd_config.d/90-confluent.conf ]; then
+	for cert in /etc/ssh/ssh*-cert.pub; do
+		echo HostCertificate $cert >> /etc/ssh/sshd_config.d/90-confluent.conf
+	done
+	echo HostbasedAuthentication yes >> /etc/ssh/sshd_config.d/90-confluent.conf
+	echo HostbasedUsesNameFromPacketOnly yes >> /etc/ssh/sshd_config.d/90-confluent.conf
+	echo IgnoreRhosts no >> /etc/ssh/sshd_config.d/90-confluent.conf
+fi
+
 TMPDIR=$(mktemp -d)
 cd $TMPDIR
 confluentpython $confapiclient /confluent-public/site/initramfs.tgz -o initramfs.tgz

confluent_osdeploy/confluent_osdeploy-aarch64.spec.tmpl

@@ -26,7 +26,7 @@ mkdir -p opt/confluent/bin
 mkdir -p stateless-bin
 cp -a el8bin/* .
 ln -s el8 el9
-for os in rhvh4 el7 genesis el8 suse15 ubuntu20.04 ubuntu22.04 coreos el9; do
+for os in rhvh4 el7 genesis el8 suse15 ubuntu20.04 ubuntu22.04 ubuntu24.04 coreos el9; do
     mkdir ${os}out
     cd ${os}out
     if [ -d ../${os}bin ]; then 
@@ -76,7 +76,7 @@ cp -a esxi7 esxi8
 %install
 mkdir -p %{buildroot}/opt/confluent/share/licenses/confluent_osdeploy/
 #cp LICENSE %{buildroot}/opt/confluent/share/licenses/confluent_osdeploy/
-for os in rhvh4 el7 el8 el9 genesis suse15 ubuntu20.04 ubuntu22.04 esxi6 esxi7 esxi8 coreos; do
+for os in rhvh4 el7 el8 el9 genesis suse15 ubuntu20.04 ubuntu22.04 ubuntu24.04 esxi6 esxi7 esxi8 coreos; do
     mkdir -p %{buildroot}/opt/confluent/lib/osdeploy/$os/initramfs/aarch64/
     cp ${os}out/addons.* %{buildroot}/opt/confluent/lib/osdeploy/$os/initramfs/aarch64/
     if [ -d ${os}disklessout ]; then

confluent_osdeploy/confluent_osdeploy.spec.tmpl

@@ -28,11 +28,15 @@ This contains support utilities for enabling deployment of x86_64 architecture s
 #cp start_root urlmount ../stateless-bin/
 #cd ..
 ln -s el8 el9
-for os in rhvh4 el7 genesis el8 suse15 ubuntu18.04 ubuntu20.04 ubuntu22.04 coreos el9; do
+cp -a el8 el10
+mv el10/initramfs/usr el10/initramfs/var
+for os in rhvh4 el7 genesis el8 suse15 ubuntu18.04 ubuntu20.04 ubuntu22.04 ubuntu24.04 coreos el9 el10; do
     mkdir ${os}out
     cd ${os}out
     if [ -d ../${os}bin ]; then 
         cp -a ../${os}bin/opt .
+    elif [ $os = el10 ]; then
+        cp -a ../el9bin/opt .
     else
         cp -a ../el8bin/opt .
     fi
@@ -42,7 +46,7 @@ for os in rhvh4 el7 genesis el8 suse15 ubuntu18.04 ubuntu20.04 ubuntu22.04 coreo
     mv ../addons.cpio .
     cd ..
 done
-for os in el7 el8 suse15 el9 ubuntu20.04 ubuntu22.04; do
+for os in el7 el8 suse15 el9 ubuntu20.04 ubuntu22.04 ubuntu24.04; do
     mkdir ${os}disklessout
     cd ${os}disklessout
     if [ -d ../${os}bin ]; then
@@ -73,12 +77,14 @@ cd ..
 cp -a esxi7out esxi6out
 cp -a esxi7 esxi6
 cp -a esxi7out esxi8out
+cp -a esxi7out esxi9out
 cp -a esxi7 esxi8
+cp -a esxi7 esxi9
 
 %install
 mkdir -p %{buildroot}/opt/confluent/share/licenses/confluent_osdeploy/
 cp LICENSE %{buildroot}/opt/confluent/share/licenses/confluent_osdeploy/
-for os in rhvh4 el7 el8 el9 genesis suse15 ubuntu20.04 ubuntu18.04 ubuntu22.04 esxi6 esxi7 esxi8 coreos; do
+for os in rhvh4 el7 el8 el9 el10 genesis suse15 ubuntu20.04 ubuntu18.04 ubuntu22.04 ubuntu24.04 esxi6 esxi7 esxi8 esxi9 coreos; do
     mkdir -p %{buildroot}/opt/confluent/lib/osdeploy/$os/initramfs
     mkdir -p %{buildroot}/opt/confluent/lib/osdeploy/$os/profiles
     cp ${os}out/addons.* %{buildroot}/opt/confluent/lib/osdeploy/$os/initramfs

confluent_osdeploy/el7-diskless/profiles/default/scripts/functions

@@ -53,6 +53,7 @@ function set_confluent_vars() {
     if [ -z "$confluent_profile" ]; then
         confluent_profile=$(grep ^profile: /etc/confluent/confluent.deploycfg | sed -e 's/[^ ]*: //')
     fi
+    export confluent_profile confluent_mgr nodename
 }
 
 fetch_remote() {

confluent_osdeploy/el7-diskless/profiles/default/scripts/getinstalldisk

@@ -1,15 +1,19 @@
+#!/usr/bin/python3
 import subprocess
 import os
 
 class DiskInfo(object):
     def __init__(self, devname):
+        if devname.startswith('nvme') and 'c' in devname:
+            raise Exception("Skipping multipath devname")
         self.name = devname
         self.wwn = None
         self.path = None
         self.model = ''
         self.size = 0
-        self.driver = None
+        self.driver = ''
         self.mdcontainer = ''
+        self.subsystype = ''
         devnode = '/dev/{0}'.format(devname)
         qprop = subprocess.check_output(
             ['udevadm', 'info', '--query=property', devnode])
@@ -44,7 +48,9 @@ class DiskInfo(object):
             elif (k == 'DRIVERS' and not self.driver
                     and v not in ('"sd"', '""')):
                 self.driver = v.replace('"', '')
-        if not self.driver and 'imsm' not in self.mdcontainer:
+            elif k == 'ATTRS{subsystype}':
+                self.subsystype = v.replace('"', '')
+        if not self.driver and 'imsm' not in self.mdcontainer and self.subsystype != 'nvm':
             raise Exception("No driver detected")
         if os.path.exists('/sys/block/{0}/size'.format(self.name)):
             with open('/sys/block/{0}/size'.format(self.name), 'r') as sizesrc:

confluent_osdeploy/el7-diskless/profiles/default/scripts/syncfileclient

@@ -1,4 +1,5 @@
 #!/usr/bin/python
+import time
 import importlib
 import tempfile
 import json
@@ -223,6 +224,7 @@ def synchronize():
         if status == 202:
             lastrsp = ''
             while status != 204:
+                time.sleep(2)
                 status, rsp = ac.grab_url_with_status('/confluent-api/self/remotesyncfiles')
                 if not isinstance(rsp, str):
                     rsp = rsp.decode('utf8')

confluent_osdeploy/el7/profiles/default/scripts/functions

@@ -53,6 +53,7 @@ function set_confluent_vars() {
     if [ -z "$confluent_profile" ]; then
         confluent_profile=$(grep ^profile: /etc/confluent/confluent.deploycfg | sed -e 's/[^ ]*: //')
     fi
+    export confluent_profile confluent_mgr nodename
 }
 
 fetch_remote() {

confluent_osdeploy/el7/profiles/default/scripts/getinstalldisk

@@ -1,15 +1,19 @@
+#!/usr/bin/python3
 import subprocess
 import os
 
 class DiskInfo(object):
     def __init__(self, devname):
+        if devname.startswith('nvme') and 'c' in devname:
+            raise Exception("Skipping multipath devname")
         self.name = devname
         self.wwn = None
         self.path = None
         self.model = ''
         self.size = 0
-        self.driver = None
+        self.driver = ''
         self.mdcontainer = ''
+        self.subsystype = ''
         devnode = '/dev/{0}'.format(devname)
         qprop = subprocess.check_output(
             ['udevadm', 'info', '--query=property', devnode])
@@ -44,7 +48,9 @@ class DiskInfo(object):
             elif (k == 'DRIVERS' and not self.driver
                     and v not in ('"sd"', '""')):
                 self.driver = v.replace('"', '')
-        if not self.driver and 'imsm' not in self.mdcontainer:
+            elif k == 'ATTRS{subsystype}':
+                self.subsystype = v.replace('"', '')
+        if not self.driver and 'imsm' not in self.mdcontainer and self.subsystype != 'nvm':
             raise Exception("No driver detected")
         if os.path.exists('/sys/block/{0}/size'.format(self.name)):
             with open('/sys/block/{0}/size'.format(self.name), 'r') as sizesrc:

confluent_osdeploy/el7/profiles/default/scripts/syncfileclient

@@ -5,6 +5,7 @@ import json
 import os
 import shutil
 import pwd
+import time
 import grp
 try:
     from importlib.machinery import SourceFileLoader
@@ -223,6 +224,7 @@ def synchronize():
         if status == 202:
             lastrsp = ''
             while status != 204:
+                time.sleep(2)
                 status, rsp = ac.grab_url_with_status('/confluent-api/self/remotesyncfiles')
                 if not isinstance(rsp, str):
                     rsp = rsp.decode('utf8')

confluent_osdeploy/el8-diskless/initramfs/usr/lib/dracut/hooks/cmdline/10-confluentdiskless.sh

@@ -155,7 +155,7 @@ fi
 ready=0
 while [ $ready = "0" ]; do
     get_remote_apikey
-    if [[ $confluent_mgr == *:* ]]; then
+    if [[ $confluent_mgr == *:* ]] && [[ $confluent_mgr != "["* ]]; then
         confluent_mgr="[$confluent_mgr]"
     fi
     tmperr=$(mktemp)
@@ -324,7 +324,7 @@ fi
 echo '[proxy]' >> /run/NetworkManager/system-connections/$ifname.nmconnection
 chmod 600 /run/NetworkManager/system-connections/*.nmconnection
 confluent_websrv=$confluent_mgr
-if [[ $confluent_websrv == *:* ]]; then
+if [[ $confluent_websrv == *:* ]] && [[ $confluent_websrv != "["* ]]; then
     confluent_websrv="[$confluent_websrv]"
 fi
 echo -n "Initializing ssh..."

confluent_osdeploy/el8-diskless/profiles/default/scripts/functions

@@ -53,6 +53,7 @@ function set_confluent_vars() {
     if [ -z "$confluent_profile" ]; then
         confluent_profile=$(grep ^profile: /etc/confluent/confluent.deploycfg | sed -e 's/[^ ]*: //')
     fi
+    export confluent_profile confluent_mgr nodename
 }
 
 fetch_remote() {

confluent_osdeploy/el8-diskless/profiles/default/scripts/getinstalldisk

@@ -1,15 +1,19 @@
+#!/usr/bin/python3
 import subprocess
 import os
 
 class DiskInfo(object):
     def __init__(self, devname):
+        if devname.startswith('nvme') and 'c' in devname:
+            raise Exception("Skipping multipath devname")
         self.name = devname
         self.wwn = None
         self.path = None
         self.model = ''
         self.size = 0
-        self.driver = None
+        self.driver = ''
         self.mdcontainer = ''
+        self.subsystype = ''
         devnode = '/dev/{0}'.format(devname)
         qprop = subprocess.check_output(
             ['udevadm', 'info', '--query=property', devnode])
@@ -44,7 +48,9 @@ class DiskInfo(object):
             elif (k == 'DRIVERS' and not self.driver
                     and v not in ('"sd"', '""')):
                 self.driver = v.replace('"', '')
-        if not self.driver and 'imsm' not in self.mdcontainer:
+            elif k == 'ATTRS{subsystype}':
+                self.subsystype = v.replace('"', '')
+        if not self.driver and 'imsm' not in self.mdcontainer and self.subsystype != 'nvm':
             raise Exception("No driver detected")
         if os.path.exists('/sys/block/{0}/size'.format(self.name)):
             with open('/sys/block/{0}/size'.format(self.name), 'r') as sizesrc:

confluent_osdeploy/el8-diskless/profiles/default/scripts/image2disk.py

@@ -10,6 +10,7 @@ import stat
 import struct
 import sys
 import subprocess
+import traceback
 
 bootuuid = None
 
@@ -426,4 +427,9 @@ def install_to_disk(imgpath):
 
 
 if __name__ == '__main__':
-    install_to_disk(os.environ['mountsrc'])
+    try:
+        install_to_disk(os.environ['mountsrc'])
+    except Exception:
+        traceback.print_exc()
+        time.sleep(86400)
+        raise

confluent_osdeploy/el8-diskless/profiles/default/scripts/imageboot.sh

@@ -1,6 +1,6 @@
 . /lib/dracut-lib.sh
 confluent_whost=$confluent_mgr
-if [[ "$confluent_whost" == *:* ]]; then
+if [[ "$confluent_whost" == *:* ]] && [[ "$confluent_whost" != "["* ]]; then
     confluent_whost="[$confluent_mgr]"
 fi
 mkdir -p /mnt/remoteimg /mnt/remote /mnt/overlay

confluent_osdeploy/el8-diskless/profiles/default/scripts/onboot.sh

@@ -16,6 +16,7 @@ if [ -z "$confluent_mgr" ]; then
 fi
 confluent_profile=$(grep ^profile: /etc/confluent/confluent.deploycfg|awk '{print $2}')
 timedatectl set-timezone $(grep ^timezone: /etc/confluent/confluent.deploycfg|awk '{print $2}')
+hostnamectl set-hostname $nodename
 export nodename confluent_mgr confluent_profile
 . /etc/confluent/functions
 mkdir -p /var/log/confluent

confluent_osdeploy/el8-diskless/profiles/default/scripts/syncfileclient

@@ -1,4 +1,6 @@
 #!/usr/bin/python3
+import random
+import time
 import subprocess
 import importlib
 import tempfile
@@ -7,6 +9,7 @@ import os
 import shutil
 import pwd
 import grp
+import sys
 from importlib.machinery import SourceFileLoader
 try:
     apiclient = SourceFileLoader('apiclient', '/opt/confluent/bin/apiclient').load_module()
@@ -227,9 +230,16 @@ def synchronize():
                 myips.append(addr)
         data = json.dumps({'merge': tmpdir, 'appendonce': appendoncedir, 'myips': myips})
         status, rsp = ac.grab_url_with_status('/confluent-api/self/remotesyncfiles', data)
+        if status >= 300:
+            sys.stderr.write("Error starting syncfiles - {}:\n".format(status))
+            sys.stderr.write(rsp.decode('utf8'))
+            sys.stderr.write('\n')
+            sys.stderr.flush()
+            return status
         if status == 202:
             lastrsp = ''
             while status != 204:
+                time.sleep(1+(2*random.random()))
                 status, rsp = ac.grab_url_with_status('/confluent-api/self/remotesyncfiles')
                 if not isinstance(rsp, str):
                     rsp = rsp.decode('utf8')
@@ -277,10 +287,21 @@ def synchronize():
                             os.chmod(fname, int(opts[fname][opt], 8))
                     if uid != -1 or gid != -1:
                         os.chown(fname, uid, gid)
+        return status
     finally:
         shutil.rmtree(tmpdir)
         shutil.rmtree(appendoncedir)
 
 
 if __name__ == '__main__':
-    synchronize()
+    status = 202
+    while status not in (204, 200):
+        try:
+            status = synchronize()
+        except Exception as e:
+            sys.stderr.write(str(e))
+            sys.stderr.write('\n')
+            sys.stderr.flush()
+            status = 300
+        if status not in (204, 200):
+            time.sleep((random.random()*3)+2)

confluent_osdeploy/el8/initramfs/usr/lib/dracut/hooks/cmdline/01-confluent.sh

@@ -1,14 +1,20 @@
 #!/bin/bash
 echo -n "" >> /tmp/net.ifaces
 echo -n "" > /tmp/01-autocons.devnode
-cat /tls/*.0 >> /etc/pki/tls/certs/ca-bundle.crt
+BUNDLENAME=/etc/pki/tls/certs/ca-bundle.crt
+if [ ! -e "$BUNDLENAME" ]; then
+    BUNDLENAME=/etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem
+    mkdir -p /etc/pki/tls/certs
+    ln -s $BUNDLENAME /etc/pki/tls/certs/ca-bundle.crt
+fi
+cat /tls/*.0 >> $BUNDLENAME
 if ! grep console= /proc/cmdline >& /dev/null; then
     autocons=$(/opt/confluent/bin/autocons)
     if [ -n "$autocons" ]; then
         echo console=$autocons |sed -e 's!/dev/!!' >> /tmp/01-autocons.conf
         autocons=${autocons%,*}
-	echo $autocons > /tmp/01-autocons.devnode
-	echo "Detected firmware specified console at $(cat /tmp/01-autocons.conf)" > $autocons
+        echo $autocons > /tmp/01-autocons.devnode
+        echo "Detected firmware specified console at $(cat /tmp/01-autocons.conf)" > $autocons
         echo "Initializing auto detected console when installer starts" > $autocons
     fi
 fi
@@ -16,4 +22,5 @@ if grep console=ttyS /proc/cmdline >& /dev/null; then
     echo "Serial console has been requested in the kernel arguments, the local video may not show progress" > /dev/tty1
 fi
 . /lib/anaconda-lib.sh
+echo rd.fcoe=0 > /etc/cmdline.d/nofcoe.conf
 wait_for_kickstart

confluent_osdeploy/el8/initramfs/usr/lib/dracut/hooks/pre-pivot/01-confluent.sh

@@ -1,5 +1,8 @@
 #!/bin/bash
 BUNDLENAME=/sysroot/etc/pki/tls/certs/ca-bundle.crt
+if [ ! -e "$BUNDLENAME" ]; then
+    BUNDLENAME=/sysroot/etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem
+fi
 while [ -h $BUNDLENAME ]; do
     BUNDLENAME=/sysroot/$(readlink $BUNDLENAME)
 done

confluent_osdeploy/el8/initramfs/usr/lib/dracut/hooks/pre-trigger/01-confluent.sh

@@ -2,6 +2,15 @@
 [ -e /tmp/confluent.initq ] && return 0
 . /lib/dracut-lib.sh
 setsid sh -c 'exec bash <> /dev/tty2 >&0 2>&1' &
+if [ -f /tmp/dd_disk ]; then
+    for dd in $(cat /tmp/dd_disk); do
+        if [ -e $dd ]; then
+            driver-updates --disk $dd $dd
+	    rm $dd
+        fi
+    done
+    rm /tmp/dd_disk
+fi
 udevadm trigger
 udevadm trigger --type=devices --action=add
 udevadm settle
@@ -20,13 +29,6 @@ function confluentpython() {
         /usr/bin/python2 $*
     fi
 }
-if [ -f /tmp/dd_disk ]; then
-    for dd in $(cat /tmp/dd_disk); do
-        if [ -e $dd ]; then
-            driver-updates --disk $dd $dd
-        fi
-    done
-fi
 vlaninfo=$(getarg vlan)
 if [ ! -z "$vlaninfo" ]; then
         vldev=${vlaninfo#*:}
@@ -61,43 +63,52 @@ if [ -e /dev/disk/by-label/CNFLNT_IDNT ]; then
         udevadm info $i | grep ID_NET_DRIVER=cdc_ether > /dev/null &&  continue
         ip link set $(basename $i) up
     done
-    for NICGUESS in $(ip link|grep LOWER_UP|grep -v LOOPBACK| awk '{print $2}' | sed -e 's/:$//'); do
-        if [ "$autoconfigmethod" = "dhcp" ]; then
-            /usr/libexec/nm-initrd-generator ip=$NICGUESS:dhcp
-        else
-            v4addr=$(grep ^ipv4_address: $tcfg)
-            v4addr=${v4addr#ipv4_address: }
-            v4plen=${v4addr#*/}
-            v4addr=${v4addr%/*}
-            v4gw=$(grep ^ipv4_gateway: $tcfg)
-            v4gw=${v4gw#ipv4_gateway: }
-            ip addr add dev $NICGUESS $v4addr/$v4plen
-            if [ "$v4gw" = "null" ]; then
-                v4gw=""
-            fi
-            if [ ! -z "$v4gw" ]; then
-                ip route add default via $v4gw
-            fi
-            v4nm=$(grep ipv4_netmask: $tcfg)
-            v4nm=${v4nm#ipv4_netmask: }
-            DETECTED=0
-            for dsrv in $deploysrvs; do
-                if curl --capath /tls/ -s --connect-timeout 3 https://$dsrv/confluent-public/ > /dev/null; then
-                    rm /run/NetworkManager/system-connections/*
-                    /usr/libexec/nm-initrd-generator ip=$v4addr::$v4gw:$v4nm:$hostname:$NICGUESS:none
-                    DETECTED=1
-                    ifname=$NICGUESS
+    TRIES=30
+    DETECTED=0
+    while [ "$DETECTED" = 0 ] && [ $TRIES -gt 0 ]; do
+        TRIES=$((TRIES - 1))
+        for NICGUESS in $(ip link|grep LOWER_UP|grep -v LOOPBACK| awk '{print $2}' | sed -e 's/:$//'); do
+            if [ "$autoconfigmethod" = "dhcp" ]; then
+                /usr/libexec/nm-initrd-generator ip=$NICGUESS:dhcp
+            else
+                v4addr=$(grep ^ipv4_address: $tcfg)
+                v4addr=${v4addr#ipv4_address: }
+                v4plen=${v4addr#*/}
+                v4addr=${v4addr%/*}
+                v4gw=$(grep ^ipv4_gateway: $tcfg)
+                v4gw=${v4gw#ipv4_gateway: }
+                ip addr add dev $NICGUESS $v4addr/$v4plen
+                if [ "$v4gw" = "null" ]; then
+                    v4gw=""
+                fi
+                if [ ! -z "$v4gw" ]; then
+                    ip route add default via $v4gw
+                fi
+                v4nm=$(grep ipv4_netmask: $tcfg)
+                v4nm=${v4nm#ipv4_netmask: }
+                DETECTED=0
+                for dsrv in $deploysrvs; do
+                    if curl --capath /tls/ -s --connect-timeout 3 https://$dsrv/confluent-public/ > /dev/null; then
+                        rm /run/NetworkManager/system-connections/*
+                        /usr/libexec/nm-initrd-generator ip=$v4addr::$v4gw:$v4nm:$hostname:$NICGUESS:none
+                        DETECTED=1
+                        ifname=$NICGUESS
+                        break
+                    fi
+                done
+                if [ ! -z "$v4gw" ]; then
+                    ip route del default via $v4gw
+                fi
+                ip addr flush dev $NICGUESS
+                if [ $DETECTED = 1 ]; then
                     break
                 fi
-            done
-            if [ ! -z "$v4gw" ]; then
-                ip route del default via $v4gw
             fi
-            ip addr flush dev $NICGUESS
-            if [ $DETECTED = 1 ]; then
-                break
-            fi
-        fi
+        done
+    done
+    for NICGUESS in $(ip link|grep LOWER_UP|grep -v LOOPBACK| awk '{print $2}' | sed -e 's/:$//'); do
+        ip addr flush dev $NICGUESS
+        ip link set $NICGUESS down
     done
     NetworkManager --configure-and-quit=initrd --no-daemon
     hmackeyfile=/tmp/cnflnthmackeytmp
@@ -175,7 +186,7 @@ if [ ! -z "$autocons" ]; then
     errout="-e $autocons"
 fi
 while ! confluentpython /opt/confluent/bin/apiclient $errout /confluent-api/self/deploycfg2 > /etc/confluent/confluent.deploycfg; do
-	sleep 10
+        sleep 10
 done
 ifidx=$(cat /tmp/confluent.ifidx 2> /dev/null)
 if [ -z "$ifname" ]; then
@@ -216,23 +227,38 @@ proto=${proto#protocol: }
 textconsole=$(grep ^textconsole: /etc/confluent/confluent.deploycfg)
 textconsole=${textconsole#textconsole: }
 if [ "$textconsole" = "true" ] && ! grep console= /proc/cmdline > /dev/null; then
-	autocons=$(cat /tmp/01-autocons.devnode)
-	if [ ! -z "$autocons" ]; then
-	    echo Auto-configuring installed system to use text console
-	    echo Auto-configuring installed system to use text console > $autocons
+        autocons=$(cat /tmp/01-autocons.devnode)
+        if [ ! -z "$autocons" ]; then
+            echo Auto-configuring installed system to use text console
+            echo Auto-configuring installed system to use text console > $autocons
             /opt/confluent/bin/autocons -c > /dev/null
-	    cp /tmp/01-autocons.conf /etc/cmdline.d/
-	else
-	    echo "Unable to automatically detect requested text console"
-	fi
+            cp /tmp/01-autocons.conf /etc/cmdline.d/
+        else
+            echo "Unable to automatically detect requested text console"
+        fi
 fi
 
-echo inst.repo=$proto://$mgr/confluent-public/os/$profilename/distribution >> /etc/cmdline.d/01-confluent.conf
+. /etc/os-release
+if [ "$ID" = "dracut" ]; then
+    ID=$(echo $PRETTY_NAME|awk '{print $1}')
+    VERSION_ID=$(echo $VERSION|awk '{print $1}')
+    if [ "$ID" = "Oracle" ]; then
+        ID=OL
+    elif [ "$ID" = "Red" ]; then
+        ID=RHEL
+    fi
+fi
+ISOSRC=$(blkid -t TYPE=iso9660|grep -Ei ' LABEL="'$ID-$VERSION_ID|sed -e s/:.*//)
+if [ -z "$ISOSRC" ]; then
+    echo inst.repo=$proto://$mgr/confluent-public/os/$profilename/distribution >> /etc/cmdline.d/01-confluent.conf
+    root=anaconda-net:$proto://$mgr/confluent-public/os/$profilename/distribution
+    export root
+else
+    echo inst.repo=cdrom:$ISOSRC >> /etc/cmdline.d/01-confluent.conf
+fi
 echo inst.ks=$proto://$mgr/confluent-public/os/$profilename/kickstart >> /etc/cmdline.d/01-confluent.conf
 kickstart=$proto://$mgr/confluent-public/os/$profilename/kickstart
-root=anaconda-net:$proto://$mgr/confluent-public/os/$profilename/distribution
 export kickstart
-export root
 autoconfigmethod=$(grep ipv4_method /etc/confluent/confluent.deploycfg)
 autoconfigmethod=${autoconfigmethod#ipv4_method: }
 if [ "$autoconfigmethod" = "dhcp" ]; then
@@ -312,4 +338,8 @@ if [ -e /lib/nm-lib.sh ]; then
         fi
     fi
 fi
+for NICGUESS in $(ip link|grep LOWER_UP|grep -v LOOPBACK| awk '{print $2}' | sed -e 's/:$//'); do
+    ip addr flush dev $NICGUESS
+    ip link set $NICGUESS down
+done
 

confluent_osdeploy/el8/profiles/default/initprofile.sh

@@ -1,5 +1,8 @@
 #!/bin/sh
-sed -i 's/centos/CentOS/; s/rhel/Red Hat Enterprise Linux/; s/oraclelinux/Oracle Linux/; s/alma/AlmaLinux/' $2/profile.yaml
+sed -i 's/centos/CentOS/; s/rhel/Red Hat Enterprise Linux/; s/oraclelinux/Oracle Linux/; s/alma/AlmaLinux/;s/fedora/Fedora Linux/' $2/profile.yaml
+if grep Fedora $2/profile.yaml > /dev/null; then
+	sed -i 's/@^minimal-environment/#/' $2/packagelist
+fi
 ln -s $1/images/pxeboot/vmlinuz $2/boot/kernel && \
 ln -s $1/images/pxeboot/initrd.img $2/boot/initramfs/distribution
 mkdir -p $2/boot/efi/boot

confluent_osdeploy/el8/profiles/default/scripts/add_local_repositories

@@ -6,6 +6,7 @@ except ImportError:
 import importlib.util
 import importlib.machinery
 import sys
+import glob
 modloader = importlib.machinery.SourceFileLoader('apiclient', '/opt/confluent/bin/apiclient')
 modspec = importlib.util.spec_from_file_location('apiclient', '/opt/confluent/bin/apiclient', loader=modloader)
 apiclient = importlib.util.module_from_spec(modspec)
@@ -27,7 +28,7 @@ with open('/etc/confluent/confluent.deploycfg') as dplcfgfile:
            _, profile = line.split(' ', 1)
         if line.startswith('ipv4_method: '):
            _, v4cfg = line.split(' ', 1)
-if v4cfg == 'static' or v4cfg =='dhcp':
+if v4cfg == 'static' or v4cfg =='dhcp' or not server6:
     server = server4
 if not server:
     server = '[{}]'.format(server6)
@@ -41,6 +42,7 @@ try:
 except AttributeError:
     f = cStringIO.StringIO(cfgdata)
     c.readfp(f)
+gpgkeys = glob.glob('/etc/pki/rpm-gpg/RPM-GPG-KEY-*')
 for sec in c.sections():
     if sec.startswith('variant-'):
         try:
@@ -56,3 +58,6 @@ for sec in c.sections():
                 repopath = repopath[1:]
             repout.write('baseurl=https://{}/confluent-public/os/{}/distribution/{}\n'.format(server, profile, repopath))
             repout.write('enabled=1\n')
+            if gpgkeys:
+                gpgkeyvals = ['file://{}'.format(x) for x in gpgkeys]
+                repout.write('gpgkey=' + ' '.join(gpgkeyvals) + '\n')

confluent_osdeploy/el8/profiles/default/scripts/functions

@@ -53,6 +53,7 @@ function set_confluent_vars() {
     if [ -z "$confluent_profile" ]; then
         confluent_profile=$(grep ^profile: /etc/confluent/confluent.deploycfg | sed -e 's/[^ ]*: //')
     fi
+    export confluent_profile confluent_mgr nodename
 }
 
 fetch_remote() {

confluent_osdeploy/el8/profiles/default/scripts/getinstalldisk

@@ -1,15 +1,19 @@
+#!/usr/bin/python3
 import subprocess
 import os
 
 class DiskInfo(object):
     def __init__(self, devname):
+        if devname.startswith('nvme') and 'c' in devname:
+            raise Exception("Skipping multipath devname")
         self.name = devname
         self.wwn = None
         self.path = None
         self.model = ''
         self.size = 0
-        self.driver = None
+        self.driver = ''
         self.mdcontainer = ''
+        self.subsystype = ''
         devnode = '/dev/{0}'.format(devname)
         qprop = subprocess.check_output(
             ['udevadm', 'info', '--query=property', devnode])
@@ -44,7 +48,9 @@ class DiskInfo(object):
             elif (k == 'DRIVERS' and not self.driver
                     and v not in ('"sd"', '""')):
                 self.driver = v.replace('"', '')
-        if not self.driver and 'imsm' not in self.mdcontainer:
+            elif k == 'ATTRS{subsystype}':
+                self.subsystype = v.replace('"', '')
+        if not self.driver and 'imsm' not in self.mdcontainer and self.subsystype != 'nvm':
             raise Exception("No driver detected")
         if os.path.exists('/sys/block/{0}/size'.format(self.name)):
             with open('/sys/block/{0}/size'.format(self.name), 'r') as sizesrc:

confluent_osdeploy/el8/profiles/default/scripts/post.sh

@@ -34,6 +34,11 @@ run_remote_python add_local_repositories
 # run_remote_python will use the appropriate python interpreter path to run the specified script
 # A post.custom is provided to more conveniently hold customizations, see the post.custom file.
 
+# This will do some serial console fixup for bad grub configuration when serial is firmware
+# managed. See script for details
+run_remote_python autoconsole
+
+
 # This will induce server side processing of the syncfile contents if
 # present
 run_remote_python syncfileclient

confluent_osdeploy/el8/profiles/default/scripts/pre.sh

@@ -90,8 +90,14 @@ touch /tmp/cryptpkglist
 touch /tmp/pkglist
 touch /tmp/addonpackages
 if [ "$cryptboot" == "tpm2" ]; then
-	LUKSPARTY="--encrypted --passphrase=$(cat /etc/confluent/confluent.apikey)"
-	echo $cryptboot >> /tmp/cryptboot
+    lukspass=$(python3 /opt/confluent/bin/apiclient /confluent-api/self/profileprivate/pending/luks.key 2> /dev/null)
+     if [ -z "$lukspass" ]; then
+        lukspass=$(python3 -c 'import os;import base64;print(base64.b64encode(os.urandom(66)).decode())')
+    fi
+    echo $lukspass > /etc/confluent/luks.key
+    chmod 000 /etc/confluent/luks.key
+    LUKSPARTY="--encrypted --passphrase=$lukspass"
+    echo $cryptboot >> /tmp/cryptboot
     echo clevis-dracut >> /tmp/cryptpkglist
 fi
 
@@ -114,8 +120,8 @@ confluentpython /etc/confluent/apiclient /confluent-public/os/$confluent_profile
 grep '^%include /tmp/partitioning' /tmp/kickstart.* > /dev/null || rm /tmp/installdisk
 if [ -e /tmp/installdisk -a ! -e /tmp/partitioning ]; then
     INSTALLDISK=$(cat /tmp/installdisk)
-    sed -e s/%%INSTALLDISK%%/$INSTALLDISK/ -e s/%%LUKSHOOK%%/$LUKSPARTY/ /tmp/partitioning.template > /tmp/partitioning
-    dd if=/dev/zero of=/dev/$(cat /tmp/installdisk) bs=1M count=1 >& /dev/null
+    sed -e s/%%INSTALLDISK%%/$INSTALLDISK/ -e "s!%%LUKSHOOK%%!$LUKSPARTY!" /tmp/partitioning.template > /tmp/partitioning
     vgchange -a n >& /dev/null
+    wipefs -a -f /dev/$INSTALLDISK >& /dev/null
 fi
 kill $logshowpid

confluent_osdeploy/el8/profiles/default/scripts/setupssh.sh

@@ -1,8 +1,12 @@
 #!/bin/sh
-grep HostCert /etc/ssh/sshd_config.anaconda >> /mnt/sysimage/etc/ssh/sshd_config
-echo HostbasedAuthentication yes >> /mnt/sysimage/etc/ssh/sshd_config
-echo HostbasedUsesNameFromPacketOnly yes >> /mnt/sysimage/etc/ssh/sshd_config
-echo IgnoreRhosts no >> /mnt/sysimage/etc/ssh/sshd_config
+targssh=/mnt/sysimage/etc/ssh/sshd_config
+if [ -d /mnt/sysimage/etc/ssh/sshd_config.d/ ]; then
+	targssh=/mnt/sysimage/etc/ssh/sshd_config.d/90-confluent.conf
+fi
+grep HostCert /etc/ssh/sshd_config.anaconda >> $targssh
+echo HostbasedAuthentication yes >> $targssh
+echo HostbasedUsesNameFromPacketOnly yes >> $targssh
+echo IgnoreRhosts no >> $targssh
 sshconf=/mnt/sysimage/etc/ssh/ssh_config
 if [ -d /mnt/sysimage/etc/ssh/ssh_config.d/ ]; then
     sshconf=/mnt/sysimage/etc/ssh/ssh_config.d/01-confluent.conf

confluent_osdeploy/el8/profiles/default/scripts/syncfileclient

@@ -1,4 +1,6 @@
 #!/usr/bin/python3
+import random
+import time
 import subprocess
 import importlib
 import tempfile
@@ -7,6 +9,7 @@ import os
 import shutil
 import pwd
 import grp
+import sys
 from importlib.machinery import SourceFileLoader
 try:
     apiclient = SourceFileLoader('apiclient', '/opt/confluent/bin/apiclient').load_module()
@@ -227,9 +230,16 @@ def synchronize():
                 myips.append(addr)
         data = json.dumps({'merge': tmpdir, 'appendonce': appendoncedir, 'myips': myips})
         status, rsp = ac.grab_url_with_status('/confluent-api/self/remotesyncfiles', data)
+        if status >= 300:
+            sys.stderr.write("Error starting syncfiles - {}:\n".format(status))
+            sys.stderr.write(rsp.decode('utf8'))
+            sys.stderr.write('\n')
+            sys.stderr.flush()
+            return status
         if status == 202:
             lastrsp = ''
             while status != 204:
+                time.sleep(1+(2*random.random()))
                 status, rsp = ac.grab_url_with_status('/confluent-api/self/remotesyncfiles')
                 if not isinstance(rsp, str):
                     rsp = rsp.decode('utf8')
@@ -277,10 +287,21 @@ def synchronize():
                             os.chmod(fname, int(opts[fname][opt], 8))
                     if uid != -1 or gid != -1:
                         os.chown(fname, uid, gid)
+        return status
     finally:
         shutil.rmtree(tmpdir)
         shutil.rmtree(appendoncedir)
 
 
 if __name__ == '__main__':
-    synchronize()
+    status = 202
+    while status not in (204, 200):
+        try:
+            status = synchronize()
+        except Exception as e:
+            sys.stderr.write(str(e))
+            sys.stderr.write('\n')
+            sys.stderr.flush()
+            status = 300
+        if status not in (204, 200):
+            time.sleep((random.random()*3)+2)

confluent_osdeploy/el8/profiles/default/scripts/tpm_luks.sh

@@ -1,4 +1,5 @@
 #!/bin/sh
 cryptdisk=$(blkid -t TYPE="crypto_LUKS"|sed -e s/:.*//)
-clevis luks bind -f -d $cryptdisk -k - tpm2 '{}' < /etc/confluent/confluent.apikey
-cryptsetup luksRemoveKey $cryptdisk < /etc/confluent/confluent.apikey
+clevis luks bind -f -d $cryptdisk -k - tpm2 '{}' < /etc/confluent/luks.key
+chmod 000 /etc/confluent/luks.key
+#cryptsetup luksRemoveKey $cryptdisk < /etc/confluent/confluent.apikey

confluent_osdeploy/el9-diskless/initramfs/usr/lib/dracut/hooks/cmdline/10-confluentdiskless.sh

@@ -120,7 +120,7 @@ fi
 ready=0
 while [ $ready = "0" ]; do
     get_remote_apikey
-    if [[ $confluent_mgr == *:* ]]; then
+    if [[ $confluent_mgr == *:* ]] && [[ $confluent_mgr != "["* ]]; then
         confluent_mgr="[$confluent_mgr]"
     fi
     tmperr=$(mktemp)
@@ -171,6 +171,13 @@ permissions=
 wait-device-timeout=60000
 
 EOC
+if [ "$linktype" = infiniband ]; then
+cat >> /run/NetworkManager/system-connections/$ifname.nmconnection << EOC
+[infiniband]
+transport-mode=datagram
+
+EOC
+fi
 autoconfigmethod=$(grep ^ipv4_method: /etc/confluent/confluent.deploycfg |awk '{print $2}')
 auto6configmethod=$(grep ^ipv6_method: /etc/confluent/confluent.deploycfg |awk '{print $2}')
 if [ "$autoconfigmethod" = "dhcp" ]; then
@@ -281,11 +288,11 @@ fi
 echo '[proxy]' >> /run/NetworkManager/system-connections/$ifname.nmconnection
 chmod 600 /run/NetworkManager/system-connections/*.nmconnection
 confluent_websrv=$confluent_mgr
-if [[ $confluent_websrv == *:* ]]; then
+if [[ $confluent_websrv == *:* ]] && [[ $confluent_websrv != "["* ]]; then
     confluent_websrv="[$confluent_websrv]"
 fi
 echo -n "Initializing ssh..."
-ssh-keygen -A
+ssh-keygen -t ed25519 -f /etc/ssh/ssh_host_ed25519_key -C '' -N ''
 for pubkey in /etc/ssh/ssh_host*key.pub; do
     certfile=${pubkey/.pub/-cert.pub}
     privfile=${pubkey%.pub}

confluent_osdeploy/el9-diskless/profiles/default/scripts/firstboot.sh

@@ -9,9 +9,16 @@ HOME=$(getent passwd $(whoami)|cut -d: -f 6)
 export HOME
 nodename=$(grep ^NODENAME /etc/confluent/confluent.info|awk '{print $2}')
 confluent_apikey=$(cat /etc/confluent/confluent.apikey)
-confluent_mgr=$(grep ^deploy_server: /etc/confluent/confluent.deploycfg|awk '{print $2}')
+confluent_mgr=$(grep ^deploy_server_v6: /etc/confluent/confluent.deploycfg|awk '{print $2}')
+if [ -z "$confluent_mgr" ] || [ "$confluent_mgr" == "null" ] || ! ping -c 1 $confluent_mgr >& /dev/null; then
+    confluent_mgr=$(grep ^deploy_server: /etc/confluent/confluent.deploycfg|awk '{print $2}')
+fi
+confluent_websrv=$confluent_mgr
+if [[ "$confluent_mgr" == *:* ]]; then
+    confluent_websrv="[$confluent_mgr]"
+fi
 confluent_profile=$(grep ^profile: /etc/confluent/confluent.deploycfg|awk '{print $2}')
-export nodename confluent_mgr confluent_profile
+export nodename confluent_mgr confluent_profile confluent_websrv
 . /etc/confluent/functions
 (
 exec >> /var/log/confluent/confluent-firstboot.log
@@ -34,7 +41,7 @@ if [ ! -f /etc/confluent/firstboot.ran ]; then
     run_remote_config firstboot.d
 fi
 
-curl -X POST -d 'status: complete' -H "CONFLUENT_NODENAME: $nodename" -H "CONFLUENT_APIKEY: $confluent_apikey" https://$confluent_mgr/confluent-api/self/updatestatus
+curl -X POST -d 'status: complete' -H "CONFLUENT_NODENAME: $nodename" -H "CONFLUENT_APIKEY: $confluent_apikey" https://$confluent_websrv/confluent-api/self/updatestatus
 systemctl disable firstboot
 rm /etc/systemd/system/firstboot.service
 rm /etc/confluent/firstboot.ran

confluent_osdeploy/el9-diskless/profiles/default/scripts/functions

@@ -53,6 +53,7 @@ function set_confluent_vars() {
     if [ -z "$confluent_profile" ]; then
         confluent_profile=$(grep ^profile: /etc/confluent/confluent.deploycfg | sed -e 's/[^ ]*: //')
     fi
+    export confluent_profile confluent_mgr nodename
 }
 
 fetch_remote() {

confluent_osdeploy/el9-diskless/profiles/default/scripts/getinstalldisk

@@ -1,15 +1,19 @@
+#!/usr/bin/python3
 import subprocess
 import os
 
 class DiskInfo(object):
     def __init__(self, devname):
+        if devname.startswith('nvme') and 'c' in devname:
+            raise Exception("Skipping multipath devname")
         self.name = devname
         self.wwn = None
         self.path = None
         self.model = ''
         self.size = 0
-        self.driver = None
+        self.driver = ''
         self.mdcontainer = ''
+        self.subsystype = ''
         devnode = '/dev/{0}'.format(devname)
         qprop = subprocess.check_output(
             ['udevadm', 'info', '--query=property', devnode])
@@ -44,7 +48,9 @@ class DiskInfo(object):
             elif (k == 'DRIVERS' and not self.driver
                     and v not in ('"sd"', '""')):
                 self.driver = v.replace('"', '')
-        if not self.driver and 'imsm' not in self.mdcontainer:
+            elif k == 'ATTRS{subsystype}':
+                self.subsystype = v.replace('"', '')
+        if not self.driver and 'imsm' not in self.mdcontainer and self.subsystype != 'nvm':
             raise Exception("No driver detected")
         if os.path.exists('/sys/block/{0}/size'.format(self.name)):
             with open('/sys/block/{0}/size'.format(self.name), 'r') as sizesrc:

confluent_osdeploy/el9-diskless/profiles/default/scripts/image2disk.py

@@ -10,8 +10,16 @@ import stat
 import struct
 import sys
 import subprocess
+import traceback
 
 bootuuid = None
+vgname = 'localstorage'
+oldvgname = None
+
+def convert_lv(oldlvname):
+    if oldvgname is None:
+        return None
+    return oldlvname.replace(oldvgname, vgname)
 
 def get_partname(devname, idx):
     if devname[-1] in '0123456789':
@@ -53,9 +61,15 @@ def get_image_metadata(imgpath):
         header = img.read(16)
         if header == b'\x63\x7b\x9d\x26\xb7\xfd\x48\x30\x89\xf9\x11\xcf\x18\xfd\xff\xa1':
             for md in get_multipart_image_meta(img):
+                if md.get('device', '').startswith('/dev/zram'):
+                    continue
                 yield md
         else:
-            raise Exception('Installation from single part image not supported')
+            # plausible filesystem structure to apply to a nominally "diskless" image
+            yield {'mount': '/', 'filesystem': 'xfs', 'minsize': 39513563136, 'initsize': 954128662528, 'flags': 'rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota', 'device': '/dev/mapper/root', 'compressed_size': 27022069760}
+            yield {'mount': '/boot', 'filesystem': 'xfs', 'minsize': 232316928, 'initsize': 1006632960, 'flags': 'rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota', 'device': '/dev/nvme1n1p2', 'compressed_size': 171462656}
+            yield {'mount': '/boot/efi', 'filesystem': 'vfat', 'minsize': 7835648, 'initsize': 627900416, 'flags': 'rw,relatime,fmask=0077,dmask=0077,codepage=437,iocharset=ascii,shortname=winnt,errors=remount-ro', 'device': '/dev/nvme1n1p1', 'compressed_size': 1576960}
+            #raise Exception('Installation from single part image not supported')
 
 class PartedRunner():
     def __init__(self, disk):
@@ -74,8 +88,17 @@ def fixup(rootdir, vols):
     for vol in vols:
         devbymount[vol['mount']] = vol['targetdisk']
     fstabfile = os.path.join(rootdir, 'etc/fstab')
-    with open(fstabfile) as tfile:
-        fstab = tfile.read().split('\n')
+    if os.path.exists(fstabfile):
+        with open(fstabfile) as tfile:
+            fstab = tfile.read().split('\n')
+    else:
+        #diskless image, need to invent fstab
+        fstab = [
+            "#ORIGFSTAB#/dev/mapper/root# /                       xfs     defaults        0 0",
+            "#ORIGFSTAB#UUID=aaf9e0f9-aa4d-4d74-9e75-3537620cfe23# /boot                   xfs     defaults        0 0",
+            "#ORIGFSTAB#UUID=C21D-B881#          /boot/efi               vfat    umask=0077,shortname=winnt 0 2",
+            "#ORIGFSTAB#/dev/mapper/swap# none                    swap    defaults        0 0",
+        ]
     while not fstab[0]:
         fstab = fstab[1:]
     if os.path.exists(os.path.join(rootdir, '.autorelabel')):
@@ -86,14 +109,14 @@ def fixup(rootdir, vols):
             if tab.startswith('#ORIGFSTAB#'):
                 if entry[1] in devbymount:
                     targetdev = devbymount[entry[1]]
-                    if targetdev.startswith('/dev/localstorage/'):
+                    if targetdev.startswith('/dev/{}/'.format(vgname)):
                         entry[0] = targetdev
                     else:
                         uuid = subprocess.check_output(['blkid', '-s', 'UUID', '-o', 'value', targetdev]).decode('utf8')
                         uuid = uuid.strip()
                         entry[0] = 'UUID={}'.format(uuid)
                 elif entry[2] == 'swap':
-                    entry[0] = '/dev/mapper/localstorage-swap'
+                    entry[0] = '/dev/mapper/{}-swap'.format(vgname.replace('-', '--'))
                 entry[0] = entry[0].ljust(42)
                 entry[1] = entry[1].ljust(16)
                 entry[3] = entry[3].ljust(28)
@@ -125,8 +148,10 @@ def fixup(rootdir, vols):
         newcfg = ifcfg.split('/')[-1]
         newcfg = os.path.join(rootdir, 'etc/NetworkManager/system-connections/{0}'.format(newcfg))
         shutil.copy2(ifcfg, newcfg)
-    shutil.rmtree(os.path.join(rootdir, 'etc/confluent/'))
-    shutil.copytree('/etc/confluent', os.path.join(rootdir, 'etc/confluent'))
+    rootconfluentdir = os.path.join(rootdir, 'etc/confluent/')
+    if os.path.exists(rootconfluentdir):
+        shutil.rmtree(rootconfluentdir)
+    shutil.copytree('/etc/confluent', rootconfluentdir)
     if policy:
         sys.stdout.write('Applying SELinux labeling...')
         sys.stdout.flush()
@@ -141,16 +166,81 @@ def fixup(rootdir, vols):
     grubsyscfg = os.path.join(rootdir, 'etc/sysconfig/grub')
     if not os.path.exists(grubsyscfg):
         grubsyscfg = os.path.join(rootdir, 'etc/default/grub')
-    with open(grubsyscfg) as defgrubin:
-        defgrub = defgrubin.read().split('\n')
+    kcmdline = os.path.join(rootdir, 'etc/kernel/cmdline')
+    if os.path.exists(kcmdline):
+        with open(kcmdline) as kcmdlinein:
+            kcmdlinecontent = kcmdlinein.read()
+        newkcmdlineent = []
+        for ent in kcmdlinecontent.split():
+            if ent.startswith('resume='):
+                newkcmdlineent.append('resume={}'.format(newswapdev))
+            elif ent.startswith('root='):
+                newkcmdlineent.append('root={}'.format(newrootdev))
+            elif ent.startswith('rd.lvm.lv='):
+                ent = convert_lv(ent)
+                if ent:
+                    newkcmdlineent.append(ent)
+            else:
+                newkcmdlineent.append(ent)
+        with open(kcmdline, 'w') as kcmdlineout:
+            kcmdlineout.write(' '.join(newkcmdlineent) + '\n')
+    for loadent in glob.glob(os.path.join(rootdir, 'boot/loader/entries/*.conf')):
+        with open(loadent) as loadentin:
+            currentry = loadentin.read().split('\n')
+        with open(loadent, 'w') as loadentout:
+            for cfgline in currentry:
+                cfgparts = cfgline.split()
+                if not cfgparts or cfgparts[0] != 'options':
+                    loadentout.write(cfgline + '\n')
+                    continue
+                newcfgparts = [cfgparts[0]]
+                for cfgpart in cfgparts[1:]:
+                    if cfgpart.startswith('root='):
+                        newcfgparts.append('root={}'.format(newrootdev))
+                    elif cfgpart.startswith('resume='):
+                        newcfgparts.append('resume={}'.format(newswapdev))
+                    elif cfgpart.startswith('rd.lvm.lv='):
+                        cfgpart = convert_lv(cfgpart)
+                        if cfgpart:
+                            newcfgparts.append(cfgpart)
+                    else:
+                        newcfgparts.append(cfgpart)
+                loadentout.write(' '.join(newcfgparts) + '\n')
+    if os.path.exists(grubsyscfg):
+        with open(grubsyscfg) as defgrubin:
+            defgrub = defgrubin.read().split('\n')
+    else:
+        defgrub = [
+        'GRUB_TIMEOUT=5',
+        'GRUB_DISTRIBUTOR="$(sed ' + "'s, release .*$,,g'" + ' /etc/system-release)"',
+        'GRUB_DEFAULT=saved',
+        'GRUB_DISABLE_SUBMENU=true',
+        'GRUB_TERMINAL=""',
+        'GRUB_SERIAL_COMMAND=""',
+        'GRUB_CMDLINE_LINUX="crashkernel=1G-4G:192M,4G-64G:256M,64G-:512M rd.lvm.lv=vg/root rd.lvm.lv=vg/swap"',
+        'GRUB_DISABLE_RECOVERY="true"',
+        'GRUB_ENABLE_BLSCFG=true',
+        ]
+    if not os.path.exists(os.path.join(rootdir, "etc/kernel/cmdline")):
+        with open(os.path.join(rootdir, "etc/kernel/cmdline"), "w") as cmdlineout:
+            cmdlineout.write("root=/dev/mapper/localstorage-root rd.lvm.lv=localstorage/root")
     with open(grubsyscfg, 'w') as defgrubout:
         for gline in defgrub:
             gline = gline.split()
             newline = []
             for ent in gline:
-                if ent.startswith('resume=') or ent.startswith('rd.lvm.lv'):
-                    continue
-                newline.append(ent)
+                if ent.startswith('resume='):
+                    newline.append('resume={}'.format(newswapdev))
+                elif ent.startswith('root='):
+                    newline.append('root={}'.format(newrootdev))
+                elif ent.startswith('rd.lvm.lv='):
+                    ent = convert_lv(ent)
+                    if ent:
+                        newline.append(ent)
+                    elif '""' in ent:
+                        newline.append('""')
+                else:
+                    newline.append(ent)
             defgrubout.write(' '.join(newline) + '\n')
     grubcfg = subprocess.check_output(['find', os.path.join(rootdir, 'boot'), '-name', 'grub.cfg']).decode('utf8').strip().replace(rootdir, '/').replace('//', '/')
     grubcfg = grubcfg.split('\n')
@@ -158,6 +248,12 @@ def fixup(rootdir, vols):
         grubcfg = grubcfg[:-1]
     if len(grubcfg) == 1:
         grubcfg = grubcfg[0]
+    elif not grubcfg:
+        grubcfg = '/boot/grub2/grub.cfg'
+        paths = glob.glob(os.path.join(rootdir, 'boot/efi/EFI/*'))
+        for path in paths:
+            with open(os.path.join(path, 'grub.cfg'), 'w') as stubgrubout:
+                stubgrubout.write("search --no-floppy --root-dev-only --fs-uuid --set=dev " + bootuuid + "\nset prefix=($dev)/grub2\nexport $prefix\nconfigfile $prefix/grub.cfg\n")
     else:
         for gcfg in grubcfg:
             rgcfg = os.path.join(rootdir, gcfg[1:])  # gcfg has a leading / to get rid of
@@ -213,10 +309,19 @@ def fixup(rootdir, vols):
         shimpath = subprocess.check_output(['find', os.path.join(rootdir, 'boot/efi'), '-name', 'shimx64.efi']).decode('utf8').strip()
         shimpath = shimpath.replace(rootdir, '/').replace('/boot/efi', '').replace('//', '/').replace('/', '\\')
         subprocess.check_call(['efibootmgr', '-c', '-d', targblock, '-l', shimpath, '--part', partnum])
+
+    try:
+        os.makedirs(os.path.join(rootdir, 'opt/confluent/bin'))
+    except Exception:
+        pass
+    shutil.copy2('/opt/confluent/bin/apiclient', os.path.join(rootdir, 'opt/confluent/bin/apiclient'))
     #other network interfaces
 
 
 def had_swap():
+    if not os.path.exists('/etc/fstab'):
+        # diskless source, assume swap
+        return True
     with open('/etc/fstab') as tabfile:
         tabs = tabfile.read().split('\n')
         for tab in tabs:
@@ -227,8 +332,14 @@ def had_swap():
                 return True
     return False
 
+newrootdev = None
+newswapdev = None
 def install_to_disk(imgpath):
     global bootuuid
+    global newrootdev
+    global newswapdev
+    global vgname
+    global oldvgname
     lvmvols = {}
     deftotsize = 0
     mintotsize = 0
@@ -260,6 +371,13 @@ def install_to_disk(imgpath):
             biggestfs = fs
             biggestsize = fs['initsize']
         if fs['device'].startswith('/dev/mapper'):
+            oldvgname = fs['device'].rsplit('/', 1)[-1]
+            # if node has - then /dev/mapper will double up the hypen
+            if '_' in oldvgname and '-' in oldvgname.split('_')[-1]:
+                oldvgname = oldvgname.rsplit('-', 1)[0].replace('--', '-')
+                osname = oldvgname.split('_')[0]
+                nodename = socket.gethostname().split('.')[0]
+                vgname = '{}_{}'.format(osname, nodename)
             lvmvols[fs['device'].replace('/dev/mapper/', '')] = fs
             deflvmsize += fs['initsize']
             minlvmsize += fs['minsize']
@@ -304,6 +422,8 @@ def install_to_disk(imgpath):
             end = sectors
         parted.run('mkpart primary {}s {}s'.format(curroffset, end))
         vol['targetdisk'] = get_partname(instdisk, volidx)
+        if vol['mount'] == '/':
+            newrootdev = vol['targetdisk']
         curroffset += size + 1
     if not lvmvols:
         if swapsize:
@@ -313,13 +433,14 @@ def install_to_disk(imgpath):
             if end > sectors:
                 end = sectors
             parted.run('mkpart swap {}s {}s'.format(curroffset, end))
-            subprocess.check_call(['mkswap', get_partname(instdisk, volidx + 1)])
+            newswapdev = get_partname(instdisk, volidx + 1)
+            subprocess.check_call(['mkswap', newswapdev])
     else:
         parted.run('mkpart lvm {}s 100%'.format(curroffset))
         lvmpart = get_partname(instdisk, volidx + 1)
         subprocess.check_call(['pvcreate', '-ff', '-y', lvmpart])
-        subprocess.check_call(['vgcreate', 'localstorage', lvmpart])
-        vginfo = subprocess.check_output(['vgdisplay', 'localstorage', '--units', 'b']).decode('utf8')
+        subprocess.check_call(['vgcreate', vgname, lvmpart])
+        vginfo = subprocess.check_output(['vgdisplay', vgname, '--units', 'b']).decode('utf8')
         vginfo = vginfo.split('\n')
         pesize = 0
         pes = 0
@@ -346,13 +467,17 @@ def install_to_disk(imgpath):
                 extents += 1
             if vol['mount'] == '/':
                 lvname = 'root'
+
             else:
                 lvname = vol['mount'].replace('/', '_')
-            subprocess.check_call(['lvcreate', '-l', '{}'.format(extents), '-y', '-n', lvname, 'localstorage'])
-            vol['targetdisk'] = '/dev/localstorage/{}'.format(lvname)
+            subprocess.check_call(['lvcreate', '-l', '{}'.format(extents), '-y', '-n', lvname, vgname])
+            vol['targetdisk'] = '/dev/{}/{}'.format(vgname, lvname)
+            if vol['mount'] == '/':
+                newrootdev = vol['targetdisk']
         if swapsize:
-            subprocess.check_call(['lvcreate', '-y', '-l', '{}'.format(swapsize // pesize), '-n', 'swap', 'localstorage'])
-            subprocess.check_call(['mkswap', '/dev/localstorage/swap'])
+            subprocess.check_call(['lvcreate', '-y', '-l', '{}'.format(swapsize // pesize), '-n', 'swap', vgname])
+            subprocess.check_call(['mkswap', '/dev/{}/swap'.format(vgname)])
+            newswapdev = '/dev/{}/swap'.format(vgname)
         os.makedirs('/run/imginst/targ')
         for vol in allvols:
             with open(vol['targetdisk'], 'wb') as partition:
@@ -361,6 +486,8 @@ def install_to_disk(imgpath):
             subprocess.check_call(['mount', vol['targetdisk'], '/run/imginst/targ'])
             source = vol['mount'].replace('/', '_')
             source = '/run/imginst/sources/' + source
+            if not os.path.exists(source):
+                source = '/run/imginst/sources/_' + vol['mount']
             blankfsstat = os.statvfs('/run/imginst/targ')
             blankused = (blankfsstat.f_blocks - blankfsstat.f_bfree) * blankfsstat.f_bsize
             sys.stdout.write('\nWriting {0}: '.format(vol['mount']))
@@ -426,4 +553,9 @@ def install_to_disk(imgpath):
 
 
 if __name__ == '__main__':
-    install_to_disk(os.environ['mountsrc'])
+    try:
+        install_to_disk(os.environ['mountsrc'])
+    except Exception:
+        traceback.print_exc()
+        time.sleep(86400)
+        raise

confluent_osdeploy/el9-diskless/profiles/default/scripts/imageboot.sh

@@ -1,6 +1,6 @@
 . /lib/dracut-lib.sh
 confluent_whost=$confluent_mgr
-if [[ "$confluent_whost" == *:* ]]; then
+if [[ "$confluent_whost" == *:* ]] && [[ "$confluent_whost" != "["* ]]; then
     confluent_whost="[$confluent_mgr]"
 fi
 mkdir -p /mnt/remoteimg /mnt/remote /mnt/overlay

confluent_osdeploy/el9-diskless/profiles/default/scripts/installimage

@@ -30,6 +30,7 @@ if [ ! -f /sysroot/tmp/installdisk ]; then
     done
 fi
 lvm vgchange -a n
+/sysroot/usr/sbin/wipefs -a /dev/$(cat /sysroot/tmp/installdisk)
 udevadm control -e
 if [ -f /sysroot/etc/lvm/devices/system.devices ]; then
     rm /sysroot/etc/lvm/devices/system.devices
@@ -40,6 +41,8 @@ echo "Port 22" >> /etc/ssh/sshd_config
 echo 'Match LocalPort 22' >> /etc/ssh/sshd_config
 echo '    ChrootDirectory /sysroot/run/imginst/targ' >> /etc/ssh/sshd_config
 kill -HUP $(cat /run/sshd.pid)
+cp /sysroot/etc/pki/ca-trust/source/anchors/* /sysroot/run/imginst/targ/etc/pki/ca-trust/source/anchors/
+chroot /sysroot/run/imginst/targ update-ca-trust
 
 chroot /sysroot/run/imginst/targ bash -c "source /etc/confluent/functions; run_remote post.sh"
 chroot /sysroot bash -c "umount \$(tac /proc/mounts|awk '{print \$2}'|grep ^/run/imginst/targ)"

confluent_osdeploy/el9-diskless/profiles/default/scripts/onboot.sh

@@ -16,6 +16,7 @@ if [ -z "$confluent_mgr" ]; then
 fi
 confluent_profile=$(grep ^profile: /etc/confluent/confluent.deploycfg|awk '{print $2}')
 timedatectl set-timezone $(grep ^timezone: /etc/confluent/confluent.deploycfg|awk '{print $2}')
+hostnamectl set-hostname $nodename
 export nodename confluent_mgr confluent_profile
 . /etc/confluent/functions
 mkdir -p /var/log/confluent

confluent_osdeploy/el9-diskless/profiles/default/scripts/post.sh

@@ -2,13 +2,24 @@
 
 # This script is executed 'chrooted' into a cloned disk target before rebooting
 #
-
+if [ -f /etc/dracut.conf.d/diskless.conf ]; then
+    rm /etc/dracut.conf.d/diskless.conf
+fi
+for kver in /lib/modules/*; do kver=$(basename $kver); kernel-install add $kver /boot/vmlinuz-$kver; done
 nodename=$(grep ^NODENAME /etc/confluent/confluent.info|awk '{print $2}')
 confluent_apikey=$(cat /etc/confluent/confluent.apikey)
-confluent_mgr=$(grep ^deploy_server: /etc/confluent/confluent.deploycfg|awk '{print $2}')
 confluent_profile=$(grep ^profile: /etc/confluent/confluent.deploycfg|awk '{print $2}')
-export nodename confluent_mgr confluent_profile
+confluent_mgr=$(grep ^deploy_server_v6: /etc/confluent/confluent.deploycfg|awk '{print $2}')
+if [ -z "$confluent_mgr" ] || [ "$confluent_mgr" == "null" ] || ! ping -c 1 $confluent_mgr >& /dev/null; then
+    confluent_mgr=$(grep ^deploy_server: /etc/confluent/confluent.deploycfg|awk '{print $2}')
+fi
+confluent_websrv=$confluent_mgr
+if [[ "$confluent_mgr" == *:* ]]; then
+    confluent_websrv="[$confluent_mgr]"
+fi
+export nodename confluent_mgr confluent_profile confluent_websrv
 . /etc/confluent/functions
+run_remote setupssh
 mkdir -p /var/log/confluent
 chmod 700 /var/log/confluent
 exec >> /var/log/confluent/confluent-post.log
@@ -16,9 +27,9 @@ exec 2>> /var/log/confluent/confluent-post.log
 chmod 600 /var/log/confluent/confluent-post.log
 tail -f /var/log/confluent/confluent-post.log > /dev/console &
 logshowpid=$!
-curl -f https://$confluent_mgr/confluent-public/os/$confluent_profile/scripts/firstboot.service > /etc/systemd/system/firstboot.service
+curl -f https://$confluent_websrv/confluent-public/os/$confluent_profile/scripts/firstboot.service > /etc/systemd/system/firstboot.service
 mkdir -p /opt/confluent/bin
-curl -f https://$confluent_mgr/confluent-public/os/$confluent_profile/scripts/firstboot.sh > /opt/confluent/bin/firstboot.sh
+curl -f https://$confluent_websrv/confluent-public/os/$confluent_profile/scripts/firstboot.sh > /opt/confluent/bin/firstboot.sh
 chmod +x /opt/confluent/bin/firstboot.sh
 systemctl enable firstboot
 selinuxpolicy=$(grep ^SELINUXTYPE /etc/selinux/config |awk -F= '{print $2}')
@@ -33,7 +44,10 @@ run_remote_parts post.d
 # Induce execution of remote configuration, e.g. ansible plays in ansible/post.d/
 run_remote_config post.d
 
-curl -sf -X POST -d 'status: staged' -H "CONFLUENT_NODENAME: $nodename" -H "CONFLUENT_APIKEY: $confluent_apikey" https://$confluent_mgr/confluent-api/self/updatestatus
+# rebuild initrd, pick up new drivers if needed
+dracut -f /boot/initramfs-$(uname -r).img $(uname -r)
+
+curl -sf -X POST -d 'status: staged' -H "CONFLUENT_NODENAME: $nodename" -H "CONFLUENT_APIKEY: $confluent_apikey" https://$confluent_websrv/confluent-api/self/updatestatus
 
 kill $logshowpid
 

confluent_osdeploy/el9-diskless/profiles/default/scripts/sample/consoleredirect

@@ -0,0 +1,15 @@
+is_rhel=false
+
+if test -f /boot/efi/EFI/redhat/grub.cfg; then
+    grubcfg="/etc/default/grub"
+    is_rhel=true
+else
+    echo "Expected File missing: Check if os redhat"
+    exit
+fi
+
+# Working on Redhat
+if $is_rhel; then
+    sed -i '/^GRUB_TERMINAL/s/serial //' $grubcfg
+    grub2-mkconfig -o /boot/grub2/grub.cfg
+fi

confluent_osdeploy/el9-diskless/profiles/default/scripts/syncfileclient

@@ -1,4 +1,6 @@
 #!/usr/bin/python3
+import random
+import time
 import subprocess
 import importlib
 import tempfile
@@ -7,6 +9,7 @@ import os
 import shutil
 import pwd
 import grp
+import sys
 from importlib.machinery import SourceFileLoader
 try:
     apiclient = SourceFileLoader('apiclient', '/opt/confluent/bin/apiclient').load_module()
@@ -227,9 +230,16 @@ def synchronize():
                 myips.append(addr)
         data = json.dumps({'merge': tmpdir, 'appendonce': appendoncedir, 'myips': myips})
         status, rsp = ac.grab_url_with_status('/confluent-api/self/remotesyncfiles', data)
+        if status >= 300:
+            sys.stderr.write("Error starting syncfiles - {}:\n".format(status))
+            sys.stderr.write(rsp.decode('utf8'))
+            sys.stderr.write('\n')
+            sys.stderr.flush()
+            return status
         if status == 202:
             lastrsp = ''
             while status != 204:
+                time.sleep(1+(2*random.random()))
                 status, rsp = ac.grab_url_with_status('/confluent-api/self/remotesyncfiles')
                 if not isinstance(rsp, str):
                     rsp = rsp.decode('utf8')
@@ -277,10 +287,21 @@ def synchronize():
                             os.chmod(fname, int(opts[fname][opt], 8))
                     if uid != -1 or gid != -1:
                         os.chown(fname, uid, gid)
+        return status
     finally:
         shutil.rmtree(tmpdir)
         shutil.rmtree(appendoncedir)
 
 
 if __name__ == '__main__':
-    synchronize()
+    status = 202
+    while status not in (204, 200):
+        try:
+            status = synchronize()
+        except Exception as e:
+            sys.stderr.write(str(e))
+            sys.stderr.write('\n')
+            sys.stderr.flush()
+            status = 300
+        if status not in (204, 200):
+            time.sleep((random.random()*3)+2)

confluent_osdeploy/esxi7/initramfs/bin/dcuiweasel

@@ -6,6 +6,7 @@ if [ ! -f /var/run/vmware/show-esx-shell-login ]; then
     chvt 2
     /etc/init.d/ESXShell start
 fi
+/opt/confluent/bin/apiclient -f
 uuid=$(vsish -e get /hardware/bios/dmiInfo|grep -A15 UUID|sed -e 's/.*://'|sed -e ':a;N;$!ba;s/\n//g' | sed -e 's/ *0x//g')
 uuid=${uuid:0:8}-${uuid:8:4}-${uuid:12:4}-${uuid:16:4}-${uuid:20:12}
 kargs=$(vsish -e get /system/bootCmdLine|grep "command line:")
@@ -116,4 +117,5 @@ profile=$(grep ^profile: /etc/confluent/confluent.deploycfg.new | sed -e 's/^pro
 mv /etc/confluent/confluent.deploycfg.new /etc/confluent/confluent.deploycfg
 export node mgr profile
 . /tmp/modinstall
+localcli network firewall load
 exec /bin/install

confluent_osdeploy/esxi7/profiles/hypervisor/profile.yaml

@@ -1,3 +1,3 @@
-label: Confluent installation of VMware ESXi %%VERSION%% Hypervisor
+label: VMware ESXi %%VERSION%% Hypervisor
 ostype: esxi
 kernelargs: runweasel

confluent_osdeploy/genesis/initramfs/opt/confluent/bin/rungenesis

@@ -1,5 +1,14 @@
 #!/bin/bash
+mkdir -p /usr/libexec /run/sshd
+if [ ! -x /usr/libexec/platform-python ]; then
+       ln -s /usr/bin/python3 /usr/libexec/platform-python
+fi
 export LANG=en_US.utf8
+
+(while :; do TERM=linux tmux a <> /dev/tty1 >&0 2>&1; done) &
+
+
+
 mkdir -p /etc/confluent
 if ! grep console= /proc/cmdline >& /dev/null; then
     autocons=$(/opt/confluent/bin/autocons)
@@ -9,13 +18,9 @@ if ! grep console= /proc/cmdline >& /dev/null; then
     if [ ! -z "$autocons" ]; then
         echo "Using $(cat /tmp/01-autocons.conf)"
         (while :; do TERM=xterm-256color tmux a <> $autocons >&0 2>&1; done) &
-        (while :; do TERM=linux tmux a <> /dev/tty1 >&0 2>&1; done) &
-    else
-        (while :; do TERM=linux tmux a <> /dev/console >&0 2>&1; done) &
     fi
 else
     (while :; do TERM=xterm-256color tmux a <> /dev/console >&0 2>&1; done) &
-    (while :; do TERM=linux tmux a <> /dev/tty1 >&0 2>&1; done) &
 fi
 (while :; do TERM=linux tmux <> /dev/tty2 >&0 2>&1; done) &
 echo -n "udevd: "
@@ -29,6 +34,13 @@ modprobe ib_umad
 modprobe hfi1
 modprobe mlx5_ib
 echo "done"
+if [ -x /usr/bin/seatd-launch -a -x /usr/bin/sway ]; then
+    export XDG_RUNTIME_DIR=/run/users/0
+    mkdir -p $XDG_RUNTIME_DIR
+    sed -i '/^output /d' /etc/sway/config
+    echo 'exec foot -t xterm -T Terminal tmux a' > /etc/sway/config.d/genesis
+    (while :; do seatd-launch sway <> /dev/tty1 >& /dev/null; done) &
+fi
 cat > /etc/ssh/sshd_config << EOF
 Port 22
 Port 3389
@@ -43,18 +55,25 @@ mkdir -p /etc/pki/tls/certs
 cat /tls/*.pem > /etc/pki/tls/certs/ca-bundle.crt
 TRIES=0
 touch /etc/confluent/confluent.info
+TRIES=5
+echo -n "Waitiing for disks..."
+while [ ! -e /dev/disk/by-label ] && [ $TRIES -gt 0 ]; do
+    sleep 1
+    TRIES=$((TRIES - 1))
+done
+echo "Done"
 if [ -e /dev/disk/by-label/CNFLNT_IDNT ]; then
     mkdir -p /media/ident
     mount /dev/disk/by-label/CNFLNT_IDNT /media/ident
     if [ -e /media/ident/genesis_bootstrap.sh ]; then
-        exec sh /media/ident/genesis_bootstrap.sh
+        exec bash /media/ident/genesis_bootstrap.sh
     fi
 fi
 if [ -e /dev/disk/by-label/GENESIS-X86 ]; then
     mkdir -p /media/genesis
     mount /dev/disk/by-label/GENESIS-X86 /media/genesis
     if [ -e /media/genesis/genesis_bootstrap.sh ]; then
-        exec sh /media/genesis/genesis_bootstrap.sh
+        exec bash /media/genesis/genesis_bootstrap.sh
     fi
 fi
 cd /sys/class/net

confluent_osdeploy/genesis/initramfs/usr/lib/dracut/hooks/cmdline/10-genesis.sh

@@ -2,15 +2,17 @@ root=1
 rootok=1
 netroot=genesis
 clear
+mount -t cgroup2 cgroup2 /sys/fs/cgroup
+mount -t efivarfs efivarfs /sys/firmware/efi/efivars
 echo PS1="'"'[genesis running on \H \w]$ '"'" >> ~/.bashrc
 echo PS1="'"'[genesis running on \H \w]$ '"'" >> ~/.bash_profile
 mkdir -p /etc/ssh
 mkdir -p /var/tmp/
 mkdir -p /var/empty/sshd
-sed -i '/^root:x/d' /etc/passwd
+sed -i '/^root:/d' /etc/passwd
 echo root:x:0:0::/:/bin/bash >> /etc/passwd
 echo sshd:x:30:30:SSH User:/var/empty/sshd:/sbin/nologin >> /etc/passwd
-tmux new-session -d sh /opt/confluent/bin/rungenesis
+tmux new-session -d bash /opt/confluent/bin/rungenesis
 while :; do
 	sleep 86400
 done

confluent_osdeploy/genesis/profiles/default/scripts/configbmc

@@ -210,16 +210,28 @@ def set_port_tsm(s, port, model):
 
 
 def set_port_xcc(s, port, model):
+    if '_' in port:
+        port_type = port.split('_')[0]
+        port_number = port.split('_')[1]  # this could be the number or the form sfp28 || rj45
+        port = port_type
     oport = port
     if port.lower() == 'dedicated':
         port = b'\x01'
     elif port.lower() in ('ml2', 'ocp'):
         port = b'\x02\x00'
-    elif port.lower() == 'lom':
-        if model == '7x58':
-            port = b'\x00\x02'
+    elif port.lower() == 'lom':  # potentially have to change to port_form.lower() == 'sfp28 || or rj45  // if it is two port sfp28 it is usually the first port that is ncsi enabled'
+        if port_number == '1' or port_number.lower() == 'sfp28':
+            port = b'\x05\x00'
+        elif port_number == '3' or port_number.lower() == 'rj45':
+            if model == '7x58':
+                port = b'\x00\x02'
+            else:
+                port = b'\x00\x00'
         else:
-            port = b'\x00\x00'
+            if model == '7x58':
+                port = b'\x00\x02'
+            else:
+                port = b'\x00\x00'
     else:
         port = port.split(' ')
         port = bytes(bytearray([int(x) for x in port]))

confluent_osdeploy/genesis/profiles/default/scripts/functions

@@ -23,6 +23,7 @@ function set_confluent_vars() {
     if [ -z "$confluent_profile" ]; then
         confluent_profile=$(grep ^profile: /etc/confluent/confluent.deploycfg | sed -e 's/[^ ]*: //')
     fi
+    export nodename confluent_mgr confluent_profile
 }
 
 fetch_remote() {

confluent_osdeploy/genesis/profiles/default/scripts/onboot.sh

@@ -10,6 +10,9 @@
 # present
 run_remote_python syncfileclient
 
+run_remote_parts onboot.d
+
+
 # Induce execution of remote configuration, e.g. ansible plays in ansible/onboot.d/
 run_remote_config onboot
 

confluent_osdeploy/genesis/profiles/default/scripts/syncfileclient

@@ -1,4 +1,6 @@
 #!/usr/bin/python3
+import random
+import time
 import subprocess
 import importlib
 import tempfile
@@ -7,6 +9,7 @@ import os
 import shutil
 import pwd
 import grp
+import sys
 from importlib.machinery import SourceFileLoader
 try:
     apiclient = SourceFileLoader('apiclient', '/opt/confluent/bin/apiclient').load_module()
@@ -227,9 +230,16 @@ def synchronize():
                 myips.append(addr)
         data = json.dumps({'merge': tmpdir, 'appendonce': appendoncedir, 'myips': myips})
         status, rsp = ac.grab_url_with_status('/confluent-api/self/remotesyncfiles', data)
+        if status >= 300:
+            sys.stderr.write("Error starting syncfiles - {}:\n".format(status))
+            sys.stderr.write(rsp.decode('utf8'))
+            sys.stderr.write('\n')
+            sys.stderr.flush()
+            return status
         if status == 202:
             lastrsp = ''
             while status != 204:
+                time.sleep(1+(2*random.random()))
                 status, rsp = ac.grab_url_with_status('/confluent-api/self/remotesyncfiles')
                 if not isinstance(rsp, str):
                     rsp = rsp.decode('utf8')
@@ -277,10 +287,21 @@ def synchronize():
                             os.chmod(fname, int(opts[fname][opt], 8))
                     if uid != -1 or gid != -1:
                         os.chown(fname, uid, gid)
+        return status
     finally:
         shutil.rmtree(tmpdir)
         shutil.rmtree(appendoncedir)
 
 
 if __name__ == '__main__':
-    synchronize()
+    status = 202
+    while status not in (204, 200):
+        try:
+            status = synchronize()
+        except Exception as e:
+            sys.stderr.write(str(e))
+            sys.stderr.write('\n')
+            sys.stderr.flush()
+            status = 300
+        if status not in (204, 200):
+            time.sleep((random.random()*3)+2)

confluent_osdeploy/rhvh4/profiles/default/scripts/getinstalldisk

@@ -1,15 +1,19 @@
+#!/usr/bin/python3
 import subprocess
 import os
 
 class DiskInfo(object):
     def __init__(self, devname):
+        if devname.startswith('nvme') and 'c' in devname:
+            raise Exception("Skipping multipath devname")
         self.name = devname
         self.wwn = None
         self.path = None
         self.model = ''
         self.size = 0
-        self.driver = None
+        self.driver = ''
         self.mdcontainer = ''
+        self.subsystype = ''
         devnode = '/dev/{0}'.format(devname)
         qprop = subprocess.check_output(
             ['udevadm', 'info', '--query=property', devnode])
@@ -44,7 +48,9 @@ class DiskInfo(object):
             elif (k == 'DRIVERS' and not self.driver
                     and v not in ('"sd"', '""')):
                 self.driver = v.replace('"', '')
-        if not self.driver and 'imsm' not in self.mdcontainer:
+            elif k == 'ATTRS{subsystype}':
+                self.subsystype = v.replace('"', '')
+        if not self.driver and 'imsm' not in self.mdcontainer and self.subsystype != 'nvm':
             raise Exception("No driver detected")
         if os.path.exists('/sys/block/{0}/size'.format(self.name)):
             with open('/sys/block/{0}/size'.format(self.name), 'r') as sizesrc:

confluent_osdeploy/suse15-diskless/initramfs/lib/dracut/hooks/cmdline/10-confluentdiskless.sh

@@ -116,7 +116,7 @@ fi
 ready=0
 while [ $ready = "0" ]; do
     get_remote_apikey
-    if [[ $confluent_mgr == *:* ]]; then
+    if [[ $confluent_mgr == *:* ]] && [[ $confluent_mgr != "["* ]]; then
         confluent_mgr="[$confluent_mgr]"
     fi
     tmperr=$(mktemp)

confluent_osdeploy/suse15-diskless/profiles/default/scripts/functions

@@ -53,6 +53,7 @@ function set_confluent_vars() {
     if [ -z "$confluent_profile" ]; then
         confluent_profile=$(grep ^profile: /etc/confluent/confluent.deploycfg | sed -e 's/[^ ]*: //')
     fi
+    export confluent_profile confluent_mgr nodename
 }
 
 fetch_remote() {

confluent_osdeploy/suse15-diskless/profiles/default/scripts/imageboot.sh

@@ -140,4 +140,5 @@ mv /lib/modules/$(uname -r) /lib/modules/$(uname -r)-ramfs
 ln -s /sysroot/lib/modules/$(uname -r) /lib/modules/
 mv /lib/firmware /lib/firmware-ramfs
 ln -s /sysroot/lib/firmware /lib/firmware
+chroot /sysroot chkstat --system --set --noheader > /dev/null
 exec /opt/confluent/bin/start_root

confluent_osdeploy/suse15-diskless/profiles/default/scripts/syncfileclient

@@ -1,4 +1,6 @@
 #!/usr/bin/python3
+import random
+import time
 import subprocess
 import importlib
 import tempfile
@@ -7,6 +9,7 @@ import os
 import shutil
 import pwd
 import grp
+import sys
 from importlib.machinery import SourceFileLoader
 try:
     apiclient = SourceFileLoader('apiclient', '/opt/confluent/bin/apiclient').load_module()
@@ -227,9 +230,16 @@ def synchronize():
                 myips.append(addr)
         data = json.dumps({'merge': tmpdir, 'appendonce': appendoncedir, 'myips': myips})
         status, rsp = ac.grab_url_with_status('/confluent-api/self/remotesyncfiles', data)
+        if status >= 300:
+            sys.stderr.write("Error starting syncfiles - {}:\n".format(status))
+            sys.stderr.write(rsp.decode('utf8'))
+            sys.stderr.write('\n')
+            sys.stderr.flush()
+            return status
         if status == 202:
             lastrsp = ''
             while status != 204:
+                time.sleep(1+(2*random.random()))
                 status, rsp = ac.grab_url_with_status('/confluent-api/self/remotesyncfiles')
                 if not isinstance(rsp, str):
                     rsp = rsp.decode('utf8')
@@ -277,10 +287,21 @@ def synchronize():
                             os.chmod(fname, int(opts[fname][opt], 8))
                     if uid != -1 or gid != -1:
                         os.chown(fname, uid, gid)
+        return status
     finally:
         shutil.rmtree(tmpdir)
         shutil.rmtree(appendoncedir)
 
 
 if __name__ == '__main__':
-    synchronize()
+    status = 202
+    while status not in (204, 200):
+        try:
+            status = synchronize()
+        except Exception as e:
+            sys.stderr.write(str(e))
+            sys.stderr.write('\n')
+            sys.stderr.flush()
+            status = 300
+        if status not in (204, 200):
+            time.sleep((random.random()*3)+2)

confluent_osdeploy/suse15/profiles/hpc/autoyast.leap

@@ -10,6 +10,12 @@ dynamic behavior and replace with static configuration.
       <hwclock>UTC</hwclock>
       <timezone>%%TIMEZONE%%</timezone>
     </timezone>
+    <firstboot>
+      <firstboot_enabled config:type="boolean">false</firstboot_enabled>
+    </firstboot>
+    <kdump>
+      <add_crash_kernel config:type="boolean">false</add_crash_kernel>
+    </kdump>
     <general>
       <self_update config:type="boolean">false</self_update>
       <mode>

confluent_osdeploy/suse15/profiles/hpc/initprofile.sh

@@ -1,4 +1,7 @@
 #!/bin/sh
+# WARNING
+# be careful when editing files here as this script is called
+# in parallel to other copy operations, so changes to files can be lost
 discnum=$(basename $1)
 if [ "$discnum" != 1 ]; then exit 0; fi
 if [ -e $2/boot/kernel ]; then exit 0; fi

confluent_osdeploy/suse15/profiles/hpc/scripts/functions

@@ -53,6 +53,7 @@ function set_confluent_vars() {
     if [ -z "$confluent_profile" ]; then
         confluent_profile=$(grep ^profile: /etc/confluent/confluent.deploycfg | sed -e 's/[^ ]*: //')
     fi
+    export confluent_profile confluent_mgr nodename
 }
 
 fetch_remote() {

confluent_osdeploy/suse15/profiles/hpc/scripts/getinstalldisk

@@ -1,15 +1,19 @@
+#!/usr/bin/python3
 import subprocess
 import os
 
 class DiskInfo(object):
     def __init__(self, devname):
+        if devname.startswith('nvme') and 'c' in devname:
+            raise Exception("Skipping multipath devname")
         self.name = devname
         self.wwn = None
         self.path = None
         self.model = ''
         self.size = 0
-        self.driver = None
+        self.driver = ''
         self.mdcontainer = ''
+        self.subsystype = ''
         devnode = '/dev/{0}'.format(devname)
         qprop = subprocess.check_output(
             ['udevadm', 'info', '--query=property', devnode])
@@ -44,7 +48,9 @@ class DiskInfo(object):
             elif (k == 'DRIVERS' and not self.driver
                     and v not in ('"sd"', '""')):
                 self.driver = v.replace('"', '')
-        if not self.driver and 'imsm' not in self.mdcontainer:
+            elif k == 'ATTRS{subsystype}':
+                self.subsystype = v.replace('"', '')
+        if not self.driver and 'imsm' not in self.mdcontainer and self.subsystype != 'nvm':
             raise Exception("No driver detected")
         if os.path.exists('/sys/block/{0}/size'.format(self.name)):
             with open('/sys/block/{0}/size'.format(self.name), 'r') as sizesrc:

confluent_osdeploy/suse15/profiles/hpc/scripts/post.d/10-remove-online-repos.sh

@@ -0,0 +1,3 @@
+#!/usr/bin/bash
+# remove online repos
+grep -lE "baseurl=https?://download.opensuse.org" /etc/zypp/repos.d/*repo | xargs rm --

confluent_osdeploy/suse15/profiles/hpc/scripts/sample/consoleredirect

@@ -0,0 +1,39 @@
+is_suse=false
+
+if test -f /boot/efi/EFI/sle_hpc/grub.cfg; then
+    grubcfg="/boot/efi/EFI/sle_hpc/grub.cfg"
+    grub2-mkconfig -o $grubcfg
+    is_suse=true
+else
+    echo "Expected File missing: Check if os sle_hpc"
+    exit
+fi
+
+# working on SUSE
+if $is_suse; then
+    start=false
+    num_line=0
+    lines_to_edit=()
+    while read line; do
+        ((num_line++))
+        if [[ $line == *"grub_platform"* ]]; then
+            start=true
+        fi
+        if $start; then
+            if [[ $line != "#"* ]];then
+                lines_to_edit+=($num_line)
+            fi
+        fi
+        if [[ ${#line} -eq 2 && $line == *"fi" ]]; then
+            if $start; then
+                start=false
+            fi
+        fi
+    done < grub_cnf.cfg
+
+    for line_num in "${lines_to_edit[@]}"; do
+        line_num+="s"
+        sed -i "${line_num},^,#," $grubcfg
+    done
+    sed -i 's,^terminal,#terminal,' $grubcfg
+fi

confluent_osdeploy/suse15/profiles/hpc/scripts/syncfileclient

@@ -1,4 +1,6 @@
 #!/usr/bin/python3
+import random
+import time
 import subprocess
 import importlib
 import tempfile
@@ -7,6 +9,7 @@ import os
 import shutil
 import pwd
 import grp
+import sys
 from importlib.machinery import SourceFileLoader
 try:
     apiclient = SourceFileLoader('apiclient', '/opt/confluent/bin/apiclient').load_module()
@@ -227,9 +230,16 @@ def synchronize():
                 myips.append(addr)
         data = json.dumps({'merge': tmpdir, 'appendonce': appendoncedir, 'myips': myips})
         status, rsp = ac.grab_url_with_status('/confluent-api/self/remotesyncfiles', data)
+        if status >= 300:
+            sys.stderr.write("Error starting syncfiles - {}:\n".format(status))
+            sys.stderr.write(rsp.decode('utf8'))
+            sys.stderr.write('\n')
+            sys.stderr.flush()
+            return status
         if status == 202:
             lastrsp = ''
             while status != 204:
+                time.sleep(1+(2*random.random()))
                 status, rsp = ac.grab_url_with_status('/confluent-api/self/remotesyncfiles')
                 if not isinstance(rsp, str):
                     rsp = rsp.decode('utf8')
@@ -277,10 +287,21 @@ def synchronize():
                             os.chmod(fname, int(opts[fname][opt], 8))
                     if uid != -1 or gid != -1:
                         os.chown(fname, uid, gid)
+        return status
     finally:
         shutil.rmtree(tmpdir)
         shutil.rmtree(appendoncedir)
 
 
 if __name__ == '__main__':
-    synchronize()
+    status = 202
+    while status not in (204, 200):
+        try:
+            status = synchronize()
+        except Exception as e:
+            sys.stderr.write(str(e))
+            sys.stderr.write('\n')
+            sys.stderr.flush()
+            status = 300
+        if status not in (204, 200):
+            time.sleep((random.random()*3)+2)

confluent_osdeploy/suse15/profiles/server/autoyast.leap

@@ -10,6 +10,12 @@ dynamic behavior and replace with static configuration.
       <hwclock>UTC</hwclock>
       <timezone>%%TIMEZONE%%</timezone>
     </timezone>
+    <firstboot>
+      <firstboot_enabled config:type="boolean">false</firstboot_enabled>
+    </firstboot>
+    <kdump>
+      <add_crash_kernel config:type="boolean">false</add_crash_kernel>
+    </kdump>
     <general>
       <self_update config:type="boolean">false</self_update>
       <mode>

confluent_osdeploy/suse15/profiles/server/initprofile.sh

@@ -1,4 +1,7 @@
 #!/bin/sh
+# WARNING
+# be careful when editing files here as this script is called
+# in parallel to other copy operations, so changes to files can be lost
 discnum=$(basename $1)
 if [ "$discnum" != 1 ]; then exit 0; fi
 if [ -e $2/boot/kernel ]; then exit 0; fi

confluent_osdeploy/suse15/profiles/server/scripts/functions

@@ -53,6 +53,7 @@ function set_confluent_vars() {
     if [ -z "$confluent_profile" ]; then
         confluent_profile=$(grep ^profile: /etc/confluent/confluent.deploycfg | sed -e 's/[^ ]*: //')
     fi
+    export confluent_profile confluent_mgr nodename
 }
 
 fetch_remote() {

confluent_osdeploy/suse15/profiles/server/scripts/getinstalldisk

@@ -1,15 +1,19 @@
+#!/usr/bin/python3
 import subprocess
 import os
 
 class DiskInfo(object):
     def __init__(self, devname):
+        if devname.startswith('nvme') and 'c' in devname:
+            raise Exception("Skipping multipath devname")
         self.name = devname
         self.wwn = None
         self.path = None
         self.model = ''
         self.size = 0
-        self.driver = None
+        self.driver = ''
         self.mdcontainer = ''
+        self.subsystype = ''
         devnode = '/dev/{0}'.format(devname)
         qprop = subprocess.check_output(
             ['udevadm', 'info', '--query=property', devnode])
@@ -44,7 +48,9 @@ class DiskInfo(object):
             elif (k == 'DRIVERS' and not self.driver
                     and v not in ('"sd"', '""')):
                 self.driver = v.replace('"', '')
-        if not self.driver and 'imsm' not in self.mdcontainer:
+            elif k == 'ATTRS{subsystype}':
+                self.subsystype = v.replace('"', '')
+        if not self.driver and 'imsm' not in self.mdcontainer and self.subsystype != 'nvm':
             raise Exception("No driver detected")
         if os.path.exists('/sys/block/{0}/size'.format(self.name)):
             with open('/sys/block/{0}/size'.format(self.name), 'r') as sizesrc:

confluent_osdeploy/suse15/profiles/server/scripts/post.d/10-remove-online-repos.sh

@@ -0,0 +1,3 @@
+#!/usr/bin/bash
+# remove online repos
+grep -lE "baseurl=https?://download.opensuse.org" /etc/zypp/repos.d/*repo | xargs rm --