SLES Upgrade to 15.

5 PowerVC
Prerequisites: ______________________________________________________________________ 3

Validate Virtual Optical Media _____________________________________________________________ 3

Upload ISO to Virtual Media repository. Needs at least 12GB available. ____________________________ 3

Create vSCSI unit to the server and prepare to start the console for the upgrade. ____________________ 4

Verify vSCSI is created ____________________________________________________________________ 5

SLES 15 SP6 OS Upgrade _____________________________________________________________ 5

Backup files ____________________________________________________________________________ 5

De-register the server to avoid issues. _______________________________________________________ 6

Restart the server and start with SMS mode. _________________________________________________ 8

Select SCSI CS-ROM ______________________________________________________________________ 9

Select Upgrade _________________________________________________________________________ 10

Leave language and keyboard defaults to English. ____________________________________________ 11

Select partition for upgrade. ______________________________________________________________ 12

Error: Cannot find any product to upgrade.________________________________________________ 13

Error: No migration product found ______________________________________________________ 14

Hit “Enter” to continue; we don’t need the SAP partitions mounted. _____________________________ 16

Accept the license agreements. ___________________________________________________________ 16

All previously used repositories will be removed, and this is OK. Hit Next. _________________________ 17

Select the following modules: Basesystem, Legacy and Server Applications. _______________________ 17

Hit “Next” to continue with the installation of the selected modules. _____________________________ 18

Review the summary ____________________________________________________________________ 18

Install packages with vendor change. ____________________________________________________ 19

Change Bootloader option to not use a specific partition. ____________________________________ 19

Press “Update” and wait for all the packages to be installed.____________________________________ 21

Bootloader recovery. _______________________________________________________________ 22

 Error while mounting /mnt: special device /dev/rootvg/lvroot does not exist. ___________________ 24

Error: grub2-mkconfig: failed to get canonical path of ‘/boot/grub2’ ___________________________ 26

Error: grub2-install: More than one install device? __________________________________________ 27

Notes after installation._____________________________________________________________ 26

Fix DNS resolvers _______________________________________________________________________ 33

Register the server on RMT _______________________________________________________________ 33

Install NCPA ___________________________________________________________________________ 33

Fix SSSD ______________________________________________________________________________ 34

Patch the server to the latest level _________________________________________________________ 28

Error: Unsupported version of key: V3____________________________________________________ 29

Error: Python packages conflicts when patching ____________________________________________ 29

Check that the salt minion is running. ______________________________________________________ 32

Fix FTP/SFTP ___________________________________________________________________________ 37

Fix RMC _______________________________________________________________________________ 31

Run Healthchecks _________________________________________________________________32

Warning: PV is using an old PV header ______________________________________________________ 32

Revision History _____________________________________________________________________33

Prerequisites:

- Validate server is managed on powervc y novalink. If not, snapshot must be taken by

Virtualization team.
- Take Capture from PowerVC GUI.
Validate Virtual Optical Media

- Validate from novalink we have access to the filer where the media is located:
[padmin@fit02a08lp969 ~]$ showmount -e 10.130.19.83|grep
/vol/vffit1_shares/install_files | grep 10.132.72
*Change for the segment of the server to be upgraded.
- Check from Novalink the vios:
[padmin@fit02a08lp969 ~]$ pvmctl vios list

Virtual I/O Servers

+---------------------------+----+---------+--------+----------+------+-----+-----+

| Name | ID | State | RMC | Ref Code | Mem | CPU | Ent |

+---------------------------+----+---------+--------+----------+------+-----+-----+

| fit02a08ap962_lp958_vios1 | 2 | running | active | | 8192 | 4 | 2.0 |

| fit02a08ap963_lp958_vios2 | 3 | running | active | | 8192 | 4 | 2.0 |

+---------------------------+----+---------+--------+----------+------+-----+-----+

[padmin@fit02a08lp969 ~]$

- Access to each vios and validate if there is at least 10G of space to mount the media.

padmin@fit02a08ap963:/home/padmin $ lsrep
Size(mb) Free(mb) Parent Pool Parent Size Parent Free
13258 9463 rootvg 1525760 1388544
Name File Size Optical Access
sles15-5_sap_iso 3795 None ro
padmin@fit02a08ap963:/home/padmin $

Note: If repository does not have enough space, you can increase it, i.e:
padmin@fit02a08ap963:/home/padmin $ chrep -size 4G
*Adds an additional 4GB to the repository
Upload ISO to Virtual Media repository. Needs at least 12GB available.

In the Novalink:

Check if it is already mounted with df command. If it’s not, mount it:

 sudo mount 10.130.29.105:/syntax_shares01/OSX /mnt

If you have space, you can proceed to create the optical device from novalink.

*We will use the SP5 ISO for this upgrade.

[padmin@fit02a08lp969 ~]$ pvmctl vom upload --data /mnt/OS/SLES15SAP/15.6/SLE-

SAP-15-SP6-Full-ppc64le-GM-Media1.iso --name sles_sap15SP6_iso --vios
fit02a08ap963_lp958_vios2

*Change the name of the VIOS in yellow accordingly.

Create vSCSI unit to the server and prepare to start the console for the upgrade.

pvmctl scsi create -p name=fit02a08ap963_lp958_vios2 --lpar

name=bac02a08ls011-6b147c9b-0000000f --stor-id name=sles_sap15SP5_iso --type
vopt

*Change the data in yellow accordingly

Verify vSCSI is created

pvmctl scsi list

SLES 15 SP6 OS Upgrade

Search for the Novalink of the server’s hosts in PowerVC and login to it with ssh with padmin
user.

Open a console to the server:

pvmctl vm list

mkvterm --id <id>

Backup files
Before starting the OS upgrade make sure you backup some important files:

cp -pv /etc/resolv.conf /tmp/resolv.conf.bck

cp -pv /etc/sysconfig/network/ifcfg-eth0 /tmp/ifcfg-eth0.bck

cp -pv /etc/sysconfig/network/routes /tmp/routes.bck

Take a snapshot of the server in PowerVC
Power off the server

shutdown -h now

Once it is powered off, go to the PowerVC console and click on “Capture”:

Click “Continue”:

Select “Boot set only” and rename the snapshot as

<hostname>_upgrade_sles_15-6_<CHG request>
Click “Capture”.

Messages like the following should be logged:

You can validate by going into “Images” and searching for your snapshot:

Now you can bring the server up again.

De-register the server to avoid issues.

Before upgrading run these commands:

 sudo SUSEConnect --de-register

sudo SUSEConnect --cleanup

sudo rm -f /etc/SUSEConnect

sudo rm -rf /etc/zypp/credentials.d/*

sudo rm -rf /etc/zypp/repos.d/*

sudo rm -f /etc/zypp/services.d/*

Reboot the server to start the upgrade.

Restart the server and start with SMS mode.

5. Select boot options

1. Select Install/Boot Device

2. CD/DVD

7. List All Devices

*It will take some minutes to scan all devices and it will show a list.
Select SCSI CS-ROM

2. Normal Mode Boot

1. Yes
Select Upgrade

Position cursor to Upgrade and hit “Enter” to start the upgrade process. Wait for it to finish.

If you get following errors, select “Continue”:

Leave language and keyboard defaults to English.

Select “Next” on Network

Select partition for upgrade.

*If you get a multipath warning, select “Yes”:

Next

Error: Cannot find any product to upgrade.

If you get

“Error: Cannot find any product to upgrade. Make sure the selected partition contains an
upgradable product.”:

OR:

The /root partition is not showing:

Reboot the system.

Then, do the following:

cd /etc/products.d

ls -l

rm /etc/products.d/baseproduct

ln -s SLES.prod /etc/products.d/baseproduct

Error: No migration product found

If you get the following error:

Reboot the server and run these commands:

sudo SUSEConnect --de-register

sudo SUSEConnect --cleanup

sudo rm -f /etc/SUSEConnect

sudo rm -rf /etc/zypp/credentials.d/*

sudo rm -rf /etc/zypp/repos.d/*

sudo rm -f /etc/zypp/services.d/*

Then, run the upgrade again.

Hit “Enter” to continue; we don’t need the SAP partitions mounted.

Accept the license agreements.

All previously used repositories will be removed, and this is OK. Hit Next.

Select the following modules: Basesystem, Legacy and Server Applications.

Hit “Next” to continue with the installation of the selected modules.

Review the summary

Some packages will probably need to be solved manually. Ensure that the previous installation
is going to be upgraded.

Hit tab to access the submenus and then use the UP/DOWN arrow keys to navigate.

Hit enter to solve the packages not solved.

Install packages with vendor change.

Change Bootloader option to not use a specific partition.

Select Propose
Select Ok
Configuration is ready.

Press “Update” and wait for all the packages to be installed.

Bootloader recovery.

Enter the recovery menu.

Restart the server and start with SMS mode.

5. Select boot options

1. Select Install/Boot Device

2. CD/DVD

7. List All Devices

*It will take some minutes to scan all devices and it will show a list, select SCSI CD-ROM

2. Normal Mode Boot

1. Yes

Select “Rescue system”

 mount /dev/rootvg/lvroot /mnt

Error while mounting /mnt: special device /dev/rootvg/lvroot does not exist.

If you see the following error:

Run:

multipath -r

multipath -l

vgchange -ay

multipath -r

multipath -l
Try to mount again:

mount /dev/rootvg/lvroot /mnt

mount /dev/rootvg/lvvar /mnt/var

mount /dev/rootvg/lvusr /mnt/usr

mount /dev/rootvg/lvtmp /mnt/tmp

mount -o bind /proc /mnt/proc

mount -o bind /sys /mnt/sys

mount -o bind /dev /mnt/dev

chroot /mnt /bin/bash

Then run:

grub2-mkconfig -o /boot/grub2/grub.cfg

Error: grub2-mkconfig: failed to get canonical path of ‘/boot/grub2’

If you get the next error:

Exit chroot and mount /mnt/boot:

exit

mount /dev/mapper/*-part2 /mnt/boot

df -hT /mnt/boot

chroot /mnt /bin/bash

Try to run the command again:

grub2-mkconfig -o /boot/grub2/grub.cfg
Then run:

grub2-install /dev/sda1 --boot-directory=/boot

Error: grub2-install: More than one install device?

If you get the following error with the last command:

Try checking that you are using two dashes “--“ instead of one large dash “–“.

Try to run the command again:

grub2-install /dev/sda1 --boot-directory=/boot

Then run:

dracut -v --force -a "multipath lvm" --regenerate-all

 Exit chroot and reboot.

exit

umount /mnt/*

umount /mnt

shutdown -r now

Run post steps script

After ISO upgrade please run the following script to perform the post steps and preparations for
the SLES 15.6 upgrade.

/collect/OSX/bin/sles_upgrades/15/poststeps.sh

This will make backup files in /tmp, fix the resolvers and change some configurations to services
like SSSD, NCPA, VFSTPD. It will also register the server to the RMT server.

Please validate if the services are running correctly after it finishes.

Patch the server to the latest level

To avoid conflicts, make the /boot partition writable with:

mount -o rw,remount /boot

Apply patch.

screen

zypper patch --with-interactive

Error: Unsupported version of key: V3

If after upgrade, you get the next warning when running zypper commands:

Warning: Unsupported version of key: V3

Run:

zypper ref | grep "gpg-pubkey-" | xargs rpm -e

Patch the server with:

zypper lr -u

zypper patch --with-interactive

Repeat this until you get "Nothing to do" message.

Error: Python packages conflicts when patching

If you are prompted with a problem regarding Python packages, check that they are the same
packages, just newer versions and select option 1.

Type "Y".
Run the patch again.

zypper patch --with-interactive

If you are prompted again with Python packages, select 1 again:

If you get packages conflicts, check that they are the same packages, just newer versions. If
that's the case, type "yes":
If you get "Nothing to do" message, the patch is complete:

Reboot the server.

shutdown -r now

Now run your tests for the server. Server is now upgraded.

Fix RMC
Run the following:

mount /collect

cd /collect/OSX/rsct/sles15

zypper in -y src* dev* rsct* Dyn*

/usr/sbin/rsct/bin/mknodeid -f

rmcctrl -z; rmcctrl -A; rmcctrl -p

/usr/sbin/rsct/install/bin/recfgct

Wait 1 min and review.

/usr/sbin/rsct/bin/rmcdomainstatus -s ctrmc

*RMC status should be fixed after 15 minutes

Check that the salt minion is running.

systemctl -l status salt-minion

systemctl -l start salt-minion

systemctl -l enable salt-minion

Run Healthchecks

/collect/OSX/bin/healthcheck.sh

If you get the following warnings in the Healthchecks:

Warning: PV is using an old PV header

Run:

vgck --updatemetadata <VG>

for each VG listed.

NOTE: ss is the replacement for netstat. All scripts must be adapted.

Post steps in case the scripts fail.

Fix DNS resolvers

The DNS resolvers file, /etc/resolv.conf will be emptied after the upgrade, restore a backup of
that file after the upgrade (this is the current workaround in the meanwhile we are investigating
issues with netconfig).

As a normal user run:

sed -i 's/^NETCONFIG_DNS_POLICY=""/NETCONFIG_DNS_POLICY="auto"/'
/etc/sysconfig/network/config

netconfig update -m dns-resolver

ls -l /tmp/resolv.conf.bck

rm /etc/resolv.conf

cp -pv /tmp/resolv.conf.bck /etc/resolv.conf

cat /etc/resolv.conf

ls -l /etc/resolv.conf

Register the server on RMT

Once on SLES 15 SP6, register the server to the RMT with the following command:

SUSEConnect --url http://153.95.208.113

Validate with:

SUSEConnect --status-text

SUSEConnect -l

Install NCPA

Lastly, apply the Saltstack job for NCPA configuration.

salt-call state.apply ncpa

 salt-call state.apply monitoring

systemctl enable ncpa_listener

systemctl restart ncpa_listener

systemctl status ncpa_listener

Fix SSSD

Check that /etc/nsswitch.conf is correctly configured with “sss” instead of “ldap”, as below:

cat /etc/nsswitch.conf

Then, check that PAM is correctly configured too with “pam_sss.so” instead of “pam_ldap.so” as
the latter is deprecated in SLES 15.

grep ldap /etc/pam.d/common-*

Change it, if necessary.

sed -i 's/pam_ldap.so/pam_sss.so/' /etc/pam.d/common-*

Stop and disable NSCD service:

systemctl stop nscd

systemctl disable nscd

systemctl status nscd

And start / enable SSSD:

systemctl start sssd

systemctl enable sssd

systemctl status sssd

Check that it actually started, and if not, fix the errors that may have raised.

If you keep getting errors and can’t get the service up, you can try with:

salt-call state.apply sssd

Fix FTP/SFTP

The service “xinetd” no longer exists in SLES 15 SP6, so we need to fix FTP/SFTP to run with
“vsftpd” service instead.

Make a copy of /etc/vsftpd.conf

cp -pv /etc/vsftpd.conf /tmp/vsftpd.conf.bck

Look for the parameter “listen” in /etc/vsftpd.conf.

grep -i listen /etc/vsftpd.conf

It should be set to “YES”. If it is equal to “NO”, change it.

listen=YES

Also check that “listen_ipv6” is either commented or set to “NO”

listen_ipv6=NO

or

#listen_ipv6=NO

Then, make sure PAM is correctly configured.

Make a backup of the file:

cp -pv /etc/pam.d/vsftpd /tmp/pam_vsftpd.bck

Then:

grep auth /etc/pam.d/vsftpd | grep -v "#"

Look for the next parameter:

auth required pam_shells.so

Change “required” to “sufficient”:

Start the vsftpd service:

systemctl status vsftpd

systemctl start vsftpd

systemctl enable vsftpd

systemctl status vsftpd

Revision History

Revision History

08/12/2024 Jose Miranda Added SSSD Fix section.

08/20/2024 Jose Miranda Added FTP/SFTP Fix section.

09/26/2024 Jose Miranda Added a section for taking snapshot in PowerVC.

Added a section for the post steps scripts after ISO upgrade.