Category: administration

ODA Quickie – How to solve “Validate kernel log level – Failed” problem during ODA patching

svenwellerLeave a comment

During an ODA X7-2S upgrade from 19.9 to 19.13 we encountered the following issue. The prepatch report mentioned that the check “Validate kernel log level” failed with the message

OS kernel log level is set to debug, this may result in a failure when patching Clusterware
If kernel OS log level is more than KERN_ERR(3) then GI patching may fail

This problem also seems to exist in versions 19.10+ . It is a problem that can not be ignored. Trying to update the server anyways will lead to an error.

Here is an example how such a prepatch report might look like

Patch pre-check report
------------------------------------------------------------------------
                 Job ID:  d94c910d-5ed5-4b02-9c65-9e525c176817
            Description:  Patch pre-checks for [OS, ILOM, GI, ORACHKSERVER]
                 Status:  FAILED
                Created:  March 14, 2023 11:52:10 AM CET
                 Result:  One or more pre-checks failed for [GI]


Node Name
---------------
ODA01

Pre-Check                      Status   Comments
------------------------------ -------- --------------------------------------
__OS__
Validate supported versions     Success   Validated minimum supported versions.
Validate patching tag           Success   Validated patching tag: 19.13.0.0.0.
Is patch location available     Success   Patch location is available.
Verify OS patch                 Success   Verified OS patch
Validate command execution      Success   Validated command execution

__ILOM__
Validate supported versions     Success   Validated minimum supported versions.
Validate patching tag           Success   Validated patching tag: 19.13.0.0.0.
Is patch location available     Success   Patch location is available.
Checking Ilom patch Version     Success   Successfully verified the versions
Patch location validation       Success   Successfully validated location
Validate command execution      Success   Validated command execution

__GI__
Validate GI metadata            Success   Successfully validated GI metadata
Validate supported GI versions  Success   Validated minimum supported versions.
Validate available space        Success   Validated free space under /u01
Is clusterware running          Success   Clusterware is running
Validate patching tag           Success   Validated patching tag: 19.13.0.0.0.
Is system provisioned           Success   Verified system is provisioned
Validate ASM in online          Success   ASM is online
Validate kernel log level       Failed    OS kernel log level is set to debug,
                                          this may result in a failure when
                                          patching Clusterware If kernel OS log
                                          level is more than KERN_ERR(3) then
                                          GI patching may fail

Validate minimum agent version  Success   GI patching enabled in current
                                          DCSAGENT version
Validate Central Inventory      Success   oraInventory validation passed
Validate patching locks         Success   Validated patching locks
Validate clones location exist  Success   Validated clones location
Validate DB start dependencies  Success   DBs START dependency check passed
Validate DB stop dependencies   Success   DBs STOP dependency check passed
Evaluate GI patching            Success   Successfully validated GI patching
Validate command execution      Success   Validated command execution

__ORACHK__
Running orachk                  Success   Successfully ran Orachk
Validate command execution      Success   Validated command execution

You can check the setting of the kernel log level like this

[root@ODA01 ~]# cat /proc/sys/kernel/printk

10      4       1       7

The first entry “10” means the loglevel is set to debug. It should be set to “3” (=error).

However changing the /proc/sys/kernel/printk file is not the correct way to solve the issue.

One must edit the file /etc/default/grub and to remove the “debug” entry there.

Then run

grub2-mkconfig -o /boot/efi/EFI/redhat/grub.cfg

Grub is the “grand unified bootloader” that activates when the ODA ( or a VM ) is started. The above command takes the default config and applies it to all relevant vm configurations.

On some ODAs (I believe older ODA X6-2) you might need to apply the change to a different configuration file. On our ODA X7-2S this file was not existent so we did not change it.

grub2-mkconfig -o /boot/grub2/grub.cfg

After this the server needs to be restarted, so that the new setting is applied.

And here is a link to a MOSC thread that helped to solve the issue.
https://community.oracle.com/mosc/discussion/comment/16906486#Comment_16906486

I hope this saves you some time, in case you encounter the same problem.

ORDS 22.x – How to setup and run multiple ORDS on single tomcat

svenweller6 Comments

Problem / Justification

The installation and configuration of ORDS has been changed quite a bit in version 22. Your hitherto existing installation and upgrade process needs to be carefully examined and probably reworked.

Here is an issue I encountered.

Imagine a database with two PDBs. One of them is the normal APEX database (test environment), the other is a regular pdb clone from a different environment, for example from development.

The goal is to run and access a different version of ORDS (and potentially APEX) in each PDB.
This post considers only ORDS, not APEX.

For simplicity we want to access the first environment using “ords” in the url and the second one using “clone”.

In older ORDS versions, it was possible to:

  1. configure ORDS in standalone mode,
  2. rename the war file and
  3. copy it into the webapps folder in tomcat.

And doing the same for the second database we could easily setup two independent versions of ords on a single tomcat.

This process does not function anymore, because the path to the configuration is not added to the ords.war file as it was previously.

Architecture overview

Solution

The following solution assumes that you already upgraded ords, including the ords meta data repository in both databases to the new version. If you don’t have that, look at the prerequisites section and the links near the end of this article.

To run two ords in standalone mode is fairly easy. Use a different port and a different --config parameter while starting ORDS (and possibly starting different versions of ORDS).

It is different for a single tomcat server whichs runs both ORDS using the same port but a different url base path.

We can configure a single ORDS to connect against multiple databases. However this means that the ords version is always the same. The goal is to run a different version of ORDS – so that ORDS matches the metadata repository in the PDB – and to use a slightly different base path. Both do not work when using the same ORDS with the same configuration.

Fortunatly we can recreate the pre 22.x behaviour by creating a new war file.

There is a documented way to create a war file similar to what we were used to. By using the ords war command we are able to inject the setting for the configuration path into a (new) war file. This setting eventually ends up in the web.xml of the tomcat deployment as we will show.

tomcat deployment

Setting the config.url parameter is possible during tomcat startup.

But doing it this way would not allow us to have two ords using a different configuration folder.

export JAVA_OPTS="-Dconfig.url=${ORDS_CONFIG}"

The alternative is to set the config.url parameter inside the web.xml file in each webapp.

We could do that manually but fortunatly oracle gives us a command to create a war file. Using it we can take this war file, put it into the webapps folder on tomcat, and the necessary web.xml file will be created.

war file creation

Warning: the following command will immediately overwrite an existing war file on tomcat in the webapps folder. If you don’t want that then go to a different target folder.

configure ords for PDB1

export PATH=/opt/sven/tst/ords/ords-22.4.0/bin:$PATH
export JAVA_HOME=/usr/java/jdk-17.0.6
cd /opt/sven/tst/tomcat/latest/webapps

ORDS_CONFIG=/opt/sven/tst/ords/ordsconfig/ords
URL_BASE_PATH=ords

ords --config ${ORDS_CONFIG} war ${URL_BASE_PATH}.war

The variables are only created for easy usage and so that they can be easily adapted for the second database.

configure clone for PDB2
export PATH=/opt/sven/tst/ords/ords-22.4.0/bin:$PATH
export JAVA_HOME=/usr/java/jdk-17.0.6
cd /opt/sven/tst/tomcat/latest/webapps
ORDS_CONFIG=/opt/sven/tst/ords/ordsconfig/clone
URL_BASE_PATH=clone

ords --config ${ORDS_CONFIG} war ${URL_BASE_PATH}.war

result: check tomcat deployments

As soon as a new war file is put into the webapps folder, the subdirectories are created. This behaviour depends on the tomcat configuration, but it is the default. So no need to restart tomcat to see the unzipped war files. We can check the result in the webapps/ords/ and webapps/clone/ folders. 

/opt/sven/tst/tomcat/latest/webapps/ords/WEB-INF/web.xml

<?xml version="1.0" encoding="UTF-8" standalone="no"?><web-app xmlns="http://xmlns.jcp.org/xml/ns/javaee" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" id="ORDS" metadata-complete="true" version="3.1" xsi:schemaLocation="http://xmlns.jcp.org/xml/ns/javaee http://xmlns.jcp.org/xml/ns/javaee/web-app_3_1.xsd">
        <display-name>Oracle REST Data Services</display-name>

        <context-param>
                <param-name>config.url</param-name>
                <param-value>/opt/sven/tst/ords/ordsconfig/ords</param-value>
        </context-param><context-param>
                <param-name>version</param-name>
                <param-value>22.4.0.r3401044</param-value>
        </context-param>

        <listener>
                <listener-class>oracle.dbtools.entrypoint.WebApplicationEntryPoint</listener-class>
        </listener>

        <servlet>
                <description>
                </description>
                <display-name>HttpEndPoint</display-name>
                <servlet-name>HttpEndPoint</servlet-name>
                <servlet-class>oracle.dbtools.entrypoint.WebApplicationRequestEntryPoint</servlet-class>
        </servlet>

        <servlet-mapping>
                <servlet-name>HttpEndPoint</servlet-name>
                <url-pattern>/*</url-pattern>
        </servlet-mapping>

        <servlet>
                <description>
                </description>
                <display-name>Forbidden</display-name>
                <servlet-name>Forbidden</servlet-name>
                <servlet-class>oracle.dbtools.entrypoint.Forbidden</servlet-class>
        </servlet>

        <servlet-mapping>

The relevant part in this and the following xml is the setting for the config.url context parameter.

/opt/sven/tst/tomcat/latest/webapps/clone/WEB-INF/web.xml

<?xml version="1.0" encoding="UTF-8" standalone="no"?><web-app xmlns="http://xmlns.jcp.org/xml/ns/javaee" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" id="ORDS" metadata-complete="true" version="3.1" xsi:schemaLocation="http://xmlns.jcp.org/xml/ns/javaee http://xmlns.jcp.org/xml/ns/javaee/web-app_3_1.xsd">
        <display-name>Oracle REST Data Services</display-name>

        <context-param>
                <param-name>config.url</param-name>
                <param-value>/opt/sven/tst/ords/ordsconfig/clone</param-value>
        </context-param><context-param>
                <param-name>version</param-name>
                <param-value>22.4.0.r3401044</param-value>
        </context-param>

        <listener>
                <listener-class>oracle.dbtools.entrypoint.WebApplicationEntryPoint</listener-class>
        </listener>

        <servlet>
                <description>
                </description>
                <display-name>HttpEndPoint</display-name>
                <servlet-name>HttpEndPoint</servlet-name>
                <servlet-class>oracle.dbtools.entrypoint.WebApplicationRequestEntryPoint</servlet-class>
        </servlet>

        <servlet-mapping>
                <servlet-name>HttpEndPoint</servlet-name>
                <url-pattern>/*</url-pattern>
        </servlet-mapping>

        <servlet>
                <description>
                </description>
                <display-name>Forbidden</display-name>
                <servlet-name>Forbidden</servlet-name>
                <servlet-class>oracle.dbtools.entrypoint.Forbidden</servlet-class>
        </servlet>

        <servlet-mapping>

Further considerations

prerequisites

The described solution assumes that you already upgraded ords, including the ords meta data repository in both databases to the new version.

The installation / migration requires to supply the old configuration folder, the new configuration folder and the database connect.

java version

ORDS 22 requires java 11 or 17. We decided to upgrade to java 17 (which is free again). I did not test if it is possible to run different java versions for each ORDS on the same TomCat. This might be a problem in the future, if the minimum java version for ORDS is increased again.

… The Oracle JDK is free to use for any Oracle product requiring Java, which includes ORDS.
That same Java can’t then be used for a non-Oracle product, that could trigger licensing issues.
Of course with Java 17, Oracle Java becomes free again. …

Jeff Smith
https://forums.oracle.com/ords/apexds/post/can-we-use-ords-free-of-cost-if-licensed-with-java-se-9569

automation

If you want to automate the installation and deployment process, below is a link to a blog post by Peter O’Brian that gives some more details about web deployments. It features a python script that is able to create a new .war file that includes the needed config path setting. It might be worth looking at, especially if you want to add more configurable options for automatic deployments.

alternative architecture

If you want to access multiple databases using the same ORDS then consider the following architecture. You will need a single configuration folder with multiple databases (pool configs) in it. I will not go into further details here, because that was not the goal of this post. But if you are interested in how to set it up, then add a comment and I might consider writing an extra post about it.

Useful links

ODA Quickie – How to solve ODABR Error: Dirty bit is set.

svenwellerLeave a comment

The problem

A little while ago during an ODA X7-2S upgrade from 19.6 to 19.9 the following error was encountered.

SUCCESS: 2021-06-04 10:02:05: ...EFI device backup saved as '/opt/odabr/out/hbi/efi.img'
INFO: 2021-06-04 10:02:05: ...step3 - checking EFI device backup
ERROR: 2021-06-04 10:02:05: Error running fsck over /opt/odabr/out/hbi/efi.img
ERROR: 2021-06-04 10:02:05: Command: 'fsck -a /opt/odabr/out/hbi/efi.img' failed as fsck from util-linux 2.23.2 fsck.fat 3.0.20 (12 Jun 2013) 0x25: Dirty bit is set. Fs was not properly unmounted and some data may be corrupt.  Automatically removing dirty bit. Performing changes. /opt/odabr/out/hbi/efi.img: 23 files, 1245/63965 clusters
INFO: 2021-06-04 10:02:05: Mounting EFI back
ERROR: 2021-06-04 10:02:06: Backup not completed, exiting...

This seems to be a known issue for Bare Metal ODAs. But the way to solve the problem is poorly documented.

The mos notes

The Oracle ODABR support document mentions the problem twice and gives slightly different solutions.

Check the “ODABR – Use Case” and the “known issues section”.

https://support.oracle.com/epmos/faces/DocumentDisplay?id=2466177.1

The document also mentions Internal Bug 31435951 ODABR FAILS IN FSCK WITH “DIRTY BIT IS SET”.

From the public ODABR document

This is not an ODABR issue. ODABR is signalling a fsck error because your (in this case) efi partition is not in expected status… 
To fix this:

unmount efi
fsck.vfat -v -a -w <efidevice>
mount efi

Unfortunatly the workaround is a bit vague and hard to understand. The efi partition is mounted as /boot/efi . The “efi device” is not the same as the mount point but can be gathered from that.


Here are the exact commands that helped me to solve the issue.

The solution

First check your filesystem (the output was taken after we repaired the issue) – your mileage may vary.

[root@ODA01 odabr]# df -h
Filesystem                          Size  Used Avail Use% Mounted on
devtmpfs                             94G   24K   94G   1% /dev
tmpfs                                94G  1.4G   93G   2% /dev/shm
tmpfs                                94G  4.0G   90G   5% /run
tmpfs                                94G     0   94G   0% /sys/fs/cgroup
/dev/mapper/VolGroupSys-LogVolRoot   30G   11G   17G  40% /
/dev/mapper/VolGroupSys-LogVolU01   148G   92G   49G  66% /u01
/dev/mapper/VolGroupSys-LogVolOpt    59G   43G   14G  77% /opt
tmpfs                                19G     0   19G   0% /run/user/1001
tmpfs                                19G     0   19G   0% /run/user/0
/dev/asm/commonstore-13             5.0G  367M  4.7G   8% /opt/oracle/dcs/commonstore
/dev/asm/reco-215                   497G  260G  238G  53% /u03/app/oracle
/dev/asm/datredacted-13             100G   28G   73G  28% /u02/app/oracle/oradata/redacted
/dev/asm/datredacted2-13            100G   74G   27G  74% /u02/app/oracle/oradata/redacted2
/dev/md0                            477M  208M  244M  47% /boot
/dev/sda1                           500M  9.8M  490M   2% /boot/efi

This shows us the “efi device” is /dev/sda1

Then we did the steps as described in the documentation:

[root@ODA01 odabr]# umount /boot/efi

[root@ODA01 odabr]# fsck.vfat -v -a -w /dev/sda1
fsck.fat 3.0.20 (12 Jun 2013)
fsck.fat 3.0.20 (12 Jun 2013)
Checking we can access the last sector of the filesystem
0x25: Dirty bit is set. Fs was not properly unmounted and some data may be corrupt.
 Automatically removing dirty bit.
Boot sector contents:
System ID "mkdosfs"
Media byte 0xf8 (hard disk)
       512 bytes per logical sector
      8192 bytes per cluster
        16 reserved sectors
First FAT starts at byte 8192 (sector 16)
         2 FATs, 16 bit entries
    131072 bytes per FAT (= 256 sectors)
Root directory starts at byte 270336 (sector 528)
       512 root directory entries
Data area starts at byte 286720 (sector 560)
     63965 data clusters (524001280 bytes)
63 sectors/track, 255 heads
         0 hidden sectors
   1024000 sectors total
Reclaiming unconnected clusters.
Performing changes.
/dev/sda1: 23 files, 1245/63965 clusters

[root@ODA01 odabr]# mount /boot/efi

After this, we could sucessfully create an ODABR snapshot

[root@ODA01 odabr]# ./odabr backup -snap -osize 50 -usize 80
INFO: 2021-06-04 12:14:49: Please check the logfile '/opt/odabr/out/log/odabr_87615.log' for more details


│▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒│
 odabr - ODA node Backup Restore - Version: 2.0.1-62
 Copyright Oracle, Inc. 2013, 2020
 --------------------------------------------------------
 Author: Ruggero Citton <ruggero.citton@oracle.com>
 RAC Pack, Cloud Innovation and Solution Engineering Team
│▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒│

INFO: 2021-06-04 12:14:49: Checking superuser
INFO: 2021-06-04 12:14:49: Checking Bare Metal
INFO: 2021-06-04 12:14:49: Removing existing LVM snapshots
WARNING: 2021-06-04 12:14:49: LVM snapshot for 'opt' does not exist
WARNING: 2021-06-04 12:14:49: LVM snapshot for 'u01' does not exist
WARNING: 2021-06-04 12:14:49: LVM snapshot for 'root' does not exist
INFO: 2021-06-04 12:14:49: Checking LVM size
INFO: 2021-06-04 12:14:49: Boot device backup
INFO: 2021-06-04 12:14:49: Getting EFI device
INFO: 2021-06-04 12:14:49: ...step1 - unmounting EFI
INFO: 2021-06-04 12:14:50: ...step2 - making efi device backup
SUCCESS: 2021-06-04 12:14:54: ...EFI device backup saved as '/opt/odabr/out/hbi/efi.img'
INFO: 2021-06-04 12:14:54: ...step3 - checking EFI device backup
INFO: 2021-06-04 12:14:54: Getting boot device
INFO: 2021-06-04 12:14:54: ...step1 - making boot device backup using tar
SUCCESS: 2021-06-04 12:15:05: ...boot content saved as '/opt/odabr/out/hbi/boot.tar.gz'
INFO: 2021-06-04 12:15:05: ...step2 - unmounting boot
INFO: 2021-06-04 12:15:05: ...step3 - making boot device backup using dd
SUCCESS: 2021-06-04 12:15:10: ...boot device backup saved as '/opt/odabr/out/hbi/boot.img'
INFO: 2021-06-04 12:15:10: ...step4 - mounting boot
INFO: 2021-06-04 12:15:10: ...step5 - mounting EFI
INFO: 2021-06-04 12:15:11: ...step6 - checking boot device backup
INFO: 2021-06-04 12:15:12: OCR backup
INFO: 2021-06-04 12:15:13: ...ocr backup saved as '/opt/odabr/out/hbi/ocrbackup_87615.bck'
INFO: 2021-06-04 12:15:13: Making LVM snapshot backup
SUCCESS: 2021-06-04 12:15:13: ...snapshot backup for 'opt' created successfully
SUCCESS: 2021-06-04 12:15:15: ...snapshot backup for 'u01' created successfully
SUCCESS: 2021-06-04 12:15:15: ...snapshot backup for 'root' created successfully
SUCCESS: 2021-06-04 12:15:15: LVM snapshots backup done successfully

Side note: We used smaller backup sizes, to circumvent issues with not having enough space for the snapshot, although there was enough space. But this was not connected to the “dirty bit” issue.

I hope this helps others to troubleshoot their ODA.