Today I did have my first presentation at the UKOUG TechFest 2019 in Brighton. Looking back it was a great day with many interesting lectures and good conversations with colleagues and partners. After a long and exhausting day I’ll take a few minutes to sum up my presentation about the PDB isolation and Security.

When you start using Oracle Multitenant, it makes sense to consider a few specific security topics. In general, the same security principles apply to Oracle Multitenant databases as to regular single tenant databases. But depending on the purpose of PDBs, it will be relatively important to implement a few security measures which are beyond of the scope of general database security.

PDB Isolation and Security at a glance

In operation, Oracle container databases use shared resources. These include a backup LAN, management LAN and other system resources. On the other hand, individual Oracle features do access system resources as user oracle. In a common environment, this is basically a security risk. In a DBAAS environment, these can be the following risks.

• PDB admin use privilege escalation.
• Excessive use of shared resources.
• Access sensitive data via shared resources e.g. backup or management LAN.
• Break out of PDB and get OS access as oracle.
• Gain access to the root container (cdb$root)
• Gain access to other PDBs.
• Gain access to the network.
• Use of critical features like.
• Administration features
• Oracle JVM
• DBMS_SCHEDULER
• External table pre-processor

In principle, the owner of each PDB has DBA privileges or equivalent privileges on his PDB. Various measures are taken to ensure security and separation from the operating system and other PDBs. This includes the following measures:

• Provide DBA role respectively customized DBA role to PDB_ADMIN
• Managing OS Access using PDB_OS_CREDENTIAL.
• Manage File Access using PDB PATH_PREFIX and CREATE_FILE_DEST.
• Restrict user operation in PDBs in an Oracle multitenant container database using lockdown profiles.

The following figure shows a diagram of the container database and security measures.

Demo and Engineering Environment

The easiest way to verify the various measures for PDB security, is to use a couple of examples. The demo and test environment for PDB isolation and security presented here is based on Oracle database in Docker containers. The Docker images are build according to the build script in the GitHub repository https://github.com/oehrlis/docker. With a few adjustments the scripts can be run in any Oracle container database. If you follow the Docker-based approach, the following basic images are required:

• Oracle Database 12 Release 2 RU April 2019
• Oracle Database 19c Release Update October 2019 (19.5.0.0)

See https://github.com/oehrlis/docker/tree/master/OracleDatabase for more Information about how to build the Docker images.

Setup Docker Container

If the appropriate docker images are available, an adequate lab environment can be set up within a very short time. Just make sure you update and adjust the docker-compose.yml file to fit your environment, before you setup the docker container. The compose file does contain two database services. One for Oracle 12c Release 2 and on for Oracle 19c (19.5.0.0).

Run docker-compose to create the Docker containers.

docker-compose up -d

Check the progress of you container and database creation

docker-compose logs -f

As soon as you the the following message your database is ready to use.

    tvd122    | ---------------------------------------------------------------

    tvd122    |  - DATABASE TTVD122 IS READY TO USE!

    tvd122    | ---------------------------------------------------------------

    ...

    tvd190    | ---------------------------------------------------------------

    tvd190    |  - DATABASE TTVD190 IS READY TO USE!

    tvd190    | ---------------------------------------------------------------

Test which PDB OS credential require some OS user. To create these user log into either of the container as root and run 01_add_pdb_os_user.sh.

    docker exec -it -u root tvd190 bash --login

    bash-4.2# /u01/config/scripts/01_add_pdb_os_user.sh 

    Found passwd utility

    Skip, group orapdb exists.

    Skip, user orapdb exists.

    Changing password for user orapdb.

    passwd: all authentication tokens updated successfully.

    Add PDB OS user orapdbsec:

    Changing password for user orapdbsec.

    passwd: all authentication tokens updated successfully.

    Add PDB OS user orapdb1:

    Changing password for user orapdb1.

    passwd: all authentication tokens updated successfully.

    Add PDB OS user orapdb2:

    Changing password for user orapdb2.

    passwd: all authentication tokens updated successfully.

    Add PDB OS user orapdb3:

    Changing password for user orapdb3.

    passwd: all authentication tokens updated successfully.

Your now ready to use you Docker base PDB isolation and security environment.

Setup regular Database Environment

Of course, the scripts can also be tested in a regular database environment. You only have to make the following adjustments.

• Create a tnsnames.ora entry for you PDB. e.g. PDBSEC.
• Create the run_id.sh scripts and folders according to 00_prepare_pdb_env.sh Script.
• Create dedicated OS user for the PDB_OS_CREDENTIAL test using the scripts 01_add_pdb_os_user.sh.
• Optional install the patch 25820082
• Adjust the demo script to match your environment e.g. directory path, pdb name etc.

Demo and Test Scripts

The demo scripts are located in the script folder. In general, they work for both 12.2 and 19c, except the lockdown profile create script.

• 00_prepare_pdb_env.sh Script to add a tnsname.ora entry and other stuff for the PDB PDBSEC.
• 01_add_pdb_os_user.sh Script to add a PDB OS user.
• 10_create_pdb.sql Create a PDB (pdbsec) used for PDB security engineering.
• 20_create_directories.sql Script to create directories.
• 30_create_datafile.sql Script to create datafiles.
• 40_create_PDB_OS_CREDENTIAL.sql Script to configure PDB_OS_CREDENTIAL.
• 41_create_ext_table.sql Script to configure table pre-processors.
• 42_create_scheduler_job.sql Script to configure external OS jobs.
• 50_create_lockdown_profiles_12.2.sql Script to create lockdown profiles for 12.2.
• 50_create_lockdown_profiles.sql Script to create lockdown profiles.
• 51_lockdown_trace_view.sql Script to create lockdown profiles for trace files.
• 53_lockdown_external_table.sql Script to create lockdown profiles external table.
• 60_dbms_sys_sql_test.sql Script to verify DBMS_SYS_SQL
• 90_drop_pdb.sql Drop PDB (pdbsec) used for PDB security engineering
• ld_profiles.sql Displays information about lockdown profiles.
• ld_rules.sql Displays information about lockdown rules in the current container.
• lpdb.sql List PDBS

Slides of the lecture

The slides for the lecture have been uploaded to slideshare. They do provide a few information about the idea and the concept behind the PDB isolation and security.

It is way too long ago since my last blog post. These were or are busy weeks for me. Any way, I finally found some time to start writing a blog post about a special setup for kerberos authentication of Oracle databases. It is about configuring kerberos authentication for multiple database servers with only one active directory account and corresponding Service Priciple Names (SPN). Additionally there is an challenge, that the keytab file should only be created with ktutil directly on the DB server. Access to a Windows server and use of ktpass.exe is not possible. I did setup a nice test case on a couple of compute instances on Oracle cloud infrastructure. During the verification of the test setup I had to realise that the kerberos authentication does not work as planned. Until now it is not possible to create a keytab file with ktutil that I can use successfully with Active Directory. The same kerberos configuration with a keytab create with ktpass.exe on the AD server does work. But that’s on other story…

The aim of this blog post is to sum up a couple of troubleshooting actions I came across. Kerberos itself is around since a couple of decades. Therefore you will find various documentation, RFC, etc. But it is not always easy to recognise what is still relevant and what not. Mainly because the implementation of Kerberos at both Oracle and Microsoft is not necessarily the same or 100% MIT Kerberos compliant. The fact that there are different versions of Oracle, MS AD and Kerberos makes it even more exciting

Basics

A basic requirement for Kerberos is the network and time configuration.

• Problem: okinit does fail with clock skew too great
• Cause: The systems involved must be synchronous in terms of system time e.g. using a NTP service to configure date / time. If the system times differ to much you will receive this error when using okinit.
• Solution: Configure proper system times using NTP service. Small time drifts can be covered by setting SQLNET.KERBEROS5_CLOCKSKEW=300 in sqlnet.ora

• Problem: Miscellaneous errors due to wrong / missing network configuration.
• Cause: Using CNAME rather A records, no DNS configuration, no revers lookkup etc
• Solution: Configure proper DNS name resolution for database service as well MS active directory service. Each system must be able to be resolved by name or IP address. Kerberos will look for service principle names based on A records.

oracle@db:/u00/app/oracle/network/admin/ [TDB190S] cd

oracle@db:~/ [TDB190S] nslookup win2016ad.trivadislabs.com

Server:     10.0.1.4

Address:    10.0.1.4#53



Name:   win2016ad.trivadislabs.com

Address: 10.0.1.4



oracle@db:~/ [TDB190S] nslookup 10.0.1.4

4.1.0.10.in-addr.arpa   name = win2016ad.trivadislabs.com.



oracle@db:~/ [TDB190S] nslookup db

Server:     10.0.1.4

Address:    10.0.1.4#53



db.trivadislabs.com canonical name = ol7db19.trivadislabs.com.

Name:   ol7db19.trivadislabs.com

Address: 10.0.1.6



oracle@db:~/ [TDB190S] nslookup 10.0.1.6

6.1.0.10.in-addr.arpa   name = ol7db19.trivadislabs.com.

Trace and Log Files

Kerberos Trace

As of Oracle 12c release 2 it is possible to enable kerberos tracing by setting KRB5_TRACE to a trace file. This logs the Kerberos calls in the current session.

export KRB5_TRACE=/u00/app/oracle/network/admin/kerberos.trc

oracle@db:~/ [TDB190S] okinit king



Kerberos Utilities for Linux: Version 19.0.0.0.0 - Production on 08-JUN-2020 20:54:16



Copyright (c) 1996, 2019 Oracle.  All rights reserved.



Configuration file : /u00/app/oracle/network/admin/krb5.conf.

Password for king@TRIVADISLABS.COM:

A sample output of a kerberos trace file:

oracle@db:~/ [TDB190S] head -10 /u00/app/oracle/network/admin/kerberos.trc

[5645] 1591649656.590082: Getting initial credentials for king@TRIVADISLABS.COM

[5645] 1591649656.590084: Sending unauthenticated request

[5645] 1591649656.590085: Sending request (199 bytes) to TRIVADISLABS.COM

[5645] 1591649656.590086: Resolving hostname ad.trivadislabs.com

[5645] 1591649656.590087: Sending initial UDP request to dgram 10.0.1.4:88

[5645] 1591649656.590088: Received answer (196 bytes) from dgram 10.0.1.4:88

[5645] 1591649656.590089: Sending DNS URI query for _kerberos.TRIVADISLABS.COM.

[5645] 1591649656.590090: No URI records found

[5645] 1591649656.590091: Sending DNS SRV query for _kerberos-master._udp.TRIVADISLABS.COM.

[5645] 1591649656.590092: Sending DNS SRV query for _kerberos-master._tcp.TRIVADISLABS.COM.

Oracle SQLNet tracing

For Kerberos troubleshooting with Oracle SQLNet it is helpful to disable ADR tracing. Not mandatory, but makes life a bit easier. Set DIAG_ADR_ENABLED in sqlnet.ora to OFF.

DIAG_ADR_ENABLED=OFF

Before KRB5_TRACE was available, okinit calls could only be traced with sqlnet.ora and TRACE_LEVEL_OKINIT. See also MOS note 162668.1. The parameter does not make sense when you already use KRB5_TRACE.

TRACE_LEVEL_OKINIT=SUPPORT

TRACE_DIRECTORY_OKINIT=/u00/app/oracle/network/

TRACE_FILE_OKINIT=okinit.trc

For further analysis you usually have to switch on SQLNet Tracing. Don’t even thing about setting an other level than SUPPORT (16). Kerberos calls are only available with the highest level.

TRACE_LEVEL_OKINIT=SUPPORT

TRACE_DIRECTORY_OKINIT=/u00/app/oracle/network/

TRACE_FILE_OKINIT=okinit.trc

Enable tracing for SQLNet clients:

TRACE_LEVEL_CLIENT=SUPPORT

TRACE_DIRECTORY_CLIENT= /u00/app/oracle/network/trc

TRACE_FILE_CLIENT=sqlnet_client.trc

Enable tracing for SQLNet Server:

TRACE_LEVEL_SERVER=SUPPORT

TRACE_DIRECTORY_SERVER= /u00/app/oracle/network/trc

TRACE_FILE_SERVER=sqlnet_server.trc

The errors in the trace files are not always obvious. You can find a few infos and hint in MOS note 185897.1. But most of the time there is no way around searching for the corresponding error or function call in Oracle Support or the search engine of choice.

Network Tracing

The next level is to trace the network calls. Depending on the environment you can directly use Wireshark. But it is much easier to first create a network dump via command line and to analyse it later using Wireshark. I use tcpdump on my OCI environment and download the trace file to my MacBook, where I then use Wireshark.

Get the available interfaces:

sudo tcpdump -D

1.nflog (Linux netfilter log (NFLOG) interface)

2.nfqueue (Linux netfilter queue (NFQUEUE) interface)

3.usbmon1 (USB bus number 1)

4.ens3

5.any (Pseudo-device that captures on all interfaces)

6.lo [Loopback]

Start tracing for interface ens3:

sudo tcpdump -i ens3 -s 65535 -w /tmp/network_okcreate.trc

Keep it running until while testing the kerberos authentication. As soon as done copy the trace file to the client an open it using Wireshark. The following picture does show a trace dump where the kerberos protocol has been selected.

A part of the kerberos packet is encrypted and not visible as you can see in following picture.

Kerberos does use the service’s secret key to encrypt these messages. You can import the keytab file into Wireshark to decrypt the messages. For this purpose the keytab file must be specified in Wireshark in the preferences. Click Edit > Preferences > Protocols > KRB5.

You now see the message content of the packet. This is in particular useful when you have to analyse issues related to ticket size, missing groups etc.

Conclusion

Unfortunately my Kerberos problem is still not solved. Nevertheless I did get the opportunity to practice a couple of Kerberos tracing methods. The introduction of KRB5_TRACE did simplify tracing a bit, but in most case you still have to use SQLNet or network tracing to find the root cause of you Kerberos problem. A direct solution is unfortunately not always found with tracing. At least you have all the relevant information to search My Oracle Support, open a service request or try your luck at googling for a solution.

Good luck with your Kerberos setup.

References

Some links related to this blog post:

• Kerberos Troubleshooting Guide [185897.1]
• Master Note For Kerberos Authentication [1375853.1]
• How to Trace Unix System Calls [110888.1]
• Tracing Okinit [162668.1]
• How to Enable Oracle SQL*Net Client, Server, Listener, Kerberos and External procedure Tracing from Net Manager [395525.1]
• Requesting kerberos TGT with OKINT errors with okinit: Clock skew too great in 12.1.0.2 [2312008.1]

When it comes to the conception and implementation of a central user administration of Oracle databases, authentication is one of the central topics. Often there is a need for integration with an existing directory service or IAM solution. Whereby usually MS Active Directory is involved. But Oracle Databases and MS Active Directories are not yet best friends. In this blog post we will explain why this is so, with a focus on authentication.

A few Basics

Oracle Databases does provide a couple of authentication methods. This includes among others the following:

• Password authentication
• OS authentication
• Kerberos authentication
• SSL authentication

All methods have their advantages and disadvantages and thus their justification. But now let’s speak about password authentication. This works basically always the same way, no matter if database or directory based. The picture below shows the schematic diagram of the password authentication process.

• The user does send the logon request with its username to the database.
• The database generates a session key, to encrypt communication.
• The client generates the password hash and sends it encrypted to the DB server.
• The database now compares the password hashes.
  • either the hash from USER$
  • or the hash from the directory server

The key aspect is that the database always verifies the password hashes. Either with the hash in the database or, in case of directory-based authentication, with the hash from the directory. The process is used when using Oracle Centrally Managed Users (CMU) but also with Oracle Enterprise User Security in combination with an Oracle Directory e.g. Oracle Unified Directory EUS AD Proxy. In case of a regular LDAP directory, the hash is read from userPassword or another attribute. However, this is not possible in MS Active Directory, where passwords are stored internally in the Security Account Manager (SAM) and cannot be read directly. This is one of the reasons why Active Directory is not fully LDAP v3 compliant. But that is an other story

This is now the moment where the Oracle password filter comes into play. Microsoft does provide a functionality within Windows called password filter. These filters provide a way to implement password policies and change notification. When a password change request is made, the Local Security Authority (LSA) calls the password filters registered on the system. Each password filter is called twice: first to validate the new password and then, after all filters have validated the new password, to notify the filters that the change has been made. The following illustration shows this process.

Oracle Password Filter

The Oracle password filter solves relatively simply the problem that the hash cannot be read. The filter uses the password change notification and stores the password accordingly in an additional LDAP attribute. The database or directory server on the other hand is then able to read the user password hash. Oracle Databases and Active Directory starts to like each other . But usually Windows or Security Admins are not so happy any more. The fact that a foreign DLL has to be installed on the domain controller sometimes causes headaches or just endless discussions…

The latest version of the password filter is delivered as EXE file opwdintg.exe. It is part of any Oracle Database binaries as of release 18c. Older version of Oracle Database, Oracle Internet Directory and Oracle Unified Directory do also include the password filter in an other form e.g. setup.exe or a jar file. Nevertheless it is crucial, that you get the latest version which is right now part of Oracle Database 19.8.0.0. This is also the valid version when you use OUD or OID, see MOS Note 2640135.1 How to Get the Latest oidpwdcn.dll (New Name orapwdfltr.dll). Alternatively you can also download a generic patch 23191994 for fusion middleware.

But what exactly happens when you install the Oracle password filter? Oracle performs the following steps during installation:

• Add an Active Directory schema extension for an additional user attribute orcleCommonAttribute. Once installed a schema extension can not be removed any more.
• Create some generic groups to control the password filter plugin. The filter will only update the orcleCommonAttribute attribute for users which are part directly or indirectly of one of the group.
  • ORA_VFR_MD5 is required when the Oracle Database WebDAV client is used
  • ORA_VFR_11G enables the use of the Oracle Database 11G password verifier
  • ORA_VFR_12C enables the use of the Oracle Database 12C password verifier
• Install the Oracle password filter DLL orapwdfltr.dll. This requires a reboot of the domain controller.

The following screenshots show the installation of the Oracle password filer.

After a reboot the installation of the Oracle password filter is finished. Now let’s see what’s new there. First we review the AD schema change. This can be done by starting the Microsoft Management Console (MMC) and open the Active Directory Schema Snap-In. See the old documentation install the Schema Snap-In if the snap-in is not available. The following screenshot does show the details about the new attribute.

In the registry we see under LSA an additional entry for the notification packages. orapwdfltr the name of the DLL installed on the domain server.

And finally the new groups and the new attribute orcleCommonAttribute.

Although the attribute orcleCommonAttribute in the picture below does only get propagated after a password reset. The user KING is part of the group Trivadis LAB Users. This group itself is member of ORA_VFR_11G.

Yeah, but it’s an Oracle tool…

In one of my many conversations with customers about these password filters I was asked if they could examine the source code. Mmm, no! It is quite common that neither Oracle nor Microsoft publish their source code. In this case Oracle uses an API or functionality defined and documented by Microsoft. But this does not convince everyone. That’s why I have tried to investigate this in detail. One of my first attempts was a test if I can decompile the DLL. This would be possible if it is written in .net or something similar, but not with C or C++. You can use an online disassembler, but the result will not help you.

Analysis of the executable installation file opwdintg.exe with exiftool, reveal that it is only a self extracting cabinet.

exiftool opwdintg.exe 

ExifTool Version Number         : 12.00

File Name                       : opwdintg.exe

Directory                       : .

File Size                       : 193 kB

File Modification Date/Time     : 2020:09:04 06:17:15+02:00

File Access Date/Time           : 2020:09:04 06:18:43+02:00

File Inode Change Date/Time     : 2020:09:04 06:17:15+02:00

File Permissions                : rw-r--r--

File Type                       : Win64 EXE

File Type Extension             : exe

MIME Type                       : application/octet-stream

Machine Type                    : AMD AMD64

Time Stamp                      : 2013:10:14 08:48:22+02:00

Image File Characteristics      : Executable, Large address aware

PE Type                         : PE32+

Linker Version                  : 11.0

Code Size                       : 32768

Initialized Data Size           : 163840

Uninitialized Data Size         : 0

Entry Point                     : 0x7f1c

OS Version                      : 6.3

Image Version                   : 6.3

Subsystem Version               : 5.2

Subsystem                       : Windows GUI

File Version Number             : 11.0.9600.16428

Product Version Number          : 11.0.9600.16428

File Flags Mask                 : 0x003f

File Flags                      : (none)

File OS                         : Windows NT 32-bit

Object File Type                : Executable application

File Subtype                    : 0

Language Code                   : English (U.S.)

Character Set                   : Unicode

Company Name                    : Microsoft Corporation

File Description                : Win32 Cabinet Self-Extractor

File Version                    : 11.00.9600.16428 (winblue_gdr.131013-1700)

Internal Name                   : Wextract

Legal Copyright                 : © Microsoft Corporation. All rights reserved.

Original File Name              : WEXTRACT.EXE            .MUI

Product Name                    : Internet Explorer

Product Version                 : 11.00.9600.16428

You can invoke the executable with two additional parameters C and T to extract the content into the directory specified with T.

c:\vagrant>opwdintg.exe /C /T:c:\vagrant\opwdintg

In the directory you will find three files:

• instpflt.bat Batch file used to install the password filter.
• etadschm.bat Batch file used to do the schema extension for orcleCommonAttribute and create the 3 AD groups.
• orapwdfltr.dll the Oracle password filter dll itself.

Even if you cannot decompile orapwdfltr.dll, you can still examine the batch files. As expected, the batch files do exactly what we have already verified graphically above. Schema extension, create groups and register Oracle password filter.

With pev, a PE file analysis toolkit, we can check other stuff like the functions exported by the DLL. As you can see in the output below, the functions correspond to Microsoft’s specifications for password filters. An indication that the DLL does what it should. However, pev provides other tools to analyse the DLL, hashes, import functions etc. But we will skip that at this point.

readpe --exports orapwdfltr.dll 

Exported functions

    Library

        Name:                            orapwdfltr.dll

        Functions

            Function

                Ordinal:                         1

                Address:                         0x1080

                Name:                            InitializeChangeNotify

            Function

                Ordinal:                         2

                Address:                         0x2ea0

                Name:                            PasswordChangeNotify

            Function

                Ordinal:                         3

                Address:                         0x1080

                Name:                            PasswordFilter

A few words about Security

But what about security? There are basically two aspects. First, the fact that the DLL is a rather critical component. There are known malware that exploit exactly this method to get the passwords. It is therefore a best practice to configure LSA security to allow only signed DLLs for LSA. Ok, besides that you should also know which DLL you have installed and why. But here we are at the point where it gets a bit difficult. Oracle has forgotten to sign orapwdfltr.dll in the past. Therefore, if LSA security is enabled the password filter will not work. See also MOS note 2612535.1 or 2616566.1. Among other things, Oracle has proposed to turn off the LSA security. Certainly not the way to go. But luckily there is already a bug 31134430 and patch 23191994 available for this issue. The fix does include a signed version of the orapwdfltr.dll, as you can see in the following code block.

signtool.exe verify /pa /v orapwdfltr.dll



Verifying: orapwdfltr.dll



Signature Index: 0 (Primary Signature)

Hash of file (sha256): 2A14712107D424FF5577EF5C3D111CF66DB40F6226047ADC4F31389D69F437EB



Signing Certificate Chain:

 Issued to: VeriSign Class 3 Public Primary Certification Authority - G5

 Issued by: VeriSign Class 3 Public Primary Certification Authority - G5

    Expires:   Wed Jul 16 16:59:59 2036

    SHA1 hash: 4EB6D578499B1CCF5F581EAD56BE3D9B6744A5E5



 Issued to: Symantec Class 3 Extended Validation Code Signing CA - G2

 Issued by: VeriSign Class 3 Public Primary Certification Authority - G5

        Expires:   Sun Mar 03 16:59:59 2024

        SHA1 hash: 5B8F88C80A73D35F76CD412A9E74E916594DFA67



    Issued to: Oracle America Inc.

    Issued by: Symantec Class 3 Extended Validation Code Signing CA - G2

            Expires:   Wed Jan 27 16:59:59 2021

            SHA1 hash: 1CB08E9B70B917E64407A4F2665799D58B171F89



The signature is timestamped: Wed Apr 22 18:33:05 2020

Timestamp Verified by:

    Issued to: DigiCert Assured ID Root CA

    Issued by: DigiCert Assured ID Root CA

    Expires:   Sun Nov 09 17:00:00 2031

    SHA1 hash: 0563B8630D62D75ABBC8AB1E4BDFB5A899B24D43



        Issued to: DigiCert SHA2 Assured ID Timestamping CA

        Issued by: DigiCert Assured ID Root CA

        Expires:   Tue Jan 07 05:00:00 2031

        SHA1 hash: 3BA63A6E4841355772DEBEF9CDCF4D5AF353A297



            Issued to: TIMESTAMP-SHA256-2019-10-15

            Issued by: DigiCert SHA2 Assured ID Timestamping CA

            Expires:   Wed Oct 16 17:00:00 2030

            SHA1 hash: 0325BD505EDA96302DC22F4FA01E4C28BE2834C5



Successfully verified: orapwdfltr.dll



Number of files successfully Verified: 1

Number of warnings: 0

Number of errors: 0

Alternatively you can also check the windows property of orapwdfltr.dll.

The other security challenge is the password hash itself. In a regular LDAP, ACIs are usually defined to restrict access to password attributes. However, no ACIs are defined when installing the Oracle password filter. It is therefore strongly recommended to restrict access to this attribute. Generally only the Oracle service accounts, which are used to setup Oracle AD integration, has to read it.

Conclusion

When my workmate Martin Berger published his blog post about the issue with LSA and the password filter, there was no official solution beside disabling LSA security. Fortunately it is a bit better in the meantime. The bug fix did found its way in the latest release of Oracle Database 19c (19.8.0.0) and in the generic fusion middleware patch 23191994. This official signed version of the password filter can be used for either Oracle Centrally Managed Users (CMU), Oracle Enterprise User Security (EUS) or Oracle Unified Directory DIP. It is a fact that this password filter means a change on the domain server. Every change represents a potential risk. Nevertheless, this change is comprehensible and is, according to Microsoft, a documented procedure. By carefully assigning the Oracle groups (ORA_VFR_11g, ORA_VFR_11C, etc), you can ensure that only those users who need the hash in orcleCommonAttribute have set it. It is also recommended to define ACIs to limit access to orcleCommonAttribute restrictively.

SSL and Kerberos authentication are basically secure methods. Additionally these authentication methods allow Single Sign On. Unfortunately, practice shows that many tools cannot handle this. Password authentication on the other hand offers greater flexibility. The Oracle password filter is not bad nor dangerous. In my humble opinion it is worth to consider this solution.

References

A few links related to this blog post:

• Blog post by my workmate Martin Berger about Oracle EUS authentication with LSA activated on AD Thanks for bringing up this topic at Trivadis as well with Oracle Support
• Oracle Enterprise User Security AD Integration
• Oracle Support Note 2640135.1 OUD 12c – How to Get the Latest oidpwdcn.dll (New Name orapwdfltr.dll)
• Oracle Support Note 2612535.1 EUS Login Failure of AD Users Proxied by OUD: LdapErr: DSID-0C090CE0, comment: Error in attribute conversion operation
• Oracle Support Note 2616566.1 OUD 11g – OIDPWDCN.DLL Plug-in Fails On AD 2012 R2 With Error “The password notification DLL oidpwdcn failed to load with error 577”
• Oracle Bug 31134430 need to have orapwdfltr.dll signed by Microsoft.
• Oracle Patch 23191994 “signed” version of the oidpwdcn.dll for Oracle Unified Directory.
• Microsoft Windows Dev Center Password Filter Programming Considerations  
• Microsoft Windows Dev Center Installing and Registering a Password Filter DLL  
• Microsoft Windows Dev Center Password Filter 
• Microsoft Windows Dev Center Password Filter Functions 
• pev the PE file analysis toolkit
• ExifTool by Phil Harvey
• stackoverflow discussion about the signtool

Since I’m always short of time for a longer blog post, I’ll just try a short one. Intended as a mini-series, I will show different configuration examples for Oracle Enterprise User Security. Today I’ll start with the configuration of EUS based proxy privileges. The environment I use is DOE, my Docker based Oracle Engineering environment. In particular the EUS configuration. For more information, see the corresponding GitHub repository oehrlis/doe respectively in the folder eus for the EUS specific environment.

Background

Database proxy privileges are used relatively often to give certain users rights to access a different schema. The user authenticates himself with his credentials and becomes a proxy user in the database. Below an example where the user RMAN, gets access to a different schema, specifically an other RMAN catalog schema (see also blog post about SEPS and RMAN).

CREATE USER rman IDENTIFIED BY welcome1;

CREATE USER rman19000 NO AUTHENTICATION QUOTA UNLIMITED ON rman_data;

GRANT RECOVERY_CATALOG_OWNER TO rman19000;

ALTER USER rman19000 GRANT CONNECT THROUGH rman; 

ALTER USER rman19000 DEFAULT TABLESPACE rman_data;

The following users were created

• RMAN1900 is the schema owner for an Oracle 19c RMAN catalog stored in the tablespace RMAN_DATA. The user is created without any authentication but with a proxy privilege for the user RMAN.
• RMAN is the user which will be used to connect to the catalog. There are other catalogs as well but not shown in this example

SQL> CONNECT rman[RMAN19000]/welcome1@CATALOG

Connected.

SQL> SHOW USER

USER IS "RMAN19000"

SQL> SELECT sys_context('userenv','PROXY_USER') PROXY_USER, sys_context('userenv','SESSION_USER') SESSION_USER FROM dual;



PROXY_USER SESSION_USER

---------- ---------------

RMAN       RMAN19000D

With pure database authentication or authorisation, the configuration of proxy users is easy. With Enterprise User Security, proxy privileges are no longer managed in the database but in the directory. Let’s take a look at that.

Database Configuration

For Enterprise User Security based proxy privileges, only ENTERPRISE USERS is specified in the database. The rest is done in the OracleContext of the directory. See also ALTER USER in Oracle® Database SQL Language Reference 19c.

ALTER USER scott GRANT CONNECT THROUGH ENTERPRISE USERS;

Enterprise User Security Configuration

The configuration can be either done via Oracle Enterprise Manager Cloud Control as documented in Oracle® Database Enterprise User Security Administrator’s Guide 19c or with the command line utility eusm. I prefer the command line utility as I often do not have an OEM by hand.

• Create the proxy permission in the directory.

eusm createProxyPerm proxy_permission="Scott Proxy" \

domain_name="OracleDefaultDomain" \

realm_dn="dc=trivadislabs,dc=com" ldap_host=eusoud.trivadislabs.com ldap_port=1389 \

ldap_user_dn=cn=eusadmin,cn=oraclecontext ldap_user_password=$(cat /u01/common/etc/eusadmin_pwd.txt)

• Define a target user for this proxy permission.

eusm addTargetUser proxy_permission="Scott Proxy" \

database_name="TEUS01" \

target_user="SCOTT" dbuser="system" dbuser_password=$(cat /u00/app/oracle/admin/TEUS01/etc/TEUS01_password.txt) \

dbconnect_string="eusdb.trivadislabs.com:1521/TEUS01.trivadislabs.com" \

domain_name="OracleDefaultDomain" \

realm_dn="dc=trivadislabs,dc=com" ldap_host=eusoud.trivadislabs.com ldap_port=1389 \

ldap_user_dn=cn=eusadmin,cn=oraclecontext ldap_user_password=$(cat /u01/common/etc/eusadmin_pwd.txt)

• Explicit granting of proxy permission to the user KING. Can also be assigned to a group.

eusm grantProxyPerm proxy_permission="Scott Proxy" \

user_dn="cn=Ben King,ou=Senior Management,ou=People,dc=trivadislabs,dc=com" \

domain_name="OracleDefaultDomain" \

realm_dn="dc=trivadislabs,dc=com" ldap_host=eusoud.trivadislabs.com ldap_port=1389 \

ldap_user_dn=cn=eusadmin,cn=oraclecontext ldap_user_password=$(cat /u01/common/etc/eusadmin_pwd.txt)

• Display the proxy permissions defined for the EUS default domain.

eusm listProxyPermissions domain_name="OracleDefaultDomain" \

realm_dn="dc=trivadislabs,dc=com" ldap_host=eusoud.trivadislabs.com ldap_port=1389 \

ldap_user_dn=cn=eusadmin,cn=oraclecontext ldap_user_password=$(cat /u01/common/etc/eusadmin_pwd.txt)

• Display information for the proxy permission Scott Proxy

eusm listProxyPermissionInfo proxy_permission="Scott Proxy" \

domain_name="OracleDefaultDomain" \

realm_dn="dc=trivadislabs,dc=com" ldap_host=eusoud.trivadislabs.com ldap_port=1389 \

ldap_user_dn=cn=eusadmin,cn=oraclecontext ldap_user_password=$(cat /u01/common/etc/eusadmin_pwd.txt)

• Display proxy permissions for the user KING.

eusm listProxyPermissionsOfUser \

user_dn="cn=Ben King,ou=Senior Management,ou=People,dc=trivadislabs,dc=com" \

realm_dn="dc=trivadislabs,dc=com" ldap_host=eusoud.trivadislabs.com ldap_port=1389 \

ldap_user_dn=cn=eusadmin,cn=oraclecontext ldap_user_password=$(cat /u01/common/etc/eusadmin_pwd.txt)

Using the Proxy Permissions

Let’s test the permissions and connect as user KING.

• Regular connection to the database as schema owner SCOTT.

SQL> CONNECT SCOTT/tiger@TEUS01

Connected.

SQL> SHOW USER

USER IS "SCOTT"

SQL> SELECT sys_context('userenv','PROXY_USER') PROXY_USER , sys_context('userenv','SESSION_USER') SESSION_USER FROM dual;



PROXY_USER  SESSION_USER

--------------- ---------------

                SCOTT

• Regular connection to the database as KING.

SQL> CONNECT king/welcome1@TEUS01

Connected.

SQL> SHOW USER

USER IS "KING"

SQL> SELECT sys_context('userenv','PROXY_USER') PROXY_USER , sys_context('userenv','SESSION_USER') SESSION_USER FROM dual;



PROXY_USER  SESSION_USER

--------------- ---------------

                KING

• Proxy connection to the database

SQL> CONNECT king[SCOTT]/welcome1@TEUS01

Connected.

SQL> SHOW USER

USER IS "SCOTT"

SQL> SELECT sys_context('userenv','PROXY_USER') PROXY_USER , sys_context('userenv','SESSION_USER') SESSION_USER FROM dual;



PROXY_USER  SESSION_USER

--------------- ---------------

KING            SCOTT

Conclusion

Configuration of proxy permissions in connection with Oracle Enterprise User Security is not as complicated as you might think. It is also useful if shared global users need access to certain schemas. For example, a power user is allowed to access the application schema.

• Oracle® Database SQL Language Reference 19c ALTER USER and CREATE USER
• Oracle® Database Enterprise User Security Administrator’s Guide 19c Granting Proxy Permissions to Enterprise Users
• Oracle® Database Enterprise User Security Administrator’s Guide 19c Enterprise User Security Manager (EUSM) Command Reference

Today I held my presentation about Oracle security best practice “Oracle passwords, but secure!” at the virtual UKOUG event. Unfortunately, this year the beautiful view of Brighton beach and the active exchange with colleagues was missing. Ok, on the other hand I was able to enjoy the first snow in Switzerland with my children.

The following blog post is a summary of my presentation with some examples, notes, references and slides.

Oracle Password Hashes

The different Oracle Database releases do provide various password verifiers. Although the older password verifiers are no longer state of the art, they are still used relatively frequently. It is therefore essential to take the appropriate measures to make password-based authentication secure. Oracle currently offers the following password hash functions:

• Oracle 10g Hash Function based on DES and an Oracle specific algorithm. It is case insensitive and does use a weak password salt i.e. the username is used as salt.
• MD5 based Hash Function used for digest authentication in XDB
• Oracle 11g Hash Function based on the SHA1 hash algorithm. But since 2005 SHA1 is no longer considered as safe. The hash function does supports case sensitive and multibyte character passwords.
• Oracle 12c Hash Function based on a de-optimised algorithm involving PBKDF2 and SHA-512. It supports case sensitive and multibyte character passwords.

The different password verifiers can be controlled by SQLNET.ALLOWED_LOGON_VERSION_SERVER respectively SQLNET.ALLOWED_LOGON_VERSION_CLIENT or by setting the passwords explicitly using ALTER USER ... IDENTIFIED BY VALUES.

Create different users with different password verifiers

CREATE USER test_10g IDENTIFIED BY VALUES 'AF310E4D20D06950';
CREATE USER test_11g IDENTIFIED BY VALUES 'S:6702B83E88D277BFC378AD6B22DD1AE01895A254470F8124A9D3C5347056';
CREATE USER test_12c IDENTIFIED BY VALUES 'T:45738A7B75C9E31ED0C533BCF4931084658A143FD7CF826B980A88EA6C4F0BE66C28DA7085BCAE386723029BA967DC4F45E9C146F6FA7C22E44BA2C1BD2F56F8C22291D417E26D4B810003F3F055EDFF';
CREATE USER test_all IDENTIFIED BY Welcome1;

In DBA_USERS you will see the different password versions

SET LINESIZE 160 PAGESIZE 200
COL username FOR a10
COL password_versions FOR a20
SELECT username, password_versions 
FROM dba_users WHERE username LIKE 'TEST_%';

USERNAME    PASSWORD_VERSIONS
----------- --------------------
TEST_10G    10G
TEST_11G    11G
TEST_ALL    10G 11G 12C
TEST_12C    12C

Or in USER$ you can find the corresponding hashes:

SET LINESIZE 160 PAGESIZE 200
COL name FOR a10
COL password FOR a16
COL spare4 FOR a64
SELECT name,password,spare4 
FROM user$ WHERE name LIKE 'TEST_%' ORDER BY 1;

NAME       PASSWORD         SPARE4
---------- ---------------- ----------------------------------------------------------------
TEST_10G   AF310E4D20D06950
TEST_11G                    S:6702B83E88D277BFC378AD6B22DD1AE01895A254470F8124A9D3C5347056
TEST_12C                    T:45738A7B75C9E31ED0C533BCF4931084658A143FD7CF826B980A88EA6C4F0B
                            E66C28DA7085BCAE386723029BA967DC4F45E9C146F6FA7C22E44BA2C1BD2F56
                            F8C22291D417E26D4B810003F3F055EDFF

TEST_ALL   4932A1B4C59EC3D0 S:ABF25107166264C8EAFE72BF02152DE17000F359CB5BAF21A6AF41477633;T
                            :62FEE108652A56D940813F54EC72D1494ACAD99F2BBDD0A578BF1F97FAB4A7E
                            B468A98B6B553E460DE21E57F6C35A930DEE027D20B33ED13D56EA0ECACB1CEA
                            94EEC8AC389561346052BB0BFF2C06647

Manually create a Oracle 10g password verifier:

SQL> @create_password_hash.sql system ieShae0

Username : system
Password : ieShae0
Hash	 : 0AD56CF5F1CB8D2A
SQL	 : alter user system identified by values '0AD56CF5F1CB8D2A';

PL/SQL procedure successfully completed.

Testing the Password Verifier

There are a couple of possibilities and tools to “verify” password hashes. Among the best known are the tools Hashcat and John the Ripper. These tools doe support a wide range of hashes as well attack methods. Below you find an example of a brute force attack for the Oracle hash we created above.

• --increment will start to brute force with shorter length e.g 4 characters
• –-custom-charset1 to define numbers and characters
• –-hash-type Oracle 7+ respectively password verifier 10g
• --show show the password

echo "0AD56CF5F1CB8D2A" >demo.hash
hashcat --attack-mode 3 --increment --increment-min 4 \
--custom-charset1 ?l?d --hash-type 3100 ./demo.hash ?1?1?1?1?1?1?1
hashcat --hash-type 3100 ./demo.hash --show

Good Practice

Here are a few good practices on Oracle passwords.

• Keep your Oracle Clients and Server up to date. Stay updated by following Critical Patch Updates, Security Alerts and Bulletins. Install security fixes in a reasonable time frame
• Consider using strong Authentication like Kerberos and SSL based authentication.
• Don’t use legacy password verifier
  • Use Oracle password file version 12.2
  • Explicitly configure ALLOWED_LOGON_VERSION_SERVER to 12a and exclusively use 12c hash values
  • Start using PBKDF2 SHA-512 for directory-based password authentication with EUS and CMU
• Revise your password policies
  • NIST, CIS, STIG and other standards are continuously adjusted.
  • Does the complexity rule still make sense or does it just reduce the amount of possibilities.
• User awareness training. Make sure your user know the principle of good and bad Use of phase phrase rather than password

Slides of the UKOUG Presentation

References

Links and references related to this blog post

• GitHub Ghist with a few more notes Oracle Passwords but secure!
• Presentation on Slideshare
• create_password_hash.sql Calculate Oracle DES based password hash from username and password.
• verify_passwords.sql Check if user in sys.user$ has a weak DES based password. Base Script for verify_alluser_passwords.sql, verify_alluser_passwords_no.sql, verify_alluser_passwords_no.sql, verify_alluser_passwords_no.sql, verify_user_password.sql and verify_user_password_no.sql.
• HashCat advanced password recovery
• John the Ripper password cracker
• Oracle® Database Security Guide 19c
• Oracle® Database Advanced Security Guide 19c
• Oracle® Database Database Net Services Reference 19c

A few hours ago Oracle published a blog post about the new version Oracle 21c. See Introducing Oracle Database 21c. It is again an innovation release with a couple of interesting new features and enhancements. The online Oracle Documentation library does provide a few information on this enhancements:

• Oracle Database 21c What’s New
• Oracle® Database Learning Database New Features 21c
• Oracle Database 21c Books

The study of documentation and blog posts is always interesting. But it gets much more exciting when you can do first hands-on with the new release. Thanks to the Oracle Cloud this is easily possible. With a few clicks in OCI you can create a DB system with Oracle 21c. Then nothing else standing in the way to test new features all night long.

Requirements

Unfortunately it is not possible to get Oracle 21c as Always Free Version. At least I did not find a way to do so. My workmate Philipp Salvisberg pointed out to me that it is indeed possible to create an Allways Free 21c ATP DB. You just have to be in the right OCI region. Any way, since I do like to test a few infrastructure feature like DB Nest, Kerberos, EUS etc I any way have to setup a DB System rather than an Oracle Autonomous Database. In order to finally get started, a few prerequisites must be met:.

• An Oracle Account and active OCI Tenant
• A few OCI credits
• Compartment where you plan to deploy your DB system
• A VCN for you DB System.

VCN

Usually create my VCN’s with Terraform. That means I create my VCNs usually with a bastion host, a public and a private network. For this I use my Terraform OCI modules which are available in the Terraform registry. See tvdlab-base, tvdlab-bastion or tvdlab-vcn. A blog post is still on my todo list . For setting up the Oracle 21c DB system I assume that a corresponding VCN is available.

I’ll use the following information for my setup

• Compartment DB21C
• VCN db21c00

DB System

To setup the 21c database go to the main page and select Bare Metal, VM, and Exadata in the menu.

Create a DB System by pressing the corresponding blue button. Do not be surprised. On the following picture you can already see a 21c DB system.

Compared to Oracle 20c preview it does not matter if you choose Bare Metal or Virtual Machine. 21c is available for both environments. This also applies to the storage management software. I do select a Virtual Machine with the Logical Volume Manager.

As I do only use the DB System for simple feature test and engineering I keep the size to the minimum. As Oracle software I choose the Enterprise Edition. Other editions including Standard Edition are also possible.

Add you public SSH key(s) to be able to access the DB system later on via SSH. Depending on your OS you can use putty or ssh-keygen for this. Since I do work on a Mac I’ll create the pair of keys with ssh-keygen.

Finalise the system configuration by selecting your VCN and a corresponding subnet. In my case it is the VCN db21c00 and the private subnet db21c00 private.

After pressing next you come to the second page of the DB System wizard, where we configure the DB itself.

Choose a fancy name for your DB or like me just TDB21c. By the default you see, that the Database Image 19c is selected. You can change it and select 21c.

Finish the DB System wizard by selecting Create DB System.

It will take now a while until you Oracle 21c Database is ready to use. A good moment to crab a cup of coffee or write a blog post . I’ll do this now as well and finish this post in a couple of minutes…

My 21C DB system is finished in the meantime and can be used.

Since I did create the VM in a private subnet, it is not possible to directly access the DB from the internet. I do have to connect using the bastion host. Either via SSH or for SQL Developer using SSH port forwarding.

Enclosed an example to connect to the new DB system via bastion host using SSH proxy command. In this example I do us the following SSH parameters:

• -A does enables forwarding of the authentication agent connection
• -J is used to specify the proxy host used to initiate the connection. In my case the bastion host.
• db21c is the alias hostname for the public IP of my bastion host
• deadalus is the hostname for my DB system
• opc is the user used to connect to the different compute instances. you can also connect directly to the oracle user, if you put the public keys in the .ssh/authorized_keys file of user oracle.

soe@gaia:~/ [ic19300] ssh -A -J opc@db21 opc@deadalus
Last login: Wed Dec  9 05:35:20 2020 from 10.0.0.2
[opc@deadalus ~]$ sudo su - oracle
Last login: Wed Dec  9 05:36:22 UTC 2020 from 10.0.0.2 on pts/0
[oracle@deadalus ~]$ sqlplus / as sysdba

SQL*Plus: Release 21.0.0.0.0 - Production on Wed Dec 9 05:42:38 2020
Version 21.1.0.0.0

Copyright (c) 1982, 2020, Oracle.  All rights reserved.


Connected to:
Oracle Database 21c Enterprise Edition Release 21.0.0.0.0 - Production
Version 21.1.0.0.0

SQL> 

Alternatively, you can do SSH port forwarding for port 1521 and connect directly to the database with SQL Developer.

soe@gaia:~/ [ic19300] ssh -A opc@db21 -L 1521:deadalus:1521
Last login: Wed Dec  9 05:33:00 2020 from yyy.yyy.yyy.yyy
--------------------------------------------------------------------
-
- Welcome to the bastion / jump host for the OCI environment
- hostname          :   db21c
- Public IP         :   xxx.xxx.xxx.xxx
- Guacamole Console :   http://secret
- Guacamole Admin   :   <USER>
- Guacamole Password:   <PASSWORD>

[opc@db21c-bastion00 ~]$ 

If you don’t want to deal with command line and SSH commands, then with newer versions of SQL Developer you can set up a connection directly via SSH port forwarding. Select SSH from the View menu and create a new SSH connection. The following details must be added:

• Name: Name of your SSH connection
• Host: Public IP of your bastion host
• Username: Usually just opc
• Key File: The private key file matching the public key used above
• Name: Name of you SSH port forwarding rule
• Host: Private IP address of the DB system
• Port: The TCP Port configured on the DB system usually just 1521
• Local Port: Use an automatically assigned local port, then you do not have to bother if it is already in use

The SSH host can now be used to configure the DB connection. You only have to select the connection type SSH.

Conclusion

In OCI you can create a new engineering system for Oracle 21c relatively quickly. With the Bastion host I have chosen a more complex but secure method. The separation of the subnets into private and public network does offer me the flexibility to test a few Features which requires additional infrastructure components. e.g. Centrally Managed Users CMU or Enterprise User Security EUS, which both require additional services such as Oracle Unified Directory or MS Active Directory. Running everything via public IP is rather a bad idea. Any way, following this approach you now have Oracle 21c Database. I wish you happy engineering.

References

• Oracle Blog Post Introducing Oracle Database 21c
• Oracle 21c Documentation Library Books
• Oracle Database 21c What’s New
• Oracle® Database Learning Database New Features 21c
• OCI white paper Bastion Hosts – Protected Access for Virtual Cloud Networks
• Oracle A-Team Blog SSH tunnel to a private VM using a Bastion host in OCI
• Martin Berger Blog Oracle Cloud Infrastructure and SSH Keys – Jump!
• Trivadis Terraform Modules in the Terraform Registry

It happens that the children are at school and the wife is at work in the hospital. For lunch a light snack from the takeaway around the corner would be perfect. No problem at work in the city, but when working from home? The offer in my area is relatively modest. Hey but why not just make a quick Piadina for lunch! For once something else than OCI, DB Security or other database stuff. Not sure what a Piadina is?

Piadina or Piada is a thin Italian flatbread, typically prepared in the Romagna historical region (Forlì, Cesena, Ravenna and Rimini). It is usually made with white flour, lard or olive oil, salt and water.

Wikipedia. 2020. “Piadina.” Last modified 23. October 2020. https://en.wikipedia.org/wiki/Piadina

The dough

Unfortunately, I can not fall back on an old family recipe. Therefore, simply a variant of the dough, which has always worked well for me. In principle, the ingredients are relatively simple and available in almost every household. One takes (or borrow from the neighbour):

• 200g white flour
• 1 tsp salt
• 1 tbsp olive oil
• 1 dl sparkling mineral water

Mix everything together into a smooth dough and let it rest ideally for 20-30 minutes. If you are starving, you can skip the 30 minutes. There is no block corruption or anything…

The roasting / baking

Then divide the dough into 4 equal parts and roll each out into a thin patty. To do this, take a rolling pin. If you don’t have one, you can use an empty and clean beer, wine or whiskey bottle from the night before . You can then briefly bake the patty in a pan without adding oil.

The filling

As a filling you can take what every you find in you fridge and fits to a Piadina. Usually you take some cheese (Parmigiano, Pecorino,…) italian raw ham, arugula, salami, dried tomatos etc…

My Piadina today looks like this

Conclusion

A Piadina always goes well. It also doesn’t take much more time to prepare than deploying an Oracle 21c DB into OCI. After enjoying 1-2 Piadina, you are strengthened for a productive and successful afternoon. Enjoy

This morning I had the great opportunity to participate in the virtual event AUSOUG Connect 2021 with my lecture Security Best Practice: Oracle passwords, but secure!. For me it was a premiere and a pleasure to be part of an Oracle event in Australia.

Oracle Password Security is a small but central topic in database security. Database security and especially passwords have been on my mind for a while. Over the time, one or the other example on that subject accumulates. Therefore it is not always easy to concentrate on the essential points. And as so often in today’s lecture, there was not enough time to go into all possible examples and demos. For this reason, I try to briefly pick up on one or the other point in this blog post.

Demo Environment

A simple Docker container with Oracle Database 21.4.0.0 is used for the Lab environment. The following sketch shows the schematic structure of the environment. The scripts for this environment and the following demos are available in the GitHub repository oehrlis/orapwd. The Docker container is based on oehrlis/docker. However, the scripts can also be used in another Oracle database container or Oracle database.

Logon Process

The login process into the Oracle database can be performed most easily with a network analysis e.g. with Wireshark. This way you can see exactly which TCP packets are sent from or to the DB server. The network traffic is collected either directly with Wireshark or with the help of tcpdump on the DB server. Here is an example of how to use tcpdump. This requires that tcpdump is installed in advanced. Command has to be run as root.

bash-4.2# mkdir -p /u01/config/tcpdump
bash-4.2# tcpdump -i eth0 -s 65535 -w /u01/config/tcpdump/tcpdump_$(date "+%Y%m%d_%H%M").dmp
tcpdump: listening on eth0, link-type EN10MB (Ethernet), capture size 65535 bytes

Then you can connect via SQL*Plus and run test queries. After finishing the tests you can stop tcpdump and analyse the created dump file with Wireshark. After loading the tcpdump file, select a TNS packet for analysis and select Fallow TCP Stream. In the new window you can see the information that is exchanged between the database server and client, as you can see in the following picture.

With unencrypted SQL*Net connections you can see very nicely the connection establishment, session key exchange and subsequently also the SQL statements which are sent from the client to the server. With an ALTER USER scott IDENTIFIED BY tiger; the statement is only parsed on the server and is therefore not encrypted.

Different Password Verifier

As explained in the lecture, you can use ALLOWED_LOGON_VERSION_SERVER or ALLOWED_LOGON_VERSION_CLIENT in sqlnet.ora to specify which version of the authentication protocol and thus which password verifier should be used. See also Oracle® Database Database Net Services Reference 21c or Oracle® Database Security Guide 21c.

Lets set ALLOWED_LOGON_VERSION_SERVER to 11 in sqlnet.ora either using vi or just append it to the end of sqlnet.ora with echo.

echo "SQLNET.ALLOWED_LOGON_VERSION_SERVER=11" >> $TNS_ADMIN/sqlnet.ora

Now create a configure a couple of users using sqlplus. Preferably one user per password verifier. Whereby we set the passwords explicitly with IDENTIFIED BY VALUES.

ALTER SESSION SET CONTAINER=pdb1;
CREATE USER test_10g IDENTIFIED BY VALUES 'AF310E4D20D06950';
CREATE USER test_11g IDENTIFIED BY VALUES
'S:6702B83E88D277BFC378AD6B22DD1AE01895A254470F8124A9D3C5347056';
CREATE USER test_12c IDENTIFIED BY VALUES
T:45738A7B75C9E31ED0C533BCF4931084658A143FD7CF826B980A88EA6C4F0BE66C28DA7085BCAE386723029BA967DC4F45E9C146F6FA7C22E44BA2C1BD2F56F8C22291D417E26D4B810003F3F055EDFF';
CREATE USER test_all IDENTIFIED BY Welcome1;

Don’t forget to grant some privileges..

GRANT CREATE SESSION TO test_10g;
GRANT CREATE SESSION TO test_11g;
GRANT CREATE SESSION TO test_12c;
GRANT CREATE SESSION TO test_all;

GRANT SELECT_CATALOG_ROLE TO test_10g;
GRANT SELECT_CATALOG_ROLE TO test_11g;
GRANT SELECT_CATALOG_ROLE TO test_12c;
GRANT SELECT_CATALOG_ROLE TO test_all;

Now lets see what we have in SYS.USER$

SET LINESIZE 160 PAGESIZE 200
COL name FOR a10
COL password FOR a16
COL spare4 FOR a20
SELECT name,password,spare4 FROM user$ WHERE name LIKE 'TEST_%' ORDER BY 1;

NAME	   PASSWORD	    SPARE4
---------- ---------------- ----------------------------------------
TEST_10G   AF310E4D20D06950
TEST_11G		    S:6702B83E88D277BFC378AD6B22DD1AE01895A2
			    54470F8124A9D3C5347056

TEST_12C		    T:45738A7B75C9E31ED0C533BCF4931084658A14
			    3FD7CF826B980A88EA6C4F0BE66C28DA7085BCAE
			    386723029BA967DC4F45E9C146F6FA7C22E44BA2
			    C1BD2F56F8C22291D417E26D4B810003F3F055ED
			    FF

TEST_ALL   4932A1B4C59EC3D0 S:FA89B6A242F2E80B1F45E2A7861D9CF49F51ED
			    34B4D2FABAA319561AEEE4;T:E66E6EEDA917E09
			    28E40C05E5B5E8D34D20AA1FDB1F108E4EE8DCE2
			    31EE59BD17BBEB58F83DD01713911A96E817BE8D
			    F28584350991611EF366CC2AEE9CBBAB668D69C8
			    03C92639BC3853F527A1B8DB3

In DBA_USERS we do see the password verifier version

SET LINESIZE 160 PAGESIZE 200
COL username FOR a10
COL password_versions FOR a20
SELECT username, password_versions FROM dba_users WHERE username LIKE 'TEST_%';
USERNAME   PASSWORD_VERSIONS
---------- --------------------
TEST_10G   10G
TEST_11G   11G
TEST_ALL   10G 11G 12C
TEST_12C   12C

As you can see Oracle did create all 3 password hashes for the user TEST_ALL as ALLOWED_LOGON_VERSION_SERVER is set to 11. We now change it to 12 and see, that the user TEST_10G can no longer connect.

host sed -i 's/SQLNET\.ALLOWED_LOGON_VERSION_SERVER.*/SQLNET\.ALLOWED_LOGON_VERSION_SERVER=12/' $cdn/admin/sqlnet.ora
SQL> connect test_10g/Welcome1@pdb1
ERROR:
ORA-01017: invalid username/password; logon denied

Warning: You are no longer connected to ORACLE

Change ALLOWED_LOGON_VERSION_SERVER to 12a will then cause, that only the user TEST_12C and TEST_ALL can log in.

host sed -i 's/SQLNET\.ALLOWED_LOGON_VERSION_SERVER.*/SQLNET\.ALLOWED_LOGON_VERSION_SERVER=12a/' $cdn/admin/sqlnet.ora
SQL> connect test_11g/Welcome1@pdb1
ERROR:
ORA-01017: invalid username/password; logon denied

SQL> connect test_12c/Welcome1@pdb1
Connected.
SQL> connect test_all/Welcome1@pdb1
Connected.

When now changing the password for user TEST_ALL, Oracle will only create the password hash for 12c.

SET LINESIZE 160 PAGESIZE 200
COL username FOR a10
COL password_versions FOR a20
SELECT username, password_versions FROM dba_users WHERE username LIKE 'TEST_%';

USERNAME   PASSWORD_VERSIONS
---------- --------------------
TEST_10G   10G
TEST_11G   11G
TEST_ALL   12C
TEST_12C   12C

Check Passwords

To check the default Oracle password we can query dba_users_with_defpwd. Here we see that in PDB1 the user SCOTT still has a default password.

ALTER SESSION SET CONTAINER=pdb1;

Session altered.

SELECT username FROM dba_users_with_defpwd;

USERNAME
-----------
SCOTT

For Oracle 10g respectively passwords stored in SYS.USER$.PASSWORD column we can also verify the password hashes using the script verify_alluser_passwords.sql. This script just calculates the a few passwords based on an embedded dictionary and compares the hashes. You see that the user TEST_10G has a week password.

SQL> @verify_alluser_passwords.sql
User			      Status	Password
----------------------------- -----------------------------
SYS			      0 	OK
AUDSYS			      8 	OK
SYSTEM			      0 	OK
OUTLN			      8 	OK
...
SCOTT			      16	OK
TVD_HR			      0 	OK
TVD_HR_SEC		      0 	OK
TEST_10G		      0 	WELCOME1
TEST_11G		      0 	OK
TEST_12C		      0 	OK
TEST_ALL		      0 	OK

Let’s create manually a Oracle 10g password hash using create_password_hash.sql. The script does use DBMS_CRYPTO to manually create a hash value.

SQL> @create_password_hash.sql system ieShae0
Username : system
Password : ieShae0
Hash	 : 0AD56CF5F1CB8D2A
SQL	 : alter user system identified by values '0AD56CF5F1CB8D2A';

When now can create a hash file for the hashcat tool to run a brute force attack on the hash.

echo "0AD56CF5F1CB8D2A:SYSTEM" > demo.hash

hashcat as a couple of options and parameters. Please see https://hashcat.net/hashcat/ for more detailed information. For now we just use the following parameters:

• --increment will start to brute force with shorter length e.g 4 characters
• --custom-charset1 to define numbers and characters
• --hash-type Oracle 7+ respectively password verifier 10g
• --show show the password

Let’s start a hashcat session

hashcat --attack-mode 3 --increment --increment-min 4 --custom-charset1 ?l?d --hash-type 3100 ./demo.hash ?1?1?1?1?1?1?1

This session is not using any dictionary or rule base attack. Is is testing all password combination according the formate defined above. This can take a moment. To speed this up you’ll would have to use a dictionary, rule based or combined attack.

hashcat (v6.1.1) starting...

/usr/local/Cellar/hashcat/6.1.1/share/hashcat/OpenCL/m03100_a3-optimized.cl: Pure kernel not found, falling back to optimized kernel
OpenCL API (OpenCL 1.2 (Sep  5 2021 22:39:07)) - Platform #1 [Apple]
====================================================================
* Device #1: Intel(R) Core(TM) i9-9980HK CPU @ 2.40GHz, skipped
* Device #2: Intel(R) UHD Graphics 630, 1472/1536 MB (384 MB allocatable), 24MCU
* Device #3: AMD Radeon Pro 5500M Compute Engine, 8112/8176 MB (2044 MB allocatable), 24MCU

/usr/local/Cellar/hashcat/6.1.1/share/hashcat/OpenCL/m03100_a3-optimized.cl: Pure kernel not found, falling back to optimized kernel
Minimum password length supported by kernel: 0
Maximum password length supported by kernel: 30

Hashes: 1 digests; 1 unique digests, 1 unique salts
Bitmaps: 16 bits, 65536 entries, 0x0000ffff mask, 262144 bytes, 5/13 rotates

Applicable optimizers applied:
* Optimized-Kernel
* Zero-Byte
* Not-Iterated
* Single-Hash
* Single-Salt
* Brute-Force

Watchdog: Hardware monitoring interface not found on your system.
Watchdog: Temperature abort trigger disabled.

Host memory required for this attack: 602 MB
Session..........: hashcat
Status...........: Cracked
Hash.Name........: Oracle H: Type (Oracle 7+)
Hash.Target......: 0AD56CF5F1CB8D2A:SYSTEM
Time.Started.....: Thu Nov 11 13:52:50 2021 (1 min, 2 secs)
Time.Estimated...: Thu Nov 11 13:53:52 2021 (0 secs)
Guess.Mask.......: ?1?1?1?1?1?1?1 [7]
Guess.Charset....: -1 ?l?d, -2 Undefined, -3 Undefined, -4 Undefined 
Guess.Queue......: 4/4 (100.00%)
Speed.#2.........:  4586.9 kH/s (9.83ms) @ Accel:8 Loops:32 Thr:8 Vec:1
Speed.#3.........:   365.5 MH/s (7.82ms) @ Accel:64 Loops:32 Thr:64 Vec:1
Speed.#*.........:   370.1 MH/s
Recovered........: 1/1 (100.00%) Digests
Progress.........: 23809572864/78364164096 (30.38%)
Rejected.........: 0/23809572864 (0.00%)
Restore.Point....: 399360/1679616 (23.78%)
Restore.Sub.#2...: Salt:0 Amplifier:3552-3584 Iteration:0-32
Restore.Sub.#3...: Salt:0 Amplifier:5920-5952 Iteration:0-32
Candidates.#2....: NPRH5I1 -> TFCIN8M
Candidates.#3....: LDD45HR -> SBRYHG0

Started: Thu Nov 11 13:52:44 2021
Stopped: Thu Nov 11 13:53:53 2021

The result can also be grabbed by using the --show command. By the way, the brute force attack on this hash just took about 1min on my MacBookPro.

hashcat --hash-type 3100 ./demo.hash --show

Verify the performance of your environment by running a benchmark on Oracle 7+ hash

hashcat --benchmark --hash-type 3100
hashcat (v6.1.1) starting in benchmark mode...

Benchmarking uses hand-optimized kernel code by default.
You can use it in your cracking session by setting the -O option.
Note: Using optimized kernel code limits the maximum supported password length.
To disable the optimized kernel code in benchmark mode, use the -w option.

OpenCL API (OpenCL 1.2 (Sep  5 2021 22:39:07)) - Platform #1 [Apple]
====================================================================
* Device #1: Intel(R) Core(TM) i9-9980HK CPU @ 2.40GHz, skipped
* Device #2: Intel(R) UHD Graphics 630, 1472/1536 MB (384 MB allocatable), 24MCU
* Device #3: AMD Radeon Pro 5500M Compute Engine, 8112/8176 MB (2044 MB allocatable), 24MCU

Benchmark relevant options:
===========================
* --optimized-kernel-enable

Hashmode: 3100 - Oracle H: Type (Oracle 7+)

Speed.#2.........:  4946.0 kH/s (78.31ms) @ Accel:128 Loops:16 Thr:8 Vec:1
Speed.#3.........:   349.2 MH/s (71.25ms) @ Accel:32 Loops:512 Thr:64 Vec:1
Speed.#*.........:   354.2 MH/s

Started: Thu Nov 11 13:57:56 2021
Stopped: Thu Nov 11 13:58:02 2021

As you see hashcat, john the ripper etc are powerful but also dangerous tools when it comes to password engineering

Recommendations

• Keep your Oracle Clients and Server up to date
  • Stay updated by following Critical Patch Updates, Security Alerts and Bulletins
  • Install security fixes in a reasonable time frame
• Consider using strong Authentication
  • Kerberos and SSL based Authentication
• Don’t use legacy password verifier
  • Use Oracle password file version 12.2
  • Explicitly configure ALLOWED_LOGON_VERSION_SERVER to 12a and exclusively use 12c hash values
• Revise your password policies
• User awareness training
• Reduce the attack vector
  • Limit access to password hash values
  • Know where you have password hash values
  • Start using NOAUTHENTICATION for schema owners where no login is required.
• Implement general database hardening

Slides to my Lecture

Disclaimer

The use of methods and tools to verify password hashes are not allowed everywhere. In particular, their use on productive environments is explicitly not recommended. Please check if the tools are allowed in the respective environment, company, country, etc. before using them. The author disclaims any liability.

This morning I had the opportunity to give a presentation on Oracle Database Vault at the DOAG conference.

Abstract

Oracle Database Vault has been on the market for a few years now. The product has been constantly improved over the years. But where is it worthwhile to use it? Which security measures can be implemented with it? And from whom does DB Vault protect me at all? In this presentation, the technical possibilities of Database Vault 19c / 21c will be explained in addition to the experiences from two customer projects. We will try to show where the use of Database Vault is worthwhile under certain circumstances and under which conditions it is not. This also includes whether protection against snakes and thieves is ensured.

PS: I asked my children what kind of presentation I should submit. The answers were snakes, thieves and cheetahs…

Questions

Question: Is Oracle Database Vault a separate product?

Answer: Yes it is an option for Oracle Database Enterprise Edition. Beside the Oracle Database Vault Option it is required to have a valid Oracle Database Enterprise Edition license.

Question: What is a REALM

Answer: A REALM is a grouping of database schemas, database objects, and database roles that must be secured for a given application. A REALM is some kind of a security zone for DB objects. User who are owner or participant of the REALM can also access the objects within a REALM. RULE and RULE SET are used to authorise user for REALMS. See also About Realms in Oracle® Database Vault Administrator’s Guide 21c.

If you have any further questions, don’t hesitate to ask them via comment on this blog post. Alternatively, you can also contact me directly.

Slides to my Lecture

Links and References

• Agenda DOAG Conference 2021
• Oracle® Database Vault Administrator’s Guide 21c html
• Oracle® Database Vault Administrator’s Guide 21c – What to Expect After You Enable Oracle Database Vault
• White paper Oracle Database Vault DBA Administrative Best Practices
• White paper Oracle Database Vault Best Practices

The tnsnames.ora is a configuration file for Oracle database name resolution. It contains network service names that are mapped to connection descriptors for the local naming method. With the help of tnsnames.ora Oracle clients respectively the users can easily access Oracle databases. The connection descriptors provides all relevant information like host, Port, service name etc.

For larger environments with multiple Oracle databases and hundreds or more clients, managing and distributing the tnsnames.ora becomes cumbersome. The management of Oracle Net Service Names can be done with the following Oracle Solutions and Tools:

• Manual management of Oracle Net Service Names in one or more tnsnames.ora files. e.g. with a version control system, NFS share etc.
• Oracle Network Service Names can be entered directly in MS Active Directory. The database names are then resolved via AD. However, this method requires a schema extension in MS Active Directory. This is usually not so easy to implement in larger environments.
• Use of an Oracle Internet Directory (OID) for the directory based administration of Oracle Net Service Names. But OID is anything but lean.

It is also a good idea to directly implement Oracle Enterprise User Security based on Oracle Internet Directory or Oracle Unified Directory. Whereby with this solution a corresponding Oracle Directory Services Plus license is required. In addition, with Oracle Enterprise User Security, authentication and authorisation are also set up centrally. In addition to the license costs, there is also the increased implementation and operating costs. For the central administration of the Oracle Net Service Names a bit much effort. Especially if you want to use Oracle Centrally Managed Users instead of Oracle Enterprise User Security. As a simple alternative for the directory-based Oracle Net Service Names resolution, an open source LDAP directory service can of course always be used. A proven solution, just not officially supported by Oracle.

Since a few days Oracle has fulfilled my long awaited Christmas wish and adjusted the Restricted Use License for OUD and OID . The changes are available immediately for all current Oracle versions. I.e. Oracle 12.2, 18c, 19c and 21c. For the older database versions, e.g. 12.1 and 11.2 the restricted use licenses have not been adjusted.

This now allows the use of Oracle Unified Directory to build an LDAP directory for Oracle Net Service Names name resolution for any Oracle Edition except Oracle Database Express Edition. With the help of my scripts on GitHub oehrlis/oudbase you can build an OUD directory within a few minutes. I will give an example of this in a later blogpost.

Conclusion

Nothing stands in the way of setting up a simple LDAP directory service, based on Oracle Unified Directory, for Oracle Net Services name resolution. This is especially good news for larger environments and Oracle Centrally Managed Users deployments, where until now the only option was to manage the Oracle Net Service Names manually or to use an OpenSource LDAP directory service.

Links to the latest Oracle® Database Database Licensing Information User Manuals:

• Oracle® Database Database Licensing Information User Manual 21c
• Oracle® Database Database Licensing Information User Manual 19c
• Oracle® Database Database Licensing Information User Manual 18c
• Oracle® Database Database Licensing Information User Manual 12c Release 2 (12.2)

Have fun setting up your Oracle Unified Directory based Oracle Net Service Names server. Stay tuned for a couple of technical information and how-to’s ….

The tnsnames.ora is a configuration file for Oracle database respectively Oracle Net Service Names resolution. It contains network service names that are mapped to connection descriptors for the local naming method. With the help of tnsnames.ora Oracle clients respectively the users can easily access Oracle databases. The connection descriptors provides all relevant information like host, port, service name etc.

For larger environments with multiple Oracle databases and hundreds or more clients, managing and distributing the tnsnames.ora becomes difficult. Especially when Oracle DataGuard or Oracle Real Application Cluster are added, where more complex connection description with failover or load balancing information is needed. A reliable Oracle Network Service and an up-to-date tnsnames.ora are crucial for a highly available access to Oracle databases. A manual copy of tnsnames.ora or a central NFS or Windows share usually does not meet this requirement.

The Solution Approach

The idea of using an LDAP directory to manage Oracle Network Service Names is not new. There are several official and unofficial approaches how this can be implemented:

• Use of an Oracle Directory for the administration of Oracle Net Service Names, e.g. Oracle Internet Directory (OID) or Oracle Unified Directory (OUD). Whereby with this solution a corresponding Oracle Directory Services Plus license is required. In addition, OID is anything but lean. Ok the license situation has slightly changed see Free Oracle Unified Directory for Oracle Net Services
• Setup of Oracle Enterprise User Security (EUS). Here, too, an Oracle Directory, i.e. OID or OUD with a corresponding license, is required. With this solution, the authentication and authorisation of the databases is also solved centrally. The setup of EUS is rather complex as it is not only to setup an LDAP server. You also have to define and implement an appropriate user and role concept. In any case, this has an impact on existing applications and use cases.
• Oracle Network Service Names can be entered directly in MS Active Directory. The database names are then resolved via AD. However, this method requires a schema extension in MS Active Directory. This is usually not so easy to implement in larger environments.
• Alternative LDAP directory servers like OpenLDAP or 389 Directory Server can be used. Also here a schema extension is needed. Since the LDAP servers are only used for the Oracle Net Service Names resolution this is not critical. Especially since a LDAP schema extension is standard procedure. The advantage of this method is that by using an OpenSource LDAP server the costs remain manageable, although it is not officially supported.

The following figure shows a schematic diagram of the Oracle Network Service Names resolution using an open source LDAP directory as an example.

For the solution presented here, we use the open source LDAP server 389 Directory Server. This is available as an open source variant via Fedora and is also part of RedHat Enterprise Linux as RedHat Directory Server (RHDS). Whereby a corresponding subscription is necessary in order to use the RedHat Directory Server (RHDS). However, the 389 Directory Server from the Fedora project also works perfectly under Oracle Enterprise Linux 8.

The solution presented here is based on the 389 Directory Server from the Fedora project. However, the installation and configuration steps can be applied more or less 1:1 to the RedHat Directory Server (RHDS) as well. Especially since the documentation from the 389 Directory Server is usually any way based on the RedHat documentation or at least references it.

Prerequisites and Requirements

The prerequisites are straight forward. The 389 Directory Server or RedHat Directory Server (RHDS) is modest in terms of system resources. Especially as LDAP server for a few 100 Oracle Net Service Names. The documentation Red Hat Directory Server 11 Release Notes does not show any specific hardware requirements. It is recommended to provide at least 2 CPU cores plus 16GB memory for productive environments. For a simple test setup also less is enough. The base operating system is OEL 8.5 or REL 8.5.

Preparation

First of all we have to make sure, that the Fedora EPEL repository is added and the 389-ds Module is enabled.

sudo dnf -y install https://dl.fedoraproject.org/pub/epel/epel-release-latest-8.noarch.rpm
sudo yum -y module enable 389-ds

It is also a good idea to open a couple of local firewall ports for LDAP and LDAPS

sudo firewall-cmd --list-all --permanent --zone=public
sudo firewall-cmd --permanent --add-service=ldap --zone=public
sudo firewall-cmd --permanent --add-service=ldaps --zone=public
sudo firewall-cmd --permanent --add-port=9090/tcp --zone=public
sudo firewall-cmd --reload
sudo firewall-cmd --list-all --permanent --zone=public

Optional we pre create the dirsrv group and assign the OS user oracle to be a member of this group. This allows certain administrative activities to be performed as user oracle.

sudo groupadd --gid 520 dirsrv
cat /etc/group
sudo usermod -a -G dirsrv oracle

Install 389 Directory Server

The installation is quite simple. Since we added the Fedora EPEL repository, we only need to install the appropriate packages and dependencies for 389 Directory Server with yum.

sudo yum install 389-ds-base sscg

Configure 389 Directory Server for Oracle Net Service

Once the packages are installed, we can create a Directory Server instance. The easiest way to do this is to use a template. The template can be created directly with dscreate.

sudo dscreate create-template /tmp/oraNet.inf

The template must then be adjusted accordingly. In particular, the following values must be set:

• full_machine_name the full qualified hostname of the LDAP directory server.
• instance_name name of the LDAP directory server instance.
• root_password password for the directory server root user.
• suffix used as base DN for the directory information tree (DIT) of the directory server

Enclosed is an example to setup an instance named oraNet on ldap1.trivadislabs.com with a base DN dc=trivadislabs,dc=com:

[general]
full_machine_name = ldap1.trivadislabs.com
start = True

[slapd]
instance_name = oraNet
port = 389
root_password = Welcome1
secure_port = 636
self_sign_cert = True
self_sign_cert_valid_months = 24

[backend-userroot]
create_suffix_entry = True
suffix = dc=trivadislabs,dc=com

The directory server instance is then created as root with the command dscreate.

sudo dscreate from-file /tmp/oraNet.inf

Check if the instance is running. dscreate does not only create the instance. It also configure the corresponding start/stop scripts.

sudo dsctl --list
sudo systemctl status dirsrv@oraNet.service

So that we can later also create corresponding Oracle Net Service Names objects in the Directory Server, the schema must be extended. For this we copy the file 90orclNet.ldif into the corresponding instance directory and restart the instance.

curl -Lf https://gist.githubusercontent.com/oehrlis/49767f09c265efc9fc3a74ee16bdfd53/raw/5c71003afe0c38040d317b9a8bc12d3eef113a75/90orclNet.ldif -o /tmp/90orclNet.ldif 
sudo cp /tmp/90orclNet.ldif /etc/dirsrv/slapd-oraNet/schema/90orclNet.ldif
sudo systemctl restart dirsrv@oraNet.service

Check the status of our new Directory Server instanz.

sudo dsctl $(dsctl --list) status
sudo dsctl $(dsctl --list) healthcheck

n the following we will execute several LDAP commands. To avoid having to enter the password interactively each time, we save it temporarily in a local file. This file is passed with the parameter -y to the LDAP command. It makes sense to delete the password file afterwards.

echo "Welcome1" | tr -d '\n' >.oraNetDirectoryManager.pwd
chmod 600 .oraNetDirectoryManager.pwd

The directory server is actually ready now. However, in order to be able to register the corresponding Oracle Net Service Names. We still need to create an Oracle Context. To do this, we simply create an object cn=OracleContext with the class orclContext in the Base DN dc=trivadislabs,dc=com.

ldapadd -h $(hostname -f) -p 389 -x -D "cn=Directory Manager" \
-y .oraNetDirectoryManager.pwd <<-EOI
dn: cn=OracleContext,dc=trivadislabs,dc=com
objectclass: orclContext
cn: OracleContext

EOI

Since the Oracle clients usually execute the resolution of the Oracle Net Service Names with an anonymous LDAP query respectively with an anonymous bind, the ACIs have to be adapted in a way that anonymous searches are allowed in the Oracle Context.

ldapadd -h $(hostname -f) -p 389 -x -D "cn=Directory Manager" \
-y .oraNetDirectoryManager.pwd <<-EOI
dn: dc=trivadislabs,dc=com
changetype: modify
add: aci
aci: (targetattr!="userPassword||authPassword")(version 3.0; acl "Anonymous read access"; allow (read,search,compare) userdn="ldap:///anyone";)

EOI

LDAP Based Oracle Net Services

We now have an empty 389 Directory Server ready to be used for Oracle Net Service Names resolution. It’s about time to add our first entry using ldapadd. In the following example we add a new entry with the name TDB02 and its Oracle Net Description String.

ldapadd -h $(hostname -f) -p 389 -x -D "cn=Directory Manager" \
-y .oraNetDirectoryManager.pwd <<-EOI
dn: cn=TDB02,cn=OracleContext,dc=trivadislabs,dc=com
objectclass: top
objectclass: orclNetService
cn: TDB02
orclNetDescString: (DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=db19)(PORT=1521))
 (CONNECT_DATA=(SERVER=DEDICATED)(SERVICE_NAME=TDB02.trivadislabs.com))(UR=A))

EOI

With ldapmodify we can also modify existing entries.

ldapmodify -h $(hostname -f) -p 389 -x -D "cn=Directory Manager" \
-y .oraNetDirectoryManager.pwd <<-EOI
dn: cn=TDB02,cn=OracleContext,dc=trivadislabs,dc=com
changetype: modify
replace: orclNetDescString
orclNetDescString: (DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=db19)(PORT=1521))
 (CONNECT_DATA=(SERVER=DEDICATED)(SERVICE_NAME=TDB02.trivadislabs.com))(UR=A))

EOI

Or search for entries using ldapsearch.

ldapsearch -h $(hostname -f) -p 389 -x -LLL -b "dc=trivadislabs,dc=com" -s sub "(&(objectclass=orclNetService)(cn=TDB0*))"

With ldapdelete the entries can be deleted afterwards.

ldapdelete -h $(hostname -f) -p 389 -x -D "cn=Directory Manager" \
-y .oraNetDirectoryManager.pwd \
cn=TDB03,cn=OracleContext,dc=trivadislabs,dc=com

Before we are able to use the LDAP based Oracle Net Service Names we have to configure Oracle Net. To do this, we need to adjust the name resolution order in $TNS_ADMIN/sqlnet.ora with the NAMES.DIRECTORY_PATH parameter and put LDAP first.

NAMES.DIRECTORY_PATH=(LDAP, TNSNAMES, EZCONNECT )

In addition, the file $TNS_ADMIN/ldap.ora must be created. In it, the LDAP server configuration must be specified as follows.

DIRECTORY_SERVERS=(ldap1.trivadislabs.com:389:636)
DEFAULT_ADMIN_CONTEXT="dc=trivadislabs,dc=com"
DIRECTORY_SERVER_TYPE=OID

Finally we can do a tnsping and check if the Oracle Net Service Name is resolved via LDAP or tnsnames.ora

oracle@db19:~/ [rdbms19] tnsping TDB02

TNS Ping Utility for Linux: Version 19.0.0.0.0 - Production on 01-MAR-2022 06:31:06

Copyright (c) 1997, 2021, Oracle.  All rights reserved.

Used parameter files:
/u01/app/oracle/network/admin/sqlnet.ora

Used LDAP adapter to resolve the alias
Attempting to contact (DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=db19)(PORT=1521))(CONNECT_DATA=(SERVER=DEDICATED)(SERVICE_NAME=TDB02.trivadislabs.com))(UR=A))
OK (10 msec)

What’s Next?

This blog post describes a simple configuration of the 389 Directory Server for Oracle Net Service. In principle, nothing stands in the way of productive use. However, it is recommended to consider 2-3 points in more detail.

• User and role concept for the administration and management of data in LDAP.
• Toolset for administration e.g. scripts, LDAP browser etc.
• High available configuration of the 389 Directory Server e.g. multiple LDAP servers with an appropriate replication configuration.
• Certificates suitable for production.
• Development and implementation of an operating and security concept. This includes backup and restore tasks, among others.
• License and subscription clarification, especially when using the RedHat Directory Server instead of the pure 389 Directory Server.

Conclusion

389 Directory Server, just as with OpenLDAP, it is relatively easy to create a central directory for the Oracle Net Service Names or tnsnames.ora. Within a few minutes you have built a stand-alone LDAP server. With a little more effort, configuring secure SSL certificates, extended directory information trees with different suffixes, as well as replication, etc. is also easily possible.

Based on this approach, we have already been able to successfully set up highly available LDAP directory servers with multiple suffixes respectively Oracle Network Service Domain Names within the scope of customer projects. Thus, nothing stands in the way of replacing the cumbersome manual administration of the tnsnames.ora files with a central directory. Especially if only the Oracle Network Service Names are in focus. If, in addition, the authentication and authorisation of the databases is to be set up centrally, there is no way around Oracle Enterprise User Security (EUS) or Oracle Centrally Managed Users (CMU). The advantage of the solution presented here is that it can be combined with Oracle Centrally Managed Users (CMU). I.e. authentication and authorisation is done with CMU via Active Directory Integration, while name resolution is based on the LDAP directory. All with manageable effort and without additional licensing costs. In contrast, a solution with Oracle Enterprise User Security (EUS) is somewhat more flexible, but also more complex and cost-intensive.

Links and References

The following links are useful in the context of this blog post, tnsnames.ora, Oracle Network Services and LDAP Directory Server.

• Oracle® Database Net Services Administrator’s Guide 21c
• Oracle® Database Net Services Reference 21c
• Oracle® Database Net Services Reference 21c – LDAP Schema for Oracle Net Services
• Oracle® Database Administrator’s Reference 21c for Microsoft Windows – Using Oracle Database with Microsoft Active Directory
• Fedora 389 Directory Server Overview
• Fedora 389 Directory Server Documentation
• Red Hat Directory Server 11 Installation Guide
• Red Hat Directory Server 11 Administration Guide
• LDAP Wiki
• Install, Configure 389 Directory Server (LDAP) on CentOS, RHEL 8
• Managing OpenLDAP a blog post by Tyler about how to use OpenLDAP client tools like ldapsearch, ldapadd etc. use full as well for 389 Directory Server.

With the introduction of Oracle Unified Audit, Oracle has completely redesigned the process of logging audit events. With the new unified audit trail, there is only one place where audit records are stored. Ok, the audit trail exists per PDB and for a read only database additionally somehow as a binary overflow file. However, the times are over in which we had to deal with AUD$, FGA_LOG$ or DVSYS.AUDIT_TRAIL$ tables and all kinds of audit files. Everything is all right, isn’t it?

Unfortunately not quite. Oracle left out the possibility to write audit records to SYSLOG in the first implementations of Oracle Unified Audit. With the release Oracle 18c later 19c this functionality was added again step by step. But before you get too excited and start switching the audit trail back to SYSLOG, you need to take a look at one or two things. The new initialisation parameters UNIFIED_AUDIT_SYSTEMLOG and UNIFIED_AUDIT_COMMON_SYSTEMLOG do not work quite the same as AUDIT_SYSLOG_LEVEL used to. In this blog post, we’ll take a closer look at audit configuration in the context of SYSLOG.

Background

Even with the use of the SYSLOG configuration for Oracle Unified Audit, the Unified Audit Infrastructure as well as the audit policies must be configured as before. In this respect, nothing changes with SYSLOG forwarding. With the new initialisation parameters below, Oracle only enables that the audit records are additionally forwarded to SYSLOG in a reduced form.

• UNIFIED_AUDIT_SYSTEMLOG specifies whether key fields of unified audit records will be written to the SYSLOG utility (on UNIX platforms) or to the Windows Event Viewer (on Windows). In a CDB, this parameter is a per-PDB static initialisation parameter. This parameter is available from Oracle 18c.
• UNIFIED_AUDIT_COMMON_SYSTEMLOG specifies whether key fields of unified audit records generated due to common audit policies will be written to the SYSLOG utility. This parameter is available only from Oracle 19c.

The following sketch shows the schematic structure of the unified audit configuration with ¡ forwarding in an Oracle Multitenant Database. Whereby the following has been configured:

• Common audit policy COMMON_ORA_LOGON for audit action LOGON
• Local audit policy LOCAL_ORA_LOGON for audit action LOGON
• UNIFIED_AUDIT_SYSTEMLOG parameter set to SYSLOG facility local0.info
• UNIFIED_AUDIT_COMMON_SYSTEMLOG parameter set to SYSLOG facility local1.info

The following use cases are drawn in the sketch:

Common User Login to CDB$ROOT

1. A common user is logged into the CDB$ROOT and COMMON_ORA_LOGON does create an audit event for audit action LOGON
2. A single audit record is created in the local UNIFIED_AUDIT_TRAIL of CDB$ROOT
3. Additionally An audit record is created in the SYSLOG facility local1.warning

Common User Login to PDB01

1. A common user is logged into the PDB01 and COMMON_ORA_LOGON as well LOCAL_ORA_LOGON does create an audit event for audit action LOGON
2. A single audit record is created in the local UNIFIED_AUDIT_TRAIL of PDB01 
3. Additionally an audit record is created in the SYSLOG facility local1.info

Local User Login to PDB01

1. A local user is logged into the PDB02 and LOCAL_ORA_LOGON does create an audit event for audit action LOGON
2. A single audit record is created in the local UNIFIED_AUDIT_TRAIL of PDB02
3. Additionally an audit record is created in the SYSLOG facility local0.warning

Note: An audit record is created in UNIFIED_AUDIT_TRAIL as well as in SYSLOG. This means that you also have to define a corresponding housekeeping for UNIFIED_AUDIT_TRAIL.

Setup Unified Audit with SYSLOG Integration

Configure SYSLOG

First we do have to configure corresponding SYSLOG destinations for our database audit information. In the following I will not go into the detailed configuration of SYSLOG respectively RSYSLOG. We only extend the configuration with two additional log files. The first thing to do is to edit the /etc/rsyslog.conf file as root user. We add two new local facilities under the RULES section.

sudo vi /etc/rsyslog.conf

# Unified Audit Rules
local0.info            /var/log/oracle_common_audit_records.log
local1.info            /var/log/oracle_audit_records.log

Afterwards the RSYSLOG service must be restarted.

sudo systemctl restart rsyslog.service 

Audit Initialisation Parameters

The Oracle initialisation parameter for the audit configuration requires an instance restart and has to be modified via CDB$ROOT. Below we just change UNIFIED_AUDIT_COMMON_SYSTEMLOG optionally we can also forward local audit records to SYSLOG by setting UNIFIED_AUDIT_SYSTEMLOG.

• Connect as SYS to CDB$ROOT and change UNIFIED_AUDIT_COMMON_SYSTEMLOG

CONNECT / AS SYSDBA
SHOW PARAMETER unified_audit_common_systemlog
ALTER SYSTEM SET unified_audit_common_systemlog='local0.info' SCOPE=SPFILE;

• Restart the whole container database

STARTUP FORCE;
SHOW PARAMETER unified_audit_common_systemlog

As of now, audit records for common audit events are forwarded to the appropriate SYSLOG facility.

Audit Policies

For simplicity, we test the audit configuration with a few simple audit policies for the audit action LOGON. Of course, audit policies can be defined for any actions.

• Create a common audit policy to log all logon events of common users in CDB$ROOT or any PDB.

CONNECT / AS SYSDBA
CREATE AUDIT POLICY common_ora_logon ACTIONS LOGON CONTAINER=ALL;
AUDIT POLICY common_ora_logon;

• Check which audit policies are enabled.

SET LINESIZE WINDOW
COL policy_name FOR A16
COL entity_name FOR A11
SELECT * FROM audit_unified_enabled_policies;
 
POLICY_NAME	 ENABLED_OPTION  ENTITY_NAME ENTITY_ SUC FAI
---------------- --------------- ----------- ------- --- ---
COMMON_ORA_LOGON BY USER	 ALL USERS   USER    YES YES

• Create a local audit policy to log all logon events of local users in a particular PDB.

ALTER SESSION SET CONTAINER=pdb1;
CREATE AUDIT POLICY local_ora_logon ACTIONS LOGON;
AUDIT POLICY local_ora_logon;

• Check which audit policies are enabled. We can now see that the local audit policy as well as the common audit policy from before are active in the PDB.

SET LINESIZE WINDOW
COL policy_name FOR A16
COL entity_name FOR A11
SELECT * FROM audit_unified_enabled_policies;
 
POLICY_NAME	 ENABLED_OPTION  ENTITY_NAME ENTITY_ SUC FAI
---------------- --------------- ----------- ------- --- ---
LOCAL_ORA_LOGON  BY USER	 ALL USERS   USER    YES YES
COMMON_ORA_LOGON BY USER	 ALL USERS   USER    YES YES

Test Audit Configuration

Let’s purge the audit trail in CDB$ROOT as well PDB1 have not too much information in the trail.

CONNECT / AS SYSDBA
BEGIN
    dbms_audit_mgmt.clean_audit_trail(
    audit_trail_type         =>  dbms_audit_mgmt.audit_trail_unified,
    use_last_arch_timestamp  =>  FALSE);
END;
/
 
ALTER SESSION SET container=PDB1;
BEGIN
    dbms_audit_mgmt.clean_audit_trail(
    audit_trail_type         =>  dbms_audit_mgmt.audit_trail_unified,
    use_last_arch_timestamp  =>  FALSE);
END;
/

First we do a login as user SYSTEM to CDB$ROOT of database TDB19C

sqlplus system@TDB19C

Lets check what we do see in the view UNIFIED_AUDIT_TRAIL

SET LINESIZE 160 PAGESIZE 200
COL event_timestamp FOR A17
COL dbusername FOR A10
COL action_name FOR A17
COL return_code FOR 999999
COL unified_audit_policies FOR A30
 
SELECT
    to_char(event_timestamp,'dd.mm.yy hh24:mi:ss') event_timestamp,
    sessionid,
    dbusername,
    action_name,
    return_code,
    unified_audit_policies
FROM unified_audit_trail
ORDER BY event_timestamp;
 
EVENT_TIMESTAMP    SESSIONID DBUSERNAME ACTION_NAME	  RETURN_CODE UNIFIED_AUDIT_POLICIES
----------------- ---------- ---------- ----------------- ----------- ------------------------------
22.03.22 09:07:03 1430860507 SYS	LOGOFF BY CLEANUP	    0
23.03.22 14:49:24 2404020191 SYS	EXECUTE 		    0
23.03.22 14:49:44 2578688223 SYSTEM	LOGON			    0 COMMON_ORA_LOGON

Lets see what we do have in the SYSLOG log file

host sudo grep -i 2578688223 /var/log/oracle_common_audit_records.log
Mar 23 14:49:44 localhost journal: Oracle Unified Audit[17838]: LENGTH: '204' TYPE:"4" DBID:"1612911514" SESID:"2578688223" CLIENTID:"" ENTRYID:"1" STMTID:"1" DBUSER:"SYSTEM" CURUSER:"SYSTEM" ACTION:"100" RETCODE:"0" SCHEMA:"" OBJNAME:"" PDB_GUID:"86B637B62FDF7A65E053F706E80A27CA"

The action number can be locked up in the table AUDIT_ACTIONS

SELECT * FROM audit_actions WHERE action=100;      

    ACTION NAME
---------- ----------------------------
       100 LOGON

Now lets see what happens when we login as SYSTEM into PDB1

CONNECT system@pdb1.trivadislabs.com

We now do see an audit record in the UNIFIED_AUDIT_TRAIL of the PDB. Active Policy for this common user is LOCAL_ORA_LOGON and COMMON_ORA_LOGON.

SET LINESIZE 160 PAGESIZE 200
COL event_timestamp FOR A17
COL dbusername FOR A10
COL action_name FOR A11
COL return_code FOR 999999
COL unified_audit_policies FOR A33
 
SELECT
    to_char(event_timestamp,'dd.mm.yy hh24:mi:ss') event_timestamp,
    sessionid,
    dbusername,
    action_name,
    return_code,
    unified_audit_policies
FROM unified_audit_trail
ORDER BY event_timestamp;

EVENT_TIMESTAMP    SESSIONID DBUSERNAME ACTION_NAME RETURN_CODE UNIFIED_AUDIT_POLICIES
----------------- ---------- ---------- ----------- ----------- ---------------------------------
23.03.22 14:54:22 1216566979 SYS	EXECUTE 	      0
23.03.22 14:55:01 3827730564 SYSTEM	LOGON		      0 LOCAL_ORA_LOGON, COMMON_ORA_LOGON

Lets see what we do have in the SYSLOG logfile

host sudo grep -i 3827730564 /var/log/oracle_common_audit_records.log
Mar 23 14:55:01 localhost journal: Oracle Unified Audit[18210]: LENGTH: '203' TYPE:"4" DBID:"817014372" SESID:"3827730564" CLIENTID:"" ENTRYID:"1" STMTID:"1" DBUSER:"SYSTEM" CURUSER:"SYSTEM" ACTION:"100" RETCODE:"0" SCHEMA:"" OBJNAME:"" PDB_GUID:"B8E3D716A96C1507E0530100007F363B"

As a final test, we log into PDB1 as local user SCOTT.

connect scott/tiger@pdb1.trivadislabs.com

There is now a new audit record for SCOTT in the UNIFIED_AUDIT_TRAIL of the PDB. Active policy for this local user is LOCAL_ORA_LOGON.

SET LINESIZE 160 PAGESIZE 200
COL event_timestamp FOR A17
COL dbusername FOR A10
COL action_name FOR A11
COL return_code FOR 999999
COL unified_audit_policies FOR A33
 
SELECT
    to_char(event_timestamp,'dd.mm.yy hh24:mi:ss') event_timestamp,
    sessionid,
    dbusername,
    action_name,
    return_code,
    unified_audit_policies
FROM unified_audit_trail
ORDER BY event_timestamp;

EVENT_TIMESTAMP    SESSIONID DBUSERNAME ACTION_NAME RETURN_CODE UNIFIED_AUDIT_POLICIES
----------------- ---------- ---------- ----------- ----------- ---------------------------------
23.03.22 14:54:22 1216566979 SYS	EXECUTE 	      0
23.03.22 14:55:01 3827730564 SYSTEM	LOGON		      0 LOCAL_ORA_LOGON, COMMON_ORA_LOGON
23.03.22 14:59:26 2954396682 SCOTT	LOGON		      0 LOCAL_ORA_LOGON

Because we have only set the parameter UNIFIED_AUDIT_COMMON_SYSTEMLOG and regular audit policies are not forwarded to SYSLOG, we do not find an entry in the corresponding SYSLOG log file.

host sudo grep -i 2954396682 /var/log/oracle_common_audit_records.log
host sudo grep -i 2954396682 /var/log/oracle_audit_records.log

Conclusion

Although many things have been simplified with Oracle Unified Audit, it is easy to lose Although much has been simplified with Oracle Unified Audit, it is easy to lose track of all the common and local audit policies in an Oracle multitenant environment. An up-to-date audit concept that takes these special cases into account is absolutely essential. This includes the use of the new initialisation parameters for SYSLOG integration. Although the information in SYSLOG is nowhere near as rich as in UNIFIED_AUDIT_TRAIL itself, this feature allows easy integration of Oracle audit events into a central repository, e.g. Splunk, Elastiksearch or similar, to create a simple audit dashboard. The true source of the audit data with information on the complete SQL statements, detailed user information etc. remains reserved for the Unified Audit trail in the database.

I have mentioned SYSLOG and RSYSLOG alternately in the blog post. The Oracle feature is basically for SYSLOG and all services based on SYSLOG. Whereas on my OCI Compute instance, where I did my tests, RSYSLOG is configured and used.

References

• Oracle® Database SQL Language Reference 21c AUDIT (Unified Auditing)
• Oracle® Database Database Reference 21c UNIFIED_AUDIT_SYSTEMLOG
• Oracle® Database Database Reference 21c UNIFIED_AUDIT_COMMON_SYSTEMLOG
• Oracle Support Document 2623138.1 How to write Unified Audit Trail Records to SYSLOG in 18c
• Oracle Support Document 1582627.1 How To Purge The UNIFIED AUDIT TRAIL