Author: mardaff

I am an Oracle DBA with experience in Maximum Availability technology such as RAC, Data Guard, Grid Control and ASM. I worked with version 9i through 12c on a number of operating systems. Next to that I have an extensive experience with PL/SQL, Oracle eBusiness Suite and shell scripting. Specialties: Oracle RAC & ASM (Configuration/Monitoring/Tuning) Oracle Data Guard (Configuration/Monitoring/Tuning) EM12c (Installation / troubleshooting) SQL & PLSQL *nix Shell scripting
Posted in Postgres

Postgres: PANIC: could not locate a valid checkpoint record

Posted on by mardaff

I recently started to play with Postgres and this was one of the errors I got from my setup when I was trying to bring up the database:

 

[Tue Nov 06 09:58:10 root@postgres-primary:~ ] $ su - postgres -c '/usr/pgsql-10/bin/pg_ctl -D $PGDATA start'
waiting for server to start....2018-11-06 09:58:37.346 UTC [21595] LOG: listening on IPv4 address "127.0.0.1", port 5444
2018-11-06 09:58:37.349 UTC [21595] LOG: listening on Unix socket "/var/run/postgresql/.s.PGSQL.5444"
2018-11-06 09:58:37.355 UTC [21595] LOG: listening on Unix socket "/tmp/.s.PGSQL.5444"
2018-11-06 09:58:37.375 UTC [21595] LOG: redirecting log output to logging collector process
2018-11-06 09:58:37.375 UTC [21595] HINT: Future log output will appear in directory "/app/pgsql/admin/test/pg_log".
stopped waiting
pg_ctl: could not start server
Examine the log output.


[Tue Nov 06 10:04:29 root@postgres-primary:/app/pgsql/admin/test/pg_log ] $ tail -f pglog_test_20181106.log
2018-11-06 09:56:12.529 UTC [21573] LOG: startup process (PID 21575) was terminated by signal 6: Aborted
2018-11-06 09:56:12.529 UTC [21573] LOG: aborting startup due to startup process failure
2018-11-06 09:56:12.530 UTC [21573] LOG: database system is shut down
2018-11-06 09:58:37.379 UTC [21597] LOG: database system was shut down at 2018-11-06 09:46:58 UTC
2018-11-06 09:58:37.379 UTC [21597] LOG: invalid primary checkpoint record
2018-11-06 09:58:37.379 UTC [21597] LOG: invalid secondary checkpoint record
2018-11-06 09:58:37.379 UTC [21597] PANIC: could not locate a valid checkpoint record
2018-11-06 09:58:37.379 UTC [21595] LOG: startup process (PID 21597) was terminated by signal 6: Aborted
2018-11-06 09:58:37.379 UTC [21595] LOG: aborting startup due to startup process failure
2018-11-06 09:58:37.381 UTC [21595] LOG: database system is shut down

 

The solution is to perform a “Write-ahead log reset”:

[Tue Nov 06 10:02:24 root@postgres-primary:/app/pgsql/admin/test/pg_log ] $ su - postgres -c '/usr/pgsql-10/bin/pg_resetwal $PGDATA'
Write-ahead log reset

 

And start the database:

[Tue Nov 06 10:06:25 root@postgres-primary:/app/pgsql/admin/test/pg_log ] $ su - postgres -c '/usr/pgsql-10/bin/pg_ctl -D $PGDATA start'
waiting for server to start....2018-11-06 10:06:33.452 UTC [21668] LOG: listening on IPv4 address "127.0.0.1", port 5444
2018-11-06 10:06:33.454 UTC [21668] LOG: listening on Unix socket "/var/run/postgresql/.s.PGSQL.5444"
2018-11-06 10:06:33.458 UTC [21668] LOG: listening on Unix socket "/tmp/.s.PGSQL.5444"
2018-11-06 10:06:33.467 UTC [21668] LOG: redirecting log output to logging collector process
2018-11-06 10:06:33.467 UTC [21668] HINT: Future log output will appear in directory "/app/pgsql/admin/test/pg_log".
done
server started

 

Advertisements
Posted in Linux, Oracle

Linux mpstat Command

Posted on by mardaff

What is mpstat?

mpstat is used to monitor cpu utilization on your system. It will be more useful if your system has multiple processors. The first processors will signed as CPU 0. The second one will be signed CPU 1 and so on.

The mpstat command writes to standard output activities for each available processor, processor 0 being the first one. Global average activities among all processors are also reported. The mpstat command can be used both on SMP and UP machines, but in the latter, only global average activities will be printed. If no activity has been selected, then the default report is the CPU utilization report:

 mpstat -P ALL 5 1

It shows the various stats for the CPUs in the system. The –P ALL options directs the command to display stats for all the CPUs, not just a specific one. The parameters 5 2 directs the command to run every 5 seconds and for 2 times. The above output shows the metrics for all the CPUs first (aggregated) and for each CPU individually. Finally, the average for all the CPUs has been shown at the end.

Let’s see what the column values mean:

%user   -Indicates the percentage of the processing for that CPU consumes by user processes. User processes are non-kernel processes used for applications such as an Oracle database. In this example output, the user CPU %age is very little.
%nice   -Indicates the percentage of CPU when a process was downgraded by nice command. The command nice has been described in an earlier installment. It brief, the command nice changes the priority of a process.
%system -Indicates the CPU percentage consumed by kernel processes
%iowait -Shows the percentage of CPU time consumed by waiting for an I/O to occur
%irq    -Indicates the %age of CPU used to handle system interrupts
%soft   -Indicates %age consumed for software interrupts
%idle   -Shows the idle time of the CPU
%intr/s -Shows the total number of interrupts received by the CPU per second

It’s possible that a poorly written application not using multi-threadd architecture runs on a multi-processor machine but does not use all the processors. As a result, one CPU overloads while others remain free. You can easily diagnose these sorts of issues via mpstat.

And now and example:

oracle@primary:PRI::$ mpstat -P ALL 5 1
Linux 3.10.0-862.11.6.el7.x86_64 (primary.localdomain) 10/15/2018 _x86_64_(2 CPU)

07:35:11 AM CPU %usr %nice %sys %iowait %irq %soft %steal %guest %gnice %idle
07:35:16 AM all 0.51 0.00 0.51 0.61 0.00 0.00 0.00 0.00 0.00 98.36
07:35:16 AM 0 0.82 0.00 0.62 0.21 0.00 0.00 0.00 0.00 0.00 98.36
07:35:16 AM 1 0.20 0.00 0.40 1.21 0.00 0.20 0.00 0.00 0.00 97.98

Average: CPU %usr %nice %sys %iowait %irq %soft %steal %guest %gnice %idle
Average: all 0.51 0.00 0.51 0.61 0.00 0.00 0.00 0.00 0.00 98.36
Average: 0 0.82 0.00 0.62 0.21 0.00 0.00 0.00 0.00 0.00 98.36
Average: 1 0.20 0.00 0.40 1.21 0.00 0.20 0.00 0.00 0.00 97.98
oracle@primary:PRI::$

Print CPU utilization using intervals

You may want to see the CPU utilization movement. To do this, you can use intervals. Here’s an example.
The below command is to show you 3 reports about CPU utilization with 3 seconds interval:

oracle@primary:PRI::$ mpstat 1 2
Linux 3.10.0-862.11.6.el7.x86_64 (primary.localdomain) 10/15/2018 _x86_64_(2 CPU)

07:40:37 AM CPU %usr %nice %sys %iowait %irq %soft %steal %guest %gnice %idle
07:40:38 AM all 1.03 0.00 0.00 0.51 0.00 0.00 0.00 0.00 0.00 98.46
07:40:39 AM all 0.51 0.00 0.51 1.03 0.00 0.00 0.00 0.00 0.00 97.95
Average: all 0.77 0.00 0.26 0.77 0.00 0.00 0.00 0.00 0.00 98.21
oracle@primary:PRI::$
Posted in EM12c, EM13c, Oracle

EM 12c, 13c: Enterprise Manager Cloud Control Agent Install Fails at Initialization Step With Error: SSH daemon (sshd) is not running on port 22

Posted on by mardaff

Screen Shot 2018-06-11 at 9.03.06 AM

This error can occur due to one of the following reasons:

1. SSH port 22 is blocked by proxy / firewall on the Target machine
2. SSH is not running on the Target machine
3. SSH daemon is running on non-default port (other than 22)

1. Verify that the SSH port 22 is not blocked from the OMS host using the following command (a blocked port will be indicated, a free port will result in no response):

telnet 22

2. Verify that SSH is running on the target host using

ps -ef | grep ssh
 netstat -anp | grep 22

The Prerequisites for 12c Agent installation using Push Method clearly states that the SSH daemon should be running on the default port (that is, 22) on all the destination hosts.

3. If SSH on the target host is running on a non-default port, then update the SSH_PORT property in $/oui/prov/resources/Paths.properties as below:

SSH_PORT=
  • If you are installing on target A where SSH is running on port 99; then you would need to make the above change. If the next Agent Deployment is on target B where SSH is running on default port 22; then you will need to revert the changes to set SSH_PORT= or SSH_PORT=22 in $/oui/prov/resources/Paths.properties file.
  • You cannot install Agents on target X and target Y at the same time, if they have SSH running on different ports.
Posted in EM13c, Oracle

Patch EM13c

Posted on by mardaff

After installing EM13c, you may want to install the latest patches as well. We’ll start with OPatch and OMSPatcher.

Download the following patches from Oracle Support:

p6880880_139000_Generic.zip => OPatch
p19999993_132000_Generic.zip => OMSPatch

Go to your patching staging area:

cd /app/oracle/media/patches
unzip p6880880_139000_Generic.zip

Check the existing versions:

[oracle@oms13c]$ $OMS_HOME/OMSPatcher/omspatcher version
OMSPatcher Version: 13.8.0.0.0
OPlan Version: 12.1.0.2.2
OsysModel build: Mon Dec 21 18:31:52 PST 2015
OMSPatcher succeeded.
[oracle@oms13c]$ $OMS_HOME/OPatch/opatch version
OPatch Version: 13.8.0.0.0
OPatch succeeded.
[oracle@oms13c]$
[oracle@oms13c]$ $AGENT_HOME/OPatch/opatch version
OPatch Version: 13.8.0.0.0
OPatch succeeded.
[oracle@oms13c agent_13.2.0.0.0]$

Now, let’s take some backups before deploying the new patch version:

cd $OMS_HOME
[oracle@oms13c]$ mv OMSPatcher OMSPatcher.OLD
[oracle@oms13c]$ mv OPatch OPatch.OLD

Upgrade the OPatch on 13.2 OMS Home :

[oracle@oms13c]$ /app/oracle/em13c/oracle_common/jdk/bin/java -jar /app/oracle/media/patches/6880880/opatch_generic.jar -silent oracle_home=/app/oracle/em13c/
Launcher log file is /tmp/OraInstall2018-05-24_09-45-32AM/launcher2018-05-24_09-45-32AM.log.
Extracting the installer . . . . Done
Checking if CPU speed is above 300 MHz.   Actual 2596.992 MHz    Passed
Checking swap space: must be greater than 512 MB.   Actual 2047 MB    Passed
Checking if this platform requires a 64-bit JVM.   Actual 64    Passed (64-bit not required)
Checking temp space: must be greater than 300 MB.   Actual 1954 MB    Passed
Preparing to launch the Oracle Universal Installer from /tmp/OraInstall2018-05-24_09-45-32AM
Installation Summary
Disk Space : Required 26 MB, Available 31,479 MB

Feature Sets to Install:
 Next Generation Install Core 13.9.1.0.1
 OPatch 13.9.1.3.0
 OPatch Auto OPlan 13.9.1.0.0
Session log file is /tmp/OraInstall2018-05-24_09-45-32AM/install2018-05-24_09-45-32AM.log
Loading products list. Please wait.
 1%
 40%
Loading products. Please wait.
 42%
 43%
 44%
 45%

.........

S_COMP_INPROGRESS_SAVE_INVENTORY
 Component : oracle.glcm.opatchauto.core.actions.classpath
S_COMP_INPROGRESS_SAVE_INVENTORY
 Component : oracle.glcm.opatchauto.core.wallet.classpath
S_COMP_INPROGRESS_SAVE_INVENTORY
Logs successfully copied to /app/oraInventory/logs.

Now let’s check the version:

[oracle@oms13c]$ $OMS_HOME/OPatch/opatch version
OPatch Version: 13.9.1.3.0
OPatch succeeded.
[oracle@oms13c]$

Upgrade the OPatch on 13.2 Agent Home

Stop the agent

./emctl stop agent

Let’s make a backup of OPatch:

cd $AGENT_HOME

mv OPatch OPatch.OLD

Run the patch:

[oracle@oms13c]$ /app/oracle/em13c/agent/jdk/bin/java -jar /app/oracle/media/patches/6880880/opatch_generic.jar -silent oracle_home=/app/oracle/em13c/agent13c

Launcher log file is /tmp/OraInstall2018-05-24_09-56-56AM/launcher2018-05-24_09-56-56AM.log.
Extracting the installer . . . . Done
Checking if CPU speed is above 300 MHz.   Actual 2596.992 MHz    Passed
Checking swap space: must be greater than 512 MB.   Actual 2047 MB    Passed
Checking if this platform requires a 64-bit JVM.   Actual 64    Passed (64-bit not required)
Checking temp space: must be greater than 300 MB.   Actual 1954 MB    Passed
Preparing to launch the Oracle Universal Installer from /tmp/OraInstall2018-05-24_09-56-56AM
Installation Summary
Disk Space : Required 26 MB, Available 31,373 MB
Feature Sets to Install:
 Next Generation Install Core 13.9.1.0.1
 OPatch 13.9.1.3.0
 OPatch Auto OPlan 13.9.1.0.0
Session log file is /tmp/OraInstall2018-05-24_09-56-56AM/install2018-05-24_09-56-56AM.log
Loading products list. Please wait.
 1%
 40%
Loading products. Please wait.
 42%
 43%
 44%

......

S_COMP_INPROGRESS_SAVE_INVENTORY
 Component : oracle.glcm.opatchauto.core.wallet.classpath
S_COMP_INPROGRESS_SAVE_INVENTORY
 Component : jsch.lib
S_COMP_INPROGRESS_SAVE_INVENTORY
Logs successfully copied to /app/oraInventory/logs.

Check the version

[oracle@oms13c]$ $AGENT_HOME/OPatch/opatch version
OPatch Version: 13.9.1.3.0
OPatch succeeded.
[oracle@oms13c]$

Start the agent

./emctl start agent

Install OMS Patcher:

This is done in the traditional way:

unzip p19999993_132000_Generic.zip -d $OMS_HOME

Check now the OPatch version

[oracle@oms13c]$ $OMS_HOME/OMSPatcher/omspatcher version
OMSPatcher Version: 13.8.0.0.2
OPlan Version: 12.2.0.1.6
OsysModel build: Tue Apr 04 07:25:39 UTC 2017
OMSPatcher succeeded.

 

OMS Patch OMS-patch-25841652 ( p25841652_132000_Generic.zip)

unzip p25841652_132000_Generic.zip -d OMS-patch-25841652

Stop OMS (but keep the weblogic components up):

[oracle@oms13c]$ ./emctl stop oms
Oracle Enterprise Manager Cloud Control 13c Release 2 
Copyright (c) 1996, 2016 Oracle Corporation.  All rights reserved.
Stopping Oracle Management Server...
Oracle Management Server Successfully Stopped
Oracle Management Server is Down
JVMD Engine is Down
[oracle@oms13c]$ ./emctl status oms
Oracle Enterprise Manager Cloud Control 13c Release 2 
Copyright (c) 1996, 2016 Oracle Corporation.  All rights reserved.
WebTier is Up
Oracle Management Server is Down
JVMD Engine is Down
BI Publisher Server is Up
[oracle@oms13c]$

Go to the folder where u have unzipped the patch and run the analyzer:

cd /app/oracle/media/patches/OMS-patch-25841652/25841652
[oracle@oms13c 25841652]$ $OMS_HOME/OMSPatcher/omspatcher apply -analyze OMSPatcher.OMS_DISABLE_HOST_CHECK=true
OMSPatcher Automation Tool
Copyright (c) 2017, Oracle Corporation.  All rights reserved.
OMSPatcher version : 13.8.0.0.2
OUI version        : 13.9.1.0.0
Running from       : /app/oracle/em13c
Log file location  : /app/oracle/em13c/cfgtoollogs/omspatcher/opatch2018-05-24_11-52-32AM_1.log
OMSPatcher log file: /app/oracle/em13c/cfgtoollogs/omspatcher/25841652/omspatcher_2018-05-24_11-52-38AM_analyze.log
Please enter OMS weblogic admin server URL(xxxxxxx:xxxx):> 
Please enter OMS weblogic admin server username(xxxxxxxxx):> 
Please enter OMS weblogic admin server password:> 
WARNING: Could not apply the patch "25414328" because the "oracle.sysman.vi.oms.plugin with version 13.2.1.0.0" core component of the OMS or the plug-in for which the patch is intended is either not deployed or deployed with another version in your Enterprise Manager system.
WARNING: Could not apply the patch "25414306" because the "oracle.sysman.emfa.oms.plugin with version 13.2.1.0.0" core component of the OMS or the plug-in for which the patch is intended is either not deployed or deployed with another version in your Enterprise Manager system.
WARNING: Could not apply the patch "25118889" because the "oracle.sysman.vt.oms.plugin with version 13.2.1.0.0" core component of the OMS or the plug-in for which the patch is intended is either not deployed or deployed with another version in your Enterprise Manager system.
WARNING: Could not apply the patch "25414263" because the "oracle.sysman.csm.oms.plugin with version 13.2.1.0.0" core component of the OMS or the plug-in for which the patch is intended is either not deployed or deployed with another version in your Enterprise Manager system.
WARNING: Could not apply the patch "25580681" because the "oracle.sysman.ssa.oms.plugin with version 13.2.1.0.0" core component of the OMS or the plug-in for which the patch is intended is either not deployed or deployed with another version in your Enterprise Manager system.
WARNING: Could not apply the patch "25414356" because the "oracle.sysman.smf.oms.plugin with version 13.2.1.0.0" core component of the OMS or the plug-in for which the patch is intended is either not deployed or deployed with another version in your Enterprise Manager system.
Configuration Validation: Success

Running apply prerequisite checks for sub-patch(es) "25414294,25414339,25414245,25765741,25739957" and Oracle Home "/app/oracle/em13c"...

Sub-patch(es) "25414294,25414339,25414245,25765741,25739957" are successfully analyzed for Oracle Home "/app/oracle/em13c"

Complete Summary

================

All log file names referenced below can be accessed from the directory "/app/oracle/em13c/cfgtoollogs/omspatcher/2018-05-24_11-52-32AM_SystemPatch_25841652_1"

Prerequisites analysis summary:

-------------------------------

The following sub-patch(es) are applicable:

             Featureset                                    Sub-patches                                                                         Log file
             ----------                                    -----------                                                                         --------
  oracle.sysman.top.oms   25414294,25414339,25414245,25765741,25739957   25414294,25414339,25414245,25765741,25739957_opatch2018-05-24_11-52-37AM_1.log

The following sub-patches are incompatible with components installed in the OMS system:
25414328,25414306,25118889,25414263,25580681,25414356

--------------------------------------------------------------------------------
The following warnings have occurred during OPatch execution:
1)  Could not apply the patch "25414328" because the "oracle.sysman.vi.oms.plugin with version 13.2.1.0.0" core component of the OMS or the plug-in for which the patch is intended is either not deployed or deployed with another version in your Enterprise Manager system.
2)  Could not apply the patch "25414306" because the "oracle.sysman.emfa.oms.plugin with version 13.2.1.0.0" core component of the OMS or the plug-in for which the patch is intended is either not deployed or deployed with another version in your Enterprise Manager system.
3)  Could not apply the patch "25118889" because the "oracle.sysman.vt.oms.plugin with version 13.2.1.0.0" core component of the OMS or the plug-in for which the patch is intended is either not deployed or deployed with another version in your Enterprise Manager system.
4)  Could not apply the patch "25414263" because the "oracle.sysman.csm.oms.plugin with version 13.2.1.0.0" core component of the OMS or the plug-in for which the patch is intended is either not deployed or deployed with another version in your Enterprise Manager system.
5)  Could not apply the patch "25580681" because the "oracle.sysman.ssa.oms.plugin with version 13.2.1.0.0" core component of the OMS or the plug-in for which the patch is intended is either not deployed or deployed with another version in your Enterprise Manager system.
6)  Could not apply the patch "25414356" because the "oracle.sysman.smf.oms.plugin with version 13.2.1.0.0" core component of the OMS or the plug-in for which the patch is intended is either not deployed or deployed with another version in your Enterprise Manager system.
--------------------------------------------------------------------------------
OMSPatcher Session completed with warnings.
Log file location: /app/oracle/em13c/cfgtoollogs/omspatcher/25841652/omspatcher_2018-05-24_11-52-38AM_analyze.log

OMSPatcher completed with warnings.
[oracle@oms13c 25841652]$

Ignore these errors and move on with deploying the patch:

$OMS_HOME/OMSPatcher/omspatcher apply OMSPatcher.OMS_DISABLE_HOST_CHECK=true

After it’s finished:

$OMS_HOME/OMSPatcher/omspatcher lspatches

Start OMS:

./emctl start oms

 

 

 

Posted in EM13c, Oracle

Install EM13c in silent mode

Posted on by mardaff

Suppose you have already the server created.

As root user install the necessary packages for EM13c:

yum install make -y
yum install binutils -y
yum install gcc -y
yum install libaio -y
yum install glibc-common -y
yum install libstdc++ -y
yum install libXtst -y
yum install sysstat -y
yum install glibc -y
yum install glibc-devel -y
yum install glibc-devel.i686 -y

Create the necessary folders (as oracle)

mkdir –p /app/oraInventory
mkdir -p /app/oracle/em13c/middleware
mkdir -p /app/oracle/em13c/agent13c
mkdir -p /app/oracle/em13c/gc_inst
mkdir -p /app/oracle/em13c/swlib

Run the installer (if you don’t have enough tmp space one can use  –J-Djava.io.tmpdir=

Don’t forget to make the em13200_linux64.bin executable: 

chmod u+x em13200_linux64.bin

If you don’t have enough tmp space don’t forget to create the path where you have space:

mkdir -p /app/em13-temp

Get the response file and edit it:

[oracle@oms13c]$  ./em13200_linux64.bin -J-Djava.io.tmpdir=/app/em13-temp -getResponseFileTemplates -outputLoc /app/oracle/media

0%.........................................................................100%

Launcher log file is /app/em13-temp/OraInstall2018-05-23_10-10-54AM/launcher2018-05-23_10-10-54AM.log.

Copying response file template(s)

to /app2/oracle/media ...

  Copying response file template software_only.rsp

  Copying response file template new_install.rsp

  Copying response file template upgrade.rsp

Finished copying response file template(s)

The log(s) can be found here: /app/em13-temp/OraInstall2018-05-23_10-10-54AM.

Make a copy of it and make sure u remove all the “” :

cp new_install.rsp my_new_install.rsp
vi my_new_install.rsp

Now you’re good to run the installer and wait for about 2h 🙂 :

[oracle@oms13c]$./em13200_linux64.bin -J-Djava.io.tmpdir=/app/em13-temp -silent -ignoreSysPrereqs -responseFile /responsefile-location/my_new_install.rsp

 

Posted in EM13c, Oracle

How to get the response file for a silent install of EM13c

Posted on by mardaff 
[oracle@oms13c]$  ./em13200_linux64.bin -J-Djava.io.tmpdir=/app/em13-temp -getResponseFileTemplates -outputLoc /app/oracle/media

0%.........................................................................100%

Launcher log file is /app/em13-temp/OraInstall2018-05-23_10-10-54AM/launcher2018-05-23_10-10-54AM.log.

Copying response file template(s)

to /app2/oracle/media ...

  Copying response file template software_only.rsp

  Copying response file template new_install.rsp

  Copying response file template upgrade.rsp

Finished copying response file template(s)

The log(s) can be found here: /app/em13-temp/OraInstall2018-05-23_10-10-54AM.