If HANA is running into a Disk Full Situation, mostly in a replication setup, it is the /hana/log volume filling up with Log Segments.
To get the actual Log Segment Situation and partial immediate help, either use:
Option A, via SQL statement:
select b.host, b.service_name, a.state, count(*) from “PUBLIC”.”M_LOG_SEGMENTS” a join “PUBLIC”.”M_SERVICES” b on (a.host = b.host AND a.port = b.port) group by b.host, b.service_name, a.state;
Option B, via OS command as <sid>adm user:
hdbnsutil -listServices –allDatabases –sapcontrol=1 | grep -P “service_name|process_id|volume|database_name” | awk ‘/^service.*service_name=/ {s=$0; next} /^service.*process_id=/ {p=$0; next} /^service.*volume=/ {split($0,v,”=”); if (v[2] != 0) {print s; print p; print $0; show_dbname=1} else {show_dbname=0}; next} /^service.*database_name=/ {if (show_dbname) {print $0; print “”}}’; for i in $(find -L /hana/log/${DIR_INSTANCE:9:3} -name ‘hdb*’ -type d) ; do echo $i ; hdblogdiag seglist $i | grep -oP ‘Formatting|Preallocated|Writing|bClosedb|Truncated|BackedUp|RetainedFree|bFreeb|Unknown|ClosedIncomplete|FreeIncomplete’ | sort | uniq -c ; done
A Sample of Option B:
Sample Setup:
physical Hostname: hdb-11-85static
virtual Hosntame: hana-11-virtual
SID: HSR
Instance ID: 77hdb-11-85static:~ # su -l <sid>adm
hsradm@hdb-11-85static:/usr/sap/HSR/HDB77> hdbnsutil -listServices –allDatabases –sapcontrol=1 | grep -P “service_name|process_id|volume|database_name” | awk ‘/^service.*service_name=/ {s=$0; next} /^service.*process_id=/ {p=$0; next} /^service.*volume=/ {split($0,v,”=”); if (v[2] != 0) {print s; print p; print $0; show_dbname=1} else {show_dbname=0}; next} /^service.*database_name=/ {if (show_dbname) {print $0; print “”}}’; for i in $(find -L /hana/log/${DIR_INSTANCE:9:3} -name ‘hdb*’ -type d) ; do echo $i ; hdblogdiag seglist $i | grep -oP ‘Formatting|Preallocated|Writing|bClosedb|Truncated|BackedUp|RetainedFree|bFreeb|Unknown|ClosedIncomplete|FreeIncomplete’ | sort | uniq -c ; done
OUTPUT:
service/hana-11-virtual/37701/service_name=nameserver
service/hana-11-virtual/37701/process_id=7329
service/hana-11-virtual/37701/volume=1
service/hana-11-virtual/37701/database_name=
service/hana-11-virtual/37703/service_name=indexserver
service/hana-11-virtual/37703/process_id=26467
service/hana-11-virtual/37703/volume=3
service/hana-11-virtual/37703/database_name=HSR
service/hana-11-virtual/37707/service_name=xsengine
service/hana-11-virtual/37707/process_id=26470
service/hana-11-virtual/37707/volume=2
service/hana-11-virtual/37707/database_name=HSR
/hana/log/HSR/mnt00001/hdb00001
21 Free
1 Writing
/hana/log/HSR/mnt00001/hdb00003.00003
20 Free
1 Writing
/hana/log/HSR/mnt00001/hdb00002.00003
22 Free
1 RetainedFree
1 Writing
Most of the log segments are in state Free and RetainedFree can removed via below commands:
Free-> This is an indicator for a similar log volume full situation in the past. Although the log volume appears to be full, the free & already allocated (e.g. indexserver) log segments will be re-used.
-> You can release the space occupied by the allocated log-segments in your file system:#1: via SQL command: ALTER SYSTEM RECLAIM LOG;
#2: via OS command: hdbcons –p <PID_of_service> ‘log release’
Via the results of the OS command in the very first step, Option B, you get back information about the PID (Process ID) and the Volume.
The Volume ID hides within the Information of “hdb0000x”, x stands for the Volume ID.
The Volume Path and the PID can be used now to run the specific release command:
SAMPLE:
hsradm@hdb-11-85static:/usr/sap/HSR/HDB77> hdbcons -p 7329 ‘log release’
SAP HANA DB Management Client Console (type ‘?’ to get help for client commands)
Try to open connection to server process with PID 7329
SAP HANA DB Management Server Console (type ‘help’ to get help for server commands)
Executable: hdbnameserver (PID: 7329)
Service: nameserver@hana-11-virtual:37701 (DB#1: SYSTEMDB)
[OK]
—
## Start command at: 2025-05-26 11:02:15.249
Released 20 segments
[OK]
## Finish command at: 2025-05-26 11:02:15.255 command took: 5.589 msec
—
[EXIT]
—
[BYE]
RetainedFree
-> If the connection to the secondary is not available, the primary system will keep writing the redo log segments in the online log area to be prepared for the delta log shipping after the connection is reestablished.
-> These log segments are marked as RetainedFree until the secondary is in sync again. , see KBA 2821539 – Volume /hana/log is full with log segments in status RetainedFree
-> You can release the space occupied by the allocated log-segments in your file system via hdbcons by executing the OS Command as <sid>adm user:
-> hdbcons –p <PID_of_service> ‘log release -r’
-> After deleting Log Segments “RetainedFree” the secondary will perform a full datashipping and log this message “Missing logsegments unavailable on primary – reinitialize”
-> Check if the Parameter global.in/[system_replication]/enable_log_retention is set to force_on_takeover or force.
-> If yes then see 3102080 – RetainedFree status log segment accumulated on replication tier site while replication status is normal
Configuration parameter related:
logshipping_max_retention_size (KBA 2526877 – SAP HANA System Replication Parameter logshipping_max_retention_size)
Log Retention and Multitarget System Replication:
SAP HANA System Replication
SAP HANA Administration Guide for SAP HANA Platform
How to Avoid Log Full Situations
To avoid log volumes running full due to non-working system replication follow the recommendations from SAP Guide:
SAP HANA Troubleshooting and Performance Analysis Guide – LogReplay: Managing the Size of the Log File at: https://help.sap.com/docs/SAP_HANA_PLATFORM/bed8c14f9f024763b0777aa72b5436f6/d04835a010d3485c87992619dce89e2d.html?locale=en-US
1) Unregister an Unused Secondary
If the secondary is disconnected for a prolonged period and if it is not to be used as a replication server anymore, then disable / unregister and decouple the secondary site and disable the primary functionality.
This will stop the RetainedFree log entries from being written.
Unregister the secondary; this is normally done from the secondary site but can be done from the primary if the secondary is not available anymore,
SAP Note 1945676 – Correct usage of hdbnsutil -sr_unregister.
hdbnsutil –sr_unregister
Disable the primary (from the primary site. In a 3 way replication Site B (secondary) is the “primary” for SITE C):
hdbnsutil –sr_disable
Execute reconfigure on the primary site:
hdbnsutil –reconfig
You can use this same procedure for a primary which, after a takeover, will no longer be used for failback.
2) Set a Maximum Retention Size
Another option to manage the log size is to set a value for the logshipping_max_retention_size parameter. If the log size reaches this limit, then RetainedFree log entries will be overwritten. Note the following points in this case:
If any RetainedFree log entries are lost, then synchronization by logreplay will no longer be possible and a full data shipment will be necessary to resynchronize the secondary.It is not possible to switch back to delta mode to resynchronize – only a full data shipping is possible.
Note that the parameter logshipping_max_retention_size is set for a service. See for the details SAP KBA 0002526877 – SAP HANA System Replication Parameter logshipping_max_retention_size.
3) Placeholder File
As a general precaution, to prevent any disk full situation from arising you can reserve a portion of the disk with an emergency placeholder file (containing any dummy values), for example, occupying 5 – 10 % of the file system. This file can then be deleted if ever necessary to quickly solve disk full situations.
4) Using the HANACleaner to automatically remove LogSegments with Status FREE, ##2399996 – How-To: Configuring automatic SAP HANA Cleanup with SAP HANACleaner
Log Segment Knowledge, Background Information & Sources
Data and Log Volumes Documentation: https://help.sap.com/docs/SAP_HANA_PLATFORM/6b94445c94ae495c83a19646e7c3fd56/be3e5310bb571014b3fbd51035bc2383.html?locale=en-US&version=2.0.02
The following log segment states exist:
FormattingThe log segment is being formatted and not yet used.PreallocatedThe log segment has been preallocated, but never used.WritingThe log segment is currently being written. ClosedThe log segment is closed, not backed up and is still required for restart. TruncatedThe log segment is not required for restart, but has not been backed up. BackedUpSegment is already backed up, but a savepoint has not yet been written. Therefore it needs to be kept for instance recovery. RetainedFreeThe log segment has been backed up and is not required for restart, but is required to resync the system replication sites. FreeThe log segment has been backed up, it is not required for restart and can be reused.
Per default each service can store up to 10240 log segments.
If this number is reached, the log reports a log full error even if there is still disk space available.
This limit can be changed: 2072410 – Enlarge limitation of log segment number of LogSegment Directory
The new value of log segments that can be allocated can be checked via SQL (only) by connecting to each individual service: SELECT HOST,PORT,VOLUME_ID,PARTITION_ID,PATH,TOTAL_SEGMENTS FROM M_LOG_PARTITIONS
SAP NOTES / KBAs Log related:
Temporary Workaround: 1679938 – DiskFullEvent on Log Volume
2409471 – SAP HANA Log Volume Full on Secondary or Tertiary Site with System Replication
2821539 – Volume /hana/log is full with log segments in status RetainedFree
2781933 – Nameserver Crash at DataRecovery::LogSegment::freeRetainedSegment When Executing Command hdbcons log release -r
2988616 – /hana/shared volume full or growing
3047379 – Monitoring hana/log volume consumption
3142505 – Limit the Maximum Size Used by HANA Log Segments
SAMPLE: ALTER SYSTEM ALTER CONFIGURATION (‘global.ini’, ‘SYSTEM’ ) SET (‘persistence’, ‘log_disk_usage_limit’) = ’85’ WITH RECONFIGURE COMMENT SAP Note 3142505. Log volume usage size limited to 85%’;
2083715 – Analyzing log volume full situations
1679938 – DiskFullEvent on Log Volume
3537839 – SAP HANA Database Startup Issue Due to Insufficient Disk Space
If HANA is running into a Disk Full Situation, mostly in a replication setup, it is the /hana/log volume filling up with Log Segments.To get the actual Log Segment Situation and partial immediate help, either use:Option A, via SQL statement:select b.host, b.service_name, a.state, count(*) from “PUBLIC”.”M_LOG_SEGMENTS” a join “PUBLIC”.”M_SERVICES” b on (a.host = b.host AND a.port = b.port) group by b.host, b.service_name, a.state;Option B, via OS command as <sid>adm user:hdbnsutil -listServices –allDatabases –sapcontrol=1 | grep -P “service_name|process_id|volume|database_name” | awk ‘/^service.*service_name=/ {s=$0; next} /^service.*process_id=/ {p=$0; next} /^service.*volume=/ {split($0,v,”=”); if (v[2] != 0) {print s; print p; print $0; show_dbname=1} else {show_dbname=0}; next} /^service.*database_name=/ {if (show_dbname) {print $0; print “”}}’; for i in $(find -L /hana/log/${DIR_INSTANCE:9:3} -name ‘hdb*’ -type d) ; do echo $i ; hdblogdiag seglist $i | grep -oP ‘Formatting|Preallocated|Writing|bClosedb|Truncated|BackedUp|RetainedFree|bFreeb|Unknown|ClosedIncomplete|FreeIncomplete’ | sort | uniq -c ; doneA Sample of Option B:Sample Setup:physical Hostname: hdb-11-85staticvirtual Hosntame: hana-11-virtualSID: HSRInstance ID: 77hdb-11-85static:~ # su -l <sid>adm
hsradm@hdb-11-85static:/usr/sap/HSR/HDB77> hdbnsutil -listServices –allDatabases –sapcontrol=1 | grep -P “service_name|process_id|volume|database_name” | awk ‘/^service.*service_name=/ {s=$0; next} /^service.*process_id=/ {p=$0; next} /^service.*volume=/ {split($0,v,”=”); if (v[2] != 0) {print s; print p; print $0; show_dbname=1} else {show_dbname=0}; next} /^service.*database_name=/ {if (show_dbname) {print $0; print “”}}’; for i in $(find -L /hana/log/${DIR_INSTANCE:9:3} -name ‘hdb*’ -type d) ; do echo $i ; hdblogdiag seglist $i | grep -oP ‘Formatting|Preallocated|Writing|bClosedb|Truncated|BackedUp|RetainedFree|bFreeb|Unknown|ClosedIncomplete|FreeIncomplete’ | sort | uniq -c ; doneOUTPUT:service/hana-11-virtual/37701/service_name=nameserverservice/hana-11-virtual/37701/process_id=7329service/hana-11-virtual/37701/volume=1service/hana-11-virtual/37701/database_name=service/hana-11-virtual/37703/service_name=indexserverservice/hana-11-virtual/37703/process_id=26467service/hana-11-virtual/37703/volume=3service/hana-11-virtual/37703/database_name=HSRservice/hana-11-virtual/37707/service_name=xsengineservice/hana-11-virtual/37707/process_id=26470service/hana-11-virtual/37707/volume=2service/hana-11-virtual/37707/database_name=HSR/hana/log/HSR/mnt00001/hdb00001 21 Free 1 Writing/hana/log/HSR/mnt00001/hdb00003.00003 20 Free 1 Writing/hana/log/HSR/mnt00001/hdb00002.00003 22 Free 1 RetainedFree 1 WritingMost of the log segments are in state Free and RetainedFree can removed via below commands:Free-> This is an indicator for a similar log volume full situation in the past. Although the log volume appears to be full, the free & already allocated (e.g. indexserver) log segments will be re-used. -> You can release the space occupied by the allocated log-segments in your file system:#1: via SQL command: ALTER SYSTEM RECLAIM LOG;#2: via OS command: hdbcons –p <PID_of_service> ‘log release’Via the results of the OS command in the very first step, Option B, you get back information about the PID (Process ID) and the Volume.The Volume ID hides within the Information of “hdb0000x”, x stands for the Volume ID.The Volume Path and the PID can be used now to run the specific release command:SAMPLE:hsradm@hdb-11-85static:/usr/sap/HSR/HDB77> hdbcons -p 7329 ‘log release’SAP HANA DB Management Client Console (type ‘?’ to get help for client commands)Try to open connection to server process with PID 7329SAP HANA DB Management Server Console (type ‘help’ to get help for server commands)Executable: hdbnameserver (PID: 7329)Service: nameserver@hana-11-virtual:37701 (DB#1: SYSTEMDB)[OK]–## Start command at: 2025-05-26 11:02:15.249Released 20 segments[OK]## Finish command at: 2025-05-26 11:02:15.255 command took: 5.589 msec–[EXIT]–[BYE] RetainedFree-> If the connection to the secondary is not available, the primary system will keep writing the redo log segments in the online log area to be prepared for the delta log shipping after the connection is reestablished. -> These log segments are marked as RetainedFree until the secondary is in sync again. , see KBA 2821539 – Volume /hana/log is full with log segments in status RetainedFree -> You can release the space occupied by the allocated log-segments in your file system via hdbcons by executing the OS Command as <sid>adm user: -> hdbcons –p <PID_of_service> ‘log release -r’ -> After deleting Log Segments “RetainedFree” the secondary will perform a full datashipping and log this message “Missing logsegments unavailable on primary – reinitialize” -> Check if the Parameter global.in/[system_replication]/enable_log_retention is set to force_on_takeover or force.-> If yes then see 3102080 – RetainedFree status log segment accumulated on replication tier site while replication status is normal Configuration parameter related: logshipping_max_retention_size (KBA 2526877 – SAP HANA System Replication Parameter logshipping_max_retention_size) Log Retention and Multitarget System Replication:SAP HANA System Replication SAP HANA Administration Guide for SAP HANA Platform How to Avoid Log Full SituationsTo avoid log volumes running full due to non-working system replication follow the recommendations from SAP Guide:SAP HANA Troubleshooting and Performance Analysis Guide – LogReplay: Managing the Size of the Log File at: https://help.sap.com/docs/SAP_HANA_PLATFORM/bed8c14f9f024763b0777aa72b5436f6/d04835a010d3485c87992619dce89e2d.html?locale=en-US 1) Unregister an Unused SecondaryIf the secondary is disconnected for a prolonged period and if it is not to be used as a replication server anymore, then disable / unregister and decouple the secondary site and disable the primary functionality.This will stop the RetainedFree log entries from being written.Unregister the secondary; this is normally done from the secondary site but can be done from the primary if the secondary is not available anymore,SAP Note 1945676 – Correct usage of hdbnsutil -sr_unregister.hdbnsutil –sr_unregisterDisable the primary (from the primary site. In a 3 way replication Site B (secondary) is the “primary” for SITE C):hdbnsutil –sr_disableExecute reconfigure on the primary site:hdbnsutil –reconfigYou can use this same procedure for a primary which, after a takeover, will no longer be used for failback.2) Set a Maximum Retention SizeAnother option to manage the log size is to set a value for the logshipping_max_retention_size parameter. If the log size reaches this limit, then RetainedFree log entries will be overwritten. Note the following points in this case: If any RetainedFree log entries are lost, then synchronization by logreplay will no longer be possible and a full data shipment will be necessary to resynchronize the secondary.It is not possible to switch back to delta mode to resynchronize – only a full data shipping is possible.Note that the parameter logshipping_max_retention_size is set for a service. See for the details SAP KBA 0002526877 – SAP HANA System Replication Parameter logshipping_max_retention_size.3) Placeholder FileAs a general precaution, to prevent any disk full situation from arising you can reserve a portion of the disk with an emergency placeholder file (containing any dummy values), for example, occupying 5 – 10 % of the file system. This file can then be deleted if ever necessary to quickly solve disk full situations.4) Using the HANACleaner to automatically remove LogSegments with Status FREE, ##2399996 – How-To: Configuring automatic SAP HANA Cleanup with SAP HANACleanerhttps://community.sap.com/t5/technology-blogs-by-members/hana-housekeeping-using-hanacleaner/ba-p/13351394 Log Segment Knowledge, Background Information & SourcesData and Log Volumes Documentation: https://help.sap.com/docs/SAP_HANA_PLATFORM/6b94445c94ae495c83a19646e7c3fd56/be3e5310bb571014b3fbd51035bc2383.html?locale=en-US&version=2.0.02 The following log segment states exist: FormattingThe log segment is being formatted and not yet used.PreallocatedThe log segment has been preallocated, but never used.WritingThe log segment is currently being written. ClosedThe log segment is closed, not backed up and is still required for restart. TruncatedThe log segment is not required for restart, but has not been backed up. BackedUpSegment is already backed up, but a savepoint has not yet been written. Therefore it needs to be kept for instance recovery. RetainedFreeThe log segment has been backed up and is not required for restart, but is required to resync the system replication sites. FreeThe log segment has been backed up, it is not required for restart and can be reused. Per default each service can store up to 10240 log segments. If this number is reached, the log reports a log full error even if there is still disk space available. This limit can be changed: 2072410 – Enlarge limitation of log segment number of LogSegment Directory The new value of log segments that can be allocated can be checked via SQL (only) by connecting to each individual service: SELECT HOST,PORT,VOLUME_ID,PARTITION_ID,PATH,TOTAL_SEGMENTS FROM M_LOG_PARTITIONS SAP NOTES / KBAs Log related: Temporary Workaround: 1679938 – DiskFullEvent on Log Volume2409471 – SAP HANA Log Volume Full on Secondary or Tertiary Site with System Replication 2821539 – Volume /hana/log is full with log segments in status RetainedFree 2781933 – Nameserver Crash at DataRecovery::LogSegment::freeRetainedSegment When Executing Command hdbcons log release -r2988616 – /hana/shared volume full or growing3047379 – Monitoring hana/log volume consumption3142505 – Limit the Maximum Size Used by HANA Log SegmentsSAMPLE: ALTER SYSTEM ALTER CONFIGURATION (‘global.ini’, ‘SYSTEM’ ) SET (‘persistence’, ‘log_disk_usage_limit’) = ’85’ WITH RECONFIGURE COMMENT SAP Note 3142505. Log volume usage size limited to 85%’; 2083715 – Analyzing log volume full situations1679938 – DiskFullEvent on Log Volume3537839 – SAP HANA Database Startup Issue Due to Insufficient Disk Space Read More Technology Blog Posts by SAP articles
#SAP
#SAPTechnologyblog
