Abstract
If you are on BW 7.x, planning to modernize your SAP BW system with SAP Datasphere BW bridge or SAP BW/4HANA (on-prem or PCE), and want to save the investments made along the history, you should consider a path which can help you to migrate your data in an automated way, without the need of reinventing the wheel or having additional manual efforts such as data reloads or reinitializations .
This can be achieved with the Remote Conversion, please see my previous blogs on the topic and the Conversion Guide:
1. SAP Datasphere, SAP BW bridge: Demystifying the Remote Conversion
2. SAP Datasphere, BW bridge: How to transfer data from SAP BW or BW/4HANA to a 2-tier tenant landscape
3. SAP Datasphere, SAP BW bridge – Remote Conversion Guide
This blog intends to provide you additional details on how to optimize the data migration by using the ‘Expert View’ of the Remote Conversion. This capability is available for both SAP Datasphere BW bridge and SAP BW/4HANA and, proven by project experience, it can improve the performance and make your migration runtime up to 20x faster in comparison with the ‘Standard View’.
Figure 1: Data migration options
Motivation
When using the Remote Conversion, one (1) program is generated for every task type of a table in scope. See the Conversion Guide section 3.4 Generation Phase for further details.
Those programs are executed by the tool considering the following sequence:
Read: data selection in the sender system. These programs run in the sender system.Move: data migration from the sender system to the receiver system. These programs run in the sender system and trigger DIA jobs in the receiver system.Conversion: data insertion into the corresponding tables of the receiver system. These programs run in the receiver system.
There are other types of programs such as DELTA_MERGE and MOVE_TO_ADSO which run in the receiver system, but not relevant for the purpose of this explanation.
Figure 2: Data transfer logic and sequence
This is the standard behaviour of the tool. This way, each table will have the corresponding phases processed sequentially.
When you get to the Migration Phase in the Conversion Cockpit, there are basically 2 ways of running the data migration.
1. “Start Migration” option
It uses the “Standard View” and “Extended View”. With this option, the tool automatically controls the sequence/scheduling of the data migration tasks for each table (read –> move –> conversion).
Although it follows a logical sequence, this option may create some idle time in both sender and receiver system, reason why you see the systems not running in full capacity at certain times. It may also create delays for the completeness of the tasks in the sender system – usually the critical path as the sender system is live/operational – making its release to take longer.
Figure 3: Start Migration
2. “Start Transformation of Identified Tables (Optional)” option
It uses the “Expert View”. With this option, it is possible to define/control the sequence and the number of jobs for table types, critical tables, etc. , by for example focusing on the read/move tasks in the sender system. This way, it’s possible to finish the sender system tasks faster, release the system and then focus on the receiver tasks afterwards. As the receiver system is usually not operational yet, this approach is commonly agreed with customers.
This option is not automated like the “Start Migration” one, it requires manual work/intervention and some experience with the data migration logic. This manual intervention is needed to optimize the sequences and keep the systems as busy as possible. By avoiding the idle times, it’s possible to considerably improve the overall migration runtimes.
Figure 4: Start Transformation of Identified Tables (Optional)
The Conversion Guide section 3.6 Migration Phase describes the option 1.
I will show you in this blog how to use the option 2.
Execution
1. Add the parameter ‘PCL_EXPERT’ = “X” to the user (s) who will execute the migration package. This will enable the ‘Expert View” option in Process Monitor of the Conversion Cockpit.
Figure 5: Parameters in SU01
2. Check the table CNVLTRL_TEC in the sender system. The field TABGROUP contains the sequence of the table types to be executed.
Figure 6: Table CNVLTRL_TEC – TABGROUP 09
In general there are 6 TABGROUP values:
TABGROUP 09: T* and some RS* tables TABGROUP 10: some SID tables such as /BI0/SCALWEEK, /BI0/SCALYEAR, /BI0/SCURRENCY, etc.TABGROUP 11: /BI0/SCALQUARTER and /BI0/SFISCYEARTABGROUP 12: /BI0/SCALMONTH and /BI0/SFISCPERTABGROUP 13: /BI0/SDATETABGROUP 30: all others
In the example I will be demonstrating, there are only tables under TABGROUP 09 and TABGROUP 30, and the content of each TABGROUP may slightly differ depending on the scope that was collected.
3. Run “Start Transformation of Identified Tables (Optional)” as indicated in the Figure 4.
4. Select “Start tasks in sender system”, “Read Tasks” and “Move Cluster Tasks”, and start the execution with the tables under TABGROUP 09.
Figure 7: Start Transformation of Identified Tables (Optional) – TABGROUP 09
The flag “Per Table in Sender System” will ensure each selected table will have at least 1 job assigned. The number of jobs to be used depends on how many jobs (BGD work processes) are available in the system. Check the transaction SM50 to verify this information. It’s possible to modify the number or DIA and BDG to optimize the execution (transaction RZ03).
5. Run “Monitor Transformation Status for Active Package (Expert)” to display/monitor the tables. You may consider to open this screen in a new session and keep it open for continuous/faster monitoring.
Figure 8: Conversion Monitor – TABGROUP 09
If you filter the monitor including the “Jobs To Do’, you can also display the tables that did not start yet.
Figure 9: Conversion Monitor for tables with status “To Do”
6. Repeat step (4), now for the tables under TABGROUP 30 (or TABGROUP 10 and every subsequent group in case your scope contains tables under those groups).
You should get the below pop up in case you did close the previous session and need to run it again. Select ‘Yes” and proceed.
Figure 10: Confirm the start of a new execution for tables
Figure 11: Table CNVLTRL_TEC – TABGROUP 30
It’s possible to select the Application Server that will process the set of tables (relevant in case there is more than one Application Server available).
Figure 12: Start Transformation of Identified Tables (Optional) – TABGROUP 30
Figure 13: Conversion Monitor – TABGROUP 30
Let’s have a look on the status of some tables, sorted by Task Type:
Figure 14: Conversion Monitor – status per Task Type
You can also use the “Quick Select” button to select and release tables by type. By having this distribution, you can increase the level of parallelization and ensure as many tables as possible to run. I am now selecting all SID tables, only the ones not yet started will be triggered.
Figure 15: Start Transformation of Identified Tables (Optional) – Quick Select
Here we can see that all SID tables were finished:
Figure 16: Conversion Monitor – SID tables
I will release 8 new jobs, without specifying any table or type. This way, the tool will randomly pick up tables which did not start yet.
Figure 17: Conversion Monitor – new tables being picked
Updating the monitor, all tables have finished the Read and Move tasks:
Figure 18: Conversion Monitor – Read and Move tasks are finished
In “Monitor: Transformation Status (Optional)”, we can confirm the Read and Move tasks were finished for all tables in scope.
Figure 19: Monitor: Transformation Status (Optional)
With the Read and Move phases finished, this means there are no further tasks to run in the sender system. The sender system can be handed over to the customer and the data loads/process chains there can be resumed in the sender system.
Before handing over the sender system, you should:
Make sure the table RSMIGROBJ in the sender system is empty. This is the lock table for the objects in scope.Run “Confirm Data Selection for Participating BW Objects is Finished in Original System” in the Task List and sync with the Conversion Cockpit.
Figure 20: Confirm Data Selection for Participating BW Objects is Finished in Original System
7. Repeat steps (3) to (6), the difference is that now you should select “Start tasks in receiver system”.
Figure 21: Conversion Monitor – insertion for TABGROUP 09
Figure 22: Conversion Monitor – insertion for TABGROUP 30
Figure 23: Overall status per phase and last table in progress
With “Verify Task Completion”, you can check if all tasks were finished, including the conversion ones. In case there is any task missing, this activity will raise an error and indicate which table/tasks still needs to be processed.
Figure 24: Verify Task Completion
In “Monitor: Transformation Status (Optional)”, all tasks for all tables in scope are now finished.
Figure 25: Monitor: Transformation Status (Optional) – all tasks finished
With that, the new data loads/process chains can now be started in the receiver system. Even if the the data loads were earlier resumed in the sender system, the delta queues were cloned and synchronized – ensuring the consistency of the data loads for both sender and receiver systems. This is one of the benefits of the Remote Conversion, offering the possibility of having both systems running in parallel, for example, during the validation/stabilization period.
Conclusion
The scope of this run contained 130 tables (/BI* and RS*) with 100 million records. Up to 8 BTC work processes were used in parallel.
Here a comparison between the runtimes of the ‘Standard View” and ‘Expert View’ for the same scope:
Standard View: 48 minutes
Read + Move tasks: 15 minutesConversion tasks: 33 minutes
Expert View: 16 minutes
Read + Move tasks: 5 minutesConversion tasks: 11 minutes
It was possible to run the data migration 3x faster for this particular scope. The larger the scope, the more this factor can be improved.
For the purpose of this blog, the conversion tasks were triggered only after the read/move ones were finished. In practice, it is also possible to parallelize all task types, allowing even faster runtimes to be achieved.
AbstractIf you are on BW 7.x, planning to modernize your SAP BW system with SAP Datasphere BW bridge or SAP BW/4HANA (on-prem or PCE), and want to save the investments made along the history, you should consider a path which can help you to migrate your data in an automated way, without the need of reinventing the wheel or having additional manual efforts such as data reloads or reinitializations .This can be achieved with the Remote Conversion, please see my previous blogs on the topic and the Conversion Guide:1. SAP Datasphere, SAP BW bridge: Demystifying the Remote Conversion2. SAP Datasphere, BW bridge: How to transfer data from SAP BW or BW/4HANA to a 2-tier tenant landscape3. SAP Datasphere, SAP BW bridge – Remote Conversion GuideThis blog intends to provide you additional details on how to optimize the data migration by using the ‘Expert View’ of the Remote Conversion. This capability is available for both SAP Datasphere BW bridge and SAP BW/4HANA and, proven by project experience, it can improve the performance and make your migration runtime up to 20x faster in comparison with the ‘Standard View’. Figure 1: Data migration optionsMotivationWhen using the Remote Conversion, one (1) program is generated for every task type of a table in scope. See the Conversion Guide section 3.4 Generation Phase for further details.Those programs are executed by the tool considering the following sequence:Read: data selection in the sender system. These programs run in the sender system.Move: data migration from the sender system to the receiver system. These programs run in the sender system and trigger DIA jobs in the receiver system.Conversion: data insertion into the corresponding tables of the receiver system. These programs run in the receiver system.There are other types of programs such as DELTA_MERGE and MOVE_TO_ADSO which run in the receiver system, but not relevant for the purpose of this explanation. Figure 2: Data transfer logic and sequenceThis is the standard behaviour of the tool. This way, each table will have the corresponding phases processed sequentially. When you get to the Migration Phase in the Conversion Cockpit, there are basically 2 ways of running the data migration. 1. “Start Migration” optionIt uses the “Standard View” and “Extended View”. With this option, the tool automatically controls the sequence/scheduling of the data migration tasks for each table (read –> move –> conversion).Although it follows a logical sequence, this option may create some idle time in both sender and receiver system, reason why you see the systems not running in full capacity at certain times. It may also create delays for the completeness of the tasks in the sender system – usually the critical path as the sender system is live/operational – making its release to take longer. Figure 3: Start Migration 2. “Start Transformation of Identified Tables (Optional)” optionIt uses the “Expert View”. With this option, it is possible to define/control the sequence and the number of jobs for table types, critical tables, etc. , by for example focusing on the read/move tasks in the sender system. This way, it’s possible to finish the sender system tasks faster, release the system and then focus on the receiver tasks afterwards. As the receiver system is usually not operational yet, this approach is commonly agreed with customers. This option is not automated like the “Start Migration” one, it requires manual work/intervention and some experience with the data migration logic. This manual intervention is needed to optimize the sequences and keep the systems as busy as possible. By avoiding the idle times, it’s possible to considerably improve the overall migration runtimes. Figure 4: Start Transformation of Identified Tables (Optional)The Conversion Guide section 3.6 Migration Phase describes the option 1.I will show you in this blog how to use the option 2.Execution1. Add the parameter ‘PCL_EXPERT’ = “X” to the user (s) who will execute the migration package. This will enable the ‘Expert View” option in Process Monitor of the Conversion Cockpit. Figure 5: Parameters in SU012. Check the table CNVLTRL_TEC in the sender system. The field TABGROUP contains the sequence of the table types to be executed. Figure 6: Table CNVLTRL_TEC – TABGROUP 09In general there are 6 TABGROUP values:TABGROUP 09: T* and some RS* tables TABGROUP 10: some SID tables such as /BI0/SCALWEEK, /BI0/SCALYEAR, /BI0/SCURRENCY, etc.TABGROUP 11: /BI0/SCALQUARTER and /BI0/SFISCYEARTABGROUP 12: /BI0/SCALMONTH and /BI0/SFISCPERTABGROUP 13: /BI0/SDATETABGROUP 30: all othersIn the example I will be demonstrating, there are only tables under TABGROUP 09 and TABGROUP 30, and the content of each TABGROUP may slightly differ depending on the scope that was collected.3. Run “Start Transformation of Identified Tables (Optional)” as indicated in the Figure 4.4. Select “Start tasks in sender system”, “Read Tasks” and “Move Cluster Tasks”, and start the execution with the tables under TABGROUP 09. Figure 7: Start Transformation of Identified Tables (Optional) – TABGROUP 09The flag “Per Table in Sender System” will ensure each selected table will have at least 1 job assigned. The number of jobs to be used depends on how many jobs (BGD work processes) are available in the system. Check the transaction SM50 to verify this information. It’s possible to modify the number or DIA and BDG to optimize the execution (transaction RZ03).5. Run “Monitor Transformation Status for Active Package (Expert)” to display/monitor the tables. You may consider to open this screen in a new session and keep it open for continuous/faster monitoring. Figure 8: Conversion Monitor – TABGROUP 09If you filter the monitor including the “Jobs To Do’, you can also display the tables that did not start yet. Figure 9: Conversion Monitor for tables with status “To Do”6. Repeat step (4), now for the tables under TABGROUP 30 (or TABGROUP 10 and every subsequent group in case your scope contains tables under those groups).You should get the below pop up in case you did close the previous session and need to run it again. Select ‘Yes” and proceed. Figure 10: Confirm the start of a new execution for tables Figure 11: Table CNVLTRL_TEC – TABGROUP 30It’s possible to select the Application Server that will process the set of tables (relevant in case there is more than one Application Server available). Figure 12: Start Transformation of Identified Tables (Optional) – TABGROUP 30 Figure 13: Conversion Monitor – TABGROUP 30Let’s have a look on the status of some tables, sorted by Task Type: Figure 14: Conversion Monitor – status per Task TypeYou can also use the “Quick Select” button to select and release tables by type. By having this distribution, you can increase the level of parallelization and ensure as many tables as possible to run. I am now selecting all SID tables, only the ones not yet started will be triggered. Figure 15: Start Transformation of Identified Tables (Optional) – Quick SelectHere we can see that all SID tables were finished: Figure 16: Conversion Monitor – SID tablesI will release 8 new jobs, without specifying any table or type. This way, the tool will randomly pick up tables which did not start yet. Figure 17: Conversion Monitor – new tables being pickedUpdating the monitor, all tables have finished the Read and Move tasks: Figure 18: Conversion Monitor – Read and Move tasks are finishedIn “Monitor: Transformation Status (Optional)”, we can confirm the Read and Move tasks were finished for all tables in scope. Figure 19: Monitor: Transformation Status (Optional)With the Read and Move phases finished, this means there are no further tasks to run in the sender system. The sender system can be handed over to the customer and the data loads/process chains there can be resumed in the sender system. Before handing over the sender system, you should:Make sure the table RSMIGROBJ in the sender system is empty. This is the lock table for the objects in scope.Run “Confirm Data Selection for Participating BW Objects is Finished in Original System” in the Task List and sync with the Conversion Cockpit. Figure 20: Confirm Data Selection for Participating BW Objects is Finished in Original System7. Repeat steps (3) to (6), the difference is that now you should select “Start tasks in receiver system”. Figure 21: Conversion Monitor – insertion for TABGROUP 09 Figure 22: Conversion Monitor – insertion for TABGROUP 30 Figure 23: Overall status per phase and last table in progressWith “Verify Task Completion”, you can check if all tasks were finished, including the conversion ones. In case there is any task missing, this activity will raise an error and indicate which table/tasks still needs to be processed. Figure 24: Verify Task CompletionIn “Monitor: Transformation Status (Optional)”, all tasks for all tables in scope are now finished. Figure 25: Monitor: Transformation Status (Optional) – all tasks finishedWith that, the new data loads/process chains can now be started in the receiver system. Even if the the data loads were earlier resumed in the sender system, the delta queues were cloned and synchronized – ensuring the consistency of the data loads for both sender and receiver systems. This is one of the benefits of the Remote Conversion, offering the possibility of having both systems running in parallel, for example, during the validation/stabilization period.ConclusionThe scope of this run contained 130 tables (/BI* and RS*) with 100 million records. Up to 8 BTC work processes were used in parallel.Here a comparison between the runtimes of the ‘Standard View” and ‘Expert View’ for the same scope:Standard View: 48 minutesRead + Move tasks: 15 minutesConversion tasks: 33 minutesExpert View: 16 minutesRead + Move tasks: 5 minutesConversion tasks: 11 minutesIt was possible to run the data migration 3x faster for this particular scope. The larger the scope, the more this factor can be improved.For the purpose of this blog, the conversion tasks were triggered only after the read/move ones were finished. In practice, it is also possible to parallelize all task types, allowing even faster runtimes to be achieved.Felipe Capano Read More Technology Blogs by SAP articles
#SAP
#SAPTechnologyblog
