Deploying Intel FlexRan on the SUSE Adaptive Telco Infrastructure Platform

4. Verify the Rancher installation

You should wait a few minutes for Rancher to be rolled out:

kubectl -n cattle-system rollout status deploy/rancher

BIOS Configuration for Dell PowerEdge XR11

In order to import the configuration file, you need to do it using the idrac web interface:

Note: This configuration depends on the hardware vendor, so please check with your hardware vendor for the best configuration to use.

Pay special attention to the parameters suggested here, as some of them could dramatically increase or decrease the performance of the tests being executed.

The next table shows the recommended configuration as a reference for the most common hardware vendors:

As specified in the output above, we have 2 types of resources available for the FlexRan workload:

• The FEC will be `intel.com/intel_fec_5g` and the value will be 1 because we bind only 1 of 2.
• The VFs will be `intel.com/intel_sriov_odu` or `intel.com/intel_sriov_oru` and the value will be 4 because we bind 4 VFs for each PF.

Basically, FlexRan will request some resources available in the host to be used as a VF for the tests we’re going to run.

Important Note: If you don’t get the resources available here, does not make sense continue with the flexran demo tests. Please, review the previous steps to ensure you have the VFs created and the SRIOV CNI plugin working properly.

FlexRan tests

References

For this article, we will use the following references to deploy the Intel FlexRan reference implementation on top of the ATIP edge cluster:

– You will also need to download the FlexRan-22.07 tarball from Intel to run the tests and mount the downloaded `tests` folder into the pre-defined containers because it’s not included into the pre-defined containers.

Note: In order to download those files, it is required to have access to the Intel website (you can ask your Intel representative).

• FlexRAN-22.07-L1.tar.gz_part00
• FlexRAN-22.07-L1.tar.gz_part01

– Container Entrypoint script (Just in case you need to change anything else like the MAC addresses for the VFs)

./extract.sh

mkdir /home/tmp_flexran

cp -R tests/ /home/tmp_flexran/

cp build/docker/docker_entry.sh /home/tmp_flexran/

The /home/tmp_flexranwill be the folder to be mounted into the containers on the next section.

Before deploying the FlexRan containers, let’s review the next steps:

– Interfaces to be used in the flexran pods yaml files, should be referenced by their resource names. You could get those as: kubectl get nodes -o json | jq '.items[].status.allocatable' as explained in the previous section (#sriov-configuration)

– The container entrypoint script contains the tests files customization for this specific environment. It will modify some parameters such as MAC addresses, `VF` information and the `dpdk` info into the `tests`. It will be explained in the next section.

– Tests should be mounted in /home/tmp_flexran/tests and exposed in /home/flexran/tests

Container entrypoint

The FlexRan containers run a script when the container is started.

The script will modify the configuration files of the FlexRan tests applications to adapt the tests to our environment.

The modifications done in this file will be:

– PCIDEVICE_INTEL_COM_INTEL_FEC_5G=$(env|grep PCIDEVICE_INTEL_COM_INTEL_FEC_5G= |awk -F '=' '{print $2}') in order to select the right PCI device for the FEC ACC100 card used for the 5G acceleration.

– export INTEL_COM_INTEL_CPULIST=$(cat /sys/fs/cgroup/cpuset/cpuset.cpus) to get the CPU list of the host machine.

Also, we need to change the CPU cores as well as the MAC addresses into the RU section because there isn’t any substitution in the entrypoint script. We need to change the following lines:

sed-i"s/ioCore=2/ioCore=62/g"config_file_o_ru.dat

sed-i"s/duMac0=[0-9,a-z][0-9,a-z]:[0-9,a-z][0-9,a-z]:[0-9,a-z][0-9,a-z]:[0-9,a-z][0-9,a-z]:[0-9,a-z][0-9,a-z]:[0-9,a-z][0-9,a-z]/duMac0=00:11:22:33:00:00/g"config_file_o_ru.dat

sed-i"s/duMac1=[0-9,a-z][0-9,a-z]:[0-9,a-z][0-9,a-z]:[0-9,a-z][0-9,a-z]:[0-9,a-z][0-9,a-z]:[0-9,a-z][0-9,a-z]:[0-9,a-z][0-9,a-z]/duMac1=00:11:22:33:00:10/g"config_file_o_ru.dat

sed-i"s/ruMac0=[0-9,a-z][0-9,a-z]:[0-9,a-z][0-9,a-z]:[0-9,a-z][0-9,a-z]:[0-9,a-z][0-9,a-z]:[0-9,a-z][0-9,a-z]:[0-9,a-z][0-9,a-z]/ruMac0=00:11:22:33:00:01/g"config_file_o_ru.dat

sed-i"s/ruMac1=[0-9,a-z][0-9,a-z]:[0-9,a-z][0-9,a-z]:[0-9,a-z][0-9,a-z]:[0-9,a-z][0-9,a-z]:[0-9,a-z][0-9,a-z]:[0-9,a-z][0-9,a-z]/ruMac1=00:11:22:33:00:11/g"config_file_o_ru.dat


sed-i"s/duMac2=[0-9,a-z][0-9,a-z]:[0-9,a-z][0-9,a-z]:[0-9,a-z][0-9,a-z]:[0-9,a-z][0-9,a-z]:[0-9,a-z][0-9,a-z]:[0-9,a-z][0-9,a-z]/duMac2=00:11:22:33:00:20/g"config_file_o_ru.dat

sed-i"s/duMac3=[0-9,a-z][0-9,a-z]:[0-9,a-z][0-9,a-z]:[0-9,a-z][0-9,a-z]:[0-9,a-z][0-9,a-z]:[0-9,a-z][0-9,a-z]:[0-9,a-z][0-9,a-z]/duMac3=00:11:22:33:00:30/g"config_file_o_ru.dat

sed-i"s/ruMac2=[0-9,a-z][0-9,a-z]:[0-9,a-z][0-9,a-z]:[0-9,a-z][0-9,a-z]:[0-9,a-z][0-9,a-z]:[0-9,a-z][0-9,a-z]:[0-9,a-z][0-9,a-z]/ruMac2=00:11:22:33:00:21/g"config_file_o_ru.dat

sed-i"s/ruMac3=[0-9,a-z][0-9,a-z]:[0-9,a-z][0-9,a-z]:[0-9,a-z][0-9,a-z]:[0-9,a-z][0-9,a-z]:[0-9,a-z][0-9,a-z]:[0-9,a-z][0-9,a-z]/ruMac3=00:11:22:33:00:31/g"config_file_o_ru.dat

Then, we need to remove (or comment with the hashtag `#` sign) the tests not covered by the flexran container version we are using (the test files are not present in the FlexRan tests suite used so the script will fail):

• flexran/bin/nr5g/gnb/l1/orancfg/sub3_mu0_20mhz_sub3_mu1_20mhz_4x4/gnb/
• flexran/bin/nr5g/gnb/l1/orancfg/sub3_mu0_20mhz_sub6_mu1_100mhz_4x4/gnb/
• flexran/bin/nr5g/gnb/l1/orancfg/mmwave_mu3_100mhz_2x2/gnb/
• flexran/bin/nr5g/gnb/l1/orancfg/mmwave_mu3_100mhz_2x2/gnb/
• flexran/bin/nr5g/gnb/l1/orancfg/sub3_mu0_20mhz_sub3_mu1_20mhz_4x4/oru/
• flexran/bin/nr5g/gnb/l1/orancfg/sub3_mu0_20mhz_sub6_mu1_100mhz_4x4/oru/
• flexran/bin/nr5g/gnb/l1/orancfg/mmwave_mu3_100mhz_2x2/oru/

For instance, you will have the next block commented (or removed):

#cd /home/flexran/bin/nr5g/gnb/l1/orancfg/sub3_mu0_20mhz_sub3_mu1_20mhz_4x4/gnb/

#sed -i "s/<systemThread>2, 0, 0</systemThread>/<systemThread>$systemthread, 0, 0</systemThread>/g" phycfg_xran.xml

#sed -i "s/<timerThread>0, 96, 0</timerThread>/<timerThread>$timerThread, 96, 0</timerThread>/g" phycfg_xran.xml

#sed -i "s/<FpgaDriverCpuInfo>3, 96, 0</FpgaDriverCpuInfo>/<FpgaDriverCpuInfo>$FpgaDriverCpuInfo, 96, 0</FpgaDriverCpuInfo>/g" phycfg_xran.xml

#sed -i "s/<FrontHaulCpuInfo>3, 96, 0</FrontHaulCpuInfo>/<FrontHaulCpuInfo>$FrontHaulCpuInfo, 96, 0</FrontHaulCpuInfo>/g" phycfg_xran.xml

#sed -i "s/<radioDpdkMaster>2, 99, 0</radioDpdkMaster>/<radioDpdkMaster>$radioDpdkMaster, 99, 0</radioDpdkMaster>/g" phycfg_xran.xml

An example of docker entrypoint script is available [here](./2023-07-29-flexran-images/flexran-dockerentrypoint.sh)

FlexRan Timer Mode

FlexRAN Timer Mode use case does not use fronthaul. RF IQ samples are read from files and write to files. The test pod is configured with two containers. One container is for `L1APP`, and another one is for `TestMAC`.

Smaller storage is built in pod as Kubernetes resources, but the storage is not enough for the test case. The test case requires 20GB storage for tests files, so it will be mounted from the host using the `tests` volume mount path.

The test case is a setup for one peak cell and two average cells. 8 cores with HT enabled are pinned for `L1APP`, the average core utilization is between 45-65%. The below figure shows the threading model for this test case.

To deploy the Test Timer Mode, we will use the following yaml file:

The following resources are required:

– intel.com/intel_fec_5g: '1' to request the FEC ACC100 resource.

– hugepages-1Gi: 16Gi to request the hugepages resource for the first container.

– hugepages-1Gi: 8Gi to request the hugepages resource for the second container.

– /home/tmp_flexran/tests folder mounted to /home/flexran/tests in order to have the tests available for the containers.

– image: intel/flexran_vdu:v22.07 to use the pre-defined container from Intel pre-defined containers.

Once the pod is deployed, you can check the status of the pod:

kubectl get pods

NAME READY STATUS RESTARTS AGE

flexran-dockerimage-release 2/2 Running 0 2m

This will launch automatically the Timer Mode Tests which contains 83 tests for the FlexRan library. It could take up to 3 hours to complete the tests. The status of the tests can be found in the container logs:

kubectl logs flexran-dockerimage-release flexran-l1app

After a successful tests execution the following output should be shown:

All Tests Completed, Total run 83 Tests, PASS 83 Tests, and FAIL 0 Tests
----------------------------------------------------------------------------
mem_mgr_display_size:
Num Memory Alloc: 11
Total Memory Size: 264,018
----------------------------------------------------------------------------

Using htop we could see the CPU usage of the containers as well as the pinned cores for this workload:

Using the Rancher UI, you can check the status of the pods and the logs of the containers.
Also, you could get the following metrics using Prometheus and grafana to check the CPU usage, CPU cores isolated used, and the Memory usage of the containers:

The following resources are required:

– intel.com/intel_fec_5g: '1' to request the FEC ACC100 resource.

– intel.com/intel_sriov_odu: '4' to request the 4 VFs for the ODU interface.

– intel.com/intel_sriov_oru: '4'to request the 4 VFs for the ORU interface.

– hugepages-1Gi: 20Gi to request the hugepages resource for the first container.

– hugepages-1Gi: 16Gi to request the hugepages resource for the second container.

– /home/tmp_flexran/tests folder mounted to /home/flexran/tests in order to have the tests available for the containers.

Once the pods are deployed, you need to launch the tests manually. You can do that using the following commands:

1. Open a terminal in the `flexran-vdu` pod and run the following commands:

kubectl exec -ti flexran-vdu-66c9bf5765-ftptg /bin/bash

cd flexran/bin/nr5g/gnb/l1/orancfg/sub6_mu1_100mhz_4x4/gnb/

./l1.sh -oru

2. Open a new terminal in the `flexran-vdu` pod and run the following commands:

kubectl exec -ti flexran-vdu-66c9bf5765-ftptg /bin/bash

cd flexran/bin/nr5g/gnb/testmac

./l2.sh --testfile=../l1/orancfg/sub6_mu1_100mhz_4x4/gnb/testmac_clxsp_mu1_100mhz_hton_oru.cfg

3. Open a third terminal to run the `flexran-vru` pod and run the following commands:

kubectl exec -ti flexran-vru-66casd2e5765-23resd /bin/bash

cd flexran/bin/nr5g/gnb/l1/orancfg/sub6_mu1_100mhz_4x4/oru/

chmod +x run_o_ru.sh; taskset -c 20 ./run_o_ru.sh

Using htop we could see the CPU usage of the containers as well as the pinned cores for this workload:

Using the Rancher UI, you can check the status of the pods and the logs of the containers.

Also, you could get the following metrics using prometheus and grafana to check the CPU usage, CPU cores isolated used, and the Memory usage of the containers: