HyperBDR is a cloud-native migration and disaster recovery product that utilizes the concept of cloud-native storage. Its core business scenario involves synchronizing the source data to cloud-native storage in a block-level differential manner. Currently, it supports both block storage and object storage. It also leverages Boot-in-Cloud patented technology to restore business systems to a usable state with just one click. This fully utilizes the orchestration capabilities of cloud-native, meeting the different needs of migration and disaster recovery scenarios.
HyperBDR currently supports nearly 10 major versions of source operating systems (Windows/CentOS/Redhat/Ubuntu/SUSE/domestic operating systems), and hundreds of minor versions. It also supports nearly 40 target cloud platforms (public cloud, private cloud, hybrid cloud, hyper-converged, virtualization, etc.), with the number still increasing. If we were to perform comprehensive testing of the source operating systems on all cloud platforms, the combination of test cases could exceed 10,000.
In such a large-scale scenario, achieving test coverage purely relying on manpower is impractical. Therefore, it is necessary to introduce automated testing methods to test the core business scenarios. This not only meets the needs of automated testing but also allows developers to assess the impact on the core processes in newly developed features during the development process, further improving product stability and reliability.
Pain Points Analysis Let’s first look at several pain points in manual testing of the HyperBDR product:
Pain Point 1: Multiple test cases but insufficient human resources Given the scale of the source and target platforms mentioned above, even for basic smoke testing, there are over a hundred test cases for a complete test scenario. For example:
Source (19 types): CentOS (6/7/8), Redhat (6/7/8), SUSE (11/12), Ubuntu (14.04/16.04/18.04/20.04), Windows (2003/2008/2012/2016/2019), Oracle Linux, domestic operating systems Target (9 types): OpenStack, AWS, Alibaba Cloud, Tencent Cloud, Huawei Cloud, Mobile Cloud, ZStack, hyper-converged products Thus, a single test scenario would involve 171 types. Some may argue that these test cases are not that many and can be completed quickly. Therefore, let’s move on to the second pain point of HyperBDR in the testing process: the issue of testing cycle.
Pain Point 2: Long testing cycles Unlike business testing, testing a single HyperBDR scenario is time-consuming. Let’s analyze the data synchronization and start-up processes without considering the time spent on resource preparation and various configurations:
Data synchronization: In simple terms, data synchronization involves reading the effective data (not allocated capacity) from the source operating system and writing it to the target cloud-native storage at the block level. The first synchronization is a full sync, and subsequent syncs are permanent incremental syncs. Taking Windows as an example, assuming the effective data is 500GB and calculating with an 80% utilization rate of the gigabit LAN bandwidth, the transfer speed would be around 800 Mbps, approximately 80 MB/s. This process would take about 1 hour and 8 minutes. Host startup: The startup time varies greatly depending on the type of cloud-native storage. For example, with Huawei Cloud’s block storage, due to snapshot mechanisms and the ability to support swapping system disks, the startup time has little relation to capacity and can be kept within 5 minutes. However, most domestic cloud platforms lack such capabilities. For instance, when Alibaba Cloud has snapshot-generated volumes, the underlying speed limit is 40 MB/s, which prolongs the recovery process. Taking object storage as an example, if we restore object storage data to block storage within the internal network, the recovery time for a 500GB effective data disk would be approximately 40 minutes. Therefore, dealing with a single host’s test takes at least 2 hours. Based on the assumed scenarios mentioned above, a day’s worth of testing would not even complete a complete test scenario. One might suggest running tests concurrently, which brings us to the third pain point: cost.
Pain Point 3: Testing cost The reason why full concurrency cannot be achieved is due to network bandwidth limitations. In our internal development environment, we only have 40Mbps of external network bandwidth. In the aforementioned test scenario, with full bandwidth utilization, it would take about 35 hours for the full data transfer of 500GB. This undoubtedly further extends the testing cycle.
Additionally, with so many source environments and different scenarios, it requires a massive amount of source computing and storage resources. As the product continues to iterate, resource consumption will increase. To solve this problem, we decided to use some public cloud environments to address the shortage of local resources. Based on the characteristics of resource usage, we mainly adopt a pay-as-you