Building a secure development process for a retailer. Part 2: SAP applications

We recently started telling you about our experience building a secure development process for a large retailer. In case you missed it, you can check out the first part about the secure development of web portals and mobile apps here. Today, we’ll give you some details of this project’s implementation in the SAP family of applications

Getting to know SAP

SAP applications were our customer’s most extensive group of systems in terms of code volume. However, we knew very little about SAP at the start of the project.

From the usual viewpoint of developers, SAP is completely disconnected from reality, existing in a world of its own. Usually, in IT, you can choose programming languages and platform architecture for specific tasks, as well as build your own development process, choosing from the tools available at each stage. In SAP, there is no freedom at all. Everything is strictly regulated, as the entire development process is implemented solely with the German vendor’s own tools. All the actions of the developers and engineers are built around the monolithic stack created by SAP.

The main difficulty we faced from the get-go was that we had no clear idea how to dump the code from such an extensive system. Our application code analyzer could check the code in the ABAP language, but at that moment we had no full-fledged integration with this software. However, we managed to solve the task rather quickly by using the Mass Downloader tool.

This required us to understand the structure of the system. As implementation engineers, we only had experience working with SAP Logon (a kind of front end to work with the platform). However, its features were sufficient to solve our problem: we could open the SAP Logon and get to a single portal with almost all necessary information such as the code, current tasks, scheduled tasks, and so on. For the developers, this system is also an integrated development environment.

One of the peculiarities of SAP implementation for the customer was its tight integration with Jira: no changes could be introduced into the application code unless the corresponding task had been created in Jira. At the same time, SAP projects had their own peculiarities of creating tasks in Jira, different from what was seen in the other two groups (checkout software and web portals/mobile apps). For example, tasks for vulnerability patching would not simply be created, but would first be discussed by the development team and planned in advance, so that they could then be officially created in Jira and accepted for patching. The reason was that it was imperative to get a task created in Jira into development within the specified timeframe, without any exceptions or delays.

Selecting a landscape for scanning

The group controlled over 20 SAP systems, each responsible for a different area in retail: logistics routes, warehouse management, accounting, production, and so on. All these systems consisted of several landscapes, from three (dev, test, and prod) to five (the above three + regression and functional testing).

Correspondingly, we had to decide together with the customer in which landscape the source code would be analyzed. On the one hand, we had to verify the code relevant to production, not to development, as it would be changing drastically in the process. On the other hand, it was imperative to find potential vulnerabilities as early as possible. In the end, we decided to scan the code before regression testing and not to analyze the production code, so as to avoid having to dump it and make changes to the production landscapes.

Scanning process

The source code extractor is a program written in ABAP. We had to download and install it in two or sometimes even three landscapes of each of the 12 systems we accepted for analysis. After the extractor had been installed, it was necessary to set up source code dumping on a schedule with the help of special built-in tools. We would download the functional modules, classes, and the source code for each system landscape separately, and only after that, the code would be sent to DerScanner for analysis.

So what was the overall process of sending code for scanning like? The customer had Unix servers (IBM IX systems to run SAP applications), and the source code was dumped there by an extractor. The code was then archived and sent via SSH protocol to the DerScanner server, which ran a scheduled task every 10 minutes to check for newly arrived code. If the code had arrived, it would be checked to see if the analysis was running in the appropriate project for a particular landscape on a particular system. If not, the code would be queued up for analysis and then analyzed. The code security manager would then go to DerScanner, review the results of the analysis for the new section of code in the relevant project, verify the vulnerabilities, and create patching tasks, which would go into Jira.

Complexities of SAP projects

In terms of customer requirements, there were no exotic complexities with SAP systems. All that was needed was to analyze the source code on a schedule once a day. However, this was rather difficult to implement as scanning for vulnerabilities was a completely foreign process for this software as a closed system.

I don’t know if other tools can be easily integrated into the SAP development process, but for us, such integration was a completely new experience; each step had to be researched, invented, and worked through. I have to hand it to the customer; SAP engineers helped us a lot.

For example, we had to modify the source code dumping program for this project, since errors would occur when dumping from SAP, which the customer’s engineers helped us figure out. For all the systems, the SAP team created a user who would send us all the public keys for all the systems via SCP, so that we could enable them as authorized on the server.

Some SAP landscapes did a full reset of all changes and settings every two weeks and rolled back to the version implemented in production so that we could do regression testing. Because of that, all scheduled tasks in DerScanner would be reset and had to be recreated. Here’s how we solved this problem together with SAP engineers. Vulnerability scanning was implemented into development and testing environments, and the tasks and settings required to create a scheduled task (configs) were implemented into the production systems, but without dumping them. Then, in case of a complete reset, the task could be quickly recreated. The configs were preserved, and you only needed to enable the scheduled task.

Summary

With the help of engineers from the customer’s team, we successfully implemented code analysis in the SAP closed system. In the process, we significantly increased our knowledge of the system and were able to create a full-fledged integration successfully, practically from scratch; it went on to be used in 12 independent SAP projects of the customer.

It is really cool when you implement a project and also learn a lot yourself, enriching your tech with new features! This was the case not only for the SAP application group but also for the retailer’s checkout software. We will tell you more about this project in our next post

By Dan Chernov, CTO, DerSecur