Building a secure development process for a retailer. Experience of integration with GK checkout software

What’s the hardest part of project work? Probably, bringing customer and contractor expectations of the process and result into accordance. When we started implementing the secure development process in the group for GK applications (checkout software) of a large retailer, we had a whole lot of time and tasks to reduce the vulnerabilities in the code. Here’s a story about what we had to actually solve and how we did it.

By the way, this is our third post on the experience of building a secure development process for a large retailer. If you missed them, you can check out the first two parts: about secure web portal and mobile app development and about secure development in the SAP application group.

About the project

This is the scope of work we were given when the customer first approached us. According to the initial project plan, we had a pretty relaxed timeline for the project. We were very excited, naively thinking that we would be able to manage several projects at a time. However, we ended up spending almost all of our time on the checkout software, as things turned out to be much more complicated than we had thought. This proved to be the most difficult integration area, as it required the most extensive process automation.

Firstly, we ran into a powerful bureaucratic machine. Each action required constant approvals, edits, reports, corrections, queries, waiting for answers and clarifications. All of that caused changes in our approaches and required a lot of work. Secondly, the customer’s representatives had neither the desire nor the time to help us.

GK

GK is essentially a framework upon which you can create your own customized checkout software. Several years prior, the customer had purchased from GK some of the source code for that software to create their own version. This area was being worked on by a very large development team. Most of the software was written in Java (90% of all projects). There were also some projects written in C++ using pretty old libraries, as the hardware was customized for this technology stack. It ran Windows Server 2008 and a set of libraries for Visual Studio, for which this version of the operating system was the last compatible.

The group had its own rigid development processes and release cycles. The branching process was similar to GitFlow, with a main development branch and feature branches for various tasks. However, feature branches could stay in development for months (often ready to merge, but abandoned) and were often merged in just before the release they were supposed to be in.

In addition to CI/CD implementation, we were also brought in Jira, for which we had to create a separate entity to classify vulnerability tasks. This task type allowed us to create vulnerability reports within Jira in accordance with the development process and work with the task tracker.

The GK direction had a long development process where a feature with some functionality would be created, and then that feature branch could go on to exist apart from the main development branch for a very long time. It could be verified, but wouldn’t be merged into master almost until the very release due to technical difficulties in regression testing, etc.

Integration solution

When we first came to the GK group to implement DerScanner, we were surprised by the huge amount of code in the projects in this area and suggested a standard option to scan the main development branch. The developers said that they too wanted to receive information on vulnerabilities and to use the analyzer, but not in the same way as the security officers. They wanted automated vulnerability scanning: a single entry point in the form of GitLab, where they would be able to see all the changes. This way, not only the main development branch but all the side branches could be automatically run for scanning. This also required the option of an automated launch from Jira and, at the request of the customer, or a semi-automated one from Jenkins or GitLab.

In the end, we decided that the easiest way to analyze a branch (especially for long feature branches) would be to run a scan in such a way that the task in Jira, when completely finished (final push), would be assigned the “In Review” status. After that, it would reach a specialist who could accept the changes and give the go-ahead to start scanning the code for vulnerabilities. Otherwise, code analysis for every push in the repository by a developer would create an immense load on both people and hardware. Developers were pushing commits in the repository at least once every five minutes, and the projects sometimes contained some 3.5 million lines of code, which could take 6–8 hours to fully scan. No amount of resources could possibly cover that.

The technology stack

Now let’s talk a little about the set of technologies we used for the project. GitLab worked as the version control system, Jenkins as the CI/CD system, Nexus as the artifact repository, and Jira as the task tracking system. The developers themselves only interacted with Jira and GitLab, while Jenkins was used by engineers and DerScanner by security officers. All information about vulnerabilities in the code was present in GitLab and Jira.

So here’s how we had to organize the scanning process: in Jira, the task was assigned the “In Review” status via Jenkins, as a build was needed. There was a whole stack of Jenkins plugins and scripts that received the webhook from Jira. That is, on top of the vulnerability-type task, we also had to create webhooks in Jira, which, when changing the project in GK, would be sent to Jenkins and then require additional configuration steps and extensive refinement of two-way integration. The process was as follows: when some task was updated in GK, Jira would check whether it matched the ones that should be sent to Jenkins, namely, whether it was a development task. Only then the webhook would proceed to Jenkins, which would parse the webhook and, based on the request body, decided which repository in GitLab and which group of jobs had to run.

GK developers had a number of requirements when creating tasks in Jira, and we tried to use them to the best of our ability to get as much information as possible. This allowed us to correctly match the repositories in GitLab and the branch we had to retrieve from a particular repository to run it for analysis. In the requirements, we made it very clear that the GitLab branch must have a Jira task number in its name. Basically, this is the best practice used everywhere: the task must have labels showing which part of the application/system the problem belongs to.

Based on the webhook which contained the project name, labels, component fields, the key of the task, etc., an automatic determination that all tasks needed to run would take place. Some of the projects would receive a “satisfactory” reply on all values and then be sent for scanning. This means GitLab received a request, checked the availability of the relevant development branch, and, upon confirmation, the system would start scanning a new part of the code in this branch using DerScanner.

Meeting the requirements

Since there were tens of thousands of vulnerabilities in GK projects, the developers did not want to see them all. Meanwhile, the code contained some fairly critical vulnerabilities: passwords inserted into the code, reflected crossite scripting, SQL injection, etc.

The developers were planning on fixing vulnerabilities in the backlog, but wanted to see changes first: that is, the vulnerabilities that had appeared in the new code compared to the previous version.

However, the mechanisms implemented in our product do not imply incremental analysis for Java: the bytecode is what gets analyzed, so it is impossible to check only the changed files, as code cohesion is lost. If a single file is changed that uses several libraries or other packages that belong to the same source code, all information about the vulnerability is lost. That is why we told the customer from the get-go that in this case, it would only be possible to analyze the source code as a whole, and suggested the following scheme.

Just like with the company’s other projects, with GK, we matched the project in GitLab with its counterpart in DerScanner and additionally correlated it with the development branch. To make it easier for developers and IS officers to search it, we created a file containing a registry of the repository group name, the specific project in that group, and the branch in DerScanner.

The developers also wanted to see comments with changes directly in GitLab, so here’s how we did that.

Integrating with GitLab

Firstly, we would scan the main development branch for each repository that the change branch came from. As the developers wanted to see the comments with changes directly in GitLab, we created a rule: should a branch be assigned the “In Review” status, it would mean that we already had a request to merge it in. The system would check that the merge request had been actually created, and if that was confirmed, DerScanner would analyze the main development branch. Then, it would scan the branch with the changes, build a diff, dump it, retrieve only the new vulnerabilities, take the critical ones (our tool allows for adjusting the criticality levels), and convert their description into a text message with a link to this occurrence in a certain project in DerScanner (see the screenshot below). These messages would be sent to GitLab as a discussion on behalf of the technical account. In the merge request, along with the modified code, one could see the trigger string directly. It was accompanied by a comment in GitLab addressed to a specific developer, reading roughly: “there is a vulnerability here — please fix or patch it.”

That way, between the two scans, we could detect changes that the developer had made. If the number of vulnerabilities had increased, that would definitely be reflected in the comments of the report. This scheme was very convenient for everyone involved, as GitLab would immediately display unresolved discussions, which meant that the merge request couldn’t be executed. The person responsible for approving the merge request would see that the developer had created a vulnerability that needed patching.

Integration with Active Directory

The developers needed rights in order to view the vulnerabilities through DerScanner. So, on top of the integration with development systems and all three scopes of work, we also integrated with Active Directory services. The customer had a huge AD with a million nested groups, which required a lot of optimization. This means that in the process of CI/CD deployment, we had to deal with a lot of related customization, adaptation and optimization tasks (see screenshot below).

Regarding the integration with Active Directory, GK developers did not want to configure access settings for a number of DerScanner projects directly in AD, as they did not own the resource. At the company, the process of granting access rights was as follows: if the developers needed to add a new person to a group, they had to write a request to tech support. Tech support would then determine who to send this request to, who was responsible for issuing such rights, whether this person could actually be granted rights, whether they were authorized or not, etc.

This was a very time-consuming process that the developers did not manage, unlike GitLab. Then, they asked us to set up the integration with AD as follows. They would not only allow system tech owners and security officers to see vulnerabilities, as has been done for other areas, but also execute rights mapping in GitLab. That is, if a person had developer rights or higher (maintainer, repository owner), they would be able to view the relevant projects in DerScanner that belonged to that repository. When a new project was created in DerScanner, the system would query the names of anyone who was a GitLab member in that repository with an access level higher than developer. The list would be dumped, the code analyzer contacted, and the parser would send a request to AD to add the list of people selected. Everyone who was a member of that repository in GitLab would be granted automatic permissions through their AD accounts to access the project in DerScanner. At the end of the chain, a PDF file would be generated with a vulnerability report from the code analyzer. The report contained a diff emailed to the GK project user list.

Summary

The integration with GK checkout software is an example of a large-scale project of implementation and automation of a secure development process in the company, built not only around the information security department but around developers as well. In this group of projects, just like in the others, parallel scanning of the main development branch took place, as security officers also created tasks in Jira and verified them. The vulnerabilities would be entered into the backlog and patched later.

The developers were given a handy tool, embedded into a familiar environment and workflow. Since they had set up a threshold of the top critical vulnerabilities which arise quite rarely, their workload did not increase too much after the implementation. In the end, everyone involved in the project was happy: both those who sought to improve code security and those who had to fix vulnerabilities.

If you have any experience of your own with secure development practices, please share it in the comments!

Reminder: this is part three of a series about building a secure development process for a large retailer. In the next post, we’ll summarize the results of this large-scale project. We’ll talk about what we had learned from it, and how our tool had evolved thanks to this experience.

By Dan Chernov, CTO, DerSecur