Building a secure development process for a retailer. Part 4 Summary of a major project

Building a secure development process for a retailer. Part 4 Summary of a major project

This is the final part of a series about building a secure development process for a large retailer. If you missed them, you can check out the previous parts: about secure web portal and mobile app development, about secure development in the SAP Application Group, and about embedding our tools in the checkout software development process. It's time to review the lumps we’d taken and summarize our experience.

Everything was pretty great. We could have ended the story right there if it weren't for one “but.” In our planning, we miscalculated our integration time. Spoiler alert: a lot of things went wrong, and as a result, the implementation became more complicated and the timeline shifted. Spoiler alert 2: it was impossible to account for all the risks in advance. And here's where I want to go into more detail.

Lessons to Be Learned

1. Considering the regulations and bureaucracy

The bigger your customer, the more regulations and bureaucracy are there. They are supposed to make things easier, but in fact, only further complicate them. We stumbled over the first stone at the tender stage. We had been asked to prepare a master plan before we had time to get acquainted with all the internal requirements and details, as well as the internal structure of the company, which would end up affecting a lot. It was a requirement of the board, without which the implementation would not have happened. We agreed and, after assessing the risks, we signed up for a fairly loose deadline, as it seemed to us at the time.

At the start, the customer warned us that they had a regulation in effect, according to which the implementation should go through three standard steps: development, testing and production. Let’s give an example of why these rules do not actually make things easier.

We needed to deploy to a large number of mobile app repositories, but not all of them had CI set up, and therefore there were many situations where we couldn’t see how the code was built. We started by getting access to the source code (unfortunately, read-only). We didn't have access to the build and testing infrastructure, so we tried to build it locally, writing internal instructions for reproducibility and insurance against the bus factor, tried to automate building and running the analysis, and constantly consulted with the retailer's developers. Finally, when we ran all the source (or built code) analyses locally in DerScanner, finalized everything, and came to the customer with a finished plan, certain unforeseen things came to light. Some repositories had changed, rendering our approaches outdated, and some team leaders simply refused to give us access to their code, demanding integration instructions to implement everything themselves.

In the end, some of the work we had done would be wasted, and some of it would be handed over to the customer to proceed on their own. If we had avoided the bureaucratic process, we would have saved a lot of time. As we saw with the in-house developers later, it could have been avoided. Toward the end of the project, we became convinced that we could just as easily have ignored these requirements without any damage to the project or our reputation. But by then, the integration had already progressed into its final debugging phase. We concluded that in the future, we would negotiate to be exempted from strict regulation compliance requirements if non-compliance would have no impact on the project whatsoever. We also realized that we should have had insisted on greater involvement by the retailer. That way, we could have learned about significant changes earlier and reacted to them more flexibly. And now, the most important conclusion: when planning timelines, it's imperative to evaluate the abundance of regulations at large organizations as a considerable risk.

2. Staff turnover is a significant hindrance to the process

In a large company, high staff turnover is another reason why we would periodically be pushed back in the time loop. Several times, we had reached agreements with decision makers only to see them leave the company later. We had to either renegotiate with a new employee to comply with previously approved agreements or talk to a higher-level manager in cases where a replacement had not yet been found.

We were once caught off guard by the categorical response, “I think this decision would be wrong. We will not do it.” This happened when we had already negotiated a full transfer of MariaDB database support with the head of database administration. Just before we handed over the project, when we had coordinated everything, it turned out that the head of the department had changed. In spite of the fact that all the agreements with the previous employee had been logged, in accordance with the regulations, we had to make a compromise and take the administration upon ourselves under the scope of product maintenance. It turned out that the customer would not take on the MariaDB support, although this was not mentioned in the project documentation.

Such cases are not always preventable, so to get out of such a situation, on the one hand, you need to be able to insist on solutions and follow them, and on the other hand, you’ve got to seek common ground and to concede.

3. Getting into the specifics will save you a lot of pain

During the implementation phase, we would often encounter planning flaws. Since the management of the retailer had asked us to prepare a master plan before the approval of the project, and after the approval we had to follow an established course, there were difficulties. At the start, we expected that everything inside the company would be arranged in a similar way (even SAP), but it turned out to be not the case. The infrastructure and how it was deployed, development regulations, charters, restrictions on purchasing products, etc. were all different. We fully understood that we would encounter different technologies during development, but we hoped for a unified approach to processes and similar concepts — and this is where we were wrong.

Having completed the project, we decided that from then on, during the exploration and analysis phase, we would be looking at each enterprise division as a separate company and delving deeply into its specifics. We would have to determine which traits were common and which were not. After this implementation, we formed a unified questionnaire that allows us to understand how the development process happens, how developers write their code, how they apply and roll out changes, what kind of infrastructure they have for the production environment, etc.

Integration of this scale is a partnership wherein all parties must agree and be able to respond flexibly to change. Nevertheless, to end up with a somewhat clear plan, it is better to go for an iterative approach in putting it together.

Product changes as a result of the project

1. Integration with AD

Prior to this project, DerScanner integration with Active Directory had not been optimized for high loads. It would slow down when connecting to directories with thousands of users, and was not very convenient to work with. In the course of the project's development, it had been improved so that it was no longer a problem. Now we can integrate with any similar system, and we won't have any performance problems.

2. Jira

We've also improved a lot on all the nuances related to Jira. Before the project, we had relied on a default implementation of Jira, but this retailer used a version of Jira that had been rewritten, adding a number of fields, changing the default task creation mechanics, and altering all the original templates. Therefore, we implemented a tool in DerScanner that allows adjusting for any kind of integration, with the ability to edit the body of a created task sent to Jira via API from DerScanner. It's a very flexible tool.

For example, when we were setting up integration with SAP, we found that our old integration with Jira was dumping some default task fields, so you tried creating a task, and it wouldn't work. You were like, “What's going on?” You opened up Jira, and it was all wrong. Jira's API would be sending wrong default fields with the wrong data type. But since the customer had processes tied to this implementation, it was necessary to solve the problems on the DerScanner side.

As a result, in SAP, we needed to define the “estimated time to do this task” fields in advance. Time estimate arrived as a string. You would go to it and see two fields. If you opened XML, you would see that in those fields, the data was supposed to be described in a certain way. In fact, it was a dictionary with many values where you had to specify the time. For example, one day should be in the “1d” format. You would fix it, try running the integration, and something else would fail. You’d then see that some field dumped as a mandatory one was indeed listed as mandatory. Despite that, in the interface, it was not necessary to fill it out. After messing around like that a bit, we finally implemented the integration — however, if we hadn't had the flexibility to customize any kind of fields, we wouldn't have been able to configure integration with such a heavily customized Jira.

3. Changing the architecture of the application

In conjunction with this project, we did considerable upgrades to the architecture of our application. This was rather labor-intensive, since we had to switch from a monolithic paradigm where the whole load was placed on the server, to a traditional backend-frontend model. Because of that, we had to take different approaches in the entire development, designing, creating our own API, and changing the technology stack. As a result, this solution significantly simplified the automation, as our new API became its main tool and entry point.

We found that there were a lot of requests to switch from MariaDB/MySQL, which had enough functionality to run the application, to PostreSQL. Especially from big businesses for which this database is very common thanks to its extensive functionality and good support. Thus, the customer at some point gave up support for MariaDB, and their database division switched to work exclusively with PostgreSQL. Therefore, we had to implement support for this database.

Summary

Overall, this customer, being a large company, helped a lot with the development of our product. This is a serious battle case that yielded us a lot of insight into how our solution should evolve. As a result of the implementation, we formulated approaches to risk assessment and project planning in the enterprise segment and created a questionnaire to collect information on the customer's infrastructure.

The project pointed out our bottlenecks, and we realized how to deal with them in the future. Our team realized what the architecture of DerScanner should look like in a year, so the product could be improved and its delivery options expanded. We learned a lot in terms of improving the user experience with the product. In some places, the user search was lacking. We should have been able to add users to the local group from LDAP and other protocols in several places related to the administration panel (in the project settings, user settings, etc.).

Everyone involved in this project on our end is now working on developing the product, and they have a common understanding of how it should be scaled for large implementations of the secure development process. We are now working on another, even more global upgrade to the architecture of the entire application, which will allow us to scale and change it more flexibly, making it more versatile and easier for users to interact with.

This is the final part of a series of articles about building a secure development process for a large retailer. We’ll be happy to answer any questions and hear about your own experiences with secure development practices in the comments!

By Dan Chernov, CTO, DerSecur