Cybercrime is one of the biggest threats to every company, and it’s becoming more and more sophisticated. It is predicted to cost the world $6 trillion in 2021. If it was measured as a country, cybercrime would be the world’s third largest economy after U.S. and China.
VMware has formed a security business unit – VMware Carbon Black – with the vision to create a world safe from cyberattacks. Its mission is to transform security through big data and behavioral analytics in the cloud. Our team’s biggest assets are the diverse profiles of experts in software engineering, testing automation, UX design, DevOps engineering, infrastructure automation engineering, site reliability engineering, data science engineering, and data analysis.
The product serves more than 25K business clients, tens of millions of devices and analyzes more than a billion events per minute. To be able to support and develop the product, the team is expanding and looking for engineers with different backgrounds.
Fifty people already work in the R&D team in Bulgaria and the number is expected to double by the end of 2021, offering various opportunities across different skillsets for senior and junior positions.
Check out the open positions here.
Strong protection with intrinsic security
VMware Carbon Black is an integrated endpoint and workload protection platform that analyzes attackers’ behavior patterns to detect and stop never-seen-before attacks.
The two consecutive VMware acquisitions of Carbon Black – a leading next-generation security cloud provider in 2019 and Octarine – a Kubernetes security startup in 2020 represent the evolution of VMware’s “intrinsic security strategy” – protecting content and applications wherever they live. Security features are built into the infrastructure and across workloads, clients, and applications.
The challenges
The team is tackling two significant technology challenges.
The huge data volume is growing exponentially – more than 20 petabytes of data in the cloud and more than a billion events per minute coming from the customers’ endpoints. The Compute cluster consists of approximately 2,000 EC2 instances in Amazon. And to make the task even more challenging –the data from hundreds of thousands of endpoints has to be collected in their data lake and processed close to real-time – minutes or hours between the detection of a suspicious event on an endpoint (a computer or a cluster) and sending an alert. A large portion of the data in the Cloud is indexed in a Solr-based search index, which allows the customers to perform low-latency near real-time search queries against the data collected from the endpoints. Simultaneously a traditional analytical data lake is also used, allowing the continuous improvement of the algorithms and the application of sophisticated data analysis. All events happening on the endpoints are streamed to Carbon Black systems, making it possible to respond quickly to new types of attacks and develop a new algorithm in a few hours.
The other complexity that the team has to address is the
growing sophistication of cyberattacks.
“If ten years ago cybersecurity software meant detecting malware by
mapping it to a huge database of classified malware, today this is not enough. The
major part of cyberattacks today does not use malware. The trend is analyzing
the behavior on the endpoints. This is an analytical problem, which requires
using data-driven workflows, AI and complex systems, which detect anomaly in
behavior in almost real-time,” says Ivan Markov, Senior Staff Engineer at
Carbon Black.
The technologies
To be able to protect our customers’ systems and develop the technology simultaneously, the Carbon Black team is using a complex tech stack that includes:
- Full text indexing cluster: Apache Solr based cluster
- Data lake: streaming ingestion based on Apache Flink/Kubernetes, Spark + AWS Glue-based batch workloads
- Athena for BI requests and SQL access
- BI/Reporting: Redash, Tableau, Mode Analytics
- SaaS: technologies from the AWS portfolio (Kinesis, ECS, EC22, Lambda)
“The list of technologies that we use goes on and on. They are cutting-edge and interesting to work with. However, what is more important is not so much the tech stack, but the problems we solve and our approach,” says Ivan. “The expertise of our team in Bulgaria is impressive.”
“Even though our organization is distributed, we are aiming to work as one team. To achieve this, we create teams based on the sistership model – usually, the teams in two geographies have a similar structure and have a common roadmap (a product or a part of the product). That allows for cooperation between the teams, but also certain independence. We call this loose coupling between the teams in strong cohesion,” explains Yavor Boychev, Senior Engineering Manager at Carbon Black. The teams have 7 to 10 members with a clearly defined ownership. “This gives us control over what we do, and we know that if something breaks, it is our responsibility to fix it,” adds Yavor.
New opportunities
The Bulgarian team works on Carbon Black Cloud, focusing on threat hunting capabilities, data analytics, platform services, and the protection of cloud native apps. “We’ll continue to further invest in these areas, as well as start new ones, including new core platform services, protection of containerized workloads, UI platform,” says Ventsislav Potchekanski, who manages VMware Carbon Black in Bulgaria.
According to Yavor Boychev, with the acquisition of Octarine, VMware enters a new and exciting stage. “Octarine is the intersection of two strategic investments for the company – the modernization of the infrastructure for the company apps (cloud-native apps & products) and cybersecurity. This is an exciting direction for our business,” adds Yavor.
VMware Carbon Black team has ambitious goals for growth – they are planning to hire 50 to 70 new people by the end of 2021 for various senior and junior engineering roles across all aspects of the tech stack used by Carbon Black – data processing and analytics with Java, Python, Big Data frameworks and SQL; microservices and compute with Kubernetes, AWS ECS, Java, Go; User Interface based on React & JavaScript”.
“We are looking for engineers with a mindset for growth and development and fitting into the distributed teams’ philosophy,” says Yavor.
