The goal of digital transformation is outcomes, not engineering

If there’s a common theme that runs throughout the content on Intersect, it’s that organizations should avoid taking on unnecessary engineering projects, especially when that work is focused on infrastructural plumbing rather than on elements that add meaningful business value. Here are a few of the places we’ve already discussed it:

It’s a theme worth bringing up again, in light of Facebook recently discussing its Tupperware container-orchestration system and Amazon Web Services announcing general availability of its recommendation-engine service, Personalize. The Tupperware blog and some subsequent interviews with Facebook engineers received a lot of attention among engineering types because it’s a cool system that runs at a massive scale. But it’s also a bespoke system that’s fine-tuned for Facebook’s business and nobody else’s. Even among enterprises with capable engineering teams, the question when they see something like Tupperware should probably be something more like, “How can we bring some of these key features to a community-driven project like Kubernetes?” rather than “How can we build something like that ourselves?”

A simple reality check would probably uncover one fundamental truth very quickly: Trying to replicate a system developed over 8 years by one of the world’s biggest and most technically savvy companies is probably not a project worth taking on. Attacking it alone is fraught with peril and might be near impossible. But by leveraging a huge and growing community, it’s not only possible, but in some cases probable that the features you want will be integrated (like when two dozen other large users also want them).

The major problem with going too far down the DIY path in the name of digital transformation—whether that’s attempting to build something complex from scratch, or even getting your hands dirtier than necessary with an existing technology like Kubernetes—is that it forces you to spend far too much energy on engineering, rather than focusing on delivering outcomes. If the goal of digital transformation is ultimately about improving the business, you want to be flexible enough to do what you need to do without feeling tied to a project that’s already cost millions of dollars and untold hours. And you certainly don’t want to incur additional technical debt of the type that makes transitioning a colossal undertaking.

Which brings us to AWS Personalize. Building good recommendation engines is still a challenging task, even in the era of deep learning and data science. Merits of any particular service aside—including Personalize—an API-based cloud service is probably the best way for most organizations to go about doing recommendations. If it works, fantastic! You just saved yourself a lot of time and money. If it doesn’t work, you probably didn’t have to invest too much time or money to find that out. You’re onto the next option that much faster and with that much more budget still in your pocket.

For more on this topic, you can also check out this fantastic blog post from Dropbox titled “Your system is not a sports team.” It’s a look into what happens when teams or individuals become too ardent for the thing they’re working on, at the cost of doing what’s right for the company. One way this rears its head is when folks fight against killing a project or moving away from a technology that isn’t cutting it anymore or isn’t worth the effort. Another more latent way is when folks get captured by “system bias” and spend their time striving for unnecessary performance gains or working on unnecessary subcomponents in lieu of work that would actually make a meaningful difference.

This anecdote from the Dropbox post—discussing the rationale behind, and reaction to, a simple thing like renaming its product-specific Magic Pocket Team as the Storage Infrastructure Team—nicely captures the spirit of engineering in a fast-moving digital world:

The method here is that this is the team of block storage domain experts at the company. If the system ever ceases to meet our needs and we need to pursue an alternative, this is the team that needs to advocate for that strategy. Magic Pocket is not a sport team and our people not a bunch of zealous fans. They’re highly skilled engineers with a responsibility to design, build, and operate the best storage system in the world. If we can’t trust that team to drive this strategy who can we trust?

Orienting a team around a mission and not a specific system is critical to ensure that their priorities are aligned with what’s best for the company.

What you need to know this week

Amazon enters, and leaves new businesses

Amazon Store Credit Builder (Amazon): Another example of a tech company leveraging its ubiquity and ability to execute to get into a new, lucrative market. And to serve a demographic that’s currently underserved by existing banks. Apple and T-Mobile also have both entered financial services, recently.

Amazon to shut down its Amazon Restaurants business in the U.S. (GeekWire): But sometimes, tech companies decide their experiments aren’t worth it and close them. That’s good news for smaller companies that can execute and are willing to stay in it for the long haul.

Digital transformation done right and not so right

Salesforce is buying data visualization company Tableau for $15.7B in all-stock deal (TechCrunch): This is a case of Salesforce recognizing that its users want better data analysis, which many are currently getting via Tableau. So rather than build something, it bought the company and its $1 billion-plus in annual revenue.

Life on the road gets a little easier as truckers adopt digital technology (Wall Street Journal): A great set of applications here, from digitizing paperwork that’s typically handles in the cab of a truck, to making repairs faster and easier. Everyone wins.

CVS Health’s grand consumer-driven healthcare plan depends on data infrastructure (ZDNet): CVS is completing its acquisition of Aetna, but it needs to get a handle on both companies' data to really maximize the opportunity. That often proves easier said than done.

JPMorgan scraps mobile banking app Finn, its attempt to lure emoji-loving millennials (CNBC): From a digital transformation perspective, a more accurate headline might be that Chase is being adaptive in the face of business realities. If Finn wasn’t working as planned, you keep the good parts and move onto the next thing.

When employees are using software that IT hasn’t approved (Harvard Business Review): Yeah, this still happens, and in addition to giving employees wiggle room within some defined parameters, a big solution appears to be tackling individual needs with specialized vendors instead of doing massive, years-to-completion omnibus deals with large vendors.

Michaels’s offers fresh lessons in the perils of being a tech laggard (Fortune): This is not a good look for any retailer in 2019. The playbook for success in retail is pretty clear, and it starts with digital.

Kubernetes thrives, Hadoop struggles

Apple joins the open source Cloud Native Computing Foundation (TechCrunch): This is good news for the CNCF projects—including Kubernetes—because Apple brings unique perspectives around scale and privacy. It’s good news for Apple because working with these technologies will make hiring easier.

The optimal Kubernetes cluster size: Let’s look at the data (The New Stack): Small, application-specific clusters is the conventional wisdom, which makes ease of provisioning them all the more important.

Hadoop runs out of gas (InfoWorld): This is a good recap of a tumultuous week or so in the big data world, and the space’s greater issues. Data is more important than ever, but it’s being used in ways that don’t always require a massive and complex system.

Security, or Congress?

You. Quest and LabCorp. Explain these medical database super-hacks, say US senators as 425,000 more people hit (The Register): At some point, Congress will grow tired of hearings and start imposing tough regulations on data breaches. The time to get to get ahead of the curve is now.

Who left a database of emails, credit cards, plain-text passwords, and more open to the web this week? Tech Data, come on down! (The Register): This happens far too frequently. See previous link about where this is headed.

Microsoft wants more security researchers to hack into its cloud (Bloomberg): Microsoft, like other cloud providers, gets it. Vulnerabilities aren’t a sign of weakness; they’re inevitable and fixing them makes you stronger.

AI: Boring, but good for business

Amazon shares how it leverages AI throughout the business (ZDNet): There’s a lot to learn from Amazon’s experiences getting started with artificial intelligence and machine learning, even if you’re not at its scale.

When AI becomes an everyday technology (Harvard Business Review): There some good advice here from one of Google’s AI business leaders, including a focus on solving problems rather than on searching for places to do AI.

How to make data and AI add up (Financial Times): Some more good advice, although it’s unclear that anyone knows for certain the right skill set for any sort of chief data officer or related position. Data governance and infrastructure, for example, are different but very critical considerations.

Small businesses aren’t rushing into AI (Wall Street Journal): It probably depends on how one defines “small.” But if the data science movement taught us anything, it’s that most truly small and even mid-sized businesses won't be hiring specialists or using API services. They'll consume AI as features of the applications they use.

The rise of industrial AI (CIO): AI is definitely beneficial in industrial settings where there’s room to optimize everything from manufacturing processes to predictive maintenance. Makes you wonder if there’s also room for better internal software development, too.

Pharma groups combine to promote drug discovery with AI (Financial Times): This harkens back to a post here a few weeks ago about how zero-sum industries are vanishing. Even drug companies are realizing the benefits of working together in areas where they probably have little collective expertise.