And Now, MLOps

We do love our acronyms. Add MLOps--also known as ML CI/CD, ModelOps, and ML DevOps--to the lexicon. It’s perhaps like AIOps in the sense of applying artificial intelligence or machine learning to the information technology operations process. 

“MLOps is an approach that marries and automates ML model development and operations, aiming to accelerate the entire model life cycle process,” says Deloitte. “MLOps features automated pipelines, processes, and tools that streamline all steps of model construction.”

As tends to happen when we develop new acronyms and concepts, we often see new firms emerging. Some might argue MLOps is more an engineering culture using DevOps principles than anything else. 

“MLOps (a compound of “machine learning” and “operations”) is a practice for collaboration and communication between data scientists and operations professionals to help manage production ML (or deep learning) lifecycle,” Wikipedia says. 

Similar to the DevOps or DataOps approaches, MLOps looks to increase automation and improve the quality of production ML while also focusing on business and regulatory requirements, Wikipedia adds.

Virtually all Surveyed European and North American Firms Buy SaaS

Nearly half of North American and European firms surveyed spend between one percent and 25 percent of their software budgets on software as a service subscriptions, a survey of 143 North American and European organization information technology spending finds. 

source: Computer Economics 

All of the U.S. firms are buying some SaaS services, as well as 98 percent of the European respondents, the report by Computer Economics reports.

Some 11 percent of North American respondents say they buy nearly all their software in SaaS mode.

Monitor Every Network "Like You Own it"

Visibility and performance management becomes more important in a multi-cloud and hybird cloud applications environment. 

FinOps Shows Cloud Maturity

It arguably shows some new level of maturity for cloud computing when execs start looking at managing cloud costs as an integral part of the larger information technology budget. That is a more-refined level of analysis than the earlier question of whether enterprises should move compute functions away from owned infrastructure and “to the cloud.”

As a practical matter, execs now confront the possibility that dispersed ordering of cloud capabilities is inefficient, as when multiple contracts for cloud resources are let by different organizations within a single enterprise, for example. In other cases the issue might revolve around which cloud features need to be purchased. 

“FinOps is essentially an operating model for the cloud world,” says J.R. Storment, executive director of the FinOps Foundation. “It brings together a prescriptive set of actions, best practices and culture that enable disparate teams like engineering, finance, IT and business to come together to get the most value out of every dollar that they spend in cloud.”

Hybrid computing approaches add more weight to the value of FinOps, as computing operations become more decentralized.  

Modern Computing Alliance Aims for "Silicon to Cloud" Seamless Enterprise Computing

What Precisely C Suites Need to Know about AI is a Difficult Matter

One reason it often is hard to get telecom executive attention about artificial intelligence is that it is a feature of other products that are purchased to support networks, or features of products sold to customers, but not a discrete product in and of itself. 

Qualcomm, for example, notes how AI is used by its chipsets to improve the accuracy of signal propagation, and therefore the positioning of antennas to support indoor networks. Useful, of course, but telecom executives do not buy chipsets or even network elements but “networks and platforms.” 

source: Qualcomm 

While AI is useful for operations as well, such as call centers and customer service; or marketing analytics, it is not the sort of “thing” the C suite typically has to know much about. 

All of that makes AI content a tricky topic for organizers of telecom events. It is quite easy to err on the side of “AI hype,” which arguably is not useful, or in the direction of “how it works,” which also is generally unhelpful. 

The only thing that really matters to C suite executives is how AI materially affects all the other parts of the business model the C suite is responsible for, from product development to marketing, sales and support; network management to capital investment and overall operating costs. 

None of that is easy to do, right now. Edge computing and internet of things are easier to program, since they immediately raise the issue of incremental new revenue, which does get attention.

Rise of FinOps

As cloud workloads continue to climb as a percentage of total workloads, public and private cloud costs are going to climb as well. At the same time, more organizations are shifting to use of multiple cloud vendors, plus on-premises, plus private cloud, all of which increases complexity and therefore cost, while increasing the risk of inefficient purchasing. 

Since different cloud and hybrid services have different pricing and billing models, and costs can change from month to month, there now is a growing emphasis on management of cloud costs. The notion that FinOps solutions are required is therefore not too surprising, from the standpoint of vendors that hope to capitalize on the “new trend” by selling FinOps solutions. 

source: Cloudera, Harvard Business Review 

source: Cloudera, 451 Research

ServiceNow, IBM Team for AIOps

ServiceNow said it will integrate its service management and operational visibility tools with IBM's Watson software that automates information technology operations using artificial intelligence, or AIOps. Available later in 2020, the solution is intended to allow enterprise IT staff to identify, fix and prevent IT issues. 

AIOps value lies in its ability to improve pattern recognition, anomaly detection and determination of causation, Gartner says. AIOps uses a big data platform to aggregate siloed IT operations data in one place, IBM says. 

IBM notes that data can include:

• Historical performance and event data

• Streaming real-time operations events

• System logs and metrics

• Network data, including packet data

• Incident-related data and ticketing

• Related document-based data

Using Watson AIOps, the average time to resolve incidents was reduced by 65 percent, according to one recent initial proof of concept project with a client, ServiceNow says. “AIOps will detect patterns a human would be unlikely to uncover, including those that reveal cause and effect,” Gartner analysts have said. 

Enterprises might use AIOps to support a pattern detection algorithm supporting customer relationship operations. In such cases, software can map the metrics from IT and business data. 

User navigation might be correlated with digital experience data, order data, sentiment data and account activity.

That would enable the building of a composite model of a customer, across all applications they use and different behaviors across multiple modes of a single application such as when they use a web browser versus a mobile device, Gartner says.

Dreamworks uses AIOps to Smooth Out Big Rendering Jobs

“Any sufficiently advanced technology is indistinguishable from magic,” novelist Arthur C. Clarke once quipped. 

But successful advanced technology use cases always are concrete, and add value because real business problems are solved. Consider the way Dreamworks uses AIOps. As a digital content business, Dreamworks arguably has an easier time than most switching to fully-remote work. 

But even so, there are real problems AIOps helps Dreamworks solve. The way the studio schedules workloads if a case in point. One specific problem is ensuring that the internal computing network does not crash when huge rendering jobs are executed, such creating 150,000 animated people in a crowd scene.

Doing all the rendering at once would affect computing performance. 

"We don't want the artists noticing that something's performance has changed," says Skottie Miller, technology fellow and vice president of platform and services architecture at DreamWorks. "We want our synthetic transaction and monitoring framework to tell us before the artists notice that something is trending in a bad direction.”

"It used to be there would be an issue and maybe an engineer noticed because they were looking for it, or maybe the system sent an alert and an engineer would go investigate it," Miller notes. "Now an issue surfaces almost always with a recommendation and, in many cases, a solution before the engineer is in the loop.”

“It lets us run with 24x7 support with fewer sets of eyeballs staring at the systems,” he says. And that is precisely the sort of use case AIOps was envisioned to support: automating the alerting process and preparing a solution without information technology staffs having to do so manually. 

Telecom AIOPs

In the connectivity business, potential applications include network operations monitoring and management, predictive maintenance, fraud mitigation, cybersecurity, customer service, marketing virtual digital assistants, customer relationship management preventive maintenance and battery optimization, for example. 

In the network operations monitoring area, that might include anomaly detection for operations, administration, maintenance and provisioning (OAM&P), performance watching and optimization, alert or alarm suppression, bother price ticket action recommendations, automated resolution of bother tickets, prediction of network faults or congestion prediction, for example. 

Artificial intelligence is a capability, not a product. 

“You don’t focus on ‘I’m going to go do AI,’ says Peter Guerra, North America chief data scientist at Accenture. “You focus on ‘I’m going to do supply chain better, and I’m going to leverage AI to do that.’” Peter Guerra, North America chief data scientist at Accenture.

Hughes Network Systems Adds AIOps to its Managed Services

We are, by most estimates, relatively early in the artificial intelligence life cycle. Most AI-based capabilities tracked by Gartner are five to 10 years away from widespread commercial use. Some might say the same is true of connectivity service provider use of AI to support their own network operations.

But it is coming. Hughes Network Systems, for example, has commercial availability of its artificial intelligence capability for supporting information technology operations (AIOps).

Integrated into the company’s HughesON Managed Network Services, the Hughes AIOps feature is already in use across more than 32,000 managed sites. The technology automatically predicts and preempts—or “self-heals”—undesirable network behavior, preventing service-disrupting symptoms in 70 percent  of cases, HNS says.

Hughes says it s the first managed services provider to deliver a self-healing WAN edge capability to enterprise customers.

source: Gartner 

Of course, AI will have impact in many other ways. At two recent sessions of the PTC Academy, a training course for mid-career telecom professionals, the point was made that artificial intelligence, more automated business processes and competitive pressures on profit margins all would combine to reduce headcount in the industry. That is not a judgment about the morality of the trend, just a prediction of what will happen. 

Nor are such observations in any way denying that new jobs that will almost inevitably be created as the automation, artificial intelligence and “substitute machines for humans” trends unfold. 

Big technology changes have happened before. Much-higher mechanization of agriculture drove most U.S. residents off farms and into urban centers, where those people and their descendants worked in new roles. A shift of value from goods to services likewise has shifted people out of factories and created new jobs elsewhere in the economy, particularly in a wide range of services roles. 

While the shift within the connectivity industry might not be that pronounced, industry headcount has been dipping for some decades, though offset by growth in the mobility segment. In the U.S. market, you can see the slow attrition of fixed network employment since the internet bubble peak and crash. 

The emergence of the mobility business as the industry growth driver was accompanied by job expansion in that segment of the business, stabilizing around 2002 and then falling after 2009. 

source: Bureau of Labor Statistics

To the extent that profit margins continue to be under pressure, and industry revenue growth anemic (less than one percent per year, globally), we should expect more substitution of machine operations for humans. 

AIOps for Dummies

AIOps is among the esoteric disciplines information technology managers encounter these days. Many would argue that is the case because information technology operations these days are complex and decentralized. 

To a large degree, that is because modern IT operations no longer are conducted “in house,” but also integrate cloud computing operations, virtual servers spinning up and down, containers sharing operating systems and providing microservices, mobile device and app support. New Internet of Things apps and devices, cloud storage, software as a service and platform as a service all contribute to complexity. 

Among the issues is that AIOps--like artificial computing generally--is not a product. It is instead a capability that enhances other products and services. “AIOps isn’t a specific off-the-shelf product,” say Tony Branton and Ted Coombs, authors of a ServiceNow edition of “AIOps for Dummies.”

Specifically, AIOps is about applying machine learning to supervision processes. Right now, that mostly means AIOps “sits on top” of existing management tools. “Machine learning is a branch of AI that uses the ability of computers to learn by analyzing data and improving answers to questions posed to it autonomously,” the authors say. 

Typically, that means applying supervised or unsupervised learning routines to security or asset management; root-cause analysis; change management; impact analysis, capacity management; performance and availability management. 

Is AIOps a Capability or a Product?

Some markets are hard to estimate because the definitions matter so much. Consider AIOps. A new research report predicts the global AIOps platform market will reach US $11.1 billion in sales by 2025, growing at about a 34 percent compound annual growth rate. 

Aside from the typical issues--how do we define platform; what is AIOps--we face some additional questions. Is artificial intelligence a capability of existing products or a new product category? If the former, then AIOps is a capability, not a product category. If the latter, is it only new products which count? 

source: Techtarget

“Major growth factors” include the growing demand for AI-based value in  IT operations, Cole Reports says. But how do we separate AI features that are embedded in existing platforms and software? Is it the full value of the retail product, or only the incremental value added by AI? 

To use one example, AIOps is not about automation in a direct sense, some would argue, but is a tool to achieve automation. AIOps essentially requires analytics conducted on big data sets, but is not directly big data. AIOps is normally part of some analytics solution for some IT operations function. 

source: dzone

The Cole Reports study suggests properly that AI will be used in information technology operations, which illustrates the definitional issue. If AI improves operations processes, can we cleanly separate AI from the value AI providers for existing operations management products, both embedded into other software and provided as stand-alone solutions?

It is not an unusual task. When estimating 5G revenue, a proper understanding of incremental revenue growth would also subtract the value of replaced 4G connections. Growing streaming revenue also has to be balanced against lower linear subscription revenue to evaluate actual net market changes. 

AIOps presents some similar issues. Definitions matter. If every analytics platform or operations system includes AIOps features, then AIOps becomes almost a meaningless term.

Netreo Unveils AIOps: Autopilot

Netreo, a supplier of  IT management announced the release of AIOps: Autopilot, said to be the first product featuring data models that combine artificial intelligence (AI) and machine learning (ML) technology with 20 years of network management system (NMS) configuration and monitoring data. 

This allows threshold baselines, event correlation rules, dependency mapping, and many other configurations to evolve and improve the longer Netreo is deployed. 

AIOps: Autopilot works with both the on-premises and native-cloud versions of Netreo’s solution, including Netreo Cloud. It runs in the background of a Netreo deployment and constantly scans the configuration to make sure it is always tuned properly, the company says. 

When issues or potential improvements are found, AIOps: Autopilot will either automatically fix the problem or provide engineers suggested remediations using AI and ML algorithms applied against previously gathered historical data. 

AIOps: Autopilot automatically learns from every successive execution and gets smarter, so that it can both reduce unnecessary alerts and preemptively correct more issues over time.

AIOps: Autopilot also comes with an array of extensions out-of-the-box to give operations teams a head start. These extensions provide the ability to:

• Automatically baseline thresholds against historical values and exceptions to minimize false-positives and alert noise.
• Model all metrics against best-practice key performance indicators (KPIs) to ensure there are no blind spots. 
• Learn system and environment changes, and change the monitoring infrastructure to automatically adapt.

Pricing for the AIOps: Autopilot add-on is $12,000 for customers that purchase a Professional Level license. Autopilot is included at the Ultimate License level.

OpsRamp Winter 2020 Release Adds More Analytics

OpsRamp announced its new OpsQ Recommend Mode for first-response and incident remediation. OpsQ Recommend Mode uses predictive analytics to reduce mean-time-to-resolution (MTTR). 

Other artificial intelligence for IT operations (AIOps) innovations in the release include visualization of alert similarity patterns and new alert stats widgets to provide transparency into machine learning-driven decisions, OpsRamp says. 

The OpsRamp Winter 2020 Release also introduces 19 new cloud monitoring integrations for Amazon Web Services (AWS), Microsoft Azure, and Google Cloud Platform (GCP), along with dynamic topology maps for Azure and GCP. 

The Winter 2020 Release introduces the OpQ Bot and a new Recommend Mode for alert escalation policies. 

Visualization of Alert Seasonality Patterns allows IT teams to  visualize seasonality patterns that OpsQ has learned. 

The Alert Stats Widget shows the total number of raw events, correlated alerts, inference alerts, auto-ticketed alerts, and auto-suppressed alerts handled by the OpsQ event management engine. 

OpsRamp also added monitoring support for AWS, Azure, and Google Cloud services, including:

• AWS – Transit Gateway, AppSync, CloudSearch, and DocumentDB
• Azure – Application Insights, Traffic Manager, Virtual Network, Route Table, Virtual Machine Scale Sets, SQL Elastic Pool, and Service Bus
• GCP – Cloud BigTable, Cloud Composer, Cloud Filestore, Firebase, Cloud Memorystore for Redis, Cloud Run, Cloud TPU, and Cloud Tasks

Cloud Topology Maps support automated topology discovery and mapping for Azure and GCP. 

Agentless Discovery and Monitoring for Windows Servers, as its name suggests, allows automated monitoring. Synthetic Monitoring provides deeper insights and analysis for troubleshooting multi-step transactions.

AIOps for Telcos: Is it Automation or Insight? Ericsson Chooses

Whether AIOps is about insight or automation often is unclear, especially because the stated goal of AIOps to gain insights that lead to automated action, supervised or unsupervised by humans. Ericsson, for example, talking about applied artificial intelligence in communications networks, prefers the “automation” label. 

In large part, that might reflect the particular challenges of the connectivity service provider industry, where many revenue and business problems grow from unhappy customers and poor customer experience (dropped calls, no coverage, slow internet speeds, poor connection quality). 

“Previous Ericsson research shows that almost 50 percent of consumer network perception is based on personal experiences of the network, indicating the huge importance of network quality as the key to customer satisfaction,” Ericsson says. 

source: Ericsson

Customer satisfaction, in turn, is believed to be causally related to word of mouth referrals, prospect perceptions of quality and value, customer churn, customer acquisition and therefore revenue, profits and market share. 

“Whether customers give you US$2 or $20 ARPU, if you do not provide the quality they expect, they will churn,” says Vicente Cotino Director of Network Operation Maintenance, Orange Spain. 

To be sure, network performance is not the only driver of satisfaction. “At least one-third of the total NPS (net promoter score, a measure of customer willingness to recommend a product or service to others) score is derived from network performance.” 

Of the service providers surveyed by Ericsson, 80 percent use NPS as a key metric in operations. Also, several service providers indicate that 40 percent to 60 percent of operations key performance indicators are business-related.

Service providers, in other words, must do many things right, beyond ensuring that the network works as it is supposed to work. What is hard to untangle is the percentage of benefit that AIOps might provide, and whether it is automating changes or detecting issues that is “more important.” 

Clearly the two are related. Action is not possible without insight; nor is insight useful unless changes can be made fast. 

In surveys undertaken by Ericsson, 80 percent of respondents say automation is key for the cost and customer experience. About 90 percent of operations personnel say AI is important in protecting customer experience. 

As always, methodology matters. I do not know how the questions were framed. It is not clear how the responses might have been different if poll takers were asked about whether better network and consumer behavior insight and prediction would protect customer experience. 

But Ericsson frames the issue as one of “automation.” That might simply be a reflection of our general confusion about effectiveness and efficiency. People often speak about the importance of efficiency, which might be expressed as getting work done faster, with fewer people, at less cost or with fewer mistakes and rework. 

Less common are evaluations of effectiveness, which might be expressed as “doing the right things” to produce value or desired outcomes. A traditional way of illustrating the difference is to note that an organization gains little to nothing by automating things that should not be done. 

In principle, AIOps as applied to a communications network requires automated insight to produce proactive network responses. So far, we seem to have gotten better at insight, while IT managers still debate the value of unsupervised action by AI-driven systems to correct and modify network operations. 

Put simply, “nobody trusts the system to behave autonomously” at the moment.

Is AIOps About Automation or Insight?

For more than a year, it seems, there has been some debate about whether AIOPs fundamentally is about automation or insight. It can be hard to discern, as a fully-developed AIOps solution, capable of providing insights, also often also is said to include the ability to proactively diagnose and resolve issues. 

Part of the problem is that the approach is so new. Right now, developers are creating new capabilities around the ingestion of data, correlation of incidents and resolution of issues. AIOps is “about making sense of large volumes of information,” says Ciaran Byrne, OpsRamp VP. 

There is an element of remediation and automation, though, he notes. 

At the current stage of development, nobody is willing to entrust AI-enabled systems to reconfigure networks, servers and software autonomously. Beyond that, though, the fundamental value of AIOps arguably is not the automated response but the move “from hindsight to insight to foresight,” says Joris DeWinne, StackState solutions architect. 

Today, enterprises monitor the health of components and systems,  but the problem has been too many notifications, he says. At the next level, monitoring adds logs and metrics. Next-generation monitoring has been added by firms such as ServiceNow, AWS and Azure, using Kubernetes, for example.

AIOps will represent the next level after that, where prediction starts to be feasible, he argues. 

AI Helps, Says Deepsense.AI

Artificial intelligence can solve a number of IT management challenges, argues Andy Thurai, Deepsense.AI U.S operations head. 

Any anomaly can signal the existence of a problem, as the default state of the infrastructure ecosystem is stable, he says. So early detection of an anomaly is usually a sign of a problem that has yet to be fully understood, he says. 

Root cause identification another advantage, when most enterprises have multiple to scores of separate monitoring systems, Thurai says. 

Service tickets analytics are another advantage of applying AI to operations. When fed data on tickets submitted to a service desk, an ML-based model can predict seasonal spikes and requests. This can help the service desk owner deploy help desk personnel as needed. So we predict the number of call center agents you need to have on site, says Thurai. 

AI also can be applied to detect seasonality trends and adjust capacity accordingly. 

Machine-powered analysis delivers insights that are beyond the reach of humans. Frequent pattern mining is possible because AI canfind correlations that are impossible for humans to detect. 

AIOps solutions also eliminate noise and allow staff to concentrate on the real underlying problems.

Incident Management on a Single Pane of Glass?

Sameer Hutheesing, AVP,  Business Development, GAVS Technologies, believes it now is possible to correlate most incident reports into a single system. That might not yet be a full AIOPs capability, allowing enterprises to close their network operations centers, but it is hard to argue this is not useful. 

video