Building machine learning or AI solutions is difficult. Besides the uncertainty involved, convincing the business stakeholders to invest in the AI solution can be extremely difficult. On top of this, there’s an added layer of complexity when we are attempting this in a regulated industry such as banking or pharmaceuticals.
Our method of approaching business AI focuses on six core areas: Business Capture, AI Problem Framing, Data Strategy, AI System Design, Feasibility Assessment & Performance Evaluation.
Advantages of using this method are:
- An upfront focus on driving business value
- Ability to identify opportunities for investment in AI/ML solutions with Value vs. Feasibility assessment
- An audit trail for the data sources & solution designs
- A framework for translating business problems into ML/AI solutions
- A reference design document for our engineers that will work on scaling the solution
1) Business Capture
In the business capture phase, we work with business SMEs or analysts to frame your business problems. Here are some guidelines for this:
- Business Area & Owner: Take these down as a point of reference, we’ll want to know who would potentially fund a full-scale deployment if your lab were successful.
- Stakeholders and their roles: we make a note of who the SME contacts are, what their role or title is, who the analysts are, the data scientists, sponsors etc.
- Problem: Write out the business problem that requires solving, we should get an understanding of this by speaking to the business SME. If the problem appears too complicated, we will break it down into simpler sub problems and address them separately. The following questions must be answered: What is driving the change? Who owns the processes? What does a successful solution look like?
- Business benefit: estimate the benefit to the business if the AI solution was to be put in place. Initially this will be a “back of the envelope” calculation based on best possible outcomes. we’ll refine this later once the prototype has been built. It is better to express this business benefit both quantitatively and qualitatively.
2) AI Problem Framing
Once we have captured your business problem, we attempt to frame it in terms of AI & ML at a high level.
- Value Proposition: We will try to understand who the end user is, what their objectives are, and how they would benefit from machine learning or AI.
- Decision Making: Intelligent systems can be effective in making decisions. we must scope out the decisions that the system is making or supplementing, the scale and the frequency. In regulated industries there are often constraints on “black box” models. So, we will need to understand how interpretable the decision-making process needs to be too.
- ML Services: Your intelligent solution might require you to leverage a combination of ML or AI services provided by a cloud platform or ML PaaS. It’s a useful exercise to scope these out at a high level and decide what they could potentially be.
- Machine Learning: Not all solutions will require a complex machine learning model. We will assess whether you will need such a model or not. More advanced models are usually more expensive in the production environment. You should note that having to build your own complex ML solutions is often more complex than leveraging off the shelf ML services.
3) Data Strategy
Data is the petrol to a machine learning engine, so it is important that we capture the data requirements in detail. Initially we consider the requirements for training and inference. If we’re training a machine learning model, what features might be predictive? If we’re just doing inference, what data is required?
Data Management
We should consider the approach to data management and procurement here. Are structured/semi structured data, unstructured data being used or both?
There are different requirements for dealing with structured and semi structured data vs. unstructured data. The way we store your data, the processing costs, volume available are some of the things to consider. In industry there are strict regulations on data and how it is used. You will want to have a strategy for complying with these. Capturing whether personal data is being used, the data source, primary contact for the data and how long you plan to store it will assist us in passing the inevitable data audits.
Data Labelling
In an industry setting we might want to take a supervised machine learning approach, but might not have labelled data. we should specify a data labelling strategy here. Should we use an outsourced service, or some other method to create the labels?
Bias Mitigation
We need to get an understanding of the bias in your data and put in place strategies for managing them. Examples of bias are:
- Outside Bias: Are your data sources from a outside agent? Outside agents tend to suppress data that might be reputationally damaging. For example, data on tobacco’s impact on health gathered by a cigarette company.
- Self-selection Bias: A form of selection bias where your data sources are from those that volunteered to provide it. This is mostly survey data.
4) AI System Design
Scope out the detail of the intelligent solution. We use the Business Capture and AI problem framing exercise to help inform you here. Things to consider are:
- Approach: Documenting the approach intended to take to solve the problem. At this point we will have a good idea of what machine learning services you require and whether or not you need to train a model.
- Performance Requirements: What are the minimum performance standards that are expected? for example, latency requirements and performance of models on test data. The performance targets should be in line with your business requirements.
- Outputs: What will the outputs be? Will your intelligent system serve inferences to another app, or will it be used to generate an insights dashboard? This will have implications for the design patterns we choose later.
- High Level Solution Design: Mapping out the solution design end-to-end. If the solution is cloud native, we should keep this at the services level and not go into detail yet about the ML pipelines (if we’re training a model). These diagrams are useful to scale up the solution for production at some point.
- Machine Learning Strategy: If we’re training a model from scratch, we will list out the models that we will try along with a baseline model. It’s worth mentioning the limitations of each model here too. We will also draw out a more detailed machine learning pipeline here including steps for data segregation, data wrangling, feature engineering, model training & evaluation loops etc. We will probably loosely define our strategy for monitoring any ML models we train once they go live. For monitoring the attributes, as a standard, we must monitor data quality, model quality/performance, model bias drift, and feature attribution.
- Model Training & Evaluation: What are the splits for Training, Validation and Test data sets? What strategies should be used to tune the hyper parameters and test the models?
- Platforms, Tooling, and Infrastructure: Think about the services you need and how much they will cost you. If you’re using a cloud platform, there are a myriad of machine learning services that you can leverage. On-Premise solutions are usually more expensive to build and maintain but they will provide more control over data operations.
5) Performance Evaluation
Whether we’re training our own models or we’re leveraging existing ML services we should be tracking their performance against the requirements you stipulated. For machine learning solutions, a scientifically rigorous strategy for evaluating model performance should be implemented to prevent overfitting and to maximise the chance of the models working properly in production.
6) Feasibility Assessment
Our end goal is to build models that can work in production and add value to your organization. Once we have scoped out all the requirements and built the prototype, we will have a better understanding of how feasible the solution is to scale. Things to consider here are:
- Data Infrastructure: Data Access, volume and quality.
- ML/Solution: Technical resources available, existing solutions, knowledge of solution.
- Processes & Systems: Are there any business process changes, system adjustments, or organisational changes required to implement the solution?
- Know-how: Is the tech and domain knowledge available? And how long will it take to upskill your team to meet requirements?
- Solution to live: How long will it take to take the solution to live?
Value vs. Feasibility
You should be able to assess the value vs. feasibility of the solution. You want to go for solutions that have high value and high feasibility, few solutions that are designed will fit into this category on the outset. If you’re an organisation that is early in the adoption curve for intelligent applications, you can scope out several of these solutions to see where there are opportunities for you to invest.
- Top Left: Aspirational projects hard to do at present but high business value.
- Bottom Left: Trash, relatively low business value and low feasibility.
- Bottom Right: Low hanging fruit, low business value but high feasibility.
- Top Right: Rockets, high business value and high feasibility.
Conclusions
Applying artificial intelligence to solve business problems is difficult. Because of all the moving parts, it is definitely worth standardising your approach as much as possible to make the process more efficient.