back ground

This article is part of a larger independent study (see below) on how product managers integrate machine learning into their products. It consists ofRyan DinglerMyselfMBA at the University of California at Berkeley, atVince LawWith the help of usinstructor.

The study aims to understand how product managers design, plan, and build products that support ML. To achieve this understanding, we interviewed 15 product development experts from various technology companies. Among the 15 companies represented, the market value of 14 companies exceeds 10 billion, 11 is publicly listed, 6 is B2C, and 9 is B2B.

Product Manager guides the ML series:

Before starting machine learning

OndetermineAfter machine learning (ML) may help solve the problem, it is important to take some steps before developing the model with the team.

According to our research, most product teams follow a framework and a structured process for evaluating ML issues. In this article, we share the four most important of these studies.

Step 1: Writing Machine Learning Hypothesis

Sometimes, product teams have a general meaning to what they expect to achieve, and they can start their ML projects. The problem with this unstructured approach is that you will learn less and will not create a framework for future use cases.

If you don't have a certain assumption, you will also risk increasing the risk (especially for smaller data sets) and find a random feature associated with your target variable. This correlation may lead to the erroneous belief that random features are relevant in your model, which may lead to a generalization of the model in production.

A reasonable assumption will include all of the following:

  1. Change what you are testing
  2. Expected result (high level)
  3. Success indicator
  4. Model output (numbers, tags, clusters, etc.)
  5. aims
  6. Prediction factor (advanced)

Example hypothesis

Improve search rankings in Dropbox with ML[1. CHANGE]Will allow users to find the correct file[2. result]Reduce 15% time[3. METRIC]. The model will score every possible file[4. MODEL OUTPUT]Use files recently shared with users and recently viewed files[6. PREDICTORS]Predict the file that the user ultimately selects[5. TARGET].

note:Be careful if you skip the hypothetical steps. We interviewed some teams that did not establish entity assumptions in advance. Their assumptions are often: "X competitors have already done this. Why can't we?" Even if your competitors have already done so, we still recommend using the above framework to create a simple assumption.

Step 2: What data do I need?

Photograph by Franki Chamaki

In general, your data needs to be informative and relevant to the problem you are trying to solve, but for ML, the data needs to be rich. A simple rule of thumb is to provide at least thousands of rows of data for a linear model and hundreds of thousands of rows of data for a neural network. If you don't have data, consider ways to get the right data or stick to a heuristic-based approach.

Assuming there is enough data, the data still has to have some patterns. The algorithm learns from these modes. You don't have to know the exact pattern before you start, but you should be able to express it qualitatively or have an intuition.

In the Dropbox search example, the product team may intuitively feel the file sharing associated with the file that the user ultimately selected. However, the team does not need to know the exact relationship or pattern before starting the project.

Finally, keep in mind that even if the amount of data is large, not all problems can be solved. As one of the social media primes we interviewed said: “It’s not the same thing to have data and get value from it.”

key point

  • The data needs to help answer your hypothesis and have some hypothetical patterns.
  • Even with the right data, you may end up with a working model.

Step 3: Start with heuristics

ML models require a lot of data, are complex, and can take a long time to develop and test before they are ready for production. Therefore, it is usually best to start with a simple set of heuristics before trying ML. Heuristic rules provide a shortcut to solving problems.

Heuristic builds are fast, simple to implement, and easy to understand. These rules can also be used as a shortcut to testing your hypothesis without spending a lot of time refining the model. To some extent, heuristic algorithms can act as prototyping tools before building a full-featured ML model.

When planning a simple news source, if the expected result of your hypothesis is to increase user engagement, then this can be achieved through heuristics that express the user's basic content. For example, any post that exceeds five in an hour will appear on the news feed. Of course, this is not a very accurate proxy for virality, so you may want to include additional heuristics in the number of annotations or shares.

In many cases, a project may stop a simple set of heuristics (no ML required). This is usually the case if the data is not sufficient for ML, or if the heuristic is simple, good, and easy to maintain. Many of the companies we interviewed began to consider ML after their heuristics were too complex. Complications can occur when you try to build more and more interdependent, overlapping and personalized heuristics.

key point

  • Heuristic scanning can be a good shortcut to testing your hypothesis.
  • If your heuristics become too complex to maintain or performance lag, consider ML.

Step 4: Risk Management Strategy

Image recognition algorithm marks inappropriate thingsToChat robot learning becomes culturally inappropriateThere are many examples of ML model errors. Many of the companies we interviewed are keenly aware of these issues and they use risk management techniques to help mitigate (rather than solve) these problems.


Many companies maintain words, phrases, groups, or organizations that may be considered bad actorsblacklist.

If you start typing "What fu" in Google search, the autocomplete result might be: "What is Funk! Band," "What is jelly, and what is the future." There are no four letters in this ML model. word. Google won't be in the companyAutofill policyFill in words or phrases that are considered inappropriate. Another example isPinterestProduce news by blacklisting anti-vaccination related posts in the search.

Prejudice in machine learning

dataCollected and explainedThe process is usually biased. Many companies use internal data to train their initial models. This method is convenient, but it has problems.

For example, in an expense report, the receipts collected by your company's personnel may not represent receipts collected by your customers worldwide. This data collection bias can reduce the generalization capabilities of your model and bias your company like yours.

Excluding variables related to prejudice is not the answer. Many product teams agree that it is best to exclude race, gender, or other variables from the model.AmazonTried this through its automated resume screening ML model, which resulted in the algorithm being biased towards women.In many data sets, there are other racial or gender proxies. "With sufficient data, class membership will inevitably be coded into other functions."-  

Professor Moritz Hardt, University of California, Berkeley

There is no magic cure for this prejudice. First, add features such as class members (gender, race, etc.) to the model to help you measure the bias that does exist. Once the deviation is measured, your solution may depend on the use case. When retraining the model, using mitigation techniques, offsetting deviations after training, and even changing the user interface may make sense.

key point

  • Consider the risks of your ML use cases and propose policies to address them.
  • Blacklists can help serve as a guardrail for ML models.
  • Please pay attention to whichType deviationMay affect your model.
  • To mitigate deviations in the model, allow the model to measure deviations and then takeMeasures to offset deviations.

This article is transferred from medium,Original address