AutoML Natural Language Beginner's guide

Link: => stopamswanros.nnmcloud.ru/d?s=YToyOntzOjc6InJlZmVyZXIiO3M6MzY6Imh0dHA6Ly9iYW5kY2FtcC5jb21fZG93bmxvYWRfcG9zdGVyLyI7czozOiJrZXkiO3M6MTk6Ikdvb2dsZSBjbG91ZCBhdXRvbWwiO30=


The performance of your model on the test set is intended to give you a pretty good idea of how your model will perform on real-world data. You may unsubscribe from these newsletters at any time. Should I optimize for precision or recall? This can help you find a good balance between false positives and false negatives.

To train a neural network, a developer provides the software with a detailed set of instructions describing the appropriate model to train, then trains it to recognize patterns within the dataset in a time-consuming, laborious process. All of this provides some common machine learning metrics to evaluate your model accuracy and see where you can improve your training data. So if the largest label has 10,000 examples, the smallest label should have at least 1,000 examples.

Hands

This product is not intended for real-time usage in critical applications. Introduction Imagine your business has a contact form on its website. Every day you get tons of messages from the form, many of which are actionable in some way, but they all come in together and it's easy to fall behind on dealing google cloud automl them since different employees handle different message types. It would be great if an automated system could categorize them so the right person sees the right comments. You need some sort of system to look at the comments and decide if they represent complaints, praise for past service, an attempt to learn more about your business, schedule an appointment, or establish a relationship. Classical programming requires the programmer to specify step-by-step instructions for the computer to follow. But this approach quickly gets infeasible. Customer comments use a broad and varied vocabulary and structure — too diverse to be captured by a simple set of rules. If you tried to build manual filters, you'd quickly find that you weren't able to categorize the vast majority of your customer comments. You need a system that can generalize to a wide variety of comments. In a scenario where a sequence of specific rules is bound to expand exponentially, you need a system that can learn from examples. Fortunately, machine learning systems are well-positioned to solve this problem. Machine learning involves using data to train algorithms to achieve a desired outcome. The specifics of the algorithm and training methods change based on the use case. There are many different subcategories of machine learning, all of which solve different problems and work within different constraints. Using supervised learning, we can train a custom model to recognize content that we care about in text. Assess your use case While putting together the dataset, always start with the use case. As you move through the guidelines for putting together your dataset, we encourage you to consider fairness in machine learning where relevant to your use case. If so, about assessing your use case for fairness considerations. Source your data Once you've established what data you will need, you need to find a way to source it. You can begin by taking into account all the data your organization collects. You may find that you're already google cloud automl the data you would need to train a model. In case you don't have the data you need, you can obtain it manually or outsource it to a third-party provider. Fair-aware: Review regulations in both your region and the locations your application will serve, as well as existing research or product information in your domain to learn about legal guidelines and common issues. The likelihood of successfully recognizing a label goes up with the number of high quality examples for each; in general, the more labeled data you can bring to the training process, the better your model will be. Target at least 1000 examples per label. Distribute examples equally across categories It's important to capture a roughly similar number of training examples for each category. Even if you have an abundance of data for one label, it is best to have an equal distribution for each label. To see why, imagine that 80% of the customer comments you use to build your model are estimate requests. With such an unbalanced distribution of labels, your model is very likely to learn that it's safe to always tell you a customer comment is an estimate request, rather than going out on a limb to try to predict a much less common label. We understand it may not always be possible to source an approximately equal number of examples for each label. High quality, unbiased examples for some categories may be harder to source. In those circumstances, you can follow this rule of thumb - the label with the lowest number of examples should have at least 10% of the examples as the label with the highest number of examples. So if the largest label has 10,000 examples, the smallest label should have at least 1,000 examples. Capture the variation in your problem space For similar reasons, try to ensure that your data captures the variety and diversity of your problem space. When you provide a broader set of examples, the model will be better able to generalize to new data. Say you're trying to classify articles about consumer electronics into topics. The more brand names and technical specifications you provide, the easier it will be for the model to figure out the topic of an article — even if that article is about a brand that didn't make it into the training set at all. Match data to the intended output for your model Find text examples that are similar to what you're planning to make predictions on. If you are trying to classify social media posts about glassblowing, you probably won't get great performance from a model trained on glassblowing information websites, as the vocabulary and style may be very different. Ideally, your training examples are real-world data drawn from the same dataset you're planning to use the model to classify. Training Set The vast majority of your data should be in the training set. After the model learning framework incorporates training data during each iteration of the training process, it uses the model's performance on the validation set to tune the model's hyperparameters, which are variables that specify the model's structure. If you tried to use the training set to tune the hyperparameters, it's quite likely the model would end up overly focused on your training data, and have a hard time generalizing to examples that don't exactly match it. Using a somewhat novel dataset to fine-tune model structure means your model will generalize better. Test Set The test set is not involved in the training process at all. Once the model has completed its training entirely, we use the test set as an entirely new challenge for your model. The performance of your model on the test set is intended to give you a pretty good idea of how your model will perform on real-world data. Manual Splitting You can also split your dataset yourself. Manually splitting your data is a good choice when you want to exercise more control over the process or if there are specific examples that you're sure you want included in a certain part of your model google cloud automl lifecycle. Evaluate Once your model is trained, you will receive a summary of your model performance. What should I keep in mind before evaluating my model. Debugging a model is more about debugging the data than the model itself. If your model starts acting in an unexpected manner as you're evaluating its performance before and after pushing to production, you should return and check your data to see where it might be improved. In this section, we will cover what each of these concepts mean. For each example, the model outputs a series of numbers that communicate how strongly it associates each label with that example. If the number is high, the model has high confidence that the label should be applied to that document. What is the Score Threshold. The score threshold refers to the level of confidence the model must have to assign a category to a test item. In the example above, if we set the score threshold to 0. If your score threshold is high, your model will classify fewer text items, but it will have google cloud automl lower risk of misclassifying text items. However, when using your model in production, you will have to enforce the thresholds you found optimal on your side. What are True Positives, True Negatives, False Positives, False Negatives. After applying the score threshold, the predictions made by your model will fall in one of the following four categories. You can use these categories to calculate precision and recall — metrics that help us gauge the effectiveness of your model. What are precision and recall. Precision and recall help us understand how well our model is capturing information, and how much it's leaving out. Precision tells us, from all the test examples that were assigned a label, how many actually were supposed to be categorized with that label. Recall tells us, from all the test examples that should have had the label assigned, how many were actually assigned the label. Should I optimize for precision or recall. Depending on your use case, you may want to optimize for either precision or recall. Let's examine how you might approach this decision with the following two use cases. Use case: Urgent documents Let's say you want to create a system that can prioritize documents that are urgent from ones that are not. A false positive in this case would be a document that is not urgent, but gets marked as such. The user can dismiss them as non urgent and move on. A false negative in this case would be, a document that is urgent, but the system fails to flag it as such. In this case, you would want to optimize for recall. This metric measures, for all the predictions made, how much is being left out. A high-recall model is likely to label marginally relevant examples, which is useful for cases where your category has scarce training data. Use case: Spam filtering Let's say you want to create a system that automatically filters email messages that are spam from messages that are not. A false negative in this case would be a spam email that does not get caught, and you see them in your inbox. Usually just a bit annoying. A false positive in this case would be an email that is falsely flagged as spam and gets removed from your inbox. If it were an important email, the user may have been adversely impacted. In this case, you would want to optimize for precision. This metric measures, for all the predictions made, how correct they google cloud automl. A high-precision model is likely to label only the most relevant examples, which is useful for cases where your category is common in the training data. Fair-aware: Evaluating your model for fairness requires understanding the impact of different types of errors - false positives and false negatives - for different user demographics. How google cloud automl I use the Confusion Matrix. We can compare the model's performance on each label using a confusion matrix. In an ideal model, all the values on the diagonal will be high, and all the other values will be low. This shows that the desired categories are being identified correctly. If any other values are high, it gives us a clue into how the model is misclassifying test items. How do I interpret the Precision-Recall curves. The score threshold tool allows you to explore how your chosen score threshold affects your precision and recall. As you drag the slider on the score threshold bar, you can see where that threshold places you on the precision-recall tradeoff curve, as well as how that threshold affects your precision and recall individually for multiclass models, on these graphs, precision and recall means the only label used to calculate precision and recall metrics is the google cloud automl label in the set of labels we return. This can help you find a good balance between false positives and false negatives. Once you've chosen a threshold that seems to be acceptable for your model on the whole, you can click individual labels and see where that threshold falls on their per-label precision-recall curve. In some cases, it might mean you get a lot of incorrect predictions for a few labels, which might help you decide to choose a per-class threshold that's customized to those labels. For example, let's say you look at your customer comments dataset and notice that a threshold at 0. For that category, you see tons of false positives. In that case, you might decide to use a threshold of 0. Google cloud automl useful metric for model accuracy is the area under the precision-recall curve. It measures how well your model performs across all score thresholds. But just in case you want to sanity check your model, there are a few ways to do it. Hopefully, this matches your expectations. Try a few examples of each type of comment you expect to receive. Fair-aware: Think carefully about your problem domain and its potential for unfairness and bias. Come up with cases that would adversely impact your users if they were found in production, and test those first. Fair-aware: If you have a use case that warrants fairness considerations, read more about how to use your model in a manner that mitigates biases or adverse outcomes. Except as otherwise noted, the content of this page is licensed under theand code samples are licensed under the. Last updated November 15, 2018.

In an ideal model, all the values on the diagonal will be high, and all the other values will be low. Jia Li said Google has benchmarked its methods compared to existing publicly known techniques to conclude that its models are better and easier to produce. A neural network that could train another neural network would take a significant burden off of researchers and engineers. Except as otherwise noted, the content of this page is licensed under the , and code samples are licensed under the. From 2014-2015 I was a Council Member of the World Economic Forum's Global Agenda Council on the Future of Government. These annotations are being integrated into our search engine to enhance the impact on Guest experience through more relevant search results, expedited discovery and product recommendations on shopDisney. Today, any company on earth can simply outsource all of its deep learning needs to Google, uploading a few dozen examples and letting Google handle the annotation, model construction and tuning.